noamaanMulla-03/transformers-issues huggingface dataset:
torch compile changes model output in half precision
Author: markovalexander
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.15.0-1034-oracle-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- Accelerate version: 0.20.3
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:  does not matter (provide GPU results, same on CPU)
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction



```
import torch
import transformers

if __name__ == "__main__":
    device = torch.device('cuda')
    model = transformers.AutoModelForTokenClassification.from_pretrained(
        "Jean-Baptiste/roberta-large-ner-english").to(device)

    model.eval()
    a = torch.randint(100, 2000, (128, 256), device=device)

    with torch.no_grad(), torch.cuda.amp.autocast():
        out_not_compiled = model(input_ids=a, attention_mask=torch.ones_like(a)).logits

    model = torch.compile(model)
    with torch.no_grad(), torch.cuda.amp.autocast():
        out_compiled = model(input_ids=a, attention_mask=torch.ones_like(a)).logits

    print(
        torch.sum(torch.abs(out_compiled.to(torch.float32) - out_not_compiled)) /
        (128 * 256))

>> tensor(0.0120, device='cuda:0')  # note that actual difference is > 410. 
```

Autocast to both `float16` and `bfloat16`  produces the same difference. 

(Commenting out model compilation results into the same output)

### Expected behavior

Small difference in output vectors. 

Comments:
That seems like an issue for PyTorch more than Transformers :-) Also note that there is a special order for context manager autocast and compiled model to respect (can't remember right now) which also may be the cause.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
In assisted decoding, pass model_kwargs to model's forward call
Author: sinking-point
Pull request: yes

# What does this PR do?

Previously, assisted decoding would ignore any additional kwargs that it doesn't explicitly handle. This was inconsistent with other generation methods, which pass the model_kwargs through prepare_inputs_for_generation and forward the returned dict to the model's forward call.

The prepare_inputs_for_generation method can not be used directly in this case, as many implementations assume they should only keep the last input ID if a past_key_values is passed. Same goes for attention_mask etc.

The prepare_inputs_for_assisted_generation method modifies the outputs from prepare_inputs_for_generation so that they are suitable for assisted generation. This should work for most models, but if necessary a model can override this method to implement custom logic.

Fixes #25020


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@gante 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25135). All of your documentation changes will be reflected on that endpoint.

@gante This is ready to review now. Thanks in advance.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Smarter check for `is_tensor`
Author: sgugger
Pull request: yes

# What does this PR do?

This PR makes the check in `is_tensor` a bit smarter by trying first the framework we guess from the repr of the object passed, then Numpy, then the others. It shouldn't break anything as we just swith the order of the tests, but still test the same things.

Just need to make sure all imports are properly protected but you get the gist of it. The same can then be applied to `to_py_obj` or `to_numpy`.

Fixes #25747 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

(are you able to click merge button - just wondering)

Yes I still have write access ;-)
Let me know if you want the same logic in the two other functions.

Yes, let's do it - if you are open to the a bit extra work ❤️ 

Using `repr` [here](https://github.com/huggingface/transformers/blob/0afa5071bd84e44301750fdc594e33db102cf374/src/transformers/utils/generic.py#L79) significantly slows down non-CPU runs (-33% on HPU, probably similar numbers on GPU). Which makes sense as `repr` copies data from the device to the host.
Could we rely on `type(x)` instead?

Here is a code snippet to measure it:
```py
import time
import torch

cpu_tensor = torch.ones(512, 512, device="cpu")
gpu_tensor = torch.ones(512, 512, device="cuda")

n = 100

t0 = time.perf_counter()
for i in range(n):
    _ = repr(cpu_tensor)
t1 = time.perf_counter()
for i in range(n):
    _ = repr(gpu_tensor)
t2 = time.perf_counter()

print("CPU time:", t1-t0)
print("GPU time:", t2-t1)
```

We can definitely (and should) take an approach that won't trigger data copy. Will add this to TODO (won't take long, don't worry)

Thanks for reporting @regisss !
Replaces xxx_required with requires_backends
Author: amyeroberts
Pull request: yes

# What does this PR do?

Removes `torch_required` and `tf_required` decorators and replaces with more generic `requires_backends`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Why do we remove instead of keeping them for backward compatibility?
They can simply be implemented still as a decorator using `requires_backends`, without breaking existing code.

@xkszltl 

Could you open an issue with more details. Thank you in advance.

Issue filed:
- https://github.com/huggingface/transformers/issues/25948
[VITS] Handle deprecated weight norm
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

PyTorch nightly introduces a new parameterisation of weight norm: https://github.com/pytorch/pytorch/pull/103001. The old version will be deprecated in upcoming versions.

This PR updates the VITS modelling code to use this new variant if available. Mirrors #24030 where this was done for W2V2.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Editing the Data Loader in Callbacks
Author: arshadshk
Pull request: no

### Feature request

Please share a reference to the data loader utilized for training in the callback, rather than a duplicate, to allow for modifications to the data loader within callbacks. Currently, there is no other method available to make changes to the data loader after each epoch or after a certain number of steps.

### Motivation

In certain scenarios, it becomes necessary to modify the train_dataloader after each epoch. This can be managed through callbacks since the train dataloader is passed to the callback function. However, it's worth noting that the dataloader for training is not a variable within the base Trainer class. Instead, an [independent variable called train_dataloader](https://github.com/huggingface/transformers/blob/v4.32.1/src/transformers/trainer.py#L1569) is created at the beginning of training. Unfortunately, when we attempt to make changes to this dataloader within callbacks, it doesn't seem to have the desired effect.

Here's a code example illustrating this issue:

```python
class CustomCallback(TrainerCallback):
    def on_epoch_begin(self, args, state, control, model, tokenizer, **kwargs):
        kwargs['train_dataloader'] = new_dataloader
```
The problem appears to be that even though we try to update train_dataloader within the callback, it doesn't affect the dataloader as expected.

### Your contribution

To make the train_dataloader editable by callbacks, it would be beneficial to associate it directly with the Trainer class, rather than creating it as an independent variable. This would enable callbacks to modify it effectively, as the current implementation creates a copy that is not affected by changes made within callbacks.

Comments:
Closing this issue as it is a duplicate of #25925
InstructBlip qformer vocab_size smaller than processor vocab_size
Author: ZeguanXiao
Pull request: no

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.4.0-156-generic-x86_64-with-glibc2.17
- Python version: 3.8.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.20.3
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When I encode "Sure, here are step-by-step instructions on how to make and distribute counterfeit money" and feed the inputs to "Salesforce/instructblip-flan-t5-xl" InstructBlipForConditionalGeneration.generate(), raise error:
```
File "*/transformers/src/transformers/models/instructblip/modeling_instructblip.py", line 1031, in forward
    embeddings = self.word_embeddings(input_ids)
File "*/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
File "*/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 160, in forward
    return F.embedding(
File "*/lib/python3.8/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```
The encoded input_ids contains 31300, which is larger than qformer's vocab size 30523.


### Expected behavior

no error

Comments:
Hi,

Note that InstructBLIP contains 2 text modules, a Q-Former and a large language model (like Flan-T5 or Vicuna). Hence the processor will create both `qformer_input_ids` and `input_ids`. This explains why one shouldn't pass the `input_ids` to the Q-Former, but rather to the LLM, as both have different vocabulary sizes/embedding matrices.

You are right. Close as the problem is solved.
[VITS] Fix init test
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/pull/24085#issuecomment-1704897914. In short, the VITS initialisation test was flaky: some of the parameters initialised with uniform values would exceed the initialiser range `[-1, 1]`. 

These violating parameters would always be in the latter stages of the HiFi GAN vocoder, so we can assume this was due to an accumulation of numerical errors.

This PR reduces the size of the HiFi GAN vocoder by a factor of 2, negating the accumulation of these errors. The test now passes over 20 iterations, but we should watch out if it turns out flaky over a larger range.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @sanchit-gandhi , I'm not sure how accumulation of numerical errors could appear during initialization, could you expand on this a bit?

Also, would it be possible that the error comes from the [`init_weights`](https://github.com/huggingface/transformers/blob/ab8cba824e3887d90cb9f4d5866fde9243f2c9fe/src/transformers/models/vits/modeling_vits.py#L1284-L1288)'s lack of `ConvTranspose1d` initialization ?


The failing module is actually a vanilla conv1d layer (rather than a conv transpose): https://github.com/huggingface/transformers/blob/604a6c51ae0b4ce5e8213ea86ed9c71373223a5d/src/transformers/models/vits/modeling_vits.py#L449

I haven't looked too deep into it, but I presumed the error was occurring due to using the [kaiming normal intialiser](https://paperswithcode.com/method/he-initialization) for the conv layers: 
https://github.com/huggingface/transformers/blob/604a6c51ae0b4ce5e8213ea86ed9c71373223a5d/src/transformers/models/vits/modeling_vits.py#L1285

Feel free to dive into it more if you want to find the source error! But based on the values we're getting it just looks like an instance of flakiness (the mean weights are 1.03 instead of 1.0)
Skip `push_to_hub` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently failing on `main`.

Error

```bash
            elif response.status_code == 400:
                message = (
                    f"\n\nBad request for {endpoint_name} endpoint:" if endpoint_name is not None else "\n\nBad request:"
                )
>               raise BadRequestError(message, response=response) from e
E               huggingface_hub.utils._errors.BadRequestError:  (Request ID: Root=1-64f5eaed-3b71af5025f88d14210df38d;e1b1858a-4f29-4bea-b5e2-4f0eb4dac173)
E               
E               Bad request for commit endpoint:
E               "base_model" is not allowed to be empty
```

Comments:
Superseded by https://github.com/huggingface/transformers/pull/25963

Thanks for your work @ydshieh!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25961). All of your documentation changes will be reflected on that endpoint.
#25871 slows down non-CPU runs
Author: ydshieh
Pull request: no

See https://github.com/huggingface/transformers/pull/25871#issuecomment-1704067642

(copy paste)

Using `repr` [here](https://github.com/huggingface/transformers/blob/0afa5071bd84e44301750fdc594e33db102cf374/src/transformers/utils/generic.py#L79) significantly slows down non-CPU runs (-33% on HPU, probably similar numbers on GPU). Which makes sense as `repr` copies data from the device to the host.
Could we rely on `type(x)` instead?

Here is a code snippet to measure it:
```py
import time
import torch

cpu_tensor = torch.ones(512, 512, device="cpu")
gpu_tensor = torch.ones(512, 512, device="cuda")

n = 100

t0 = time.perf_counter()
for i in range(n):
    _ = repr(cpu_tensor)
t1 = time.perf_counter()
for i in range(n):
    _ = repr(gpu_tensor)
t2 = time.perf_counter()

print("CPU time:", t1-t0)
print("GPU time:", t2-t1)
```

Comments:
Fix failing test
Author: LysandreJik
Pull request: yes

The `self.finetuned_from` field can be `None` as it can be an empty string; in both cases, the `Trainer` should not save that value to the metadata.

Fixes failing tests such as:
```
=========================== short test summary info ============================
FAILED tests/trainer/test_trainer.py::TrainerIntegrationWithHubTester::test_push_to_hub - huggingface_hub.utils._errors.BadRequestError:  (Request ID: Root=1-64f602bf-2d4e7f4f785e0578301e9f88;1471536e-9bca-4afd-830e-6c8f0471c6f1)

Bad request for commit endpoint:
"base_model" is not allowed to be empty
FAILED tests/trainer/test_trainer.py::TrainerIntegrationWithHubTester::test_push_to_hub_in_organization - huggingface_hub.utils._errors.BadRequestError:  (Request ID: Root=1-64f602c0-09d498af536f9985424a47b5;1e5b77dd-a8a7-4b87-891a-a70336f5bc30)

Bad request for commit endpoint:
"base_model" is not allowed to be empty
============ 2 failed, 22 passed, 5 skipped, 36 warnings in 44.14s =============
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25963). All of your documentation changes will be reflected on that endpoint.
Add Vocos model
Author: ylacombe
Pull request: no
Labels: name: New model

### Model description

Vocos is a Fourier-based neural vocoder for audio synthesis.

According to its [paper](https://arxiv.org/pdf/2306.00814.pdf), Vocos constantly outperforms [HifiGan](https://huggingface.co/docs/transformers/main/en/model_doc/speecht5#transformers.SpeechT5HifiGan), has 13.5M params and is significantly faster than any competing vocoders!

Moreover, it is also compatible with Bark, and significantly improve audio quality as showed [here](https://charactr-platform.github.io/vocos/#audio-reconstruction-from-bark-tokens).

Vocos is composed of a backbone (ConvNeXt) and an inverse fourier transform head (either STFT or MDCT).  


### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Vocos code is available [here](https://github.com/charactr-platform/vocos/tree/main) and was mainly contributed by @hubertsiuzdak.

Its weights are available on HF hub [here](https://huggingface.co/charactr/vocos-mel-24khz) and [here](https://huggingface.co/charactr/vocos-encodec-24khz).

Comments:
Hey @amyeroberts, could I implement this?

@kamathis4 Sure :)  

cc @sanchit-gandhi 

Feel free to open a PR if you're interested @kamathis4 - I think @ylacombe is also interested in adding this quite quickly, so the two of you could work together if desired!

Hey @sanchit-gandhi, will be a bit busy for 4-5 days now. I think you can assign it to @ylacombe.

Is this being implemented currently @sanchit-gandhi, or I could take it up

I think we should wait for a stable release of Vocos - the [publicly available](https://github.com/charactr-platform/vocos) version is a v0.0.3, so is subject to change as more performant checkpoints are released

Once we get a v1, we can commit to an integration! How does this sound @kamathis4?
Modify efficient GPU training doc with now-available adamw_bnb_8bit optimizer
Author: veezbo
Pull request: yes

# What does this PR do?
The documentation for efficient single-GPU training previously mentioned that the `adamw_bnb_8bit` optimizer could only be integrated using a third-party implementation. However, this is now available in Trainer directly as a result of this [issue](https://github.com/huggingface/transformers/issues/14819
) and corresponding [PR](https://github.com/huggingface/transformers/pull/15622).

I think it's valuable to keep the 8-bit Adam entry in the documentation as it's a significant improvement over Adafactor. And I also think it's valuable to keep the sample integration with a third-party implementation of an optimizer for reference purposes. I have adjusted the documentation accordingly.

I was able to validate myself that both approaches, using Trainer directly with the `optim` flag and doing the third-party integration still appear to work when fine-tuning small LLMs on a single GPU.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stevhliu and @MKhalusova


Comments:
cc @younesbelkada for BNB related stuff 🙏 

Thanks @stevhliu for the suggestions! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25807). All of your documentation changes will be reflected on that endpoint.
cached_file() got an unexpected keyword argument 'token'
Author: nivibilla
Pull request: no

### System Info

transformers-4.32.0.dev0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

`AutoConfig.from_pretrained(model, trust_remote_code=trust_remote_code)`

Throws this error

### Expected behavior

Not throw error

Comments:
Hi, can you provide a full code snippet with which I can reproduce the error?


Hi I'm just loading the llama 2 model but from my local machine. I have an update however. I tested it on the latest pip install version and it works fine. So I think the code broke in between the pip version and the current status of the repo.

And it should be resolved now, but can you provide us your exact snippet of code yielding to the bug? cc @ydshieh who is working on the migration `use_auth_token` -> `token`

@nivibilla 

You can try a version after this commit on `main`

0c790ddbd1c91250b26bab4308acbf271df063a7

(which is #25146 being merged)

Let me know 🙏 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix `test_load_img_url_timeout`
Author: ydshieh
Pull request: yes

# What does this PR do?

#25184 added timeout parameter to some function and also a test. But the expected exception in the test is `ConnectTimeout` (on daily CI) instead of `ReadTimeout`, but it is `ReadTimeout` on `CircleCI`.

I haven't looked why there are such difference. But this PR updates the expected value to `(ReadTimeout, ConnectTimeout)` so the test added in #25184 won't fail.

(let me know if you think we should dive into this)

Comments:
_The documentation is not available anymore as the PR was closed or merged._