noamaanMulla-03/transformers-issues huggingface dataset:
Add back in generation types
Author: amyeroberts
Pull request: yes

# What does this PR do?

#28494 removed some custom types in the `generation.utils` module. This has caused downstream issues in other libraries, notably Coqui-TTS c.f #28649

This PR adds them back in so they're still importable. 

cc @gante for reference when you're back 

Comments:
@amyeroberts uhmmm, I didn't think anyone would be importing these auxiliary types 🤔 

How can we deprecate a type variable? I have no idea how to throw a warning if a user uses it as a type :D  

@gante Hyrum's law strikes again! I didn't think they'd be used either, which is partly why I wasn't checking rigorously in the PR review. Normally, something like this would have minimal impact which would be easy to remedy on the user side, and removing would be OK. However in this case the issue had a big impact, with lots of activity in the issue, which is why we decided to add back in. 

I don't think there's any easy way to deprecate a type unfortunately. Or, at least I don't know and wasn't able to find one. 

@amyeroberts note to self: don't create auxiliary types ☠️
fix: suppress `GatedRepoError` to use cache file (fix #28558).
Author: scruel
Pull request: yes

# What does this PR do?
The repo will have some optional files missed, and we have `_raise_exceptions_for_missing_entries=False` to suppress such errors, however for gated repo we won't be able to know if files exists or not without passing the `token` parameter (or env variable), even we already fully downloaded the repo, we will still won't be able to use it.
For those optional files, we will suppress exceptions, and for those required, we will still get errors, this PR is for keeping this behavior same if `GatedRepoError` occurs.

Fixes #28558

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @ArthurZucker @ydshieh


Comments:
Yeah, I think we should also have the same processing logic for `RepositoryNotFoundError` exception, should I commit it in this PR?
I would also like to merge three private parameters into one parameter called `optional_file`, but consider I also not quite familiar with this repo, I decided not to do so, even IMO such change will be just fine.
Any feedback will be welcome!

Hi.

- confirmed this PR works with the example case (which fails without this PR) - i.e. for gated repositories
- and indeed, so far this PR won't work with private repositories (`RepositoryNotFoundError`)
- I was originally concerned that we will miss the precise information about a file couldn't be found on the Hub.
  - However, this PR only adds `_raise_exceptions_for_gated_repo` to the places where we already have `_raise_exceptions_for_xxx`, so this PR doesn't add extra concern I mentioned above (and if there is anything we  can improve, it could be done in a separate PR by us)

So it's ✅ for me.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28566). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: deprecate old src imports
Author: gante
Pull request: yes

# What does this PR do?

We have 3 thin wrappers for `generate`, one for each framework, whose sole purpose is to import the mixin from `src/transformers/generation(_flax/_tf)_utils.py`. In other words, to import from `src` according to the codebase before [this PR](https://github.com/huggingface/transformers/pull/20096).

Since this is a `src` import (and not a `from transformers import X`), I believe this can be safely removed before v5.

Comments:
Phi for sequence classfication gives nan 
Author: Alymostafa
Pull request: no

### System Info

I tried fine-tuning the Phi-1_5 model for a sequence classification task, but it gives ```nan``` in the validation loss and ```no log``` in the training loss. I tried fp16 & float 32, and both have the same output. 
@younesbelkada 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint,torch_dtype=torch.float32, device_map="cuda", num_labels=1, trust_remote_code=True)

args = TrainingArguments(
    f"{model_name}-finetuned-{task}",
    evaluation_strategy = "steps",
    eval_steps=100,
    save_strategy = "steps",
    learning_rate=2e-5,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_train_epochs=6,
    fp16=True,
    load_best_model_at_end=True,
    metric_for_best_model=metric_name,
)


trainer = Trainer(
    model,
    args,
    train_dataset=encoded_dataset["train"],
    eval_dataset=encoded_dataset['validation'],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics_for_regression
)

trainer.train()

### Expected behavior

loss as a numbers not nan or no logging

Comments:
Hi @Alymostafa 
I suspect there might be something off with the trust_remote_code version of phi_1-5, can you try with the official transformers version which is pushed here: https://huggingface.co/susnato/phi-1_5_dev ?
Can you also try to train with bf16 mixed precision? i.e. you need to load the model in fp32 and pass `bf16=True` in `TrainingArguments`

> Hi @Alymostafa
> I suspect there might be something off with the trust_remote_code version of phi_1-5, can you try with the official transformers version which is pushed here: https://huggingface.co/susnato/phi-1_5_dev ?
> Can you also try to train with bf16 mixed precision? i.e. you need to load the model in fp32 and pass `bf16=True` in `TrainingArguments`

Hello, may I ask: is it necessary to load model in fp32 for amp?

I think it would work for both scenoarios (loading it in fp32 or fp16) but is preferable to load it in fp32

> I think it would work for both scenoarios (loading it in fp32 or fp16) but is preferable to load it in fp32

Thanks.👍

@younesbelkada Thanks, it works.
 Is it possible to add a note on the Microsoft repo page to use the model you mentioned? That would be useful for other people.
What is the minimum video card with large memory required to run the mixtral-8x7b model
Author: zysNLP
Pull request: no

I mean the model that just came out：mistralai/Mixtral-8x7B-Instruct-v0.1，looks like a lot of parameter files，what is the minimum nvidia graphics card video memory required?

Comments:
Hi @zysNLP, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
In the RoPE paper they don't compute actual softmax
Author: wendlerc
Pull request: no

Is this a feature or a bug?

https://github.com/huggingface/transformers/blob/03cc17775b961d16cc4d0d7ab0c8487120d0b708/src/transformers/models/llama/modeling_llama.py#L429C9-L429C22

In RoPE paper equation 19: https://arxiv.org/pdf/2104.09864.pdf they don't compute the actual softmax.

Best,
Chris

Comments:
Ok I guess that's because they implement a linear attention.
[`Siglip`] protect from imports if sentencepiece not installed
Author: amyeroberts
Pull request: yes

# What does this PR do?

Complement to #28636.

Mistake on my part - when testing importing from top level `from transformers import *`, environment was running on different python / site packages that originally thought (sentence piece was installed) covering up these requirements. 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28737). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Seq2SeqTrainingArguments.__init__() got an unexpected keyword argument 'save_only_model'
Author: nic-olo
Pull request: no

### System Info

![image](https://github.com/huggingface/transformers/assets/89006260/a6c4672e-4d3c-460a-9407-2b3de3301d73)


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

    training_args = Seq2SeqTrainingArguments(
        output_dir=DIR,

        # Parameters
        per_device_train_batch_size=hyperparameters["batch_size"],
        per_device_eval_batch_size=hyperparameters["batch_size"],
        learning_rate=hyperparameters["learning_rate"],
        weight_decay=hyperparameters["weight_decay"],
        num_train_epochs=hyperparameters["nb_epochs"],
        fp16=False,
        optim="adamw_torch",

        # Logging
        logging_dir=f"{DIR}/training_logs",
        logging_strategy="epoch",
        # report_to=["wandb", "tensorboard"],
        report_to=["tensorboard"],
        # Saving
        save_strategy="epoch",
        # Evaluating
        evaluation_strategy="epoch",
        predict_with_generate=True,
        generation_max_length=550,
        generation_num_beams=3,
        save_safetensors=True,
        save_total_limit=1,
        # metric_for_best_model='eval_loss',
        load_best_model_at_end=True,
        save_only_model=True,
        # metric_for_best_model="Weighted_comb",
        # greater_is_better=True,
    )


### Expected behavior

Comments:
Fix `DepthEstimationPipeline`'s docstring
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #28729

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28733). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @Wauplin Will update it (along with `top_k` too maybe!)

FYI:

Also changed

> Assign labels to the image(s) passed as inputs.

to 

> Predict the depth(s) of the image(s) passed as inputs.

and removed `top_k` part

https://github.com/huggingface/transformers/pull/28733/commits/22ff25c6cec0eba56ea78646c3b88ec9f728e86a
Depth Estimation Pipeline docstrings are wrong
Author: osanseviero
Pull request: no

The docstring is a paste from the image classification pipeline it seems https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/depth_estimation.py#L54-L83 . The correct output, as per https://huggingface.co/docs/transformers/main/tasks/monocular_depth_estimation, should be a dictionary with an image and a tensor

cc @amyeroberts @ydshieh 

Comments:
Thanks @osanseviero ! Indeed, will open a PR to fix this
[Docs] Fix Typo in English & Japanese CLIP Model Documentation (TMBD -> TMDB)
Author: Vinyzu
Pull request: yes

Fixes Typo in TMBD to TMDB (for "TheMovieDatabase")


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

[/] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
[/] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
[/] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
[/] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28751). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Enable Gradient Checkpointing in Deformable DETR
Author: FoamoftheSea
Pull request: yes

# What does this PR do?

Gradient Checkpointing is not currently supported by Deformable DETR, but with slight modifications I was able to get it working in both the encoder and decoder stages, which both independently led to noticeable reductions in VRAM usage during training. This makes a default Deformable DETR configuration trainable on a 4GB GPU.

@amyeroberts 

Comments:
Note that fix-copies made changes to DETA model because it has the "# copied from" comments over methods taken from Deformable DETR. If we don't want to change that model, we could instead remove those comments and revert the changes to that file.

Hi @FoamoftheSea, thanks for working on this!

There's another open PR which addresses enabling gradient checkpointing for DETA -- #28615. What I'd suggest is removing the relevant `# Copied from` headers in DETA and the changes in its modeling file. 

This way we can include both of your respective contributions without clashes.

@amyeroberts All done!

No problem! 😀

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28686). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Models with a sentencepiece tokenizers have problems with special tokens and encode decode
Author: ekgren
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1vujbKaRkIpk7qli7eUKAZQDRksHSRW51?usp=sharing

### Expected behavior

Huggingface tokenizers with sentencepiece in the back have inconsistent encoding decoding behaviour. If you encode and decode a string with special characters white spaces are inserted.

Expected behaviour would be to get the exact same string back. 

This is both present with the Llama2 tokenizer, the gpt-sw3 tokenizers and more

Comments:
#26678 fixed this, I can't push everything to the hub but Llama tokenizer will have a fix soon. 
This is a duplicate of #26455

Thank you for all the hard work @ArthurZucker, closing this issue then!
Trainer stuck at step 19(out of 100000) in Jupyter notebook
Author: sr5434
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): 2.13.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@pacman100 @muellerz 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] My own task or dataset (give details below)
- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)

### Reproduction

Run this code: https://github.com/sr5434/CodegebraGPT/blob/main/Train_CodegebraGPT.ipynb
Run it on a Lambda Labs 1xA6000

### Expected behavior

It has been 2 hours, and the model started out pretty quickly. However, it has frozen up and appears to be stuck at 19 steps. The GPU and CPU are both at 100% utilization.

Comments:
It turns out the kernel was busy, but checkpoints were still being saved.
Fix windows err with checkpoint race conditions
Author: muellerzr
Pull request: yes

# What does this PR do?

Windows doesn't like python trying to open files, so makes the additional race condition check to be non-windows based

Fixes # (issue)

https://github.com/huggingface/transformers/pull/28364


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
@amyeroberts I'm not a windows expert but after looking the only alternative would to be to use the atomic `os.replace()` since it's atomic *except* it doesn't work on folders, so there's not really a good solution here for this. 

I don't think we have too many multi-node windows users (as this seems fairly uncommon, I'd expect more UNIX-based multinode users), so it's okay for now. If a user reports an issue hopefully more users will know a windows-based solution, or we can try a hacky-workaround using `replace()` and checking for a particular file that would exist in the folder. 

Let me know if you're comfortable with this, else I can include the hack in here

@muellerzr Sounds good to me! Thanks for looking into this 🤗 

Just FYI, this is not a race condition, the error is because you're trying to call os.open() on a directory (not a file), which doesn't work in Windows. See: https://stackoverflow.com/questions/21785127/open-a-directory-on-windows-permission-denied

@adamkolar Yep - this PR is addressing the permissions issue with Windows. The permission issue arose from a check that was introduced to handle race conditions when renaming checkpoints #28364 

@amyeroberts I see, my bad
[🛠️] Fix-whisper-breaking-changes 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Should fix the backward compatibility issue with `model.config.forced_decoder_ids = ...` and should help users who want to generate with timestamps. 

Fixes #21937 and #21878

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Test are failing because we do not check the `model.config` updating and will look good! 

The same should now be applied to both the `TF` and the `Flax` version as the overwriting of the `generate` function is also supported. Will open a follow up PR for these 

For TF and flax #21334
[`docs`] Improve visualization for vertical parallelism
Author: petergtz
Pull request: yes

# What does this PR do?

The documentation says "We refer to this Model parallelism as “Vertical” because of how models are typically visualized.", but then visualizes the model horizontally. This change visualizes the model indeed vertically.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@stevhliu @sgugger

Comments:
Mark test_constrained_beam_search_generate as flaky
Author: amyeroberts
Pull request: yes

# What does this PR do?

Test occasionally fails on CI runs e.g. https://app.circleci.com/pipelines/github/huggingface/transformers/83241/workflows/6cb424b9-229b-412f-abfd-71cc6cfc7392/jobs/1073186/tests#failed-test-0

Marking as flaky to trigger retries to help prevent failing CI runs on unrelated PRs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28757). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Pin pytest version <8.0.0
Author: amyeroberts
Pull request: yes

# What does this PR do?

pytest released a new major version, 8 two days ago: https://pypi.org/project/pytest/8.0.0/

This breaks doctest runs on CI e.g. https://app.circleci.com/pipelines/github/huggingface/transformers/83241/workflows/7ca5119f-b434-4c93-89fb-28378e63c449/jobs/1073188

Pinning until we make our doctests compatible. 

Comments:
@amyeroberts you are too fast to address my first comment, so I am not sure if you see my 2nd one



src/transformers/dependency_versions_table.py need to be updated by running make deps_table_update


> @amyeroberts you are too fast to address my first comment, so I am not sure if you see my 2nd one
> 
> src/transformers/dependency_versions_table.py need to be updated by running make deps_table_update

I didn't - but I've run and pushed the change now :) 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28758). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.