noamaanMulla-03/transformers-issues huggingface dataset:
device agnostic trainer testing
Author: ji-huazhong
Pull request: yes

# What does this PR do?
Part of https://github.com/huggingface/transformers/issues/25654#issuecomment-1783704306

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @ydshieh 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27131). All of your documentation changes will be reflected on that endpoint.

Results on CI runner looks good.
make tests of pytorch_example device agnostic
Author: ji-huazhong
Pull request: yes

# What does this PR do?

Part of https://github.com/huggingface/transformers/issues/25654


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

cc @ydshieh @fxmarty @zhangsibo1129 @arsalanu 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27081). All of your documentation changes will be reflected on that endpoint.

@amyeroberts  This is a continuation of (the merged) #25870. There might be more such PRs in the future.
remove the obsolete code related to fairscale FSDP
Author: ji-huazhong
Pull request: yes

# What does this PR do?

As the title says, this PR introduces two modifications:
1. remove the gradient clipping code related to fairscale FSDP, as fairscale FSDP has been removed. See: https://github.com/huggingface/transformers/pull/25702 
2. updated the error message when mixing `--bf16` and `--half_precision_backend apex` as the `cuda_amp` option is no longer available. I'm not sure how to write it more appropriately, so I just use `auto` instead.



Comments:
@muellerzr and @ArthurZucker could you please take a look at this PR? Thanks :D

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26651). All of your documentation changes will be reflected on that endpoint.

@amyeroberts sorry for bothering, could you please review this PR?
Remove broken links to s-JoL/Open-Llama
Author: CSRessel
Pull request: yes

# What does this PR do?

I noticed a broken link in the README, for the source code of the Open-Llama model. This looks like it matches the docs where there's been a warning added:

> This model is in maintenance mode only, so we won't accept any new PRs changing its code.

To reduce confusion (and link rot!) this PR updates all references to that code, by clarifying that the original source was removed.

For the previous discussion on model deprecation,  see: https://github.com/huggingface/transformers/pull/24922

## Before submitting

- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Potential reviewers might be @stevhliu or @MKhalusova since it's docs, or @amyeroberts who I've seen approve a lot of README edits today, or @sgugger has previously reviewed PR's related to Open-Llama.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27164). All of your documentation changes will be reflected on that endpoint.
[docstring] Fix docstring for AltCLIPTextConfig, AltCLIPVisionConfig and AltCLIPConfig
Author: AksharGoyal
Pull request: yes

# What does this PR do?

 - Fixed docstrings for AltCLIPTextConfig, AltCLIPVisionConfig and AltCLIPConfig
 - Cleaned few docstrings

Fixes #26638

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.





Comments:
Hi @amyeroberts I have made the change. Let me know if anything else is needed from my side. For some reason removing AltCLIPVisionConfig is giving errors so I let it stay there. Any tip on resolving it would be helpful.

@AksharGoyal Thanks! For the removal of the `AltCLIPVisionConfig` what errors are you getting?

@amyeroberts This is what I see when I go to one of the [failed workflows](https://app.circleci.com/pipelines/github/huggingface/transformers/76713/workflows/18e1d257-983d-4f62-8733-8827d768d425/jobs/975389)
```
2023-10-29 19:05:35.437483: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/circleci/transformers/src/transformers/deepspeed.py:23: FutureWarning: transformers.deepspeed module is deprecated and will be removed in a future version. Please import deepspeed modules directly from transformers.integrations
  warnings.warn(
Traceback (most recent call last):
  File "utils/check_docstrings.py", line 1231, in <module>
    check_docstrings(overwrite=args.fix_and_overwrite)
  File "utils/check_docstrings.py", line 1223, in check_docstrings
    raise ValueError(error_message)
ValueError: There was at least one problem when checking docstrings of public objects.
The following objects docstrings contain templates you need to fix: search for `<fill_type>` or `<fill_docstring>`.
- AltCLIPVisionConfig

Exited with code exit status 1

```

@AksharGoyal The issue is arising because some of the parameters in `AltCLIPVisionConfig`'s  docstring aren't correctly filled in. You'll see now in the configuration file `<fill_docstring>` has been added in places where descriptions are needed 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27128). All of your documentation changes will be reflected on that endpoint.
Trigger CI if `tiny_model_summary.json` is modified
Author: ydshieh
Pull request: yes

# What does this PR do?

When `tests/utils/tiny_model_summary.json` is changed, currently it won't trigger tests, as it is not a python file.

However, we expect CI being triggered, as the modification of this file usually means there are new models enabling pipeline testing.

(So far, sometimes a PR (usually mine) may have green CI but later red CI on `main` due to this)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Shorten the conversation tests for speed + fixing position overflows
Author: Rocketknight1
Pull request: yes

Some of the conversation tests were still overflowing the maximum position embeddings for very short models like `BlenderBotSmall`. I reduced the limits a lot, which also speeds up those tests! cc @ydshieh

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @Rocketknight1 . Thanks a lot. I think it's probably better to simply skip this test for blenderbot small.

Could you remind me why the test was never failing before?

@ydshieh the test never failed before because the `ConversationalPipeline` class wasn't working very well, and didn't actually generate any text in these tests! 

Thanks. Then let's just skip it for this model. You can search `def is_pipeline_test_to_skip` to see how to skip pipeline tests. 

@ydshieh are you sure? I think the test makes more sense with very short outputs anyway - we don't need 20 tokens to confirm the pipeline works!

That's true, but making it this way also hide the fact of some models fail this test. Having a skip make it explicit. We can definitely use `2` instead of `20` if we do have a gain of running time (not that usually the model used for pipeline testing is very small, and not really obvious to say we will have a gain).

BlenderBotSmall can pass the test! The problem only arises because the test `BlenderBotSmallConfig` has `max_position_embeddings=20`. Maybe I keep the test at a higher number, but increase the `BlenderBotSmallConfig` in the test to have more position embeddings, so it passes more easily?

Hey! Let's just skip it for now it's blocking a few prs! 

I'll merge #27013 so you have time to work on the fix

cc @ydshieh @ArthurZucker this should be ready to go now! I've skipped the test for `BlenderBotSmall` as @ydshieh suggested because the problem is in the `hf-internal-testing` checkpoint, and I don't think the model has much usage anymore. The model actually works fine - the test checkpoint just has a very low value for `max_position_embeddings`.

I also increased `max_new_tokens` back to `5` in the general test and removed the workarounds that @ArthurZucker added. I don't think it needs to go higher - it just makes the test slower, and doesn't tell us anything new!

Quick ping again @ydshieh @ArthurZucker! This PR should be ready to go

Damm, I made review but forgot to commit the comments. Sorry 

@ydshieh Changes made as you suggested - I used `is_pipeline_test_to_skip`. I left the updated sequence lengths alone because even though they don't fix our `tiny-random` models, they do fix some other tests, especially for `BlenderBot`.
NEFTune Support pls
Author: nivibilla
Pull request: no
Labels: name: Feature request

### Feature request

The TRL library introduced support for NEFTune in this [pr](https://github.com/huggingface/trl/pull/871) 

### Motivation

Improves Supervised Fine-tuning performance

See paper : https://arxiv.org/abs/2310.05914

### Your contribution

I can try to port code over from TRL.

Comments:
Hey @nivibilla, we integrate well with TRL, which leverages a hefty number of components from the `transformers` library. What is your issue with the current TRL implementation/what would you like to enable by moving the support from TRL directly to `transformers` ?

a lot of my scripts and pipelines for finetuning use native transformers. ideally I don't want to have to update all of them to use trl if possible. 

@younesbelkada could you share a bit about how one could go about leveraging NEFTune with `transformers` models?

I think it's important that the two frameworks complete each other rather than compete for the same features with differing implementations, so let's take a look at leveraging TRL simply for `transformers` models rather than importing the implementation within `transformers`.

currently in TRL we monkey-patch the forward pass of the emebdding layer with a new one, but with https://github.com/huggingface/trl/pull/889 we'll have a cleaner implementation that leverage forward post hooks which is cleaner. I will open a PR to add that feature in Trainer and leverage that feature in TRL
[FEAT] Add Neftune into transformers Trainer
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title

Fixes: https://github.com/huggingface/trl/issues/923
Fixes: https://github.com/huggingface/transformers/issues/26899

This PR adds NEFTune: a new technique for enhancing Supervised fine-tuning results results proposed in: https://arxiv.org/abs/2310.05914 

![Screenshot 2023-10-13 at 17 36 38](https://github.com/huggingface/trl/assets/49240599/81995c1d-16c3-4cfc-9fbc-0701a9a87531)


I propose a very simple API which is as simple as passing a valid `neftune_noise_alpha` argument when initializing the `TrainingArguments`. To avoid any surprising behaviour, we should revert to the original forward method at the end of the training. This is handled inside the inner training loop that attaches the correct forward hook before the beginning of training, and makes sure to remove it right after training the model. 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Added a test and a relevant documentation section, this PR is ready for final review!
fix: Fix typical_p behaviour broken in recent change
Author: njhill
Pull request: yes

A recent PR https://github.com/huggingface/transformers/pull/26579 fixed an edge case out-of-bounds tensor indexing error in TypicalLogitsWarper, and a related behaviour change was made that we thought fixed a long-standing bug w.r.t. the token inclusion cutoff.

However after looking more closely, I am pretty certain that the original logic was correct and that the OOB fix should have been made differently.

Specifically the docs state that it should include the "smallest set of tokens that add up to P or higher" and so `last_ind` should actually be one more than the index of the last token satisfying `(cumulative_probs < self.mass)`.

We still need a max clamp in case that last token is the very last one in the tensor.

Comments:
@gante sorry about this! I observed that it can actually make a significant difference to the output when typical_p is used.

(the CI failed for unrelated reasons, rerunning failed jobs)
[`Quantization` / `tests` ] Fix bnb MPT test
Author: younesbelkada
Pull request: yes

# What does this PR do?

Addresses: https://github.com/huggingface/transformers/pull/27145#discussion_r1376240625 

https://github.com/mosaicml/llm-foundry/issues/703 being merged, we can now safely unskip the test and make sure to point the trust remote code model into the commit that contains the fix

cc @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[docs] Update CPU/GPU inference docs
Author: stevhliu
Pull request: yes

Implements the proposal suggested in #26723, namely:

- consolidating the inference on one/many GPUs docs into one page
- removing the inference on specialized hardware page
- cleanups and updates to the inference docs to provide more context about the how/what of the optimization techniques, code examples so users don't have to skip around to other pages, etc.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lvwerra did you participate in writing the original document as well? Do you have feedback on this refactor?

Nice overhaul! I think it's also an opportunity to mention Optimum as this falls right into what we are doing:
- I think the GPU inference page should briefly mention inference with ORT and link to: https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu
- Same for CPU with a link to: https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models
- And for Intel CPUs I would also refer to Optimum Intel: https://huggingface.co/docs/optimum/intel/index

WDYT?

Cc @echarlaix @mfuntowicz 

> @lvwerra did you participate in writing the original document as well? Do you have feedback on this refactor?

Only did the general structure of that part so leave it to others to comment on content :) 

> Nice overhaul! I think it's also an opportunity to mention Optimum as this falls right into what we are doing:
> 
> * I think the GPU inference page should briefly mention inference with ORT and link to: https://huggingface.co/docs/optimum/onnxruntime/usage_guides/gpu
> * Same for CPU with a link to: https://huggingface.co/docs/optimum/onnxruntime/usage_guides/models
> * And for Intel CPUs I would also refer to Optimum Intel: https://huggingface.co/docs/optimum/intel/index
> 
> WDYT?
> 
> Cc @echarlaix @mfuntowicz

Totally aligned with your suggestion @regisss, let us  know if we can help @stevhliu 

Thanks for the feedback y'all! 

Let me know what you think of the new Optimum sections @regisss @echarlaix :)
🌐 [i18n-ZH] Translate tflite.md into Chinese
Author: yyLeaves
Pull request: yes

# What does this PR do?
Translate tflite.md into Chinese
part of #20095  
## Who can review?
Documentation: @stevhliu 
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27134). All of your documentation changes will be reflected on that endpoint.
fix docstring in get_maskformer_resize_output_image_size
Author: wesleylp
Pull request: yes

# What does this PR do?
Fixes docstring in `get_maskformer_resize_output_image_size`.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 
@rafaelpadilla 


Comments: