noamaanMulla-03/transformers-issues huggingface dataset:
adding dynamic categorical feature option
Author: guyko81
Pull request: yes

# What does this PR do?

Adding dynamic categorical feature to time_series_transformer
// Have not tested yet!

Comments:
awesome @guyko81 will have a look! this will be great!


@kashif can you help me please?
when I try to add the variables based on the example I struggle getting the dynamic_categorical_features split into past and future values.
When I add it to the time features
```
def create_instance_splitter(
    config: PretrainedConfig,
    mode: str,
    train_sampler: Optional[InstanceSampler] = None,
    validation_sampler: Optional[InstanceSampler] = None,
) -> Transformation:
    assert mode in ["train", "validation", "test"]

    instance_sampler = {
        "train": train_sampler
        or ExpectedNumInstanceSampler(
            num_instances=1.0, min_future=config.prediction_length
        ),
        "validation": validation_sampler
        or ValidationSplitSampler(min_future=config.prediction_length),
        "test": TestSplitSampler(),
    }[mode]

    return InstanceSplitter(
        target_field="values",
        is_pad_field=FieldName.IS_PAD,
        start_field=FieldName.START,
        forecast_start_field=FieldName.FORECAST_START,
        instance_sampler=instance_sampler,
        past_length=config.context_length + max(config.lags_sequence),
        future_length=config.prediction_length,
        time_series_fields=["time_features", "observed_mask", "dynamic_categorical_features"],
    )
```
and use it like this:
```
    if config.num_dynamic_categorical_features > 0:
        PREDICTION_INPUT_NAMES.append("past_dynamic_categorical_features")
        PREDICTION_INPUT_NAMES.append("future_dynamic_categorical_features")
```
I got a shape error:
```
RuntimeError: stack expects each tensor to be equal size, but got [89, 1568] at entry 0 and [161, 1568] at entry 1
```

But when I try to simply add it like this:
```
    if config.num_dynamic_categorical_features > 0:
        PREDICTION_INPUT_NAMES.append("dynamic_categorical_features")
```
I got an error like this:
```
RuntimeError: stack expects each tensor to be equal size, but got [1568, 2] at entry 0 and [1566, 2] at entry 16
```

So the second version goes longer, however when a time series is shorter (1566 long vs 1568) it throws an error. I'm just not familiar with gluons enough to feel how to create past and future dynamic_categorical_features.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Ignore all exceptions from signal in dynamic code
Author: sgugger
Pull request: yes

# What does this PR do?

#25613 shows we can have different exception types coming from `signal` when asking the user if they want to trust remote code or not. This PR ignores them all.
As a reminder this is only a convenience function trying to help the user that did not set `trust_remote_code=True`, so the errors all become "Please set `trust_remote_code`"

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix PEFT integration failures on nightly CI
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes all failing CIs when PEFT is installed that has the error:

```bash
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
```

The failing tests are `test_from_pretrained_no_checkpoint` tests that calls `xxx.from_pretrained()` with model_id explicitly being to None. When PEFT is installed (which is the case in the daily CI runners since #25077 ) `from_pretrained` will look for adapter files in the model repo, and a check that ignores `model_id` if it is set to None was forgotten. 

cc @ydshieh @sgugger 


Comments:
Sounds great, thank you all !

_The documentation is not available anymore as the PR was closed or merged._
[`Llama2`]  Add support for Llama 2
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Add Support for Llama2 ! 🔥 🔥 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24891). All of your documentation changes will be reflected on that endpoint.

Somehow this commit will easily lead the llama model to be overflow in fp16 during training. Hope someone can take a look :)

@fe1ixxu you can have a look at #25065 for more details on this!
Run doctest for new files
Author: ydshieh
Pull request: yes

# What does this PR do?

The effect could be see in [this run](https://app.circleci.com/pipelines/github/huggingface/transformers/70724/workflows/f9ba5d54-6d64-4630-b500-ba1cf28baade/jobs/887892/artifacts) (see the artifact).

From Sir @ArthurZucker 

> new files should always be tested

> the documentation_tests.txt is only for legacy models taht are not tested

> but whenever someone adds something, (a new pipeline, a new model etc) should be automatic

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker also mentioned `a` to me. I am happy with that, but we have to be a bit careful not to include files that has no doc example in that opposite list (if we want to use that list as an indication of **missing** doctest - it depends what we interpret it and use it)

Added a `not_doctested.txt`. This list needs some more work, in particular `src/transformers/pipelines`. We should remove them out from this file so we can test them - but we have to make sure they work first!

(so far on `main`, they are not doctested)

I would prefer to separate that work from this PR.

I am going to merge this PR without removing `utils/documentation_tests.txt` yet. It's still used by the doctest workflow file, and changing that file to use the opposite logic needs some extra work.

Will do it in a follow up PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25588). All of your documentation changes will be reflected on that endpoint.
correct TTS pipeline docstrings snippet
Author: ylacombe
Pull request: yes

This PR aims to correct the TTS pipeline code snippet [that was failing](https://github.com/huggingface/transformers/actions/runs/5898104885/job/15998670461).

I simply wrote a correct snippet instead!


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


## Who can review?
Hey @ydshieh and @ArthurZucker, WDYT of that ?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @ArthurZucker , thanks for the quick review !

Just a QQ before addressing your comments, [there is already a line](https://github.com/ylacombe/transformers/blob/b64cccafd178f7bbf83da2a75c8697836d60f533/utils/documentation_tests.txt#L491-L492) `src/transformers/pipelines/` in `documentation_tests.txt`.

I'm thus afraid to be redundant if I add `src/transformers/pipelines/text_to_audio.py` in `documentation_tests.txt`. WDYT?

 

Let me check, probably it's a logic error in test fetcher.

cc @ydshieh as he suggested this. If it's there it should indeed be tested and no need to add it again. 

Can you also check the `empty` CI @ydshieh 

well, that line is added after the new doc test logic was merged.

For this PR, let's just add the full path to the file to `documentation_tests.txt` and move on to merge this.
Will address the test fetcher later.

empty CI is expected as no code is changed - if I understand correctly

> For this PR, let's just add the full path to the file to documentation_tests.txt and move on to merge this.

It's done now!

@ylacombe Let's merge this unless something else to be done?

@ydshieh I agree, can you do it?
OpenAIGPTModel raises exception with `afn="Relu"`
Author: nathan-chappell
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 
@younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers.models.openai import OpenAIGPTConfig, OpenAIGPTModel
OpenAIGPTModel(OpenAIGPTConfig(n_vocab=1, n_embed=1, afn="relu"))(torch.eye(1, dtype=int))

# raises:
# AttributeError: 'ReLU' object has no attribute 'size'
```

### Expected behavior

I would expect no error to be raised and the calculation to be performed.  For example, the following code behaves as expected:

```python
OpenAIGPTModel(OpenAIGPTConfig(n_vocab=1, n_embed=1, afn="gelu"))(torch.eye(1, dtype=int))
```

Comments:
Hey thanks for reporting and providing a very efficient snippet! Indeed the relu is not initialized thus you have the error. Reviewing your PR now

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Skip doctest for some recent files
Author: ydshieh
Pull request: yes

# What does this PR do?

skip doctests for `Idefics` and some `peft` stuff.

They are not `documentation_tests.txt` (to be removed soon) anyway.

`Idefics` cause runner crash --> we need to see why this happens (and mark it a slow doctest manually if this is necessary)



Comments:
_The documentation is not available anymore as the PR was closed or merged._
🚨🚨🚨 changing default threshold and applying threshold before the rescale
Author: rafaelpadilla
Pull request: yes

# What does this PR do?

Fixes #25490 by applying the threshold before the rescaling.

Here's a breakdown:
OwlViT model scales confidences scores, so that the max score will always adjust to `1.000`. 

To illustrate, let's consider a scenario where the threshold is set at `threshold=0.6` and detected scores (in descending order) are `[0.4259, 0.0707, 0.0441, ...]`. After the rescaling process, they will become `[1.0000, 0.0734, 0.0039, ...]`, and by applying the threshold, the bounding box associated to the first score (initially `0.4259`) is retained.  This is not the expected behavior, since it initially had a low score of `0.4259`. 

With this behavior, at least one object is left (the one with the highest score), regardless its detection score.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Proper way to monkey patch a customized model no in transformers?
Author: lucasjinreal
Pull request: no

### Feature request

Hi, nowadays more and more model emerging out, many of them hard to merge transformers, but we have more and more customized patchs such as condensing rotary, xformers attn etc.

What is the best to monkey patch a customized model which not inside transformers library but just loaded with AutoModel?

### Motivation

need guidance

### Your contribution

current not

Comments:
Hi @lucasjinreal, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

@amyeroberts Please help me find someone answer this issue, I posted serveral on forum none of them got response. than k u

@lucasjinreal Another place to ask questions like this is in [our discord](https://discord.com/invite/hugging-face-879548962464493619). 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Added paper links in logitprocess.py
Author: pranith7
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
#24783 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Comments:
cc @gante 

Thanks for your PR! Please run `make style` on your branch to fix the auto-formatting issues.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25482). All of your documentation changes will be reflected on that endpoint.

@sgugger I have addressed the above change.

cc @gante 

cc @gante  updated the links for the paper

Thanks @gante for your support and this is my first step towards open source.
fix documentation for CustomTrainer
Author: minhtriet
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #25542


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.


HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25635). All of your documentation changes will be reflected on that endpoint.
`push_to_hub` with `branch` or `revision` keyword argument
Author: zanussbaum
Pull request: no
Labels: name: Feature request

### Feature request

In `datasets`, you can upload a dataset to a `branch`. In the `transformers` package, it doesn't seem like `branch` or `revision` [are supported](https://huggingface.co/docs/transformers/v4.28.1/en/main_classes/model#transformers.PreTrainedModel.push_to_hub)

### Motivation

To push a model to the hub and with a revision seems a little harder. It seems like I would need to find the cache directory of the model and use `upload_folder` from `huggingface_hub` to upload to the correct revision.

I could very well be missing the right documentation but I can't seem to figure out how/where to do this

### Your contribution

Maybe a PR? 

Comments:
cc @sgugger 

If you want to contribute a PR to add this, it would be welcome!

Any updates on this @zanussbaum ?
[DOCS] MusicGen Docs Update
Author: xNul
Pull request: yes

# What does this PR do?

Adds a note about token and generation limitations to the MusicGen docs so users know how to use the MusicGen model.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@sanchit-gandhi 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25510). All of your documentation changes will be reflected on that endpoint.

@sanchit-gandhi all done?
[TYPO] Update perf_train_gpu_one.mdx
Author: cakiki
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@osanseviero 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[TYPO] Update perf_train_gpu_one.mdx
Author: cakiki
Pull request: yes

# What does this PR do?
Fixes typo. negligable -> negligible

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@osanseviero 

Comments:
_The documentation is not available anymore as the PR was closed or merged._