noamaanMulla-03/transformers-issues huggingface dataset:
[Time-Series] time-series patching
Author: elisim
Pull request: no
Labels: name: New model

### Model description

"time-series patching" refers to the process of segmentation the series into subseries-level patches which are served as input tokens to the transformer. It's really similar to what's done in ViT, but for time-series. This idea was first propsed in a recent ICLR paper:

[A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://arxiv.org/abs/2211.14730)
code: https://github.com/yuqinie98/PatchTST

@kashif @NielsRogge 

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

@yuqinie98

Edit: I think that "new model" is not the best label to this issue, maybe there is a better label for this?

Comments:
PatchTST is in Gluon, thanks to @kashif . Closing here :) https://github.com/awslabs/gluonts/pull/2748

Oh I apologize that I just notice this issue. Somehow I haven't seen it previously... And I appreciate so much for your attention on the PatchTST! Anything I could do for it? 
Also, glad to see the Gluonts application!

@yuqinie98 not a problem... i will get PatchTST added to transformers next

@yuqinie98 also note that the model on gluonts is not exactly the patchTST implementation as your paper (or the implementations in tsai and neuralforecast):
 - it doesn't carry the multivariate time series around but rather each entry in the batch is some random time series from the data
  - it is probabilistic 
  - the model is given the mean and std of the window of each time series as input (a bit like the non-stationary transformer) 
  - a lag variant (instead of `torch.unfold`) based on the `freq` is also implemented
  
  the points I raised in my email to you still stand with respect to the paper
Check for case where `auxiliary_head` is `None` in `UperNetPreTrainedModel`
Author: mmurray
Pull request: yes

# What does this PR do?

`UperNetConfig` has an option called [`use_auxiliary_head`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/upernet/configuration_upernet.py#L45). When `use_auxiliary_head` is `False`, the [`auxiliary_head` is set to `None`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/upernet/modeling_upernet.py#L359). However,  [`UperNetPreTrainedModel` assumes that `auxiliary_head` is not `None`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/upernet/modeling_upernet.py#L314), resulting in an error when loading a pretrained upernet model with `use_auxiliary_head=False` (see issue #25513 ).

This pull request updates `UperNetPreTrainedModel` to appropriately handle the case where `auxiliary_head` is `None`.

Fixes #25513 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
`UperNetPreTrainedModel` throws an `AttributeError` when `use_auxiliary_head=False`
Author: mmurray
Pull request: no

### System Info

- `transformers` version: 4.31.0 (also tried on 4.32.0.dev0)
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.29
- Python version: 3.8.11
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 1.10.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code snippet:
```
from transformers import UperNetForSemanticSegmentation
model = UperNetForSemanticSegmentation.from_pretrained("openmmlab/upernet-swin-base", use_auxiliary_head=False)
```

Resulting error / stack trace:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/mike/.pyenv/versions/grip/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2700, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/mike/.pyenv/versions/grip/lib/python3.8/site-packages/transformers/models/upernet/modeling_upernet.py", line 362, in __init__
    self.post_init()
  File "/home/mike/.pyenv/versions/grip/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1108, in post_init
    self.init_weights()
  File "/home/mike/.pyenv/versions/grip/lib/python3.8/site-packages/transformers/models/upernet/modeling_upernet.py", line 314, in init_weights
    self.auxiliary_head.init_weights()
AttributeError: 'NoneType' object has no attribute 'init_weights'
```

### Expected behavior

I expect that the model should initialize with no error (and with the auxiliary head unused internally).

Comments:
add __repr__  to the BitsAndBytesConfig class
Author: ranchlai
Pull request: yes

# What does this PR do?

add __repr__  to the BitsAndBytesConfig class for better visualization and debugging
before 
```python
>>> print(bnb_config)
>>> BitsAndBytesConfig(quant_method=<QuantizationMethod.BITS_AND_BYTES: 'bitsandbytes'>)

```
after
```python
>>> print(bnb_config)
>>> BitsAndBytesConfig {
  "bnb_4bit_compute_dtype": "bfloat16",
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": true,
  "llm_int8_enable_fp32_cpu_offload": false,
  "llm_int8_has_fp16_weight": false,
  "llm_int8_skip_modules": null,
  "llm_int8_threshold": 6.0,
  "load_in_4bit": true,
  "load_in_8bit": false,
  "quant_method": "bitsandbytes"
}
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25517). All of your documentation changes will be reflected on that endpoint.
Make training args fully immutable
Author: muellerzr
Pull request: yes
Labels: name: trainer

# What does this PR do?

This PR ensures that the `TrainingArguments` are a fully immutable dataclass after the `__post_init__` has been ran. We'll find that the tests suddenly fail now 😉 Should be merged after https://github.com/huggingface/transformers/pull/25390

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger can you give it one final look please 😄
Use dynamic past key-values shape in TF-Whisper
Author: Rocketknight1
Pull request: yes

TF-Whisper uses the static past key-values shape for some conditionals, which causes issues when compiling with past key-values whose shape is not a known constant at compile time.

This PR swaps it to the dynamic runtime shape for correct compilation.

Fixes #25522 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Avoid CI runs under users' own CircleCI personal account
Author: ydshieh
Pull request: yes

# What does this PR do?

Sometimes the tests will run in the CircleCI of the user and not run our tests since they don't have access to our resources. One example is on [this PR](https://github.com/huggingface/transformers/pull/20479#issuecomment-1369690668) where the "real" tests were not run.

**We can make the new job `check_circleci_user` required (too) - once this PR is merged into `main`**

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20981). All of your documentation changes will be reflected on that endpoint.

@ydshieh  Unfortunately, I encountered the problem you mentioned last night and I don't know how to solve it. The link provided in your MR ( [https://support.circleci.com/hc/en-us/articles/360008097173-Troubleshooting-why-pull-requests-are-not-triggering-jobs-on-my-organization- ](https://support.circleci.com/hc/en-us/articles/360008097173-Troubleshooting-why-pull-requests-are-not-triggering-jobs-on-my-organization-) ) has expired. Can you tell me how to solve this problem.? This is my PR [#25519](https://github.com/huggingface/transformers/pull/25519 )  After I submitted it, the circle ci began to execute automatically, but this issue occurred afterwards.
Training speed slows down to a half when double batchsize
Author: YTianZHU
Pull request: no

### System Info

same version in the requirements.txt

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run any given llama training code in the repo, if I double --per_device_train_batch_size, the training speed slows down to a half, which is not as expected

### Expected behavior

if I double --per_device_train_batch_size, the training speed should maintain almost the same

Comments:
I don't know where you read that but this is only true for very small batch sizes. It's logical that the model takes twice as long to compute the outputs when you have twice the amount of data.

Sorry for the unclearness, I actually use small batch sizes, like --per_device_train_batch_size=2, 4, 8, 16. Or, is it already relatively large batch sizes for the nlp task? New to nlp.
Draft
Author: lishukan
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
Marian: post-hack-fix correction
Author: gante
Pull request: yes

# What does this PR do?

### Context
#25294 removes a Marian-only hack, where the `logits` of the `pad_token_id` were being set to `-inf` before the `logsoftmax`. This was perceived as safe to do, as `bad_word_ids` is set to the `pad_token_id` in Marian. `bad_word_ids` sets the scores (i.e. after `logsoftmax`) to `-inf`

### Why was that a problem?
The changed position of the operation (setting to `-inf` before/after `logsoftmax`) meant that the log probabilities after the `logsoftmax` were now NOT summing to 1, i.e. not normalized. Since beam search selects the beams with the highest scores, this meant that the higher `logits[:, pad_token_id]` were before being set to `-inf`, the higher distribution shift would be observed due to the lack of normalization.

This issue materializes as minor output differences, as seen in our CI, and minor performance degradation, as discussed in #4290

### Do we have to revert the PR and reintroduce the hack?
No 🙌  Long after the hack in #4290 was introduced, we have added a flag to renormalize the scores after applying the logits processors, which happens after the `logsoftmax`. Applying this flag is equivalent to applying the hack.

⚠️ This means that Marian's generation configs should be updated with this flag as well, in addition to this PR (assuming it is accepted :) )

👉 all `slow` tests are back to green after this change.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Layout lmre
Author: yang0369
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
Any reason why BART does not have a ForTokenClassification variant?
Author: BramVanroy
Pull request: no
Labels: name: Core: Modeling

Are there any theoretical constraints to create a ForTokenClassification variant for BART? In my current project I am using a sequence classification head + a token classification head, so I would like to implement the token classification part manually. However, since it is not implemented in the repository, I wonder if there are any particular reasons why one should not do this. 

Comments:
No theoretical reasons, go for it!



@BramVanroy, did you implement BartForTokenClassification?

@vgaraujov Sorry, no I did not.
📝 Add parameter names to code examples in README
Author: kadirnar
Pull request: yes

@sgugger, @stevhliu and @MKhalusova 

Updated the code examples in the README file to include parameter names for better clarity and readability. Previously, the examples were missing parameter names, which could lead to confusion. By adding the parameter names, it becomes easier for users to understand and utilize the code correctly.

Thanks! 🙌

Comments:
> Thanks for your PR. I don't believe this makes those basic examples easier to understand, however, so I would leave things as is.

Hi @sgugger, thank you for your feedback. It might be nice to specify the 'model' parameter of the pipeline function. This is how I will update all the tasks in the https://huggingface.co/tasks section.

Example(task=depth-estimation):
huggingface/hub-docs#890

If you want, I can close this pull request.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🚨🚨🚨  Fix rescale ViVit Efficientnet
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes the rescaling logic for both EffiicentNet and ViVit. 

EfficientNet: The values were being rescale between [-0.5, 0.5] 

ViVit: values were being rescale between [-7.689350249903882e-06, 0.9999923106497501] as scale was being treated as having a value 255 in the `rescale` method, rather than 1/255. 

**This is a breaking change** and will affect the model outputs for both these models. However, it is a bug fix and should improve model predictions. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

If I'm not mistaken this PR is problematic as it breaks the `rescale` function. If I rebase my PR https://github.com/huggingface/transformers/pull/24796 to `main` the torchvision transforms vs transformers image transforms equivalency test fails: `tests/models/idefics/test_image_processing_idefics.py::IdeficsImageProcessingTest::test_torchvision_numpy_transforms_equivalency`

@amyeroberts I'm guessing this was issue causing bad efficientnet predicons that I observed? 

FWIW I haven't ecountered any vision models that require separate codepaths for rescale vs mean/std [-1, 1] is mean 0.5, std 0.5, and [0, 1] is mean 0, std 1. ...

Not sure if this is still relevant, but 

- @rwightman  could you let us know your `main` commit sha?
- cc @amyeroberts (if not yet look the last above comment)


@ydshieh I managed to track down the issue with EfficientNet down to the softmax which was being applied to the logits #25501 (sorry, I should have linked here as well). 

@rwightman Agreed, the design was something I either suggested or approved, so my bad. The good thing is, using `offset` isn't necessary, and users can simply pass in different means and stds to achieve the desired rescaling as you suggested.