noamaanMulla-03/transformers-issues huggingface dataset:
BART-fusion
Author: jnj2102
Pull request: no
Labels: name: New model

### Model description

BART- fusion, a novel model for generating lyric interpretations from lyrics and music audio that combines a large-scale pre-trained language model with an audio encoder. It uses a cross-modal attention module to incorporate the audio representation into the lyrics representation to help the pre-trained language model understand the song from an audio perspective, while preserving the language model’s original generative performance.  Please see the paper here: https://arxiv.org/abs/2208.11671 

### Open source status

- [X] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Here is the code repository for the paper: https://github.com/ldzhangyx/BART-fusion/tree/main.  The weights should be available in the checkpoints: https://drive.google.com/drive/folders/18EUUx-KT9xGJ1uq2UoOgj0X9BpngNn_T

Comments:
@sgugger what do you think of this request? If you think it's a good addition to the repo, I can take this task on.

cc @sanchit-gandhi and @hollance 

Hey @jnj2102! Thanks for the feature request - while I think it's a cool model, I'm not sure it's best suited in the `transformers` library directly since the original repository has quite low usage (20 stars) and the paper as well (4 citations). If you're really keen on using this model, you could explore adding it to the Hub, e.g. as done with the [MERT](https://huggingface.co/m-a-p/MERT-v1-95M) model. WDYT?

Hi! No problem. How do you  add a model to the Hub? I’ll check out the MERT
model too.

On Fri, Jun 2, 2023 at 11:29 AM Sanchit Gandhi ***@***.***>
wrote:

> Hey @jnj2102 <https://github.com/jnj2102>! Thanks for the feature request
> - while I think it's a cool model, I'm not sure it's best suited in the
> transformers library directly since the original repository has quite low
> usage (20 stars) and the paper as well (4 citations). If you're really keen
> on using this model, you could explore adding it to the Hub, e.g. as done
> with the MERT <https://huggingface.co/m-a-p/MERT-v1-95M> model. WDYT?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/23781#issuecomment-1573929450>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP2N4VD6EPP5HNQTGMZ6ALXJIBFDANCNFSM6AAAAAAYPVMXB4>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
-- 
Best wishes,

Jami


Hey Jami! Awesome - there's info on using custom code on the Hub here: https://huggingface.co/docs/transformers/v4.27.1/en/custom_models#using-a-model-with-custom-code. Let me know if you have any questions, more than happy to help here!
Pin `deepspeed` to `0.9.2` for now
Author: ydshieh
Pull request: yes

# What does this PR do?

DeepSpeed 0.9.3 has some issue, and introduced many more failures. See for example [here](https://github.com/huggingface/transformers/actions/runs/5166856163/jobs/9307371184) and [there](https://github.com/huggingface/transformers/actions/runs/5166856163/jobs/9307371242).

Let's pin `deepspeed==0.9.2` for now 🙏 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @pacman100
Fix parallel mode check
Author: muellerzr
Pull request: yes

# What does this PR do?

Fixes a check that relies on `distributed_state` when it might not be there

Fixes # (issue)

Should fix https://github.com/huggingface/transformers/issues/23390

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
changed unused args from error to warning
Author: Daryl149
Pull request: yes

the value error now breaks the code, while it can run perfectly without the unused arguments. This happens for example in https://huggingface.co/OpenAssistant/falcon-40b-sft-mix-1226/discussions/2

# What does this PR do?

Fixes issue referenced here: https://huggingface.co/OpenAssistant/falcon-40b-sft-mix-1226/discussions/2


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Library:


- generate: @gante




Comments:
This feels like a limitation of being able to discern between a typo (error) and an actual unused argument (warning), but I understand :(
Add check for tied parameters
Author: SunMarc
Pull request: yes

# What does this PR do 

This is the fix in the transformers library of this [PR](https://github.com/huggingface/accelerate/pull/1529). 

It will fix the case where a user uses their own device map (in the `from_pretrained` method) but forget that parameters that are tied together should be on the same device. We return an error showing which parameters should be on the same device.




Comments:
The test are failing because I used a function that i have added recently in accelerate.utils. Should we use the main for the tests @sgugger ? 

_The documentation is not available anymore as the PR was closed or merged._
ZeroDivisionError on `trainer.evaluate` if model and dataset are tiny
Author: tomaarsen
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.16
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger

cc: @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Consider the following snippet:
```python
from torch import nn
from transformers import Trainer
from datasets import Dataset

model = nn.Identity()
eval_dataset = Dataset.from_dict({"tokens": [1]})

trainer = Trainer(
    model,
    eval_dataset=eval_dataset,
)
metrics = trainer.evaluate()
print(metrics)
```
(Sometimes) results in 
```
Traceback (most recent call last):
  File "[sic]\demo.py", line 13, in <module>
    metrics = trainer.evaluate()
  File "[sic]\transformers\trainer.py", line 3043, in evaluate
    speed_metrics(
  File "[sic]\transformers\trainer_utils.py", line 354, in speed_metrics
    samples_per_second = num_samples / runtime
ZeroDivisionError: float division by zero
```
This is rarely an issue - only when models and datasets are tiny. The reason I am invested in resolving this is testing purposes. See for example this [Action](https://github.com/lvwerra/trl/actions/runs/5179991753/jobs/9351434458) on TRL. To keep the testing efficient, the TRL maintainers chose a small model and dataset - which sometimes caused this flaky test.

### Expected behavior

I would expect any of these:
```
1. {'eval_runtime': 0.0, 'eval_samples_per_second': 0.0, 'eval_steps_per_second': 0.0}
2. {'eval_runtime': 0.0, 'eval_samples_per_second': None, 'eval_steps_per_second': None}
3. {'eval_runtime': 0.0, 'eval_samples_per_second': torch.inf, 'eval_steps_per_second': torch.inf}
4. {'eval_runtime': 0.0}
```
Note that these cases would essentially never occur other than during tests. With other words, I think all are fine as long as there's no exception. However, I prefer option 4 personally, but I am open to suggestions. For simplicity, I'll push a simple PR to implement 4.

- Tom Aarsen

Comments:
Prevent ZeroDivisionError on `trainer.evaluate` if model and dataset are tiny 
Author: tomaarsen
Pull request: yes

Closes #24048

Hello!

## Pull Request overview
* Prevent ZeroDivisionError on `trainer.evaluate` if model and dataset are tiny 

## Details
Please see #24048 for details.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Tests would be quite flaky for this.


## Who can review?

@sgugger, @younesbelkada 

- Tom Aarsen

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add TimmBackbone model
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds a new model TimmBackbone to use for loading timm weights for use in the AutoBackbone API.

Example usage: 
```
from transformers import AutoBackbone

# Loads a transformers model 
backbone = AutoBackbone.from_pretrained("microsoft/resnet-18")

# Loads a timm checkpoint
backbone = AutoBackbone.from_pretrained("resnet18", use_timm_backbone=True)
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh Merging now. I'll address any comments or requests for changes in a follow up PR.
Use TruncatedNormal from Keras initializers
Author: hvaara
Pull request: yes

# What does this PR do?

This PR updates the types of `get_initializer` to use `TruncatedNormal` from `tf.keras.initializers`. Before this change the type was set to `tf.initializers.TruncatedNormal`, while `tf.keras.initializers.TruncatedNormal` is what was returned from the function.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Revamp test selection for the example tests
Author: sgugger
Pull request: yes

# What does this PR do?

After the revamp of the test_fetcher, the test collection of the example tests wasn't working anymore: if only an example is changed, the corresponding tests are not run. This PR adapts the example tests collection to the new test fetcher and makes sure they are appropriately run. It's more fine-grained than the previous approach which ran all example tests as soon as when a diff was discovered: here the tests are run when the modifications impact the test examples.

To see this in action:
- at the [first commit](https://app.circleci.com/pipelines/github/huggingface/transformers/65204) the diff only impacts repo utils, so only the test repo utils job is ran (no example or other test jobs).
- at the [second commit](https://app.circleci.com/pipelines/github/huggingface/transformers/65207) the diff has a change in the PyTorch `run_glue`, so the PyTorch example test job is ran, but only on the trainer examples (not the no_trainer ones since they are not touched).
- at the [third commit](https://app.circleci.com/pipelines/github/huggingface/transformers/65208), we remove the fake example modif and add a fake change in the Trainer, which impacts all examples.

cc @ydshieh for when you're back from vacation.

Comments:
Act on deprecations in Accelerate no_trainer examples
Author: muellerzr
Pull request: yes

# What does this PR do?

As title states, acts on deprecation that will be going through in this PR https://github.com/huggingface/accelerate/pull/1537 to avoid nightly failures

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Oops, missed one
Author: muellerzr
Pull request: yes

# What does this PR do?

Tried to be careful, missed one 🙃 

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix device placement for model-parallelism in generate for encoder/de…
Author: sgugger
Pull request: yes

…coders

When using model parallelism with encoder/decoder models, there is an issue in `generate` using the encoder in isolation. That encoder won't output logits on the same device as the inputs like the whole model does, and we get mismatched device errors.

Repro:
```py
from transformers import AutoModelForSeq2SeqLM

model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M", device_map="auto")
inputs = {"input_ids": torch.tensor([[256047, 94124, 248079, 15697, 248203, 2]], device=0), 'attention_mask': torch.tensor([[1, 1, 1, 1, 1, 1]], device=0), 'forced_bos_token_id': 256079}

model.generate(**inputs, max_length=4000)
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Remote code improvements
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a few improvements to the code on the Hub API, mainly:
- if `trust_remote_code` is unset, we ask the user to validate or not (with a timeout of 30s for users launchin script)
- in case of conflict between code on the Hub and local code (for when Falcon or MPT are added in Transformers for instance),  `trust_remote_code=False` will execute the local code, `trust_remote_code=True` the one on the Hub

Comments:
_The documentation is not available anymore as the PR was closed or merged._
🌐 [i18n-KO] Translated `bertology.mdx` to Korean
Author: wonhyeongseo
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `bertology.mdx` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `bertology.mdx` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
@sgugger, @ArthurZucker, @eunseojo May you please review this PR?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

* Live preview is not working, although checks have passed. I confirmed there are no gotchas using GitHub Codespace. (We should create a pre-built environment (where all `pip install` commands are completed) for mentees and new-comers, preferably with `preview --[lang]` commands built-in as well)

Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

@sgugger, @ArthurZucker, @eunseojo May you please review this PR?