noamaanMulla-03/transformers-issues huggingface dataset:
Adds `TRANSFORMERS_TEST_BACKEND`
Author: vvvm23
Pull request: yes

# What does this PR do?

Allows specifying arbitrary additional import following first `import torch`. This is useful for some custom backends, that will require additional imports to trigger backend registration with upstream torch. See https://github.com/pytorch/benchmark/pull/1805 for a similar change in `torchbench`.

If the specified backend does not exist, we throw a helpful error. I have updated the docs to include this new variable.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger would you mind taking a look at this? It relates to my previous PR.

Comments:
cc @ydshieh 

> I would like to see a real example of usage, please.

I have now provided example usage with the `torch_npu` backend. This is actually already in upstream so wouldn't be needed, but usage would be the same for backends not in upstream.

> I am also wondering if it's always only one backend needed. What happens if we need multiple ones?

I feel just having one is fine for now, I am struggling to imagine a use-case for multiple, unless a single worker somehow had CPU, GPU, and other backends all on one physical machine.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25655). All of your documentation changes will be reflected on that endpoint.
[`from_pretrained`]  Simpler code for peft
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Simplifies the logic when loading a PEFT model: the `_adapter_model_path` is used instead of `maybe_has_adapter_model_path` and `has_adapter_config`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks! Can confirm that the tests are now green! @younesbelkada some are really fast do you not want to remove the `slow` mention for them? 

@ArthurZucker sorry the tests are still failing, probably something wrong happened during the merge, can you double check? 🙏  I can also have a look if you want

Edit: just made https://github.com/huggingface/transformers/pull/25733
Fix `test_model_parallelism`
Author: ydshieh
Pull request: yes

# What does this PR do?

It's not complete yet, but I lost my faith for the many places needed in `vilt` and `esm`. Need your opinion 🙏 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

OK, so will update the `_no_split_module_class`. Thanks
[idefics] idefics-9b test use 4bit quant
Author: stas00
Pull request: yes

I didn't think that the nightly CI gpu is small when I initially wrote this test.

This PR updates the slow idefics-9b quality test to use 4bit quantization so it reduces memory needs from 20GB to 7GB.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25734). All of your documentation changes will be reflected on that endpoint.
ImageProcessor - check if input pixel values between 0-255
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds a warning in the image processor if the user is passing in images that have already had their pixel values rescale between 0 and 1 and `do_rescale=True`. Additionally adds information in the docstring.

This has caused some confusion and error reports recently, when users have been passing in images that have already been partially transformed. 

Related issues: #25195, #24857, #23096


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@rafaelpadilla @ArthurZucker Thanks bot for your detailed reviews! I've added the suggested updated: making the function name consistent with others `_is_scaled_image` -> `is_scaled_image` and any missing doc updates.
[ASR Pipe Test] Fix CTC timestamps error message
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

The test [`test_chunking_and_timestamps`](https://github.com/huggingface/transformers/blob/70b49f023c9f6579c516671604468a491227b4da/tests/pipelines/test_pipelines_automatic_speech_recognition.py#L1153) failed on the nightly run when checking that the argument `return_timestamps=True` gave the correct error message for CTC models. This is because the string pattern `(char)` in the error message was closing the regex expression. Removing it allows the test to pass as expected.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`from_pretrained`] Fix failing PEFT tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

Addresses: https://github.com/huggingface/transformers/pull/25726#issuecomment-1691783489
In fact it is important to overwrite `pretrained_model_name_or_path` with the `base_model_name_or_path` attribute of the adapter config otherwise `from_pretrained` will try to load the config file from the adapter model_id where it should look for  it at `base_model_name_or_path`

now all PEFT tests are green

cc @sgugger @ArthurZucker 

Comments:
Let's make them not slow otherwise we are shooting in the dark 😉 

I agree it sounds important to have it, happy to add a PEFT circleCI job that runs `tests_torch` as well as PEFT tests, I can do that in a follow up PR, what do you think @sgugger @ydshieh  (offline we have agreed to not do it, maybe we should after these failures?)

_The documentation is not available anymore as the PR was closed or merged._

We can have a new job to see how long it takes.

I would try to keep the new job to run only the PEFT tests but not to re-run the `tests_torch`.

Let's hear our @sgugger 's wisdom too. We will miss it.


Merging as this is blocking for an upcoming PR #25738 , will address future comments in a follow up PR!
Packing without cross-contamination
Author: ToddMorrill
Pull request: no

### Feature request

Is there something within Hugging Face that prevents latter subsequences from attending to earlier subsequences when you use packing? Is there a way to implement attention masking so that subsequences only attend to tokens within the subsequence within a packed example?

As it currently stands:
1. the attention mask fed into transformers is a 1d sequence and we need to be able to pass a 2d sequence to specify the appropriate attention mask with multiple sequences
2. this will interact with positional embeddings, because the position should be relative to the start of the example, not the sequence it's packed into
3. this will impact the loss calculation at the boundaries of examples. In particular, EOS tokens shouldn't have loss calculated for predicting the start of the next example.
4. and there may be other impacts I'm not thinking of.

There appear to be a few challenges to overcome but nevertheless it seems like an important feature to have.

### Motivation

I find it unsettling that when packing, we're just simply letting the latter subsequences' tokens attend to the first subsequences tokens. Packed sequences could have nothing to do with one another and I don't want to contaminate examples. At the same time, I don't want to give up the throughput gains of packing sequences.

I suppose I could sort my dataset by length to minimize the wasted computation (i.e. pack approximately equal length examples into batches together) as a decent solution. I'm not sure if this will impact model performance in any way though.

This feature request has been raised several times: https://github.com/huggingface/trl/issues/302
https://github.com/huggingface/transformers/issues/17726
I think tensorflow implements this and GraphCore talks about it [here](https://www.graphcore.ai/posts/introducing-packed-bert-for-2x-faster-training-in-natural-language-processing) and in their [paper](https://arxiv.org/abs/2107.02027).

### Your contribution

This doesn't strike me as a "first contribution" but if someone wants to coach me, I can give it a shot.

Comments:
Also in #6661

Hi @ToddMorrill

For existing models, I am afraid that it's unlikely to make changes for this feature. If there are new models that support this natively in the original modeling code, that could be the case when that model is ported into `transformers`.

It’s a real bummer because it seems like an important feature to have.
ALIBI position embedding support for other models ( BERT, ELECTRA )
Author: theblackcat102
Pull request: no

### Feature request

Since BLOOM is using ALIBI for position embedding, is there any plans to add ALIBI to other existing models such as BERT, ELECTRA etc?

ALIBI have so far proven to be [extrapolate sequence length beyond training sequence](https://arxiv.org/pdf/2108.12409.pdf) (figure 4). I think this idea should be added to existing architecture since existing models such as BERT, ELECTRA already supports more than one types of position embedding strategy.

### Motivation

If existing models such as BERT, Roberta or ELECTRA supports ALIBI position strategy, this will motivate community pretrained BERT which supports beyond 512 sequence length. One of the largest impact would be long form input sequence in summarization task.

### Your contribution

I have implemented ALIBI mechanism on my own private transformer fork and could submit a PR if this idea is accepted.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@theblackcat102 I would be quite interested in that. Could you also link your fork?

I would be quite interested to train BERT et al for longer sequences. 

@nreimers, you may take a look on MosaicAI's composer package. It provides a simple way to do a network surgery to add ALiBi into either BERT (RoBERTa) or GPT models.

I tried on GPT-2 model and it changes position embedding quite easily.  The below figure compares two models (blue w/ default PE while orange w/ ALiBi PE) that are trained on L=256 tokens but tested on longer sequences. We can find ALiBI supports "train short while test long" as described in their paper.

![image](https://github.com/huggingface/transformers/assets/4459248/d8cdc706-37b0-48f4-ac97-e2239765f312)


https://docs.mosaicml.com/projects/composer/en/stable/method_cards/alibi.html

Also, tried the similar idea on BERT-base model, however, when using sin/cos absolute PE, I cannot find the perplexity explosion effect as shown in GPT-2 above. Any suggestion for finding a good metric to measure a MLM model?

```shell
len: 256 ==> PPL: 1.0095623592512963
len: 257 ==> PPL: 1.009994390628342
len: 512 ==> PPL: 1.0602131151848846
len: 768 ==> PPL: 1.0771209222866058
```
host memory still occupied after huggingface model deleted
Author: linlifan
Pull request: no

### System Info

mainline

### Who can help?
@sgugger 

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
import torch 
from transformers import AutoConfig, AutoTokenizer, BloomForCausalLM, AutoModelForCausalLM
import time
import psutil
import gc

def get_host_memory():
    memory_allocated = round(psutil.Process().memory_info().rss / 1024**3, 3)
    print("cpu"," memory used total: ", memory_allocated, "GB")


def load_and_run():
    s_ = time.time()
    config = AutoConfig.from_pretrained('bigscience/bloom-560m')
    model = AutoModelForCausalLM.from_config(config)
    e_ = time.time()
    print("model loading time: %f " % (e_ - s_))
    get_host_memory()
    del model
    del config
   
load_and_run()
gc.collect()
for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
                print(type(obj), obj.size())
        except:
            pass
get_host_memory()

### Expected behavior

right now:
after model loaded:
cpu  memory used total:  3.327 GB
after model deleted:
cpu  memory used total:  3.364 GB

expected, host memory should be freed after model deleted.
```

Comments:
Hey! Would you mind asking this question on [the forum](https://discuss.huggingface.co/)? 

> Hey! Would you mind asking this question on [the forum](https://discuss.huggingface.co/)?

thank you @ArthurZucker  .  I move the question to the forum.
🌐 [i18n-KO] Translated `visual_question_answering.md` to Korean
Author: wonhyeongseo
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다! -->
# What does this PR do?

Translated the `visual_question_answering.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에, 이 아래에 리뷰를 요청할 팀원들을 멘션해주세요! -->
Team OSSCA, may you please review this PR? @bolizabeth, @nuatmochoi, @heuristicwave, @mjk0618, @keonju2, @harheem, @HongB1, @junejae, @54data, @Sunmin0520, @seank021, @augustinLib, @sronger, @TaeYupNoh, @kj021, @eenzeenee

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
May you please review this PR? @sgugger, @ArthurZucker, @stevhliu

Comments:
안녕하세요!

🔥 번역에 참여해주신 모든 분들께 정말 감사드립니다. 이번 번역에서는 특정 이미지에 대해서 설명이 나와 보면서 번역이 알맞는지 확인해야했습니다. 오랜만에 번역하는 것이다보니 오류가 많을 것으로 예상되는데요. 어떤 부분이든 모두 주저없이 말씀해주세요 :)

리뷰는 일방적인 수용이 아닌 서로간의 대화를 전제로 이루어집니다. 때문에 "어라, 이건 내가 맞아!"라고 주장해주시면 오히려 더 재밌는 리뷰 시간이 될 것입니다. 지금까지 최고의 리뷰를 꾸준히 올려주시는 @nuatmochoi 님 🥇 , @heuristicwave 님 🥈 , @sim-so 님 🥉 도 모두 리뷰 자유롭게 올려주세요. 감사합니다. 💯 😆

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25679). All of your documentation changes will be reflected on that endpoint.
[`PEFT`] Fix PeftConfig save pretrained when calling `add_adapter`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes a tiny issue when building a demo for @NielsRogge in https://github.com/huggingface/peft/issues/796 
when attaching a fresh new adapter, we need to manually add the attribute `base_model_name_or_path` otherwise we can't load back the base model.

We did not flagged it before in the tests because we were only testing the case when we call `load_adapter` then `save_pretrained` which works fine because we attach an existing PeftConfig.

Added also a nice test

cc @ArthurZucker @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add conversion support for CodeLlama 34B
Author: AlpinDale
Pull request: yes

There hasn't been any architectural changes, and I was able to convert the 34B by simply adding in the intermediate size for the model. I have [converted the 34B model](https://huggingface.co/alpindale/CodeLlama-34B-hf) and it appears to work well.

# What does this PR do?

This PR should make it possible to convert the [CodeLlama](https://ai.meta.com/research/publications/code-llama-open-foundation-models-for-code/) models, recently released by Meta AI.




Comments:
Doesn't seem to work for 7B and 13B variants.

Conversion doesn't account for the modified tokenizer and the new rope theta value. Closing the PR until someone else implements the new changes.

See #25740
Support specifying revision in push_to_hub
Author: jmif
Pull request: yes

Resolves https://github.com/huggingface/transformers/issues/22867 by adding revision to `create_commit` call and proactively creating the branch before committing.

Change hasn't been discussed or approved in an issue AFAIK.  Haven't written tests but happy to once the general approach is approved.  I've tested manually and the changes work.

Comments:
cc @sgugger 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25578). All of your documentation changes will be reflected on that endpoint.

Thanks again!

@jmif what is the syntax for using the revision?

I tried:
```
model.push_to_hub(adapter_model_name, revision="GPTQ")
```
but that just pushed to main.

@RonanKMcGovern could you open a new issue with the output of `transformers-cli env` and a snippet for reproduction?
Seems to be a bug in documentation with Speech Encoder Decoder Model Training
Author: greeshmasmenon
Pull request: no

### System Info

@sanchit-gandhi  - Can you please take a look at the below?

The documentation [here](https://huggingface.co/docs/transformers/main/model_doc/speech-encoder-decoder) doesn't seem to execute. It looks like there is a call being made to `loss = model(**input_features).loss` when input_features has not been initialized yet. 

```
from transformers import AutoTokenizer, AutoFeatureExtractor, SpeechEncoderDecoderModel
from datasets import load_dataset

encoder_id = "facebook/wav2vec2-base-960h"  # acoustic model encoder
decoder_id = "bert-base-uncased"  # text decoder

feature_extractor = AutoFeatureExtractor.from_pretrained(encoder_id)
tokenizer = AutoTokenizer.from_pretrained(decoder_id)
# Combine pre-trained encoder and pre-trained decoder to form a Seq2Seq model
model = SpeechEncoderDecoderModel.from_encoder_decoder_pretrained(encoder_id, decoder_id)

model.config.decoder_start_token_id = tokenizer.cls_token_id
model.config.pad_token_id = tokenizer.pad_token_id

# load an audio input and pre-process (normalise mean/std to 0/1)
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values

# load its corresponding transcription and tokenize to generate labels
labels = tokenizer(ds[0]["text"], return_tensors="pt").input_ids

# the forward function automatically creates the correct decoder_input_ids
loss = model(**input_features).loss
loss.backward()
```

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Execute the code anywhere. It is an official example.

### Expected behavior

To execute and show the backward loss. Instead, you get the error - `NameError: name 'input_features' is not defined
`

Comments:
The line where the model takes in the input features should actually be:

`loss = model(input_values=input_values, labels=labels).loss`

Hey! Thanks for noticing 🤗 Would you  like to open a PR and fixe this? You could also try running the doctests to find if there are other parts that are not up to date 

Sure, I will submit a PR by the end of the week.

On Wed, Aug 23, 2023 at 7:59 AM Arthur ***@***.***> wrote:

> Hey! Thanks for noticing 🤗 Would you like to open a PR and fixe this? You
> could also try running the doctests to find if there are other parts that
> are not up to date
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/25673#issuecomment-1689327003>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AYNGKNFMG5WEYMM2TSEORUTXWWL3TANCNFSM6AAAAAA32UADQA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


-- 
Thanks and Regards,
Greeshma S Menon


Awesome, thanks @greeshmasmenon!