noamaanMulla-03/transformers-issues huggingface dataset:
Generate: multi-device support for contrastive search
Author: gante
Pull request: yes

# What does this PR do?

Fixes #24634 

In multi-gpu settings, the past KV cache may be scattered across devices -- the cache corresponding to a layer sits in the same device as the layer itself, and different layers may be in different devices.

In contrastive search, we must apply indexing operations on the past KV cache. The indexes are in a tensor, which sits on the same device as the model outputs by default. Applying these indexes on the past KV cache currently results in an exception if the model is split across devices (see the issue linked above).

This means we either move the indexing tensor to all possible devices or keep the tensor on CPU. Indexing is typically CPU-heavy on PyTorch, so the benchmarks on my end indicate that moving the indexing tensor to the CPU enables multi-device contrastive search without noticeable throughput degradation 🙌 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

For future reference, here's the benchmark code:
```py
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
from tqdm import tqdm

# Other configuration options
DEVICE = "cuda:0"
NUM_RUNS = 10
MAX_NEW_TOKENS = 1000
TEXT_INPUT = "def sieve_of_eratosthenes():"

# Load the model and prepare generate args
repo_id = "huggyllama/llama-7b"
model = AutoModelForCausalLM.from_pretrained(repo_id, device_map="auto", load_in_4bit=True)

assistant_model = None
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
model_inputs = tokenizer(TEXT_INPUT, return_tensors="pt").to(DEVICE)

generate_kwargs = {
    "max_new_tokens": MAX_NEW_TOKENS,
    "top_k": 10,
    "penalty_alpha": 0.6,
}

# Warmup
print("Warming up...")
for _ in range(2):
    gen_out = model.generate(**model_inputs, **generate_kwargs)
print("Done!")


# Measure OR Stream
def measure_generate(model, model_inputs, generate_kwargs):
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)
    torch.cuda.reset_peak_memory_stats(DEVICE)
    torch.cuda.empty_cache()
    torch.cuda.synchronize()

    start_event.record()
    for _ in tqdm(range(NUM_RUNS)):
        gen_out = model.generate(**model_inputs, **generate_kwargs)
    end_event.record()

    torch.cuda.synchronize()
    max_memory = torch.cuda.max_memory_allocated(DEVICE)
    print("Max memory (MB): ", max_memory * 1e-6)
    print("Throughput (tokens/sec): ", (NUM_RUNS * MAX_NEW_TOKENS) / (start_event.elapsed_time(end_event) * 1.0e-3))

measure_generate(model, model_inputs, generate_kwargs)
```

On my end, with a RTX3090, I get 150 tokens/s before and after these changes.

@gante Thanks for adding the script! ❤️
Generate: force cache with `inputs_embeds` forwarding
Author: gante
Pull request: yes

# What does this PR do?

Fixes the issue raised in [this comment](https://github.com/huggingface/transformers/issues/23042#issuecomment-1618513599).

The issue and the solution is described in the comment added alongside the change :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
LoRA training adapter_model.bin is 888 bytes always
Author: kallewoof
Pull request: no

### System Info

Ubuntu 22.04
Python 3.10.11
transformers 4.30.2
peft 0.4.0.dev0
accelerate 0.20.3

### Who can help?

@ArthurZucker 
@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code is at https://github.com/kallewoof/alpaca-lora/blob/202306-ooba-imports/finetune.py

You need a text file with text content (i.e. not instruction-based). Anything goes.

Run the above script: `python ./finetune.py --base_model=MODEL --raw_data_path=TEXTFILE --batch_size=32 --micro_batch_size=1 --num_epochs=2 --lora_r=128 --lora_alpha=256 --cutoff_len=512 --overlap_len=256 --save_steps=1`


### Expected behavior

Expected: After one (each) iteration, a checkpoint should be saved with an adapter_model.bin file that contains the LoRA weights.

Got: The checkpoints are made, but the adapter_model.bin file is only 888 bytes and does not grow (there are two versions; both are wrong, one contains only the LoRA stuff, the other one contains optimizer.pt, rng_state.pth, scheduler.pt, etc. which I have no idea why they are saved for a LoRA.)

Note: using the exact same conda environment, Oobabooga is able to generate LoRAs after loading a model in 4 bit with double quant, where the above finetune.py fails to do so. I have also verified that the resulting raw data dataset is identical between the two code bases.

Comments:
cc @younesbelkada 

I backtracked until I was on the original tloen/alpaca-lora script, using the original dataset on int8 model, and I am still seeing the 888 byte adapter_model.bin file, along with a bunch of other files that you normally don't see in a LoRA output (optimizer.pt, etc).

I am closing this as I don't think I will be able to provide adequate feedback for a clean fix, and I've moved on to a different approach. Sorry for wasted time.
Causal language modeling documentation is wrong? 
Author: JoaoLages
Pull request: no

I just noticed that [on this page](https://huggingface.co/docs/transformers/tasks/language_modeling) we do not add any end-of-speech token (EOS) to the end of the texts.
This means we are training a model that does not shut up! The EOS token should be added!

Comments:
Thanks a lot! Transfering this to transformers

It's just an example that we keep as simple as possible. You can customize it to your needs for your own trainings.

An example that is **wrong**, let's not try to argue that it isn't 😅 . In that example, you are interested in training to generate sentences, but you are actually training the model to never stop generating...

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update perf_train_gpu_one.mdx
Author: aasthavar
Pull request: yes

# What does this PR do?

Minor changes - Corrected a word's spelling. Changed markdown syntax for heading and for URL to be seen as a link.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR. 

@sgugger, @stevhliu and @MKhalusova


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23101). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`torch.compile` is ignored when using DeepSpeed
Author: xplip
Pull request: no

### System Info

Since #22279, `torch.compile` is called at the end of `_wrap_model`. However, [these lines](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#LL1397C1-L1398C34) immediately return the deepspeed engine, so `torch.compile` is never executed, even when asking for it in the training args. I don't think this is intended because DeepSpeed does not automatically run `torch.compile`, but please correct me if I am wrong. @stas00 @sgugger 

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install transformers and deepspeed with torch 2.0
2. Run any HF transformers training script with `--deepspeed ds_config.json --torch_compile`
3. See logs, no `torch.compile` logs to be found. this means training is slower with zero 0 than without using deepspeed (due to the lack of speedup from compilation)

### Expected behavior

`torch.compile` should still be called somewhere in the trainer when using DeepSpeed 

Comments:
I don't think DeepSpeed support `torch.compile` yet, so this was done intentionally. If the situation has changed, we can of course revisit.

What Sylvain said, wrt the 2 not working together.

But it's not that Deepspeed doesn't support `torch.compile`, it's rather that `torch.compile` is very immature - somewhere between alpha and beta state based on my experiments - many other things break with `torch.compile` besides Deepspeed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Simplifying Output from Text Classification Pipelines
Author: 907Resident
Pull request: no

### Feature request

> This feature request references the content discussed in a [HF thread](https://discuss.huggingface.co/t/i-have-trained-my-classifier-now-how-do-i-do-predictions/3625).

I was just wondering if there is a particular reason why the output of the `pipe` shown above is a double list?

For instance, the output of the following:

```python
from transformers import TextClassificationPipeline

model = ...
tokenizer = ...
pipe = TextClassificationPipeline(model=model, tokenizer=tokenizer, return_all_scores=True)
# outputs a list of dicts like [[{'label': 'NEGATIVE', 'score': 0.0001223755971295759},  {'label': 'POSITIVE', 'score': 0.9998776316642761}]]
res = pipe("I love this movie!")

```

is a list that has _two_ square brackets (i.e., `[[ ... ]]`). This means the indexing process to grab say the negative score requires:

```python
neg = res[0][0]["score"]
```

This could be enhanced by simply returning a single dictionary object:

```python
res = {"label":["NEGATIVE", "POSITIVE"], "score":[0.0001, 0.9998]}

```

### Motivation

This idea came from reading a [HF discussion thread](https://discuss.huggingface.co/t/i-have-trained-my-classifier-now-how-do-i-do-predictions/3625). It was two years ago, so I did not want to reopen the conversation there. 

Also, I think this is a feature addition, but feel free to correct me if I am wrong. 

### Your contribution

I do not currently have plans to submit a PR, but if there is interest from the HF team, then I will take a harder look and comment here if I can make the change and submit a PR.

My initial guess is that this is not something bothers many users. 

Comments:
This is also something we cannot change without breaking the code of many many users :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix audio feature extractor deps
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

The PR #21998 refactored many of the audio feature extractors to use a numpy backed for log Mel feature extraction (as opposed to `torchaudio` as was done previously). However, some of the feature extractors still required the `"speech"` backend for import, which states `torchaudio` as its sole dependency:
https://github.com/huggingface/transformers/blob/6eedfa6dd15dc1e22a55ae036f681914e5a0d9a1/src/transformers/utils/import_utils.py#L648-L650

This PR updates these four feature extractors to no longer require `"speech"`, since they're now numpy only.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
documentation_tests.txt - sort filenames alphabetically
Author: amyeroberts
Pull request: yes

# What does this PR do?

Reorganises the file names listed in `documentation_tests.txt` so that they are in alphabetical order. This is to address two things: 
* Make it obvious where to add new files
* Make it easier to spot if certain files are missing. For example, I didn't notice until recently that modleing_imagegpt.py wasn't included (its config was). 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh I added a quick check in `utils/check_doctest_list.py` - let me know what you think :)

Yes! Eventually, we can provide a fix option to modify the file to sort and deduplicate the lines. But again, the PR itself is complete and can be merged already.
Pytorch type hints
Author: IMvision12
Pull request: yes

# What does this PR do?

Added Type hints

## Who can review?
@Rocketknight1 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20112). All of your documentation changes will be reflected on that endpoint.

Sure, I will add all type hints for pytorch and ping you

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20112). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20112). All of your documentation changes will be reflected on that endpoint.

@Rocketknight1 I think I've covered all PyTorch models!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20112). All of your documentation changes will be reflected on that endpoint.

@IMvision12 Amazing, thank you! I just finished reviewing and it all looks good, so I'm going to merge now. Once we have full type hint coverage we can add tests to ensure that it stays that way in future, and then start using the type hints in library checking, so this should help a lot!
Handle `g_state` in RWKV's customized CUDA kernel to overcome sequence length limitation
Author: Blealtan
Pull request: no
Labels: name: Feature request

### Feature request

Handle `g_state` in RWKV's customized CUDA kernel enables backward pass with a chained forward. As such, the maximum `context_length` will not hinder longer sequences in training, and the behavior of WKV backward is coherent with forward.

For BF16 kernels, see [here](https://github.com/Blealtan/RWKV-LM-LoRA/tree/dev-infctx/RWKV-v4neo/cuda). Credits to icecuber on RWKV Discord channel (searching for `chunked GPT mode` in the history will show the original code).

### Motivation

The current implementation of RWKV dedicates to a `max_seq_length`, propagating the sequence length parameter down to the CUDA kernel. It can be problematic with longer input sequences. By supporting `g_state` backward, we can fix the maximum sequence length inside CUDA kernel and instead call it several times until the complete sequence gets processed. Also, given the forward pass already supports state chaining, the backward should also support this.

> Some not so related advertising:
> In [my recent experiments](https://github.com/Blealtan/RWKV-LM-LoRA/tree/dev-infctx), I'm building upon the state chaining functionality (or chunked GPT mode, per icecuber's wording) to achieve near-constant VRAM training with arbitrary sequence length. The basic idea is to do forward pass of the entire model once a piece and perform checkpointing for each piece, so that at the cost of the forward pass repeated twice we get any long sequence trained within fixed VRAM. If `g_state` is supported in `transformers`, it will be easy to port that here.

### Your contribution

I can help by submitting the PR, but only later. I'm not locking that in case anyone has the time earlier than me.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update warning messages reffering to post_process_object_detection
Author: rafaelpadilla
Pull request: yes

# What does this PR do?

Noticed that `post_process` will be replaced by `post_process_object_detection` in v5.
However, the (old) `post_process` does not threshold the bounding box scores (it has the same effect if using `threshold=0`).
But the (new) `post_process_object_detection` has a threshold parameter which, depending on the model, has different default values.
When this change occurs, users will have fewer boxes detected if the default threshold of `post_process_object_detection` is not `0`.

This PR includes:
1) Alerting the usage of threshold in existing warning messages of vision models, so when users stop calling `post_process` and start calling `post_process_object_detection`, their results will not be affected.   
2) Changing `owlvit.md` as it was not making usage of the (new) `post_process_object_detection`. 

I searched for other .md files and docstrings that will be affected when `post_process` stops working. But noticed that only `owlvit.md` will produce wrong results if not calling `post_process_object_detection` with the correct threshold. All others (e.g. `modeling_conditional_detr.py`, `modeling_deformable_detr.py`, `modeling_deta.py`, `modeling_detr.py`, `zero_shot_object_detection.md`, etc) already explicitly use a threshold and won't be affected.


## Before submitting
- [ x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add `finetuned_from` property in the autogenerated model card
Author: sgugger
Pull request: yes

# What does this PR do?

We already extract the info but it wasn't properly put in the metadata. This PR adds it.
cc @julien-c @osanseviero for the exact name of the tag to use.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

What did you have in mind exactly? The only text added is 
```
"This model is a fine-tuned version of f" [{self.finetuned_from}](https://huggingface.co/{self.finetuned_from}) on "
```
which doesn't contain the `finetuned_from` tag.

@sgugger i would rename the variable everywhere

That's a breaking change in Transformers for cosmetic reasons only, so a big no no from my side.

I would go ahead with this PR as-is :fire:
Make warning disappear for remote code in pipelines
Author: sgugger
Pull request: yes

# What does this PR do?

Currently, loading a pipeline with a model that has its code on the Hub will result in a warning that the model is not in the right auto class. This PR adds the custom class in the auto mapping so that this warning is not triggered.

Related to #24598

Comments:
_The documentation is not available anymore as the PR was closed or merged._