noamaanMulla-03/transformers-issues huggingface dataset:
Need more understanding of the function: get_visual_embeddings(image_path)
Author: dheeman00
Pull request: no

Dear Developers

I was looking through the code for image embedding and found this section of the code in the visual_bert:

# this is a custom function that returns the visual embeddings given the image path
visual_embeds = get_visual_embeddings(image_path)

This is a user-defined function. But I want to know what are the existing imaging embedding process should work with visual_bert?

Some suggestions will be helpful.

Thanks
Dheeman

Comments:
Hey @dheeman00!

You should check out the following complete example for a fully working pipeline with VisualBERT: https://github.com/huggingface/transformers/tree/master/examples/research_projects/visual_bert

cc @gchhablani

Hey @LysandreJik 

I looked into the repo. But I think I found a better example that can be helpful for others.

https://huggingface.co/docs/transformers/model_doc/deit

deit directly represents a vector from a given image. 

cc @NielsRogge too

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> Hey @LysandreJik
> 
> I looked into the repo. But I think I found a better example that can be helpful for others.
> 
> https://huggingface.co/docs/transformers/model_doc/deit
> 
> deit directly represents a vector from a given image.

Hey @dheeman00

You said _deit directly represents a vector from a given image. I would like to know if you can help me to understand which part is creating the visual embedding because I'm kinda lost. I wonder if you can bring me some help finding the `visual_embeds = get_visual_embeddings(image_path)` equivalent.

I saw that the forward operation return 

`hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, num_channels, height, width).`

So if I do:

```
from transformers import AutoImageProcessor, LevitModel
import torch
from datasets import load_dataset

dataset = load_dataset("huggingface/cats-image")
image = dataset["test"]["image"][0]

image_processor = AutoImageProcessor.from_pretrained("facebook/levit-128S")
model = LevitModel.from_pretrained("facebook/levit-128S")

inputs = image_processor(image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs, output_hidden_states=True)

hidden_state= outputs.hidden_state
embedding = list(last_hidden_states.shape)[0]
```
embedding will contain the vector representation of the image in a vector space?

Thank you in advance, I would appreciate your help :D
Upload (daily) CI results to Hub
Author: ydshieh
Pull request: yes

# What does this PR do?

Let's start to upload the results to a Hub dataset [hf-internal-testing/transformers_daily_ci](https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/tree/main/2024-05-31) and use this dataset to build a visualization dashboard 🚀 !

So far I take the advantage of the daily CI workflow run - upload directly from there instead of fetching the finished workflow run afterward.

I didn't change any format in the raw reports (but processed in the workflow), but I believe it contains everything required to perform visualization/analysis.

[A dummy run](https://github.com/huggingface/transformers/actions/runs/9320789057) gives [hf-internal-testing/transformers_daily_ci](https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/tree/main/2024-05-31)
 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31168). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review. I will merge once I applied the same changes to

> utils/notification_service_quantization.py

(sorry, I forget this file quite frequently)
Rename sanity_evaluation to eval_on_start
Author: Qubitium
Pull request: yes

# What does this PR do?

Rename ambiguous training.arg `sanity_evaluation` to `eval_on_start`.

Reasons:

1. The new training arg name is not clear at all. What is `sanity`? Who decides on sanity? The actual code is quite useful and simple but the var name doesn't actually reflect what the code does. 
2. The new `eval_on_start` is clear on the intention without the need to peek at doc. 

Options on naming:

1. Based on existing eval args and naming convention I chose `eval_on_start`. `evaluation_on_start` was considered but since `eval_` is non-ambigious and already part of the arg list, it is more concise. 


## Who can review?

@muellerzr and @SunMarc




Comments:
@amyeroberts it hasn't been released yet, so we're fine. Only merged 3 days ago: https://github.com/huggingface/transformers/pull/31146

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31192). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue with model.generate returning outdated hidden states for earlier tokens
Author: Joe-Hall-Lee
Pull request: no
Labels: name: Generation

### Description
I have been using the generate method from transformers library to generate text and retrieve hidden states. However, I encountered an issue with the output of the hidden states in the model.generate method. Specifically, it seems like the hidden states for previously generated tokens are not updated when a new token is generated.
Here's the code I am using:

```
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch


def generate_with_final_hidden_state(model, input_ids, attention_mask, max_output_length, tokenizer):

    model.eval()

    outputs = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        max_new_tokens=max_output_length,
        output_hidden_states=True,
        return_dict_in_generate=True,
        temperature=0.0,
        top_p=1.0
    )

    generated_ids = outputs.sequences
    hidden_states = outputs.hidden_states

    last_layer_hidden_states = [state[-1] for state in hidden_states]

    for idx, state in enumerate(last_layer_hidden_states):
        print(f"Shape of hidden state at step {idx}:", state.shape)

    final_hidden_state = torch.cat(last_layer_hidden_states, dim=1)

    if generated_ids.shape[1] > final_hidden_state.shape[1]:
        generated_ids = generated_ids[:, :-1]

    batch_size, total_length = generated_ids.shape

    mask = torch.ones(batch_size, total_length,
                      dtype=torch.long, device=input_ids.device)
    input_length = input_ids.shape[1]
    mask[:, :input_length] = 0

    return generated_ids, final_hidden_state, mask

# Other parts of the code are omitted for brevity.

```
Upon inspecting the shape of hidden_states, I observed the following:

- hidden_states is a tuple of length new_tokens + 1.

- Each element in hidden_states is also a tuple of length num_hidden_layers + 1.

- For the last layer, the first Tensor has dimensions [batch_size, input_length, hidden_size], and the remaining Tensors have dimensions [batch_size, 1, hidden_size].

This indicates that the hidden state of newly generated tokens is appended to hidden_states, but the hidden states of earlier tokens are not updated accordingly. As a result, directly concatenating these hidden states to form an embedding may not be appropriate because the hidden states of earlier tokens may not be the latest.
### Expected behavior
When generating sequences, it would be more meaningful if the hidden states for all tokens up to the current step were updated and returned. This would ensure that each token's hidden state is up-to-date, reflecting the current context and generated tokens.
### Environment
transformers version: 4.31.0
Platform: Linux
Python version: 3.10.12
PyTorch version: 2.1.2

Comments:
cc @gante @zucchini-nlp 

I've realized that I made a mistake in my previous description. The hidden states of previously generated tokens are not supposed to be updated when generating each new token. Apologies for the confusion.
NotImplementedError
Author: vincent507cpu
Pull request: no

### System Info

I was trying to run **https://github.com/dvlab-research/LongLoRA/blob/main/fine-tune.py**, but I first encountered `ValueError: Tokenizer class ChatGLMTokenizer does not exist or is not currently imported.` (it's been solved by switching line 135 and line 141 snippests). Then I had a new error (please see below).

Could anyone help me? Thanks a lot!

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`python fine-tune.py --output_dir ./output_models/chatglm3-6b-32k`

### Expected behavior

```
/home/zwj/GitHub/LongLoRA-main/llama_attn_replace.py:464: UserWarning: Flash attention is only supported on A100 or H100 GPU during training due to head dim > 64 backward.ref: https://github.com/HazyResearch/flash-attention/issues/190#issuecomment-1523359593
  warnings.warn(
Loading checkpoint shards: 100%|█████████████████████████████████████████████| 7/7 [00:02<00:00,  3.38it/s]
Traceback (most recent call last):
  File "/home/zwj/GitHub/LongLoRA-main/fine-tune.py", line 217, in <module>
    train()
  File "/home/zwj/GitHub/LongLoRA-main/fine-tune.py", line 165, in train
    smart_tokenizer_and_embedding_resize(
  File "/home/zwj/GitHub/LongLoRA-main/fine-tune.py", line 81, in smart_tokenizer_and_embedding_resize
    model.resize_token_embeddings(len(tokenizer))
  File "/home/zwj/miniconda3/envs/longlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1811, in resize_token_embeddings
    model_embeds = self._resize_token_embeddings(new_num_tokens, pad_to_multiple_of)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zwj/miniconda3/envs/longlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1832, in _resize_token_embeddings
    self.set_input_embeddings(new_embeddings)
  File "/home/zwj/miniconda3/envs/longlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1610, in set_input_embeddings
    base_model.set_input_embeddings(value)
  File "/home/zwj/miniconda3/envs/longlora/lib/python3.11/site-packages/transformers/modeling_utils.py", line 1612, in set_input_embeddings
    raise NotImplementedError
NotImplementedError
```

Comments:
Hi Vincent,
I've meet the same problem,  which also raise with no set_input_embeddings. Could you please tell me how to fix it?
Thanks.
Diff converter v2
Author: ArthurZucker
Pull request: yes

# Introducing `diff_converter`, a linter to go from a modular file to a single model single file.

The `diff_converter` tool is here to replace our old `Copied from` statements, while keeping our core `transformers` philosophy:
- single model single file
- explicit code
- standardization of modeling code
- readable and educative code
- simple code
- least amount of modularity

This additionally unlocks the ability to very quickly see the differences between new architectures that get developed. While many architectures are similar, the "single model, single file" policy can obfuscate the changes. With this diff converter, we want to make the changes between architectures very explicit. 

## What it is:

It's a Python script that converts a `diff_my_model.py` file into `configuration_my_model.py` and `modeling_my_model.py` files using [AST](https://docs.python.org/3/library/ast.html). 

The `diff_my_model.py` file is a new way to write new models using inheritance (which had always been banned from `transformers`) to simplify model addition and improve readability. 

The **converter**, allows us to "unpack" the inheritance and obtain the traditional "single-file" `modeling` file. 
This diff is not to be imported from nor used directly, and isn't present in the inits of `transformers`. It will likely be renamed to `_diff_my_model.py` to clarify this; however, it can be used to debug the model implementation while contributing a new model.

## Supported syntax:

The `examples` folder holds examples of syntax that are usable, most notably:
- `super_kwargs` to replace with the parent class' kwargs
- `super().__function__()` calls, which will be replaced by the source code of the parent class, de-duplicating similar code
-  inheritance: base class dependencies are automatically fetched and written to the modeling file, including renamed decorators.

⚠️⚠️⚠️ This feature is in its infancy. It will be developed over the new few months in order to smoothen out the contribution of new models. ⚠️⚠️⚠️

If the conversion of you diff file does not go as planned / you encounter any issue with it, ping @ArthurZucker 🤗 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30868). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts everything is adressed, merging and can adresse more stuff later on!

FIxed by #31202
Fixing `name 'torch' is not defined` in `bitsandbytes` integration
Author: jamesbraza
Pull request: yes

With `transformers==4.41.2` and this:

```python
from transformers import AutoModelForCausalLM, QuantoConfig

AutoModelForCausalLM.from_pretrained(
    "microsoft/phi-2", low_cpu_mem_usage=True, quantization_config=QuantoConfig(weights="int8")
)
```

I hit this traceback:

```none
  File "/Users/user/code/repo/venv/lib/python3.12/site-packages/transformers/integrations/bitsandbytes.py", line 331, in <module>
    def dequantize_bnb_weight(weight: torch.nn.Parameter, state=None):
                                      ^^^^^
NameError: name 'torch' is not defined
```

---

An alternate would be requiring importers to first run `import torch`

Cc @younesbelkada @SunMarc @amyeroberts @ArthurZucker based on recent PR reviewers for this file

Comments:
Failing test is unrelated! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31243). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Adaptive Decoding Support
Author: zwhong714
Pull request: no
Labels: name: Feature request

name: Generation

### Feature request

Adaptive decoding balances the diversity and coherence of open-ended text generation, making it an excellent method compared to top-k, top-p, and other decoding algorithms. For more details, the paper is available [here](https://arxiv.org/abs/2402.18223), and the code can be accessed [here](https://github.com/zwhong714/adaptive_decoding).

### Motivation

During the generation process, the distribution predicted by the language model generally falls into two categories. The first is a flattened distribution, indicating that the LM has multiple potential choices for the next token. The second is a sharp distribution, suggesting that the model's options are more limited. Ensuring that the model dynamically understands the current state is crucial for generating highly diverse and coherent sentences.

### Your contribution

idea done.

Comments:
cc @gante
Reenable SDPA's FA2 During Training with torch.compile
Author: warner-benjamin
Pull request: yes

This PR resolves #30010 and completes #30070 by reenabling the SDPA Flash Attention 2 kernel for `torch.compile` when the model is training. During eval, SDPA dispatches to the efficient kernel with the same logic as in #30070.

This PR will prevent SDPA Attention models from using a low amount of memory during training in eager mode but using a large amount or OOM'ing when compiling due to using the wrong SDPA kernel. It shouldn't affect exporting or generation when the model is in eval mode.

Moving the `is_causal` dispatch logic from inline to an if statement is required to support both `fullgraph=True` and `dynamic=True`. The current code errors out with `dynamic=True` due to `q_len > 1` not being the correct bool type. But wrapping it in a bool `bool(q_len>1)` to fix dynamic breaks `fullgraph=True`.

The Llama tests that I could run either all pass or fail in the same state as on main (`LlamaIntegrationTest::test_conversion` & `LlamaIntegrationTest::test_compile_static_cache`). I couldn't run Gemma tests due to a model gating error despite having access to Gemma.

Comments:
Tagging @ArthurZucker and @younesbelkada for review.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30442). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

fyi @fxmarty when you come back

Not sure why the CI errored out after these formatting changes. Locally I still have `LlamaIntegrationTest::test_conversion` & `LlamaIntegrationTest::test_compile_static_cache` failing, both which also fail on main. Every other run slow test passes for Llama.

@warner-benjamin can you make sure the CI is green? Failing ones seem unrelated, maybe merging main will do

This PR causes a dynamo graph break at `torch.all(attention_mask == 1)` when running with torch.compile in training mode, due to the dynamic control flow. Is there a way to get around this?

@tombousso Yes. Why is it an issue for you? Do you see perf degradation?

AFAIK there is no obvious way around it, maybe using newer APIs from https://github.com/pytorch/pytorch/pull/114823 & maybe other PRs

https://pytorch.org/docs/main/cond.html may be the way?

Yes, I was seeing perf degradation. I was hoping to get a graph with no breaks to make it easier to see what's going on, and to give the compiler the best opportunity to make optimizations.

@tombousso Could you open an issue for that?
Reorder methods added to Cache classes
Author: poedator
Pull request: yes

# What does this PR do?
Implements methods for StaticCache and DynamicCache to reorder along tokens axis and  resize(static only).
These are useful for cache compacting, specifically in speculative decoding applications.

testing:
somehow tests/models/llama/test_modeling_llama.py::LlamaModelTest::test_torch_fx fails because of `if pad > 0` in pad size calculations. See the test log here: https://gist.github.com/poedator/5f9fa03e853dcd8940e8bd40d64e2a9d
What would be the best way to avoid this error?

Tagging @gante @ArthurZucker @younesbelkada 


Comments:
Hi @poedator 👋  Thank you for opening the PR! 

As you know, `transformers` is a large library, so we must be conservative when adding new features to ensure we can maintain them. To be considered for addition, a feature must be one of the following:

1. It is a very small feature, with negligible code footprint and maintenance requirements;
2. It is a feature that is widely requested by the community, ideally linking requests for the feature;
3. The feature substantially improves a certain aspect of the library and/or model usage, and there is data to back the claim.

Unfortunately, I can't fit this feature in any of the above buckets at the moment. Let me know if the situation changes, or if you have data to support your proposed feature, based on the criteria above 🤗 (for instance -- if you are using these changes for a specific purpose, can you illustrate it with code?)

> Hi @poedator 👋 Thank you for opening the PR!
> ...
> Unfortunately, I can't fit this feature in any of the above buckets at the moment. Let me know if the situation changes, or if you have data to support your proposed feature, based on the criteria above 🤗 (for instance -- if you are using these changes for a specific purpose, can you illustrate it with code?)

These features would be useful in more advanced speculative decoding methods, where the Draft model  generates a tree of continuations, which is validated by Target model in single pass. Then there is a need to compact the cache of both models to keep only the needed tokens. 
I use a monkey patch in my code for this. Will call you to check it out once I publish it. 

@poedator yes please, ping me when you have results! If they are SOTA (or close to it), I'd be happy to add it!

(our apologies for being kind of gatekeepers -- maintenance will get out of hand if we are not picky with new features 🤗 )

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Benchmark GitHub Actions workflow
Author: ydshieh
Pull request: yes

# What does this PR do?

Benchmark GitHub Actions workflow.

[Workflow run page](https://github.com/huggingface/transformers/actions/runs/9318192661/job/25650027316)
[Dataset on the Hub](https://huggingface.co/datasets/hf-internal-testing/benchmark_results/tree/main)

(Not done yet)
- Maybe change decoding steps from 128 to 1024
- Run against Llama2 too

We are mostly only interested in `summary.json` file.

However, I am uploading the whole directory of an experiment in each run. The reason is to keep the benchmark config files available if we ever need to access this information. However, this seems a bit too much (too many files for which most of time they remain the same across different dates)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31163). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `MistralIntegrationTest`
Author: ydshieh
Pull request: yes

# What does this PR do?

- `test_speculative_generation`: it was failed due to 9efec114 (Jan 19 2024) then 2e27291c (May 13 2024). I would trust those 2 PRs and simply update the expected outputs (cc @gante )
- `test_model_7b_generation` and `test_model_7b_logits`: it is from my PR #29905 where I changed `dtype` and/or `load_in_4bit` in `from_pretrained` in the tests but forgot to update the expected output values for T4.

Also, 

```python
        del model
        backend_empty_cache(torch_device)
        gc.collect()
```
this is not helping and worse, with it, we actually get `GPU OOM` (here for `test_model_7b_long_prompt_sdpa`). After removing them, the tests are all passing now.

> 111 passed, 48 skipped, 276 warnings in 205.46s (0:03:25)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31231). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.