noamaanMulla-03/transformers-issues huggingface dataset:
Multi input of owlv2 cause RuntimeError: Boolean value of Tensor with more than one value is ambiguous
Author: zhangjingxian1998
Pull request: no
Labels: name: bug

name: Vision

### System Info

- `transformers` version: 4.41.1
- Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.1
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.0.dev20240318+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@jla524 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
import requests
from PIL import Image
from io import BytesIO
from transformers import AutoProcessor, Owlv2ForObjectDetection

processor = AutoProcessor.from_pretrained('google/owlv2-base-patch16-ensemble')
model = Owlv2ForObjectDetection.from_pretrained('google/owlv2-base-patch16-ensemble')

url = 'http://demandware.edgesuite.net/sits_pod41/dw/image/v2/AAJI_PRD/on/demandware.static/-/Sites-londondrugs-master/default/dw99e32df7/products/L8191009/large/L8191009.JPG?sw=556&amp;sh=680&amp;sm=fit'
response = requests.get(url, timeout=2)
image = Image.open(BytesIO(response.content)).convert('RGB')
target_sizes = [image.size, image.size]
texts = ["human"]

inputs = processor(text=[texts, texts], images=[image,image], return_tensors="pt")
outputs = model(**inputs)
results = processor.post_process_object_detection(outputs=outputs, threshold=0.1, target_sizes=target_sizes)
```
It error at
https://github.com/huggingface/transformers/blob/9d35edbb30625489bf286a9b15aed0c5a3119c1c/src/transformers/models/owlv2/image_processing_owlv2.py#L531

### Expected behavior

Two image and Text lists input, two results should output.

Comments:
cc @qubvel if you have bandwidth

Hi @zhangjingxian1998 thanks for reporting the issue 🤗 

It seems like this use case was missed in https://github.com/huggingface/transformers/pull/30686 while fixing the bounding box rescaling issue. I prepared a fix
Checkpoint saving by different evaluation criterias
Author: daehuikim
Pull request: no
Labels: name: Feature request

### Feature request

I would like to be able to save the best checkpoint separately for each criteria in the evaluation.
For example```save_best_checkpoint=[loss,bleu,rouge]``` then we can get 3 different best checkpoint in result repo.

### Motivation

There can be 2 different best check points when training a model. For example, LOSS might be the lowest at check point 10000 and BLEU SCORE might be the highest at check point 20000.  However, the current TrainingArgument doesn't seem to support saving for these two different best checkpoints as far as I know.

### Your contribution

I did not make any PR yet. 
I am just suggesting idea
Thank you!

Comments:
cc @muellerzr
[SuperPoint, PaliGemma] Update docs
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR updates the docs of 2 models:

* SuperPoint: as the wrong class was used, and a figure + link to a demo notebook is also added.
* PaliGemma: add a figure, usage tips and resources section.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31025). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Watermark: fix tests
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As reported by @gante , watermarking slow tests is failing for different hardwares. This PR removes the hard check for the generated ids, since we already check if watermarking algorithm works correctly in the "test_logits_processor.py". So now we only check for the detector in the slow test.

The unrelated changes are from `make style` I guess

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30961). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docs / PEFT: Add PEFT API documentation
Author: younesbelkada
Pull request: yes

# What does this PR do?

Currently in the PEFT integration docs, the API documentation is missing, this PR adds it as discussed internally: https://huggingface.slack.com/archives/C04L3MWLE6B/p1716843071520559

cc @osanseviero @stevhliu 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31078). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Render chat template tojson filter as unicode
Author: CISC
Pull request: yes

# What does this PR do?

<details>
<summary>Rant about lack of special capabilties templates...</summary>

I wish more models that have extra capabilities like f.ex. function calling would start using the power available in chat templates instead of implementing their own inference mini-stacks (`Mistral`, `Berkeley`, et al.; I'm looking at you) that you would have to add as model-specific dependencies to use those features "as intended".

Shout out to `Cohere` for actually making an effort here, however it's a shame they did not follow it to completion to also include the final `tool` role to enable the full round-trip with natural language response to the function call (see my Mistral `GGUF` for a fully functional example of this).
</details>

I've been using the jinja2 `tojson` filter in a couple of chat templates to render function calling instructions, which is working very well, however sometimes this will introduce escaped unicode characters, which is undesirable.

I'm mainly using `GGUF`s, so have not used `tojson` with `transformers` yet, but I imagine this could be a useful feature for many recent models if only they started using chat templates properly (see rant).

This PR simply changes the default parameters of `tojson` so that the JSON is rendered in unicode.

Examples of (`GGUF`) models using `tojson`:
* [openfunctions-v2](https://huggingface.co/CISCai/gorilla-openfunctions-v2-SOTA-GGUF)
* [Mistral-7B-Instruct-v0.3](https://huggingface.co/CISCai/Mistral-7B-Instruct-v0.3-SOTA-GGUF)

Submitted similar PR to abetlen/llama-cpp-python#1486

## Who can review?

@ArthurZucker

Comments:
The failing tests seem to be an unrelated issue in `main` branch, and not this PR.

Also @CISC and @junrae6454, since you're both interested in tool use with chat templates, we're actually planning an overhaul of that, and several of the model templates as well. Please see PR #30621, and let me know if you have any feedback!

@Rocketknight1 Oh, hey, that might impact some of my ongoing work in `llama-cpp-python`, thanks for the heads up!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31041). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merging without core maintainer review because it's a one-line fix that only affects the `tojson` filter in Jinja, so it shouldn't have any other side-effects on the library. cc @amyeroberts and @ArthurZucker for visibility anyway!
FIX / OPT: Fix OPT multi-GPU training for `OPTForQuestionAnswering`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/31037

In order to perform OPT multi-GPU training (when the model is split across multiple GPUs), `device_map='auto'` the lm_head might be on another device than the first device, leading to device mismatch error. One could make sure to set a custom device_map in order to set the lm_head on the first device but this simple fix covers the usecase where people load the model with device_map="auto"

Note we apply a similar trick for all models when computing the LM-loss e.g.: https://github.com/huggingface/transformers/blob/22dab246c5605d2fca0597eed84d16a783bc3e22/src/transformers/models/llama/modeling_llama.py#L1196 we just forgot to do it for QuestionAnswering classes

 cc @amyeroberts 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31092). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
skip `test_multi_gpu_data_parallel_forward` for `vit` and `deit`
Author: ydshieh
Pull request: yes

# What does this PR do?

Since `torch==2.3+cu121`, although this test passes, many subsequent tests have `CUDA error: misaligned address`.

If `nvidia-xxx-cu118` stuff are also installed, no failure (even with `torch==2.3+cu121`).

This test is known to be problematic, likely due to the `nn.DataParallel` (which is not recommended to be used by `torch` doc).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31086). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ydshieh If `nn.DataParallel` isn't recommended for use, then could we adapt or remove these tests instead?

Yeah, we think to change it to `DistributedDataParallel` but never work on it or remove the test as we never feel the priority strongly. If you are ok, let's continue to skip them and come back to it once:

- we have better CI states on slack (even by skip failing tests and track them on GitHub issue pages)
- AMD and nightly (torch) CI rework

The above two are more important.

As you can see, even with `nn.DataParallel`, it works under some conditions (here, if `nvidia-xxx-cu118` are installed) etc. And I am not also 100% sure it's really `nn.DataParallel` or not, it's just from my experience.

@ydshieh Could you open an issue to track this and for the change to `DistributedDataParallel` to be made?

Sure, opened an issue with a short description
Fix PretrainedConfig docstring with deprecated resume_download
Author: albertvillanova
Pull request: yes

Fix PretrainedConfig docstring with deprecated resume_download, so that it is aligned with all the others. See:
- #30620

Follow-up on:
- #31007

CC: @Wauplin 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31014). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docs / Quantization: Redirect deleted page
Author: younesbelkada
Pull request: yes

# What does this PR do?

Specifically addresses: https://github.com/huggingface/transformers/pull/30942#issuecomment-2133473745

cc @LysandreJik @SunMarc @stevhliu 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31063). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
TST: Fix instruct-blip tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

With the recent CI update to torch 2.3, this PR fixes the expected results for instruct-blip tests

cc @ydshieh 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31088). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I'll let @ydshieh comment on this but I don't think it's a breaking change, it often happens that some logits / generated texts slightly differ from one minor version to another

`InstructBlipModelIntegrationTest::test_inference_vicuna_7b` is from the torch version.

`tests/models/instructblip/test_modeling_instructblip.py::InstructBlipModelIntegrationTest::test_inference_flant5_xl` is not, it fails since Dec. 9 2023, but I can't really see what's going on there.

It's not the fp32 that is fixed in #27867

OK thanks ! Merging ! 

Yeah, finally got it, it is caused by #27808 (commit `b31905d1`)

(`test_inference_flant5_xl `)
new model request: DeepSeek-V2
Author: Atry
Pull request: no
Labels: name: New model

### Model description

Tech Report: https://arxiv.org/abs/2405.04434
Code: https://huggingface.co/deepseek-ai/DeepSeek-V2/blob/e0828e3cc0a03408724b80c3cc92c8e072db8d01/modeling_deepseek.py

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
I am taking this up
LLAMA3's special_token is not the same in a file as it is used
Author: rangehow
Pull request: no



In https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/special_tokens_map.json
We have
```json
{
"bos_token": "
",
"eos_token": "
"
}
```
```json
{
  "eos_token_id": 128001,
}
```
While
`print(tokenizer.eos_token,tokenizer.eos_token_id)`-> `
 128009`

Is here something wrong?

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just load llama3 tokenizer

### Expected behavior

Clarify this diffrent setting

Comments:
 mistake : (
Similarity search bug - urgent
Author: ssdidis
Pull request: no

### System Info

Exception has occurred: AttributeError
'HuggingFaceEmbeddings' object has no attribute 'encode'
  File "/home/ubuntu/new_d.py", line 38, in perform_similarity_search
    query_vector = embeddings.encode(['High blood pressure is caused by eating salt'])
                   ^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/new_d.py", line 58, in run_indexing_pipeline
    distances, indices = perform_similarity_search(query, faiss_index, embeddings_model)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/new_d.py", line 63, in <module>
    run_indexing_pipeline()
AttributeError: 'HuggingFaceEmbeddings' object has no attribute 'encode'

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

def perform_similarity_search(query_text, index, embeddings, top_k=5):
    """Perform similarity search in the FAISS index for a given query text."""
    # Use the embeddings object to embed the query_text into a vector.
    # Ensure the text is passed as a list and the result is accessed correctly.
    query_vector = embeddings.encode(['High blood pressure is caused by eating salt'])
    # Reshape the query_vector for compatibility with FAISS search method if necessary.
    # FAISS expects the query vector to be a 2D array.
    if len(query_vector.shape) == 1:
        query_vector = query_vector.reshape(1, -1)
        
    # Search the index using the reshaped query_vector.
    distances, indices = index.search(query_vector, top_k)  # Search the index for the top_k closest vectors
    return distances, indices

### Expected behavior

1. the similarity search function should return data that is most similar to the query text.

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Caching Past Key values of any length for Vision LLM's
Author: saikoneru
Pull request: no
Labels: name: Feature request

name: Generation

### Feature request

Allowing passing past key values during the forward pass of more than one token similar to the text large language models. 

### Motivation

According to the documentation [here](https://huggingface.co/docs/transformers/en/model_doc/llava_next#transformers.LlavaNextForConditionalGeneration.forward.past_key_values) one could in theory pass past key values of the prompt to speed up the forward pass. However, I think that the cached forward pass only happens when the input_ids has **only** one new token as described in the condition [here](https://github.com/huggingface/transformers/blob/573565e35a5cc68f6cfb6337f5a93753ab16c65b/src/transformers/models/llava_next/modeling_llava_next.py#L816). Changes to this making it consistent with the text LLMs (thanks for that) would be highly appreciated.

### Your contribution

If you can give me any pointers on how I can realign cache, create the attention masks, etc., It would also be very helpful

Comments:
@saikoneru hey! 

If I understand you correctly, you want to use past_kv to continue generation in VLMs. Since VLMs consist of a vision tower and an LLM as a backbone, passing past_kv should work out-of-the-box for them. But currently there are some issues because of the way how VLMs are implemented in transformers, and simply passing one new token will not work.

If you're trying to use the `generate()`, have a look at #30809 which had some working examples on NanoLLaVa. If you want to pass inputs directly to the forward pass, then you have to expand and align (attn_mask, position_ids, and optionally labels) to match the size of prev past_kv, i.e. take into account special vision tokens. 

@zucchini-nlp Thank you for your reply. I need to pass it directly to the forward function (for **reranking situations**). Here I can cache the prompt and rank the candidates faster. I managed to use the forward pass for one new token but will see how to adapt it for more than one token. Thanks again.
Quantized KV cache: update quanto
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

The latest version of `quanto` was apparently refactored affecting quant-cache. This PR updates QuantoQuantizedCache to work with the latest version of quanto

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31052). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@younesbelkada right, it's not compatible with older versions. I can add a warning or raise an error asking to update quanto to the latest version, if that's usual workflow in these situations? :)

Since the quanto cache is not part of a release yet, I think we should just force users to use quanto >= 0.2.0 - for that I would just check the quanto version using something like this: https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L1576C53-L1576C79 and raise an error at the cache init if the quanto minimum version is not met
With respect to the imports we'll need to make sure to import `AffineQuantizer` and `MaxOptimizer` only if quanto version is greater than 0.2.0 

> LGTM ! I think it will be interesting to rerun the benchmark with quanto for int4 kv-cache quantization. @dacorvo said that the latency of models quantized with qint4 weights is drastically reduced (almost divided by two).

The new kernels only work when doing a matmul between fp16 inputs and int4 weights. In this case the KV are just dequantized, and I haven't bound the kernel for that (the CUDA dequantize method is included in quanto though so maybe you can give it a try).

> > LGTM ! I think it will be interesting to rerun the benchmark with quanto for int4 kv-cache quantization. @dacorvo said that the latency of models quantized with qint4 weights is drastically reduced (almost divided by two).
> 
> The new kernels only work when doing a matmul between fp16 inputs and int4 weights. In this case the KV are just dequantized, and I haven't bound the kernel for that (the CUDA dequantize method is included in quanto though so maybe you can give it a try).

More specifically: https://github.com/huggingface/quanto/blob/f545e01443767b8920609bfcde5417fe064eedfd/quanto/library/ext/cuda/awq/dequantize.cuh#L14
This is the fast int4 -> fp16 dequantizer for packed int4 "a la AWQ". This should be bound in pybind and called here: https://github.com/huggingface/quanto/blob/f545e01443767b8920609bfcde5417fe064eedfd/quanto/tensor/qbits/awq/qbits.py#L32

@dacorvo i see, thanks! I'll check it out!
FEAT: Add mistral v3 conversion script
Author: younesbelkada
Pull request: yes

# What does this PR do?

Adds the conversion script to convert mistral-v3 models

cc @ArthurZucker 

Fixes: #31093


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30981). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Build docker images for tokenizers main branch
Author: LysandreJik
Pull request: yes
Labels: name: WIP

:warning: Do not merge this PR!

PR 1/2: in order to run the full test suite (slow tests included), with the `main` branch of the `tokenizers` library, rebase this PR on `main`. Once the workflows have finished, head to PR 2/2 here: https://github.com/huggingface/transformers/pull/16712

---

This PR is one of two items to run the full test suite for the tokenizers current `main` branch.

In order to re-run, rebuild the docker images, publish them to the docker hub, and rebase this PR on the `main` branch of this repository.

Steps done in order to create this PR:
- Edit the dockerfiles so that they successfully install `tokenizers` from source in the container
- Edit the `build-docker-images.yml` action to:
    - Have it push these images to the Docker Hub.
    - Remove all non important images
    - Edit the identifier to contain `internal` as a prefix, and `tokenizers-main` as a suffix.
- Convert these images to private visibility so that it does not surprise users

These images will be built on each commit, so rebasing this branch on `main` will retrigger the workflow

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16708). All of your documentation changes will be reflected on that endpoint.