Add nvidia/LocateAnything-3B
Author: IMvision12
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46991)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46991)
<!-- ci-dashboard-badge:end -->

# LocateAnything-3B : end-to-end visual grounding demo

@ArthurZucker  @zucchini-nlp

This PR adds [`nvidia/LocateAnything-3B`](https://huggingface.co/nvidia/LocateAnything-3B)

## Overview : the six versatile tasks

A single model handles detection, referring, pointing, OCR, document-layout and GUI grounding,
selected purely by the instruction prompt:

---

## 1. Detection (open-vocabulary, multi-category)

**Image:** [Chinatown street scene](https://www.ilankelman.org/stopsigns/australia.jpg) (1300×876)
**Prompt:** `Locate all the instances that matches the following description: stop sign</c>car</c>person.`

```
<ref>stop sign</ref><box><165><132><361><422></box>
<ref>car</ref><box><528><546><917><761></box>
<ref>person</ref><box><365><526><380><590></box><box><381><526><396><570></box><box><473><531><488><599></box><box><522><543><550><646></box><box><573><533><583><549></box>
```

→ **7 boxes**: 1 stop sign, 1 car, 5 people. Multiple categories are joined with the `</c>` separator.

<img width="1300" height="876" alt="1_detection" src="https://github.com/user-attachments/assets/2299d986-a65c-41a0-8305-6c4655cef811" />

---

## 2. Multi-Object Referring

**Image:** [SUV trunk with grocery bags](https://huggingface.co/datasets/hf-internal-testing/sam2-fixtures/resolve/main/groceries.jpg) (800×534)
**Prompt:** `Locate all the instances that match the following description: brown paper grocery bag.`

```
<ref>brown paper grocery bag</ref><box><457><331><558><640></box><box><549><331><637><640></box><box><631><331><731><637></box><box><726><319><804><637></box>
```

→ **4 boxes** for a free-form referring phrase (not a fixed class name).

<img width="800" height="534" alt="2_referring" src="https://github.com/user-attachments/assets/221e0bfd-960c-4224-9ffc-3e0d84b5c174" />

---

## 3. Pointing

**Image:** [White pickup truck](https://huggingface.co/datasets/hf-internal-testing/sam2-fixtures/resolve/main/truck.jpg) (1800×1200)
**Prompt:** `Point to: the wheels of the truck.`

```
<ref>the wheels of the truck</ref><box><225><636></box><box><315><615></box><box><726><599></box><box><840><556></box>
```

→ **4 points** (a point is a `<box>` with two coordinates instead of four).

<img width="1800" height="1200" alt="3_pointing" src="https://github.com/user-attachments/assets/80e26007-79a9-4988-8f1f-7d7e46140328" />

---

## 4. OCR / Scene-Text Detection

**Image:** [Directional signboard](https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/general_ocr_001.png) (720×1150)
**Prompt:** `Detect all the text in box format.`

```
<ref>绿洲仕格维花园公寓</ref><box><19><423><720><478></box>
<ref>上海斯格威铂尔又</ref><box><44><346><505><393></box>
<ref>打浦路2529335号</ref><box><105><471><552><508></box>
<ref>打浦路15号</ref><box><272><389><555><424></box>
<ref>大酒店</ref><box><505><338><677><379></box>
```

→ **5 text regions**, each returned with the **recognized text** inside `<ref>…</ref>` in addition to its box.

<img width="720" height="1150" alt="4_ocr" src="https://github.com/user-attachments/assets/415da82a-2f2f-4373-86aa-2dd12f5d4e2f" />

---

## 5. Layout Grounding (document layout analysis)

**Image:** [Two-column document page](https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/layout_demo.jpg) (1654×2339)
**Prompt:** `Locate all the instances that matches the following description: title</c>text.`

```
<ref>title</ref><box><209><294><522><340></box>
<ref>text</ref><box><209><81><540><278></box><box><209><362><540><619></box><box><562><81><891><325></box><box><562><363><894><519></box><box><562><531><885><586></box><box><208><819><398><827></box><box><208><753><886><787></box><box><208><692><894><738></box>
```

→ **9 boxes**: 1 `title` region + 8 `text` blocks across both columns.

<img width="1654" height="2339" alt="5_layout" src="https://github.com/user-attachments/assets/87dcbe06-9d19-44f4-832f-ef2fc074bd6b" />

---

## 6. GUI Grounding (box + point)

**Image:** [Game battle UI](https://static.wikia.nocookie.net/essentialsdocs/images/7/70/Battle.png) (512×384)
**Prompt (box):** `Locate the region that matches the following description: the FIGHT button.`
**Prompt (point):** `Point to: the RUN button.`

```
<ref>the FIGHT button</ref><box><543><784><692><866></box>     # box
<ref>the RUN button</ref><box><865><934></box>                 # point
```

→ The model boxes the **FIGHT** button and points to the **RUN** button in the same UI.

<img width="512" height="384" alt="6_gui" src="https://github.com/user-attachments/assets/0ed6a57f-426f-4f6c-bf1f-b3f5e4ce538e" />


---

```python
import torch
from transformers import AutoProcessor, LocateAnythingForConditionalGeneration

model_id = "nvidia/LocateAnything-3B"
processor = AutoProcessor.from_pretrained(model_id)
model = LocateAnythingForConditionalGeneration.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto"
)

messages = [{"role": "user", "content": [
    {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},
    {"type": "text", "text": "Locate all the instances that matches the following description: stop sign</c>car</c>person."},
]}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

generated_ids = model.generate(**inputs, generation_mode="hybrid", max_new_tokens=512)
text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
boxes = processor.parse_boxes(text, 1300, 876)
```

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, locateanything

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46991)
**Latest run:** [28498572997:2](https://github.com/huggingface/transformers/actions/runs/28498572997)
**Result:** `success` | **Jobs:** 14 | **Tests:** 66,309 | **Failures:** 0 | **Duration:** 18h 37m

<!-- ci-dashboard-recap:end -->

@ArthurZucker @zucchini-nlp Ready for review! 🚀

Duplicate i think, @molbap  and @SangbumChoi were already working on it 

Thanks for pointing this out! I wasn't aware of the ongoing work by @molbap and @SangbumChoi.

I will close the PR Thanks

Linking the PRs in case you want to take a look, though I am not sure which one is more up-to-date 

https://github.com/huggingface/transformers/pull/46300 and https://github.com/huggingface/transformers/pull/46776

Sure