Uniformize zero-shot object detection postprocessing methods
Author: qubvel
Pull request: no
Labels: name: Feature request

name: Vision

name: Processing

## Uniformizing Zero-Shot Object Detection post-processing  

### Introduction  
Currently, we have four **zero-shot object detection models** in the Transformers library:  

- **OwlVit**  
- **OwlV2**  
- **Grounding Dino**  
- **OmDet Turbo**  

Each model uses slightly different **postprocessing arguments** and produces **different output formats**, which complicates user experience and makes it harder to use them in pipelines.  

To address these inconsistencies, proposed a **unified postprocessing interface** for all four models. This will enhance usability, reduce confusion, and enable seamless integration with existing pipelines.  


## Comparison of Postprocessing Methods  

Below is a comparison of the current `postprocessing` methods and their arguments:  

| **Model**         | **Postprocessing Method**                                                                                                  | **Key Arguments**                                                                                                                                                                   |
|--------------------|--------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **OwlVit / OwlV2** | post_process_object_detection       | `outputs`, `threshold`, `target_sizes`                                                                                                                                            |
| **Grounding Dino** | post_process_grounded_object_detection    | `outputs`, `input_ids`, `box_threshold`, `text_threshold`, `target_sizes`                                                                                                         |
| **OmDet Turbo**    | post_process_grounded_object_detection | `outputs`, `classes`, `score_threshold`, `nms_threshold`, `target_sizes`, `max_num_det`                                                                                |

### Suggested Changes to Arguments  

To standardize postprocessing across all models, the following suggestions are proposed:  

1. **Standardize Method Naming:**  

Use a single method, `post_process_grounded_object_detection`, for all models for text-guided object detection. For backward compatibility, retain additional methods (e.g., OwlVit/OwlV2’s `post_process_object_detection`) with a deprecation cycle.  

2. **Unify Required Arguments:**  

Make `outputs` the only required argument.  
     - For **Grounding Dino**, pass `input_ids` inside the `outputs` parameter.  
     - For **OmDet Turbo**, make `classes` optional to provide additional flexibility.  

3. **Rename Threshold Parameters:**  

Standardize parameter names (`score_threshold` and `box_threshold`) to a single name: `threshold`. These parameters perform the same function (filtering detections by confidence score), so a uniform name reduces confusion.  

4. **Add `text_labels` Argument:**  

Introduce an optional `text_labels` parameter to map detected labels (integer IDs) to their corresponding text names.  

### Final Unified Method Signature  

The new method would look like this:  

```python  
def post_process_grounded_object_detection(
    self,
    outputs,
    threshold: float = ...,
    target_sizes: Optional[Union[TensorType, List[Tuple]]] = None,
    text_labels: Union[List[str], List[List[str]]] = None,
    <additional model-specific params>
)
```  

---

## Postprocessing Outputs  

### Current outputs by post processing  

| **Model**         | **Current Output Format**                                                                                         |
|--------------------|------------------------------------------------------------------------------------------------------------------|
| **OwlVit / OwlV2** | `{"scores": score, "labels": label, "boxes": box}`<br> *(labels are integer class IDs)*                          |
| **Grounding Dino** | `{"scores": score, "labels": label, "boxes": box}`<br> *(labels are text names decoded of detected objects from `input_ids`)*         |
| **OmDet Turbo**    | `{"scores": score, "classes": class, "boxes": box}`<br> *(classes are text names of detected objects)*           |

---

#### Suggested unified output format  

The output format will be standardized to:  

```python  
{
    "scores": score,
    "labels": label,        # Integer class IDs
    "boxes": box,           # Detected bounding boxes
    "text_labels": text     # Optional: text labels 
}
```  

### Detailed Model Changes  

#### **OwlVit / OwlV2**  
Current:  
```python  
{"scores": score, "labels": label, "boxes": box}
```  

Proposed:  
```python  
{
    "scores": score,
    "labels": label,
    "boxes": box,
    "text_labels": text
}
```  

#### **Grounding Dino**  
Current:  
```python  
{"scores": score, "labels": label, "boxes": box}
```  

Proposed:  
```python  
{
    "scores": score,
    "labels": text,  # Will be set to `None` with deprecation cycle
    "boxes": box,
    "text_labels": text
}
```  

#### **OmDet Turbo**  
Current:  
```python  
{"scores": score, "classes": class, "boxes": box}
```  

Proposed:  
```python  
{
    "scores": score,
    "labels": label,         # Add integer labels
    "boxes": box,
    "text_labels": text,     # Copy of current `classes`
    "classes": text         # Retain temporarily, remove with deprecation cycle
}
```  

Feel free to provide feedback on the suggested changes!

### Motivation

This will enhance usability, reduce confusion, and enable integration with existing zero-shot object detection pipelines.

### Your contribution

I will work on this and already have draft PRs.

Comments:
Thanks @qubvel for all that work! One of my projects actually uses all 4 architectures and my code just got so much cleaner <3

Thanks for the feedback 🤗