Delete README.md with huggingface_hub
Browse files
README.md
DELETED
|
@@ -1,85 +0,0 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
base_model:
|
| 4 |
-
- Qwen/Qwen2.5-VL-3B-Instruct
|
| 5 |
-
- Qwen/Qwen2.5-VL-7B-Instruct
|
| 6 |
-
- lmms-lab/llava-onevision-qwen2-7b-ov
|
| 7 |
-
tags:
|
| 8 |
-
- robotics
|
| 9 |
-
- vision-language-action-model
|
| 10 |
-
- vision-language-model
|
| 11 |
-
library_name: transformers
|
| 12 |
-
|
| 13 |
-
# Collection Metadata (Referencing InternRobotics/VLN-PE style)
|
| 14 |
-
repo: InternRobotics/RoboInter-VLM
|
| 15 |
-
type: "checkpoint-collection"
|
| 16 |
-
description: "Collection of RoboInterVLM checkpoints and configs fine-tuned on RoboInter-VQA."
|
| 17 |
-
checkpoints:
|
| 18 |
-
- name: RoboInterVLM_qwenvl25_3b
|
| 19 |
-
path: RoboInterVLM_qwenvl25_3b/
|
| 20 |
-
notes: "Lightweight Qwen2.5-VL model"
|
| 21 |
-
- name: RoboInterVLM_qwenvl25_7b
|
| 22 |
-
path: RoboInterVLM_qwenvl25_7b/
|
| 23 |
-
notes: "Stronger performance Qwen2.5-VL backbone"
|
| 24 |
-
- name: RoboInterVLM_llava_one_vision_7B
|
| 25 |
-
path: RoboInterVLM_llava_one_vision_7B/
|
| 26 |
-
notes: "LLaVA-OneVision (SigLIP + Qwen2) backbone"
|
| 27 |
-
---
|
| 28 |
-
|
| 29 |
-
# RoboInterVLM: Vision-Language Model Checkpoints for RoboInter Manipulation Suite
|
| 30 |
-
|
| 31 |
-
Model checkpoints of **RoboInterVLM**, developed as part of the [RoboInter](https://github.com/InternRobotics/RoboInter) project. These models are fine-tuned on the [RoboInter-VQA](https://huggingface.co/datasets/InternRobotics/RoboInter-VQA) dataset for intermediate representation understanding and generation in robotic manipulation.
|
| 32 |
-
|
| 33 |
-
## Available Checkpoints
|
| 34 |
-
|
| 35 |
-
| Checkpoint | Base Model | Architecture | Parameters | Description |
|
| 36 |
-
|---|---|---|---|---|
|
| 37 |
-
| `RoboInterVLM_qwenvl25_3b` | [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) | Qwen2.5-VL | ~3B | Lightweight Qwen2.5VL model, suitable for efficient deployment |
|
| 38 |
-
| `RoboInterVLM_qwenvl25_7b` | [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) | Qwen2.5-VL | ~7B | Larger Qwen2.5-VL backbone for stronger performance |
|
| 39 |
-
| `RoboInterVLM_llava_one_vision_7B` | [LLaVA-OneVision-Qwen2-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | LLaVA-OneVision (SigLIP + Qwen2) | ~7B | LLaVA-OneVision backbone with SigLIP vision encoder |
|
| 40 |
-
|
| 41 |
-
All checkpoints are stored in `safetensors` format with `bfloat16` precision.
|
| 42 |
-
|
| 43 |
-
## Supported Tasks
|
| 44 |
-
|
| 45 |
-
These models are jointly trained on general VQA and three categories of our curated VQA tasks:
|
| 46 |
-
|
| 47 |
-
- **Generation**: Predicting intermediate representations such as trajectory waypoints, gripper bounding boxes, contact points/boxes, object bounding boxes (current & final), etc.
|
| 48 |
-
- **Understanding**: Multiple-choice visual reasoning about contact states, grasp poses, object grounding, trajectory selection, movement directions, etc.
|
| 49 |
-
- **Task Planning**: High-level task planning including next-step prediction, action primitive recognition, success determination, etc.
|
| 50 |
-
|
| 51 |
-
## Usage
|
| 52 |
-
|
| 53 |
-
### Qwen2.5-VL Checkpoints
|
| 54 |
-
For loading and inference with the Qwen2.5-VL checkpoint, please refer to the [RoboInterVLM-QwenVL](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterVLM/RoboInterVLM-QwenVL) codebase. We provide a fast loading example below:
|
| 55 |
-
|
| 56 |
-
```python
|
| 57 |
-
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
|
| 58 |
-
|
| 59 |
-
model_path = "InternRobotics/RoboInterVLM_qwenvl25_3b" # or RoboInterVLM_qwenvl25_7b
|
| 60 |
-
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
|
| 61 |
-
model_path, torch_dtype="auto", device_map="auto"
|
| 62 |
-
)
|
| 63 |
-
processor = AutoProcessor.from_pretrained(model_path)
|
| 64 |
-
```
|
| 65 |
-
|
| 66 |
-
### LLaVA-OneVision Checkpoint
|
| 67 |
-
|
| 68 |
-
For loading and inference with the LLaVA-OneVision checkpoint, please refer to the [RoboInterVLM-LLaVAOV](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterVLM/RoboInterVLM-LLaVAOV) codebase, as it requires custom model classes.
|
| 69 |
-
|
| 70 |
-
### Training & Evaluation
|
| 71 |
-
|
| 72 |
-
For full training and evaluation pipelines, please refer to:
|
| 73 |
-
|
| 74 |
-
- **Qwen2.5-VL models**: [RoboInterVLM-QwenVL](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterVLM/RoboInterVLM-QwenVL)
|
| 75 |
-
- **LLaVA-OneVision model**: [RoboInterVLM-LLaVAOV](https://github.com/InternRobotics/RoboInter/tree/main/RoboInterVLM/RoboInterVLM-LLaVAOV)
|
| 76 |
-
- **VQA Dataset**: [RoboInter-VQA](https://huggingface.co/datasets/InternRobotics/RoboInter-VQA)
|
| 77 |
-
|
| 78 |
-
## Related Resources
|
| 79 |
-
|
| 80 |
-
- **Project**: [RoboInter](https://github.com/InternRobotics/RoboInter)
|
| 81 |
-
- **Annotation Data**: [RoboInter-Data](https://huggingface.co/datasets/InternRobotics/RoboInter-Data)
|
| 82 |
-
- **VQA Dataset**: [RoboInter-VQA](https://huggingface.co/datasets/InternRobotics/RoboInter-VQA)
|
| 83 |
-
## License
|
| 84 |
-
|
| 85 |
-
Please refer to the original licenses of [RoboInter](https://github.com/InternRobotics/RoboInter), [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct), and [LLaVA-OneVision](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|