metadata
license: apache-2.0
pipeline_tag: image-to-text
Tokenize Anything via Prompting
Ting Pan1,2*, Lulu Tang2*, Xinlong Wang2¶, Shiguang Shan1
We present Tokenize Anything via Prompting, a unified and promptable model capable of simultaneously segmenting, recognizing, and captioning arbitrary regions, with flexible visual prompts (point, box and sketch). The model is trained with exhaustive segmentation masks sourced from SA-1B, coupled with semantic priors from a pre-trained EVA-CLIP with 5 billion parameters.
Installation
See Github Page.
Models
Model weights
V1.1 Release Notes
- Three versions of the model are available with different image encoders.
- Use a longer pre-training and fine-tuning schedule (improved segmentation and caption performance).
- Apply weight decay for all bias parameters (avoid FP16 overflow in QK matmul).
- Sample point prompts from predicted mask instead of GT box during VG training.
| Model | Description | Schedule | MD5 | Weights |
|---|---|---|---|---|
| tap_vit_h | ViT-H TAP v1.1 model | (100% SA-1B, 180k), (VG, 50ep) | 4bdfb9 | 🤗 HF link |
| tap_vit_l | ViT-L TAP v1.1 model | (100% SA-1B, 180k), (VG, 50ep) | c1d41f | 🤗 HF link |
| tap_vit_b | ViT-B TAP v1.1 model | (100% SA-1B, 180k), (VG, 50ep) | 707f80 | 🤗 HF link |
V1.0 Release Notes
- Two versions of the model are available with different image encoders.
- Original paper results.
| Model | Description | Schedule | MD5 | Weights |
|---|---|---|---|---|
| tap_vit_l | ViT-L TAP v1.0 model | (50% SA-1B, 90k), (VG, 25ep) | 03f8ec | 🤗 HF link |
| tap_vit_b | ViT-B TAP v1.0 model | (50% SA-1B, 90k), (VG, 25ep) | b45cbf | 🤗 HF link |
Concept weights
Note: You can generate these weights following the Concept Guide.
| Concept | Description | Weights |
|---|---|---|
| Merged-2560 | Merged concepts | 🤗 HF link |
| LVIS-1203 | LVIS concepts | 🤗 HF link |
| COCO-80 | COCO concepts | 🤗 HF link |
License
Citation
@article{pan2023tap,
title={Tokenize Anything via Prompting},
author={Pan, Ting and Tang, Lulu and Wang, Xinlong and Shan, Shiguang},
journal={arXiv preprint arXiv:2312.09128},
year={2023}
}
Acknowledgement
We thank the repositories: SAM, EVA, LLaMA, FlashAttention, Gradio, Detectron2 and CodeWithGPU.