Salesforce/llama3-clip-pretrained-mantis-taco-8b

🌮 TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action

🌐 Website | 📑 Arxiv | 🤗 Model Weights | 💻 Demo

If you like our project or are interested in its updates, please star us :) Thank you! ⭐

Model description

We introduce TACO as a family of multi-modal large action models designed to improve performance on such complex, multi-step and multi-modal tasks. During inference, TACO produces chains-of-thought-and–action (CoTA), executes intermediate steps by invoking external tools such as OCR, depth estimation and calculator, then integrates both the thoughts and action outputs to produce coherent responses. Our TACO models outperform the instruction-tuned baseline across 8 benchmarks, achieving a 3.6% improvement on average, with gains up to 15% in MMVet tasks involving OCR, mathematical reasoning and spatial reasoning.

Figure 1. TACO vs. other multi-modal models

Usage

See our Github repository.

Intended uses & limitations

This model is intended to be used on complex, multi-step and multi-modal question answering tasks. It is trained to answer visual questions with some of the following 15 actions：OCR, LocalizeObjects, GetObjects, EstimateRegionDepth, EstimateObjectDepth, Crop, ZoomIn, QueryLanguageModel, GetImageToImagesSimilarity, GetImageToTextsSimilarity, GetTextToImagesSimilarity, DetectFaces, QueryKnowledgeBase, Calculate, and SolveMathEquation. Additionally, the Terminate action is also supported for the model to provide a final answer.

For other types of tasks that don't benefit from the actions above, you might need to train a new model or further finetune it with other actions.

Training and evaluation data

See our paper for details.

Training procedure and hyperparameters

See our paper for details.

Training results

See our paper for details.

License information

This release is for research purposes only in support of an academic paper. This repository is licensed under the noncommercial license CC-BY-NC 4.0. This model was built with Meta Llama 3, which is licensed under the Meta Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.

Citation

Please cite us if you find our repository helpful. Thank you!

@misc{ma2024tacolearningmultimodalaction,
      title={TACO: Learning Multi-modal Action Models with Synthetic Chains-of-Thought-and-Action}, 
      author={Zixian Ma and Jianguo Zhang and Zhiwei Liu and Jieyu Zhang and Juntao Tan and Manli Shu and Juan Carlos Niebles and Shelby Heinecke and Huan Wang and Caiming Xiong and Ranjay Krishna and Silvio Savarese},
      year={2024},
      eprint={2412.05479},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2412.05479}, 
}