UnifiedVisionGPT: Streamlining Vision-Oriented AI through Generalized
  Multimodal Framework
Authors: name: Chris Kelly; name: Luhui Hu; name: Cindy Yang; name: Yu Tian; name: Deshun Yang; name: Bang Yang; name: Zaoshan Huang; name: Zihao Li; name: Yuexian Zou
Published: 2023-11-16 13:01:25

Abstract: In the current landscape of artificial intelligence, foundation models serve
as the bedrock for advancements in both language and vision domains. OpenAI
GPT-4 has emerged as the pinnacle in large language models (LLMs), while the
computer vision (CV) domain boasts a plethora of state-of-the-art (SOTA) models
such as Meta's SAM and DINO, and YOLOS. However, the financial and
computational burdens of training new models from scratch remain a significant
barrier to progress. In response to this challenge, we introduce
UnifiedVisionGPT, a novel framework designed to consolidate and automate the
integration of SOTA vision models, thereby facilitating the development of
vision-oriented AI. UnifiedVisionGPT distinguishes itself through four key
features: (1) provides a versatile multimodal framework adaptable to a wide
range of applications, building upon the strengths of multimodal foundation
models; (2) seamlessly integrates various SOTA vision models to create a
comprehensive multimodal platform, capitalizing on the best components of each
model; (3) prioritizes vision-oriented AI, ensuring a more rapid progression in
the CV domain compared to the current trajectory of LLMs; and (4) introduces
automation in the selection of SOTA vision models, generating optimal results
based on diverse multimodal inputs such as text prompts and images. This paper
outlines the architecture and capabilities of UnifiedVisionGPT, demonstrating
its potential to revolutionize the field of computer vision through enhanced
efficiency, versatility, generalization, and performance. Our implementation,
along with the unified multimodal framework and comprehensive dataset, is made
publicly available at https://github.com/LHBuilder/SA-Segment-Anything.