Autoregressive Models in Vision: A Survey
Authors: name: Jing Xiong; name: Gongye Liu; name: Lun Huang; name: Chengyue Wu; name: Taiqiang Wu; name: Yao Mu; name: Yuan Yao; name: Hui Shen; name: Zhongwei Wan; name: Jinfa Huang; name: Chaofan Tao; name: Shen Yan; name: Huaxiu Yao; name: Lingpeng Kong; name: Hongxia Yang; name: Mi Zhang; name: Guillermo Sapiro; name: Jiebo Luo; name: Ping Luo; name: Ngai Wong
Published: 2024-11-08 17:15:12

Abstract: Autoregressive modeling has been a huge success in the field of natural
language processing (NLP). Recently, autoregressive models have emerged as a
significant area of focus in computer vision, where they excel in producing
high-quality visual content. Autoregressive models in NLP typically operate on
subword tokens. However, the representation strategy in computer vision can
vary in different levels, i.e., pixel-level, token-level, or
scale-level, reflecting the diverse and hierarchical nature of visual data
compared to the sequential structure of language. This survey comprehensively
examines the literature on autoregressive models applied to vision. To improve
readability for researchers from diverse research backgrounds, we start with
preliminary sequence representation and modeling in vision. Next, we divide the
fundamental frameworks of visual autoregressive models into three general
sub-categories, including pixel-based, token-based, and scale-based models
based on the strategy of representation. We then explore the interconnections
between autoregressive models and other generative models. Furthermore, we
present a multi-faceted categorization of autoregressive models in computer
vision, including image generation, video generation, 3D generation, and
multi-modal generation. We also elaborate on their applications in diverse
domains, including emerging domains such as embodied AI and 3D medical AI, with
about 250 related references. Finally, we highlight the current challenges to
autoregressive models in vision with suggestions about potential research
directions. We have also set up a Github repository to organize the papers
included in this survey at:
https://github.com/ChaofanTao/Autoregressive-Models-in-Vision-Survey.