PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language
  Instructions
Authors: name: Weifeng Lin; name: Xinyu Wei; name: Renrui Zhang; name: Le Zhuo; name: Shitian Zhao; name: Siyuan Huang; name: Junlin Xie; name: Yu Qiao; name: Peng Gao; name: Hongsheng Li
Published: 2024-09-23 17:59:46

Abstract: This paper presents a versatile image-to-image visual assistant, PixWizard,
designed for image generation, manipulation, and translation based on free-from
language instructions. To this end, we tackle a variety of vision tasks into a
unified image-text-to-image generation framework and curate an Omni
Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction
templates in natural language, we comprehensively include a large set of
diverse vision tasks such as text-to-image generation, image restoration, image
grounding, dense image prediction, image editing, controllable generation,
inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers
(DiT) as our foundation model and extend its capabilities with a flexible any
resolution mechanism, enabling the model to dynamically process images based on
the aspect ratio of the input, closely aligning with human perceptual
processes. The model also incorporates structure-aware and semantic-aware
guidance to facilitate effective fusion of information from the input image.
Our experiments demonstrate that PixWizard not only shows impressive generative
and understanding abilities for images with diverse resolutions but also
exhibits promising generalization capabilities with unseen tasks and human
instructions. The code and related resources are available at
https://github.com/AFeng-x/PixWizard