TIP: Text-Driven Image Processing with Semantic and Restoration
  Instructions
Authors: name: Chenyang Qi; name: Zhengzhong Tu; name: Keren Ye; name: Mauricio Delbracio; name: Peyman Milanfar; name: Qifeng Chen; name: Hossein Talebi
Published: 2023-12-18 17:02:30

Abstract: Text-driven diffusion models have become increasingly popular for various
image editing tasks, including inpainting, stylization, and object replacement.
However, it still remains an open research problem to adopt this
language-vision paradigm for more fine-level image processing tasks, such as
denoising, super-resolution, deblurring, and compression artifact removal. In
this paper, we develop TIP, a Text-driven Image Processing framework that
leverages natural language as a user-friendly interface to control the image
restoration process. We consider the capacity of text information in two
dimensions. First, we use content-related prompts to enhance the semantic
alignment, effectively alleviating identity ambiguity in the restoration
outcomes. Second, our approach is the first framework that supports fine-level
instruction through language-based quantitative specification of the
restoration strength, without the need for explicit task-specific design. In
addition, we introduce a novel fusion mechanism that augments the existing
ControlNet architecture by learning to rescale the generative prior, thereby
achieving better restoration fidelity. Our extensive experiments demonstrate
the superior restoration performance of TIP compared to the state of the arts,
alongside offering the flexibility of text-based control over the restoration
effects.