Learning Action and Reasoning-Centric Image Editing from Videos and
  Simulations
Authors: name: Benno Krojer, name: Dheeraj Vattikonda, name: Luis Lara, name: Varun Jampani, name: Eva Portelance, name: Christopher Pal, name: Siva Reddy

An image editing model should be able to perform diverse edits, ranging from
object replacement, changing attributes or style, to performing actions or
movement, which require many forms of reasoning. Current general
instruction-guided editing models have significant shortcomings with action and
reasoning-centric edits. Object, attribute or stylistic changes can be learned
from visually static datasets. On the other hand, high-quality data for action
and reasoning-centric edits is scarce and has to come from entirely different
sources that cover e.g. physical dynamics, temporality and spatial reasoning.
To this end, we meticulously curate the AURORA Dataset
(Action-Reasoning-Object-Attribute), a collection of high-quality training
data, human-annotated and curated from videos and simulation engines. We focus
on a key aspect of quality training data: triplets (source image, prompt,
target image) contain a single meaningful visual change described by the
prompt, i.e., truly minimal changes between source and target images. To
demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on
a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks.
Our model significantly outperforms previous editing models as judged by human
raters. For automatic evaluations, we find important flaws in previous metrics
and caution their use for semantically hard editing tasks. Instead, we propose
a new automatic metric that focuses on discriminative understanding. We hope
that our efforts : (1) curating a quality training dataset and an evaluation
benchmark, (2) developing critical evaluations, and (3) releasing a
state-of-the-art model, will fuel further progress on general image editing.