Appearance and Pose-Conditioned Human Image Generation Using Deformable GANs

In this paper, we address the problem of generating person images conditioned on both pose and appearance information. Specifically, given an image <inline-formula><tex-math notation="LaTeX">$x_a$</tex-math><alternatives><mml:math><mml:msub><mml:mi>x</mml:mi><mml:mi>a</mml:mi></mml:msub></mml:math><inline-graphic xlink:href="lathuiliere-ieq1-2947427.gif"/></alternatives></inline-formula> of a person and a target pose <inline-formula><tex-math notation="LaTeX">$P(x_b)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>P</mml:mi><mml:mo>(</mml:mo><mml:msub><mml:mi>x</mml:mi><mml:mi>b</mml:mi></mml:msub><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="lathuiliere-ieq2-2947427.gif"/></alternatives></inline-formula>, extracted from an image <inline-formula><tex-math notation="LaTeX">$x_b$</tex-math><alternatives><mml:math><mml:msub><mml:mi>x</mml:mi><mml:mi>b</mml:mi></mml:msub></mml:math><inline-graphic xlink:href="lathuiliere-ieq3-2947427.gif"/></alternatives></inline-formula>, we synthesize a new image of that person in pose <inline-formula><tex-math notation="LaTeX">$P(x_b)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>P</mml:mi><mml:mo>(</mml:mo><mml:msub><mml:mi>x</mml:mi><mml:mi>b</mml:mi></mml:msub><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="lathuiliere-ieq4-2947427.gif"/></alternatives></inline-formula>, while preserving the visual details in <inline-formula><tex-math notation="LaTeX">$x_a$</tex-math><alternatives><mml:math><mml:msub><mml:mi>x</mml:mi><mml:mi>a</mml:mi></mml:msub></mml:math><inline-graphic xlink:href="lathuiliere-ieq5-2947427.gif"/></alternatives></inline-formula>. In order to deal with pixel-to-pixel misalignments caused by the pose differences between <inline-formula><tex-math notation="LaTeX">$P(x_a)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>P</mml:mi><mml:mo>(</mml:mo><mml:msub><mml:mi>x</mml:mi><mml:mi>a</mml:mi></mml:msub><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="lathuiliere-ieq6-2947427.gif"/></alternatives></inline-formula> and <inline-formula><tex-math notation="LaTeX">$P(x_b)$</tex-math><alternatives><mml:math><mml:mrow><mml:mi>P</mml:mi><mml:mo>(</mml:mo><mml:msub><mml:mi>x</mml:mi><mml:mi>b</mml:mi></mml:msub><mml:mo>)</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="lathuiliere-ieq7-2947427.gif"/></alternatives></inline-formula>, we introduce <italic>deformable skip connections</italic> in the generator of our Generative Adversarial Network. Moreover, a <italic>nearest-neighbour loss</italic> is proposed instead of the common <inline-formula><tex-math notation="LaTeX">$L_1$</tex-math><alternatives><mml:math><mml:msub><mml:mi>L</mml:mi><mml:mn>1</mml:mn></mml:msub></mml:math><inline-graphic xlink:href="lathuiliere-ieq8-2947427.gif"/></alternatives></inline-formula> and <inline-formula><tex-math notation="LaTeX">$L_2$</tex-math><alternatives><mml:math><mml:msub><mml:mi>L</mml:mi><mml:mn>2</mml:mn></mml:msub></mml:math><inline-graphic xlink:href="lathuiliere-ieq9-2947427.gif"/></alternatives></inline-formula> losses in order to match the details of the generated image with the target image. Quantitative and qualitative results, using common datasets and protocols recently proposed for this task, show that our approach is competitive with respect to the state of the art. Moreover, we conduct an extensive evaluation using off-the-shell person re-identification (Re-ID) systems trained with person-generation based augmented data, which is one of the main important applications for this task. Our experiments show that our Deformable GANs can significantly boost the Re-ID accuracy and are even better than data-augmentation methods specifically trained using Re-ID losses.