Image-to-Image Translation用
GAN手法のサーベイ
2018/08/29
産業技術総合研究所 人工知能研究センター
金崎 朝子

1

この資料について• 画像変換系のGANについて調べる機会があったので、せっかくなのでまとめました。
• なおECCV2018以降はノーチェック。
• アーキテクチャとコスト関数にのみフォーカス。
• 手法の比較が目的のため、詳しい説明は省略。
• 初心者向け

2

1. 基礎的な手法の紹介

3

Autoencoder (AE)𝒙encoder𝒛decoder𝒙′: 目的関数4

Variational Autoencoder (VAE)• D. P. Kingma, M.Welling, “Auto-encoding variationalbayes,” in: Proceedings of International Conference on
Learning Representations (ICLR), 2014.
• D. J. Rezende, S. Mohamed, D. Wierstra, “Stochasticbackpropagation and approximate inference indeep generative models,” in: Proceedings ofInternational Conference on Machine Learning (ICML),
2014.

noise 𝝐𝒙

encoder𝝁𝑞𝜙 𝒛 𝒙𝝈𝒛

decoder𝑝𝜃 𝒙 𝒛𝒙′

KLダイバージェンス
5Generative Adversarial Nets (GAN)• I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D.
Warde-Farley, S. Ozair, A. Courville, Y. Bengio,“Generative adversarial nets,” in: Proceedings ofAdvances in Neural Information Processing Systems(NIPS), 2014.

real𝒙
data
discriminator𝐷𝒛

generator𝐺

realfake𝒙′

randomnoise←GANのmin, maxはこのノリなので以下省略

6VAE-GAN• A. B. L. Larsen, S. K. Sønderby, H. Larochelle, O.
Winther, “Autoencoding beyond pixels using alearned similarity metric,” in: Proceedings ofInternational Conference on Machine Learning(ICML), 2016.

real𝒙
data
discriminator

noise 𝝐𝒙

encoder𝑞𝜙 𝒛 𝒙𝝁𝝈𝒛

decoder𝑝𝜃 𝒙 𝒛

𝐷real
fake𝒙′

7

Adversarial Autoencoder (AAE)• A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow,“Adversarial autoencoders,” in: Proceedings ofInternational Conference on Learning
Representations (ICLR), 2016.

𝒙

encoder𝑞𝜙 𝒛 𝒙𝒛𝒛

好きな事前分布を用意⇒decoder𝑝𝜃 𝒙 𝒛𝒙′

discriminator𝐷01𝑝(𝒛)8

Adversarial Variational Bayes (AVB)
noise 𝝐• L. Mescheder, S. Nowozin, A. Geiger, “Adversarial
variational bayes: Unifying variational
autoencoders and generative adversarialnetworks,” in: Proceedings of International
Conference on Machine Learning (ICML), 2017.

𝒙encoder𝑞𝜙 𝒛 𝒙𝒛

decoder𝑝𝜃 𝒙 𝒛𝒙′※AAEはAVBの特殊なケース
𝒛discriminator𝐷01𝑝(𝒛)9

ALI, BiGAN
𝒙

encoder𝐺𝒛 (𝒙)𝒛※同じモデル𝒙𝒛

𝐷(𝒙, 𝒛)𝒙′𝒛𝒛

decoder𝐺𝒙 (𝒛)𝒙′real fake

ALIBiGAN• V. Dumoulin, I. Belghazi, B. Poole, O. Mastropietro,
A. Lamb, M. Arjovsky, A. Courville, “Adversariallylearned inference,” in: Proceedings ofInternational Conference on Learning
Representations (ICLR), 2017.

• J. Donahue, P. Krähenbühl, T. Darrell,“Adversarial feature learning,” in:Proceedings of International Conference on
Learning Representations (ICLR), 2017.

10

2. Image-to-Image Translation11画像から画像への変換
𝒙

ソースドメイン𝒚ターゲットドメイン12

pix2pix• P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros,“Image-to-image translation with𝒛
conditional adversarial networks,” in:noise
Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition(CVPR), 2017.
𝒙

real𝒚 + 𝒙
datadiscriminator𝐷generator𝐺

real
fake𝒚′ + 𝒙

conditional GAN

13

pix2pixHD• T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J.
Kautz, B. Catanzaro, “High-resolution
image synthesis and semantic
manipulation with conditional gans,” in:Proceedings of IEEE Conference on
Computer Vision and Pattern Recognition(CVPR), 2018.
マルチスケールのGとDを使うことで高解像度にFM: Feature Matching

14

CycleGAN, DiscoGAN, DualGAN※同じモデル

𝐺𝒚𝒙
𝐷𝒙𝒚𝐺𝒙

𝐷𝒚𝒙generator

𝐺𝒚𝒚′generator𝐺𝒙𝒙′𝒚𝐺𝒙

DiscoGAN

T. Kim, M. Cha, H. Kim, J. K. Lee, J. Kim,
J.-Y. Zhu, T. Park, P. Isola, A. A. Efros,“Unpaired image-to-image translation “Learning to discover cross-domain
relations with generative adversarial
using cycle-consistent adversarial
networks,” in: Proceedings ofnetworks,” in: Proceedings ofInternational Conference on Computer International Conference on Machine
Learning (ICML), 2017.
Vision (ICCV), 2017.

𝒙′

generator

𝐺𝒚𝒚′𝐷𝒙𝐷𝒚
cycle-consistency loss

CycleGAN

generator

cycle-consistency loss

DualGANZ. Yi, H. Zhang, P. Tan, M. Gong,“Dualgan: Unsupervised dual
learning for image-to-imagetranslation,” in: Proceedings ofInternational Conference on
Computer Vision (ICCV), 2017.

15

CoGAN• M.-Y. Liu, O. Tuzel,“Coupled generative
adversarial networks,” in:Proceedings of Advances in
Neural Information
Processing Systems (NIPS),2016.

GAN 1𝒛generator

𝐺1

𝑔1 𝒛

GAN 2𝒛

𝐺2

𝐷1

real
fakeweight sharing

weight sharing

generator

discriminator𝑔2 𝒛

discriminator

𝐷2

real
fake16

UNIT• M.-Y. Liu, T. Breuel, J. Kautz,“Unsupervised image-toimage translation
networks,” in: Proceedingsof Advances in Neural
Information Processing
Systems (NIPS), 2017

VAEとCoGANの組み合わせ

cycle consistency

noise 𝝐𝒙1 encoder𝐸1𝝁𝝈weightsharing𝒙2

encoder𝐸2𝒛𝝐𝝁𝝈𝒛

generator
𝐺1𝒙1→1
1or𝒙2→1
2

discriminator
𝐷1

real
fake

discriminator
𝐷2

real
fake

weightsharing

generator
𝐺2

𝒙1→21or𝒙2→2217

Triangle GAN
𝒙𝒚generator

𝐺𝒙

𝒙′𝒙′

𝒚𝒚

𝐷1 (𝒙, 𝒚)𝒙𝒚′𝒙generator

𝐺𝒚𝒚′real fake• Z. Gan, L. Chen, W. Wang, Y. Pu, Y. Zhang,H. Liu, C. Li, L. Carin, “Triangle generative
adversarial networks,” in: Proceedings ofAdvances in Neural Information Processing
Systems (NIPS), 2017.

𝐷2 (𝒙, 𝒚)

𝐺𝒙𝐺𝒚cGANとBiGAN（= ALI）の組み合わせ18

今回紹介しなかったもの• Domain Transfer Network (DTN) [Taigman+, ICLR2017]https://arxiv.org/abs/1611.02200• BicycleGAN [Zhu+, NIPS2017]https://junyanz.github.io/BicycleGAN/• Unsupervised Image-to-Image Translation with Generative Adversarial Networks[Dong+, NIPS2017]https://arxiv.org/pdf/1701.02676.pdf• Triple GAN [Li+, NIPS2017]: Triangle GANと間違われやすいが画像間変換ではない
https://arxiv.org/abs/1703.02291• StarGAN [Choi+, CVPR2018]: マルチドメインを同時に扱える
https://github.com/yunjey/StarGAN19

おわりに• 同時期に全く同じ手法が複数出ていて地獄• 名前が似てても全く別物のものが多いのでちゃんと読んだ方がいい• 間違い等があればご連絡くださいm(_ _)m20