DCGAN > blog > Image Completion with Deep Learning in TensorFlow

自分がやろうとしている事項に関連しそうな論文、ブログなどをあさっている時に読んだ資料の抜粋です。

「今の自分が必要としているものではない」と思い始めたので、途中までの整理です。
将来、読む必要が生じる可能性もなきにしもあらずであるため、記事にしておきます。

<hr>

[Image Completion with Deep Learning in TensorFlow](https://bamos.github.io/2016/08/09/deep-completion/)

> Content-aware fill is a powerful tool designers and photographers use to fill in unwanted or missing parts of images. Image completion and inpainting are closely related technologies used to fill in missing or corrupted parts of images. 

画像の劣化した部分や消失した部分をDCGANを使って埋め合わせする方法のようだ。

自分が解きたい問題と関係あるだろうか。
線形方程式の初期値の分布(粒子のdipoleごとの初期値）が正確に得られない状況の中でDCGANを使って、ある誤差での分布をまず取得し、次に線形方程式を解いていくことで、希望精度に持っていく方針はどうだろうか。
粒子の特性（サイズ、形状、複素屈折率など）を変更した場合、初期値の分布は分からない。それを計算済の特性セットの結果を用いてDCGANで埋め合わせを行う、という考え。

リンク先では以下の論文を紹介している。
[Semantic Image Inpainting with Perceptual and Contextual Losses](https://arxiv.org/abs/1607.07539) by Raymond Yeh, Chen Chen, Teck Yian Lim, Mark Hasegawa-Johnson, Minh N. Do

以下、記事内容の抜粋。

>  I’ve added [ML-Heavy] tags to sections to indicate that the section can be skipped if you don’t want too many details. 

詳細を読みたくない場合は[ML-Heavy]タグの部分をスキップするとのこと。

 
### Step 1: Interpreting images as samples from a probability distribution

> 1. Contextual information: You can infer what missing pixels are based on information provided by surrounding pixels.
2. Perceptual information: You interpret the filled in portions as being “normal,” like from what you’ve seen in real life or from other pictures.

コンテキストの場合は周囲の情報から推論する。
認知(Perceptual)の場合は実世界や他の画像から埋めた部分を「正常」と解釈する。

> Here’s the probability density function (PDF) for a normal distribution. 

正規分布の確率密度関数を考える。

> The key relationship between images and statistics is that we can interpret images as samples from a high-dimensional probability distribution. 

撮影した画像のピクセルというのは、ある確率密度関数からのサンプルとみなす。
その分布を持つことで「正常」という判断をするとある。

> Our images will be 64 pixels wide and 64 pixels high, so our probability distribution has 64⋅64⋅3≈12k dimensions.

(中略)

> Given x=1, what is the most probable y value? We can find this by maximizing the value of the PDF over all possible y values with x=1 fixed.

(中略)

>  it’s difficult and often intractable for more complex distributions over images. The complexity partly comes from intricate [conditional dependencies](https://en.wikipedia.org/wiki/Conditional_dependence): 

(中略)

### Step 2: Quickly generating fake images

> another well-studied idea in statistics is to learn how to generate new (random) samples with a [generative model](https://en.wikipedia.org/wiki/Generative_model). Generative models can often be difficult to train or intractable, but lately the deep learning community has made some amazing progress in this space.
Yann LeCun gives a great introduction to one way of training generative models (adversarial training) in [this Quora post](https://www.quora.com/What-are-some-recent-and-potentially-upcoming-breakthroughs-in-deep-learning/answer/Yann-LeCun?srid=nZuy), describing the idea as the most interesting idea in the last 10 years in machine learning:

(中略)

> In this post we’ll only focus on Generative Adversarial Nets (GANs).

### [ML-Heavy] Generative Adversarial Net (GAN) building blocks

> we’d like to define a function G(z) that produces samples from our original probability distribution.

### Using G(z) to produce fake images

> The DCGAN paper showed how a DCGAN can be trained on a dataset of bedroom images. Then sampling G(z)G(z) will produce the following fake images of what the generator thinks bedrooms looks like.

### Existing GAN and DCGAN implementations

> There are many great GAN and DCGAN implementations on GitHub you can browse:

実際のコードが色々紹介されている。

> Moving forward, we will build on [carpedm20/DCGAN-tensorflow](https://github.com/carpedm20/DCGAN-tensorflow).

### Running DCGAN on your images

> If you skipped the last section, but are interested in running some code: The implementation for this portion is in my [bamos/dcgan-completion](https://github.com/bamos/dcgan-completion.tensorflow).tensorflow GitHub repository. I strongly emphasize that the code in this portion is from Taehoon Kim’s [carpedm20/DCGAN-tensorflow](https://github.com/carpedm20/DCGAN-tensorflow) repository.

(中略)

> As a warning, if you don’t have a CUDA-enabled GPU, training the network in this portion may be prohibitively slow.

(中略)

### Step 3: Finding the best fake image for image completion

### Image completion with DCGANs

> Now that we have a trained discriminator D(x)D(x) and generator G(z)G(z), how can we use them to complete images? In this section I present the techniques in Raymond Yeh and Chen Chen et al.’s paper [“Semantic Image Inpainting with Perceptual and Contextual Losses,”](https://arxiv.org/abs/1607.07539)