Creating a highly accurate 3D image easily with one photo!  (You can now edit the depth in PNG.)[1-E]

(For this article, I just tried running the code on github below.)

#Overview
**One photo**
**Highly accurate 3D image ↓ Easy creation**.


(↓ This is also an example of PNG editing. It does not mean anything at all, but this pizza sharply processed the depth PNG. There is a part where the image is accidentally dragged, but due to the nature of the image of pizza, it can be hidden well...)

<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/7c250f80-43d5-3e1b-7538-9574183e5d3b.gif" width="480">



<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/9105c5cf-2014-235f-1d5c-9567ba0d9b25.gif" width="360">

(↓ An example of PNG editing explained in the latter half of the book. I don't care, but I thought that this technique was well done... I explained it separately.)
<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/1960782b-0cb8-d31f-6943-4a0f796810f0.gif" width="480">

Below is a paper and github.


https://arxiv.org/pdf/2004.04727.pdf
Paper ``3D Photography using Context-aware Layered Depth Inpainting''
Meng-Li Shih1

github is below.
https://github.com/vt-vl-lab/3d-photo-inpainting

Paper abstract

>We propose a method for converting a single RGB-D input image into a 3D photo — a multi-layer representation
for novel view synthesis that contains hallucinated color
and depth structures in regions occluded in the original
view. We use a Layered Depth Image with explicit pixel
connectivity as underlying representation, and present a
learning-based inpainting model that synthesizes new local color-and-depth content into the occluded region in a
spatial context-aware manner. The resulting 3D photos
can be efficiently rendered with motion parallax using standard graphics engines. We validate the effectiveness of our
method on a wide range of challenging everyday scenes and
show less artifacts compared with the state of the arts.


It will be added,
2020/05/23 ** About 3 days ago, the depth could be edited in PNG. **
I tried it.

# I tried (work)
(The github code worked easily in my windows environment. Maybe a GPU is needed. No need for a good GPU. Processing time was 1-2 minutes.)

As a creation procedure,
Put the image you want to process in the image folder
Just type the following command.

```
python main.py --config argument.yml
```

For publication here, the original photo is "High Quality Free Image Material"
https://pixabay.com/ja/
Obtained from (Similar to the King Kong in the overview)

Since some objects are moved (shaking) from one photo, there are of course **parts where there is no information**, but they seem to be generated by force.
great!!

**I'm sorry, it may take about 30 seconds to display. .. .. **

**All inputs are single stills**
**1 image!**





![ダウンロードgr.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/d8734a88-2e4d-5ec5-7428-e7a393ba89cd.gif)




![ダウンロードgirl.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/4164d366-3865-a977-acb7-7b0dd594422b.gif)

<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/0da47736-65cb-8d0b-84bf-887222c285f0.gif" width=480>

![ダウンロード66a.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/4978f345-af9f-6140-fd2c-2880ffd450b1.gif)


![ダウンロードbeach.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/dc8c8758-2368-a5ee-47e2-5ec2045706b0.gif)


# I tried (additional work)
I used the paintings published by **the Metropolitan Museum of Art**.

![ダウンロードart2.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/f271b0f1-a2e3-75c1-b743-56e88162ac5a.gif)


![ダウンロードart3.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/5313fa79-f418-b5bd-40e7-ba052c8f224c.gif)

<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/39e9cf20-8b62-ece4-8fd5-32219191484f.gif" width=480>

# I tried (failed. CG work etc.)

Again, the original photos were taken from "High Quality Free Image Materials". ↓
https://pixabay.com/ja/



↓**Failure?**. (Rather, computers don't work well together...)

![ダウンロードdoragon.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/c7b6a98e-f0e6-31ed-eb3d-65bf4005cdbf.gif)

↓ **Failure**. **Escher**. Also, balloons may be difficult. Maybe you could name it "**Balloon problem**".
However, the picture is beautiful. .. ..

![ダウンロードeee.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/5d2e19a1-dfa9-2a1f-85ae-54236e425dbc.gif)

↓ Somehow the texture. .. .. Failure.
![ダウンロードkuma.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/f8d06bd6-3c75-8f14-424a-a308ba5f3b5e.gif)

↓ Normal failure.
![ダウンロードdog.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/937933a3-603e-6a32-4d7f-f9b7e59a1898.gif)

↓ Normal failure.
Separately, examples of improvements are shown in the following articles.
[・・・【3】。勝手にdepthのみ feat. intel-isl の。](https://qiita.com/torinokaijyu/items/6889598b732851c2e8fd)


![ダウンロードjump.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/48905d3b-113c-886f-dfaa-2c9ceb78ebfc.gif)

#(2020/05/23) It was possible to edit the depth in PNG about 3 days ago.
I edited the depth.

Step 1:
In argument.yml,
Change depth_format to png and execute.
```

```
...
depth_format:'.png'
...
```

Then the depth will be saved as PNG. (I don't want to display this png ↓ at all... but it's probably displayed larger by the way...)

![graffiti-745071 - コピー.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/41ecdffa-b58a-ffbc-da3c-3dfecd66025f.png)

Step 2:
Edit the PNG.
(At the bottom of the car, deep (= black) where the depth is shallow.)
See the results below to see if you can do such a crude edit.

![graffiti-745071.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/e0f227c8-37a6-b6e0-68df-1f6d206c95ec.png)

Step 3:
Set require_midas to False so that the edited PNG is used.

```
require_midas: False
```

Then rerun the conversion.

![ダウンロードedit.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/64bb82f9-d9c7-3e90-0b39-07766e2b011d.gif)

As you intended, the depth of the bottom of the car is deeper. However, just like PNG,
It didn't look good with rough PNG editing.
I think that people who can do paint processing normally are fine. I can't. With a brush
I painted it on and off. .. ..
Although it may be difficult to convey what I did because the work is rough, you can edit the depth expressed as PNG as a screen when you are not satisfied with the automatically detected depth.

As an example of editing PNG, edge enhancement of PNG or **sharpening example** is the pizza at the beginning.

#PNG editing example, result added (2020/06/07)

Acclaim failed.
**Apply another image depth PNG** and see if there are any interesting effects. "

↓ (very) failed
<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/41ef2581-0964-3bfc-bad8-2a1ba0ba6ed5.gif" width=480>

Image for depth used here (Source: https://pixabay.com/en/)

<img src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/635153/e2503326-b37e-15de-6cf0-5d5c17f1bcd2.jpeg" width=200>

⇒ Again, what I was able to recognize.

* If you want to feel it originates from the ground, this method is impossible.
* Depth is very flat. Originally, it's supposed to be like that.


↓ I reversed the depth in black and white. (Very) failure → failure, improved to some extent.

<< See the top of this article for images >>>

# Information that may be useful
* For GFORCE GTX1050Ti (dedicated GPU memory 4.0GB),
  **May not work due to lack of GPU memory**.
This is **not dependent on image size**,
Probably something related to the amount of memory required for 3D representation in the image contents?
　I have not found a solution. It may be manageable by changing parameters.
  Since it was a red car, about 3.4GB of dedicated GPU memory was used and about 2.7GB of white was used.
Regarding this memory problem, why not use Google Colab or something?
* I showed the editing of depth information by PNG, but surely the default is numpy?.
　If you can handle that data (numpy) rattling, it may be possible to edit that.
* I feel that the depth information expressed in PNG is normalized somewhere. I feel that the result was the same even if I changed the depth range of PNG. :star:

#Summary

I don't understand how technically great.
However, I wonder if it is forcible.
(In the first place, people's perception when looking at a photograph is a formidable hindrance, so...)
I need to study the technology here a little more.
(For the time being, I understand the processing habits, so I think I can make a little more fun work.)
At this stage, I have no idea what it can be used for.
**Please try it yourself. "Indoor selfie" is recommended**

#Summary 2
Described after "I tried (additional work)".
Unfortunately, **I got tired of seeing it in no time. .. .. ..**
It's tough if you don't have an output for that purpose. .. .. ..
It's just a **effect**. .. .. That makes me tired. Furthermore, we look forward to the next level of technology.
However, I'm tired of it, but I'm interested in technology, although I don't understand it at all. .. ..
It was easy to edit the depth.

#from now on
I think it is better to display the name of the pixabay author, separately.
Is there a different way to show the processed image? ? ?
**If you have any comments, please.**

reference:
[写真1枚で高精度な3D画像簡単作成やってみた【2】。(depthをnumpyで加工してみる)](https://qiita.com/torinokaijyu/items/e761c00c87d6a00b8c30)
[](https://qiita.com/torinokaijyu/items/6889598b732851c2e8fd)

[写真1枚で高精度な3D画像簡単作成やってみた【3】。勝手にdepthのみ 別の手法（intel-isl のMiDaS他）。](https://qiita.com/torinokaijyu/items/6889598b732851c2e8fd)

:star2:  [写真1枚で高精度な3D画像簡単作成やってみた【0】。(空間の捉え方を確認した)](https://qiita.com/torinokaijyu/items/d93b83b6a135f1b9a660)

:new:[写真1枚で高精度な3D画像簡単作成やってみた【-1】。(隠れた領域が、本当に見えるのか？)](https://qiita.com/torinokaijyu/items/20a5a478881c9d00f9ac)