3D Photography Inpainting: Exploring Art with AI.
By: Vlad Alex, Merzmensch
Tags: Machine Learning, Artificial Intelligence, AI, Data Visualization, Art

3D Photography Inpainting: Exploring Art with AI.

Usage of the new model, with examples and Colab Notebook.

Da Vinci, “The Last Supper”. Converted with 3D Photo Inpainting by Merzmensch

We are living in a Great Epoche of Experiments. Well, science, society, and culture experience new narratives all the time. Think about Renaissance or Dadaist movements. But data science, empowered by the current state of the computational periphery, allows us to do something beyond our imagination. Machine learning was the initial start of the new journey into the unknown. Deep (and unsupervised) learning is a step further.

Trained on specific datasets, AI can visualize its perception of reality — or augment existing concepts. Sure, it depends on training sources and capabilities. But the same is true about us humans — we understand what we know. And we know what we understand.

A Glimpse behind the Artwork.

I always wondered about the ability of AI to interpret reality — and art — in a particular case. Also: isn’t it an exciting thing to explore “hallucination” by machine?

Like our brain complements the lack of information in a blind spot (see also: Scotoma) with “filling in” non-perceived visuals, so AI “inpaints” missing areas with something it “assumes” should be there.

Sometimes it works in surprising directions. ArtBreeder, a StyleGAN based web application “corrected” my uploaded image in an unexpectable way:

AI Experiment “Image Inpainting” by NVIDIA AI delivered me some creepy results:

Here is another fantastic inpainting glitch, observed by Jonathan Fly:

3D Ken Burns effect by Simon Niklaus et al. provided us with a wholly new thrilling dimension — using a single image, it analyzes the occlusion, depth, and geometry overall, which allows us to “enter inside” a photo.

You can bring historic photos to life:

Or even you can take a step further, behind the image:

3D Ken Burns effect works perfectly on photos or images with a clear perspective. It becomes complicated when depth is not easily definable:

3D Photography Inpainting

With their paper “3D Photography using Context-aware Layered Depth Inpainting” (PDF) has a team from Virginia Tech, National Tsing Hua University, and Facebook brought the exploration of an image to another level.

On their Project Website, you can see a variety of demonstration videos.

And here is a video comparison between various existing models:

The novel method with multi-layered depth brings more dimension into the game. The background behind the movable objects is still filled in with inpainting, but the three-dimensionality of the results are stunning.

Using CNN-based depth estimation and image preprocessing, the model continues invisible areas in a connected way: context-aware continuation of lines and patterns works very convincingly.

Compare the image by Magritte I did with the new model:

Or sneak inside the photo portrait of Salvador Dalí and Frederico Garcia Lorca:

Exploring art with 3D Photo Inpainting

As you see, the interpretation of hidden layers by machine is fascinating.

The results in the demo video are more than convincing, but that modified portrait of Picasso intrigued me at most:

Screencap from 3D Photography Inpaiting demo video (Source), Photographer: Herbert List / Magnum Photos

The photo used here depicts Picasso in front of “La Cuisine” (1948) in his Rue des Grands-Augustins studio. And the painting areas behind the artist are filled in by AI.

So what does happen within the brain of the machine? I did some experiments and was overwhelmed.

The Landscapes behind Jesus in Da Vinci’s “The Last Supper” are clear and unique:

Another Magritte — feel the space in surreal dreams:

René Magritte is famous for his illusions and plays with layers of realities. In his work “L’Heureux donateur” the beholder cannot recognize, whether it’s a silhouette of a man with a hat painted over with a night scene — or is it a cut-out in the shapes of the man, leading us to the landscapes beyond.

3D Photo Inpainting brings its interpretation:

Andrew Wyeth’s desolate world becomes more vivid:

Piranesi’s utopian post-urban structures get their depth: