Content Creation of Immersive Media — Part 2
By: James Zhang
Tags: 3d Modeling, Content Creation, Virtual Reality, AI, Image

Synthetic Media

Synthetic Media from GauGAN

Painting, movies, photos, or posts on social media are artifacts by expressing ourselves to others. Some carry emotions of suffering or joys while the others contain knowledge. However, the more versatile and expressive the media is, the more professions are required to master. The problem is magnified in the domain of immersive media, where engaging content creation is only accessed by a minority of people.

Can we democratize the creation of content?

In fact, in the era of AI, intelligent assistants are created to understand high-level inputs so that we can focus on the most intuitive interaction such as strokes, semantics, or natural language. This paradigm is sometimes called “Synthetic Media”.

In the following sections, I will present contemporary AI-based content creation methods.

Image to Image Translation:

Speaking of AI for content creation, there is an algorithm that dominates this field — GAN.

Intuitively, we can think of GAN as two AIs compete together: one is an artist who generates the content, and the other is a critic who judges the content. If the critic assesses the content based on realism, the artist will be guided to create a photorealistic one.

Concept of GAN, from https://www.google.com/url?sa=i&url=https%3A%2F%2Ftowardsdatascience.com%2Fgan-papers-to-read-in-2020-2c708af5c0a4&psig=AOvVaw2xZygYkrNjBiRqN91WpRbC&ust=1607939322241000&source=images&cd=vfe&ved=0CAIQjRxqFwoTCMDutPPWyu0CFQAAAAAdAAAAABAJ

We can also condition the artist in a situation.

In the image-to-image translation, the conditioned case is an image, and the algorithm will try to convert the image to a desired one.

GauGAN by Nvidia

GauGAN, an algorithm developed by Nvidia, translates semantic maps to realistic nature scenes. Without expertise in colorizing and stroking details, we can focus on simplified drawing.

Similarly, Chimera Painter, invented by Google in 2020, adopts the same method. Instead of generating photorealistic scenes, video game-like style mythical monsters are spawned by the computer program, which showcases the versatility of image-to-image translation.

Chimera Painter by Google

AI Content Creation in 3D:

If AI can help us generate 2D images, why stop there? Why not use it to create 3D models?

This interactive terrain generation algorithm can help us design terrains to populate the virtual world in just a few seconds without knowing the details. Rivers, ridges, and valleys can be created with just a few stokes.

Or we can use the same image-to-image translation methods to create facades of buildings. Coupled with automated 3D model algorithms, the algorithm can generate realistic and diversified cities out of the blue.

How about instead of input simplified images, real human portraits are entered and generate precise 3D models?

With only an image from a single view, the AI can inference the 3D avatar even with correct backsides.

Another paradigm is manipulating content in high-level contexts such as semantic, texture, or style, which including style transfer and semantic manipulation.

Style Transfer:

Style transfer converts an image to an arbitrary style, which can be commonly seen on filters of photo editing apps. With the advantage of AI, obscure styles of specific artistic expression or semantics like nighttime and daytime can also be transferred.

For example, Alteration, a 360 film created by Facebook partnered with the production company OKIO Studio, the Saint George VFX studio, and director Jérôme Blanquet, utilize style transfer to convey the eerie atmosphere.

Semantic Manipulation:

from StyleGAN

For semantic manipulation, you probably have seen the AI filter that alters the ages of human portraits.

StyleGAN, in addition to making a person aging, hairstyle, skin color, or accessories can all be modified by tuning and parameters. Will a high-level understanding of the features of human faces, StyleGAN enables you to mix styles. With selected images, a target image can be created while preserving the characteristics of input faces.

Dreamscape introduces this concept to the domain of buildings. By manipulating the artistic styles or semantics, we can explore a virtual world with approximately infinite variations. Imagine future content can be manipulated at will, to suit different purposes and create diversified situations fostering human empathy.

Summary:

AI-based algorithms have the potential to democratize content creation and change our communication methods, but this also means fake content can be populated more easily. DeepFake, which can easily create false speeches of action of celebrity, is this kind. Further, we can’t deny issues such as weaponizing synthetic media to propagate hate speeches, or the synthesized content are biased. These open questions are crucial to the flourishing of synthetic media.