The Uncanny Valley & Deep Fakes in Star Wars
By: Jonah Nazarian
Tags: Star Wars, Uncanny Valley, De Ageing, Deepfake Videos, Deepfakes

The Uncanny Valley & Deep Fakes in Star Wars

⚠️Spoiler warning for all Star Wars canon including The Mandalorian season finale.⚠️

Wow…that was all I could say after finishing The Mandalorian season finale. From the moment I saw the X-wing fly into the imperial hanger, I knew I was in for a treat. I paid attention to every detail hoping that my favorite Jedi had come to rescue Din and the gang. 🚀

First, his emerald lightsaber extended; my first clue that the Jedi was Skywalker. The cloaked being swiftly cut through the dark troopers as if they are younglings. 🗡

Then we see a shot of the Jedi’s gloved hand. At this point, it is clear that Luke is the knight. He continues on, taking out the entire platoon until finally, he gets his own classic Star Wars hallway scene.

Like Father Like Son; Luke Emulates Vader’s Famous Hallway Scene

“This is it,” I thought to myself. The blast doors are opened for Luke, he enters and retracts his lightsaber. On the edge of my couch I wonder, “Will he be de-aged, or will it be a recast? Hopefully, it looks good.”🤞

De-aged Luke Holding Grogu

Finally, he unhoods himself to Star Wars fans around the world, Mark Hamill looking just as young as he was in the original trilogy! How was this transformation possible? Has Hamill channeled his inner Jedi and midichlorians to become younger? Nope, it’s all just some good ol’ Hollywood de-aging.👼

More specifically, it was a deep fake🥸. Rather than using a 3D model and a face capture suit like past movies, the team behind The Mandalorian decided to use newer and improved technology. Deep fakes are made using generative adversarial networks (GAN), a type of artificial intelligence that is able to map the face of one person onto another’s while making it accurate to the lighting and reflections in the environment.

It achieves these feats by allowing two machine learning models to train each other. While one GAN creates the pseudo videos, the other is tasked with detecting any inconsistencies. This process repeats until the GAN cannot detect any more remnants that the video is fabricated.

Given enough time and money, you can make anyone say anything⏳ 💰

Although it was a magical scene, I couldn’t help but get distracted by Luke’s synthetic face which appeared as if it didn’t belong in real life. Rather, it was something out of a video game. Consequently, Luke’s unnatural movement made me a bit uncomfortable and taken aback.

This discomfort is due to a phycological concept known as the “Uncanny Valley” which was thought of by Masahiro Mori, a Japanese professor. The concept states that as a synthetic version of something becomes more like its legitimate counterpart, they become more realistic — however after reaching a certain point, the inverse is true. 📈📉📈

The Uncanny Valley Relating to Robotics

Although the theory originally was concerning robots, it can also apply to animation and digital editing. Unfortunately, Disney’s use of deep fake technology to reduce Skywalker’s age was not precise enough to make it out of the notorious valley. This was most likely due to budget and time constraints. 💸

When it comes to faces, something that we see and interact with every day, it is important to get every detail correct, or else it will be noticeable to the viewer. Even if we do not notice it deliberately, our brain will subconsciously alert us to the odd face. Something as simple as lips being too stagnant may be enough to trigger this response. 🤐

The “Uncanny Valley” effect can be traced back to the subthalamic nucleus (STN), a part of your frontal lobe that also is related to Parkinson’s Disease. When jerky or unnatural movements are witnessed, there is more activity in the STN. 🧠

Diagram Featuring the Sub-Thalamic Nucleus

This development leads scientists to consider that when detecting unnatural movements, our brain believes that there is a disconnect between our visual input and our brain’s definition of normal movements.🧑‍🔬

Unfortunately for Star Wars fans, this was not the first time they felt the unease produced by de-aged actors. In Rogue One similar de-aging technology had been used. Although Luke’s scene was a step in the right direction, there is still a lot to be done before we comfortably see late actors reprising their roles as if they are still 20.

To recreate the late Peter Cushing, who played “Grand Moff Tarkin” in the original trilogy, Rogue One implemented several VFX tricks.

First, a stand-in actor was scanned in a complex light stage and a “Medusa Rig.” These scans allowed the post-production team and VFX artists the chance to alter the actor’s face and see what he looked like in different lighting.

De-aged Tarkin and Leia

Next during filming, the stand-in wore a head-mounted camera rig to capture his facial movements. Then in post-production, “Industrial Light and Magic” used 3D editing software to map parts of Cushing’s face onto the actor’s. 🎥

Final vs Original

Lastly, animators use CGI to make the facial expressions more accurate to Cushing’s original performances and polish the overall aesthetic. A similar process was used for the recreation of Carrie Fisher’s character “Leia.”

Although Luke’s de-aging is subjectively worse than Leia’s in Rogue One, it is still an extremely promising step for Disney. Now that the studio has begun the use of deep fakes, I am confident that in higher production movies and shows, the force will be strong with de-aging and the “Uncanny Valley” will be overcome. 🌄