First Order Motion Model. An efficient framework by Aliaksandr... | by Merzmensch | Towards Data Science

Seeing used to be believing. Thank AI, we finally have to farewell this cute and naive, but dangerous faith. Because it actually never was. In the XXth century, photos were retouched by repressive regimes. With Deep Learning, we experience new ways to re-illustrate reality. It is not a danger; it’s a chance.
Among various methods, the framework and paper “First Order Motion Model for Image Animation” by Aliaksandr Siarohin et al. captivates through its brilliant idea:
Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class. (Source, my emphasis)
The key points are set along with transformations (similar to a puppet tool in Photoshop, or like sensors on motion capture suit) — and so the trained movement can be transferred to a target image.
The requirement is the same object category.
Shortly put, the unsupervised learning approach analyzes the motion data in source footage, universalizes it and, applies to target footage.
It also allows face-swap in quite a different way than face2face-approach. While face2face engages the face detector and applies the facial features on the target image, the framework “First Order Motion Model” goes another way:
Motion is described as a set of keypoints displacements and local affine transformations. A generator network combines the appearance of the source image and the motion representation of the driving video. In addition, we proposed to explicitly model occlusions in order to indicate to the generator network which image parts should be inpainted (source).
And it works astonishingly well.
You can try it out either using the GitHub repository or Colab Notebook.
I tried my luck on Nefertiti using the footage of AI pioneer Geoffrey Hinton. This footage is delivered with Notebook. You can use another video material. It has to fit into specific requirements and sizes.
The result was more than convincing:
In opposite, applying my own photo delivers some glitches, especially regarding glasses. I suppose, these patterns are missing in the footage and so the allocation of keypoints sometimes fails:
Using a painting by Arcimboldo, we can see how the motion assignment works — some of the graphical features are still detected as background and are not animated.
The most interesting effect happens if we use an image without physiognomical patterns — or even abstract art. I used work by Pollock:
Here we can observe how the model is trying to orient itself in the chaos of structures — and gets it up to a certain point.
Paper demonstrates other face-swap experiments:
Probably if we combine this method with StyleTransfer — or even with Deep Painterly Harmonization, we will achieve even more coherent results.
The first thought coming to your mind if you see such models is surely: DeepFake. We are already too biased (and keep biasing our Neural Networks).
Can we still believe in visuals? Can we distinguish between real and fake in the Digital Age? The answer is: “we cannot anymore”. But: the answer still does not end on that point.
This naive belief in the truth behind an image — so seductive because so simple — is a dangerous deception. There are zillions of truths behind an image. Even a raw image without any traces of post-production is not a truth, but a particularly chosen perspective. A glimpse of the unknown ontology. Pars pro toto — and we should take care of its semantical fragility.
We have to re-think our relationship with concepts of truth. Because there are so many truths like people. Sure, it’s easy to deceive somebody, faking an image or video footage and presenting it as a real thing. And people do it already — from friendly pranks till presidential uses.
But it’s up to us to believe it or not. It’s up to us to research and to analyze more than before. Sure, it’s more work for our brain and perception, but that’s the reality. Understanding the world is not that easy as we used to believe in all the past centuries.
Believing in images is a comfortable self-deception. And I am thankful to all AI efforts to disrupt this belief.
Sources:
Framework scheme and GIF demonstrations, marked with 1) :Website // Paper
Framework scheme and GIF demonstrations, marked with 1) :Website // Paper
@InProceedings{Siarohin_2019_NeurIPS,  author={Siarohin, Aliaksandr and Lathuilière, Stéphane and Tulyakov, Sergey and Ricci, Elisa and Sebe, Nicu},  title={First Order Motion Model for Image Animation},  booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},  month = {December},  year = {2019}}
2. Video footage. Interview with Geoffrey Hinton: https://www.youtube.com/watch?v=XG-dwZMc7Ng