AI Learns Real-Time 3D Face Reconstruction | Two Minute Papers #245
The paper "Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network" and its source code is available here:
https://arxiv.org/abs/1803.07835
https://github.com/YadiraF/PRNet

Addicted? Pick up cool perks on our Patreon page! - https://www.patreon.com/TwoMinutePapers

A few comments with some of the best applications:
Lowell Camp - "This technology could be used for consumer-budget markerless facial motion capture, and if a follow-up paper enhances it with audio analysis for tongue posing, then it would require very little touch-up beyond a little temporal filtering." 
Milleoiseau - "VOIP in game but with face tracking."
Evan - "Could this be used for some kind of automatic lip-reading system for deaf viewers to view live events?"
Matan - "Monitor emotions for product improvement."
Idjles Erle - "Reconstructing ancestors faces from photos that are 150 years old. Working out from old photos who is more likely rested to whom."
Morph Verse - "Maybe create a toolsets for artists to support easy correct anatomy tools in characters with facial and body features, for faster workflow in apps like Blender or 3ds."
Bernard van Tonder - "Encourage others to watch educational content:
Let celebrities/sport idols teach important subjects by mapping their faces and voices onto people's faces in educational videos."
Adam de Anda - "Online shopping could get much more personalized. Send a selfie and be able to see sunglasses, hats, jewelry etc on your own face and able to rotate the image. Damn this actually pretty solid"

We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Andrew Melnychuk, Brian Gilman, Christian Ahlin, Christoph Jadanowski, Dennis Abts, Emmanuel, Eric Haddad, Esa Turkulainen, Geronimo Moralez, Lorin Atzberger, Malek Cellier, Marten Rauschenberg, Michael Albrecht, Michael Jensen, Nader Shakerin, Raul Araújo da Silva, Rob Rowe, Robin Graham, Ryan Monsurate, Shawn Azman, Steef, Steve Messina, Sunil Kim, Torsten Reil.
https://www.patreon.com/TwoMinutePapers

One-time payment links are available below. Thank you very much for your generous support!
PayPal: https://www.paypal.me/TwoMinutePapers
Bitcoin: 13hhmJnLEzwXgmgJN7RB6bWVdT7WkrFAHh
Ethereum: 0x002BB163DfE89B7aD0712846F1a1E53ba6136b5A
LTC: LM8AUh5bGcNgzq6HaV1jeaJrFvmKxxgiXg

Thumbnail background image credit: https://pixabay.com/photo-1722556/
Splash screen/thumbnail design: Felícia Fehér - http://felicia.hu

Károly Zsolnai-Fehér's links:
Facebook: https://www.facebook.com/TwoMinutePapers/
Twitter: https://twitter.com/karoly_zsolnai
Web: https://cg.tuwien.ac.at/~zsolnai/
 Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. Today, we have two extremely hard problems on the menu. One is facial alignment and the other is 3D facial reconstruction. For both problems, we have an image as an input and the output should be either a few lines that mark the orientation of the jawline, mouth and eyes, and in the other case, we are looking for a full 3D computer model of the face. And all this should happen automatically without any user intervention. This is extremely difficult because this means that we need an algorithm that takes a 2D image and somehow captures 3D information from this 2D projection much like a human would. This all sounds great and would be super useful in creating 3D avatars for Skype calls or scanning real humans to place them in digital media such as feature movies and games. This would be amazing, but is this really possible? This work uses a convolutional neural network to accomplish this and it not only provides high quality outputs, but it creates them in less than 10 milliseconds per image, which means that it can process a hundred of them every second. That is great news indeed because it also means that doing this for video in real time is also a possibility. But not so fast because if we are talking about video, no requirements arise. For instance, it is important that such a technique is resilient against changes in lighting. This means that if we have different lighting conditions, the output geometry the algorithm gives us shouldn't be widely different. The same applies to camera and pose as well. This algorithm is resilient against all 3 and it has some additional goodies. For instance, it finds the eyes properly through glasses and can deal with cases where the jawline is occluded by the hair or in furate shape when one side is not visible at all. One of the key ideas is to give additional instruction to the convolutional neural network to focus more of its efforts to reconstruct the center regions of the face because that region contains more discriminative features. The paper also contains a study that details the performance of this algorithm. It reveals that it is not only 5 to 8 times faster than the competition, but also provides higher quality solutions. Since these are likely to be deployed in real world applications very soon, it is a good time to start brainstorming about possible applications for this. If you have ideas beyond the animation movies and games line, let me know in the comment section. I will put the best ones in the video description. Thanks for watching and for your generous support and I'll see you next time.
[{"start": 0.0, "end": 4.0, "text": " Dear Fellow Scholars, this is Two Minute Papers with K\u00e1roly Zsolnai-Feh\u00e9r."}, {"start": 4.0, "end": 7.0, "text": " Today, we have two extremely hard problems on the menu."}, {"start": 7.0, "end": 12.0, "text": " One is facial alignment and the other is 3D facial reconstruction."}, {"start": 12.0, "end": 17.0, "text": " For both problems, we have an image as an input and the output should be either a few lines"}, {"start": 17.0, "end": 22.0, "text": " that mark the orientation of the jawline, mouth and eyes, and in the other case,"}, {"start": 22.0, "end": 26.0, "text": " we are looking for a full 3D computer model of the face."}, {"start": 26.0, "end": 30.0, "text": " And all this should happen automatically without any user intervention."}, {"start": 30.0, "end": 35.0, "text": " This is extremely difficult because this means that we need an algorithm that takes a 2D image"}, {"start": 35.0, "end": 41.0, "text": " and somehow captures 3D information from this 2D projection much like a human would."}, {"start": 41.0, "end": 47.0, "text": " This all sounds great and would be super useful in creating 3D avatars for Skype calls"}, {"start": 47.0, "end": 53.0, "text": " or scanning real humans to place them in digital media such as feature movies and games."}, {"start": 53.0, "end": 56.0, "text": " This would be amazing, but is this really possible?"}, {"start": 56.0, "end": 59.0, "text": " This work uses a convolutional neural network to accomplish this"}, {"start": 59.0, "end": 66.0, "text": " and it not only provides high quality outputs, but it creates them in less than 10 milliseconds per image,"}, {"start": 66.0, "end": 70.0, "text": " which means that it can process a hundred of them every second."}, {"start": 70.0, "end": 76.0, "text": " That is great news indeed because it also means that doing this for video in real time is also a possibility."}, {"start": 76.0, "end": 81.0, "text": " But not so fast because if we are talking about video, no requirements arise."}, {"start": 81.0, "end": 86.0, "text": " For instance, it is important that such a technique is resilient against changes in lighting."}, {"start": 86.0, "end": 91.0, "text": " This means that if we have different lighting conditions, the output geometry the algorithm gives us"}, {"start": 91.0, "end": 95.0, "text": " shouldn't be widely different. The same applies to camera and pose as well."}, {"start": 95.0, "end": 100.0, "text": " This algorithm is resilient against all 3 and it has some additional goodies."}, {"start": 100.0, "end": 105.0, "text": " For instance, it finds the eyes properly through glasses and can deal with cases"}, {"start": 105.0, "end": 111.0, "text": " where the jawline is occluded by the hair or in furate shape when one side is not visible at all."}, {"start": 111.0, "end": 116.0, "text": " One of the key ideas is to give additional instruction to the convolutional neural network"}, {"start": 116.0, "end": 120.0, "text": " to focus more of its efforts to reconstruct the center regions of the face"}, {"start": 120.0, "end": 123.0, "text": " because that region contains more discriminative features."}, {"start": 123.0, "end": 128.0, "text": " The paper also contains a study that details the performance of this algorithm."}, {"start": 128.0, "end": 132.0, "text": " It reveals that it is not only 5 to 8 times faster than the competition,"}, {"start": 132.0, "end": 135.0, "text": " but also provides higher quality solutions."}, {"start": 135.0, "end": 139.0, "text": " Since these are likely to be deployed in real world applications very soon,"}, {"start": 139.0, "end": 143.0, "text": " it is a good time to start brainstorming about possible applications for this."}, {"start": 143.0, "end": 148.0, "text": " If you have ideas beyond the animation movies and games line, let me know in the comment section."}, {"start": 148.0, "end": 150.0, "text": " I will put the best ones in the video description."}, {"start": 150.0, "end": 171.0, "text": " Thanks for watching and for your generous support and I'll see you next time."}]