Artistic Style Transfer For Videos | Two Minute Papers #68
Artificial neural networks were inspired by the human brain and simulate how neurons behave when they are shown a sensory input (e.g., images, sounds, etc). They are known to be excellent tools for image recognition, any many other problems beyond that - they also excel at weather predictions, breast cancer cell mitosis detection, brain image segmentation and toxicity prediction among many others. Deep learning means that we use an artificial neural network with multiple layers, making it even more powerful for more difficult tasks. 

This time they have been shown to be apt at reproducing the artistic style of many famous painters, such as Vincent Van Gogh and Pablo Picasso among many others. All the user needs to do is provide an input photograph and a target image from which the artistic style will be learned.

And now, onto the next frontier: transferring artistic style to videos!

_________

The paper "Artistic style transfer for videos" is available here:
http://arxiv.org/abs/1604.08610

The implementation of this technique is also available:
https://github.com/manuelruder/artistic-videos

Recommended for you:
Deep Neural Network Learns Van Gogh's Art - https://www.youtube.com/watch?v=-R9bJGNHltQ
Deep Learning Program Learns to Paint - https://www.youtube.com/watch?v=UGAzi1QBVEg
From Doodles To Paintings With Deep Learning   - https://www.youtube.com/watch?v=jMZqxfTls-0

Sintel Movie copyright: Blender Foundation
https://durian.blender.org/sharing/

WE WOULD LIKE TO THANK OUR GENEROUS SUPPORTERS WHO MAKE TWO MINUTE PAPERS POSSIBLE:
Sunil Kim.
https://www.patreon.com/TwoMinutePapers

Subscribe if you would like to see more of these! - http://www.youtube.com/subscription_center?add_user=keeroyz

The thumbnail background image was taken from the corresponding paper.
Splash screen/thumbnail design: Felícia Fehér - http://felicia.hu

Károly Zsolnai-Fehér's links:
Facebook → https://www.facebook.com/TwoMinutePapers/
Twitter → https://twitter.com/karoly_zsolnai
Web → https://cg.tuwien.ac.at/~zsolnai/
 Dear Fellow Scholars, this is two minute papers with Kato Ejona Ifehir. We have previously talked about a technique that used a deep neural network to transfer the artistic style of a painting to an arbitrary image, for instance, to a photograph. As always, if you're not familiar with some of these terms, we have discussed them in previous episodes and links are available in the description box, make sure to check them out. Style transfer is possible on still images, as there's currently no technique to apply this to videos, it is hopefully abundantly clear that a lot of potential still lies dormant inside. But can we apply this artistic style transfer to videos? Would it work if we would simply try? For an experienced researcher, it is flagrantly obvious that it's an understatement to say that it wouldn't work. It would fail in a spectacular manner, as you can see here. But with this technique, it apparently works quite well. To be frank, the results look gorgeous. So how does it work? Now, don't be afraid, you'll be presented with a concise but deliberately obscure statement. This technique preserves temporal coherence when applying the artistic style by incorporating the optical flow of the input video. Now, the only question is what temporal coherence and optical flow means? Temporal coherence is a term that was used by physicists to describe, for instance, how the behavior of a wave of light changes or stays the same if we observe it at different times. In computer graphics, it is also an important term because oftentimes we have techniques that we can apply to one image, but not necessarily to a video, because the behavior of the technique changes drastically from frame to frame, introducing a disturbing flickering effect that you can see in this video here. We have the same if we do the artistic style transfer because there is no communication between the individual images of the video. The technique has no idea that most of the time we are looking at the same things and if so, the artistic style would have to be applied the same way over and over to these regions. We are clearly lacking temporal coherence. Now, onto optical flows. Imagine a flying drone that takes a series of photographs while hovering and looking around the bovus. To write sophisticated navigation algorithms, the drone would have to know which object is which across many of these photographs. If we have slightly turned, most of what we see is the same and only a small part of this new image is new information. But the computer doesn't know that as all it sees is a bunch of pixels. Optical flow algorithms help us achieving this by describing the possible motions that give us photograph B from photograph A. In this application, what this means is that there is some interframe communication. The algorithm will know that if I color this person this way a moment ago, I cannot drastically change the style of that region on a whim. It is now easy to see why naively applying such techniques to many individual frames would be a flippant attempt to create beautiful smooth-looking videos. So now, it hopefully makes a bit more sense. This technique preserves temporal coherence when applying the artistic style by incorporating the optical flow of the input video. Such great progress in so little time. Last time I've mentioned Cram from the comments section and this time I'd like to comment related giraffe for his insightful comments. Thanks for being around and I've definitely learned from you fellow scholars. I am really loving the respectful and quality discussions that take place in the comments section and it is really cool that we can both learn from each other. Thanks for watching and for your generous support and I'll see you next time.
[{"start": 0.0, "end": 4.4, "text": " Dear Fellow Scholars, this is two minute papers with Kato Ejona Ifehir."}, {"start": 4.4, "end": 14.6, "text": " We have previously talked about a technique that used a deep neural network to transfer the artistic style of a painting to an arbitrary image, for instance, to a photograph."}, {"start": 14.6, "end": 23.2, "text": " As always, if you're not familiar with some of these terms, we have discussed them in previous episodes and links are available in the description box, make sure to check them out."}, {"start": 23.2, "end": 34.8, "text": " Style transfer is possible on still images, as there's currently no technique to apply this to videos, it is hopefully abundantly clear that a lot of potential still lies dormant inside."}, {"start": 34.8, "end": 40.8, "text": " But can we apply this artistic style transfer to videos? Would it work if we would simply try?"}, {"start": 40.8, "end": 47.3, "text": " For an experienced researcher, it is flagrantly obvious that it's an understatement to say that it wouldn't work."}, {"start": 47.3, "end": 52.0, "text": " It would fail in a spectacular manner, as you can see here."}, {"start": 52.0, "end": 60.0, "text": " But with this technique, it apparently works quite well. To be frank, the results look gorgeous. So how does it work?"}, {"start": 60.0, "end": 65.6, "text": " Now, don't be afraid, you'll be presented with a concise but deliberately obscure statement."}, {"start": 65.6, "end": 74.2, "text": " This technique preserves temporal coherence when applying the artistic style by incorporating the optical flow of the input video."}, {"start": 74.2, "end": 79.2, "text": " Now, the only question is what temporal coherence and optical flow means?"}, {"start": 79.2, "end": 90.8, "text": " Temporal coherence is a term that was used by physicists to describe, for instance, how the behavior of a wave of light changes or stays the same if we observe it at different times."}, {"start": 90.8, "end": 109.0, "text": " In computer graphics, it is also an important term because oftentimes we have techniques that we can apply to one image, but not necessarily to a video, because the behavior of the technique changes drastically from frame to frame, introducing a disturbing flickering effect that you can see in this video here."}, {"start": 109.0, "end": 117.0, "text": " We have the same if we do the artistic style transfer because there is no communication between the individual images of the video."}, {"start": 117.0, "end": 128.0, "text": " The technique has no idea that most of the time we are looking at the same things and if so, the artistic style would have to be applied the same way over and over to these regions."}, {"start": 128.0, "end": 131.0, "text": " We are clearly lacking temporal coherence."}, {"start": 131.0, "end": 140.0, "text": " Now, onto optical flows. Imagine a flying drone that takes a series of photographs while hovering and looking around the bovus."}, {"start": 140.0, "end": 148.0, "text": " To write sophisticated navigation algorithms, the drone would have to know which object is which across many of these photographs."}, {"start": 148.0, "end": 155.0, "text": " If we have slightly turned, most of what we see is the same and only a small part of this new image is new information."}, {"start": 155.0, "end": 167.0, "text": " But the computer doesn't know that as all it sees is a bunch of pixels. Optical flow algorithms help us achieving this by describing the possible motions that give us photograph B from photograph A."}, {"start": 167.0, "end": 172.0, "text": " In this application, what this means is that there is some interframe communication."}, {"start": 172.0, "end": 181.0, "text": " The algorithm will know that if I color this person this way a moment ago, I cannot drastically change the style of that region on a whim."}, {"start": 181.0, "end": 190.0, "text": " It is now easy to see why naively applying such techniques to many individual frames would be a flippant attempt to create beautiful smooth-looking videos."}, {"start": 190.0, "end": 201.0, "text": " So now, it hopefully makes a bit more sense. This technique preserves temporal coherence when applying the artistic style by incorporating the optical flow of the input video."}, {"start": 201.0, "end": 205.0, "text": " Such great progress in so little time."}, {"start": 205.0, "end": 214.0, "text": " Last time I've mentioned Cram from the comments section and this time I'd like to comment related giraffe for his insightful comments."}, {"start": 214.0, "end": 218.0, "text": " Thanks for being around and I've definitely learned from you fellow scholars."}, {"start": 218.0, "end": 226.0, "text": " I am really loving the respectful and quality discussions that take place in the comments section and it is really cool that we can both learn from each other."}, {"start": 226.0, "end": 235.0, "text": " Thanks for watching and for your generous support and I'll see you next time."}]