Pursue to Experience the Touching in Jay-Chou's World
In this article, I’ll illustrate the work what I done. First, the motivation will be discussed. Second, I will introduce the fundamental…
Category: Machine Learning
Author: Sunner Li

Pursue to Experience the Touching in Jay-Chou's World
Figure 1. The sport field watching scenario in different image space
In this article, I’ll illustrate the work what I done. First, the motivation will be discussed. Second, I will introduce the fundamental theory of my work. At last, I will show the revised version of my idea, and the result of my work. The copyright of the video and the related image is own by JVR Music International Ltd. and Jay Chou. The original video site is here. If there’s any issue about the copyright, please inform me and I will remove these images.
Motivation
Figure 2. The shot of waiting-for-you MV
In 18, January 2018, the famous singer Jay Chou released his first single video in the latest year. The name of this song is called waiting for you(等你下課). For this precious day, he gave the warming blessing to the fans:
這首歌是送給自己、也是送給你們的生日禮物
Maybe everyone has this touching in your life. In our young, we might do some crazy but memorable things with the girl we like. However, the girl might not like you very much. After few years. You might lay on the sport field and watch the sky. Also, we not only remember the letter which write to the girl we miss, and the things we done, but also wish you can realize: there’s a person which very care about you in these years.
I was touching by the song, the lyrics and the screen. Each frame shows the unique feeling toward my heart. The first idea is that I want to live in Jay Chou’s world, and try to change the ending of this period of emotion. However, we cannot live in the film. How to try to live in the film is the topic issue in this work.
Fundamental Idea about Style Transfer
At first, I try to gain the touching in Chou’s video. Next, we can transfer the question into the style transfer problem. In recent years, the scientist try to use deep learning technique to solve this problem. The pioneer about these approach is in this article[1].
To give a brief classification of these methods, there’re two big types of method to solve the problem. The first one is pure convolutional neural network (pure-CNN) based. There’re two components in this approach. The one component is render network which is just as the usual hour-glass shape (If you don’t know what I mean, please refer to my previous article). The another one is feature extraction network, and the VGG-16 is commonly used in this part. The real-world image and style image will extract the symbol of feature by the feature extraction network, and the loss can be computed.
The other method is generative adversarial network (GAN) based. In this approach, there’re two components in the structure: generator and discriminator. The generator will try to render the image, and the discriminator will judge if it’s in the real image space or in style image space.
Classical Model: CycleGAN
Figure 3. The idea of CycleGAN
In the end of March 2017, there’s a classical model purposed which called Cycle-Consistent Adversarial Networks (CycleGAN)[2]. This project got thousands of stars on GitHub pages in the short time. The most creative idea is that it can reach unpaired training image. For example, you cannot find two horse and zebra doing the same action in the same environment and background. However, we can reach horse-zebra transformation by CycleGAN. This model can also achieve some others tasks, including image to photo and summer to winter which is shown in Figure 3.
Figure 4. The loss computation topology of CycleGAN
The training path of CycleGAN is shown in Figure 4. In our application scenario, we can split the image as two image space: real-image space (real space) and waiting-for-you space (wait space). In the whole structure, there’re four components: wait-to-real generator, real discriminator, real-to-wait generator and wait discriminator. The corresponding generator will transfer the input image to the target image domain.
There’re two paths in this topology. In the first path, the waiting image will be transferred by wait-to-real generator firstly. Then the real discriminator will judge if the transfer result is at real image domain or not. Next, this image will be transferred back into waiting space by real-to-wait generator, and the result is called reconstructed image. At last, we can compute the lasso loss (l1 loss) between original image and reconstructed image.
The other path in this topology is the reversed direction which opposed to the first path. The real image will be transferred by real-to-wait generator, then judge if it’s at wait image domain or not by wait discriminator. Next, the generated image will be transferred back by wait-to-real generator, and compare the l1 loss at the end.
Figure 5. The identity mapping loss in CycleGAN
There’s another key issue. To make the generated image keep the original color tone, the author also included the identity mapping loss. The idea of identity mapping loss is that the image will keep the same structure and intensity by generator whose target image space is as the same as the input’s. The idea of identity mapping loss is shown in Figure 5. By adding this l1 loss, the generator can ensure the identity mapping between the same image domain.
Mask-CycleGAN
Figure 6. The shot of screen and mask
However, after I observe the image in wait space, the obstacle I imagine is that the cover text might influence the transfer result. In the most frames of the video, the text just give the hint about the lyrics. However, during the loss computing, we will still compute the difference between these text region, and it’s not necessary for the image domain transfer problem! Thus, I select the pixels which is too white, and do the dilation for one times. The result is called mask by myself, and the mapping between waiting image and mask is illustrated in Figure 6.
Figure 7. The loss computation topology of Mask-CycleGAN
The revised version of loss computation topology is shown in Figure 7. As we can see, before the computation of l1 loss in wait space, the both image will multiply the mask to remove the text region. However, this change will make the variance of those text region too high. In order to control the variance. We set the probability parameter α to control this revision. In our experiments, α is 0.5.
Result
In our experiments, the hardware we use is GTX 1080-ti. There’re two models will be trained in this works: usual CycleGAN and Mask-CycleGAN. Each model train for 10 epochs, and it should spend around 1 day to complete the training. The training data we use in wait space is the screen in the waiting-for-you video. The image is captured for each 10 frame, and the number of image in waiting space is 749.
On the other hands, we use MS-COCO dataset to become the training data in real space. In the original paper, the natural image was used to do the photo-image transfer training. However, there’s some unnatural object which occur in our waiting space, such as building and school. In order to increase the shared semantic between the two image space, we drop the original Flickr dataset and adopt MS-COCO which capture more wild semantic in the real world.
Figure 8. The render result 1
The Figure 8 shows the render results. The 1st and 2nd rows show the first path (waiting →real →waiting)we mentioned in our previous section. On the other hand, the 3rd and 4th rows shows the second path(real →waiting → real). The left part is the render result of usual CycleGAN, and the right side is the render result of Mask-CycleGAN. The Figure 9 shows another training result.
Figure 9. The render result 2
As you can see, the color tone in real world is very strong. However, since there’re a lot of night star images in waiting space, the color tone in waiting world is rather dark. For the texture field, the image in real world space is more detailed. Conversely, the image in waiting space is more smooth. The feeling of waiting space is just like using pastel painting on the screen with soft impression.
Figure 10. The loss curve of two models
We can compare the result from the loss curve. The Figure 10 shows the loss curve of both strategies. As you can see, the loss of both models have convergence trend as the time extending. It proves that our idea can work in real case!
Figure 11. The render result toward NCTU sport field
At last, I show the render result toward our school’s sport field in Figure 11. The left one is the original image, and the right side is the render result. Unfortunately, we can just learn the blur and dark painting toward the waiting space. It still has some distance to reach the style in Jay Chou’s world.
Increase Shared Semantic
However, as you can see in the Figure 8 and Figure 9, the result of render isn’t perfect. There two reason I guess. First, the MS-COCO dataset doesn’t have enough shared semantic that the generator can learn. In original CycleGAN paper, the author also discussed that the transformation between cat and dog will be fail since they didn’t have enough shared information.
Second, the adversarial loss I adopt is least square loss. According to the original paper of LSGAN[3], the least square loss will give more large penalty to the image if it’s fake, and the trend is shown in Figure 12. As the result, the image might become timid to generate the image with specific pastel style.
Figure 12. The penalty circustance between usual GAN and LSGAN
In order to overcome these two disadvantages, I try to use original CycleGAN structure with ResNet-6 generator and DCGAN discriminator. Next, I try to use original binary cross entropy loss as the adversarial loss. At last, I use another dataset which is collected by ourselves. This dataset contains the image which comes from the cover of original MV. Moreover, we adopt the latest waiting-for-you MV which Jay Chou released in 21 February, 2018 that can increase the diversity and shared message.
In this dataset, there are 1782 images in real world modality and 2039 images in waiting-for-you image modality. Moreover, there are some fade out images which is much dark than the other one. In order to prevent the render result becoming dark, we ignore these images with low brightness.
Figure 13. The image of different dataset
The circumstance between three modalities is shown in Figure 13. The left most image comes from MS-COCO dataset, and the middle image comes from our collected dataset. As you can see, the middle image has more share information toward the right most image rather than the left one.
Figure 14. The Mask-CycleGAN training result after changing dataset
The training result after adopting our collected dataset as training data is shown in Figure 14. The left column is the original image which locates in real world modality. Additionally, the middle column is the transfer results and the right column is the reconstruction results. As you can see, the generator can have much better performance than the usual one.
So what’s the problem that causes the worse performance in the initial implementation? At last, we try to use least square error again, and the result is shown in Figure 15. As you can see, the result has less variance than Figure 14, and the contour of object is more clear than the Figure 14 too. As the result, we make a simple inference: increasing the share semantic of image could increase the property which locates in different image domain!
Figure 15. The Mask-CycleGAN training result after adopting least square loss
Conclusion
In this article, we implement the CycleGAN to try to transfer the image to the waiting-for-you image space. Second, to avoid to fit the lyrics region in the film image, the mask idea is purposed to get rid of the computation of specific region. At last, we shows that the generator can transfer the image with soft painting impression.
The whole code and pre-trained model can be downloaded in the project site, and the re-implementation which adopt our collected dataset can be found here. Additionally, the dataset which collected by ourself can be found here. You can also do the video transfer to transfer the whole video to the waiting-for-you image space, and create your unique touching!
Reference
[1] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge, “Image Style Transfer Using Convolutional Neural Networks,” In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, USA, 27–30 June, 2016, pp. 2414–2423.
[2] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros, “Unpaired Image-to-Image Translation using CycleConsistent Adversarial Networks,” In International Conference on Computer Vision (ICCV), Venice, Italy, 22- 29, October, 2017, pp. 2223–2232.
[3] Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley, “Least Squares Generative Adversarial Networks,” arXiv: 1611.04076 [cs.CV], April, 2017.