Realistic Face Images from Sketches Using Deep Learning | by Frank Xu | Towards Data Science

Have you ever thought about the limit of what Deep Learning and AI could do? 8 years ago, AlexNet achieved a top-5 error of 15.3% in the ImageNet 2012 Challenge, which was incredible at the time thanks to CNN and GPU training. 3 years later in 2015, ResNet-152 shows only 3.57% error, which is better than human error for this task at 5%. Also in 2015, DCGAN shown the world that Deep Learning algorithms can not only classify objects, but also create new ones. In late 2017, transformer architecture introduced by Google not only brought faster training and better results than LSTMs in the NLP field, but also challenged the Computer Vision world by bringing the attention module.
Thanks to all of the people who contributed to the Deep Learning field, we have more and more applications that we could not have thought of before, which brings up today’s topic, generating realistic face images from hand sketches. Before anything, credit goes to the researchers, and thanks for the open-source code. I will briefly introduce the model architecture, then we’ll go hands-on where you can deploy this model locally!
As shown in the architecture above, the model is separated into three parts, which are the Component Embedding (CE) Module, Feature Mapping (FM) Module, and Image Synthesis (IS) Module. An input of hand sketch face image of size 512 by 512, is first decomposed into five components: “left-eye”, “right-eye”, “nose”, “mouth”, and “remainder”. The “eye”s, “nose” and “mouth” are separated by taking heat windows size of 128, 168 and 192, while the “remainder” is literally the remainder part of the sketch. The five components are then feature-encoded using 5 auto-encoders with latent descriptors of 512 dimensions. The feature vectors of components are considered as the point samples of the underlying component manifolds, and are used to refine the hand-drawn sketch by projecting its individual parts to the corresponding component manifolds using K nearest neighbors, as shown in the Manifold Projection part.
The individual feature vectors of components are projected to manifolds to increase their plausibility. In the FM module, instead of decoding each component vector back to image then synthesis on the component-level, the authors chose to fuse the vectors’ sketches into one complete face then generate the complete image, as it helps with a more consistent result in terms of both local details and global styles. Given the combined feature vector maps, the IS module converts them to a realistic face image using a conditional GAN architecture. They also applied a two-stage training method, wherein stage 1 only the five individual auto-encoders in the CE module were trained using different component sketches. Then in stage 2, the parameters in stage 1 are fixed, and the FM/IS modules work together to generate face images through training GAN.
Before anything, If you are just looking for a quick test-drawing with the application, make sure to check out the project homepage, where the original author created a web-based testing interface.
The motivation for starting this blog came from me trying to implement along with the original GitHub Repository. Since Jittor requires Ubuntu >= 16.04, while most of the people are using MacOS/Windows, one of the easiest ways is to use Docker to build a working environment from scratch.
System: MacOS/Windows/Linux (Any OS that supports docker)
RAM: 8 GB minimum, the running model will be taking up around 6 GB
Disk Space: 5 GB minimum
GPU: None is fine, except you want to do further training
Download and install Docker in your operating system.
Here we are going to construct our docker image for running the model. First create a Dockerfile with following contents.
vim Dockerfile
Then build the docker image and run it. In the same directory of Dockerfile,
# Build the docker image called ubuntu_for_deepfacedocker build -t ubuntu_for_deepface .# Start a container called ubuntu_env and memory limit of 8GBdocker run -it --name ubuntu_env --memory=8g ubuntu_for_deepface
Alternatively, you can pull the pre-compiled docker image that I created to avoid creating and compiling Dockerfile.
# Pull the docker image from Docker Hubdocker pull frank0124/drawing-to-face# Start a container called ubuntu_env based on the pulled imagedocker run -it --name ubuntu_env --memory=8g frank0124/drawing-to-face
You should be able to see the terminal/CMD prompt changed to
root@xxxxxxxxxxxx:/#
Which means you are in the up-and-running docker container. Docker can be frustrated to deal with in the beginning, but you can master it if you utilize the search engines and read through some documentation.
The original REPO is already available in the container. Here I also provide a simplified version of code by removing unnecessary features from my GitHub. In the container,
git clone https://github.com/franknb/Drawing-to-Face.gitcd /Drawing-to-Face/Params# If you are using the original code, simplycd /DeepFaceDrawing-Jittor/Params
Then download the pre-trained weights,
wget https://www.dropbox.com/s/5s5c4zuq6jy0cgc/Combine.zipunzip Combine.zip && rm Combine.zipwget https://www.dropbox.com/s/cs4895ci51h8xn3/AE_whole.zipunzip AE_whole.zip && rm AE_whole.zipcd ..
cd /Drawing-to-Facepython3.7 example.py
After it runs through, you should expect generated images in the /results directory in the running container. It would be tough to view the result .jpg files directly from docker, which requires you to set up X11 socket before running the docker. Here I recommend the simple “copy and paste” method.
Open up a new terminal/CMD on your own operating system,
# This line of code copies the results folder to your current directory in your OSdocker cp ubuntu_env:/Drawing-to-Face/results ./
Then you can simply go to that current directory where you run your terminal/CMD, and find the pasted images.
Alternatively, if you are running the original test_model.py, check this page for detailed instructions.
Now you know the model could run on your own environment, it’s time to draw and run more tests.
First, you want to create some 512 by 512 sized sketch jpg files. You can draw it wherever you want, on an iPad or even on paper then take a picture of it. Then make sure it has right format and size, I recommend Photoshop for this process for easy size/format control. If you are not ready to make face sketches all by yourself, please also refer to this testing interface. It has a shadow-based sketch board, which could give you some references for drawing. Put all the images you want to test in a directory/folder called test.
Now, inside the docker, you want to remove the existing test images to avoid running them again.
# In the running containerrm /Drawing-to-Face/test/*
Lastly, open up terminal/CMD again and find the test directory that you created, which contains images you want to test.
# In your own OSdocker cp test ubuntu_env:/Drawing-to-Face/
Next steps for you? Maybe read through the code for the model part to gain a better understanding of how it works, or build an App that utilizes the algorithm.
Let’s look at some model outputs above. All the cases are generated with all refinement parameters set at 0.5, which is the secret sauce of this model to enable better image generation. There are in total 6 parameters (for sex, eye1, eye2, nose, mouth and remainder) you can set for one picture generated. In example.py that I provided, I set all refinement parameters to 0.5 and sex to 0 (male), while in the original test_model.py, the author gave a matrix of 30 parameters for 5 testing images.
The first sketch on the left has a little bit of cartoon-ish touch on it, specifically in the bigger eyes, slightly higher eyebrows and weird looking nose. In the generated images, we can see the model struggled to put the eye and eyebrows on. For the hair, face, mouth and ears, the model did wonderful work in making the results as real as possible. The same goes with the second sketch from the left, with exaggerated eye/mouth shapes, and the model really strikes the balance between sketch and reality.
Sketches 3 to 5 from the left are more “realistic”, with the comparison with the first two. We also ended up with better (more realistic) generated images, closing to the quality of the teaser image in the front.
This model output above is based on the 5th sketch above, where sex is set at female. We did a full transformation of all refinement parameters set from 0 to 1. As stated in paper (Refinement parameter = 1- Confidence parameter),
wbc = 1 (confidence parameter) means a full use of an input sketch for image synthesis, while by setting wbc = 0 we fully trust the data for interpolation. This blending feature is particularly useful for creating faces that are very different from any existing samples or their blending.
We can see that the sketched hairstyle (with bangs) isn’t available from existing samples or their blending, but we can generate images with bangs by setting the parameter around 0.5, halfway between faithful and realistic, which enables us to get a sketch-like but realistic output.
You can check out all the source code at my GitHub.