# A Joint Training Model for Face Sketch Synthesis
## 1 Introduction  
Face sketch synthesis is a key branch of face style transformation, which generates face sketches for given input photos with the help of face photo-sketch pairs as the training dataset [1]. It has achieved wide applications in both law enforcement and digital entertainment. For example, sketches drawn according to the description of victims or witnesses can help identify a suspect by matching the sketch against a mugshot dataset from a police department. Face sketch synthesis reduces the texture discrepancy between photos and sketches for the face recognition procedure [2] and thus increases the recognition accuracy [3]. In digital entertainment, people are increasingly preferring to use face sketches as their portrait in social media; the sketch synthesis technique can also simplify animation production [4]. During the past two decades, various sketch synthesis methods have been proposed. The exemplar-based method is an important category of existing synthesis approaches. It synthesizes sketches for test photos by utilizing photo-sketch pairs as training data. The exemplar-based method mainly consists of neighbor selection and reconstruction weight representation [5]. In the neighbor selection process, K nearest training photo patches are selected for a test photo patch. In the reconstruction weight representation, a weight vector between the test photo patch and the selected photo patches is calculated. The target sketch patch can be obtained using weighted averaging of the K training sketchpatches corresponding to the selected photo patches with the calculated weight vector. The final sketch is obtained by averaging all the generated sketch patches. Exemplar-based face sketch synthesis originates from the Eigen-transformation research by [PERSON] et al. [6]. In their work, all training photo-sketch pairs were used to generalize the target sketch. Principal component analysis was adopted to learn the weight coefficients by projecting the input test photo onto the training photos. It is difficult to represent nonlinear relationships between face photos and sketches by only learning one holistic reconstruction model. Thus, [PERSON] et al. [7] proposed a locally linear embedding (LLE)-based sketch synthesis method to estimate the nonlinear mapping with piecewise linear mappings. The LLE method works at the image patch level, in which K nearest training photo patches are searched in terms of Euclidean distance for each test photo patch. However, the LLE method suffers from a serious noise problem. To resolve this problem, [PERSON] et al. [8] formulated face sketch synthesis into a spatial sketch denoising problem and calculated the reconstruction weight using the conjugate gradient solver. To describe the dependency relationship between neighboring sketches, [PERSON] et al. [9] introduced a multi-scale Markov random fields (MRF) model to represent the neighboring constraint between adjacent sketch patches using a compatibility function. However, this method only chose the best single sketch patch from the training data for the test photo patch, meaning it could not synthesize new sketch patches. Additionally, the optimization process in the MRF model is an NP-hard problem. To overcome these limitations, [PERSON] et al. [10] extended the MRF model by introducing the linear combination of nearest neighbors to structure a Markov weight fields (MWF) model, which is capable of synthesizing new sketch patches that do not exist in the training dataset. In reference [11], a sparse representation-based face sketch synthesis method was proposed by [PERSON] et al. [PERSON] et al. [12] proposed a multiple representation-based face sketch synthesis method, which is able to obtain high-quality sketch images. However, this method is time-consuming because of the online neighbor selection. Recently, [PERSON] et al. [13] proposed a state-of-the-art face sketch synthesis method, based on random sampling and locality constraint (RSLCR).