# A Joint Training Model for Face Sketch Synthesis
## 1 Introduction
They randomly sampled the training photo and sketch patches in place of a neighbor search, and then employed the locality constraint to model the distinct correlations between the test photo patch and sampled photo patches while calculating the reconstruction weight coefficients. However, the target sketch patch reconstruction obtained by weighted-averaging the hundreds of sampled sketches can be regarded as a low-pass filter process, which results in blurred synthesized sketches. In addition, the Bayesian inference was utilized in reference [14] to incorporate the neighboring constraint in both the neighbor selection and reconstruction weight representation, which can obtain impressive performance. Apart from the exemplar-based method, deep learning techniques were also applied to face sketch synthesis, generating new trends. [PERSON] et al. [15] first proposed a fully convolutional network (FCN) to learn end-to-end mapping from photos to sketches. It consisted of six convolutional layers with rectified linear units as activation functions. [PERSON] et al. [16] utilized the branched FCN to structure a decomposition representation learning framework for sketch synthesis. Additionally, the generative adversarial network (GAN) [17] was developed for image style transformation (e.g., photo-to-sketch generation or vice versa). The deep learning-based sketch synthesis methods can preserve textural structure well; however, serious noise effects occur in the synthesized results. This is mainly because of the limited available of training data, which is insufficient to train large networks robustly [18]. A common problem with these exemplar-based approaches is that they ignore the role of training sketches when calculating reconstruction weights. This is because the basic assumption of these exemplar-based methods is that a photo patch and its corresponding sketch patch have a similar geometric manifold structure. If two photo patches are similar, then their sketch patch counterparts are also similar. However, owing to potential misalignment, the reconstruction weight obtained from the test photo patch and selected training photo patches may not be suitable for sketch patches reconstruction [19]. We propose a new exemplar-based method, the joint training model, to solve the problem. In our method, the training photo and the sketch patches are concatenated. Instead of directly using the sketch patches, the high-pass filtered component of the training sketches are adopted to reduce the effect of the modality difference between a photo and a sketch. Then, we employ the offline random sampling method to select the joint training photo and the sketch patches for the test photo patch. Moreover, a modified locality constraint is designed to calculate the reconstruction weight. With the obtained reconstruction weight, the target sketch patch can be synthesized. Experimental results indicate that the proposed joint model significantly eliminates noise and improves the synthesized sketch quality. It also preserves the detail information of the test photo, which other methods cannot do. Therefore, synthesized face sketch images using our method achieve higher accuracy in face sketch recognition. The contributions of this paper are summarized as below. 1. To consider the training sketches during the reconstruction weight representation process, a joint training model is proposed to integrate the training photo and sketch information. 2. We design a modified locality constraint that modulates the reconstruction weight through the distance between the high-pass filtered images of test patches and the sampled training sketch patches. 3. The proposed method yields high quality sketches with more detail information and less noise over the wide range of datasets, promoting the accuracy of the sketch-based suspect identification. The organization of the rest of the paper is as follows. Section 2 introduces the relevant works, including some example-based sketch synthesis methods, which are the basic works of the proposed method.