Title: UNLABELED DISENTANGLING OF GANS WITH GUIDED SIAMESE NETWORKS

Abstract: Disentangling underlying generative factors of a data distribution is important for interpretability and generalizable representations. In this paper, we introduce two novel disentangling methods. Our first method, Unlabeled Disentangling GAN (UD-GAN, unsupervised), decomposes the latent noise by generating similar/dissimilar image pairs and it learns a distance metric on these pairs with siamese networks and a contrastive loss. This pairwise approach provides consistent representations for similar data points. Our second method (UD-GAN-G, weakly supervised) modifies the UD-GAN with user-defined guidance functions, which restrict the information that goes into the siamese networks. This constraint helps UD-GAN-G to focus on the desired semantic variations in the data. We show that both our methods outperform existing unsupervised approaches in quantitative metrics that measure semantic accuracy of the learned representations. In addition, we illustrate that simple guidance functions we use in UD-GAN-G allow us to directly capture the desired variations in the data.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are generative model estimators, where two neural networks (generator and discriminator) are trained in an adversarial setting, so that likelihood-based probabilistic modeling is not necessary. This works particularly well for sampling from a complex probability distribution, such as images. Although GANs yield realistic looking images (Radford et al., 2015) , the original formulation in (Goodfellow et al., 2014) only allows for randomly sampling from the data distribution without disentangled structural or semantic control over the generated data points.

One way to disentangle the generation process is to use conditional GANs (Mirza & Osindero, 2014; Odena et al., 2017) . These models modify the generator by conditioning it with supervised labels. Then, they either take the same labels as input in the discriminator (Mirza & Osindero, 2014) and measure the image-label compatibility, or classify the correct label at the output, given the generated image (Odena et al., 2017) . Conditional GANs rely on a dataset with labels, which might not always be available or might be time-consuming to collect.

In this paper, we propose two GAN-based methods that learns disentangled representations without using labeled data. Our first method, Unlabeled Disentangling GAN (UD-GAN), generates image pairs, then embeds them with Siamese Networks (Chopra et al., 2005) , and finally learns a distance metric on a disentangled representation space. Whereas our second method, UD-GAN-G, uses guidance functions to restrict the input to our siamese networks, so that they capture desired semantic variations.

2. RELATED WORK:

There have been many studies on learning disentangled representations in generative models, which can be grouped into the level of supervision/labeled data they require.

Disentangled representations (supervised). In (Zhu et al., 2014; Yang et al., 2015) , the identity and the viewpoint of an object are disentangled via reconstructing the same object from a different viewpoint and minimizing a reconstruction loss. Whereas in (Kingma et al., 2014; Makhzani et al., 2016) , the style and category of an object is separated via autoencoders, where an encoder embeds the style of an input image to a latent representation, and a decoder takes the category and style input to reconstruct the input image. In (Tran et al., 2017; Yin et al., 2017) , autoencoders and GANs are combined to decompose identity and attribute of an object, where the disentangled representation is obtained at the encoder outputs, and image labels are used at the output of the discriminator.

Disentangled representations (semi-supervised). In (Reed et al., 2014) , they clamp the hidden units for a pair of images with the same identity but with different pose or expression to have the same identity representation. Whereas in (Kulkarni et al., 2015) , synthesized images are used to disentangle pose, light, and shape of an object by passing a batch of images where only one attribute varies and the rest of the representation is clamped to be the same. These techniques only require a batch of samples with one attribute different at a time.

Disentangled representations (unsupervised). InfoGAN (Chen et al., 2016) is an unsupervised technique that discovers categorical and continuous factors by maximizing the mutual information between a GAN's noise variables and the generated image. β-VAE (Higgins et al., 2017) and DIP-VAE (Kumar et al., 2018) are unsupervised autoencoder-based techniques that disentangle different factors in the latent representation of an encoded image. In β-VAE, the KL-divergence between the latent and a prior distribution is weighted with a factor β > 1 to encourage disentanglement in the posterior latent distributions. Wheres in DIP-VAE, the covariance matrix of the latent distribution is encouraged to be an identity matrix, thus leading to uncorrelated latent representations.

For all of the unsupervised methods, after a model is trained, a human needs to investigate which factors map to which semantic property. In addition, as the methods are unsupervised, not all desirable factors might be represented. In contrast, our method builds on existing approaches with two important modifications: (i) We operate on pairs of similar/dissimilar image pairs. (ii) We compute the image embeddings using separate networks, which allows us to guide the disentangling process with information restriction.

3. UNLABELED DISENTANGLING GAN:



3.1. BACKGROUND: GENERATIVE ADVERSARIAL NETWORKS:

In GANs, the generator, G(.), maps a latent variable z, which has an easy-to-sample distribution, into a more complex and unknown distribution, such as images. On the other hand, the discriminator D(.) tries to distinguish real images from the ones that are generated by G. In (Goodfellow et al., 2014) , the training is performed as a minimax game as follows:

EQUATION

where P R and P Z are the probability distributions of real images and the latent variable z, respectively. We train our GAN by using the loss in equation 1. In order to increase stability, we modify the generator loss by maximzing log(D(G(z))), instead of minimizing the second term in equation 1.

3.2. A NOVEL GAN ARCHITECTURE: UD-GAN:

In a standard GAN setting, all of the variation in the distribution of real images is captured by the latent variable z. However, a single dimension or a slice of z does not necessarily have a semantic meaning. In this paper, our target is to slice the latent variable into multiple vectors, where each vector controls a different semantic variation.

Our network architecture is visualized in Figure 1 . In our method, the latent vector z = [q 1 , q 2 , ..., q N A ] is the concatenation of N A knobs, {q i } N A i=1 , which represent different attributes we aim to disentangle. One can add a final variable that captures the variation (and the noise) that is not picked up by the knobs. In our experiments, this additional variable did not have a notable effect. In our notation, q i refers to all of the knobs, except q i . In order to train our model, first, for each q i , we sample two different vectors, q

(1) i and q

(2) i from Unif(-1, 1). If we would attempt to form a batch by combinatorially concatenating all knob samples, we get a batch size of 2 N A , which grows exponentially with the number of attributes. To avoid this computational burden, we train our model through stochastic sampling of one attribute at a time. For example, if the i th attribute is chosen, we generate four images as shown in Figure 1 . The image pairs that are generated with the same q i vectors, {x 11 , x 12 } or {x 21 , x 22 }, should have the same i th attribute, regardless of the values of q i . We can ensure this via embedding the generated image pairs into a representation space with Siamese Networks (Chopra et al., 2005) , which are denoted as φ i (.), and then learning a distance metric on the embedding vectors by employing Contrastive Loss (Hadsell et al., 2006 ). An optional guidance function is used to restrict the information that goes into a siamese network, thus letting us approximate a desired representation space.

The guidance is disabled for our unsupervised UD-GAN approach. Whereas for UD-GAN-G, the guidance is a simple, user-defined function, which is discussed in Section 3.3.

We use a Contrastive Loss function to pull similar image pairs together, and push dissimilar pairs apart as follows:

EQUATION

where, L φ i is the Contrastive Loss for the i th Siamese Network φ i (.), the function

ρ i (x ni1 , x ni2 ) = φ i (x ni1 ) -φ i (x ni2

) 2 is a shorthand for embedding distance between x ni1 and x ni2 , and γ

(1,2) i

is an adaptive margin of the form γ

(1,2) i = q (1) i -q (2) i 2 .

Using an adaptive margin makes the distance between two latent samples semantically meaningful and we empirically found that it improves the training stability.

The discriminator network D is not modified and is trained to separate real and generated image distributions. Donahue et al. (2018) use a similar latent variable slicing for capturing illumination and pose variations of a face with a fixed identity. Their discriminator needs image pairs, which must be labeled for real images, to judge the quality and identity of the faces. Our method does not require any labels for the real images. Instead, we create similar and dissimilar image pairs via concatenating latent variables and generating image batches. Our final loss function is:

EQUATION

where, L W GAN is the GAN loss described in equation 1, λ φ i is the weight of the embedding loss, and the sampling of the latent variables depends on i and is performed as described above.

3.3. DISENTANGLING WITH GUIDANCE FUNCTIONS: UD-GAN-G:

A guidance function reduces the information content that flows into a siamese network and causes the corresponding embedding space to capture only the variations in the restricted input. For example, consider we want to capture the hair-related attributes in the CelebA dataset (Liu et al., 2015) , which contains aligned images of human faces. By cropping every region but the top part of a generated image, we are able to guide φ top (.) to learn only the variations in the "Hair Color" as shown in the first row of Figure 2 . Note that, the knob q top (that corresponds to φ top (.)) changes the hair color not only at the cropped part of the image but as a whole. This is due to the interplay between the adversarial part of our loss (see equation 3), which enforces global realism in images, and the contrastive loss, which administers disentangled representations. As shown in Figure 2 , different guidance functions leads to capturing different variations in the CelebA dataset.

Crop Guidance

φ top q top φ miu q miu φ mil q mil φ bot q bot

Varying Latents -1.0 1.0 Varying latent variables that correspond to guided siamese networks captures desired variations.

3.4. PROBABILISTIC INTERPRETATION:

We can gain a probabilistic interpretation of our method on a toy example. Let us assume a problem, where we want to generate images of colored polygons (see Figure 1 ), where there are two independent factors of variation: shape and color, which we want to capture using two knobs q i and q j , respectively. When we set q j to a certain value and vary q i , we want to generate polygons with the same color, but different shapes, and vice versa.

Let P be the probability distribution of colored polygons. For each attribute, P can be decomposed into a mixture distribution as follows:

P = Ni k=1 π (k) i Q (k) i ← for attribute i, P = Nj k=1 π (k) j Q (k) j ← for attribute j (4)

where, Q

(k) i is a mixture component and π (k) i

is its corresponding probability of choosing it, and N i is the number of different values an attribute (in our example, i corresponds to shape) can take. A similar explanation can be made for attribute j, i.e. color. For the sake of this analysis, we accept that for each attribute, P can be decomposed into different discrete mixture distributions as shown in Figure 3 . For this specific case, Q

(1) i and Q

(2) i are the distributions of colored squares and colored diamonds, respectively. For the color attribute, which is indexed by j, each Q (k) j corresponds to a distribution of polygons with a single color (i.e., green polygons).

Our contrastive loss in equation 2 has two terms. The first term is minimizing the spread of each mixture component

Q (k)

i . This spread is inversely related to disentanglement. If all samples from

Q (1) i Q (2) i γ (1,2) i (a) φ i -space (shape) Q (1) j Q (2) j Q (3) j Q (4) j γ (2,4) j (b) φ j -space (color)

Figure 3 : Illustration of the embedding spaces and separated probability distributions after training our model.

Q (k) i

are mapped to the same embedding vector, the effect of j (and any other attribute) on the representation φ i (.) disappears and disentangling is achieved. During training, we stochastically go through all embedding spaces and minimize their spread, thus resulting in a disentangled representation in Table 9 in Appendix G.

The second term in equation 2 separates all Q (k) i from each other using an adaptive margin γ (1,2) i . This margin depends on the difference between input latent pairs, so that the resulting embedding space is smooth. In other words, we separate rectangles, circles, and ovals from each other, but circles should be closer to ovals than squares, due to their relative similarity. In the following, we focus on the shape attribute that is represented by i, however, derivations carry over to the color attribute j.

In order to separate the probability distributions over image embeddings, one can maximize a divergence between all pairs from Q (k) i . One way to measure the distance between these distributions is to use the unbiased estimator of the energy distance (Székely & Rizzo, 2004) :

EQUATION

The energy distance in equation 5 can be interpreted as an instance of Maximum Mean Discrepancy (Bińkowski et al., 2018) and resembles the Contrastive Loss (Hadsell et al., 2006) . We can rewrite equation 5 using the Contrastive Loss in equation 2 as follows:

D E = -L φ i + 1 4 2 nj =1 2 mj =1 ρ i (x 1nj , x 2mj ) 2 + max(0, γ (1,2) i -ρ i (x 1nj , x 2mj )) 2 (6)

Each element in the second sum is quadratic function and has its minimum at

ρ i (x 1nj , x 2mj ) = γ (1,2) i

/2 and the value of the minimum is γ

(1,2) i 2 /2. So, we can rewrite equation 6 as follows:

EQUATION

Therefore, as the margin γ

(1,2) i depends only on the input latent variables and is not trainable, minimizing our embedding loss L φ i maximizes the lower bound for the energy distance D E . This corresponds to learning a Siamese Network φ i (.) that separates two probability distributions Q

(1) i and Q

(2) i , i.e., colored squares and colored diamonds, from each other and minimizes the spread of each distribution, thus resulting in disentangling the effect of j from i. The same derivation can be made for the color attribute. After jointly training the Siamese Networks, we can achieve the embedding spaces represented in Figure 3 . An example of one such disentangled embedding space is illustrated for MNIST (LeCun & Cortes, 2010) digits in Figure 4 in Appendix B.

4. EXPERIMENTS:



4.1. SETUP:

We perform our experiments on a server with Intel Xeon Gold 6134 CPU, 256GB system memory, and an NVIDIA V100 GPU with 16GB of graphics memory. Our generator and discriminator architectures are outlined in our Appendix A. Each knob is a 1-dimensional slice of the latent variable and is sampled from Unif(-1, 1). We use ADAM (Kingma & Ba, 2014) as an optimizer for our training with the following parameters: learning rate=0.0002 and β 1 = 0.5. We will release our code after the review process.

Datasets. We evaluate our method on two image datasets: (i) the CelebA dataset (Liu et al., 2015) , which consists of over 200,000 images of aligned faces. We cropped the images to 64 × 64 pixels in size. (ii) the 2D Shapes (Higgins et al., 2017) , which is a dataset that is synthetically created with different properties, such as shape, scale, orientation, and x-y locations. Both datasets are divided into training and test sets with a 90%-10% ratio. The weight values for the contrastive loss is λ φ = 1 for the CelebA dataset and λ φ = 5 for the 2D shapes dataset. We use a 32 and 10-dimensional latent variables for the CelebA and the 2D Shapes datasets, respectively.

Baselines. We have two versions of our algorithm. UD-GAN refers to the results that are obtained without any guidance at the input of our siamese networks, whereas UD-GAN-G represents a guided training. We compare our method against β-VAE (Higgins et al., 2017) , DIP-VAE (Kumar et al., 2018) , and InfoGAN (Chen et al., 2016) to compare against both autoencoder and GAN-based approaches. We get the quantitative and visual results for β-VAE and DIP-VAE from (Higgins et al., 2017) and (Kumar et al., 2018) , and use our own implementation of InfoGAN for training and testing. The same generator/discriminator architecture is used for InfoGAN and our method.

Guidance. For the CelebA dataset, the first 28 of 32 latent knobs are unguided and therefore are processed by the same siamese network that outputs a 28-dimensional embedding vector1 . Whereas the remaining four knobs correspond to four siamese networks (φ top , φ miu , φ mil , φ bot ) that are guided with cropped images in Figure 2 . For the 2D shapes dataset, we have 10 knobs, where the first 7 dimensions are unguided. In order to guide the remaining three networks, we estimate the center of mass ( Mx , My ) and the size Ŝ of the generated object and feed them to our siamese networks, φ X ( Mx ), φ Y ( My ), and φ S ( Ŝ). More information for this computation can be found in Appendix D.

4.2. RESULTS:

Disentanglement Metric. This metric was proposed by Higgins et al. (2017) and measures whether learned disentangled representations can capture separate semantic variations in a dataset. In β-VAE and DIP-VAE, this representation is the output of the encoder, i.e., the inferred latent variable. For InfoGAN, we use the representation learned by the discriminator. In our method, we use the concatenated outputs of our siamese networks, which we denote as φ(.).

The disentanglement metric scores for different methods are illustrated in Table 1 . Here, we can see that both of our methods outperforms the baseline on the CelebA dataset. All of the baseline approaches relate the latent variables to generated images on per-image basis. Whereas our approach attempts to relate similarities/differences of latent variable pairs to image pairs, which provides a discriminative image embedding, where each dimension is invariant to unwanted factors (Hadsell et al., 2006) .

For both datasets, our guided network (UD-GAN-G) performs better than our unguided approach, especially on the CelebA dataset. This might be due to the correlations between irrelevant attributes. For example the correlation coefficient between "Wearing Lipstick" and "Wavy Hair" attributes is 0.36, although they are not necessarily dependent. One of our guided networks receive the cropped image around the mouth of a person, which prevents cluttering it with hairstyle. Therefore, this guidance provides better disentanglement and results in an improved score as shown in Table 1 . Due to containing simple synthetic images, our disentanglement scores for the 2D shapes dataset are very high. The reason we get 100.0 score on our guided method is because of the guidances we choose, which are highly correlated with the ground truth labels, as shown in Table 7 in Appendix D. 2018) introduced a binary classification metric for the CelebA attributes that project a test image embedding onto average embedding vectors of attributes. In Table 2 , we compare our method against baseline approaches on CelebA attribute classification accuracy using the aforementioned projection vector. Similar to the results in Table 1 , our guided approach slightly outperforms our unguided method and the other completely unsupervised techniques. This is because some attributes in the CelebA dataset can be spatially isolated via cropping, which leads to a better classification performance. For example, the attributes that are related to hair (Black Hair, Blond Hair, Wavy Hair) and mouth (Mouth Slightly Open, Wearing Lipstick) are captured better by the guided approach, because our top and bottom crops (see Figure 2 ) are detaching the effects of other variations and are making attributes less correlated. The accuracy on the attribute "Bangs" is worse on the guided approach. This might be due to heuristic cropping we perform that divides the relevant image region into two slits. Visual Comparison. In Table 3 , we illustrate images generated by different methods on the CelebA dataset. Each of the three rows capture the change in a semantic property: smile, azimuth, and hair color, respectively. Within each image group, a latent dimension is varied (from top to bottom) to visualize the semantic change in that property. Compared to adversarial methods, such as InfoGAN and UD-GAN-G, the DIP-VAE method generates blurrier images, due to the data likelihood term in VAE-based approaches, which is usually implemented as a pixel-wise image reconstruction loss. In GAN-based approaches, this is handled via a learnable discriminator in an adversarial setting. In Table 1 and 2, we quantitatively show the advantage of using our guided approach. Another advantage is to have better control over the captured attributes. For example, in all unsupervised approaches (including UD-GAN), we need to check which latent dimension represents corresponds to which visual attribute. In some cases, a semantic attribute might not be captured due to the correlated nature of a dataset. Whereas, in UD-GAN-G, we directly obtain the variations in smile, azimuth, and hair color through cropping the bottom, middle, and top part of our images, respectively. Thanks to our guidance in Figure 2 , we can directly manipulate these three attributes using the knobs q bot , q mil , and q top as shown in Table 3 .

The same trend is true for the 2D Shapes dataset results in Table 4 . Although the X and Y positions and the scale of the synthetic object is captured by both our unsupervised and guided approaches, the guidance we choose directly captures the desired feature on in advance chosen knobs q X , q Y , and q S , respectively. Table 3 : Images generated by varying a latent dimension, which corresponds to a semantic property.

DIP-VAE:

InfoGAN UD-GAN-G Smile -1.0     q bot     1.0 Azimuth -1.0     q mil     1.0 Hair Color -1.0     q top     1.0

Table 4: Generated images for the 2D Shapes dataset by varying a latent dimension, which corresponds to a semantic property (first row: UD-GAN, second row: UD-GAN-G).

X Position Y Position Scale:

UD-GAN UD-GAN-G -1.0     q X     1.0 -1.0     q Y     1.0 -1.0     q S     1.0 4.3 DISCUSSION

In completely unsupervised approaches, there is no guarantee to capture all of the desired semantic variations. The main premise behind UD-GAN-G is to find very simple, yet effective ways to capture some of the variation in the data. This weak supervision helps us to obtain proxies to certain semantic properties, so that we get the desired features without training the model multiple times with different hyperparameters or initializations.

In the aligned the CelebA dataset, each face is roughly centered around the nose. This reduces the variation and simplifies the problem of guidance design, as we show in Figure 2 . In more complex scenarios, where the objects can appear in a large variety of scales, translations, and viewpoints, one can use a pre-trained object detection and localization method, such as YOLO (Redmon et al., 2015) , as a guidance network. This enables us to use the knowledge obtained from a labeled dataset, such as ImageNet (Russakovsky et al., 2015) to disentangle a new unlabeled dataset. Note that backpropagating the gradients of a deep network into an image might cause adversarial samples (Szegedy et al., 2014) . However, the discriminator can alleviate this by rejecting problematic images.

In order to backpropagate the gradients from the siamese networks to the generator, the guidance function we use needs to be differentiable. This might pose a limitation to our method; however, differentiable relaxations can instead be used to guide our network. For example, one can employ differentiable relaxation of the superpixel segmentation in (Jampani et al., 2018) to disentangle a low-level image segmentation.

Our latent variables are sampled from a uniform distribution. In addition, image similarity is measured by using L2-distance between a pair of image embeddings. We experimented with modeling some latent dimensions as categorical variables. However, we encountered training stability issues, due to computing the softmax loss between two learnable categorical image embeddings, instead of one embedding and one fixed label vector as it is usually done. We plan to tackle that problem in our future work.

5. CONCLUSION:

In this paper we introduced UD-GAN and UD-GAN-G, novel GAN formulations which employ Siamese networks with contrastive losses in order to make slices of the latent noise space disentangled and more semantically meaningful. Our experiments encompassed guided and unguided approaches for the embedding networks, and illustrated how our methods can be used for semantically meaningful image manipulation. Our qualitative and quantiative results confirm that our method can adjust well to the intrinsic factors of variation of the data and outperform the current state-of-the-art methods on the CelebA and 2D Shapes datasets. In future work, we plan to investigate more powerful forms of embedders, e.g. extracting information from pre-trained networks for semantic segmentation and landmark detection. This allows for even more powerful novel image manipulation techniques.

:

Note that each dimension is still an independently evaluated

Figures and Tables:

Figure 1: The flowchart of our architecture. Sampled latents from different attributes are combined into latent vectors. Generated images are grouped with respect to different attributes (here, represented by shape) by Siamese Networks (denoted as φ i ).

Figure 2: (left) Four of our siamese networks are guided with differently cropped images. (right) Varying latent variables that correspond to guided siamese networks captures desired variations.

Disentanglement metric scores(Higgins et al., 2017), which measure how strongly and independently the dataset attributes are captured by a method.

Table content: <table><tr><td>Method</td><td colspan="2">2D Shapes CelebA</td></tr><tr><td>β-VAE</td><td>99.2</td><td>7.1</td></tr><tr><td>InfoGAN</td><td>88.4</td><td>12.3</td></tr><tr><td>DIP-VAE</td><td>98.7</td><td>14.8</td></tr><tr><td>UD-GAN</td><td>99.1</td><td>15.4</td></tr><tr><td>UD-GAN-G</td><td>100.0</td><td>16.5</td></tr><tr><td colspan="2">CelebA Attribute Classification. Kumar et al. (</td><td/></tr></table>

CelebA attribute classification accuracy.

Table content: <table><tr><td>Method</td><td>Arched Eyebrows</td><td>Attractive</td><td>Bangs</td><td>Black Hair</td><td>Blond Hair</td><td>Heavy Makeup</td><td>Male</td><td>Mouth Slightly Open</td><td>No Beard</td><td>Wavy Hair</td><td>Wearing Hat</td><td>Wearing Lipstick</td></tr><tr><td>β-VAE</td><td colspan="12">71.6 72.6 90.6 79.3 89.1 79.3 83.5 76.1 86.9 67.8 95.9 82.4</td></tr><tr><td>DIP-VAE</td><td colspan="12">73.7 73.2 90.9 80.6 91.9 81.5 85.9 75.9 85.3 71.5 96.2 84.7</td></tr><tr><td>InfoGAN</td><td colspan="12">74.7 73.8 90.9 80.9 91.5 82.6 87.4 76.9 88.7 74.3 97.3 86.9</td></tr><tr><td>UD-GAN</td><td colspan="12">75.0 74.8 90.5 82.1 91.5 84.2 86.2 79.7 87.5 75.1 96.5 85.6</td></tr><tr><td colspan="13">UD-GAN-G 75.0 75.5 90.2 82.3 92.1 84.2 89.9 82.2 87.7 75.6 96.7 87.3</td></tr></table>

Peer review (iclr2019):
 [Edit] I changed my rating from 4 to 5 based on the author responses. ======= This paper proposed a GAN that learns a disentangled factors of variations in unsupervised (or weakly-supervised) manner. To this end, the proposed method incorporates a contrastive loss together with Siamese network, which encourages the generator to output smaller variations in samples if they are drawn by varying the same latent factors. The proposed idea is evaluated on simple datasets such as MNIST and centered faces, and show that it is able to learn disentangled latent codes by incorporating some heuristics. Although the paper presents an interesting and reasonable idea, I think the paper is incomplete and in the proof-of-concept stage. In terms of method, the guidance for learning Siamese networks are designed heuristically (e.g. edges, colors, etc.) which limits its applicability over various datasets; I think that designing more principled approach to build such guidances from data should be one of the key contributions of the paper. In terms of evaluation, the authors only presented a few qualitative results on simple datasets, which is not comprehensive and convincing. In conclusion, I suggest a reject of this paper due to the lacks of comprehensive study and evaluation.