Year: 2019

Abstract:
We study the problem of learning to map, in an unsupervised way, between domains A and B, such that the samples b ∈ B contain all the information that exists in samples a ∈ A and some additional information. For example, ignoring occlusions, B can be people with glasses, A people without, and the glasses, would be the added information. When mapping a sample a from the first domain to the other domain, the missing information is replicated from an independent reference sample b ∈ B. Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image. Our solution employs a single two-pathway encoder and a single decoder for both domains. The common part of the two domains and the separate part are encoded as two vectors, and the separate part is fixed at zero for domain A. The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term. Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains. We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc.

heading: 1 INTRODUCTION
text: In the problem of unsupervised domain translation, the algorithm receives two sets of samples, one from each domain, and learns a function that maps between a sample in one domain to the analogous sample in the other domain (Zhu et al., 2017a; Kim et al., 2017; Yi et al., 2017; Benaim & Wolf, 2017; Liu & Tuzel, 2016; Liu et al., 2017; Choi et al., 2017; Conneau et al., 2017; Zhang et al., 2017a;b; Lample et al., 2018). The term unsupervised means, in this context, that the two sets are unpaired.
In this paper, we consider the problem of domain B, which contains a type of content that is not present at A. As a running example, we consider the problem of mapping between a face without eyewear (domain A) to a face with glasses (domain B). While most methods would map to a person with any glasses, our solution is guided and we attach to an image a ∈ A, the glasses that are present in a reference image b ∈ B. In comparison to other guided image to image translation methods, our method is considerably simpler. It relies on having a latent space with two parts: (i) a shared part that is common to both A and B, and (ii) a specific part that encodes the added content in B. By setting the second part to be the zero vector for all samples in A, a disentanglement emerges. Our analysis shows that this
leads to the ability to train with a simple, straightforward domain confusion term, while enjoying the generalization guarantees that would otherwise require a more elaborate loss.

heading: 1.1 PREVIOUS WORK
text: In image to image translation, the transformations are often captured by multi-layered networks of an encoder-decoder architecture. The existing solutions often assume a one to one mapping between the domains, i.e., that there exists a function y such that given a sample a in domain A, maps it to an analog sample in domain B. In fact, the circularity based constraints by Zhu et al. (2017a); Kim et al. (2017); Yi et al. (2017) are based on this assumption, since going from one domain to the other and back, it is assumed that the original sample is obtained, which requires no loss of information. However, to employ an example made popular by Zhu et al. (2017a), when going from a zebra to a horse, the stripes are lost, which results in an ambiguity when mapping in the other direction.
This shortcoming was identified by the literature, and a few contributions present many to many mappings. These include the augmented CycleGAN Almahairi et al. (2018), which adds a random vector to each domain. A completely different approach is taken by the NAM method (Hoshen & Wolf, 2018), in which multiple solutions are obtained by considering multiple random initializations. In our work, multiplicity of outcomes arises from using a reference (guide) image.
A powerful way to capture relations between the two domains, is by employing separate autoencoders for the two domains, but which share many of the weights (Liu & Tuzel, 2016; Liu et al., 2017). This leads to a shared representation: while the low-level image properties, such as texture and color, are domain-specific and encoded/decoded separately, the mid- and top-level properties are common to both domains and are processed by identical replicas of the same layers. In our work, we rely on a shared encoder for both domains in order to enforce a shared representation.
Our method performs one-sided mapping, from A to B, and does not learn the mapping in the other direction. Benaim & Wolf (2017) perform one-sided mapping using a distance based constraint, which we do not employ. That method is symmetric in the two domains and in many experiments, the distance constraint is used in tandem with the cycle constraint. In our case, the two domains are asymmetric and one domain contains an added content. Our method is inherently asymmetric, reflecting the asymmetry between the source image and the guide image, which contains the additional content. The work by Taigman et al. (2017); Hoshen & Wolf (2018) map in an asymmetric way, but rely on the existence of a perceptual distance, which we do not employ.
Guided Translation The most relevant work contains very recent and concurrent methods in which the mapping between the domains employs two inputs: a source image a and a reference (guide or attribute) image b. Tab. 1 compares these methods to ours along two axes: (i) where sharing occurs in the architecture, and (ii) the number of trained sub-networks of each type. The sharing can be of layers between different encoders and decoders, following, e.g., Liu & Tuzel (2016); sharing of a common part of a latent space; and using the same encoder or decoder for multiple
domains. The networks are of four types: encoders, which map images to a latent space, generators (also known as decoders), which generate images from a latent representation, discriminators that are used as part of an adversarial loss, and other, less-standard, networks.
It is apparent that our method is considerably simpler than the literature methods. The main reason is that our method is based on the emergence of disentanglement, as detailed in Sec. 4. This allows us to to train with many less parameters and without the need to apply excessive tuning, in order to balance or calibrate the various components of the compound loss.
The MUNIT architecture by Huang et al. (2018), like our architecture, employs a shared latent space, in addition to a domain specific latent space. Their architecture is not limited to two domains1 and unlike ours, employs separate encoders and decoders for the various domains. The type of guiding that is obtained from the target domain in MUNIT is referred to as style, while in our case, the guidance provides content. Therefore, MUNIT, as can be seen in our experiments, cannot add specific glasses, when shifting from the no-glasses domain to the faces with eyewear domain.
The EG-UNIT architecture by Ma et al. (2018) presents a few novelties, including an adaptive method of masking-out a varying set of the features in the shared latent space. In our latent representation of domain A, some of the features are constantly zero, which is much simpler. This method also focuses on guiding for style and not for content, as is apparent form their experiments.
The very recent DRIT work by Lee et al. (2018) learns to map between two domains using a disentangled representation. Unlike our work, this work seems to focus on style rather than content. The proposed solution differs from us in many ways: (1) it relies on two-way mapping, while we only map from A to B. (2) it relies on shared weights in order to ensure that the common representation is shared. (3) it adds a VAE-like (Kingma & Welling, 2014) statistical characterization of the latent space, which results in the ability to sample random attributes. As can be seen in Tab. 1, the solution of Lee et al. (2018) is considerably more involved than our solution.
DRIT (and also MUNIT) employ two different types of encoders that enforce a separation of the latent space representations to either style or content vectors. For example, the style encoder, unlike the content encoder, employs spatial pooling and it also results in a smaller representation than the content one. This is important, in the context of these methods, in order to ensure that the two representations encode different aspects of the image. If DRIT or MUNIT were to use the same type of encoder twice, then one encoder could capture all the information, and the image-based guiding (mixing representations from two images) would become mute. In contrast, our method (i) does not separate style and content, and (ii) has a representation that is geared toward capturing the additional content.
The work most similar to us in its goal, but not in method, is the PairedCycleGAN work by Chang et al. (2018). This work explores the single application of applying the makeup of a reference face to a source face image. Unfortunately, the method was only demonstrated on a proprietary unshared dataset and the code is also not publicly available, making a direct comparison impossible at this time. The method itself is completely different from ours and does not employ disentanglement. Instead, a generator with two image inputs is used to produce an output image, where the makeup is transfered between the input images, and a second generator is trained to remove makeup. The generation is done separately to k = 5 pre-segmented facial regions, and the generators do not employ an encoder-decoder architecture.
Lastly, there are guided methods, which are trained in the supervised domain, i.e., when there are matches between domain A and B. Unlike the earlier one-to-one work, such as pix2pix Isola et al. (2017b), these methods produce multiple outputs based on a reference image in the target domain. Examples include the Bicycle GAN by Zhu et al. (2017b), who also applied, as baseline in their experiments, the methods of Bao et al. (2017); Gonzalez-Garcia et al. (2018).
Other Disentanglement Work InfoGAN (Chen et al., 2016) learns a representation in which, due to the statistical properties of the representations, specific classes are encoded as a one-hot encoding of part of the latent vector. In the work of Lample et al. (2017); Hadad et al. (2018), the representation is disentangled by reducing the class based information within it. The separate class based information is different in nature from our multi-dimensional added content. Cao et al. (2018),
1Ours method can be readily extended to multiple target domains B1, . . . , Bk, but this is not explored here.
which builds upon Hadad et al. (2018), performs guided image to image translation, but assumes the availability of class based information, which we do not.

heading: 2 PROBLEM SETUP
text: We consider a setting with two domains A = (XA, DA) and B = (XB , DB). Here, XA,XB ⊂ RM and DA, DB are distributions over them (resp.). The algorithm is provided with two independent datasets SA = {ai}m1i=1 and SB = {bj} m2 j=1 of samples from the two domains that were sampled in the following manner: SA i.i.d∼ Dm1A and SB i.i.d∼ Dm2B (1) We denote, DA,B := DA ×DB the distribution of sampling (a, b) for a ∼ DA and b ∼ DB independently. We assume a generative model, in which b is specified by a sample a and a specification c from a third unknown domainC = (XC , DC) of specifications whereDC is a distribution over the metric space XC ⊂ RN . Formally, there is an invertible function u(b) = (u1(b), u2(b)) ∈ XA×XC that takes a sample b ∈ XB and returns the content u1(b) of b and the specification u2(b) of b. The goal is to learn a target function y : XA × XB → XB such that:
y(a, b) ∼ DB where: a ∼ DA, b ∼ DB ,a |= b and u(y(a, b)) = (a, u2(b)) (2)
Informally, the function y takes two samples a and b and returns the analog of a in B that has the specification of b. For example, A is the domain of images of persons, B is the domain of images of persons with sunglasses and C is the domain of images of sunglasses. The function y takes an image of a person and an image of a person with sunglasses and returns an image of the first person with the specified sunglasses. For simplicity, we assume that the target function is extended to inputs (b1, b2) ∈ X2B and u(y(b1, b2)) = (u1(b1), u2(b2)). In other words, b1 and b2 are mapped to a third b that has the content of b1 and the specification of b2. In particular, u(y(b, b)) = (u1(b), u2(b)) = u(b) and, therefore, y(b, b) = b.
Note that within Eq. 2, there is an assumption on the underlying distributionsDA andDB . Using the concrete example, our framework assumes that the distribution of images of persons with sunglasses and the distribution of images of persons without them is the same, except for the sunglasses. Otherwise, the distribution of the samples generated by y when resampling a would not be the same as DB . Note that we do not enforce this assumption on the data, and only employ it for our theoretical results, to avoid additional terms.
For two functions f1, f2 : X → R and a distribution D over X, we define the generalization risk between f1 and f2 as follows:
RD[f1, f2] := Ex∼D[`(f1(x), f2(x))] (3)
For a loss function ` : RM × RM → [0,∞). Typically, we use the L1(u,v) := ‖u − v‖1 or L2(u,v) := ‖u− v‖22 losses. The goal of the algorithm is to return a hypothesis h ∈ H, such that h : RM × RM → RM , that minimizes the generalization risk,
RDA,B [h, y] = E(a,b)∼DA,B [`(h(a, b), y(a, b))] (4)
This quantity measures the expected loss of h in mapping two samples a ∼ DA and b ∼ DB to the analog y(a, b) of a, that has the specification u2(b) of b. The main challenge is that the algorithm does not observes paired examples of the form ((a, b), y(a, b)) as a direct supervision for learning the mapping y : XA × XB → XB .

heading: 3 METHOD
text: In order to learn the mapping y, we only use an encoder-decoder architecture, in which the encoder receives two input samples and the decoder produces a single output sample that borrows from both input samples. As we discuss in Sec. 2, the goal of the algorithm is to learn a mapping h = g◦f ∈ H such that: g ◦ f(a, b) ≈ y(a, b). Here, f serves as an encoder and g as a decoder. The encoder f in our framework is a member of a set of encoders F , each decomposable into two parts and takes the following form:
f(a, b) = (e1(a), e2(b)) (5)
e1(a) ∼ e1 ◦DA
e1(b) ∼ e1 ◦DB
a ∼ DA
b ∼ DB
0
e2(b)
f(a,a)
f(b, b)
disc
e1
e1
0
e2
g
g
Figure 1: An illustration of the domains and functions employed in our work. DA and DB are the distributions of images in the two domains. e1 and e2 the two pathways of the encoder, g is the decoder, which is applied to f , which aggregates the output of the two encoder pathways. There are only three constraints used while training, shown in red: (i) a reconstruction loss in domain A, comparing g ◦ f(a,a) with a, (ii) a reconstruction loss in domain B, and (iii) a measure of the discrepancy between the distributions e1 ◦ DA and e1 ◦DB , measured with a domain confusion term.
where e1 : RM → RE1 serves as an encoder of shared content and e2 : RM → RE2 serves as an encoder of specific content. Here, E1 and E2 are the dimensions of the encodings. The decoder, g is a member of a set of decodersM. Each member ofM is a function g : RE1+E2 → RM . In order to learn the functions f and g, we apply the following min-max optimization:
min f∈F,g∈M max d∈C {LA + LB − λLD} , (6)
for some weight parameter λ > 0, of the following training losses (see Fig. 1):
LA = 1
m1 ∑ a∈SA ‖g(e1(a), 0E2)− a‖1 (7)
LB = 1
m2 ∑ a∈SB ‖g(e1(b), e2(b))− b‖1 (8)
LD = 1
m1 ∑ a∈SA l(d(e1(a)), 0) + 1 m2 ∑ b∈SB l(d(e1(b)), 1) (9)
where 0E2 is the vector of zeros of length E2, d is a discriminator network, and l(p, q) = −(q log(p) + (1 − q) log(1 − p)) is the binary cross entropy loss for p ∈ [0, 1] and q ∈ {0, 1}. The discriminator d is a member of a set of discriminators C that locates functions d : RM → [0, 1]. The discriminator d is trained to minimize LD and Eq. 9 is a domain confusion term (Ganin et al., 2016) encouraging the distribution e1 ◦ DA to be similar to the distribution e1 ◦ DB . Here and elsewhere, the composition f ◦ D of a function f and a distribution D denotes the distribution of f(x) for x ∼ D.

heading: 4 ANALYSIS
text: In this section, we provide a theoretical analysis for the success of the proposed method. For this purpose, we recall a few technical notations (Cover & Thomas, 2006): the expectation and probability operators symbols E,P, the Shannon entropy (discrete or continuous)H(X) := −EX [logP[X]], the conditional entropy H(X|Y ) := H(X,Y ) −H(Y ), the (conditional) mutual information (discrete or continuous) I(X;Y |Z) := H(X|Z)−H(X|Y,Z), the Kullback-Leibler (KL) divergence DKL(p‖q) := Ex∼p[log(p(x)/q(x))], and the total correlation TC(z) := DKL (P[z]‖ ∏ i P[zi]), where P[zi] is the marginal distribution of the i’th component of z. In particular, TC(z) is zero if and only if the components of z are independent, in which case we say that z is disentangled. For two distributions D1 and D2, we define the C-discrepancy between them to be discC(D1, D2) := supc1,c2∈C |RD1 [c1, c2]−RD2 [c1, c2]| = supc1,c2 |Ex∼D1`(c1(x), c2(x))−Ex∼D2`(c1(x), c2(x))|. The discrepancy behaves as an adversarial distance measure between two distributions, where d(x) = `(c1(x), c2(x)) is the discriminator that tries to differentiate between D1 and D2, for c1, c2 ∈ C. This quantity is being employed in Chazelle (2000); Ben-david et al. (2006); Mansour et al. (2009); Cortes & Mohri (2014).

heading: 4.1 GENERALIZATION BOUND
text: Thm. 1 upper bounds the generalization risk, based on terms that can be minimized during training, as well as on approximation terms. It is similar in fashion to the classic domain adaptation bounds proposed by Ben-David et al. (2010); Mansour et al. (2009).
Theorem 1. Assume that the loss function ` is symmetric and obeys the triangle inequality. Then, for any autoencoder h = g ◦ f ∈ H, such that f(x1,x2) = (e1(x1), e2(x2)) ∈ F is an encoder and g ∈M is a decoder, the following holds,
RDA,B [h, y] ≤RDB,B̂ [h, y] + ming∗∈M
{ RDA,B [g ∗ ◦ f, y] +RDB,B̂ [g ∗ ◦ f, y] } + discM(f ◦DA,B , f ◦DB,B̂)
(10)
where DB,B̂ is the distribution of (b, b) where b ∼ DB .
(The proofs can be found in the appendix.) Thm. 1 provides an upper bound on the generalization riskRDA,B [h, y], which is the argument that we would like to minimize. The upper bound is decomposed of three terms: a reconstruction error, an approximation error and a discrepancy term. The first term,
RDB,B̂ [h, y] = E(b,b)∼DB,B̂ [`(g ◦ f(b, b), y(b, b))] = Eb∼DB [`(g ◦ f(b, b), b)] (11)
is the reconstruction error for samples b ∼ DB . Since we do not have full access to DB , we minimize its empirical version (see Eq. 8). The second term, ming∈M { RDA,B [g ◦ f, y] +RDB,B̂ [g ◦ f, y] } , measures the minimal error obtained by a best fitting g ∈ M, such that g ◦ f ≈ y for inputs (a, b) ∼ DA,B and for inputs (b, b) ∼ DB,B̂ . Similar to (Ben-David et al., 2010; Mansour et al., 2009), this term is assumed to be small and is decreased as M’s capacity is increased. The third term, discM(f ◦DA,B , f ◦DB,B̂), is the discrepancy between the distributions f ◦DA,B and f ◦DB,B̂ . This term is small, if the distributions of (e1(a), e2(b)) (for a ∼ DA and b ∼ DB independently) and (e1(b), e2(b)) (for b ∼ DB) are close to each other. Since e1(a) and e2(b) are independent of each other (from the factorization DA,B = DA ×DB), if this term is small, then, e1(b) and e2(b) weakly depend on each other. Moreover, if the discrepancy term is zero, then, e1(b) and e2(b) are independent of each other.
While one can minimize the discrepancy term explicitly, by minimizing it with respect to e1 and e2, using a discriminator, we found empirically that this confusion term, which involves both parts of the embedding, is highly unstable. Instead, we show theoretically and empirically that there is a high likelihood for a disentangled representation (where e1(b) and e2(b) are independent) to emerge, and the discrepancy term can be replaced with the following discrepancy discM′(e1 ◦ DA, e1 ◦ DB), which measures the closeness between the distributions of e1(a) and of e1(b) for a ∼ DA and b ∼ DB , as is done in Eq. 9. Here, M′ is a set of discriminators that are similar in complexity to the ones in M. This discrepancy is simpler than the one in Eq. 10, since it does not involve a comparison of e2 between two distributions, nor the interaction between e1 and e2.
In Lem. 1, we show that if e1(b) and e2(b) are independent, then, disc(f ◦ DA,B , f ◦ DB,B̂) ≤ disc(e1◦DA, e1◦DB). Therefore, if a disentangled representation occurs, we can minimize disc(f ◦ DA,B , f ◦DB,B̂), by minimizing disc(e1 ◦DA, e1 ◦DB) instead.
Lemma 1. Let M be the set of neural networks of the form: c(x) = φ(Wr . . . φ(W2φ(W1x + q))), where, Wi ∈ Rdi×di+1 for i ∈ {1, . . . , r − 1}, q ∈ Rd2 and d1 = E1 + E2. In addition, φ(x1, . . . , xk) = (φ1(x1), . . . , φ1(xk)), for k ∈ N, (x1, . . . , xk) ∈ Rk and a non-linear activation function φ1 : R → R. LetM′ be the same asM with d1 = E1 (instead of d1 = E1 + E2). Let f(x) = (e1(x), e2(x)) be an encoder and assume that: e1(b) |= e2(b). Then,
discM(f ◦DA,B , f ◦DB,B̂) ≤ discM′(e1 ◦DA, e1 ◦DB) (12)

heading: 4.2 EMERGENCE OF DISENTANGLED REPRESENTATIONS
text: The following results are very technical and inherit many of the assumptions used by previous work. We therefore state the results informally here and leave the complete exposition to the appendix.
First, we extend Proposition 5.2 of Achille & Soatto (2018) from the case of multiclass classification to the case of autoencoders. In their work, the aim is to show the conditions in which a mid-level representation f of a multi-class classification neural network h = c ◦ f is both disentangled, i.e., TC(f(b)) is small, and is minimal, i.e., I(f(b); b) is small, where b is an input random variable. In their Proposition 5.2, they focus on a linear representation, i.e., f = W is a linear transformation, and they introduce a tight upper bound on the sum TC(Wb) + I(Wb; b).
In the general case, their goal is to show that for a neural network h = c◦f , both quantities TC(f(b)) and I(f(b); b) are small, when f that is a high level representation of the input. Unfortunately, they were unable to show that both terms are small simultaneously. Therefore, in their Cor. 5.3, they extend the bound of their Proposition 5.2 to show that only the mutual information I(f(b); b) is small and assume that the components of each mid-level representation of b in the layers of f are uncorrelated, which is a very restrictive assumption.
In our Lem. 2, we provide an upper bound for TC(f(b)) that is similar in fashion to their bounds. The main differentiating factor is that we deal with an autoencoder h = g◦f . In this case, the mutual information I(h(b); b) is expected to be large, since h(b) is trained to recover b and by the dataprocessing inequality, I(f(b); b) ≥ I(h(b); b) and therefore, I(f(b); b) is also large in this case. As a result, no upper bound is given on the mutual information term I(f(b); b). This is unlike the information bottleneck principle, which guides the classification setting of Achille & Soatto (2018).
Our upper bound on TC(f(b)) includes the term−I(h(b); b), and consequently, TC(f(b)) tends to be smaller (increased disentanglement) as I(h(b); b) increases. Additionally, the upper bound sums a term d1 · q(α). Here, d1 is the dimension of f(b), α denotes the amount of regularization in the weights of f and q(α) is monotonically increasing as α tends to zero. Therefore, the disentanglement tends to be larger for an autoencoder h = g◦f , such that f is regularized and the mutual information I(h(b); b) is large. In our analysis, we do not require f to be a linear transformation and we do not assume that the components of each mid-level representation of b in the layers of f are uncorrelated. Lemma 2 (Informal). Let b ∼ DB be a distribution and h = g ◦ f an autoencoder. Let d1 be the dimension of f(b) and d2 the dimension of the layer previous to f(b). Under some assumptions on the weights of the encoder, there is a monotonically decreasing function q(α) for α > 0 such that:
TC(f(b)) ≤ d1 · q(α)− I(h(b); b) +O (d1/d2) (13)
Eq. 13 bounds the total correlation of f(b), which measures the amount of dependence between the components of the encoder on samples in B. The bounds has three terms: d1 · q(α), −I(h(b), b) and O (d1/d2). In this formulation, α denotes the amount of regularization in the weights of f . In addition, q(α) is monotonically increasing as α tends to zero. The term I(h(b); b) measures the mutual information between the input b and output h(b) of the autoencoder h. Since the mutual information is subtracted in the right hand side, the larger it is, the smaller TC(f(b)) should be. The last term, O(d1/d2) measures the ratio between the dimension of the output of f and the dimension of the previous layer of f . Thus, this quantity is small whenever there is a significance reduction in the dimension in the application of the last layer of f .
Therefore, there is a tradeoff between the amount of regularization in the weights of f and the mutual information I(h(b); b). If there is small regularization, then, the autoencoder is able to produce better reconstruction h(b) ≈ b, and therefore, a larger value of I(h(b); b). On the other hand, small regularization leads to a higher value of q(α).
The bound relies on the mutual information between the inputs and outputs of the autoencoder to be large. The following lemma provides an argument why this is the case when the expected reconstruction error of the autoencoder is small. Lemma 3 (Informal). Let b ∼ DB be a distribution over a discrete set XB and h = g ◦ f an autoencoder. Assume that ∀x1 6= x2 ∈ XB : ‖x1 − x2‖1 > ∆. Then,
I(h(b); b) ≥ (
1− E[‖h(b)− b‖1] ∆
) H(b)− √ E[‖h(b)− b‖1] (14)
The above lemma asserts that if the samples in DB are well separated, whenever the autoencoder has a small expected reconstruction error, E[‖h(b) − b‖1], then, the mutual information I(h(b); b) is at least a large portion of H(b). Therefore, we conclude that if the autoencoder generalizes well, then, it also maximizes the mutual information I(h(b); b).
To conclude the analysis: for a small enough reconstruction error, when training the autoencoder, the mutual information between the autoencoder’s input and output is high (Lem. 10), which implies that the individual coordinates of the representation layer are almost independent of each other (Lem. 9). When using part of the representation to encode the information that exists in domain A (the shared part), the other part would contain coordinates that are weakly dependent of the features encoded in A. In such a case, we can train with a GAN that involves only the shared representation (Lem. 1). That way, we can upper bound the generalization error expressed in Thm. 1, using relatively simple loss terms, as is done in Sec. 3.

heading: 5 EXPERIMENTS
text: We evaluate our method on three additive facial attributes: eyewear, facial hair, and smile. Images from the celebA face image dataset by Yang et al. (2015) were used, since these are conveniently annotated as having the attribute or not. The images without the attribute (no glasses, or no facialhair, or no smile) were used as domain A in each of the experiments. Note that three different A domains were used. As the second domain B, we used the images labeled as having glasses, having facial hair, or smiling, according to the experiment.
Our underlying network architecture adapts the architecture used by Lample et al. (2017), which is based on (Isola et al., 2017a), where we use Instance Normalization (Ulyanov et al., 2016) instead of Batch Normalization (Ioffe & Szegedy, 2015), and without dropout. Let Ck denote a ConvolutionInstanceNorm-ReLU layer with k filters, where a kernel size of 4 × 4, with a stride of 2, and a padding of 1 is used. The activations of the encoders e1, e2 are leaky-ReLUs with a slope of 0.2 and the deocder g employs ReLUs. e1 has the following layers C32, C64, C128, C256, C512, C512−d; e2 has a slightly lower capacity C32, C64, C128, C128, C128, Cd, where d = 25. The input images have a size of 128 × 128, and the encoding is of size 512 × 2 × 2 (split between the e1 and e2). g is symmetric to the encoders and employs transposed convolutions for the upsampling.
In the first set of experiments, we add the relevant content from a random image b ∈ B into an image from a. The results are given in Fig. 2, and appendix Fig. 7 and 8. We compare with two guided image translation baselines: MUNIT (Huang et al., 2018) and DRIT (Lee et al., 2018). We used the published code for each method and despite our best effort, these methods fail on the task of content addition. In almost all cases, the baseline methods apply the style of the guide and not the added content.
It should be noted that the simplicity of our approach directly translates to more efficient training than the two baselines methods. Our method has one weighting hyperparameter, which is fixed throughout the experiments. MUNIT and DRIT each has several weighting hyperparameters (since they use more loss terms) and these require attention and, need to change between the experiments, both in our runs and in the authors’ own experiments. In addition, our method has a lower memory footprint and a much shorter duration of each iteration. The statistics are reported in Tab. 2 and as can be seen, the runtime and memory footprint of our method are much closer to the Fader network by Lample et al. (2017), which cannot perform guided mapping, than to MUNIT and DRIT.
Since the performance of the baselines is clearly inferior in the current setting, we did not hold a user study comparing different algorithms. Instead, we compare the output of our method directly with real images. Two experiments are conducted: (i) can users tell the difference between an image from domain B and an image from domain A that was translated to domain B, and (ii) can users tell
the difference between an image from domain B and the same image, after replacing the attribute’s content (glasses, smile, or facial-hair) with that of another image from B. The experiment was performed with n = 30 users, who observed 10 pairs of images each, for each of the tests.
The results are reported in Tab. 3. As can be seen, users are able to detect the real image over the generated one, in most of the cases. However, the success ratio varies between the three image translation tasks and between the two types of comparisons. The most successful experiments, i.e., those where the users were confused the most, were in the facial hair (“beard”) category. In contrast, when replacing a person’s glasses with those of a random person, the users were able to tell the real image 74% of the time.
Fig. 3 and appendix Fig. 9 and 10 show the type of images shown in the experiment where users were asked to tell an image from domain B from an hybrid image that contains a face of one image from this domain, and the attribute content from another image from it. As can be seen, most mixand-match combinations seem natural. However, going over the rows, which should have a fixed attribute (e.g., the same glasses), one observes some variation. This unwanted variation arises from the need to fit the content to the new face.
The method does have, as can be expected, difficulty dealing with low quality inputs. Examples are shown in Fig. 4, including a misaligned source or guide image. Also shown is an example in which the added content in the target domain is very subtle. These challenges result in a lower quality output. However, the output in each case does indicate some ability to overcome the challenging input.
To evaluate the linearity of the latent representation e2(b), we performed interpolation experiments. The results are presented in Fig. 5. As can be seen, the change is gradual as we interpolate linearly between the e2 encoding of the two guide images shown on the left and on the right.
In the supplementary appendix, we provide many more translation examples, see Fig. 11, 12, and 13.
While the previous experiments focused on the guided addition of content (mapping from A to B), our method can also be applied in the other direction, from B to A. This way, the specific content in image b ∈ B is removed. In our method, this is achieved simply by decoding a representation of the form (e1(b), 0).
The advantage of mapping in this direction is the availability of additional literature methods to compare with, since no guiding is necessary. In Fig. 6, we compare the results we obtain for removing a feature with the Fader network method of Lample et al. (2017). As can be seen, the removal process of our method results in less residuals.
To verify that we obtain a better quality in comparison with that of the published implementation of Fader networks, we have applied both an automatic classifier and a user study. The classifier is trained on the training set of domain A and B, using the same architecture that is used by Lample et al. (2017) to perform model selection.
Tab. 4 presents the mean probability of class B provided by the classifier for the output of both Fader network and our method. As can be seen, the probability to belong to the class of the image before the transformation is, as desired, low for both methods. It is somewhat lower on average in our method, despite the fact that our method does not use such a network during training.
The user study was conducted on n = 20 users, each examining 20 random test set triplets from each experiment. Each triplet showed the original image (with the feature) and the results of the two algorithms, where the feature is removed. The users preferred our method over fader 92% of the time for glasses removal, 89% of the time for facial hair removal, and 91% of the time for the removal of a smile.

heading: 6 CONCLUSIONS
text: When converting between two domains, there is an inherent ambiguity that arises from the domainspecific information in the target domain. In guided translation, the reference image in the target domain provides the missing information. Previous work has focused on the missing information that is highly tied to the texture of the image. For example, when translating between paintings and photos, DRIT adds considerable content from the reference photo. However, this is unstructured content, which is not well localized and is highly related to subsets of the image patches that exist in the target domain. In addition, the content from the reference photo that is out of the domain of paintings is not guaranteed to be fully present in the output.
Our work focuses on transformations in which the domain specific content is well structured, and guarantees to replicate all of the domain specific information from the reference image. This is done using a small number of networks and a surprisingly simple set of loss terms, which, due to the emergence of a disentangled representation, solves the problem convincingly.

heading: ACKNOWLEDGEMENTS
text: This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC CoG 725974). The theoretical analysis in this work is part of Tomer Galanti’s Ph.D thesis research conducted at Tel Aviv University.

heading: A ADDITIONAL FIGURES

heading: B PRELIMINARIES
text: B.1 NOTATIONS AND TERMINOLOGY
In this section we provide notations and terminology that are were not introduced in Sec. 4 but are necessary for the proofs of the claims in this section.
We say that three random variables (discrete or continuous) X1, X2, X3 form a Markov chain, indicated with X1 → X2 → X3, if P[X3|X2, X1] = P[X3|X2]. The Data Processing Inequality (DPI) for a Markov chain X1 → X2 → X3 ensures that I(X1;X3) ≤ min (I(X1;X2), I(X2;X3)). In particular, it holds for X2 = f(X1) and X3 = g(X2), where f, g are deterministic processes.
We denote by x ∼ logN (µ, σ2) a random variable that is distributed by a log-normal distribution, i.e., log x ∼ N (µ, σ2). We consider that the mean and variance of a log-normal distribution logN (µ, σ2) are exp(µ + σ2/2) and (exp(σ2) − 1) exp(2µ + σ2) respectively. We denote by W ⊙ U := (Wk,j · Uk,j)k≤m,j≤m the Hadamard product of two matrices W ,U ∈ Rm×n. For a given vector x ∈ Rm, we denote dim(x) := m and for a matrix W ∈ Rm×n, we denote dim(W ) := mn. In addition, we denote x2 = x ⊙ x = (x21, . . . , x 2 m) and E[x] = (E[x1], . . . ,E[xm]). The indicator function, is denoted by 1 [x] for a boolean variable x ∈ {true, false} (i.e., 1 [x] = 1 if x = true and 1 [x] = 0 o.w).
B.2 LEMMAS
In this section, we provide useful lemmas that aid in the proofs of our main results.
Lemma 4. Let x = (x1, . . . , xn) ∈ Rn be a random vector. Let µ1, . . . , µn : R→ R be continuous invertible functions and we denote µ(x) := (µ1(x1), . . . , µn(xn)). Then, TC(x) = TC(µ(x)).
Proof. First, we consider that:
TC(x) = DKL
( P[x] ∥∥∥ n∏ i=1 P[xi] ) = DKL ( P[x] ∥∥∥P[x̄]) (15) where, x̄ := (x̄1, . . . , x̄n) is a vector of independent random variables, such that x̄i is distributed, according to the marginal distribution of xi.
KL-divergence is invariant to applying continuous invertible transformations, i.e., DKL(X‖Y ) = DKL(µ(X)‖µ(Y )) for µ that is continuous and invertible. Therefore,
TC(x) = DKL
( P[µ(x)] ∥∥∥P[µ(x̄)]) = DKL(P[µ(x)]∥∥∥ n∏ i=1 P[µi(x̄i)] ) = TC(µ(x)) (16)
Lemma 5. Let p ∈ [0, 1]. Then, H(p) ≤ 2 log(2) √ p(1− p).
Proof. See (https://math.stackexchange.com/users/44121/jack daurizio).
The following lemma is a modification of Claim 2.1 in (Regev, 2013).
Lemma 6. Let X and Y be two random variables. Assume that there is a function (i.e., a deterministic process) F , such that P[F (Y ) = X] ≥ q ≥ 1/2. Then, I(X;Y ) ≥ qH(X)−H(q).
Proof. By the data processing inequality,
I(X;Y ) ≥ I(X;F (Y )) = H(X)−H(X|F (Y )) = H(X)−H(1 X=F (Y ), X|F (Y )) = H(X)− (H(1 X=F (Y )|F (Y )) +H(X|1 X=F (Y ), F (Y )))
(17)
Since conditioning does not increase entropy, H(1 X=F (Y )|F (Y )) ≤ H(1 X=F (Y )) ≤ H(q). In addition,
H(X|1 X=F (Y ), F (Y )) =P[1 X=F (Y ) = 0] ·H(X|1 X=F (Y ) = 0, F (Y )) + P[1 X=F (Y ) = 1] ·H(X|1 X=F (Y ) = 1, F (Y ))
=P[1 X=F (Y ) = 0] ·H(X|1 X=F (Y ) = 0, F (Y )) ≤(1− q)H(X|1 X=F (Y ) = 0, F (Y )) ≤(1− q)H(X)
(18)
Therefore, we conclude that, I(X;Y ) ≥ qH(X)−H(q).
Lemma 7. Let b ∼ DB be a distribution over a discrete set XB ⊂ RM and hv : RM → RM is a (possibly random) function. Assume that ∀x1 6= x2 ∈ XB : ‖x1−x2‖1 > ∆. Let P[‖hv(b)−b‖1 ≤ ∆] ≥ q ≥ 1/2. Then, I(hv(b); b) ≥ qH(b)−H(q).
Proof. Let F (u) := arg minx∈XB ‖u − x‖1. Since the members of XB are ∆-distant from each other, if ‖hv(b)− b‖1 ≤ ∆, then, F (hv(b)) = b. Therefore, we have:
P[F (hv(b)) = b] ≥ P[‖hv(b)− b‖1 ≤ ∆] ≥ q ≥ 1/2 (19)
By Lem. 6, for X :← b, Y :← hv(b) and F :← F , we have: I(hv(b); b) ≥ qH(b)−H(q).
The following lemma is an example of three uncorrelated variables X,Y, Z, such that there is a dimensionality reducing linear transformation over them that preserves all of their information.
Lemma 8. Let X and Y be two independent uniform distributions over [−1, 1] and Z = (X+Y )2. Then, the transformation T (x, y, z) = (x, y) satisfies I(X,Y, Z;T (X,Y, Z)) = H(X,Y, Z) and Cov(X,Y ) = Cov(X,Z) = Cov(Y, Z).
Proof. Since X and Y are independent, their covariance is zero. By the definition of X and Y , we have: E[X] = E[Y ] = E[X3] = 0. Therefore,
Cov(Y, Z) = Cov(X,Z)
= E[X(X + Y )2]− E[X]E[(X + Y )2] = E[X3] + 2E[X2]E[Y ] + E[X]E[Y 2]− E[X]E[(X + Y )2] = 0
(20)
Finally, we consider that T is a homeomorphic transformation T : (x, y, (x + y)2) 7→ (x, y) (between the manifolds {(x, y, z) | x, y ∈ [−1, 1], z = (x + y)2} and [−1, 1]2) and mutual information is invariant to applications of homeomorphic transformations, i.e., I(X;Y ) = I(µ(X); ν(Y )) for homeomorphisms µ and ν over the sample spaces ofX and Y (resp.). Therefore, I(X,Y, Z;T (X,Y, Z)) = I(X,Y, Z;X,Y, Z) = H(X,Y, Z).

heading: C PROOFS OF THE MAIN RESULTS
text: Theorem 1. Assume that the loss function ` is symmetric and obeys the triangle inequality. Then, for any autoencoder h = g ◦ f ∈ H, such that f(x1,x2) = (e1(x1), e2(x2)) ∈ F is an encoder and g ∈M is a decoder, the following holds,
RDA,B [h, y] ≤RDB,B̂ [h, y] + ming∗∈M
{ RDA,B [g ∗ ◦ f, y] +RDB,B̂ [g ∗ ◦ f, y] } + discM(f ◦DA,B , f ◦DB,B̂)
(10)
where DB,B̂ is the distribution of (b, b) where b ∼ DB . Proof. Let g∗ ∈ arg ming∈M { RDA,B [g ◦ f, y] +RDB,B̂ [g ◦ f, y] } . Since the loss ` obeys the triangle inequality,
RDA,B [g ◦ f, y] ≤ RDA,B [g ◦ f, g∗ ◦ f ] +RDA,B [g∗ ◦ f, y] (21)
By the definition of discrepancy,
RDA,B [g ◦ f, y] ≤ RDB,B̂ [g ◦ f, g ∗ ◦ f ] +RDA,B [g∗ ◦ f, y] + discM(f ◦DA,B , f ◦DB,B̂) (22)
Again, by the triangle inequality,
RDA,B [g ◦ f, y] ≤RDB,B̂ [g ◦ f, y] +RDA,B [g ∗ ◦ f, y] +RDB,B̂ [g ∗ ◦ f, y] + discM(f ◦DA,B , f ◦DB,B̂)
=RDB,B̂ [g ◦ f, y] + ming∈M { RDA,B [g ◦ f, y] +RDB,B̂ [g ◦ f, y] } + discM(f ◦DA,B , f ◦DB,B̂)
(23)
Lemma 1. Let M be the set of neural networks of the form: c(x) = φ(Wr . . . φ(W2φ(W1x + q))), where, Wi ∈ Rdi×di+1 for i ∈ {1, . . . , r − 1}, q ∈ Rd2 and d1 = E1 + E2. In addition, φ(x1, . . . , xk) = (φ1(x1), . . . , φ1(xk)), for k ∈ N, (x1, . . . , xk) ∈ Rk and a non-linear activation function φ1 : R → R. LetM′ be the same asM with d1 = E1 (instead of d1 = E1 + E2). Let f(x) = (e1(x), e2(x)) be an encoder and assume that: e1(b) |= e2(b). Then,
discM(f ◦DA,B , f ◦DB,B̂) ≤ discM′(e1 ◦DA, e1 ◦DB) (12)
Proof. By the definition of discrepancy, and since e1(b) |= e2(b), we have:
discM(f ◦DA,B , f ◦DB,B̂)
= sup c1,c2∈M ∣∣∣Ee1(a),e2(b)`(c1(e1(a), e2(b)), c2(e1(a), e2(b))) − Ee1(b),e2(b)`(c1(e1(b), e2(b)), c2(e1(b), e2(b)))
∣∣∣ = sup
c1,c2∈M ∣∣∣Ee2(b){Ee1(a)`(c1(e1(a), e2(b)), c2(e1(a), e2(b))) − Ee1(b)`(c1(e1(b), e2(b)), c2(e1(b), e2(b))) }∣∣∣ (24)
By |E[x]| ≤ E[|x|] and Ex[supy∈Y f(x,y)] ≤ supy∈Y Ex[f(x,y)], we have:
discM(f ◦DA,B , f ◦DB,B̂)
≤Ee2(b) sup c1,c2∈M ∣∣∣{Ee1(a)`(c1(e1(a), e2(b)), c2(e1(a), e2(b))) − Ee1(b)`(c1(e1(a), e2(b)), c2(e1(b), e2(b)))
}∣∣∣ ≤ sup
y∈RE2 sup c1,c2∈M ∣∣∣{Ee1(a)`(c1(e1(a),y), c2(e1(a),y)) − Ee1(b)`(c1(e1(b),y), c2(e1(b),y)) }∣∣∣ (25)
By the definition ofM, for any c ∈ M and a fixed vector, y ∈ RE2 , there is a function u ∈ M′, such that for every x ∈ RE1 , we have: c(x,y) = u(x). Therefore, we can rewrite the last equation as follows:
discM(f ◦DA,B , f ◦DB,B̂)
≤ sup u1,u2∈M′ ∣∣∣{Ee1(a)`(u1(e1(a)), u2(e1(a))− Ee1(b)`(u1(e1(b)), u2(e1(b))}∣∣∣ =discM′(e1 ◦DA, e1 ◦DB)
(26)
C.1 EMERGENCE OF DISENTANGLED REPRESENTATIONS
In this section, we employ the theory of (Achille & Soatto, 2018) in order to show the emergence of disentangled representations, when an autoencoder h = g ◦ f generalizes well. Our analysis shows that by learning an autoencoder such that the encoder f has log-normal regularization, there is a high likelihood of learning disentangled representations. We note that until now, we treated the autoencoder h as a function of two variables (a, b or b, b). From now on, if the two variables are the same, then, we simply write hv(b). In addition, by Eq. 11, instead of writing, RDB,B̂ [g ◦ f, y] we can simply write RDB [h, Id] := EfEb∼DB [`(g ◦ f(b), b)]. The framework of Achille & Soatto (2018) relies on a few assumptions, which are imported to our case. The algorithm is provided with m i.i.d samples SB = {bi}mi=1 from the distribution DB and trains an encoder-decoder neural network hv = gu ◦ fw, such that the parameters of the encoder, fv , have log-normal perturbations. Formally, we learn a mapping hv of the form hv(x) = φ(W 2tφ(W 2t−1 . . . φ(W 1x))), where W i ∈ Rdi+1×di , for i ∈ {1, . . . , 2t − 1}, d1 = d2t = M , w = (W t, . . . ,W 1), u = (W 2t, . . . ,W t+1) and v = (W 2t, . . . ,W 1). Here, φ(x1, . . . , xm) = (φ1(x1), . . . , φ1(xk)) is a non-linear activation function φ1 : R → R extended for all m ∈ N and (x1, . . . , xk) ∈ Rk. We assume that φ1 : R → R is a homeomorphism (i.e., φ1 is invertible, continuous and φ−11 is also continuous). The encoder fw is the composition of the first t layers and the decoder gu is composed of the last t layers of hv . Following Achille & Soatto (2018), we assume that the posterior distribution p(W ki,j |SB) is defined as a Gaussian dropout,
W ki,j |SB ∼ ki,j · Ŵ ki,j (27)
where i ∈ Rdi+1×di and ki,j ∼ logN (−α/2, α), for k ∈ {1, . . . , t}. We consider that the mean and variance of logN (−α/2, α) are 1 and exp(α) − 1 respectively. Here, Ŵ k is a learned mean of the k’th layer of the encoder. We denote the output of the k’th layer of fw by zk, i.e., zk = φ(W kφ(W k−1 . . . φ(W 1x))).
Implicit Emergence of Disentangled Representations The following lemma is a corollary of Proposition 5.2 in (Achille & Soatto, 2018) for our model. Lemma 9. Let k ∈ {1, . . . , t}, yk = W kφ(W k−1 . . . φ(W 1b)) and zk = φ(yk), for W k = k ⊙ Ŵ k, where ki,j ∼ logN (αk/2, αk). Further, assume that the marginals of P[yk] and P[yk|zk−1] are Gaussians, the components of zk−1 are uncorrelated and that their kurtosis is uniformly bounded. Let q(α) := − 12 log (1− exp (−α)). Then,
TC(zk) ≤ dim(zk) · q(αk)− I(hv(b); b) +O (
dim(zk) dim(zk−1)
) (28)
Proof. By Proposition 5.2 in Achille & Soatto (2018), we have:
TC(yk) + I(yk; zk−1)
dim(yk) ≤ q(αk) +O
( 1
dim(zk−1)
) (29)
By Lem. 4, we have, TC(yk) = TC(zk). In addition, dim(yk) = dim(zk). Therefore, TC(zk) + I(yk; zk−1) ≤ dim(zk) · q(αk) +O (
dim(zk) dim(zk−1)
) (30)
Finally, by the data processing inequality, for X1 := b, X2 := zk−1, X3 := yk and X4 := hv(b), we have, I(hv(b); b) ≤ I(yk; zk−1). The bound follows from Eq. 30 and the last observation.
Lem. 9 provides an upper bound on the total correlation of the k’th layer of the autoencoder hv(b). This bound assumes that the marginal distributions of yk and yk|zk−1 are Gaussians. This is a reasonable assumption if dim(zk−1) is large by the central limit theorem. We also assume that there are no pair-wise linear correlations between the components of zk−1. In some sense, this assumption can be viewed as minimality of zk−1. Informally, if there is a strong linear correlation between two components of zk−1, then, we can throw away one of them and keep most of the information. On the other hand, if the components of zk−1 are uncorrelated, the existence of a dimensionality
reducing linear transformation W such that I(Wzk−1; zk−1) = H(zk−1) is still a possibility (for a concrete example, see Lem. 8). Hence, the next layer, zk, is still useful, in order to compress the representation and can still preserve all the information. We also assume that the kurtosis of the components of zk−1 is uniformly bounded. This is a technical hypothesis that is always satisfied, if the components of zk−1 are sub-Gaussian or with uniformly bounded support.
The function q(αk) is monotonically increasing as αk tends to 0. Therefore, this bound realizes a trade-off between the mutual information of hv(b) and b and the amount of perturbations in the encoder. If the perturbations in the encoder are stronger, then, the ability of the autoencoder to reconstruct b decays. On the other hand, if the perturbations are small, then, q(α) increases.
We consider that the bound decreases, as I(hv(b); b) increases. It is reasonable to believe that since hv(b) is an autoencoder that is being trained to reconstruct b, I(hv(b); b) is maximized implicitly. The problem of training an autoencoder that maintains a reconstruction that has high mutual information with respect to the input has recently attracted a considerable attention. Several methods for maximizing this term explicitly were proposed in (Zhao et al., 2017; Phuong et al., 2018). In the following lemma, we show that if the samples of DB are well separated, then, the mutual information I(hv(b); b) is implicitly maximized as the reconstruction, hv(b) ≈ b improves. Lemma 10. Let b ∼ DB be a random vector over a discrete set XB ⊂ RM and hv : RM → RM an autoencoder. Assume that ∀x1 6= x2 ∈ XB : ‖x1 − x2‖1 > ∆ and RDB [hv(b), Id] ≤ ∆2 . Then,
I(hv(b); b) ≥ (
1− RDB [hv(b), Id] ∆
) H(b)− √ RDB [hv(b), Id] (31)
Proof. First, assume by contradiction that: P[‖hv(b)− b‖1 ≤ ∆] < (
1− RDB [hv(b), Id] ∆
) (32)
In other words, P[‖hv(b)− b‖1 > ∆] > RDB [hv(b), Id]/∆. In particular, we arrive at a contradiction: RDB [hv, Id] = Eb[‖hv(b)− b‖1] ≥ P[‖hv(b)− b‖1 > ∆] ·∆ > RDB [hv, Id] (33) By the above contradiction and the hypothesis that RDB [hv(b),Id]∆ ≤ 1/2, we have, q ≥(
1− RDB [hv(b),Id]∆ ) ≥ 1/2. Therefore, by Lem. 7,
I(hv(b); b) ≥ (
1− RDB [hv(b), Id] ∆
) H(b)−H ( 1− RDB [hv(b), Id]
∆
) (34)
Additionally, by Lem. 5, we have:
H ( 1− RDB [hv(b), Id]
∆
) ≤ 2 log(2) √( 1− RDB [hv(b), Id]
∆
) · RDB [hv(b), Id]
∆ ≤ 2 log(2) √ RDB [hv(b), Id] ∆ ≤ √ RDB [hv(b), Id] ∆
(35)
Finally,
I(hv(b); b) ≥ (
1− RDB [hv(b), Id] ∆
) H(b)− √ RDB [hv(b), Id]
∆ (36)
The above lemma asserts that if the samples in DB are well separated, whenever the autoencoder has a small expected reconstruction error RDB [hv(b), Id], then, the mutual information I(hv(b); b) is at least a large portion of H(b). Therefore, we conclude that if the autoencoder generalizes well, then, it also maximizes the mutual information I(hv(b); b). Finally, we note that we cannot directly apply Lem. 10 to bound the mutual information in Lem. 9. That is because, in Lem. 9, we assume that the components of yk are distributed normally, which implies that the distribution of b must be continuous. On the other hand, in Lem. 10 we assume that b is distributed according to a discrete distribution. In order to reduce this friction, instead of assuming that the each component of zk is distributed according to a normal distribution, we can assume that it is distributed according to a discrete approximation of a normal distribution.

Review summaries:
This paper tackles the task of content transfer. For a given type of images (frontal face shots), the goal is to transfer a particular localized property (e.g. glasses or facial hair) extracted from one image to another image of the same type (difference face). This is also known as the problem of guided image-to-image translation. 

The paper proposes an unsupervised approach for mapping two sets of objects, A and B, such that set B contains all the information that is in set A and some additional information. The paper learns a latent space which encodes: (a) information which is shared in both sets, and (b) the additional content present in B. This is done by employing a two-pathway encoder and a decoder for both the sets. Experiments on problems such as adding glasses or facial hair to faces shows that the proposed method performs better than existing disentanglement approaches.