user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Overcoming the Disentanglement vs Reconstruction Trade-off via Jacobian Supervision

Paper content:
A major challenge in learning image representations is the disentangling of the factors of variation underlying the image formation.
This is typically achieved with an autoencoder architecture where a subset of the latent variables is constrained to correspond to specific factors, and the rest of them are considered nuisance variables.
This approach has an important drawback: as the dimension of the nuisance variables is increased, image reconstruction is improved, but the decoder has the flexibility to ignore the specified factors, thus losing the ability to condition the output on them.
In this work, we propose to overcome this trade-off by progressively growing the dimension of the latent code, while constraining the Jacobian of the output image with respect to the disentangled variables to remain the same.
As a result, the obtained models are effective at both disentangling and reconstruction.
We demonstrate the applicability of this method in both unsupervised and supervised scenarios for learning disentangled representations.
In a facial attribute manipulation task, we obtain high quality image generation while smoothly controlling dozens of attributes with a single model.
This is an order of magnitude more disentangled factors than state-of-the-art methods, while obtaining visually similar or superior results, and avoiding adversarial training.
A desired characteristic of deep generative models is the ability to output realistic images while controlling one or more of the factors of variation underlying the image formation.
Moreover, when each unit in the model's internal image representation is sensitive to each of these factors, the model is said to obtain disentangled representations.
Learning such models has been approached in the past by training autoencoders where the latent variables (or a subset of them) are constrained to correspond to given factors of variation, which can be specified (supervised) or learned from the data (unsupervised) BID22 BID29 BID15 .
The remaining latent variables are typically considered nuisance variables and are used by the autoencoder to complete the reconstruction of the image.There exists one fundamental problem when learning disentangled representations using autoencoders, sometimes referred to as the "shortcut problem" BID29 .
If the dimension of the latent code is too large, the decoder ignores the latent variables associated to the specified factors of variation, and achieves the reconstruction by using the capacity available in the nuisance variables.
On the other hand, if the dimension of the latent code is small, the decoder is encouraged to use the specified variables, but is also limited in the amount of information it can use for reconstruction, so the reconstructed image is more distorted with respect to the autoencoder's input.
BID29 showed that this trade-off between reconstruction and disentangling can indeed be traversed by varying the dimension of the latent code.
However, no principled method exists to choose the optimal latent code dimension.The shortcut problem was also addressed by using additional mechanisms to make sure the decoder output is a function of the specified factors in the latent code.
One approach, for example, consists in swapping the specified part of the latent code between different samples, and using adversarial training to make sure the output distribution is indeed conditioned to the specified factors BID22 BID19 BID29 .
However, adversarial training remains a difficult and unstable optimization problem in practice.Based on these observations, we propose a method for avoiding the shortcut problem that requires no adversarial training and achieves good disentanglement and reconstruction at the same time.Our method consists in first training an autoencoder model, the teacher, where the dimension of the latent code is small, so that the autoencoder is able to effectively disentangle the factors of variation and condition its output on them.
These factors can be specified in a supervised manner or learned from the data in an unsupervised way, as we shall demonstrate.
After the teacher model is trained, we construct a student model that has a larger latent code dimension for the nuisance variables.
For the student, we optimize the reconstruction loss as well as an additional loss function that constrains the variation of the output with respect to the specified latent variables to be the same as the teacher's.In what follows, we consider autoencoder models (E, D), that receive an image x as input and produce a reconstructionx : D(E(x)) =x.
We consider that the latent code is always split into a specified factors part y ∈ R k and a nuisance variables part z ∈ R d : E(x) = (y, z), D (y, z) =x.Consider a teacher autoencoder (E T , D T ), with nuisance variables dimension d T , and a student DISPLAYFORM0 Because the dimension of the nuisance variables of the student is larger than in the teacher model, we expect a better reconstruction from it (i.e. ||x −x S || < ||x −x T ||, for some norm).At
the same time, we want the student model to maintain the same disentangling ability as the teacher as well as the conditioning of the output on the specified factors. A
first order approximation of this desired goal can be expressed as DISPLAYFORM1 where j ∈ {1...H
· W · C}, H, W and C are the dimensions of the output image, and i ∈ {1...k} indexes over the specified factors of variation.In this paper we propose a method to impose the first-order constraint in (1), which we term Jacobian supervision. We show two applications of this method. First, we propose an unsupervised algorithm that progressively disentangles the principal factors of variation in a dataset of images. Second, we use the Jacobian supervision to train an autoencoder model for images of faces, in which the factors of variation to be controlled are facial attributes. Our resulting model outperforms the state-of-theart in terms of both reconstruction quality and facial attribute manipulation ability.
A natural trade-off between disentanglement and reconstruction exists when learning image representations using autoencoder architectures.
In this work, we showed that it is possible to overcome this trade-off by first learning a teacher model that is good at disentangling and then imposing the Jacobian of this model with respect to the disentangled variables to a student model that is good at reconstruction.
The student model then becomes good at both disentangling and reconstruction.
We showed two example applications of this idea.
The first one was to progressively learn the principal factors of variation in a dataset, in an unsupervised manner.
The second application is a generative model that is able to manipulate facial attributes in human faces.
The resulting model is able to manipulate one order of magnitude more facial attributes than state-of-the-art methods, while obtaining similar or superior visual results, and requiring no adversarial training.
For the autoencoder utilized for experiments in Section 3, we used the following architecture.
For the encoder: DISPLAYFORM0 where F (I, O) indicates a fully connected layer with I inputs and O outputs.
For the first teacher model (k = 2, d = 0), we also used BatchNorm after the encoder output.The decoder is the exact symmetric of the encoder, with a Tanh layer appended at the end.We used Adam (Kingma & Ba, 2014 ) with a learning rate of 3e −4 , a batch size of 128 and weight decay coefficient 1e −6 .

assistant: text: A method for learning image representations that are good for both disentangling factors of variation and obtaining faithful reconstructions.