Title: ENCODER DISCRIMINATOR NETWORKS FOR UNSUPERVISED REPRESENTATION LEARNING

Abstract: Learning representations of data samples in an unsupervised way is needed whenever computers have to reason about unlabeled data. Applications range from compressing and denoising data to super-resolution, generating new samples from a given sample distribution and much more. In this work, we use information entropy and a little game to motivate a new encoder discriminator architecture in order to learn unsupervised latent representations. Inspired by the game "Taboo", we train an encoder network to generate a meaningful representation of one particular sample of a dataset. Using this description, a discriminator network then has to retrieve the same sample from the whole dataset. We show that learning in this manner on many different samples repeatedly minimizes the information entropy given the latent description and, thus, forces the encoder network to make precise descriptions that can be interpreted by the discriminator. We provide first results of this method on the MNIST and the Fashion MNIST dataset.

1. INTRODUCTION AND RELATED WORK:

While in supervised machine learning the goal is to find a mapping of samples onto predefined labels, in unsupervised learning algorithms not only have to find a mapping but also -and in particularto find meaningful representations themselves. The reasons to do this are manifold: Hinton & Salakhutdinov (2006) show how to generate compressed representations of data using deep autoencoders. Vincent et al. (2008) show how autoencoders can be used to denoise images. This requires finding the hidden structure of the data behind the noise and thus representations that are independent of noise added to the input. Dong et al. (2016) present a way to increase the resolution of lower resolution input images. Most of these encoder-decoder methods optimize their output to match the training data with respect to some metric (usually L 2 ). These metrics however do not capture the statistics of images well and hence often lead to blurry results in the case of model uncertainty.

Another trend in unsupervised learning are generative adversarial networks (GANs) as introduced by Goodfellow et al. (2014) . By training a discriminative loss function to distinguish between samples that originate from the dataset and samples coming from a generator network, new samples can be generated from a latent noise input. These samples capture the statistics of the dataset and often look surprisingly realistic. Radford et al. (2015) showed how the latent noise input can be related to the generator output and was able to perform vector arithmetic for visual concepts in that latent space. However in these generator-discriminator methods, finding a latent representation for a visual concept is difficult -especially in the presence of mode collapse. Furthermore, retrieving a latent representation of a particular sample is impossible since no encoder is given.

In this work, we present a new architecture that combines the benefits of the encoder part of autoencoders with the discriminator part of GANs in order to generate latent sample descriptions that are in good accordance with the statistics of the dataset. We motivate the network using the game "Taboo" -a word guessing game where words have to be described without using forbidden phrases -and show how this leads to optimal representations by means of minimizing information entropy. In this section, we introduce encoder discriminator networks as a new way to learn latent representations in an unsupervised way. First, the architecture is motivated by a cooperative game similar to "Taboo". Then, we want to reveal how the game can be implemented and our underlying network architecture. In the end of this section, we want to give a more thorough explanation of the theoretical aspects and derive the loss function from an information entropy argument.

2.1. "TABOO" -A COOPERATIVE GAME:

Taboo is a famous game in which players in a team try to find words collaboratively: First, one player picks a random card from a pile and keeps it secret. This card contains a word which the player has to describe as precisely as possible so that it can be guessed by the teammates. To make the game harder, the player is constrained by several words which must not appear in the description. The team which makes the best descriptions and finds the words the fastest wins the game.

The encoder discriminator architecture follows this analogy: First, the encoder picks a random sample from a dataset which contains, for example, an image that needs to be found by the discriminator. Then, the encoder has to make an as precise as possible description of this sample but is constrained by the dimensionality and boundedness of the latent description space. The loss function is minimized when the descriptions of the encoder are precise enough and interpretable by the discriminator network such that the discriminator can retrieve the sample from the whole dataset with high confidence. So in contrast to GANs, where the discriminator is trained adversely, in this setting the discriminator works collaboratively with the encoder and thus the network can be trained end-to-end.

2.2. ENCODER DISCRIMINATOR ARCHITECTURE:

Figure 1 shows the outline of an Encoder Discriminator network. First, a random sample of the dataset is picked and fed into an encoder network. This encoder network generates a latent description of the picked sample and passes it to a discriminator network. The discriminator network takes the sample description and compares it with the original as well as other samples from the dataset. For each sample, the discriminator returns a confidence value of how certain the network is, that the latent description indeed comes from that sample. These confidence values are then passed to a softmax function which computes the predicted probabilities of the samples to be the original randomly picked sample. Finally, the aim of this network is to minimize the cross entropy between the picked samples and the computed probability distribution. A more detailed overview of the architectures used in the encoder and discriminator networks is given in Figure 2 . 

2.3. DISCRIMINATOR LOSS:

In this section, we want to motivate our method from an information entropy stand point. So let us first assume, we have an encoder function, latent(x j ), which maps x j from sample space onto a latent description space. Then we can introduce the probability distribution p(x i = x j |latent(x j )) over x i which describes the probability that sample x i is equal to x j given the latent representation of x j . The latent description is most precise if latent(x j ) is chosen such that the entropy of p(x i = x j |latent(x j )) is minimized:

H(p(x i = x j |latent(x j ))) = - xi p(x i = x j |latent(x j )) log(p(x i = x j |latent(x j )))

This entropy value is minimized when there is no x i other than x j which is mapped on the same latent(x j ). In other words, this means latent(x j ) provides a unique description of x j . Now, we can sum the entropy of p(x i = x j |latent(x j )) over all x j in order to enforce latent(x j ) to provide good descriptions for all samples:

Ĥ = xj H(p(x i = x j |latent(x j )))

Ĥ is minimized by latent(x j ), if latent(x j ) provides a unique description for every x j . This condition alone, however, is not sufficient to also learn meaningful representations since a simple hashing function which returns a unique hash value for every sample would give perfect results with respect to Ĥ. In the end, we want to make sense of the latent description and this is where the discriminator network q(latent(x j ), x i ) comes in. By training q on p, we can ensure that the descriptions of the encoder are interpretable by the discriminator. By minimizing the cross entropy between p and q, we can train q to become closer and closer to p. Now the loss function becomes:

L(latent, q) = xj ,xi -p(x i = x j |latent(x j )) log(q(latent(x j ), x i )) = xj ,xi -δ i,j log(q(latent(x j ), x i ))

Here δ i,j is 1 for i=j and 0 otherwise. This is a valid assumption since the probability that latent(x j ) maps two different samples of the dataset onto exact the same position in the continuous latent space is basically zero. Thus, p(x i = x j |latent(x j )) becomes δ i,j . As q approaches p, the loss function approximates the entropy Ĥ as defined above. Hence, training on this loss end-to-end not only optimizes the discriminator q with respect to the cross-entropy but also the encoder to make as precise as possible descriptions. Note, however, that q can not be calculated with one separate sample alone but always needs other samples to be properly normalized in the softmax layer.

2.4. LATENT DISTANCE:

With the results obtained above, we can introduce a probabilistic measure of distance, which might be helpful to find related samples in a dataset:

d(x i , x j ) = log q(latent(x i ), x i ) q(latent(x i ), x j )

Note, that this definition of a distance is neither commutative nor positive definite and should not be confused with the L 2 distance in latent space. It rather is a measure of how likely it is that the discriminator confuses x j with x i given the latent description of x i . The smaller the distance, the more likely is a confusion and the more similar are x i and x j . If x i = x j , the distance is 0.

3. EXPERIMENTS:

We performed two experiments using the proposed encoder discriminator architecture. The first one was performed on the MNIST (Modified National Institute of Standards and Technology) database (LeCun et al. ( 2010)). The second was done on the Fashion MNIST dataset (Xiao et al. (2017) ).

3.1. TRAINING DETAILS:

The network was trained on batches of 100 samples and the latent space was chosen to be two dimensional and bounded between zero and one. We furthermore added Gaussian noise and random affine transformations to the samples before they were passed to the encoder and discriminator networks. This greatly improves the performance since it enforces the encoder to learn more robust representations.

3.2. MNIST:

The MNIST dataset contains 60.000 training and 10.000 test images of handwritten digits. Figure 3 shows how the encoder network maps samples of the 10 different digit classes onto the latent space.

The encoder clearly separates the different digit classes and thus learns useful abstraction features. However there are still problems in particular in between the digits classes 8 and 5 as well as 4 and 9. This matches our intuition that these classes should be the hardest to discriminate. Another observation is that the encoder fills the available latent space "densely", because the encoder has to use the available latent space as efficiently as possible. This is in contrast to other common clustering algorithms that usually try to cover semantic differences in the form of larger L 2 distances in latent space. Figure 4 shows some MNIST images plotted in the latent space. As one can clearly see in the class of ones, the encoder also learns additional features (like the digit angle) that help the discriminator to also distinguish between samples of the same class. The Fashion MNIST dataset is a dataset containing 60.000 training and 10.000 test images of 10 different product categories available at Zalando. These categories comprise T-shirts/tops, trousers, pullovers, dresses, coats, sandals and more. The idea behind this dataset is that different classes of product categories contain samples with higher variability than different digits of the original MNIST dataset and thus are harder to cluster. Figure 5 shows how the encoder network maps samples of the different product categories onto the latent space. As for the MNIST dataset, the encoder fills the latent space densely and is capable of separating several categories. Problems arise in particular in between the categories "Pullover", "Coat", "Shirt" -categories that are sometimes even for the human eye hard to differentiate. Figure 6 shows some Fashion MNIST images plotted in the latent space. In this plot one can clearly observe that on top of sorting for categories, the encoder 

4. CONCLUSION:

In this work, we motivated a new encoder discriminator architecture by taking inspiration from the game "Taboo" and using the concept of information entropy. First promising experiments on the MNIST and Fashion MNIST dataset show that this approach is capable of clustering samples of the same class. We thus conclude that this encoder discriminator networks can learn unsupervised meaningful abstractions that might be helpful in tasks like image classification or scene understanding.

In the future this architecture could be applied to larger datasets such as ImageNet and a higher dimensional latent space in order to perform for example unsupervised transfer learning (Pan et al. ( 2010)) for classification tasks. Another application could be to use the output of the encoder network as the source of latent noise for generator networks in GANs. By passing also this latent description to the discriminator, the problem of mode collapse could be addressed.

:



Figures and Tables:

Figure 1: Visualization of the Encoder Discriminator concept

Figure 2: Architecture of the Encoder and Discriminator networks. The convolutional layers use relu activation functions, the fully connected layers use sigmoid activations.

Figure 3: Test labels of the MNIST set plotted in the two dimensional latent space. A plot of the samples themselves in latent space can be found in Figure 4

Figure5: Test labels of the Fashion MNIST set plotted in the two dimensional latent space. A plot of the samples themselves in latent space can be found in Figure6

Figure 6: Test samples of the Fashion MNIST set plotted in the two dimensional latent space. Best viewed in electronic form

Peer review (iclr2019):
 The paper proposes a novel encoder-based unsupervised representation learning. The network consists an encoder and a discriminator. The encoder learns to generate sample-specific codes, and discriminator tries to assign the generated code to the corresponding sample, given the correct sample and a set of incorrect samples. The method is tested on two relatively small datasets: MNIST and Fashion MNIST. The essence of the method is novel and make sense. The text is short but clear. the method technically sounds. However, the experimental part lacks a quantitative evaluation as well as a comparison to state-of-the-art methods, which makes the assessment of the proposed model difficult. Qualitative results are informative but not enough. Another disappointing point in experiments is that the method is evaluated on two relatively small datasets, with very small latent space (d=2). I expected two major experiments: (1) applying the model on a bigger dataset, as mentioned in future works, (2) consider this game as a self-supervised representation learning method and use it as an initialization for a supervised task. It gives the opportunity to compare it against others as well. other points: - the paper lacks a comprehensive literature review. the topic is well studied and deserves to cite more than 8 references. - some details are missing in the text e.g. fully-connected size, training hyper-parameters, number of incorrect samples fed to discriminator during training, training time, number of parameters. - ablation study: which one is a better feature learning network: Encoder or convolutional layers of Discriminator? In general, I believe that some significant analyses are missing and it is not possible to accept a paper without having additional insights.