Title: LOG HYPERBOLIC COSINE LOSS IMPROVES VARIA-TIONAL AUTO-ENCODER

Abstract: In Variational Auto-Encoder (VAE), the default choice of reconstruction loss function between the decoded sample and the input is the squared L 2 . We propose to replace it with the log hyperbolic cosine (log-cosh) loss, which behaves as L 2 at small values and as L 1 at large values, and differentiable everywhere. Compared with L 2 , the log-cosh loss improves the reconstruction without damaging the latent space optimization, thus automatically keeping a balance between the reconstruction and the generation. Extensive experiments on MNIST and CelebA datasets show that the log-cosh reconstruction loss significantly improves the performance of VAE and its variants in output quality, measured by sharpness and FID score. In addition, the gradient of the log-cosh is a simple tanh function, which makes the implementation of gradient descent as simple as adding one sentence in coding.

1. INTRODUCTION:

Unsupervised generative modeling aims to represent probability distributions over input data. The explicit approach usually employs certain parametric model to estimate and approximate the unknown data distribution p data , but the effect is sensitive to the model selection, for which expert knowledge is often needed. In a different vein, implicit generative models aim to generate samples from the data distribution without estimating the distribution itself. This greatly weaken the dependency on the model selection, and finds numerous applications such as simulating possible futures in reinforcement learning Finn & Levine (2017) , predicting the next frame of a video sequence Lotter et al. (2015) and image super-resolution Ledig et al. (2017) . Prominent examples of implicit generative models include Boltzmann machines Aarts & Korst (1988) , Generative Adversarial Network (GAN) Goodfellow et al. (2014) and Variational Auto-Encoder (VAE) Kingma & Welling (2013) . Among these methods, VAE enjoys an efficient and stable training process, and the encoder component of VAE gives dimension reduction and feature learning as a by-product. Meanwhile, it is also observed that images generated by VAE are often blurry. One potential reason is that the model fails to represent high-dimensional probability distributions accurately, which is the target issue we aim to address in this paper.

1.1. VARIATION AUTO-ENCODERS:

We now give a very brief account of VAE. In VAE, there is an encoder that converts the real input data x to a latent code z ∼ q φ (z|x), and a decoder that reconstructs an image x ∼ p θ (x|z) from the latent code z, where q and p are encoder and decoder, with parameter φ and θ respectively. VAE has two objectives: one is to match the decoded samples x to the input x, and the other is to maintain the posterior q φ (z|x) of the hidden code vector z to a given prior distribution p(z). Combining these two, VAE uses the following loss function for datapoint x.

L(θ, φ; x) = -E q φ (z|x) log p θ (x|z) + D KL (q φ (z|x) p(z)).

(1)

Here the first term is the reconstruction loss, which reduces to the squared L 2 loss x -x 2 2 if we assume that the decoder predicts a Gaussian distribution at each pixel i.e. p θ (x|z) = N (x, σ 2 I). The second term of the objective matches the distribution of latent code to a prior p(z), which by default is a standard normal distribution.

1.2. RECONSTRUCTION LOSS:

To improve accuracy of representing probability distributions, many variants of VAE have been proposed. Much attention has been paid to improving the design of the second term, for example, changing the prior on the latent code (e.g. S-VAE Davidson et al. (2018) , VAE with householder flow Tomczak & Welling (2016)), and changing the loss between the latent distribution and the prior (e.g. AAE Makhzani et al. (2016) , WAE Tolstikhin et al. (2018) ).

An alternative approach for improvement is through the reconstruction loss, i.e. the first term in Eq. ( 1), which measures the error between the decoded sample and the input data points. Although the squared L 2 loss is widely used as reconstruction loss in VAE setting, it has long been observed that this loss function results in problems such as blurry images Mathieu et al. (2015) . Several new metrics are proposed. Ridgeway et al. (2015) applied the structural-similarity (SSIM) index Wang et al. (2004) as a reconstruction metric of an auto-encoder. Larsen et al. ( 2016) employs a GAN discriminator to compute the reconstruction loss in VAE. Dosovitskiy & Brox (2016) combined a GAN discriminator with the perceptual loss which computes distances between image features extracted by deep neural networks. These experiments demonstrate the potential of improving VAE by re-designing the reconstruction loss. However, disadvantages are also evident: The first one Ridgeway et al. (2015) only demonstrated results on gray-scale images, which are very blurry. The last two Larsen et al. (2016) ; Dosovitskiy & Brox (2016) reported that their produce images suffer from severe high-frequency artifacts, possibly due to feature loss as discussed in Mahendran & Vedaldi (2015) . In addition, as other GAN-based methods, introducing a discriminator makes the training harder and instable.

In this paper, we propose an element-wise reconstruction loss based on a simple log-cosh function

EQUATION

where a ∈ R + is a parameter and log is the natural logarithm. In Sec. 4 we show empirically that even if the model trained with the log-cosh reconstruction loss achieved a larger error measured by L 2 loss, it significantly outperforms the model trained with squared L 2 loss. The result also indicates that squared L 2 loss can be ineffective in evaluating high-dimensional data like images, which will be discussed in more details later. 

2. LIMITATIONS OF TRADITIONAL LOSS:

Given two vectors x and x, the squared L 2 loss and the L 1 loss are defined as

EQUATION

EQUATION

respectively, where x i stands for the i-th entry of x. The loss functions also extend to matrices by treating them as long vectors obtained from concatenating the rows. When applied in the VAE setting, x and x are the input of encoder and output of decoder, respectively.

As mentioned in the last section, the squared L 2 loss widely used in VAE comes naturally from multivariate Gaussian distribution p θ (x|z) ∝ exp(-x -x 2 2 ). Other than the aforementioned issue of causing blurry images, the squared L 2 loss is also well known to be sensitive to large noise, and it has long been argued that in image space, it is ineffective in evaluating visual quality Wang & Bovik (2009) and makes the training to easily get stuck in local minima Zhao et al. (2017) .

If we choose a decoder with zero-mean Laplace distribution p θ (x|z) ∝ exp(-x -x 1 ), we will obtain a loss function in terms of the L 1 norm. The L 1 loss is used as image to image loss in some variants of VAE and GAN, such as AEGAN Rosca et al. (2017) and cycle GAN Zhu et al. (2017) . The gradient of L 1 loss is ±1 at differential points. The L 1 loss is generally robust to noise and, when used as a regularizer, encourages sparsity in solution. It is found to be quite useful in computer vision tasks such as deblurring Cai et al. (2009) ; Freeman et al. (2009) ; Xu & Jia (2010) , since it appears to enable the solver to escape from local minima, although the exact reason for this is still elusive. In addition, Zhao et al. (2017) shows that L 1 can outperform L 2 when directly used as a loss function for neural networks in image restoration applications such as denoising Jain & Seung (2009) , deblurring Xu et al. (2014) , demosaicking Wang (2014) , and super-resolution Dong et al. (2014) . However, an obvious disadvantage of the L 1 loss is that it is not differentiable if any entry x i -xi = 0, causing oscillation between ±1 during the training process.

In this paper, we will show that our proposed log-cosh loss takes the advantage of L 1 while overcoming the problem of non-differentiability. Experiments show that the log-cosh loss outperforms both L 1 and L 2 loss.

3. LOG-COSH LOSS:



3.1. PROPERTIES OF LOG-COSH LOSS:

Based on the log-cosh function shown in Eq. ( 2), we propose to use the log-cosh loss as the reconstruction error, taking advantages of both squared L 2 loss and L 1 loss while overcome their limitations. The log-cosh loss is defined as

EQUATION

where a ∈ R + is a hyper-parameter. We will first show that the log-cosh function behaves like L 2 around origin and like L 1 at other points.

Proposition 1. The log-cosh function f (t; a) for a > 0 approximates |t| -1 a log 2 when t → ∞, and approximates 0.5at 2 when t → 0. More precisely,

f (t; a) = 1 a log(cosh(at)) = 1 a log e at + e -at 2 →      |t| - 1 a log 2, when |t| → ∞ 0.5at 2 , when |t| → 0 (6)

Proof. When |t| → ∞, we have f (t; a) = 1 a log((e a|t| +e -a|t| )/2) → 1 a log(e a|t| /2) = |t|-1 a log 2. When |t| → 0, the Taylor expansion gives f (t; a) = 0.5at 2 + O(t 4 ). For general t, one has bounds |t| -

1 a log 2 ≤ f (t; a) ≤ |t| -1 a log 2 + 1 a .

The log-cosh loss can be viewed as a smoothed version of L 1 loss differentiable everywhere. The gradient of f (t; a) is

EQUATION

where tanh(t) = (e t -e -t )/(e t + e -t ) is the hyperbolic tangent function and σ(t) = 1/(1 + e -t ) is the sigmoid function, a commonly used activation function in deep networks.

3.2. MOTIVATION OF APPLYING LOG-COSH LOSS TO VAE:

The log-cosh loss applied to VAE can be understood in the following way, which was also how we obtained the method. In general we hope to balance the reconstruction accuracy and latent space optimization in VAE. If the squared L 2 loss is used for the reconstruction, then the reconstruction accuracy deteriorates in the region of small reconstruction error, as L 2 penalizes these small errors too lightly, making the objection function Eq.( 1) to be dominated by the second term (the KL divergence term). To solve this problem, simply increasing the relative weight of the reconstruction loss does not work well, since it harms the optimization in the latent space when the reconstruction error is large. Putting both large-error and small-error regions into consideration, a feasible solution is to increase the weight of reconstruction loss when it is small, but at the same time to clip the gradient of the loss with an upper bound so that the reconstruction penalty does not always increase linearly with the error. A smooth approximation of this is the tanh function, whose integral turns out to be the log-cosh function. The resulting log-cosh loss becomes more sensitive to small errors because it behaves as a 0.5a-scaled squared L 2 loss around the origin. At the same time, it avoids penalizing too much when error is large when it behaves likes the L 1 loss. Therefore, we are motivated to implement the log-cosh loss to improve the reconstruction accuracy of VAE without damaging its latent space optimization.

The implementation of the log-cosh loss in VAE is quite simple, as the gradient is explicitly available as shown in Eq. ( 7). In tensorflow the implementation is as simple as replacing the reconstruction loss in the back-propagation with

x * tf .stop gradient(2 * tf .sigmoid(2a * (x -x)) -1),

where x and x are the input of encoder and output of decoder, respectively. This form also avoids overflow or underflow during the computation, as it directly combines the gradient of the log-cosh loss (w.r.t decoder samples) and the gradient of the samples (w.r.t parameters).

4. EXPERIMENTS:

We empirically compare the log-cosh loss to the squared L 2 loss in this section, leaving the comparison to L 1 to Appendix B. All the objective functions studied are explicitly given in Appendix A.1.

The experiments are run on two datasets: (1) MNIST LeCun et al. (1998) , which consists of 70k images of handwritten digits with size 28 × 28, and (2) CelebA Liu et al. (2015) , which contains roughly 203k celebrity images cropped and resized to 64 × 64 as many previous works. We quantify the performance of VAE using two measures. The first one is sharpness, for which every image is convolved with the Laplace filter

0 1 0 1 -4 1 0 1 0

, and then the variance of the activations is computed as the sharpness score. The second one is Fréchet Inception Distance (FID) Heusel et al. (2017) , for which the coding layer of a classification model trained by us is used to extract features and the Fréchet distance Fréchet (1957) is computed between distributions of the layer obtained from two groups of samples. FID captures the similarity of generated images to real ones in terms of visionrelevant features. We demonstrate that the log-cosh loss performs well on different architectures, including VAE Kingma & Welling (2013) and Wasserstein Auto-Encoders (WAE) Tolstikhin et al. (2018) , with multi-layered perceptrons (MLP) networks and convolutional (Conv) networks.

4.1. EXPERIMENTAL SETUP:

In all the experiments we use an Euclidian latent spaces Z = R d and Gaussian prior distributions p θ (z) = N (Z; 0, I d ). Following Tolstikhin et al. (2018) , we use d = 8 for MNIST and d = 64 for CelebA. For MNIST, we report results for MLP architecture and Conv architecture. For CelebA, we only use convolutional networks, considering that VAE implemented with MLP always performs badly on complex images like human faces. In all models, we use the sigmoid function at the output layer of the decoder.

We test the quality of reconstructed samples and generated samples. The reconstructed samples are the outputs of the decoder given vectors z encoded from input image samples. The generated samples are obtained by decoding random noise vectors z sampled from standard Gaussian. The sharpness we obtain is averaged across 10 3 randomly selected images, and the FID is computed based on 10 4 randomly generated samples. For further details on experiment setup, please refer to Appendix A.

4.2. RESULTS:

Numerical results on MNIST dataset are summarized in Table . 1, from which we can see that the logcosh loss improves the sharpness up to 245% and the FID up to 78%. In all models, the log-cosh loss consistently outperforms the squared L 2 loss. Larger sharpness stands for images of higher quality while smaller FID means that the distribution of generated samples is closer to the distribution of real dataset in terms of visual-relevant features. One can also observe that WAE improves sample quality compared with plain VAE, but the log-cosh loss can further improve WAE. We show generated samples in Fig. 2 . As can be seen, many samples generated by traditional methods are quite blurry; some blurry examples are highlighted using colored boxes. After replacing the squared L 2 loss with our log-cosh function, we get sharper examples with a clearer background.

The reconstructed samples are shown in Fig. 3 , which further confirms the benefit of the log-cosh loss. Our method is able to reconstruct sharp samples accurately. We use boxes to highlight some corresponding samples that are either wrongly reconstructed or extremely blurry when the squared L 2 loss function is used. We also conduct interpolation experiments, i.e., linearly interpolate between latent vectors and decode samples from these vectors. The interpolation results are reported in Appendix C, and suggest that our method enforces well-behaved manifolds in the latent space.

Results on CelebA dataset are summarized in Table . 2. In VAE, the log-cosh loss outperforms the squared L 2 loss by a factor of ≈ 30% in both sharpness and FID. In WAE, the sharpness does not improve much, while the FID score is significantly improved by 30%. This indicates that our method better approximates the real dataset distribution. Generated samples shown in Fig. 4 further confirms this: Some samples of traditional methods are very dark and blurry, and some suffer from serious artifacts as marked by boxes. In comparison, our method generates many fewer such examples. In Fig. 5 we show the reconstruction results, and we find our method to reconstruct input images more accurately. For example, the images reconstructed by traditional method and marked by boxes have either a wrong facial expression or a wrong gender, while our method reconstructs these features accurately. 

4.3. DISCUSSION:

In this section, we present a more in-depth analysis of the results shown in Sec. 4.2.

4.3.1. IS L 2 A GOOD DISTANCE MEASURE FOR SIMILAR PICTURES?:

We study the titled question in the context of VAE by examining the L 2 and log-cosh loss functions. Take VAE (MLP) as an example, and use the squared L 2 and the log-cosh as loss functions. We keep track of the quality of the parametrized networks N i L2 and N i log-cosh after each mini-batch i during the training. Here the quality is measured by the squared L 2 loss when running the networks on the testing set, and this quality comparison is recorded in Fig. 6 (a). We can see that both training processes converge, and the L 2 loss of the N i L2 networks is consistently smaller than that of the N i log-cosh networks. However, as we have seen from both the sharpness/FID scores and visual comparison in Sec. 4.2, the N i log-cosh networks actually generate better figures than the N i L2 networks. We also showcase one specific example in the figure, where two pictures of digit 3 are reconstructed. The upper one is clearly better, but it has a larger L 2 error. This suggests that the L 2 loss function is not a good measure for similar pictures in the context of VAE.

4.3.2. BALANCE BETWEEN RECONSTRUCTION AND LATENT SPACE:

In this section we explain that the log-cosh loss contributes to finding a better balance between the reconstruction and the latent space optimization, thus improving the performance of VAE. Recall that VAE simultaneously minimizes two losses: the reconstruction loss which measures the error between input of encoder and output of decoder, and the latent space loss which matches the distribution of latent vectors to a given prior distribution. The two losses are conflicting each other, and decreasing one often causes the other to increase. The trade-off between reconstruction and latent space is crucial because the goal is to accurately reconstruct the data points and at the same time to obtain a compact manifold on latent space for generation.

We show in Sec. 3 that the log-cosh loss behaves like a 0.5a-scaled squared L 2 loss at small errors, and like a L 1 loss at large errors. When the reconstruction error is small, a large a makes it visible in the objective function and in turn further decreases the reconstruction error. When the error is large, the gradient of log-cosh loss is bounded by L 1 , which avoids harming latent space much. We (c) Generation quality measured by FID (smaller is better) of models trained with log-cosh and λ-scaled squared L 2 loss. We show reconstructed digit 3 of models trained with L 2 on the top and log-cosh on the bottom. The curvature at 0 is a for log-cosh with parameter a and 2λ for λscaled squared L 2 . Since the log-cosh loss behaves like a 0.5a-scaled squared L 2 loss only when the error is small and is not sensitive to large outliers, it does not harm optimization of latent space much and increasing a improves reconstruction and generation simultaneously. In contrast, without our loss, simply scaling the L 2 does improve reconstruction but severely harm generation. Moreover, the reconstructed samples are still worse than our method.

show in Fig. 6 (b) that the latent space error of model trained with the log-cosh loss (a = 100) does not increase much compared with the one trained with L 2 . To further investigate the relationship between reconstruction and latent space, we train models with increasing parameter a and draw the reconstructed digits and FID of generation in Fig. 6(c ), which shows that our method improves reconstruction and generation simultaneously. This confirms that the log-cosh loss improves reconstruction without breaking its balance with latent space optimization. In contrast, without our log-cosh loss, the λ-scaled squared L 2 does improve the reconstruction sometimes with increasing λ. However, the scaled L 2 is quite sensitive to outliers, which harms optimization of latent space especially when the reconstruction error is large, thus severely decreasing generation quality as shown in Fig. 6(c) .

4.3.3. MORE APPLICATIONS AND FUTURE WORK:

The advantages of the log-cosh loss should make it useful in more applications. We provide an example of text removal in Appendix D, where the log-cosh loss consistently outperforms the L 2 and L 1 loss. It is worth trying to apply the log-cosh loss to various image restoration tasks where L 1 can outperform L 2 , as demonstrated by Zhao et al. (2017) . More tasks worth exploring include denoising Jain & Seung (2009) , deblurring Xu et al. (2014) , demosaicking Wang (2014) , superresolution Dong et al. (2014) , etc.

5. CONCLUSION:

In this paper, we propose to use the log-cosh loss as the reconstruction error of VAE. 1 We provide theoretical justifications, and conduct extensive experiments. Empirical results demonstrate that compared with traditional L 2 loss, the log-cosh loss improves the reconstruction without damaging the latent space optimization, thus automatically keeping a balance between the reconstruction and the generation. As a result, our method significantly improves the performance of VAE on image data in terms of sharpness and FID score. We also show the usefulness of the log-cosh loss for text removal, and call for more systematic studies of the loss function on other tasks in vision and general machine learning problems.

A FURTHER DETAILS ON EXPERIMENTS:



A.1 OBJECTIVE FUNCTIONS:

In all implementations, we just change the reconstruction loss term while keep the loss term of latent space unchanged. We list all objective functions here, including VAE and WAE with the squared L 2 , L 1 and log-cosh loss as the reconstruction loss.

L VAE-L2 (θ, φ; x) = λ • E z∼q φ (z|x) x∼p θ (x|z) x -x 2 2 + D KL (q φ (z|x) p(z)) L VAE-L1 (θ, φ; x) = λ • E z∼q φ (z|x) x∼p θ (x|z) x -x 1 + D KL (q φ (z|x) p(z)) L VAE-log -cosh (θ, φ; x) = λ • E z∼q φ (z|x) x∼p θ (x|z) L log -cosh (x, x) + D KL (q φ (z|x) p(z)) L WAE-L2 (θ, φ; x) = λ • E z∼q φ (z|x) x∼p θ (x|z) x -x 2 2 + D MMD (q φ (z|x) p(z)) L WAE-L1 (θ, φ; x) = λ • E z∼q φ (z|x) x∼p θ (x|z) x -x 1 + D MMD (q φ (z|x) p(z)) L WAE-log -cosh (θ, φ; x) = λ • E z∼q φ (z|x) x∼p θ (x|z) L log -cosh (x, x) + D MMD (q φ (z|x) p(z)) (8)

where λ > 0 is a hyperparameter which by default equals to 1 in the context of VAE, the L log -cosh is the log-cosh loss defined in Eq. ( 5), D KL is the KL-divergence and D MMD is a divergence called the maximum mean discrepancy Tolstikhin et al. (2018) . To be accurately, the loss function of WAE should be given w.r.t batches of x and now the second term is D MMD ( x q φ (z|x)p(x)dx p(z)).

A.2 MNIST:

In all experiments on MNIST, we use mini-batches of size 100 and train the model for 40 epochs. The activation function is Rectified Linear Unit (ReLU). All models are trained by Adam with the learning rate η = 10 -3 . The log-cosh reconstruction loss Eq.( 2) is implemented with parameter a = 100. VAE (MLP) and WAE (MLP) use the relatively simple neural networks of multi-layered perceptrons (MLP). The encoder architecture is: x ∈ R 28×28 → F C 500 → F C 500 → F C 8 , and the decoder architecture is:

z ∈ R 8 → F C 500 → F C 500 → F C 28×28 .

In the VAE (Conv) and WAE (Conv), we use convolutional architectures with 4 × 4 convolutional filters and batch normalization Ioffe & Szegedy (2015) . The encoder architecture is:

x ∈ R 28×28 → Conv 128 → Conv 256 → Conv 512 → Conv 1024 → F C 8 , and the decoder architecture is: z ∈ R 8 → F C 7×7×1024 → Deconv 512 → Deconv 256 → Deconv 1 .

Here the Conv k stands for convolution with k filters, the Deconv k stands for deconvolution with k filters, and the F C k stands for fully connected layer with k neurons.

A.3 CELEBA:

In all experiments on CelebA, we use mini-batch size 100, ReLU activation, and Adam training algorithm as the MINST experiment. For the learning rate, we follow WAE Tolstikhin et al. (2018) by initializing it to η = 10 -3 and decreasing it by factor of 2 after 30 epochs, and further by factor of 5 after first 50 epochs. The reported VAE (Conv) was trained for 68 epochs and WAE (Conv) for 55 epochs. We use a = 100 in VAE (Conv) and a = 2.5 in WAE (Conv).

We use convolutional architectures with 4 × 4 convolutional filters and batch normalization Ioffe & Szegedy (2015) . The encoder architecture is: 

x ∈ R 64×64×3 → Conv 128 → Conv 256 → Conv 512 → Conv 1024 → F C 64 ,

B EXPERIMENTAL RESULTS OF L 1 LOSS:

In this section, we test the L 1 loss on MNIST and CelebA dataset, and compare the numerical results with the L 2 loss and the log-cosh loss reported in Sec. 4. Samples of model trained with 3 and Table 4 , where the improvement percentages are computed w.r.t the squared L 2 loss.

One can see that on the MNIST dataset, the log-cosh loss performs much better than the L 2 loss, as expected. It is also observed that the L 1 loss also performs very well, and sometimes even slightly better than the log-cosh loss. This is because 90% pixels of MNIST data are very close to binary value {0, 1} and the outputs of our networks are in the interval (0, 1), always falling into one side of the ground truth value (0 or 1). Thus the typical issue of the L 1 loss oscillating between nondifferential point does not occur often here, and thus the advantage of log-cosh being smooth around 0 does not exhibit benefits than L 1 . In comparison, for colored images or black-white images with gray pixels, the ground truth in general is somewhere between 0 and 1, and an output of a network may fall left or right of this value, suffering from the oscillation of derivatives of L 1 . Indeed, we can see that on CelebA, the L 1 loss performs much worse than the log-cosh loss in both scores (Table 4 ) and the visualization (Fig. 8 ).

C INTERPOLATION EXPERIMENTS ON MNIST:

We randomly take two samples (x 1 , x 2 ) from real dataset and feed them to the encoder to get the latent vectors (z 1 , z 2 ). Then we linearly interpolate between z 1 and z 2 in the latent space and use them to decode images. The decoded images are shown in Fig 9 . The log-cosh loss enforces well behaved manifold in latent space, which is obviously better than squared L 2 loss. It is promising to apply the log-cosh loss to various image restoration applications such as denoising, deblurring, demosaicking, and super-resolution. Here we provide an example of text removal, and show that the log-cosh loss consistently outperforms the L 2 and L 1 loss. We use a recently proposed denoising framework called noise2noise Lehtinen et al. (2018) , which learns to restore good images by only going through bad ones. The network architecture is based on the idea of convolutional auto-encoder, consisting of convolutional and deconvolutional layers. The training dataset consists of 291 images widely used (e.g. in Schulter et al. (2015) ; Kim et al. (2016) ). Among these, 200 images of the training set come from the Berkeley Segmentation Dataset Martin et al. (2001) and the rest 91 images come from Yang et al. (2010) . Following Lehtinen et al. (2018) , we add texts to the images and let the model learn to remove these texts. After training, we test the trained model on the Urban 100 dataset Huang et al. (2015) which consists of 100 high resolution images. We use the SSIM index Wang et al. (2004) which is often used as a measure to evaluate the similarity between the restored images and the target images in the denoising tasks. We compare the squared L 2 loss and the L 1 loss to the log-cosh loss, and the results are shown in Fig. 10 . We can see that the log-cosh loss consistently outperforms both the squared L 2 loss and the L 1 loss. Actually, the log-cosh loss fails to outperform the L 1 loss in only 1 out of the 100 tested images, illustrated by the red point in Fig. 10 .

We list some restored images in Fig. 11 to show how the log-cosh loss stands out. The L 2 loss may lead to a severe color shift, because the L 2 loss encourages the outputs to shift toward the mean color of the noisy images, which is locally distorted by texts. The L 1 loss encourages to take the median and does not suffer from color shift, but severe artifacts can appear, as highlighted by red boxes in Fig. 11 . The log-cosh loss releases the problems by averaging observations around the median. It behaves like the L 1 at the beginning of training when the error is large, thus avoiding the color shift. In addition, when the error is small, The log-cosh loss behaves like the L 2 and encourages to smooth the color by taking the mean, thus reducing the artifacts introduced by the L 1 loss. As highlighted by red boxes, in the results of the log-cosh loss, the artifacts are removed or sometimes still exist but their color is much closer to the background. 

:

We also propose to use the L1 loss for special image cases in which most pixels of input images are very close to 0 or 1; see Appendix B for detailed discussions on this.

Figures and Tables:

The plots of f (t; a) in interval[-1, 1]   for different a are shown in Fig.1. The function is close to L 1 for large |t| and close to L 2 for small |t|, thereby combining the smoothness advantage of L 2 , and robustness and image sharpness advantage of L 1 . In addition, the derivative of the function is simply the sigmoid function, rendering a very efficient training and very simple implementation.

Figure 1: Plots of f (t; a) = 1 a log(cosh(at)) with a = 1, 10 and 100.

Figure 2: Random generated samples of VAEs trained on MNIST dataset.

Figure 3: Reconstructed samples of VAEs trained on MNIST dataset. For comparision, some corresponding samples are marked by boxes with the same color.

Figure 4: Random generated samples of VAEs trained on CelebA dataset. Samples that are very dark and blurry or suffer from serious artifacts are highlighted.

Figure 5: Reconstructed samples of VAEs trained on CelebA dataset. For comparision, some corresponding samples are marked by boxes with the same color.

Figure 6: (a) Reconstruction error during training.For comparison, we use squared L 2 to measure the error of both models. We show two corresponding reconstructed sample here: While the upper one is clearly better, it has a larger L 2 error. (b) Latent space error measured by D KL (q φ (z|x) p θ (z)). (c) Generation quality measured by FID (smaller is better) of models trained with log-cosh and λ-scaled squared L 2 loss. We show reconstructed digit 3 of models trained with L 2 on the top and log-cosh on the bottom. The curvature at 0 is a for log-cosh with parameter a and 2λ for λscaled squared L 2 . Since the log-cosh loss behaves like a 0.5a-scaled squared L 2 loss only when the error is small and is not sensitive to large outliers, it does not harm optimization of latent space much and increasing a improves reconstruction and generation simultaneously. In contrast, without our loss, simply scaling the L 2 does improve reconstruction but severely harm generation. Moreover, the reconstructed samples are still worse than our method.

and the decoder architecture is:z ∈ R 64 → F C 8×8×1024 → Deconv 512 → Deconv 256 → Deconv 128 → Deconv 3 .

Figure 7: Samples of VAEs trained with L 1 reconstruction loss on MNIST dataset.

Figure 8: Samples of VAEs trained with L 1 reconstruction loss on CelebA dataset.

Figure 10: SSIM index of images restored with different loss. Each point corresponds to a restored image. Larger SSIM index is better. The green line indicates equal SSIM index. The log-cosh loss fails to outperform the L 1 loss in only 1 out of the 100 tested images as highlighted by red point

Figure11: Text removal results of models trained with the squared L 2 , L 1 and log-cosh loss. We list 1-SSIM here to show the difference between the restored images and the target images. The results of the L 2 loss suffers from severe color shift. The results of the L 1 loss suffer from severe artifacts. Both problems are released by the log-cosh loss.

Sharpness (larger is better) and FID (smaller is better) on MNIST dataset.

Table content: <table><tr><td colspan="2">Algorithm</td><td colspan="3">Sharpness Reconstructed Samples Genarated Samples</td><td>FID</td></tr><tr><td>VAE (MLP)</td><td colspan="2">Squared L 2 0.052 Log-cosh 0.168 (+223%)</td><td colspan="2">0.040 0.138 (+245%)</td><td>200 65 (-68%)</td></tr><tr><td>VAE (Conv)</td><td colspan="2">Squared L 2 0.068 Log-cosh 0.156 (+129%)</td><td colspan="2">0.063 0.146 (+132%)</td><td>114 25 (-78%)</td></tr><tr><td>WAE (MLP)</td><td colspan="2">Squared L 2 0.082 Log-cosh 0.158 (+93%)</td><td colspan="2">0.078 0.149 (+91%)</td><td>101 35 (-65%)</td></tr><tr><td>WAE (Conv)</td><td colspan="2">Squared L 2 0.111 Log-cosh 0.166 (+50%)</td><td colspan="2">0.100 0.159 (+59%)</td><td>56 24 (-57%)</td></tr><tr><td/><td>VAE (MLP) VAE (MLP)</td><td>VAE (Conv) VAE (Conv)</td><td>WAE (MLP) WAE (MLP)</td><td>WAE (Conv) WAE (Conv)</td></tr><tr><td>Squared Squared</td><td/><td/><td/></tr><tr><td>L L</td><td/><td/><td/></tr><tr><td>2 2</td><td/><td/><td/></tr><tr><td>Log-cosh Log-cosh</td><td/><td/><td/></tr></table>

Sharpness (larger is better) and FID (smaller is better) on MNIST dataset. All the percentages are computed w.r.t squared L 2 loss.

Table content: <table><tr><td colspan="2">Algorithm</td><td colspan="2">Sharpness Reconstructed Samples Genarated Samples</td><td>FID</td></tr><tr><td/><td colspan="2">Squared L 2 0.052</td><td>0.040</td><td>200</td></tr><tr><td>VAE (MLP)</td><td>Log-cosh</td><td>0.168 (+223%)</td><td>0.138 (+245%)</td><td>65 (-68%)</td></tr><tr><td/><td>L 1</td><td>0.175 (+237%)</td><td>0.149 (+273%)</td><td>61 (-70%)</td></tr><tr><td/><td colspan="2">Squared L 2 0.068</td><td>0.063</td><td>114</td></tr><tr><td>VAE (Conv)</td><td>Log-cosh</td><td>0.156 (+129%)</td><td>0.146 (+132%)</td><td>25 (-78%)</td></tr><tr><td/><td>L 1</td><td>0.162 (+138%)</td><td>0.154 (+144%)</td><td>22 (-81%)</td></tr><tr><td/><td colspan="2">Squared L 2 0.082</td><td>0.078</td><td>101</td></tr><tr><td>WAE (MLP)</td><td>Log-cosh</td><td>0.158 (+93%)</td><td>0.149 (+91%)</td><td>35 (-65%)</td></tr><tr><td/><td>L 1</td><td>0.162 (+98%)</td><td>0.155 (+99%)</td><td>37 (-63%)</td></tr><tr><td/><td colspan="2">Squared L 2 0.111</td><td>0.100</td><td>56</td></tr><tr><td>WAE (Conv)</td><td>Log-cosh</td><td>0.166 (+50%)</td><td>0.159 (+59%)</td><td>24 (-57%)</td></tr><tr><td/><td>L 1</td><td>0.161 (+45%)</td><td>0.152 (+52%)</td><td>31 (-45%)</td></tr></table>

Sharpness (larger is better) and FID (smaller is better) on CelebA dataset. All the percentages are computed w.r.t squared L 2 loss.

Table content: <table><tr><td>Algorithm</td><td>Sharpness Reconstructed Samples Genarated Samples</td><td>FID</td></tr></table>

Peer review (iclr2019):
 The paper proposes a new loss function which can be used in the reconstruction term of various auto-encoder architectures. The pixel-wise cost function .ell(X, X*) = f(X - X*; a) is defined for pairs of two input images X and X* and has one positive real-valued hyperparameter a. For small values of t the function f(t; a) behaves like a quadratic function, while for large t it behaves like |t|. As a consequence, it is smooth, everywhere differentiable (like L2) while not penalizing outliers too hard (like L1). The authors present several experiments conducted on MNIST and Celeba datasets, demonstrating that a simple change of a conventional pixel-wise squared L2 distance with the proposed log-cosh cost function improves the FID scores of generated samples as well as the visual quality of reconstructions (including "the sharpness"). I would say this is clearly an empirical study (even though the authors claim they provide "theoretical justifications", they are rather hand wavy), which is not a bad thing in this case. The message of the paper is very clear and I think the authors did a good job in selling their point. The main (and, perhaps, the only) contribution is the proposal to use the log-cosh function as the reconstruction cost. And this proposal is well justified by the set of experiments. However, there are several major issues: (1.1) The objective functions reported in appendix A.1 corresponding to WAE have in fact nothing to do with WAE. In WAE the regularizer penalizes the divergence between the prior distribution p(a) and *the aggregated posterior* distribution .int_x q(z|x) p(x) dx. In other words, D_MMD(q(z|x) || p(z)) in Eq. 8 should be replaced with D_MMD(.int_x q(z|x) p(x) dx || p(z)) in order to result in the WAE model. In summary, if the authors indeed used objectives reported in Eq. 8 of Appendix A, they were actually not using WAE but rather some other sort of regularized auto-encoders, which in a way are quite similar to VAEs. (1.2) I am surprised to see the reported FID scores for the Celeba data set. Having worked with this data set myself in combination with VAEs and WAEs, I am impressed with the extremely low FID scores: 46 for the vanilla L2 VAE and 30 for the L2 WAE. Note that while in the appendix the authors say they follow the architectural choices provided in [1] while performing the "L2 WAE Celeba" experiment, the authors arrive at FID=30 compared to FID=55 reported in the "Wasserstein Autoencoders" paper. Also, based on my experience, achieving FID=46 on CelebA with a vanilla VAE is very impressive. Note that the authors use 10^4 of samples to evaluate the FID scores, which is exactly the same as in [1]. This size is known to be large enough to reduce the variance of FID, so the difference (55 - 30) can not be explained by the fluctuations of FID. Therefore, I ask the authors to (anonymously) share the code and/or checkpoints of the 2 particular trained models: L2 VAE and L2 WAE trained on Celeba. Other comments: (2.1) Note that the reconstruction cost function in VAE should be normalized for every value of the code Z, as it corresponds to the logarithm of the likelihood (density) function -log p(X|Z). L2 and L1 costs both correspond to the well known likelihood (decoder) models (Gaussian and Laplace). However, it is hard to say what decoder model (what type of conditional distribution p(X|Z) ) would give rise to the proposed log-cosh function. In particular, the normalizing constant is not known and may depend on Z. In other words, by exchanging the L2 cost with the log-cosh loss in the VAE one looses the theoretical guarantees supporting VAE, including the fact that the objective is the lower bound on the marginal log likelihood. While this is not necessarily a problem (unless one uses the value of the objective as the bound on the marginal log likelihood, which is not the case in this paper), I would suggest mentioning it. Notice that, for instance, in WAE this problem does not appear, as the reconstruction term there does not involve any likelihood functions and thus does not need to be normalized. (2.2) In Figure 2 I don*t see why the authors did not highlight bad samples in the second row corresponding to their proposed method? I see many badly looking images there. Say, (4, 9) in VAE (MLP) and (8, 9) in VAE (Conv) and (6, 1) in WAE (MLP) and (2, 10) in WAE (Conv), where (i, j) means i-th row, j-th column, indexing starting from 1. (2.3) How would the Huber loss perform and how does it compare to the proposed loss? [1] Wasserstein Autoencoders. Tolstikhin et al., ICLR, 2018.