You Only Train Once: Loss-Conditional Training of Deep Networks
Authors: Alexey Dosovitskiy
Josip Djolonga
Venue:  (2020)
Keywords: deep learning
image generation

Abstract: In many machine learning problems, loss functions are weighted sums of several terms. A typical approach to dealing with these is to train multiple separate models with different selections of weights and then either choose the best one according to some criterion or keep multiple models if it is desirable to maintain a diverse set of solutions. This is inefficient both at training and at inference time. We propose a method that allows replacing multiple models trained on one loss function each by a single model trained on a distribution of losses. At test time a model trained this way can be conditioned to generate outputs corresponding to any loss from the training distribution of losses. We demonstrate this approach on three tasks with parametrized losses: beta-VAE, learned image compression, and fast style transfer.

Published as a conference paper at ICLR 2020
YOU ONLY TRAIN ONCE: LOSS-CONDITIONAL TRAINING OF DEEP NETWORKS
Alexey Dosovitskiy & Josip Djolonga Google Research, Brain Team
(adosovitskly, josipd @googgle..og.ccoo
ABSTRACT
In many machine learning problems, loss functions are weighted sums of several terms. A typical approach to dealing with these is to train multiple separate models with different selections of weights and then either choose the best one according to some criterion or keep multiple models if it is desirable maintain a diverse set of solutions. This is inefficient both at training and at inference time. We propose method that allows replacing multiple models trained one loss function each by single model trained on distribution of losses. At test time model trained this way can be conditioned to generate outputs corresponding to any loss from the training distribution of losses. We demonstrate this approach on three tasks with parameterized losses: B-V,,, learned image compression, and fast style transfer.
INTRODUCTION
When designing a model for machine learning problem at hand, a practitioner often thinks of multiple qualities that the model should have, For example, addition to requiring a good prediction performance, it may be desirable to have a classifier that is simple, so that it would better generalize to unseen data. Similarly, when training a lossy image compression model, one optimizes both the size of the compressed images and their quality. In such scenarios, multiple loss functions have to be minimized simultancously, each of them modelling a different facet the considered problem.
Typically, it is not possible to simultaneously optimize all these -osses-either due to limited model capacity, or because some of the losses may be fundamentally in conflict (for instance, image quality and the compression ratio in image compression). One thus has decide how to balance the losses during optimization. Such multi-objective problems are most commonly scalarized by linearly com- bining the losses, with weights defining the trade-off between the loss terms. However, the exact values of the chosen weights can strongly affect the performance of the model and their tuning can be cumbersome. It might be also unpredictable how the weights affect the final values of the indi- vidual losses; moreover, the actual goal may be to optimize a down-stream metric that is not being explicitly minimized during training (for instance, classification accuracy). Furthermore, in many tasks there may not be single optimal model, but it is instead important to make different trade-offs at test time. For instance, image compression, to achieve a fine-graines coverage of rate-distortion trade-offs, one would first have to train at least ten models with different parameter settings, and then inference time store them all in memory and select the appropriate one depending on the required compression level. This is inefficient both at training and at inference time. However, intuitively, the models trained for different loss variants are related and could share a large fraction of computation. Is it possible to improve the efficiency by making use of this model redundancy?
In this paper we propose a simple and broadly applicable approach that efficiently deals with multi- term loss functions and, more generally, arbitrarily parameterized loss functions. Instead of training one model for each parameter combination the loss, we train one model that covers the whole space of different loss weightings. We achieve this by (i) training the model losses instead of single loss, and (ii) conditioning the model outputs on the parameters of the
distribution of
The code will he released at www.gi wwi.tth...... com www.:ggghhb..ccmoogglleeeeeeeeee httpa////googgeeeeeeeaeaaeeeeeea htwp://ggoooogg.eeeeeeeeec////t

---

Published as a conference paper at ICLR 2020
loss function. This way, inference time the conditioning vector can be varied, allowing us to traverse the space of models corresponding to different loss functions. We dub the method You Only Train Once (YOTO). The conceptual simplicity of the approach makes it easily applicable to many problem domains, with only minimal changes to existing codebases.
We experimentally showcase the efficacy of the proposed optimization scheme on image synthesis problems, where multi-loss problems are especially widespread. We first show that (5-)variational autoencoders (Kingma & Welling 2014 Higgins al. 2017) can be trained for large range of 3 parameters at the cost of training a single fixed-3 model. Then, we train a single deep image compression model (Ballé al. 2018) that can adjust the rate-distortion trade-off on the fly at test time. Finally, we apply YOTO to style transfer (Gatys et al. [2016] Ghiasi et al. (017)) where it is provides a clear benefit given that there is no clear single optimum and that the effect of the various losses on the stylized image is difficult to characterize.
2 RELATED WORK
Losses with multiple terms are most commonly associated with multi-task learning (Caruana 1997 Kokkinos 2017; Zamir et al. 2018). where the goal is to train a single model that simultancously solves several learning tasks. We refer the reader to Ruder (2017) for a recent review. Most related to our approach are methods that condition the network architecture on the task at hand (Rebuffi et al.) 2017 Mallya et al. 2018), which an be seen as a special case of our method with "one hot" loss weight vectors. While ideas from our work could also be applied to the multi-task learning scenario, here we choose to focus on problems where it is desirable to learn a diverse set of models, rather than single well- well-performing model. Recently, Brault et al. (2019) studied simultaneous optimization of multiple loss functions in the context of multi-task kernel learning. Our work is similar in spirit, but differs drastically both technically and in the application domains.
Conditional normalization schemes have recently been used for a range of supervised learning tasks, including Conditional Batch Normalization (CBN) for visual question answering (de Vries et al.) 2017]. Conditional Instance Normalization and Adaptive Instance Normalization (AdaIN) for fast style transfer (Dumoulin al., 2017 Huang & Belongie, [017]] Ghiasi et al., 2017). Feature- Linear Modulation (FiLM) for visual reasoning (Perez et al.] 2018), and a variety of conditioning schemes for generative models (Karras al. [ Brock et al. al..|221111. 2019). comprehensive overview of conditioning methods is provided by DDmoulln et al. (2018). All these methods condition convolutional network on some side information by linearly transforming the activations with co- efficients computed from the side information. The side information typically informs the model about the prediction target: for instance, it can be the text of a question in the question answering setting the style image in style transfer. The only exception we are aware of is the recent work of Babaeizadeh & Ghiasi (2018), which conditions a style transfer network on the parameters of the loss function, same as in method. The approach is technically very similar, but formulate it as a generally applicable technique and evaluate on several tasks both qualitatively and quantitatively.
Conditional models are also often used in sensorimotor control train single model capable of addressing a family of tasks, such as throwing darts at different targets (da Silva et al.) 2012). block stacking and ball hitting (Deisenroth et al., 2014), navigating towards different locations (Schaul eet al. 2015), moving objects with robotic arm (Andrychowicz et al. (0177), driving in the desired direction (Codevilla et al. 2018) or trading off the reward components (Dosovitskiy & Koltun 2017]. A single model, conditioned on the task parameters, is trained to simultaneously maximize the rewards for a distribution of tasks. While technically similar to our setting, the conceptual difference is that between the tasks the training targets are changing (either rewards or expert actions used as labels for imitation learning), rather than the loss function itself, as in our case.
Another recent line of related work explores the use hypemnetorrs - networks that generate the weights of other networks - for hyperparameter optimization. Lorraine & Duvenaud (2018) pro- pose to parametrize the weights of a network by hypernetwork, which is conditioned on the values of the hyperparameters. The model can then be trained on a distribution of hyperparameters, and the hyperparameters can be optimized via gradient descent the validation data. MacKay et al. (2019) improve the scalability of the method by proposing efficient approximations to hypernet- works. These methods technically similar to our approach, but they focus hyperparameter
2

---

Published as a conference paper at ICLR 2020
Training:
Testing:
y
Model Fe
Fa 0.
Loss C
L(y, Fo(mr, 0),)
Model Fe Fa 0)
A (A-PP)
Figure 1: Illustration of the proposed method. At training time (left), for each training data point loss parameters À are sampled from a distribution Px, and the model is conditioned on these. Then, test time (right), the model can be conditioned on any desired loss parameters to achieve behavior corresponding to the loss function with these parameters.
optimization, while we are more interested in obtaining the full set of solutions corresponding to the different loss coefficients.
Recently, Shoshan al. (2019) and Wang et al. (2019) addressed the problem of interpolating be- tween predictions two or more image generation models. Both methods start by training several separate models and then propose different ways of interpolating between them: either by using specialized adaptor blocks in the networks, or by directly interpolating the network weights. We, in contrast, train only single model by directly optimizing the performance over distribution of losses. This conceptually simpler, can he easily integrated in existing models, and requires training a single model instead of several.
METHOD
We consider the following learning scenario: given a training distribution of pairs I, y Pa..y with a E X C RRx E Y C IRdv, and a loss function L(:,)) : Y x + IR, we aim to learn model F X W. with parameters 0, such that its predictions ý F(x, O) minimize the expected value of the loss L(y, ÿ) over the dataset. This formulation is very broad and covers a variety machine learning problems, including (self-)supervised learning learning, and some types of generative models. We focus on this setting for a more streamlined presentation, although the proposed method can be applied to models that go beyond this formulation.
Instead of single fixed loss function .,,,)), assume that we are interested in family of losses C(. A). parameterized by vector À of losses is weighted sum of several loss terms: C( parameters, such as, for instance, the power p in 8 losses, might also be included the vector A. In practice, one then typically minimizes the loss independently for each choice of the parameter vector X:
Rdd, The most common case of such a family
Σ, A (.) However, other
0% arg min Em.yy Cly, F(x 0), A).
Instead of fixing A, we propose to solve an optimization problem where the parameters À are sam- pled from a distribution Px. Hence, during training time the model observes many losses, and can learn to utilize the relationships between them to optimize all of them simultaneously. The distri- bution PA can seen as a prior over the losses, and we discuss its choice in the next subsection. At inference time, the joint model can be conditioned on arbitrary desired parameter value X, yielding the corresponding predictions. We thus solve the following optimization problem:
= arg min
Cly, Folx, 0, A),
In the limit of infinite model capacity, this new proposed optimization problem is equivalent to the original independent optimization scheme, in the sense that the minimal values for both problems coincide. Further discussion, as well as the proof, are provided in the appendix.
Proposition 1. Consider two optimization problems:
(1) min Bay-Pa Cly, F(x),
(2)
Ex- Eay-Pe Cly. G(x, A), GEC(XXA)
min
FEC(%)
3

---

Published as a conference paper at ICLR 2020
where C(S) denotes the space of continuous functions S. Assume there exists continuous function F (x, X) such that for every X E the function (- 2) solves the problem (1). Then, if G is a solution of the problem (2), G (x) also minimizes the problem (1) almost surely w.r.t. Px-
This theoretical result shows that the method is fundamentally as powerful as the standard per- parameter training, while only requiring single model. However, in practice the infinite capacity assumption does not hold, so implementation details become important, which we describe next.
3.1 PRACTICAL DETAILS
In this work we focus on training deep networks with methods based on stochastic gradient descent. We therefore approximate the expectations in iquation[lusing Monte Carlo estimates: we estimate the expectation over the data distribution by averaging over training set D = {(,,, y;) and the expectation over the loss parameters by sampling from the loss parameter distribution:
0- arg min
L(y,, A.),
A
(3)
In our experiments, we sample a new loss parameter for each training data point every time it is encountered during training.
The distribution of the loss parameters to be used for training is typically not given and has to be chosen. Clearly, the support of the training distribution has to include the range of parameter values we are interested in covering, but this still leaves lot of freedom in the specific choice of the distribution. In our experiments we use the log-uniform distribution, which is commonly used for sampling the parameters of machine learning models (Bergstra & Bengio 2012).
In our experiments the model parameters, we use Feature-wise Linear Modulation (FiLM) (Perez al. (018). First, select the layers of the network to be conditioned (can be all layers or a subset). Next, we condition each the layers the given weight parameters X. Assume the layer outputs feature map f of dimensions W x H x C, with W and H corresponding to the spatial dimensions and C' to the channels. We feed the parameter vector À to two multi-layer perceptrons (MLPs) M, and M,, to generate two vectors, a and μu, of dimensionality C each. We then multiply the feature map channel-wist by a and add Iu to get the transformed feature map f:
convolutional network. To condition the network on the loss
Fagk = ozfigk + pbs a = (A),), # M,(A).
(4)
EXPERIMENTS
We evaluate the proposed method both quantitatively and qualitatively on three problems with multi- term loss functions: 3-VAE, learned image compression, and fast style transfer. All details regarding the architectures and the training, as well as additional results, are provided the appendix.
4.1 3- VARIATIONAL AUTOENCODERS
Our first problem is the loss of the 3-variational autoencoder (Higgins al. 2017)
(5) where (=(2 x)) is the amortized approximate posterior implemented by an encoder network with parameters Φ, Po =) is stochastic decoder network with parameters 0, and p(z) is the prior distribution, in our case a standard multivariate Gaussian. The loss a single positive parameter trading off between the reconstruction quality and the divergence between the approximate posterior and the prior.
Ca.vAE(w, 2, ④, 0, 3) = (=g(( log pa (x - - ( x)) x)]l((=)),
We consider two settings: the CIFAR- 10 dataset (Krizhevsky 2009) with Gaussian outputs, and the Shapes3D dataset (Burgess & Kim 2018) with Bernoulli outputs. In both cases we use convolu- tional encoder-decoder networks with Gaussian latents. We condition after each downsampling upsampling operation. Both for YOTO and weight models, we train a set networks with varying capacities, by proportionally changing the width of all layers. This is done with the goal
4

---

Published as a conference paper at ICLR 2020
700
YOTO, width
Model
1250
600
YOTO, 02 wiith xt
1000
YOTO, wodth
Fall loss ditereecc
Fixed weights,
1005
400
P'ted weights, widtb
Fized weights, wadth x2
500
weights,
300
Method
250
YOTO
Fized weights
100
-4000
-3000
-2000
-1 1.000
10000
Reconstruction loss
Log2 of beta
(a) Frontier on CIFAR-10.
(b) Full losses on CIFAR -
150
YOTO, wodth x0.5
YOTO, wodth xa.0
1.00
30
Fixed weights, width x0.5
Pixed weights,
Full loss diferracce
168s
Fixed weights, width x4.
Method
YOTO
Fixed weights
15
-50
3500
3900
4100
Reconstraattin loss
Log2 beta
(c) Frontier Shapes3d.
(d) Full losses Shapes3D.
Figure 2: Quantitative 8- VAE results on CIFAR-10 (a,b), and Shapes3D (c,d) for models of varying capacity (width). most cases, the proposed method performs close models trained indepen- dently for each loss weight value. This is especially the case for high capacity models.
of measuring how does the capacity of YOTO model affect its performance relative to set of fixed-weight models. We train each network 3 times and report means and standard deviations over these three runs.
Quantitative results are shown in Figure For each dataset, we provide two plots: the frontier of the two loss components plotted against each other ((a), (c)), and the value of the full loss plotted against the value of the parameter 3 ((b), (d)). The first plot allows to assess how well the proposed model fits the weighted sum of the losses. To better highlight the differences between the methods, when plotting the full loss, we normalize it by subtracting the loss of one of the models from all other models.
loss terms, while the second one shows the actual optimization objective
the
On both datasets, we observe several common tendencies. First, for small-capacity models the pro- posed method somewhat underperforms relative to the fixed-weight models, especially CIFAR- 10. This is not surprising, since it is difficult for small model to cover all loss variants. Second, both the YOTO models and the ixee-weight ones improve substantially with increasing model ca- pacity, so set of fixed-wer weight models of a certain width is consistently matched or outperformed by a twice wider YOTO model (this is additionally highlighted Figure plots the average full loss versus the width of the network). Third, the larger the network capacity, the closer the perfor- mance of fxxedmodels and the YOTO model. In particular, on both datasets for the widest networks the two frontiers become very close. This is quite intuitive and matches the theoretical result that in the limit infinite model capacity the proposed method should be as powerful as training separate fixed- fixed-weight weight models.
Qualitative results are shown in Figure 3 Both for reconstruction and sampling, for each value of the weight 3 that show YOTO produces results closely resembling those of the corresponding
5

---

Published as a conference paper at ICLR 2020
YOTO
Fixed weight
orig.
B 128
B = 512
(a) Image reconstructions using the trained 3- 9-VAE models.
B = 128
B = 512
(b) Samples from the trained 3-VAE models.
Figure 3:
Qualitative VAE results on the Shapes3D dataset. For each loss weight B, the recon- struction and sampling results of YOTO are very similar to those of the separately trained models.
-000
4000
Method
3900
YOTO
Fixed weights
Flond weights
600
3940
-800
p-all lo0e
3920
Pall
3900
-100
1200
3880
11400
3820
1.0
2.0
Widthh
Width factor
(a) CIFAR-10.
(b) Shapes3D.
Figure 4: Loss averaged over all loss weight values function of the model capacity. We compare the proposed method set of fixiht models on two datasets: CIFAR-10 (a) and Shapes31) (b). We vary the capacity of the network by multiplying the width of the network by factor. The performance of the proposed method overall matches that of fixed-based models with roughly 1.5x wider model. Moreover, the performance of the proposed method gets closer to that of the fixed-weight models when the capacity of the model is increased.
independently trained model. An interesting property of the YOTO model is that sampling with the same noise vector for different beta values generates similar images at different quality levels.
Additional results, including comparison to two simple baselines single fxeed-weigt model and a baseline based on interpolation), an ablation study of the proposed method, and qualitative results on CIFAR-10, are shown in the appendix.
4.2 LEARNED IMAGE COMPRESSION
We build the learned compression model of Ballé et al. (2018). The model is similar varia- tional autoencoder, but with few modifications that allow for the quantization the latent repre- sentation and its compressed image encoding. Moreover, it includes "hyperprior" learned

---

Published as a conference paper at ICLR 2020
40
38
38
36
PSNR
PSNR
34
34
30
YOТО
30
width x1.33
28
YOTO width x1.33
Fixed weight
Fixed weight
28
26
0.2
0.4
0.6
0.0
1.0
1.2
1.4
1.6
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Bits per pixel
Bits per pixel
(a) PSNR on Tecnick.
(b) PSNR Kodak.
Figure 5: Quantitative compression results on the Tecnick and Kodak datasets. A basic YOTO model under-performs compared to a of fixed-weight models, but wider network nearly matches the performance of the fixed-weight models, especially in the high-quality regime.
by second variational autoencoder on top of the latents of the first one. The loss function has single parameter the "rate-distortion weight", trading off compression rate versus reconstruction quality. We refer the reader to (Ballé et al.] 2018) for details. The architecture of the model includes four networks: a convolutional encoder and hyper-encoder, as well as a transposed convolutional decoder and hyper-decoder. The networks are relatively shallow, with three convolutional layers each. We apply conditioning to all layers of the model. Same as for VAEs, we train three models for each setting to estimate the stability of training. Further details are provided in the appendix.
We evaluate the compression models on two datasets: Kodak (Kodak 1993) and Tecnick (Asuni & Giachetti] 2014). The quantitative results Tecnick shown in Figure 5 while the results Kodak are deferred to the appendix. We show rate-distortion plots with standard PSNR on pixel values measure of compression quality and bits per pixel measure of compression rate. We plot three models per method, but since the training is stable, there difference between them almost not visible. The gap between separate fixed-weight models ("Fixed weight") and single joint model of the same size ("YOTO") is fairly small, but consistent across all compression rates and especially pronounced the higher quality regime. However, making the model larger ("YOTO width x x1.33") recovers a large fraction of the performance, in particular the wider model matches the performance of smaller fixed-weight model in the high quality regime. However, the high- models.
compression regime there is still small gap from the
We hypothesize that the reason for relatively worse performance of YOTO on the compression task compared to 3-VAEs is that the networks used by [Bllé et al. (2018) are relatively shallow. They therefore lack the representational power to efficiently make use the conditioning input. We expect that could devise combinations of architectures and conditioning methods for the compression task which would lead a smaller gap between the fixed-weigt and YOTO models; however, this additional tuning is beyond the scope of this paper.
4.3 STYLE TRANSFER
The final problem we consider style transfer (Gatys al. 2016) given a content image x. and a style image T.. the goal is to synthesize stylized image that contains the same content as æ., but style that resembles that of æ,, This typically cast as an optimization problem we want to find an image that when passed through a pre-trained neural network has activations similar to those æ., but whose aggregate feature statistics resemble those of x.. There is lot freedom in deciding which layers of the network to use and how weigh them. These parameters are not easy to tune, since we are optimizing for visual appearance, which cannot easily quantified. Hence, a single model capable of representing all loss variants would be of great benefit. These experiments are similar to those presented by Babaeizadeh Ghiasi (2018), but with more focus on a quantitative evaluation, same as other applications reported above.

---

Published as a conference paper at ICLR 2020
16-36
VOTT opecatst
Saam stole Janaza
Dland
Re-32
proepppmtttonn midels
inderpolition beadln
De 100
10
10
1A,-10
"A
AA, 1,
Condgarrtion
2e954
Lends
20116
(a) Performance of YOTO when we unilaterally vary each weight. We show means and standard deviations over four runs.
Content lona
(b) Sum of all style losses plotted against the content loss.
Figure 6: Quantitative results on style transfer. In both cases YOTO performance approaches or matches that of separate models trained per loss parameter vector.
Fixed YOTO
(a) Varying the content coefficient Ac
(h) Varying one of the style coefficients As.3
Figure 7: Qualitative comparison of image stylization models image from the validation set of ImageNet. The first row shows the results of YOTO trained on all parameters, and the second row shows the results of fixed-weight models trained independently for each loss parameters. In both cases the parameter values increase from left to right. YOTO generates results very similar to the separate models, while training just single model.
We build on the work of Ghiasi et al. (2017). who propose training single deep network that can stylize images in arbitrary styles. To this end, the content image is fed to the stylization network itself, while the style image is fed to a different "style prediction" network that extracts a feature em- bedding. This embedding is then used modulate the stylization network via conditional instance normalization. We refer the reader to (Ghiasi et al. 2017) for a detailed explanation.
Following Ghiasi al. (2017), we extract the activations from the VGG network (Simonyan & Zisserman 2015). and define total of six losses: four style losses C..1 (y. ...,..., L x.,, one content loss C. (y) and total variation loss C,, (y) encouraging smoothness between the neighbouring pixels of y. The final loss is then linear weighting of these, which we will denote as C(y) > Ag,; a..)) ) Xaha( c) + (v) We train a single YOTO model over five of these parameters (while keeping the TV loss fixed 10*), with log-uniform distributions. To this end, in addition to the style embeddings we condition the stylization network on the vector of loss parameters X. We sample the content images form ImageNet (Deng al. (009) and use 14 pointillism paintings as the style images. Further details are provided in the appendix.
We start by reporting quantitative results, As the problem has 5 loss parameters, it is not possible to visualize the frontier with respect all losses. Instead, we take two visualization approaches. First, we show the full loss achieved by different methods on a set of loss parameter values selected as follows: we start with the default parameter settings of Ghiasi al. (201)) A >x.2 = A
10-3, A = 10-2 A = 2, and then either increase or decrease one of these parameter by factor of 10 while keeping the rest fixed. This adds up to total of 10 parameter values: variants per each of the 5 parameters. To better understand the variability of the learned models due to the noisy training process we further trained each model four times. We provide the results in Figure 6al As a baseline, for each parameter configuration default parameters (* ("Single model baseline"). The YOTO model trained on all parameters closely matches the performance of fixed-weing weight models trained for each of the parameter configurations.
report the full loss value for the model trained with the
Next, we focus on just one aspect of the loss surface: the orfde-off between the content loss and the sum of style losses. In Figure 6b plot the frontier of these values corresponding different
8

---

Published as a conference paper at ICLR 2020
methods. Here add YOTO "specialist" model to the comparison, which has been trained with only 1 parameter, the content loss λę. Moreover, we report a baseline method that simply interpolates the images produced by two models trained with the extreme values of the content weight. Both YOTO models are close to the fixed-wight models, with the frontier of the "specialist" model closely tracing the fixed models. The interpolation baseline performs very poorly.
Finally, we show select qualitative results in Figure 7 where we compare fixed weight models to YOTO model trained all five parameters. We vary the content weight Ae and the third style weight Aa.3. which contribute most to the full loss. We can see that YOTO captures the effect of changing each of these weights and obtains results similar to those of the separately trained models.
5 CONCLUSION
We have presented an approach to training single deep model to fit parametric family of loss functions. This way, instead of training multiple models corresponding to different loss variants, we can train single model that provides solutions for all loss variants. We demonstrated the success- ful application of the method on three applications: 3-VAE, learned image compression, and fast style transfer. These initial results are promising, but several challenges remain. First, it would be interesting to devise provably effective strategies of selecting the loss parameter distribution, espe- cially for losses with multiple parameters. Second, further experiments with various architectures and conditioning methods can lead to an even smaller discrepancy between the separate models and our YOTO model. Third, the method could be used for many other applications, within or be- yond the domain of image generation. Fourth, one might use YOTO-trained models to initialize usual fixed-loss training or, vice-versa, use pre-trained fixed-loss model to initialize the training of YOTO. Finally, we believe that this work can also open a new family of theoretical questions centered around the analysis of problems that have to optimize a continuum of loss functions. We see all these directions as exciting avenues for future research.
ACKNOWLEDGEMENTS
We sincerely thank Johannes Ballé for the help with setting up the image compression experiments and insightful discussions. We also thank Sebastian Nowozin, Rodolphe Jenatton, Joan Puigcerver, Sylvain Gelly, and others in Google Brain for useful discussions and the support of the project.
REFERENCES
Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. In NeurIPS, 2017.
Nicola Asuni and Andrea Giachetti. TESTIMAGES; Large-scale archive for testing visual devices and basic image processing algorithms. In Andrea Giachetti (ed.), Smart Tools and Apps for Graphics, 2014.
Mohammad Babaeizadeh and Golnaz Ghiasi. Adjustable real-time style transfer. In ICLR Deep- GenStruct Workshop, 2018.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. In ICLR, 2018
C. Berge. Topological Spaces: Including Treatment of Multi-valued Functions, Vector Spaces and Convexity. 1963.
James Bergstra and Yoshua Bengio. Random search for 2012.
optimization. JMLR, 13,
Romain Brault, Alex Lambert, Zoltán Szabó, Maxime Sangnier, and Florence A'Allhé Buue Infinite task learning in rkhss. In A/STATS, 2019.

---

Published as a conference paper at ICLR 2020
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gihhibbcooo/e/pmm//////ss 2018.
Rich Caruana. Multitask learning. Mach. Learn., 28(1), 1997.
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End- to-end driving via conditional imitation learning. In International Conference Robotics and Automation (ICRA), 2018.
B.C. da Silva, G.D. Konidaris, and A.G. Barto. Learning parameterized skills. ICML, 2012.
Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. Modulating early visual processing by language. In NeurIPS, 2017.
M. P. Deisenroth, P. Englert, J. Peters, and D. Fox. Multi-task policy search for robotics. In ICRA, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy and Vladlen Koltun. Learning by predicting the future. In ICLR, 2017.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR, 2017.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 2018.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2414-2423, 2016.
Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. Explor- ing the structure of real-time, arbitrary neural artistic stylization network. In BMVC, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAB: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Xun Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In ICCV, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
Kodak. Kodak lossless true color image suite (PhotoCD PCD0992), 1993, URL http::/rOk.
us/ hts/graphics/koaak
lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In CVPR), 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. arXiv preprint
2018.
Matthew MacKay, Paul Vicol, Jonathan Lorraine, David Duvenaud, and Roger B. Grosse. Self- tuning networks: Bilevel optimization of hyperparameters using structured best-response func- tions. In ICLR, 2019.
10

---

Published as a conference paper at ICLR 2020
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to mul- tiple tasks by learning to mask weights. In ECCV, 2018.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual reasoning with general conditioning layer. In AAAI, 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NeurIPS, 2017.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. ArXiv, 2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In NeurIPS, 2015.
Alon Shoshan, Roey Mechrez, and Lihi Zelnik-Manor. Dynamic-net: Tuning the objective without re-training. ICCV, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy. Deep network interpolation for continuous imagery effect transition. In CVPR, 2019.
Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.

---

Published as a conference paper at ICLR 2020
APPENDIX
A ARCHITECTURE AND TRAINING DETAILS
A.1 BETA-VAE
We use convolutional encoder-decoder networks on both datasets. We use leaky rectified linear layer (Rel.U) non-linearity in all networks, except for units predicting stangard deviation of Gaussians where we use softmax non-linearity. On CIFAR-10 the network inputs have size 32 32 pixels, on Shapes3D - 64 x 64 pixels. We now describe the base network architectures, corresponding to "width x1.0".
On CIFAR-I0 both the encoder and the decoder consist of 3 blocks of 2 convolutional layers each, with the first layer in each block having stride 2. All kernels have spatial size 3 3,, and the number of channels is, respectively, 8, 8, 16, 16, 32, 32 (and increased proportionally for the "wider" models). The convolutional encoder is followed by fully connected layers with 256 and 512 units respectively, where the output of the latter layer is split in two 256-dimensional vectors encoding the mean and the variance of the approximate posterior. The decoder starts two fully connected layers with 256 and 512 units, the output of the latter layer is reshaped to a 4 4 32 feature map and further processed by a decoder that is symmetric to the encoder.
On S both the encoder and the decoder consist of convolutional layers each, all with stride 2. All kernels have spatial size 3 x 3. and the number of channels is. respectively, 8. 16, 32, 64 (and increased proportionally for the "wider" models). The convolutional encoder followed by fully connected layers with 256 and 20 units respectively, where the output of the latter layer is split in two 10-dimensional vectors encoding the mean and the variance of the approximate posterior. The decoder starts two fully connected layers with 256 and 1024 units, the output of the latter layer is reshaped to a x x 64 feature map and further processed by a decoder that symmetric to the encoder.
On Shapes3D we train all models for a total of 600, 000 mini- batch iterations, and we multiply the learning rate by 0,5 after 300, 000, 390, 000, 480, 000, and 570, 000 iterations. On CIFAR-10 use a proportionally twice longer schedule. We tuned the learning rates by sweeping over the values (5 10-5 10-1 2 10-a - 10-4, 8 10-1) and ended up using the learning rates 1 10-1 on CIFAR- 10 and 2 10-4 on Shapes3ID. We use mini-batches of 128 samples on CIFAR-10 10 and 64 samples on Shapes3D. We use weight decay of 10 in all models.
For YOTO models, we condition the last layer of each convolutional block. The conditioning MLP has one hidden layer with 256 units on Shapes3D and 512 units on CIFAR-10. At training time we sample the 8 parameter from log-normal distribution on the interval [0.125, 1024.] for Shapes3D and on the interval [0.125, 512.] for CIFAR-10.
A.2 IMAGE COMPRESSION
We train the model proposed by Ballé et al. 2018 and we refer the reader to that paper for architecture details. We train the models with the standard MSE reconstruction loss. All models have 192 channels in all layers, except for the "width 11.33" ones that have 256 channels. We use mini-batches of 8 samples. We train for 2 million mini-batch iterations with the learning rate of 10-4. We condition all convolutional layers in all networks using MLPs with hidden layer of 128 units. During training we sample the rate-distortion weight log-uniformly from the interval [1.2 10 2.6 10 10-
A.3 FAST STYLE TRANSFER
We build the work of Ghiasi et al. (0177), and use the exact same network architecture and training protocol. We optimize for million steps with a learning rate of 10 10-3 The ranges that we used were 10.1, 100] for the content parameters, [10- 10 '1 for the first three style parameters and [10-3, 1] for the fourth one - we always sampled from a bog-uniform distribution.
We condition the models by 1) sending the logarithms of the weights to MLP with single hidden layer of size 512 with ReLu activations and an output dimension of size, 2) concatenating these inputs the inception features of the style image, 3) for each layer modulation use MI.P
12

---

Published as a conference paper at ICLR 2020
YOTO
Fixed weight
oni.
[ 1
Ø 16
B 64
256
(a) Reconstruction
16
β 64
256
(b) Sampling
Figure 8: Qualitative VAE results on the CIFAR-10 10 dataset. Both for reconstruction and sampling, the YOTO results are qualitatively very similar to those of separate models trained for each value of the loss weight.
with single layer of dimension 256 and ReI.U activations, on which two affine maps are applied to compute the scale and shift.
ADDITIONAL RESULTS
B.1 BETA-VAE
Figure shows qualitative results for 3-VAE on CIFAR-I0. Same as for Shapes3ID, both recon- structions and samples from the YOTO model are qualitatively very similar to those generated with models trained separately for each weight.
We compare the proposed method to two baselines. The first one is naive baseline training VAE with fixed B, and computing the loss value this fixed model for all other /9 values ("Single model baseline"). We select the fixed so that it minimizes the average validation loss over all values. The second bascline interpolates between two models trained with fixed weights. Namely, train two beta- VAEs for the extreme values of beta, and interpolate both the latents and the reconstructions produced by these models. We then measure the KI and reconstruction losses for these interpolated values. Note that this interpolation baseline is not VAE any more: the latents and the reconstrutcions are averaged independently, so the interpolated reconstruction cannot be generated from the interpolated latent. Therefore, it is not clear how would one sample from such model, and we do not report these results in the main paper to avoid confusion.
The results are shown in Figure Surprisingly, on Shapes3D the baseline performs very close the fixed-model and YOTO frontiers, and sometimes even outperforms them. In contrast, on CIFAR-10 the baseline is much worse than the other methods. We believe the reason is that while averaging the logits for Bernoulli outputs on Shapes3D is quite meaningful, averaging the standard deviations of Gaussian outputs on CIFAR- 10 strongly affects Gaussian log likelihood.
Timing, We timed the training of a fixed-weight 3- and a YOTO model. To this end, we trained both models model to be only 8% slower than its fixed-weight counterpart: 4.64 vs 5,04 training steps per second with the "width x2" network architecture.
single CPU core over 200 mini-bate iterations. We found the YOTO
13

---

Published as a conference paper at ICLR 2020
3500
aRa Interpolation baseline, width xd.0
600
*9e
Single model baseline, width sd.0
3000
Poll difference
500
2500
400
300
1500
1000
-100
-500
-5.00 0
-4000
-500
-2000
-1000
1000
2000
-2
0
10
Reconstruction Josa
Log2 of beta
(a) CIFAR-10 frontier
(b) CIFAR-10 full loss
300
Modal
YOTO,
25
Interpolatoon baselnn, width XL.0
Single model basdiino, withi xd.0
Full loss difference
150
KL loss
1000
-50
-100
3400
3500
3600
3500
3,800
30:00
4000
Reconstruction
(c) 3DShapes frontier
(d) 3DShapes full loss
Figure 9: Quantitative comparison of the proposed method with baselines for beta- VAE on CIFAR- 10 (a,b), and Shapes3D (c,d). The proposed method outperforms the "single model" baseline by large model. The "interpolation" baseline is reported for completeness, but not a fair baseline, since it does not correspond VAE.
24
24
22
MSS-SSM PSNR
20
MS. SSIM PSNR
20
18
18
16
14
14
YOTO
12
YOTO
YOTO width x1.33
YOTO width x1.33
weight
10
Fixed weight
1.50
Bits per pixel
Bits pixel
(a) MS-SSIM on Tecnick.
(b) MS-S on Kodak.
Figure 10: Quantitative results of compression with the MS-SSIM metric on Tecnick (a) and Kodak (b).
14

---

Published as a conference paper at ICLR 2020
200
Modal
800
400
Weight per batch
Unttorm sampling
300
No log inputs
Pull loss. differmons
400
Ieput conditiosing
conditioning
200
XL boke
300
-100
-200
-5000
-1000
-3000
-2000
-1000
1000
2000
Reconstruction Joss
Log2 of beta
Frontier.
(b) Full losses.
Figure 11: Ablation study of the CIFAR- 10 3-VAE. The distribution of the loss weight used train- ing time has the largest impact on the performance. Moreover, conditioning the model by providing the loss weights as inputs to the model underperforms compared to more advanced conditioning methods. Other variants all perform close to the full method.
B.2 ABLATION STUDY
We now ablate different design choices made in our implementation of YOTO to check their im- portance. We perform this analysis the CIFAR-10 dataset, with the "width x4" architecture. The design decisions we study are: the weight distribution used during training (log-uniform or uniform), the method of sampling the loss weights (per sample or per mini-batch), the method of conditioning the network on the weights (FiLM, concatenating the conditioning vectors with the feature maps, or feeding the conditioning vectors to the network as an additional input), pre-proue pre-processing of the conditioning inputs (taking their logarithm or not).
The results are presented in Figure [1] Clearly, the choice the parameter distribution during train- ing is the most important choice, and log-uniform distribution performs much better than uniform. Other choices are of less importance, but more advanced conditioning schemes have an advantage over simply feeding the loss parameters as one of the network inputs.
B.3 STYLE TRANSFER
Figure shows additional qualitative results on style transfer for different content and style images.
C
THEORY
We start by proving Proposition from the main paper and then comment on the cases when the conditions of the proposition satisfied.
Proof of Proposition I. Denote the values minimized in optimization problems (1) and (2) by Vi(()
and (22()) respectively. From the definition of F* it follows that Vi (I((-A)) > (E (-,",,,,)))))))) A)) for any
À A and G C(% A). Therefore, V2(G) V2(F*), F* minimizes (2). Thus, for any G* that minimizes (2):
0 = Vz(G*) - V2(FF) EAN (V (F(FF ())]
(6)
Moreover, (.,A)) - (f > 0 for all E A. Expectation of non-negative function is 0 only if it equals almost surely, which means that (G V1(F* (.,)) a.s., q.e.d. O
The most restrictive condition of the proposition is the continuity of the function (+,-,)) as a func- tion of A. Indeed, generally arg max a multi-valued mapping that can be shown to be hemi- continuious using Berge's 's Maximum theorem (Berge 1963) under weak technical conditions. How- ever, there does not necessarily exist a continuous selection of this multi-valued mapping. Its ex- istence can be proven under very restrictive conditions, such as the uniqueness of the minimizer,
15

---

Published as a conference paper at ICLR 2020
Fixed YOTO
YOTO
Fixed
Fixed YOTO
Fixed YOTO
Fixed YOTO
(a) Varying the content coefficient A
Varying of the style coefficients A
Figure 12: Additional qualitative comparison of image stylization models on images from the val- idation set of ImageNet. The first row shows the results YOTO trained on all parameters, the second row of models trained independently per configuration. In both cases parameter value increases from left ot right. YOTO results are very similar those of independently trained models.
16

---

Published as a conference paper at ICLR 2020
which clearly do not hold for optimization problems encountered in deep learning. While empiri- cally the existence a continuous F seems natural, it remains to seen theoretical guarantees can be provided for realistically complex scenarios.
Another direction for extending Proposition 1 is explicitly incorporating function approximation. With function approximation, it would likely not be possible to prove that minimizers of the two problems coincide almost surely, but rather that they can be made arbitrarily close with a probability arbitrarily close 1. Overall, we believe our work raises range of interesting theoretical questions to be addressed in future research.
17


Reviews:
[{"review_id": "r1lkpJl6qS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe authors propose a methodology to train a single Deep Neural Network (DNN) that minimize a parametrized (weighted) sum of different losses. Since the model is itself conditioned by the weights, it allows to train a single model for all weights values, instead of retraining when the weights change.\n\nExperiments suggest that this methodology does not degrade the performances to much w.r.t. retraining on every weight changes.\n\nThe proposed conditioning of the layers is done via a reparametrization (FiLM, Perez et al. 2018) of the weights with a scale $\\sigma(\\lambda)$ and a bias $\\mu(\\lambda)$ where $\\mu$ and $\\sigma$ are MLP. This allows to condition the layer on $\\lambda$, while keeping the number of parameters low.\n\nNovelty\n----------\nThe idea of integrating a family of loss functions with model conditioning has also been proposed by Brault et al. [1], in the context of multi-task kernel learning. Hence I believe this work should be acknowledged.\n\nAs the product of kernels is the tensor product of their feature maps, it would suggest to condition the network's layers by taking the tensor product of the weights with respect to an MLP on $\\lambda$. This could be applied on each layers or simply on the last Fully Connected layer. Note however that it would drastically increase the number of parameters and hence not be a viable solution (or maybe with some channel pooling?).\n\nReferences:\n\n[1] Infinite Task Learning in RKHSs; Brault, Romain and Lambert, Alex and Szabo, Zoltan and Sangnier, Maxime and d'Alche-Buc, Florence; Proceedings of Machine Learning Research; 2019.\n\nQuality\n----------\nThe paper is self content and well written.\n\nExperiments are well detailed and seems to be reproducible. I would be a great addition to release the code in a public repository (with a link in the paper or appendices) if the paper is accepted.\n\nI would also suggest the following experiments:\n    * An experiment showing the time penalty induce by training the loss conditional model. The authors claims that training multiple separate models is inefficient compared to their proposed method. While it seems obvious, it deserve an experiment as one of the claim.\n    * The authors propose to sample one $\\lambda$ per SGD iteration. However it ma be useful to sample more of them. Especially when the set of $\\lambda$ is large (high dimensional)\n    * Possibly use a pre-trained model and only tune the $\\sigma(\\lambda)$, $\\mu(\\lambda)$ MLPs\n\nOverall my decision is weak accept, the paper lacks of novelty and the experiments could be more extensive.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1gGP5untH", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Due to the late rebuttal, I was not able to respond during discussion time.\n\nQ1, Q2) sound and look convincing.\nQ3) I still cannot find details on this in the paper. How the validation set was chosen? What was the size? The authors need to make all the experiments fully reproducible.\nQ4, Q5) ok\n\nMy concerns were sufficiently addressed in the current revision, and I will increase my score. However, the paper still feels close to a borderline, but probably, tending to \"accept\". \n\nAlso, I agree with Review #4, that also wondering about the application of the proposed method to hyperparameter search (Suggestion 1 in my review). Even if this would not be a sota in hyperparameter search, it feels like missing the opportunity to make the paper much stronger, by adding one more nice property to the proposed model.\n\n-----\n\nGenerative models often use a loss function that is a weighted sum of different terms, e.g., data-term and regularizer. Let's denote these weights as λ. The paper proposes a method for learning a single model that approximates the result produced by a generative model for a range of loss-term weights λ. The method uses the following mechanisms i) λ-conditioned layers ii) training with a stochastic loss function, that is induced by a (log-uniform) distribution over λ. The performance of the model is demonstrated on the following problems learning β-VAE, image compression, and style transfer. The models clearly demonstrate an ability to approximate problem solutions for a range of coefficients. The paper is clearly written. The experiments, however, need future discussion.\n\n1) The beta-VAE experiments (sec. 4.1)\n\nQ1. While models demonstrate a reasonable behavior on Shapes3d dataset. The samples and reconstructions on CIFAR10 (Figure 8) indicate that all models are not trained well. If this is the case, conclusions might be misleading, since the approximating output of undertrained models might be much simple comparing to well-trained ones. Authors may want to provide a comparison of the trained models with conventional VAEs (with β=1), the reference figures for CIFAR10 are provided, for example, in https://arxiv.org/abs/1606.04934.\n\nQ2. Wider YOTO seems to help a lot, but, what happens to the baseline models of increased size?\n\n\"We select the fixed β so that it minimizes the average loss over all β values.\"\n\nQ3. Was it done directly on a test set, or were validation-data used?\n\n2) Image compression (sec. 4.3)\n\n\"Finally, a wider model trained with a larger batch size (“YOTO wider batch16”) closely follows the fixed weight models in the high compression regime and outperforms them in the high quality regime.\" (Figure 5)\n\nQ4. How is this compared to the baseline with batch16?\n\nQ5. Authors also may want to provide std for provided metrics. The difference does not look statistically significant. \n\nSuggestion 1: It might also be interesting to see if we can use this technique to perform a hyperparameter search. Train the model, select one the best performing set of hyperparameters, and then train models with this best value.\n\nOverall, the paper proposes an interesting technique, that surprisingly, can work for a range of hyperparameters, and potentially have a high practical impact. However, the empirical evaluation is half-baked, specifically has certain methodological drawbacks e.g., perhaps undertrained beta-VAE model, absence of standard deviations while comparing (close) numerical results, and comparing models with different optimization parameters -- the performance difference might be due to optimization. \n\nI recommend to reject the paper, however, I will appreciate discussions with authors and other reviewers, and will consider changing my score in case of reasonable argumentation.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJegbBKCFr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The problem tackled by the paper is related to the sensitivity of deep learning models to hyperparameters. While most of the hyperparameters correspond to the choice of architecture and optimization scheme, some influence the loss function. This paper assumes that the loss function consists of multiple weighted terms and proposes the method of finding the optimal neural network for each set of parameters by only training it once.\n\nThe proposed method consists of two aspects: the conditioning of the neural network and the sampling of the loss functions' weights. Feature-wise Linear Modulation is used for conditioning and log-uniform distribution -- for sampling.\n\nMy decision is a weak accept.\n\nIt is not clear to me if the choice of performance metrics is correct. In many practical scenarios, we would prefer a single network that performs best under a quality metric of choice (for example, perceptual image quality) to an ensemble of networks that all are good at minimizing their respective loss functions. Therefore, the main performance metric should be the following: how much computation is required to achieve the desired performance with respect to a chosen test metric.\n\nMoreover, it might be obvious that the proposed method would be the best w.r.t. this metric, compared to other hyperparameters optimization methods, since it only requires a neural network to be trained once with little computational overhead on top. But then its performance falls short of the \"fixed weight\" scenario, where a neural network is trained on a fixed loss function and requires to raise the complexity of the network to achieve similar performance.\n\nTherefore, obtaining a neural network that would match the desired performance in the test time and would have a similar computational complexity requires more than \"only training once\", with more components, such as distillation, required to be built on top of the proposed method. The title of the paper is, therefore, slightly misleading, considering its contents.\n\nAlso, it is slightly disappointing that the practical implementation of the method does not allow a more fine-grained sampling of weights, with uniform weights sampling shown to be degrading the performance. This implies that the method would have to be either applied multiple times, each time searching for a more fine-grained approximation for the best hyperparameters, or achieve a suboptimal solution.\n\nBelow are other minor points to improve that did not affect the decision:\n-- no ImageNet experiments for VAE\n-- make plots more readable (maybe by using log-scale)\n-- some images are missing from fig. 7 comparison", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes and validates a simple idea of training a neural network for a parametric family of losses, using a popular AdaIN mechanism.
Following the rebuttal and the revision, all three reviewers recommend acceptance (though weakly). There is a valid concern about the overlap with an ICLR19-workshop paper with essentially the same idea, however the submission is broader in scope and validates the idea on several applications.
Decision: Accept (Poster)