Image Generation using Continuous Filter Atoms
Authors: Ze Wang
Seunghyun Hwang
Zichen Miao
Qiang Qiu
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Conditional image generation
neural ordinary differential equations
convolutional filter decomposition
continuous image generation

Abstract: In this paper, we model the subspace of convolutional filters with a neural ordinary differential equation (ODE) to enable gradual changes in generated images. Decomposing convolutional filters over a set of filter atoms allows efficiently modeling and sampling from a subspace of high-dimensional filters. By further modeling filters atoms with a neural ODE, we show both empirically and theoretically that such introduced continuity can be propagated to the generated images, and thus achieves gradually evolved image generation. We support the proposed framework of image generation with continuous filter atoms using various experiments, including image-to-image translation and image generation conditioned on continuous labels. Without auxiliary network components and heavy supervision, the proposed continuous filter atoms allow us to easily manipulate the gradual change of generated images by controlling integration intervals of neural ordinary differential equation. This research sheds the light on using the subspace of network parameters to navigate the diverse appearance of image generation.

Image Generation using Continuous Filter Atoms
Ze Wang" Seunghyun Hwang" Zichen Miao Qiang Qiu
Purdue University (zewang, hwang229, miaoz, qqiu)epurdue. edu
Abstract
In this paper, model the subspace of convolutional filters with neural ordinary differential equation (ODE) to enable gradual changes in generated images. Decom- posing convolutional filters over set of filter atoms allows efficiently modeling and sampling from subspace of high-dimensional filters. By further modeling filters atoms with neural ODE, we show both empirically and theoretically that such introduced continuity can be propagated to the generated images, and thus achieves gradually evolved image generation. We support the proposed framework of image generation with continuous filter atoms using various experiments, includ- ing image-to-image translation and image generation conditioned on continuous labels. Without auxiliary network components and heavy supervision, the proposed continuous filter atoms allow to easily manipulate the gradual change of gen- erated images by controlling integration intervals of neural ordinary differential equation. This research sheds the light on using the subspace of network parameters to navigate the diverse appearance of image generation.
Introduction
Conditional image generation has been widely studied in recent years due to its numerous applications including image segmentation I12 [. style transfer [57 20 image inpainting [33 52 image
super-resolution [54] 44 image registration M. and image synthesis [31] Despite extensive research and applications in these fields, limited progress has been made on conditional image generation using continuous or closely spaced labels due to the difficulties pointed out in M e.g.. the absence of real images for some labels. It is shown in [ that the standard training with empirical risk minimization does not apply well to continuous labels. Moreover, it remains challenging encourage the generation diversity, especially without heavy supervision, while maintaining output fidelity. Previously, generation diversity is mainly encouraged using explicit regularization terms to convey diversity with additional latent codes 26 34 However, explicit regularizations inevitably introduce often cause either poor diversity noticeable sacrifices terms of generation quality and the correspondence to input conditions.
additional hyperparameters, and sub-optimal hyperparameters
On the other hand, achieving generation diversity by modeling the deep network parameter space has attracted limited attention [47]. This direction is mainly challenged by the prohibitive cost on both modeling of and sampling from the very nigh-dimensional space of convolutional filters in modern image generative networks. Motivated by the observation that a convolutional filter can be well approximated by linear combination of low-dimensional filter atoms, BasisGAN 47 shows in image generation that, the high-dimensional filter space each layer can be well approximated by low-dimensional filter subspace, which significantly reduces the cost of both parameter modeling and sampling. In this way, each sampling of the modeled parameter subspace results in one deterministic transformation from the input condition to the desired target domain, and the diverse outputs are achieved by sampling multiple times. More importantly, is empirically shown in 47 that without
"Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

any explicit regularizations, such parameter subspace modeling can work as plug-and and-play module to convert deterministic conditional image generation network to produce diverse images with appealing stochasticity, without any auxiliary network components or regularization terms.
However, we observe that the filter subspace modeling in - suffers from several major limitations: First, this method is sensitive to network configurations of the basis generator, as minor changes to the parameters can highly likely to trigger mode collapse in the parameter distribution, which results in point estimation to the subspace of parameter and removes diversity in the output images. Furthermore, BasisGAN is incapable of modeling a continuous space, so that gradual changes to the generated parameters cannot be obtained by simply interpolating the latent codes.
In this paper, we adopt subspace view to convolutional filters by performing atom- tlom-coefficients Then we further model the filter subspace using a neural able sample from this continuous subspace
decomposition, as in 65 ordinary differential equation (ODE), that a series of filter atoms at arbitrarily fine resolution. The low-dimensional filter subspace allows extremely efficient learning and modeling using ODE. We show both empirically and theoretically that, continuous transition in the filter subspace can be propagated naturally to generated images, and thus produce visually appealing images with gradually varying appearance. More importantly, we show that the continuity introduced to the filter subspace and the induced generation smoothness allow continuous manipulation of generated appearance with only discrete samples as supervision. which is task that is used to be considered intractable by training with standard empirical risk minimization 8 These appealing properties of the proposed method enable various significant applications through only standard training, without requiring any auxiliary network components, additional regularization terms, or heavy supervision.
46 4
We perform various experiments on conditional image generation, with conditions being in the form of images, labels, or both. We list here several example applications that are enabled by the proposed image generation using continuous filter atoms from an ODE atom generator:
Continuous image synthesis that covers a wide range of gradually varying appearance with high fidelity and accurate correspondence to the input condition.
Sequential image synthesis with explicitly specified starting and ending points to allow flexible appearance manipulation without heavy supervision.
Interpolation of generated image appearance arbitrarily fine resolution.
An effective approach to generating images conditioned on continuous labels.
2
Related Work
Conditional image generation. Image generation is widely studied in the machine learning com- munity, from restricted Boltzmann machines [39 to variational autoencoders 121;: in particular variants with conditions 30
show promising results. Empowered by GANs M. conditional 47 have demonstrated superior performance on image generation with the restriction of condition. Recently, conditional image generation with conditions on continuous labels has attracted attention, and new training schemes [ are proposed to improve continuous conditional image generation. From the perspective of the dynamic of parameter subspace, our method provides a simple yet effective solution.
40 42
generative adversarial networks (cGANs) 17
33
48
[
50
57
26
Continuous image translation. Many works have discussed improving the diversity of generated images in image-to-image translation tasks 27
34
3
26
45
15
53 [
However, continuous
image translation that aims to generate intermediate images between the source and the target images 2n
49 (0O have manipulated the attribute vectors either by interpolation or linear transformations to continuously control the latent space. 1I 251 employ multiple generative networks in the middle of the pathway from source to target image to generate intermediate results. (6 (3) disentangle the latent space as content and style codes and interpolate the styles, in order to produce intermediate samples. However, all the aforementioned methods rather need to import auxiliary neural networks or carry extra burdens on model training by supervisions. Moreover, synthesizing intermediate images
is not yet widely studied. Some previous works [55 51
introducing additional regularization without relying on interpolation of either domain labels, or disentangled representations, which can discourage the stochasticity of generated images, is still considered big challenge.

---

3 Continuous Filter Atoms
In this section, we start with brief review of neural ODEs and atom-coefficient filter decomposition, which serve as key ingredients in the proposed image generation approach. We then present in detail that, with the effectiveness being theoretically justified, how the proposed continuous atom generation using neural ODEs achieves gradual image generation.
3.1 Neural ODEs
Neural ordinary differential equations (ODEs) are introduced in 4 approach to interpreting the neural network as system of ODEs, where each ODE represents an underlying dynamic of the hidden elements. Especially, neural ODEs are frequently adopted to enforce continuous dynamics in latent representations, for example, 19 32 propose to use neural ODEs to smoothly connect the latent space of video frames; adopts ODE-RNNS to model time-continuous hidden dynamics to handle irregularly sampled time-series data.
Neural ODEs model a latent state z(tx) as z(ta) z(to) + Shh f(z(t), 0) dt, where dx(t)
dt
f(x(1),4; 4; 0) is modeled as a neural network parametrized by 0. Then, the latent state at an arbitrary point 4g can be obtained
ODESolve(z(to), f. (to. t,),
(I)
where to and t, denote the start and the end of the integral interval, respectively. The simplest method for approximating the solutions of the ordinary differential equations with a given initial value is the Euler's method, which is first-order integrator with fixed step size. In practice, families of Runge-Kutta methods, e.g., the Midpoint method and fourth-order Runge-Kutte [22]. are preferred
due to their superior convergence and stability. Note that all the experiments in this paper follow the
implementation in [4] for Runge- Kunge-Kutta of fifth-order of Dormand Seeddd-rreee-eeee Shamp--h with adaptive step size (
For any given integral interval (tz,tz) where i, E {O...T), ODESolver always outputs a unique solution for the integral of the continuous dynamics, long the ordinary equation f is uniformly Lipschitz continuous in z and t 7 Parametrizing f as a neural network equipped with Lipschitz continuous non-linear functions will meet the requirements. Thus, in paper, modeling filter subspace using neural ODEs guarantees continuity as we will theoretically justify later.
3.2 Convolutional Filter Decomposition
Directly modeling the space of high-dimensional convolutional filters in an image generation network practically prohibitive in terms of both computation and parameter scale 47]. Inspired by the observation that a convolutional filter can be well-approximated as linear combination of filter bases [35, as shown in 47 a subspace view to the convolutional filters can be adopted in image generation network by decomposing each convolutional filter as a linear combination of oow-dimensional filter atoms. Specifically, given an [ I-sized convolutional filter F exx xtxl with c' input and output channels, respectively, we can decompose the filter over a set of m atoms D as = AD, where D E |gxxix and A E Ieee' xm are the filter subspace coefficients. As we will show later, by modeling the continuity of a low-dimensional filter subspace using neural ordinary differential equations, we achieve efficient training and modeling of gradual appearance changes in conditional image generation.
3.3 Continuous Atom Generation for Smooth Appearance Modeling
In this paper, we aim for modeling gradual changes in generated images with respect to continuous input conditions. Instead of seeking new ways of training, provide a unique view by modeling the transformation from an input condition to the corresponding output with parameters instantiated from an underlying continuous space, which subsequently leads generated images with gradual changes.
We adopt the aforementioned atom-coefficient decomposition, which permits low computational modeling of filter subspace as also validated in [47]. However, modeling filter subspace with standard neural networks can still suffer from mode collapse often caused by sub-optimal network
configurations, and the continuity is also difficult to be guaranteed by standard training as discussed

---

Illustration of the proposed image generation using continuous filter atoms. Note that only single convolu- tional layer is shown here for easier view- ing. We decompose each convolutional fil- ter as F = AD, where the coefficients A R xm remain learned determinis- tically, and the filter atoms D E F x are sampled from a continuous atom space modeled with neural ODE. As shown in the red box, continuously sampled atoms can produce images with gradual appearance changes.
Figure 1:
Governd leage
Revettadte Films
Abxa Colfiints
Aue Spac.
in [8] We therefore adopt a neural ODE based filter atom generator with the continuity property discussed in Section 3.1 and show theoretically that such atom continuity can be propagated to the
deep network outputs to achieve conditional image generation with gradual changes.
Specifically, we decompose the i-th layer filters F into respective atoms D' and coefficients A as F' = A'D'. Over layers of filters {F' we jointly model the corresponding atoms {D'} Ii using atom generation. We assume that all instantiations of filter atoms reside in a continuous space parametrized as (t) where is a time variable, i.e., the generated atoms at the time point can be expressed as (t). In the limit of small step c, the local dynamic within this atom space then be
(t) + E dD (t((t) (t)/dt. Therefore, given an initial state of atoms within this space denoted D (to), and modeling the derivative of d'D (t) dtt using neural network f(.,;))
expressed as D (t + e) = (t))
dD(t) (I))),,;))
any samples within this atom space can then be obtained by an integral of the function values of from to to the desired ending point. The described dynamic leads naturally to a neural ordinary differential equation, where the instantiation at a time point f,, 8 >> 0, in this continuous dynamic can be obtained through a continuous atom generator by
((1.) = (()))) D(1))
| [(D(I),40)d..
(3)
Note that the initial state ((t)) here are free parameters to be optimized in the training jointly with 0 that parametrizes f. Starting from the initial state D (ft)). an instantiation of filter atoms Dit,)) is defined to be a unique solution to the ODE initial value problem at some condition lx. by Picard's existence theorem. Consequently, as l varies, different instantiations of atoms are generated from this modeled continuous filter subspace, and our corresponding atom generator equipped with ODESolver learns the underlying continuous dynamics of the filter subspace.
Given the subspace of filters modeled as an underlying continuous dynamic through neural ODE, we prove the Lipschitz-type continuity transducted from atoms to output features in the i-th layer. For simplicity, let c = = 1, and the argument extends. Given an input image X(u) (u E U, C Z?), define the local input norm ||æ]|,,NN == <Ew'E, z(u - 22)2)1/2 1/2 and the convolution (I, W) = ΣWEN, I(u u') w')), where Na c. U is a local Euclidean grid centered at u. The i-th layer function Fi xi-1 + X' can be expressed as,
=
of Σ A + +b²),
(4)
Am
where D denotes the k-th atom in the i-th layer, A is the corresponded k-th coefficient, and & denotes the bias at the i-th layer.
Theorem 1. Suppose Dj and D22 are two continuously generated atoms, (i.e., 3M > 0. D
Dall2 < M t2 J. and assume the activation function a is non-expansive which holds for ReLU, then F is continuous in D',
| - X1 s (11A")12A) M [[(
- with sup well
| 12.N.:
(5)
4

---

207
30F
2F'
F(
F(
F(ts)
D(fu-al
P(G]]
D(-
10.1
D
Eo
1g
Figure 3: Visualization of interpolated samples the rotating chair example. As the neural ODE-based atom generator learns smooth trajectory of rotation, image generation at any desired degrees is now al- lowed. The images generated the blue dots interpolation results for unseen rotation angles.
Figure 2: An example of generating rotat- ing chairs with the proposed continuous filter atoms. Each tinct angle along the roll axis, and is given as an explicit condition the atom generation.
the figure represents a dis-
in which X = Di F D2; A²) outputs correspond to two atoms.
Proof. The proof is provided in the Appendix Section
Notably, modeling the atom subspace of filters offers significant advantages on efficiency. Under the rule of neural ODE, the dimensionality of (()) cannot be varied, meaning the dimension of the input and the output of the ODESolver remain strictly consistent. This additional restriction makes modeling the space of full-size filters practically infeasible due to the computation and memory constraints, Mcanwhile, by modeling the filter subspace instead, we significantly reduce the cost of learning the corresponding neural ODE. Typically, atoms across layers introduce few hundred parameters only and significantly simplify the training and inference of atom generator T.
3.4 Image Generation with Continuous Labels
Depending on the specific applications of the proposed continuous atom generation as we will show next, the ending points of the ODE integral can be: 1) explicitly given to strictly control the behavior of continuous dynamics, 2) stochastically sampled from prior distribution, e.g., Uniform or Gaussian, to enhance the diversity of the generated images.
Explicit continuous conditions. Compared the previous conditional GAN methods where label information is vectorized or one-hot encoded, the proposed method permits minimal preprocessing the continuous labels, which fed into the continuous atom generator simply in the forms of raw floating numbers preserving the continuity of labels. Moreover, unlike [2S which can only take finite discretized conditions, such as categorical class information, continuous filter atoms do not hold any such restrictions. Therefore, by sampling filter atoms from the subspace of convolutional filters modeled by neural ODE, with the desired ending points integral specified by the continuous labels, the continuous filter atoms can be evaluated with any values within valid range of the regression labels. To illustrate this, we present an interpolation example Figure 3 and show that, with the underlying dynamics of the rotated chair images captured by continuous atom generator trained on limited angels only, the proposed method allows interpolation to the modeled dynamic arbitrarily fine resolution, and generates rotated chairs any angels that are not exposed to the model during training.
Stochastic continuous conditions. The stochastic modeling of integration interval permits gen- erated images with diverse appearances, and gradual transitions among the generated images. As we will show in Section 4.1 while the proposed method cannot directly enforce perceptual diverse samples as in 47 , the sampled sequences with gradual appearance changes jointly offer decent diversity while maintaining outstanding fidelity and correspondence to the input conditions.
3.5 Conditional Image Generation with Continuous Atoms
By simply plugging in the proposed continuous atom generator to off-the-shelf GANs, we show next that variants of conditional generative adversarial models (cGAN) [ 28 with continuity can
5

---

be easily obtained. The training does not involve any auxiliary network components or additional regularizations.
Conditional on images. Image generation conditional on images, also known as image-to-image
translation 17. aims generating a target image of a certain domain with high fidelity and desirable while maintaining correspondence w.r.t. an imput image condition. We introduce image generation conditional on input images with the output space modeled as a smooth sequence through continuous atom filters, thus sampling the continuous filter atoms allows generating images with gradual appearance changes without losing the correspondence to the conditions. Formally, representing the condition images as x, which are usually samples from an empirical distribution p(x), we aim at modeling a conditional distribution p(y|x). By plugging in the continuous atom generator [t:: Do. f), the conditional distribution is learned to be modeled by generator D. where contains all remaining parameters in the generator, e.g., the coefficients A in each decomposed convolutional layer. And the input to the discriminator ID is now trained to evaluate the compatibility of a pair of images, one from the source domain ~ p(x), the other one comes from either true target distribution y p(y)x), or the generated approximation & D. (x: T(t; Po,0).
diversity 47 2 E
The training can be realized
min max L(D, G) = xx (D y)]]+ D
(6)
Ex--p(() [og((1 - D(x, Go.6DD. T(; 8))))
where (((6o; denotes uniform distribution with the range specified by in and t-1-
Conditional on labels. As pointed out S cGANs are good conditional image generation with discrete labels but can struggle at continuous labels. In our method, without leveraging any customized training schemes as in [Н. we show that the continuous image generation with standard empirical risk minimization can be significantly improved by placing assumption on the gradual appearance change across continuous labels, and trained with continuous atom generation given continuous labels as the inputs. Formally, define the scalar regression label sampled continuously from x ~ p(z), and plug in the continuous atom generator to standard cGAN to model the desired target distribution p(y(z). The image generation is then achieved G. D.D. (z; T(ax; Do. 0)), with an input vector z ~ -N I) carrying randomness as in standard cGANs. Comparing to ( the input to the continuous atom generator is replaced as ox, where is a predefined scalan that linearly scales the input continuous labels. And we use a standard discriminator with each input consists of an image and label. The training can be realized
min max (DD, G) =E G
y))}+ Ex aerp Dzx Go..DD (a T(ar; Do. 0))))
Conditional on both images and labels. For image-to taage-to-image translation tasks, a few works have explored imposing conditions on both images and labels to achieve scalable image translation while introducing diversity and manipulations. Specifically, 455 and (5] impose additional domain attributes navigate the image translation. However, the domain attributes they condition are all categorical labels, and none of those methods can adopt continuous or raw format (e.g, float numbers) labels to capture the gradual changes in the desired target image space. Our method permits using the continuous labels fed into the atom generator to explicitly control sampling in the continuous filter subspace, while maintaining consistent input image conditions. Therefore, the proposed method is able to achieve image generation with gradual appearance changes w.r.t. the input continuous label conditions, and preserve strong correspondence to the source image. To the best of our knowledge, our method is the first work to present gradually changing image-to-ing tomiee translation results conditioned on both images and continuous labels. Formally, having both x p(x) and p(x). which refer to the input image and the scalar label, respectively, we employ our continuous atom generator, T(ox; Do, A), to get the desired target distribution p(y|.r, x). We then formulate our image generator as Go xx; T ((a;; Do. f)). The training can be realized as
min max L(D, G) -xxxp(xx) |0
(8)
E (oc ID G....Do (x; ((ax; Do, A))))

---

Input
Ground truth
Generated samples with gradual changes
Figure 4: Continuous image-to-image translation. The network is trained without any auxiliary loss functions or regularizations, From top to bottom, the image to image translation tasks are: edges + handbags, edges- + shoes, nights days, labels - facades, and maps satellite.
Table 1: Quantitative results on image generation conditional on images only.
Sabels->Facedee
Map-lSatellitc
Night-Day
Edge- LPIPS
Edge->Handbag
Methods
LPIPS
FID
LPIPS
FID
LPIPS f FID
FID
LPIPS
FID
BicycleGAN
0.1413
98.85
0.1150
145.78
0.103
120.63
0.139
72.49
0.184
96.28
MSGAN
0,1894
92.84
0.2189
152.43
0,176
107.90
0.167
60.28
0.228
89.96
BasisGAN
0.2648
88.70
0.2417
35.54
0.184
102.56
0.242
64.17
0.350
88.76
Ours
0,2712
87.75
0.1803
98.72
0.264
106.95
0.237
57.29
0.276
38.07
Experiments
In this section, we present results and comparisons in two experimental settings. We start with continuous image-to-image translation tasks where we show that by directly plugging in the proposed continuous atom generator, we can either condition the generated images solely on images, or on both images and labels, where each of the cases refers to 6 and (8). respectively. In both cases, we quantitatively and qualitatively show that our method achieves continuous image generation with high fidelity and diversity. Then, we proceed to image generation tasks conditional on continuous labels following 7
We compare the proposed method with [8] and [28] on the face generation experiments. Implementation details are in Appendix Section B More qualitative results are in
Appendix Section C
4.1 Continuous Image-to-image Translation
The introduced continuous atom generation can be naturally applied to the image-to-image translation experiments (47 as a simple plug-and-play module. Specifically, in image-to-image translation tasks with paired samples, we adopt ResNet [K3 based generator network following the implementation of Pix2Pix [7]. All the convolutional filters the ResNet blocks are decomposed into atom-coefficients where an individual atom of each block is generated continuously with neural ODE. Note that single neural ODE is utilized throughout the entire network, enforcing the network to be governed by single continuous dynamic. These settings are also applied in unpaired to-gag translation tasks, where we replace our baseline model from the Pix2Pix to ResNet-based CycleGAN (7).
Continuous image-to-image image translation conditional on images. We present results on various datasets including edges + handbags, edges + shoes, maps + satellite, night -+ days, and labels - facades. Compared to previous methods working on the same datasets, our method does not focus on generating images with strong diversity. Instead, we show that by plugging in the introduced continuous filter atoms, images with smooth intermediate transitions can be obtained by directly
7

---

Table 2: Quantitative results on image generation conditional on both images and labels under the configuration of image-to toe-tto--mmaa image translation tasks with paired inputs. Diversity and fidelity are measured using LPIPS and FID, respectively. As Pix2Pix deterministic model, LPIPS scores for Pix2Pix are all marked as ' and all the scores resulted in zero.
Datasets
RC49
RCA20
RA20
RL20
RT20
Methods
Pix2Pix
Ours
Pix2Pix
Ours
Pix2Pix
Qurs
Pix2Pix
Ours
Pix2Pix
Ours
LPIPS
0.1127
0.0931
0.068
0.1089
0.1213
FID 4
251.49
195.01
218.88
85.06
165.01
73.82
227.04
124.83
206.49
70.92
10
20
30"
40"
500
60"
70
S0P
90"
Input
Gradually rotating samples
Figure 5: Continuous soaagetoo-image translation conditional on both images and labels with paired samples. From top to bottom, the trained datasets are: RA-20, RCA-20, RL-20, and RT-20. The generated images of all datasets rotate from degree 0.1 to 89.9 at the interval of 5 degree, along the yaw axis. All models are trained with Pix2Pix by plugging in our continuous atom generator.
training the network without any additional supervision and regularizations. As shown qualitatively Figure f and quantitatively in Table these smooth transitions in the generated appearance can naturally lead to satisfying diversity without compromising the quality of the generated images and the correspondence to the input conditions.
Continuous image-to-image translation conditional on regression labels. We present results of image-to-image translation tasks conditional on regression labels on variety of datasets and experimental settings. Following (). the generated images are conditional on explicitly given regression labels, while keeping correspondence to the input image. We start with experiments on image-to- toage-to-imaae translation on paired samples, and then proceed to the unpaired samples.
Paired samples. In this experiment, we plug in our continuous filter atoms to the Pix2Pix 17 model and conduct continuous conditional image-to-image translation tasks on five different synthetic datasets, RC-49 [ RA-20, RCA-20 RT-20, RL-20, where objects in each dataset are rendered to rotate between 0 and 360" with 0. interval. Since there are very few datasets of 3-D objects with continuous labels, we created new datasets RA-20, RCA-20, RT-20, RL-20, by rendering 'Airplanes' 'Cars', 'Tables', 'Laptops' categories of ShapeNet, respectively, following the manuals of RC-49, We add further descriptions of datasets in the Appendix Section B.I During training, we only use samples with rotation degrees below 90", and fix the 0 image as the source image with varying target images. We adopt the gap of rotation degree between the source and the target image as an external condition for atom generation. We show in the Figure 5 that the generated images rotate smoothly according to the given rotation degrees. Especially for the results of RC-49, although we have organized the training data in 5 interval, it is noteworthy that our method generated interpolation results at the interval of 2.5 degree in the Figure 3 Table 2 also demonstrates that the generated images of our proposed method hold higher level of fidelity than the baseline.
Unpaired samples. For translation tasks on unpaired datasets, we present results on UTKFace [56] AiceringAngle cells-200, and Waymo [4]] dataset. Other than the Waymo dataset 41 all the datasets are composed of images with continuous labels, e.g., the age for UTCea the angle of the steering wheel of a car for the Steering/Agle, and the number of cells for Cells-200, Waymo 41 dataset has four different time labels for each image, which are dawn, daytime, dusk, and night. We add more detailed descriptions results of aging face experiments on the CACD2000 3 at Appendix Section In the unpaired amage-to-image translation tasks, using continuous labels can lead to confusion training, as the
the datasets in the Appendix Section B.T with additional

---

Table Quantitative results on conditional image generation when image and label were both given as input conditions under the configuration of unpaired image-to-image translation task. We present and Cel1200. Diversity and fidelity are measured using LPIPS and FID, respectively. The LPIPS scores for the CycleGAN are marked as '-',
results on UTKFaae [56]. CACD2000(3), Steering
as the all LPIPS scores resulted in zero, with no meaningful diversity observed.
Datasets
Cell200
SteeringAngle
UTKFace
CACD2000
Methods
CycleGAN
Ours
CycleGAN
Ours
CycleGAN
Ours
CycleGAN
Ours
LPIPS f
0.1355
0.0544
0.0052
0.0018
FID 4
30.44
22.22
73.03
52.39
47.24
41.55
58.43
57.15
Input
Generated samples with gradual changes
Figure 6: Continuous imagc-to-image translation conditional on both image and labels with unpaired samples. From top to bottom, the datasets are: UTKFace (young - old), SteeringAngle (steering angle at -80 -> steering angle at 809), Cell200 (one cell - two-hundred cells), and Waymo (dawn -+ day). All models are trained with CycleGAN by plugging in our continuous atom generator.
tendency of continuous labels can be violated within unpaired samples. For instance, in the human face translation experiment, due to different levels of aging among people, it is more natural to have the age group as a class, rather than treating each age as a class. Therefore, the target domain is divided into a specific interval of labels, so that each sub-group of the target domain can be assigned distinct labels instead. The distinct labels are then directly used as a condition for modeling the continuous subspace of atoms as in 8 Despite the use of distinct labels, we highlight that it is still possible to capture continuous variation between the generated images via interpolation in the continuous filter subspace. During the training, images with certain conditions,c.gg teenagers for the UTKFace, -650 -80 for SteeringAngle, one number of cell for Cclls-200, and dawn time for Waymo, are fixed as source domain input. Figure 3 shows the gradual changing between the condition of the target domain. While keeping a certain amount of consistency within the input image, smooth transitions between generated images are observed. In particular, the result of the Waymo dataset at the fourth row of Figure D well captures progressive changes over time in the generated images from dawn to daytime, through interpolation in filter subspace. Table also highlights our proposed method's superior fidelity over the baseline model. Additional training details are presented in the Appendix Section B
4.2 Conditional Image Generation with Regression Labels
We present face image generation with the condition only on regression label specifying the age. The training objective follows 7 strictly. We construct a conditional image generation network by replacing the Adaptive Instance Normalization (AdaIN) 14] layers with standard batch normalization layers, and replace convolutional filters with the proposed continuous atom generation networks. As shown in the qualitative results in Figure 7 by fixing the input latent code and changing only

---

60
Figure 7: Continuous image generation conditional on regression labels. From left to right, each column are images generated with the same atoms sampled from the continuous atom space and model the appearance of face at particular age from 0 to 60. By fixing in 7. each row shows the gradual appearance changes w.r.t. continuously sampled atoms.
Table 4: Quantitative results conditional image generation when the label is given an input condition. We present results on UTKFace [56 with age as regression labels.
Dataset
Methods
Intra-FIh FID +
NIQE 4
Diversity f
Label Score +
cGAN
4.516
2.315
0.254
11.087
UTKFace
CcGAN
0.425
1.725
1.298
7.452
Ours
0.432
1.749
1.321
7.399
the condition scalar æ, we are able to generate sequences of images with gradual age changes. And switching z allows changing the identity of the face sequence. Comparable quantitative results in Table Table[iith with CcGAN 8 further demonstrate the effectiveness.
5
Conclusion
In this paper, we presented both theoretically and empirically that, continuous dynamics of con- volutional filters can be effectively modeled in the filter subspace by neural ordinary differential equation, to subsequently achieve conditional image generation with gradual appearance changes. The introduced continuous filter atom generation enables continuous image generation conditional on images, labels, or both. We demonstrated its effectiveness using both superior quantitative and qualitative results.
6 Acknowledgements
This work was supported by the DARPA TAMI program under No. HR00L 12190038.
References
[1] Moab Arar, Yiftach Ginger, Dov Danon, Amit Bermano, and Daniel Cohen- Or. Unsupervised multi- modal image registration via geometry preserving longe-to--mmae translation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 13410-1322 2020.
[2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 4rXiv:15/2.03012 2015.
[3] Bor-Chun Chen, Chu-Song Chen, and Winston H Hsu. Cross-age reference coding for age-invariant face recognition and retrieval. In European Conference Computer Vision, pages 768-783. Springer, 2014.
[4] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Neural Information Processing Systems, 2018
10

---

[5] Yunjey Choi, Miaje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain miimge translation. In IEEE Conference on Computer Vixion and Puttern Recognition, pages 8789-8797, 2018.
[6] Yunjey Choi. Youngjung Uh. Jaejun Yoo. and Jung-Woo Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In IEEE Conference on Computer Vision and Pattern Recognition, pages 8188-8197. 2020.
[7] Earl Coddington and Norman Levinson. Theory of ordinary differential equations. Tata McGraw Hilll Education, 1955.
[8] Xin Ding, Yongwei Wang, Zuheng Xu, William J Welch, and Jane Wang. Cegan: Continuous con- ditional generative adversarial networks for image generation. International Conference of Learning Representations, 20210.
[9] John R Dormand and Peter Prince. family of emhedded runge-kutta formulae. Journal of computational and applied mathematics, 6( (69)229, 1980.
[10] Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. Dlow: Domain flow adaptation and generalization. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2477-2486, 2019.
[11] lan Goodfellow, Jean Pouget-Abadie Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Advances in Newral Information Processing Systems, 2014.
[12] Xi Guo, Zhicheng Wang, Qin Yang, Weifeng Lv, Xianglong Liu, Qiong Wu, and Jian Huang. Gan-hased virtual-to-reat image translation for urban scene semantic segmentation. Neurocomputing, 394:127-133. 2020.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778 2016.
[14] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE International Conference Computer Vision, pages 1501-1510, 2017.
[15] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised imagetto-image translation. In European Conference Computer Vision, pages 172 189, 2018.
[16] Zhizhong Huang, Shouzhen Chen, Junping Zhang, and Hongming Shan. Pfa-gan: Progressive face aging with generative adversarial network. IEEE Transactions on Information Forensics and Security, 2020.
[17] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou. and Alexei A Efros. Image-to-iman translation with conditional adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1125- 2017.
[18] Dakai Jin, Ziyue Xu, Youbao Tang. Adam P Harrison, and Daniel J Mollura. Ct-realistic lung nodule simu- lation from 3d conditional generative adversarial networks for robust lung segmentation. In International Conference on Medical Image Computing and Computer-Assisse Intervention, pages 732-740. Springer, 2018.
[19] David Kanaa, Vikram Voleti, Samira Kahou, Christopher Pal, and Mila CIFAR. Simple video generation using neural odes. In Advances in Neural Information Processing Systems, 2019.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[21] Diederík Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 131.6111 2013.
[22] Wilhelm Kutta. Beitrag zur naherungsweisen integration totaler differentialgleichungenn Z. Math. Phys., 46:435-453, 1901.
[23] Hsin- Ying Lee. Hung- Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Diverse image- to-image translation via disentangled representations. In European Conference on Computer Vision, pages 35-51, 2018.
[24] Antti Lehmussola, Pekka Ruusuvuori, Jyrki Selinummi, Heikki Huttunen, and Olli Hli-Harja. Computa- tional framework for simulating fluorescence microscope images with cell populations. JEEE transactions medical imaging, 26(7):1010-10 1016, 2007.
11

---

[25] Wallace Lira, Johannes Merz. Daniel Ritchie, Daniel Cohen-Or, and Hao Zhang. Ganhopper: Multi-hop gan for unsupervised image-to-image translation. In European Conference on Computer Vision, pages 363-379. Springer. 2020.
[26] Qi Mao, Hsin- Ying Lee, Hung- Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In IEFE Conference on Computer Vision and Pattern Recognition, pages 1429-1437,
[27] Qi Mao, Hung- Yu Tseng, Hsin-Ying Lec, Jia-Bin Huang, Siwei Ma, and Ming- Hsuan Yang. Continuous and diverse image-to-ing to-image translation via signed attribute vectors. arXiv preprint 1.X0v22011.01215, 2020.
[28] Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 1rIiv:1411.17 2014.
[29] Takeru Miyato and Masanori Koyama. 1OX.::802..066.7 2018.
egans with projection discriminator.
arXiv preprint
[30] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint 161v::601.00679, 2016.
[31] Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen, Image-to to-iimge translation: Methods and applica- tions. arXiv preprint arXiv:2101.0800 2021.
[32] Sunghyun Park, Kangyeol Kim, Junsoo Lee, Jacgul Choo, Joonscok Lee, Sookyung Kim, and Edward Choi. Vid-ode: Continuous-tine video generation with neural ordinary differential equation. arXiv preprint arXiv:2010.055 2020.
[33] Deepak Pathak. Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In IEEE Conference Computer Vision and Pattern Recognition, pages 2536-25444, 2016.
[34] Pengda Qin, Weiran Xu, and William Yang Wang. Dsgan: Generative adversarial training for distant supervision relation extraction. arXiv preprint arXiv:1805.000 1805.09929 2018.
[35] Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, al. DCFNet: Deep neural network with decomposed convolutional filters. In International Conference on Machine Learning, pages 4198-4207. PMLR, 2018.
[36] Andrés Romero, Pablo Arbeláez. Lue Van Gool, and Radu Timofte. Smit: Stochastic multi-label image- to-image translation. In IEEE International Conference on Computer Vision Workshops, pages 0-0, 2019.
[37] Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. Latent odes for iregularly-sampled time series. arXiv preprint arXiv: arXiv:/907.00 1907.03907, 2019.
[38] Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Scribbler: Controlling deep image synthesis with sketch and color. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2017.
[39] Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986.
[40] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, pages 3483-3491 2015.
[41] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, al. Scalability in perception for autonomous driving: Waymo open dataset. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2446-2454, 2020.
[42] Aaron Van den Oord, Nal Kalehbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, ct al. Conditional image generation with pixelenn decoders. Advances Neural Information Processing Systems, pages 4790-4798, 2016.
[43] Ting-Chun Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In IEEE Conference on Computer Vision and Pattern Recognition, 2018
12

---

[44] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In European Conference on Computer Visian Workshops, pages 0-0, 2018.
[45] Yaxing Wang. Abel Gonzalez-Gared Joost van de Weijer, and Luis Herranz. Sdit: Scalable and diverse cross-domain image translation. In ACM International Conference on Multimedia, pages 1267-1276. 2019.
[46] Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu, A dictionary approach to domain-invariant invariant learning in deep networks. Advances in Neural Information Processing Systems, 2020.
[47] Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. International Conference of Learning Representations, 2020.
[48] Wang, Zichen Miao, Hu, and Qiang Qiu. Adaptive convolutions with per pixel dynamic filter atom. In IEEE International Conference on Computer Vision, pages 12302-12341 12311, 2021.
[49] Po- Wei Wu, Yu-Jing Lin, Che-Han Chang, Edward Y Chang, and Shih- Wei Liao. Relgan: Multi-domain image-to-image tramslation via relative attributes. In IEEE International Conference on Computer Vision, pages 5914-59122 1122--92,222 2019.
[50] Wenqi Xian, Patsorn Sangkloy. Varun Agrawal, Amit Raj. Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Texturegan: Controlling deep image synthesis with texture patches. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1456-8465, 2018.
[51] Taihong Xiao, Jiapeng Hong, and Jinwen Ma. Dna-gan Learning disentangled representations from multi-attribute images. arXiv preprint
2017.
[52] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In IEEE Conference Computer Vision and Pattern Recognition, pages 5505 5514, 2018.
[53] Xiaoming Yu, Yuanqi Chen, Thomas Li, Shan Liu, and Ge Li. Multi-mapping image-to-image translation learning disentanglement, arXiv preprint arXiv:1909. 1900007877. 2019.
[54] Yuan Yuan, Siyuan Liu, Jiawei Zhang, Yongbing Zhang, Chao Dong. and Liang Lin. Unsupervised image super-resolution using cycle-in in-cycle generative adversarial networks. JEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 701-710, 2018.
[55] Jianfu Zhang, Yuanyuan Huang. Yaoyi Li, Weijie Zhao, and Liqing Zhang. Multi-attribute transfer via disentangled representation. AAAI Conference on Artificial Intelligence, volume 33, pages 9195-9202, 2019.
[56] Zhifei Zhang, Yang Song, and Hairong Qi. Age progresson/vegressionn by conditional adversarial autoen- coder. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5810-5818, 2017.
[57] Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image- to tongagoo-mmage translation using vycle-consisten adversarial networks. In IEEE international Conference on Computer Vision, pages 2223 2233, 2017.
13


Reviews:
[{"review_id": "s6t9kPKcqF2", "reviewer": "Reviewer_aPp5", "summary": "The authors build on previous conditional image generation models, which are based on conditional Generative Adversarial Networks (cGAN). The authors use the observation made by previous work [1, 2], that the kernel $F$ of a convolution layer can be replaced with a low-rank, two-component tensor factorization $F = AD$. $A$ is called the coefficient tensor, and has dimensions [channel_in x channel_out x m]. $D$ has dimensions [m x kernel_width x kernel_height] and represents the $m$ \"atoms\" of the convolution kernel. \n\nThe authors' goal is to devise a cGAN, such that the generator's output is a smooth function of its conditioning input. To this end, the authors propose to model \"atoms tensor\" $D = D(t)$ using a neural ODE, which turns the atoms into a smooth function of the introduced fictitious time variable $t$. Given some further mild requirements, the authors show that smoothness in the filter atoms induces smoothness in the outputs. Based on this theoretical contribution, they propose novel ways to design cGANs that can be conditioned on images, on some real-valued label or both. \n\nThe authors verify the effectiveness of their method on several different tasks, for which they achieve results that are comparable or exceed the state-of-the-art while using much fewer parameters than most comparable methods.\n\n[1] Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. DCFNet: Deep neural network with decomposed convolutional filters. ICML 2018\n\n[2] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. ICLR 2020", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\nThe core idea of the work, i.e. to model the filter atoms using a neural ODE, and then show that the smoothness from the atoms \"propagates\" to the model's output is a novel and nice, though perhaps somewhat unsurprising contribution.\nIn my view, the most significant contribution of the work is how this key result is utilised to elegantly allow conditioning on labels whose domain is a subset of the real line.\nThe methodology seems reasonable, and the authors perform a reasonable extensive empirical verification of their method, and achieve good results.\n\n# Weaknesses\nThe writing suffers from several mistakes on many levels. It has several spelling and grammatical mistakes, incorrect word usage, incorrect, hard-to-parse and sometimes seemingly superfluous sentences. I also believe that some of the formulae are incorrectly stated.\nTo give a non-exhaustive list of examples:\n- On several occasions, the word \"graduate\" is used incorrectly to mean \"gradual\", including in Fig 1.\n- Lines 42-43: \"First, this method is sensitive to network configurations, and mode collapse, which results in a point estimation to the subspace of parameter, can often be observed.\" - I cannot parse this sentence.\n- Line 74: \"[...] with the restriction of condition.\" - I am not sure what the restriction is supposed to mean here.\n- Lines 76-77: \"From the perspective of the dynamic of parameter subspace, [...]\" - While later sections somewhat clarify what this means, this is extremely confusing at the beginning.\n- Lines 99-101: \"[16, 27] propose ...\" - This sentence belongs to the Related Works section, not the Background section.\n- Lines 147-155: The authors seem to explain neural ODEs, even though they have already done so in section 3.1\n- Lines 166: I believe that in the definition of the local input norm we the terms in the sum should be $x(u')$ instead of $x(u - u')$, otherwise for every pixel $u$ its local norm would be the local norm around $x(0)$.\n\nBeyond these mistakes, I found the general content layout to be confusing. It took several readings to understand the core idea of \"propagating\" the Lipschitz continuity from the filter atoms to the outputs, because of the poor explanation.\n\n# Summary\nIn my opinion, this paper proposes and executes a couple of very nice ideas, which are unfortunately heavily overshadowed by the poor writing. If the authors can heavily improve the writing, I would be happy to recommend acceptance, though I believe this would require major restructuring of the paper.\n\n# After the rebuttal\nThe authors have addressed my technical concerns, and after some discussion with the other reviewers, I have more confidence that the writing can be improved for the camera-ready version of the paper, hence I raised my score to recommend acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion about limitations or social impact in the work. Including savings on the parameter count, or inference speed would have been useful. Some explanation and explorations of the theoretical consequences of Theorem 1 would be very useful. Comments on the stability of training and its scalability would be also important.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5"}, "extra_text": {}}, {"review_id": "okFL2Q8AZQe", "reviewer": "Reviewer_FG2b", "summary": "The presented paper deals with the problem of diversity in produced samples of image generation models. The proposed solution employs the recent framework of decomposing convolutional filters over a set of learned filter atoms. To make the results diverse, one multiplier of such decomposition is modeled with neural ODEs, thus allowing for continuous changes in the outputs of the generator (the related theorem was proved). Importantly, the solution may be integrated into many architectures as a drop-in replacement for standard convolutions.\n\nThe authors have conducted experiments on image-to-image translation and image generation with continuous labels. The reported results demonstrate that the ODE-driven filter decomposition leads to meaningful results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Strengths. \n\n    a. Neural ODE is a celebrated way of modeling time-continuous dynamics and it looks like a reasonable tool for generative models with non-discrete labels. Also, the framework of Filter Atoms is an interesting alternative for other common conditioning methods like AdaIN, weight (de)modulation, etc. Therefore, the motivation for this paper is pretty clear.\n\n    b. The paper is well-written, all the concepts are explained in a self-contained manner. \n\n2. Weaknesses.\n\n    a. First of all, I have some concerns about experiments from the section \"img2img translation conditional on images\" (edges -> handbags, etc., line 269). The naturalness of learning the mapping between a one-dimensional time variable and color/satellite style/etc is at least questionable to me. The approach of BasisGAN that models such styles as samples from a multivariate Gaussian seems more reasonable, and this method is well-established in img2img literature (e.g., see [1, 2, 3]). This is possibly why BasisGAN mostly outperforms the authors' model in this task (Table 1). Unfortunately, the authors do not show qualitative results for this problem other than edges -> handbags.\n\n    b. The results on UTKFace and CACD2000, presented in the appendix, look not very convincing to me. The presented approach seems to perform not that well on human faces translation. As Table 3 claims the presented approach outperforms CycleGAN, it would be interesting to demonstrate also the CycleGAN outputs to compare the samples qualitatively.\n\n    c. Why do the authors need to use discrete \"distinct labels\" (line 304). What exactly makes truly continuous labels \"intractable\" for the ODE-based approach?\n\n    d. A lot of recent papers [4, 5] have shown the efficiency of positional embedding as a tool for feeding continuous values into the neural network. I think this could be a much simpler baseline aiming to check the usefulness of ODE for continuous conditioning: one may positionally encode the time variable with random or learned frequencies to a vector, then process it with some multi-layer perceptron and obtain a conditioning vector for AdaIN or weight (de)modulation [6]. Such a baseline may be used for experiments described at lines 278 and 312.\n\n    e. The paper lacks hi-res experiments, unlike BasisGAN, which compared itself with the pix2pixHD approach for segmentation -> cityscapes problem. Is the ODE-based approach scalable? How is inference time affected with ODE solver being involved?\n\n3. Suggestions.\n\n    a. The recent CoMoGAN [7] paper also deals with the conditioning on a continuous one-dimensional time variable to model daytime light changes (among other tasks). Although this paper had not been released by the time of the NeurIPS deadline, I think that comparison with CoMoGAN would make the submission stronger.\n\n4. Rating (pre-rebuttal)\n\n    With all the pros and cons listed above, I tend to rate this paper below the threshold (score = 5). To my mind, more evidence to use exactly an ODE-based solution is required. Some of the results look not impressive making the quality of the model questionable. I ask the authors to address the concerns in the rebuttal. \n\n5. Updates (post-rebuttal)\n\n    After reading the other reviews, as well as the authors' feedback, I would like to raise my score from 5 to 6. The main reason that prevents me from giving a higher rating is the lack of a baseline of the same type, i.e., able to perform continuous changes w.r.t. a `time` variable (e.g., one suggested in the review). Without such a comparison, the paper is just a proof of concept without any evidence that ODE is the best method for the considered tasks.\n\n\nReferences\n1. Huang et al. Multimodal Unsupervised Image-to-image Translation. ECCV, 2018.\n2. Lee et al. DRIT++: Diverse Image-to-Image Translation viaDisentangled Representations. IJCV, 2020.\n3. Anokhin et al. High-Resolution Daytime Translation Without Domain Labels. CVPR, 2020.\n4. Tancik et al. Fourier Features Let Networks Learn High-Frequency Functions in Low Dimensional Domains. NeurIPS, 2020.\n5. Sitzmann et al. Implicit Neural Representations with Periodic Activation Functions. NeurIPS, 2020.\n6. Karras et al. Analyzing and Improving the Image Quality of StyleGAN. CVPR, 2020.\n7. Pizzati et al. CoMoGAN: continuous model-guided image-to-image translation. CVPR, 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have no suggestions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Q75VZfRA3RH", "reviewer": "Reviewer_t6CV", "summary": "The paper describes a novel approach of decomposing convolutional filters into a set of filter atoms, which are modeled by neural ODEs. The authors show how the continuity allows generating gradually changing images. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Defining the parameters via a composition of filter atoms and implementing these filter atoms via neural ODEs seems like a good idea, introducing continuity without extreme computational costs. The paper is a bit out of my domain of expertise, so I can’t comment much on novelty and evaluation.\n\nStrengths\n1. The paper is a pleasure to read. It seems very well organized. \n1. The model allows efficient training by leveraging a low-dimensional subspace of filter atoms. \n1. Gradual changes of generated images can be learned without the need for auxiliary modules or extra supervision.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Weaknesses/Questions\nTable 4 doesn’t have the best performing models highlighted in bold.\nTypos:\n1. Line 39: stochasiticity\n1. Line 330: graduate -> gradual\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "HUYjfUkgFog", "reviewer": "Reviewer_Nnbn", "summary": "This paper designs continuous filter atoms to enable gradual changes for image generation, by using the ordinary differential equation. This is a clever strategy that decomposes convolutional filters with atoms, and it is significant for the high-dimensional filters. Specifically, those continuous filter atoms are helpful to obtain smooth propagated image generation.  Various experiments on image to image translation and conditioned image generation verify the performances of the introduced filter atoms. Besides, the proposed filter atoms allow users to manipulate images in a new way that controls the integration interval. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using continuous filter atoms from an ODE atom generator is very interesting and meaningful. I believe that it is a very effective and useful method for image generation. The experiments on conditional image generation show that filter atoms cover a wide range of gradually varying appearance and accurate correspondence to the input condition.  Therefore, it allows flexible appearance manipulation without supervision. Also, it can generate image appearance arbitrarily conditioned on continuous labels.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The experimental setting can be improved. The quantitative evaluation is limited. The user study and ablation experiments are missing.\n\n-Paper writing needs to be improved. For example:\n- L.039, \"stochasiticity\" --> \"stochasticity\"\n- L.063, \"varying appearance\" --> \"varying appearances\"\n- L.066, \"without heavy supervisions\" --> \"without heavy supervision\"\n- L.075, \"has attract attention\" --> \"has attracted attention\"\n- L.189, \"atoms along the learned continuous\" --> \"atoms along with the learned continuous\"\n- L.248, \" get the desirable target distribution\" --> \" get the desired target distribution\"\n- L.275, \"this smooth transitions\" --> \"these smooth transitions\"\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "72"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes using continuous filter atoms (based on Neural ODE) for image generation.  These continuous filter atoms can enable smoother image synthesis compared to convolutional filters. The paper has received positive reviews. Many reviewers find the idea interesting and the results encouraging. The core idea is novel. There are concerns regarding the writing and baselines. The rebuttal addressed most of the concerns. The AC agreed with the reviewers’ consensus and recommended accepting the paper. The authors are encouraged to improve their writing according to the reviewers' comments. 
Decision: Accept (Spotlight)