Image Generation using Continuous Filter Atoms
Year: 2022

Abstract:
In this paper, we model the subspace of convolutional filters with a neural ordinary differential equation (ODE) to enable gradual changes in generated images. Decomposing convolutional filters over a set of filter atoms allows efficiently modeling and sampling from a subspace of high-dimensional filters. By further modeling filters atoms with a neural ODE, we show both empirically and theoretically that such introduced continuity can be propagated to the generated images, and thus achieves gradually evolved image generation. We support the proposed framework of image generation with continuous filter atoms using various experiments, including image-to-image translation and image generation conditioned on continuous labels. Without auxiliary network components and heavy supervision, the proposed continuous filter atoms allow us to easily manipulate the gradual change of generated images by controlling integration intervals of neural ordinary differential equation. This research sheds the light on using the subspace of network parameters to navigate the diverse appearance of image generation.

heading: 1 Introduction
text: Conditional image generation has been widely studied in recent years due to its numerous applications including image segmentation [12, 18], style transfer [57, 20], image inpainting [33, 52], image super-resolution [54, 44], image registration [1], and image synthesis [31]. Despite extensive research and applications in these fields, limited progress has been made on conditional image generation using continuous or closely spaced labels due to the difficulties pointed out in [8], e.g., the absence of real images for some labels. It is shown in [8] that the standard training with empirical risk minimization does not apply well to continuous labels. Moreover, it remains challenging to encourage the generation diversity, especially without heavy supervision, while maintaining output fidelity. Previously, generation diversity is mainly encouraged using explicit regularization terms to convey diversity with additional latent codes [26, 34]. However, explicit regularizations inevitably introduce additional hyperparameters, and sub-optimal hyperparameters can often cause either poor diversity or noticeable sacrifices in terms of generation quality and the correspondence to input conditions.
On the other hand, achieving generation diversity by modeling the deep network parameter space has attracted limited attention [47]. This direction is mainly challenged by the prohibitive cost on both modeling of and sampling from the very high-dimensional space of convolutional filters in modern image generative networks. Motivated by the observation that a convolutional filter can be well approximated by a linear combination of low-dimensional filter atoms, BasisGAN [47] shows in image generation that, the high-dimensional filter space in each layer can be well approximated by a low-dimensional filter subspace, which significantly reduces the cost of both parameter modeling and sampling. In this way, each sampling of the modeled parameter subspace results in one deterministic transformation from the input condition to the desired target domain, and the diverse outputs are achieved by sampling multiple times. More importantly, it is empirically shown in [47] that without
⇤Equal contribution.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
any explicit regularizations, such parameter subspace modeling can work as a plug-and-play module to convert a deterministic conditional image generation network to produce diverse images with appealing stochasticity, without any auxiliary network components or regularization terms.
However, we observe that the filter subspace modeling in [47] suffers from several major limitations: First, this method is sensitive to network configurations of the basis generator, as minor changes to the parameters can highly likely to trigger mode collapse in the parameter distribution, which results in a point estimation to the subspace of parameter and removes diversity in the output images. Furthermore, BasisGAN is incapable of modeling a continuous space, so that gradual changes to the generated parameters cannot be obtained by simply interpolating the latent codes.
In this paper, we adopt a subspace view to convolutional filters by performing atom-coefficients decomposition, as in [35, 47, 46, 48]. Then we further model the filter subspace using a neural ordinary differential equation (ODE), so that we are able to sample from this continuous subspace a series of filter atoms at arbitrarily fine resolution. The low-dimensional filter subspace allows extremely efficient learning and modeling using ODE. We show both empirically and theoretically that, continuous transition in the filter subspace can be propagated naturally to generated images, and thus produce visually appealing images with gradually varying appearance. More importantly, we show that the continuity introduced to the filter subspace and the induced generation smoothness allow continuous manipulation of generated appearance with only discrete samples as supervision, which is a task that is used to be considered intractable by training with standard empirical risk minimization [8]. These appealing properties of the proposed method enable various significant applications through only standard training, without requiring any auxiliary network components, additional regularization terms, or heavy supervision.
We perform various experiments on conditional image generation, with conditions being in the form of images, labels, or both. We list here several example applications that are enabled by the proposed image generation using continuous filter atoms from an ODE atom generator:
• Continuous image synthesis that covers a wide range of gradually varying appearance with high fidelity and accurate correspondence to the input condition.
• Sequential image synthesis with explicitly specified starting and ending points to allow flexible appearance manipulation without heavy supervision.
• Interpolation of generated image appearance at arbitrarily fine resolution.
• An effective approach to generating images conditioned on continuous labels.

heading: 2 Related Work
text: Conditional image generation. Image generation is widely studied in the machine learning community, from restricted Boltzmann machines [39] to variational autoencoders [21]; in particular variants with conditions [30, 40, 42] show promising results. Empowered by GANs [11], conditional generative adversarial networks (cGANs) [17, 33, 38, 43, 50, 57, 26, 47] have demonstrated superior performance on image generation with the restriction of condition. Recently, conditional image generation with conditions on continuous labels has attracted attention, and new training schemes [8] are proposed to improve continuous conditional image generation. From the perspective of the dynamic of parameter subspace, our method provides a simple yet effective solution.
Continuous image translation. Many works have discussed improving the diversity of generated images in image-to-image translation tasks [47, 34, 6, 26, 23, 45, 15, 53, 5]. However, continuous image translation that aims to generate intermediate images between the source and the target images is not yet widely studied. Some previous works [55, 51, 27, 49, 10] have manipulated the attribute vectors either by interpolation or linear transformations to continuously control the latent space. [16, 25] employ multiple generative networks in the middle of the pathway from source to target image to generate intermediate results. [6, 36] disentangle the latent space as content and style codes and interpolate the styles, in order to produce intermediate samples. However, all the aforementioned methods rather need to import auxiliary neural networks or carry extra burdens on model training by introducing additional regularization or supervisions. Moreover, synthesizing intermediate images without relying on interpolation of either domain labels, or disentangled representations, which can discourage the stochasticity of generated images, is still considered a big challenge.

heading: 3 Continuous Filter Atoms
text: In this section, we start with a brief review of neural ODEs and atom-coefficient filter decomposition, which serve as key ingredients in the proposed image generation approach. We then present in detail that, with the effectiveness being theoretically justified, how the proposed continuous atom generation using neural ODEs achieves gradual image generation.

heading: 3.1 Neural ODEs
text: Neural ordinary differential equations (ODEs) are introduced in [4] as an approach to interpreting the neural network as a system of ODEs, where each ODE represents an underlying dynamic of the hidden elements. Especially, neural ODEs are frequently adopted to enforce continuous dynamics in latent representations, for example, [19, 32] propose to use neural ODEs to smoothly connect the latent space of video frames; [37] adopts ODE-RNNs to model time-continuous hidden dynamics to handle irregularly sampled time-series data. Neural ODEs model a latent state z(ts) as z(ts) = z(t0) + R ts t0
f(z(t), t;✓) dt, where dz(t)dt = f(z(t), t;✓) is modeled as a neural network parametrized by ✓. Then, the latent state at an arbitrary point ts can be obtained as
z(ts) = ODESolve(z(t0), f, (t0, ts), ✓), (1)
where t0 and ts denote the start and the end of the integral interval, respectively. The simplest method for approximating the solutions of the ordinary differential equations with a given initial value is the Euler’s method, which is a first-order integrator with a fixed step size. In practice, families of Runge-Kutta methods, e.g., the Midpoint method and fourth-order Runge-Kutta [22], are preferred due to their superior convergence and stability. Note that all the experiments in this paper follow the implementation in [4] for Runge-Kutta of fifth-order of Dormand-Prince-Shampine with adaptive step size [9].
For any given integral interval (ti, tj) where i, j 2 {0...T}, ODESolver always outputs a unique solution for the integral of the continuous dynamics, as long as the ordinary equation f is uniformly Lipschitz continuous in z and t [7]. Parametrizing f as a neural network equipped with Lipschitz continuous non-linear functions will meet the requirements. Thus, in our paper, modeling filter subspace using neural ODEs guarantees continuity as we will theoretically justify later.

heading: 3.2 Convolutional Filter Decomposition
text: Directly modeling the space of high-dimensional convolutional filters in an image generation network is practically prohibitive in terms of both computation and parameter scale [47]. Inspired by the observation that a convolutional filter can be well-approximated as a linear combination of filter bases [35], as shown in [47], a subspace view to the convolutional filters can be adopted in image generation network by decomposing each convolutional filter as a linear combination of low-dimensional filter atoms. Specifically, given an l ⇥ l-sized convolutional filter F 2 Rc⇥c0⇥l⇥l with c0 input and c output channels, respectively, we can decompose the filter over a set of m atoms D as F = AD, where D 2 Rm⇥l⇥l, and A 2 Rc⇥c0⇥m are the filter subspace coefficients. As we will show later, by modeling the continuity of a low-dimensional filter subspace using neural ordinary differential equations, we achieve efficient training and modeling of gradual appearance changes in conditional image generation.

heading: 3.3 Continuous Atom Generation for Smooth Appearance Modeling
text: In this paper, we aim for modeling gradual changes in generated images with respect to continuous input conditions. Instead of seeking new ways of training, we provide a unique view by modeling the transformation from an input condition to the corresponding output with parameters instantiated from an underlying continuous space, which subsequently leads to generated images with gradual changes.
We adopt the aforementioned atom-coefficient decomposition, which permits low computational modeling of filter subspace as also validated in [47]. However, modeling a filter subspace with standard neural networks can still suffer from mode collapse often caused by sub-optimal network configurations, and the continuity is also difficult to be guaranteed by standard training as discussed
in [8]. We therefore adopt a neural ODE based filter atom generator with the continuity property discussed in Section 3.1, and show theoretically that such atom continuity can be propagated to the deep network outputs to achieve conditional image generation with gradual changes.
Specifically, we decompose the i-th layer filters Fi into respective atoms Di and coefficients Ai, as Fi = AiDi. Over k layers of filters {Fi}k1 , we jointly model the corresponding atoms {Di}k1 using atom generation. We assume that all instantiations of filter atoms reside in a continuous space parametrized as D(t) where t is a time variable, i.e., the generated atoms at the time point t can be expressed as D(t). In the limit of a small step ✏, the local dynamic within this atom space can then be expressed as D(t+ ✏) = D(t) + ✏ · dD(t)/dt. Therefore, given an initial state of atoms within this space denoted as D(t0), and modeling the derivative of dD(t)/dt using a neural network f(·, ·; ✓):
dD(t) dt = f(D(t), t; ✓), (2)
any samples within this atom space can then be obtained by an integral of the function values of f from t0 to the desired ending point. The described dynamic leads naturally to a neural ordinary differential equation, where the instantiation at a time point ts, s > 0, in this continuous dynamic can be obtained through a continuous atom generator T by
D(ts) = T (ts;D(t0), ✓) = D(t0) + Z ts
t0
f(D(t), t; ✓)dt. (3)
Note that the initial state D(t0) here are free parameters to be optimized in the training jointly with ✓ that parametrizes f . Starting from the initial state D(t0), an instantiation of filter atoms D(ts) is defined to be a unique solution to the ODE initial value problem at some condition ts, by Picard’s existence theorem. Consequently, as t varies, different instantiations of atoms are generated from this modeled continuous filter subspace, and our corresponding atom generator equipped with ODESolver learns the underlying continuous dynamics of the filter subspace.
Given the subspace of filters modeled as an underlying continuous dynamic through neural ODE, we prove the Lipschitz-type continuity transducted from atoms to output features in the i-th layer. For simplicity, let c = c0 = 1, and the argument extends. Given an input image X(u) (u 2 U ,U ⇢ Z2) , define the local input norm ||x||2,Nu := ( P u02Nu x(u u
0)2)1/2 and the convolution hx,wiNu :=P u02Nu x(u u
0)w(u0), where Nu ⇢ U is a local Euclidean grid centered at u. The i-th layer function F i : Xi 1 ! Xi can be expressed as,
Xi(u) = F i(Xi 1,Di;Ai) = ( mX
k=1
AikhXi 1,Di,kiNu + bi), (4)
where Di,k denotes the k-th atom in the i-th layer, Aik is the corresponded k-th coefficient, and b i denotes the bias at the i-th layer. Theorem 1. Suppose Di1 and Di2 are two continuously generated atoms, (i.e., 9M > 0, ||Di1 Di2||2  M |t1 t2|), and assume the activation function is non-expansive which holds for ReLU, then F i is continuous in Di,
||Xi1 Xi2||2  (||Ai||2 ) p |U| · ||(Di1 Di2)||2, with = sup
u2U ||Xi 1||2,Nu , (5)
in which Xi1 = F i(Xi 1,Di1;Ai), Xi2 = F i(Xi 1,Di2;Ai) are outputs correspond to two atoms. Proof. The proof is provided in the Appendix Section A.
Notably, modeling the atom subspace of filters offers significant advantages on efficiency. Under the rule of neural ODE, the dimensionality of D(t) cannot be varied, meaning the dimension of the input and the output of the ODESolver remain strictly consistent. This additional restriction makes modeling the space of full-size filters practically infeasible due to the computation and memory constraints. Meanwhile, by modeling the filter subspace instead, we significantly reduce the cost of learning the corresponding neural ODE. Typically, atoms across layers introduce a few hundred parameters only and significantly simplify the training and inference of atom generator T .

heading: 3.4 Image Generation with Continuous Labels
text: Depending on the specific applications of the proposed continuous atom generation as we will show next, the ending points of the ODE integral can be: 1) explicitly given to strictly control the behavior of continuous dynamics, or 2) stochastically sampled from a prior distribution, e.g., Uniform or Gaussian, to enhance the diversity of the generated images.
Explicit continuous conditions. Compared to the previous conditional GAN methods [28, 29], where label information is vectorized or one-hot encoded, the proposed method permits minimal preprocessing to the continuous labels, which are fed into the continuous atom generator simply in the forms of raw floating numbers preserving the continuity of labels. Moreover, unlike [28] which can only take finite discretized conditions, such as categorical class information, continuous filter atoms do not hold any such restrictions. Therefore, by sampling filter atoms from the subspace of convolutional filters modeled by neural ODE, with the desired ending points of integral specified by the continuous labels, the continuous filter atoms can be evaluated with any values within a valid range of the regression labels. To illustrate this, we present an interpolation example in Figure 3, and show that, with the underlying dynamics of the rotated chair images captured by continuous atom generator trained on limited angels only, the proposed method allows interpolation to the modeled dynamic at arbitrarily fine resolution, and generates rotated chairs at any angels that are not exposed to the model during training.
Stochastic continuous conditions. The stochastic modeling of integration interval permits generated images with diverse appearances, and gradual transitions among the generated images. As we will show in Section 4.1, while the proposed method cannot directly enforce perceptual diverse samples as in [47], the sampled sequences with gradual appearance changes jointly offer decent diversity while maintaining outstanding fidelity and correspondence to the input conditions.

heading: 3.5 Conditional Image Generation with Continuous Atoms
text: By simply plugging in the proposed continuous atom generator to off-the-shelf GANs, we show next that variants of conditional generative adversarial models (cGAN) [11, 28] with continuity can
be easily obtained. The training does not involve any auxiliary network components or additional regularizations.
Conditional on images. Image generation conditional on images, also known as image-to-image translation [17], aims at generating a target image of a certain domain with high fidelity and desirable diversity [47, 26, 34] while maintaining correspondence w.r.t. an input image condition. We introduce image generation conditional on input images with the output space modeled as a smooth sequence through continuous atom filters, thus sampling the continuous filter atoms allows generating images with gradual appearance changes without losing the correspondence to the conditions. Formally, representing the condition images as x, which are usually samples from an empirical distribution p(x), we aim at modeling a conditional distribution p(y|x). By plugging in the continuous atom generator T (t;D0, ✓), the conditional distribution is learned to be modeled by generator G ,✓,D0 , where contains all remaining parameters in the generator, e.g., the coefficients A in each decomposed convolutional layer. And the input to the discriminator D is now trained to evaluate the compatibility of a pair of images, one from the source domain x ⇠ p(x), the other one comes from either true target distribution y ⇠ p(y|x), or the generated approximation G ,✓,D0(x; T (t;D0, ✓)). The training can be realized as
min G max D L(D,G) =Ex⇠p(x),y⇠p(y|x)[log(D(x,y))]+
Ex⇠p(x),t⇠U(t0,t 1)[log(1 D(x,G ,✓,D0(x; T (t;D0, ✓))))], (6)
where U(t0, t 1) denotes uniform distribution with the range specified by t0 and t 1. Conditional on labels. As pointed out in [8], cGANs are good at conditional image generation with discrete labels but can struggle at continuous labels. In our method, without leveraging any customized training schemes as in [8], we show that the continuous image generation with standard empirical risk minimization can be significantly improved by placing an assumption on the gradual appearance change across continuous labels, and trained with continuous atom generation given continuous labels as the inputs. Formally, we define the scalar regression label as x sampled continuously from x ⇠ p(x), and plug in the continuous atom generator to standard cGAN to model the desired target distribution p(y|x). The image generation is then achieved as G ,✓,D0(z; T (↵x;D0, ✓)), with an input vector z ⇠ N (0, I) carrying randomness as in standard cGANs. Comparing to (6), the input to the continuous atom generator is replaced as ↵x, where ↵ is a predefined scalar that linearly scales the input continuous labels. And we use a standard discriminator with each input consists of an image and a label. The training can be realized as
min G max D L(D,G) =Ex⇠p(x),y⇠p(y|x)[log(D(x,y))]+
Ex⇠p(x),z⇠N (0,I)[log(1 D(x,G ,✓,D0(z; T (↵x;D0, ✓))))]. (7)
Conditional on both images and labels. For image-to-image translation tasks, a few works have explored imposing conditions on both images and labels to achieve scalable image translation while introducing diversity and manipulations. Specifically, [45] and [5] impose additional domain attributes to navigate the image translation. However, the domain attributes they condition on are all categorical labels, and none of those methods can adopt continuous or raw format (e.g, float numbers) labels to capture the gradual changes in the desired target image space. Our method permits using the continuous labels fed into the atom generator to explicitly control sampling in the continuous filter subspace, while maintaining consistent input image conditions. Therefore, the proposed method is able to achieve image generation with gradual appearance changes w.r.t. the input continuous label conditions, and preserve strong correspondence to the source image. To the best of our knowledge, our method is the first work to present gradually changing image-to-image translation results conditioned on both images and continuous labels. Formally, having both x ⇠ p(x) and x ⇠ p(x), which refer to the input image and the scalar label, respectively, we employ our continuous atom generator, T (↵x;D0, ✓), to get the desired target distribution p(y|x,x). We then formulate our image generator as G ,✓,D0(x; z; T (↵x;D0, ✓)). The training can be realized as
min G max D L(D,G) =Ex⇠p(x),y⇠p(y|x,x)[log(D(x,y))]+
Ex⇠p(x),x⇠p(x)[log(1 D(x,G ,✓,D0(x; T (↵x;D0, ✓))))]. (8)

heading: 4 Experiments
text: In this section, we present results and comparisons in two experimental settings. We start with continuous image-to-image translation tasks where we show that by directly plugging in the proposed continuous atom generator, we can either condition the generated images solely on images, or on both images and labels, where each of the cases refers to (6) and (8), respectively. In both cases, we quantitatively and qualitatively show that our method achieves continuous image generation with high fidelity and diversity. Then, we proceed to image generation tasks conditional on continuous labels following (7). We compare the proposed method with [8] and [28] on the face generation experiments. Implementation details are in Appendix Section B. More qualitative results are in Appendix Section C.

heading: 4.1 Continuous Image-to-image Translation
text: The introduced continuous atom generation can be naturally applied to the image-to-image translation experiments [47] as a simple plug-and-play module. Specifically, in image-to-image translation tasks with paired samples, we adopt a ResNet [13] based generator network following the implementation of Pix2Pix [17]. All the convolutional filters of the ResNet blocks are decomposed into atom-coefficients where an individual atom of each block is generated continuously with neural ODE. Note that a single neural ODE is utilized throughout the entire network, enforcing the network to be governed by a single continuous dynamic. These settings are also applied in unpaired image-to-image translation tasks, where we replace our baseline model from the Pix2Pix to ResNet-based CycleGAN [57].
Continuous image-to-image translation conditional on images. We present results on various datasets including edges ! handbags, edges ! shoes, maps ! satellite, night ! days, and labels ! facades. Compared to previous methods working on the same datasets, our method does not focus on generating images with strong diversity. Instead, we show that by plugging in the introduced continuous filter atoms, images with smooth intermediate transitions can be obtained by directly
training the network without any additional supervision and regularizations. As shown qualitatively in Figure 4 and quantitatively in Table 1, these smooth transitions in the generated appearance can naturally lead to satisfying diversity without compromising the quality of the generated images and the correspondence to the input conditions.
Continuous image-to-image translation conditional on regression labels. We present results of image-to-image translation tasks conditional on regression labels on a variety of datasets and experimental settings. Following (8), the generated images are conditional on explicitly given regression labels, while keeping correspondence to the input image. We start with experiments on image-to-image translation on paired samples, and then proceed to the unpaired samples.
Paired samples. In this experiment, we plug in our continuous filter atoms to the Pix2Pix [17] model and conduct continuous conditional image-to-image translation tasks on five different synthetic datasets, RC-49 [8], RA-20, RCA-20, RT-20, RL-20, where objects in each dataset are rendered to rotate between 0 and 360 with 0.1 interval. Since there are very few datasets of 3-D objects with continuous labels, we created new datasets RA-20, RCA-20, RT-20, RL-20, by rendering ‘Airplanes’, ‘Cars’, ‘Tables’, ‘Laptops’ categories of ShapeNet, respectively, following the manuals of RC-49. We add further descriptions of datasets in the Appendix Section B.1. During training, we only use samples with rotation degrees below 90 , and fix the 0 image as the source image with varying target images. We adopt the gap of rotation degree between the source and the target image as an external condition for atom generation. We show in the Figure 5, that the generated images rotate smoothly according to the given rotation degrees. Especially for the results of RC-49, although we have organized the training data in 5 interval, it is noteworthy that our method generated interpolation results at the interval of 2.5 degree in the Figure 3. Table 2 also demonstrates that the generated images of our proposed method hold a higher level of fidelity than the baseline.
Unpaired samples. For translation tasks on unpaired datasets, we present results on UTKFace [56], SteeringAngle, cells-200, and Waymo [41] dataset. Other than the Waymo dataset [41], all the datasets are composed of images with continuous labels, e.g., the age for UTKFace, the angle of the steering wheel of a car for the SteeringAngle, and the number of cells for Cells-200. Waymo [41] dataset has four different time labels for each image, which are dawn, daytime, dusk, and night. We add more detailed descriptions on the datasets in the Appendix Section B.1 with additional results of aging face experiments on the CACD2000 [3] at Appendix Section B.2. In the unpaired image-to-image translation tasks, using continuous labels can lead to confusion in training, as the
Table 3: Quantitative results on conditional image generation when image and label were both given as input conditions under the configuration of unpaired image-to-image translation task. We present results on UTKFace [56], CACD2000[3], SteeringAngle, and Cell200. Diversity and fidelity are measured using LPIPS and FID, respectively. The LPIPS scores for the CycleGAN are marked as ’-’, as the all LPIPS scores resulted in zero, with no meaningful diversity observed.
Datasets Cell200 SteeringAngle UTKFace CACD2000
Methods CycleGAN Ours CycleGAN Ours CycleGAN Ours CycleGAN Ours
LPIPS " - 0.1355 - 0.0544 - 0.0052 - 0.0018 FID # 30.44 22.22 73.03 52.39 47.24 41.55 58.43 57.15
Input Generated samples with gradual changes
Figure 6: Continuous image-to-image translation conditional on both image and labels with unpaired samples. From top to bottom, the datasets are: UTKFace (young ! old), SteeringAngle (steering angle at -80 ! steering angle at 80 ), Cell200 (one cell ! two-hundred cells), and Waymo (dawn ! day). All models are trained with CycleGAN by plugging in our continuous atom generator.
tendency of continuous labels can be violated within unpaired samples. For instance, in the human face translation experiment, due to different levels of aging among people, it is more natural to have the age group as a class, rather than treating each age as a class. Therefore, the target domain is divided into a specific interval of labels, so that each sub-group of the target domain can be assigned distinct labels instead. The distinct labels are then directly used as a condition for modeling the continuous subspace of atoms as in (8). Despite the use of distinct labels, we highlight that it is still possible to capture continuous variation between the generated images via interpolation in the continuous filter subspace. During the training, images with certain conditions,.e.g, teenagers for the UTKFace, -65 to -80 for SteeringAngle, one number of cell for Cells-200, and dawn time for Waymo, are fixed as a source domain input. Figure 6 shows the gradual changing between the condition of the target domain. While keeping a certain amount of consistency within the input image, smooth transitions between generated images are observed. In particular, the result of the Waymo dataset at the fourth row of Figure 6 well captures progressive changes over time in the generated images from dawn to daytime, through interpolation in filter subspace. Table 3 also highlights our proposed method’s superior fidelity over the baseline model. Additional training details are presented in the Appendix Section B.

heading: 4.2 Conditional Image Generation with Regression Labels
text: We present face image generation with the condition only on a regression label specifying the age. The training objective follows (7) strictly. We construct a conditional image generation network by replacing the Adaptive Instance Normalization (AdaIN) [14] layers with standard batch normalization layers, and replace convolutional filters with the proposed continuous atom generation networks. As shown in the qualitative results in Figure 7, by fixing the input latent code z, and changing only
the condition scalar x, we are able to generate sequences of images with gradual age changes. And switching z allows changing the identity of the face sequence. Comparable quantitative results in Table 4 with CcGAN [8] further demonstrate the effectiveness.

heading: 5 Conclusion
text: In this paper, we presented both theoretically and empirically that, continuous dynamics of convolutional filters can be effectively modeled in the filter subspace by neural ordinary differential equation, to subsequently achieve conditional image generation with gradual appearance changes. The introduced continuous filter atom generation enables continuous image generation conditional on images, labels, or both. We demonstrated its effectiveness using both superior quantitative and qualitative results.

heading: 6 Acknowledgements
text: This work was supported by the DARPA TAMI program under No. HR00112190038.

Review summaries:
The authors build on previous conditional image generation models, which are based on conditional Generative Adversarial Networks (cGAN). The authors use the observation made by previous work [1, 2], that the kernel $F$ of a convolution layer can be replaced with a low-rank, two-component tensor factorization $F = AD$. $A$ is called the coefficient tensor, and has dimensions [channel_in x channel_out x m]. $D$ has dimensions [m x kernel_width x kernel_height] and represents the $m$ "atoms" of the convolution kernel.   The authors' goal is to devise a cGAN, such that the generator's output is a smooth function of its conditioning input. To this end, the authors propose to model "atoms tensor" $D = D(t)$ using a neural ODE, which turns the atoms into a smooth function of the introduced fictitious time variable $t$. Given some further mild requirements, the authors show that smoothness in the filter atoms induces smoothness in the outputs. Based on this theoretical contribution, they propose novel ways to design cGANs that can be conditioned on images, on some real-valued label or both.   The authors verify the effectiveness of their method on several different tasks, for which they achieve results that are comparable or exceed the state-of-the-art while using much fewer parameters than most comparable methods.  [1] Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. DCFNet: Deep neural network with decomposed convolutional filters. ICML 2018  [2] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. ICLR 2020

The presented paper deals with the problem of diversity in produced samples of image generation models. The proposed solution employs the recent framework of decomposing convolutional filters over a set of learned filter atoms. To make the results diverse, one multiplier of such decomposition is modeled with neural ODEs, thus allowing for continuous changes in the outputs of the generator (the related theorem was proved). Importantly, the solution may be integrated into many architectures as a drop-in replacement for standard convolutions.  The authors have conducted experiments on image-to-image translation and image generation with continuous labels. The reported results demonstrate that the ODE-driven filter decomposition leads to meaningful results.

This paper designs continuous filter atoms to enable gradual changes for image generation, by using the ordinary differential equation. This is a clever strategy that decomposes convolutional filters with atoms, and it is significant for the high-dimensional filters. Specifically, those continuous filter atoms are helpful to obtain smooth propagated image generation.  Various experiments on image to image translation and conditioned image generation verify the performances of the introduced filter atoms. Besides, the proposed filter atoms allow users to manipulate images in a new way that controls the integration interval.