Image Generation using Continuous Filter Atoms

1 Introduction . Conditional image generation has been widely studied in recent years due to its numerous applications including image segmentation [ 12 , 18 ] , style transfer [ 57 , 20 ] , image inpainting [ 33 , 52 ] , image super-resolution [ 54 , 44 ] , image registration [ 1 ] , and image synthesis [ 31 ] . Despite extensive research and applications in these fields , limited progress has been made on conditional image generation using continuous or closely spaced labels due to the difficulties pointed out in [ 8 ] , e.g. , the absence of real images for some labels . It is shown in [ 8 ] that the standard training with empirical risk minimization does not apply well to continuous labels . Moreover , it remains challenging to encourage the generation diversity , especially without heavy supervision , while maintaining output fidelity . Previously , generation diversity is mainly encouraged using explicit regularization terms to convey diversity with additional latent codes [ 26 , 34 ] . However , explicit regularizations inevitably introduce additional hyperparameters , and sub-optimal hyperparameters can often cause either poor diversity or noticeable sacrifices in terms of generation quality and the correspondence to input conditions . On the other hand , achieving generation diversity by modeling the deep network parameter space has attracted limited attention [ 47 ] . This direction is mainly challenged by the prohibitive cost on both modeling of and sampling from the very high-dimensional space of convolutional filters in modern image generative networks . Motivated by the observation that a convolutional filter can be well approximated by a linear combination of low-dimensional filter atoms , BasisGAN [ 47 ] shows in image generation that , the high-dimensional filter space in each layer can be well approximated by a low-dimensional filter subspace , which significantly reduces the cost of both parameter modeling and sampling . In this way , each sampling of the modeled parameter subspace results in one deterministic transformation from the input condition to the desired target domain , and the diverse outputs are achieved by sampling multiple times . More importantly , it is empirically shown in [ 47 ] that without ⇤Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . any explicit regularizations , such parameter subspace modeling can work as a plug-and-play module to convert a deterministic conditional image generation network to produce diverse images with appealing stochasticity , without any auxiliary network components or regularization terms . However , we observe that the filter subspace modeling in [ 47 ] suffers from several major limitations : First , this method is sensitive to network configurations of the basis generator , as minor changes to the parameters can highly likely to trigger mode collapse in the parameter distribution , which results in a point estimation to the subspace of parameter and removes diversity in the output images . Furthermore , BasisGAN is incapable of modeling a continuous space , so that gradual changes to the generated parameters can not be obtained by simply interpolating the latent codes . In this paper , we adopt a subspace view to convolutional filters by performing atom-coefficients decomposition , as in [ 35 , 47 , 46 , 48 ] . Then we further model the filter subspace using a neural ordinary differential equation ( ODE ) , so that we are able to sample from this continuous subspace a series of filter atoms at arbitrarily fine resolution . The low-dimensional filter subspace allows extremely efficient learning and modeling using ODE . We show both empirically and theoretically that , continuous transition in the filter subspace can be propagated naturally to generated images , and thus produce visually appealing images with gradually varying appearance . More importantly , we show that the continuity introduced to the filter subspace and the induced generation smoothness allow continuous manipulation of generated appearance with only discrete samples as supervision , which is a task that is used to be considered intractable by training with standard empirical risk minimization [ 8 ] . These appealing properties of the proposed method enable various significant applications through only standard training , without requiring any auxiliary network components , additional regularization terms , or heavy supervision . We perform various experiments on conditional image generation , with conditions being in the form of images , labels , or both . We list here several example applications that are enabled by the proposed image generation using continuous filter atoms from an ODE atom generator : • Continuous image synthesis that covers a wide range of gradually varying appearance with high fidelity and accurate correspondence to the input condition . • Sequential image synthesis with explicitly specified starting and ending points to allow flexible appearance manipulation without heavy supervision . • Interpolation of generated image appearance at arbitrarily fine resolution . • An effective approach to generating images conditioned on continuous labels . 2 Related Work . Conditional image generation . Image generation is widely studied in the machine learning community , from restricted Boltzmann machines [ 39 ] to variational autoencoders [ 21 ] ; in particular variants with conditions [ 30 , 40 , 42 ] show promising results . Empowered by GANs [ 11 ] , conditional generative adversarial networks ( cGANs ) [ 17 , 33 , 38 , 43 , 50 , 57 , 26 , 47 ] have demonstrated superior performance on image generation with the restriction of condition . Recently , conditional image generation with conditions on continuous labels has attracted attention , and new training schemes [ 8 ] are proposed to improve continuous conditional image generation . From the perspective of the dynamic of parameter subspace , our method provides a simple yet effective solution . Continuous image translation . Many works have discussed improving the diversity of generated images in image-to-image translation tasks [ 47 , 34 , 6 , 26 , 23 , 45 , 15 , 53 , 5 ] . However , continuous image translation that aims to generate intermediate images between the source and the target images is not yet widely studied . Some previous works [ 55 , 51 , 27 , 49 , 10 ] have manipulated the attribute vectors either by interpolation or linear transformations to continuously control the latent space . [ 16 , 25 ] employ multiple generative networks in the middle of the pathway from source to target image to generate intermediate results . [ 6 , 36 ] disentangle the latent space as content and style codes and interpolate the styles , in order to produce intermediate samples . However , all the aforementioned methods rather need to import auxiliary neural networks or carry extra burdens on model training by introducing additional regularization or supervisions . Moreover , synthesizing intermediate images without relying on interpolation of either domain labels , or disentangled representations , which can discourage the stochasticity of generated images , is still considered a big challenge . 3 Continuous Filter Atoms . In this section , we start with a brief review of neural ODEs and atom-coefficient filter decomposition , which serve as key ingredients in the proposed image generation approach . We then present in detail that , with the effectiveness being theoretically justified , how the proposed continuous atom generation using neural ODEs achieves gradual image generation . 3.1 Neural ODEs . Neural ordinary differential equations ( ODEs ) are introduced in [ 4 ] as an approach to interpreting the neural network as a system of ODEs , where each ODE represents an underlying dynamic of the hidden elements . Especially , neural ODEs are frequently adopted to enforce continuous dynamics in latent representations , for example , [ 19 , 32 ] propose to use neural ODEs to smoothly connect the latent space of video frames ; [ 37 ] adopts ODE-RNNs to model time-continuous hidden dynamics to handle irregularly sampled time-series data . Neural ODEs model a latent state z ( ts ) as z ( ts ) = z ( t0 ) + R ts t0 f ( z ( t ) , t ; ✓ ) dt , where dz ( t ) dt = f ( z ( t ) , t ; ✓ ) is modeled as a neural network parametrized by ✓ . Then , the latent state at an arbitrary point ts can be obtained as z ( ts ) = ODESolve ( z ( t0 ) , f , ( t0 , ts ) , ✓ ) , ( 1 ) where t0 and ts denote the start and the end of the integral interval , respectively . The simplest method for approximating the solutions of the ordinary differential equations with a given initial value is the Euler ’ s method , which is a first-order integrator with a fixed step size . In practice , families of Runge-Kutta methods , e.g. , the Midpoint method and fourth-order Runge-Kutta [ 22 ] , are preferred due to their superior convergence and stability . Note that all the experiments in this paper follow the implementation in [ 4 ] for Runge-Kutta of fifth-order of Dormand-Prince-Shampine with adaptive step size [ 9 ] . For any given integral interval ( ti , tj ) where i , j 2 { 0 ... T } , ODESolver always outputs a unique solution for the integral of the continuous dynamics , as long as the ordinary equation f is uniformly Lipschitz continuous in z and t [ 7 ] . Parametrizing f as a neural network equipped with Lipschitz continuous non-linear functions will meet the requirements . Thus , in our paper , modeling filter subspace using neural ODEs guarantees continuity as we will theoretically justify later . 3.2 Convolutional Filter Decomposition . Directly modeling the space of high-dimensional convolutional filters in an image generation network is practically prohibitive in terms of both computation and parameter scale [ 47 ] . Inspired by the observation that a convolutional filter can be well-approximated as a linear combination of filter bases [ 35 ] , as shown in [ 47 ] , a subspace view to the convolutional filters can be adopted in image generation network by decomposing each convolutional filter as a linear combination of low-dimensional filter atoms . Specifically , given an l ⇥ l-sized convolutional filter F 2 Rc⇥c0⇥l⇥l with c0 input and c output channels , respectively , we can decompose the filter over a set of m atoms D as F = AD , where D 2 Rm⇥l⇥l , and A 2 Rc⇥c0⇥m are the filter subspace coefficients . As we will show later , by modeling the continuity of a low-dimensional filter subspace using neural ordinary differential equations , we achieve efficient training and modeling of gradual appearance changes in conditional image generation . 3.3 Continuous Atom Generation for Smooth Appearance Modeling . In this paper , we aim for modeling gradual changes in generated images with respect to continuous input conditions . Instead of seeking new ways of training , we provide a unique view by modeling the transformation from an input condition to the corresponding output with parameters instantiated from an underlying continuous space , which subsequently leads to generated images with gradual changes . We adopt the aforementioned atom-coefficient decomposition , which permits low computational modeling of filter subspace as also validated in [ 47 ] . However , modeling a filter subspace with standard neural networks can still suffer from mode collapse often caused by sub-optimal network configurations , and the continuity is also difficult to be guaranteed by standard training as discussed in [ 8 ] . We therefore adopt a neural ODE based filter atom generator with the continuity property discussed in Section 3.1 , and show theoretically that such atom continuity can be propagated to the deep network outputs to achieve conditional image generation with gradual changes . Specifically , we decompose the i-th layer filters Fi into respective atoms Di and coefficients Ai , as Fi = AiDi . Over k layers of filters { Fi } k1 , we jointly model the corresponding atoms { Di } k1 using atom generation . We assume that all instantiations of filter atoms reside in a continuous space parametrized as D ( t ) where t is a time variable , i.e. , the generated atoms at the time point t can be expressed as D ( t ) . In the limit of a small step ✏ , the local dynamic within this atom space can then be expressed as D ( t+ ✏ ) = D ( t ) + ✏ · dD ( t ) /dt . Therefore , given an initial state of atoms within this space denoted as D ( t0 ) , and modeling the derivative of dD ( t ) /dt using a neural network f ( · , · ; ✓ ) : dD ( t ) dt = f ( D ( t ) , t ; ✓ ) , ( 2 ) any samples within this atom space can then be obtained by an integral of the function values of f from t0 to the desired ending point . The described dynamic leads naturally to a neural ordinary differential equation , where the instantiation at a time point ts , s > 0 , in this continuous dynamic can be obtained through a continuous atom generator T by D ( ts ) = T ( ts ; D ( t0 ) , ✓ ) = D ( t0 ) + Z ts t0 f ( D ( t ) , t ; ✓ ) dt . ( 3 ) Note that the initial state D ( t0 ) here are free parameters to be optimized in the training jointly with ✓ that parametrizes f . Starting from the initial state D ( t0 ) , an instantiation of filter atoms D ( ts ) is defined to be a unique solution to the ODE initial value problem at some condition ts , by Picard ’ s existence theorem . Consequently , as t varies , different instantiations of atoms are generated from this modeled continuous filter subspace , and our corresponding atom generator equipped with ODESolver learns the underlying continuous dynamics of the filter subspace . Given the subspace of filters modeled as an underlying continuous dynamic through neural ODE , we prove the Lipschitz-type continuity transducted from atoms to output features in the i-th layer . For simplicity , let c = c0 = 1 , and the argument extends . Given an input image X ( u ) ( u 2 U , U ⇢ Z2 ) , define the local input norm ||x||2 , Nu : = ( P u02Nu x ( u u 0 ) 2 ) 1/2 and the convolution hx , wiNu : =P u02Nu x ( u u 0 ) w ( u0 ) , where Nu ⇢ U is a local Euclidean grid centered at u . The i-th layer function F i : Xi 1 ! Xi can be expressed as , Xi ( u ) = F i ( Xi 1 , Di ; Ai ) = ( mX k=1 AikhXi 1 , Di , kiNu + bi ) , ( 4 ) where Di , k denotes the k-th atom in the i-th layer , Aik is the corresponded k-th coefficient , and b i denotes the bias at the i-th layer . Theorem 1 . Suppose Di1 and Di2 are two continuously generated atoms , ( i.e. , 9M > 0 , ||Di1 Di2||2 M |t1 t2| ) , and assume the activation function is non-expansive which holds for ReLU , then F i is continuous in Di , ||Xi1 Xi2||2 ( ||Ai||2 ) p |U| · || ( Di1 Di2 ) ||2 , with = sup u2U ||Xi 1||2 , Nu , ( 5 ) in which Xi1 = F i ( Xi 1 , Di1 ; Ai ) , Xi2 = F i ( Xi 1 , Di2 ; Ai ) are outputs correspond to two atoms . Proof . The proof is provided in the Appendix Section A . Notably , modeling the atom subspace of filters offers significant advantages on efficiency . Under the rule of neural ODE , the dimensionality of D ( t ) can not be varied , meaning the dimension of the input and the output of the ODESolver remain strictly consistent . This additional restriction makes modeling the space of full-size filters practically infeasible due to the computation and memory constraints . Meanwhile , by modeling the filter subspace instead , we significantly reduce the cost of learning the corresponding neural ODE . Typically , atoms across layers introduce a few hundred parameters only and significantly simplify the training and inference of atom generator T .

Summary: The authors build on previous conditional image generation models, which are based on conditional Generative Adversarial Networks (cGAN). The authors use the observation made by previous work [1, 2], that the kernel $F$ of a convolution layer can be replaced with a low-rank, two-component tensor factorization $F = AD$. $A$ is called the coefficient tensor, and has dimensions [channel_in x channel_out x m]. $D$ has dimensions [m x kernel_width x kernel_height] and represents the $m$ "atoms" of the convolution kernel.   The authors' goal is to devise a cGAN, such that the generator's output is a smooth function of its conditioning input. To this end, the authors propose to model "atoms tensor" $D = D(t)$ using a neural ODE, which turns the atoms into a smooth function of the introduced fictitious time variable $t$. Given some further mild requirements, the authors show that smoothness in the filter atoms induces smoothness in the outputs. Based on this theoretical contribution, they propose novel ways to design cGANs that can be conditioned on images, on some real-valued label or both.   The authors verify the effectiveness of their method on several different tasks, for which they achieve results that are comparable or exceed the state-of-the-art while using much fewer parameters than most comparable methods.  [1] Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. DCFNet: Deep neural network with decomposed convolutional filters. ICML 2018  [2] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. ICLR 2020