Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes a novel approach of decomposing convolutional filters into a set of filter atoms, which are modeled by neural ODEs. The authors show how the continuity allows generating gradually changing images. 

main_review: Defining the parameters via a composition of filter atoms and implementing these filter atoms via neural ODEs seems like a good idea, introducing continuity without extreme computational costs. The paper is a bit out of my domain of expertise, so I can’t comment much on novelty and evaluation.

Strengths
1. The paper is a pleasure to read. It seems very well organized. 
1. The model allows efficient training by leveraging a low-dimensional subspace of filter atoms. 
1. Gradual changes of generated images can be learned without the need for auxiliary modules or extra supervision.

limitations_and_societal_impact: Weaknesses/Questions
Table 4 doesn’t have the best performing models highlighted in bold.
Typos:
1. Line 39: stochasiticity
1. Line 330: graduate -> gradual
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper designs continuous filter atoms to enable gradual changes for image generation, by using the ordinary differential equation. This is a clever strategy that decomposes convolutional filters with atoms, and it is significant for the high-dimensional filters. Specifically, those continuous filter atoms are helpful to obtain smooth propagated image generation.  Various experiments on image to image translation and conditioned image generation verify the performances of the introduced filter atoms. Besides, the proposed filter atoms allow users to manipulate images in a new way that controls the integration interval. 
main_review: The idea of using continuous filter atoms from an ODE atom generator is very interesting and meaningful. I believe that it is a very effective and useful method for image generation. The experiments on conditional image generation show that filter atoms cover a wide range of gradually varying appearance and accurate correspondence to the input condition.  Therefore, it allows flexible appearance manipulation without supervision. Also, it can generate image appearance arbitrarily conditioned on continuous labels.
limitations_and_societal_impact: - The experimental setting can be improved. The quantitative evaluation is limited. The user study and ablation experiments are missing.

-Paper writing needs to be improved. For example:
- L.039, "stochasiticity" --> "stochasticity"
- L.063, "varying appearance" --> "varying appearances"
- L.066, "without heavy supervisions" --> "without heavy supervision"
- L.075, "has attract attention" --> "has attracted attention"
- L.189, "atoms along the learned continuous" --> "atoms along with the learned continuous"
- L.248, " get the desirable target distribution" --> " get the desired target distribution"
- L.275, "this smooth transitions" --> "these smooth transitions"
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors build on previous conditional image generation models, which are based on conditional Generative Adversarial Networks (cGAN). The authors use the observation made by previous work [1, 2], that the kernel $F$ of a convolution layer can be replaced with a low-rank, two-component tensor factorization $F = AD$. $A$ is called the coefficient tensor, and has dimensions [channel_in x channel_out x m]. $D$ has dimensions [m x kernel_width x kernel_height] and represents the $m$ "atoms" of the convolution kernel. 

The authors' goal is to devise a cGAN, such that the generator's output is a smooth function of its conditioning input. To this end, the authors propose to model "atoms tensor" $D = D(t)$ using a neural ODE, which turns the atoms into a smooth function of the introduced fictitious time variable $t$. Given some further mild requirements, the authors show that smoothness in the filter atoms induces smoothness in the outputs. Based on this theoretical contribution, they propose novel ways to design cGANs that can be conditioned on images, on some real-valued label or both. 

The authors verify the effectiveness of their method on several different tasks, for which they achieve results that are comparable or exceed the state-of-the-art while using much fewer parameters than most comparable methods.

[1] Qiang Qiu, Xiuyuan Cheng, Guillermo Sapiro, et al. DCFNet: Deep neural network with decomposed convolutional filters. ICML 2018

[2] Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. ICLR 2020
main_review: # Strengths
The core idea of the work, i.e. to model the filter atoms using a neural ODE, and then show that the smoothness from the atoms "propagates" to the model's output is a novel and nice, though perhaps somewhat unsurprising contribution.
In my view, the most significant contribution of the work is how this key result is utilised to elegantly allow conditioning on labels whose domain is a subset of the real line.
The methodology seems reasonable, and the authors perform a reasonable extensive empirical verification of their method, and achieve good results.

# Weaknesses
The writing suffers from several mistakes on many levels. It has several spelling and grammatical mistakes, incorrect word usage, incorrect, hard-to-parse and sometimes seemingly superfluous sentences. I also believe that some of the formulae are incorrectly stated.
To give a non-exhaustive list of examples:
- On several occasions, the word "graduate" is used incorrectly to mean "gradual", including in Fig 1.
- Lines 42-43: "First, this method is sensitive to network configurations, and mode collapse, which results in a point estimation to the subspace of parameter, can often be observed." - I cannot parse this sentence.
- Line 74: "[...] with the restriction of condition." - I am not sure what the restriction is supposed to mean here.
- Lines 76-77: "From the perspective of the dynamic of parameter subspace, [...]" - While later sections somewhat clarify what this means, this is extremely confusing at the beginning.
- Lines 99-101: "[16, 27] propose ..." - This sentence belongs to the Related Works section, not the Background section.
- Lines 147-155: The authors seem to explain neural ODEs, even though they have already done so in section 3.1
- Lines 166: I believe that in the definition of the local input norm we the terms in the sum should be $x(u')$ instead of $x(u - u')$, otherwise for every pixel $u$ its local norm would be the local norm around $x(0)$.

Beyond these mistakes, I found the general content layout to be confusing. It took several readings to understand the core idea of "propagating" the Lipschitz continuity from the filter atoms to the outputs, because of the poor explanation.

# Summary
In my opinion, this paper proposes and executes a couple of very nice ideas, which are unfortunately heavily overshadowed by the poor writing. If the authors can heavily improve the writing, I would be happy to recommend acceptance, though I believe this would require major restructuring of the paper.

# After the rebuttal
The authors have addressed my technical concerns, and after some discussion with the other reviewers, I have more confidence that the writing can be improved for the camera-ready version of the paper, hence I raised my score to recommend acceptance.

limitations_and_societal_impact: There is no discussion about limitations or social impact in the work. Including savings on the parameter count, or inference speed would have been useful. Some explanation and explorations of the theoretical consequences of Theorem 1 would be very useful. Comments on the stability of training and its scalability would be also important.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The presented paper deals with the problem of diversity in produced samples of image generation models. The proposed solution employs the recent framework of decomposing convolutional filters over a set of learned filter atoms. To make the results diverse, one multiplier of such decomposition is modeled with neural ODEs, thus allowing for continuous changes in the outputs of the generator (the related theorem was proved). Importantly, the solution may be integrated into many architectures as a drop-in replacement for standard convolutions.

The authors have conducted experiments on image-to-image translation and image generation with continuous labels. The reported results demonstrate that the ODE-driven filter decomposition leads to meaningful results.
main_review: 1. Strengths. 

    a. Neural ODE is a celebrated way of modeling time-continuous dynamics and it looks like a reasonable tool for generative models with non-discrete labels. Also, the framework of Filter Atoms is an interesting alternative for other common conditioning methods like AdaIN, weight (de)modulation, etc. Therefore, the motivation for this paper is pretty clear.

    b. The paper is well-written, all the concepts are explained in a self-contained manner. 

2. Weaknesses.

    a. First of all, I have some concerns about experiments from the section "img2img translation conditional on images" (edges -> handbags, etc., line 269). The naturalness of learning the mapping between a one-dimensional time variable and color/satellite style/etc is at least questionable to me. The approach of BasisGAN that models such styles as samples from a multivariate Gaussian seems more reasonable, and this method is well-established in img2img literature (e.g., see [1, 2, 3]). This is possibly why BasisGAN mostly outperforms the authors' model in this task (Table 1). Unfortunately, the authors do not show qualitative results for this problem other than edges -> handbags.

    b. The results on UTKFace and CACD2000, presented in the appendix, look not very convincing to me. The presented approach seems to perform not that well on human faces translation. As Table 3 claims the presented approach outperforms CycleGAN, it would be interesting to demonstrate also the CycleGAN outputs to compare the samples qualitatively.

    c. Why do the authors need to use discrete "distinct labels" (line 304). What exactly makes truly continuous labels "intractable" for the ODE-based approach?

    d. A lot of recent papers [4, 5] have shown the efficiency of positional embedding as a tool for feeding continuous values into the neural network. I think this could be a much simpler baseline aiming to check the usefulness of ODE for continuous conditioning: one may positionally encode the time variable with random or learned frequencies to a vector, then process it with some multi-layer perceptron and obtain a conditioning vector for AdaIN or weight (de)modulation [6]. Such a baseline may be used for experiments described at lines 278 and 312.

    e. The paper lacks hi-res experiments, unlike BasisGAN, which compared itself with the pix2pixHD approach for segmentation -> cityscapes problem. Is the ODE-based approach scalable? How is inference time affected with ODE solver being involved?

3. Suggestions.

    a. The recent CoMoGAN [7] paper also deals with the conditioning on a continuous one-dimensional time variable to model daytime light changes (among other tasks). Although this paper had not been released by the time of the NeurIPS deadline, I think that comparison with CoMoGAN would make the submission stronger.

4. Rating (pre-rebuttal)

    With all the pros and cons listed above, I tend to rate this paper below the threshold (score = 5). To my mind, more evidence to use exactly an ODE-based solution is required. Some of the results look not impressive making the quality of the model questionable. I ask the authors to address the concerns in the rebuttal. 

5. Updates (post-rebuttal)

    After reading the other reviews, as well as the authors' feedback, I would like to raise my score from 5 to 6. The main reason that prevents me from giving a higher rating is the lack of a baseline of the same type, i.e., able to perform continuous changes w.r.t. a `time` variable (e.g., one suggested in the review). Without such a comparison, the paper is just a proof of concept without any evidence that ODE is the best method for the considered tasks.


References
1. Huang et al. Multimodal Unsupervised Image-to-image Translation. ECCV, 2018.
2. Lee et al. DRIT++: Diverse Image-to-Image Translation viaDisentangled Representations. IJCV, 2020.
3. Anokhin et al. High-Resolution Daytime Translation Without Domain Labels. CVPR, 2020.
4. Tancik et al. Fourier Features Let Networks Learn High-Frequency Functions in Low Dimensional Domains. NeurIPS, 2020.
5. Sitzmann et al. Implicit Neural Representations with Periodic Activation Functions. NeurIPS, 2020.
6. Karras et al. Analyzing and Improving the Image Quality of StyleGAN. CVPR, 2020.
7. Pizzati et al. CoMoGAN: continuous model-guided image-to-image translation. CVPR, 2021.
limitations_and_societal_impact: I have no suggestions.
review_title: null

Meta-review: This paper proposes using continuous filter atoms (based on Neural ODE) for image generation.  These continuous filter atoms can enable smoother image synthesis compared to convolutional filters. The paper has received positive reviews. Many reviewers find the idea interesting and the results encouraging. The core idea is novel. There are concerns regarding the writing and baselines. The rebuttal addressed most of the concerns. The AC agreed with the reviewers’ consensus and recommended accepting the paper. The authors are encouraged to improve their writing according to the reviewers' comments. 
Decision: Accept (Spotlight)