Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summarize what the paper claims to contribute. 
The paper develops a spatio-temporal network that is defined in terms of continuous spatial functions and continuous temporal dynamics. The approach is meant to bring deep networks closer to biological neural models. The model is tested on CIFAR-10, and on a variation of CIFAR-10 in which blocks of pixels are blacked out. The methodological novelty consists of combining several existing approaches (continuous kernels, kernel-scale learning, and neural ODEs). 

List strong and weak points of the paper. 
Strong points:
-	I think the motivation is very strong. Deep convolutional networks are increasingly used in brain modelling, but they are somewhat disconnected from earlier computational neuroscience in ways that this paper tries to address. 
-	The pattern completion test with blacked-out pixels is a nice way to employ spatiotemporal dynamics in image recognition, and the results are promising. 
-	It is very interesting that the distribution of learned scales reflects the distribution of receptive-field sizes in primary visual cortex. 

Weak points: 
-	The feature-map evolution results in Fig. 4 are interesting, and a few more examples are given in the appendix, but it would also be nice to see mean +/- SD dynamics across many images, to complement Figure 3C more thoroughly. 
-	The model doesn’t outperform controls on CIFAR-10, although it does perform moderately better with blacked-out blocks of 6x6 pixels or more. 
-	Performance of the model and baselines on CIFAR-10 is not strong, which raises the question of how compatible the approach is with higher performance. 
-	The choice of CIFAR-10 as a test of the approach does not seem to be well motivated. A task with a temporal component might give the dynamic parts of the model more to do. 

Clearly state your recommendation (accept or reject) with one or two key reasons for this choice. 
I recommend to accept the paper. There is a growing body of work that compares deep CNNs to biological neural networks, and the conventions of discrete time and space in CNNs unfortunately distance this work somewhat from much previous computational neuroscience. I think the paper helps to close this gap. 

Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. 
-	Could you please expand on the motivation for continuous space with respect to biological vision? Ultimately vision is based on discrete photoreceptors, and generally on populations of discrete cells. Relatedly, the motivation for continuous time is more obvious, but I think it would also help to comment on spikes in this context. 
-	The dynamic equation in Eq. 2 seems to be autonomous. How does input affect h? 
-	In Section 3.2, the learned scales are shown to increase with network depth. This is compared with biological receptive fields, which grow through the visual hierarchy. But I had understood the scale to correspond to the kernel size rather than the receptive field size (which grows in deep networks even if all the kernels are the same size). Does sigma correspond to kernel or receptive field size? 

Provide additional feedback with the aim to improve the paper. 
-	Analytic tractability is mentioned on page 1 as an advantage of some continuous models in neuroscience, and it seems to be implied at that point that such benefits are sought in the paper, but it doesn’t seem that the approach ultimately offers much hope in this sense. If there is some potential here, please expand. 
-	I didn’t understand “… time (or network depth)” on pg. 4.  
-	Also on pg. 4, the sampling domain of the filter is given, but not the sampling frequency. 
-	I found section A.1 relatively hard to follow. In particular, Eq. 5 seems to be meant to motivate the filter family, but I didn’t follow the argument (or maybe I missed the point entirely). Also, aside from general interest I didn’t understand how the paragraph that contains Eq. 6 related to the rest of the paper. 
-	It wouldn’t hurt to define DOPRI. 
-	I didn’t follow the last paragraph of A.2. 

review_title: Nice bridge between deep learning and computational neuroscience.  

final_score: rating: 7: Good paper, accept
review_content: ## Review


### Summary

The authors define continuous deep networks by expressing 2D-convolutional filters as a linear combination of Gaussian function and its derivatives. By combining this description with the previously proposed neural ODE framework they obtain a spatio-temporally continuous description of a convolutional neural network.

There are 3 main contributions:  

1. they are able to estimate the support width of the filters and they show that  it increases with the network depth as observed in the visual cortex  
2. they show that their network performs as well as alternative non-continuous neural networks on CIFAR-10 while having less parameters  
3. they exploit the temporal dynamics to resolve a pattern completion task

Overall, I think the work is good but I am not as enthusiastic as the authors about the importance of the work for neuroscience and machine learning.

### Strengths

* The paper is well written and easy to understand. The goals and contributions of the work are clearly stated.
* The quantitative results are marginally good.
* The qualitative results (filter support width, pattern completion and contrast robustness) are interesting and relevant to neuroscience


### Weaknesses

* I fail to understand how the work could be relevant to neuroscience beyond what is presented here. What is so important about using spatially continuous filters that couldn't be done with discrete filters ?
* The pattern completion task is not fully conducted. It would be great to reconstruct the missing part of the input.
* The increase of filter support width with network depth correlates with what is known for the visual cortex but I fail to understand how it could be relevant to current work in experimental neuroscience. What is the benefit of learning continuously changing support for machine learning ? About the relation to biology, the increase in size might be more related to the specific task on which the network is trained than to what is observed in the visual cortex.
* The observed contrast robustness is not compared to other neural networks nor discussed in the light of experimental neuroscience observations.

### Minor comments

* The text in the figures is way too small. It should be the same size as the main text.


review_title: Review DCN

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

The authors propose a hierarchical model of neural ODEs, which they fit to CIFAR10. They find performance on par with ResNets, and include qualitative analyses on the filters learned by the models, their ability to fill in occluded features, and robustness to contrast at test time.


Strengths:

The filter parameterization is interesting. I can imagine this improving sample efficiency in certain contexts — perhaps the authors should seek out those kinds of tasks to complement their CIFAR results?

The discussion does a nice job of explaining the issues that neural ODEs have when scaling to large image datasets.


Weaknesses:

You spend time discussing spatio-temporal receptive fields throughout. Why? Your models are applied to 2d images.

The authors are missing a huge literature on (a) recurrent convolutional networks, and (b) using these networks to simulate classical vs. extra classical receptive field effects.

ResNet-Blocks is the original ResNet? Maybe change "blocks" to a citation? Or V1/V2 depending on which implementation it is (unclear from the text).

There's essentially no difference between the performance of any of the models tested. Is it possible to scale to ImageNet? It is important to show that the proposed method does *something* different than the standard ResNet. The authors attempted to add some qualitative experiments towards this goal in Fig 4, but those results are not very convincing. I think to show filling-in you'd want to show reconstruction in RGB space.

review_title: Review

Meta-review: This paper received 1 weak accept, 1 accept, and 1 weak reject.

All reviewers questioned the motivation for continuous space/time with respect to biological vision. Obviously, discrete approximations used in machine vision are approximations but it is not clear from the paper or the authors’ response that this severely limits the ability of deep nets to predict neural data in ways that their continuous nets would not. 

In addition, I have to confess that I did not really understand the argument made by the authors in their revision. In any case, the burden should be on the authors to go beyond general statements and to really demonstrate that the proposed models provide actual insights for neuroscience since the performance in terms of machine vision on CIFAR10 is underwhelming (the authors have to find a low data regime and even then the reviewers stated that the baselines used are not strong baselines, the reduction in the number of parameters is quite small relative to methods for actually reducing the number of parameters). 


Clearly, the work has potential as noted by the reviewers. The authors suggest that “DCNs can be used to model the temporal profiles of neuronal responses, which are known to not be constant even when the experimental stimuli are static images: for example, spatial frequency responses change over time in response to stationary gratings (Frazor et al., 2004). Similar observations are made for the contrast response function (Albrecht et al., 2002). Such temporal profiles cannot be simulated in conventional CNNs. “ This sounds like an interesting set of neuroscience data that the authors could be indeed leveraging to demonstrate the benefit of their approach. My recommendation would be to add those in a revision of this paper which will significantly strengthen the work. I would add that the concepts of temporal and spatial continuity are independent and the authors should consider studying them separately to provide more in-depth analyses and convincing results. 

As it stands, the paper has clear potential but it does not make a sufficient contribution to either ML or neuroscience and hence, I recommend the paper to be rejected.

Decision: Reject