Deep Continuous Networks
Authors: Nergis Tomen
Silvia Laura Pintea
Jan van Gemert
Venue:  (2021)
Keywords: continuous representations
neuroscience
convolutional neural networks
gaussian scale-space
learnable scale
receptive field size
neural ODEs
pattern completion

Abstract: CNNs and computational models of biological vision share some fundamental principles, which, combined with recent developments in deep learning, have opened up new avenues of research in neuroscience. However, in contrast to biological models, conventional CNN architectures are based on spatio-temporally discrete representations, and thus cannot accommodate certain aspects of biological complexity such as continuously varying receptive field sizes and temporal dynamics of neuronal responses. Here we propose deep continuous networks (DCNs), which combine spatially continuous convolutional filter representations, with the continuous time framework of neural ODEs. This allows us to learn the spatial support of the filters during training, as well as model the temporal evolution of feature maps, linking DCNs closely to biological models. We show that DCNs are versatile. Experimentally, we demonstrate their applicability to a standard classification problem, where they allow for parameter reductions and meta-parametrization. We illustrate the biological plausibility of the scale distributions learned by DCNs and explore their performance in a pattern completion task, which is inspired by models from computational neuroscience. Finally, we suggest that the continuous representations learned by DCNs may enable computationally efficient implementations.

Under review as a conference paper at ICLR 2021
DEEP CONTINUOUS NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
CNNs and computational models of biological vision share some fundamental principles, which, combined with recent developments in deep learning, have opened up new avenues of research neuroscience. However, contrast to bi- ological models, conventional CNN architectures are based on temportempp discrete representations, and thus cannot accommodate certain aspects of biolog- ical complexity such as continuously varying receptive field sizes and temporal dynamics of neuronal responses. Here we propose deep continuous networks (DCNs), which combine spatially continuous convolutional filter representations, with the continuous time framework of neural ODEs. This allows us to learn the spatial support of the filters during training, as well model the temporal evolution of feature maps, linking DCNs closely to biological models. We show demonstrate their applicability
that DCNs are versatile. Experimentally, standard classification problem, where they allow for parameter reductions and peetaaparametrizationn We illustrate the biological plausibility of the scale distri- butions learned by DCNs and explore their performance in a pattern completion task, which is inspired by models from computational neuroscience. Finally, we suggest that the continuous representations learned by DCNs may enable compu- tationally efficient implementations.
INTRODUCTION
Computational neuroscience and computer vision have a long and mutually beneficial history of cross-pollination of ideas (Sejnowski] 2020; Cox Dean 2014). The current state-of-the-art in computer vision relies heavily on deep neural networks (DNNs), and in particular convolutional neural networks (CNNs), from which multiple analogies can be drawn to biological eircuits (Kiet- zmann et al.] 2018). Specifically, recent advances in DNNs have enabled researchers to learn more accurate models the response properties of neurons in the visual cortex (Klindt et al.. 2017 Ca- dena et al. 2019 Ecker et al. 2019), well to test decades old hypotheses from neuroscience in the domain of computer vision (Lindsey et al.] 2019). However, contrary to biological models, CNNs typically operate in the domain of spatio- temporally discrete signals, and employ appropri- ately discretized kernels, as a natural part of digital image processing.
In computational neuroscience, the other hand, large scale neural network models of the visual system often adopt continuous, closed-form expressions to describe spatio-temporal receptive fields, as well the interaction strength between populations of neurons (Dayan & Abboti 2001]. Among others, such descriptions serve to limit the scope and parameter space of a model, by utilizing prior information regarding receptive field shapes (Jones & Palmer 1987) and principles of perceptual grouping (Li 1998). In addition, the choice of continuous and often tain some analytical tractability in complex models involving large number of coupled populations. Our approach draws inspiration from such computational models to propose continuous representa- tions of receptive fields in CNNs, where both the shape and the scale of the filters are trainable in the continuous domain.
sollytiifffufiiossssss help re-
In a complementary fashion, recent influential work in deep learning has introduced neural ordinary differential equations (ODEs) (Lu et al., 2018: Ruthotto & Haber 2019: Chen et al. 2018) which propose a continuous time (or depth) interpretation of CNNs. Such continuous time models both offer end-to-end training capabilities with backpropagation which are highly applicable to computer vision problems (e.g. by way of adopting ResNet blocks (He et al. (2066), ), as well as help bridge the gap computational biology where networks are often modelled as dynamical systems which

---

Under review as a conference paper at ICLR 2021
evolve according differential equations. In this work we aim to extend the impetus of the contin- uous time neural ODEs to the apatio-temporal domain.
To that end we introduce deep continuous networks (DCNs), which are spatio-temporally continuous in that the neurons have spatially well-defined receptive fields based on scale-spaces and Gaussian derivatives (Florack et al. 1996) and their activations evolve according equations of motion com- prising convolutional layers. We combine spatial and temporal continuity in a network with neural ODEs by learning linear weights for a set of analytic basis functions (as opposed to pixel-based weights), which can also intuitively be parametrized as a function of time, or network depth.
The following outlines our main contributions: (i) We provide theoretical formulation of spatio- temporally continuous deep networks building on Gaussian derivative basis functions and neural ODEs; (ii) We demonstrate the applicability of DCN models, namely, that they exhibit a reduction in parameters, and can be used to parametrize convolutional filters as function of time in straight- forward fashion, while achieving performance comparable with or better than ResNet and ODE-Net baselines; (iii) We show that filter scales learned by DCNs are consistent with biological observa- tions and we propose that the combination of our design choices for spatial and temporal continuity may be helpful in studying the emergence of biological receptive field properties as well as high- level phenomena such as pattern completion: (iv) We suggest that the continuous representations learned by DCNs may be leveraged for computational savings.
We believe DCNs can bring together two communities as they provide a test bed for hypotheses and predictions pertaining to both biological systems as well as pushing the boundaries of biologically inspired computer vision.
2 DEEP CONTINUOUS NETWORKS
2.1 NEUROSCIENTIFIC MOTIVATION
There is little doubt that modern deep learning frameworks will be conducive to effective and insight- ful collaborations between neuroscience and machine learning (Richards al. (019). In particular in vision research, CNNs are becoming increasingly popular for modelling early visual areas (Batty)
al. 2017 Ecker et al., 2019 Lindsey al. 2019). Here we propose model which can facili- tate such investigations by linking the end- end trainable but discrete CNN architectures with the biologically more plausible and
continuous computational models.
Structured receptive fields. Classical receptive fields (RFs) of cortical neurons display complex response properties with wide array of selectivity structures already at early visual areas (Van den Bergh et al.) 2010). Such response properties may also vary greatly based on multiple factors. For example the RF size (spatial extent) is known to depend on eccentricity (Harvey & Dumoulin 2011) and visual area (Smith et al. (00T) and may even change with depth within a cortical layer (Bauer jet al. 1999). Similarly, studies have shown that receptive field size and spatial frequency selectivity of neurons may co-vary with input contrast (Sceniak et al.) (2002).
Based on these observations, we aim to build a model which can accommodate the biological realism better than conventional CNNs, by explicitly modelling the RF size trainable parameter. To that end, we adopt a Gaussian scale-space representation for the convolutional filters, which we call structured receptive fields (SRFs) (Jacobsen et al. 2016). Previously, Gaussian scale-spaces have been proposed as a plausible model of biological receptive fields and feature extraction in low- level vision (Florack et al. 1992. Lindeberg & Florack 1994, Lindeberg 1993). Here, we are inspired partially by computational models which investigate the origin of response properties in the visual system, by employing RFs and recurrent interaction functions which scale as difference of Gaussians (Somers et al. 1995; Ernst et al.) 200T). Partially, we are motivated by the success of algorithms which utilize Gaussian scale-spaces (Loowe) 2004).
Neural ODEs. Studies have shown that both the contrast (Albrecht et al.) 2002) and spatial fre- quency (Frazor et al., (2044) response functions cortical neurons display characteristic temporal profiles. However, temporal dynamics are not incorporated into typical feed-forward CNN models. In addition, it has been suggested that lateral interactions play an important role in the generation of complex and selective neuronal responses (Angelucci & Bressloff 2006). Such activity dynamics are often computationally modeled using recurrently coupled neuronal populations whose activa- tions evolve according coupled differential equations (Ben-Ya Yishai et al. 1995 Ernst al. 2001
2

---

Under review as a conference paper at ICLR 2021
Fa(2: (cl.yy: y: 2.28)
U 0.90)
ii
dij
G
6
G
G
6
C
G
Go
GL.
Ghe
G
(1,
G (2,
Figure SRF filters based on N-jet filter approximation. Convolutional filters are defined as the weighted sum of Gaussian derivative basis functions up to order 2, with corresponding scales oi = 2.28 (left) and 02 = 0.90 (right). Our DCN models learn both the coefficients a, and the scale end-to-end during training.
For temporally continuous computational representations consistent with biological models, we adopt the framework of neural ODEs (Chen et al. 2018). Neural ODE interpretation ResNet mod- els presents opportunity to explicitly model the temporal dynamics of feature extractors in DNNs. Under certain assumptions, neural ODEs can be interpreted as recurrent interactions (Rousseau et al., 2019), which are biologically plausible (Liao & Poggio 2016). Unlike ODEs based pixel-based convolutional filters, the combination with structured filters (SRFs) also provide an intuitive way to parametrize the evolution of the kernels function of depth.
Ultimately, we see spatio-temporally continuous representations end-to-end trainable networks as link between modern CNN architectures and computational models of biological vision. Although for small spatio-lemporal scales it may be more appropriate to use discrete descriptions of biological neurons, such as photoreceptors populations located in spatially discrete locations, or consider temporally distinct spiking dynamics, computational models using continuous population activity or rate-based models provide reasonably good explanations phenomena observed at the network or systems level (Ben- Yishai et al., 1995, Dayan & Abbott 2001]. We believe such larger scale computational models align well with the purposes of computer vision.
2.2 STRUCTURED RECEPTIVE FIELDS
We the multiscale local N-jet formulation (Florack al. 1996) define the filters in convo- lutional layers. Structured receptive fields (SRFs) based on the Gaussian N-jet basis functions are highly applicable CNNs, as they represent Taylor expansion of the input image or feature maps in a local neighbourhood in space and scale, and can be used to approximate pixel-based filters (Appendix A.I. This means that each filter F(x, yi o)) in the network is a weighted sum of N basis functions, which are partial derivatives of the isotropic two-dimensional Gaussian function (="1=²)
The scale, or the spatial extent, of the filter is explicitly modelled in the a parameter of the Gaussian, which also indirectly determines the spatial frequency response of the SRF (Figure TT.
G(I, y: 2:.2 2.2
Formally, the N-jet formulation of an SRF filter F(x, y) is given by:
++j HJSN N Σ
N Σ
gits ai dr
F. F,
aig GLu (I.V: σ)
G (z. σ), (1)
051,05
050,551
where Gld (I, o) are the partial derivatives of the Gaussian G(a, σ) with respect to and y. is the degree of the Taylor polynomial which determines the basis order, and a encodes the expansion coefficients.
Defined this way, SRFs have favourable properties over pixel-based filters. SRF filters are steerable by the coefficients and the basis functions are spatially separable. Likewise, due their spatially continuous description, the filters can be trivially scaled, or rotated, without interpolation.
Figure T shows the N-jet approximation of two filters in two different scales di and Ø2. We note that both the coefficients a and the scale o are learnable filter parameters. Instead fixing the scale o priori and optimizing for as in Jacobsen al. (2016) and Sosnovik et al. (2020). integrate

---

Under review as a conference paper at ICLR 2021
both these parameters in the network optimization, thus learning not only the shape but also the spatial support the filters.
2.3 NEURAL ODES
We model the continuous evolution of feature maps within an 'ODE block'. Formally, an ODE block contains a stack of M convolutional layers. each with its convolutional filters w with m = 1......4 M. followed by normalization Gnorn (.) and non non-linear activation CELU(.) functions. Following the notations ofeh et al. (0018) and Ruthotto Haber (019)) we define the equations of motion for the feature states h E R as:
dh((t) f w d Gnorm [g(( (e(ww))((Kg) (w + Jd('))t) + (2dd
(2)
dt
where (x) = CELU Gavem (x)). the linear operators Km E TRxx denote the convolution operators parametrized by w and Jm E IR" x IR denote the linear explicit t-terms parametrized by dm The filters w d (0) are functions of some learnable parameters 0. As typical CNN convolution operator, K., (w")) with 2 input and 2 output channels can be written as
K (w
12
(3)
2,2
with Wi the convolutional kernels for input channel and output channel j of the m-th convolution. In this definition, the discrete depth of feed-forward networks such ResNets is reimagined continuous dimension defined by time t, where the input image defines the initial conditions h()). For the rest of this paper, we stick to the original ODE-Net interpretation that the number of function evaluations performed by the numerical ODE solver is analogous to network depth.
In accordance with conventional ResNet blocks, we pick M 2. Based on the original imple- mentation by Chen et al. (2018). Gnorn is defined as group normalization (Wu & He 2018). For generalized compatibility with neural ODEs and the adjoint method, we choose a non- acti- vation function with theoretically unique and bounded adjoint, namely continuously differentiable exponential linear units, or CELU (Barron 2017). Similarly, we keep the linear dependence of the equations of motion ODE solver to solve the equations of motion for predefined time interval / E [0, using the adaptive step size DOPRI method.
time (or network depth) t. Finally, we adapt the GPU implementation of
2.4 DEEP CONTINUOUS NETWORKS WITH SRFS AND NEURAL ODES
We formulate deep continuous networks (DCNs) by employing learnable, continuous SRF filter descriptions to define the weights in the evolution of a neural ODE. This means that for DCNs, each wy in Eq. 2 is discretization of the continuous SRF filter (a, y/ a)) given in Eq. sampled in [-2o, (2). We define the spatial axes such that the width and height of a pixel is 8_, 8, - and the sampling rate is (8_") ). ajt and "t trainable filter parameters, where oj; 0t is shared between the filters in a convolutional layer unless stated otherwise. All our code available af
Network architecture and training. We construct DCNs by stacking ODE blocks separated by downsampling blocks (Figure 2). Each downsampling block is sequence of normalization, non- linear activation and strided convolution. We use a convolutional layer for increasing the channel dimensionality at the input level and employ global average pooling and a fully connected layer at the output level. We train our networks using cross entropy loss and the CIFAR- dataset (Krizhevsky 2009). (See Appendices A.3 for further details regarding training parameters.)
As baseline without spatial continuity, compare DCN performance to the ODE-Net introduced in Chen et al. (2018), where the convolutions within the ODE blocks are performed using discrete, pixel-based kernels, with & x k parameters. As baseline without temporal continuity, we define the 'Resokttblockk' model where the ODE blocks are replaced by generic, discrete ResNet blocks, comprising two convolutional layers and skip connection, with comparable number of parameters
'https:/github.com// com//tticcccn/oooccfffff//////// "cdee will be made available github
4

---

Under review as a conference paper at ICLR 2021
to the ODE-Net. This is similar to the baseline model used in Chen et al. (2018). In the ResNet-SRF. blocks model, we provide the discrete-ting secrrtt-timeeeeee and continuous- space baseline by replacing the filter definition of ResNet-blocks with SRF definitions.
We test two versions of DCNs and BesNet-SRF-bloche to quantify the viability of SRF filters outside of the ODE blocks. In DCN-ODE and ResNet-SRF-blocks we use the SRF filters only within the ODE (ResNet) blocks, and for the remaining layers we use discrete kernels with the same hyperparameters as the baselines. In the second version, DCN-full (ResNet-SRR-fulle SRF-full), we use spatially continuous kernels ev- erywhere, including the downsampling layers.
Input Image (32x32x3)
look conv,
ODE Block 1, 32
Downsampling block 1, 64
ODE Block 64
Finally, we investigate the case where we drop scale sharing within layer, and optimize the scale parameter oj independently for each input channel and output channel j. which we call DCN dzz-
Downsampling block 2, 128
ODE Block 3, 128
Meta- cesta-aaramtrizatioon DCNs enable us to parametrize the trainable filter parameters a and a as a function of time This both enables the kernels to vary smoothly over time, and lets us define temporal dy- namics for the neuronal responses in our network. We test the viability of such models by introducing DCN variants where and/or are de- fined using linear or quadratic functions of t and learnable parameters b, G, ags by Ga and ba (Table 3)
normalization
nonlinear activation
Global Average Pooling
fc, 10
Figure 2: DCN model ar- chitecture with CIFAR-10 input images. Downsam- pling is performed via convolutions with stride 2. The convolutional kernel size k depends model, and is learned during training in DCNs. The equations of motion (Eq. 2 are solved within ODE blocks.
3 EXPERIMENTAL ANALYSIS
3.1 PARAMETER REDUCTION AND DATA EFFICIENCY
the
Similar to biological models, where analytical receptive fields limit the scope of the model using prior information, find that DCNs are more parameter efficient compared to baseline networks. Evaluated on CIFAR-10, DCNs perform on par with baselines, despite using SRFs of a small basis order 2, which means each filter shape is defined by only 6 free parameters opposed 9 for the conventional x 3 ker- nels (Table [ In addition, we find that parameter reduction via the use of SRFs with small basis order also leads to data efficiency. When trained on subset of CIFAR-10 images (small-data regime), DCNs outperform the discrete baseline networks (Table 2a).
As an additional demonstration of the versatility of the spatio-temporal representations learned by DCNs, we perform an image reconstruction experiment. We use the feature maps generated by the networks trained on the CIFAR- classification task (output of ODE Block in Fig. 2). as input to a decoder network. The decoder networks are composed of 2 DCN-ODE, ODE-Net or ResNet blocks, separated by bilinear upscaling layers and 1 x 1 convolutions to reduce the output dimensionality
Continuity
Model
Accuracy (%)
# Parameters
Spatial
Temporal
ResNet-blocks
89.01 + 0.17
555K
ResNet-SRF-block
88.26 1 0.03
426K
ResNet-SRF-full ODE-Net (Chen et al. 2018)
V
x
89.32 1 0.39
323K
X
89.60 + 0.28
560K
DCN-ODE
Y
89.46 1 0.16
429K
DCN-full
89.18 + 0.32
326K
DCN 034
89.74 = 0.30
472K
Table 1: CIFAR- 10 validation accuracies of DCN models, averaged over 3 runs, compared base- line models. ODE-Net and ResNet-blocks baselines introduced in Chen et al. (2018). DCNs perform on par with spatially and/or temporally discrete basclines, despite having a lower number of trainable parameters.
5

---

Under review as a conference paper at ICLR 2021
Model
# images per class
Model
Reconstruction Loss (%)
52
103
512
1024
ResNet-blocks
39.8 = 0.6
49.0 + 0.2
70.4 1.2
76.8 - 0.7
ResNet-blocks
21.0 = 0.4
ODE-Net
41.7 = 1.2
48.6 + 0.5
71.7 1.5
77.4 17.4:00 0.5
ODE-Net
20.2 = 1.3
DCN-ODE
44.5 = 0.8
54.2 0.8
75.5
1 0.8
79.7 = 0.3
DCN-ODE
17.1 = 0.3
(b)
Table 2: (a) Validation accuracies on a subset of CIFAR-I0 (small-data regime) for the DCN-ODE model versus baselines when the training data is limited. The DCN model outperforms spatially and/ or temporally discrete baselines as parameter efficiency leads to data efficiency. (b) DCNs achieve
lower MSE loss in the reconstruction task than baselines on the CIFAR-10 validation set, despite using a smaller number of parameters. All results are averaged over 3 runs.
to three (RGB). We train the networks to reconstruct CIFAR-10 images using mean squared error (MSE) loss. We find that the DCN models outperform discrete baseline models on the validation set (Table 2)), despite having lower number of parameters as before. Additional details and example images are shown Appendix A.4
Similarly, we find that meta-paramaen parametrized DCN variants match the classification performance of baselines and may outperform DCNs with static weights (Table 4 This is an interesting finding since we test only few models, with little hyperparameter optimization, which indicates that DCNs can potentially be used parametrize the dependence of convolutional kernel weights on network depth, for further parameter reduction.
3.2 LINK WITH BIOLOGICAL MODELS
Scale fitting. As advantage conventional CNNs, it is possible to directly investigate the optimal receptive field (RF) size each DCN block after training, since DCNs fit the kernel scale explicitly. We observe upward trend in the SRF scale with the depth of the convolutional layer within the network (Fig. Bb). While the RF size grows with depth also in conventional CNNs, typically grows linearly over convolutions, as the kernel size constant across layers. This is limitation in most CNNs that the visual system does not necessarily have. DCNs, on the other hand, can learn RF sizes which grow non-linearly the behaviour in downstream visual areas (Smith et al. 2001). In addition. we plot the distribution of learned 5; in different ODE blocks of the model DCN oj (Fig.3)) Note that the scale parameter controls the bandwidth of the SRF filters and is thus related to their spatial frequency response. We find that the o; distributions after training are approximately log-normal and display a positive skew, which is consistent with the scale and spatial frequency tuning distributions in the primate visual system (Yu et al. 2010). We believe these results are promising for bridging the gap between deep learning and traditional models of biological systems.
function of depth, which seems to be in line with
Pattern completion. Established models from computational neuroscience, with continuous tem- poral dynamics and well-defined recurrent interaction structures, such as the Ermentrout-Cower model (Bressloff et al. (001), or neural field models (Amari 1977). display interesting high-level phenomena such spontaneous pattern formation and travelling waves (Coombes) 2005). Such models employ local, distance-dependent interactions, similar to the SRF based ODE blocks in the DCN formulation. Based on this resemblance, we hypothesize that DCNs can perform well in the case of locally missing information in images, through pattern completion at the feature map level.
Model
Parametrization
Model
Accuracy (%)
DCN (t) DCN (12)
20t+b
DCN (t)
89.97 + 0.30 89.93 = 0.28 89.88 +L 0.25
2at2+bt+ +b++c
DCN (t)) DCN (() and a(t)
DCN (t),
20, aut ba
Table 3: peramerrrattiiizaioon of filter parame- ters a and a as function of time in different DCN variants.
Table 4: CIFAR-10 validation accu- racies for DCN models with meta- parametrization.

---

Under review as a conference paper at ICLR 2021
DCN-ODE
3500
block conv1
Validation Accuracy (%)
90
1.6
DCN-full
block 1: conv2
ResNett SRF blocks
block 2: conv1
1,4
Reidott-SRF-N
fiiters
2500
2: conv2
80
block 3: convl
1.2
2000
block
1.0
#
1500
70
DCN
1000
ResNet-blocks
0.6
500
baseline
mask 4x4
mask 6x6
mask 8xB
mask 10:10
mask 12x12
Black mon
0.4
Block I. come
block 2: covv
Baok zi comv
plak x comr
block 31 curorr
2
Learned scale
(b)
(c)
Figure 3: (a) Learned values increase with depth within the network. (b) ojs distributions within the ODE blocks display a positive skew in line with biological observations. (c) CIFAR-10 valida- tion accuracies on the pattern completion task with increasing mask size.
We test this hypothesis by masking n X n pixels on the CIFAR-10 validation set at test time. The masks have pixel values, and are placed the center of the image. We find that when confronted with a small patch missing information, DCNs can generate feature maps similar to those obtained
from intact images. Specifically, we observe that the difference D(t) = 1 Σ hh' between the feature maps generated by an intact image (t) and masked image h im..as (t), normalized by the amplitude of the intact image A, is reduced within an ODE block (Figure 4b) In terms of the overall classification performance with masked images test time, we find that DCNs are marginally more robust against zero-masking than baselines (Fig. 3P).
masked (t)
3.3 CONTRAST ROBUSTNEESS AND COMPUTATIONAL EFFICIENCY
The selectivity of neuronal responses is invariant to contrast in mammalian vision (Sclar & Freeman 1982 Skottun et al., 1987). However, we observe that DCN and ODE-Net models are sensitive to changes in input contrast. This is not unexpected since ODE blocks compute the solution to the initial value problem posed by the equations of motion and the input feature maps, and contrast variations change the initial conditions. To quantify this sensitivity we vary the contrast e of the input images at test time, where for each image H; in the CIFAR-10 validation set we define the input as H, cH,, When naively changing the input contrast this way, we find that the validation accuracy decays rapidly for both models (solid lines in Fig. 4p, top).
Empirically, we notice that, with the appropriate choice of normalization functions, the input contrast has direct effect on the time scales of the solution h(t). Based on this observation, we heuristi- cally test whether scaling the integration time interval T (used during training) of ODE block by e for each input image T = eT, can improve contrast robustness at test time. We find that with the scaled integration interval DCN validation accuracy relatively robust against changes in contrast compared to naive baselines and ODF-Net, until << 1 when time scales become too fast and the ODE solver becomes unstable for all models (dashed lines in Fig. 4p, top).
Interestingly. we observe reduction in the number of function evaluations (NFEs) in ODE block for e < (Fig. 4p, middle). Furthermore, we show that as long as the error tolerance of the ODE blocks are not decreased, this effect can be exploited by scaling the input feature maps of all ODE blocks by с for significant computational savings. We find that decreasing c leads to considerable efficiency improvements, where total NFEs be reduced from 102 to 60, (for c and 0.06), with less than 0.5% loss in accuracy (Fig. 4b, bottom).
RELATED WORK
Spatially continuous filter representations. Structured filters have been traditionally used in com- puter vision for extracting image structure at multiple scales. N-jet filter basis is first introduced by Florack et al. 1996) based on previous work Gaussian scale-spaces (Florack et al. 1992 Lin-
deberg 2013). We use the N-jet basis in order to approximate convolutional filters, as they enable a spatially continuous representation with explicit scale parameter d.
Similar to the N-jet basis, a set of oriented multi-scale wavelets, called a steerable pyramid, is proposed by (Simoncelli et al. (1992) and complex wavelets have been used by Bruna & Mallat
7

---

Under review as a conference paper at ICLR 2021
Input image
masked Image
80
hl)(()
h mm (
D(t)
40
20
EdB survel.
DCE block
3.30
DCN
(7)
h in (n (7)
((t)
1.0-
NEE
708
ODE
300
0.5
Contrast Level
Integration time
(a)
(b)
Figure 4: (a) Pattern completion in the DCN feature maps for an example image. Feature maps are shown for a single channel in ODE block 1. We find that the difference ((1) between the feature maps htdh (t) of an intact image and h iin.masked() (t) of masked image is reduced as f + T. We also show the mean ((1) for 1000 validation images (bottom right), where the shaded area is the standard deviation over different images. Example feature maps from other models are provided in Appendix A.5 (b) Top: In terms of CIFAR- 10 validation performance, DCNs are more robust than baseline ODE-Nets changes in input contrast at test time. Interestingly. the number of functionn evaluations (NFEs) in the first ODE block (middle) or the whole DCN network (hottom) can be reduced considerably by modulating c.
(2013) and Mallat (0122) as part of scattering transforms. CNN filters based on linear combinations of Gabor wavelets adopted by Luan al. (2017), while Worrall al. (2017) propose circular harmonics, as spatially continuous filter representations.
Similar to our approach, |Shelhamer et al. (2019) combine free-form filters with Gaussian kernels, thus learning the filter resolution. Likewise, Xiong et al.] (2020) learn filter sizes using Gaussian kernels optimized using variational inference. Here, we use the N-jet framework based on Gaussian derivatives as in InDoabbsen et al. (2016), however our main motivation is retaining compatibility with biological models. Also, unlike unlike|Jcooben Jacobsen al. (2016) we learn the scale parameter a during training.
Continuous time representations in deep networks. Along with work by Lu et al. (2018) and Ruthotto & Haber (209), networks continuous in the temporal (or depth) dimension have been proposed by CChn et al. (20T8) under the name neural ordinary differential equations (ODEs). They propose ODE-Nets based on the ResNet formulation (He et al. 2016) for classification tasks, which used as a baseline in this paper. In this work we focus mainly on an image classification task, how- ever, there is extensive ongoing work on generative models and normalizing flows using the neural ODE continuous time interpretation (Salman et al., 2018 Grathwohl et al.) 2019). We note that DCNs can be readily incorporated into continuous flow models, as well as other spatio-temporally continuous DNN interpretations based partial differential equations (Ruthotto & Haber 2019).
Even though the adjoint method described in Chen et al. (2018) offers considerable computational savings, especially terms of memory, recent work has improved upon both in terms of stability, computational efficiency and performance (Dupont et al. 2019 Finlay et al. 2020 Zhuang et al., 2020b). Likewise, the contrast robust formulation of DCNs, well as the synergy between the O(T memory complexity of the adjoint method and spatially separable SRF filters (the implementations of which may otherwise inflate the memory cost) provide potential computational benefits over conventional CNNs where the number of function evaluations is fixed.
Other studies have suggested that, similar to our DCN variants where the filter definitions are inde- pendent of time, neural ODEs based on ResNet architectures with weight sharing can be interpreted as recurrent neural networks (Rousseau et al. 2019 Kim et al., 2016), which bridges the gap be- tween deep learning, dynamical systems and the primate visual cortex (Massaroli al. 2020 Liao & Poggio 2016). Similar to these works, we capitalize on the parallels between neural ODEs and the dynamical systems approach of the computational models of recurrent, biological circuits in order to extend them to fully continuous networks, where not only the depth of the network is continuous but also the shape and spatial resolution of the filters are end-to-end trainable.
8

---

Under review as a conference paper at ICLR 2021
CNNs and RNNs models of biological networks. There is extensive prior work CNNs and recurrent neural networks (RNNs) for modeling biological computation. The visual cortex is highly recurrent (Dayan & Abbott 2001 Liao & Poggio (016) which is thought to be responsible for complex neuronal dynamics (Ben- Yishai et al. 1995; Angelucei & Bressloff 2006). Accordingly,
computational models with lateral connections (Sompolinsky et al. [988] Ernst et al. 200T) and
more recently RNNs (Laje & Buonomano, 2013; Mante et al., 2013 Mastrogiuseppe & Ostojic 2018) have been extensively used as models of biological neural computation. For example the first- order reduced and controlled error (FORCE) algorithm. have been used to reproduce the dynamics of different biological circuits (Sussillo & Abboti, 2009 Laje & Buonomano/ 2013] Carnevale et al.)
2015 Rajan et al. 2016 Enel et al. 2016). Similarly, optimization via gradient-based algorithms such as the Hessian- free method (HF) or stochastic gradient-descens (SGD) have been adopted to train recurrent networks (Mante et al. 2013] Barak et al.) 2013; (ong et al.] (016) to replicate experimental observations. It has also been suggested to use spiking recurrent networks (Kim & Chow 2018 Kim et al., 2019) and incorporate synaptic dynamics (Ba et al. 2016; Miconi et a
2018) for improved phystological realism. Finally, recurrent convolutional networks (RCNNs) have been proposed (Liang & Hu 2015 Spoerer et al. 2017 Hu & Mihalas 2018), which can emulate biological lateral connectivity structures and extra-classical receptive field effects.
In contrast to typical RNNs, our model is based on the ResNet inspired model of neural ODEs, and in its current form (Eq.. 2. does not accept time-variant input. In that sense, the spatio-temporal dynamics of DCNs refer to the dynamics of the neurons and not the input. Although with weight sharing DCNs can be thought of as recurrent networks (Rousseau et al. 2019) and can be modified to process time-variant input (such as videos), in this paper we consider the DCN models to be an extension of conventional feed-forward CNNs, with extended temporal dynamics and continuous spatial representations, which are applicable to feed-forward models of the visual system similar to works by Lindsey et al. (2019): Bcker et al. (2019); Schrimpf et al. 20188; Zhuang et al. (2020a).
5 DISCUSSION
We introduce DCNs, CNN models which learn spatio-temporally continuous representations, con- sistent with biological models. We showed that DCNs can match baseline performance in an image classification task and outperform baselines in the small-data regime and in reconstruction task, while using smaller number of parameters. Similarly, we have proposed different methods of meta- parametrization of the convolutional filter as a function of time, which may not only be viable for network compression purposes, but also for modelling the temporal profiles of biological responses. As a further link with biological models, we have demonstrated that the learned scale distributions in DCNs are compatible with experimental observations, which seems promising for the applica- bility of DCN models to future neuroscientific investigations regarding the emergence of RF sizes. In addition, we have presented the capability of DCNs to reduce errors in feature maps caused by masking. Finally, we have empirically shown an interplay between the input contrast to ODE blocks and the time scales of the solutions, which can be capitalized on for computational savings.
However, one of the biggest limitations of DCNs is that they may become unstable during training. This is a combined problem of neural ODE models and scale fitting, which may lead exploding filter sizes with larger learning rates. Especially for meta- metaa-prametrizztion it would be advisable to clip the integration time and filter parameters within reasonable range. Nevertheless, we believe there are exciting future research opportunities involving DCNs. A rigorous mathematical analysis of the nonlinear system of ODEs is beyond the scope of this paper, however, neural ODE formula- tions provide interesting opportunities for establishing a theoretical understanding of DNNs based on dynamical systems. The interplay of input contrast and integration time is one such observation which requires further scrutiny. Similarly, our choice of filters based on well-behaved Gaussian derivatives allow for further analytical investigations, unlike conventional CNNs.
Similarly, DCNs offer interesting possibilities for biological modelling. Although there is still large gap between modern CNNs, DCNs and the complex biological realism, the inbuilt smooth evolution of filters in DCNs can be used, for example, to incorporate response dynamics such as synaptic depression or short-term potentiation (similar to previous work by Ba et al. (2016); Miconi et al. (2018)). Likewise, the equations of motion can be modified to reflect axonal delays generate oscillations. In short, we believe by offering a link between dynamical systems, biological models and CNNs, DCNs display an interesting potential to bring together ideas from both fields.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Duane G Albrecht, Wilson S Geisler, Robert A Frazor, and Alison M Crane. Visual cortex neurons of monkeys and cats: temporal dynamics of the contrast response function. Journal of neurophys- iology, (8(2):888-913) 2002.
Shun-ichi Amari. Dynamics of pattern formation in cybernetics, 27(2):77-87, 1977.
type neural fields. Biological
Alessandra Angelucci and Paul Bressloff. Contribution of feedforward, lateral and feedback connections the classical receptive field center and extra-classical receptive field surround of primate v1 neurons. Progress in brain research, 154:93-120, 2006.
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin lonescu. Using fast weights to attend to the recent past. Advances in Neural Information Processing Systems, pp- 2016.
Omri Barak, David Sussillo, Ranulfo Romo, Misha Tsodyks, and LF Abbott. From fixed points to chaos: three models of delayed discrimination. Progress in neurobiology, 103:214-222, 2013.
Jonathan T Barron. Continuously differentiable exponential linear units. CoRR, 2017.
Eleanor Batty, Josh Merel, Nora Brackbill, Alexander Heitman, Alexander Sher, Alan Litke, EJ Chichilnisky, and Liam Paninski. Multilayer recurrent network models primate retinal ganglion cell responses. ICLR, 2017.
Ute Bauer, Michael Scholz, Jonathan B Levitt, Klaus Obermayer, and Jennifer Lund. A model for the depthede of receptive field size and contrast sensitivity of cells in layer 4c of macaque striate cortex. Vision research, 39(3):613 1999.
Rani Ben- Yishai, R Lev Bar-Or, and Haim Sompolinsky. Theory of orientation tuning in visual cortex. Proceedings of the National Academy Sciences, (094)384443845 1995.
Paul C Bressloff, Jack D Cowan, Martin Golubitsky, Peter J Thomas, and Matthew C Wiener. Geo- metric visual hallucinations, euclidean symmetry and the functional architecture of striate cortex. Philosophical Transactions of the Royal Society of London. Series Biological Sciences, 356 (1407):299-33)) -300, 2001.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. TPAMI, 35(8):1872 1886, 2013.
Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque V1 responses to natural images. PLoS computational biology, 15(4):e100689, 2019.
Federico Carnevale, Victor de Lafuente, Ranulfo Romo, Omri Barak, and Néstor Parga. Dynamic control of response criterion in premotor cortex during perceptual detection under temporal un- certainty. Neuron, 66(4):1067-1077 2015.
Ricky T. Q Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary dif- ferential equations. NeurIPS, 2018.
Stephen Coombes. Waves, bumps, and patterns in neural field theories. Biological cybernetics, 93 (2):91-108, 2005.
David Daniel Cox and Thomas Dean. Neural networks and neuroscience-isspreed computer vision. Current Biology, 24(18):9911-R92, 2014.
Peter. Dayan and F. Abbott. Theoretical neuroscience computational and mathematical model- ing of neural systems. Massachusetts Institute of Technology Press, 1st edition, dec 2001. ISBN 0262041995
John R Dormand and Peter J Prince. family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
10

---

Under review as a conference paper at ICLR 2021
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural ODEs. In NeurIPS, pp- 3140-3150, 2019.
Alexander S Ecker, Fabian H Sinz, Emmanouil Froudarakis, Paul G Fahey, Santiago A Cadena, Edgar Y Walker, Erick Cobos, Jacob Reimer, Andreas S Tolias, and Matthias Bethge. A rotation- equivariant convolutional neural network model of primary visual cortex. In International Con- ference on Learning Representations, 2019.
Pierre Enel, Emmanuel Procyk, René Quilodran, and Peter Ford Dominey, Reservoir computing properties of neural dynamics in prefrontal cortex. PLoS compulational biology, 2(6))e100496) 2016.
UA Ernst, KR Pawelzik, C Sahar-Pikielny, and MV Tsodyks. Intracortical origin of visual maps. Nature neuroscience, 4(4):431-436, 2001.
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How train your neural ODE. arXiv preprint
2020.
Luc Florack, Bart Ter Haar Romeny, Max Viergever, and Jan Koenderink. The gaussian scale-space paradigm and the multiscale local jet. IJCV, 18(1):61-75, 1996.
Luc MJ Florack, Bart M ter Haar Romeny, Jan Koenderink, and Max A Viergever. Scale and the differential structure of images. Image and Vision Computing, 10(6))376-388, 1992.
Robert A Frazor, Duane Albrecht, Wilson Geisler, and Alison M Crane. Visual cortex neurons of monkeys and cats: temporal dynamics of the spatial frequency response function. Journal of Neurophysiology, 91(6):2077-2627 2004.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. ICLR, 2019.
Ben M Harvey and Serge Dumoulin. The relationship between cortical magnification factor and population receptive field size in human visual cortex: constancies in cortical architecture. Journal of Neuroscience, 31(38):36044 136611 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings the IEEE international conference on computer vision, pp. 1026-10344 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Brian Hu and Stefan Mihalas. Convolutional neural networks with extra-classical receptive fields. CoRR, abs/1810.1 1594, 2018. http:/// /ttp:///axvvvvv.a owg./rgg//......... abs////b///t /tabbbs/81l 11594
Jorn-Henrik Jacobsen, Jan Gemert, Zhongyu Lou, and Arnold WM Smeulders. Structured receptive fields in CNNs. In CVPR, pp. 2610-2619, 2016.
Judson P Jones and Larry A Palmer. An evaluation of the two-dimensional gabor filter model of simple receptive fields cat striate cortex. Journal of neurophysiology, 58(6):1233-11)) 1258, 1987.
Tim Kietzmann, Patrick McClure, and Nikolaus Kriegeskorte. Deep neural networks in computa- tional neuroscience. BioRxiv, pp. 133504, 2018.
Christopher M Kim and Carson C Chow. Learning recurrent dynamies in spiking networks. Elife, 7:e37124, 2018.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1637-1645, 2016.
Robert Kim, Yinghao Li, and Terrence J Sejnowski. Simple framework for constructing functional spiking recurrent neural networks. Proceedings of the national academy of sciences, 116(45): 2281 1-22820, 2019.
11

---

Under review as a conference paper at ICLR 2021
David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identifica- tion for large populations separating "what" and "where". Advances in Neural Information Processing Systems, pp. 1550--316,, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical University of Toronto, Department of Computer Science, 04 2009.
Rodrigo Laje and Dean V Buonomano. Robust timing and motor patterns by taming chaos in recurrent neural networks. Nature neuroscience, 16(7):925-933, 2013.
Zhaoping Li. neural model of contour integration the primary visual cortex. Neural Computa- tion, 10(4))903-944 1998.
Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In /EEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3367-3375, 2015. doi: 10. 1109/CCPR.2015.729855 URL httph:tp///oooo org / 1tt///10..11//PPPP11 2215.7298958
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint arXiv: 160.006000 2016.
Tony Lindeberg. Discrete derivative approximations with scale-space properties: A basis for low- level feature extraction. Journal of Mathematical Imaging and Vision, 3(4):349-376, 1993.
Tony Lindeberg. Scale-space theory in computer vision, volume 256. Springer Science & Business Media, 2013.
Tony Lindeberg and Luc Florack. Foveal scale-space and the linear increase of receptive field size as a function of eccentricity. KTH Royal Institute of Technology, 1994.
Jack Lindsey, Samuel Ocko, Surya Ganguli, and Stephane Deny. A unified theory of carly visual representations from retina to cortex through anatomically constrained deep CNNs. In Interna- tional Conference on Learning Representations, 2019.
David G Lowe. Distinctive image features from ccale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pp. 3276-3285. PMLR, 2018
Shangzhen Luan, Baochang Zhang, Chen Chen, Xianbin Cao, Qixiang Ye, Jungong Han, and Jianzhuang Liu. Gabor Convolutional Networks. CoRR, 2017.
Stéphane Mallat. Group invariant scattering. Communications on Pure and Applied Mathematics, 6331099 2012.
Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 503(74744:78-88, 2013.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural ODEs. arXiv preprint arXiv:2002.08077 2020.
Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron, 99(3):609 623, 2018
Thomas Miconi, Kenneth O. Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In Proceedings of the 35th International Conference on Machine Learning, ICML, volume 80, pp. 3556-3565, 2018.
Kanaka Rajan, Christopher D Harvey, and David Tank. Recurrent network models sequence generation and memory. Neuron, 128 142, 2016.
12

---

Under review as a conference paper at ICLR 2021
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte. Peter Latham, Grace W. Lind- say, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiota Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P. Kording. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1761-1770 2019.
François Rousseau, Lucas Drumetz, and Ronan Fablet. Residual networks as flows of diffeomor- phisms. Journal of Mathematical Imaging and Vision, pp. 1-11, 2019.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pp. 1-13, 2019.
Hadi Salman, Payman Yadollahpour, Tom Fletcher, and Kayhan Batmanghelich. Deep diffeomor- phic normalizing flows. CoRR, 2018.
Michael P Sceniak, Michael J Hawken, and Robert Shapley. Contrast-dependent changes in spatial frequency tuning of macaque vI neurons: effects of changing receptive field size. Journal of Newrophysiology, (8(3):1333-1373 2002.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James DiCarlo. Brain-score score: Which artificial neural network for object recognition is most brain-like? bioRxiv preprint, 2018. URL/https: /www wwwpiiiwwwwwoorooooooooeeee 10. 1111/407000///
G Sclar and RD Freeman. Orientation selectivity in the cat's striate cortex is invariant with stimulus contrast. Experimental bruin research, 46(3):457-461, 1982.
Terrence J Sejnowski. The unreasonable effectiveness of deep learning in artificial intelligence. Proceedings of the National Academy of Sciences, 2020.
Evan Shelhamer, Dequan Wang, and Trevor Darrell. Blurring the line between structure and learning to optimize and adapt receptive fields. CoRR, 2019.
Eero P Simoncelli, William T Freeman, Edward H Adelson, and David J Heeger. Shiftable multi- scale transforms. IEEE transactions on Information Theory, 38(2):587-607, 1992.
Bernt C Skottun, Arthur Bradley, Gary Sclar, Izumi Ohzawa, and Ralph D Freeman. The effects of contrast on visual orientation and spatial frequency discrimination: comparison of single cells and behavior. Journal of neurophysiology. 57(3):773-766, 1987.
Andrew T Smith, Krish Devi Singh, AL Williams, and Mark Greenlee. Estimating receptive field size from fmrr data in human striate and extrastriate visual cortex. Cerebral cortex. 11(12): 1182-1190, 2001.
David C Somers, Sacha B Nelson, and Mriganka Sur. An emergent model of orientation selectivity in cat visual cortical simple cells. Journal of Neuroscience, 15(8):5448-544 5465, 1995.
Haim Sompolinsky, Andrea Crisanti, and Hans-Jurgen Sommers. Chaos in random neural networks. Physical review letters, 61()):259 1988.
H Francis Song, Guangyu R Yang, and Xiao-Jing Wang. Training excitatory-inhibitor recurrent neural networks for cognitive tasks: a simple and flexible framework. PLoS computational biol- ogy, 12(2)e 2016.
Ivan Sosnovik, Michat Szmaja, and Arnold Smeulders. Scale-equivariant steerable networks, ICLR, 2020.
Courtney J Spoerer, Patrick McClure, and Nikolaus Kriegeskorte. Recurrent convolutional neural networks: a better model of biological object recognition. Frontiers in psychology, 8:1551, 2017.
13

---

Under review as a conference paper at ICLR 2021
David Sussillo and Larry F Abbott. Generating coherent patterns of activity from chaotic neural networks. Neuron, 63(4):544- -57444455,,20 2009.
Gert Van den Bergh, Bin Zhang, Lutgarde Arckens, and Yuzo M Chino. Receptive-field properties of v1 and v2 neurons in mice and macaque monkeys. Journal of Comparative Neurology, 518 (11):2051-2077 2010.
Daniel E. Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance. In CVPR, July 2017.
Yuxin Wu and Kaiming He. Group normalization. ECCV, pp. 3-19, 2018.
Zhitong Xiong, Yuan Yuan, Nianhui Guo, and Qi Wang. Variational context-deformable convnets for indoor scene parsing. In CVPR, pp. 3992 4002, 2020.
Hsin-Hao Yu, Richa Verma, Yin Yang, Heath A Tibballs, Leo L Lui, David H Reser, and Mar- cello GP Rosa. Spatial and temporal frequency tuning in striate cortex: functional uniformity and specializations related to receptive field eccentricity. European Journal of Neuroscience, (1)6): 1043-1062, 2010.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael Frank, James DiCarlo, and Daniel Yamins. Unsupervised neural network models of the ventral visual stream. bioRxiv, 2020a.
Juntang Zhuang, Nicha Dvornek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ODE. arXiv preprint arXivv20066022493, 2020b.
A APPENDIX
A.1 GAUSSIAN MULTISCALE LOCAL N-JET
Based on the Schwartz theory of smooth test functions, the Gaussian scale-space paradigm states that the derivatives Lig.., (7; o)) of function Lo()) with respect to the spatial variables 2;, with i 1....... d, and at scale a is given by the convolution
Li (z; σ) = Lo * O,,..., G(I;
(4)
where D.,...., is the n-th order partial derivative and G(ž: or) is the normalized, isotropic Gaussian kernel with standard deviation o; a and mean μ4; = 0 (Florack et al. 1996). Note that Lz(z) does not need to a smooth function, and therefore the Gaussian scale-space paradigm can applied to obtain local image derivatives in different scales, where z denote the spatial coordinates and can be interpreted as the coordinate in the scale dimension.
We build upon this definition of local image derivatives build our structured receptive fields (SRFs), similar to Jacobsen et al.] (2016). The main idea we leverage is that using a Taylor ap- proximation, one decompose an input image into a superimposition of its local derivatives. This decomposition can then be performed by local convolution kernels in CNNs, where the relative weight of each derivative order can be learned during training. order to show this, we observe that the N-th order Taylor expansion of an image L I22 + IR around a point (a, b) is given by
a) 6) gits
L(x,y)= Σ
Ou L(a,b)
(5)
dzj
where the partial derivatives of L. with respect to x and y can be interpreted as LI,...,, 1, (ar, y do) and L,,..2, (x, yy oo) from equation 4 at some original scale do. This means that, via the linearity of convolution, and under the assumption that L(x, y;; o)) does not diverge, it is equivalent to either use the N-th order derivatives of the input image, or use the N-th order derivatives of the Gaussian scale The local N-jets can thus be seen to be
function G d),, to perform the decomposition parametrized by the coefficients in the expansion in equation By definition, we consider the Taylor expansion coefficients be covariant derivatives of the image [xx,, y, a), which independent
14

---

Under review as a conference paper at ICLR 2021
of the local coordinate system, to reach the filter approximations given in equation where the coefficients take the form ai.j-
In short, the multiscale local N-jet provides a natural decomposition of an image in local neigh- borhood in the spatial and scale dimensions. Under convolution with G(ă; o). this decomposition provides framework for defining convolutional filters in CNN using N-th order Taylor polyno- mials. The SRF filters we use are based on this N-jet definition and allow us to learn the scale, spatial frequency and orientation of the filters during training, which are fundamental properties of biological receptive fields (Jones & Palmer 1987; Lindeberg 1993).
In addition, the Gaussian scale-space formulation of SRFs (Jacobsen et al. 2016) lead to theoret- ically interesting properties which strengthen the motivation for our choice of filters based on the Gaussian N-jet. For example, the semi-group property of Gaussian scale-spaces indicates for Gaussian derivative kernel Gr. (z, t) parametrized by the variance t = o²
(z, G (I, (1 (') G (z, 15 G(r, y:
(6)
where the superscripts (i.j) denote the partial derivatives with respect to x and y. This means that translation in scale dimension Gaussian kernel. For DCNs, this means that we have a solid understanding of the scale of the feature maps (in the sense of the Gaussian scale-space) every layer in the network, as long as we know the learned value of the ass. In the absence of SRFs with an explicit scale parameter, this information lost.
be achieved through convolution with simple (0-th order)
Similarly, SRF filters based on Gaussian derivatives are steerable by the coefficients asy. For exam- ple, filter with orientation 0 can be described using second order basis functions
GØ 02.0220 a1,1G 0o.2 cos? ( 2 cos(O) sin(O)G' sin 60) (7)
Finally, the set of Gaussian derivative basis functions G (xx, y; o) are spatially separable (-,),,;;)))) G¹ (c;
(8)
which is useful for computational efficiency in numerical applications.
A.2 TRAINING PROCEDURE
The basic architecture of all our models is given in figure 2 Unless otherwise stated, in all models we use group normalization (Wu & He| 2018) with 32 groups every layer as the normalization func- tion. As the nonlinear activation, ron 2017). or CELU for DCN models. Based on the original ODE-Net implementation (Chen et al.) 2018), for ODE-Net models and ResNet-block models, we use rectified linear units (ReLU). Like- wise, when defining the integration time interval T, we stick to the original implementation, with T for ODE-Net models, whereas we use T 2 for DCN models. Otherwise, all the hyperpa- rameters are kept constant between models. For a brief overview of hyperparameter optimization, see appendix A.3
continuously differentiable exponential linear units (Bar-)
All models are trained for 100 epochs on the standard CIFAR-10 training set (Krizhevsky 2009) using cross-entropy loss. As data augmentation. we use random translations up to 4 pixels each spatial dimension and random horizontal flips. Optimization is performed using SGD with a mini- batch size of 128, initial learning rate 10-1, momentum 0.9, and a learning rate decay by a factor of 0.1 at epochs 40 and 70. For continuous time models based on neural ODEs, we use the adjoint method for backpropagating the losses and ODE solvers with error tolerance set to 10-3,
For convolutional layers with pixel-based k x & filters, the weights are initialized using the standard Kaiming uniform initialization (He et al., 2015). For layers using SRF filters, the initial a values are randomly sampled from a normal distribution with mean 0 and standard deviation O. and initial values are sampled from a normal distribution with mean and standard deviation 2/3.
For the restricted CIFAR-10 experiments (small-data regime), we pick the total number of training images to match our mini-batch size of 128, or otherwise have a minimal number of samples in the final batch (which is dropped in each epoch). In order to confirm convergence, for the training set sizes [520, 1030, 5120], we increased the number of training epochs by a factor of [10, 5, respectively.
15

---

Under review as a conference paper at ICLR 2021
For peta--parmetrrized models the initial values for the learnable parameters follow normal distri- butions N ((,. a ~ (N 2/3). b ~ (N0 0.1), for the DCN o model; a ~ (/0, 2/3), b (0, 2/3), e (f00,.11 0.1) for the DCN (t9)) model; and a, ~ 20,33),, b, (0,0 Gx ~ (0, 0.1) and b, ~ (0, 0.05) for the DCN o()), (t)) model.
For all our models using SRF filters, except for the DCN oij model, we use scale sharing within convolutional layer such that o = am for all m. This makes the GPU implementation trivial, as all filters within a layer are sampled in the interval 1-20", 20n",, and hence the convolutional kernel sizes within layer uniform. However, for the DCN oig model, GPU implementation would be highly inefficient if we truncated the kernels at fixed factor of ojj independently for each input and output channel j. Therefore for the DCN diy model we fix the kernel size at 7 7, but we still learn the shape and the scale (bandwidth) of the filters during training.
We use the Dormand Prince (DOPRI) method (Dormand & Prince 1980) as the numerical ODE solver. The DOPRI method is an explicit, adaptive solver of the Runge- Kutta family, which uses 6 function evaluations to compute fourth- and fifth-order accurate solutions to ODEs, along with an error estimate. The size of the adaptive steps taken by the solver can be regulated by specifying an error tolerance on this error estimate.
As is the case with most modern ODE solvers, the GPU implementation of the DOPRI solver that we use (Chen et al. 2018) considers the input arguments for the integration time interval E [0. T] (or t E [0, 2] in the case of all DCN models) as soft bounds. This means that the DOPRI algorithm may explore time points outside of this interval, based on its internal error estimation, and may then employ interpolation to return solutions within the specified bounds. In the meta parametrized mod- els, where for example a is an explicit function of I, this may lead to very large or very small kernel sizes, ordinarily unexpected within the integration time interval. In order avoid numerical insta- bility and memory issues in the veta-parametrized models, we scale down and clip the integration time when passing into the parameter definitions as o (Tfelip) and (Ttclip). We clip the values in the interval [-0.5, 2.5] and use т = 0.5.
A.3 HYPERPARAMETER TUNING
As mentioned in appendix A.2 we share all the design choices and hyperparameters between all DCN and baseline ODE-Net models, except for the nonlinear activation function and integration time interval T.
This difference in design choices arises since for the ODE-Net baseline we stay faithful the origi- nal ODE-Net implementation. where ReLU is the nonlinear activation function and T 1, whereas for DCN models we use CELU as the activation function, due to its generalized compatibility with the adjoint method, and T 2. In order to verify that our design choices do not provide un- fair advantage over the ODE-Net baseline, we run some control experiments, where we vary the activation function and T in the ODE-Net baseline.
We find that the change of activation functions or integration interval T do not provide significant increase to the CIFAR-10 classification performance in the ODE-Net baseline (Table 5)
Model
Accuracy (%)
DCN-ODE
89.46 + 0.16
ODE-Net (baseline)
89.60 0.28
ODE-Net with T = 2
89.50 + 0.07
ODE-Net with CELU
89.33 0.16
ODE-Net with CELU and T = 2
89.25 + 0.30
Table CIFAR- 10 validation accuracy (averaged over 3 runs) in the control experiments testing the effect of DCN model design choices on the bascline ODE-Net.
16

---

Under review as a conference paper at ICLR 2021
A.4 CIFAR-10 10 IMAGE RECONSTRUCTION
For the reconstruction task, we use the network architectures with 3 DCN-ODE blocks (Fig. 2 or for the baseline networks ODE-Net or ResNet-blocks, trained the CIFAR-10 classification task as an encoder. We use the feature maps at the end of ODE Block 3 as the input to a small decoder network. The decoder DCN network is made up of ODE block with 128 input channels. followed by bilinear upscaling. 1 X 1 convolution to reduce the number of channels to 64, another ODE block, followed by bilinear upscaling, normalization, non-linear activation and 1 convolution to generate the output in RGB space.
We implement reconstruction as regression task and use the mean squred error (MSE) as the loss function. Otherwise, the training parameters are the same as the classification experiments: We use SGD optimizer with a mini-batch size of 128, learning rate 10 and momentum 0.9, together with the adjoint method and an error tolerance of 10 10-3
The reconstruction of some example images (randomly selected) from the CIFAR-10 validation set by the DCN and baseline networks are shown in figure
A.5 PATTERN COMPLETION FEATURE MAPS
In figure 6 we show the feature map evolution within the first ODE block (or ResNet block) of different models with and without masking of some example input images. Size of the mask depicted here is x 6 pixels and the example images were chosen so as to have the mask located close to the middle of the object. We picked some channels with visible mask-related artifacts in the input feature maps to the first ODE (ResNet) block. Since there is no feature map trajectory in the LesNet-blocks model, but only one input and one output feature map, the difference between the feature maps of the intact and masked image is given as a scatter plot of two data points connected by red line.
Figure 7 depicts the average difference of intact and masked feature maps (itt) averaged over 1000 images and the standard deviation for the DCN and baseline networks.
17

---

Under review as a conference paper at ICLR 2021
Input Image
ResNet- Blocks
DCN-ODE
ODE-Net
Figure 5: Example CIFAR-10 10 validation images and their reconstruction by the DCN-ODE model as compared to baseline models.
18

---

Under review as a conference paper at ICLR 2021
image
5.5
D&1
D09
masced Image
h 1r
by hora
5.4
masked Nytuem
Integ
Inograt:
Iputtmape
D19
aDExet
maaked Imaga
hv (m..ener
masked image
h woreeea Los
Image
Inout Image
Restict-aiose alocks
masced Imagee
hy www.
h h
marked Imagee
0.5
Figure 6: Feature map evolution within the first ODE block (or ResNet block) of different models.
DCN
ODE-Net
ResNet-Blocks
1.0
1.0
1.0
D(t)
D(t)
D(t)
0.5
0.5
0.5
0.0
0.0
0
2
integration time
integration time
integration time
Figure 7: Evolution of the mean difference (t) between feature maps of an intact input image and a masked input image, averaged over 1000 images in the CIFAR- 10 validation set. The shaded areas (or in the case of ResNet, the errorbars) show the standard deviation.
19


Reviews:
[{"review_id": "cz3g-4wZF5_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Review\n\n\n### Summary\n\nThe authors define continuous deep networks by expressing 2D-convolutional filters as a linear combination of Gaussian function and its derivatives. By combining this description with the previously proposed neural ODE framework they obtain a spatio-temporally continuous description of a convolutional neural network.\n\nThere are 3 main contributions:  \n\n1. they are able to estimate the support width of the filters and they show that  it increases with the network depth as observed in the visual cortex  \n2. they show that their network performs as well as alternative non-continuous neural networks on CIFAR-10 while having less parameters  \n3. they exploit the temporal dynamics to resolve a pattern completion task\n\nOverall, I think the work is good but I am not as enthusiastic as the authors about the importance of the work for neuroscience and machine learning.\n\n### Strengths\n\n* The paper is well written and easy to understand. The goals and contributions of the work are clearly stated.\n* The quantitative results are marginally good.\n* The qualitative results (filter support width, pattern completion and contrast robustness) are interesting and relevant to neuroscience\n\n\n### Weaknesses\n\n* I fail to understand how the work could be relevant to neuroscience beyond what is presented here. What is so important about using spatially continuous filters that couldn't be done with discrete filters ?\n* The pattern completion task is not fully conducted. It would be great to reconstruct the missing part of the input.\n* The increase of filter support width with network depth correlates with what is known for the visual cortex but I fail to understand how it could be relevant to current work in experimental neuroscience. What is the benefit of learning continuously changing support for machine learning ? About the relation to biology, the increase in size might be more related to the specific task on which the network is trained than to what is observed in the visual cortex.\n* The observed contrast robustness is not compared to other neural networks nor discussed in the light of experimental neuroscience observations.\n\n### Minor comments\n\n* The text in the figures is way too small. It should be the same size as the main text.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AbjIHxP2p0i", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarize what the paper claims to contribute. \nThe paper develops a spatio-temporal network that is defined in terms of continuous spatial functions and continuous temporal dynamics. The approach is meant to bring deep networks closer to biological neural models. The model is tested on CIFAR-10, and on a variation of CIFAR-10 in which blocks of pixels are blacked out. The methodological novelty consists of combining several existing approaches (continuous kernels, kernel-scale learning, and neural ODEs). \n\nList strong and weak points of the paper. \nStrong points:\n-\tI think the motivation is very strong. Deep convolutional networks are increasingly used in brain modelling, but they are somewhat disconnected from earlier computational neuroscience in ways that this paper tries to address. \n-\tThe pattern completion test with blacked-out pixels is a nice way to employ spatiotemporal dynamics in image recognition, and the results are promising. \n-\tIt is very interesting that the distribution of learned scales reflects the distribution of receptive-field sizes in primary visual cortex. \n\nWeak points: \n-\tThe feature-map evolution results in Fig. 4 are interesting, and a few more examples are given in the appendix, but it would also be nice to see mean +/- SD dynamics across many images, to complement Figure 3C more thoroughly. \n-\tThe model doesn’t outperform controls on CIFAR-10, although it does perform moderately better with blacked-out blocks of 6x6 pixels or more. \n-\tPerformance of the model and baselines on CIFAR-10 is not strong, which raises the question of how compatible the approach is with higher performance. \n-\tThe choice of CIFAR-10 as a test of the approach does not seem to be well motivated. A task with a temporal component might give the dynamic parts of the model more to do. \n\nClearly state your recommendation (accept or reject) with one or two key reasons for this choice. \nI recommend to accept the paper. There is a growing body of work that compares deep CNNs to biological neural networks, and the conventions of discrete time and space in CNNs unfortunately distance this work somewhat from much previous computational neuroscience. I think the paper helps to close this gap. \n\nAsk questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. \n-\tCould you please expand on the motivation for continuous space with respect to biological vision? Ultimately vision is based on discrete photoreceptors, and generally on populations of discrete cells. Relatedly, the motivation for continuous time is more obvious, but I think it would also help to comment on spikes in this context. \n-\tThe dynamic equation in Eq. 2 seems to be autonomous. How does input affect h? \n-\tIn Section 3.2, the learned scales are shown to increase with network depth. This is compared with biological receptive fields, which grow through the visual hierarchy. But I had understood the scale to correspond to the kernel size rather than the receptive field size (which grows in deep networks even if all the kernels are the same size). Does sigma correspond to kernel or receptive field size? \n\nProvide additional feedback with the aim to improve the paper. \n-\tAnalytic tractability is mentioned on page 1 as an advantage of some continuous models in neuroscience, and it seems to be implied at that point that such benefits are sought in the paper, but it doesn’t seem that the approach ultimately offers much hope in this sense. If there is some potential here, please expand. \n-\tI didn’t understand “… time (or network depth)” on pg. 4.  \n-\tAlso on pg. 4, the sampling domain of the filter is given, but not the sampling frequency. \n-\tI found section A.1 relatively hard to follow. In particular, Eq. 5 seems to be meant to motivate the filter family, but I didn’t follow the argument (or maybe I missed the point entirely). Also, aside from general interest I didn’t understand how the paragraph that contains Eq. 6 related to the rest of the paper. \n-\tIt wouldn’t hurt to define DOPRI. \n-\tI didn’t follow the last paragraph of A.2. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2lskEF1iXJ9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe authors propose a hierarchical model of neural ODEs, which they fit to CIFAR10. They find performance on par with ResNets, and include qualitative analyses on the filters learned by the models, their ability to fill in occluded features, and robustness to contrast at test time.\n\n\nStrengths:\n\nThe filter parameterization is interesting. I can imagine this improving sample efficiency in certain contexts — perhaps the authors should seek out those kinds of tasks to complement their CIFAR results?\n\nThe discussion does a nice job of explaining the issues that neural ODEs have when scaling to large image datasets.\n\n\nWeaknesses:\n\nYou spend time discussing spatio-temporal receptive fields throughout. Why? Your models are applied to 2d images.\n\nThe authors are missing a huge literature on (a) recurrent convolutional networks, and (b) using these networks to simulate classical vs. extra classical receptive field effects.\n\nResNet-Blocks is the original ResNet? Maybe change \"blocks\" to a citation? Or V1/V2 depending on which implementation it is (unclear from the text).\n\nThere's essentially no difference between the performance of any of the models tested. Is it possible to scale to ImageNet? It is important to show that the proposed method does *something* different than the standard ResNet. The authors attempted to add some qualitative experiments towards this goal in Fig 4, but those results are not very convincing. I think to show filling-in you'd want to show reconstruction in RGB space.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 1 weak accept, 1 accept, and 1 weak reject.

All reviewers questioned the motivation for continuous space/time with respect to biological vision. Obviously, discrete approximations used in machine vision are approximations but it is not clear from the paper or the authors’ response that this severely limits the ability of deep nets to predict neural data in ways that their continuous nets would not. 

In addition, I have to confess that I did not really understand the argument made by the authors in their revision. In any case, the burden should be on the authors to go beyond general statements and to really demonstrate that the proposed models provide actual insights for neuroscience since the performance in terms of machine vision on CIFAR10 is underwhelming (the authors have to find a low data regime and even then the reviewers stated that the baselines used are not strong baselines, the reduction in the number of parameters is quite small relative to methods for actually reducing the number of parameters). 


Clearly, the work has potential as noted by the reviewers. The authors suggest that “DCNs can be used to model the temporal profiles of neuronal responses, which are known to not be constant even when the experimental stimuli are static images: for example, spatial frequency responses change over time in response to stationary gratings (Frazor et al., 2004). Similar observations are made for the contrast response function (Albrecht et al., 2002). Such temporal profiles cannot be simulated in conventional CNNs. “ This sounds like an interesting set of neuroscience data that the authors could be indeed leveraging to demonstrate the benefit of their approach. My recommendation would be to add those in a revision of this paper which will significantly strengthen the work. I would add that the concepts of temporal and spatial continuity are independent and the authors should consider studying them separately to provide more in-depth analyses and convincing results. 

As it stands, the paper has clear potential but it does not make a sufficient contribution to either ML or neuroscience and hence, I recommend the paper to be rejected.

Decision: Reject