Title: Recurrent neural network dynamical systems for biological vision
Venue: NeurIPS 2024
Accepted: yes
Keywords: convolutional neural networks

recurrent neural networks

neuroscience

vision

cognition

perception
Abstract: In neuroscience, recurrent neural networks (RNNs) are modeled as continuous-time dynamical systems to more accurately reflect the dynamics inherent in biological circuits. However, convolutional neural networks (CNNs) remain the preferred architecture in vision neuroscience due to their ability to efficiently process visual information, which comes at the cost of the biological realism provided by RNNs. To address this, we introduce a hybrid architecture that integrates the continuous-time recurrent dynamics of RNNs with the spatial processing capabilities of CNNs. Our models preserve the dynamical characteristics typical of RNNs while having comparable performance with their conventional CNN counterparts on benchmarks like ImageNet. Compared to conventional CNNs, our models demonstrate increased robustness to noise due to noise-suppressing mechanisms inherent in recurrent dynamical systems. Analyzing our architecture as a dynamical system is computationally expensive, so we develop a toolkit consisting of iterative methods specifically tailored for convolutional structures. We also train multi-area RNNs using our architecture as the front-end to perform complex cognitive tasks previously impossible to learn or achievable only with oversimplified stimulus representations. In monkey neural recordings, our models capture time-dependent variations in neural activity in higher-order visual areas. Together, these contributions represent a comprehensive foundation to unify the advances of CNNs and dynamical RNNs in vision neuroscience.

Recurrent neural network dynamical systems for biological vision

Abstract

In neuroscience, recurrent neural networks (RNNs) are modeled as continuous-time dynamical systems to more accurately reflect the dynamics inherent in biological circuits. However, convolutional neural networks (CNNs) remain the preferred architecture in vision neuroscience due to their ability to efficiently process visual information, which comes at the cost of the biological realism provided by RNNs. To address this, we introduce a hybrid architecture that integrates the continuoustime recurrent dynamics of RNNs with the spatial processing capabilities of CNNs. Our models preserve the dynamical characteristics typical of RNNs while having comparable performance with their conventional CNN counterparts on benchmarks like ImageNet. Compared to conventional CNNs, our models demonstrate increased robustness to noise due to noise-suppressing mechanisms inherent in recurrent dynamical systems. Analyzing our architecture as a dynamical system is computationally expensive, so we develop a toolkit consisting of iterative methods specifically tailored for convolutional structures. We also train multi-area RNNs using our architecture as the front-end to perform complex cognitive tasks previously impossible to learn or achievable only with oversimplified stimulus representations. In monkey neural recordings, our models capture time-dependent variations in neural activity in higher-order visual areas. Together, these contributions represent a comprehensive foundation to unify the advances of CNNs and dynamical RNNs in vision neuroscience.

1 Introduction

Dynamical systems have long been a cornerstone in the field of neuroscience, tracing their origins back to the modeling of single neurons [1, 2]. Early biophysical models provided critical insights into the dynamic behavior of neurons. Building on this foundation, neuroscientists progressed to constructing networks of dynamical neurons [3-12]. The idea was to understand not just how individual neurons operate but how networks of neurons interact to produce complex behaviors [5-7, 12] and cognitive functions [8-10]. The rise of deep learning and artificial recurrent neural networks (RNNs) marked a significant turning point in this endeavor. Neuroscience quickly adopted RNNs due to their ability to model time-dependent processes [13-29], much like how biological circuits process information over time. In these models, each artificial neuron can behave according to some biophysical dynamics, whether through rate coding [13, 14, 18, 30] or spiking mechanisms [19, 20]. This alignment with

biological plausibility made RNNs a powerful tool in neuroscience. Today, advances in computational power and learning algorithms have enabled these networks to be trained on diverse and complex tasks, ranging from motor control [14, 24] to cognitive functions [15, 16, 18, 27, 29-31]. Modern RNNs are capable of learning intricate patterns in data, making them invaluable for modeling a wide array of neural processes. We provide details of key references in Appendix A [1-5, 7, 13, 30, 32]. RNNs are not the only models used in neuroscience. Convolutional neural networks (CNNs) have also been widely used, particularly in the domain of biological vision [33-56]. CNNs excel at processing spatial hierarchies in images, making them ideal for object recognition and scene understanding tasks. While there are recurrent CNNs that integrate temporal dynamics into the spatial processing capabilities of CNNs [36, 38, 41-44, 49, 51, 57], the specific advances of dynamical RNNs in neuroscience have largely remained inapplicable to vision models. To bridge this gap, we propose a hybrid architecture that integrates the continuous-time recurrent dynamics of RNNs with the image processing capabilities of CNNs, which we refer to as CordsNet ( Co nvolutional R NN d ynamical s ystem). Briefly, our contributions and results are:

Dynamical expressivity analysis. Werigorously compare CordsNet with other recurrent dynamical architectures by training them all on multiple cognitive tasks in neuroscience. We find that CordsNet can achieve the same range of dynamical regimes as other architectures.

New training algorithm. Training a continuous-time model is computationally expensive. We propose a computationally cheaper algorithm to efficiently initialize CordsNets and successfully train them on standard image classification benchmarks like ImageNet.

Autonomous and robust inference. We show that our trained models can perform inference autonomously and exhibit robustness to noise compared to static and discrete-time CNNs, which are actually just properties inherent to dynamical systems.

Analytical toolkit. There are many ways to analyze CordsNets, but they are memory intensive. We thus develop a toolkit consisting of iterative and dimensionality-reduction methods needed to analyze the model within reasonable computational limits and show some examples.

Image-computable models. We demonstrate the effectiveness of CordsNet as the front-end of an image-computable multi-area model by training such models to perform tasks using the actual stimuli seen by subjects in experiments rather than abstract inputs.

Prediction of neural activity. We find that CordsNets (trained on ImageNet) are able to predict temporal signatures in neural activity of higher-order visual areas (V4 and IT).

Together, our contributions represent a comprehensive and directed effort to bring decades of advancements in dynamical systems to vision neuroscience.

2 Model architecture

We first briefly introduce CNNs and dynamical RNNs in neuroscience. A typical convolutional layer in a CNN consists of a 2D-convolution, normalization (commonly batch normalization [58]) and a non-linearity (such as ReLU [59]), although the order of these operations can change [60]. For two consecutive convolutional layers x l -1 and x l , this can be written as:

where σ is the non-linear activation function. For a single image, the output of a convolutional layer is characterized by three dimensions (channels, height, and width), which can be interpreted as the latent state of the CNN (Figure 1A, left). On the other hand, a continuous-time RNN dynamical system typically found in neuroscience literature [28] is described by:

where T represents a diagonal matrix of neuron time constants, r represents the latent state of the RNN, W rec is the recurrent weight matrix, b represents the vector of neuron biases, and -r is the leaky term which mimics the dynamics of biological neurons. h inp is the external input to the network that first undergoes a linear transformation through W inp (Figure 1A, middle). Just like in CNNs, σ is the non-linear activation function. Note that there are no artificial normalizing operations as they are generally incompatible with continuous-time dynamical systems. The latent state of the network can be simulated forward in time by Euler's method with a suitable choice of time discretization.

A

convolution

normalization

non-linearity

Convolutional neural network

Dynamical recurrent neural network

linear

continuous-time

recurrent

dynamics

B

Neural activity

Real

Imag

Stable regime

Examples

Eigenvalues

Neural activity

Real

Imag

Oscillatory regime

Examples

Eigenvalues

Neural activity

Chaotic regime

Examples

Eigenvalues

C

Perceptual decision-making

CordsNet

PC1

PC2

PC3

Low-rank RNN

PC1

PC2

PC3

D

Multi-sensory decision-making

CordsNet

PC1

PC2

PC3

Sparse RNN

PC1

PC2

PC3

Memory-pro delayed response

(Ring attractor)

Stimulus

PC1

Stimulus

PC2

Output

CordsNet (Hybrid architecture)

convolution

continuous-time

recurrent

dynamics

Unlike regular RNNs, dynamical RNNs typically evolve (and are therefore backpropagated) over hundreds of time steps [28].

In CordsNet, which is our proposed hybrid architecture, every convolutional layer is replaced by a dynamical RNN (Figure 1A, right). The activations x l now represent firing rates r l . In order to preserve the convolutional structure, the input and recurrent weight matrices ( W inp and W rec ) are now convolutions ( Convinp and Convrec ) , while h inp and b are reshaped to have convolutional structures. This can be summarized as:

In the rest of this work, we will show that this proposed extension is highly non-trivial and brings the best (and worst) features of dynamical RNNs into CNNs.

2.1 Dynamical characteristics

Since convolutions are linear operations, they can be expressed as a 2-D weight matrix operating on a flattened 1-D vector, just like in equation (2), which we describe in detail in Appendix B.1. However, the convolutional recurrent weight structure of CordsNet spans only a small subspace of all possible recurrent weight matrices, so there is a need to verify if such a restriction would limit the range of dynamical properties that our networks can express. We know that (sufficiently large) Gaussian-initialized fully-connected networks using tanh activations exhibit different dynamical regimes based on initialization variance [4, 61]. We successfully replicate this in CordsNets to express stable, oscillatory and chaotic behaviors (Figure 1B) under the same conditions. Further analysis of dimensionality and autocorrelations can be found in Appendix B.2.

Multiple variants of dynamical RNNs have been trained on cognitive tasks in neuroscience [15, 28], and thus we want to know if CordsNets will produce similar solutions when trained on such tasks despite the restrictions from the convolutional weight structure. For this investigation, we choose

the same set of five tasks that was previously adopted for analyzing low-rank dynamical RNNs [62], consisting of a perceptual decision-making task [63], a parametric working memory task [64], a multi-sensory decision-making task [65], a contextual decision-making task [13] and a delayed matchto-sample task [66]. We independently train CordsNets, fully-connected RNNs, low-rank RNNs [62] and sparsely-connected RNNs [15] on these tasks across different activation functions, learning rates, network sizes and initializations. We find that trained networks of all architectures produce similar neural trajectories when aligned using canonical correlation analysis (Figure 1C), suggesting that CordsNets are able to utilize the same dynamical motifs as other established architectures in neuroscience to perform cognitive tasks. We also provide a more rigorous and quantitative analysis using recently proposed metrics on representational similarity [67] in Appendix B.3 which further supports this conclusion. In fact, from these metrics, we find that our networks produce more similar solutions to fully-connected RNNs than low-rank or sparsely-connected RNNs.

RNNs can perform tasks requiring long-term dependencies by storing information in memory. A dynamical RNN typically achieves this using attractors in neural activity space [68, 69]. We show that our networks can represent well-known classes of attractors such as ring (Figure 1D), line attractors, and discrete fixed-points (see Appendix B.4). Taken together, our results suggest that the convolutional recurrent weight structure of CordsNet does not constrain its expressiveness as a recurrent dynamical system.

3 Training and results

We now focus on the functionality of CordsNets as an image recognition model. We train networks of four different sizes, named CordsNet-RX, where X ∈ [2 , 4 , 6 , 8] represents the number of recurrent layers. Exact model specifications can be found in Appendix C.1, where we also review important design choices. We train our models on MNIST [70], Fashion-MNIST [71], CIFAR-10 [72], CIFAR100 and ImageNet [73], each with dataset-specific augmentations [74] as detailed in Appendix C.2. The neuron time constant is set to 10 ms (constant for all neurons), and the network is simulated with 2 ms time steps. The loss function that we aim to minimize is computed as:

Wefirst simulate the networks for 200 ms (time steps 0 to 100 ) without any input so that they converge to some steady state spontaneous activity ( spontaneous ). A batch of images is then presented for 200 ms (time steps 100 to 200 ). During this time, the cross-entropy loss is computed for the last 60 ms (time steps 170 to 200 ) and combined using a log-weighted sum. Finally, the networks are simulated for another 200 ms (time steps 200 to 300 ), and the mean-squared error between activity in the last 20 ms (time steps 290 to 300 ) and spontaneous activity is added to the loss. This additional term encourages the networks to return to spontaneous activity after stimulus presentation. This loss function has been carefully designed to produce a mono-stable solution so that the network may perform accurate inference indefinitely across time, which we will elaborate on in the next section. We also provide a detailed ablation study of every coefficient and every term in Appendix D. Simulating these networks for 300 steps across time is computationally expensive. For comparison, we compute the total multiply-accumulate operations (MACs) for 79 well-known CNN models found in torchvision.models library and compare them against their parameter counts (Figure 2C). Our largest model, CordsNet-R8, has approximately the same number of parameters as ResNet-18, the smallest model of the ResNet series. In contrast, our smallest model, CordsNet-R2, requires more MACs to simulate than ViT-H-14, the largest vision transformer in the model library currently.

While training the models by computing and minimizing the loss function in equation (4) is inevitable, we can reduce the number of training iterations needed by carefully initializing our models. We do this in three computationally cheaper steps (Figure 2B). We first train a feedforward CNN model without the recurrent component and replace it with a one-time convolutional layer. We also include batch normalization here to improve training efficiency. In the second step, we fold the batch normalization [75] into the convolution operation and replace the one-time convolutional layers with dynamical linear RNNs. Batch normalization folding is done in the following way:

A

avgpool 3x3, /2

softmax

linear

flatten

conv 3x3, 128

conv 3x3, 128, /2

conv 3x3, 64

conv 3x3, 64, /2

conv 3x3, 64

conv 3x3, 64

conv 3x3, 128

conv 3x3, 128

classification

B

D

C









log

10

(MACs)

log

10

(parameters)

ViT-H-14

ResNet-18

R8

R4 R6

R2

CordsNet

ResNet and variants

Vision transformers

Others - DenseNet,

EfficientNet, RegNet,

Swin, VGG, etc.

convolution

normalization

non-linearity

one-time

convolution

convolution

non-linearity

linear

recurrent

dynamics

convolution

non-linear

recurrent

dynamics

normalization folding

analytical initialization

parametric annealing

Accuracy (%)

Epochs





CordsNet-R2

0

8.3%





CordsNet-R4

0

19.7%





CordsNet-R6

0

29.1%





CordsNet-R8

0

41.2%

Accuracy (%)

Epochs

CordsNet

Fully-connected RNN

A dynamical linear RNN is described by equation (2), except that σ is simply the identity function. Such a network can be analytically solved for every time step t :

We derive its steady-state solution, which can be approximated and compared to a convolutional layer:

The parameters of the linear RNN are W conv and b , which we optimize by minimizing the meansquared error between its steady state r ∞ and the output of the convolutional layer, After optimization, we train the full linear model (end-to-end) using a reduced cost function:

We present the image for 200 ms (time steps 0 to 100 ) and compute the weighted classification loss for the last 60 ms (time steps 70 to 100 ). We do not need to compute the spontaneous penalty term because linear RNNs are almost guaranteed to be mono-stable during training. Finally, in the third step, we replace the identity activation function with a parametric ReLU non-linearity [76]

with parameter a , where PReLU ( x ) = max (0 , x ) + a × min (0 , x ) . We train the networks starting from a = 1 and gradually reducing a until a = 0 at the end of training. This provides us with an initialization of the original model for further fine-tuning. We additionally perform two control experiments, where we train both fully-connected RNNs (with matched parameter counts) and CordsNets directly without our initialization method. In both controls, the number of training epochs is determined by matching the time required for CordsNets to be trained directly with the time required for CordsNets to be initialized using our method. Additional details about this time benchmark can be found in Appendix C.3. Test accuracies of all experiments can be found in Table 1 (for ImageNet, the validation accuracy is shown instead). We draw three main conclusions:

Initialization vs Control (C). Our initialization method consistently produces models with higher test accuracies compared to models that were directly trained on every dataset except for MNIST, highlighting the effectiveness of our initialization approach. Figure 2D compares the ImageNet validation accuracy of CordsNets trained directly (purple circles) across all epochs against the validation accuracy of the models initialized using our method (purple dashed lines). MNIST classification is simple and does not require extensive feature extraction; it may therefore benefit from a more straightforward training approach.

Control (R) vs Control (C). When trained for the same number of epochs, CordsNets R6/R8 consistently outperforms their parameter-matched fully-connected RNN counterparts on all datasets by a significant margin. Fully-connected RNNs perform remarkably poorly on ImageNet due to their poor scaling with image size (Figure 2D, green circles). However, the results become more ambiguous when it comes to smaller models on simpler datasets. We attribute this to the flexibility of fully-connected models, which allow them to attain early and fast gains in performance, especially when learning simpler features in smaller datasets.

Fine-tuned vs CNN. Our fine-tuned CordsNets are able to attain accuracies that are slightly lower than (but reasonably close to) their feedforward CNN counterparts, suggesting that we have successfully trained a continuous-time dynamical system with competitive image processing capabilities. Closing the performance gap with CNNs remains a goal for future work.

A

B

C

D

CordsNet

feature map

CNN

feature map

Presented

stimulus

Time

CordsNet

CNN

DT-CNN

Normalized activations

Stimulus 1 softmax output

Time (ms)

Time (ms)

0

0

0.7

0.5

0.5

0

0

1

0

1

0

1

0

800

0

800

0

800

0

800

0

800

0

800

*

*

*

*

*

*

Normalized activations

Stimulus 2 softmax output

Time (ms)

Time (ms)

0

0

1

0.5

1

0

0

1

0

1

0

1

0

800

0

800

0

800

0

800

0

800

0

800

*

*

*

*

*

*

Stimulus 1

(400 ms)

Stimulus 2

(400 ms)

No stimulus

(400 ms)

Stimulus 2

(400 ms)

Stimulus 1

(400 ms)

No stimulus

(400 ms)

MSE

MSE

0

0

0.3

2

0

24000

0.70

0.029

Counts

CNN

CordsNet

0.1

0.5 0.1

0.5

0.7

0.2

CNN

CordsNet

all error bars x300

0○○○00

4 Model analysis

As a continuous-time model, the feature maps of CordsNets will change over time. We handpick a single interpretable feature map from CordsNet-R8 that depicts a gradual evolution over time when the input image is changed (Figure 3A). To emphasize the rich temporal dynamics that trained CordsNets express, we consider an additional comparison with CORNet-RT [38], a class of discretetime recurrent CNN trained on ImageNet. In addition to architectural differences, the two models are trained differently. CordsNet has been trained to classify images over some time interval (and expected to perform accurately indefinitely across time), while CORNet-RT has been trained to classify for a single time step (and expected to be accurate for that particular time step).

We compare the temporal activities of the two models (as well as the feedforward CNN) in two scenarios. In the first scenario, an image is presented for 400 ms and then removed for the next 400 ms (Figure 3B, left). The feedforward CNN, having no concept of time, responds with the same activations across time and drops to zero when there is no input. The activity in CORNet-RT varies over time and spikes briefly at the time step, during which it is trained to accurately classify the input image. It is not able to make an accurate prediction at all other times. In contrast, the activity in CordsNet rises and stays at a fixed level for as long as the image is presented and returns to some baseline activity when the image is removed. In the second scenario, an image is presented for 400 ms, followed by a different image for 400 ms (Figure 3B, right). For its particular time step,

A

B

Iteration 25

Iteration 50

Iteration 75

Iteration 100

Iteration 300

True values

Imaginary

-1

1

-1

1

Real

-1

1

Real

-1

1

Real

-1

1

Real

-1

1

Real

-1

1

Real

C

Layer 1

0

1

0

400

Re

Re

Im

Im

Layer 2

0

1

0

400

Re

Re

Im

Im

Layer 3

0

1

0

400

Re

Re

Im

Im

Layer 4

0

1

0

400

Re

Re

Im

Im

Time (ms)

Normalized activations

Top 100 eigenvalues

Slow model

Fast model

Fast

Slow

Dimension 1

Dimension 2

Irish water spaniel: 94.9%

Curly-coated retriever: 0.0%

Fast

Slow

Dimension 1

Dimension 2

Irish water spaniel: 0.0%

Curly-coated retriever: 99.3%

Fast

Slow

Dimension 1

Dimension 2

Irish water spaniel: 37.9%

Curly-coated retriever: 50.0%

CORNet-RT correctly classifies the first image but cannot correctly classify the second image, even though it can correctly classify the second image if it were presented first. CordsNet can correctly classify both images and once again maintains the correct classification throughout the duration of the stimulus. These results showcase the autonomous nature of CordsNet compared to discrete-time CNNs - it can perform inference indefinitely in time, self-reset to a baseline state, and react flexibly to stimuli changes, all while being governed by a single differential equation.

In the feedforward CNN, when temporally uncorrelated white noise is applied to the input image at each time step, model activations deviate from baseline noiseless values, which results in a decrease in classification accuracy (Figure 3C-D, green). When the same noisy images are instead presented to CordsNet, its deviations are attenuated by more than an order of magnitude compared to those of the feedforward CNN. It also robustly maintains its classification accuracy at high noise levels (Figure 3CD, purple). This is a result of natural noise attenuation from continuous-time dynamics and filtering by the recurrent weights. We can rewrite the dynamics as an update equation to depict these effects:

We next present an analysis of our trained models from a dynamical systems point of view. For a dynamical RNN, this typically involves performing some form of matrix decomposition on its recurrent weight matrix or applying dimensionality-reduction techniques on neural trajectories. For CordsNet, both of these approaches are particularly challenging due to the size of the networks. Memory limitations prevent the full recurrent weight matrix from being expanded from its kernel representation. One solution that we found is to compute the eigenvalues of the recurrent weight matrix directly from its kernel form using Arnoldi iteration [77] (Figure 4A), which we use to uncover an important dynamical motif present in our trained networks. We notice that the dynamical characteristics of our networks are different when we train them to correctly classify images at different times. When trained to classify early, the networks exhibit an oscillatory behavior, but not when trained to classify late (Figure 4B, left). We attribute this to the effect of transient overshooting,

500 ms

600 ms

1000 ms

B

Cat

Dog

60% dog

40% cat

Example inputs

Output

Lever press

Categorical discrimination task

600 ms

600 ms

600 ms

Non-match trials only

C

D

Visual feature at

various locations

Example inputs

Output

Eye saccade

Oculomotor delayed-response task

Video stimuli of moving dots

at various coherence levels

Example inputs

Output

Eye saccade

Perceptual decision-making task

A

Visual features of

varying quantities

Example inputs

Output

Lever press

Numerical quantification task

500 ms

800 ms

1000 ms

1200 ms

Non-match trials only

1200 ms

1000 ms

500 ms

3000 ms

500 ms

600 ms

600 ms

1000 ms

Video stimulus

Neural activity

0

1

0

1000

Delay period (ms)

Neural activity

0

1

0

1000

Delay period (ms)

Neural activity

0

1

0

1000

Stimulus period (ms)

Neural activity

0

1

0

3000

Delay period (ms)

1

2

3

4

5

Cat morphs

Left

Right

Dog morphs

8 positions

where being in the oscillatory regime results in an early overshoot in activity, thereby speeding up the propagation of the signal. In the linear approximation, according to equation (6), this would manifest if the eigenvalues of ( W conv -I ) of the model trained for faster inference contain larger imaginary components. We confirm this to be the case (Figure 4B, right). It is also possible to perform dimensionality reduction within reasonable memory limits, but these methods require some adaptation for convolutional operations. Therefore, we release a toolkit consisting of the aforementioned functions that are specifically tailored for convolutional weights. These convenient tools unlock many possible approaches for analyzing CordsNets. For example, we identify a particular dimension in CordsNet-R8 which represents whether the model perceives an image as an Irish water spaniel or a curly-coated retriever (Figure 4C).

5 Applications

CordsNet is an ideal front-end of any RNN dynamical system that requires image-computability. To demonstrate this, we connect the final layer of CordsNet-R8 with a fully-connected RNN (with 512 neurons) and train only the fully-connected RNN on a set of cognitive tasks that explicitly requires visual information processing (Figure 5). We use the actual images the monkeys see as the input to our multi-area model. This is done either by obtaining the stimuli set from the authors of the experiments [78, 79] or generated according to the specifications described in the original experimental papers [80, 81]. In our trained models, we found interpretable neurons in the fully-connected RNN layer, such as neurons tuned to stimulus quantity (Figure 5A, right), cats or dogs (Figure 5B, right), spatial positions on the screen (Figure 5C, right) and motion direction (Figure 5D, right).

A

C

B

V1

V2

V4

IT

Neural recordings from primate visual system

Model activations from artificial neural network

V4

0

1

0

300

IT

0

1

0

300

Normalized activity

Time since stimulus (ms)

CordsNet activity

Neural recordings

R2

R4

R6

R8

0

0.13

Pearson correlation

Area V4 (Penultimate layer)

***

***

***

***

R2

R4

R6

R8

0

0.1

Area IT (Final layer)

**

**

**

**

Base score between CordsNet activations and neural recordings

Score obtained from temporally-shuffled data

Brain-Score is a benchmarking framework designed to evaluate the performance of artificial neural networks in terms of their ability to model and predict neural and behavioral data from the brain [39]. In particular, model activations from CNN layers have been compared with time-averaged neural data from various parts of the primate visual system (Figure 6A). With our continuous-time model, we want to know if our models are able to model temporal signatures in brain activity. To do so, we omit the time-averaging step when computing Brain-Scores (see Appendix E for details). Using the same training-test splits, we additionally fit the same model activations with temporally-shuffled neural data. Due to the lagged response in neural data, we have to temporally shift our model activations such that the time in which activity rises in response to an input is aligned with neural data. There are occasions where no shifting is necessary, such as in the final two layers of CordsNet-R8 (Figure 6B) on V4 and IT neural activity [82]. All CordsNet models score higher on unshuffled data across V4 and IT (paired t-test, all p-values < 10 -4 , Figure 6C), suggesting that the models are capturing temporal structures within the neural data. For the sake of transparency, we also state that we have performed the same test on a different dataset with V1 and V2 activities [83] but obtained inconclusive results.

6 Discussion and conclusion

We have presented CordsNet, a hybrid architecture combining the strengths of CNNs and dynamical RNNs to process visual information with continuous-time dynamics. CordsNet exhibits various dynamical behaviors, such as oscillations and chaos (Figure 1B). In our analysis of its image processing capabilities, we showcase its ability to classify images indefinitely across time (Figure 3B, right), as well as its robustness to noise (Figure 3D, right). We have also effectively used CordsNet as a means to build image-computable dynamical systems capable of performing cognitive tasks (Figure 5). Finally, CordsNet has been successful at modeling the temporal signatures of neural activity in higherorder visual areas like V4 and IT (Figure 6C).

Limitations. The main limitation of CordsNet lies in its substantial memory requirements (Figure 2C). Memory-efficient training algorithms have been explored [84]. Another limitation is the difficulty of analyzing CordsNet as a dynamical system, primarily due to how it is unfeasible to convert the recurrent kernel weights into its corresponding full-size recurrent weight matrix.

Conclusion. CordsNet bridges a crucial gap between dynamical systems and vision neuroscience. While most of cognitive neuroscience continues to build upon decades of RNN research, vision neuroscience remains dominated by CNN models due to the inability of RNNs to efficiently process visual information. CordsNet has resolved this limitation by being a dynamical RNN with image processing capabilities.

Author contributions

W.S. designed the architecture. W.S, A.B. and P.R. trained and analyzed the networks. All authors designed the study, took part in discussions, interpreted the results, and wrote the paper.

Acknowledgments and Disclosure of Funding

This work was supported by the NIH grant R01MH062349, Office of Naval Research grant N00014, James Simons Foundation grant NC-GB-CULM-00003138 and NYU High Performance Computing. A.B. was supported by the Swartz Foundation.

References

Hodgkin, A. L. & Huxley, A. F. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology 117 , 500-544 (1952).

Knight, B. W. Dynamics of encoding in a population of neurons. The Journal of General Physiology 59 , 734-766 (1972).

van Vreeswijk, C. & Sompolinsky, H. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science 274 , 1724-1726 (1996).

Sompolinsky, H., Crisanti, A. & Sommers, H. J. Chaos in random neural networks. Physical Review Letters 61 , 259-262 (1988).

Seung, H. S. How the brain keeps the eyes still. Proceedings of the National Academy of Sciences 93 , 13339-13344 (1996).

Bulsara, A. R., Elston, T. C., Doering, C. R., Lowen, S. B. & Lindenberg, K. Cooperative behavior in periodically driven noisy integrate-fire models of neuronal dynamics. Physical Review E 53 , 3958-3969 (1996).

Zhang, K. Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory. Journal of Neuroscience 16 , 2112-2126 (1996).

Amit, D. J. & Brunel, N. Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. Cerebral Cortex 7 , 237-252 (1997).

Amit, D. J. & Brunel, N. Dynamics of a recurrent network of spiking neurons before and following learning. Network: Computation in Neural Systems 8 , 373-404 (1997).

Brunel, N. & Sergi, S. Firing frequency of leaky intergrate-and-fire neurons with synaptic current dynamics. Journal of Theoretical Biology 195 , 87-95 (1998).

Vogels, T. P., Rajan, K. & Abbott, L. F. Neural network dynamics. Annu. Rev. Neurosci. 28 , 357-376 (2005).

Sussillo, D. & Abbott, L. F. Generating coherent patterns of activity from chaotic neural networks. Neuron 63 , 544-557 (2009).

Mante, V., Sussillo, D., Shenoy, K. V. & Newsome, W. T. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature 503 , 78-84 (2013).

Sussillo, D., Churchland, M. M., Kaufman, M. T. & Shenoy, K. V. A neural network that finds a naturalistic solution for the production of muscle activity. Nature neuroscience 18 , 1025-1033 (2015).

Song, H. F., Yang, G. R. & Wang, X.-J. Training excitatory-inhibitory recurrent neural networks for cognitive tasks: a simple and flexible framework. PLoS computational biology 12 , e1004792 (2016).

Song, H. F., Yang, G. R. & Wang, X.-J. Reward-based training of recurrent neural networks for cognitive and value-based tasks. Elife 6 , e21492 (2017).

Cueva, C. J. & Wei, X.-X. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. arXiv preprint arXiv:1803.07770 (2018).

Masse, N. Y., Yang, G. R., Song, H. F., Wang, X.-J. & Freedman, D. J. Circuit mechanisms for the maintenance and manipulation of information in working memory. Nature neuroscience 22 , 1159-1167 (2019).

Kim, R., Li, Y. & Sejnowski, T. J. Simple framework for constructing functional spiking recurrent neural networks. Proceedings of the national academy of sciences 116 , 22811-22820 (2019).

Kim, R. & Sejnowski, T. J. Strong inhibitory signaling underlies stable temporal dynamics and working memory in spiking neural networks. Nature neuroscience 24 , 129-139 (2021).

Cueva, C. J. et al. Low-dimensional dynamics for working memory and time encoding. Proceedings of the National Academy of Sciences 117 , 23021-23032 (2020).

Keller, A. J. et al. A disinhibitory circuit for contextual modulation in primary visual cortex. Neuron 108 , 1181-1193 (2020).

Kleinman, M., Chandrasekaran, C. & Kao, J. A mechanistic multi-area recurrent network model of decision-making. Advances in neural information processing systems 34 , 23152-23165 (2021).

Saxena, S., Russo, A. A., Cunningham, J. & Churchland, M. M. Motor cortex activity across movement speeds is predicted by network-level strategies for generating muscle activity. Elife 11 , e67620 (2022).

Soo, W. & Lengyel, M. Training stochastic stabilized supralinear networks by dynamics-neutral growth. Advances in Neural Information Processing Systems 35 , 29278-29291 (2022).

Stroud, J. P., Watanabe, K., Suzuki, T., Stokes, M. G. & Lengyel, M. Optimal information loading into working memory explains dynamic coding in the prefrontal cortex. Proceedings of the National Academy of Sciences 120 , e2307991120 (2023).

Goudar, V., Peysakhovich, B., Freedman, D. J., Buffalo, E. A. & Wang, X.-J. Schema formation in a neural population subspace underlies learning-to-learn in flexible sensorimotor problemsolving. Nature Neuroscience 26 , 879-890 (2023).

Soo, W., Goudar, V. & Wang, X.-J. Training biologically plausible recurrent neural networks on cognitive tasks with long-term dependencies. Advances in Neural Information Processing Systems 36 , 32061-32074 (2023).

Fascianelli, V. et al. Neural representational geometries reflect behavioral differences in monkeys and recurrent neural networks. Nature Communications 15 , 6479 (2024).

Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T. & Wang, X.-J. Task representations in neural networks trained to perform many cognitive tasks. Nature Neuroscience 22 , 297-306 (2019).

Liu, Y. & Wang, X.-J. Flexible gating between subspaces in a neural network model of internally guided task switching. Nature Communications 15 , 6497 (2024).

Hopfield, J. J. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences 79 , 2554-2558 (1982).

Yamins, D. L. et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences 111 , 8619-8624 (2014).

Kriegeskorte, N. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science 1 , 417-446 (2015).

Yamins, D. L. & DiCarlo, J. J. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience 19 , 356-365 (2016).

Spoerer, C. J., McClure, P. & Kriegeskorte, N. Recurrent convolutional neural networks: a better model of biological object recognition. Frontiers in psychology 8 , 1551 (2017).

Nayebi, A. et al. Task-driven convolutional recurrent models of the visual system. Advances in neural information processing systems 31 (2018).

Kubilius, J. et al. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv (2018).

Schrimpf, M. et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv (2018).

Kubilius, J. et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems 32 (2019).

Kar, K., Kubilius, J., Schmidt, K., Issa, E. B. & DiCarlo, J. J. Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior. Nature neuroscience 22 , 974-983 (2019).

Kietzmann, T. C. et al. Recurrence is required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences 116 , 21854-21863 (2019).

Spoerer, C. J., Kietzmann, T. C., Mehrer, J., Charest, I. & Kriegeskorte, N. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS computational biology 16 , e1008215 (2020).

van Bergen, R. S. & Kriegeskorte, N. Going in circles is the way forward: the role of recurrence in visual inference. Current Opinion in Neurobiology 65 , 176-193 (2020).

Zhuang, C. et al. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences 118 , e2014196118 (2021).

Lindsay, G. W. Convolutional neural networks as a model of the visual system: Past, present, and future. Journal of cognitive neuroscience 33 , 2017-2031 (2021).

Ratan Murty, N. A., Bashivan, P., Abate, A., DiCarlo, J. J. & Kanwisher, N. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nature communications 12 , 5540 (2021).

Storrs, K. R., Kietzmann, T. C., Walther, A., Mehrer, J. & Kriegeskorte, N. Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting. Journal of cognitive neuroscience 33 , 2044-2064 (2021).

Nayebi, A. et al. Recurrent connections in the primate ventral visual stream mediate a trade-off between task performance and network size during core object recognition. Neural Computation 34 , 1652-1675 (2022).

Benigno, G. B., Budzinski, R. C., Davis, Z. W., Reynolds, J. H. & Muller, L. Waves traveling over a map of visual space can ignite short-term predictions of sensory input. Nature Communications 14 , 3409 (2023).

Thorat, S., Doerig, A. & Kietzmann, T. C. Characterising representation dynamics in recurrent neural networks for object recognition. arXiv preprint arXiv:2308.12435 (2023).

Nayebi, A., Rajalingham, R., Jazayeri, M. & Yang, G. R. Neural foundations of mental simulation: Future prediction of latent representations on dynamic scenes. Advances in Neural Information Processing Systems 36 (2024).

Linsley, D. et al. Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex. Advances in Neural Information Processing Systems 36 (2024).

Doerig, A. et al. Visual representations in the human brain are aligned with large language models. arXiv preprint arXiv:2209.11737 (2024).

Salehi, S., Lei, J., Benjamin, A. S., Müller, K.-R. & Kording, K. P. Modeling attention and binding in the brain through bidirectional recurrent gating. bioRxiv 2024-09 (2024).

McNeal, N., Deb, M. & Murty, A. R. Small-scale adversarial perturbations expose differences between predictive encoding models of human fmri responses. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models (2024).

Liang, M. & Hu, X. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , 3367-3375 (2015).

Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).

Nair, V. & Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning , 807-814 (2010).

Franceschi, D. D. & Jang, J. H. Demystifying batch normalization: Analysis of normalizing layer inputs in neural networks. In Optimization and Learning: Third International Conference, OLA 2020, Cádiz, Spain, February 17-19, 2020, Proceedings 3 , 49-57 (Springer, 2020).

Mastrogiuseppe, F. & Ostojic, S. Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron 99 , 609-623 (2018).

Dubreuil, A., Valente, A., Beiran, M., Mastrogiuseppe, F. & Ostojic, S. The role of population structure in computations through neural dynamics. Nature neuroscience 25 , 783-794 (2022).

Gold, J. I. & Shadlen, M. N. The neural basis of decision making. Annu. Rev. Neurosci. 30 , 535-574 (2007).

Romo, R., Brody, C. D., Hernández, A. & Lemus, L. Neuronal correlates of parametric working memory in the prefrontal cortex. Nature 399 , 470-473 (1999).

Raposo, D., Kaufman, M. T. & Churchland, A. K. A category-free neural population supports evolving demands during decision-making. Nature neuroscience 17 , 1784-1792 (2014).

Miyashita, Y. & Chang, H. S. Neuronal correlate of pictorial short-term memory in the primate temporal cortexyasushi miyashita. Nature 331 , 68-70 (1988).

Williams, A. H., Kunz, E., Kornblith, S. & Linderman, S. Generalized shape metrics on neural representations. Advances in Neural Information Processing Systems 34 , 4738-4750 (2021).

Battista, A. & Monasson, R. Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks. Physical Review Letters 124 , 048302 (2020).

Driscoll, L. N., Shenoy, K. & Sussillo, D. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience 1-15 (2024).

Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine 29 , 141-142 (2012).

Xiao, H., Rasul, K. & Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).

Krizhevsky, A. & Hinton, G. Learning multiple layers of features from tiny images. The CIFAR-10/CIFAR-100 dataset (2009).

Deng, J. et al. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , 248-255 (Ieee, 2009).

Cubuk, E. D., Zoph, B., Shlens, J. & Le, Q. V. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719 2 , 7 (2019).

Jacob, B. et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition , 2704-2713 (2018).

He, K., Zhang, X., Ren, S. & Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , 1026-1034 (2015).

Arnoldi, W. E. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quarterly of applied mathematics 9 , 17-29 (1951).

Freedman, D. J., Riesenhuber, M., Poggio, T. & Miller, E. K. Categorical representation of visual stimuli in the primate prefrontal cortex. Science 291 , 312-316 (2001).

Nieder, A., Freedman, D. J. & Miller, E. K. Representation of the quantity of visual items in the primate prefrontal cortex. Science 297 , 1708-1711 (2002).

Roitman, J. D. & Shadlen, M. N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. Journal of neuroscience 22 , 9475-9489 (2002).

Funahashi, S., Bruce, C. J. & Goldman-Rakic, P. S. Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. Journal of neurophysiology 61 , 331-349 (1989).

Majaj, N. J., Hong, H., Solomon, E. A. & DiCarlo, J. J. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience 35 , 13402-13418 (2015).

Freeman, J., Ziemba, C. M., Heeger, D. J., Simoncelli, E. P. & Movshon, J. A. A functional and perceptual signature of the second visual area in primates. Nature neuroscience 16 , 974-981 (2013).

Linsley, D., Karkada Ashok, A., Govindarajan, L. N., Liu, R. & Serre, T. Stable and expressive recurrent vision models. Advances in Neural Information Processing Systems 33 , 10456-10467 (2020).

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer:[Yes]

Justification: We have made 6 claims. The claim on dynamical expressitivity analysis can be found in Section 2, the claim on new training algorithm can be found in section 3, the autonomous nature of our model can be observed in Figure 3, the analytical toolkit is described in Section 4, the image-computable models can be found in Figure 5, and the neural data analysis can be found in Figure 6.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have included a dedicated part in our discussion to discuss limitations, where we reference other relevant parts of the paper where we presented the limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All training details required to reproduce our results can be found in Appendices B and C.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, the link to our code is provided in the supplementary.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: For all experiments, we have provided details in both the main text and the supplementary.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have done a paired t-test for our claims in Figure 6.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided hardware specifications and time benchmarks in Appendix C.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly cited all sources of experimental data that we have used.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a hybrid architecture that integrates continuous-time recurrent neural networks (RNNs) with convolutional neural networks (CNNs), named CordsNet, to improve biological realism in vision models. The authors claim that CordsNet matches CNN performance on benchmarks like ImageNet while showing increased robustness to noise. They also present a toolkit for analyzing these models and demonstrate the model's ability to capture time-dependent neural activity.\n\nSTRENGTHS:\n1. The proposed model demonstrates connections to neural behavior and shows effectiveness in dealing with continuous-time modeling. It might inspire future biologically feasible network design.\n2. The sharing of weights for convolutional and recurrent operations is novel since these are typically done sequentially in a backbone.\n3.  The introduction of tools for analyzing convolutional structures in dynamical systems is potentially helpful for future research, but more details are needed.\n\nWEAKNESSES:\n1. The comparative result can be conducted more comprehensively. Some comparative results are not conducted on models with the same temporal processing power (Figure 3 only CNN shown).  Comparison to similar temporal models such as convolutional RNN  and vanilla RNN are lacking. These models can likely exhibit similar behaviors while requiring a less complex training scheme. \n2. The choice of image datasets is questionable. The dataset is in discrete-time space, and continuous-time training might not provide any advantages over traditional CNN. The proposed model enables intra-batch information flow between images, which can provide certain robustness to discrete data. Still, the advantages should be more prominent when images in a batch are naturally sequential and have dense temporal information.\n\nQUESTIONS:\nWhat’s your model’s advantage over a simple sequential design of convolutional preprocessing followed by a dynamical RNN?\n\nLIMITATIONS:\n1. The use case of the proposed model is limited and the advantage over discrete-time models is not clear. I believe continuous-time vision tasks are needed to fully demonstrate the effectiveness of this method. Tasks such as video prediction, and video feature extraction might be good candidates. Small-scale experiments can be also conducted on augmented image datasets with more temporal information.\n2. The proposed model is novel, but if it provides advantages over existing hybrid models remains questionable. It shares similar design intuitions to convolutional RNN, effectively mimicking the visual preprocessing of visual information followed by temporal processing in a later stage in the brain. Such designs are widely used in the form of CNN + vision Transformer in video processing (XMem, Cheng & Schwing, 2022) and video generation (Video Diffusion Model, Ho et al., 2022) domain where both spatial and temporal processing are conducted concurrently.  However, a comparison with such an existing design is not provided. The provided CNN baseline is relatively weak compared to the above-mentioned recent architecture. \n3. Intuitively the spontaneous activity attraction is functionally similar to weight decay, whether this regularization term provides advantages remains questionable. Ablation studies can be conducted to provide more rationales behind your design choice since multiple design options are present in your model.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBiological neural networks are continuous-time dynamical systems. However, the previous models that are carefully designed to explain biological networks are discrete-time systems or even non-dynamical ones like convolutional neural networks (CNNs). However, the model that best explains biological vision and achieves high performance on downstream tasks is CNNs. This work fills the gap by creating novel networks that combine CNNs and continuous-time recurrent neural networks called CordsNet. CordsNet is evaluated on a standard ImageNet classification task and also across several standard cognitive neuroscience tasks.\n\nSTRENGTHS:\n1. Biological realism during inference: CordsNet might be one of the most biomimetic deep learning models, as its dynamics are more realistic than other state-of-the-art models like CORnet-RT and CORnet-S. It is motivated by neural dynamics models.\n\n2. A broad range of cognitive tasks are studied: Beyond the ImageNet classification task, this work investigates the responses of CordsNet on several standard cognitive tasks that neuroscientists use to study humans and animals.\n\n3. Benefit of recurrence dynamics over CNN: CordsNet demonstrates the advantage of classification with noisy images over CNN. This suggests the missing piece to make computer vision models more robust, like humans.\n\n4. Great visualization: The visualization is very insightful and provides intuition on how CordsNet works.\n\nWEAKNESSES:\n1. Other methods like CORnet-RT and CORnet-S are not evaluated on the cognitive tasks that plug RNN. Therefore, we cannot see if CordsNet is better than others on this benchmark. I believe they can perform the same tasks but they may have different characteristics or different representations than CordsNet\n\n2. CordsNet is trained to predict stimuli at a certain duration, but it cannot accurately predict other durations. \n\n3. The task is supervised image classification where the models learn from rich human labels. Moreover, brains perform with minimal or no strong supervision from others. Instead, brains may learn in a self-supervised manner. It is possible to explore self-supervised learning tasks that are more like what brains do.\n\n4. Embodied tasks are not included: To study the benefits and characteristics of neural dynamics, embodied tasks may be more interesting.\n\nQUESTIONS:\n1. Citation [36] is wrong. That one is CORnet-S, not CORnet-RT [1].\n\n2. Why are brain scores low compared to the leader board on the same public datasets? Is the score a neural predictivity score defined in the Brain Score paper [2] or a different metric?\n\n3. How are brain-scores of the proposed method compared to CORnet-S and CORnet-RT? How many total parameters of the CordsNet compared to those models?\n\nReference\n\n[1] Kubilius, Jonas, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel LK Yamins, and James J. DiCarlo. \"Cornet: Modeling the neural mechanisms of core object recognition.\" BioRxiv (2018): 408385.\n\n[2] Schrimpf, Martin, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar et al. \"Brain-score: Which artificial neural network for object recognition is most brain-like?.\" BioRxiv (2018): 407007.\n\nLIMITATIONS:\nA lot if memory is required during training.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a recurrent convolutional neural network architecture and training algorithm that simulates the biological visual system of mammals. The technical novelty is mostly in the training algorithm which is meant to mimic biological systems, with a first stage of spontaneous activities followed by learning and then again spontaneous activities. Since this process is too computational intensive, the authors introduce approximations, such as initializing from a supervised trained network. The empirical analysis shows that the method is not too far from standard supervised learning while being robust to noise and while exhibiting a behavior that better matches biological systems. Also, the approach is used to predict brain activity data as an application. \n\nDisclaimer: My background is machine learning rather than comp. neuroscience. I am not qualified to assess the impact that this work can have in that community and my assessment is mostly limited to the ML side of this contribution.\n\nSTRENGTHS:\n+ simple and intuitive architecture\n+ generally well written and fluent paper\n+ the loss definition (in its original formulation of eq. 2) is very interesting and novel\n+ nice analysis of how this work relates and contributes to the field of comp. neuroscience\n+ overall motivation and research topic\n\nWEAKNESSES:\n- important technical details are missing. For instance, in sec. 2 only few lines are used to give a high level description of the architecture. Too much material that should be in the main paper is placed in the appendix (e.g., comparison to supervised CNNs mentioned in the abstract) . Overall the paper is not self-contained and misses critical details.\n- there is a big approximation between the original loss function and what actually is minimized in practice. If the goal is to simulate a biological system, it is unclear to me how the approximation made fit in the context of that goal.\n- overall, claims are not entirely supported. For instance, it is not true that on ImageNet the performance is \"comparable\" to a standard CNN. A difference of 5% is big in that context. Some accomplishments of this work should be toned down a bit.\n- missing references: There is prior work on recurrent CNNs that is not cited. For instance, \nRecurrent Convolutional Neural Network for Object Recognition by Liang et al. CVPR 2015 uses the same architecture as far as I can tell.\n- missing ablations: The authors well ablated the contribution of the recurrent and convolutional part of their model. However, their approach makes lots of design choices which are not very well justified empirically. For instance, what happens if fewer recurrent iterations are used? or what happens if only the top-most fully connected layer are made recurrent? what happens if some terms of the loss are removed? etc.\n\nQUESTIONS:\nI am curious whether the model oscillates when there are multiple interpretation of the input. In the simplest setting, the input could be linearly \"mixed-up\". I am also curious whether \"easy\" examples require less iterations to converge.\n\nLIMITATIONS:\nNo concern.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
In this work, the authors proposed CordsNet (Convolutional RNN dynamical system), which provides incorporates a conventional neuroscientific model of RNN incorporating the convolutional layer.

STRENGTHS:
1. The authors analyzed the proposed dynamic convolutional RNN from different aspects. 

2. Derived batch normalization for the case of linear dynamic RNN

WEAKNESSES:
1. The authors failed to compare it with a series of works named Deep Equilibrium Models (DEQs), which model a layer or a block of layers including the non-linearities as a fixed-point problem.  CordsNet is limited to the linear part of the layer while DEQs are not. And DEQs achieved SoTA performance on many large-scale tasks.

2. In the comparison seems Convolutional LSTM/GRU is missing, which is an important family of RNN with Conv layers. Could the authors explain why they are not necessary to be included in the benchmarks?

3. The contribution points are really scattered and don’t feel elaborated enough. Maybe it’s more suitable as a journal paper with more elaborated experimental details for each section.

QUESTIONS:
1. Is the analytical formulation of CordNets limited to only linear layer?

LIMITATIONS:
The authors discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInspired by neuroscience, the authors propose to introduce recurrent connections in conventional convolutional neural networks (CNNs). The resulting model shows comparable performance with regular CNNs, but exhibits higher noise robustness. The authors also developed a toolkit to analyze the resulting architecture, using iterative and dimensionality reduction methods. The architecture is used to model complex cognitive tasks, mimicking the higher order visual areas in the monkey.\n\nSTRENGTHS:\n- This paper seamlessly weaves inspiration from neuroscience, dynamical systems theory, learning algorithms, analysis methods, performance, and modeling and replicating results from neuroscience experiments.\n\n- Introducing continuous dynamics into CNN+RNN seems to exhibit really interesting properties.\n\n- Analysis of the dynamical behavior by finding the eigenvalues directly from the recurrent weight matrices is interesting. This has been used to uncover some really interesting dynamical patterns dependent on the time of classification (fater or slower inference).\n\n- The use of the model in various visual cognitive tasks (in V4 and IT) is also quite impressive.\n\nWEAKNESSES:\n- Too much technical details are hidden in the appendix. Please at least provide a brief sketch in the main text.\n\nQUESTIONS:\nQuestions\n\n- Isn't the task in Fig 5D handled by area MT? Have you considered computing the BrainScore for this? In general, can results in Fig 5 be compared to the experimental literature?\n\nComments\n\n- There is an earlier recurrent CNN than the one cited: Ming Liang and Xiaolin Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3367–3375, 2015.\n\nLIMITATIONS:\nThe discussion/conclusion provides adequate assessment.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}