CircuitNet: A Generic Neural Network to Realize Universal Circuit Motif Modeling
Authors: Yansen Wang
XINYANG JIANG
Kan Ren
Caihua Shan
Xufang Luo
Kaitao Song
Dongsheng Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Bio-inspired neural network
Deep Learning

Abstract: The successes of artificial neural networks (ANNs) are largely attributed to mimicking the human brain structures. Recent advances in neuroscience revealed that neurons interact with each other through various kinds of connectivity patterns to process information, in which the common connectivity patterns are also called circuit motifs. However, many existing ANNs can only model one or two circuit motifs in their architectures, so that their performance may drastically vary among different types of machine learning tasks. 
In this paper, we propose a new type of neural network inspired by the architectures of neuronal circuits, namely Circuit Neural Network (CircuitNet). In CircuitNet, a group of densely connected neurons, namely circuit motif unit (CMU), form the basic unit of the network, which is capable of modeling universal circuit motifs by adjusting the weights within the CMUs. Compared with traditional feed-forward networks, CircuitNet has the ability to model more types of neuron connections such as feed-back and lateral motifs.
Inspired by the locally dense and globally sparse structure of the human brain, several iterations of signal transmission among different CMUs are achieved by sparse connections through the input ports and output ports of different CMUs. 
Experiments have demonstrated that CircuitNet can outperform popular neural network architectures in function approximation, reinforcement learning, image classification, and time series forecasting tasks.

Under review as a conference paper at ICLR 2023
CIRCUITNET: A GENERIC NEURAL NETWORK TO RE- ALIZE UNIVERSAL CIRCUIT MOTIF MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
The successes of artificial neural networks (ANNs) are largely attributed to mim- icking the human brain structures. Recent advances in neuroscience revealed that neurons interact with each other through various kinds of connectivity patterns to process information, in which the common connectivity patterns are also called circuit motif's. However, many existing ANNs can only model one or two circuit motifs in their architectures, so that their performance may drastically vary among different types of machine learning tasks. In this paper, propose of neural network inspired by the architectures of neuronal circuits, namely Cir- cuit Neural Network (CircuitNet). In CircuitNet, group of densely connected neurons, namely circuit motif unit (CMU), form the basic unit of the network, which is capable of modeling universal circuit motifs by adjusting the weights within the CMUs. Compared with traditional feed-forward networks, CircuitNet has the ability model more types of neuron connections such as feed-back and lateral motifs. Inspired by the locally dense and globally sparse structure of the human brain, several iterations of signal transmission among different CMUs are achieved by sparse connections through the input ports and output ports of dif- ferent CMUs. Experiments have demonstrated that CircuitNet can outperform popular neural network architectures in function approximation, reinforcement learning, image classification, and time series forecasting tasks.
type
INTRODUCTION
In the past decades, artificial neural networks (ANN) (MeCulloch & Pitts 1943) have been widely used as function estimators to solve regression and classification problems, which massively push deep learning forward in vastly different fields such as computer vision (He et al., 2016). natural language processing (Vaswani et al.) 2017), deep reinforcement learning (Mnih et al. (015)), etc. The successes of ANN are largely attributed to mimicking the simplified human brain structures. For example, the original multi-layer perceptrons (MLP) are collections of neurons organized as layers (Minsky & Papert] 1969] Rosenblati] 1958), and signals are controlled and transmitted be- tween layers via linear transformations and non-linear activation functions, just like the synapses in biological brain. Recently, the network architectures become rather complex, but their basic units such as convolutional layers and recurrent layers are still different abstractions of human ner- vous systems (Fukushima& & Miyake, (982, Hubel & Wiesel 1968; Lindsay 2021] Kietzmann et al.) 2019 van Bergen & (2eggeskorte|22220)
In recent years, understanding on both deep learning and neuroscience has made great advance, and it is time to rethink how artificial neural network designs can be further inspired by neuroscience. Following previous ANNs, we simplify the signal from a neuron as real number, and focus more on how to model the signal transmission and connectivity patterns among neurons. Recent findings in neuroscience (Luo 202 Peters) 1991 Standring 2021 Swanson, 2012) emphasized the role of specific patterns of synaptic connectivity in neuron communication across different brain regions. These patterns are analogous the connection of neurons in ANNs, and in the rest of this section we will introduce how these findings can inspire the ANN design.
One line of advances in neuroscience revealed that neurons interact with each other through vari- ous kinds of connectivity patterns, namely the circuit motifs (Luo) 202T). There are four types of most common circuit motifs, including feed-forward excitation and inhibition, feed-back inhibition,

---

Under review as a conference paper at ICLR 2023
lateral inhibition, and mutual inhibition, as shown in Figure 1 However, neurons in most existing ANNs (including those with residual connections (He et al. 2016)) that incorporate merely feed- forward networks only function as the wred-forward excitation and inhibition patterns. Even if some recurrent structures, e.g., RNNs (Hochreiter & Schmidhuber, 1997; Cho et al. (014). are able to model cyclic patterns, they may fail with complex interactions among upstream neurons before information comes in.
Another typical property of the biological nervous system is the local density and global spar- sity (Swanson 2012). Although single neuron can have thousands of synapses, most of them are located within a small region and form a functional group for specific tasks (Hawkins & Ahmmad, 2016). Only a few of them travel to other functional groups as bridges between regions (Peters) 1991 Standring 2021 Swanson 2012). In conclusion, the biological nervous system has a locally dense and globally sparse connection structure. This is typically not true in ANNs where if we consider the vectors in a network as functional groups with some specific meanings, connections within group are mostly missing but those between groups are of high density. For instance, the attentions between vectors in the original type of transformer form a complete graph and are of the highest density (Vaswani al., 2017). On the other hand, many parameters within neural net- work have been proved to be redundant (Glorot et al.) [011] [Schwarz et al. [021]] Wen et al. [016]] Yoon & Hwang 2017) and the global sparsity of the biological neurons can provide insight for us to reduce the complexity of the neural network (Mocanu et al. 2018 Pessoa 2014 Strogatz
2001
As a result, the design of our proposed ANN follows two principles that previous work does not fully consider: 1) the low-level interaction pattern among neurons should contain different types of circuit motifs (universal circuit motifs), and 2) neuron connections should follow locally-dense and globally-sparse pattern to form the high-level structure. In this paper, we propose a new type of architecture named Circuit Neural Network (CireuitNet) for neural network design. CircuitNet uses densely connected neurons as basic functional structure called Circuit Motif Unit (CMU) to sup- port universal circuit motifs. The communication among different CMUs is achieved by the sparse connections between subsets of special neurons in CMUs, namely input/output ports. Experiments on both synthetic and real-world datasets prove that, with comparable or even fewer parameters, CircuitNet can outperform popular neural network architectures in various types of tasks, demon- strating its effectiveness and generalizability in machine learning. Notably, CircuitNet outperforms multi-layer perceptron in function approximation, image classification and reinforcement learning tasks, outperforms MLP-based, CNN- and Transformer- bbasedd neural networks in image clas- sification tasks on CIFAR-10, CIFAR-100 and ImageNet datasets, and outperforms CNN-based, RNN-based and Transformer-based neural networks in time series forecasting tasks.
2 CIRCUIT NEURAL NETWORK
In this section, we introduce the detailed implementation of the proposed Circuit Neural Network, including how to model the signal transmission between neurons in Circuit Motif Unit (CMU) to support universal circuit motifs and the overall locally dense globally sparse connection structure.
2.1 NEURON SIGNAL TRANSMISSION FOR UNIVERSAL CIRCUIT MOTIFS
In this paper, the signal from a neuron is simplified as a real number. Given two or more connected neurons, their interaction is achieved by signal transmission, which is modeled as a transformation function g(;:)) -). As shown in Figure I we aim at designing a transformation function that can model four types of neuron connection patterns (i.e., circuit motifs), including feed- forward,, mutual, feed- back and lateral motifs, corresponding to the four patterns in recent biological findings (Luo 2021). In feed-forward and mutual motifs, the input signal depends merely on the state of the previous neuron, while in the other two motifs, an input signal may be affected by the state of another neuron or the target neuron itself. In the following paragraphs, we present the design of our proposed transformation function and claborate why this design can model different circuit motifs.
Linear Transformation One of the simplest transformation used to pass signals from neuron 4y
to xy can be written as:
g( Wer
(1)
2

---

Under review as a conference paper at ICLR 2023
(a) Feed-forward
(h) Mutual
(c) Feed-back
(d) Lateral
Figure 1: Different motifs among neurons Luo 2021). A neuron in green is the target neuron in
motif and is updated in every iteration based on the signals from other neurons.
Wi a learnable parameter indicating the intensity of influence from ž to 24. Most of the previous ANNs also follow the idea of linear transformation because is straightforward yet effective mod- eling feed-forward and mutual motifs where input signals to neuron are considered independently and added up simply.
Neuron-wise Attention Linear transformation modeling the feed-forward and mutual motifs as- sumes that the influence of an input signal is independent with other signals. However, this is some- times over-simplified since there are also feed-back and lateral connections in human brains where the intensity of the input signal may be influenced by signals from other neurons and the target ron itself. To equip CircuitNet with the ability to model these more complicated patterns, we need find some transformations which take the interactions between input signals into consideration.
One concept we can borrow from traditional ANNs to model the interactions is the self-attention mechanism (Vaswani al.) 2017), but in CircuitNet, it is calculated between neurons (scalars) rather than groups (vectors). Therefore, we name it neuron- wise attention and formulate it as: g 04) VjX,
exp( ( (K) Σ. (Qp((QxX) (K,))
Ozj softmax ( (Q (Kx))(,X))) (KjX))
X contains the states of all the M neurons in group of connected neurons and Vi, K,, Qi RIxM are learnable linear projection weights for calculating value, key and query (which are scalars) for the i-th neuron.
This type of transformation can used to model the feed-back motif well because the influence of the input signal is adjusted by the target neuron state.
Product between Neurons Another way to model the interactions between input signals to directly caleulate the product of every pair of neurons, i.e.
g(a,;,y, 2x) Wonaeyya
where Wajk is a learnable parameter to model the intensity of the combined influence from æj, 24. It is divided by 2 eliminate the influence to the gradient caused by the quadratic term. Note that j = i or &o = i is allowed in this formula so that the lateral motif (input signal is affected by another neuron) and the feed-back motif (input signal affected by the target neuron) can be modeled at the same time.
In order to model the universal circuit motifs, different transformations will used together to get more comprehensive depiction of the circuit motifs. For example, we can use the product between neurons and the linear transformation together and write the transformation function as below:
g( TJ 2h)
2 W Wo + Viya + Vi
(5)
In this way, all four patterns illustrated in Figure 1 can be modeled properly.
2.2 LOCALLY DENSE AND GLOBBLLY SPARSE NEURON CONNECTIONS
In the previous section, we have introduced how CircuitNet models the signal transmission among the connected neurons. Now introduce how neurons connected in the proposed CircuitNet.

---

Under review as a conference paper at ICLR 2023
Model Ounpott
CircuitNet
(CMU)
Model impuls
Input
Output
Port
Port
Figure 2: CircuitNet architecture. As generic neural network, CircuitNet can be used in different tasks, take different forms of inputs and give corresponding outputs (left). It consists of several Circuit Motif Units (CMUs) connected with each other through their ports and maintains a global sparsity (middle). Within single CMU, neurons are interconnected densely in complicated man- ner in order to model a universal circuit motif, e.g., the green line connecting two neurons can be a linear transformation and the red lines be transformation involving 3 neurons (right).
Circuit Motif Unit (CMU) Following the locally- and globally-sparse principle, CircuitNet contains clusters of neurons forming several densely connected neural circuit groups as its basic units, namely Circuit Motif Units (CMU). X' {X, X2, XN3 with N units and X; (Ia T12, xaM) with M neurons per unit. By applying the transformation function introduced in the previous section and dense connection, CMU is able model all types of circuit motifs, i.e., universal circuit motifs. These CMUs are then sparsely connected by their input/output ports. The overall architecture CircuitNet and the structure of CMU are shown in Figure
As result, after initialized by the task-specific input, different CMUs cooperate with each other by a certain number of iterations of intra-unit and inter-unit signal transmission. Specifically, when we look at a single unit, its state is updated based on two sources of signal input: the internal update among neurons within the CMU with the aforementioned transformations, and the received signals from other CMUs with edges to it, as follows:
X f(g (8) (k-1); X, (1- + Sn hij (Xj
(k- (k-1)) + b, ()),
(6)
where e denotes the set of connected CMU pairs. An edge Ejt E means the i-th CMU will receive signals from the j-th CMU at every iteration. X; denotes the state of the i-th CMU at iteration k. Pi (k) (::-) :) is the transformation function described in Section 2.1 to model the universal circuit motif, b (.) is the transformation function to process the signals from other CMUs, and f(-) the normalization and activation function. b, is the bias term. To simplify the notation without loss of generality, & may be omitted elsewhere in the formulas when unconcerned about specific CMU or iteration.
Inter-unit Signal Transmission To enable inter-unit interactions by signal transmission, specific sub-set of neurons from the CMU are responsible for sending out signals based on their own states, and another set for receiving signals from others. We call these neurons input and output ports, respectively, In this way, the update of neurons in a CMU through ports can be written as:
hag (Xj) Pin.i (Wji(Pout out. Xz)).
Here, Wii ji is learnable linear transformation from the j-th CMU the i-th. Pin. P out are binary vectors with non-zero elements, identifying which p neurons are the input output ports, and is the element-win multiplication which picks these neurons out when processing the signals received or sent by the CMU. Note that we do not set the strict limit that one neuron can only work one role a CMU, which means the input port and output port may overlap on some neurons. This can actually help the signal transmit faster between CMUs without direct edge between them.
3 ADAPTATION OF CIRCUITNET IN SPECIFIC APPLICATIONS
As generic network architecture, we expect that CircuitNet can be adapted to all kinds tasks just like the MI.P does. In order to CircuitNet as general function approximator, different portions
4

---

Under review as a conference paper at ICLR 2023
of the input data should be fed into different Circuit Motif Units as initialization. Ideally, each CMU is responsible for a specific sub-task. After that, CircuitNet will perform several iterations of signal transmission as aforementionedd and task-specific aggregation transformation will finally extract the information from the network and make a prediction.
The general structure of CircuitNet is quite different from traditional ANNS although they both take the neurons as the most basic building blocks, so that some commonly-used techniques may or may not be applied directly in CircuitNet. Before diving into the details of the experiments, in this section, we briefly show typical cases of how CircuitNet can be adapted in specific tasks. More details on the designs of our model under different circumstances are included in Appendix B
3.1 DEEP REINFORCEMENT LEARNING TASKS
One of the tasks we choose to evaluate CircuitNet is deep reinforcement learning (DRL). Here, choose a range of challenging continuous control tasks with proprioceptive observations (e.g.. positions & velocities), which have been widely used as the benchmarks in DRL research (Schulman al., 2015; 2017: Haarnoja et al., 2018). In these tasks, MLPs are usually used as backbones for policy and value networks. Thus, we directly substitute MLPs with CircuitNet without changing other parts in the learning process.
3.2 IMAGE CLASSIFICATION
Another chosen task is image classification in computer vision investigate how CircuitNet behaves large-scale structured data. Individual CMUs CircuitNet are designed to work for specific tasks or embed specific meanings. In computer vision tasks, it is natural to use CMUs to represent features in different locations. Specifically, CircuitNet will start the signal transmission after a convolutional layer initializes neurons states in arranned CMUs with representations of image regions.
Previous MLPs work well on small datasets, but can hardly scale up due to the gradient vanishing problem (Nair& & Hinton| 2010). Therefore, many advanced techniques have been proposed to help ANNs to work with deeper networks (He et al. [2166] [Simonyan & Zisserman 2014). Correspond- ingly, also adopt similar ideas to help CircuitNet perform better when the numbers of CMUs and iterations of signal transformation are large including weight sharing, normalization, and feature aggregation. Also, to stabilize the training when we stack the quadratic terms over large number of iterations, we propose a novel clipped sinusoidal activation function ClipSin(r), as follows:
A + e(xx 3),
<< -3,
ClipSin(1)
π x), A Asin(
- -3 53
(8)
A+ 1 3),
3.
The activation function is half period of sinusoidal function within [-3, 3], and is clipped linear function outside with small slope to prevent zero gradients. The amplitude is controlled by hyperparameter A. Practically, we set 0.01 and A 3.388235. In this way, we can avoid the gradient exploding problem caused by ReLU activation or the gradient vanishing problem caused by Tanh activation when optimizing deep CircuitNet models. theoretical justification on why we suppose it to work and how we choose the hyperparameters can be found in Appendix
3.3 TIME SERIES FORECASTING
As a general neural network architecture, CireuitNet can also be applied to sequence data in an autoregressive manner just like the Recurrent Neural Network (RNN) and its variants (Hochreiter & Schmidhuber, 1997 Cho et al.) 20144). To demonstrate how this can be achieved, we adopt CircuitNet to several time series forecasting tasks.
The modification of CircuitNet to an autoregressive model is simple and straightforward. First, during every iteration, the state of specific CMU will be updated with both the signals from all the CMUs and the information from the input at this timestep. Second, since the number of iterations to transmit signals may vary according the lengths of the input, the parameters shared across iterations. In this way, we rewrite Eq. (6) to an auto-regressive form as:
x (k) = (gg.(1( (k- 1). X 1)) + E S
h4p (X,
(k (K-1)) + es (x())) + bz).
(9)
5

---

Under review as a conference paper at ICLR 2023
e,(tk) o (Wim,,itt) + bin,i).
(10)
In this form of the state update equation, Win,i and bin,i are the cevatiooninvvrian learnable param- eters to map the k-th input in the sequence to the i-th group. After consuming the whole sequence, the final states of the CMUs are used to predict the following data points by linear forecasting layer. We call the CircuitNet in this form Recurrent CircuitNet (R-CircuitNet) to distinguish with the non-autoregresssve version introduced in Section 2
4 EXPERIMENTS AND ANALYSIS
In this section, we present the experiment results and analysis several tasks. Introductions to the datasets and the metrics used to compare CircuitNet with baselines can be found in the Appendix [A] Although our primary goal is not beat the state-of-the-art models, we list them as a comparison. A further study on the limitations and possible improvements are presented in the Appendix D.3
4.1 SYNTHETIC EXPERIMENTS
The universal approximation theorem states that a feed-forward network with bounded depth can be an arbitrarily good approximator for any continuous functions if the width is allowed to grow (Hornik et al. 1989 Scarselli & Tsoi 1998). However, in practice, it may take a lot of neurons and parameters fit rather simple function such the multiplication between two numbers, if linear transformation is the only allowed computation before activation function.
Because the Circuit Motif Units are able model universal circuit motifs, expect our network can be used to better approximate some complex functions with equal number of neurons or param- eters than MLP. Therefore, design four types of functions and generated the synthetic datasets, upon which we conduct the experiments to demonstrate the function approximation ability of CMU. Appendix A. includes details of the synthetic functions.
4.1.1 RESULTS ON SYNTHETIC DATASETS
The performance of the two different models on the synthetic datasets is presented in Table Mod- els under direct comparison for the same dataset have similar numbers of parameters. From the table, we can see that CMU obtains a significantly lower eтror than MLP across all the functions, indicating the better ability of CMU to perform universal function approximation by modeling the complex interactions between signals.
Table 1: Fitting emrors the synthetic datasets. n denotes the number of neurons.
Univariate Exponential
Univariate Periodic
Multivariate Linear
Multivariate Quadratic
Model
RRSE
Model
RRSE
Model
RRSE
Model
RRSE
MLP (u=7)
0.0021
MLP (a=64)
0.0057
(a=60)
0.0029
MLP (m=129)
0.0077
CMU (n=3)
0.0013
CMU (m=15)
0.0027
(n=15)
0.0010
CMU (n=25)
0.0062
4.1.2 VISUALIZATION OF THE LOss LANDSCAPES
To gain a deeper insight into the optimization process, we plot in Figure 3 the loss landscapes of MLP and CMU on the two multivariate datasets with GradVis (Chatzimichailiiis al. 2019). in which the parameter space is projected onto the plane formed by the eigenvectors corresponding to the largest two eigenvalues of the Hessian matrix.
From the figure, we can see that the landscapes are rather smooth for MLP and CMU, On the multivariate linear dataset, both models start to converge properly to the minimal point within a few epochs. However, the behavior of CMU differs with MLP on multivariate quadratic dataset. While CMU is still able to find the right direction to converge within 10 epochs, MLP drifts for over 50 epochs before the final converging process, Around these points, the eigenvalues of the Hessian matrix are close to zero or negative and the MLP model struggles to escape from the relatively flat saddle points (Dauphin et al. 2014: Chatzimichailidis et al. 2019). This suggests that by modeling different kinds of motifs, CMU can not only approximate the function more accurately, but also shape the loss landscape advantageously for quicker convergence.

---

Under review as a conference paper at ICLR 2023
(a) Linear, MI.P
(b) Linear, CMU
(c) Quadratic, MLP
(d) Quadratic, CMU
Figure 3: Loss landscape and the optimization trajectory of MLP and CMU in the first 100 epochs.
Croatie Croattes
-006
DrcathesLar
CroattedLLP
Exerpement ittrection
Le
Ant-v2
(b) HalfCheetah-v2
(c) Hopper-v2
Figure 4: Learning curves of CircuitNet and MLP on the first three continuous control benchmarks. contains the other two without any influence the conclusion.
Appendix C.4
4.2 DEEP REINFORCEMENT LEARNING
The goal of DRL algorithms is to achieve a lower sample complexity or higher sample efficiency, iee, to obtain a near-optimal policy using less interactions with the environments. We compare CircuitNet with MLP as the backbones the Soft Actor-Critic (SAC) (Haarnoja et al. 2018) policy on several challenging continuous control tasks from the OpenAl gym benchmark suite (Brockman ce al. 2016). Besides, to compare between different kinds of transformation methods, we run the SAC policy with several variants of our models. Refer to Appendix B.2 for more training details.
The learning curves on these continuous control benchmarks are shown in Figure 4 L, P. A after Cir- cuitNet represents the types of transformations (linear transformation, product between neurons, neuron-wise attention) respectively. Among the shown environments here, CircuitNet achieves significantly better performance than MLP 2 benchmarks and comparable on the third envi- ronment. It's notable that with product between neurons, CircuitNet fails in DRL tasks. This may be because the overly small number of neurons in CMU (in order achieve comparable number of parameters) conversely harms the expressiveness of the network.
4.3 IMAGE CLASSIHCATION
We conduct experiments on the image classification tasks on MNCST & Cortes] 2010
CIFAR- 10.0R-100100(Kizzeeeee et al. 2009) and ImageNet Deng et al. (2009) to demonstrate how Cir-
cuitNet performs after scaling up with techniques listed in Section
Table 2: Experiment results on MNIST.
Model
Accuracy
# Params (M)
3-layer MLP, 500 + 150 hidden units (LeCun et al.
1998
0.976
0.47
2-layer MLP, 800 hidden units (Simard el al. 2003
0.984
0.64
CircuitNet
0.993
0.14
Ensemble of ensembles et al. 2020
0.999
We design tiny CircuitNet containing 64 neurons and 48 ports in every CMU and for every batch of inputs with two iterations of signal transmissions. As shown in Table 2 comparing with two MLP models, CircuitNet achieves higher accuracy with less than 30% of parameters.
MNIST
CIFAR-10/100 For CIFAR datasets, we design larger CircuitNet containing 32 neurons and 24 ports in each CMU with 9 iterations of signal transmissions. As shown in Table 3 com-

---

Under review as a conference paper at ICLR 2023
Table 3: Experiment results CIFAR-10 and CIFAR-100.
Accuracy
Model
# Params (M)
# FLOPs (M)
CIEAR-I0 10
CIFAR-100
12-layer MLP
0.572
0.246
5.2
5.5
ResNet-15 (e
0.870
VII (Dosovitsury
2020
0.809
0.542
9.6
1283
Swin Transformer Li et 2021
0.846
0.554
26.7
1049
MLP-Mixer Tolstiknin
0.801
0.547
406
CircuiNNet-L
0.858
0.588
15.5
39.7
Circuideet-LAA
0.853
0.588
15.7
74.9
CircuiNeet-LeF
0.913
4.0
41.3
Efficient udaptive ensembling Bruno
2022
0.996
0.968
Table 4: Experiment results on ImageNet. The baseline results were from (Han et al. 2022
Model
Params (M)
FLOPs (B)
Aecuracy
ResNer- 18 (He al. 2016
12
1.8
0.706
ViT-B/16
Dusovissiyy
2020
86.4
55.5
0.779
PVT-Tiny Wang el al. 2021
13.2
1.9
0.751
ResMLP-S12 [oouvron al.
2022
16.7
0.770
Poolformer-SI1 2 Yu
0.772
Pyramid ViG-i Han
2022
10.7
1.7
0.782
CircuitNet
11.0
1.8
0.785
CoCa Yu et 2022a
2100
0.910
pare CircuitNet with some the existing SOTA image classification models, including CNN-based method (ResNet), transformer-based methods (ViT and Swin Transformer) and MLP-based method (MLP-Mixer). Since CIFAR-10/100 are relatively small, transformer-base methods achieve worse performance due lack of training data, e.g.. without pretraining ImageNet. CircuitNet outper- forms all the compared methods when applying product to model the interactions between neurons. Meanwhile, by doubling number of neurons and ports, CircuitNet achieves comparable performance with only linear or neuron-wis wise attention (i.c., CircuitNet-L and CireuitNet-L+A),
ImageNet As shown in Table 4 by further scaling up CircuitNet to 12 transmission iterations, on the large-scale ImageNet dataset, CircuitNet achieves better performance than existing state-of- the-art methods (with comparable or larger number of parameters) including CNN-based method (ResNet), MLP-based method (ResMLP), transformer-based methods (ViT-B/16, PVT-Tiny, and Poolformer-S12) and very recent GNN-based method (Pyramid ViG-Ti).
4.4 TIME SERIES FORECASTING
To test the ability of CircuitNet in the autoregressive manner, i.e., Recurrent CircuitNet (R- CircuitNet), to be applied sequential data, we include several multivariate time series forecasting tasks and compare our model to various kinds of time series forecasting methods.
The time series forecasting results are presented Table Statistical methods (ARIMA (Box al., 2015), GP (Roberts et al. 2013)), CNN-based model (TCN (Lea et al., (017)). RNN-based model (GRU (Coo et al.] 2014)), transformer pased-modedl (Autoformer Wu et al. 2021). Informer (Zhou al. (021)). and hybrid model (LSTNet (Lai et al. 2018)) all listed for comparison and they tend to be effective in one or two specific settings. As for R-CircuitNee, the three different variants perform similarly and outperform strong baselines by considerable margin across half of the tasks, and can still get comparable numbers others.
5 RELATED WORKS
Many previous neural network architectures designed by closely mimicking biological neural organizations (Bengio et al. 2013)). One of the earliest prior arts might be single/multi-lay percep- tron (Rosenblatt 1958 Minsky Papert 1969), which mathematical simplification of signals
8

---

Under review as a conference paper at ICLR 2023
Table 5: Experiment results time series datasets with different forecasting horizons (6, 24).
Solar-energy
Pems-bay
Metr-la
Model
6
24
24
RRSEL
R21
RRSE
R2;
RRSE.
R21
RRSE!
R21
RRSE.
R21
RRSEL
R21
ARIMA (Box
2015
0.202
0
0.365
0.847
0.532
0.741
0.548
0.723
0.575
0.687
0.742
0.441
GP Roberts et at. 2013
0.225
0.944
0.388
0.836
0,544
0.732
0.532
0.712
0.572
0.685
0.738
0.437
LST [ La et al 2018
0.218
0.950
0.360
0.857
0.521
0.752
0.560
0.706
0.564
0.697
0.732
0.475
GRU
Cho at
0.219
0,950
0.355
0.875
0.529
0.747
0.573
0.703
0.517
0.759
0.797
0.429
TCN
Lea a. 2017
0.210
0,954
0.445
0.804
0,487
0.785
0.551
0.718
0.570
0.698
0.705
0,545
Informer (Zhu et a.. 2021
0.239
0.948
0,358
0.889
0,522
0.755
0.547
0.733
0,579
0.697
0.747
0,516
R-CircuitNett-L
0.216
0,956
0,382
0.861
0.378
0.869
0.545
0.730
0,450
0.818
0.692
0.570
R-CireuitNet-L-AA R CircuitNet-I.o LPP
0.212
0.957
0.409
0.841
0.377
0.870
0.561
0.712
0,448
0.820
0.667
0,600
0.211
0.958
0.515
0.748
0.380
0.868
0.563
0.711
0.447
0.820
0.656
0.613
Autoformer Wu al. 2021
0.212
0.960
0.432
0.852
0.452
0.782
0.543
0.711
0.565
0.762
0.692
0.548
being received from dendrites and sent down to the axon once activated. Convolutional neural net- works (Fukushima& & Miyake|| 1982 LeCun et al. 1998 Krizhevsky et al. 2012) apply filters over an input image, which can learn feature maps akin to the planes of S-cells in the Neocognitron (Hubel & Wiesel 1968; Fukushima & Miyake, 1982 Lindsay 202T). Recurrent neural networks (Hochreiter
& Schmidhuber 1997 Cho et al. (2144) achieved great successes in modeling sequential dependen- cies among the data due to mimicking the recurrent connections within and between stages of the cascade the human brain (Kietzmann et al. 2019 van Bergen & Kriegeskorte| 2020p. In spiking neural networks (Maass 1997, Wožniak et al., 2020), each neuron fires when its input crosses threshold, which was inspired by the occurrence of information transmission when a membrane po- tential reaches threshold in the cortex (Gerstner & Kistler 2002] Ghosh-Dastidarr & Adeli 2200)). Inspired by visual information representation in human visual cortex (Qiao et al. 2018). the capsule neural network adds structures called "capsules" to CNN, each of which represents an object or ob- ject part and the combinations of which form more stable representations. Although the above neural network architectures have achieved huge successes in specific tasks, they are not generic enough to model the combinations of different types of neural circuits, e.g., a combination of feedforward excitation/inhibition, lateral inhibition and mutual inhibition in human brains (Luo 2021).
Besides neural network architectures, the successes of many popular operations in deep learning can also be explained from the perspective of biological neural processes. For instance, attention mech- anism has become an integral part in many sequence modeling algorithms, which can capture both local dependencies and long- -aaneaagee dependencies in the sequence or space (Bahdanau et al. 2014] Vaswani et al.) 2017, Veličković et al. 2018). In the biological visual systems, attention mechanism is also considered as an important ability to flexibly control with limited computational resources in nervous systems, in which neurons that represent the attended features have higher firing rates than those represent unattended features (Treue & Trujillo 1999; Lindsay 2020). Dropout (Srivas- lava et al. 2014) is another popular technique for preventing deep neural networks from overfitting, which is analogy to the characteristic of biological neural networks randomly switching neurons on
or off during training and inference (Faisal et al.. 2008 Deco et al. 2009). Normalizations, such as batch normalization (Toffe & Szegedy] 2015) and weight normalization (Salimans & Kingmal| (0166). are popular techniques facilitate the training of deep neural networks. This phenomenon has been found from invertebrates to mammals as a canonical neural computation, in which the responses of some neurons are divided by common factor reflecting the summed activity of a pool of neu- rons (Carandini & Heeger, 2012). The above works are orthogonal to the proposed CircuitNet, and we leverage these operations to improve the performance of CircuitNet in machine learning tasks.
CONCLUSION
This paper proposes a new type of neural network inspired by the architecture of neural circuits, where groups of densely connected neurons, namely Circuit Motif Units, form the basic units for modeling universal circuit motifs and a locally-dense and globally-spane
high-level network struc- ture is applied inspired by the connections among functional regions of brain. Experiments on synthetic and real datasets demonstrate that CircuitNet can outperform popular neural network ar- chitectures, such as MLP, CNN, RNN, and transformer, in several kinds of machine learning tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sanghyeon An, Minjun Lee, Sanglee Park, Heerin Yang, and Jungmin So. of simple convolutional neural network models for mnist digit recognition. arXiv:2008, 10400, 2020.
An ensemble arXiv preprint
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
Yoshua Bengio, Dong-Hyun Lee, Jorg Bornschein, Thomas Mesnard, and Zhouhan Lin. Towards biologically plausible deep learning, arXiv preprint arXiv:1502.04155 2015.
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Antonio Bruno, Davide Moroni, and Massimo Martinelli. Efficient adaptive ensembling for image classification. arXiv preprint arXiv:2206.00 073994944 07394, 2022.
Matteo Carandini and David Heeger. Normalization a canonical neural computation. Nature Reviews Neuroscience, 13(1):51-62, 2012.
Avraam Chatzimichailidis, Janis Keuper, Franz-Josef Pfreundt, and Nicolas R Gauger. Gradvis: Visualization and second order analysis of optimization surfaces during the training of deep neural networks. In 2019 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC), pp. 66-74. IEEE, 2019.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. Advances in neural information processing systems, 27, 2014.
Gustavo Deco, Edmund Rolls, and Ranulfo Romo. Stochastic dynamics as a principle of brain function. Progress in neurobiology, 88(1):1-16, 2009.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
A Aldo Faisal, Luc PJ Selen, and Daniel M Wolpert. Noise in the nervous system. Nature reviews neuroscience, (14):292-303) 2008.
Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp- 267-285. Springer, 1982.
Wulfram Gerstner and Werner M Kistler. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press, 2002.
Samanwoy Chosh-Dastidas and Hojjat Adeli. Spiking neural networks. International journal of
neural systems, 19004)295-338, 2009.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011.
10

---

Under review as a conference paper at ICLR 2023
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. PMLR, 2018.
Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: An image is worth graph of nodes. arXiv preprint 2iXiv:2206600222, 2022.
Jeff Hawkins and Subutai Ahmad. Why neurons have thousands of synapses, theory of sequence memory neocortex. Frontiers in neural circuits, pp. 23, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Kurt Horiik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, (65):359-366, 1989.
David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture of monkey striate cortex. The Journal of physiology, 195(1):215-243, 1968.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Tim C. Kietzmann, Courtney J. Spoerer, Lynn K. A. Sörensen, Radoslaw M. Cichy, Olaf Hauk, and Nikolaus Kriegeskorte. Recurrence is required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences, 116(43):21854- 21863, 2019. doi: 10. 1073/pnas....... 19055441 1997S//4491905544111 16. 10. 20.1073/pnass wet.aass.905550 1905544116
URL httts:::: http://wwww.a..//////////..
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR (Poster), 2015.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pp. 95-104, 2018
Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolu- tional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 156-165, 2017.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URLhttp: /nttp://yann.
lecun. wttp://////ccmmmmmmc//eccmm
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-2) 22224, 1998.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting, arXiv preprint rrXiv:1707.01922 2017.
Grace W Lindsay. Attention in psychology, neuroscience, and machine learning. Frontiers in com- putational neuroscience, pp. 29, 2020.
Grace W Lindsay. Convolutional neural networks as a model of the visual system: Past, present, and future. Journal of cognitive neuroscience, 33(10):017--031, 2021.

---

Under review as a conference paper at ICLR 2023
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Viston. pp. 10012-10022, 2021.
Liqun Luo. Architectures of neuronal circuits. Science, 37366559):eabg72855, 2021.
Andrew L. Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, pp. 3. Citeseer, 2013.
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):1659-1611, 1997.
Warren McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 15-3:1,5-,33, 1943.
Marvin Minsky and Seymour Papert. Perceptrons. 1969.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level 2015.
control through deep reinforcement learning, Nature, 518(7540):529-533
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):1-12, 2018.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, ICML' 10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 978 605589077.
Luiz Pessoa. Understanding brain networks and brain organization. Physics of life reviews, 11(3): 400-435, 2014.
Alan Peters. The fine structure of the nervous system. Neurons and their supporting cells, 1991.
Kai Qiao, Chi Zhang, Linyuan Wang, Jian Chen, Lei Zeng, Li Tong, and Bin Yan. Accurate re- construction of image stimuli from human functional magnetic resonance imaging based on the decoding model with capsule network architecture. Frontiers in Neuroinformatiis, 12, 2018.
Stephen Roberts, Michael Osborne, Mark Ebden, Steven Reece, Neale Gibson, and Suzanne Aigrain. Gaussian processes for time-series modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371 10550, 2013.
Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
Tim Salimans and Durk P Kingma. Weight normalization: simple reparameterization accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
Franco Scarselli and Ah Chung Tsoi. Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results. Neural networks, 1 11(1):15-77, 1998.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1899.1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv.1707.06347, 2017.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Teh. Power- propagation: A sparsity inducing weight reparameterisation. Advances in Neural Information Processing Systems, 34:28889-2888 28908, 2021.
Patrice Y Simard, David Steinkraus, John C Platt, al. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Con- ference Document Analysis and Recognition, pp. 958 988-963,22 2003.
12

---

Under review as a conference paper at ICLR 2023
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint
2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (5(1):192--195)) 2014.
Susan Standring. Gray's anatomy e-book: the anatomical basis of clinical practice. Elsevier Health Sciences, 2021.
Steven H Strogatz. Exploring complex networks. Nature, 410(6825):268 276, 2001.
Larry W Swanson. Brain architecture: understanding the basic plan. Oxford University Press, 2012.
lyy O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34:24261- 24272, 2021.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmip: Feed- forward networks for image classification with data-efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Stefan Treue and Julio C Martinez Trujillo. beature-based attention influences motion processing gain in macaque visual cortex. Nature, 399(6736):555-57)) 1999.
Ruben S van Bergen and Nikolaus Kriegeskorte. Going in circles is the way forward: the role of recurrence in visual inference. Current Opinion in Neurobiology, 65:176-193, 2020. ISSN 0959-4388. Whole-brain interactions between neural circuits.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liö, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568-578, 2021.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances neural information processing systems, 29, 2016.
Stanistaw Wožniak, Angeliki Pantazi, Thomas Bohnstingl, and Evangelos Eleftheriou. Deep learn- ing incorporating biologically inspired neural dynamics and in-memory computing. Nature Ma- chine Intelligence, 2(6))325-333) 2020.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419 222330, 2021.
Jachong Yoon and Sung Ju Hwang. Combined group and exclusive sparsity for deep neural net- works. In International Conference on Machine Learning, pp. 3958-3966. PMLR, 2017.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.019177 2022a.
13

---

Under review as a conference paper at ICLR 2023
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10819-10829, 2022b.
Zarita Zainuddin and Ong Pauline. Function approximation using artificial neural networks. WSEAS Transactions on Mathematics, 7(6):333-338, 2008.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng,
Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11106-11115, 2021.
A DETAILS OF THE DATASETS
A.1 SYNTHETIC DATASETS
The synthetic datasets are generated via:
y= (x),
where f(-) is a known function which the model needs to fit. We include both univariate and multi- variate functions in our synthetic dataset.
The univariate functions from Zainuddin & Pauline (0088) with proper normalization. The train set and test set are generated by evenly sampling x within [-1, 1].
The multivariate functions are generated by sampling randomly ag from 41(-1, 1) and ik. Ou from (1, 2, .... 10). The train set and test set are generated by randomly sampling X; from 1).
The details of the datasets are presented in Table 6 We leverage the Root Relative Squared Error (RRSE) to evaluate the fitting quality:
Σ m==1
l1
Dm 12
RRSE
(11)
Em=1
l1
where M the size of the test set and y is the average of Vmn
Table 6: Synthetic datasets
Dataset Name
= f(x: or f(X)
# Train
# Test
Dimension of X
Range of X
Univariate Exponential
(x + 1)e -3=33 3x-3/50 /50
100
200
[-1, 11
Univariate Periodic
sin(ATx)e -xx
100
200
[-1, I]
Multivariate Linear
Ekm Nx
2,000
200
20
[-1, 1]
Multivariate Quadratic
20 km ax Xin Xon
2,000
200
[-1, 1]
A.2 REINFORCEMENT LEARNING FOR CONTINUOUS CONTROL
The continuous control experiment on 5 different benchmarks are fetched directly from the OpenAI gym benchmark suite (Brockman al. 2016). The dimensions of the observation space and action space are listed in Table
A.3 IMAGE CLASSIFICATION
CircuitNet is evaluated on three public image classification datasets, namely MNIST (LeCun & Cortes, 2010), Krizhekry et al., 2009) and ImageNet (Deng al. 2009). MNIST handwriting digit recogmition dataset using 60,000 images as training set and 10,000 images as test set, with ten categories, ( 9. The CIFAR-10 dataset consists of 50,000 images of size 32 32 in
14

---

Under review as a conference paper at ICLR 2023
-103
110
(a) Univariate Exponential
(b) Univariate Periodic
Figure 5: The two univariate functions synthetic datasets.
Table 7: Continuous control environments.
Environment
Observation dimension
Action dimension
Ant-v2
111
8
HalfCheetah-v2
17
6
Hopper-v2 v2
11
3
Humanoid-v2
376
17
Walker2d-v2
17
training set and 10,000 images test with 10 classes. CIFARI00 has the same train/test split but contains 100 classes in total. For ImageNet dataset we use the most common ImageNet (ILSVRC) subset that contains 1,000 object classes with totally 1,281,167 training images and 50,000 validation images.
A.4 TIME SERIES FORECASTING
We have leveraged three common benchmarking datasets for time series forecasting task. Solar (Lai et al. 2018) includes the solar power production records in 2006, which are sampled every 10 minutes from 137 PV plants Alabama State. Pems-bay (Li et al. 2017) contains average traffic speed records measured by 325 sensors the Bay Area ranging from Jan 2017 to May 2017, offered by California Transportation Agencies (CalTrans). Metr-la (Li et al. 2017) contains average traffic speed measured by 207 loop detectors the highways of Los Angeles County ranging from Mar 2012 to Jun 2012, provided by the Los Angeles Metropolitan Transportation Authority.
In addition the Root Relative Squared Error (RRSE) in Equation of determination (R²) as the evaluation metric:
we also include coefficient
R
1
ΣΣΣ
M
N
Il 1l
y
(12)
MNT
g
Here, M is the size of test sets, N the number of variables, and T is the forecasting horizon. yf is the t-th future value of the i-th variable for the m-th sample in the test set, and ÿ is the average over all test samples.
B MODEL CHOICES AND TRAINING DETAILS
We list the details of training for reproducing.
B. SYNTHETIC DATASETS
We use an MLP with layers for this experiment. This is the closest structure to our CMU, where input data is first mapped CMU via linear function, then iteration of signal transmission
15

---

Under review as a conference paper at ICLR 2023
conducted with both linear transformation, neuron- neuron-wise attention, and product between neurons, and the I-dimension result is obtained by another linear function. Since modeling universal motifs introduces more parameters, list models with similar number of parameters rather than hidden neurons for fair comparison in Table
When training with gradient descent optimizer in the univariate datasets, find that both models will occasionally stuck some local minima. Therefore, we leverage the Adam optimizer Kingma & Ba (2015). As for the multivariate functions, because we need obtain the optimization trajactory for analysis, we still use the mini-batch gradient descent algorithm with batch size of 128. The learning rate was set to 1 x 10-2 with 0.9 momentum, and all the models are optimized for 10,000 epochs at maximum to ensure convergence.
B.2 REINFORCEMENT LEARNING FOR CONTINUOUS CONTROL
In this paper, we use Soft Actor-Critic (SAC) (Haarnoja et al. 2018), popular off-policy deep reinforcement learning (DRL) algorithm with high sample efficiency can be similarly applied to other Reinforcement Learning algorithms using neural networks as value function approximators or policy networks. We use CircuitNet as backbones for the policy functioo and two Q-functions in the SAC algorithm. Intuitively, for tasks where the states and actions have specific and known meanings, we may divide the inputs into groups based on their meanings and each group corresponds to the input of CMU in CircuitNet. However, the physical meaning of the states and actions are not exposed in selected tasks, so we simply map all the input dimensions to groups with differently initialized linear layers. The CircuitNets used in DRL tasks are relatively small and do not have many CMUs, so the connections among CMUs can be designed as a complete graph without introducing too many parameters.
example, but our method
The detailed hyperparameters are listed Table The sizes of the network are designed to match the number of parameters of the MLP baseline in the original paper. As for the learning rate and the activation function, we conducted grid search for every possible value in the sets and the best results of each model is reported. Because of the high fluctuation in deep reinforcement learning, all the experiments with 4 different seeds to present the error bar in the figure.
Table 8: Training details for deep reinforcement learning
Soft Actor Critic
Optimizer
Mini-batch gradient descent
Batch size
256
Learning rate
{3 x 10
x 10
10-4,14400 5.5 x 10 5, x 10 6,1x 1 10 10-4]
Discount
0.99
Starting Steps
10000
Replay buffer size
x 106
Model Details
# CMUs
4
# Iterations of signal transmission
# Neurons CMU
CircuitNet-L
64
CirtuitNet-L+A
40
CettuitNet-L+*
32
# Ports per CMU CircuitNet-L. CireuitNet-L+A
48
30
CircuitNet-L+P
24
Activation function
[ReLU, ClipSin}
B.3 IMAGE CLASSIFICATION
B.3.1 GENERAL DESIGN CHOICES
16

---

Under review as a conference paper at ICLR 2023
Topology of CMUs We first map the input image to CMUs based on the regions. Specifically, a convolution layer is applied to the input image and extracts representations for different regions.
Then we directly use the representations to initialize the states of the neurons in CMUs. As a result, each CMU corresponds to region of the input image after initialization and the CMUs are arranged grids.
For MNIST and CIFAR datasets, to transmit signals among CMUs, we use 2 2*-hop connections (k N), meaning that the (i )-th CMU in every row and every column can communicate with the (i-1) 1) x 2*) |th and [ 1) x ]th CMUs. This topology keeps the global sparsity with (NN connections while enabling arbitrary two CMUs to communicate within Oloo N) iterations, where N is the total number of CMUs.
For ImageNet dataset, because of the huge number of CMUs, even grid connections will introduce too many edges. Thus we follow et al. (2022) and consider the correlation between CMU signals by adding edges between CMUs with top-K 9) lowest output signal distance to accelerate signal transmission process.
Weight Sharing In order to reduce the dimension of the parameter space and utilize the spatial- invariant property for the image classification task, weights are shared across CMUs in the same iteration transmit signals within the units. However, since the topological position of CMUs are not completely symmetric, inter-umit weights between ports are not shared.
Weight Normalization
Normalization is a useful technique to stabilize the training process. Be- sides batch normalization, in CircuitNet, we also consider the weight normalization (Salimans & Kingma 2016) as an alternative option. If we rewrite the update of neurons as:
e (V (k-1))
(13)
instead of using V directly, we use its direction W to do the transformation:
V
(14)
IVI
Here, (k-1) is a vector whose elements can be calculated by the states from the previous iteration k 6-1 1 and constants and V is the learnable parameters in this iteration.
We use this type of normalization to address the problem caused by the different number of inputs for different neurons due to the incoming edges and different roles of neurons. Without the weight normalization technique, this problem becomes even harmful when parameters are shared across CMUs which are topologically asymmetric.
Feature Aggregation Compared the number of categories in image classification datasets, the hidden size in every iteration is still too high to project onto the output space. Therefore, we apply learnable linear aggregation function similar to pooling layers merge different CMUs together based on the spatial locations after certain iterations.
B.3.2 TRAINING DETAILS FOR DIFFERENT DATASETS
MNIST We design tiny CireuitNet that contains 4 x 4 groups with each CMUs containing 64 neurons and 48 ports, and the signal transmission is conducted for two iterations.
CIFAR CircuitNet-L+P contains 16 x 16 CMUs with each CMU containing 32 neurons and 24 ports, the signal transmission conducted for 9 iterations. To reduce the computation cost and accelerate the signal transmission, we merge the neighboring 4 CMUs at the 3rd and 6th signal transmission iteration to reduce the total CMU numbers, which cut the number CMUs to 1/4 after each merge operation. CircuitNet-L simply leverages Equation (I) to transmit signals and CircuitNet-L+A replace the product transformation in Equation (5) with neuron-wise attention in
② and 3 In order to make the these two CircuitNet variants to have comparable number of parameters and/or FLOPs, the CMU size is increased to 64, 52 neurons and the port size is increase to 48, 39 ports, respectively.
17

---

Under review as a conference paper at ICLR 2023
Table 9: Training details for Image Classification. Hyperparameters in braces searched.
Optimization
Optimizer
Adam, SGD)
Maximum learning rate
{1 x 10-1, x 10
x 10-3 5 x 10-1, x 10-1)
# Warm-up iterations
{0. 5}
Learning rate decay
Cosine Annealing Scheduler
Number of epochs
600
Model Details
# CMUs
MNIST
4
CIFAR
16 x
16
ImageNet
112
112
# Iterations of signal transmission
MNIST
CIFAR
9
ImageNet
12
# Neurons CMU MNIST
CIFAR
CircuitNet-L
64
CireuitNet--++A
52
CircuitNet-L++P
32
ImageNet # Ports per CMU MNIST
48
48
CIFAR
CircuitNet-L
48
CireuitNet-L+A
39
CireuitNet-L++P
24
ImageNet
36
Activation function
{Tanh, ReLU, ClipSin}
Normalization
(Batch Normalization, Weight Normalization]
ImageNet We further scale-up the CircuitNet to achieve better performance ImageNet contain- ing 12 transmission iterations splitted to stages with 2, 2, 4 and iterations, respectively. Following Han al. 2022). suppress over-smoothing, linear transformation added both before and after the signal transmission. Furthermore, before feed inputs into the CircuitNet, we add Stem structure containing 3 convolution layers that down- down-sample the original image 112 112 feature-map. Each feature vector in the feature-map corresponds to a initial state of a CMU.
Currently, for ImageNet, CireutNet applies only linear transformation for intra-CMU signal trans- mission but already achieves comparable performance with existing state-of-the-as methods.
For training implementation, random-cru cropom--rpppaa and horizontal- -lip data augmentations are applied. The hyperparameters are summarized in Table g
B.4 TIME SERIES FORECASTING
The recurrent CircuitNet initialize all the states of CMUs with zeros at the beginning, i.c., X (0) 0. Then it consumes the incoming data point, maps CMUs, and updates the states according to Equation 9 After the whole time series are proceeded, we obtained the feature vector of the sequence by down-sampling from the final states of all the CMUs.
The tasks introduced in A.4 are multivariate time series forecasting tasks. However, in order to limit the extra designs to the minimal, simplify the multivariate inputs to univariate time series.
18

---

Under review as a conference paper at ICLR 2023
Specifically, use the same recurrent CircuitNet generate the feature vectors for all the variate sequences separately, and aggregate them before the forecasting layer.
In our experiment, have 12 CMUs for the recurrent CircuitNet. Thus the connections among CMUs are designed as a complete graph similar to the case reinforcement learning. summary of hyperparameters are included in Table TO We run all the experiments with 3 different seeds and the average results are reported.
Table 10: Training details for time series forecasting. Hyperparameters in braces are searched.
Optimization
Optimizer Loss function
Adam MSELoss
Learning rate
1 x 10-3 3 1 10-5
Weight decay Number of epochs
1000
Model Details
Neurons per CMU
{4, 8, 16)
Ports per CMU
0.75 x Neurons per CMU
Total Neurons
(16. 32, 48} {1, 2, 4, 8}
Dimension of feature vector per variate
Activation function
Tanh
C ACTIVATION DESIGN FOR QUADRATIC TERMS IN CMU
In the image classification task, the signals will transmit for several iterations within CMU. When the number of iterations becomes large, it may cause the gradient vanishing or gradient exploding problem, prohibits the model to converge correctly. all the operations are linear, this problem has already been solved effectively with Rectified Linear Units (ReLU)(Krzhevsky et al. 2012). However, when quadratic terms are introduced Equation and (5), we need seek another solution to address this problem. As mentioned in Section 3 instead of traditional activation functions, we use clipped sinusoidal activation function ClipSin(x):
A A+c +(x+3)),
-3,
ClipSin(x)
A Asin =z)
- < x S 3,
(15)
A A+e(x - 3), > 3.
And we set é = 0.01 and A = 3.388235. We provide some theoretical justification on this design here. We will start by investigating how ReLU may prevent the gradient from vanishing or exploding and why it fails in quadratic situations. Then we introduce how to find satisfactory function and why the proposed ClipSin may be useful. Finally, show how we determine the parameters.
C.1 RELU ACTIVATION LINEAR TRANSMISSIONS
The linear transmission of signals in iteration i with an activation function can be written as:
(k-1) a x + +b))),
(16)
or if we focus on a specific neuron:
(k+1)
(k) (1) ums xt + b((1)) o (x.(1+)))
(17)
The derivative of neurons between iterations is:
(k++1)
= o' (2
(k++))
(18)
0r (k)
19

---

Under review as a conference paper at ICLR 2023
1-
Figure 6: Plot of y ClipSin(x)
According to the chain rule, the derivatives will be multiplied together with some additional terms introduced by normalization and the optimization target. When the activation function a-)) is the ReLU function, o-) is either 1. This means the ReLU function is essentially turning on off the paths to pass the gradient without bringing in any bias on the gradient scales.
C.2 GRADIENTS IN QUADRATIC TRANSMISSIONS
When we have the linear and quadratic terms the same time, the equation of transmission will be rewritten as:
(
0l (1))
(1) +b(*),
(19)
(k+1)
(1) (k) miy "
(R) ot + ((A)) o (x
(20)
Here, X the vector containing the product between every two elements X (x), T2
R", i.e. =(f,, ..., Inn x2 TR? Now the derivative of neurons between adjacent
iterations is:
Osm (k+1)
of (I (2((++1)
(k) (wol) min w (A) (8) + 2omn (0 (.(e) )a
0. (k)
o' (2(2((+)1)
(wien + w (0 196) + wottin (xx((11) + (k)
(k)
o' (x((++1)))
(1)
(21)
we apply the chain rule once, we get:
Ox (k+1)
(k+1)
am(k) (k)
Oz (k-1)
dxl (k) dxn (k-1)
= of d2(2/(ttt)
Σ
(k) of (2)
(K) (k-1)
=o(
(k-1) 1)
(K) + w (x
((k) (2(( )o'(x ((0))
In
+ wo
mil o (xt (x)))(()))) (2)) + und 0r (2((11)
(K)
(22)
20

---

Under review as a conference paper at ICLR 2023
can find 3 factors from the activation function that influence the gradient scale: ( ((x(()) (a ( ( ( (a)))"))o (x("), and o' o' (x)())) and similar terms will be multiplied when
From Equation (22
apply the chain rule to more iterations. Then we can analyze how different activation functions
behave:
(x) = (a). When x < 0, (x)) 0 and it turns off the gradient propagation path properly. However, when a (z) = (x)) x' are both unbounded. This will cause gradient explosion or gradient vanishing based on the input scale.
0,y 2 O, even if we have o'xx) o'vy) 1,
(αx) tanh(æ), then we have [o(x)] < L. lo'(3)) S 1, and (z(z) < 2v3 ~ 0.385.
The occurrence of gradient vanishing is doomed.
Therefore, we must come up with new activation function which can properly locate all (x)). lo (x))1, and (a())' (x)) around 1.
C.3 CLIPPED SINUSOIDAL ACTIVATION FUNCTION
The simplest function find to be satisfactory is the sinusoidal function A Asi(wwx). Let (x))
A sin (wz), we have:
(x)) IAw cos(w S Aw,
(23)
(o(y)o = | sin(wy) cas (wx)) S A A
(24)
la(x)o' (x:) 1 A sin (2wrr)
2 A
(25)
which can all be probably bounded and scaled by carefully choosing A and w.
However, one thing annoys is the fact that sin(r) not monotonic, so we choose to use only half a period of the function in [- Zor Z and use linear function with small slope outside this region. The complete format of the Clipped Sinusoidal Activation Function ClipSin(x) is:
E(x +
2w Asin(wx:),
2w
ClipSin(1)
S x S
(26)
2w
2w
+ c(x
π
)
π
2w
2w
C.4 CHOICE OF PARAMETERS
The intuition of selecting the parameters is to make the scales of the terms in Equation 23
and 255 close to 1 possible, at least, in the sense of expectation. Supposing that we can have a normalization layer before the activation function that normalize x .(k) to the standard normal distribution (0, 1), we first choose z so that
2.1. i.e. [-3, 3], contains the majority of (k) Also, considering that most terms in Equation 22 are (x (1)) }ا(( ((k)), we force:
z
((0. ~N [| sinwwr (k)) (os( (k) | =
(27)
Solving the equation and discarding the negative root, we get:
3
A = 2e T2
3.388235.
(28)
rerfi 6/2
where erfi(2) = -i eif((z) is the imaginary enror function. This is the amplitude of the sinusoidal function we use in our model. As for the slope of the linear function e, we follow the design in Leaky al. 2013) set it to 0.01.
21

---

Under review as a conference paper at ICLR 2023
0000
1750
Average
Average
2382
MLP
9000
MLP
CircuitNet-I
CircuitNet-l
CircuitNet-L+A
2900
CircuitNet-+AA
-2000
CircuitNet-L+P
CircuitNet-L+P
Environment interactions
168
Environment interactions
(a) Ant-v2
(b) HalfCheetah--2
Average retam
3000
0930
3000
MLP
MLP
2003
CircuitNet-L
CircuitNet-L
CircuitNet-L+A
1000
CircuitNet-L+A
CircuitNet-L+P
CircuitNet-L+9
203
Environment interactions
Environment interactions
(c) Hopper-v2
(d) Humanoid-v2
0003
4020
MLP
2003
CircuitNet
CircuitNet-L+A
CircuitNet-L+P
Environment interactions
1m
(e) Walker2d-v2
Figure 7: Learning curves of CircuitNet and MLP all 5 continuous control benchmarks.
D ADDITIONAL EXPERIMENT RESULTS AND ANALYSIS
D.1 ADDITIONAL EXPERIMENT RESULTS OF DEEP REINFORCEMENT LEARNING IN MORE ENVIRONMENTS
For the additional two environments, our model outperforms the baseline on Walker2d-v2 and achieved comparable results on Humanoid-v2.
We present results on all environments in Figure 7
D.2 ABLATION STUDIES
In order to investigate the effect of modeling different types of motifs real tasks, we compared variants of CircuitNet in experiments, [R-)CicuitNet-L, Re-JCiuitNet--LtA and (R-)CicuitNet-
L+P. The transition functions used for intra-unit signal transmissions and motifs to model are listed
in Table I1
22

---

Under review as a conference paper at ICLR 2023
Table 11; Description of different CircuitNet variants. N is the number of neurons per Unit
Variant CircuitNet
Intra-unit Signal Transmisson
Motifs Model
Parameters per CMU
(R XCircuiNett L.
Linear
Feed forward, Mutual
N²
(R CCicuiiNet-LLA
Linear. Neuron-wise attention
Foed-forward, Mutual, Food-back
1N2
JCreuilNet-L.
Linear. Product between Neurons
Feed forwardd Mutual, Feed back, Lateral
N N2
Table 12: Part of the experiment results on CIFAR-10 10 and CIFAR-100.
Accuracy
Model
CIFAR-10
CIFAR-100
Best of compared baselines
0.870
0.570
CercuiiNet-L.
0.858
0.588
CircuitNet-L+AA
0.853
0,588
CireuitNet-L++P
0,913
0.649
Table 13: Part of the experiment results on time series datasets with different forecasting horizons (6, 24).
Solar-energy
Pems-bay
Metr-la
Model
RRSEJ
R21
RRSEE
R21
RRSEŞ
R21
RRSE4
R2
|RRSE]
R21
RRSE!
R21
Best of compared baselines
0,202
0,960
0.355
0.889
0.452
0.782
0.532
0.733
0.517
0.762)
0.692
0.548
R-CircuitNet-L
0.216
0.956
0.382
0.861
0.378
0.869
0.545
0.730
0.450
0.818
0.692
0.570
R-CincuitNet-Le L+A
0.212
0.957
0.841
0.377
0.870
0.561
0.448
0.820
0.667
0.600
R R-CincuitNet-LAe L+P
0.211
0.958
0.515
0.748
0.380
0.868
0.563
0.711
0.447
0.820
0.656
0.613
We attached some results from Table and Table 5 here for convenience.
Deep Reinforcement learning (Figure both Circuit-L and Circuit-L+A achieved bet- ter or comparable results to the MLP baseline. However, with product between neurons, CircuitNet fails in DRL tasks. This may be explained by that the overly small number of neurons in CMU (in order to achieve comparable number of parameters) conversely harms the expressiveness of the network.
Image Classification (Table 12 on Cifar-10/Cifar- 100 dataset, Circuit-L. and Circuit-L+A perform similarly with ResNet-18 18 (He al., 2016) of comparable number of parameters and FLOPs. However, when product between neurons are added as a transition function, Circuit-L+P has a significant improvement on performance with comparable FLOPs and less parameters.
Time Series Forecasting (Table 13 All the varients achieve better results than the baselines on at least half the settings. However, we don't observe a consistent trend which variant of CircuitNet dominate the others among all the datasets.
In conclusion, even if the transition function linear function, the design of local density and global sparsity helps CircuitNet achieve superior or comparable performance to the baselines. As for the comparison among variants, there isn't 't a consistent conclusion which the best due the trade-off between capturing more interactions among neurons and involving more neurons while maintaining a similar number of parameters.
D.3 LIMITATIONS AND POSSIBLE DIRECTIONS FOR IMPROVEMENTS
Recent years, there are an enormous number works proposed to improve the performance neu- ral networks in different tasks and some of them have achieved state-of-the-art performances. Quite a few branches of techniques have been proved to be very effective and widely used in the ma- chine learning community, however, considering the fact that they almost completely orthogonal to CircuitNet, we neither used them in CircuitNet implementations nor listed them our experi- ments. These works include large-scale pre-training, advanced data augmentation, automatic neural
23

---

Under review as a conference paper at ICLR 2023
architecture search, etc. We optimistically expect further improvement with exquisite task seec-iiecific designs and these advanced techniques.
24


Reviews:
[{"review_id": "gV_JeF_kQSo", "reviewer": "Reviewer_oKif", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work explores how circuit motifs observed in biology can be built into artificial neural networks and finds they improve performance vs standard ANNs.\n", "summary_of_the_review": "An interesting and novel study with good empirical results. ", "strength_and_weaknesses": "Strengths\n- Viewing circuit motifs as computational building blocks is an interesting and important research direction with relevance for both AI and neuroscience.\n- The model presented is well tested on a number of benchmarks and performs well.  \n\nWeaknesses\n- The biological inspirations of this paper (feedforward excitation/inhibition, lateral inhibition and mutual inhibition Luo, 2021) rely on the presence of Dale's law in their definition. However Dale's law is absent in these networks, making it unclear how this work connects to biology. \n- It is unclear which circuit motifs underlie the performance of CMUs. There are no ablation studies, nor analysis of the CMUs that have been learned. For example it may be that neuron-wise attention is required for all of the results, in which case the \"generic\" nature of CMUs is a bit misleading. \n- How neuron-wise attention maps on to biology is unclear. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gPVqfLNcinw", "reviewer": "Reviewer_zeXY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a locally dense, globally sparse alternative to the feedforward neural network architecture. The proposed architecture consists of interconnected blocks where the neurons within each block are connected via feedforward, attention, and product-between-neurons mechanisms. The authors compare the performance of their model to that of the feedforward networks in function approximation, reinforcement learning, computer vision, and time series forecasting tasks, where they report an increased performance compared to the baseline models of their choice.", "summary_of_the_review": "The paper addresses the important task of improving the deep learning models’ performance and resource usage by introducing sparse and modular architecture. The Authors have compared their model to a variety of classic baseline approaches on multiple standard datasets, showing competitive performance. At the same time, the model’s scores fall below current leading scores for the used datasets (though, those likely required finetuning specific to each dataset) – this needs to be mentioned in the text. For these reasons, I recommend borderline acceptance.", "strength_and_weaknesses": "The goal of this paper is important, as there is growing evidence that the sparsity/modularity of neural network models may lead to improved performance with less computation. The authors have performed a rigorous evaluation of their model’s performance on various datasets where they compared it to the performances of multiple baseline models, published previously.\n\nAmong the others, the strengths of this submission are:\n\n-The paper contains a comprehensive Methods appendix which includes thorough descriptions of datasets and models. The baseline models are thoroughly referenced.\n\n-In the synthetic dataset for the function approximation, the Authors use a large enough class of the functions to be approximated, which goes beyond simple linear or quadratic functions.\n\n-CircuitNet is compared to milestone models in vision (e.g. ResNet, ViT) and time series forecasting (e.g. ARIMA, etc.) tasks. Those models include other sparse, modular approaches (e.g. ViT) that naturally have a similarly small number of parameters.\n\nWhile the strength of the proposed approach is in using a single architecture to outperform multiple baselines, meticulously optimized by previous Authors, it should be noted in the text that the current SOTA is higher than the reported performance on the used datasets. For example, on ImageNet it’s 91% and not the ballpark of 78% (https://paperswithcode.com/sota/image-classification-on-imagenet); on CIFAR-10 it’s above 99% (https://paperswithcode.com/sota/image-classification-on-cifar-10) etc. Understandably, these results required bells and whistles specific to each of the said datasets and do not devalue the results in this paper – though they still need to be mentioned and discussed not to mislead the paper readers.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SD5y28_uXQ", "reviewer": "Reviewer_CjTa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes CMU, a motif unit formed by a group of densely connected neocons, which is inspired by the structure of human brain. Experiment results suggests better performance. ", "summary_of_the_review": "Overall, I really like the idea of CMU and its potential of providing a more robust NN unit to build more capable networks. The experiment conducted in varies types of applications backs the idea that CMU net can be more accurate and maintain a reasonable size given it follows the design rule of locally dense and globally sparse. The paper feels most robust and novel. Clarity in explaining some CMU operation and experimental discussion can be slightly improved. ", "strength_and_weaknesses": "Strength:\n* The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.\n* Related works and background knowledge related are covered and discussed. \n* Experiments are conducted extensively in comparing the proposed CMU net against other networks in a variety of tasks.\n\nWeekness:\n* The clarity can be further improved. For instance, the experiment section can add a table or explain in the comments on the difference between CMU, CMUL+A and +P. \n* Experiment section can benefit from a small summary of why CMU can outperform in many different tasks while maintain very small size.\n\nQuestions:\n* While the learning curve compares CMU against MLP in training speed, how much more or less training computing cost per iteration does CMU need compare to traditional ANNs?\n* For eqn 3 (attention), what doe value, key and query represents?\n* For the clipsin activation function, the appendix suggests it is somewhat inspired by leakyRELU. Is it possible to have the [-3,3] linearly for less computation cost?\n* Can the authors clarify how does CMU handle conv layers, I did not quite follow the part in 3.2.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject