Dynamic Neural Response Tuning
Authors: Tian Qiu
Wenxiang Xu
Lin Chen
Linyun Zhou
Zunlei Feng
Mingli Song
Venue: ICLR 2024 poster (2024)
Keywords: biological neuron
artificial neural network
information transmission
information aggregation
neural response
dynamic tuning
activation function
neural representation
neuronal
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Artificial Neural Networks (ANNs) have gained widespread applications across various areas in recent years. The ANN design was initially inspired by principles of biology. The biological neural network's fundamental response process comprises information transmission and aggregation. The information transmission in biological neurons is often achieved by triggering action potentials that propagate through axons. ANNs utilize activation mechanisms to simulate such biological behavior. However, previous studies have only considered static response conditions, while the biological neuron's response conditions are typically dynamic, depending on multiple factors such as neuronal properties and the real-time environment. Therefore, the dynamic response conditions of biological neurons could help improve the static ones of existing activations in ANNs. Additionally, the biological neuron's aggregated response exhibits high specificity for different categories, allowing the nervous system to differentiate and identify objects. Inspired by these biological patterns, we propose a novel Dynamic Neural Response Tuning (DNRT) mechanism, which aligns the response patterns of ANNs with those of biological neurons. DNRT comprises Response-Adaptive Activation (RAA) and Aggregated Response Regularization (ARR), mimicking the biological neuron's information transmission and aggregation behaviors. RAA dynamically adjusts the response condition based on the characteristics and strength of the input signal. ARR is devised to enhance the network's ability to learn category specificity by imposing constraints on the network's response distribution. Extensive experimental studies indicate that the proposed DNRT is highly interpretable, applicable to various mainstream network architectures, and can achieve remarkable performance compared with existing neural response mechanisms in multiple tasks and domains. Code is available at https://github.com/horrible-dong/DNRT.

Published as a conference paper at ICLR 2024
DYNAMIC NEURAL RESPONSE TUNING
Tian Qiu, Wenxiang Xu, Lin Chen, Linyun Zhou, Zunlei Feng", Mingli Song Zhejiang University (tqiu, xuwx1 996, 1 siin_chen,, zhou 1 yaxx, zunlei feng, songml ) juzju. edu cn
ABSTRACT
Artificial Neural Networks (ANNs) have gained widespread applications across var- ious areas in recent years. The ANN design was initially inspired by principles biology. The biological neural network's 's fundamental response process comprises information transmission and aggregation. The information transmission in biologi- cal neurons is often achieved by triggering action potentials that propagate through axons. ANNs utilize activation mechanisms to simulate such biological behavior. However, previous studies have only considered static response conditions, while the biological neuron's 's response conditions are typically dynamic, depending on multiple factors such as neuronal properties and the real-time environment. There- fore, the dynamic response conditions of biological neurons could help improve the static ones of existing activations in ANNs. Additionally, the biological neuron's 's aggregated response exhibits high specificity for different categories, allowing the nervous system to differentiate and identify objects. Inspired by these biological patterns, we propose novel Dynamic Neural Response Tuning (DNRT) mecha- nism, which aligns the response patterns of ANNs with those of biological neurons. DNRT comprises Response-Adaptive Activation (RAA) and Aggregated Response Regularization (ARR), mimicking the biological neuron's 's information transmission and aggregation behaviors. RAA dynamically adjusts the response condition based on the characteristics and strength of the input signal. ARR devised to enhance the network's 's ability to learn category specificity by imposing constraints on the network's ' response distribution. Extensive experimental studies indicate that the proposed DNRT is highly interpretable, applicable various mainstream network architectures, and can achieve remarkable performance compared with existing neural response mechanisms multiple tasks and domains. Code is available https: / com coom/hore hwwr/oorrbbb wirbbbl.. bble --onng/// / /DNRT.
INTRODUCTION
In recent years, Artificial Neural Networks (ANNs) (McCulloch & Pitts, 1943; Jain et al., 1996; LeCun et al., 2015; Abiodun et al., 2018) have made significant milestones in various computer vision tasks (Voulodimos et al., 2018), such image classification, well numerous other tasks and domains (Wu et al., 2020; Zhang et al., 2023).
The idea of ANNs was initially derived from the perception pattern of the human brain (McCulloch & Pitts, 1943). Multilayer Perceptron (MIP) (McClelland et al., 1987), a classical ANN model that neurons its basic unit, resembles that of biological neural networks and led significant breakthroughs. Subsequently, Convolutional Neural Networks (CNNs) (LeCun et al., 1998) were developed to simulate local visual perception. Today, the advent of abundant computing power has led to further progress in the popular Vision Transformers (ViTs) (Han et al., 2022) that integrate attention mechanisms (Vaswani et al., 2017) and MLP structures (McClelland al., 1987).
The above ANNs utilize activation mechanisms (Dubey et al., 2022) that simulate the behavior of biological neurons transmitting the information. When the external signal reaches specific intensity, the activation mechanism transmits the signal to the next layer; otherwise, in most cases, it suppresses the transmission of the signal. Rectified Linear Unit (ReLU) (Glorot et al., 2011) is a commonly used activation mechanism in deep learning, with a fixed response threshold of zero, allowing only
"oorreponniigg author.

---

Published as a conference paper at ICLR 2024
sensory neuron
motor neuron
reality
membrang pocemial (mV)
response value
w/o modication wi modication
idcal
time (ms)
channel
Information Transmission
response distribution (class I)
reality (class
reality
probubility
resporaee distributien (class 2)
333
response
ideal (class I)
ideal (class 2)
response value
channel
Biological Neurons
(b) Information Aggregation
Artificial Neural Network
Figure 1: Observations on biological neurons and the way they have inspired the proposed Dynamic Neural Response Tuning (DNRT). The observations include two aspects of information processing: information transmission and information aggregation. (a) The threshold membrane potential required for transmitting information through neurons depends on the characteristics of neurons and the real- time environment due to its dynamically changing nature. (b) The information aggregated by neurons The proposed DNRT aims to align the response patterns of AÑNs with
is highly those of biological neurons, thus addressing the ANN's corresponding observed imperfections.
inputs greater than zero to pass while other imputs are set to zero. Another classical activation form is Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel. 2016), typically used in Transformers. suggests that the information transmission depends the fixed priori input distribution, i.e., input value ranked later among all inputs is more likely to be suppressed, whose curve is also predefined, similar ReLU but better simulating real-world scenarios.
However, as shown in Figure I(a), neuroscience research observed that the threshold membrane potential depends on neuronal properties and the real-time environment due its dynamically changing nature (Kandel et al., 1991; Fontaine et al., 2014; Azouz & Gray, 2000; Sun, 2009; Arancio al., 1995; Knowland et al., 2017). Similarly, and the strength of inputs for different objects or categories also varies. Therefore, the input strength cannot fully indicate the relevance of the input information. Considering these issues, we believe that the dynamic response pattern of biological neurons can be applied to improve the static response pattern the existing activation mechanisms in ANNs.
ANNs, different neurons have different functions,
Additionally, as shown in Figure 1(b), after the signals aggregate in the biological neuron, the response distributions exhibit significant differences across categories, called the response specificity neurons (Kreiman et al., 2000). In experiments with ANNs, we have observed similar phenomena. By training different categories of samples, the ANN can learn specificity similar to that of neurons. Furthermore, ANNs with poor performance tend to present Gaussian distribution with higher variance under the stimuli from a specific category. The Gaussian distribution is a common assumption (Fischer, 201 that reflects the presence of error in ANN's responses, and the distribution variance reflects the magnitude of this error. So, the high Gaussian variances indicate the specificity of the response is not strong enough, making difficult to distinguish between different categories through their responses, posing significant challenge for the ANN to make correct judgments.
This paper explores the application of biological neural response patterns in transmitting and aggre- gating information within various ANN architectures. A novel mechanism called Dynamic Neural Response Tuning (DNRT) is proposed. DNRT comprises Response-Adaptive Activation (RAA) and Aggregated Response Regularization (ARR), mimicking the biological information transmission and aggregation. RAA dynamically adjusts the response conditions based on the characteristics and strength of input signals, simulating the corresponding behavior of biological neurons. ARR imposes constraints on the distribution of aggregated neural responses based on historical statistics on category signals, enhancing the neural networks' ability to learn category specificity. Experiments on mainstream visual classification networks indicate that the proposed DŇRT outperforms existing neural response mechanisms in enhancing classification accuracy and helps to differentiate the neural
2

---

Published as a conference paper at ICLR 2024
response under different categories of stimuli. Additionally, DNRT exhibits the capacity to generalize effectively to other tasks and domains, showcasing its remarkable versatility.
The contribution of this paper is the simulation of dynamic biological neural response patterns through Disponse-Adaptivv Activation and Aggregated Response Regularization to address corresponding observed imperfections in ANNs. The study provides new concepts and techniques for constructing ANNs and valuable insights into in-depth neural information processing research. Extensive experi- ments across various mainstream ANN architectures, tasks, and domains indicate that the proposed method surpasses existing neural response mechanisms, delivering outstanding performance.
2
RELATED WORK
2.1 NEURAL RESPONSE IN NEUROSCIENCE
Neural response refers to the manner in which the nervous system reacts to stimuli from external sources (Kandel et al., 2000). The two primary processes involved in neural response are information transmission and aggregation (Kandel et al., 1991).
Information transmission. Through the axons of biological neurons, signals are transmitted the neuron's 's terminal, and neurotransmitters are released. Then, the signal is transmitted to other neurons through synapses (Kandel et al., 2000). The information transmission depends on the response condition. For spiking neurons, when the stimulus surpasses the threshold, an action potential that travels to the terminal is produced, resulting in the release of neurotransmitters (Hodgkin & Huxley, 1952). The neuron will not generate action potential if the stimulus is below the threshold. Neurons with different functions possess differing response thresholds (Kandel et al., 1991). The response threshold of neurons also be influenced by extrinsic elements such medications (Arancio et al., 1995) and nervous system ailments (Knowland et al., 2017). Moreover, the dynamic threshold is a regulatory mechanism of biological neurons, which has been widely observed in various nervous systems (Fontaine et al., 2014; Azouz & Gray, 2000; Sun, 2009).
Information aggregation. Biological neurons receive impulses from multiple other neurons and integrate these signals. Signal aggregation can occur at various levels, from local to global. reflecting the cerebral cortex's highly complex interconnectivity between neurons (Kandel et al., 1991). These aggregation processes form the neural network responsible for representing complex emotions, thoughts, and behaviors in a high-dimensional space based on the information received from this network (Kandel et al., 1991). Stimuli of the same category usually generate similar responses, and for different categories of stimuli, the information aggregated by biological neurons exhibits significant differences (Kreiman et al., 2000), indicating the response specificity of neurons.
2.2 ACTIVATION MECHANISM
The activation mechanism (Dubey et al., 2022) plays a pivotal role in ANNs as it defines how neurons respond to input signals and transforms them into output signals. Each activation varies in mathematical properties and nonlinear forms, allowing neurons to simulate different neural phenomena, including excitation, inhibition, and modulation. According to the prevailing activation response rule, irrelevant features are expected to be suppressed. while relevant features gain amplified influence, which can achieve data sparsity, diminish redundant information, and enable better feature distinction. Mathematically, activation mechanisms are categorized into different types, including logistic Sigmoid and Tanh variants (Xu et al., 2016; LeCun et al., 1998), Rectified Linear Unit variants (Glorot et al., 2011: 1; Maas et al., 2013; He et al., 2015; Xu et al., 2015; Jiang et al., 2018; Chen et al., 2020), Exponential Linear Unit variants (Clevert et al., 2015; Klambauer et al., 2017; Trottier et al., 2017), Softplus variants (Dugas al., 2000; Misra, 2019), probabilistic variants (Hendrycks Gimpel, 2016; Xu et al., 2015; Jiang al., 2018), multi-dimensional normalization nonlinearities (Ballé et al., 2015) and others (Maass, 1 1997; Wang et al., 2022).
The activations ELU (Clevert et al., 2015). SELU (Klambauer et al., 2017), and SiLU (Ramachandran et al., 2017) mentioned above, and others (Liu et al., 2020), contain learnable parameters that can adapt to various data distributions, avoiding gradient vanishing and explosion. thereby enhancing the convergence speed and precision of ANNs However, these parameters can only regulate response strength and do not influence response-triggering conditions. The threshold of spiking biological
3

---

Published as a conference paper at ICLR 2024
neurons can dynamically adjust based on environmental conditions (Fontaine al., 2014; Azouz & Gray, 2000; Sun, 2009; Arancio et al., 1995; Knowland et al., 2017). This attribute is absent in previous ANNs' activation mechanisms, indicating the need for improvement.
OBSERVATIONS
This section explores the observed phenomena in biological neurons and ANNs. The phenomena with the biological neurons' working patterns will help fix ANNs' corresponding observed imperfections.
3.1 OBSERVATION 1: INFORMATION TRANSMISSION RESPONSE
As illustrated in Figure I(a), biological neurons exhibit dynamic responses. Their response condition, Le., the threshold membrane potential, which determines the transmission of information between neurons, depends on various factors, including the type of neurons and their internal and external context. In ANNs, samples of given category are fed record the distributions of its responses upon passing through an activation with fixed response threshold, in which the horizontal coordinate is the element's (channel's) index within the feature vector, the vertical coordinate is the response value, and the area indicates the response density. We observe that some channels' response distributions are truncated. These channels and their inputs are generally considered irrelevant. Most inputs fall below the pre-set response condition (the threshold) and correctly get suppressed; however, some inputs in these channels still meet the condition and mistakenly trigger activation responses because the response condition for activation is pre-set and fixed according to prior knowledge. These incorrect responses may interfere with the network's final decision. The dynamic response conditions in biological neurons inspire improve the existing activation mechanisms in ANNs.
3.2 OBSERVATION 2: INFORMATION AGGREGATION RESPONSE
As illustrated in Figure 1(b), for different categories of stimuli, the information aggregated by biological neurons exhibits significant differences, indicating the response specificity neurons. Regarding the working pattern in ANNs, aggregates the activated signals transmitted by all neurons to produce a response output. The aggregated response distributions are captured after feeding samples approximately Gaussian-
of the same category. The observation indicates that the distributions shaped at each channel. However, the variances these Gaussian distributions are relatively large. We usually assume Gaussian distribution to represent the presence of error in ANN's responses. The variance of this distribution corresponds to the magnitude of the error. So, the high Gaussian variances indicate the response is pretty noisy and overlaps between categories, implying that the specificity differences between categories are not significant. As a result, the ANN is more likely to misrecognize these categories, requiring further optimization. The flass-ppecffic aggregated responses of biological neurons inspire us to devise techniques to enhance the aggregated ones of ANNs.
4 DYNAMIC NEURAL RESPONSE TUNING
4.1 RESPONSE- ACTIVATION
Inspired by the dynamic response conditions observed in biological neurons, a general adaptive response mechanism is devised to replace the static ones in current ANN activations.
Define an activation A that operates under static response conditions. Taking GELU (Hendrycks & Gimpel, 2016) as example of A, each input x has probability of (fxx) to be activated and probability of (1 ((x)) to be restrained (set to zero), and the activation probability Φ(x) rises as the input gets larger. The A's output w.r.t. the input æ can be expressed in a form of mathematical expectation:
A(x)
((x) 1x + (1 - -0x)))))--)))))))))()(())) (xx).
Here, the probability ((z) P(X < .x) is the cumulative distribution function of the standard normal distribution X ~ (/(0,)) (Hendrycks Gimpel, 2016). Since the curve of d(x:) is static/fxeed, any input A will consistently follow the same response condition, potentially leading to the
4

---

Published as a conference paper at ICLR 2024
phenomenon described in Observation ($3.1), where some irrelevant signals may incorrectly trigger an activation response and have a detrimental impact on the network's 's final decision.
To address this issue, we propose a Desponse-Adaptii Activation (RAA) mechanism that allows the original activation to adjust its response condition according to different external inputs.
Assume that an irrelevant input x" is going incorrectly trigger high response x" - =(x")) under the original response condition of A. In accordance with the intended meaning of A, with x" determined, it is necessary minimize the activation probability o(x") much possible. To accomplish this, an offset Arr can be imposed on x". Since the & is monotonic increasing, in this case, Ar" should be negative, then obtain new response x" Oxx" + Ax*), which is lower than the original response x" . (xx") ) and suppresses the irrelevant input x". On the other hand, when relevant input x' is going to be incorrectly suppressed under the original response condition of A, a positive offset Ax" should be applied. Hence, the issue lies in the calculation of Ar.
straightforward solution directly treating the offset År as learnable parameter a. Denote the Response-Adaptive Activation as RAA, which can be expressed as follows:
RAA(x)
((x + a).
(2)
Even though this technique offers adaptive response conditioning, the activation mechanism's 's re- sponse condition is still somewhat inflexible since a uniformly influences each input.
A more dynamic alternative is determining the offset Ar based on the input itself by introducing a mapping from the input to its desired offset. Then, the second (not the final) version of RAAA can be expressed as follows:
RAA(x) x. F(x + f(x))
(3)
zx D(x Φ + (wx + b)),
where the learnable scalars w and b represent the weight and bias of the linear mapping f- They initialized zero ensure an initial offset of zero. Now, RAA allows the activation response condition to adapt dynamically to different inputs, which is crucial for achieving better performance.
However, will not adopt Eq. 3 since such element-wise operation requires too many parameters and computations. In order to efficiently assess signal relevance with minimal computational costs, we further refine the RAA's granularity to the feature vector level. Generally, neural networks extract features from input data, usually presented as feature sequence {x:} of length L, where each feature x; [Ti, I2, X2,...,,a]] is d-dimensional vector. Consequently, final version of RAA can be expressed as follows:
RAA(x) = x Φ(x + (x))
= = F( + [x x
where the d-dimensional mapping vector parameters used in the model, which are also initialized with zero to ensure an initial offset of zero.
[wr, W2.. w2]2 and scalar bias b are learnable
The basic concept and thought of RAA hold broad applicability. With negligible parameters and computations introduced, RAA gains the ability to respond to input signals in an adaptive manner. Furthermore, this behavior is not limited to specific context and can also be extended to other static activation forms such as ReLU (Glorot et al., 2011) (011)(em etc.
4.2 AGGREGATED RESPONSE REGULARIZATION
According to Observation 2 ($3.2), the biological aggregated responses present high category speci- ficity. For stimuli of a given category, the ANN's aggregated response exhibits a Gaussian distribution in each channel but with a high degree of variance. more focused response can enhance feature specificity, thus leading the network to make more accurate decisions between categories.
To address this issue, we propose a statistics-based regularization technique called Aggregated Response Regularization (ARR). Let K denote the total number of categories of the given dataset. For input data of each category l E (1, ..., K), the historical mean/average aggregated responses M2 within the network tracked separately and in real time. To avoid a rise in space complexity, the historical mean is updated by using a moving mean technique as follows:
4 (1 m) 11 m x
(5)
5

---

Published as a conference paper at ICLR 2024
where the vector x = [2,, TT,] rd]" is the network's aggregated response time t when inputting sample from category k, and d is the vector's dimension. _ and μ4 represent the historical mean response vectors of category l at times t - and t, respectively. The hyperparameter m is the momentum for updating the moving mean. Therefore, only K vectors (Ma1k=1 1 need to be maintained throughout the entire process, whose memory overhead is negligible.
F2,
Next, the network's aggregated response can be made more concentrated by utilizing the historical mean. To achieve this, at each time I, a loss constraint Larr is applied between the aggregated response x and its historical mean ul f-1 as follows:
||x
Larr
d
where d is the dimension of the vector x and μ. Constraining the new aggregated response to be closer to the historical mean is equivalent to reducing the variance of the Gaussian distribution.
deep ANN includes multiple layers, and the aggregated response after the activation in each layer is capable of calculating the mean value of historical aggregated responses and applying regularization to new ones. Therefore, the final loss L can be expressed as
L = Lrask + A. J Ej=
where Ltask is the primary loss for the specific task; for example, in the context of a standard classification task, Ltask represents the cross-entropy loss. J is the number of layers in the network that have ARR applied, and À is the balanced parameter. Introducing this new loss function helps the network to enhance category response specificity. Especially when dealing with small datasets, the proposed ARR can better leverage limited data and avoid the risk of overfitting. Moreover, as training strategy, the proposed approach does not affect the inference speed.
4.3 NEURAL NETWORK LEARNING
The proposed Desponse-Adaptig Activation (RAA) mechanism can replace the network's original ones, and the proposed Aggregated Response Regularization (ARR) operates on the network's global feature vector aggregated after the activation. For simple MLPs, the output directly represents the global feature vector. Networks that extract feature maps/sequences compute the global feature vector by taking the global average of the feature maps/sequences along the channels. Regarding some Transformer models that incorporate a class token, simply peel off the class token separately as the global feature vector while applying the corresponding constraint.
5 EXPERIMENTAL, STUDY
Datasets. In the main experiments, we adopt five datasets, including MNIST (LeCun et al., 1998), CIFAR- 10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), ImageNet- 100 (Deng et al., 2009), and ImageNet-IK K (Deng et al., 2009), to verify the effectiveness of the proposed DNRT.
Compared methods. We conduct comparisons between DNRT and existing neural response mecha- nisms equipped with different mainstream activations (see $2.2), including Softplus (Dugas et al., 2000), ELU (Clevert al., 2015), SELU (Klambauer al., 2017). SiLU (Ramachandran et al., 2017), ReLU (Glorot et al., 201 I), GELU (Hendrycks & Gimpel, 2016), and GDN (Ballé et al., 2015).
Experimental settings. The image size of MNIST and CIFAR- 10. 100) remains 28x28 and 32x32, while the images in ImageNet- (100,, 100, 1K] are uniformly scaled to 224x224. In the proposed ARR, the momentum m for updating the moving mean is empirically set to 0. and the balanced parameter varies depending on networks and datasets (see Appendix A.2). All experiments use the same data augmentations provided by timm (Wightman, 2019), AdamW optimizer with weight decay of 0.05, drop-path rate of 0, gradient clipping norm of 1. 1.0, and cosine annealing learning rate scheduler with linear warm-up. Except for simple MLPs, which are trained for only 50 epochs from scratch, other networks are trained for 300 epochs from scratch. The automatic mixed precision is adopted to speed up the training. All other training settings, including batch size, learning rate, warm-up epochs, and on, are kept identical throughout each set of comparative experiments. In the displayed tables, for example, "GELU" denotes the network with the GELU activation and the original response aggregation mechanism, and "DNRT" denotes the network with the proposed RAA and ARR.

---

Published as a conference paper at ICLR 2024
Table 1: Top-1 accuracy (%) across the MNIST, CIFAR-10, and CIFAR-100 datasets using the proposed DNRT on a simple MLP with only one input layer, one hidden layer, and one output layer.
Top-1 Acc %
Softplus
ELU
SELU
SiLU
ReLU
GELU
GDN
DNRT
MNIST
MLP
97.3
97.5
97.6
97.0
97.1
97.5
97.5
98.0
CIFAR-I0
MLP
51.7
51.4
50.6
52.8
52.1
52.9
44.4
53.6
CIFAR-I00
MLP
24.8
24.7
24.4
26.1
25.4
25.6
19.8
26.1
Table Top-1 accuracy (%) across the CIFAR-IO, CIFAR-100, and ImageNet-100 datasets using the proposed DNRT on Vision Transformer (VIT) and its variants.
Top-1 Acc/ %
Softplus
ELU
SELU
SiLU
ReLU
GELU
GDN
DNRT
ViT-Tiny
84.3
82.0
79.4
85.5
89.9
89.2
81.8
92.1
DeiT-Tiny
84.7
81.4
79.9
86.6
89.6
89.2
83.0
92.4
CIFAR-10
CaiT-XXS
82.5
80.7
78.4
86.6
89.4
88.7
80.0
91.6
PVT-Tiny
90.6
89.3
85.4
92.5
93.0
92.8
82.8
94.3
TNT-Small
88.3
85,4
83.7
90.5
90.9
91.2
85.1
92.5
ViT-Tiny
62.4
60.0
57.5
65.5
65.7
65.4
59.4
71.4
Deif-Tiny
63.4
60.0
58.3
67.1
67.0
67.0
59.8
71.4
CIFAR-100
CaiT-XXSS
60.4
59.3
55.8
63.9
65.8
65.5
56.2
70.6
PVT--Tiny
69.5
69.3
65.7
70.2
70.9
70.6
64.4
71.6
TNT-Small
65.2
63.8
60.9
65.1
65.4
64.4
62.5
71.9
ViT-Tiny
74.1
68.9
66.4
74.1
75.4
76.4
67.9
80.9
DeiT-Tiny
75.3
69.4
67.0
75.1
75.6
74.6
66.3
81.1
ImageNet-
CaiT-XXS
70.9
69.1
65.9
76.1
76.0
76.7
69.5
80.4
PVT-Tiny 79.5
77.1
76.1
79.5
81.9
81.4
75.8
84.1
TNT-Small
78.9
79.3
76.4
77.6
79.9
77.2
76.9
82.3
Table 3: Top-1 accuracy (%) across the CIFAR-IO, CIFAR-100, and ImageNet-100 datasets using the proposed DNRT on various CNN architectures.
Top-1 Ace %
Softplus
ELU
SELU
SiLU
ReLU
GELU
GDN
DNRT
AlexNet
76.1
84.3
82.6
84.3
86.0
85.2
86.4
VGG-11
89.6
90.4
89.1
91.7
92.2
92.0
89.3
92.6
CIFAR-10
ResNet- 18
94.0
93.9
93.0
95.2
95.0
95.2
82.3
95.5
MobileNet
90.0
89.8
87.7
89.7
87.4
89.4
86.6
90.0
ShuffleNet
90.2
90.2
87.8
90.8
89.4
90.9
87.6
91.2
AlexNet
44.0
57.6
55.7
57.3
57.2
57.4
56.5
59,2
VGG-11
64.7
68.8
66.0
70.8
70.2
70.7
70.2
71.0
CIFAR-100
ResNet- 8
75.6
75.5
74.7
75.7
75.7
75.8
71.7
76.5
MobileNet
66.3
67.5
64.1
66.0
66.0
66.2
55.3
66.9
ShuffleNet
68.0
68.2
63.8
68.0
66.3
68.2
57.3
68.7
AlexNet
74.8
77.5
75.7
77.8
76.3
77.7
74.4
78.8
VGG-11
80.1
83.6
80.6
87.0
87.7
87.7
85.3
88.7
ImageNet-
ResNet-18 18
85.4
84.9
83.9
85.7
84.9
85.9
80.2
86.8
MobileNet
80.5
80.7
77.5
80.9
80.6
81.0
73.6
81.7
ShuffleNet
83.0
82.3
79.4
82.9
81.6
82.6
75.3
83.4
5.1 DNRT ON MLP
As shown in Table 1, the proposed DNRT is firstly validated on the original ANN model, known as MLP (McClelland et al., 1987), due to its working mechanism being akin to the human brain's perception mechanism. A basic MLP compiled with one input layer, one hidden layer, and one output layer is built, with an activation applied after the hidden layer. DNRT can replace existing neural response mechanisms in MLP. The results indicate that the proposed DNRT yields the best performance on the classical MNIST, CIFAR- 10, and CIFAR- 100 datasets. It affirms the feasibility of incorporating the biological signal transmission and aggregation characteristics into ANNs.

---

Published as a conference paper at ICLR 2024
Table 4: Generalization effect of the proposed DNRT across various tasks: node classification on the DGraph dataset and tackling highly imbalanced data on the Long-Tailed CIFAR- 10 dataset.
Node classification
Tackling highly imbalanced data (3=10, 50, 100)
AUC %
GCN
GraphSAGE
Top-1
Acc %
ViT-Tiny
ResNet-32
ReLU
72.5
75.1
ReLU GELU
53.0,50.0 44.9 52.6, 49.3, 44.4
86.8, 83.5, 80.0
GELU
71.9
74.4
86.5, 83.3, 79.6
DNRT
74.9
76.7
DNRT
56.5, 53.6, 47.3
87.7, 84.1, 82.0
Table 5: Ablation study of the proposed DNRT on the CIFAR-I00 dataset.
Top-1 Ace / %
ReLU
GELU
RAA
ReLU + ARR
GELU + ARR
DNRT (RAA + ARR)
ViT-Tiny ResNet-18
65.7 75.7
65.4
66.5
69.6 76.3
68.5
71.4 76.5
75.8
76.1
76.2
5.2 DNRT ON VITS
The popular Vision Transformer (ViT) and its variants incorporate multiple MLP structures. We incorporate the proposed DNRT mechanism into them. Table shows the top-1 accuracy (%) of DNRT on CIFAR-IO, CIFAR-100, and ImageNet-100 datasets across five different ViT architectures: ViT (Dosovitskiy et al., 2020), DeiT (Touvron al., 2021a), CaiT (Touvron al., 2021b), PVT (Wang et al., 2021) and TNT (Han et al., 2021). DNRT can replace existing neural response mechanisms in Transformer's MIPs, where RAA replaces the original activation, and ARR is performed on the class token aggregated after the activation. The results consistently illustrate that DNRT remarkably outperforms the baselines. Morcover, the results on ImageNet-IK can be found Appendix A.3.
5.3 DNRT ON CNNS
The proposed DNRT is also evaluated on various mainstream CNNs, including AlexNet (Krizhevsky et al., 2017), VGG (Simonyan & Zisserman, 2014), ResNet (He et al., 2016), MobileNet (Howard al., 2017) and ShuffleNe((V2) (Ma al., 2018), where RAA replaces the original activation, and ARR is applied to the globally pooled feature maps aggregated after the activation. The results in Table 3 highlight the versatility of DNRT handling different CNN architectures.
5.4 GENERALIZATION TO OTHER TASKS
Apart from the standard image classification task, the proposed DNRT can also perform various other tasks or domains, including non-computer vision tasks like special tasks like tackling highly imbalanced data. The results are remarkable.
node classification, and some
Node classification. We conduct additional experiments on the node classification task. The DGraph dataset (Huang al., 2022) is employed to evaluate the proposed DNRT. GCN (Kipf & Welling, 2016) and GraphSAGi AGE (Hamilton et al., 2017) are chosen GNN models, each comprising two layers with 64-dimensional feature space. Table 4 (left side) shows the results.
Tackling highly imbalanced data. Additionally, the proposed DNRT is quite capable of addressing the negative impact brought by the highly imbalanced category distribution. A Long-Tailed CIFAR- 10 dataset is generated with reduced training examples per class while the validation set remains unchanged. The imbalance rate 3 denotes the ratio of the most to the least frequent category's 's sample size, i.c., B = TTmaz/Nmin The sample size of different classes decays exponentially. In this case, the primary loss item Lrask employed in Eq. is the "Balanced Softmax Loss (BSI.).. The results for the imbalance rate B=10, 50, 100 are presented in Table 4 (right side).
5.5 ABLATION STUDY
Table presents the ablation results on the proposed DNRT. Both RAA and ARR contribute to enhancing the network, and their combination leads to further improvement in performance.
8

---

Published as a conference paper at ICLR 2024
w/o RAA
w/o ARR
6
10
16
2
8
14
18
20
W/ RAA
ARR
3
0
4
0
10
16
20
-3
12
14
18
20
(a) Visual results of RAA
(b) Visual results ofA
Figure 2: Response distributions the class token in VIT's MLP block with and without the proposed DNRT on the CIFAR-10 10 dataset. (a) MLP's activation responses. (b) MLP's aggregated responses. The baseline is the network with the GELU activation (the default used in ViT) and the original response aggregation mechanism. The horizontal coordinate represents the element's (channel's) index within the feature vector, the vertical coordinate represents the response value, and the depicts the response density. The figure displays the response distributions of the first 20 elements (channels) in each vector (the class token). For more results, please see Figure Appendix A.4.
5.6 NEURAL RESPONSE VISUALIZATION
Figure displays the response distributions on the class token ViT's MLP block with and without the proposed DNRT. As shown in Figure 2(a), when using traditional activation mechanism with static response condition, almost all channels exhibit a large amount of response since some irrelevant features that should not be activated are mistakenly activated. In contrast, RAA reduces the activation of irrelevant features and achieves considerably sparser activation response, indicating the network's ability to self-regulate response conditions and extract key features from input data with higher precision. Figure 2(b) illustrates the effect of imposing the ARR on VIT's class token. Under the guidance of ARR, the MLP's aggregated response the class token becomes more concentrated, enabling the network to emphasize task-related features, thus improving its generalization ability. These visual results are in line with expectations, providing clear insights into the internal mechanism of DNRT and elucidating why DNRT can elevate the performance of ANNs. More visual results and discussions can be found in Figure 4 and Appendix A.4.
CONCLUSION
Inspired by the principles of biology and neuroscience, we propose novel mechanism called Dynamic Neural Response Tuning (DNRT). AapponsiveAAdaptiii Activation (RAA) and Aggregated Response Regularization (ARR) are devised simulate the biological neuron's 's behaviors of transmitting and aggregating information, dynamically adjusting response conditions, and regularizing the gathered activated signals, respectively. The experiments demonstrate that the proposed DNRT is more effective than existing neural response mechanisms across various ANN architectures, including MLP, ViTs, and CNNs. Furthermore, DNRT showcases its superiority diverse tasks and domains, encompassing image classification, non-computer vision tasks like node classification, and some special tasks like tackling highly imbalanced data. A more precise simulation of response characteristics biological neural networks can further enhance ANNs. Overall, the findings of this paper have profound implications for the ANN's future. Moving forward, we are going to delve deeper into the analysis of both biological neural networks and representation-level patterns within ANNs devise more advanced neural response tuning mechanisms.
ACKNOWLEDGMENTS
This work supported by National Natural Science Foundation of China (U20B2066) and Ningbo Natural Science Foundation (2022]182, 2023J281).

---

Published as a conference paper at ICLR 2024
REFERENCES
Oludare Isaac Abiodun, Aman Jantan, Abiodun Esther Omolara, Kemi Victoria Dada, Nachaat Ab- dElatif Mohamed, and Humaira Arshad. State-of-the-ar artificial neural network applications: A survey. Heliyon, 4(1 1):009938, 2018.
O Arancio, ER Kandel, and RD Hawkins. Activity-dependen long-term enhancement of transmitter release by presynaptic 3', -cycyli gmp in cultured hippocampal neurons. Nature, 376(6535); 74-80, 1995.
Rony Azouz and Charles M Gray. Dynamic spike threshold reveals a mechanism for synaptic coincidence detection in cortical neurons in vivo. Proceedings of the National Academy of Sciences, (1114)::11088155 2000.
Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. Density modeling of images using generalized normalization transformation. CoRR, abs/1511.0622 2015.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic relu. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pp. 351-367. Springer, 2020.
Djork-Arn Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: 1,07299, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.11929 2020.
Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. Activation functions in deep learning: A comprchensive survey and benchmark. Neurocomputing, 2022.
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems, 13, 2000.
Hans Fischer. A history of the central limit theorem: from classical to modern probability theory, volume 4. Springer, 2011.
Bertrand Fontaine, José Luis Peña, and Romain Brette. Spike-threshold adaptation predicted by membrane potential dynamics in vivo. PLoS computational biology, (M44)e1003555 2014.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
William I.. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. NIPS, 2017.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34:15908 155590-5919,,,,,1111119199999119 2021.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87-110, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint N.XY::6606608415. 2016.
Alan L Hodgkin and Andrew Huxley. quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of physiology. 117(4):500, 1952.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: pte.//4444..// 2017.
Xuanwen Huang, Yang Yang, Yang Wang, Chunping Wang, Zhisheng Zhang, Jiarong Xu, and Lei Chen. Dgraph: large-scale financial dataset for graph anomaly detection. ArXiv, abs/2207.03574 033579 2022.
Anil K Jain, Jianchang Mao, and K Moidin Mohiuddin. Artificial neural networks: A tutorial. Computer, 29(3):31-44, 1996.
Xiaoheng Jiang. Yanwei Pang. Xuelong Li, Jing Pan, and Yinghong Xie. Deep neural networks with elastic rectified linear units for object recognition. Newrocomputing, 275: 133-1139, 2018.
Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, Sarah Mack, et al. Principles of neural science, volume 4. McGraw-hill New York, 2000.
Eric R Kandel et al. Nerve cells and behavior. Principles of neural science, 3:18-32, 1991.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ArXiv, 609.0007,, 2016.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Daniel Knowland, Varoth Lilascharoen, Christopher Pham Pacia, Sora Shin, Eric Hou-Jen Wang, and Byung Kook Lim. Distinct ventral pallidal neural populations mediate separate symptoms of depression. Cell, 170(2):284-297, 2017.
Gabriel Kreiman, Christof Koch, and Itzhak Fried. Category-specific visual responses of single neurons in the human medial temporal lobe. Nature neuroscience, 3(9):946-953, 2000.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-2324 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((5533:436 444, 2015.
Aizhu Liu, Haigen Hu, Tian Qiu, Qianwei Zhou, Qiu Guan, and Xiaoxin Li. Exploring optimal adaptive activation functions for various tasks. In 2020 IEEE International Conference Bioin- formatics and Biomedicine (BIBM). pp. 2290-2297. IEEE, 2020.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. Proceedings of the European Conference on Computer Vision pp. 116-131, 2018.
Andrew L Maas, Awni Y Hannum, Andrew Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Atlanta, Georgia, USA, 2013.
11

---

Published as a conference paper at ICLR 2024
Wolfgang Maass. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9):1659-1677) 1997.
James L McClelland, David E Rumelhart, PDP Research Group, et al. Parallel Distributed Processing, Volume 2: Explorations in the Microstructuree of Cognition: Psychological and Biological Models, volume 2. MIT press, 1987.
Warren McCulloch and Walter Pitts. A logical calculus the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 5:115-133, 1943.
Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:1908. 086811,...88.....1 2019.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1109.155: 2014.
Qian-Quan Sun. Experience--epeeddent intrinsic plasticity interneurons of barrel cortex layer iv.
Journal of Neurophysiology, (02(5):2955-297) 2009.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers distillation through attention. International conference on machine learning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 32 42, 2021b.
Ludovic Trottier, Philippe Giguere, and Brahim Chaib-Dras Parametric exponential linear unit for deep convolutional neural networks. In 2017 16th IEEE international conference on machine learning and applications (ICMLA), pp. 207-214. IEEE, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, Eftychios Protopapadakis, et al. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018, 2018
Siqi Wang, Tee Hiang Cheng. and Meng-Hiot Lim. Ltmd: Learning improvement of spiking neural networks with learnable thresholding neurons and moderate dropout. Advances in Neural Information Processing Systems, 35:28350-28362 2022.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
568-578, 2021.
Ross Wightman. Pytorch image models. http://ghhhbcooonngggamaaaaatoeo
2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 24, 2020.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv: 1005 00853, 2015.
Bing Xu, Ruitong Huang, and Mu Li. Revise saturated activation functions. arXiv preprint 1602.00580 2016.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailen learning: A survey. IEEE Transactions Paltern Analysis and Machine Intelligence, 2023.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 COMPUTATIONAL. COSTS
This section discusses the computational costs the proposed DNRT terms trainable parameters. GPU overhead, computational complexity, and inference speed. An experiment is also conducted to validate that the primary performance improvement comes from the RAA intrinsic mechanism rather than simply the extra parameters.
Firstly, in the proposed DNRT, ARR a training strategy that does not introduce any parameters. RAA needs negligible additional parameters to judge the relevance of input signals. Table 6 compares DNRT's trainable parameters and GPU overhead with those of the original networks. The networks fed 224x224-pixel images with batch size of 512, and the GPU occupancy is recorded. Notably, the activation mechanism used in original networks should be implemented manually as our RAA. Utilizing counterparts directly from pre-made libraries (like torch.nn) can result in unfair comparisons due to their high optimization at the low level. The experiment is conducted on an NVIDIA A100 (80G). For most VīTs and CNNs, the increase in trainable parameters ranges from 0.008% 0.18%, which is negligible, and the GPU occupancy during both training and inference is also negligible. The extremely low computational costs indicate that DNRT has broad practical value.
Table 6: Trainable parameters (M) and GPU overhead (GiB) of the networks with the proposed DNRT compared to those of the original networks. "RAA" denotes the network with RAA only. "DNRT" denotes the network with both RAA and ARR.
Trainable Parameters M
Training GPU Memory / GiB
Inference GPU Memory GiB
Network Metric
Original
RAA
DNRT
Original
RAA
DNKT
Original
RAA
DNRT
ViT-Tiny
5.38
5.39
5.39
19.76
19.90
19.91
4.18
4.19
4.19
DeiT-Tiny
5.91
5.92
5.92
20.04
20.19
20.20
4.28
4.28
4.28
CaiT-XXS
11.96
11,98
11.98
47.67
47.82
47.84
3.86
3.86
3.86
PVT-TTiny
13.23
13.24
13.24
44.57
47.21
47.46
14.21
14.21
14,21
TNT-Small
23.76
23.77
23.77
69.39
69.69
71.15
7.84
7.85
7.85
AlexNet
61.10
61.11
61.11
7.87
8.42
8.42
3.99
4.36
4.36
VGG-11
132.87
132.88
132.88
51.38
55.41
55.59
21.22
27.44
27.44
R8
11.69
11.69
11.69
14,30
15.59
15.61
5.47
5.47
5.47
MobileNet
3.50
3.51
3.51
21.35
22.45
22.46
9.26
9.34
9.34
ShuffleNet 2,28
2.28
2.28
11.99
13.46
13.47
4.44
4.46
4.46
Moreover, the impact of DNRT on computational complexity and inference speed is also minimal. Here, present numerical comparisons for the network's computational complexity, measured in "FLOPs" (floating-poin operations), and inference speed, measured in "Latency" (the time required for model inference single image). The "Latency" is obtained, average, from the model inferring 224x224-pixd images on an NVIDIA 3090. The results are shown in Table 7.
Table Computational complexity (FLOPs / G) and inference speed (Latency ms) of the networks with the proposed DNRT compared to those of the original networks.
Network Metrie
ReLU (FLOPs)
ReLU (Latency)
DNRT (FLOPS)
DNRT (Latency)
ViT-Tiny TNT-Small
1.078 G
4.5 ms
1.080 G
4.6 ms
4.849 G
10.1 ms
4.856 G
10.5 ms
Additionally, concerning that the performance improvement brought by RAA may stem from the increased number of parameters itself, to verify the primary contributors to the performance improve- ment, we conduct another group of experiments where pure extra layer of linearity (ex-L) is added into ViT-T's MLPs, with the same number of parameters as RAA. The comparison on the CIFAR- 100 dataset is shown in Table 8. The performance on RAA outperforms the pure ex-L. which indicates the primary improvement stems from the RAA intrinsic mechanism (climinating redundan//mmeleant signals) rather than simply the extra parameters.
13

---

Published as a conference paper at ICLR 2024
Table 8: Verification of the primary contributor to the performance improvement the CIFAR-100 dataset. The "ex-L" a pure extra layer of linearity with the same number of parameters as RAA.
Method
ReLU
GELU
cx-L ReLU
ex-L + GELU
RAA
Trainable Parameters (with ViT-Tiny)
5380132
5380132 65.4
5389360 65.6
5389360
5389360
Top-1 Acc %
655.7
65.7
66.5
A.2 HYPERPARAMETE SELECTION
Momentum for updating the moving mean I in Eq. In our experiments, the momentum m is empirically set to 0.1 without too much consideration. Theoretically, high value of m risks making the mean value unstable, and conversely, low value of m smoothens the mean value update, but may cause the mean value to lag behind. As illustrated in Figure 3(a), the network's performance is not sensitive to the m between around 0.1 and 0.8, and extreme m values lead to negative effects.
Balanced parameter À for the ARR loss Carr in Eq. 7. The optimal balanced parameter À for Larr specific to individual tasks. The relation between and the classification accuracy training CIFAR- 100 with ViT-Tiny is depicted in Figure 3(b), where the optimal À is roughly 20, and too large À results in negative side effects. For other trials, we found the optimal to around 10 when training CIFAR-10 with ViT-Tiny and TNT-Small, approximately when training CIFAR-100 with CarT-XXS, etc. Overall, the network's s performance is not sensitive to À if not too large. Searching the optimal for each task can be consuminn,,, but one thing is for sure: small values do not hurt accuracy. Hence, for the experiments on ImageNet-IK, we set the default to 1.
(a)
(b)
Top-1 Acc %
Top-1 Ace %
70
68
66
64
66
0.0
0.1
0.2
0.5
0.8
1.0
20 2
100
150
Figure (a) Top-1 accuracy (%) w.r the momentum m for updating the moving mean in Eq. 5 when training the CIFAR-100 dataset using the ViT-Tiny model. (b) Top-1 accuracy (%) w.r. the balanced parameter À for Larr Eq. 7 when training the CIFAR- 100 dataset using the ViT-Tiny model.
A.3 EXPERIMENTS IMAGENET-IK
Table 9 presents the top-1 accuracy (%) achieved by the proposed DNRT on various ViT architectures when evaluated on the ImageNet- dataset. As with the results in Table 2, the ones on ImageNet- KK continue to highlight the consistent superiority of DNRT over the baseline methods, showcasing the scalability and robustness of the proposed approach.
Table 9: Top-1 accuracy (%) on the ImageNet- 1K dataset using the proposed DNRT on Vision Transformer (ViT) and its variants.
Top-1 Acc/
ViT-Tiny
DeiT-Tiny
CaiT-XXS
PVT-Tiny
TNT-Small
ReLUU
70.9
73.2
74.0
73.7
73.4
GELU
70.4
73.0
73.6
73.5
73.3
ImageNet-IK
DNRT
73.0
73.5
75.3
76.1
75.6
A.4 MORE NEURAL RESPONSE VISUALIZATION
This section extends $5.6 "Neural Response Visualization" of the main paper, providing more neural response visualization results in Figure 4. As shown in Figure 4(a), with the static response activation. the presented responses are active on almost every channel. Some irrelevant inputs that should not be activated mistakenly activated, and of course, few relevant inputs that should be activated might
14

---

Published as a conference paper at ICLR 2024
be suppressed. After using the proposed RAA, the activation responses become significantly sparser, implying that most irrelevant features are suppressed, which can reduce the network's 's learning difficulty and improve the network's s interpretability. Figure 4(b) shows the aggregated response distribution of ViT's MLP on the class token across different input categories. In the absence ARR, the variance of aggregated responses each channel of the class token is significantly high, making it challenging to differentiate class tokens for different objects visually. The use of ARR makes the response of each channel more focused, enabling the distinguishabiltyy of responses across different categories, thus facilitating the network's 's final decisions.
w/o RAA (mlp-1)
w/ RAA (mlp-1)
0
10
20
30
40
50
10
20
30
40
50
w/o RAA (mlp-4)
RAA (mlp-4)
0
30
50
w/o RAA (mlp-10) 10)
w/ RAA (mlp-10)
0
10
20
30
40
50
10
20
30
40
50
Visual results of RAA
w/o ARR (airplane)
w/ ARR (airplane)
3
3
10
20
30
40
50
-3
10
20
30
40
50
w/o ARR (bird)
w/ ARR (bird)
3
20
30
40
-3
10
20
30
40
50
w/o ARR (dog)
w/ ARR (dog)
0
-3
20
30
40
10 20 30
40
50
(b) Visual results of ARR
Figure Response distributions the class token in ViT's MLP block with and without the proposed DNRT on the CIFAR- 10 dataset. (a) MLP's activation responses in ViT layer-1, 4, and 10. (b) MLP's aggregated responses to airplane, bird. and dog images. The baseline the network with the GELU activation (the default used in VIT) and the original response aggregation mechanism. The horizontal coordinate represents the element's (channel's) index within the feature vector, the vertical coordinate represents the response value, and the area depicts the response density. The figure displays the response distributions of the first 50 elements (channels) in each vector (the class token).
15

---

Published as a conference paper at ICLR 2024
A.5 COMPARISON WITH SNNS
Leaky Integrate-and-Finn (LIF) (Maass, 1997) is used as the activation mechanism in Spiking Neural Networks (SNNs), which simulates the process of potential integration and output in biological neurons. When the potential accumulates to certain threshold, it triggers a pulse output, after which the potential gradually returns to the baseline level through the leakage mechanism. The dynamic nature of LIF is reflected in two aspects:
Time dependence of potential. At each moment, there will be potential accumulation or release. The potential varies over time. However, it is currently still in the exploration stage, without demonstrating performance advantages over existing approaches.
2. Learnable thresholds. The original LIF in SNN (Maass, 1997) used a fixed threshold, and recently has started supporting learnable thresholds (LIF-LT) (Wang et al., 2022), which shares similarities with our DNRT approach. However, LIF-LT simply determines the threshold by scalar learnable parameter, which still lacks flexibility. In LIF-LT, neurons in the same layer share the same threshold and cannot adapt it based on the contextual input. As result, it is unable to avoid irrelevant features incorrectly triggering activation responses. In contrast, DNRT judges the relevance each input signal and adjusts the threshold accordingly for each signal, offering greater flexibilityy compared to LIF-LT's learnable threshold.
Additionally, Table 10 presents the numerical comparison the CIFAR-10 dataset. In terms of performance, the mechanism of DNRT remains advantageous.
Table 10: Top-1 accuracy (%) of the proposed DNRT compared with SNN's Leaky Integrate-and and-FFe (LIF) mechanisms on the CIFAR-10 dataset.
Top-1 Acc / %
Softplus
ELU
SELU
SiLU
ReLU
GELU
GDN
LIF
LIF-LT
DNRT
ResNet-19
93.8
94.4
93.2
95.2
95.1
95.2
82.1
92.9
93.5
95.6
A.6 OVERHTTING TEST
Deep ANNs are susceptible to suffering severe overfitting when the available data highly insufficient, resulting in limited performance on the testing set. The proposed ARR mechanism is capable of alleviating this challenge. As previously mentioned, when the network's performance is poor, the aggregated response will exhibit Gaussian distribution with high variance. ARR leverages historical response statistics and adds loss constraints to new responses, thereby achieving more focused response distributions within the same category. We evaluate the performance of the ViT-Tiny model with and without the proposed ARR on the CIFAR- 10 dataset when using weak data augmentations only consisting of "random horizontal flipping" and "normalization" Figure 5 shows the loss and accuracy curves. Specifically, the loss curve on the testing set provides further evidence that ARR effectively overcomes overfitting. This is because the response constraint can effectively generalize the response clustering from the training set to the testing set.
2-
60
1
ARR (Irsin)
woo ARR
ARR (tmin)
wio ARR
w/ ARR (lust)
40
w/ ARR
0-
0
25
50
75
100
125
150
0
25
50
75
100
125
150
(a) Loss CE over epochs
(b) Top-1 Acc % over epochs
Figure 5: The classification performance of ViT-Tiny with and without the proposed ARR the CIFAR-IO dataset using weak data augmentations comprising only "random horizontal flipping" and "normalization" The model without ARR experiences severe overfitting, while the one with ARR overcomes it and yields massive performance gains.
16


Reviews:
[{"review_id": "73QUIDyJPY", "reviewer": "Reviewer_Hui7", "summary": "The authors developed a Dynamic Neural Response Tuning (DNRT) mechanism inspired by the dynamic response conditions of biological neurons, addressing the limitations of static response conditions in existing activation functions. DNRT consists of Response-Adaptive Activation (RAA) and Aggregated Response Regularization (ARR), which mimic biological neurons' information transmission and aggregation behaviors, respectively. RAA adjusts the response condition based on input signal strength and characteristics, while ARR enhances the network's ability to learn category specificity. Experiments show that DNRT is interpretable, compatible with various network architectures, and outperforms existing activation functions in multiple tasks and domains.", "questions": "1. In the presented experiments, it is crucial to disclose the number of parameters employed for each model. To ensure a fair comparison, the parameters for the baseline models must be comparable. Please provide this information in a clear and concise manner.\n\n2. It is imperative that the chosen baseline models are well-established and representative of the current state of the art in deep learning. Notably, state-of-the-art deep learning methods have demonstrated the ability to achieve accuracy levels exceeding 85% on the ImageNet dataset. Please confirm that the selected baselines adhere to this standard.\n\n3. Apart from the aspect of biological plausibility, it is essential to elucidate the motivations behind the development of RAA and AAR for the deep learning community. Please provide a comprehensive explanation of the underlying inspirations.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors developed a Dynamic Neural Response Tuning (DNRT) mechanism inspired by the dynamic response conditions of biological neurons, addressing the limitations of static response conditions in existing activation functions. DNRT consists of Response-Adaptive Activation (RAA) and Aggregated Response Regularization (ARR), which mimic biological neurons' information transmission and aggregation behaviors, respectively. RAA adjusts the response condition based on input signal strength and characteristics, while ARR enhances the network's ability to learn category specificity. Experiments show that DNRT is interpretable, compatible with various network architectures, and outperforms existing activation functions in multiple tasks and domains.", "weaknesses": "1. The author have established a series of baseline experiments; however, the performance demonstrated in these experiments appears to be inferior to the results reported in previous literature.\n\n2. RAA method introduces an additional parameter, W, which may compromise the fairness of comparisons with other activation functions.\n\n3. The authors did not state clearly of the significance of the study. What is the inspiration of this study for neuroscience, or for the development of deep learning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DLOdhq30Cp", "reviewer": "Reviewer_sidQ", "summary": "The paper delves into the realm of Artificial Neural Networks and their foundational principles inspired by biological neural systems. A focal point is the discernible distinction between the static response conditions in traditional ANNs and the inherently dynamic nature of biological neurons. Recognizing this disparity, the authors propose the Dynamic Neural Response Tuning (DNRT) mechanism.\n\nDNRT is a two-pronged approach comprising:\n\n1. Response-Adaptive Activation (RAA): A novel activation function that dynamically modulates its response based on the characteristics and magnitude of the incoming signal. Unlike traditional activation functions, which maintain a static response, RAA adapts, mirroring the dynamic response conditions observed in biological neurons.\n\n2. Aggregated Response Regularization (ARR): A mechanism designed to refine the aggregation of signals in the network. ARR aims to enhance the network's proficiency in discerning and classifying distinct categories, thereby boosting performance.\n\nTo validate the effectiveness of DNRT, the authors embarked on an extensive experimental evaluation across diverse architectures, including basic MLPs, ViTs and CNNs, Their findings suggest that DNRT not only integrates seamlessly across these architectures but also often outperforms traditional activation functions in terms of performance.", "questions": "Could the authors elaborate on the specific biological studies or evidence that influenced the design of DNRT? How closely does DNRT emulate the dynamic behaviors observed in biological neurons?\n\nAre there specific architectures, tasks, or datasets where DNRT might not provide significant benefits or might even be counterproductive? What are the potential challenges or drawbacks associated with DNRT?\n\nHow are the parameters in the Response-Adaptive Activation (RAA) introduced and managed? How do these parameters impact network complexity, and are there challenges in optimizing these parameters during training?\n\nBeyond the experimental setups, how does DNRT fare in real-world applications or larger, more complex datasets? Are there scalability concerns or other practical challenges?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper introduces the DNRT mechanism, a fresh perspective in the realm of neural network activation functions. While traditional activation functions have been studied extensively, the idea of aligning ANNs with the dynamic response patterns of biological neurons offers a distinct and innovative approach. The two-pronged design of DNRT, with Response-Adaptive Activation (RAA) and Aggregated Response Regularization (ARR), presents a unique combination of ideas, each addressing specific challenges in neural network design. The inspiration drawn from the dynamic nature of biological neurons and the attempt to emulate that in ANNs is a commendable original endeavor.\n\nThe paper showcases a meticulous and rigorous experimental evaluation across various architectures, solidifying its claims about the efficacy of DNRT. The ablation studies provide deeper insights into the individual and combined contributions of the components of DNRT, enhancing the paper's technical quality. \n\nThe paper is well-structured, with a clear flow from motivation to proposal, followed by experimental validation. Concepts like RAA and ARR are explained with precision, allowing readers to grasp the core ideas effectively. While there's room for more intuitive visualizations, the current presentation ensures that readers familiar with the domain can understand the proposed mechanism and its implications.\n\nAddressing the static nature of traditional activation functions and attempting to bridge the gap with the dynamic responses of biological neurons holds significant implications for the field of neural networks.\nIf DNRT can be consistently shown to enhance performance across various architectures, as suggested by the paper's results, it can lead to a paradigm shift in how activation functions are designed and implemented.", "weaknesses": "While the paper draws inspiration from the dynamic nature of biological neurons, a deeper exploration into the biological underpinnings could have been beneficial. More direct parallels between the DNRT mechanism and real-world neural behaviors would strengthen the paper's foundational claims. Consider integrating more biological studies or references that showcase the parallels between DNRT and actual biological neural behaviors, reinforcing the biological accuracy of the proposed mechanism.\n\nThe paper provides comprehensive experimental evaluations, but a detailed discussion on potential limitations or scenarios where DNRT might not be as effective is missing. Understanding these limitations can provide a more balanced view and guide future work. Dedicate a section to discuss potential limitations, challenges in broader applicability, or scenarios where traditional activation functions might still be preferable.\n\nWhile the paper does address prior work in the domain, a more detailed comparison or discussion regarding how DNRT builds upon or differs from existing methods would offer readers a clearer perspective on its novelty. The related work section could be expanded to include more direct comparisons with existing activation functions or mechanisms, highlighting the unique contributions of DNRT.\n\nThe paper mentions the adaptability of RAA and hints at additional parameters, but a clear breakdown of how DNRT introduces and manages these parameters, especially in comparison to traditional activation functions, is lacking. Provide a subsection detailing the parameterization of RAA, including how these parameters are learned, their impact on network complexity, and potential challenges in optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tOubMlFvz1", "reviewer": "Reviewer_2aiC", "summary": "The authors propose a method using novel activation functions which allows neurons in an ANN to provide outputs that cluster around particular classes more closely, by analogy with some observed behaviors of biological neurons.", "questions": "Bibliography: Perhaps reformat the bibliography for easier searching, eg numbered, last name first, initials only except for last name.\n\nAbstract: \n\nIs this first sentence an accurate summary of why ANNs have done so well? It seems there is much more to the matter.\n\n\"is achieved by triggering action potentials\": Often. Neurotransmitters and non-spiking neurons can also have prominent roles.\n\n\"depending on neuron properties and\": There is a broad literature on neurotransmitters and how they modulate neural behavior.\n\n\"dynamically adjusts\": my sense from the paper is that the activation function is modified to weight inputs differently, but once trained the response does not change dynamically, ie its response properties are fixed. If this is correct, \"dynamically\" is not a correct word.\n\n\"ARR is devide\": Maybe give some brief detail about this, to parallel the detail given about RAA is the sentence above.\n\n1. Paragraph 1: Perhaps find more comprehensive reviews for ANN progress (eg Goodfellow, etc).\n\n1. paragraph 3 \"otherwise it suppresses\": This is true for ReLU, but not for other activation functions eg tanh. Also, this would be a good place to cite some good literature reviews of flavors of activation functions.\n\n\"An input value ranked later among all inputs\": Does this refer to time-dependent neural response? I did not see this addressed as a technique or effect in the paper. \n\n\"always adheres to a gaussian curve\": Always? This framing is too simple. Also in this paragraph: the distinction between when referring to ANNs or BNNs is unclear. \n\n\"It is a a common ... final decision.\": This strikes me as vague. Can the sentence be sharpened?\n\n2.1 \"surpasses the threshold\": For spiking neurons (see above comment about non-spiking types and local spread of neurotransmitters).\n\n2.2 \"pivotal role in neural networks\": Does this refer to ANN, BNN, or both?\n\nBottom of page 3, \"Biological NNs...\": This is an important observation. However, does the proposed method really emulate this, or does it modify the activation function to a new, static, form?\n\n3.1, last sentence: While BNNs feature dynamically changing activation functions, does the proposed method do this (same as previous comment)? Perhaps this is really a statement about my lack of understanding of the proposed method.\n\n3.2, first sentence: I believe this is too simple a description.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The program idea, to make individual neurons in an ANN responsive to context or input distribution or similar is a good one.", "weaknesses": "General review context: Please note that I am simply another researcher, with some overlap of expertise with the content of the paper. In some cases my comments may reflect points that I believe are incorrect or incomplete. In most cases, my comments reflect spots where an average well-intentioned reader might stumble for various reasons, reducing the potential impact of the paper. The issue may be the text, or my finite understanding, or a combination. These comments point to opportunities to clarify and smooth the text to better convey the intended story. I urge the authors to decide how or whether to address these comments. I regret that the tone can come out negative even for a paper I admire; it's a time-saving mechanism for which I apologize.\n\nThe text is somewhat confusing to me. Examples: 1. It does not clearly distinguish between ANNs and biological NNs (BNNs), which confuses the exposition; 2. The method is not (for me) explained clearly enough, so I did not come away with a clear sense of what it entailed.\n\nI suspect that the literature grounding is incomplete (I'm not expert).\n\nReviewer limitation: I had trouble understanding this paper, the mechanics of the method, and why it would be beneficial. So perhaps I am not an optimal reviewer. Or perhaps this reflects the need for some degree of rewriting.\n\nNotes to ICLR: \n\n1. Please include line numbers in the template. They make reviewing much easier! \n\n2. Please reformat the default bibliography style to make searching the bib easier! eg numbered, last name first, initials only except for last name.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r5iAThj4SS", "reviewer": "Reviewer_9nGD", "summary": "The authors' rebuttal have essentially addressed my concerns, so I have raised the score of the paper to a weak accept.\n\n=========================================================================================\n\nThis paper proposes a novel Dynamic Neural Response Tuning (DNRT) mechanism inspired by the dynamic response conditions of biological neurons. The DNRT mechanism comprises Response-Adaptive Activation (RAA) and Aggregated Response Regularization (ARR), which mimic the information transmission and aggregation behaviors of biological neurons. The authors demonstrate that the proposed DNRT can be applied to various mainstream network architectures and can achieve remarkable performance compared with existing response activation functions in multiple tasks and domains. The paper contributes to the simulation of dynamic biological neural response patterns and provides new concepts and techniques for constructing Artificial Neural Networks (ANNs).", "questions": "Can the authors provide more in-depth comparisons between the proposed DNRT mechanism and existing activation functions, discussing the advantages and disadvantages of each approach? Additionally, please include a comparison with spiking neural networks, as they also employ dynamic activation functions.\n\nHow does the DNRT mechanism generalize to other tasks and domains beyond those presented in the experiments? Are there any limitations or challenges in applying DNRT to different network architectures?\n\nWhat are the potential implementation challenges and computational costs associated with using the DNRT mechanism in real-world applications? Are there any strategies to mitigate these challenges?\n\nConsidering the mainstream view that \"computation is all you need,\" it is important to evaluate the additional computational costs and bandwidth bottlenecks introduced by DNRT. This is a concern as activation functions like Softplus and ELU are not widely used due to their lower computational efficiency compared to ReLU. Additionally, there are ongoing discussions about replacing the softmax layer in transformers with ReLU for improved computational efficiency. Could the authors provide an assessment of the impact of DNRT on model inference speed when replacing ReLU?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: The paper presents an innovative approach to mimicking the dynamic response conditions of biological neurons in ANNs, which is a significant departure from existing static activation functions.\n\nQuality: The proposed DNRT mechanism is well-motivated and grounded in neuroscience research. The Response-Adaptive Activation (RAA) and Aggregated Response Regularization (ARR) components are clearly explained and justified.\n\nClarity: The paper is well-written and organized, with clear explanations of the DNRT mechanism, its components, and the motivation behind it. The figures and tables help illustrate the concepts and results effectively.\n\nSignificance: The DNRT mechanism has the potential to improve the performance of ANNs in various tasks and domains, as demonstrated by the extensive experiments conducted by the authors. The paper also provides valuable insights into neural information processing research.", "weaknesses": "Limited comparison with existing methods: While the paper demonstrates the performance of DNRT compared to existing activation functions, it would be helpful to see a more in-depth analysis and discussion of the differences between DNRT and other approaches, including spiking neural networks (SNNs). SNNs are based on neural dynamics and employ dynamic activation functions, which share similarities with the ideas presented in this paper.\n\nLack of generalization analysis: The paper claims that DNRT is applicable to various mainstream network architectures, but it would be beneficial to provide a more detailed analysis of how the proposed method generalizes to other tasks and domains beyond those presented in the experiments.\n\nPotential implementation challenges: The paper does not discuss the possible challenges in implementing the proposed DNRT mechanism in real-world applications or the computational cost associated with its use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)