Learning to Act through Activation Function Optimization in Random Networks
Authors: Joachim Winther Pedersen
Sebastian Risi
Venue: Submitted to ICLR 2023 (2023)
Keywords: artificial neural networks
activation functions
neural diversity

Abstract: Biological neural networks are characterised by a high degree of neural diversity, a trait that artificial neural networks (ANNs) generally lack.  
Additionally,  learning in ANNs is typically synonymous with only modifying the strengths of connection weights.    
However, there is much evidence from neuroscience that different classes of neurons each have crucial roles in the information processing done by the network. In nature, each neuron is a dynamical system that is a powerful information processor in its own right. In this paper we ask the question, how well can ANNs learn to perform reinforcement learning tasks only  through the optimization of neural activation functions, without any weight optimization? 
We demonstrate the viability of the method and show that the neural parameters are expressive enough to allow learning three different continuous control tasks without weight optimization.
These results open up for more possibilities for synergies between synaptic and neural optimization in ANNs in the future.

Under review as a conference paper at ICLR 2023
LEARNING TO ACT THROUGH ACTIVATION FUNCTION OPTIMIZATION IN RANDOM NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Biological neural networks are characterised by high degree of neural diversity, a trait that artificial neural networks (ANNs) generally lack. Additionally, learning in ANNs is typically synonymous with only modifying the strengths of connec- tion weights. However, there is much evidence from neuroscience that different classes of neurons each have crucial roles the information processing done by the network. In nature, each neuron is dynamical system that is a powerful infor- mation processor in own right. this paper we ask the question, how well can ANNs learn to perform reinforcement learning tasks only through the optimization neural activation functions, without any weight optimization? We demonstrate expressive
the viability of the method and show that the neural parameters enough to allow learning three different continuous control tasks without weight optimization. These results open up for more possibilities for synergies between synaptic and neural optimization in ANNs in the future. Code is available from [anonymised].
INTRODUCTION
Artificial neural networks (ANNs) have been shown to able learn a wide variety of different tasks (Schmidhuber) 2015). With inspiration from their biological counterparts (Hassabis et al.) 2017). ANNs have dramatically pushed the boundaries for what is achievable for artificial intel- ligence technologies. ANNs are trained by tuning a large number of parameters, each of which provides a small contribution to the final output of the network. Likewise, most, but not all (Tit- ley al. 2017). learning and behavioral changes are manifested in the biological brain as long- or short-term potentiation or depression of synapses between neurons (Stiles) 2000).
Neurons of the human brain are characterised by high degree of diversity (Lillien 1997 Soltesz et al. 2006), and different classes of neurons respond differently to the incoming signals is (Izhieeicch 2003 A single biological neuron is sophisticated processor in its own rights (lzhikevich 2007 Poirazi et al. 2003), with information processing occurring at several steps both in its dendrites al. 2018: Magee 2000), cell body and
terminals (Kamiya & Debanne 2020 Rama et al. 2018). Neurons of various classes interconnect in intricate circuits (Breedlove) & Watson 2013 Kandel al. 2000). This suggests that at Ieast part of the explanation behind
BBeaulieu-Laroche
the impressive ability of biological networks to learn and retain knowledge must be found in the
interplay between the abundance of different neuron types (Nusser 2018).
While the diversity of biological neurons is well documented, in ANNs it is common to have a single activation function used by all hidden neurons. Intrigued by the interesting properties of randomly-initialistt networks in both machine learning (Gaier & Hal 2019 Najarro & Risi 2020) Ulyanov et al. 2018) and neuroscience (Lindsay et al., (017)) we are interested in the computa- tional expressivity of only optimizing parameterized neural activation functions without any weight optimization. As described below, our approach allows every neuron in our ANNs to be a unique
dynamical system.
We apply our method to three diverse continuous control tasks. The simpler CartPoleSwingee
Gaier & Ha (2019), the locomotion of bipedal robot (Brockman et al. 2016). and basedd car racing task with procedurally generated tracks (Brockman et al. 2016). The results show that the method performs well all three tasks, outperforming weight-optimized networks that have

---

Under review as a conference paper at ICLR 2023
similar number of adjustable parameters in two out of three tasks. Surprisingly, optimized activa- tion functions in random networks even outperform a weight-optimized network with many more adjustable parameters in the more challenging CarRacing-ve environment.
While there cxist previous work that deals with optimizing activation function parameters (Agostinelli et al., 2014 Sipper 2021 Bingham & Mikkkulainen, 2022), to the best of knowl- edge, here we show for the first time that optimizing expressive activation functions alone allows solving challenging RL tasks. We hope that our results inspire further research in approaches that do not only see ANN optmization as the optimization of weight parameters and challenging some of our assumptions on what means for such systems to learn.
Neural Input
Neuron
Neural Output
Input
P1
P2
Ps
Output
Neural State
P
Ps
P6
New State
1
P7
Ps
Pg
Figure 1: Iilustration of the proposed neural activation function. The parameters p; are optimized in order to achieve expressive function. These parameters used integrate the input with neural state and a bias term through a vector-matrix multiplication. As input, the neuron takes an input value propagated from the previous layer, its neural state, and constant value of one bias term. outputs a value to propagated to the next layer, as well as its own new state. For further details see Section
2 RELATED WORK
Neurocentric Optimization. Biases in neural networks is an example of neurocentric parameters. When an ANN optimized solve any task, the values of the network's weights and biases are gradually being tuned until functional network has been achieved. The network most commonly one bias value for each neuron and the weight parameters thus greatly outnumber these neu- rocentric bias parameters. It is well known that the function of biases to translate the activations the network (Benítez al., 1997), and ease the optimization of the network. Another well known example of neurocentric parameters where a parameter learned to determine the slope the function in the case of negative inputs. Introducing this per neuron customization of the activation functions shown to improve perfor- mance of networks with little extra computational cost. Neurocentric parameter optimization can also be found within the field of plastic neural networks. In one setting of their experiments, [rzelai & Floreano| (200I) optimized plasticity rules for each neuron (they referred to this as "node encod- ing") such that each incoming synapse of neurocentric parameters is therefore far from new. However, in contrast to earlier work, in this paper we explore the potential of solely optimizing the activation functions of a randomly initialized network without ever adapting the weights.
found in the PReLU activation functions (He et al., 2015)
node was adapted by common plasticity rule. The idea
Activation Functions Neuroevolution. Not all ANNs have single activation function for all hidden neurons. Some version of Ceouu-Evolution of Augmented Topologies (NEAT) (Stanley & Mikkulainenl 2002, Papavasileiou et al., (02II) allow for different activation functions on each neu- ron. The NEAT algorithm searches through networks with increasing complexity over the process of evolution. Starting from a simple network structure, each new network has a chance adding a new neuron the network. When from a number of predetermined functions. In newer versions NEAT, mutations allow activation functions of neurons be changed even after it was initially added (Hagg et al. (017). This resulted
neuron is added, it can be allocated random activation function
2

---

Under review as a conference paper at ICLR 2023
in more parsimonious networks. In their weight agnostic neural network (WANN) work. Gaier & Ha (2019) used NEAT to find network structures that could perform well, even when all weights had the same value. Notably, as part of the evolved structure was the possibility of neurons to have different activation functions, and different neurons could thus respond differently to the same in- coming signals. This likely extended the expressiveness of the WANNS considerably. We the work presented below as being in similar spirit as that of Gaier and Ha, in that we are also exploring the capabilities of component of neural networks in the absence of traditional weight optimization. In the work presented in this paper, all networks have a standard fully connected structure. Further, we do not choose from a set of standard activation functions, but introduce stateful neurons with several parameters to tune for each individual activation function.
Lottery Tickets & Supermasks. The Lottery Ticket Hypothesis (Frankle & Carbin 2018 states that the reason large ANNs are trainable is that large network contains a combinatorial number of subnetworks, one of them is likely to be easily trainable for the task at hand. The hypothesis is based on the finding that after having trained a large network, it is usually possible to prune large portion of the parameters without suffering significant loss in performance (Li et al. 2016). An even stronger take the Lottery Ticket Hypothesis states that due to the sheer number of subnetworks that present within a large network, it is possible to learn a binary mask on top of the weight matrices of a randomly initialized neural network, and in this manner get network that can solve the task at hand (Malach et al.] 2020, Wortsman et al.] 2020 Ramanujan et al., (020). This has even been shown to be possible at the level of neurons; with large enough network initialization, a network can be optimized simply by masking out a portion of the neurons in the network (Wortsman et al. 20000] Malach et al.] 2020). Learning an activation function for each neuron could be seen as learning a sophisticated mask on top each neuron, something that has been demonstrated to be possible. However, the idea of learning mask on top of random network relies on the random network being large enough (Malach et al. 2020), so that the chance of it containing a useful subnetwork high. Masking neurons is a less expressive mask method than masking weights: it is equivalent to masking full columns of the weight matrices instead of strategically singling out weights in the weight matrix. As such, the method of masking neurons require larger random networks in order to be successful. In this paper, we optimize neurons' activation functions in relatively small networks. This is possible because the activation functions themselves are much more expressive than a binary mask.
3 SEARCHING ONLY FOR ACTIVATION FUNCTIONS IN RANDOM NETWORKS
Typically, optimization of ANNs has been framed as the learning of distributed representations (Bengio et al., 2013) that can become progressively more abstract with the depth of the network. Optimization of weights, is process of fine-tuning the iterative transformation of one representation into another in order to end with a new, more useful representation of the input.How How the intermediate layers layersrespone respond to given input depends the specific configuration of the weight matrix responsible for transforming the input as well as their activation function.
Randomly-initialisedd networks can already perform useful computations (Ulyanov al. 2018 He let al. 2016; Hoccreiter & Schmidhuber] 1997). When activation functions are trained specifically to interpret signals from a fixed random matrix, as is the case in this paper, the initially arbitrary transformations will become meaningful to the function, as long as there exist detectable patterns between the input the function receives and the output the function passes on and is evaluated on. Whether a pattern is detectable depends on the expressiveness of the function. With this in mind, it reasonable to assume that if activation functions of the neurons in the network are made more ex- pressive, they can result in useful representations even when provided with arbitrary transformations of the input.
Motivated by the diversity of neuron types in animal brains (Soltesz et al.) 2006), we aim to test how well RL agents can perform if we only optimize the parameters of a network's 's activation functions while not optimizing its weights at all. An illustration of the activation function representation we optimize in this paper is shown in Fig. T Each activation function consist of a small three-by- three matrix of values to be optimized. Representing a neuron by small matrix means that the neuron can take more than a single value as input, as well as outputting more than one value. We utilize this to endow each neuron with state. The state of the neuron is integrated with the input
3

---

Under review as a conference paper at ICLR 2023
through the optimized neural parameters. Part of the neuron's output becomes the new state of the neuron, which is fed back to the neuron with the next input. This turns our activation functions into dynamical systems. Presented with the same input value at different points in the neuron's 's history can thus yield different outputs. We find that stateful neurons provide a convenient and efficient way of equipping network with some memory capabilities.
When a representation has been transformed by a weight matrix, each activation function receives one value of the transformation, same as in traditional ANNs. This input value is concatenated with the current state of the neuron and a third value that is always equal to one. Together, these form a vector. The output of neuron is the vector- multiplication of this vector and the matrix that makes up the parameterized neuron. From the perspective of a single neuron, this can be written as:
[ Mf,5 Lanh(mi [af M177 117))
Here, zi.; is the input value, M7 1 is the state of the neuron, and 11 the neuron with I denoting the current layer in the network, i the placement of the neuron in the layer, and t is the current time step. A value of I is concatenated as a bias term for the neuron. The hyperbolic tangent function used for non-linearity and to restrict output values to be in [-1, 1]. 4[41,J is the value that is propagated through weights connecting to the subsequent layer, and hf, is the updated state of the neuron. o, the third value in the output of the neuron is discarded. As the three-by-three matrix has nine values in total, we need to optimize nine parameters for each neuron in the network.
4 EXPERIMENTS
We optimize activation function parameters in otherwise standard fully connected feedforward neu- ral networks. All the these networks have two hidden layers, containing 128 and 64 neurons, re- spectively. We use learned activation functions layer. The sizes of the input and output layers vary with the environments described below. The fixed weight values seeds, excepts for the weight-optimized models in the environment, which we ran twice due to its longer training times.
all neurons, including in the input and output
sampled from A(0,, 0.5). We ran each experiment three times different
As baselines, we optimize weights of standard feedforward networks. For these we run two differ- settings: one has similar number of adjustable parameters the number of parameters in the activation function approach (Small FFNN Architecture). In order to get the number of weights be similar to activation function parameters, the widths and depths of these networks have to be considerably smaller than the random networks used in the main experiments. In the second baseline setting (Same FFNN Architecture), we train weights of networks that have the same widths and depths as the networks in the main experiments, and thus many more adjustable parameters. Un- less stated otherwise, the activation function for all baseline experiments is the hyperbolic tangent function for all neurons.
4.1 ENVIRONMENTS
We test the effectiveness of only optimizing the parameters of the activation functions in randomly initialized networks in three diverse continuous control tasks: the CartPoleSwingUp environment
Gaier& & Ha 2019), the Bipedal Walker environment, and the Car Racing environment (Brockman ct al. 2016), which are described below:
CartPoleSwingUp. This enviromment is a variation of the classic control task (Barto et al. 1983 where a cart rewarded for balancing a pole for as long as possible. In the CartPo Ses ngUp variation, an episode starts with the pole hanging downwards, and in order to score points, the agent must move the cart such that the pole gets to an upright position. From there, the task is to keep balancing the pole. We use the implementation of Gaier and Ha (Gaier & Hal 2019). The agent gets five values as input, and must output a single value in [-1,1] in order to move the cart left and right. With these input- and output layers. the total number of neurons in the network becomes 198, and we optimize 1.792 parameters. For a feedforward network with a similar number of weights we optimize network with two hidden layers of size 48 and 32, respectively.
4

---

Under review as a conference paper at ICLR 2023
056g
Figure 2: Environments used for experiments.
Left::
CartPoleSwingUp.
Middle:
BipedalWalker-v3 Right: CarRacing-vO.
Table 1: Convolutional Layer Parameters.
Layer
Layer 2
Input Channels
6
Output Channels
6
8
Kernel Size
5
Stride
2
Activation Function
tanh
tanh
Bias
Not used
Not used
Bipedal Walker. To get maximum score in the sipedalWalker-v environment (Brockman ct al. 2016), two-legged robot needs to learn to walk as efficiently and robustly as possible. The terrain is procedurally generated with small bumps that cause the robot to trip if its gait is too brittle. Falling over results a large penalty to the overall score the episode. Observations environment consist of 24 values, including LIDAR detectors and information about the robots joint positions and speed. For actions, four values continuous values in [-1, 1] are needed. This result in network with 220 neurons altogether. To get network with a similar number weights as there are parameters in the main experiment, we train a feedforward network with two hidden layers, both containing neurons.
this
Car Racing, In the CarRacing domain (Brockman et al. 2016), car must learn to navigate through procedurally generated tracks. The car is rewarded getting as far as possible while keeping inside the track at all times. To control the car, actions consisting of three continuous values are needed. One of these is in [-1, (for steering left and right), while the other two are in [O, 1], one for controlling the gas, the other for controlling the break. The input is top down image of the car and its surroundings, as shown in Fig. 4.1 For the experiments in this paper, we wanted to focus the effectiveness of the activation functions in fully connected networks. We therefore followed a strategy closely mimicking that used by Najarro & Risi (2020) to get a flat representation of the input image. We normalize the input values and resize the image into the shape of 3x84.884. The image is then send through two convolutional layers, with the parameters of these specified in Table I After both layers two-dimensional max pooling with kernel size 2 and stride 2 was used to gradually reduce the number of pixels.
The output of the convolutional layers flattened. resulting in vector containing 648 values. This vector is then used as input to fully connected feedforward network with our proposed activation functions. Importantly, the parameters the convolutional layers stay fixed after initialization and are never optimized. The output layer has three neurons. With the much larger input layer, this network has 844 neurons and 7,596 adjustable parameters. Since two of the action values should be in [0, 1]. sigmoid function is used for these two output neurons in the place of the hyperbolic tangent function as shown in Equation I.
the same strategy of convolutional layers with fixed parameters to get a flat input for the feedforward networks, the weights of which we are optimizing. Since the input is so large, the feedforward network can only have single hidden layer of size 12 to get a similar number of adjustable parameters as in the main approach. This results in a network with 7,827 parameters, including weights and biases. The activation function for all neurons in the
For the baseline experiments,
5

---

Under review as a conference paper at ICLR 2023
network is the hyperbolic tangent function, except for two of the output neurons, which are activated by the sigmoid function.
4.2 OPTIMIZATION DETAILS
For parameter optimization, we use combination of Genetic Algorithm (GA) and Covariance Matrix Adpatation Evolution Strategy (AMA-ES)Haansee (2006). More specifically, we GA is used for the first 100 generations of the optimization. The best solution found by the GA then used as starting point for the CMA-ES algorithm. For both algorithms, we use off-the-shed
implementations provided by Ha 2017 and Hansen (206). For all experiments, the GA uses population size of 512, and its mutations are drawn from a normal distribution (0,, 1). All other hyperparameterss are the default parameters of the implementation. The large sigma means that the GA can cover large area, but in coarse manner. The use of an initial GA phase is meant to kick-start the optimization for subsequent fine-tuning by CMA-ES. For the CMA-ES algorithm, we use a population size of 128, and set weight decay to zero. Other than that, hyperparameters are the default parameters of the implementation. The total number of generations 1,400 for the Car Racing environment, and 4,000 for the pidda 1 Wa ker and PortPoleSwinggee Sgg environments.
For the large weight-optimized networks, CMA-ES becomes impractical to use due to its use of the covariance matrix of size 22 with N being the number of parameters to optimize. For this reason, we instead use Evolution Strategy (sometimes referred to as "OpenES" (Ha (0176)) as described in Salimans et al. (2017) and. We use a population size of 128 and otherwise use the default parameters of the implementation Ha 20175).
5 RESULTS
Evaluations of the most successful runs of each experimental setting are summarized in Table 2 All experimental settings achieved good scores on the CartPoleSwingfi task. Only the weight- optimized network with hidden layers of size 128 and 64 managed to average a score above 300 points over 100 episodes in the B Bipedal 1 Wa ker-v3 environment, However, the activation func- tions alone came close with just a fraction of the number of optimized parameters. The smaller weight-optimized network's 's average score of 235 is indicative of the agent having learned to walk to the end of the level in most cases, but in an inefficient manner.
In the Ca CarRacing v environment, our method scored the highest, though none of the approaches reached an average score above 900 over 100 episode, which is needed for the task to be considered solved. Training curves for all experimental settings can be found in Appendix A.
5.1 INVESTIGATING LEARNED ACTIVATION FUNCTIONS
To get a better idea how neural network with random weights but optimized activation functions is solving the task, we plotted all activation functions of a layer of the champion network (Fig. (5.2) of the CarRacing environment. The figure shows that while several of the found activation functions look like the standard form we would expect from a hyperbolic tangent function with a bias, many of the other types of functions also emerged after optimization. We functions with strong oscillatory behaviors, some the whole input space, and some only in smaller sections. Other functions have extra peaks and valleys compared to the standard hyperbolic tangent. This allows these functions to respond with more nuance to input and create richer representations. Analysing these networks further to get more detailed understanding of how the solve the task an interesting future research direction.
5.2 COMPARISON TO WEIGHT AGNOSTIC NETWORKS
An approach similar in spirit to ours is the weight agnostic neural network approach by Gaier & Ha (2019). As detailed in Section 2 in WANNs, only the architecture of the neural network is learned (including choosing an activation function from predefined set for each neuron) while avoiding weight training. While an apples-to apples comparison is not possible (due to different optimization algorithms), is interesting to see how these two methods compare. The optimized activation functions tend to score better in all three environments. Since parameters are added to the

---

Under review as a conference paper at ICLR 2023
Table of Results. Means and standard deviations over 100 episodes, and number of pa- rameters for each experimental setting. In the "Activation Functions' models, only parameters of activation functions are optimized. In the other models, only the weight parameters of feedforward neural networks are optimized. Scores are evaluated with the most successful run in each setting.
Table 2:
For context, results from Weight Agnostic Neural Networks (WANN) (Gaier Ha 2019) are cluded as another method that does not optimize weights.
Model
Score
# Params.
Activation Functions
916 + 79
1,782
CartPoleSwingup
Small FFNN Architecture
927 = 83
1,889
Same FFNN Architecture
922 + 73
9,089
WANN
732 +: 16
N/A
Activation Functions
295 + 63
1,980
BipedalWalker
Small FFNN Architecture
235 = 13
1,988
Same FFNN Architecture
318 = 46
11,716
WANN
261 + 58
N/A
Activation Functions
812 - 95
7,587
CarRacing
Small FFNN Architecture
770 + 167
7,827
Same FFNN Architecture
752 + 171
91,523
WANN
608 = 161
N/A
WANN models during optimization, we cannot compare the numbers of adjustable parameters in these models to the that the activation functions. However, these results suggest that it might be easier to optimize customizable activation functions for each position in fully connected network, than it to learn a network structure from scratch.
In the future both approaches could be complementary. We imagine that extending the WANN approach with more expressive activation functions could allow their evolved neural architectures to become significantly more compact and higher-perfonnn performing.
DISCUSSION AND FUTURE WORK
In this paper, we introduced an approach to optimize activation functions for stateful neurons. Training these alone yielded neural networks that can control agents in the CartPole Poo.eSwiinggg CarRacing, and Bipeda Wa Mere environments, even when the weights of the network were never optimized. The largest weight-optiming method in the CartPoll prooeewwinggh and Bi BipedalWa Malalaall Walker environments. This is not surprising; weight optimization of ANNs now has a long history of success in plethora of domains. When using random transformations, there is a risk of getting a degraded signal, something that can be compensated for easily by tuning the weights of the transformations. Nevertheless, the proposed ac- tivation function alone did enable meaningful behavioral changes in the agents. Surprisingly, the op- timized activation functions achieved the best score of the experimental settings in the Ca rRa e ng task. The failure of the larger weight-optimized network to perform well here might be explained by the relatively low population size compared the number of parameters being optimized (128, and 91,523, respectively). This population size was the same for all experimental settings to ensure that all models are evaluated the same number of times the environments during optimization. The advantage of having smaller number of adjustable parameters also came into display in that the larger models could not be optimized by the more power CMA-ES method.
network achieved superior scores compared to our
Our results come with some caveats. Specifically, they should be viewed as a proofs- wwoofs-of-core ooe-oof--ccooeepee lack- ing extensive hyperparameter search. Hyperparameters that are likely important for achieving better performance include network depth and layer width, as well as the distribution from which weights of the random networks are drawn. Here, we were mainly interested in showing performance of the activation functions in relatively small networks. Further, we only tested a few hand-selected distributions for the random weights in preliminary experiments before settling on the distributions used in the experiments described above. Therefore, these areas were there is likely room im- provements. As part of the proposed activation function, we included a persistent neural state that is fed back the input of the neuron the subsequent time step. This provides endow the network

---

Under review as a conference paper at ICLR 2023
Figure 3: Diverse Activation Functions. Displayed are each of the 64 activation functions of the second hidden layer optimized to solve the CarRacing task. In blue are the activations that are passed on the next layer. In orange are the neural states. One thousand inputs are given from -3 to 3 in an ordered manner, from most negative to most positive. Given the updatable neural state, the ordering of the imputs matter, and a different ordering would have yielded different plots.For all plots, the neural state is initialized as zero before the first input. Several of the found activation functions look like we would expect the hyperbolic tangent function with bias and/or the possibility of negative sign to look. Few functions seem unresponsive to the input. However, many functions are clearly both responsive and different from the standard form of the hyperbolic tangent function. all of the input space. Other functions
We find functions that have oscillatory behavior in are non non-monotonic and have peaks and valleys particular areas.
with a memory mechanism. Memory local neural states is unusual in ANNs, but much more common in more the biologically inspired Spiking Neural Networks (SNNs) (Tavanaei et al.) 2019 Pfeiffer & Pfeil 2018 Izhikevich 2006). Such a neural state is most useful for data with temporal element such agents acting in environment. It is reasonable assume that the same approach would have limited use in tasks with unordered data. However, for RL tasks, stateful neurons pro- vide relatively cheap way of allowing the network to have traditional recurrent neural networks (RNNs), like LSTMs (Hochreiter & Schmidhuber, 1997), for the tasks used in this paper, would likely result the need of many adjustable parameters than the number of parameters in our activation functions. Combining stateful neurons with RNNs could result interesting memory dynamics on different timescales.
memory capacity. Setting up
The ability to improve performance while leaving weights random opens up the possibility for future work of combining activation functions like the ones proposed here with the approach of masking weights Frankle & Carbin (2018) mentioned in Section 2 An advantage of using masks is that one can train masks for different tasks on the same random network (Wortsman al. 2020). With pre-trained activation functions, it should be possible to bias the random network to perform actions that are generally useful within a specific task distribution. One could then train masks on top the untouched weights in order to perform well specific tasks, conceivably more efficient than with generic activation functions.
Another potentially interesting avenue for future work is to combine optimization of activation func- tions with synaptic plasticity functions [Soltoggio et al. (2018). A lot work has been done in the area of learning useful Hebbian-liks learning rules (Chalmers) 1991 Miconil
2016 Mouret &
Tonelli 2014 Floreano & Mattiussi. 2008
2008) Najarro & Risi 2020 Risi & Stanley) 2012; Soltoggio
al. 2008 Tonelli & Mouret 2013 Wang 2019). Less work in has explored the interaction
8

---

Under review as a conference paper at ICLR 2023
between learning rules and neural activation in ANNs, despite the fact that most learning rules take the neural activations as inputs. It seems likely that more expressive activation functions would in turn result in more expressive updates of weights via sedivtt-ddependent learning, and thus more powerful plastic neural networks.
Having shown that the proposed activation functions achieve well- pelf-orrnnnnng networks when optimized alone, a future experiments will explore the co-evolution of neural and synaptic param- eters. It will be interesting see whether synergistic effect arises between these two sets of pa- rameters. If both weights and activation functions are optimized together, will the there be as much diversity in the resulting set of activation functions, will the need for such diversity decrease?
Despite being initially inspired by biological neural networks, ANNs are still far from their bio- logical counter parts countless aspects. While the work presented here does claim to have presented biologically plausible approach, we do believe that inspiration from biological intelli- gence still offers great opportunities explore new variations of ANNs that can ultimately lead to interesting and useful results.
ACKNOWLEDGEMENTS
Removed for blind review.
REPRODUCIBILITY STATEMENT
The code reproduce every experiment in this paper is available [removed].
REFERENCES
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi. Learning activation func- tions to improve deep neural networks. arXiv preprint arXiv:1412.6688 2014.
Andrew Barto, Richard Sutton, and Charles Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions systems, man, and cybernetics, (5):834-846, 1983.
Lou Beaulicu-Laroche Enrique HS Toloza, Marie-Sophic Van der Goes, Mathicu Lafourcade, Der- rick Barnagian, Ziv M Williams, Emad Eskandar, Matthew Frosch, Sydney S Cash, and Mark Harnett. Enhanced dendritic compartmentalizzttoon in human cortical neurons. Cell, 175 (3):643-651, 2018
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1822 2013.
José Manuel Benítez, Juan Luis Castro, and Ignacio Requena. Are artificial neural networks black boxes? IEEE Transactions on neural networks, 1
156-1 164, 1997.
Garrett Bingham and Risto Miikkulainen. Discovering parametric activation functions. Neural Networks, 148:48-65 2022.
S Mare Breedlove and Neil V Watson. Biological psychology: An introduction to behavioral, cog- nitive, and clinical neuroscience. Sinauer Associates, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, arXiv preprint arXiv:1606.0 (606.0/540. 2016.
David J Chalmers. The evolution of learning: An experiment in genetic connectionism. In Connec- tionist Models, pp. 81-90. Elsevier, 1991.
Dario Floreano and Claudio Mattiussi. Bio- inspired artificial intelligence: theories, methods, and technologies. MIT press, 2008

---

Under review as a conference paper at ICLR 2023
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 2018.
Adam Gaier and David Ha. Weight agnostic neural networks. Advances in neural information processing systems, 32, 2019.
David Ha. Evolving stable strategies. blog.otoro.ue. 2017a. URL htlhtt:://bboog.oorro net/
David Ha. A visual guide evolution strategies. blog. otoro. net, 2017b.
Alexander Hagg, Maximilian Mensing, and Alexander Asteroth. Evolving parsimonious networks by mixing activation functions. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 425-432 2017.
Nikolaus Hansen. The ema evolution strategy: a comparing review. Towards new evolutionary computation, pp- 75-102, 2006.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspireed artificial intelligence. Neuron, 95(2):245-258, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1023 1034, 2015.
Kun He, Yan Wang, and John Hopcroft. A powerful generative model using random weights for the deep image representation. Advances in Neural Information Processing Systems, 29, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8);
1735-17800 1997.
Eugene Izhikevich. Simple model spiking neurons. IEEE Transactions on neural networks, 14(6):156--1555 157-6699-55,,0000000 2003.
Eugene M Izhikevich. Polychronization: computation with spikes. Neural computation, 18(2): 245-282, 2006.
Eugene M Izhikevich. Dynamical systems in neuroscience. MIT press, 2007.
Haruyuki Kamiya and Dominique Debanne. Axon neurobiology: Fine-scale dynamics of mi- crostructure and function. Frontiers in Cellular Neuroscience, 14:300, 2020.
Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, and Sarah Mack. Principles of neural science, volume 4. McGraw-hill New York, 2000.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.
Laura Lillien. Neural development: instructions for neural diversity. Current Biology, R13) R171, 1997.
Grace W Lindsay, Mattia Rigotti, Melissa R Warden, Earl K Miller, and Stefano Fusi. Hebbian learning in a random network captures selectivity properties of the prefrontal cortex. Journal of Neuroscience. 11245) 2017.
Jeffrey C Magee. Dendritic integration of excitatory synaptic input. Nature Reviews Neuroscience, 1(3):181-19,, 2000.
Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pp. 6682- 6691. PMLR, 2020.
Thomas Miconi. Learning to learn with backpropagation hebbian plasticity. arXiv preprint arXiv:1609.02225 2016.
10

---

Under review as a conference paper at ICLR 2023
Jean-Baphe Cean-Baptiste Mouret and Paul Tonelli. Artificial evolution of plastic neural networks: few key concepts. In Growing adaptive machines, pp. 251-261. Springer, 2014.
Elias Najarro and Sebastian Risi. Meta-learning through hebbian plasticity in random networks. Advances in Neural Information Processing Systems, 33, 2020.
Zoltan Nusser. Creating diverse synapses from the same molecules. Current opinion in neurobiol- ogy, $1:8-15,2018,
Evgenia Papavasileiou, Jan Cornelis, and Bart Jansen. A systematic literature review of the suc- cessors of "neuroevolution of augmenting topologies". Evolutionary Computation, 29(1):1-73, 2021.
Michael Pfeiffer and Thomas Pfeil. Deep learning with spiking neurons: opportunities and chal- lenges. Frontiers in neuroscience, pp. 774, 2018.
Panayiota Poirazi, Terrence Brannon, and Bartlett W Mel. Pyramidal neuron two-layer neural network. Neuron, 37(6))9899999)) 990999999999999 2003.
Sylvain Rama, Mickael Zbili, and Dominique Debanne. Signal propagation along the axon. Current opinion neurobiology, 51:37-44, 2018.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden in a randomly weighted neural network? In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp- 11893-11902, 2020.
Sebastian Risi and Kenneth O Stanley. A unified approach to evolving plasticity and neural geom- etry. In The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2012.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative reinforcement learning. arXiv preprint arXiv:1703.03866 2017.
Jürgen Schmidhuber. Deep learning neural networks: An overview. Neural networks, 61:85-117, 2015.
Moshe Sipper. Neural networks with à carte selection of activation functions. SN Computer Science, 2(6):1 9,011199,222 2021.
Ivan Soltesz et al. Diversity in the neuronal machine: order and variability in interneuronal micro- circuits. Oxford University Press, 2006.
Andrea Soltoggio, John Bullinaria, Claudio Mattiussi, Peter Dürr, and Dario Floreano. Evolution- ary advantages of neuromodulated plasticity in dynamic, reward-base scenarios. In Proceedings of the 11th international conference on artificial life (Alife XI), number CONF, pp. 569-576. MIT Press, 2008
Andrea Soltoggio, Kenneth O Stanley, and Sebastian Risi. Born to learn: the inspiration, progress, and future of evolved plastic artificial neural networks. Neural Networks, 108:48 67. 2018.
Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2):99-127, 2002.
Joan Stiles. Neural plasticity and cognitive development. Developmental neuropsychology, 18(2):
237-272, 2000.
Amirhossein Tavanaei, Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothée Masquelier, and Anthony Maida. Deep learning in spiking neural networks. Neural networks, 1 111:47-63, 2019.
Heather K Titley, Nicolas Brunel, and Christian Hansel. Toward neurocentric view of learning. Neuron, 95(1):19-33 2017.
Paul Tonelli and Jean- -an--attstte Mouret. On the relationships between generative encodings, regu- larity, and learning abilities when evolving plastic artificial neural networks. PLOS ONE, 8(11): 1-12, 11 2013. doi: htn.:///oomml..poo.009999991 URL / journal pone. 0079138
11

---

Under review as a conference paper at ICLR 2023
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Joseba Urzelai and Dario Floreano. Evolution of adaptive synapses: Robots with fast adaptive behavior in new environments. Evolutionary compulation, 9(4):495-524, 2001.
Lin Wang, Junteng Zheng. and Jeff Orchard. Evolving generalized modulatory learning: Unifying neuromodulation and synaptic plasticity. JEEE Transactions on Cognitive and Developmental Systems, 12(4):797-8) 808, 2019.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-15184, 2020.
A
APPENDIX
A.1
TRAINING CURVES
Swing-Up: Activation Functions
Swing-Up: Small Architecture
Swing-Up: Same Architecture
600
Waaler Activation Functions
Bipeda/Walker: Small Architecture
DipedalWa SeeddllWallie ker: Same Architecture
activstoo pos. mean
small net, top meon
some
PoD mean
-300
actiutton pas. bedt
activation pap. eval
email zep
email net. zop
-200
pio mvd.
1806
1000
2018
200
2007
2000
28
200
200
1005
1800
200
2000
Car Racing: Activation Functions
Car Racing: Small Architecture
HipedalWalkee ker: Same Architecture
activatio pas. mean
small net. sop. mean
SErTh netl: pop. meen
activation pos. zest
221
small net. cop. best
300
SarT rill pop- best
octivation poo. aval.
small not. sop. eval.
Sart ri6l. pop. eval
200
800
1000
1200
1480
200
400
680
680
3300
1000
140
201
1080
1240
1400
Figure 4: Training curves for all experimental settings. Means and standard deviations for popula- tions of runs on different seeds. Each setting was run three times except for the weight-optimizze models in the Car Racing envinronment, which were run twice. Every 50th generation, the current solution was evaluated 64 times (orange).
12


Reviews:
[{"review_id": "v50O7_C9lN", "reviewer": "Reviewer_5XEX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper augments standard neural networks with recurrent, parametrizable \"activation functions\". They show that randomly initialized networks with frozen weights can be trained with some degree of success by simply optimizing the parameters of these \"activation functions\". The results are competitive with normal, weight-optimized neural networks with similar number of parameters, and also in some cases with neural networks with an equivalent number of weights (and thus many more parameters), when trained with evolutionary algorithms.", "summary_of_the_review": "To my understanding, the  actual method is quite different from what is advertised. The  paper says that optimizing activation functions alone produces competitive results. The actual finding is that adding trainable *recurrence* to frozen-weight networks shows non-trivial performance on a specific type of tasks (dynamic control).  This might be of some interest, but even then, the experiments may not fully reflect the actual performance in comparison to trained-weight networks with modern RL algorithms.", "strength_and_weaknesses": "Strengths:\n\n- The paper proposes an idea which I think is novel, and the result was not obvious a priori. \n\nWeaknesses:\n\n1) My main concern is that the rhetoric does not match the actual content of the paper. The paper does not do what it says on the tin! We are told that networks are to be trained \"only through the optimization of neural activation functions\". But the  proposed method  does much more than just \"optimizing activation functions\": it adds a recurrent neural state to the neurons! This is quite different from what would typically be understood  as \"optimizing the activation function\" alone, and  may be the main source of performance.\n\nThe impact of adding recurrence is particularly acute because all the experiments involve dynamic control  tasks, in which  introducing recurrence to otherwise purely feedforward network could  certainly provide an advantage. It is unlikely that the method would provide similar performance on tasks that don't have a time dimension, such as image classification!\n\n2) Another concern is that of fair comparison. All networks  were trained with evolutionary algorithms (CMA-ES for the small ones, and plain ES for the large weight-optimized one). The authors claim that being able to use CMA-ES (due to the small number of trained parameters in comparison to the size of the architecture) is an advantage of the method.\n\nBut in reality, dynamic control tasks are more likely to be solved  with offline RL algorithms (like PPO or  SAC) rather than evolution.\n\nI note that, because the proposed method introduces recurrence, it would be difficult to train with offline RL methods based on replay buffers and would  likely require policy-gradient methods like A3C, which are more stable but less sample-efficient. Thus, while the method does allow certain algorithms to be used, it also prevents the use of other algorithms, which are much more popular in the literature and more sample efficient by design. \n\nHow do the various architectures perform when trained with their best possible  algorithms? What would be the outcome of training  the weight-optimized networks with, say, SAC (or some other modern RL algorithm) rather than evolution?  Is it possible that the apparent competitive results obtained for the method are an effect of unduly restricting weight-optimized network to less-efficient evolutionary algorithms?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ulIDXkdNlQU", "reviewer": "Reviewer_3uev", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a type of neural network architecture that augments individual units in the network with an additional state variable and reparameterizes them using a 3x3 weight matrix. Each individual unit receives input that is the output of a linear transformation (as in standard networks), but instead of passing that output through a pointwise nonlinear activation function the inputs are transformed through a small state update equation. The network compares the performance of these networks against feedforward networks on three RL tasks.", "summary_of_the_review": "Overall, I had issues with the motivation and implementation of the main idea. I think changing the framing to focus on the interesting bit of the architecture (adding recurrent memory to individual units), and further clarifying the neuroscientific connections (are there examples in neuroscience of mechanisms within neurons that can implement recurrent state, and how those might lead to computational benefits?) are needed.", "strength_and_weaknesses": "Strengths: The paper is generally well written, and describes it main idea clearly along with the corresponding experiments.\n\nWeaknesses: I have concerns with the motivation and architectural designs for how this paper implements its main idea, as well as with the overall impact of the results.\n\n- Motivation: The paper introduces the idea of diversity in neuroscience (specifically diverse cell types), and uses this to motivate the need for diverse activation functions in artificial networks. I find this connection pretty weak and uninspiring. I want to know _why_ we see cell type diversity in biology, specifically what is the computational benefit from having different cell types (this has been studied and discussed for many different brain regions in the neuroscience literature). Unsurprisingly, the _reasons_ for cell type diversity are as diverse as the cell types themselves. Furthermore, the mechanistic implications of cell type diversity extend beyond just activation functions (this too has been discussed heavily in the neuroscience literature). Overall, I found the motivations this paper takes from neuroscience to be weak and too vague to be useful.\n\n- Architectural designs: The proposed architecture, to me, looks very different from an \"activation function\". When I hear that term, I think of a point-wise (1D) nonlinearity in a neural network. Here, the paper defines an activation function as a small recurrent unit! Equation (1) in the paper looks just like a small vanilla RNN. That's fine (exploring what happens when you augment neurons with recurrent state is interesting), but this implementation is too different (in my mind) from the focus on \"activation functions\" in the title and abstract.\n\n- Memory: One big difference between the introduced architecture and the feedforward baselines is that the new architecture has memory. This is discussed briefly, but to me an important control is what happens if you remove the memory mechanism from the network? That is, are the observed improvements just due to having memory (state)? I suspect so, and if so, the title/abstract/introduction should be rewritten to emphasize this.\n\nSuggestions for improvement:\n- Better clarify the network architecture. How is it different from an RNN? What is the role of memory?\n- Are the performance differences across networks due to a failure to optimize, or the fact that the networks have different computational capacities? I.e. is it possible to find a setting of the parameters that can solve the task, but it is difficult to do so with the given optimization method?\n- More thorough hyperparameter comparisons are necessary to understand the limitations of the method.\n- Can we develop some deeper understanding of what these networks learn, and how that differs from both standard feedforward networks and standard recurrent networks? This is crucial to know how interesting these networks are.\n- Better clarify the connection between the motivation (diversity in biological networks) and these architectural choices. At the moment, reads like “biology is complicated, so let’s try complicated networks”—which is fairly shallow.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "5Xvb-th4o8", "reviewer": "Reviewer_ah36", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach to the training of neural networks where, instead of adjusting weights between neurons, they adjust nonlinearities on the output of each neuron. The nonlinearities are applied via a product between a learnable matrix and a vector consisting of the neuronal input, an updateable memory term, and a bias term – all followed by a fixed nonlinear function. The authors test their model on three standard RL tasks and find that the model’s performance is similar to that of the conventional neural networks with a matching number of layers/parameters.", "summary_of_the_review": "Although the paper pursues an important goal of introducing a new neural architecture for deep learning, additional experiments are needed to show what are the advantages of using this architecture. Until such experiments are performed, the results deem preliminary for ICLR, yet may be sufficient to be reported elsewhere, e.g. as a preprint.", "strength_and_weaknesses": "The goal of designing a new neural network architecture is worthy and timely, as it extends a growing body of work on network architectures including the Lottery Ticket Hypothesis and the Weight-Agnostic Networks, known to NeurIPS/ICML communities and thoroughly referenced in the text. The model has been tested on a standard set of reinforcement-learning benchmarks, allowing us to compare the results to those from other models. The new approach is complementary to the existing models and can be easily combined with them.\n\nIf I’m not missing anything, the message of the paper is that the proposed architecture, while being different from the existing ones, shows similar performance. While the result is interesting in its own right, more analysis is typically expected in ICLR submissions to assess the scope and the usefulness of the proposed model. Specifically, what are the possible use cases where applying such a model would be advantageous? If the main benefit of this model is in the reduced number of parameters, then a comparison with up-to-date compression approaches is needed. Should the main benefit of the model be in the improved performance, a comparison with the top-scoring models would be necessary for the considered RL tasks. For now, the best-reported performance of the models does not reach the current standards*, and in two out of three tests the best performance was not achieved with the proposed model.\n\n* huggingface.co/spaces/huggingface-projects/Deep-Reinforcement-Learning-Leaderboard\n\nI understand that the authors aimed to compare the models in a simple setting and did not have a goal of exceeding the current best results in selected RL tasks, however, the possible advantages of the proposed model appear most important for large high-performance networks and not for small networks that don’t need to be compressed and whose performance may be improved by the conventional means. It is not obvious if the reported results generalize to more complex high-performance networks.\n\nMinor comments:\n\n-As the proposed model uses memory and matrix multiplication to define the activation function, it is important to show that both components are needed for the reported performance.\n\n-It would be also great to have a description of how the proposed approach is different from (a subclass of) RNNs which also have memory and use matrix products before the nonlinearity.\n\n-For the Weight-Agnostic Networks, the number of parameters can be estimated as the number of coordinates of nonzero connections, i.e. the size of the netlist representation of the graph.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-ZSerhvshfM", "reviewer": "Reviewer_xrLZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by classes of biological neurons, the authors explore if RNNs can be trained to perform simple RL tasks via neuron activation function optimization. \n", "summary_of_the_review": "While an interesting research direction, I do not think that an activation function is being learned here and the biological connections are undeveloped. Results are limited and underwhelming. \n", "strength_and_weaknesses": "Strengths\n- Learning via activation functions is an interesting and important direction of research, as is the connection to biological neuron response functions.\n- Paper is generally well written. \n\nWeaknesses\n- I disagree that the model presented here is best described as “learning activation functions”, as a parametric activation function is not being learned. Instead the model is in the space of models that could be realized by a standard RNN.\n- No biological justification is given for the model’s form. I think the biological motivations and relevance of this paper are overstated. How neural firing types or classes are related to what is learned by the network is unclear and not addressed. \n- No justification is given for why RL is an appropriate task that this model is either specifically suited for, or to test properties of the model.\n- Related work section is not as relevant as Sipper, Agostinelli and Bingham papers referenced in the introduction.  \n- Why are weights sampled a gaussian with 0.5 std?\n- Synergies with standard weight optimization should be demonstrated, currently no performance benefit is presented. Results are underwhelming.\n- Methods & implementation details unclear and no link to code. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject