Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operators
Authors: Scott C Lowe
Robert Earle
Jason d'Eon
Thomas Trappenberg
Sageev Oore
Venue: NeurIPS 2022 Accept (2022)
Keywords: activation
activation functions
logit
probabilistic
Bayesian
Boolean
logic
dendritic computation

Abstract: The choice of activation functions and their motivation is a long-standing issue within the neural network community. Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds score of presence of features within the stimulus. We derive logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR, and XNOR for independent probabilities. Such theories are important to formalize more complex dendritic operations in real neurons, and these operations can be used as activation functions within a neural network, introducing probabilistic Boolean-logic as the core operation of the neural network. Since these functions involve taking multiple exponents and logarithms, they are computationally expensive and not well suited to be directly used within neural networks. Consequently, we construct efficient approximations named $\text{AND}_\text{AIL}$ (the AND operator Approximate for Independent Logits), $\text{OR}_\text{AIL}$, and $\text{XNOR}_\text{AIL}$, which utilize only comparison and addition operations, have well-behaved gradients, and can be deployed as activation functions in neural networks. Like MaxOut, $\text{AND}_\text{AIL}$ and $\text{OR}_\text{AIL}$ are generalizations of ReLU to two-dimensions. While our primary aim is to formalize dendritic computations within a logit-space probabilistic-Boolean framework, we deploy these new activation functions, both in isolation and in conjunction to demonstrate their effectiveness on a variety of tasks including tabular classification, image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.

Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operators
Scott Lowel 22., Robert Earle! .Jsoon d'Eon
Thomas Trappenberg'
Sageev Oore!.2
Faculty of Computer Science
Vector Institute for Artificial Intelligence Toronto, Ontario Canada
Dalhousie University Halifax, Nova Scotia Canada
"Correspondence: scottclowe www.lll.oommmmoommmmmmmmm
Abstract
The choice of activation functions and their motivation is long-standing issue within the neural network community. Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds score of presence features within the stimulus. We derive logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR, and XNOR for inde- pendent probabilities. Such theories are important to formalize more complex dendritic operations in real neurons, and these operations can be used as activation functions within a neural network, introducing probabilistic Boolean-logic as the operation of the neural network. Since these functions involve taking multiple exponents and logarithms, they are computationally expensive and not well suited to be directly used within neural networks. Consequently, construct efficient approximations named AND (the AND operator Approximate for Independent Logits), OR and XNORALL which utilize only comparison and addition opera- tions, have well-behaved gradients, and can be deployed as activation functions neural networks. Like MaxOut, AND AII. and OR are generalizations of ReLU to two-dimensions. While our primary aim to formalize dendritic computa- tions within logit-space mrobabilistic-Booleee framework, deploy these activation functions, both in isolation and in conjunction to demonstrate their effec- tiveness on a variety tasks including tabular classification, image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.
Introduction
Non-linear activation functions are essential in artificial neural networks (ANNs) to form higher-order representations, since otherwise the network would be degeneratively equivalent to single linear layer. Most activation functions represent simple non-linearity despite evidence of much more complex non-linear integration and computations in dendrites (Hentschel et al., 2004; London & Häusser, 2005; Payeur et al., 2019). For example, Gidon et al. (2020) recently demonstrated that a single biological neuron compute the XOR of its inputs, property long known to be lacking from artificial neurons (Minsky & Papert, 1969). While simple activation functions work in ANNs in the that more complex operations of neurons, understanding the function and impact of advanced dendritic operations networks is important. In this work, we add some of this behaviour to neural activations, corresponding to shifting
be formed from the combination of several layers
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

some of the network's 's complexity from its global structure to the neural level. There is far more complexity in biological neurons than in the abstractions that we consider here, but we make a step in the direction of using more complex neurons in ANNs. We also develop a theoretical underpinning for higher-order activation functions (e.g. like MaxOut, Goodfellow et al., 2013) in probabilistic
framework. We hypothesize that such architectures will be more parameter-effciien in situations where their assumptions hold.
Neuronal representations within ANNS are commonly understood as logits, representing the log-odds score of presence (versus absence) of features within the stimulus. From Bayesian perspective, a ReLU-like operation corresponds to the removal of all evidence for the lack of feature. Under the logit interpretation of ANN potentiations, this seems unreasonable. This can be seen in some of the ways we interact with neural networks: we must apply batch-norm before activations and not after them; when doing transfer learning from an embedding space we must use pre-activation potentiations instead of activations. Can ANNs do better if we design an architecture which treats potentiations as logits?
We thus derive logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR, and XNOR for independent probabilities. Networks constructed in this way can be interpreted as performing logical operations using point-estimates of probabilities, in a similar manner to a Bayesian network. This brings operations from the symbolism framework of AI (which is more similar to deliberative thinking, or System 2 of Kahneman, 201 1111 into the connectionist framework of ANNs (which is more like instinctive, System 1, thinking). We also construct computationally feasible approximations to these functions with well-hehaved gradients. These new activation functions, which are generalizations of ReLU to two-dimensions, are then applied on benchmark datasets to demonstrate their effectiveness on a diverse range of tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning. The new principled approach we present introduces new ways to redistribute computation from the network into the neuronal mechanisms, and build more parameter-efficient models. We demonstrate the effectiveness of activation functions based on these ideas, and expect future work to build on this.
2 Background
Early artificial neural networks featured either logistic-sigmoid or tanh as their activation function, motivated by the idea that each layer of the network is building another layer of abstraction of the stimulus space from the last layer. Each neuron in a layer identifies whether certain properties or features are present within the stimulus, and the pre-activation (potentiation) value of the neuron indicates a score or logit for the presence of that feature. The sigmoid function, a(x) = 1/(++= was hence a natural choice of activation function, since as with logistic regression, this will convert the logits of features into probabilities. There is evidence that this interpretation has merit. Previous work has been done to identify which features neurons are tuned to. Examples include LSTM neurons tracking quotation marks, line length, and brackets (Karpathy et al., 2015) and sentiment (Radford et al., 2017); projecting features back to the input space to view them (Olah et al., 2017); and interpretable combinations of neural activities (Olah et al., 2020).
Sigmoidal activation functions no longer commonly used within machine learning between layers of representations, though sigmoid is still widely used for gating operations which scale the magnitude of other features in an attention-like manner. The primary disadvantage of the sigmoid activation function is its vanishing gradient - as the potentiation rises, activity converges to platcau, and hence the gradient goes to zero. This prevents feedback information propagating back through the network from the loss function to the early layers of the network, which consequently prevents it from learning to complete the task.
The Rectified Linear Unit activation function (Fukushima, 1980; Jarrett et al., 2009; Nair & Hinton, 2010), ReLU(xx= = max(0, .r), does not have this problem, since in its non-zero regime it has gradient of 1. Another advantage of ReLU is it has very low computational demands. Since it both effective and efficient, it has proven to be a highly popular choice of activation function. The chief drawback to ReLU is that it has no sensitivity to changes across half of its input domain, which prevents updates on stimuli which trigger its "off" state and can even lead to neuronal death'. Variants of ReLU have emerged, aiming to smooth out its transition between domains and provide
'Toouh this problem is very when using BatchNorm to stabilize feature distributions.

---

gradient in its inactive regime. These include ELU (Clevert et al., 2016), CELU (Barron, 2017), SELU (Klambauer et al., 2017), GELU (Hendrycks & Gimpel. 2020), SiLU (Elfwing al., 2018; Ramachandran et al., 2017); and Mish (Misra, 2019). However, all these activation functions still bear the general shape of ReLUU and truncate negative logits.
Fuzzy logic operators are generalizations of Boolean logic operations to continuous variables, using rules similar to applying logical operators in probability space. Prior work has explored networks of fuzzy logic operations, including some which use an activation function constituting a learnable interpolation between fuzzy logic operators (Godfrey & Gashler, 2017). The activation functions we introduce here are motivated similarly to fuzzy logic operators, but designed to operate in logit space instead of in probability space, which better reflects the behaviour and space of pre-activation units.
Bayesian neural networks (BNNs) are probabilistic models which can represent the uncertainty in their model parameters, and hence uncertainty in their outputs. This differs from standard ANNs which use only a single point-estimate of each model parameter, and hence also in their neural activations. By making their priors explicit and modelling their uncertainty, Bayesian networks can be better calibrated and less vulnerable to overfitting than ANNs. However, BNNs are more challenging to train, and cannot reasonably be scaled up to the deep architectures which are possible with standard ANNs and necessary in order to learn a sufficiently complex model to solve highly complex tasks.
In this work we contribute to the theoretical underpinning of neural activation functions by developing activation functions based on the principle that neurons encode logits - scores that represent the presence of features in the log-odds space. In $3 we derive and define these functions in detail for different logical operators, and then consider their performance on numerous task types including parity ($ 4.,,, image classification (S 4.3 and & 4.4), transfer learning ($ (.5,,, abstract reasoning (Appendix $A.18), soft-rule guided classification as exemplified by the Bach chorale dataset ($ ($4.2), and compositional zero-shot learning (Appendix $A.19). These tasks were selected to (1) survey the performance of the new activations on existing benchmark tasks, and (2) evaluate their performance on tasks which we suspect in particular may require logical reasoning and hence benefit from activation functions which apply these logical operations to logits.
3 Derivation
Manipulation of probabilities in logit-space is known to be more efficient for many calculations. For instance, the log-odds form of Bayes' Rule (Appendix Eq. 9) states that the posterior logit equals the prior logit plus the log of the likelihood ratio for the new evidence (the log of the Bayes factor). Thus, working in logit-space allows us to perform Bayesian updates on many sources of evidence simultaneously. merely by summing together the log-likelihood ratios for the evidence (see Appendix A.3).. A weighted sum may be used if the amount of credence given to the sources differs and this is precisely the operation performed by a linear layer in a neural network.
When considering sets of probabilities, a natural operation is to measure the joint probability of two events both occurring the AND operation. Suppose our input space is x E [0, 1)2, and the goal is V i, and y < 0 otherwise, using model with weight vector and bias term b, such that // = T r + b. This can be trivially solved with the weight matrix w = [1, 1] and bias term b - 1.5. However, since this is only linear separator, the solution can not generalize
to output y > 0 if 2;
the case > 0 iff x; > 0.. Similarly, let us consider how the OR function is solved with a linear layer. and y < 0 otherwise. The binary case be trivially solved with the weight matrix w = [1, 1] and bias term = - 01.5. The difference between the solution for OR and the solution for AND only an offset our bias term. In each case, if the input space is expanded beyond binary to R²², the output can be violated by changing only one of the arguments.
Our goal is output y > 0 if 3x;
3.1 AND
Suppose we are given x and as the logits for the presence (vs absence) of two events, X and Y. These logits have equivalent probability values, which can be obtained using the sigmoid function, (u)) = (1 + e -- Let us assume that the events X and Y are independent of each other. In this case, the probability of both events occurring (the joint probability) is P(X,Y) Y) = P(X A Y)= = P(X) P(Y)- oxx) (z). However, we wish to remain in logit-space, and must determine the logit
3

---

of the joint probability, logit(P(XX,)) Y). This is given by
ANDn. (P 2)21)) 1g) log
p where p p=
1 T
o (z)) oly)
log
(z) o
which we coin ANDH. the AND operator for independent logits (L.). This 2d function is illustrated as contour plot (Fig. 1, left panel). Across the plane, the order of magnitude of the output is the same as least one of the two inputs, scaling approximately linearly.
The approximately linear behaviour of the function is suitable for use as an activation function (no vanishing gradient), however taking exponents and logs scales poorly from computational perspective. Hence, we developed a computationally efficient approximation as follows. Observe that we can loosely approximate ANDIL with the minimum function (Fig. 1, right panel). This is equivalent to assuming the probability of both X and Y being true equals the probability of the least likely of X and Y being true na've approximation which holds well in three quadrants of the plane, but overestimates the probability when both X and Y are unlikely. In this quadrant, when both X and Y are both unlikely, a better approximation for AND the sum of their logits.
We thus propose ANDALL a linear-approximate AND function for independent logits (AIL, i.e. approximate IL).
x + y: min a y),
x < 0, y <
AI
otherwise
As shown in Fig. 1 (left, middle), we observe that their output values and shape are very similar.
ANDa
ANDAIL
min
4
+
+6
2
+2
0
+0
-2
6
B
2
2
4
2
2
4
2
4
Figure I: Heatmap comparing the outputs for the exact logit-space probabilistic-and function for independent logits, ANDn y): our constructed approximation, AND (z, y); and max(æ, y).
3.2 OR
Similarly, can construct the logit-space OR function, for independent logits. For a pair of logits and y. the probability that either of the corresponding events is true is given by p = 1 1 1 This can be converted into logit
p __.) with p = 1= - (-p)
ORn 3) (P V Y).1y) log
(3)
which can roughly approximated by the max function. This equivalent to setting the probability of either of event X or Y occurring to be equal to the probability of the most likely event. This underestimates the upper-right quadrant (below), which we can approximate better as the sum the input logits, yielding
x + V.
x> .>0,
y otherwise
ORAH( =
max(2; (z, v),

---

ORI
OR
max
+10
4
+8
+6
2
0
+
2
-2
6
-4
8
10
-4
-
2
4
4
-2
2
4
4
-2
2
4
Figure 2: Comparison of the exact logit-space probabilistic-or function for independent logits,
(R,(x, y); our constructed approximation, ORAIL (x. y); and max( y)-
3.3 XNOR
We also consider the construction of logit-space XNOR operator. This is the probability that X and Y occur either together, or not at all, given by
XNORI logit(((X#)) log 1--p)
(5)
where - (ly) + ((-x)o((-)) We can approximate this with XNORAI(I.g)) 9) == sgn(zy) min(|2], (v1),
(6)
which focuses on the logit of the feature most likely to flip the expected parity (see Fig. 3). We could use other approximations, such as the sign-preserving geometric mean,
SionedGemmean(1 [r, y) := sgn(xy) [ry],
(7)
but this matches XNORH. less closely, and has divergent gradient along both = 0 and
0.
XNORa
XNORAu
SignedGeomean
+10
4
+6
2
+4
0
+0
2
-2
-4
8
10
2
2
-2
Figure 3: Comparison of the exact logit-space probabilistic-xnor function for independent logits, XNORn y): our constructed approximation, XNORAn y); and SignedGeomean((z, y).
3.4 Discussion
By working via probabilities, and assuming inputs encode independent events, we have derived logit-space equivalents of the Boolean logic functions, AND, OR, and XNOR. Since these are computationally demanding to compute repeatedly within a neural network, we have constructed approximations of them: ANDAIL. ORAIL. and XNORALL Like ReLU, these involve only com- parison, addition, and multiplication operations which are cheap to perform. In fact, AND AII. and ORAII. generalization of ReLU to extra dimension, since ORAn. y 0) max((, 0).
The majority activation functions are one-dimensional, f - In contrast to this, our proposed activation functions all two-dimensional, I I.. They must be applied to pairs of features
5

---

from the embedding space, and will reduce the dimensionality of the input space by factor of 2. This behaviour is the same as seen in MaxOut networks (Goodfellow et al., 2013) which use max as their activation function; MaxOut(z,, y; k) == max(x. (z. y). Similar to MaxOut, our activation functions could be generalized to higher dimensional inputs, f IR + R. by considering the behaviour of the logit-space AND, OR, XNOR operations with regard to more inputs. For simplicity, we restrict this work to consider only k = 2, but note these activation functions also generalize to higher dimensions.
3.5 Ensembling
By using multiple logit-Boolean activation functions simultaneously alongside each other, permit the network multiple options of how to relate features together. When combining the activation functions, we considered two strategies (illustrated in Appendix $ $A.6).
In the partition (p) strategy, we split the nhe dimensional pre-activation embedding equally into m partitions, apply different activation functions on each partition, and concatenate the results together. Using AIL activation functions under this strategy, the output dimension will always be half that of the input, as it is for each AIL activation function individually. In the duplication (d) strategy, apply m different activation functions in parallel to the same n. elements. The output is consequently larger, with dimension nec If desired, we can counteract the 2 + +1 reduction of AlL activation functions by using two of them together under this strategy. A negative weight in a network is equivalent to the logit-NOT operator. Hence with sufficient width and depth, multi-layer network using only the OR activation function should be able to replicate any probabilistic logic circuit.
Utilizing AND AIL., ORAHI. and XNORAIL simultaneously allows our networks to access logit-space equivalents of of the 16 Boolean logical operations with only a single sign inversion (in either one of the inputs or the output). Including the bias term and skip connections, the network has casy access to logit-space equivalents of all 16 Boolean logical operations.
We hypothesised that training a network with all three of our activation functions in an ensemble in this manner could yield better results since the network would not need to expend layers having to combine ORıL operations together to yield other Boolean operations.
3.6 Normalization
Our AIL activation functions close approximations exact AND, OR, and XNOR operations in logit-space. However, when deploying non-linearities within neural network, it is important that the activation functions have a gain of in order to improve stability during training (Klambauer et al., 2017), a property the AlL activations do not possess. We constructed normalized variants of the exact and approximate logit operators, dubbed NIL and NAIL respectively, by subtracting the expected mean and dividing by the expected standard deviation, assuming the operands are sampled independently from the standard normal distribution, (0, 1). For more details, see Appendix A.77.
4 Experiments
We evaluated the performance of our AlL activations, both individually and together in an ensemble, on a range of benchmark tasks. Since AND and ORAI. are equivalent when the sign of operands and outputs can be freely chosen, we only show results for ORAIL- We compared against three primary basclines: (1) ReLU, (2) max(x,) y) = MaxOut([a [zz y]; k 2), and (3) the concatenation of max(x, y) and min(x, y). denoted { [axx, Min (d)). The [Max, Min (d)) ensemble is equivalent to GroupSort with a group size of (Chernodub & Nowicki, 2017; Anil et al., 2019), sometimes referred to as the MaxMin operator; is comparable to the concatenation of OR and AND under our duplication strategy.
Source code for our experiments can found at nttps://githhe httttthub..ooDDaaaaa coom/Dalhousiell houshouuueee. Our python package which provides an implementa-
logi cal cal_aalaaa i_ccti vat in conn_exeermeette tion of these the activation functions is available at httpp:://itthhi thub. com/Da hous e pytorch-logit -ogit-oogic.. which is also available PyPl package pytorch- cewoorccc--oiittt Logit-logic.

---

4.1 Parity
In a simple initial experiment, we constructed a synthetic dataset whose labels could be derived directly by stacking the logical operation XNOR. Each sample had four input logits, and target value equal to the parity of the number of positive inputs. A very small model using XNOR with two hidden layers (of 4, then 2 neurons) should be capable of perfect classification accuracy on this dataset with a sparse weight matrix by learning to nest pairwise binary relationships. We trained such an MLP with either ReLU or XNORAIL activations.
The MLP with XNORAIL activation learned a sparse weight matrix able to perfectly classify any input combination, shown in Fig. 4. In comparison, with ReLU the network was only able to produce 60% classification accuracy. The accuracy with ReLU was improved by increasing the MLP width/depth, but this did not result in sparse weight matrix. This exper- iment demonstrates that XNORAIL can be utilized by a net- work to find the simplest relationship between inputs that satis- fies the objective. For additional results, see Appendix $A.
Outputs
Layer 2 Post-act
Layer 2
Pre-act
4.2 MLP Bach Chorales and Logit Independence
Layer
Post-act
The Bach Chorale dataset (Boulanger-Lecaaddowss)) et al., 2012) consists of 382 chorales composed by JS Bach, each ~ measures long, totalling ~83 000 notes. Represented as discrete sequences of tokens, has served as a benchmark for music processing for decades, from heuristic methods to HMMs, RNNs, and CNNs (Mozer, 1990; Hild et al., 1991; Allan & Williams, 2005; Liang, 2016; Hadjeres et al., 2017; Huang et al., 2019). The chorales are comprised of 4 voices (melodic lines) whose behaviour is guided by soft musical rules depending on the prior movement of that and other voices, e.g. "two voices fifth apart ought not to move in parallel with one another". We tasked 2-layer MLPs with determining whether a short four-part musical excerpt is taken from Bach chorale. Negative examples were created by stochastically corrupting chorale excerpts (see Appendix 13). We found (OR, AND, XNORAIL (d)) had highest accuracy, but the re- sults were not statistically significant (p 0.1 between best and worst, two-tailed Student's 1-test, 10 random inits; see Appendix S 10).
Layer 1 Pre-act
Inputs
ReLU
XNOR A.
Figure 4: Visualization of weight matrices learnt by two-layer MLPs on a binary classification task, where the target output the parity of the inputs. Line widths indicate weight magnitudes (orange: +ve, blue: -ve). MLP with ReLU: 60% accuracy;
XNORATL: 100% accuracy.
Additionally, we investigated the independence between logits in the trained pre-activation embed- dings. We expect that an MLP which is optimally engaging its neurons would maintain independence between features in order to maximize information. We measured correlations between adjacent pre-activations (paired operands for the logical activation functions), and also between non-adjacos -djjceee pairs of pre- mreaacivations. Our results indicate the network learns features which independent when they are not passed to the same 2D activation, and anti mnti-correlated features when they are. For more details, Appendix $ 14.14.
4.3 CNN and MLP on MNIST
We trained 2-layer MLP and 6-layer CNN models on MNIST with ADAM (Kingma & Ba, 2015), I-cycle schedule (Smith & Topin, 2017; Smith, 2018), and using hyperparameters tuned through a random search against a validation set comprised of the last 10k images of the training partition.
The MLP used two hidden layers, the widths of which were varied together to evaluate the perfor- mance for range of model sizes. The CNN used six layers of 3x3 convolution layers, with 2x2 max pooling (stride 2) after every other conv layer, followed by flattening and three MLP layers. The layer widths were scaled up to explore a range of model sizes (see Appendix $ A.15 for more details).
For the MIP, XNORAI. performed best along with SignedGeoment t-test), ahead of all other activations (p < 0.01 for each; Fig. 5 left panel) when considering the best
1p<0.. 0.1, two-tailed Student's
7

---

MNIST, MLP (2 hidden layers)
MNIST, CNN (6 layer)
98.8
99.6
Test accuracy (%)
Test accuracy (%)
98.6
98.4
ReLU
99.4
SILU
SigmedGeomean
99.3
98.2
XNORAL
ORwL
OR, XNORa (d)
99.2
98.0
OR, AND, XNORai (d)
10a
10c
107
104
10a
10c
107
Number of parameters
Number of parameters
Figure 5: We trained CNN on MNIST, MLP on Mattened-MNISS using ADAM (1 (1-cycle, 10 ep). hyperparams determined by random search. Mean (bars: std dev) of n weight inits.
performance across all widths (see Appendix $ AA.10 for methodology). However when the width reduced below 2 10a there is transition and XNOR-shaped activations perform worst. We hypothesize this may be because smaller widths embeddings are over-saturated and have individual units corresponding multiple features, whilst XNOR activations may require single-feature units to perform best.
With the CNN, five activation configurations (IOR, AND, XNORAIL (p)). [OR, XNORAIL (d/p)), Max, and SiLU) performed best (p < 0.05 for other activations, two-tailed Student's 's t-test; Fig. right panel; Appendix $ 10). CNNs which used ORAIL Max (alone or in an ensemble) maintained high performance with an order of magnitude fewer parameters (3 10 than others 10 params).
4.4 ResNet50 on CIFAR-10/100
We explored the impact of our activation functions on deep networks by deploying them in a pre- activation ResNet50 model (He et al., 6a, We exchanged all ReLU activations in the network to a candidate activation while maintaining the size of the pass-through embedding. We experimented with changing the width the network, scaling up the embedding space and all hidden layers by common factor, The network was trained on CIFAR-10/-10 for 100 epochs using ADAM (Kingma Ba, 2015), 1-cycle (Smith, 2018; Smith Topin, 2017). See Appendix AA.1 for further details.
For both CIFAR-10 and -100, SiLU, ORAIL. and Max outperform ReLU across a wide range of width values (see Fig. 6). These three activation functions hold up their performance best as the number of parameters reduced. Since SiLU was discovered by a search of 1 activation functions for this type of architecture and task (Ramachandran et al., 2017), we expected it to perform well in this setting. Yet, we find ORAIL and Max perform better than ReLU and comparable with SiLU generally. Meanwhile, other AIL activations perform similarly to ReLU when the width is thin, and slightly worse than ReLU when the width is wide. When used on its own and not part of an ensemble, the XNORAI, activation function performed poorly (off the bottom of the chart), indicating it is not suited for this task.
4.5 Transfer learning
considered transfer learning on several image classification datasets. We used ResNetl model (He et al., 2016a) pretrained on ImageNet-Ik, The weights were frozen (not fine-tuned) and used to generate embeddings of samples from other image datasets. We trained a two-layer MLP classify images from these embeddings using various activation functions. For compre-

---

CIFAR10, 0, ResNet50
CIFAR100, ResNet50
96.0
80
95.5
78
95.0
Test accuracy (%)
Test accuracy (%)
94.0
74
PRoLU
PReLU
93.0
SILU
SILU
Max
Max
92.5
ORae
ORu
OR XNORw (d)
OR, XNORA (d)
OR AND, XNOR (d)
OR AND, XNORA (d)
91.5
68
10"
10a
102
10a
Number of parameters
Number of parameters
Figure 6: ResNet50 on CIFAR-10/100, varying the activation function used through the network. The width was varied to explore range of network sizes (see text). Trained for 100 ep. with ADAM, using hyperparams as determined by random search on CIFAR-100 with width factor 2. Mean (bars: std dev) of n n= weight inits.
hensive set of baselines, we compared against every activation function built into PyTorch 1.10 (see Appendix A.177.. To make the number of parameters similar, we used width of 512 for activation functions with 1- mapping (e.g. ReLU), width 650 for activation functions with 2 1 mapping (e.g. Max, ORAn.), and a width of 438 for {OR, AND, XNORAI. (d)). See Appendix 17.17 for more details.
Our results are shown in Table 1. We found that all our (N)AIL activation functions outperformed ReLU on every transfer task. Normalization had little impact the performance of OR,IL. but large impact on X.OOR.IL-- The best overall performance was attained by the exact XNORNIL- closely followed by our approximate X.NRRNAIL and ensembles containing either of these. Surprisingly, our approximation ORL outperformed the exact form OR.IL- The proposed activation functions were beaten only by PReLU, on Stanford Cars. On Caltech101, all MLPs were beaten by linear layer, suggesting our MLPs were overfitting for that dataset. For further discussion, see Appendix $ $A.17.
4.6 Additional results
For results on tabular data, abstract reasoning and compositional zero-shot learning tasks, please see Appendix 12, Appendix $ $A.18 and Appendix $A.19, respectively.
Conclusion
In this work, motivated and introduced novel activation functions analogous to Boolean operators in logit-space. We designed the AIL functions, fast approximates to the true logit-space probabilistic Boolean operations, and demonstrated their effectiveness wide range of tasks.
Although our activation functions assume independence (which is generally approximately true for pre-activation features Icarnt with ID activations), we found the network learnt to induce anti- correlations between features which were paired together by our activations (Appendix 1A.14). This suggests exact independence of the features is not essential to the performance of our proposed activations.
We found XNOR.ALL was highly effective for shallow MLPs. Meanwhile, ORAIL was highly effective for representation learning scenarios which involve manipulating high-level features extracted by embedding network, we find that using XNOR.1I, or an ensemble of AlL activation functions together works best, and that the duplication ensembling strategy outperforms partitioning. In this work restricted ourselves to
setting of deep ResNet architecture trained on images. In

---

Table Transfer learning from a frozen ResNet- 18 architecture pretrained on ImageNet-Ik to other computer vision datasets. Mean (std. eтror) of n = 5 random initializations of the MLP (same pretrained encoder). Bold: best. Underlined: top two. Italic: no sig. diff. from best (two-sided Student's 1-test, p 0.05). Background: linear color scale from ReLU baseline (white) to best (black).
Test Accuracy (%)
Activation function
Call01
CIFARIO
CIFARIOO
Flowers
SuCCrr
STL-10
SVHN
Lincar layer only
88.35
78.56+0.00
+7.39+0.09
92. 32.3+=.22 20
33.51
91.08-00.02
46.60+0.14
ReLU
20.5
81, 13.330..s
58.0420.11
20.71
30.97
94.6220.06
53.2620.06
PR
87.8310 21
81 11.0320-1
58.9010 18
98.7710.1 1o
39.84.0 18
94.54g0-05
53,47.10 os
SELU GBLUU
87 17.7440.00 09
79. 13.33
58.2420.00 os
92. 00.27=00 202720010 18
37.51=0 17
94,53=0.07
30.94+0.12
87.1010.
81.3910.00
58.51 10.1 13
01.5110.1
33.4310 15
94.62.1000
53,4431 25
Mish
87.11
81.09
58.37 +0.10
91.61+0.15
33,75
94.61
53.05+0.12
Tanh
87.48.1..0
80.5610.07
57.35110.0
90.3210.8
29.51.0.12
94.6320007
51.8610.00
Max
86 86.9620.20
81.76 14
58. 38.6020.12
90.98
33.3720.10
94, 24,70=..00
53.5320.16
Max, Min (d)
87.23 13
82.31
59.05110.10
91.6810 18
34.91
94.64.0.01
53.91
SignedGeomean
87 47.03
51.45
11.80
91.34
26.37 48
94.68
37. 37.16
XNORm.
10.01
79.6220.09
57.1410.07
84.7620-43
1.34 11
51.9812011 94.5120.0
ORI
87.1s
79.75
58.07
91. 12:12.0.0
33. 12:0.20.
94.60
51.21+0.17
XNORNn.
87.25
82.8810.0s
60.78
93.26 26
39.47 20
94.83 un 96
55.34 to
ORNE.
79.61
38.4480.10
21.0580.22 29
35.82+0.0 04
50,99/20.15 94.58
OR, ANDsm. (d)
86 12
00.09
60-00007
11.4+=00
37.03+0.1
91.65=0000
52, 32.19+0.00
OR, XNORgi (d)
87.82.6.1 19
82.4710.05
60.0010 11
92, 22,93 12
39.2210 17
94.6320.00
34.8710.0
OR, AND, XNORNu (d)
8741
82,84
60 60.3820.10 10
92, 28.98g.11 17
39,4220.18 18
94.7|=0.03
15.1/2012
XNORAI
86.97+0.18
$1.83+0.06
58. 40
90.93=0.15
32.56=0.10
94.71=0.06
53.75+0.14
ORAIL
87.45a0.3
88
1010 09
9/2 02.00 15
36.01 a. 12
94.6920 04
53.68 14
XNORNAIL
87 27.61+0.22
82, 27.3840.07
59.77+0.13
93.07=0.20 20
39.77+0.0
94.52=0.00
53.91
ORNAH.
87. 19
00
59.4020.0
92.
37.32
94, 64.6522000
53.8210
OR, ANDNAIL (d)
87 67.6220.11
82, 28.281000
59.7110.0s
92. 10.00 20
$7.70 12
94.6 20.05
53.8610 10
OR, XNORNAI (d)
87.85+0.22
82.52+o.11
60.02+0.10
12.12
39.64=0.00
94.75-00.0
54.13+0.05
AND, AN (d)
87 34
82.6720.00
60.01 21
93. 23.1222. 21
39.65a0.1 34
94. 94.7820.0
34.5820
only considering using single activation function (or ensemble) throughout the network, however our results together indicate stronger results may found by using OR.AIL for feature extraction and either XNOR.IL or ensemble [OR, XNOR.IL. (d)) for later higher-order reasoning layers.
Our work shows there is more learn about the importance of more complex activation functions, both for ANN applications and for non-linear dendritic integration in biological neuronal networks.
Acknowledgments and Disclosure of Funding
We are grateful to Chandramouli Shama Sastry, Eleni Triantafillou, and Finlay Maguire for insightful discussions.
Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, and companies sponsoring the Vector Institute https: //vectorinstitise a httppartters/..../am/. and in part by ACENET https: //t////ceeeetteeeeeee ca/ and Compute Canada ht / www. computecanade ca/. Additionally, we gratefully ac- knowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
References
Moray Allan and Christopher Williams. Harmonising chorales by probabilistic inference. L. Saul, Y. Weiss, and I.. Bottou (eds.), Advances in Neural Information Processing Systems, volume 17.
MIT Press, 2005.
Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- 291-301. PMLR, 09-15 Jun 2019, 4.8:50..85// 1811.03881
10

---

David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 511-520. PMIR, 10-15 Jul 2018. 400.50.8850//rrivv. wttpaaaxii...8....00 1807.04255.
Jonathan T. Barron, Continuously differentiable exponential linear units. arXiv: 100.07744,7 2017. 000:10048500/7744.7445
arXiv preprint,
Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-localized abstract reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12557-12565,
June 2021. doi: 10. 20055000axxi.22000094.....
Jock Blackard and Denis Dean. Comparative accuracies of artificial neural networks and discriminant analysis in predicting forest cover types from cartographic variables. volume 24, pp. 131-151, 12 1999. doi: 10.
Jock A. Blackard. Comparison of Neural Networks and Discriminant Analysis in Predicting Forest Cover Types. PhD thesis, Colorado State University, USA, 1998. AA19921979.
Nicolas
Yoshua Bengio, and Pascal Vincent. Modeling temporal de- pendencies in digh-dimensional sequences: Application to polyphonic music generation and transcription. In Proceedings of the 29th International Coference on International Conference Machine Learning, ICML' 12, pp. 1881-1888 Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.
Artem Chernodub and Dimitri Nowicki. Norm form-preserving orthogonal permutation linear unit activa- tion functions (OPLU). arXiv preprint, uuXiv:160.003311 2017. doi: 10.40.50/////rii.....om 1604.02313.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep net- work learning by exponential linear units (ELUs). arXiv preprint, arXiv:1511.0728), 2016. fot:100485550rxxx11551....2
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised the fourteenth International Conference on Artificial Intelligence and Statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
feature learning. In Proceedings of the
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Auto ment: Learning augmentation policies from data. arXiv preprint, arXiv:1805.0955 2018. doi: 1e..00..55///riiv..lll. 1805 100500000.
Stefan Elfwing. Eiji Uchibe, and Kenji Doya. Siggoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018. ISSN 0893-6080. doi: 10. 101.0..111....... www..:.....mee............ 12.011. Special issue on deep reinforcement learning.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):594 611, 2006.
François Fleuret, Ting Li, Charles Dubout, Emma K. Wampler, Steven Yantis, and Don- ald Geman. Comparing machines and humans on visual categorization test. Proceed- ings of the National Academy of Sciences, 108(43): 17241-1176611762 2011. ISSN 0027-8424. doi:10.1073/pnas 109168108.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193- 20222 Apr 1980. ISSN 1432-0770. doi: 100.1000000//F0344455511
Albert Gidon, Timothy Adam Zolnik. Pawel Fidzinski. Felix Bolduan, Athanasia Papoutsi, Panayiota Poirazi, Martin Holtkamp, Imre Vida, and Matthew Evan Larkum. Dendritic action poten- tials and computation in human layer 2/3 cortical neurons. Science, 367(647))833-87, 2020. 1 126/sccienee..ax623..
Luke B. Godfrey and Michael S. Gashler. A parameterized activation function for learning fuzzy logic operations in deep neural networks. 20/7 IEFE International Conference on Systems, Man, and Cybernetics (SMC), pp. 740-745, 2017, doi: 10000.0.110 122666
11

---

Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp.
1319-1327, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Gaëtan Hadjeres, François Pachet, and Frank Nielsen. DeepBach: stecrable model for Bach chorales generation. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1362-1371. PMLR, 06-11 Aug 2017. doi:10.455 10w.00/80//////..1111111 161.00100..
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, New York, NY, USA, June 2016a. IEEE. 102110.119/C//PP20106
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 630-645, Cham, 2016b. Springer International Publishing. ISBN 978-3-319- 46493-0. doi:
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint. arXiv:1606.08415, 2020. 000:10.455501166.0088
H. E. Hentschel, Alan Fine, and C. S. Pencea. Biological computing with diffusion and excitable calcium stores. Mathematical Biosciences & Engineering, 1(1):147-159, 06 2004. doi: 10..139344/mmb.2004.1144
Hermann Hild, Johannes Feulner, and Wolfram Menzel. HARMONET: A neural net for harmonizing chorales in the style of J.S Bach. In Proceedings of the 4th International Conference on Neural Information Processing Systems, NIPS'91, pp. 267-274, San Francisco, CA, USA, 1991. Morgan Kaufmann Publishers Inc. ISBN 1558602224.
Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 1567-1574, May 2021. 000110048550/arxx2202...663
Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts. Aaron Courville, and Douglas Eck. Coun- terpoint by convolution. arXiv preprint, 1X03v:900.00222 2019. doi: 10.485500/a 1itii..99.... 1903.07227.
Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations image collections. In 2015 IEFE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1383-1391, 2015. 100....000/ 7.297444.
Kevin Jarrett, Koray Kavukcuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th International Conference on Computer Vision, pp. 2146-2153, 2009. doi: 10. 20110///C..20005455555
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1988-1997,, July 2017. 100100..00//CVPR.211...2
H M Dipu Kabir, Moloud Abdar, Abbas Khosravi, Seyed Mohammad Jafar Jalali, Amir F Atiya, Saeid Nahavandi, and Dipti Srinivasan. SpinalNet: Deep neural network with gradual input. IEEE Transactions Artificial Intelligence, pp. 1-13, 2022. 10911..11999AA.2222..111
Daniel Kahneman. Thinking, fast and slow: Thinking, fast and slow. Farrar, Straus and Giroux, New York, NY. US, 2011. ISBN 0-374-27563-7 (Hardcover); 1-4229-69355.0 (PDF); 978-0-374-275631 (Hardcover); 978-1-4299-6935-4 (PDF).
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint, 15x.:1500000078 2015. doi: 000.485550aaxii..5500000000
12

---

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
10i.10.8550//rxiiv. 1412.6980.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In I. Guyon, U. Von Luxburg. S. Bengio. H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. doi: 1ee41004450/17066 00050//17060211135
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554-561, 2013. 1oi: 109//CC/
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-23244 1998.
Feynman Liang. Bachbot: Automatic composition in the style of Bach chorales. University of Cambridge, 8:19-48, 2016.
Michael London and Michael Häusser. Dendritic computation. Annual Review of Neuroscience, 28 (1):503-532, 2005. 0oi:10.1166/anmueere 146/annurev.neuro.28...8 wtw.::.....666....111.
Marvin Minsky and Seymour Papert. Perceptrons. Perceptrons. M.I.T. Press, Oxford, England, 1969.
Diganta Misra. Mish: A self regularized non-monotonic neural activation function. arXiv preprint, arXiv:1908. 08681, 2019, doi: 168555..850// 1900i...........0 1908.0861..
Michael C Mozer. Connectionist music composition based on melodic, stylistic, and psychophysical constraints. University of Colorado, Boulder, Department of Computer Science, 1990.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve Restricted Boltzmann Machines. In Johannes Fürnkranz and Thorsten Joachims (eds.), Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pp. 807-814. Omnipress, 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729, 2008. doi: 10. 109//CVGIP.2000..11
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 1/022991/diittil..00007..
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: introduction to circuits. Distill, 2020. doi: 20311//istilll000000
D. B. Owen. A table of normal integrals. Communications in Statistics SSiulation and Computation, 94):389-419, 1980. 01011001080003611118000888
Alexandre Payeur, Jean-Clauda Béique, and Richard Naud. Classes of dendritic information processing. Current Opinion in Neurobiology, 58:78-88 85, October 2019. ISSN 0959-4388. dbthhtps//do..rr//100010....
www.:///..cob..01..... 20..20099000000 Computational Neuroscience.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc' Aurelio Ranzato. Task-driven modular networks for zero-shot compositional learning. In Proceedings of the IEFF/CVF Interna- tional Conference on Computer Vision (ICCV), October 2019, doi: 48..8850//rxi.111199 190....599000.......
13

---

Alec Radford, Rafal Jozefowicz. and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint, arXiw:1704.01444, 2017. dbt:10.485500mxxv...7..... 170400144,
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. arXiv preprint, arXiv:1710.05944 2017. 08511000885077710059944
John C Raven and JH Court. Raven's 's progressive matrices. Western Psychological Services Los Angeles, CA, 1938.
Jason D. M. Rennie, Lawrence Shih, Jaime Teevan, and David R. Karger. Tackling the poor assumptions of Naive Bayes text classifiers. In Tom Fawcett and Nina Mishra (eds.), Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA, pp. 616 623. AAAI Press, 2003.
Leslie N. Smith. A disciplined approach to neural network hyper- paeerpaaameters: Part Iearn- ing rate, batch size, momentum, and weight decay. arXiv preprint, arXiv:1803.09220 2018. doi: 1..555...50///rxxiv. wot.:xxii...1..00000 1803.09820.
Leslie N. Smith and Nicholay Topin. Super-convergence:: Very fast training of residual networks using large learning rates. arXiv preprint, arXiv: DeeXiv:1700.00 170000708710110 120, 2017. doi: 10..11450//axx..11111 1w 700.007172222 07.0071200
UCI Machine Learning Repository. Covertype data set, Aug 1998. URL https:///rcchi:
ics. wui.ccii... htt.iedddmmmaaaaaaeeee Soapeetts//oorrrree /Covertype.
Chi Zhang, Feng Gao. Baoxiong Jia, Yixin Zhu, and Song-Chup Zhu. RAVEN: A dataset for relational and analogical visual reasoning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5312-5322, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] The methodology we discuss
and any societal impacts we could discuss
would be highly general and completely speculative at this stage.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See $4.1-4.5 and Appendix $ AA111A..1.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] All results figures and tables have error over several seeds, and the number of seeds used, clearly indicated.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.99. The time to run each experiment is also indicated in the README files which accompany the code.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) your work uses existing assets, did you cite the creators? [Yes] See Appendix $..8. Appendix Table 3.
14

---

(b) Did you mention the license of the assets? [No] It is surprisingly difficult to discover the license under which popular ML datasets, such CIFAR- 10, are released! Most of the popular datasets within the community do not appear to be released under open licenses.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code replicate results is included in the supplemental material zip.
(d) Did you discuss whether and how consent was obtained from people whose data you't 'r using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We describe the nature of the datasets we are using contains personally
using. There is reason expect that the data identifiable information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "rvZUu7-T7t7", "reviewer": "Reviewer_BBqK", "summary": "In this paper the authors presents new activation functions based on a relaxation of the logical operators AND, OR, XNOR. Moreover, the authors present approximations that are more efficient based on simple max, min and addition operations.  This is motivated by the behavior of the operators under the logit space equivalence. As these binary operators are reduction operations, i.e., they convert two inputs into one, the authors connect to the MaxOut. Furthermore, they provide ensemble alternatives to adapt neural network architectures to use the presented boolean activation functions. Finally, the authors provide empirical results showing how the different potential combinations of logical activation functions behave for different datasets and architectures.", "questions": "Do you think that these activation functions could provide a path for explainability of neural networks?\n\nI see this paper somehow related to continuous logic functions, and I'm wondering whether composition of more complex logical functions would also make sense, e.g., introducing an implication activation, etc.\n\n", "limitations": "no negative societal impact", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Here, the authors present new activation functions. One weakness could be that they make independency assumptions, however, this does not seem to negatively affect the performance of the models. \n\nThe empirical evaluation is sound, and demonstrates the activation functions behave well and even improve the performance in some cases. They achieve this without significantly adding more parameters by using ensembles of activation functions. This is an interesting contribution as previously (MaxOut) would significantly increase the number of weights in order to have the same number of outputs. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qPgC6A4N-5q", "reviewer": "Reviewer_59zS", "summary": "This paper introduces a set of novel activation functions for\nartificial neural networks, inspired by the Boolean logic gates for\nAND, OR and XNOR.  After defining and motivating the functions, the\npaper derives approximated versions that are more computationally\nefficient, and demonstrates their use on a few test problems.\n", "questions": "1. Why is it important for network to possess Boolean activation\n  functions, given that large enough networks can approximate any\n  arbitrary function?\n2. Of all the combinations of activation functions explored in the\n  experiments section, which one should I pick to use in my next deep\n  net architecture? Is there a principle, a strategy or a rule of\n  thumb that I could use to choose one (or more) activation function\n  without having to train and test my network with all possible\n  alternatives?\n", "limitations": "I couldn't find much in terms of explicit analysis of limitations.\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe question addressed by the paper is reasonably original,\nand the proposed activation functions are derived and described\nclearly.\n\n### Weaknesses\nIn my view the main weakness of the paper is insufficient\nmotivation and significance. Why is it interesting to have activation\nfunctions that mimic logical gates? No clear motivation is given,\neither from a theoretical standpoint (e.g., networks with such\nactivation functions may be expected to learn more efficiently for\nsome reason), or from the standpoint of biological inspiration (a few\nwords are spent on dendritic computation, but logical operations are\nnot what defines dendritic computation - they are just one particular\nabstraction that is sometimes applied to it). At the same time, in the\nexperiments section there is no clear sign of particular empirical\nadvantages conferred by the proposed activation functions. Therefore,\nto summarize, the logical activation functions, although defined in a\nconsistent and reasonable way, lack a-priori theoretical motivation as\nwell as a-posteriori demonstrations of empirical usefulness.\n\nI'll now expand on what I meant by \"no sign of empirical advantages\"\nabove. In my view, the results section presents an overly-optimistic\ninterpretation of the empirical results. Throughout the various\nexamples, a range of choices of activation functions is explored, but\nno choice is shown to be consistently better than the others across\ntasks (perhaps with the exception of learning the parity of the\ninputs; but that case is more of a sanity check, as it is the natural\noperation implemented by the XNOR gate). Indeed:\n- in 4.2 (MLP on Bach chorales) there is no statistically significant\n  advantage;\n- in 4.3 (MNIST) the results are mixed and overinterpreted: for the\n  MLP the text says that one of the proposed gates is matched for best\n  place with one of the alternatives considered for comparison, but\n  this is not even evident from the plot (there are hyperparameter\n  regimes where that choice of gate is the one that performs the\n  worst). For the CNN case, a different choice of logical-gate\n  activations is claimed to be among the ones performing best, but\n  again from the plot it looks like the \"Max\" activation function is\n  the best overall.\n- in 4.4, the gate that was identified as the best performer in\n  4.3/MLP is now the worst performer, so much so that it is not even\n  included in the plot. The other gates all produce mixed results with\n  no clear winner. The activation that most consistently performs\n  better than others in this section seems to be SiLU, but in the text\n  it is grouped together with Max and ORAIL (one of the proposed\n  gates). For some reason the results of this section don't have error\n  bars.\n- section 4.5 (transfer learning) also fails to find a scenario where\n  the proposed activations consistently perform better than the\n  alternatives.\n\nFinally, the experiments section is at times made hard to interpret by\nlack of methodological details. For instance, it is not clear to me\nhow the number of parameters is balanced across activation function\nchoices, since the different structure of 1-to-1 and 2-to-1\nactivations (e.g. ReLU vs the activations introduced in this paper)\nimply a different organization of the network. Moreover, some p-values\nare given without giving any information on the test (such as line\n262, \"For the MLP, XNORAIL performed best along with signed_geomean (p\n< 0.1)\", and some results seem to lack any statistical analysis/error\nbars (e.g. Figure 6).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cb9HfeVekCM", "reviewer": "Reviewer_VL66", "summary": "This paper presents a new set of activation functions based on the approximation of  logit arithmetic AND/OR/XOR operation. The effectiveness of the proposed activity functions are demonstrated on many tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.", "questions": "* Does it improve or worsen the training time? \n* Why is the performance of AIL better than IL in most cases(such as show in Table 1)? Especially in cases like stfCars that XNOR_IL has barely any accuracy. \n* Can different layers use different settings? (some use XNOR, some use XNOR/OR d, etc)\n\n", "limitations": "The authors did discuss the limitations of the proposed method, such as the method being suitable for certain tasks based on experimental results. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**:\n\n* (+) To my knowledge, the idea proposed in this paper that uses  n->1 activation functions via approximating logit-space boolean operations is novel and interesting\n\n**Quality**:\n\n* (+) The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.\n* (+) Related works are covered and discussed.\n* (+)  Experiments are conducted extensively in different type of tasks and the results are discussed thoroughly.\n\n\n**Clarity**:\n\n* (+) This paper is well written and organised.\n* (-) it would be better to have a table accompanying figures such as figure 6 as the result can be difficult to read given the amount of legend. Especially for figure 5, 6, the table for legend given is impossible to distinguish between silu and OR, XNOR d\n\n\n\n**Significance**:\nAs demonstrated in the experimental results, the accuracy gain in CIFAR datasets in the transfer learning tests are quite big. I would assume that the proposed activation functions can be quite effective in certain types of tasks. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept