Logical Activation Functions: Logit-space equivalents of Boolean Operators
Authors: Scott C Lowe
Robert Earle
Jason d'Eon
Thomas Trappenberg
Sageev Oore
Venue: ICLR 2022 Submitted (2022)
Keywords: activation functions
logits

Abstract: Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds score of presence (versus absence) of features within the stimulus. Under this interpretation, we can derive the probability $P(x_0 \cap x_1)$ that a pair of independent features are both present in the stimulus from their logits. By converting the resulting probability back into a logit, we obtain a logit-space equivalent of the AND operation. However, since this function involves taking multiple exponents and logarithms, it is not well suited to be directly used within neural networks. We thus constructed an efficient approximation named $\text{AND}_\text{AIL}$ (the AND operator Approximate for Independent Logits) utilizing only comparison and addition operations, which can be deployed as an activation function in neural networks. Like MaxOut, $\text{AND}_\text{AIL}$ is a generalization of ReLU to two-dimensions. Additionally, we constructed efficient approximations of the logit-space equivalents to the OR and XNOR operators. We deployed these new activation functions, both in isolation and in conjunction, and demonstrated their effectiveness on a variety of tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.

Under review as a conference paper at ICLR 2022
LOGICAL ACTIVATION FUNCTIONS:
LOGIT-SPACE
EQUIVALENTS OF BOOLEAN OPERATORS
Anonymous authors Paper under double-blind review
ABSTRACT
The choice of activation functions and their motivation is long-stani bong-standing issue within the neural network community. Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds of presence (versus absence) of features within the stimulus. In this work, we investigate the implications on activation functions of considering features to be logits. We derive operations equivalent AND, OR, and XNOR for log-odds ratio representations of independent probabilities. Since these functions involve taking multiple exponents and logarithms, they are not well suited to be directly used within neural networks. Consequently, we construct efficient approximations named ANDAn. (the AND operator Approximate for Independent Logits), ORAI.. and XNOR which utilize only comparison and addition operations and can be deployed as activation functions in neural networks. Like MaxOut, AND AIL and ORAIL generalizations of ReLU to two-dimensions. We deploy these new activation functions, both in isolation and in conjunction, and demonstrate their effectiveness variety of tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.
INTRODUCTION
An activation function is non-linearity which is interlaced between linear layers within artificial neural network. The non-linearity essential in order for higher-order representations to form, since otherwise the network would be degeneratively equivalent to single linear layer.
Early artificial neural networks were inspired by biological neural networks, with the activation function analogous to neuron's 's need to exceed potentiation threshold in order to fire an action potential. Biological neurons are have long been known to be more complex than this simple abstraction, including features such non-linearities in dendritic integration. Recent work has demonstrated that a single biological neuron can compute the XOR of its inputs (Gidon et al. 2020), property long known to be lacking in artificial neurons (Minsky & Papert 1969). This suggests that gains in artificial neurons can be made by using activation functions which operate on more than one input to the neuron at once.
The earliest artificial neural networks featured either logistic-sigmoid or tanh as their activation function. These activation functions were motivated by the idea that each layer of the network is building another layer of abstraction of the stimulus space from the last layer. Each neuron in layer identifies whether certain properties or features are present within the stimulus, and the pre-activation (potentiation) value of the neuron indicates score logit for the presence of that feature. The sigmoid function, (xx) 1/(1+= "), was hence a natural choice activation function, since as with logistic regression, this will convert the logits of features into probabilities.
There is some evidence that this interpretation has merit. Previous work has been done to identify which features neurons are tuned to. Examples include LSTM neurons tracking quotation marks, line length, and brackets (Karpathy et al, 2015); LSTM neurons tracking sentiment (Radford et al. 2017); methods for projecting features back to the imput space view them (Olah et al. 2017); and interpretable combinations of neural activities (Olah et al. 2020). Analogously, within biological neural networks, neurons tuned to respond more strongly to certain stimuli, and less strongly

---

Under review as a conference paper at ICLR 2022
others. At high representation levels, concept cells respond predominantly to certain concepts, such one's grandmother, or Jennifer Aniston (Gross
2002 Quian Quiroga et al.) 2005).
Sigmoidal activation functions no longer commonly used within machine learning between layers of representations, though sigmoid is still widely used for gating operations which scale the magnitude of other features in attention-like manner. The primary disadvantage of the sigmoid activation function is its vanishing gradient - as the potentiation rises, activity converges to plateau, and hence the gradient goes zero. This prevents feedback information propagating back through the network from the loss function to the early layers of the network, which consequently prevents it from learning to complete the task.
The Rectified Linear Unit activation function (Fukushima 1980 Jarrett et al. 2009 Nair & Hinton
2010), (z)U(z)= = max(0, x²), does not have this problem, since its non-zero regime has a gradient of 1. Another advantage of ReLU is it has very low computational demands. Since it is both effective and efficient, it has proven to be highly choice of popular activation function. The chief drawback to ReLU is it has no sensitivity to changes across half of its input domain, and on average passes no gradient back to its inputs. This can lead to neurons dying if their weights make them never reach their activation threshold. If sigmoid is equivalent to converting our feature logits into probabilities, then the ReLU activation function equivalent to truncating our logits and denying any evidence for the absence of a feature. We hypothesise that omitting negative evidence in this way is undesirable and reduces the generalisation ability of Rel.U-based networks.
Variants of ReLU have emerged, aiming to smooth out its transition between domains and provide a gradient in its inactive regime. These include ELU (Clevert et al.) 2016), CELU (Barron 2017).
SELU (Klambauer et al. 2017), GELU (Hendrycks & Gimpel| 2020), SiLU (Elrwing al. [017
Ramachandran et al.] 2017). and Mish (019). However, all these activation functions still
2019).
bear the general shape of ReLU and truncate negative logits.
Fuzzy logic operators are generalizations of boolean logic operations to continuous variables, using rules similar to applying logical operators in probability space. Previous work has explored networks of fuzzy logic operations, including neural networks which use an activation functions that constitutes a learnable interpolation between fuzzy logic operators (Godfrey & Gashler 2017). In this work, we introduce activation functions which are similar to fuzzy logic operators, but derived for working in logit space instead in probability space.
In this work we set out to develop activation functions based on the principle that neurons encode logits -icss--oo scores that represent the presence of features in the log-odds space. In [Section we derive and define these functions in detail for different logical operators, and then consider their performance on numerous task types including parity (Section I. image classification (Sections 3.4 and 3.5). transfer learning (Section 3.6). abstract reasoning (Appendix A.14), soft-rule guided classification as exemplified by the Bach chorale dataset (Section 3.3). and compositional zero-shot learning (Ap) pendix A.15). These tasks were selected to (I) survey the performance of the new activations on existing benchmark tasks, and (2) evaluate their performance on tasks which we suspect in particular may require logical reasoning and hence benefit from activation functions which apply these logical operations to logits.
DERIVATION
Manipulation of probabilities in logit-space is known to be more efficient for many calculations. For instance, the log-odds form of Bayes' Rule (Equation 9) states that the posterior logit equals the prior logit plus the log of the likelihood ratio for the new evidence (the log of the Bayes factor). Thus, working in logit-space allows to perform Bayesian updates on many sources of evidence simultancously, merely by summing together the log-likelihood ratios for the evidence. A weighted sum may be used if the amount of credence given to the sources differs and this is precisely the operation performed by a linear layer in a neural network.
When considering sets of probabilities, a natural choice of operation perform is measuring the joint probability of two events both occurring - the AND operation for probabilities. Suppose our input [0, 1J2, and the goal is output y 0 if x; = 1 i and y < 0 otherwise, using model
space is
'Though this problem is very rare when using NatchNorm stabilise feature distributions.
2

---

Under review as a conference paper at ICLR 2022
with weight vector and bias term b, such that y w b. This can be trivially solved with the weight matrix w = [1, 1] and bias term b = -1.5. However, since this is only linear separator, the solution not generalise to the
y > 0 iff
O...
Similarly, let us consider how the OR function solved with a linear layer. Our goal is to output y > be trivially solved with the weight matrix w = [1, 1] and bias term b = -0.5. The difference between this and the solution for AND is only an offset our bias term. In each case, if the input space is expanded beyond binary to I2, the output can be violated by changing only one of the arguments.
if x.; and y otherwise. The binary
2.1 AND
Suppose we are given x and y as the logits for the presence (vs absence) of two events, X and Y. These logits have equivalent probability values, which can be obtained using the sigmoid function, a(u) = (1 + e --) Let us assume that the events X and are independent of each other. this case, the probability of both events occurring (the joint probability) is P(X,Y) P(X A Y P(X) P(V)= = (x) (g).
However, we wish to remain in logit-space, and must determine the logit of the joint probability, (ogit(P(X, Y)). This is given by
1- p_) where = o=
AND == (a(i(P(X)))))a log
o (xx) (v)
llog
(1)
1- (1)o(z)o())
which we coin as AND the AND operator for independent logits (IL). This 2d function is illustrated contour plot FFiuure I] left). Across the plane, the order of magnitude of the output is the same as at least one of the two inputs, scaling approximately linearly.
The approximately linear behaviour of the function suitable for use as an activation function (no vanishing gradient), however taking exponents and logs scales poorly from a computational perspective. Hence, we developed a computationally efficient approximation that we can loosely approximate AND with the minimum function (Figure 1 right panel). This is equivalent assuming the probability of both and Y being true equals the probability of the least likely of X and Y being true naive approximation which holds well in three quadrants of the plane, but overestimates the probability when both and Y are unlikely. In this quadrant, when both X and Y are both unlikely, a better approximation for ANDIL is the sum of their logits.
follows. Observe
We thus propose ANDAIL AIL, a linear-approximate AND function for independent logits (AIL, i.c. approximate IL).
y, ANDAu (min(a,y)) v): otherwise
0, 0
As shown in Figure [left, middle), we observe that their output values and shape are very similar.
ANDa
AND
min
10
18
+0
- 2
2
4
-4
2
2
-4
4
Figure 1: Heatmap comparing the outputs for the exact logit-space probabilistic-and function for independent logits, ANDu v): our constructed approximation, AND (x,, y); and max(r, v).
3

---

Under review as a conference paper at ICLR 2022
2.2 OR
Similarly, we can construct the logit-space OR function, for independent logits. For a pair of logits x and y, the probability that either of the corresponding events is true is given by p 1 (- (- This can be converted into a logit as
p (-p)
ORn (2, y) = logit (P(X V 1) log
where p = d(-x) o(-))
which can roughly approximated by the max function. This is equivalent to setting the probability of either of event X or Y occurring be equal to the probability of the most likely event. This underestimates the upper-right quadrant (below), which we can approximate better as the sum of the input logits, yielding
I > 0, y>>
ORAIL (x,
(4)
max((x,0) v), otherwise
ORe
ORAi
max
+10
+2
++0
-4
-2
4
-4
2
2
-4 -2
4
Figure 2: Comparison of the exact logit-space probabilistic-or function for independent logits,
ORn(x, y); our constructed approximation, ORAIL y); and max y).
2.3 XNOR
We also consider the construction of a logit-space XNOR operator. This is the probability that X and Y occur either together, or all, given by
XNORI. (z, v) = logit(P(XX))) log 1-
p
(5)
where p (z) (ly) + (((-))((-)) We can approximate this with XNORAn y) = sgn(zy) min([a].
(6)
which focuses on the logit of the feature most likely to flip the expected parity (Figure 3)
could use other approximations, such as the sign-preserving geometric mean,
signed.geomean (z, y) : sgn(xy) V |zy], but note that the gradient of this is divergent, both along æ = O and along y = O.
2.4 DISCUSSION
By working via probabilities, and assuming inputs encode independent events, we have derived logit-space equivalents of the boolean logic functions, AND, OR, and XNOR. Since these computationally demanding to compute repeatedly within neural network, we have constructed approximations of them: ANDAIL. ORAIL. and XNORAIL Like ReLU, these involve only com- parison, addition, and multiplication operations which are cheap to perform. In fact, AND AIL and ORAII generalization of ReLU to extra dimension, since OR (z., 0) maz 0).
The majority of activation functions are one-ooneeet mee-immensional, f R + R. In contrast to this, our proposed activation functions all two-dimensional, f: I IR. They must be applied pairs of features
4

---

Under review as a conference paper at ICLR 2022
XNORa
XNOR
signed geomean
+8
+6
+4
+2
-2
-4
10
Figure 3: Comparison of the exact logit-space probabilistic-xnor function for independent logits, (NORIL(E, y): our constructed approximation, XNORAIL y); and signed_geomean y).
from the embedding space, and will reduce the dimensionality the input space by factor This behaviour is the same as seen in MaxOut networks (Goodíellow et al. 2013) which use max their activation function, MaxOut(r, y; k) == max(x, y). Similar to MaxOut, our activation functions could be generalised to higher dimensional inputs, I R, by considering the behaviour of the logit-space AND, OR, XNOR operations with regard to more inputs. For simplicity, we restrict this work to consider only k k=2 2, but note these activation functions also generalize higher dimensions.
2.5 ENSEMBLING
By using multiple logit- bogit-boolaan activation functions simultaneously alongside each other, we permit the network multiple options how relate features together. When combining the activation functions, we considered two strategies.
In the partition (p) strategy, we split the n. dimensional pre-activation embedding equally into m partitions, apply different activation functions on each partition, and concatenate the results together. Using AIL activation functions under this strategy, the output dimension will always be half that of the input, it is for each AIL activation function individually.
In the duplication (d) strategy, we apply m different activation functions in parallel to the same n. elements. The output is conscquently larger, with dimension ne. If desired, 2-+1 reduction of AIL activation functions by using two of them together under this strategy.
counteract the
Utilising AND AIL. ORAIL and XNORAIL simultaneously allows networks to access logit-space equivalent of 12 of the 16 boolean logical operations with only a single sign inversion (in either one of the inputs the output). Including the bias term and skip connections, the network has easy access to logit-space equivalents of all 16 boolean logical operations.
3 EXPERIMENTS
We evaluated the performance of our AII. activation functions, both individually and together in an ensemble, range of benchmarking tasks. Since AND and ORAIL equivalent when the sign of operands and outputs can be freely chosen, we evaluate only on ORAIL and not both.
We compared the AIL activation functions against three primary baselines: (1) ReLU, (2)
max(x, y) = MaxOut([2, y]; k= 2). and (3) the concatenation of max(æ, y) and min (2, y), denoted (Max, Min (d)). The [Max, Min (d)) ensemble is equivalent GroupSort with group size of (Anil et al. 2019: Chernodub & Nowicki] 2017). sometimes referred to as the MaxMin operator. and concatenation of OR and ANDAI.. under our duplication strategy.
is comparable
3.1 PARITY
In simple initial experiment, constructed synthetic dataset whose labels could be derived directly using the logical operation XNOR. Each sample in this dataset consisted of four input logit values, with label that derived by converting each logit to probability space, rounding the
5

---

Under review as a conference paper at ICLR 2022
nearest integer, and taking the parity over this set of binary digits (i.e. true when we have an number of activated bits, false otherwise).
A very small MLP model with two hidden lay- ers (the first with four neurons, the second with two neurons) should be capable of per- fect classification accuracy on this dataset with sparse weight matrix by learning to extract pairwise binary relationships between inputs us- ing XNORALL AIL. We trained such an MLP, for 100 epochs using Adam, one-cycle learning rate schedule, max LR 0.01, weight decay 10-1
Outputs
Layer 2 Post-act
Layer 2 Pre-act
The two-layer MLP using the XNORAIL acti- vation learned a sparse weight matrix able to perfectly classify any input combination, shown in Figure 4] In comparison, an identical network setup using ReLU was only able to produce 60% classification accuracy. Though this accuracy could be improved by increasing the MLP width or depth, the weight matrix was still not sparse. This experiment provides example situation where XNORAIL is utilized by a network to directly extract information about the relation- ships between network inputs. For additional results, see Appendix A.7
15
Layer 1
Post-act
Layer 1 Pre-act
09 1200
Inputs
ReLU
XNOR
AIL
3.2 MLP ON COVERTYPE
Figure 4: Visualisation of weight matrices learnt by two-layer MLPs binary classification task, where the target output the parity of the inputs. Widths of lines indicate weight magnitudes (or- ange: +ve values, blue: -ve). Network with ReLU: 60% accuracy. Network with XNORAIL: 100% accuracy.
We trained small one, two, and three-layer MLP models on the Covertype dataset (UCI Machine
Learning Repository 1998] Blackard & Dean
1998) from the UCI Machine Learning Repos- itory. The Covertype dataset is classification task consisting of 581 012 samples of forest cov- erage across 7 classes. Each sample has 54 at- tributes. We used fixed random 80:20 trainttest split; for training details and accuracy vs model size, see Appendix A.8
For each activation function, we varied the number of hidden units per layer to investigate how the activation functions affected the performance of the networks as its capacity changed. We did not use weight decay or data augmentation for this experiment, and so the network exhibits clear overfitting with larger architectures. As shown in Figure 16 Appendix A.8 XNORAIL performs significantly best on Covertype (p < 10-5) for MLP with 2 or 3 layers, followed by networks which include XNOR AIL alongside with other activations, and .gommenn The duplication strategy outperforms partition.
3.3 MLP ON BACH CHORALES AND LOGIT INDEPENDENCE
The Bach Chorale dataset (Bouldnger_Lwwanowwss
Lewandowski et 2012 consists of 382 chorales composed by JS Bach, each 12 measures long, totalling approximately 83,000 notes. Represented as discrete sequences of tokens, it has served as a benchmark for music processing for decades, from heuristic methods to HMMs, RNNs, and CNNs (Mozer 1990, Hild et al.) 1992; Allan & Williams) 2005 Liang] 2016 Hadjeres et al. 2017 Huang et al. 2019). The chorales are comprised of voices (melodic lines) whose behaviour is guided by soft musical rules that depend on the prior movement of that voice as well as the movement of the other voices. An example of one such rule is that two voices fifth apart ought not to move in parallel with one another. The task choose here is determine whether a given short four- four-part musical excerpt is taken from a Bach chorale or not. During training, stochastically corrupt chorale excerpts to provide negative examples (see Appendix A.9). We

---

Under review as a conference paper at ICLR 2022
trained 2-layer MLP discriminators with single sigmoid output and exchanged all other activation functions (summarized in Figure 17). We found that {OR, ANNDA (d)) performed best, but that overall the results were comparable (p 0.1 between best and worst, Student's t-test between 10 random initialisations).
Additionally, we investigated the independence between logits in the trained pre-activation embed- dings. We would expect that an MLP which is optimally engaging its neurons would maintain independence between features in order maximize information. To capture the existence of correla- tions, we took the cosine similarity between rows of the weight matrix. Since the inputs to all features in given layer are the same, this is equivalent measuring the similarity between corresponding pair of pre-activation features. We performed two experiments. In the first, we measured correlations between all pairwise combinations, and the second we took correlations between only adjacent areaaciiatiions that would be paired for the logical activation functions. For these experiments we used 2 hidden layers and a width that showed maximal performance for each activation function. The results are shown in Appendix A.10
3.4
CNN AND MLP ON MNIST
MNIST, MLP (2 hidden layers)
MNIST, CNN (6 layer)
98.8
99.6
98.6
99.5
Test: aoourey (86)
Test BOCuracy
SILL
Rell
Max
98.4
rigned gromenn
98.3
XNORAL
98.2
ORAi
OR. AND (6)
99.3
XNORAL (s)
98.1
XNOR
AND, XNOR
99.2
98.0
AND, XNORA
10
10
Number of parametors
Number of parameters
Figure 5: We trained CNN on MNIST MLP on flattenedMMNISTT using Adam (I-cycle, 10 ep), hyperparams determined by random search. Mean (bars: std dev) of n 40 weight inits.
We trained 2-layer MLP and 6-layer CNN models on MNIST with Adam (Kingma & Ba 2017).
1-cycle schedule (Smith & Topin 2017 Smith 2018), and using hyperparameters tuned through random search against a validation set comprised of the last 10k images of the training partition.
The MLP used two hidden layers, the widths of which were varied together to evaluate the perfor- mance for a range model sizes. The CNN used six layers of 3x3 convolution layers, with 2x2 max pooling (stride 2) after every other conv layer. The output was flattened and followed by three MLP layers. The widths of the layers were scaled up approximately linearly to explore a range of model sizes (see Appendix A. for more details).
For the MLP, XNORAI, performed best along with signed ggoomean (p < 0.1), ahead of all other activations (p 0.01; Figure 5 left panel). With the CNN, five activation configurations ((OR, AND, XNORAIL (p)). {OR, XNORAL (d/p)). Max, and SiLU) performed best (p < 0.05; Figure 5] right panel). Additionally, ensemble) maintained high performance with an order of magnitude fewer parameters (3 x 104) than networks which did not (3 x 10 params).
note that CNNs which used ORAIL Max (alone or in an
3.5 RESNET5O ON CIFAR- 10/100
We explored the impact of our AIL activation functions the performance of deep networks by deploying them within a pre-activation ResNet50 model (He et al.) 2015) 2016). We exchanged all ReLU activation functions in the network to candidate activation function while maintaining the size of thss-th embedding. We experimented with changing the width of the network, scaling up the embedding space and all hidden layers by common factor. The network trained
7

---

Under review as a conference paper at ICLR 2022
CIFAR- 10/-100 for 100 epochs using Adam (Kingma & Ba 2017) 1-cycle (Smith 2018 Smith
& Topin 2017). Hyperparameters were optimized through a random search against a validation partition of CIFAR- 100 for fixed width factor = 2 only. The same hyperparameters were reused for experiments of changing width, and for CIFAR-10. 10. We used width factors of 0.5, 1, 2, and 4 for 1-1 activation functions (ReLU, etc), widths of 0.75, 1.5, 3, and 6 for 2 2+> 1 activation functions (Max, etc), and widths of 0.4, 0.75, 1.5, and 3 for {ORAIL. ANDAIL. XNORAIL (d)).
CIFAR-10, ResNet50
CIFAR-100, ResNet50
96.0
79
95.5
Test accuracy ESA
95.0
Relli
PReLU
94.5
SIE1I
76
76
Max
94.0
Max,
75
ORAL
OR, AND (d)
93.5
OR, AND, XNOR (4)
93.0
OR, XNOR (p)
OR, AND. XNOR (g)
107
10'
107
10"
Number of parameters
Number parameters
Figure 6: ResNet50 on CIFAR-10/100, varying the activation function. The same activation function (or ensemble) was used through the network. The width was varied to explore range of network sizes (see text). Trained for 100 ep. with Adam, using hyperparams determined by random search on CIFAR- 100, w = 2, only.
For both CIFAR-10 and 100, we find that SiLU, ORAIL. and Max outperform ReLU across a wide range of width values, as shown in Figure 6 These three activation functions hold up their performance best as the number of parameters is reduced. Note that SiLU was previously discovered as an optimal activation function for deep residual image models (Ramachandran et al. (017), and so it is expected to perform highly in this setting. Meanwhile, other AlL activation functions perform similarly to ReLU when the width is thin, and slightly worse than ReLU when the width is wide. When used on its own and not part of ensemble, the XNORAIL activation function performed very poorly (off the bottom of the chart), indicating it is not well suited for this task.
3.6 TRANSFER LEARNING
We considered the task of transfer learning on several image classification datasets. We used a ResNet18 model (He et al., 2015) pretrained on ImageNet-Ik. The weights were frozen (not fine-tuned) and used to generate embeddings of samples from other image datasets. We trained two- layer MLP to classify the images from these embeddings, using various activation functions. For a comprehensive set of baselines, we compared against every activation function built into PyTorch 1.10. To make the number of parameters similar, we used a width of 512 for activation functions with 1-+1 mapping (e.g. ReLU), a width of 650 for activation functions with a 2-+1 1 mapping (c.g. Max, ORAI.,, and a width of 438 for {OR, AND, XNORAIL (d)). See Appenniii A.13 for more details.
Our results are shown in Table I We found that all our activation functions outperformed ReLU on every transfer task. In particular, our three activation ensembles using the duplication strategy performed very highly across all 7 transfer learning tasks overall, (OR, AND, XNORAIL (d)) appears to perform best (top for 3 datasets, second on 3), followed by {OR, ANDAI. (d)). Our activation functions were beaten on Caltech Oxford Flowers, and Stanford Cars, and only by PReLU. For these three datasets, linear readout outperformed an MLP with ReLU, and on 1altech101 the linear layer performed best. This suggests that PRcLU excels only on particularly simple tasks. For further discussion, see Appendix A.13
3.7 ADDITIONAL RESULTS
For results on abstract reasoning and compositional zero-shot learning, please see Appendix A.14 and Appendix A.15 respectively.
8

---

Under review as a conference paper at ICLR 2022
Table 1: Transfer learning from a frozen ResNet-18 architecture pretrained on ImageNet-Ik to other computer vision datasets. An MLP head with two layers of pre-activation width of either 438, 512, or 650 (depending on activation function, to keep the number of params approximately constant) was trained, without re-training the pretrained base model. Trained with SGD, cosine annealed LR 0.01, for 25 epochs. Mean (standard error) of n=5 random initializations of the MLP (same pretrained network). Bold: best. Underlined: second best. Italic: no significant difference from best (two-sided Student's t-test, > 0.05). Background: linear color scale from ReLU baseline (white) to best (black).
Test Accuracy (%)
Activation function
Caltech101
CIFARIO
CIFAR100
Flowers
Cars
STL-10
SVHN
Linear layer only
88.35
78.56+0.0r 09
57.39
92.32+o 20
33.51=0.00 as
94.68=0.02
45.42=0.00
ReLU LeakyReLU
86.582o.17
81.6310 as
58.0420.11
90.71 26
30.97 20
94.6220.06
51.33920.0 0.
86 1s
81.67+0.11
58.01+0.00
90.73+0.32
31.09=0.24
94.61=0.05
51.40=0.0s
PReLU
87 83
81.03+0.13
58,90=0.1s
93. 93.17=0.15
39.84=0015
51,42=0.00 0.
Sohplus
86. 162o 18
79. 18.1322..m os
56.5810.07
89.39 29
21.2320.15
94.6320.03
47.44 os
ELU
87. 18 0o
80,44+0.00 os
58.08=0.10
91.71=0 14
34.70= os
94.55=0.05
50.07 0.007
CELU
17.1880..0
80,4410
58.0820.11
91 14
34.70.0
94.5520.00 05
50.07
SELU
87.74 +0.00
79.93
58.24:0.06
92.27 +2.70..
37.5 1
94.53=9.a7
49.38
GELU
87 37.10=0.1 15
81.39+0.0 09
58.51=0.15
91.51+0.13 15
33.43=0.11
94.62=0.06 06
51,56=0.0 os
SiLU
86.91
80.5310
58.1422.11 12
91.37a0 18
32.1520.1
94.5920
50.6910 0a
Hardswish
87.1250.12
80.1020.10
58.2520.10
91.5620..
33.17:0.2s
94.6220.05
50.09
Mish
87.11+0.12
81.09+0.11
58.37=0.10
91.61+0.15
33.75=0.14
94.61=0.05
51.29
Softsign
81.4720.18
80.03 09
54.8420.09
82.3430.22 22
17.33:0.10
94. 94.70-...5
49.48
Tanh
87.18=0.06
80.56+0.07
57.35=0.05
90,32+0.20 20
29.51=0.12
94.63=0.07
50.15+0.0s
GLU
86.7120.51
79.19
37.6420.10
90.34go.19
27.04
94.5720.05
2o.17
Max
80.96=0.20
81,76+0.14
58 12
90.98
33.37=0.15
94.70-0.00
51.36+0.12
Max, Min (d)
87.2320.15
82.3/
59.05
91.682o.1
34.9120.2
94.6420.05
51.72:0.0
XNORAI.
86.97
81 81.8300.o
58, 10
90.93+0.15
32.36=0.19
94.71=0.m
51.54+0.00
ORAIL
87.45
81 88
59, 10
92.00
36.01 18
94, 14.6920.0 0.
51.52
OR, AID (d)
87.43go.1 11
82.3N1o os
59.90 59.90_noos
92.07 18
37. 16
94.552o.05
$2.11
OR, XNORAn. (p)
87. 87.422011 12
81 $1.9220
59.0920 10
91. 91.932011 12
35.9920 17
68.6N1o 03
51.4810
OR, XNORAIL (d)
27.0920.21
82 22:201
91.902o.10 59.44.6.07
36.88
94.692..a
52.02 20
OR, AND, XNORAIL (p)
87 44
81.78+0.0 06
79.27=c.1
91 .1.98=0.29
35.90-0.00
94. 94.66-0.0s
51.47+0.18
OR, AND, XNOR (d)
87.4920 13
82.50
59,8320 12
92.3720.05 05
37.60 20
94.72.0.02
02
DISCUSSION
In this work we motivated and introduced novel activation functions analogous to boolean operators in logit-space. We designed the AlL functions, fast approximates to the true logit-space functions equivalent to manipulating the corresponding probabilities, and demonstrated their effectiveness on a wide range of tasks.
Although our activation functions assume independence (which is generally approximately true for the pre-activation features learnt with 1d activation functions), found the network learnt to induce anti- correlations between features which were paired together by our activation functions (Appendix A0 This suggests that the assumption of independence is not essential to the performance of our proposed activation functions.
We found that the XNORAIL activation function was highly effective in the setting of shallow net- works. Meanwhile, the ORAIL activation function was highly effective for representation learning in the setting of deep ResNet architecture trained on images. In scenarios which involve manipulating high-level features extracted by an embedding network, we find that using an ensemble of AlL. activation functions together works best, and that the duplication ensembling strategy outperforms partitioning. In this work we have restricted ourselves only considering using single activation function (or ensemble) throughout the network, however our results together indicate that stronger re- sults may be found by using ORAIL. for feature extraction and an ensemble of (OR, AND, XNOR (d)) for later bigher-order reasoning layers within the network.
The idea we propose is nascent and there is great deal of scope for exploring other forms of activation functions that combine multiple pre-activation features by utilizing "higher-order activation functions".

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have shared code to run all experiments considered in this paper with the reviewers via an anony- mous download URL. The code base contains detailed instructions on how to setup each experiment, including downloading the datasets and installing environments with pinned dependencies. It should be possible to reproduce all our experimental results with this code. For the final version of the paper, we will make the code publicly available on an online repository and share a link to within the paper.
REFERENCES
Moray Allan and Christopher Williams. Harmonising chorales by probabilistic inference. In L. Saul, Y. Weiss, and I.. Bottou (eds.), Advances in Neural Information Processing Systems, volume 17. MIT Press, 2005.
Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation, 2019.
David G. T. Barrett, Felix Hill, Adam Santoro, Ari S. Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks, 2018.
Jonathan T. Barron. Continuously differentiable exponential linear units, 2017.
Yaniv Benny, Niv Pekar, and Lior Wolf. Scale-localized abstract reasoning, 2021.
Jock A. Blackard. Comparison of Neural Networks and Discriminant Analysis in Predicting Forest Cover Types. PhD thesis, Colorado State University, USA, 1998. AA19921979.
Jock A Blackard and Denis J Dean. Comparative accuracies of neural networks and discriminant analysis in predicting forest cover types from cartographic variables. Second Southern Forestry GIS Conference, pp. 189-199, 1998.
Nicolas
Yoshua Bengio, and Pascal Vincent. Modeling temporal dependen- cies in high- dimensional sequences: Application to polyphonic music generation and transcription, 2012.
Artem Chernodub and Dimitri Nowicki. Norm-preservee orthogonal permutation linear unit activa- tion functions (oplu), 2017.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus), 2016.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011, 1.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. CoRR, 0tbs/1805.09955 2018. URL htll:ttp://// //arxiv
org/abs/ 1805. 09501
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Weggoid-weighted linear units for neural network function approximation in reinforcement learning, 2017.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions pattern analysis and machine intelligence, 594-:1144611,, 2006.
François Fleuret, Ting Li, Charles Dubout, Emma K. Wampler, Steven Yantis, and Donald Ge- man. Comparing machines and humans on visual categorization test. Proceedings of the National Academy of Sciences, 108(43): URL http::/ttt/.te /tttpp::/www.... pnas org/ wtt.aaas...gccooteeeeeeeee www.eetttttt//330/
2011. ISSN 0027-8424. doi:
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):1933))))) 202. Apr
1980. ISSN 1432-0770. doi: 10. 10.00077/F00344255. URL http:////ooo.001/.10000
BF 8F00342251
10

---

Under review as a conference paper at ICLR 2022
Albert Gidon, Timothy Adam Zolnik, Pawel Fidzinski, Felix Bolduan, Athanasia Papoutsi, Panayiota Poirazi, Martin Holtkamp, Imre Vida, and Matthew Evan Larkum. Dendritic action poten- tials and computation in human layer 2/3 cortical neurons. Science, 367(6473):83 87, 2020. doi: 10. 126/sciemcc..axx6.399 URL htttps://wwe sci ence. o sc www.eneecax2..
Luke Godfrey and Michael S. Gashler. A parameterized activation function for learning fuzzy logic operations in deep neural networks. CoRR, abs/1708.08857) 2017. htt.htttp///ax..
org/abs/1708. 08557
lan Goodfellow,
David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1319-1327, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL htt.Ltttps://proceeiiiog m www.ppres/v2B/ggee http://v2/gooodeeeeeeeoeee 1 ht.l
Charles G. Gross. Genealogy of the "grandmother cell" The Neuroscientiss, 8(5):512-518, 2002. 2002. doi: 10.1 00.117/10735802237111 Utt.att..//ooooor////11/11033333
PMID: 12374433.
Gaëtan Hadjeres, François Pachet, and Frank Nielsen. Deepbach: a steerable model for bach chorales generation, 2017.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL httt:://pa//x org /a////////// 111//1111111 03385
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs), 2020.
Hermann Hild, Johannes Feulner, and Wolfram Menzel. Harmonet: A neural net for harmonizing chorales in the style of JS Bach. In Advances in neural information processing systems, pp. 267-274, 1992.
Sheng Hu, Yuqing Ma, Xianglong Liu, Yanlu Wei, and Shihao Bai. Stratified rule-aware network for abstract visual reasoning. 2020.
Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, and Douglas Eck. Coun- terpoint by convolution, 2019,
Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations in image collections. I Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1383-1391, 2015.
Kevin Jarrett, Koray Kavukcuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In 2009 JEEE /2th International Conference on Computer Vision, pp. 2146-2153, 2009. doi: 10.1 101100/...2009555554400
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
H M Dipu Kabir, Moloud Abdar, Seyed Mohammad Jafar Jalali, Abbas Khosravi, Amir F Atiya, Sacid Nahavandi, and Dipti Srinivasan. Spinalnet: Deep neural network with gradual input, 2020.
Daniel Kahneman. Thinking, fast and slow. Thinking, fast and slow. Farrar, Straus and Giroux. New York, NY. US, 2011. ISBN 0-374-27563-7 (Hardcover); 144299-6935-44 (PDF); 978-0-3888 978-0-374-27563-1 (Hardcover); 978- 118-114299--6 14299-6935-2 (PDF).
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks, 2015.
11

---

Under review as a conference paper at ICLR 2022
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Proceedings of the 3t international conference neural information processing systems, pp. 972 172-981,22 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227-22324 1998.
Feynman Liang. Bachbot: Automatic composition in the style of Bach chorales. University of Cambridge, 8:19-48, 2016.
Marvin Minsky and Seymour Papert. Perceptrons. Perceptrons. M.I.T. Press, Oxford, England, 1969.
Diganta Misra. Mish: self regularized non-monotonic neural activation function, 2019.
Michael C Mozer. Connectionist music composition based on melodic, stylistic, and psychophysical constraints. University of Colorado, Boulder, Department Computer Science, 1990.
Vinod Nair and Geoffrey Hinton. Rectified linear units improve restricted boltzmann machines. volume 27, pp. 807-814, 06 2010.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 1/391//diitlll.000000 https://distill.pub/// /fbb/20//ffauure-visaaiizzz
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 00.23915/distill000022
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Mare' Aurelio Ranzato. Task-driven modular networks for zero-shot compositional learning, 2019.
Quian Quiroga, Reddy, G Kreiman, Koch, and Fried. Invariant visual representation by single neurons in the human brain. Nature, 435:1 102-1107, 2005. doi: 10. 10.10///attr
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment, 2017
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions, 2017.
John C Raven and JH Court. Raven's 's progressive matrices. Western Psychological Services Los Angeles, CA, 1938.
Jason Rennie, Lawrence Shih, Jaime Teevan, and David Karger. Tackling the poor assumptions of naive bayes text classifiers. Proceedings of the Twentieth International Conference Machine Learning, 41, 07 2003.
Leslie N. Smith. A disciplined approach to neural network haper-parameters: Part learning rate, batch size, momentum, and weight decay. CoRR, abs/1803 009.009200 2018. URL http:
7/arxiv. www.ass.gaaaaa....o....o.....o.r 803. 000.0000000000000000000 09820
12

---

Under review as a conference paper at ICLR 2022
Leslie N. Smith and Nicholay Topin. Super-convergence:: Very fast training of residual networks using large learning rates. CoRR, abs/1708.07120, 2017. URL http:://axxxv...r//aaa..
1708.07120
UCI Machine Learning Repository, Covertype data set, Aug 1998. URL https://archive ics. uci www.m.dummm.. perduu/m/dattaettttss /oooss/tpeee
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. RAVEN: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 LINEAR RELU EXAMPLES
In Figure we show a representation what 2-d linear layer followed by the ReLU activation function looks like. The output is the same up rotation.
+4
+3
+3
2
+2
2
+2
+1
+1
0
+0
-2
-3
4
-4
4
-5
-4
2
-2
+6
2
+4
0
-2
6
-4
8
Figure 7: ReLU unit, and ReLU units followed by linear layer to try to approximate ORE, leaving a dead space where the negative logits should be.
we apply second linear layer on top of the outputs of the first two units, can try approximate the logit AND or OR function. However, the solution using ReLU leaves quadrant of the output space hollowed out as zero due its behaviour at truncating away information.
A.2 SOLVING XOR
A long-standing criticism of artificial neural networks is their inability to solve XOR with single layer (Minsky & Papert 1969). Of course, adding single hidden layer allows a network using ReLU solve XOR. However, the way that it solves the problem is to join two of the disconnected regions together in stripe (see Figure 8). Mcanwhile, our XNORAIL is trivial able to solve the XOR problem without any hidden layers. For comparison here, we include one hidden layer with units for each network. Including a layer before the activation function makes the task harder for XNORAIL which must learn how project the input space order to compute the desired separation. Also, including the linear layer allows the network to generalise to rotations and offset versions of the task.
14

---

Under review as a conference paper at ICLR 2022
10
08
0.8
-1
02
02
-2
-2
0.0
0.0
-2
2
(a) ReLU
(b) XNORAAI
Figure 8: Solving XOR with a single hidden layer of 2 units, using either ReLU or XNORAIL activation. Circles indicate negative (blue) and positive (red) training samples. The heatmaps indicate the output probabilities of the two networks.
A.3 LINEAR LAYERS AND BAYES' RULE IN LOG-ODDS FORM
Bayes' Theorem or Bayes' Rule is given by
P(H P(X P(H) P(X)
(8)
In this case, update the probability of our hypothesis, H, based the event of the observation of a new piece of evidence, X. Our prior belief for the hypothesis is P(H). and posterior is PH (X). To update our belief from the prior and yield the posterior, we multiply by the Bayes factor for the evidence which is given by P(X(M)/P(x).
Converting the probabilities into log-odds ratios (logits) yields the following representation of Bayes' Rule.
P(H) P(HI)) log ((H))) )
P(X + log P(
log P(HOIX)
(9)
Here, HC is the complement H (the event that the hypothesis is false), and P(HO) P(H). Our prior log-odds ratio is log ((P(H)/(()))) and our posterior after updating based on the observation of new evidence X is log (P(=)))//((E()))) To update our belief from the prior and yield the posterior, we add the log-odds Bayes factor for the evidence which is given by log (P(X(1))/((x18)))
In log-odds space, a series of updates with multiple pieces of independent evidence can be performed once with summation operation.
x)) P(HC ())
P(H) P(HC) )
P(X, H P(X,(HO)
log
log
Σ log
(10)
This is operation can represented by the linear layer an artificial neural network, 2 = by + w Here, the bias term by log ((P(H)/P(H*)) is the prior for hypothesis (the presence of the feature represented by the k-th neuron), and the series of weighted inputs from the previous layer, Was T; provide evidential updates. This is also equivalent to the operation of multinomial naïve Bayes classifier, expressed in log-space, if we choose Wi: = log Pke (Rennie et al., 2003).
A.4 DIFFERENCE BETWEEN AIL AND IL FUNCTIONS
Here, measure and show the difference between the true logit-space operations and our AlL approximations, shown Figure 9 Figure 10 and Figure 11
In each case, observe that the magnitude of the difference is never more than 1, which occurs along the boundary lines in AIL. Since the magnitude of the three functions increase as we move away from the origin, the relative difference decreases in magnitude as the size of a and y increase.
15

---

Under review as a conference paper at ICLR 2022
AND
ANDAK
+10
+8
+2
0
+0
-2
-2
-4
-10
-4
-2
2
4
4
-2
0
2
AND AND
(AND,. ANDAR) / |ANDal
+0.7
40.5
+0.3
+0.1
0.1
0.3
4
-0.9
2
4
2
Figure 9: Heatmaps showing ANDIL AND AIL,
AIL. their difference, and their relative difference.
ORg
ORwl
+10
48
+6
+2
+0
-2
-2
4
-4
-10
-4
-2
2
4
-2
0
2
ORa ORRu
(OR ORau) |ORa]
+0.9
+0.7
+0.5
+0.3
40.1
0.1
4
009
-2
4
4
2
Figure 10: Heatmaps showing ORI, ORAIL. their difference, and their relative difference.
16

---

Under review as a conference paper at ICLR 2022
XNOR
XNORAL
+10
+8
+6
+2
+0
-2
-2
-4
110
-4
-2
2
4
4
2
0
2
XNORL XNORAR
(XNORL XNORw) / [
+0.9
+0.7
2
+0.5
+0.3
+0.1
-0.1
-4
-0.9
-2
-4
Figure 11: Heatmaps showing XNOR XNORAIL, their difference, and their relative difference.
17

---

Under review as a conference paper at ICLR 2022
A.5 GRADIENT AIL AND IL FUNCTIONS
We show the gradient of each of the logit-space boolean operators and their AIL approximates in Figure 12 Figure 13 and Figure 14 By the symmetry of each of the functions, the derivative with respect to is a reflected copy of the gradient with respect to æ.
We find that the gradient of each AIL function closely matches that of the exact form. Whilst there are "dead" regions where the gradient is zero, this only occurs for one of the derivatives at a time (there is always a gradient with respect least one of and y).
d/dx AND
d/dx ANDAI
+0.9
+0.7
+0.5
+0.3
0.3
0.5
-07
00.9
-4
-2
2
4
-4
-2
2
4
d/dy AND
d/dy ANDAIL
40.9
+0.7
2
+0.1
0.1
0.3
0.5
-0.9
-2
2
Figure 12: Heatmaps showing the gradient with respect and y of ANDIL and AND AIL-
18

---

Under review as a conference paper at ICLR 2022
d/dx OR2
d/dx ORai
40.9
+0.7
+0.5
+0.3
0.3
0.5
-0.7
-0.9
-4
-2
4
-
-2
2
d/dy ORe
d/dy ORai
+0.9
+0.7
+0.1
0.1
0.3
-2
0.5
-4
-09
-2
2
Figure 13: Heatmaps showing the gradient with respect to and y of ORR ORAIL-
d/dx XNOR=
d/dx XNORAK
+0.9
+0.7
+0.5
+0.3
-03
-2
0.5
0.7
0.9
-4
-2
4
4
-2
2
d/dy XNORg,
d/dy XNORAK
40.9
+0.7
+0.1
0.1
-0.
-0.5
-
-2
4
2
Figure 14: Heatmaps showing the gradient with respect and of XNORn, xnor XNORAIL.
A.6 DATASET SUMMARY
The datasets used in this work are summarised in Table 2
We used the same splits for 1altech101 used in Kabir al. 2020
19

---

Under review as a conference paper at ICLR 2022
Table 2: Dataset summaries.
No Samples
Dataset
Train
Test
Classes
Reference
Bach Chorales
229
77
2
Boulanger- Lewandowwki et al. (0012)
Caltech101
6162
1695
101
Fei-F'rr el al
(0006)
CIFAR-10 10
00000
10 000
10
Krizhessky
2009
CIFAR- 100
000000
100000
100
Knzzevsky
2009)
Covertype
864810
116 202
7
Blackard
1998%
Blackard & Dean 1998
I-RAVEN
6000
2000
Hu ct al.
2020
MTT-States
30838
12 995
245 obj. 115 attr
Isola ct al.
2015
MNIST
600000
10
10
LcCun (1998)
Oxford Flowers
6552
818
102
Nilsback & Zisserman
2008
Stanford Cars
8144
8041
196
Krause et al.
(2013)
STL-10
5000
8000
10
Coates al.
2011
SVHN
73 257
26 032
10
Netzer al.
2011
The Covertype dataset was chosen as a dataset that contains only simple features (and not pixels of an image) which could study simple MLP architecture, and selected based on its popularity on the UCI MI. reposiorr.
The Bach Chorales dataset chosen because in addition to being in continued use by ML researchers for decades - presents an interesting opportunity to consider a task where logical activation functions are intuitively applicable, as it is a relatively small dataset that has also been approached with rule-based frameworks, e.g. the expert system by Ebcioglu (1988).
MNIST, CIFAR-10, CIFAR-100 are standard image datasets, commonly used. We used small MLP and CNN architectures for the experiments on MNIST so we could investigate the performance of the network for many configurations (varying the size of the network). We used ResNett50,, very common deep convolutional architecture within the computer vision field, on CIFAR-10/100 to evaluate the performance in the context of a deep network.
The datasets used for the transfer learning task are all common and popular natural image datasets, with some containing coarse- coarse-grained classification (CIFAR- -OFAR--0),, others fine-gred grainer (Stanford Cars), and with varying dataset size 15000-75000 training samples). We chose to do an experiment involving transfer learning because it is a common practical situation where one must train only a small network that handles high-level features, and is the sort of situation which involves manipulating high-level features, relying on the pretrained network to do the feature extraction.
We considered other domains where logical reasoning is involved component of the task, and isolated abstract reasoning and compositional zero-shot learning as suitable tasks.
For abstract reasoning, we wanted to an IQ style test, and determined that I-RAVEN was state-of-te the-art dataset within this domain (having fixed some problems with the pre-existing RAVEN dataset). We determined that the SRAN architecture from the paper which introduced I-RAVEN was still state-of-the-art on this task, and so used this.
Another problem domain in which we thought it would be interesting study our activation functions was compositional zero-shot learning (CZSL). This is because the task inherently involves combining an attribute with object (i.e. the AND operation). For CZSL, we looked at SOTA methods on The best performance was from SymNet, but this was only implemented in TensorFlow and our code was set up in PyTorch so we built our experiments on the back of the second-best instead, which is the TMN architecture. In the TMN paper, they used two datasets: MTT-States and UT-Zappos- overfitting on MIT-States after around 6 epochs, but on UT-Zappos-1 it was overfitting after the first or second epoch (one can not tell beyond the fact the val performance is best for the first epoch). In the context of zero-shot learning, number tasks in the dataset. Because there are multiple samples for each noun/adjective pair, and each noun only appears with a handful of adjectives and vice versa, there is in way fewer tasks in one epoch than there are images. Hence it is possible for zero-shot learning model to overfit to the training tasks in less than one epoch (recall that the network includes pretrained ResNet model
https:/paaerrswitccdeeccom
our preliminary experiments, we found that the network started
epoch uses every image once, but there are also only a finite
20

---

Under review as a conference paper at ICLR 2022
for extracting features from the images). For simplicity, we dropped UT-Zappos-I and focused on MIT-States.
A.7 PARITY EXPERIMENTS
Following on from the parity experiment described in the main text (Section 3.1). we also introduced a second synthetic dataset with a labelling function that, while slightly more complex than the first, was still solvable by applying the logical XNOR operation to the network inputs. In this dataset increased our number of inputs to 8, and derived our labels by applying a set of nested XNOR operations:
(XNORI (XNORIL (2ng, .ry) XNORIL XNORRI (333 ENORI((NOOI (To, .7), XNORR (2o, x1)).
For this more difficult task also reformulated our initial experiment into regression problem, as the continuous targets produced by this labelling function are more informative than the rounded binary targets used in the first experiment. We also adjusted our network setup have an equal number of neurons at each hidden layer as we found that this significantly improved model performance 2 We again trained using the same model syper-parametten for 100 epochs.
While this time the model was not able to Iearn a sparse weight matrix that exactly reflected our labelling function (see Figure 15), the model was again able leverage the XNORAIL activation function to significantly outperform an identical model utilizing the ReLU activation function.
Outputs
Ourputs
Layer
Activations
Activations
Hidden Layer Inputs
Hidden Layer 3 Imputs
Hidden Layer Activations
Hidden Layer Activations
Hidden Layer 2 Inputs
Hidden Layer Impues
Hidden Layer Activations
Hidden Layer
Activations
Layer
Layer Imputs
Ieputs
(a) XNORAIL learned weight matrix
(b) ReLU learned weight matrix
Figure 15: Training results, regression experiment on second synthetic dataset
We found that a simple model with three hidden layers, each with eight neurons, utilizing XNORAIL was able to go from a validation RMSE of 0.287 the beginning of training to validation RMSE of 0.016 after 100 epochs. Comparatively, an identical model utilizing the ReLU activation function was only able to achieve a validation RMSE of 0.271 after 100 epochs. In order for our ReLU network to match the validation RMSE of our 8-neuron-pers lay-eerr XN XNORAIL model, we had to increase the model size by 32 times to 256 neurons at each hidden layer.
We hypothesize that, because our XNORAIL activation function reduces the number of hidden layer neurons by factor of k having reduced number of neurons each layer creates a bottleneck in the later layers of the network which restricts the amount of information that made its way through the final layer
21

---

Under review as a conference paper at ICLR 2022
Cavertype, MLP (1 hidden layer)
Covertype, MLP hidden layers)
Covertype, MLP (3 hidden layers)
97
94
96
Text accurmayyy 6%
90
rignad
Tiest
94
-44
88
UR. AND (D)
OR. XNOR (p)
OOL XNOO 10.
86
OR. AND 2200. 13)
09. AND. XNOG (4)
104 Number of parameters
10F
1
10
107
92
104
10f Numbor of parmettrs
10a
107
Number of porameters
Figure 16: We trained MLPs the Covertype dataset, with fixed 80:20 random split. Trained with Adam, 50 ep., 1 I-cycle, using LRs determined automatically with LR-finder. Mean (bars: std dev) of n 10 weight inits.
A.8 COVERTYPE
Our one, two, and three-layer MLPs were trained using I-cycle (Smith & Topin 2017 Smith 2018) for 50 epochs, with a batch size of 1024, without weight decay. We used a subset 15% of the training data for validation. The learning rate was selected using an automated learning rate finder approach, verified against the validation partition. For our final experiments, the validation partition was included in the training data. No data augmentation was performed.
A.9 BACH CHORALE TRAINING DETAILS
For each of the 4 voices, we restricted the available pitches to 3 octaves, resulting in 37 one-hot tokens (including silence). Since we only planned on feeding small time-windows the chorale into model, for pre- pre-processing, converted the data into shape of (seq_len, 4, 37), where set seq_len to 4.
To generate training examples for the discriminator, transposed by {-5, .4,.., 5, 6} semitones, chosen uniformly at random, and there was a 0.5 probability that the sample was corrupted by the following method:
Choose 2-3 notes in the (seq-len x 4) window to be corrupted
For each note, corrupt by the following mixture distribution:
(p 0.6) Sample a pitch from Gaussian centered the existing note, with
3 semitones, forcing the new pitch to be distinct
(p 0.2) Copy pitch from the current voice, forcing the new pitch to be distinct
(p 0.2) Extend the previous note in time
- (p 0.1) Sample uniformly from all 37 possible tokens
A.10 CORRELATIONS BETWEEN PRE- ACTIVATIOS
Results on correlations between weights in the JSB Chorale models are shown in Figure 18 We found that when taking all pre-activations into account, every activation function generally showed independence between features. Interestingly, the cosine similarities between inputs that were paired together for the bivariate activation functions showed anticorrelation almost all cases where Max or ORAIL were used, and other cases generally showed more correlation than ReLU.
We found that randomly selected pairs of preactivation features within the same layer have correlations that are given by Gaussian-like distribution centered around zero. This was the case for all of the activation functions we tested. The behaviour of randomly selected pairs features is thus reasonably consistent with the assumption of independence which have made. We also investigated the correlation between the pairs of preactivation features which were passing into our wwo-dimensional activation functions. Here, found the correlation structure is different, and the correlation depends
22

---

Under review as a conference paper at ICLR 2022
Chorales, MLP (2 hidden layers)
92
Tast accuracy (9)
90
10
10
10'
Number of paramsters
Figure 17: We trained 2-layer MLPs discriminators on JSB Chorales using Adam (constant LR 1 x 10-3 150 ep.), Mean (bars: std dev) of n = 10 weight inits.
Cosina similrties between preastivations (1st layer)
Max
Mac. Min
agrec geatwar
30VDRax
-1.0
-0.5
0.0
66
10
-05
0.0
05
1-
-1.
-0.5
00
0.5
-10
-0.
0/9
0.5
-1.
-25
08
65
-10
-65
88
65
Offic AND (1)
DM XMORAA (6)
Offi. XNON (6)
ORaa AND XND 150
ORfa. AND XN (4)
-1.0
-05
0.0
85
10
-1.
-85
0.0
05
14
-1.
-0.
60
0.5
-1.0
-25
0.3
0.5
-1.
-25
0.
85
Cosino similartis belween paired preactivations ayer)
SeLUI
Max
Mic. Min
agyeed SwiTwar
XXORRax
ORA
-1.0
-0.5
0.5
05
1.0
-1.0
-05
0.0
055
1.0
+1.0
-0.5
80
0.5
+10
-23
03
03
-1.
-25
03
85
-18
-05
Offic. ANDau (1)
DRfas XNOHRA (b)
Dffis XNDRRa (5)
GRaa AND XNDR (p)
ORta AN XND (4)
-1.0
-355
0.5
8.
1.0
1.0
-05
0.0
0.5
1.0
1..
-0.
60
0.3
-1.0
-23
00
03
10
-1.
-25
23
25
1.2
Figure 18: Cosine similarities between pre-activation weights of two activation functions in the first layer of MLP trained on JSB Chorales.
on the activation function being used. With Max and ORAIL. activations, the network learns to make the columns of the weight matrix (and hence the preactivation scores for the pair of features) be inversely correlated. With XNORAIL. the network learns features which are either positively or negatively correlated (a wider distribution of correlations than seen with random pairs features). We observe that (in all cases) the network learns to make the features passed to the AIL activation functions be correlated instead of independent, despite our assumption of independence. So it appears clear that the assumption of independence is violated, but also that it doesn't 't really matter because the network choosing to break the assumption and induce these correlations between the features get better performance.
23

---

Under review as a conference paper at ICLR 2022
A.11 CNN AND MLP ON MNIST
In this experiment trained MLP and CNN models for 10 epochs the MNIST dataset using Adam optimizer, one-cycle learning rate schedule, and cross entropy loss, with batch size of 256. We augmented our training samples using random affine transformation from: rotation of +0 degrees, scale of factor 0.8 to 1.2, translation with max absolute fraction for horizontal and vertical directions of 0.08, and shear parallel to the x-axes of +0.3.
for the optimizer and scheduler were selected through a random search of the The hyper-parameter space. We chose to do random search instead of grid search because it typically
yields better results for the same number of test cases.
For the hyper- parameter search, we trained on the first 50 000 samples of the training partition and used the final 10 000 samples as validation set. We ran the search for four iterations, with each iteration sampling 120 different hyper- paremmerreetrr settings. The initial bounds for our hyper-paramett samples were set as described in [Table 3] These bounds were chosen to be suitably wide such that the optimal configuration should be contained within them for all activation functions considered.
Table 3: Hyper- param-- random search parameters.
Plyper-parameter
Variable
Sampling (initial bounds)
Adam betal
10"
x Uniform( -3. - 0.5)
Adam beta2
10"
x Uniform( -5--1)
Adam epsilon
10a
x Uniform( -10, -6)
Weight decay
10"
x ~ Uniform( -7.
One-cycle max LR
10"
x Uniformó (-06rrmm-440))) -4.0)
One-cycle peak
x (alform(0.11...
The bounds on our uniform random variable were tightened at each iteration by selecting the top-5 performing aperrpparaeeter settings, taking the mean E and weighted standard deviation across those settings, and re-setting the bounds for the next iteration to be equal to F a.5 After the fourth iteration, we ran a final iteration where we selected the top-10 10 performing hyper-parameter settings across the four previous iterations, and then re-ran these for another 120 seeds (i.e. random weight initializations). We then selected the hyper-paramete settings which had the highest performance across all 120 seeds.
We found that the XNOR activation function required quite different dyper-parameee than the other activation functions, but OR and Max activation functions used similar parraepprssesers to ReLU. However, we have reason to believe that the proposed AlL activation functions are more susceptible than others to the choice of selected.
or the way in which sppeer-prrmmters are
Our MLP model consisted of two hidden layers of equal size with batch norm applied to each. The number of neurons in each layer was set, taking into consideration the current activation function being tested, to ensure the number of trainable parameters in the network remained static across experiments
Our CNN model consisted of six layers, where each layer was comprised of a set of 2D convolution filters (kernel size 3, stride padding 1), batch normalization, and a non-linear activation. The network also applied a pooling layer (kernel size 2, stride 2) at the end of every second layer. After the 6 CNN layers the network flattens the output and applies three linear layers. The number of output channels (ie. pre-activations) for the six 2D convolutions were [c. 2c, 4c, 4c, 8c, 8c]. and the number of pre-activation neurons produced by the subsequent linear layers were [32c, 16c]. Similar to the MLP model, was chosen ensure the number of trainable parameters in the network remained fixed across experiments. When varying the number of network parameters in our experiments, if the number of parameters dropped below 1 x 106 in the CNN model, we changed the structure of the convolution layers and linear layers to [c, c, c, c, 1.5c, 1.5c] and [2c, 1.5c), respectively. This ensured that each layer had at least two channels for our activation functions to aggregate.
Once our optimal syper-parametere were selected for both our MLP and CNN models, we then trained each model several times with varying number of trainable parameters. The reason we vary
24

---

Under review as a conference paper at ICLR 2022
the number of parameters in the network instead of the network size/structure is because network using our logical activation functions can have a significantly different number of parameters than an identical network using ReLU activation, because our logical activations aggregate across neurons at each layer.
A.12 RESNET5O ON CIFAR-10/100
For this experiment we trained a ResNet50 model for 100 epochs on CIFAR-10 and CIFAR-100 using Adam optimizer, one-cycle learning rate schedule, cross entropy loss, and augmentations derived for CIFAR-10 by AutoAugment (Cubuk al. 2018). with batch size of 128. The hyper-parameters for the optimizer and scheduler were determined through a random search of the hyper parameter space on the CIFAR- 100 datase The hyper--arameee search was tuned against partition of 10% of the CIFAR- 100 training samples. Our hyper-parameter search follows a similar approach to the one used for our MLP and CNN models on MNIST (Appendix A. but due compute constraints with our larger models here we only trained 100 seeds each iteration, and only re-examined the top 10 seeds in the final round.
For PReLU, we did not perform a hyper-parameter search and merely re-used the same hyper- parameters as discovered in the search with ReLU.
Once our optimal pyppee-paametees were selected we trained a ResNet50 model with automatic mixed precision using four different width parameters: 0.5, 1, 2, and Because activation func- tions such as XNORAIL reduce the number of post-activation neurons by a factor of 2, and the {OR, AND, XNORAIL (d)) activation function increases the number of post-activation neurons by factor of 3/2, we adjust the width parameter for these activation function to result in all networks having approximately the same number of parameters. For the 2 - 1 activation functions (Max, ORAIL. XNORAIL {OR, XNOR (p)), and {OR, AND, XNOR ATI. (p))) we use width values of 0.75, 1.5, 3, and 6. For the 2 2+ +3 activation function {OR, AND, XNORAIL (d)), we use width values of 0.4, 0.75, 1.5, and 3.
A.13 TRANSFER LEARNING
We used pretrained ResNet50 model taken from the pytorch hub. The 2-layer ML.P head was trained for 25 epochs on each dataset. Since we are interested in transfer learning in data-limited regime for these experiments, we used only small amount of data augmentation. We applied horizontal flip 0.5), scaling (0.7 to 1), aspect ratio stretching (5/1 to 4/3), and colour jitter (intensity 0.4) only. Pixel intensity normalization was done against the ImageNet mean and std.
The MLP head was optimized using SGD, momentum 0.9, We used a batch size of 128, maximum learning rate 0.01, and weight decay 1 x 10 - Before training began, we passed one epoch worth of inputs through the network without updating the weights in order to refresh the batch normalization statistics to the new dataset. We also performed one epoch of training with a warmup learning rate of 1 x 10-5 before commencing training at the maximum learning rate of 0.01. The learning rate was decayed with cosine annealing schedule over 24 epochs. We report the performance of the final model the end of the 25 epochs.
We used a pre-activation width of 512 neurons for ReLU and other and activation functions which map 1 + 1. To approximately match up the number of parameters, we used width of 650 for activation functions which map 2-11 and 438 for {OR, AND, XNORAIL (d)} which maps features 3-+2. These values control the total number of parameters in the head to be the same for datasets with 100 classes (the median number of classes in the datasets we considered). Our results with widths 138/512//500 and 6600 parameters are shown Table The precise widths and number of trainable parameters used for these experiments are shown in Table 4
Since the transfer learning experiments were performed without retraining the base network, we found (Table 1) the performance is limited by the features which the base network produces, whose relevance depends on the other lap between the domain of the pretraining task (ImageNet) and the new task. This information bottleneck, and variation in its utility, means the variation between
We used the same set of hyper-parameters from this search for both the CIFAR-10 and CIFAR-100 experiments.
25

---

Under review as a conference paper at ICLR 2022
performance different datasets is much larger than the variation for different activation functions within the same dataset.
For transfer tasks which involve coarse-grained discrimination images which are similar to ImageNet, the embedding generated by the pretrained model is already sufficient to separate the classes in the new dataset. Examples of this are Caltech101, where linear layer beats out using additional layers with non-linearities; and STL-100, where our best model is on-par with the linear model. The results these two transfer learning tasks are simple be of interest to study, The performance is limited by the information retained by the pretrained embedding, but it appears that what task relevant information is there readily available with a linear layer without needing additional logic to interpret it.
Other transfer learning tasks we attempted less trivial and show larger difference in performance across the activation functions, and gap from the linear layer readout model. The Stanford Cars dataset involves fine-grained discrimination between different car models, for which features generated by model pretrained ImageNet are not effective. The SVHN dataset, which contains images house numbers, coarse-grained ImageNet, which makes the transfer-learning task more difficult. Additionally, our use of random horizontal reflections during training will have hindered performance on this dataset by notably increasing the task complexity, since numerals are not invariant under reflection.
uses images which are outside the domain of
Table 4: Hidden pre-activation widths and number of trainable parameters used in transfer learning experiments (corresponding to results shown in Table
No Trainable Parameters
Activation function
Map
Width
Caltech101
CIFARI0
CIFARI00
Flowers
Cars
STL-10
SVHN
Linear layer only
52k
5k
51k
52k
101k
5k
ReLU
1-11
512
577k
530k
577k
578k
626k
530k
530k
LeakyReLUU
577k
530k
PReLU
1-+1
512
577k
530k
577k
578k
626k
530k
530k
Softplus
1-+1
512
577k
530k
577k
578k
626k
530k
530k
ELU
1-+1
512
577k
530k
S77k
578k
626k
S30k
530k
CELU
1-+1
577k
530k
577k
626k
530k
530k
SELU
1-11
512
577k
530k
577k
578k
626k
530k
530k
GELU
1-1
512
577k
530k
S77k
578k
626k
S30k
530k
SiLU
1-+1
577k
530k
S77k
578k
626k
530k
530k
Hardswish
1-11
512
577k
530k
577K
578k
626k
530k
530k
Mish
1-+1
512
577k
530k
577k
578k
626k
530k
530k
Softsign
1 1-+1
512
577k
530k
577k
578k
626k
530k
530k
Tanh
1-1
512
577k
530k
577k
578k
626k
530k
530k
GLU
2-+1
650
578k
549k
578k
579k
609k
549k
549k
Max
2-+1
650
578k
549k
578k
579k
609k
549k
549k
Max, (d)
577k
530k
XNORAIL
2++1
578k
549k
578k
549%
549K
ORAI..
2-+1
650
578k
549k
578k
579k
609
549k
549k
OR, ANDAIL (d)
1-+1
512
577k
530k
S77k
578k
626k
530k
530k
OR. XNORAIL (p)
2-+1
578k
549k
578k
579k
609k
549%
549k
OR, XNORAR. (d)
1-+1
512
577k
530k
577k
578k
626k
530k
530k
OR, AND, XNORAIL (p)
2-+1
650
578k
549K
578k
579k
609k
$49k
549k
OR.AND, XNORAH. (d)
2-33
438
579k
519k
S79k
580k
642k
519%
519k
Additionally, we ran the experiment again with a width of w = 512 for the 2 activation functions.
The results with constant pre-activation width 512 shown in Table 5] By using constant pre-activation width, the number of parameters used is not consistent across activation functions. The number of trainable parameters in each experiment is shown in lab.transfer-mpara--512.
We found that reducing the width from 650 to 512 (and hence reducing total number trainable param- eters) reduced the performance of 2-+1 activation functions XNOR ORAII. {OR, XNOR (p)). and (OR, AND, XNOR (p)). Increasing the width from 438 to 512 increased the perfor- mance of (OR, AND, XNOR (d)).
26

---

Under review as a conference paper at ICLR 2022
Table Transfer learning with from a frozen ResNet-18 architecture pretrained on ImageNet- to other computer vision datasets. As with Table but in this case we show results where the MLP has pre-activation width of 512. Note that although the pre-activation width is constant, the number of parameters in the network is not consistent between experiments. The number of parameters is shown in Table
Mean (standard error) of inits of the MLP (same pretrained network).
Test Accuracy (%)
Activation function
Caltech101
CIFARIO
CIFAR100
Flowers
Cars
STL-10
SVHN
Linear layer only
78.5610.
27.3920.09
92.32
33.51.0.0
45.42.0
86 26.5820
81.6320
58.0420.11
90.71
30.97 a6
94.62.0
51.399
LeakyReLLU
86.6020.13
81.6710.11
58.0120.09
90.7320.52
31.09.0.21
94.612o.0s
51.4010.0
PRcLU
87.3 27.83-0.22 21
81.03+0.15
58 18
93.7 +0.19
39.34-0.15
94.54-0.0
11.4220..0
Softlus
86. 86.1020.1 18
79. 13.1322oom
56.5820.07
89.3920.29
21.232013
94.6320.0
47.44=0
ELU
87.18=0.00
80.44+0.0 as
58.08=0.10
91.71+0.14
34.70=o..0 as
94.55=o.05
50.07
CELU
87.1820
80,4410.0
58,08
94
34.70=0
94.55+0.00
50.07
SELU
87. N7.74.0.00
79.9310.15
58.2420.06
92.27.0 15
$7.5 10.1
94.5320.0r
49.38 2.
GELU
87.10
81.39
58.51=0.15
91.51+0.15
33.43=0.15
94.62=0.08
51.56=o.m
SiLU
86.91=0.11
80.53+0.11
18
91.37=0.1
32.15=0.17
91.59=0.05
50.69
Hardswish
87.12=0.12
10.0000.10
58.25e0.10
91.56+0.25
33.17=0.29
94.62=0.05
50.09=0.09
Mish
87.11
81.0920.13
58.37e0.10 10
91.61=0.1 15
33.7522014
94.61=0.05
51. 31.29920
Softsign
81.4720.18
80.0310 oe
54.8420.00
82.3410 22
17.3320.10
94.70_o.as
49. 49.48..0.0
Tanh
87.4820.06
80.5620.07
57.3520.05
90.32:0.00
29.5120.12
94.6320.07
50.1520.00
86.3420.16
79.3510.0
57.2220.12
90.1010.0
26.7220.15
94.63g0.05
48.51
Max
86.80 08.6620.11
81,5620
58, 58.1220.10
90.59
32.80=0.0
04.6520.05
51.15
Max, Min
87 67.2310
12.3/
59.0520.10
91.6810
34.91
94.6420.0
51.7220.0
XNORAIL
86 18.422011
81.74+0
57, 77,88
90.50
31.55=0
94.78=0.0z
51.23+0:10
ORAIL.
87.2880.00
81.811000
58.6810.0
91.7810
$5.28 on
90.6720.07
51.27
OR, AND (d)
87.43
$2,38
92.07 18
37. 16-01
94.3620.05
52.77
OR, XNORAIL (p)
87 26.26..00
81 8680
58, 68.7220.00
91
35. 25.27=000
64.6..000
51.25=0
OR, XNORAI. (d)
87.092o.2
82.21 82.201
59.44
91.9010
36.88
94.6920.00
52//2
OR, AND, XNOR (p)
87.0320.19
81.7520.0 07
58.722b.0s
91 01.8520.14 14
35.46.0.14
94.6920.04
51.3220.00
OR, AND, XNOR (d)
87 26
02.4520.00
60.20
$2.11
37.89=0.1
94.65=0.04
$2.7950.0
Table 6: Number of trainable parameters used in transfer learning experiments (corresponding to results shown in Table 5).
Trainahle Parameters
Activation function
Map
Width
Caltech101
CIFARIO
CIFARI00
Flowers
STL-10
SVHN
Linear layer only
52k
Sk
51k
52k
101k
5k
Sk
ReLU
1-+1
512
577k
530k
577k
578k
626k
530k
530k
LeakyReLU
1-1
512
577k
530k
577k
578k
626k
530k
530k
PReLU
1-11
577k
530k
577k
578k
626k
530k
530k
Softplus
1-+1
512
577k
530k
577k
578k
626k
530k
530k
ELU
1-+1
512
577k
530k
577k
578k
626k
S30k
530k
CELU
1-+1
512
577k
530k
577k
578k
626k
530k
530k
SELU
1-1
577k
530k
S77k
578k
626k
530k
530k
GELU
577k
530k
S77k
578k
626k
530k
530k
SiLU
1-+1
512
577k
530k
577k
578k
626k
530k
530k
Hardswish
1-11
577k
530k
577k
578k
626k
530k
530k
Mish
577k
530k
S77k
578k
626k
530k
530k
Softsign
577k
530k
S77k
530k
530k
Tanh
1-11
512
577k
530k
577k
578k
626k
530k
530k
GLU
2-+1
512
420k
397k
420k
420k
445k
397k
397k
Max
2-+1
512
420k
397k
420k
420k
445k
397k
397k
Max, Min (d)
1-+1
512
577k
530k
577k
578k
626k
530k
530k
XNORAIL
2++1
512
420k
397k
420k
420k
445k
397k
397k
OR
2-+1
420k
397k
420k
420k
445k
397k
OR, AND (d)
1-11
512
577k
530k
S77k
578K
626k
530k
530k
OR, XNORAIL (p)
2-+1
512
420k
397k
420k
420k
445k
397k
397k
OR, XNORAiL (d)
1-+1
512
577k
530k
S77k
578k
626k
.30k
530k
OR. AND, XNORAIL (p)
2-+1
420k
397k
420k:
420k
397k
397k
OR. AND, XNORAnL (d)
2-+3
512
734k
664k
733k
735k
807k
6644
664k
27

---

Under review as a conference paper at ICLR 2022
A.14 ABSTRACT REASONING
Abstract reasoning is challenging for neural networks to learn because their structure and objective function are more effective for tasks which come instinctively humans ("System I" of Kahneman 201T), such as object recognition, as opposed to demanding ("System 2") logic tasks.
In recent years, several challenges have been proposed evaluate the ability of neural networks to perform abstract reasoning. The Raven's Progressive Matrices (Raven & Court, 1938) are long-standing IQ test, which have been emulated by BBarrttt al. (2018) with Procedurally Generated Matrices (PGM). and then the RAVEN task by (Zhang et al. 2019). Recently Hu et al. (020)) and Benny al. (202T) have improved on that task with I-RAVEN and RAVEN-FAIR, respectively, both aiming to make the task more balanced. Other abstract reasoning tasks have also been proposed
(Fleuret et al. 201 Johnson et al. 2017 Barrett al. 2018).
We considered the application of AlL activation functions in the context of the I-RAVEN task, by adapting the Stratified Rule-Aware Network (SRAN) of Hu et al. (2020) to include our AIL activation functions. We first added LayerNorm to the network, and then swapped out the seven ReLU activations the gating module. The architecture for the three ResNet-18 18 (He et al. 2015) base models were unchanged. Where necessary, the number of units per layer modified to facilitate the change in dimensionality caused by our activation function. The networks were trained using the same procedure described by Hu al. 2020)).
Table 7: Performance of SRAN-based models on the I-RAVEN dataset (Hu et al. 2020). Bold: best. Underlined: second best. Background: color scale from worst in to best, linear with accuracy value.
I-RAVEN Test Acc (%)
Activation function
Params
Acc
Center
2x2G
3x3G
O-IC
O-IG
L-RR
U-D
ReLU, Base SRAN (Hu et al. 2020
44.0M
60.8
78.2
50.1
42.4
68.2
46.3
70.1
70.3
ReLU, SRAN=LayeiNor PReIUU
45.6M
63.0
84.0
50.0
42.5
60.9
48.3
73.5
72.3
45.6M
54.5
69.2
45.4
40.5
64.4
47.6
57.5
57.3
CELU
45.6M
56.6
73.5
16.5
41.0
66.6
46.1
62.8
59.7
SELU
45.6M
53.5
67.2
43.8
40.1
62.8
44.5
58.0
58.1
GELU
45.6M
61.4
80.8
49.2
42.5
60.2
48.3
70.3
69.2
SiLU
45.6M
59.4
78.0
47.2
41.4
66.3
47.8
69.1
66.2
Max
44.6M
57.8
76.3
45.2
39.7
65.3
48.7
64.6
64.7
Max, Min (d)
45.6M
60.2
80.2
49.5
41.9
66.5
47.0
68.5
67.7
XNORAIL
44.6M
57.7
74.7
46.0
40.0
60.8
47.7
65.6
63.2
ORAII.
44.6M
64.3
84.4
19.5
44.0
71.5
47.1
76.5
77.0
OR, ANDAIL (d)
45.6M
57.5
74.7
45.6
41.3
68.3
44.9
64.5
63.0
OR, XNORAn. (p)
44.6M
59.8
80.5
45.8
41.4
67.2
48.2
67.5
68.1
OR, XNOR (d)
45.6M
62.8
83.7
49.1
43.3
49.5
73.8
72.2
OR, AND, XNORAn. (p)
44.6M
55.0
47.2
65.0
45.0
61.0
57.5
We found the network using ORA activation function performed best overall, and across most of the subtasks. The second best performing activation functions were {ORAI.. XNORAn. (d)) and
ReLU.
A.15 COMPOSITIONAL ZERO-SHOT LEARNING
Zero-shot learning encompasses all problems which involve completing novel tasks which the subject has never seen before. The subject must infer both the task and its solution based on their previous experiences (a meta-learning task). Compositional zero-shot learning is a subset of zero-shot learning which involves combining knowledge about multiple stimulus properties together in novel pairings. For instance, if the network has been trained on "sliced bread," "sliced pear," and "caramelized pear," is able to classify images of "caramelized bread" despite having never seen an example of this before?
We based our experiments on the Task-driven Modular Networks (TMN) proposed by Purushwalkam et al. (2019). We used the code shared by the authors, but were unable replicate the results they reported in the paper when using a different random seed (see Table 8). We adapted this network by changing out all the ReI.U activation functions in the gate and module networks with different
28

---

Under review as a conference paper at ICLR 2022
activation function. Because the modules each terminate with an activation function, we needed to double the size of the hidden layer for some of our networks in order to maintain the dimensionality of the output. Consequently, some experiments had around 50% more parameters in total than others.
Experiments were performed on the MIT-States dataset (Isola ct al. 2013).Wo We trained and tested on the corresponding partitions of the dataset as introduced by Purushwalkam al. (2019). We used the same paradigm as PPuuushwalkkmm et al. (219)): Adam (Kingma & Ba (017) with a learning rate of 0,001 for the module network and 0.01 for the gating network, momentum 0.9, batch size 256, weight decay 5 x 10-5 We evaluated the network using the AUC between seen and unseen samples (Purushwalkam et al.] 2019). The network was trained until the validation AUC had plateaued, determined by not increasing for 5 epochs. We selected the model from the epoch with highest validation AUC to apply to the test set. The best performance was typically attained after around epochs.
As shown in Table 8.
we find that the trio of activation functions applied in parallel, {OR, AND, XNORAI. (p)). performs best.
Table 8: Performance of TMN-based networks at compositional zero-shot learning (CZSL) on the MIT-States dataset. Mean (standard error) of n- random initializations.
MIT- Test AUC (%)
Activation function
Mapping
Params
Top-1
Top-2
Top-3
TMN (Purushwalkam et al. 2019
1-1
2.9
7.1
11.5
TMN repro. (ReLU)
1-1
438k
2.47 + 0.07
6.42 + 0.09
10.27 - 0.11
Max
2-+1
650k
2.42 + 0.07
6.37 + 0.08
10.28 = 0.06
Max, Min (d)
1-1
438k
2.53 + 0.06
6.69 0.11
10.61 0.14
XNOR
2-+1
650
1.22 + 0.05
3.47 + 0.12
5.82 - 0.19
OR
2+11
650
2.65 - 0.05
6.80 + 0.08
10.78 - 0.13
OR, ANDAM. (d)
1-1
438k
2.61 + 0.05
6.73 0.05
10.77 + 0.12 = 1.78=0.11 8.35 = 0.24
OR, XNORAI (p)
2-1
650k
2.65
6.80 0.08
OR, XNORAIL (d)
1-1
438k
1.89 0.13
5.12
ARD XNOR (p)
2-1
650k
2.67 0.10
6.95 0.14
10.96 - 0.16
Since we could not flexibly scale the total number of parameters in the network with the original architecture, we modified the TMN architecture by adding an additional linear layer to the end of each module which projects from the activation function to the embedding space. The modules would otherwise terminate with an activation function, which makes it difficult to handle activation functions which map 2 1. We dub the modified version of the network "TMNx."
Comparing the TMN results in Table 8jto TMNx in Table 9, we can see that adding the extra linear layer improved performance of the network in of itself. Intuitively, this makes sense since the output of the TMN modules are weighted with the output of the gating network and then summed, and this weighting and summing of evidence is best performed with on logits instead of the truncated output of ReLU units. But also, performance may have improved just because the model became larger.
We performed a hyperparameter search across the training parameters for the new network against the validation set using the ReLU activation function only. We adopted the hyperparameters discovered for ReLU for all other activation functions. The batch size was reduced to 128 due to the increase in size of the model. The discovered hyperparameters were learning rate of 3 x 10-3 for both the module and gating network, and a weight decay of x 10-5, Other training hyperparameters, such as the ratio of negative samples to present, were left unchanged.
In order to match the number of parameters in the network, we used the original TMN hidden width of 64 for the module and gater networks with activation functions which map 1-> 1, and increased this to 70 for activation function which map 2 1, to maintain the total number of trainable parameters. To investigate comprehensive set of baselines, we compared against every activation function implemented in Py ToTch v1.10. The results are shown in Table 9
We found that the ELU family of activations (ELU, CELU, SELU) performed best across all three top-k values. Our best activation functions were {OR, AND (d)). followed by (OR, AND, XNORAi. (p)), which outperformed ReL.U. The other AlL. activation functions
29

---

Under review as a conference paper at ICLR 2022
Table Performance of TMNX networks compositional zero-shot learning (CZSL) on the MIT- States dataset. Pre-activation width of 64 or 70 (depending on activation function, to keep the number of parameters approximately constant). Mean (standard error) of n random initializations. Bold: best. Underlined: second best. Italic: no significant difference from best (two-sided Student's t-test, p> 0.05). Background: color scale from second-worst column best, linear with accuracy value.
MIT-States Test Accuracy (%)
Activation function
Mapping
Params
Top-I
Top-2
Top-3
ReLU
1-+1
1.15M
2.76 0.08
7.11
11.26
LeakyReLU
1-11
1.15M
2.72 0.09
10.95
PReLU
1.15M
2.83 0.06
7.25 0.14
11.44
0.17
Softplus
1-1
L.5MM
0.11
7.40
11.75
ELU
1-1 L.5MM
3.13 + 0.05
+.14
12.04 + 0.20
CELU
1-11
1,15M
3.17 = 0.06
7.81
0.15
12.00 0.19
SELU
1-1
1.15M
3.05
0.04
7.77
177:0.11 0.10
12.37 + 0.16
GELU
1-+1
1,15M
2.81 0.06
7.19
11.34
0.15
SILU
1-11
1.5M
2.87 0.08
7.34 0.14
11.60
1:00100 0.18
Hardswish
1-1
1.15M
2.84 0.08
7.23 0.11
11.48 + 0.11
1-11
1.15M
2.90
7.37
11.59
Softsign
1-1
1,15M
2.90
0.04
7.28
0.09
11.53 + 0.18
Tanh
1-11
1,1SM
2.93
0.09
7.47 1 0.16
11.64 1: 0.20
GLU
2-+1
1.16M
2.59
0.06
0.15
10.91
0.24
Max
2-+1
1,16M
2.45 + 0.08
6.52 1: 0.18
10.53 1 0.29
Max, Min (d)
1-1
1.15M
2.53 + 0.10
6.61 + 0.15
0.15 10.65
XNORAI.
2-11
1.16M
1.88
0.06
4.94 0.15
7,99
0.19
ORAIL
2-11
1.16M
2.61 0.08
6.83 1 0.12
10.97
0.20
OR, AID (d)
1-11
1.15M
2.81 1 0.06
7.12 0.10
11.27
0.14
OR, XNORAIL (p)
2-11
1.16M
2.61 0.08
6.83 0.12
10.97
0.20
OR, XNORAIL (d)
1+1
1.15M
2.46 0.00
6.51 +.51+0011 0.11
10.45 + 0.17
OR, AND, XNORAI. (p)
2-+1
1.16M
0.111
7.15
0.19
1.32 0.22
OR, AND, XNORAIL (d)
2-+3
1.22M
2.13
6.95 0.16
11.03 + 0.09
considered peformed less well, and XNOR performed particularly poorly this task. This suggests this is domain where logical activation functions are less well suited.
30


Reviews:
[{"review_id": "gEilDIAxLn", "reviewer": "Reviewer_k3Hn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Summary\n\nThis paper introduces three novel activation functions for neural networks, by modifying the logit-space versions of AND, OR, and XNOR in order to only use addition and comparison operations.  A large number of experiments show that in certain architectures and on certain tasks, some combinations of these activation functions deliver good performance.  While the results are somewhat interesting, I find the activation functions under-motivated (e.g. the motivations assume independence of input features) and worry that many of the experimental settings are designed to favor their new functions.  Similarly, the breadth of experiments means that each one is relatively under-reported in its details.  It would be helpful to have a more developed theory of these activation functions and more experiments in settings that are more standard.\n\nStrengths:\n\t* Very large set of experiments\n\t* Novel design of new activation functions\n\nWeaknesses:\n\t* New activations not sufficiently motivated (hinging on assuming independence of input dimensions)\n\t* _Too many_ experiments which are (a) insufficiently described and explained, and (b) may be biased towards favoring the new activations\n\n# Minor comments\n\n* p 4: \"assuming inputs encode independent events\": the motivations for the activations all depend on this independence assumption, which does not seem justified to me.  Can the authors explain why this is desirable?\n\n* p 5: \"access to all 16 boolean functions\".  Since {and, or, not} is a complete basis for boolean functions, why not just look at an equivalent of not?\n\n* Experimental details under-reported: e.g. a $p$-value is reported in 3.2, but it's unclear what the sample of results is (different random seeds?).  And in Section 3.5, it's unclear where to find the actual results supporting the claims in the second paragraph.  And in 3.6, there is merely a giant table of results, without any explanation of the results contained therein.\n\n\n# Typographic comments\n\n* p 2: \"plus the the log\" --> \"plus the log\"\n\n* p 2: the sentence starting \"If we suppose\" doesn't have a consequent / main verb", "summary_of_the_paper": "This paper introduces three novel activation functions for neural networks, by modifying the logit-space versions of AND, OR, and XNOR in order to only use addition and comparison operations.  A large number of experiments show that in certain architectures and on certain tasks, some combinations of these activation functions deliver good performance.  While the results are somewhat interesting, I find the activation functions under-motivated (e.g. the motivations assume independence of input features) and worry that many of the experimental settings are designed to favor their new functions.  Similarly, the breadth of experiments means that each one is relatively under-reported in its details.  It would be helpful to have a more developed theory of these activation functions and more experiments in settings that are more standard.", "summary_of_the_review": "New activation functions are introduced and tested in a very wide range of experimental settings, but I find the motivations not sufficiently developed and the experimental results under-reported and motivated; more theoretical development would help assess this contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W-q5izungRZ", "reviewer": "Reviewer_M59f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I do not think the paper and the idea is well motivated. The paper states that sigmoid is less applied and that ReLU is now applied more. But, the introduction does not explain why or make the connection to why we need these Boolean logic based activation functions. Some theoretical analysis would be helpful.\n\nThere is mention of biological networks, but it is not well cited by the paper.\n\nIn the first paragraph, I found these states somewhat loose:\n- \"an activation function is a non-linearity...\"\n- \"this will convert the logits of features into probabilities.\"\n\nAlthough several experiments were performed, I did not find the reason for why certain datasets & architectures were chosen as it relates to the presented activation functions. Also, why and when would the presented activation function perform better or worse than the used baselines? \n\nIn Table 1, I noticed it is interesting that the variance across (not +/- values themselves) the various activation configurations appear low (e.g., CIFAR10 concentrated around 82.x, STL-10 concentrated around 94.x).\n\nIn Figure 5, I was curious to know why the hyper-parameters were determined by random search. Was this uniformed? How does other popular methods (e.g., line search) affect the results?\n", "summary_of_the_paper": "The paper develops and presents approximations of three Boolean logic based activation functions: AND, OR, XNOR. These activation functions are based on the principle that neurons encode logits to represent presence of features in the log-odds space (logit-space). The formulations are simple and straightforward. Empirical evaluation is conducted on several applications using the presented activations (under individual and ensemble configuration), where ReLU, max, min (with adjustments) are the competitive baselines.\n", "summary_of_the_review": "While the methods presented are simple, the paper needs more work on motivating the methods presented and connecting it to the empirical evaluations.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Fjnr2pAQGSS", "reviewer": "Reviewer_FLuV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The paper is clearly written.\n\n- The core idea is novel and sounds. \n\n- The paper provides strengths and weaknesses of the proposed approach with diverse experimental setup. \n\n- The experiments include a good amount of tasks and data types from the image classification to transfer learning, abstract reasoning, and zero-shot learning on music, images, etc. \n\n### Main questions and suggestions\n1. Does \"Linear Layer only\" in Table 1, 5, 6 mean without any activation function?  In these tables, linear layers performs surprisingly well on some datasets with a very small amount of parameters (101k) compared to the models with the proposed activation functions (around 700k). \n    1) Does this mean that the tasks/datasets are too simple? Or is there another reason that linear layers work reasonably well? Could authors comment on this phenomenon?\n    2) Is there a reason to use a small model for the \"linear layer only\" model? Otherwise, similar parameter setup seems fair for comparison. \n\n2. The performance of MaxOut is comparable to ${OR_{AIL}}$ for image-based tasks. Does this mean that most of the pre-activations become negative? Could authors provide more analysis between Maxout and $OR_{AIL}$? Plotting a maximum response histogram of the activations (similar to the one in the MaxOut networks paper (Goodfellow et al., 2013)) would be one way to compare their behaviors. \n\n3. Experiments on more complex data like Imagenet would be useful to see how the proposed approach scales and whether the model is able to learn higher-order representation. \n\n### Other suggestions\n4. Authors mainly applied the proposed activation functions for MLP and CNN-based models. Can these activation functions be used for other common models like RNNs/GRU or transformer?\n\n5. The related works listed in the paper seem limited. Adding previous works on proposing other activation functions would be helpful.  \n\nMinor:\nTable 1 exceeds the paper size. The size needs to be reduced. \n", "summary_of_the_paper": "The paper proposes new activation functions based on the approximation of AND, OR, XOR operations. In addition, ensembling strategies are proposed to combine features from these operators. The experiments on image classification, transfer learning, abstract reasoning, and zero-shot learning tasks show that the proposed activation functions perform reasonably well, especially OR for ResNet and an ensemble for zero-shot and transfer learning.", "summary_of_the_review": "In general, the paper provides great idea, insights, and analysis. I added my major concerns in main questions and suggestions above. I am willing to increase my score once my concerns are addressed. \n\n---------------------------------\n### After rebuttal\n\nI read the responses and other reviews. I generally like the idea of this paper and how it's written. However, my concerns are not fully addressed in the response, and I also agree with some of other reviewers' concerns. I keep my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DRirzUtG_h6", "reviewer": "Reviewer_d99Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is technically sound and well-written, with some very minor comments on the latter. \n\nIn particular: \n- You should list the activation functions you reference in the introduction by their names.\n- Table 1 breaks the margins, this is unacceptable. The same holds true for some tables in the appendix.\n- In section 2.4 you mention “For simplicity, we restrict ourselves to considering only k = 2, but note that these activation functions are generalisable to higher dimensions.”, and this is a perfectly valid argument. However you continue with “On the understanding that Goodfellow et al. (2013) found k = 2 yielded the best results for MaxOut networks, we anticipate the same will hold for the AIL activation functions.” Here I recommend you either drop that sentence, or provide more evidence for the conjecture, either via more experiments or a stronger argument of why you believe that is the case. \n- Although Figs. 1-3 are great and help the reader understand better. Figs. 4-6 would benefit from improving the formatting. The label on Fig. 4, could be further improved to describe what we are seeing there. What do the colors mean? And in general for figs. 4-6 the numbers are too small.\n\nOn the content side, I do see the connection to MaxOut, however your activations are potentially better when they fall in the x+y regime, as the gradient would back-propagate on both sides, unlike MaxOut where you always only update either x or y. I believe this is potentially an advantage and could be mentioned in the paper. But at the same time, that makes me wonder whether it would behave the same as maxout for the k>2 experiments.\n\nThe biggest problem that I see with the paper is the weak empirical section when it comes to stacking your activation functions against other alternatives. There are many other alternatives and you even mention some of them in the introduction. However, you do not compare against them. Furthermore, the recent advances in learnable activation functions further improve the performance of DNNs beyond what ReLU can do.\nFinally, I believe the work in [1] deserves at least some comments in your paper as related work.\n\n[1] Godfrey, L. B., & Gashler, M. S. (2017, October). A parameterized activation function for learning fuzzy logic operations in deep neural networks. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 740-745). IEEE.\n", "summary_of_the_paper": "In this paper presents a new set of activation functions based on a relaxation of the logical operators AND, OR, XNOR. This is motivated by the behavior of the operators under the logit space equivalence. \nMoreover, the authors present approximations that are computationally more efficient based on simple max, min and addition operations.\nAs these binary operators are reduction operations, i.e., they convert two inputs into one, the authors note the similarity to the MaxOut and provide different alternatives on how to adapt the architectures to use the boolean activation functions.\nFinally, the authors provide empirical results showing how the different potential combinations of logical activation functions behave for different datasets and architectures.", "summary_of_the_review": "I like the activation functions introduced by the authors in this paper. They seem like a nice extension to MaxOut with semantical meaning on the operations they carry.\n\nThe weakness I see here are that the empirical section should compare against other activation functions and the missing related work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approximate Boolean logic activation functions. A number of comparison experiments showed intriguing differences for problems with potential logical structures. Authors suggest a probabilistic rational/motivation, i.e., computation in logit-space, however, more theoretical investigation is critically needed to answer why they perform the way they do. There are a lot of activations in the literature, so perhaps it is not easy to make a distinct contribution in performance. Despite the large number of experiments the reviewers were not convinced on how they support authors claims and contributions. The reviewers and AC strongly encourage the authors to keep the direction and improve the paper for another conference.
Decision: Reject