Year: 2021

Abstract:
The choice of activation functions and their motivation is a long-standing issue within the neural network community. Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds score of presence (versus absence) of features within the stimulus. In this work, we investigate the implications on activation functions of considering features to be logits. We derive operations equivalent to AND, OR, and XNOR for log-odds ratio representations of independent probabilities. Since these functions involve taking multiple exponents and logarithms, they are not well suited to be directly used within neural networks. Consequently, we construct efficient approximations named ANDAIL (the AND operator Approximate for Independent Logits), ORAIL, and XNORAIL, which utilize only comparison and addition operations and can be deployed as activation functions in neural networks. Like MaxOut, ANDAIL and ORAIL are generalizations of ReLU to two-dimensions. We deploy these new activation functions, both in isolation and in conjunction, and demonstrate their effectiveness on a variety of tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.

text: The choice of activation functions and their motivation is a long-standing issue within the neural network community. Neuronal representations within artificial neural networks are commonly understood as logits, representing the log-odds score of presence (versus absence) of features within the stimulus. In this work, we investigate the implications on activation functions of considering features to be logits. We derive operations equivalent to AND, OR, and XNOR for log-odds ratio representations of independent probabilities. Since these functions involve taking multiple exponents and logarithms, they are not well suited to be directly used within neural networks. Consequently, we construct efficient approximations named ANDAIL (the AND operator Approximate for Independent Logits), ORAIL, and XNORAIL, which utilize only comparison and addition operations and can be deployed as activation functions in neural networks. Like MaxOut, ANDAIL and ORAIL are generalizations of ReLU to two-dimensions. We deploy these new activation functions, both in isolation and in conjunction, and demonstrate their effectiveness on a variety of tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.

heading: 1 INTRODUCTION
text: An activation function is a non-linearity which is interlaced between linear layers within an artificial neural network. The non-linearity is essential in order for higher-order representations to form, since otherwise the network would be degeneratively equivalent to a single linear layer.
Early artificial neural networks were inspired by biological neural networks, with the activation function analogous to a neuron’s need to exceed a potentiation threshold in order to fire an action potential. Biological neurons are have long been known to be more complex than this simple abstraction, including features such as non-linearities in dendritic integration. Recent work has demonstrated that a single biological neuron can compute the XOR of its inputs (Gidon et al., 2020), a property long known to be lacking in artificial neurons (Minsky & Papert, 1969). This suggests that gains in artificial neurons can be made by using activation functions which operate on more than one input to the neuron at once.
The earliest artificial neural networks featured either logistic-sigmoid or tanh as their activation function. These activation functions were motivated by the idea that each layer of the network is building another layer of abstraction of the stimulus space from the last layer. Each neuron in a layer identifies whether certain properties or features are present within the stimulus, and the pre-activation (potentiation) value of the neuron indicates a score or logit for the presence of that feature. The sigmoid function, σ(x) = 1/(1+e−x), was hence a natural choice of activation function, since as with logistic regression, this will convert the logits of features into probabilities.
There is some evidence that this interpretation has merit. Previous work has been done to identify which features neurons are tuned to. Examples include LSTM neurons tracking quotation marks, line length, and brackets (Karpathy et al., 2015); LSTM neurons tracking sentiment (Radford et al., 2017); methods for projecting features back to the input space to view them (Olah et al., 2017); and interpretable combinations of neural activities (Olah et al., 2020). Analogously, within biological neural networks, neurons are tuned to respond more strongly to certain stimuli, and less strongly to
others. At high representation levels, concept cells respond predominantly to certain concepts, such one’s grandmother, or Jennifer Aniston (Gross, 2002; Quian Quiroga et al., 2005).
Sigmoidal activation functions are no longer commonly used within machine learning between layers of representations, though sigmoid is still widely used for gating operations which scale the magnitude of other features in an attention-like manner. The primary disadvantage of the sigmoid activation function is its vanishing gradient — as the potentiation rises, activity converges to a plateau, and hence the gradient goes to zero. This prevents feedback information propagating back through the network from the loss function to the early layers of the network, which consequently prevents it from learning to complete the task.
The Rectified Linear Unit activation function (Fukushima, 1980; Jarrett et al., 2009; Nair & Hinton, 2010), ReLU(x) = max(0, x), does not have this problem, since in its non-zero regime it has a gradient of 1. Another advantage of ReLU is it has very low computational demands. Since it is both effective and efficient, it has proven to be a highly choice of popular activation function. The chief drawback to ReLU is it has no sensitivity to changes across half of its input domain, and on average passes no gradient back to its inputs. This can lead to neurons dying1 if their weights make them never reach their activation threshold. If sigmoid is equivalent to converting our feature logits into probabilities, then the ReLU activation function is equivalent to truncating our logits and denying any evidence for the absence of a feature. We hypothesise that omitting negative evidence in this way is undesirable and reduces the generalisation ability of ReLU-based networks.
Variants of ReLU have emerged, aiming to smooth out its transition between domains and provide a gradient in its inactive regime. These include ELU (Clevert et al., 2016), CELU (Barron, 2017), SELU (Klambauer et al., 2017), GELU (Hendrycks & Gimpel, 2020), SiLU (Elfwing et al., 2017; Ramachandran et al., 2017), and Mish (Misra, 2019). However, all these activation functions still bear the general shape of ReLU and truncate negative logits.
Fuzzy logic operators are generalizations of boolean logic operations to continuous variables, using rules similar to applying logical operators in probability space. Previous work has explored networks of fuzzy logic operations, including neural networks which use an activation functions that constitutes a learnable interpolation between fuzzy logic operators (Godfrey & Gashler, 2017). In this work, we introduce activation functions which are similar to fuzzy logic operators, but derived for working in logit space instead of in probability space.
In this work we set out to develop activation functions based on the principle that neurons encode logits — scores that represent the presence of features in the log-odds space. In Section 2 we derive and define these functions in detail for different logical operators, and then consider their performance on numerous task types including parity (Section 3.1), image classification (Sections 3.4 and 3.5), transfer learning (Section 3.6), abstract reasoning (Appendix A.14), soft-rule guided classification as exemplified by the Bach chorale dataset (Section 3.3), and compositional zero-shot learning (Appendix A.15). These tasks were selected to (1) survey the performance of the new activations on existing benchmark tasks, and (2) evaluate their performance on tasks which we suspect in particular may require logical reasoning and hence benefit from activation functions which apply these logical operations to logits.

heading: 2 DERIVATION
text: Manipulation of probabilities in logit-space is known to be more efficient for many calculations. For instance, the log-odds form of Bayes’ Rule (Equation 9) states that the posterior logit equals the prior logit plus the log of the likelihood ratio for the new evidence (the log of the Bayes factor).Thus, working in logit-space allows us to perform Bayesian updates on many sources of evidence simultaneously, merely by summing together the log-likelihood ratios for the evidence. A weighted sum may be used if the amount of credence given to the sources differs — and this is precisely the operation performed by a linear layer in a neural network.
When considering sets of probabilities, a natural choice of operation to perform is measuring the joint probability of two events both occurring — the AND operation for probabilities. Suppose our input space is x ∈ [0, 1]2, and the goal is to output y > 0 if xi = 1∀ i, and y < 0 otherwise, using model
1Though this problem is very rare when using BatchNorm to stabilise feature distributions.
with a weight vector w and bias term b, such that y = wTx+ b. This can be trivially solved with the weight matrix w = [1, 1] and bias term b = −1.5. However, since this is only a linear separator, the solution can not generalise to the case y > 0 iff xi > 0 ∀ i. Similarly, let us consider how the OR function solved with a linear layer. Our goal is to output y > 0 if ∃xi = 1, and y < 0 otherwise. The binary case can be trivially solved with the weight matrix w = [1, 1] and bias term b = −0.5. The difference between this and the solution for AND is only an offset to our bias term. In each case, if the input space is expanded beyond binary to R2, the output can be violated by changing only one of the arguments.

heading: 2.1 AND
text: Suppose we are given x and y as the logits for the presence (vs absence) of two events, X and Y . These logits have equivalent probability values, which can be obtained using the sigmoid function, σ(u) = (1 + e−u)−1. Let us assume that the events X and Y are independent of each other. In this case, the probability of both events occurring (the joint probability) is P(X,Y ) = P(X ∧ Y ) = P(X) P(Y ) = σ(x)σ(y).
However, we wish to remain in logit-space, and must determine the logit of the joint probability, logit(P(X,Y )). This is given by
ANDIL := logit(P(X ∧ Y )x⊥y) = log ( p
1− p
) , where p = σ(x)σ(y),
= log
( σ(x)σ(y)
1− σ(x)σ(y)
) , (1)
which we coin as ANDIL, the AND operator for independent logits (IL). This 2d function is illustrated as a contour plot (Figure 1, left). Across the plane, the order of magnitude of the output is the same as at least one of the two inputs, scaling approximately linearly.
The approximately linear behaviour of the function is suitable for use as an activation function (no vanishing gradient), however taking exponents and logs scales poorly from a computational perspective. Hence, we developed a computationally efficient approximation as follows. Observe that we can loosely approximate ANDIL with the minimum function (Figure 1, right panel). This is equivalent to assuming the probability of both X and Y being true equals the probability of the least likely of X and Y being true — a naı̈ve approximation which holds well in three quadrants of the plane, but overestimates the probability when both X and Y are unlikely. In this quadrant, when both X and Y are both unlikely, a better approximation for ANDIL is the sum of their logits.
We thus propose ANDAIL, a linear-approximate AND function for independent logits (AIL, i.e. approximate IL).
ANDAIL(x, y) :=
{ x+ y, x < 0, y < 0
min(x, y), otherwise (2)
As shown in Figure 1 (left, middle), we observe that their output values and shape are very similar.

heading: 2.2 OR
text: Similarly, we can construct the logit-space OR function, for independent logits. For a pair of logits x and y, the probability that either of the corresponding events is true is given by p = 1−σ(−x)σ(−y). This can be converted into a logit as
ORIL(x, y) := logit(P(X ∨ Y )x⊥y) = log ( p
1− p
) , where p = 1− σ(−x)σ(−y) (3)
which can be roughly approximated by the max function. This is equivalent to setting the probability of either of event X or Y occurring to be equal to the probability of the most likely event. This underestimates the upper-right quadrant (below), which we can approximate better as the sum of the input logits, yielding
ORAIL(x, y) :=
{ x+ y, x > 0, y > 0
max(x, y), otherwise (4)

heading: 2.3 XNOR
text: We also consider the construction of a logit-space XNOR operator. This is the probability that X and Y occur either together, or not at all, given by
XNORIL(x, y) := logit(P(X⊕̄Y )x⊥y) = log ( p
1− p
) , (5)
where p = σ(x)σ(y) + σ(−x)σ(−y). We can approximate this with XNORAIL(x, y) := sgn(xy) min(|x|, |y|), (6)
which focuses on the logit of the feature most likely to flip the expected parity (Figure 3).
We could use other approximations, such as the sign-preserving geometric mean, signed geomean(x, y) := sgn(xy) √ |xy|, (7)
but note that the gradient of this is divergent, both along x = 0 and along y = 0.

heading: 2.4 DISCUSSION
text: By working via probabilities, and assuming inputs encode independent events, we have derived logit-space equivalents of the boolean logic functions, AND, OR, and XNOR. Since these are computationally demanding to compute repeatedly within a neural network, we have constructed approximations of them: ANDAIL, ORAIL, and XNORAIL. Like ReLU, these involve only comparison, addition, and multiplication operations which are cheap to perform. In fact, ANDAIL and ORAIL are a generalization of ReLU to an extra dimension, since ORAIL(x, y = 0) = max(x, 0).
The majority of activation functions are one-dimensional, f : R→ R. In contrast to this, our proposed activation functions are all two-dimensional, f : R2 → R. They must be applied to pairs of features
from the embedding space, and will reduce the dimensionality of the input space by a factor of 2. This behaviour is the same as seen in MaxOut networks (Goodfellow et al., 2013) which use max as their activation function, MaxOut(x, y; k) := max(x, y). Similar to MaxOut, our activation functions could be generalised to higher dimensional inputs, f : Rk → R, by considering the behaviour of the logit-space AND, OR, XNOR operations with regard to more inputs. For simplicity, we restrict this work to consider only k=2, but note these activation functions also generalize to higher dimensions.

heading: 2.5 ENSEMBLING
text: By using multiple logit-boolean activation functions simultaneously alongside each other, we permit the network multiple options of how to relate features together. When combining the activation functions, we considered two strategies.
In the partition (p) strategy, we split the nc dimensional pre-activation embedding equally into m partitions, apply different activation functions on each partition, and concatenate the results together. Using AIL activation functions under this strategy, the output dimension will always be half that of the input, as it is for each AIL activation function individually.
In the duplication (d) strategy, we apply m different activation functions in parallel to the same nc elements. The output is consequently larger, with dimension mnc. If desired, we can counteract the 2→1 reduction of AIL activation functions by using two of them together under this strategy. Utilising ANDAIL, ORAIL and XNORAIL simultaneously allows our networks to access logit-space equivalent of 12 of the 16 boolean logical operations with only a single sign inversion (in either one of the inputs or the output). Including the bias term and skip connections, the network has easy access to logit-space equivalents of all 16 boolean logical operations.

heading: 3 EXPERIMENTS
text: We evaluated the performance of our AIL activation functions, both individually and together in an ensemble, on a range of benchmarking tasks. Since ANDAIL and ORAIL are equivalent when the sign of operands and outputs can be freely chosen, we evaluate only on ORAIL and not both.
We compared the AIL activation functions against three primary baselines: (1) ReLU, (2) max(x, y) = MaxOut([x, y]; k=2), and (3) the concatenation of max(x, y) and min(x, y), denoted {Max,Min (d)}. The {Max,Min (d)} ensemble is equivalent to GroupSort with a group size of 2 (Anil et al., 2019; Chernodub & Nowicki, 2017), sometimes referred to as the MaxMin operator, and is comparable to the concatenation of ORAIL and ANDAIL under our duplication strategy.

heading: 3.1 PARITY
text: In a simple initial experiment, we constructed a synthetic dataset whose labels could be derived directly using the logical operation XNOR. Each sample in this dataset consisted of four input logit values, with a label that was derived by converting each logit to probability space, rounding to the
nearest integer, and taking the parity over this set of binary digits (i.e. true when we have an even number of activated bits, false otherwise).
A very small MLP model with two hidden layers (the first with four neurons, the second with two neurons) should be capable of perfect classification accuracy on this dataset with a sparse weight matrix by learning to extract pairwise binary relationships between inputs using XNORAIL. We trained such an MLP, for 100 epochs using Adam, one-cycle learning rate schedule, max LR 0.01, weight decay 1× 10−4. The two-layer MLP using the XNORAIL activation learned a sparse weight matrix able to perfectly classify any input combination, shown in Figure 4. In comparison, an identical network setup using ReLU was only able to produce 60% classification accuracy. Though this accuracy could be improved by increasing the MLP width or depth, the weight matrix was still not sparse. This experiment provides an example situation where XNORAIL is utilized by a network to directly extract information about the relationships between network inputs. For additional results, see Appendix A.7.

heading: 3.2 MLP ON COVERTYPE
text: We trained small one, two, and three-layer MLP models on the Covertype dataset (UCI Machine Learning Repository, 1998; Blackard & Dean, 1998) from the UCI Machine Learning Repository. The Covertype dataset is a classification task consisting of 581 012 samples of forest coverage across 7 classes. Each sample has 54 attributes. We used a fixed random 80:20 train:test split; for training details and accuracy vs model size, see Appendix A.8.
For each activation function, we varied the number of hidden units per layer to investigate how the activation functions affected the performance of the networks as its capacity changed. We did not use weight decay or data augmentation for this experiment, and so the network exhibits clear overfitting with larger architectures. As shown in Figure 16, Appendix A.8, XNORAIL performs significantly best on Covertype (p<10−5) for MLP with 2 or 3 layers, followed by networks which include XNORAIL alongside with other activations, and signed geomean. The duplication strategy outperforms partition.

heading: 3.3 MLP ON BACH CHORALES AND LOGIT INDEPENDENCE
text: The Bach Chorale dataset (Boulanger-Lewandowski et al., 2012) consists of 382 chorales composed by JS Bach, each ∼12 measures long, totalling approximately 83,000 notes. Represented as discrete sequences of tokens, it has served as a benchmark for music processing for decades, from heuristic methods to HMMs, RNNs, and CNNs (Mozer, 1990; Hild et al., 1992; Allan & Williams, 2005; Liang, 2016; Hadjeres et al., 2017; Huang et al., 2019). The chorales are comprised of 4 voices (melodic lines) whose behaviour is guided by soft musical rules that depend on the prior movement of that voice as well as the movement of the other voices. An example of one such rule is that two voices a fifth apart ought not to move in parallel with one another. The task we choose here is to determine whether a given short four-part musical excerpt is taken from a Bach chorale or not. During training, we stochastically corrupt chorale excerpts to provide negative examples (see Appendix A.9). We
trained 2-layer MLP discriminators with a single sigmoid output and exchanged all other activation functions (summarized in Figure 17). We found that {OR,AND,XNORAIL (d)} performed best, but that overall the results were comparable (p<0.1 between best and worst, Student’s t-test between 10 random initialisations).
Additionally, we investigated the independence between logits in the trained pre-activation embeddings. We would expect that an MLP which is optimally engaging its neurons would maintain independence between features in order to maximize information. To capture the existence of correlations, we took the cosine similarity between rows of the weight matrix. Since the inputs to all features in a given layer are the same, this is equivalent to measuring the similarity between corresponding pair of pre-activation features. We performed two experiments. In the first, we measured correlations between all pairwise combinations, and in the second we took correlations between only adjacent pre-activations that would be paired for the logical activation functions. For these experiments we used 2 hidden layers and a width that showed maximal performance for each activation function. The results are shown in Appendix A.10.

heading: 3.4 CNN AND MLP ON MNIST
text: We trained 2-layer MLP and 6-layer CNN models on MNIST with Adam (Kingma & Ba, 2017), 1-cycle schedule (Smith & Topin, 2017; Smith, 2018), and using hyperparameters tuned through a random search against a validation set comprised of the last 10k images of the training partition.
The MLP used two hidden layers, the widths of which were varied together to evaluate the performance for a range of model sizes. The CNN used six layers of 3x3 convolution layers, with 2x2 max pooling (stride 2) after every other conv layer. The output was flattened and followed by three MLP layers. The widths of the layers were scaled up approximately linearly to explore a range of model sizes (see Appendix A.11 for more details).
For the MLP, XNORAIL performed best along with signed geomean (p < 0.1), ahead of all other activations (p < 0.01; Figure 5 left panel). With the CNN, five activation configurations ({OR,AND,XNORAIL (p)}, {OR,XNORAIL (d/p)}, Max, and SiLU) performed best (p<0.05; Figure 5, right panel). Additionally, we note that CNNs which used ORAIL or Max (alone or in an ensemble) maintained high performance with an order of magnitude fewer parameters (3× 104) than networks which did not (3× 105 params).

heading: 3.5 RESNET50 ON CIFAR-10/100
text: We explored the impact of our AIL activation functions on the performance of deep networks by deploying them within a pre-activation ResNet50 model (He et al., 2015; 2016). We exchanged all ReLU activation functions in the network to a candidate activation function while maintaining the size of pass-through embedding. We experimented with changing the width of the network, scaling up the embedding space and all hidden layers by a common factor. The network was trained on
CIFAR-10/-100 for 100 epochs using Adam (Kingma & Ba, 2017), 1-cycle (Smith, 2018; Smith & Topin, 2017). Hyperparameters were optimized through a random search against a validation partition of CIFAR-100 for a fixed width factor w = 2 only. The same hyperparameters were reused for experiments of changing width, and for CIFAR-10. We used width factors of 0.5, 1, 2, and 4 for 1→1 activation functions (ReLU, etc), widths of 0.75, 1.5, 3, and 6 for 2→1 activation functions (Max, etc), and widths of 0.4, 0.75, 1.5, and 3 for {ORAIL, ANDAIL, XNORAIL (d)}.
For both CIFAR-10 and 100, we find that SiLU, ORAIL, and Max outperform ReLU across a wide range of width values, as shown in Figure 6. These three activation functions hold up their performance best as the number of parameters is reduced. Note that SiLU was previously discovered as an optimal activation function for deep residual image models (Ramachandran et al., 2017), and so it is expected to perform highly in this setting. Meanwhile, other AIL activation functions perform similarly to ReLU when the width is thin, and slightly worse than ReLU when the width is wide. When used on its own and not part of an ensemble, the XNORAIL activation function performed very poorly (off the bottom of the chart), indicating it is not well suited for this task.

heading: 3.6 TRANSFER LEARNING
text: We considered the task of transfer learning on several image classification datasets. We used a ResNet18 model (He et al., 2015) pretrained on ImageNet-1k. The weights were frozen (not fine-tuned) and used to generate embeddings of samples from other image datasets. We trained a twolayer MLP to classify the images from these embeddings, using various activation functions. For a comprehensive set of baselines, we compared against every activation function built into PyTorch 1.10. To make the number of parameters similar, we used a width of 512 for activation functions with 1→1 mapping (e.g. ReLU), a width of 650 for activation functions with a 2→1 mapping (e.g. Max, ORAIL), and a width of 438 for {OR,AND,XNORAIL (d)}. See Appendix A.13 for more details. Our results are shown in Table 1. We found that all our activation functions outperformed ReLU on every transfer task. In particular, our three activation ensembles using the duplication strategy performed very highly across all 7 transfer learning tasks — overall, {OR,AND,XNORAIL (d)} appears to perform best (top for 3 datasets, second on 3), followed by {OR,ANDAIL (d)}. Our activation functions were beaten on Caltech101, Oxford Flowers, and Stanford Cars, and only by PReLU. For these three datasets, a linear readout outperformed an MLP with ReLU, and on Caltech101 the linear layer performed best. This suggests that PReLU excels only on particularly simple tasks. For further discussion, see Appendix A.13.

heading: 3.7 ADDITIONAL RESULTS
text: For results on abstract reasoning and compositional zero-shot learning, please see Appendix A.14 and Appendix A.15, respectively.

heading: 4 DISCUSSION
text: In this work we motivated and introduced novel activation functions analogous to boolean operators in logit-space. We designed the AIL functions, fast approximates to the true logit-space functions equivalent to manipulating the corresponding probabilities, and demonstrated their effectiveness on a wide range of tasks.
Although our activation functions assume independence (which is generally approximately true for the pre-activation features learnt with 1d activation functions), we found the network learnt to induce anticorrelations between features which were paired together by our activation functions (Appendix A.10). This suggests that the assumption of independence is not essential to the performance of our proposed activation functions.
We found that the XNORAIL activation function was highly effective in the setting of shallow networks. Meanwhile, the ORAIL activation function was highly effective for representation learning in the setting of a deep ResNet architecture trained on images. In scenarios which involve manipulating high-level features extracted by an embedding network, we find that using an ensemble of AIL activation functions together works best, and that the duplication ensembling strategy outperforms partitioning. In this work we have restricted ourselves to only considering using a single activation function (or ensemble) throughout the network, however our results together indicate that stronger results may be found by using ORAIL for feature extraction and an ensemble of {OR,AND,XNORAIL (d)} for later higher-order reasoning layers within the network. The idea we propose is nascent and there is a great deal of scope for exploring other forms of activation functions that combine multiple pre-activation features by utilizing “higher-order activation functions”.

heading: REPRODUCIBILITY STATEMENT
text: We have shared code to run all experiments considered in this paper with the reviewers via an anonymous download URL. The code base contains detailed instructions on how to setup each experiment, including downloading the datasets and installing environments with pinned dependencies. It should be possible to reproduce all our experimental results with this code. For the final version of the paper, we will make the code publicly available on an online repository and share a link to it within the paper.

heading: A APPENDIX

heading: A.1 LINEAR RELU EXAMPLES
text: In Figure 7, we show a representation of what a 2-d linear layer followed by the ReLU activation function looks like. The output is the same up to rotation.
If we apply a second linear layer on top of the outputs of the first two units, we can try to approximate the logit AND or OR function. However, the solution using ReLU leaves a quadrant of the output space hollowed out as zero due to its behaviour at truncating away information.

heading: A.2 SOLVING XOR
text: A long-standing criticism of artificial neural networks is their inability to solve XOR with a single layer (Minsky & Papert, 1969). Of course, adding a single hidden layer allows a network using ReLU to solve XOR. However, the way that it solves the problem is to join two of the disconnected regions together in a stripe (see Figure 8). Meanwhile, our XNORAIL is trivial able to solve the XOR problem without any hidden layers. For comparison here, we include one hidden layer with 2 units for each network. Including a layer before the activation function makes the task harder for XNORAIL, which must learn how to project the input space in order to compute the desired separation. Also, including the linear layer allows the network to generalise to rotations and offset versions of the task.

heading: A.3 LINEAR LAYERS AND BAYES’ RULE IN LOG-ODDS FORM
text: Bayes’ Theorem or Bayes’ Rule is given by
P(H|X) = P(X|H) P(H) P(X) . (8)
In this case, we update the probability of our hypothesis, H , based on the event of the observation of a new piece of evidence, X . Our prior belief for the hypothesis is P(H), and posterior is P(H|X). To update our belief from the prior and yield the posterior, we multiply by the Bayes factor for the evidence which is given by P(X|H)/P(X).
Converting the probabilities into log-odds ratios (logits) yields the following representation of Bayes’ Rule.
log
( P(H|X)
P(HC |X)
) = log ( P(H)
P(HC)
) + log ( P(X|H)
P(X|HC)
) (9)
Here, HC is the complement to H (the event that the hypothesis is false), and P(HC) = 1 − P(H). Our prior log-odds ratio is log ((P(H)/P(HC)), and our posterior after updating based on the observation of new evidence X is log (P(H|X)/P(HC |X)). To update our belief from the prior and yield the posterior, we add the log-odds Bayes factor for the evidence which is given by log (P(X|H)/P(X|HC)).
In log-odds space, a series of updates with multiple pieces of independent evidence can be performed at once with a summation operation.
log
( P(H|x)
P(HC |x)
) = log ( P(H)
P(HC) ) + ∑ i log ( P(Xi|H) P(Xi|HC) ) . (10)
This is operation can be represented by the linear layer in an artificial neural network, zk = bk +wTk x. Here, the bias term bk = log ((P(H)/P(HC)) is the prior for hypothesis (the presence of the feature represented by the k-th neuron), and the series of weighted inputs from the previous layer, wki xi provide evidential updates. This is also equivalent to the operation of a multinomial naı̈ve Bayes classifier, expressed in log-space, if we choose wki = log pki (Rennie et al., 2003).

heading: A.4 DIFFERENCE BETWEEN AIL AND IL FUNCTIONS
text: Here, we measure and show the difference between the true logit-space operations and our AIL approximations, shown in Figure 9, Figure 10, and Figure 11.
In each case, we observe that the magnitude of the difference is never more than 1, which occurs along the boundary lines in AIL. Since the magnitude of the three functions increase as we move away from the origin, the relative difference decreases in magnitude as the size of x and y increase.

heading: A.5 GRADIENT OF AIL AND IL FUNCTIONS
text: We show the gradient of each of the logit-space boolean operators and their AIL approximates in Figure 12, Figure 13, and Figure 14. By the symmetry of each of the functions, the derivative with respect to y is a reflected copy of the gradient with respect to x.
We find that the gradient of each AIL function closely matches that of the exact form. Whilst there are “dead” regions where the gradient is zero, this only occurs for one of the derivatives at a time (there is always a gradient with respect to at least one of x and y).

heading: A.6 DATASET SUMMARY
text: The datasets used in this work are summarised in Table 2.
We used the same splits for Caltech101 as used in Kabir et al. (2020).
The Covertype dataset was chosen as a dataset that contains only simple features (and not pixels of an image) on which we could study a simple MLP architecture, and was selected based on its popularity on the UCI ML repository.
The Bach Chorales dataset was chosen because — in addition to being in continued use by ML researchers for decades — it presents an interesting opportunity to consider a task where logical activation functions are intuitively applicable, as it is a relatively small dataset that has also been approached with rule-based frameworks, e.g. the expert system by Ebcioglu (1988).
MNIST, CIFAR-10, CIFAR-100 are standard image datasets, commonly used. We used small MLP and CNN architectures for the experiments on MNIST so we could investigate the performance of the network for many configurations (varying the size of the network). We used ResNet-50, a very common deep convolutional architecture within the computer vision field, on CIFAR-10/100 to evaluate the performance in the context of a deep network.
The datasets used for the transfer learning task are all common and popular natural image datasets, with some containing coarse-grained classification (CIFAR-10), others fine-grained (Stanford Cars), and with a varying dataset size (5000—75000 training samples). We chose to do an experiment involving transfer learning because it is a common practical situation where one must train only a small network that handles high-level features, and is the sort of situation which involves manipulating high-level features, relying on the pretrained network to do the feature extraction.
We considered other domains where logical reasoning is involved as a component of the task, and isolated abstract reasoning and compositional zero-shot learning as suitable tasks.
For abstract reasoning, we wanted to use an IQ style test, and determined that I-RAVEN was a state-of-the-art dataset within this domain (having fixed some problems with the pre-existing RAVEN dataset). We determined that the SRAN architecture from the paper which introduced I-RAVEN was still state-of-the-art on this task, and so used this.
Another problem domain in which we thought it would be interesting to study our activation functions was compositional zero-shot learning (CZSL). This is because the task inherently involves combining an attribute with an object (i.e. the AND operation). For CZSL, we looked at SOTA methods on https://paperswithcode.com. The best performance was from SymNet, but this was only implemented in TensorFlow and our code was set up in PyTorch so we built our experiments on the back of the second-best instead, which is the TMN architecture. In the TMN paper, they used two datasets: MIT-States and UT-Zappos-1. In our preliminary experiments, we found that the network started overfitting on MIT-States after around 6 epochs, but on UT-Zappos-1 it was overfitting after the first or second epoch (one can not tell beyond the fact the val performance is best for the first epoch). In the context of zero-shot learning, an epoch uses every image once, but there are also only a finite number of tasks in the dataset. Because there are multiple samples for each noun/adjective pair, and each noun only appears with a handful of adjectives and vice versa, there is in a way fewer tasks in one epoch than there are images. Hence it is possible for a zero-shot learning model to overfit to the training tasks in less than one epoch (recall that the network includes a pretrained ResNet model
for extracting features from the images). For simplicity, we dropped UT-Zappos-1 and focused on MIT-States.

heading: A.7 PARITY EXPERIMENTS
text: Following on from the parity experiment described in the main text (Section 3.1), we also introduced a second synthetic dataset with a labelling function that, while slightly more complex than the first, was still solvable by applying the logical XNOR operation to the network inputs. In this dataset increased our number of inputs to 8, and derived our labels by applying a set of nested XNORIL operations:
XNORIL( XNORIL(XNORIL(x2, x5),XNORIL(x3, x4)),
XNORIL(XNORIL(x6, x7),XNORIL(x0, x1)).
For this more difficult task we also reformulated our initial experiment into a regression problem, as the continuous targets produced by this labelling function are more informative than the rounded binary targets used in the first experiment. We also adjusted our network setup to have an equal number of neurons at each hidden layer as we found that this significantly improved model performance2. We again trained using the same model hyper-parameters for 100 epochs.
While this time the model was not able to learn a sparse weight matrix that exactly reflected our labelling function (see Figure 15), the model was again able to leverage the XNORAIL activation function to significantly outperform an identical model utilizing the ReLU activation function.
We found that a simple model with three hidden layers, each with eight neurons, utilizing XNORAIL was able to go from a validation RMSE of 0.287 at the beginning of training to a validation RMSE of 0.016 after 100 epochs. Comparatively, an identical model utilizing the ReLU activation function was only able to achieve a validation RMSE of 0.271 after 100 epochs. In order for our ReLU network to match the validation RMSE of our 8-neuron-per-layer XNORAIL model, we had to increase the model size by 32 times to 256 neurons at each hidden layer.
2We hypothesize that, because our XNORAIL activation function reduces the number of hidden layer neurons by a factor of k, having a reduced number of neurons at each layer creates a bottleneck in the later layers of the network which restricts the amount of information that made its way through to the final layer

heading: A.8 COVERTYPE
text: Our one, two, and three-layer MLPs were trained using 1-cycle (Smith & Topin, 2017; Smith, 2018) for 50 epochs, with a batch size of 1024, without weight decay. We used a subset of 15% of the training data for validation. The learning rate was selected using an automated learning rate finder approach, verified against the validation partition. For our final experiments, the validation partition was included in the training data. No data augmentation was performed.

heading: A.9 BACH CHORALE TRAINING DETAILS
text: For each of the 4 voices, we restricted the available pitches to 3 octaves, resulting in 37 one-hot tokens (including silence). Since we only planned on feeding small time-windows of the chorale into our model, for pre-processing, we converted the data into a shape of (seq len, 4, 37), where we set seq len to 4.
To generate training examples for the discriminator, we transposed by {−5,−4, . . . , 5, 6} semitones, chosen uniformly at random, and there was a 0.5 probability that the sample was corrupted by the following method:
• Choose 2-3 notes in the (seq len× 4) window to be corrupted • For each note, corrupt by the following mixture distribution:
– (p = 0.6) Sample a pitch from a Gaussian centered on the existing note, with σ = 3 semitones, forcing the new pitch to be distinct
– (p = 0.2) Copy a pitch from the current voice, forcing the new pitch to be distinct – (p = 0.2) Extend the previous note in time – (p = 0.1) Sample uniformly from all 37 possible tokens

heading: A.10 CORRELATIONS BETWEEN PRE-ACTIVATIONS
text: Results on correlations between weights in the JSB Chorale models are shown in Figure 18. We found that when taking all pre-activations into account, every activation function generally showed independence between features. Interestingly, the cosine similarities between inputs that were paired together for the bivariate activation functions showed anticorrelation in almost all cases where Max or ORAIL were used, and other cases generally showed more correlation than ReLU.
We found that randomly selected pairs of preactivation features within the same layer have correlations that are given by a Gaussian-like distribution centered around zero. This was the case for all of the activation functions we tested. The behaviour of randomly selected pairs of features is thus reasonably consistent with the assumption of independence which we have made. We also investigated the correlation between the pairs of preactivation features which were passing into our two-dimensional activation functions. Here, we found the correlation structure is different, and the correlation depends
on the activation function being used. With Max and ORAIL activations, the network learns to make the columns of the weight matrix (and hence the preactivation scores for the pair of features) be inversely correlated. With XNORAIL, the network learns features which are either positively or negatively correlated (a wider distribution of correlations than seen with random pairs of features). We observe that (in all cases) the network learns to make the features passed to the AIL activation functions be correlated instead of independent, despite our assumption of independence. So it appears clear that the assumption of independence is violated, but also that it doesn’t really matter because the network is choosing to break the assumption and induce these correlations between the features to get better performance.

heading: A.11 CNN AND MLP ON MNIST
text: In this experiment we trained MLP and CNN models for 10 epochs on the MNIST dataset using Adam optimizer, one-cycle learning rate schedule, and cross entropy loss, with batch size of 256. We augmented our training samples using a random affine transformation from: rotation of ±10 degrees, scale of factor 0.8 to 1.2, translation with max absolute fraction for horizontal and vertical directions of 0.08, and shear parallel to the x-axes of ±0.3. The hyper-parameters for the optimizer and scheduler were selected through a random search of the hyper-parameter space. We chose to do random search instead of grid search because it typically yields better results for the same number of test cases.
For the hyper-parameter search, we trained on the first 50 000 samples of the training partition and used the final 10 000 samples as a validation set. We ran the search for four iterations, with each iteration sampling 120 different hyper-parameter settings. The initial bounds for our hyper-parameter samples were set as described in Table 3. These bounds were chosen to be suitably wide such that the optimal configuration should be contained within them for all activation functions considered.
The bounds on our uniform random variable x were tightened at each iteration by selecting the top-5 performing hyper-parameter settings, taking the mean x̄ and weighted standard deviation σ across those settings, and re-setting the bounds for the next iteration to be equal to x̄±1.5σ. After the fourth iteration, we ran a final iteration where we selected the top-10 performing hyper-parameter settings across the four previous iterations, and then re-ran these for another 120 seeds (i.e. random weight initializations). We then selected the hyper-parameter settings which had the highest performance across all 120 seeds.
We found that the XNOR activation function required quite different hyper-parameters than the other activation functions, but OR and Max activation functions used similar hyper-parameters to ReLU. However, we have no reason to believe that the proposed AIL activation functions are more susceptible than others to the choice of hyper-parameters or the way in which hyper-parameters are selected.
Our MLP model consisted of two hidden layers of equal size with batch norm applied to each. The number of neurons in each layer was set, taking into consideration the current activation function being tested, to ensure the number of trainable parameters in the network remained static across experiments
Our CNN model consisted of six layers, where each layer was comprised of a set of 2D convolution filters (kernel size 3, stride 1, padding 1), batch normalization, and a non-linear activation. The network also applied a pooling layer (kernel size 2, stride 2) at the end of every second layer. After the 6 CNN layers the network flattens the output and applies three linear layers. The number of output channels (ie. pre-activations) for the six 2D convolutions were [c, 2c, 4c, 4c, 8c, 8c], and the number of pre-activation neurons produced by the subsequent linear layers were [32c, 16c]. Similar to the MLP model, c was chosen to ensure the number of trainable parameters in the network remained fixed across experiments. When varying the number of network parameters in our experiments, if the number of parameters dropped below 1× 106 in the CNN model, we changed the structure of the convolution layers and linear layers to [c, c, c, c, 1.5c, 1.5c] and [2c, 1.5c], respectively. This ensured that each layer had at least two channels for our activation functions to aggregate.
Once our optimal hyper-parameters were selected for both our MLP and CNN models, we then trained each model several times with varying number of trainable parameters. The reason we vary
the number of parameters in the network instead of the network size/structure is because a network using our logical activation functions can have a significantly different number of parameters than an identical network using ReLU activation, because our logical activations aggregate across neurons at each layer.
A.12 RESNET50 ON CIFAR-10/100
For this experiment we trained a ResNet50 model for 100 epochs on CIFAR-10 and CIFAR-100 using Adam optimizer, one-cycle learning rate schedule, cross entropy loss, and augmentations derived for CIFAR-10 by AutoAugment (Cubuk et al., 2018), with batch size of 128. The hyper-parameters for the optimizer and scheduler were determined through a random search of the hyper parameter space on the CIFAR-100 dataset3. The hyper-parameter search was tuned against a partition of 10% of the CIFAR-100 training samples. Our hyper-parameter search follows a similar approach to the one used for our MLP and CNN models on MNIST (Appendix A.11), but due to compute constraints with our larger models here we only trained 100 seeds each iteration, and only re-examined the top 10 seeds in the final round.
For PReLU, we did not perform a hyper-parameter search and merely re-used the same hyperparameters as discovered in the search with ReLU.
Once our optimal hyper-parameters were selected we trained a ResNet50 model with automatic mixed precision using four different width parameters: 0.5, 1, 2, and 4. Because activation functions such as XNORAIL reduce the number of post-activation neurons by a factor of 2, and the {OR,AND,XNORAIL (d)} activation function increases the number of post-activation neurons by a factor of 3/2, we adjust the width parameter for these activation function to result in all networks having approximately the same number of parameters. For the 2→ 1 activation functions (Max, ORAIL, XNORAIL, {OR,XNORAIL (p)}, and {OR,AND,XNORAIL (p)}) we use width values of 0.75, 1.5, 3, and 6. For the 2→3 activation function {OR,AND,XNORAIL (d)}, we use width values of 0.4, 0.75, 1.5, and 3.

heading: A.13 TRANSFER LEARNING
text: We used a pretrained ResNet50 model taken from the pytorch hub. The 2-layer MLP head was trained for 25 epochs on each dataset. Since we are interested in transfer learning in a data-limited regime for these experiments, we used only a small amount of data augmentation. We applied horizontal flip (p=0.5), scaling (0.7 to 1), aspect ratio stretching (3/4 to 4/3), and colour jitter (intensity 0.4) only. Pixel intensity normalization was done against the ImageNet mean and std.
The MLP head was optimized using SGD, momentum 0.9. We used a batch size of 128, maximum learning rate 0.01, and weight decay 1× 10−4. Before training began, we passed one epoch worth of inputs through the network without updating the weights in order to refresh the batch normalization statistics to the new dataset. We also performed one epoch of training with a warmup learning rate of 1× 10−5 before commencing training at the maximum learning rate of 0.01. The learning rate was decayed with a cosine annealing schedule over 24 epochs. We report the performance of the final model at the end of the 25 epochs.
We used a pre-activation width of 512 neurons for ReLU and other and activation functions which map 1→ 1. To approximately match up the number of parameters, we used a width of 650 for activation functions which map 2→1, and 438 for {OR,AND,XNORAIL (d)} which maps features 3→2. These values control the total number of parameters in the head to be the same for datasets with 100 classes (the median number of classes in the datasets we considered). Our results with widths 438/512/650 and ∼600k parameters are shown in Table 1. The precise widths and number of trainable parameters used for these experiments are shown in Table 4.
Since the transfer learning experiments were performed without retraining the base network, we found (Table 1) the performance is limited by the features which the base network produces, whose relevance depends on the other lap between the domain of the pretraining task (ImageNet) and the new task. This information bottleneck, and variation in its utility, means the variation between
3We used the same set of hyper-parameters from this search for both the CIFAR-10 and CIFAR-100 experiments.
performance on different datasets is much larger than the variation for different activation functions within the same dataset.
For transfer tasks which involve coarse-grained discrimination on images which are similar to ImageNet, the embedding generated by the pretrained model is already sufficient to separate the classes in the new dataset. Examples of this are Caltech101, where linear layer beats out using additional layers with non-linearities; and STL-10, where our best model is on-par with the linear model. The results on these two transfer learning tasks are too simple to be of interest to study. The performance is limited by the information retained by the pretrained embedding, but it appears that what task-relevant information is there is readily available with a linear layer without needing additional logic to interpret it.
Other transfer learning tasks we attempted are less trivial and show a larger difference in performance across the activation functions, and a gap from the linear layer readout model. The Stanford Cars dataset involves fine-grained discrimination between different car models, for which features generated by a model pretrained on ImageNet are not effective. The SVHN dataset, which contains images of house numbers, is coarse-grained but uses images which are outside the domain of ImageNet, which makes the transfer-learning task more difficult. Additionally, our use of random horizontal reflections during training will have hindered performance on this dataset by notably increasing the task complexity, since numerals are not invariant under reflection.
Additionally, we ran the experiment again with a width of w = 512 for the 2→1 activation functions. The results with constant pre-activation width w = 512 are shown in Table 5. By using a constant pre-activation width, the number of parameters used is not consistent across activation functions. The number of trainable parameters in each experiment is shown in tab:transfer-nparam-512.
We found that reducing the width from 650 to 512 (and hence reducing total number of trainable parameters) reduced the performance of 2→1 activation functions XNORAIL, ORAIL, {OR,XNORAIL (p)}, and {OR,AND,XNORAIL (p)}. Increasing the width from 438 to 512 increased the performance of {OR,AND,XNORAIL (d)}.

heading: A.14 ABSTRACT REASONING
text: Abstract reasoning is challenging for neural networks to learn because their structure and objective function are more effective for tasks which come instinctively to humans (“System 1” of Kahneman, 2011), such as object recognition, as opposed to demanding (“System 2”) logic tasks.
In recent years, several challenges have been proposed to evaluate the ability of neural networks to perform abstract reasoning. The Raven’s Progressive Matrices (Raven & Court, 1938) are a long-standing IQ test, which have been emulated by Barrett et al. (2018) with Procedurally Generated Matrices (PGM), and then the RAVEN task by (Zhang et al., 2019). Recently Hu et al. (2020) and Benny et al. (2021) have improved on that task with I-RAVEN and RAVEN-FAIR, respectively, both aiming to make the task more balanced. Other abstract reasoning tasks have also been proposed (Fleuret et al., 2011; Johnson et al., 2017; Barrett et al., 2018).
We considered the application of AIL activation functions in the context of the I-RAVEN task, by adapting the Stratified Rule-Aware Network (SRAN) of Hu et al. (2020) to include our AIL activation functions. We first added LayerNorm to the network, and then swapped out the seven ReLU activations in the gating module. The architecture for the three ResNet-18 (He et al., 2015) base models were unchanged. Where necessary, the number of units per layer was modified to facilitate the change in dimensionality caused by our activation function. The networks were trained using the same procedure as described by Hu et al. (2020).
We found the network using ORAIL activation function performed best overall, and across most of the subtasks. The second best performing activation functions were {ORAIL, XNORAIL (d)} and ReLU.

heading: A.15 COMPOSITIONAL ZERO-SHOT LEARNING
text: Zero-shot learning encompasses all problems which involve completing novel tasks which the subject has never seen before. The subject must infer both the task and its solution based on their previous experiences (a meta-learning task). Compositional zero-shot learning is a subset of zero-shot learning which involves combining knowledge about multiple stimulus properties together in novel pairings. For instance, if the network has been trained on “sliced bread,” “sliced pear,” and “caramelized pear,” is it able to classify images of “caramelized bread” despite having never seen an example of this before?
We based our experiments on the Task-driven Modular Networks (TMN) proposed by Purushwalkam et al. (2019). We used the code shared by the authors, but were unable to replicate the results they reported in the paper when using a different random seed (see Table 8). We adapted this network by changing out all the ReLU activation functions in the gate and module networks with a different
activation function. Because the modules each terminate with an activation function, we needed to double the size of the hidden layer for some of our networks in order to maintain the dimensionality of the output. Consequently, some experiments had around 50% more parameters in total than others.
Experiments were performed on the MIT-States dataset (Isola et al., 2015).We trained and tested on the corresponding partitions of the dataset as introduced by Purushwalkam et al. (2019). We used the same paradigm as Purushwalkam et al. (2019): Adam (Kingma & Ba, 2017) with a learning rate of 0.001 for the module network and 0.01 for the gating network, momentum 0.9, batch size 256, weight decay 5× 10−5. We evaluated the network using the AUC between seen and unseen samples (Purushwalkam et al., 2019). The network was trained until the validation AUC had plateaued, determined by not increasing for 5 epochs. We selected the model from the epoch with highest validation AUC to apply to the test set. The best performance was typically attained after around 5 epochs.
As shown in Table 8, we find that the trio of activation functions applied in parallel, {OR,AND,XNORAIL (p)}, performs best.
Since we could not flexibly scale the total number of parameters in the network with the original architecture, we modified the TMN architecture by adding an additional linear layer to the end of each module which projects from the activation function to the embedding space. The modules would otherwise terminate with an activation function, which makes it difficult to handle activation functions which map 2→1. We dub the modified version of the network “TMNx.” Comparing the TMN results in Table 8 to TMNx in Table 9, we can see that adding the extra linear layer improved performance of the network in of itself. Intuitively, this makes sense since the output of the TMN modules are weighted with the output of the gating network and then summed, and this weighting and summing of evidence is best performed with on logits instead of the truncated output of ReLU units. But also, performance may have improved just because the model became larger.
We performed a hyperparameter search across the training parameters for the new network against the validation set using the ReLU activation function only. We adopted the hyperparameters discovered for ReLU for all other activation functions. The batch size was reduced to 128 due to the increase in size of the model. The discovered hyperparameters were a learning rate of 3× 10−3 for both the module and gating network, and a weight decay of 1× 10−5. Other training hyperparameters, such as the ratio of negative samples to present, were left unchanged.
In order to match the number of parameters in the network, we used the original TMN hidden width of 64 for the module and gater networks with activation functions which map 1→1, and increased this to 70 for activation function which map 2→1, to maintain the total number of trainable parameters. To investigate a comprehensive set of baselines, we compared against every activation function implemented in PyTorch v1.10. The results are shown in Table 9.
We found that the ELU family of activations (ELU, CELU, SELU) performed best across all three top-k values. Our best activation functions were {OR,ANDAIL (d)}, followed by {OR,AND,XNORAIL (p)}, which outperformed ReLU. The other AIL activation functions we
considered peformed less well, and XNOR performed particularly poorly on this task. This suggests this is a domain where logical activation functions are less well suited.

Review summaries:
The paper proposes new activation functions based on the approximation of AND, OR, XOR operations. In addition, ensembling strategies are proposed to combine features from these operators. The experiments on image classification, transfer learning, abstract reasoning, and zero-shot learning tasks show that the proposed activation functions perform reasonably well, especially OR for ResNet and an ensemble for zero-shot and transfer learning.

The paper develops and presents approximations of three Boolean logic based activation functions: AND, OR, XNOR. These activation functions are based on the principle that neurons encode logits to represent presence of features in the log-odds space (logit-space). The formulations are simple and straightforward. Empirical evaluation is conducted on several applications using the presented activations (under individual and ensemble configuration), where ReLU, max, min (with adjustments) are the competitive baselines. 

This paper introduces three novel activation functions for neural networks, by modifying the logit-space versions of AND, OR, and XNOR in order to only use addition and comparison operations.  A large number of experiments show that in certain architectures and on certain tasks, some combinations of these activation functions deliver good performance.  While the results are somewhat interesting, I find the activation functions under-motivated (e.g. the motivations assume independence of input features) and worry that many of the experimental settings are designed to favor their new functions.  Similarly, the breadth of experiments means that each one is relatively under-reported in its details.  It would be helpful to have a more developed theory of these activation functions and more experiments in settings that are more standard.

In this paper presents a new set of activation functions based on a relaxation of the logical operators AND, OR, XNOR. This is motivated by the behavior of the operators under the logit space equivalence.  Moreover, the authors present approximations that are computationally more efficient based on simple max, min and addition operations. As these binary operators are reduction operations, i.e., they convert two inputs into one, the authors note the similarity to the MaxOut and provide different alternatives on how to adapt the architectures to use the boolean activation functions. Finally, the authors provide empirical results showing how the different potential combinations of logical activation functions behave for different datasets and architectures.