Logical Activation Functions: Logit-space equivalents of Boolean Operators

The choice of activation functions and their motivation is a long-standing issue within the neural network community . Neuronal representations within artificial neural networks are commonly understood as logits , representing the log-odds score of presence ( versus absence ) of features within the stimulus . In this work , we investigate the implications on activation functions of considering features to be logits . We derive operations equivalent to AND , OR , and XNOR for log-odds ratio representations of independent probabilities . Since these functions involve taking multiple exponents and logarithms , they are not well suited to be directly used within neural networks . Consequently , we construct efficient approximations named ANDAIL ( the AND operator Approximate for Independent Logits ) , ORAIL , and XNORAIL , which utilize only comparison and addition operations and can be deployed as activation functions in neural networks . Like MaxOut , ANDAIL and ORAIL are generalizations of ReLU to two-dimensions . We deploy these new activation functions , both in isolation and in conjunction , and demonstrate their effectiveness on a variety of tasks including image classification , transfer learning , abstract reasoning , and compositional zero-shot learning . 1 INTRODUCTION . An activation function is a non-linearity which is interlaced between linear layers within an artificial neural network . The non-linearity is essential in order for higher-order representations to form , since otherwise the network would be degeneratively equivalent to a single linear layer . Early artificial neural networks were inspired by biological neural networks , with the activation function analogous to a neuron ’ s need to exceed a potentiation threshold in order to fire an action potential . Biological neurons are have long been known to be more complex than this simple abstraction , including features such as non-linearities in dendritic integration . Recent work has demonstrated that a single biological neuron can compute the XOR of its inputs ( Gidon et al. , 2020 ) , a property long known to be lacking in artificial neurons ( Minsky & Papert , 1969 ) . This suggests that gains in artificial neurons can be made by using activation functions which operate on more than one input to the neuron at once . The earliest artificial neural networks featured either logistic-sigmoid or tanh as their activation function . These activation functions were motivated by the idea that each layer of the network is building another layer of abstraction of the stimulus space from the last layer . Each neuron in a layer identifies whether certain properties or features are present within the stimulus , and the pre-activation ( potentiation ) value of the neuron indicates a score or logit for the presence of that feature . The sigmoid function , σ ( x ) = 1/ ( 1+e−x ) , was hence a natural choice of activation function , since as with logistic regression , this will convert the logits of features into probabilities . There is some evidence that this interpretation has merit . Previous work has been done to identify which features neurons are tuned to . Examples include LSTM neurons tracking quotation marks , line length , and brackets ( Karpathy et al. , 2015 ) ; LSTM neurons tracking sentiment ( Radford et al. , 2017 ) ; methods for projecting features back to the input space to view them ( Olah et al. , 2017 ) ; and interpretable combinations of neural activities ( Olah et al. , 2020 ) . Analogously , within biological neural networks , neurons are tuned to respond more strongly to certain stimuli , and less strongly to others . At high representation levels , concept cells respond predominantly to certain concepts , such one ’ s grandmother , or Jennifer Aniston ( Gross , 2002 ; Quian Quiroga et al. , 2005 ) . Sigmoidal activation functions are no longer commonly used within machine learning between layers of representations , though sigmoid is still widely used for gating operations which scale the magnitude of other features in an attention-like manner . The primary disadvantage of the sigmoid activation function is its vanishing gradient — as the potentiation rises , activity converges to a plateau , and hence the gradient goes to zero . This prevents feedback information propagating back through the network from the loss function to the early layers of the network , which consequently prevents it from learning to complete the task . The Rectified Linear Unit activation function ( Fukushima , 1980 ; Jarrett et al. , 2009 ; Nair & Hinton , 2010 ) , ReLU ( x ) = max ( 0 , x ) , does not have this problem , since in its non-zero regime it has a gradient of 1 . Another advantage of ReLU is it has very low computational demands . Since it is both effective and efficient , it has proven to be a highly choice of popular activation function . The chief drawback to ReLU is it has no sensitivity to changes across half of its input domain , and on average passes no gradient back to its inputs . This can lead to neurons dying1 if their weights make them never reach their activation threshold . If sigmoid is equivalent to converting our feature logits into probabilities , then the ReLU activation function is equivalent to truncating our logits and denying any evidence for the absence of a feature . We hypothesise that omitting negative evidence in this way is undesirable and reduces the generalisation ability of ReLU-based networks . Variants of ReLU have emerged , aiming to smooth out its transition between domains and provide a gradient in its inactive regime . These include ELU ( Clevert et al. , 2016 ) , CELU ( Barron , 2017 ) , SELU ( Klambauer et al. , 2017 ) , GELU ( Hendrycks & Gimpel , 2020 ) , SiLU ( Elfwing et al. , 2017 ; Ramachandran et al. , 2017 ) , and Mish ( Misra , 2019 ) . However , all these activation functions still bear the general shape of ReLU and truncate negative logits . Fuzzy logic operators are generalizations of boolean logic operations to continuous variables , using rules similar to applying logical operators in probability space . Previous work has explored networks of fuzzy logic operations , including neural networks which use an activation functions that constitutes a learnable interpolation between fuzzy logic operators ( Godfrey & Gashler , 2017 ) . In this work , we introduce activation functions which are similar to fuzzy logic operators , but derived for working in logit space instead of in probability space . In this work we set out to develop activation functions based on the principle that neurons encode logits — scores that represent the presence of features in the log-odds space . In Section 2 we derive and define these functions in detail for different logical operators , and then consider their performance on numerous task types including parity ( Section 3.1 ) , image classification ( Sections 3.4 and 3.5 ) , transfer learning ( Section 3.6 ) , abstract reasoning ( Appendix A.14 ) , soft-rule guided classification as exemplified by the Bach chorale dataset ( Section 3.3 ) , and compositional zero-shot learning ( Appendix A.15 ) . These tasks were selected to ( 1 ) survey the performance of the new activations on existing benchmark tasks , and ( 2 ) evaluate their performance on tasks which we suspect in particular may require logical reasoning and hence benefit from activation functions which apply these logical operations to logits . 2 DERIVATION . Manipulation of probabilities in logit-space is known to be more efficient for many calculations . For instance , the log-odds form of Bayes ’ Rule ( Equation 9 ) states that the posterior logit equals the prior logit plus the log of the likelihood ratio for the new evidence ( the log of the Bayes factor ) .Thus , working in logit-space allows us to perform Bayesian updates on many sources of evidence simultaneously , merely by summing together the log-likelihood ratios for the evidence . A weighted sum may be used if the amount of credence given to the sources differs — and this is precisely the operation performed by a linear layer in a neural network . When considering sets of probabilities , a natural choice of operation to perform is measuring the joint probability of two events both occurring — the AND operation for probabilities . Suppose our input space is x ∈ [ 0 , 1 ] 2 , and the goal is to output y > 0 if xi = 1∀ i , and y < 0 otherwise , using model 1Though this problem is very rare when using BatchNorm to stabilise feature distributions . with a weight vector w and bias term b , such that y = wTx+ b . This can be trivially solved with the weight matrix w = [ 1 , 1 ] and bias term b = −1.5 . However , since this is only a linear separator , the solution can not generalise to the case y > 0 iff xi > 0 ∀ i . Similarly , let us consider how the OR function solved with a linear layer . Our goal is to output y > 0 if ∃xi = 1 , and y < 0 otherwise . The binary case can be trivially solved with the weight matrix w = [ 1 , 1 ] and bias term b = −0.5 . The difference between this and the solution for AND is only an offset to our bias term . In each case , if the input space is expanded beyond binary to R2 , the output can be violated by changing only one of the arguments . 2.1 AND . Suppose we are given x and y as the logits for the presence ( vs absence ) of two events , X and Y . These logits have equivalent probability values , which can be obtained using the sigmoid function , σ ( u ) = ( 1 + e−u ) −1 . Let us assume that the events X and Y are independent of each other . In this case , the probability of both events occurring ( the joint probability ) is P ( X , Y ) = P ( X ∧ Y ) = P ( X ) P ( Y ) = σ ( x ) σ ( y ) . However , we wish to remain in logit-space , and must determine the logit of the joint probability , logit ( P ( X , Y ) ) . This is given by ANDIL : = logit ( P ( X ∧ Y ) x⊥y ) = log ( p 1− p ) , where p = σ ( x ) σ ( y ) , = log ( σ ( x ) σ ( y ) 1− σ ( x ) σ ( y ) ) , ( 1 ) which we coin as ANDIL , the AND operator for independent logits ( IL ) . This 2d function is illustrated as a contour plot ( Figure 1 , left ) . Across the plane , the order of magnitude of the output is the same as at least one of the two inputs , scaling approximately linearly . The approximately linear behaviour of the function is suitable for use as an activation function ( no vanishing gradient ) , however taking exponents and logs scales poorly from a computational perspective . Hence , we developed a computationally efficient approximation as follows . Observe that we can loosely approximate ANDIL with the minimum function ( Figure 1 , right panel ) . This is equivalent to assuming the probability of both X and Y being true equals the probability of the least likely of X and Y being true — a naı̈ve approximation which holds well in three quadrants of the plane , but overestimates the probability when both X and Y are unlikely . In this quadrant , when both X and Y are both unlikely , a better approximation for ANDIL is the sum of their logits . We thus propose ANDAIL , a linear-approximate AND function for independent logits ( AIL , i.e . approximate IL ) . ANDAIL ( x , y ) : = { x+ y , x < 0 , y < 0 min ( x , y ) , otherwise ( 2 ) As shown in Figure 1 ( left , middle ) , we observe that their output values and shape are very similar .

Summary: This paper introduces three novel activation functions for neural networks, by modifying the logit-space versions of AND, OR, and XNOR in order to only use addition and comparison operations.  A large number of experiments show that in certain architectures and on certain tasks, some combinations of these activation functions deliver good performance.  While the results are somewhat interesting, I find the activation functions under-motivated (e.g. the motivations assume independence of input features) and worry that many of the experimental settings are designed to favor their new functions.  Similarly, the breadth of experiments means that each one is relatively under-reported in its details.  It would be helpful to have a more developed theory of these activation functions and more experiments in settings that are more standard.