Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a set of novel activation functions for
artificial neural networks, inspired by the Boolean logic gates for
AND, OR and XNOR.  After defining and motivating the functions, the
paper derives approximated versions that are more computationally
efficient, and demonstrates their use on a few test problems.

strengths_and_weaknesses: ### Strengths
The question addressed by the paper is reasonably original,
and the proposed activation functions are derived and described
clearly.

### Weaknesses
In my view the main weakness of the paper is insufficient
motivation and significance. Why is it interesting to have activation
functions that mimic logical gates? No clear motivation is given,
either from a theoretical standpoint (e.g., networks with such
activation functions may be expected to learn more efficiently for
some reason), or from the standpoint of biological inspiration (a few
words are spent on dendritic computation, but logical operations are
not what defines dendritic computation - they are just one particular
abstraction that is sometimes applied to it). At the same time, in the
experiments section there is no clear sign of particular empirical
advantages conferred by the proposed activation functions. Therefore,
to summarize, the logical activation functions, although defined in a
consistent and reasonable way, lack a-priori theoretical motivation as
well as a-posteriori demonstrations of empirical usefulness.

I'll now expand on what I meant by "no sign of empirical advantages"
above. In my view, the results section presents an overly-optimistic
interpretation of the empirical results. Throughout the various
examples, a range of choices of activation functions is explored, but
no choice is shown to be consistently better than the others across
tasks (perhaps with the exception of learning the parity of the
inputs; but that case is more of a sanity check, as it is the natural
operation implemented by the XNOR gate). Indeed:
- in 4.2 (MLP on Bach chorales) there is no statistically significant
  advantage;
- in 4.3 (MNIST) the results are mixed and overinterpreted: for the
  MLP the text says that one of the proposed gates is matched for best
  place with one of the alternatives considered for comparison, but
  this is not even evident from the plot (there are hyperparameter
  regimes where that choice of gate is the one that performs the
  worst). For the CNN case, a different choice of logical-gate
  activations is claimed to be among the ones performing best, but
  again from the plot it looks like the "Max" activation function is
  the best overall.
- in 4.4, the gate that was identified as the best performer in
  4.3/MLP is now the worst performer, so much so that it is not even
  included in the plot. The other gates all produce mixed results with
  no clear winner. The activation that most consistently performs
  better than others in this section seems to be SiLU, but in the text
  it is grouped together with Max and ORAIL (one of the proposed
  gates). For some reason the results of this section don't have error
  bars.
- section 4.5 (transfer learning) also fails to find a scenario where
  the proposed activations consistently perform better than the
  alternatives.

Finally, the experiments section is at times made hard to interpret by
lack of methodological details. For instance, it is not clear to me
how the number of parameters is balanced across activation function
choices, since the different structure of 1-to-1 and 2-to-1
activations (e.g. ReLU vs the activations introduced in this paper)
imply a different organization of the network. Moreover, some p-values
are given without giving any information on the test (such as line
262, "For the MLP, XNORAIL performed best along with signed_geomean (p
< 0.1)", and some results seem to lack any statistical analysis/error
bars (e.g. Figure 6).

questions: 1. Why is it important for network to possess Boolean activation
  functions, given that large enough networks can approximate any
  arbitrary function?
2. Of all the combinations of activation functions explored in the
  experiments section, which one should I pick to use in my next deep
  net architecture? Is there a principle, a strategy or a rule of
  thumb that I could use to choose one (or more) activation function
  without having to train and test my network with all possible
  alternatives?

limitations: I couldn't find much in terms of explicit analysis of limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new set of activation functions based on the approximation of  logit arithmetic AND/OR/XOR operation. The effectiveness of the proposed activity functions are demonstrated on many tasks including image classification, transfer learning, abstract reasoning, and compositional zero-shot learning.
strengths_and_weaknesses: **Originality**:

* (+) To my knowledge, the idea proposed in this paper that uses  n->1 activation functions via approximating logit-space boolean operations is novel and interesting

**Quality**:

* (+) The paper technically sounds correct and claims well supported by theoretical analysis and experimental results.
* (+) Related works are covered and discussed.
* (+)  Experiments are conducted extensively in different type of tasks and the results are discussed thoroughly.


**Clarity**:

* (+) This paper is well written and organised.
* (-) it would be better to have a table accompanying figures such as figure 6 as the result can be difficult to read given the amount of legend. Especially for figure 5, 6, the table for legend given is impossible to distinguish between silu and OR, XNOR d



**Significance**:
As demonstrated in the experimental results, the accuracy gain in CIFAR datasets in the transfer learning tests are quite big. I would assume that the proposed activation functions can be quite effective in certain types of tasks. 

questions: * Does it improve or worsen the training time? 
* Why is the performance of AIL better than IL in most cases(such as show in Table 1)? Especially in cases like stfCars that XNOR_IL has barely any accuracy. 
* Can different layers use different settings? (some use XNOR, some use XNOR/OR d, etc)


limitations: The authors did discuss the limitations of the proposed method, such as the method being suitable for certain tasks based on experimental results.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper the authors presents new activation functions based on a relaxation of the logical operators AND, OR, XNOR. Moreover, the authors present approximations that are more efficient based on simple max, min and addition operations.  This is motivated by the behavior of the operators under the logit space equivalence. As these binary operators are reduction operations, i.e., they convert two inputs into one, the authors connect to the MaxOut. Furthermore, they provide ensemble alternatives to adapt neural network architectures to use the presented boolean activation functions. Finally, the authors provide empirical results showing how the different potential combinations of logical activation functions behave for different datasets and architectures.
strengths_and_weaknesses: Here, the authors present new activation functions. One weakness could be that they make independency assumptions, however, this does not seem to negatively affect the performance of the models. 

The empirical evaluation is sound, and demonstrates the activation functions behave well and even improve the performance in some cases. They achieve this without significantly adding more parameters by using ensembles of activation functions. This is an interesting contribution as previously (MaxOut) would significantly increase the number of weights in order to have the same number of outputs. 



questions: Do you think that these activation functions could provide a path for explainability of neural networks?

I see this paper somehow related to continuous logic functions, and I'm wondering whether composition of more complex logical functions would also make sense, e.g., introducing an implication activation, etc.


limitations: no negative societal impact
review_title: null

Meta-review: metareview: The review ratings were above the acceptance threshold. The reviewers valued quite positively the originality of this paper that studied advantages of multivariate activate functions, as well as extensive numerical experiments across a wide range of tasks in order to explore their effectiveness. Upon reading the reviews, the author responses, subsequent discussion between the reviewers and the authors, as well as the paper itself, I thought that the restriction of the consideration in this paper to those derived from approximation of Boolean operators on independent Bernoullis was not well motivated nor described. At the same time, the empirical evidences demonstrating the potential usefulness of the proposal are quite interesting, so that I would like to recommend acceptance of this paper, and would expect further discussion on this subject among the attendees of the conference.
confidence: Certain
Decision: Accept