Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper develops and presents approximations of three Boolean logic based activation functions: AND, OR, XNOR. These activation functions are based on the principle that neurons encode logits to represent presence of features in the log-odds space (logit-space). The formulations are simple and straightforward. Empirical evaluation is conducted on several applications using the presented activations (under individual and ensemble configuration), where ReLU, max, min (with adjustments) are the competitive baselines.

main_review: I do not think the paper and the idea is well motivated. The paper states that sigmoid is less applied and that ReLU is now applied more. But, the introduction does not explain why or make the connection to why we need these Boolean logic based activation functions. Some theoretical analysis would be helpful.

There is mention of biological networks, but it is not well cited by the paper.

In the first paragraph, I found these states somewhat loose:
- "an activation function is a non-linearity..."
- "this will convert the logits of features into probabilities."

Although several experiments were performed, I did not find the reason for why certain datasets & architectures were chosen as it relates to the presented activation functions. Also, why and when would the presented activation function perform better or worse than the used baselines? 

In Table 1, I noticed it is interesting that the variance across (not +/- values themselves) the various activation configurations appear low (e.g., CIFAR10 concentrated around 82.x, STL-10 concentrated around 94.x).

In Figure 5, I was curious to know why the hyper-parameters were determined by random search. Was this uniformed? How does other popular methods (e.g., line search) affect the results?

summary_of_the_review: While the methods presented are simple, the paper needs more work on motivating the methods presented and connecting it to the empirical evaluations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces three novel activation functions for neural networks, by modifying the logit-space versions of AND, OR, and XNOR in order to only use addition and comparison operations.  A large number of experiments show that in certain architectures and on certain tasks, some combinations of these activation functions deliver good performance.  While the results are somewhat interesting, I find the activation functions under-motivated (e.g. the motivations assume independence of input features) and worry that many of the experimental settings are designed to favor their new functions.  Similarly, the breadth of experiments means that each one is relatively under-reported in its details.  It would be helpful to have a more developed theory of these activation functions and more experiments in settings that are more standard.
main_review: # Summary

This paper introduces three novel activation functions for neural networks, by modifying the logit-space versions of AND, OR, and XNOR in order to only use addition and comparison operations.  A large number of experiments show that in certain architectures and on certain tasks, some combinations of these activation functions deliver good performance.  While the results are somewhat interesting, I find the activation functions under-motivated (e.g. the motivations assume independence of input features) and worry that many of the experimental settings are designed to favor their new functions.  Similarly, the breadth of experiments means that each one is relatively under-reported in its details.  It would be helpful to have a more developed theory of these activation functions and more experiments in settings that are more standard.

Strengths:
	* Very large set of experiments
	* Novel design of new activation functions

Weaknesses:
	* New activations not sufficiently motivated (hinging on assuming independence of input dimensions)
	* _Too many_ experiments which are (a) insufficiently described and explained, and (b) may be biased towards favoring the new activations

# Minor comments

* p 4: "assuming inputs encode independent events": the motivations for the activations all depend on this independence assumption, which does not seem justified to me.  Can the authors explain why this is desirable?

* p 5: "access to all 16 boolean functions".  Since {and, or, not} is a complete basis for boolean functions, why not just look at an equivalent of not?

* Experimental details under-reported: e.g. a $p$-value is reported in 3.2, but it's unclear what the sample of results is (different random seeds?).  And in Section 3.5, it's unclear where to find the actual results supporting the claims in the second paragraph.  And in 3.6, there is merely a giant table of results, without any explanation of the results contained therein.


# Typographic comments

* p 2: "plus the the log" --> "plus the log"

* p 2: the sentence starting "If we suppose" doesn't have a consequent / main verb
summary_of_the_review: New activation functions are introduced and tested in a very wide range of experimental settings, but I find the motivations not sufficiently developed and the experimental results under-reported and motivated; more theoretical development would help assess this contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper presents a new set of activation functions based on a relaxation of the logical operators AND, OR, XNOR. This is motivated by the behavior of the operators under the logit space equivalence. 
Moreover, the authors present approximations that are computationally more efficient based on simple max, min and addition operations.
As these binary operators are reduction operations, i.e., they convert two inputs into one, the authors note the similarity to the MaxOut and provide different alternatives on how to adapt the architectures to use the boolean activation functions.
Finally, the authors provide empirical results showing how the different potential combinations of logical activation functions behave for different datasets and architectures.
main_review: The paper is technically sound and well-written, with some very minor comments on the latter. 

In particular: 
- You should list the activation functions you reference in the introduction by their names.
- Table 1 breaks the margins, this is unacceptable. The same holds true for some tables in the appendix.
- In section 2.4 you mention “For simplicity, we restrict ourselves to considering only k = 2, but note that these activation functions are generalisable to higher dimensions.”, and this is a perfectly valid argument. However you continue with “On the understanding that Goodfellow et al. (2013) found k = 2 yielded the best results for MaxOut networks, we anticipate the same will hold for the AIL activation functions.” Here I recommend you either drop that sentence, or provide more evidence for the conjecture, either via more experiments or a stronger argument of why you believe that is the case. 
- Although Figs. 1-3 are great and help the reader understand better. Figs. 4-6 would benefit from improving the formatting. The label on Fig. 4, could be further improved to describe what we are seeing there. What do the colors mean? And in general for figs. 4-6 the numbers are too small.

On the content side, I do see the connection to MaxOut, however your activations are potentially better when they fall in the x+y regime, as the gradient would back-propagate on both sides, unlike MaxOut where you always only update either x or y. I believe this is potentially an advantage and could be mentioned in the paper. But at the same time, that makes me wonder whether it would behave the same as maxout for the k>2 experiments.

The biggest problem that I see with the paper is the weak empirical section when it comes to stacking your activation functions against other alternatives. There are many other alternatives and you even mention some of them in the introduction. However, you do not compare against them. Furthermore, the recent advances in learnable activation functions further improve the performance of DNNs beyond what ReLU can do.
Finally, I believe the work in [1] deserves at least some comments in your paper as related work.

[1] Godfrey, L. B., & Gashler, M. S. (2017, October). A parameterized activation function for learning fuzzy logic operations in deep neural networks. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 740-745). IEEE.

summary_of_the_review: I like the activation functions introduced by the authors in this paper. They seem like a nice extension to MaxOut with semantical meaning on the operations they carry.

The weakness I see here are that the empirical section should compare against other activation functions and the missing related work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes new activation functions based on the approximation of AND, OR, XOR operations. In addition, ensembling strategies are proposed to combine features from these operators. The experiments on image classification, transfer learning, abstract reasoning, and zero-shot learning tasks show that the proposed activation functions perform reasonably well, especially OR for ResNet and an ensemble for zero-shot and transfer learning.
main_review: ### Strengths
- The paper is clearly written.

- The core idea is novel and sounds. 

- The paper provides strengths and weaknesses of the proposed approach with diverse experimental setup. 

- The experiments include a good amount of tasks and data types from the image classification to transfer learning, abstract reasoning, and zero-shot learning on music, images, etc. 

### Main questions and suggestions
1. Does "Linear Layer only" in Table 1, 5, 6 mean without any activation function?  In these tables, linear layers performs surprisingly well on some datasets with a very small amount of parameters (101k) compared to the models with the proposed activation functions (around 700k). 
    1) Does this mean that the tasks/datasets are too simple? Or is there another reason that linear layers work reasonably well? Could authors comment on this phenomenon?
    2) Is there a reason to use a small model for the "linear layer only" model? Otherwise, similar parameter setup seems fair for comparison. 

2. The performance of MaxOut is comparable to ${OR_{AIL}}$ for image-based tasks. Does this mean that most of the pre-activations become negative? Could authors provide more analysis between Maxout and $OR_{AIL}$? Plotting a maximum response histogram of the activations (similar to the one in the MaxOut networks paper (Goodfellow et al., 2013)) would be one way to compare their behaviors. 

3. Experiments on more complex data like Imagenet would be useful to see how the proposed approach scales and whether the model is able to learn higher-order representation. 

### Other suggestions
4. Authors mainly applied the proposed activation functions for MLP and CNN-based models. Can these activation functions be used for other common models like RNNs/GRU or transformer?

5. The related works listed in the paper seem limited. Adding previous works on proposing other activation functions would be helpful.  

Minor:
Table 1 exceeds the paper size. The size needs to be reduced. 

summary_of_the_review: In general, the paper provides great idea, insights, and analysis. I added my major concerns in main questions and suggestions above. I am willing to increase my score once my concerns are addressed. 

---------------------------------
### After rebuttal

I read the responses and other reviews. I generally like the idea of this paper and how it's written. However, my concerns are not fully addressed in the response, and I also agree with some of other reviewers' concerns. I keep my score.
review_title: null

Meta-review: The paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approxiThe paper is well written and deals with a simple yet interesting introduction of approximate Boolean logic activation functions. A number of comparison experiments showed intriguing differences for problems with potential logical structures. Authors suggest a probabilistic rational/motivation, i.e., computation in logit-space, however, more theoretical investigation is critically needed to answer why they perform the way they do. There are a lot of activations in the literature, so perhaps it is not easy to make a distinct contribution in performance. Despite the large number of experiments the reviewers were not convinced on how they support authors claims and contributions. The reviewers and AC strongly encourage the authors to keep the direction and improve the paper for another conference.
Decision: Reject