Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper proposed a Sparse Activation Function Search (SAFS) method for optimizing the activation functions and training hyperparameters of Sparse Neural Networks (SNNs). The authors show that the accuracy drop in SNNs, especially at high pruning ratios, can be attributed to the use of ReLU as the default activation function and the use of the same hyperparameters as dense counterparts. The authors analyze the impact of activation functions and training hyperparameters on the performance of SNNs and propose a new AutoML approach, SAFS, to optimize the activation functions and training hyperparameters of SNNs. The results demonstrate significant improvements in the accuracy of VGG-16, ResNet-18, and EfficientNet-B0 networks trained on CIFAR-10 and ImageNet-16 datasets, especially at high levels of sparsity.
overall_review: Positive:
1) Novel approach: The paper proposes a novel approach that combines hyperparameter optimization (HPO) and activation function search for sparse neural networks (SNNs). This approach is a significant contribution as there is no efficient approach for optimizing activation functions in SNN training.

2) Significant improvement in accuracy: The paper shows that the SAFS approach results in up to 8.88% and 6.33% absolute improvement in the accuracy of VGG-16 and ResNet-18 models, respectively, over the default training protocols, especially at high pruning ratios. This improvement is substantial and demonstrates the effectiveness of the proposed approach.

3) Addresses underexplored factors: The paper identifies and addresses two underexplored factors that impact the accuracy of SNNs, namely the choice of activation functions and training hyperparameters. By analyzing these factors and proposing the SAFS approach, the paper makes a significant contribution to the field of SNNs and deep learning.

Negative:
1) Limited comparison: This paper only compares its proposed approach, Sparse Activation Function Search (SAFS), with the default training protocols and the magnitude pruning method in 2015 for three popular DNN models trained on two datasets. More comparisons with other state-of-the-art methods and datasets would provide a more comprehensive evaluation of SAFS.

2) Lack of thorough analysis: The paper briefly mentions the impact of activation functions and training hyperparameters on the performance of sparse CNN architectures, but it does not provide a thorough analysis of these factors. A more detailed investigation of these factors could provide insights into why SAFS is effective and could lead to further improvements.
review_summary: The idea in this paper is novel, but empirical evaluation needs to be improved, including more experiments and more implementation details.
actions_required_to_increase_overall_recommendation: see Technical Quality And Correctness and Overall Review
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper proposed to search for activations and tune hyperparameters for sparse neural networks. Searching for layerwise activation functions is a new perspective to improve the performance of sparse networks. It showed the effect of different activation functions. 
overall_review: This paper provides a new perspective to boost the performance of sparse networks. I am not sure if searching for activations really works. I am not sure if we can simply tune parameters to get the similar performance gain as the proposed SAFS. 
review_summary: My initial decision is borderline accept. I raised some concerns, especially if we can simply use HPO to get the similar performance gain as the proposed method. I am not sure if we really need to search for activations for sparse networks. I hold room to decrease scores if my major concerns are not addressed after rebuttal. 
actions_required_to_increase_overall_recommendation: Please add experimental results to ease my concerns.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper introduces a technique for improving sparse neural networks by optimizing the hyperparameters and activation function in addition to the sparsity mask, which are often ignored in the sparse neural network literature. This search space over activation functions comprises a set of unitary functions with learned scaling of the inputs and outputs, and a hill-climbing algorithm is used to optimize the activation function, with experiments conducted on CIFAR-10 and a resized version of ImageNet. 
overall_review: Strengths:
- This combination of research areas appears to be novel and fills an often-overlooked gap in the literature (tuning activations and hyperparameters of sparse neural networks)
- The current set of experimental results seems promising
- The ablation study of the different stages of the proposed search method is a useful experiment to include. 

Weaknesses:
- As an empirical work, it would be ideal to experiment with more than two datasets. Currently, only CIFAR-10 and resized ImageNet are evaluated. On the other hand, this is potentially understandable as these experiments are likely rather expensive. 

review_summary: The idea is novel and is well explored on the two datasets that are included, CIFAR-10 and resized ImageNet, however, the evaluation could be improved by including more datasets (particularly since this is an empirical paper). 
actions_required_to_increase_overall_recommendation: I would suggest that the authors include at least one additional dataset in their empirical evaluation.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper proposes SAFS, an automated search for activation functions for sparse neural networks in conjunction with hyperparameter optimization. This method is a two-step procedure, which uses the already pruned neural network as starting and initialization point; first, the activation function for each layer in the network is searched using Late Acceptance Hill Climbing. Second, the hyperparameter for this identified network is optimized.  This paper evaluates SAFS on three different common CNNs (ResNet-18, VGG16, and EfficientNet-B0) for CIFAR-10 and ImageNet16 image classification and shows improved results over pruning baselines, which use the default HP of the dense network. 
overall_review: Strengths:
This paper proposes an interesting research question with sound experiments.
Good motivation and clear assessment of why this is important (given the studies in Sections 5.1 and 5.2)
The idea to search for activation functions and HPO in conjunction is novel. 

Weaknesses:
How does this method scale to larger networks, e.g., ResNet-50, and larger image resolutions (Imagenette, ImageNet 100) with the focus on how the input image resolutions influence this method. 

review_summary: While the idea of combining activation functions and HPO for sparse neural networks is novel and great, I believe some additional assessments and experiments are useful to improve this paper.
 

actions_required_to_increase_overall_recommendation: Additional experiments regarding the scalability of the proposed method or at least a discussion: 

-	Larger networks like ResNet-50.
-	Larger Datasets with larger resolutions, Imagenette or ImageNet100 

Different activation functions as ablation for the used dense networks. 

**Post-Rebuttal:**
I updated my score from 6 to 7.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: In the space of sparse neural networks (SNNs), this paper identifies the lack of SNN-specific development of activation functions among other design choices and proposes Sparse Activation Function Search (SAFS), a method for optimizing layer-wise SNN activation functions in a mixed-integer search space. 
overall_review: **Positive aspects:**

* The problem is motivated clearly.

* Aside from missing experiments below, the authors cover a good range of experiments and results, from explorations to define the problem in Figure 4, main results in Table 1, insightful analysis in Figure 5, and some ablations in Table 2.

**Negative aspects:**

* There is minimal mention (Table 6) nor any comparison of algorithmic cost of SAFS versus baselines and other approaches. This is important information for readers and practitioners to consider the trade-off of computational cost versus performance gain. This is also crucial for Figure 4(a): were all runs given equal computational resources? If not, you need to rerun these experiments but use the equivalent computational cost of SAFS for each baseline model with a single activation function. Computational cost information should also be included at least for all experiments in Table 2. 

* The choice of LAHC over other discrete search algorithms is only supported by Figure 6. Figure 6(a) doesn't seem very conclusive due to noise I don't completely understand Figure 6(b): is this the same data but with some sort of smoothed/windowed average? This choice could be further supported by including it in the ablation studies.

* The idea of Section 5.6 seems interesting but not yet completely developed. Do the patterns you noted extend to other backbones and tasks? With only 3 runs represented in Figure 5 (and only their averages, no measures of variance), it's not clear how much signal there is compared to noise. Also, I don't understand what is being plotted: what are the units of the colorbar?

* In Section 5.2, it's not clear what you are using "three-fold validation" here for, since you are comparing networks with different activation functions (with no inner optimizations) and Figure 4(a) shows test accuracy, not validation accuracy. 

* The breadth of backbones and tasks is relatively narrow. How does this method fair in other modalities, network types, etc.? Can the same unary operator search space be used, or does it need to be adapted to the backbone and input modality?

**Further Experiments:** (could be included in follow-up work)

* Your ablation studies are a good start, but since Figure 4(b) shows significant gain from just HPO, I would like to also see an ablation that only removes HPO (as well as an ablation that only includes HPO). 

* Your search space is well divided into selecting a unary operator and it's two continuous parameters. Your algorithm approaches these decisions in series, but I think tackling them in parallel could be more effective. Instead of having two stages, instead initialize $\alpha$ and $\beta$ for each activation function and each layer. Then, the matrices containing these parameters could be optimized by SGD simultaneously to the discrete optimization of $\Psi$.


**Minor comments:**

* Line 69: "A number of studies have proposed the necessity of pruning..." - the word "necessity" doesn't feel right here. Perhaps "...have proposed to prune..." instead?

* Figures 1 and 2 appear out of order.

* For Figure 5, right-align the bottom labels and rotate them by the right side, rather than the center, using something like this: 
```plt.setp(axs[i].get_xticklabels(), rotation=45, ha="right", rotation_mode="anchor")```
review_summary: This paper is a clear and impactful study on activation function optimization for SNNs. It could be further supported by additional experiments detailed above. I think it is worth accepting, especially if the recommended actions are taken.
actions_required_to_increase_overall_recommendation: * Adding experiments and results requested (in order of importance):
  * Comparison of computational costs for Table 1 and Figure 4(a)
  * Ablation of LAHC vs. other algorithms and of HPO for Table 2
  * Experiments outside of tiny natural image classification
  * Compare to parallel optimization of the unary operator and its parameters
* Expand and clarify analysis in Section 5.6

In response to the additions discussed below and in the general response, I have raised my score from 7 to 8.
review_title: null

Decision: Accept(Main Track)