Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an approach that supports better performance when out of distribution cases occur. It does so by letting neurons be of only compact support and thus if the input is out of distribution (OOD) it is expected to be outside that support and therefore the output will be zero. This is used to detect OOD examples. A parameter alpha is used in the algorithm that determines the size of the support. When it is small then the network acts very similarly to a regular network and when it increases the support is limited. Therefore, to make training stable, they start with a small alpha and then increase it throughout the training. 
Then in inference, a value for alpha should be selected that balances the classification accuracy and the OOD detection. For small alpha the classification accuracy is relatively good, yet, the OOD detection is virtually zero. When alpha is very large the OOD detection rate is very high but the classification accuracy deteriorates significantly. 

The strategy proposed in this work is quite interesting and might be useful. Yet, I have some concerns:

To make the comparison fair for all the methods, the same test error should be used. As can be seen from the graphs, the OOD decreases significantly when the Test error decreases. 

Also, a discussion on the selection of alpha is missing. How it should be selected in a real application? 

Another issue is related to some recent works 

How this work compares to 
-Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In International Conference on Neural Information Processing Systems, 2018


-https://proceedings.neurips.cc/paper/2019/hash/1e79596878b2320cac26dd792a6c51c9-Abstract.html

- https://openaccess.thecvf.com/content_CVPR_2020/html/Hsu_Generalized_ODIN_Detecting_Out-of-Distribution_Image_Without_Learning_From_Out-of-Distribution_Data_CVPR_2020_paper.html


-https://openaccess.thecvf.com/content_ICCV_2019/html/Yun_CutMix_Regularization_Strategy_to_Train_Strong_Classifiers_With_Localizable_Features_ICCV_2019_paper.html


Review Update:
I am raising my score but I think the following changes are still required in the paper: 
1. Add a section about the choice of alpha (current addition is lacking) discussing the robustness of alpha selection and how easy it is to select it on real problems.
2. Compare to the methods mentioned.
3. Display the proposed approach on deep networks. Does the proposed approach apply there? Given that current SOTA uses very deep networks (with skips) it is important to be able to use the proposed method with such networks.




review_title: The

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, authors propose compact support neurons to prevent high confidence responses from examples that are away from the training data. The design and training of such a neuron seem novel. 

The reviewer has the following comments:

1. In (3), what is R2? The missing definition of R2 adds difficulty to fully interpret subsequent equations. 

2. Authors start with the RBF neuron, and then replace it with a compact support neuron. However, the relationship between these two seems not very well-explained.

3. From experiments, the proposed method shows effective for the out-of-distribution (OOD) sample detection task. Authors may like to use cifar-10 and cifar-100 as the example to further explain the out-of-distribution detection. For example, how the conceptual overlap across these two datasets contribute to the results? How does a typical CNN perform for OOD detection?

4. While appreciating proposed OOD detection capability, the reviewer is not fully convinced that a compact support network is in general beneficial to knowledge representation. Of course, such a debate might appear slightly out-of-scope here. 
review_title: This paper proposes compact support neurons for deep networks.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary

The authors propose a new neural network unit and training algorithm in order to improve OOD detection. Units can smoothly be changed between standard (dot-product) and RBF type through a shape hyperparameter. During training, this hyperparameter is slowly moved in the direction of the RBF shape. Empirical comparisons on three OOD problems are presented, showing that the proposed approach is competitive.


### Score
The authors propose a novel algorithmic idea for OOD detection and show that it is competitive. However, the results show that their approach is highly sensitive to the choice of the shape hyperparameter alpha. In practice, robust ways of choosing and annealing the shape hyperparameter alpha will be needed for the method to be useful. As detailed below, I think this aspect is not addressed in satisfactory detail. Hopefully the weaknesses and comments will be addressed by the authors during rebuttal, currently, I view the paper as marginally below acceptance.

### Pros
+ New methods to reliable detect OOD samples are potentially impactful, since they can be applied across different domains
+ The authors present a novel algorithmic idea to the best of my knowledge
+ CSNNs overall show good performance when compared against alternative approaches

### Cons
- The results show that the authors method is highly sensitive to the choice of the shape hyperparameter alpha. For their empirical comparisons the authors chose alpha as follows: “Based on these insights, for each dataset we chose the classifier corresponding to the largest α where the test error takes a value comparable to the other methods compared, and reported the AUROC and NZ values in Table 1”. This selection strategy can hardly be transferred to practice — it would mean training another OOD method first, in order to select a suitable alpha. For the proposed algorithm to be useful in practice, it is central to devise and demonstrate robust schemes of finding the shape hyperparameter that do not rely on running another method. Else, methods that can be more easily used and show comparable performance, such as deep ensembles, may be preferable despite higher training times
- Similar to the UCQ paper, it would be worthwhile to report performance of larger ensemble methods, e.g., ensemble of 10 networks
- No variability in performance metrics is provided in Table 1, runs should be repeated and reported with different seeds and measures of variability (e.g. the standard error of mean) should be reported

### Additional comments
- My understanding from reading the paper is that neurons with zero outputs are pruned. In figure 9, the amount of neurons with non-zero outputs (NZ curve) seems to quickly go down to 0. Wouldn’t this imply that all neurons in the output layer are pruned, i.e., how can such a network still be used?
- The curves for train and test error of MNIST in Figure 9 seem to be identical, is there an explanation for this?
- Reference section should be revised carefully, e.g.: Goodfellow et al. (2014) was published in ICLR 2015, Hendrycks et al. (2017) misses venue


### Update

The authors have not sufficiently addressed my comments regarding the choice of alpha in practice. While a possible strategy was proposed, it has not been evaluated empirically, see comments below for more details. I am thus maintaining my score.
review_title: Novel algorithm for OOD detection, choice of alpha unclear in practice 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The main contribution is that the author porposed a formulation which connects the RBF neuron and RELU neuron in neural network to overcome the difficulties in training a DNN model with RBF layer(s). 

pros:
- This paper is well written and easy to understand. The illustration summarizes well its effect of OOD predictions.
- The formulation of eqn.3 is interesting and naturally connect RBF and RELU neurons with a parameter $\alpha$. 

cons:
- The essential idea of this compact support network is based on the assumption that $L_2$ distance of the representation space is meaningful. Is that correct? How about other metric? How about to pre-calculate the distance on dataset in the representation space, and reject those samples which are away from the class center?
- The performance gain of proposed method comparing to other baseline methods which can also stable the training of RBF network is marginal (espeically in Table.1).
- The backbone and the training datasets of RBF network are quite small, could the proposed method stable the training of large network with large dataset (such as training deeper ResNet on Imagenet)?
review_title: Interesting work that connects RBF neuron and RELU neuron in neural network

Meta-review: The paper presents an approach that supports better performance when out of distribution cases occur, by letting neurons be of only compact support and thus if the input is out of distribution (OOD). 

Pros:
- The proposed strategy is interesting and may be useful.

Cons:
- The choice of the parameter alpha, whose value is crucial to the success in experiments, is left murky. The approach suggested by the authors was not validated experimentally. 
- There is insufficient comparison to recent works.
Decision: Reject