Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: ** post rebuttal start **

After reading reviews and authors' response, I decided not to change my score.
I remark that this paper still requires a lot of revision; comparison in the main paper is somewhat unfair and all new results are in the appendix.
Also, the performance of their replication of the prior method is far lower than reported. In worst case, the performance gain from the compared method would be from their incorrect implementation on the prior works. In this kind of case, I suggest the authors to put {the numbers in the original paper} as well as {their replication} and claim that they fail to replicate the number. Ideally, if their method is evaluated in the same condition, it should outperform prior works in any case.


Detailed comments:

1-(a). Adversarial attack and OOD (which is hard to detect) are closely related: they are both in off-manifold. Their main difference would be, while adversarial attack is very close to the clean data in the data space, OOD is relatively far from the in-distribution in the data space.
Though it does not talk about OOD, you may refer to [R1] for analysis in perspective of manifold. The difficulty of OOD detection can be considered to be coming from overlapped manifolds in the latent space.

[R1] Stutz et al. Disentangling Adversarial Robustness and Generalization. In CVPR, 2019.


1-(b). Though the numbers are much lower than those reported in the original paper, I am happy to see the fair comparison. However, according to the original paper, simple FGSM is used for validation, so I am not sure such a huge difference can actually happen. In this kind of case, I suggest the authors to put {the numbers in the original paper} as well as {their replication} and claim that they fail to replicate the number.


2. I am happy to see that their revised method has better performance.

** post rebuttal end **


- Summary:
This paper proposes an out-of-distribution detection (OOD) method under constraints that 1) no OOD is available for validation and 2) model parameters should be unchanged. They specifically address a problem of the state-of-the-art method satisfying the constraints, and propose a new distance metric inspired by data compression. Experimental results on several benchmarks with different deep neural network architectures support their claim.


- Decision and supporting arguments:
Weak reject.

1. The problem setting is clear and their approach is interesting and makes sense. However, the method for comparison is not properly set. As the authors addressed, Mahalanobis detector proposed by Lee et al. (2018b) requires validation to determine weights for feature ensembling, but the validation can be done without OOD data by generating adversarial samples as proposed in the same paper. Although Table 2 in Lee et al. (2018b) shows that the performance is not better than the case when we have an explicit OOD data for validation, it reasonably works well. Therefore, rather than comparing with the vanilla version (only using last latent space) or the alternative "assemble" method (concatenating all average-pooled features), they had to compare their method with the model validated by adversarial samples, which essentially satisfies the constraints.

2. Also, I wonder the main body of the proposed method itself is really effective or some minor tweak they made is essential. According to Table 2 in the submission, their method is better than "Mahalanobis vanilla" only when all components are applied. Though the idea is interesting, I am skeptical about the effectiveness of the proposed method.


- Comments:
1. As addressed by the authors, feature concatenation ("assemble") is not effective for the Mahalanobis method but the proposed method. How about to do an ablation study about weighted averaging vs. concatenation on the proposed method as well? Again, weights can be validated by adversarial samples to satisfy the constraints.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: After reading the other reviews and comments, I appreciate the effort by the Authors, but it looks like the paper still needs some work before being ready. So, I have decided to maintain my rating.

===================

The work proposes a system for detecting out-of-distribution images for neural networks under strict limitations of not retraining the network or tuning parameters with out of distribution validation data in mind using Compression Distance in a novel way.  The authors evaluate the method broadly and against the state of the art and provide a thorough explanation of the background material and formulation. 

This work is strong in it’s cleverness, novelty, and evaluation when limited to the class of solution stipulated in section 2; however, it is not clear or presented why this restrictive choice is or could be necessary.    For this reason, I am borderline unless that caveat is addressed as described below, in which case I would be happy to accept. 

It is unclear to me as it was not presented in the work when or if the problem that is being solved by the paper is particularly either important or frequent.  Obviously, not having to retrain is more efficient than having to and not having to validate on out-of-distribution samples is helpful in times when we don’t know them ahead of time, but it is unclear to me if it is the case that we will have a situation in which both of the above are, for instance, not possible at all.  To me, if the work could be changed to compare against works which are not so tightly constrained, not for the purposes of holding it to the same standard but to understand it’s relative standing, or to better justify the very strict constraints which somehow, despite out-of-distribution detection being a popular upcoming topic, apparently only has one other paper that matches it.
 		
The paper could use some extra proofreading, for instance the first sentence of the abstract doesn’t make much grammatical sense especially with the first phrase included. 
It may be nice to cite works such as http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.132.6389&rep=rep1&type=pdf and others in that vein as this is certainly not the first work to involve compressive principles in image classification related tasks. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new framework for out-of-distribution detection, based on global average pooling and spatial pattern of the feature maps to accurately identify out-of-distribution samples. MAHALANOBIS distance based methods were discussed, and the shortcomings of using mahalanobis distance were given (i.e. assumption that features are independent, etc). They propose to use compression based distance measures off the shelf from standard compression techniques to detect spatial feature patterns in feature space and demonstrate its effectiveness on several datasets and comparison with baselines is reported and well discussed.

The motivation of the proposed approach is clear, and the method seems novel. However, the experiments could have been done in a more complex setting. The out-of distribution samples pose a danger in safety critical applications such as autonomous driving, for example a car deployed in environment that it has not seen during the training might crash. So, it would be interesting to see the performance of both baselines and proposed approach in those settings where inputs are similar in nature but very different in some aspects. I do not necessarily see something wrong with the paper, but I'm not convinced of the significance (or sufficient efficiency) of the approach. There is also  theoretical guarantees showing exhaustiveness of the proposed methods in detecting all possible out-of distribution examples. 

review_title: Official Blind Review #2

Meta-review: This paper proposes an out-of-distribution detection (OOD) method without assuming OOD in validation.

As reviewers mentioned, I think the idea is interesting and the proposed method has potential. However, I think the paper can be much improved and is not ready to publish due to the followings given reviewers' comments:

(a) The prior work also has some experiments without OOD in validation, i.e., use adversarial examples (AE) instead in validation. Hence, the main motivation of this paper becomes weak unless the authors justify enough why AE is dangerous to use in validation. 

(b) The performance of their replication of the prior method is far lower than reported. I understand that sometimes it is not easy to reproduce the prior results. In this case, one can put the numbers in the original paper. Or, one can provide detailed analysis why the prior method should fail in some cases.

(c) The authors follow exactly same experimental settings in the prior works. But, the reported score of the prior method is already very high in the settings, and the gain can be marginal. Namely, the considered settings are more or less "easy problems". Hence, additional harder interesting OOD settings, e.g., motivated by autonomous driving, would strength the paper.

Hence, I recommend rejection.
Decision: Reject