Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: A surprising observation in the OOD literature is that OOD detection based on density estimation fails on some image datasets since the learned density tends to be higher on OOD data. A hypothesis for this phenomenon is that the likelihood is dominated by the local features of the image, resulting in the high-level features being ignored. Motivated by this observation, this paper proposes a model that learns local features of images. This model is used for two different applications: (1) by combining it with a full density estimation model, it gives a better OOD detection method, achieving state-of-the art performance (2) by pairing the model with the artihmetic coding algorithm, it gives a dataset compression method that achieves state-of-the-art compression rates on in- or out-of-distribution datasets.
main_review: Both the ideas of utilizing the problematic and puzzling phenomenon of failures of density-based OOD detectors for a different approach (compression) and likelihood-ratio of the full and local model are novel and valuable. The reported results are impressive, making this method a promising approach for both these tasks. In the compression task, being able to achieve such results with way smaller models is also impressive.

Questions/comments:

- I am wondering how the runtime of different methods compare. The paper does comment on how the runtime of their proposed method scales and regarding its parallelizability, but an additional table of results showing the exact time for compression/decompression is useful.

- Why are many entries in table 3 missing? (although, they are irrelevant for all but the last column, since the AUROC of the proposed method 1)

---
Post-author response update: I appreciate authors taking the time to respond and provide table of run-times. I have read other reviews and responses and decided to keep my score the same.
limitations_and_societal_impact: I cannot think of limitations beyond what is mentioned in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new probabilistic image modeling approach based on a local autoregressive model. The local autoregressive model is implemented like the masked convolution operation in the pixelCNN model. The model is applied to two downstream tasks of OOD detection and lossless image compression. The OOD score is computed using the ratio between the likelihood from the full model versus the one from the local model. The idea is simple but shows some improvement over previous models. 


main_review: [Update] I have read the author responses and comments by the other reviewers. My previous concerns about clarity and significance have been partly resolved, so I'm changing my score from 5 to 6. 
--------------------------------------------------------------------- 

The main idea comes from the intuition that the local features are shared between different image datasets and dominate the generalization ability, which is interesting. 

In terms of clarity, it was quite hard to follow the main idea of the paper at first, probably because it is not clearly introduced how the core model is connected to the two downstream tasks which seem quite unrelated. Moreover, the target tasks are explained in sections 2, 4, and 5 while the main component of the proposed approach is explained in sec 3. Also, the methods and results are mixed in the same section.  For better readability, the overall structure needs to be improved, for example, the authors may provide more introduction about the overall flow of the paper in the beginning and rearrange the following materials. 

It is not clear from Table 3  if the performance gain over the previous methods for OOD detection is significant. Actually, it is not clear where the main focus of this paper is.  It might be better to choose one target task (probably OOD detection) and put more emphasis on that by investigating the performance behavior in more details. 

As the author argues, time complexity is an important factor in the case of lossless compression.  I wonder how the actual encoding and decoding time compares with those of other methods. 
limitations_and_societal_impact: There is no explicit discussion on the limitations or societal impact of the proposed work, but there appears to be no direct negative societal impact from this work. It will be helpful if more discussion on the limitations of the proposed method is included.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Summary: This paper study on applying generative models on unsupervised out-of-distribution detection. Authors hypothesize that local features common to both in-distribution and out-of-distribution data dominate the likelihood of generative models. The authors show evidence of the hypothesis via comparing the fully autoregressive model’s likelihood and autoregressive model with local dependencies. Motivated by the results, the authors propose a ratio between the likelihood of model trained with full-dependencies and the likelihood of model trained with local dependencies. The authors show that the proposed metric outperforms conventional generative-model-based methods on CIFAR-10 and FashionMNIST domains. Finally, the authors apply local autoregressive models on universal lossless compression. The proposed compression scheme outperforms conventional methods.

main_review: UPDATE : 
I acknowledge that I have read the author response as well as the other reviews.
Although authors replied my concerns on the novelty, I still believe the main intuition of the paper, "directly model the in-distribution dataset, using only local feature information" is straighforward application from the idea of previous works. 
Due to the previous reasoning, I insist deleting section 3.2 and Figure 1. 
However, I understand the virtue of the method on unsupervised novelty detection, which doesn't need specific augmentations.
Furthermore, I appreciate the clarifications on lossless compression.
Therefore, I am changing my score to 5 and confidence to 4.

=========================

UPDATE (2):

While this is relatively minor, I question the following points in the reply.

Our major goal of OOD detection experimental work is to show that our simple and principled approach can perform comparatively strongly. We believe our experimental results strongly support this claim. At the time of submission, we are not aware of any data augmentation free, model ensemble free methods, capable of consistently outperforming our approach.

=> Consider looking at the paper "Multiscale score matching for out of distribution detection" [https://arxiv.org/pdf/2010.13132.pdf]

While the experiments are performed only on CIFAR-10 vs SVHN, the stark contrast on the performance reasons me whether the scheme proposed by the authors is competitive, or just a method that can be outperformed by a better generative model.
===========================================

Update (3)

I acknowledge that I read the response from the authors as well as the other reviews. I believe that the methodology proposed in the paper shows State-of-the-art performance among augmentation-free unsupervised out-of-distribution detection methods. However, I'm still on the side that the main novelty of the paper is relatively minor. Therefore, I still stick to the current ratings. 

==========================================

Pros
-       Application on the universal lossless compression task is interesting, while this is not the first work to apply generative models on lossless compression. Furthermore, the proposed method outperforms conventional methods
Cons 

-       My deepest concern on the paper is that the proposed hypothesis is not new and published in NeurIPS 2020. We can find in [1] where the authors verified that GLOW and RealNVP encode representations overfitted on local pixel correlations. I would not be surprised that the same thing happens to PixelCNN since they all failed in unsupervised anomaly detection [2] and they both exhibit an encoder-decoder-like structure. As a result, this paper sounds like an extension of existing ideas to a different generative model. 
 
-       Experiment results are limited. In table 3, the authors compare their proposed metric against diverse OOD detection methods. However, the authors compared their results against comparably weak baselines. I can find other published practices [3] [4] in unsupervised novelty detection that outperform the result. 

Overall, I think the main idea of this paper is nothing different from the idea of [1], and the experimental results are not compelling or convincing compared to recently published unsupervised novelty detection methods [4]. Therefore, I am leaning towards rejecting the paper.

Comments

-       Authors should cite [1].

-       I think FashionMNIST is comparably too “easy” dataset for verifying out-of-distribution detection performance. I suggest experimenting on diverse in-distribution datasets, e.g., TinyImageNet, CelebA, LSUN. Furthermore, there are too many blanks in the experiment result that it is hard to convince that the proposed metric improves over conventional methods. Furthermore, it would be beneficial to compare the performance of the paper against more diverse unsupervised OOD detection methods.

-       I would like to see the effect of variation in local horizontal length(l) and kernel size(k) as means of OOD detection performance across different in-distribution datasets. It would be beneficial to show whether the optimized hyperparameters can be applied in other unobserved datasets.


References.   
[1] Why normalizing flows fail to detect out-of-distribution data. Polina Kirichenko∗ , Pavel Izmailov∗ , Andrew Gordon Wilson

[2] Do deep generative models know what they don’t know? Eric Nalisnick et al.

[3] Novelty detection via blurring. Sungik Choi and Sae-Young Chung

[4] CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instance. Jihoon tack, Sangwoo mo, Jongheon Jeong, and Jinwoo Shin.

limitations_and_societal_impact: While authors did not focus on explaining the potential negative social impact of their work, I believe that is not such an issue.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper discussed the counterintuitive phenomenon in OOD detection using generative models, and proposed a new method called non-local feature density which fixed the issue and achieved the SOTA performance, because non-local features are shown to contain more semantic information. Further, the authors proposed an image compressor method, NeLLoC, based on the local features, because local features are shown to constitute a domain prior for all images so that NeLLoC is more generalizable to OOD images. 
main_review: - Originality: the authors provides a new method for fixing the issue of higher likelihood of OOD inputs in the generative model based OOD detection methods. The general idea of the proposed method is similar to the previous methods: the full generative models include both local (non-semantic) and non-local features (semantic). Local image features appear largely common across the real-world image manifold and can be treated as a domain-prior;  only non-local features which typically contain semantic image information, are useful for OOD detection. So the designed OOD score is the unnormalized likelihood of non-local features, i.e. p(x|full model)/p(x|local model).  
Compared with the existing likelihood ratio methods [35] and [38,39], the proposed method is more direct to address the issue of common local features dominating the likelihood. But the proposed method seems to be specific to the PixelCNN model architecture. I wonder if the method can be generalized to other generative models.

- Quality: the paper is well written, and the results are clearly presented. 

- Clarity: It would be great if the algorithm for NeLLoC can be presented, describing the exact computational steps for compression or decompression.

- Significance: The proposed method is specifically designed based on the PixelCNN model. It would be great if the method can be generally applied to other types of generative models. 

- Minor comments:
- Figure 3 (a) shows a slight overlap between the IND and OOD distributions, but Table 3 shows the AUROC for the same experiment is 1.00.
- Typo: eq(3), k->d


*** Update: Thank the authors for your responses. It is great that the framework can be applied to other autoregressive models such as Transformer. I wonder if Transformer model can achieve even better performance. After reading other reviews and responses, I decided to keep my score. ***
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: The main concerns about this work shared by the reviewers were around novelty and presentation. One reviewer felt that a key idea underlying the work has already featured in several other works in recent years, and thus the novelty of the work is limited. I am inclined to agree with this, however, I believe this work could be a useful reference to help more people understand the intricacies of OOD detection and the biases of likelihood-based models (especially autoregressive models), and the practical demonstration that a tiny local model is all you need for efficient lossless image compression, is valuable in itself.

Therefore, I have decided to recommend acceptance. This is of course conditional on the authors including in the manuscript all the additional results they have provided, and reworking the parts that were deemed unclear as set out in the authors' responses.
Decision: Accept (Poster)