Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper outlines an approach to out of distribution [OOD] example identification. The approach makes use of the idea of explicitly separating high-level semantic content and low-level image frequency content and training separate classifiers whose outputs are combined to make the OOD decision. The semantic content extraction network uses a novel approach to data pre-processing intended to make it easier to identify OOD elements.
strength_and_weaknesses: The overall analysis of the problem is interesting and provides a justification for the design choices and dual-pathway classifier. The angular projection of the training data to enhance OOD detection is also an interesting approach to capturing the concept of the training set boundaries.

The use of rectangular frequency binning, and discarding orientation, as a texture descriptor is very rough. Empirically, it seems to work well enough, but a more sophisticated set of features may improve performance. Does the process make use of an integral image to do the computation of rectangular sums? It's also not clear if the computation is being done on the base DFT or the power spectrum. The text for figure 4 implies it is the power spectrum.

The acronym SVDD does not seem to be defined in the text.

The balance of the two systems is evaluated at only lambda = {0, 0.5, and 1}.  It's not clear from the results that equal weighting will necessarily be the best option.

The new benchmark data set seems to be needed, given the performance on the prior data sets. It seems tailored to demonstrating this particular differentiation of the task (high level versus low level features).  My question is whether it has broader applicability.
clarity,_quality,_novelty_and_reproducibility: The explicit separation of high and low level features seems to be novel and enables a specific architecture. The new benchmark data set is a novel contribution.
summary_of_the_review: The conceptual approach leads to a specific architecture that shows good performance on both the existing data sets and a new proposed benchmark data set. The concept of thinking about OOD detection as having separable problem characteristics enables new approaches to the problem.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a definition of the in distribution data (and by extension OOD), where the authors decompose the definition of the ID into texture and semantics. This decompostion provides the flexibility to different scenarios by determining which view of the ID more suited for a given scenario. The texture pipeline is built form 2D FFT and the semantic component is build from ResNet and multi-SVDD.
strength_and_weaknesses: The main strenghts of the paper are as follows:

1) The paper is well-written and the main ideas are presented in a succinct manner.
2) A problem of interest to the community is tackled in the paper, where better definition of in-distribution and OOD data will allow for deployment of deep learning based solutions in real-life scenarios.


The main weakness in my opinion is as follows:
1) I don't believe frequency based mehtods are a good proxy for texture of a region. In fact most texture segmentation methods I am aware of don't use frequency based features.
2) In principle if frequency components are indeed important for the OOD data, we should be able to learn it with a neural network. FFTs are just scaled dot product of the input with FFT basis.
3) I am not totally sold on the distinciton of the semantics and the textures. How do we decouple the texture from the semantics. I did not get it from the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and the main ideas are effectively presented. The method presented in the paper is new, however, I am not sure of it's efficacy.


The codes are not provided with the paper.
summary_of_the_review: I am not totally sold on the efficacy of the method and I dont agree with the main thesis of the paper, i.e., separation of the texture and the semantic component of the images and use it to identify in and out of distribution data. As such I will lean toward rejecting the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes a novel approach for OOD detection based on two branches. One branch operates on DFT coefficients and the other on pixels. The pixel branch combines A ResNet-18 backbone, a multi-SVDD [ghafoori20icdm] layer initialized with unit centers, and a normalizing flow. The frequency branch is directly evaluated by a normalizing flow. The authors argue that the pixel branch is suitable for detecting semantic OODs, while the frequency branch is suitable for texture OODs. 


strength_and_weaknesses: Strengths
1. Somewhat original approach to an important problem. 
2. Competitive performance with respect to approaches that train with class labels (Table C.1) on very small images.
3. Initialization at the unit sphere and disentanglement loss appear to help (Table 6)

Weaknesses

1. \lambda=1.0 wins in almost all conventional OOD experiments (Table 1). The exception is C10-LSUN*. However, this result does not repeat in Table C.1 where \lambda=1 is better than \lambda=0 in both C10-LSUN configurations. If \lambda=1.0 always wins, then the contribution of the texture branch is unclear.
2. Table 6 does not reveal \lambda; it should be performed for all three values of \lambda
3. CIFAR-10 and CIFAR-100 have shared and related classes (dog - wolf, cat - tiger etc); this is not a suitable pair for OOD detection.
4. OOD detection on small images has been largely solved (Table C.1). Further work should evaluate on tougher benchmarks such as Species, SMIYC or FIshyscapes. 
5. The distinction between anomalies and OOD samples does not reflect the current usage in the community. For instance, UCSD anomaly detection dataset [a] contains OOD objects. A recent review explicitly states that anomalies, outliers and novelties are synonyms but distinguishes low-level and high-level anomalies [b].
6. How does one choose K?

[a] Anomaly Detection in Crowded Scenes. CVPR 2010.
[b] A Unifying Review of Deep and Shallow Anomaly Detection. Proc. IEEE 2021.

Comments

The equation in 3.1 (λ · log p(T (x)|x) + (1 − λ) · log p(S(x)|x)) is different than (9).

Missing related work

The proposed approach appears related to [c]. The main difference is that [c] uses labels while the present approach appears to use a form of self-training.

Unit centers appear related to [d].   

Detecting semantic OOD samples with a convolutional model is not easy due to texture bias [e,f]. It would be good to discuss how to deal with that problem (eg. [c]).

It would be sensible to discuss applicability for dense anomaly detection [g].

[c] Hybrid Models for Open Set Recognition. ECCV 2020.
[d] ArcFace: Additive Angular Margin Loss for Deep Face Recognition
[e] ImageNet-trained CNNs are biased towards texture. ICLR 2019.
[f] Why Normalizing Flows Fail to Detect Out-of-Distribution Data. NeurIPS 2020.
[g] SegmentMeIfYouCan - A Benchmark for Anomaly Segmentation. NeurIPS 2021.
clarity,_quality,_novelty_and_reproducibility: The paper is mostly well written. The disentanglement loss appears novel however it would be good to show its contribution for different lambdas. 

I am not convinced that the "texture" branch is useful, especially since I can not see whether it could be useful in large images.  

It is difficult to appreciate experiments from Tables 2-5 since they have not been tackled in the literature and some of them feel artificial. If the paper is accepted, I would propose moving them to the supplement and bringing Table C.1 to the main paper. 


summary_of_the_review: As written in strengths/weaknesses, there is competitive experimental performance, interesting novelty (L_disentangle), but insufficient evidence about the contribution of the texture branch.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduces a new framework for OOD detection by defining more broadly the in-distribution (ID) samples using the notion of texture and semantic. Based on this definition, they create new benchmark scenarios. Finally, they propose a new method which perform well on both old and new benchmarks.
strength_and_weaknesses: Strength:
- a first attempt to clarify OOD detection task
- use of SOTA tools to design their models (resnet +SSVD for semantic, frequency features for textures and normalizing flow to combine both)
- achieve SOTA performances on different datasets
- ablation study is conducted

Weaknesses:
- I guess OOD detection is also meaningful for data that are not images. As such it seems only applicable to images.
clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well written and clear. The figures are guiding the reader smoothly along the manuscript.

Section 3.3: why do you introduce polar coordinate to say afterward that you are not using those ? Then you mention f_r after equation (5) but it is not present in this equation. In the end, it is obscure to me what is T_i... Also T cannot belong to R^{2/w} and I don't think R should be indexed by w because it is the image size... Well, I think this section should be corrected...

Most citations should be in parentheses.

Quality:

Good.

The current framework and results might not be discussed enough. How does this work affects similar tasks such as anomaly detection, novelty detection, open set recognition or outlier detection ?


Novelty:

Their model is a combination of existing tools. The combination is original though. 

Otherwise, I am not expert enough to judge beyond the authors claim.

Reproducibility:

I haven't seen any claim that the code will be released online upon acceptance.
summary_of_the_review: I would tend to recommend acceptance though I lack of expertise to have a strong opinion. The authors should really improve section 3.3. Apart from that my final decision will depend on other reviewers' opinion and the coming discussion.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new approach to OOD dectection based on both DFT coefficients and pixels.

All reviewers found the idea interesting. However, there were concerns about the usefulness of the texture branch raised by two reviewers (UnLe has an issue with using frequency based methods to characterize the texture while xaty was concerned about the good performance of lambda=1).

After further discussion with the reviewers about this issue, the conclusion was reached that this was too limiting and as such I recommend rejection. I encourage the authors to leverage all the comments made for a future resubmission.
justification_for_why_not_higher_score: There was concern about the usefulness of the texture branch. Without that branch, the proposed approach loses its appeal.
Decision: Reject