Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper decomposed out-of-distribution into texture and semantics and proposed a model that extracts the texture and semantic information separately and then combines them via normalizing flow-based method.
main_review: My biggest concerns about this paper is the correctness of assumptions, derivations and fairness of comparison.

1. in section 3.1 'we assume that both variables have conditional independence given x'. Is it a valid assumption? 
2. Equation 1 is wrong. It should be p(x|t)p(x|s)p(t)p(s) / p(x)p(t, s). The priors were ignored. And I still doubt the likelihood can be decomposed wrt. texture and semantics. For example, the texture can determine whether it is a horse or zebra, grassland or sand. 
3. in section 3.2, 'p_theta(x|t) should focus on extracting texture information alone, while p_theta(x|s) solely on the semantic counterpart', the model design does not guarantee this.
4. Figure 4 shows corrupted C10 has more frequency components at 10-20, which is not surprising. I am surprised that C10 and C100 have the same spectrum. Have you tried corrupting this C10 image by alpha-blending C10 and C100 together? Or cut C10 into pieces and rearrange the spatial orders.
5. In table 1, what is the comparison of in-distribution detection of the proposed methods against the listed methods?
6. In table2&3, looks like lambda=0 performs worse than method in comparison because lambda=0 is texture mode and it is testing texture discrepancy. In table 4, when testing semantics discrepancy, lambda=1 works worse because it is semantics mode. It is unfair to compare their best models with other methods, because the methods with best lambda include the prior knowledge whether OOD is wrt. texture or semantics. In table 2&3 when you choose lambda=1, you are explicitly telling the model that OOD is wrt. texture. Other methods do not have such kind of knowledge.
summary_of_the_review: I have some questions that could influence my understanding of the basic blocks. I cannot make a clear decision until seeing the answers.

==========================

Based on other reviewers' comments and authors' responses, I would like to lower the score to reject this paper. The false claims and equations give me an impression the authors do not understand what they are doing. I would recommend another round of revision and rebuttal for authors to build a solid story instead of rushing this work into publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The submission proposes to evaluate OOD detection problems with regards to two aspects — detect distributional shift in texture, or that in object-identity. A Fourier transform is used to identify changes in texture, and a modification of SVDD is used to identify changes in object identity, by building density models on top of a PCA-reduction of the extracted features in both cases. Experiments are conducted which showcase that the two components can detect textural vs. object-identity shift while not mis-identifying one type of shift for the other.
main_review: The submission makes the case that practical applications can sometimes call purely for a detection of textural distributional shift, and at other times purely for an object-identity shift. While this certainly makes some sense, I think this perspective is a bit too narrow: all of practical OOD detection in computer vision would not fit into one of these two buckets. For example, one might be interested in the composition of scenes, where the presence of two objects that don’t usually co-occur is considered OOD. Or one might be interested in objects poses, where a car that is overturned on the road is considered OOD. This point is made in [1], where the meaning of the word “semantic” is associated with the context, thus placing every practical OOD/anomaly detection problem into first specifying context and then developing a model that captures the notion of what is semantic within that context (which would include texture, if that was the feature of interest). I believe this is a broader and more encompassing view of the point the submission is attempting to make. However, [1] did not explore beyond the context of object-identity, and this is worth exploring further.


In Eq. (1), I think one cannot drop the $p(x)$ term from the denominator, since we are modelling a (conditional) distribution on the variable $x$. $p(t, s)$, $p(t)$, and $p(s)$ may be dropped if one is not concerned with normalization, but I think $p(x)$ ought to remain since it is not uniform. It is also not completely clear to me that texture and semantics should be treated as independent, as in the MNIST-KMNIST experiment (regardless of their conditional independence given x), because a change in semantics ought to also entail a change in textures most often (since a novel object will most likely have a different texture associated with it). But I partially agree that sometimes we might want to ignore stylistic changes if the object stays the same. An earlier attempt at evaluating non-semantic and semantic shift exists [2].


From Eq. (9) it appears that \lambda = 0.0 corresponds to “semantic”-shift, but the experiments seem to suggest the reverse. Could you clarify?


I’m not sure if it is notationally accurate to say that a realNVP model on top of PCA-reduced features of an image can be referenced as the conditional distribution $p(x | f(x))$. It might be more correct to say $p(f(x) | x)$ is what’s being learned.


Method-wise, the novel contribution is really the bit about building density models on top of PCA-reduced features from existing methods for OOD detection [3,4]. The experiments seem to suggest this can lead to fairly good performance. An ablation study is required for supporting the proposed choice of initialization for multi-SVDD.


[1] Detecting semantic anomalies, Ahmed et al.

[2] Generalized odin, Hsu et al.

[3] Deep multi-sphere SVDD, Ghafoori et al.

[4] Amplitude-phase recombination: rethinking robustness, Chen et al.

[5] Transfer-based semantic anomaly detection, Deecke et al.

summary_of_the_review: The submission is commendable in joining a set of recent papers [1,2,5] in pushing for properly benchmarking and defining OOD detection. The novel perspective in this particular submission seems to be the suggestion to disentangle texture from object-identity in a way that detecting one does not imply detection of the other, which is an interesting and thought-provoking view, albeit somewhat narrow. The idea of using density models on PCA-reduced features is demonstrated to work well. The submission scores low on clarity of notation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an OOD setting emphasizing on texture and semantics. The authors propose an OOD detection method which disentangles texture and semantics. The method achieves SoA performances.
main_review: Strengths:
- The separation of texture and semantics is reasonable
- SoA performance 
- Easy-to-understand writing and illustration

Questions and weaknesses:
- Is there any case in which texture is part of semantics?
- How do we know the optimization of (2) will extract semantic information but not the texture information?
- It looks like the method with only semantics (no texture enforced, lambda=1.0) works the best in most cases. Is it still important to enforce texture information extraction?
- Lack of comparison to latest methods (e.g. [A], [B], [C])
[A] Zisselman et al., Deep Residual Flow for Out of Distribution Detection, CVPR20
[B] Lin et al., MOOD: Multi-level Out-of-distribution Detection, CVPR21
[C] Zaeemzadeh et al., Out-of-Distribution Detection Using Union of 1-Dimensional Subspaces, CVPR21

summary_of_the_review: On the good side, the proposal of OOD problem with texture and semantics is reasonable and the proposed method achieves SoA performance. However it still lacks comparison with some latest methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose to decompose the out of distribution sample detection task into two-fold: texture and semantic. Extensive discussions and experiments are presented to demonstrate the efficacy of the proposed decomposition idea.
main_review: ## Strength
### 1. The paper is well organized and written, such that it is easy to follow.
### 2. The proposed texture and semantic decomposition idea is technically sound and correct. The motivation of the idea is clearly explained. The experiment is extensive and sufficient to justify the proposed idea.

##Weakness
### 1. The paper assumes that the texture and semantic information are independent. In my understanding, this assumption is a little strong. How to make sure the assumption is valid or when the assumption is valid. This may need some justifications.

summary_of_the_review: Overall the paper is good and ready for publication.
review_title: null

Meta-review: This paper proposes a categorization of out-of-distribution examples by texture and semantics, and proposed a model that extracts the texture and semantic information separately before combining them via a normalizing flow-based method to obtain good results. While the categorization provides some interesting perspectives, most reviewers found the assumptions too strong, and there are some issues with the derivation. Reviewers have some positive feedbacks on the proposed algorithm for OOD, but also expressed concerns about the fair comparison with more recent baselines. The paper, in its current form, is not ready for the publication, but the authors are encouraged to improve the paper with reviewers' suggestions and resubmit.
Decision: Reject