Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new out-of-distribution (OOD) detection method that makes use of known OOD data. Given a classifier with an encoder that outputs features, and a linear classification layer, the idea is to decorrelate the features of in-distribution and OOD samples in a class-wise manner. In order to achieve this, the authors use the Hilbert-Schmidt Independence Criterion (HSIC) to measure the independence between in-distribution (of a given class) and OOD features. A classifier is then trained on the usual classification loss plus the independence objective. However, since computing the HSIC metric involves computing the features of $N$ in-distribution samples in every class, which can get expensive with large $N$, the authors use a buffer to reuse old features and save compute. The method, which is called Conditional-i, is evaluated on OOD detection benchmarks in image classification and NLP, and achieves better performance than other baseline methods.
strengths_and_weaknesses: Strengths:
- The proposed method is very similar to HOOD, as the authors mentioned, however, the differences (class-wise computation of HSIC, and usage of memory bank), are made clear, and there is evidence of improvement in the experiments.
- The proposed method seems to consistently perform better than other OOD detection methods.

Weaknesses:
- There is no guarantee that decorrelating the features of in-distribution and known OOD data will lead to improved feature distinction between in-distribution and unknown OOD data.
- There is no mention in the paper how hyperparameters ($\lambda$ and $\tau$) were tuned for each of the methods. However, it seems from the ablation experiments that the used values conveniently match up with the best performance on the test OOD datasets. If it is true that the hyperparameters were tuned on the test set, the results presented in the paper aren’t trustworthy.

questions: - As mentioned previously, how were the values of $\lambda$ and $\tau$ decided? Was a separate validation OOD dataset used?
- How reasonable are the assumptions for Theorem 1, especially the ones involving $g$’s?
- The authors mention in lines 267-268 that “This hopefully may better generalize the training to unseen test OODs as long as the generative process of the test OOD is genuinely different”. What does a different generative process mean, and how can we define it? I’m not sure I believe this statement since Theorem 1 doesn’t guarantee anything about the unknown OOD case.
- In the experiments, have the authors tried using just the test scoring method from Equation 9 on a classifier trained normally with cross entropy?
- It doesn’t seem fair comparing Conditional-i-generative with methods that don’t use any auxiliary data (even if it’s synthetic). Have the authors tried outlier exposure with the generated OOD data?
- Have the authors removed overlapping images between train in-distribution, train OOD and test OOD? For example, the tiny images dataset contain CIFAR-10 and CIFAR-100.
- What $\hat{C}$, $N$ values were used for each experiment?
- What were the test classification accuracies achieved by Conditional-i on the in-distribution dataset?

Other comments:
- Line 204: “matrics” -> “metrics” 
- Line 218: Typo in $p_M(\mathbf{z}) = \sum_{c=1}^C p_\theta(\mathbf{z}^{(\xi)},\mathbf{q}|\xi=c)p(\xi=c)$ -> $p_M(\mathbf{z}) = \sum_{c=1}^C p_\theta(\mathbf{z}^{(\xi)}|\xi=c)p(\xi=c)$
- Table 1 and 2 caption: “trainnig” -> “training”
- It would be nice to include the setup that was used to compute the values in Table 4, 5, and Figure 1.

limitations: The authors used the Tiny Images dataset, which was withdrawn for ethical reasons.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a conditional independence model called Conditional-I for OOD detection. The authors introduce an integrated loss function to reduce the dependence between inliers features of each class and outlier features, so as to distinguish the OOD samples by a relevant independence measure during testing. The authors provide theoretical guarantees of the conditional independence hypothesis. The experiments demonstrate the better performance of the proposed method on CV and NLP tasks in the cases that training OOD data are available or not.

strengths_and_weaknesses: Strengths:

The work relies on an explicit independence hypothesis, i.e., the inliers  extract only little predictive information from outliers during training. The authors extend the assumption to the conditional independence which encourages each IND class to produce independent features from outliers. Moreover, theoretical arguments and supporting analyses are provided for this independence assumption. The theoretical derivations basically follow the work from [2] and [38], which seem to be correct and convincing to me.

Weaknesses:

1.The innovation of the article seems limited to me,  mainly since the work shares the same perspective as [2]. Both the models build upon the probabilistic formulation and applies the Hilbert-Schmidt Independence Criteria (HSIC).  It may be good to clarify a bit more on how novel the paper is compared from [2].

2.There is a lack of qualitative experiments to demonstrate the validity of the conditional independence model. 
a)It is better to provide some  illustrative experimental results to demonstrate that minimising HSICcond-i could indeed perform better than minimising HSIC_HOOD. Possibly, one toy dataset can be used to demonstrate the separability of inlier features and outlier features.

b)The authors propose a new test metric, however, lacking the correctness test and comparative experiments with other metrices. 
It may be better to provide some visualization results or schematic diagram, which could  make  readers easier to understand.

3.Current experimental results seem not very convincing to me. Some critical comparative results are missing. 
a)Under the setting of unseen OOD training data, DIN [34], Mahalanobis distance [31], Energy [36], their original papers did not use fake/augmented OOD training data. These settings need to be clarified in the paper. Moreover, the impact of using different augmentation methods on the result could  be explored in the ablation.

b)In CIFAR-100, the experimental setup appears to be consistent with that of HOOD [2]. However, in Table 1 (unseen OOD training data), the HOOD’s results are missing.  In [2], the results of HOOD are superior to those of Conditional-I's. 

c)In Table 2 (unseen OOD training data), the HOOD’s results are also missing. 

d)There are missing both Conditional-i-generative and HOOD results for the NLP OOD detection tasks.
As missing the results of the most relevant methods,  the present experiments could not  convince me of  the validity of the improvements.

4.The memory bank architecture is one contribution, but the authors do not provide quantitative results of introducing the memory bank architecture.
questions: 1.In Section 4.2, the authors propose that Theorem 1 explains the independence assumption is beneficial for OOD detection.  Is there any clue to prove that conditional independence within a class is more effective than restricting independence between inliers and outliers? Are there any empirical experiments to support this idea?

2.The impact of using different augmentation methods on the result should be explored in the ablation. In Image OOD detection tasks, why use two different methods of generating fake outlier samples? Why not generate them in a uniform way?

3.How about the HOOD results on the unseen OOD training data setting?

4.How much computational cost can be saved by using the memory bank architecture?

limitations: The authors seemed not to  discuss the limitations of the proposed model.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a novel training method for improving the out-of-distribution (OOD) detection performance of a classification model. It is motivated by the hypothesis that the mutual information (dependency) between OOD data and in-distribution (ID) data should be small. Specifically, the proposed method relies on the concept of class conditional independence between ID features and OOD features and utilities maximum mean discrepancy (MMD) to enforce this condition, leading to a composite loss function and training strategy. The method evolves into two variants where one requires unlabeled OOD data and the other uses generated OOD data through strong data augmentation. 
strengths_and_weaknesses: Strength 
1. The paper is very written, especially on addressing the similarity to a contemporary work and descriptions of small implementation details that are important for the success of this method in practice. 
2. The method is thetically motivated and empirically verified on large datasets including both CV and NLP tasks. 
3. The paper includes ablation study on various aspects of the algorithm such as the number of classes for enforcing independence during training and the hyperparameter for adjusting the contribution of the loss component for enforcing independence.

Weakness: 
1. As with all methods requiring OOD data during training, the proposed method necessarily entails overhead on storing training-data, sampling/generating OOD data and additional computation such as the Gaussian Kernel.
2. The performance against a specific type of OOD data of the proposed method might be highly dependent on the type of OOD exposure or augmentation used during training. 
3. It’s not clear how the proposed composite loss function would affect the discriminativeness (accuracy) of the model on in-distribution data. 

questions: 1. Could the authors comment on the accuracy of this method? 
2. Could the authors comment on the sensitivity of the method using different OOD data during training or different augmentation strategies? 

limitations: The authors addressed potential negative impact in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel method to address OOD detection by independent component analysis (ICA) techniques. This novel method named Conditional-i builds upon the probabilistic formulation, and applies the Hilbert-Schmidt Independence Criteria that offers a convenient solution for optimizing variable dependencies.

Basic Idea:
For different ID class-conditional distributions, the paper uses MMD to estimate the dependence between ID for c-class and OOD. Main formulas are in Eqs. 3 and 4. 

Theorem 1 explains why the independence assumption is beneficial for OOD detection.

Experiments have shown Conditional-i is better than some classical methods. 

strengths_and_weaknesses: Strengths:
1. Experiments show Conditional-i is better than some classical methods. 
2. The idea that uses independent component analysis (ICA) techniques seems to be novel.


Weakness:
(Major) 1. Poor writing. Many notations are not standard. I spend many times to understand your notations, like lines 127-128. What is p_{theta}? it is a mixture of ID and OOD distributions? You should revise your notations to be more readable.

(Major) 2. In Eq 11, do you assume that the number of OOD data is N, which is same with the number of c-class ID data? 

(Major) 3. If we have not any OOD data, how you train your model? It is really difficult to find where you have described it. Could you tell me which line you have introduced it?

(Major) 4. Please compare the SOTA method Gradenorm (On the importance of gradients for detecting distributional
shifts in the wild.)

(Minor) 5. The idea of  HOOD is similar with this paper. You use an extremely similar idea to submit two papers?



questions: (Major) 1. Poor writing. Many notations are not standard. I spend many times to understand your notations, like lines 127-128. What is p_{theta}? it is a mixture of ID and OOD distributions? You should revise your notations to be more readable.

(Major) 2. In Eq 11, do you assume that the number of OOD data is N, which is same with the number of c-class ID data? 

(Major) 3. If we have not any OOD data, how you train your model? It is really difficult to find where you have described it. Could you tell me which line you have introduced it?

(Major) 4. Please compare the SOTA method Gradenorm (On the importance of gradients for detecting distributional
shifts in the wild.)

(Minor) 5. The idea of  HOOD is similar with this paper. You use an extremely similar idea to submit two papers?


limitations: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: metareview: This paper proposes an out-of-distribution (OOD) detection method, where the features of in-distribution (ID) samples and those of the OOD samples are decorrelated in a class-wise manner by using HSIC.  A theoretical guarantee for decorrelation is provided.  The propposed method, Conditional-i, is evaluated on OOD detection benchmarks in image classification and NLP, and shows SOTA results.

Reviewers had many major concerns, e.g., novelty from HOOD, missing baselines with OOD exposure, hyperparameter tuning, and insufficient theory to support the proposed method, which the authors addressed well, and most of the reviewers have been convinced. 

The paper is well-written and the mathematical notation seems fine for machine learners.  However, the relation between Theorem 1 and Assumption 1 is weird.  Any theorem must always hold (otherwise it's not a theorem), and the sentence "Theorem 1 holds if Assumption 1 holds" doesn't make sense.  If the claim in Theorem 1 requires Assumption 1, Assumption 1 should appear first, and Theorem 1 should state that "under Assumption1, it holds that ...".
confidence: Certain
Decision: Accept