Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes two sample test method that measures the maximum mean discrepancy (MMD) of two sample sets for out-of-distribution sample set detection. The main idea is to use self-supervised features for MMD. On top of the self-supervised features, the paper proposes a method called MMD-CC (MMD with clean calibration) that reduces the variance of MMD in the cost of computations which is effective when sample set sizes are small.
Then, the authors propose a method to apply MMD and MMD-CC to a single outlier instance detection (anomaly detection) by simulating a two sample test on a single test sample by populating the test sample with multiple transformations (augmentations).
Two terms are used to measure anomaly score; intra similarity score which measures the similarity between transformed test samples, and out similarity score which measures the similarity between transformed training and test samples. The calibration step is proposed to find a balancing hyper-parameter between the two similarity score terms. Instead of using the similarity score directly as an anomaly score, the paper proposes to perform hypothesis testing and use the p-value as an anomaly score. The proposed methods are evaluated on 4 datasets for out-of-distribution detection tasks (a task that assumes a set of test samples are given), and evaluated on two image datasets and 3 adversarial scenarios (on the ImageNet validation set) for anomaly detection tasks (single instance detection).

strength_and_weaknesses: ## Strength

### Various experiments support the effectiveness of the method

### Diverse applications

Out-of-distribution detection from two sample sets, single instance anomaly detection, and adversarial attach detections are shown in the experiment section.

### Diverse datasets are used for the experiments.

Four datasets are used for out-of-distribution detection tasks (tasks that assume a set of test samples are given).  Two image datasets are used for anomaly detection.
Three adversarial scenarios are shown (on the ImageNet validation set) for anomaly detection tasks (single outlier instance detection).

### Ablation studies

Some necessary ablation studies are presented in the paper including varying sample set sizes (Figure1 and Table1), and the number of transformations used for anomaly detection (figure 2).

### The limitations of the paper are addressed.



## Weakness

### Computational cost

The paper also addresses their computational cost requirement. Since two sample test is a statistical test on two sample sets, their computation requires quadratic order in terms of sample sizes. Moreover, the paper proposes to repeat this test multiple times to reduce the MMD variance.

### Pretraining on a large-scale dataset required

The method relies on well trained feature representation. The experiments are performed on a pre-trained ResNet 50 or a self-trained ResNet 50.

### Missing ablation studies

1 Computing calibration parameter gamma in Eq (7) costs n_trs * n_trs * |X_val1| * ||X_val2| computations. How much is this calibration process important? What is the performance gain from this process? What if gamma=1 is used without any calibration process?

2 The paper uses hypothesis testing instead of directly using the score in equation (8). How much performance gain was obtained from this?

### (Minor point) The method may be sensitive to different transformations.

In the first paragraph of section 5.1, transformations except ‘color jittering, gaussian blur, and gray scaling’ are used. What is the rationale behind this? Is the method sensitive to different types of transformations?

clarity,_quality,_novelty_and_reproducibility: ### Clarity

1 Are S_p(1) and S_p(2) same size in Algorithm 1? Are they half size of S_q?

2 Section 4.1 mentions that 100 samplings are used for the experiment. Does it mean 50/50 from in/out of distributions (total 100 test cases) or 100/100 (200 test cases)? Or is the ratio of in/out distribution cases not equal to 0.5? (not equal number of test cases for in and out-of-distributions?)

3 What is the ‘learned similarity’? The similarity is defined as cosine similarity in Equation (1). Does learned similarity denote the cosine similarity with learned feature representation?

### Quality

The paper is well-written although there are some missing ablation studies and experiment details.

### Novelty

The paper’s idea mainly relies on MMD and is technically similar to classic MMD for out-of-distribution detection.
However, the paper proposes several parts to adapt the idea to anomaly detection.

### Reproducibility

Although most of the necessary processes are described in the paper, it will not be easy to reproduce the paper without making the code publicly available since the algorithm requires multiple steps to compute and multiple hyper-parameters (including sample set sizes) are involved in reproducing the experiments.

summary_of_the_review: The paper proposes a two-sample-test-based out-of-distribution detection and anomaly detection.
The experiments are extensive with meaningful results. Although there are a few missing ablation studies and details, the paper is well-written overall.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the detection of out-of-distribution (OOD) instances (novel classes and adversarial attacks) in vision using similarities learned via self-supervised contrastive learning (SimCLRv2) with a maximum mean discrepancy (MMD) two-sample test. This approach is first used to show that there exists a significant distributional shift between CIFAR10 and CIFAR10.1 (which has been proposed as a corrected update of CIFAR10). Then, an extension of the approach for single samples is proposed via using augmentations of a sample to create a set, called CADet (Contrastive Anomaly Detection). An experimental evaluation on OOD detection of (i) novel classes (ImageNet vs. iNaturalist, ImageNet-O) and (ii) adversarial examples (generated with PGD, CW, and FGSM attacks) shows that CADet performs similar on detecting unknown classes and favorably on detecting adversarial attacks over previous methods.
strength_and_weaknesses: + I find it interesting to see that self-supervised embeddings improve adversarial attack detection over using supervised embeddings, also for other previous methods. I'm not aware of previous work making this observation.
+ The paper is well integrated into the existing literature, making a comparison to recently established OOD methods.

- Overall, I find the experimental results to be not too convincing. For detecting novel classes, the proposed method does not show a consistent improvement over previous methods (the supervised embeddings also being better over the self-supervised one in 2/3 cases). The adversarial examples detection is more convincing, which I think should be extended/elaborated on.
- While CADet empirically seems to work particularly well for adversarial attacks, I am missing some explanation or intuition on why this might be the case. I think the paper might be stronger focusing and digging deeper on this case, since the differences over previous methods here are intriguing.
clarity,_quality,_novelty_and_reproducibility: + Overall, the paper is well written and easy-to-follow.
+ All training details seem to be given for the reproducibility of results.

- I find the technical and methodological novelty of the paper to be overall rather low, presenting a combination of existing self-supervised embedding learning (SimCLRv2) with a MMD two-sample test. Novelty is given in extending this idea to single samples via augmentation and the proposed extension of MMD.
- The space used in the main paper could be optimized in my opinion. Some background could maybe trimmed a bit to provide more space/details on the proposed methods (most content on pages 1--4 is background). Algorithms 2 + 3 could be put into the appendix.

---

*Additional Comments*
* p.1: "Anomaly detection methods generally rely on [...]" I'd rather say OOD detection methods here, since anomaly detection is more general, and OOD detection is a specific anomaly detection problem for which specific methods have been proposed. 
* p.2: "Contributions: Our main contributions are as follows:" Double colon (remove first).
summary_of_the_review: The paper includes interesting findings regarding differences between using self-supervised vs. supervised embeddings for OOD, in particular wrt adversarial examples, but is limited in providing further analysis and insights into why one might expect the shown improvements in detecting adversarial examples.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel out-of-distribution detection algorithm inspired by MMD-based two-sample testing. The paper first shows that contrastive learning, when used with MMD two-sample test, can be highly effective in recognizing distribution shifts in image statistics. Then, the paper extends this finding to build an out-of-distribution detection algorithm that can detect images that contain an object that is not in the training set or is adversarially modified to fool an image classifier.
strength_and_weaknesses: # Strength

* As far as I know, the application of MMD-based two-sample testing on OOD detection is a novel approach. There have been some efforts to apply the two-sample testing idea to OOD detection, but they were not as successful as this paper.
* The paper is well-written. The expositions are clear and easy to follow.
* The paper does a good job of summarizing the existing literature and providing the technical information required to reproduce the experiments.
* The paper provides a helpful discussion on the limitations and future directions of the work in Section 6.3.

# Weaknesses

I think the followings are more like questions rather than weaknesses.

* ("Supervised" adversarial attack) Although it is stated in Section 5.1, it would be nicer to emphasize that the adversarial samples considered in this paper are generated with respect to a supervised classifier. The reason is that an adversarial sample can also be generated with respect to an OOD detector, and OOD detectors are known to be very vulnerable to such attacks, see for example, https://arxiv.org/abs/2106.04260 . Nonetheless, limiting the scope of the paper to the "supervised" adversarial attack would not be the reason for rejection, I think.
* (Adversarial detection performance of contrastive representation) I am worried that the comparison between supervised representation and contrastive representation in Table 4 is not fair, because these adversarial samples are generated with respect to the supervised network being tested. From the information provided in the paper, it is not clear whether the attack would transfer to a network trained with contrastive learning. Even though it translates well, the attack is usually the strongest against a network the attack is targeting. Therefore, a supervised network can be said to be handicapped. I believe it is a little bit too early to conclude that contrastive representation is more effective in adversarial detection. A few more supporting arguments should help resolve this issue.
* (The number of inlier samples in testing) For me, it is surprising that so few inlier samples are needed to perform successful two-sample testing with respect to outlier classes, and I wonder if there is a good explanation regarding this. Naively, if there are 1,000 classes in the in-distribution dataset, shouldn't we need at least 1,000 samples that represent each class? 
clarity,_quality,_novelty_and_reproducibility: The paper is overall very clear, and the proposed method is presented in a sound way. I believe the successful application of MMD-based two-sample testing on OOD detection has strong novelty and significance. The paper contains enough information to reproduce the empirical results. Meanwhile, I do have a few concerns regarding the empirical results, particularly in Table 4. 
summary_of_the_review: It is a well-written paper with algorithmically novel contributions. It would be a more appealing paper if the performance in Table 3 was stronger and the fair comparisons were made in Table 4.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present research on the intersection of outlier detection using MMD on two sets of samples and self-supervied pretraining.
First of all they use self-supervision to pretrain a similarity function which is then to be used for outlier detection using MMD and derived statistics. 

They present two improvements over Liu et al. 'Learning deep kernels for non-parametric two-sample tests.' when computing statistics. One is a better calibration of the p-value calculation. The other is a different way to compute the MMD statistic, in which they double the size of the validation set, and likely are able to achieve a lower variance of the p-scores if the test distribution is different from the validation distribution.

They show two results in these parts. One is stronger evidence for a distribution shift between CIFAR10 and CIFAR10.1 than shown in previous work. The second are results for outlier detection against sets of outliers taken form other datasets and obtained by adversarial optimization.

The third contribution they state as contrastive anomaly detection. This is meant to identify a single test sample whether it could be an outlier. 

In this they take the test sample, and create a set of augmented variants from it, using augmentations similar to those used in self supervised pretraining. They do the same with a set of validation samples. Then they define a statistic somewhat reminiscent of the one used in MMD to obtain a score, which then is used to compute a p-value. The p-value is based on comparing scores computed between two augmented validation sets from the in distribution, and a score computed between one augmented validation set and one set obtained by augmenting the test sample.

They show results in Table 3 and 4 which are mixed for outlier detection and good for adversarial detection.

strength_and_weaknesses: Strengths: 
1 well written paper
2 experiments with out-of sample and adversarial data
3 an improvement in detection in setups where one has a set of test samples due to the two statistical changes
4 incremental novelty in combining self-supervision and MMD

weaknesses:

1 There might be problems in the formulation of the third contribution:

1.1 First of all, a naming criticism: The similarity is fixed. This the authors also acknowledge themself. This is okay but it is unclear why it is named contrastive, as no change or training of similarity is performed based on the augmentations of the test sample. Yes, they initialize the similarity using contrastive learning, but the test sample is not involved in this first step. It is averaging over copies obtained by data augmentation, and thus not more contrastive than the original MMD with self-supervised initialization. 

Contrastively-initialized outlier detection is more precise here.

1.2 Secondly, and most importantly, equation (6) departs from the idea of an MMD score substantially. The MMD score as in eq (4) measures a difference between self-similarities and cross-similarities. Equation (6) performs an addition where MMD as in eq (4) would perform a subtraction.

Comparing to eq (4) also a self-similarity term between validation data is missing. For thresholding it can be omitted, but it would impact the variance of the estimators. However the main issue is the mentionned addition in place of substraction of the cross-similarity.

equation (6) becomes by that a statistic without any motivation behind it. Why it is chosen as addition when (4) suggest a substraction ?

1.3 Furthermore by matching equation (7) with algorithm 2 (on page 7) one can see that the calibration to compute gamma uses only data from the same in-distribution. But then asymptotically gamma should become 1, as m^{out} and m^{in} are computed over data drawn from the same distribution an thus their variance should be the same.

If gamma is not 1, as in Table 5, then something in the implementation seems to be unexpected (although gamma seems to be a constant across the datasets in Table 5).

1.2 and 1.3 look as if they wanted to do something slightly different. gamma becomes non-trivial if it involves a mix out validation and augmented variants of the test sample.


2 it would be good to try adversarial detection with other types of adversarial attacks, which are generated by different principles

3 self-supervised outlier detection is not novel itself (https://openaccess.thecvf.com/content/CVPR2021/papers/Li_CutPaste_Self-Supervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.pdf , https://arxiv.org/abs/2103.12051 ), but see strengths, point 4
clarity,_quality,_novelty_and_reproducibility: It is well written and easily readable. 

it has two small improvements over standard MMD, which seem to be novel.

self-supervision for outlier detection as a whole is not novel  (https://openaccess.thecvf.com/content/CVPR2021/papers/Li_CutPaste_Self-Supervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.pdf , https://arxiv.org/abs/2103.12051 ) but its evaluation in the context of MMD seems to have novelty, which is incremental..
summary_of_the_review: The paper has two smaller statistical improvements over MMD. It advocates the use of self-supervision for outlier detection when combined with MMD. That is an insight.

The strongest issue with respect to acceptance in the eyes of this reviewer is the last contribution on single sample outlier detection which seems to depart from MMD without any motivation for doing so.

*edit* after reading the rebuttal, the reviewer increased his score from 5 to 6 , 

however, if accepted to ICLR please clarify that it is inspired by MMD but differs in that it gives up / departs from the geometric interpretation used in MMD (when switching the sign) + provide the motivation stated in the rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes using self-supervised contrastive learning to detect OOD examples based on an MMD-inspired two-sample test.

The reviewers agree that the paper tackles an important problem with its new adaptation of the classical MMD to OOD detection. 

However, the reviewers also point out some areas of improvement for the paper, including a more comprehensive and convincing comparison between supervised and contrastive-based approaches. Moreover, reviewers would have liked to see a better motivation for certain design choices of the method. Finally, the computational costs of the proposed method may limit its practical value. 


justification_for_why_not_higher_score: While the paper is quite interesting, there are multiple issues that should be addressed before being ready for publication as outlined in the latter portion of the meta-review.
Decision: Reject