Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper shows that the representation embeddings in the transformer-based models  do not fit the Gaussian distributional assumption. As a result, previous OOD detection methods relying on such assumptions can not work. 

This paper proposed the method (SIREN) to shape the representations into compact class-conditional vMF distributions, which unifies the distributional model between training and testing. And the proposed method is effective for both transformer-based and CNN-based models.

strengths_and_weaknesses: Strengths：

- SIREN shapes the representations for OOD detection under a coherent distributional model, which does not depend on the distribution of model representation embedding.
- SIREN unifies the distributional model between training and testing for OOD detection and shows effectiveness for both transformer-based and CNN-based models.

Weaknesses and questions：

- Why choose the vMF distributions as the coherent distributional model, rather than other distributions.

- The performance on CNN-based object detection is insufficient (table 2), and the author needs to provide more results. What's more, the VOS performs better than SIREN in CNN-based object detection, which needs some explanation.

  | Method | FPR95       | AUROC       |
  | ------ | ----------- | ----------- |
  | SIREN  | 65.45/67.33 | 84.60/81.51 |
  | VOS    | 49.02/52.11 | 88.66/85.36 |

- From figure 6, the SIREN improves the embedding quality, does this influence the performance in ID data？ And it's better to show the performance under the setting about: fixed detector and only training the projection head.
questions: Please see the "strengths and weakness".
limitations: This paper didnot describe the limitations of this work. Every paper has limitations and corner cases. I recommend the authors to supplement the limitations of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel OOD detection framework SIREN relied on the vMF distributions, which unifies representation learning and OOD detection. Detailed experiments and solid ablation studies demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: **Strengths**:
1. This paper is organized well and the oveall representation is sophisticated.
2. The method is plausible, and the experimental results perform well.
3. Extensive experiments and detailed ablation analysis.
4. The theoretical proof of the pivotal distribution deviation vector K has been explained in the appendix.

**Weaknesses**:
1. The originality is a bit limited, and the impact on the field is also incremental.

       Let's briefly summarize this paper, the authors introduce a pretext task of shaping the final image map, then exploit the corresponding maximum class-conditional likelihood to classify whether the current image is OOD or not. Thus, this apporach can still be coverd as an OOD method based on uncertainty and fixed thresholds.

2. Tets-time OOD dectection can hardly be highligthed as a new contribution, casue it is more like a test solution specific to this method.

3. Updating the mean of a distribution by using prototype is not fresh for me, in fact, it has appeared in the following paper:

    > Wang, Haobo, et al. "PiCO: Contrastive Label Disambiguation for Partial Label Learning." arXiv preprint arXiv:2201.08984 (2022).

questions: 1. This article select the vMF distribution to shape the image embedding, but does not clarify why the vMF distribution is the chosen one, and whether any Gaussian-like distribution is feasible?
limitations: Limitations have been adequately addressed. No serious concerns.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: The paper's main assumption is that we need to construct representations during training that are compatible with the detection distributions we observe during inference.

Unlike the Mahalanobis approach, which is an inference-only solution because it does not affect the network's training, the paper proposes constructing a loss to train the network to produce consistent distribution in training and test sets.

It proposed to train the network to impose the representations to follow a mixture of von Mises-Fisher (vMF) distributions and to make both train and test distributions compatibles to improve OOD detection performance.

strengths_and_weaknesses: **Weaknesses:**

_"Shaping representations" and "leverage from the learned representations" is what loss-based approaches (IsoMax, Scaled Cosine, SNGP, DUQ, and IsoMax+) have been doing since 2019. The fact that these loss-based approaches for OOD detection were not even cited may explain why the paper understand "shaping representations" and "leverage from the learned representations" as a significant novelty. Unfortunately, it is clearly not the case._ 

_Unfortunately, the paper entirely ignores the last three years of relevant advances in loss-based approaches that are completely reshaping the area after the Mahalanobis inference-based approach faded out. The paper insistently compares against Mahalanabos, a four years old approach that has been outperformed very easily in the last three years._

**1. Making representation compatible with the OOD Detection procedure is at least three years old.** 

The Mahalanobis paper is from 2018. At that time, inference-based metric learning approaches for OOD were trending. Since 2019, we have observed that loss-based methods are now mainstream. Therefore, training the model using a loss to make training representation and test/inference distribution compatible, which is the central claim of the paper's novelty, is not novel at all.

Unfortunately, the paper lacks to realize that training the model to produce train and inference representations compatible with performing OOD has been the mainstream approach since 2019 and was proposed by IsoMax, Scaled Cosine, DUQ, SNGP, and more recently by DisMax. Kapa appears similar to essentially the entropic scale of IsoMax.

**2. Forcing representation to the unit hypersphere is not novel.**

Forcing representation to the unit hypersphere during training to improve OOD detection was done in Scaled Cosine and IsoMax+, and more recently by DisMax. 

**3. Very limited novelty.**

Considering that point #1 above showed that leveraging representations/distributions learned during training to perform OOD detection has already been proposed. Moreover, noticing that point #2 above showed that unit hypersphere representations have also already been proposed, we conclude that the novelty of the proposed approach is very limited.

**4. Neither cited nor compared with similar previous approaches.**

Considering that the paper essentially proposes a loss to train the network to improve OOD detection, we believe that the paper should have mentioned and compared with (at least) IsoMax, Scaled Cosine, DUQ, IsoMax+, and SNGP. Considering that this did not happen, we do not know if using vMF helps in any way.

**5. Results for CIFAR10 do not look impressive.**  Apparently, the results for CIFAR10 are not remarkable compared to the approaches mentioned above (a direct comparison is absolutely mandatory). We would also like to see results for CIFAR100. 

**6. Results for CIFAR10 need to show classification accuracy.** We also would like to see classification accuracy results to evaluate a possible classification accuracy drop, which is very usual in loss-based approaches.

**7. We need much more experiment results using CIFAR10. Furthermore, we need CIFAR100 results as well.** CIFAR10 and CIFAR100 still are the _de facto_ standard datasets on which major OOD detection approaches have published results. The paper requires much more experimentation using CIFAR10 and CIFAR100 datasets. Classification accuracy need also to be shown and analyzed, as classification accuracy drop is a significant issue when using loss-based approaches for OOD detection.

**8. Unlike recent approaches, the proposed solutions do not allow regular pure end-to-end backpropagation-based training.**

**9. The proposed approach presents much more hyperparameters than current state-of-the-art approaches.**

**10. Results should show mean and standard deviations.**

IsoMax: https://arxiv.org/abs/1908.05569

IsoMax (journal): https://arxiv.org/abs/2006.04005

Scaled Cosine: https://arxiv.org/abs/1905.10628

DUQ: https://arxiv.org/abs/2003.02037

SNGP: https://arxiv.org/abs/2006.10108

IsoMax+: https://arxiv.org/abs/2105.14399

DisMax: https://arxiv.org/abs/2205.05874

After rebuttal: I will keep my score because the differences between the proposed model and the cited training-based approaches are not significant. No direct comparison against the methods we presented was shown. Finally, the majority of the concerns we presented were not covered in the rebuttal. Neither citing nor comparing against the previous related work for the last four years is hard to accept.

questions: We would like the authors to update the paper considering the comments presented on the above topics.
limitations: The paper does not properly comment on the approach's limitations. Unlike many other cited approaches, it does not allow pure end-to-end backpropagation training. It also should better explain the hyperparameter validation procedures required for the alpha, beta, and the dimension of the last layer.
review_title: null

Meta-review: metareview: This work proposes a new unified distributional model to address out of distribution detection and improves over the state of the art.

While the approach shares notable similarities with other works in the domain, the idea of creating a unified distributional representation also at the intermediate features, especially in the context of transformers is new.

While this work can be criticized for some missing comparisons with related works, the overall approach seems both sounds and novel and of general interest. Therefore, I suggest its acceptance for NeurIPS 2022.
confidence: Less certain
Decision: Accept