Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper presents a framework called AutoPatch for segmenting visual anomalies. Essentially, it is a neural architecture search method that enables efficient segmentation of visual anomalies without any training. By leveraging a pre-trained supernet, a black-box optimization algorithm can directly minimize FLOPS and maximize wAP on a small validation set of anomalous examples. Finally, compelling results on the widely studied MVTec dataset are presented, demonstrating that AutoPatch outperforms the current state-of-the-art method. 
overall_review: Overall, the motivation of this paper is clear: try to balance performance and FLOPS for segmentation tasks using a weight-sharing NAS method. 

strength:
- motivation is pretty clear and the problem this paper tries to solve is important. 
- the presentation of the proposed method is clear

weakness:
- the method is a combination of weight-sharing NAS and NN score. Besides, the weight-sharing NAS part only involves inference, that is to say, this method takes a pre-trained weight-sharing supernet and directly uses it to do the multi-objective search. The novelty is limited. 
- the evaluation is also limited. For example, a comparison with model compression technique is encouraging, although the result will be a single point in the Pareto front. 
review_summary: Given the novelty and empirical evaluation is limited, the reviewer does not support acceptance of this paper. 
actions_required_to_increase_overall_recommendation: - discuss more about the proposed metric
- the method is very sensitive to the random seed ( please correct the reviewer if the understanding is wrong), which makes the proposed method less reliable
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper proposes a novel application of NAS to visual anomaly segmentation called AutoPatch. AutoPatch uses nearest neighbor scoring, weight-sharing NAS, and weighted average precision to provide a pareto-optimal model for image segmentation. The main novelty introduced with AutoPatch is the use of a NAS algorithm specifically tailored for visual anomaly segmentation. AutoPatch appears to improve performance in a multi-objective space, and is able to accurately identify a near-optimal neural architecture for that problem with only a few examples of the anomaly.
overall_review: Strengths:

The authors introduce a new method for visual anomaly segmentation, through the use of a NAS algorithm searching for subnetworks from MobileNetv3, in order to generate near pareto-optimal network architectures after being given only a few examples of a visual anomaly in a dataset. A novel weighted average precision metric is being used in this space in order to create a more robust performance indicator.

Weaknesses:

One minor weakness within the paper is the limited scope of experimentation. While there are good and potentially generalizable results shown on the MVTec dataset, this would be more substantial with further experimentation on other datasets.
This is partially addressed within the future works section along with another weakness I agree with. Synthesizing artificial anomalies could provide this angle for further experimentation. Also, further ablation studies on the generalization of the model on larger anomaly set sizes would improve the empirical strength of this paper.

review_summary: This paper introduced a novel application for NAS, and a novel metric for image anomaly segmentation. It was presented in a clear manner with robust experimentation. There is a downside in its limitation to the field of visual anomaly segmentation, but this can be overcome with the usage of weighted average precision to address class imbalance.
actions_required_to_increase_overall_recommendation: I would recommend re-organizing the paper. Moving Figure 2 to the appendix, and moving the appendix after the citations would help improve the readability of the paper.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors propose AutoPatch, a NAS method that efficiently segments visual anomalies. It proposes a new metric, weighted average precision (wAP), to measure anomaly segmentation quality, and shows that AutoPatch outperforms the current state-of-the-art method with significantly fewer computational resources using only one example per anomaly type. The research question is how well Pareto optimal neural architectures can be approximated with limited anomalies. The authors conclude that while promising, reliable NAS for anomaly segmentation remains a complex challenge that depends on the specific anomalies available and can exhibit considerable variance.
overall_review: Positives: The application of NAS on Anomaly Segmentation is interesting. The presented approach is technically sound. 
The empirical results presented in this paper are attractive and prove that NAS methods on Anomaly Segmentation are effective.

Negatives: The authors compared their work only with one previous method. Is it the only method in the literature? It is also unclear how to construct a small size of the training set with one example per anomaly type. 
review_summary: The reviewer is confident of the review. The reviewer has reviewed the search space, algorithm, and evaluation criteria to check the correctness. The reviewer has concerns with the novelty of the approach taken to search the neural architecture. 
actions_required_to_increase_overall_recommendation: The authors could improve the paper by giving more insights into the search method used in the paper by comparing it to other search methods. 

The authors should also give more details on the efficiency of the search algorithm. What is the search time? 

What is the reason for AutoPatch to obtain better performance than PatchCore? Is it because of the search space or the search method? 

The authors talk about inference time multiple times in the paper. However, there is no mention of the inference time of the searched models. The inference time should also be compared with PatchCore and previously searched or manually designed models. 

It is also highly recommended to show the architecture of searched models for different scenarios. The authors did not discuss how the searched architectures look or explain them.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: In this paper, authors applied neural architecture search to the visual anomaly segmentation problem to optimize the neural architecture for the best weighted Average Precision (wAP) performance with respect to FLOPS. The proposed AutoPatch utilizes MobileNetV3 as its backbone and optimizes various width, expansion ratio, kernel size, extraction block, and patch size values for each stage using FLOPS and wAP based Multi-Objective Tree-structured Parzen Estimator (MOTPE). The experimental results show that AutoPatch outperforms the existing state-of-the-art (SOTA) PatchCore on the MVTec dataset, demonstrating superior performance.
overall_review: The strengths of this paper lie in finding a highly efficient neural network structure for visual anomaly segmentation using neural architecture search, as well as proposing a new metric for performance measurement and increasing the learning efficiency to enable learning with fewer anomaly samples. The proposed technique has demonstrated performance surpassing the existing state-of-the-art, PatchCore, in experimental results. However, the lack of evaluation on multiple datasets and the absence of ablation study act as limitations.
review_summary: Although this paper has some shortcomings in experimental design and performance validation, it is technically correct, and the proposed technique consistently demonstrates better performance compared to the comparing methods. Therefore, my recommendation is a weak accept.
actions_required_to_increase_overall_recommendation: I think it would be recommended to include comparisons with techniques other than PatchCore, comparison experiments on different datasets, and a thorough ablation study.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: In this paper, the authors propose an architecture search method, AutoPatch, which is designed to search for effective segmentation algorithms for anomalies, especially under the scenario of only having relatively few anomalies. With this, they begin by proposing a new weighted average precision (wAP) metric to serve as an alternative to AUROC and AUPRO. The motivation for this is in some industrial applications you might not be able to set some fixed False Positive Rate (FPR) and wish to be flexible over multiple thresholds. The authors then propose an extension of Once-for-All to their scenario where they can extract features from training images, without any additional training, to help segment images during inference, and use MOTPE to maximize their proposed weighted mAP metric subject to minimize the number of flops used. The authors then analyze their method in comparison with an existing method for the problem of anomaly segmentation across different amounts of anomalies.
overall_review: Pros:
+ The wAP metric seems well-motivated and is an unintuitive way to tackle the problem of class imbalance under the scenario when you don't have a fixed false positive rate.
+ The neural architecture search seems very thorough and the formulation of solving their black-box optimization scheme with MOTPE.
+ The author's proposed method doesn't require any additional training, something that I believe is underplayed 
+ The authors do a good job of answering their proposed question of "How close can they get to the Pareto frontier'.

Cons:
- The combination of the new wAP metric alongside proposing the neural architecture search scheme slightly complicates the comparison to the baseline chosen.
- The use of the name 'Weighted Average Precision' seems overloaded to something that is common elsewhere in the literature and refers to something different which is slightly confusing.
- One of the motivations in section 3 is "However, these metrics lack flexibility for diverse requirements and could potentially lead to inaccurate performance indications in the absence of an appropriate FPR limit"; however, it's unclear from the setup that there exist such scenarios where an ~some FPR limit cannot be set that is semi-optimal.
- The use of the memory bank here seems to be a major limitation because the amount of memory required to both store the examples and the amount of computing required to search them is linear in the amount of training data. The authors do recognize this in their limitations; however, I could not find details of the necessary size of the memory bank that was used to achieve their results and I think that might be a crucial factor in their performance.
- Unless I misunderstood, it seems like this method is very prone to failing under domain shift, where images are at inference.

review_summary: Overall, my current recommendation for this paper is a Weak Reject. The authors in this work solve a specific problem with pretty intuitive architectural search solutions and introduce a nice metric to help address their setup of anomaly detection. The main weaknesses of the paper stem from the lack of comparison using traditional metrics to their chosen baseline and some issues with their modeling. While they don't need to include any additional training, they have a method that is bandlimited by the number of training frames you can store, which makes it also seemingly brittle to domain shift. I also strongly believe that wAP here is causing some confusion and that the authors could benefit by changing the naming. I could be misunderstanding something here about their intended use case, and do think that for their problem setup, the authors offer a strong piece of work.
actions_required_to_increase_overall_recommendation: I am happy to work with the authors to consider increasing my score if the authors can help me better assess the quality of this paper:

1. How does their method compare to their baseline using the metrics that were proposed by PatchCore?

2. How many frames are really necessary to store in order to attain good performance? This seems to be a fundamental question that I didn't see answered in the main text.

3. Does their method suffer from issues in generalization under even small amounts of domain shift?

4. I'm curious as to the effect of the selection of examples chosen for k = 1, 2, 4. I understand that this randomization is done over seeds but am wondering if there was additional work to verify the average performance over different k for both the proposed model and the baseline. Additionally, were the same examples used between the proposed method and the baseline in the evaluation?
review_title: null

Decision: Accept(Main Track)