Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: The paper proposes an efficient multi-objective NAS method for concealed weapon detection in X-rays. The contributions are twofold. The paper first introduces computationally cheap heuristics to make genetic multi-objective NAS algorithms  for this application efficient and effective. The approach is evaluated on publicly available and private datasets. Secondly the paper proposes an approach to effectively ensemble multiple architectures discovered. The results show that their approach outperforms state-of-the-art methods for concealed weapon detection. 
overall_review: Positives:
The application and the necessity of automl for it is well motivated 
The paper studies the effectiveness of automl in a real-world industry application.
Exploiting zero cost proxies for multi-objective NAS is fairly novel to the best of my knowledge

Negatives:
Background and example definitions of “threats and modifications” is nor provided.
It is not clear why optimizing for different proxy scores is formulated as a multi-objective problem. In traditional multi-objective NAS the objectives are often user centric (eg: hardware efficiency, performance quality). Why is optimization of different zero cost proxy scores simultaneously desirable to search for optimal architectures? How were the proxy metrics selected?
Evaluation on a private dataset makes some results presented not reproducible. Could the authors evaluate on more publicly available benchmarks? 
The paper does not provide details on the search space for the task. Is it a set of all the architectures/operations from table 1?
For the results table 1,2,3 it would be nice to evaluate across multiple search seeds. 
Could the author’s provide an intuition on why the performance gains do not translate to semantic segmentation in table 4?
Presentation of the approach could be improved.
review_summary: I find the application quite interesting and well motivated. However I think the paper misses some important technical details and description of terminologies used. This makes reading the paper and deriving connections between different parts difficult.  
actions_required_to_increase_overall_recommendation: Convincing answers to my questions above could lead to an increase in score.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper introduces MEOW, a multi-objective NAS approach for weapons detection in X-ray images that are commonly used in security applications. The proposed approach combines neuroevolution NAS with proxy scores to find better-performing architectures in terms of training time, size/parameters for concealed weapons detection in X-ray images. NSGA-II is used to evolve architectures, and two complementary proxy scores (NASWOT and SYNFLOW) are modified to evaluate the performance of each architecture. These proxy scores are used as objectives within NSGA-II to generate an optimal and ensemble architecture from the pareto front. The MEOW approach results in a considerable speedup over other AutoML approaches, and results in a new ensemble and optimal architecture that are improve multi-class, multi-label mean prediction accuracy by 10% compared to popular CV architectures. The benefits of using MEOW are demonstrated on the publicly available SIXray dataset and the proprietary Residuals dataset. 
overall_review: The paper is very well written and easy to follow. Concepts are explained clearly and using NAS for weapons detection in X-ray images is technically interesting. THe proposed approach combined multi-obective search with proxy scores to evaluate discovered architecture performance. Experiments on two datasets resulted in accuracy improevemts and NAS search time speedups. The authors also mention that the datasets ued for experimentations are larger than those used in related works, making the speedup remarkable. These finding have implications for the broader AutoML community in developing neuroevolution NAS approaches for specific applications. 


The negative aspects of this paper pertain to the limited number of runs (the authors used 5 runs of NSGA-II). More runs could lead to deeper understanding of performance, and differences in architectures found. 

review_summary: This paper presented an approach that used multi-objective search with proxy scores as objectives for NAS. The paper also demonstrated the effectiveness of MEOW on a relatively new problem - that of concealed weapons detection in X-ray images. The results improved accuracy by 10% and led to an atleast 8x speedup compared to existing CV architectures. Furthermore, the datasets used for evaluations were larger than those used in related work. 

actions_required_to_increase_overall_recommendation: Please see comments above.
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: The paper proposes a new method to generate neural network architectures for the task of concealed weapon detection from X-ray images.
It combines multiple efforts from computer vision, first by adopting a neural architecture search procedure that doesn't involve training the model being evaluated and thus saving a considerable amount of time.
Second, the paper uses different methods of assessing the quality of the evaluated model, making up a multi-objective optimization problem.
This optimization problem is solved with an evolutionary search strategy where each input encodes the architecture of the evaluated model.

Further, in order to make use of different solutions found during the optimization phase, an ensemble is built from the top performing models, which avoids over-committing to a specific model.
The experiment section shows the proposed method performs well on two weapon detection tasks, especially better than other benchmarks on rare instances depicting uncommon weapons.
overall_review: The paper presents a promising approach to NAS for concealed weapon detection, an important problem.
The experiments show results in favor of the proposed approach and contain interesting insights.

Clarity is a concern for the paper, as a lot of details are currently missing, making it difficult to fully appreciate the proposed algorithm.
More setup is also needed to fully crystalize the different metrics being used to evaluated the models in the experiment section.
As of now, "performance" is quite a vague term that's being used liberally in the paper.

In terms of technical insights, the paper can benefit from further ablation studies that pinpoint what exactly makes the proposed method work.
Is it the multi-objective approach or simply just one of the scores being optimized?
What are the different ways of building the final ensemble and how does that affect performance?
review_summary: I think the paper contributes a reasonable approach to an important problem and presents promising results.
However, more details are needed to make the submission a strong one, and I recommend a weak rejection.
actions_required_to_increase_overall_recommendation: More clarity about the evaluation metrics being used in the experiment section will help the paper.
I will appreciate more intuition building in Section 3, maybe via a small illustration, where the different objectives are explained.

Ablation studies for the multi-objective aspect will make me get a better insight.
Does optimizing individual objectives work just as well?
Can we do better than picking out the top solutions during optimization to build an ensemble?
review_title: null

Decision: Accept(Main Track)