Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper proposes a novel framework for NAS that simultaneously optimizes for performance and “introspectability,” and a multi-objective evolutionary NAS algorithm called XNAS that optimizes over this pareto frontier. The main improvement introduced with this framework is the increased robustness of this kind of multi-objective optimization over traditional accuracy-only optimization. The authors conducted experiments on the multi-objective performance of XNAS on 3 image classification tasks, and performed analysis on the network motifs and model introspectability, trustworthiness, and debuggability.
overall_review: Strengths:

This paper includes a well-thought out experimental design that showcases the effectiveness of using the bi-level optimization problem consisting of the model accuracy and class-wise disentanglement using an average cosine distance for all pairwise comparisons (Section 3.2). XNAS utilizes this performance measure to search for architectures that are prone to separation between classes through the NSGA-II NAS algorithm. This framework for multi-objective optimization is shown to be more robust to identifying mispredictions, improving model explainability, and improving hypervolume in this multi-objective task. 

Weaknesses:

Showcasing the model’s performance is good, but using the hypervolume as a comparison against the single objective optimization (figure 2) is somewhat misleading. In this case, it might come across as a means of proof for the improvement of the model when all the figure is showing is that the new framework optimizes over a new objective.
Another weakness in this paper is that it does not describe in detail the thought process behind the design of this interpretability objective and its expected use. There is a mention (lines 261-262) that discuss the motivation behind DNN trustworthiness and disentanglement, but there is no clearly described motivation elsewhere.
Figure 3 is very small, and dense. Subsequently, it is difficult to grasp information from it.
One minor point is on the specificity of modality. By focusing on image classification tasks, it showcases a good general use case scenario, but this model would probably aim to see production into tasks where the higher prediction confidence or trustworthiness is useful (i.e. higher risk tasks) or the better model debuggability is useful, which is not covered in the 3 tasks.

review_summary: Overall, this paper includes thorough experimentation on image classification using the new framework for multi-objective NAS. While there are still some considerations about the overall motivational design behind this research, it clearly identifies a path to create debuggable models through multi-objective NAS. 
actions_required_to_increase_overall_recommendation: For figure 2, it might be better to showcase the difference between only the interpretability of the two optimization tasks rather than the hypervolume. This would showcase the models.

Extending the experiments to cover more applicable tasks might include facial recognition tasks, or other “higher-risk” classification tasks.

For the lack of clear motivation (not in the field of research but in the actual workflow for this research), putting in a description of the effectiveness of leveraging the new explainability outputs over the single objective would be beneficial.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: The authors propose multi-objective NAS that optimizes both performance and interpretability. The main insight presented by the authors is quantifying the introspectability as the disentanglement between latent representations for different data classes. Experiments show good trade-off can be achieved between accuracy and interospectability.

overall_review: Positives:
- The paper is well-written and easy to follow.
- The paper addresses a timely problem, and the high-level idea is promising.

Negatives:
- Technical novelty is limited since the main technical contribution is defining a metric via disentanglement.
- Not clear how feasible it is to extend the metric to beyond classification use cases.
- Evaluation is done on a limited scope.
review_summary: Overall, I believe the authors are exploring the right direction, but the detailed approaches as well as the evaluation can be further enhanced.
actions_required_to_increase_overall_recommendation: Suggest the authors to further discuss the insight behind the metric definition, compare with other existing approaches, and elaborate on how the approach can be applied to other tasks.
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: The motivation of this paper is to optimize task performance and interpretability jointly.  This paper proposed an interesting metric, introspectability, to qualify the model's interpretability. Introspectability measures cosine similarity between activation vectors. It is not clear why introspectability can be used to qualify interpretability.  The well-known multiobjective evolutionary algorithm NSGA-II is used to optimize two objectives. The algorithm is demonstrated in NAS-Bench-201 on MNIST, CIFAR10, and ImageNet-16-120. 
overall_review: This paper provides a new application of multi-objective NAS. But the algorithm contribution is weak. The model's interpretability metric, introspectability, does not make sense to me. I hope the authors can clarify it. 
review_summary: This paper proposed a new application of multi-objective NAS. Algorithms are not novel. How to qualify the model's interpretability is an interesting point. But I have concerns about it as I mentioned. So, my initial rating is borderline reject. I keep some space to adjust the score if the authors can solve my concern. Because the multi-objective NAS part is not new. I expect strong contributions from other sides, ie. model interpretability. 
actions_required_to_increase_overall_recommendation: I raised concerns about introspectability. If the authors can better explain it and provide some toy examples, I'd like to increase the score.
review_title: null

Decision: Accept(Workshop)