Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The authors develop OFA^3, a continuation of OFA^2.
OFA^2 is first trained to create a large network that during search time, given the accuracy and latency requirement, outputs the best model (subnetwork) it can find for the required latency and accuracy. 
The authors use all the subnetworks that OFA^2 outputs, and then using evoulutionary algorithm (EA) they output the best ensemble of these subnetworks for the given latency-accuracy requirement. 
Their EA is based on the NSGA-II algorithm. The DNA is a bit-string whose size matches the number of subnetworks output by the OFA^2 model. 
They start with all bits set to 1. The mutation flips a bit at random with some probability p=1%, the crossover operator is the uniform crossover so every gene is chosen uniformaly at random from either of the parents, and the selection algorithm is based on non-dominated sorting and crowding distance metrics, as in NSGA-II. 
The authors then experiment using two options: sum of latencies of the subneworks in the ensemble, and the maximum latency of the subnetworks in the ensemble. In the case of sum of latencies, they find out that it may be better to use a single architecture rather than an ensemble in most of the cases. For the maximum latency, the authors show that their results dominate any other configuration of the available sub-networks, taking accuracy and latency as the conflicting objective.

overall_review: Strengths
========
S1. An interesting approach to combine the subnetworks of OFA^2 into an ensemble. 
S2. The approach is simple, the paper is well written and easy to follow, the figures are explanetory.

Weaknesses
==========
W1. There is not even a single comparison table in the paper. Please add some comparison table, with numerical values. In my opition, it is important for the evaluation of the paper as well as a way to compare future work to your own work.

W2. The evolutionary algorithm is pretty simple / straightforward, maybe consider several alternatives and show what is the best.

W3. Can you add ablation study.

Not listed as a weakness, but, the approach uses the subnetworks computed by OFA^2, and run an EA on the output of OFA^2. Can you find an approach that is more entangled with the training and search procedure of OFA^2, I find it interesting as well.

review_summary: See the above strengths and weaknesses.
actions_required_to_increase_overall_recommendation: Please add a table of experiments.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper proposes OFA³, a strategy to build high-performance ensembles for multi-objective architecture search. This paper is based on OFA², which uses the evolutionary search algorithm NSGA-II for accuracy and latency multi-objective search and returns a set of solutions. The proposed method further improves on the latter search method by selecting the best sub-networks for forming efficient ensembles. The method is evaluated on ImageNet. 
overall_review: Strengths:
The experiments show good results in finding architectures with high accuracy and low latency in the proposed setting. 
This paper tackles an important research question. 

Weaknesses:
There are some open questions (see Technical Quality And Correctness), which would when answered improve the overall assessment and novelty of the approach in comparison to other multi-objective search strategies. 
A clear description of the strategy is missing. 


review_summary: My main concern is that there are no comparisons with other methods except for OFA² and most importantly the contribution is not clear to me. 

Additionally, the submission checklist is filled rather sparsely. 

actions_required_to_increase_overall_recommendation: In order to increase the overall recommendation, this paper would need to answer the open questions in Technical Quality And Correctness and clearly state the novelty.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: The authors propose an ensemble technique, OFA3, which addresses the problem of automatically selecting the optimal subset of the already obtained non-dominated subnetworks.
OFA3 is an extension of the OFA and OFA2-NAS frameworks that formulates the selection of neural networks to form ensembles as a multi-objective optimisation problem and automatically finds efficient ensembles from OFA2.
overall_review: The paper represents an interesting evolution of existing methods.
The performance and results presented show potential for future application.
However, the overall impression is marred by occasional formal errors.
Furthermore, the application raises questions that need to be clarified in order to allow a better assessment of the method and its performance.

review_summary: Unfortunately, the positive impressions of the paper are marred by the formal errors and criticisms of the application. In particular, small details in the formal presentation, such as errors in chapter titles or careless mistakes in the graphics, detract from the good impression. In addition, the sometimes superficial presentation in the experiments and method section detract from the overall impression.
actions_required_to_increase_overall_recommendation: - Please improve the presentation of the figures. These are minor issues, but their increased occurrence deteriorates the good impression.
- Fix the naming issues of chapter 4 and 5. 
- The application needs an overhaul. In particular, a comparison over several runs and/or initialisations.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper proposes the framework OFA^3 to select efficient ensembles out of an already obtained set of single neural networks. It optimizes for accuracy and latency by utilizing a well-established evolutionary multi-objective optimization algorithm. 
overall_review: The paper gives a clear introduction into the topic and a good overview of related work. The methodology primarily explores established concepts, which, while informative, offers limited originality. The results are insightful but suggest that the performance of the final ensembles is reported on the same dataset that they were searched on, thus could be overfitting.
review_summary: The primary contribution of this paper is the formulation of a multi-objective problem to form ensembles with pre-existing neural networks. The resulting performance improvements are modest and the overall impact of the framework is limited.
actions_required_to_increase_overall_recommendation: (already addressed above)
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: The authors have introduced OFA3, which is an extension of the OFA and OFA2 NAS frameworks. OFA3 utilizes a multi-objective optimization approach to select neural networks for forming ensembles, and it can automatically identify efficient ensembles from OFA2. This approach helps in discovering architectures that exhibit better top-1 accuracy and latency. The authors have also conducted experiments related to ensembles, including tests with encodings and operators in the EA. According to the experimental results, OFA3 outperforms OFA2 and OFA in most cases while also reducing computational costs.
overall_review: Positive aspects: The paper attempts to address the ensemble formation problem as a multi-objective problem, which is a novel approach. The internal ablation study that compares sum latency, max latency, and voting scheme is also a valuable contribution.

Negative aspects: Upon reading the paper, it is difficult to understand why this methodology is necessary. Additionally, the explanation of the multi-objective problem is excessive and should be condensed into a single section. The formulation of the methodology is lacking in detail and could benefit from a more thorough and reader-friendly (or mathematical) presentation. Lastly, the paper lacks experimental results to support the claims made in the conclusion. It would be beneficial to provide metrics such as computational cost or performance improvement in a table format.
review_summary: The evidence presented for why the proposed methodology is necessary may not be sufficient to convince readers. To better support the claims, the authors should provide additional results such as direct comparisons with OFA and OFA2, the extent of performance improvement, and the computational cost aspect mentioned in conclusion. This would help in providing a more convincing argument for the necessity of the proposed methodology.
actions_required_to_increase_overall_recommendation: - Fill in a comparison table for the search time.
- Add experiments comparing with OFA and OFA2.
- Compare with other multi-objective methods.
review_title: null

Meta-review: metareview: 
The paper proposes an ensembling strategy on top of the OFA approach to bundle sub-networks. Reviewers agreed that this is an interesting approach. However, in its current state, the paper contains several shortcomings. First, the novelty is limited and the main approach is a rather straight-forward extension of previous work. Ensembling is well-known to outperform individual models and the proposed approach is based on basic majority voting / averaging of predictions. Second, several reviewers pointed out issues in the empirical evaluation of the method. For example there is no comparison to other methods than OFA2. Even tough OFA’s performance has been benchmarked in the original paper, the paper would be more rigorous if it contains it owns benchmarking and ideally confirms the reported results.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Reject