Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: N/A for reproducibility reviewers
overall_review: N/A for reproducibility reviewers
review_summary: N/A for reproducibility reviewers
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper proposes to select algorithms based, not only on instance features (as is classically the case) but also based on algorithm features that are derived from the algorithm’s source code using program analysis. Given these features, they then selects the algorithm from the portfolio that is predicted to perform best according to a single model of algorithm performance across instance and algorithm feature space. They validate the usefulness of these program features empirically on ASlib and further analyze feature importance. 
overall_review: Reasons to accept:
- Interesting/novel premise (despite prior art): The paper proposes a general extension to the algorithm selection framework, extending it with algorithm features, exploiting the fact that algorithms are often not truly a black box, yet are treated as such. This proposed framework is interesting, and despite prior art pursuing this line of work exists (see technical quality), it is to the best of my knowledge the first method using features derived from program analysis. Furthermore, in meta-algorithmics in general this white box approach is relatively unexplored.
- Clear representation: The paper is well-written, there is a clear line of argument throughout, making it easy to follow.

Reasons to reject:
- Limited relevance to AutoML (see Impact on AutoML) 
- Not entirely convinced of usefulness features: It is not obvious to me why the algorithm features (mostly complexity measure) considered in the paper would be useful. Except for l260 the paper provides little motivation of why these features would be useful. While the experimental results do provide some evidence that these features are useful, I do not agree that these are conclusive (l235).
- Failing to provide code (see reproducibility)

review_summary: This paper has an interesting premise, and the authors present their study thereof in a clear way. Therefore, I feel that this paper, despite its shortcomings (limited relevance AutoML, non-conclusive results, missing code), would make a fine contribution to the conference, assuming the issues raised (related work, softening some claims) are addressed.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper introduces an approach to allow an in-depth analysis of the algorithm selector. It proceeds by investigating the underlying surrogate model. Concretely, the authors proposed using new features to describe algorithms. The latter algorithm features combined with instance features are fed to a surrogate model to predict the most promising algorithm. Finally, the paper presents an empirical validation of the approach on ASlib, showing the merits of the added algorithm features while yielding new insights.

overall_review: The contribution of this paper is technically novel and solid. Moreover, the proposed approach yields superior performance than the considered baseline (algorithm selectors without algorithm features). 
The paper also provides insight into the relative gain of each feature set (instance, code, and AST), suggesting the merits of the algorithm features. Finally, further analysis is also presented to investigate the importance of algorithm features. 

Beyond the importance of features, I wonder if the authors did observe patterns that characterize the relationship between instance features and features of the best algorithm. In my opinion, such an insight provides the underlying decision being made by the selector.
review_summary: From my perspective, this paper presents solid contributions both technically (analyzing algorithm features) and empirically (yielding superior performances). For this reason, I recommend acceptance. 

Nevertheless, I am not an expert in the Algorithm Selection domain and thus can not situate/assess the paper w.r.t the current state-of-the-art algorithm selectors, justifying my review confidence score set to 3.
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper proposes to use features based on the algorithm's implementation (=source code) in automated algorithm portfolio selection.
The authors use standard software analysis features such as abstract syntax tree (AST) to extract it from the algorithms. They use standard algorithm portfolios and two standard tasks and evaluate their approach on a standard library ASPLib.
overall_review: I do appreciate the idea to consider algorithm features and that is certainly a positive point about this paper.
However, how useful are the chosen features? What do they say about the actual functionality of the algorithm?
What if exactly the same algorithm is implemented in some way that changes the features but not its functionality?

To me I fail to see why and how such features would help with algorithm selection - well now you obviously can argue that the results are good and yes they are certainly encouraging. Are the baselines good enough?  For instance the name-based embedding dummy variable I mentioned above - would that make a difference? I do not want to sound too negative but I just cannot see the relation.

To me it is more a conceptual issue. If you would have said we analyzed the paper or github history for each solver and extracted the main functionality (like used 'binary resolution' or had 'XOR' support or the relationship - this is cryptominisat 2.0 which comes from cryptominisat 1.0 and is a derivative of MiniSat or look at the parameters documentation) and used those features to do algorithm selection I would argue that would be a lot more meaningful.

The argument that this is great since one can build exactly one model for algorithm portfolios is true but nothing special either. You do cite a lot of work, but for instance not consider work such as "Probabilistic Matrix Factorization for Automated Machine Learning
Nicolo Fusi, Rishit Sheth, Huseyn Melih Elibol, NeurIPS, 2018. 
I mention this one explicitly since it not only build a single model, but is actually a suitable match for your setup since it is essentially uses a recommender system. Why is that relevant? Just consider the Netflix setup - here movies are algorithms and users are problem inputs. This seems like one could do an interesting analysis (like in Netflix) what kind of algorithms like which problems - which has also been previously done. However, it would help if the features that characterize the algorithms are actually somewhat meaningful.

review_summary: I do like the overall idea of using algorithm features, but I do not see how the chosen features provide a meaningful characterization of the algorithm for algorithm selection. One can argue that the the results seem reasonable modulo some more sophisticated 'dummy features', but to me it definitively does not add any meaningful transparency to the black-box algorithm.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: Automatically selecting an algorithm to optimally solve a set of problems is known as the per-instance algorithm selection problem.  This meta-optimization problem has been especially studied in the context of optimization problems (e.g., SAT solving). Important improvements of state-of-the-art solving techniques have been achieved in both discrete and continuous optimization domains. Most of the time, the general methodology consists in using features that are associated with the problem instances and to predict the performances of a portfolio of algorithms (that may consist of different algorithms or different settings of an algorithm). 

In this paper, the authors propose to use features that describe the solving algorithms. It means that some kind of knowledge is required while previous methods claimed that they considered black-box solving techniques. Nevertheless, the authors propose general code-based features that can be easily extracted from the available softwares. They then perform an extensive set of experiments on established benchmarks that highlight the benefit of their proposed approach.
overall_review: Pro : 

- The originality of the proposed features extraction and use from solvers’ code

- Fair and well-chosen experimental process

- Obtained performances that assess the benefits of the approach 

- Quality of the writing 

- The approach could be extended to other algorithm selection problems (e.g. in ML)

Cons:

- The proposed approach is not suitable for parameter tuning of solvers (which is a possible target of algorithm selection methods

- The explanation of the relevance of the solvers’ features could be difficult to produce


review_summary: This is an interesting work whose motivations are clearly stated. The contribution is relevant with state-of-the-art techniques. The method can be adapted to other domains. Nevertheless, it already provides interesting results for discrete optimization solving. The experiments are based on a significant set of benchmarks. The presentation of the experimental results is very satisfactory.
review_title: null

Meta-review: metareview: Technical quality: Sound, no major issues

Clarity: Paper is clear and easy to read. However, lacks real efforts to understand why the chosen features work

Originality: Obtaining features from program analysis of the algorithm source code is original

Significance: While the proposed approach improves over baselines (mostly an ablation study), this is a bit of a weak point, since no real attempt is undertaken to understand why such features would be useful.

This paper adresses automatic algorithm selection from a portfolio, in order to best solve a problem instance. The main novelty is the use of algorithm features extracted from the source code by program analysis (e.g., features derived from the AST). The authors present an evaluation on a number of standard benchmarks of problem instances, where it is shown that the new algorithm features improve performance in most cases, compared to only using instance features and an algorithm indicator.

Reviewers agree that the use of features extracted from program analysis is interesting and original, given that algorithm source code is often available and standard tools can be used to extract them. Also, the empirical evaluation is deemed sound, even though some more baselines could have been compared against (e.g., simple algorithm features used in previous work).

The paper lacks a motivation why the extracted algorithm features would be useful, apart from the empirical evidence (which could have been stronger). The feature importance study by forward selection closely follows prior work (Bischl etal, 2016) and remains inconclusive. More effort from the authors to tease out why these features works, would have made the work a lot stronger. As it stands, it remains a purely empirical observation. It is therefore recommended the authors tone down statements like "opening the black box", or at least clearly explain what they mean.

The authors are strongly encouraged to publish their code, so others can explore these type of algorithm features as well.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept