Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: In this work, the authors investigate how the makeup of an algorithm portfolio affects automated algorithm selection (AAS) from that portfolio. In particular, they explore meta-representation (SHAP and performance2vec) driven selection techniques for building such a portfolio. They ultimately show discrepancies in portfolios selected based on each meta-representation approach, and qualitative differences in AAS performance.
overall_review: Strengths
- New and interesting perspective on data-driven portfolio selection
- Personalized version of their algorithm demonstrates better performance over existing greedy selection paradigm

Weaknesses
- No novel contribution on the meta-representation learning side
- P2V and SHAP serve as reasonable baselines, but it's been shown many times that SHAP is not a robust explainability method -- I'm confident that its feature attributions serve as a grounded source of truth here either
- Non-personalized versions of their method don't improve upon baseline
review_summary: Overall, this work presents an interesting approach to portfolio selection. While some of the experimental results are not very convincing, and the choice of meta-representation learning methods is not perfect (and there is also no work done on novel meta-representation learning methods), the idea itself is worth introducing to the community.
actions_required_to_increase_overall_recommendation: Ideally, develop a new meta-representation learning method that avoids the need to learn an instance-specific portfolio.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors evaluate how the performance of automated algorithm selection (AAS) depends on the portfolio of algorithms to choose from. There exists a tradeoff in the size of the portfolio: a larger portfolio contains the best algorithm with higher probability, but selecting the best algorithm out of a large number of candidates is also harder. The authors propose constructing a similarity graph from a set of algorithms based on their meta-representations, computed using performance2vec or Shapley values (SHAP). Then, given a certain constraint on the portfolio size, graph algorithms like Maximal Independent Sets (MS) and Dominating Sets (DS) can be leveraged to select a portfolio that is representative and nonredundant. 

The authors conclude that performane2vec selects a smaller portfolio but achieves better AAS performance relative to the true optimal algorithm, while SHAP selects a larger portfolio but achieves worse AAS performance. The authors further conclude that the personalized method, where portfolios are constructed for each problem, generally outperforms other baselines. 

All of the proposed methods outperform the case of having a full portfolio.

overall_review: The methodology is valid and innovative. The experiments are well-designed and thoroughly conducted and the insights drawn from the experiments are explained well (e.g., Figure 4).

The methodology could be improved in terms of clarity of explanation. The paper may also benefit from adding a section on related work. 

review_summary: The methodology is valid and innovative, and the experimental results demonstrate better behavior than other baselines. The authors also provide insight into interpreting the experimental results.There are some issues with clarity mentioned above. Therefore I recommend weak accept.

actions_required_to_increase_overall_recommendation: - clarity issues mentioned above
- adding a section on related work
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: It is well known that the effectiveness of automated algorithm selection heavily relies on the selection of a suitable algorithm portfolio. This task involves finding a balance between the flexibility of a large portfolio and the increased complexity of the AAS process. Typically, algorithms for the portfolio are chosen using a greedy selection approach based on their performance in some reference tasks. 

The purpose of the paper is to explore alternative, data-driven techniques for algorithm portfolio selection. Tha authors introduce a method that involves learning meta-representations of algorithm behavior, constructing a graph based on the similarity of these representations, and applying a graph algorithm to select a final portfolio of diverse and non-redundant algorithms. 

The effectiveness of two distinct meta-representation techniques is assessed by selecting complementary portfolios from a set of 324 CMA-ES algorithm variants for optimizing classic BBOB problems. The experimental results show that the performance based on performance2vec representations tend to favor small portfolios with negligible error in the AAS task, while the SHAP-based representations offer more flexibility but decreased AAS performance. Personalized portfolios perform comparably or slightly better than the greedy approach and outperform the full portfolio in all scenarios.

overall_review: Pro : 

The study's examination of the trade-off between the flexibility provided by a vast portfolio and the difficulty of selecting a subset of relevant algorithms is intriguing, and the proposed approach is relevant, even if not entirely novel.

The use of meta-representation of algorithm performance provides an innovative way of achieving AAS.

The experiments are thoroughly detailed, and the results are comprehensively analyzed, providing valuable insights for practitioners.

The graph representation utilized in this study provides interesting visualizations of the resulting portfolios, offering opportunities for a better understanding of the portfolio's building process.

The discovery that portfolios customized to individual preferences using Shapley meta-representations offer superior performance and greater flexibility is a noteworthy finding.


Cons : 

The initial presentation of the problem could have been more comprehensive, providing more detailed background information and context.

While the state of the art is briefly discussed, a more thorough analysis could have been beneficial to contextualize the proposed approach further.

Despite being addressed, the computational cost induced by the proposed approach remains a significant limitation that requires careful consideration.

Experiments are limited to one kind of algorithm. The performance scope could be biased by this restriction. 

review_summary: The main contribution of this paper is the assessment of two meta-representation techniques, namely SHAP and performance2vec, to choose portfolios from CMA-ES variations for BBOB. Two types of portfolios have been considered: those related to the algorithm's overall behavior and those personalized to the algorithm's behavior for each problem. The experiments show that Performance2vec-based portfolios favor smaller sizes with minimal error, while SHAP-based portfolios are more adaptable but have lower performance. Personalized portfolios offer comparable or marginally better outcomes in nearly all situations compared to the classical greedy approach of selecting the best-performing algorithms based on reference tasks. 

The paper focuses on BBO and specifically addresses the AAS problem for optimization, which is a well-known and extensively researched topic. The proposed methodology is highly pertinent, and the results obtained are intriguing for the BBO optimization community. Although some sections could have benefited from additional elaboration, the paper is well-written overall.

actions_required_to_increase_overall_recommendation: The impact is limited. The possible use for wider and more different portfolios could have been examined.
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: The paper investigates "data-driven" approach for portfolio selection in the context of black-box optimization. The authors evaluated two ways to characterize algorithms -- performance2vec and Shapley based characterization. The paper also explores the idea of "personalized" portfolio building where a portfolio is built for each problem type. The experiments show that using algorithm meta-representation to build portfolios doesn't usually help with algorithm selection. However personalized portfolios yield the best performance.
overall_review: Negative:  
- mostly exploratory work  
- I'm not convinced that Shapley values can serve as meta-representation of algorithms  
- No clear motivation (why is this a problem to begin with?)

Positive:  
- Nice experimental setup 

review_summary: Overall, I think the paper is a nice exploratory work of portfolio selection but there are a couple of comments I have about the paper (see in other sections). I'm not convinced that it has a large impact in the AutoML.
actions_required_to_increase_overall_recommendation: I think the paper would benefit from better motivation for the work. For example, you mention that you used 324 algorithms in your study and the introduction allures to the problem of large number of algorithms in full portfolio. Maybe mention the number of algorithms in the beginning? I think that can motivate the paper better.  
I'm not convinced that Shapley values can serve as meta-representation of algorithms. Can you provide your reasoning why you chose to use Shapley values?
review_title: null

Decision: Accept(Main Track)