Learning to Rank for AutoML: enhancing pipeline selection with ranking information

Authors: Hernan Ceferino Vazquez, Pucci Romero Tobias, Verónica Bogado, Jorge Sánchez
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: AutoML, Learning-to-Rank, Pipeline selection, Meta-learning, Ranking, Bayesian optimization, Monte Carlo tree search, OpenML datasets, Metric-agnostic, Machine Learning

Abstract:
This paper introduces a learning-to-rank (LTR) framework to address the problem of pipeline selection in automated machine learning systems. The traditional approach to AutoML involves learning to predict the performance of various pipelines on a given task based on data acquired from previous tasks (i.e., meta-learning), which can be complex due to the need for different models for each task-specific metric. The proposed framework aims to select the best pipeline based on ranking rather than estimating a target metric, aligning more closely with the ultimate goal of the task (i.e., selecting pipeline candidates in order, from more to least promising). This approach enables more robust, metric-agnostic solutions that are easier to compare using ranking metrics like NDCG and MRR. The paper evaluates LTR strategies on public OpenML datasets, demonstrating a clear advantage for ranking-based methods. Additionally, the integration of LTR with Bayesian optimization and Monte Carlo tree search is explored, leading to improvements in the ranking metrics. Finally, the study found a strong correlation between ranking metrics (e.g., NDCG and MRR) and AutoML metrics, such as the task objective metric and the time to find the best solution, providing insights into how ranking-based methods could enhance AutoML systems.

Decision: reject
Reviewer scores: 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730161300392

thread_type:
official_review

<thread_object>
creator: reviewer_p4ta

<content>
summary:
The paper introduces a learning-to-rank (LTR) framework to improve pipeline selection in automated machine learning (AutoML). Traditional AutoML methods predict performance metrics to select optimal pipelines, which can be limiting and resource-intensive. Instead, this framework uses ranking-based selection, focusing on ranking pipelines by performance potential rather than estimating specific metrics. By aligning selection with ranking metrics such as NDCG and MRR, this method creates a more robust, metric-agnostic approach.

The paper evaluates LTR with sequential optimization techniques like Bayesian optimization and Monte Carlo tree search on OpenML datasets, finding significant improvements over the score-based methods. Key contributions include:
* Introducing LTR approach for AutoML pipeline selection.
* An empirical comparison demonstrating the advantages of ranking-based over score-based methods.
* Insights into the correlation between ranking metrics and traditional AutoML objectives.

strengths:
1. Paper is well written, pleasing-to-read.
2. Comprehensive experiments, with details available in the appendix.

weaknesses:
1. I personally enjoyed the paper and the work of replacing autoML task with learning to rank. However, my main worry is that the impact of the work could be limited. I might be biased. But my feeling is that attentions have been attacked to LLM and related works. The paper might be better justified for its impact if authors can show some evidence like introducing LTR improves AutoML for LLM training or prompting, etc. This may well beyond the scope of the work.
2. MCTS results missing on OpenML-Weka in Figure 1. Didn’t find any explanation for the missing results there either.
3. Minor issue: “green > 0, dark green > 10” in Line 347 and “yellow < 0, orange < 10”  in Line 348 should be better with percent signs like “green > 0, dark green > 10%” and “yellow < 0, orange < 10%”.

questions:
1. Why rank-based approach show consistently (slightly) worse performance on D2 AMLB classification task in Table 2? Any intuition? Can rank-based approach be useful for specific tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730501558928

thread_type:
official_review

<thread_object>
creator: reviewer_vi55

<content>
summary:
This paper proposes an approach for AutoML which is based on learning the rank/position of candidate configurations rather than their absolute performance score. Multiple regression models are evaluated for learning to map a configuration to a position/rank. Experiments on several AutoML benchmarks show improved performance compared to score-based models.

* I am not an expert on AutoML, not sure why this paper was assigned to me. I can only provide a low-confidence review.

strengths:
* The empirical results seem appealing in terms of achieving better performance compared to a score-based approach.
* The paper is clearly written.

weaknesses:
* The proposed approach is using a regression model with ranks instead of scores, which is different from a ranking algorithm which outputs a permutation. The authors state this clearly in the paper, however they do sometimes call this a learning-to-rank problem (e.g., in the title and in line 512), which often refers to ranking losses (pairwise/listwise) and not regression using ranks. It may be interesting to see how a ranking loss based approach would compare to the regression based one.
* The proposed approach covers only search spaces with a predefined set of configurations that is enumerable. Large search spaces are not handled.
* The use of positions instead of performance scores seems somewhat incremental (replacing scores by ranks).

questions:
* Table 2: is there an explanation of the missing entries for D1 in the table? Maybe I missed it.
* In Related Work, there is some recent work from neural architecture search, a related problem in AutoML, using ranking instead of scoring for training predictors. Some examples are:
  * Renas: Relativistic evaluation of neural architecture search, Xu et al. (2021)
  * DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning, Zheng et al. (2024)
  * FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer, Hwang et al. (2024)
Can you please comment on how this relates to your work?

Minor/typos:
* Line 135: perhaps give MSE (s_ij - f_ij)^2 as an example loss?
* Line 255: “This set of OpenML tasks includes 104 tasks, split *evenly* between 71 classification and 33 regression problems” — what do you mean by “evenly”?
* Table 2: Consider adding a legend for colors in addition to or instead of the caption description. Also, consider sorting first by dataset and then by model, it seems like the similarity within a dataset is higher than within a model. This would also be consistent with the arrangement figures 1 and 2.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705269835

thread_type:
official_review

<thread_object>
creator: reviewer_7x7t

<content>
summary:
Overall the paper proposes an improvement in Automation of ML pipeline by predicting rank of an approach in the pool given config and data instead of predicting a metric score or formulating as a true ranking problem. They show promising results on different datasets and metrics. The experiment section is fairly elaborate.

strengths:
Fairly elaborate experiment section.
Simple change to existing approach.

weaknesses:
Writing can be improved significantly. It is a bit hard to read and understand the contributions as they are mixed with previous work and formulation. 
Novelty is low.

questions:
It is said that the approach is metric agnostic, you say that because the approach only cares about rank and not the metric value right? even though thats the case the rank order is determined by the metric right?? Would is still be metric agnostic? or is it metric-value agnostic?

In experiments given you are predicting rank order only would it make sense to try some classifier also instead of just regressors??

Would it also make sense to compare your approach to a model trained with some ranking loss function instead only comparing with score-based approach??

In BO approach used, you mention its pretrained, can you clarify how?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a learning-to-rank framework for AutoML pipeline selection which changes the traditional score prediction approach to ranking-based optimization. By focusing on pipeline ranking, the approach offers a more direct, efficient metric-agnostic solution. The comprehensive experiments across diverse datasets demonstrate improvements in ranking metrics such as NDCG and MRR. They also explored the integration of LTR with Bayesian optimization and Monte Carlo tree search, which further enhance the method’s performance.
The strengths of this paper is its clear motivation and detailed experimental evaluation. However, the novelty is somewhat limited, as the primary contribution involves a shift to rank-based optimization without sufficient comparison to existing ranking-based approaches. Additionally, the method is restricted to predefined search spaces, limiting its applicability to larger or dynamic configurations. These issues undermine the impact of the present paper.