Title: Self-supervised Learning to Predict Optimizer Performance: A Trade-off Study Between Ground Truth Requirements and Prediction Quality
Primary area: optimization
Keywords: self-supervised learning, benchmarking, landscape analysis, single-objective black-box optimization, problem representations

Abstract: Predicting optimizer performance typically requires costly ground-truth performance data, since each label requires multiple executions of optimization algorithms, which are computationally expensive to obtain on a large set of problems. We investigate whether self-supervised learning (SSL) can reduce this labeling burden while preserving performance. Using four landscape representations (ELA, DeepELA, DoE2Vec, TransOptAS), three learners (Random Forest (RF), SCARF, TabNet), and four train–test splitting strategies with increasing distribution shift (I, R, PC, P), we quantify the label–quality trade-off for predicting the performance of five Differential Evolution (DE) and five Particle Swarm Optimization (PSO) configurations in a controlled comparative study. We find that across settings, SSL -- especially SCARF on learned representations -- matches or surpasses supervised RF with only 50–75\% labels. Under moderate shift (PC), SSL yields substantial benchmarking gains: for multiple configurations, we can omit around 80\% of function evaluations without worsening MAE, with improvements visible already at 25\% labels. Computed ELA remains a strong supervised baseline in low-shift regimes (I/R), whereas all representations amplify SSL benefits under shift (PC). The zero-shot split (P) remains challenging, motivating richer problem-space augmentation. The results confirm that SSL benefits are portfolio-agnostic (across DE and PSO). A practical guideline emerges: use RF+ELA for easy, in-distribution tasks, while for harder distribution shifts, apply Doe2Vec with SCARF pre-training and fine-tuning on 50–75\% of labels, which achieves performance close to fully supervised models while reducing the need for costly function evaluations.

TL;DR: Self-supervised learning enables accurate optimizer performance prediction with far fewer labels, especially under moderate train-test distribution shift.