Title: COSMOS: Are Performance–Cost Tradeoffs Predictable in Model--Strategy Selection?
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Strategy Selection, Prediction, Model Adaptation, Routing, Scaling Law

Abstract: Large language models (LLMs) achieve excellent performance across numerous tasks by using a diverse array of adaptation strategies. However, optimally selecting a model and adaptation strategy under resource constraints is challenging and often requires extensive experimentation. We investigate whether it is possible to accurately predict both performance and cost without expensive trials. To address this strategy selection problem, we introduce COSMOS, a unified prediction framework that efficiently estimates adaptation outcomes at minimal cost. We instantiate and study the capability of our framework via a pair of powerful predictors: embedding-augmented lightweight proxy models to predict fine-tuning performance, and low-sample scaling laws to forecast retrieval-augmented in-context learning. Evaluations across eight representative benchmarks demonstrate that COSMOS instantiations achieves high prediction accuracy while reducing computational costs by 92.72% on average, and up to 98.71% in resource-intensive scenarios. Our results show that efficient prediction of adaptation outcomes is possible and can substantially reduce the computational overhead of LLM deployment while maintaining performance standards.

TL;DR: We show that performance–cost tradeoffs are predictable, which enables cost-effective model–strategy selection.