Title: LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, In-Context Learning, Hyperparameter Optimization, Meta-Learning

Abstract: Model and hyperparameter selection are critical but challenging in machine learning, typically requiring expert intuition or expensive automated search. We investigate whether large language models (LLMs) can act as in-context meta-learners for this task. By converting each dataset into interpretable metadata, we prompt an LLM to recommend both model families and hyperparameters. We study two prompting strategies: (1) a zero-shot mode relying solely on pretrained knowledge, and (2) a meta-informed mode augmented with examples of models and their performance on past tasks. Across synthetic and real-world benchmarks, we show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search, and that improvements from meta-informed prompting demonstrate their capacity for in-context meta-learning. These results highlight a promising new role for LLMs as lightweight, general-purpose assistants for model selection and hyperparameter optimization.

TL;DR: This paper explores using large language models as in-context meta-learners for model and hyperparameter selection in tabular classification and regression tasks.