Title: Assessing Large Language Models (LLMs) as Foundational Recommenders: A Multi-Domain, Multi-Dataset Benchmark
Primary area: foundation or frontier models, including LLMs
Keywords: benchmark, foundation models, recommender system, multi-domain

Abstract: Recent advances in large language models (LLMs) have raised the question of whether these models can serve as foundational recommenders across diverse domains. To systematically investigate this potential, we introduce RecBench-MD, a comprehensive benchmark for assessing the recommendation capabilities of LLMs from a multi-domain and multi-dataset perspective. Our benchmark encompasses 15 datasets spanning 10 domains, including e-commerce, entertainment, and social media, and evaluates 21 state-of-the-art LLMs under zero-resource, fine-tuning, and transfer-learning scenarios. Through extensive experiments, we reveal that (i) in-domain fine-tuning consistently delivers the strongest performance, (ii) cross-dataset transfer provides practical benefits in emerging recommendation contexts, and (iii) multi-domain training enhances the adaptability of LLM-based recommenders. These findings highlight both the opportunities and limitations of positioning LLMs as foundational recommenders. To support future research, we will publicly release all code and data.

TL;DR: we collect 15 datasets for evaluation and finetuning on foundational recommenders, and investigate over eight multi-domain evaluation settings