Title: COMPASS: A Multi-Turn Benchmark for Tool-Mediated Planning & Preference Optimization
Primary area: datasets and benchmarks
Keywords: multi-turn tool calling, preference optimization, reasoning and planning

Abstract: Real-world LLM agents must master strategic tool orchestration and user preference optimization through multi-turn interactions to assist users with complex planning tasks. We introduce COMPASS (Constrained Optimization through Multi-turn Planning and Strategic Solutions), a benchmark that evaluates agents on realistic travel planning scenarios. We cast travel planning as a constrained preference optimization problem, where agents must satisfy hard constraints while simultaneously optimizing soft user preferences. To support this, we build a realistic travel database covering transportation, accommodation, and ticketing for 20 U.S. National Parks and a comprehensive tool ecosystem that mirrors commercial booking platforms. Evaluating state-of-the-art models, we uncover two critical gaps: (i) an acceptable–optimal gap, where agents reliably meet constraints but fail to optimize preferences, and (ii) a plan-coordination gap, where performance collapses on multi-service (flight, hotel) coordination tasks, especially for open-source models. By grounding reasoning and planning in a practical, user-facing domain, COMPASS offers a benchmark that directly measures an agent’s ability to optimize user preferences in realistic tasks, bridging theoretical advances with real-world impact.

TL;DR: COMPASS is a benchmark that tests whether LLM agents can intelligently optimize user preferences in realistic, tool-based travel planning tasks.