OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows

## Abstract

Abstract Autonomous agents powered by large language models (LLMs) are increasingly deployed in real-world applications requiring complex, long-horizon workflows. However, existing benchmarks predominantly focus on atomic tasks that are self-contained and independent, failing to capture the long-term contextual dependencies and multi-interaction coordination required in realistic scenarios. To address this gap, we introduce OdysseyBench , a comprehensive benchmark for evaluating LLM agents on long-horizon workflows across diverse office applications including Word, Excel, PDF, Email, and Calendar. Our benchmark comprises two complementary splits: OdysseyBench+ with 300 tasks derived from real-world use cases, and OdysseyBench-Neo with 302 newly synthesized complex tasks. Each task requires agent to identify essential information from long-horizon interaction histories and perform multi-step reasoning across various applications. To enable scalable benchmark creation, we propose HomerAgents , a multi-agent framework that automates the generation of long-horizon workflow benchmarks through systematic environment exploration, task generation, and dialogue synthesis. Our extensive evaluation demonstrates that OdysseyBench effectively challenges state-of-the-art LLM agents, providing more accurate assessment of their capabilities in complex, real-world contexts compared to existing atomic task benchmarks. We believe that OdysseyBench will serve as a valuable resource for advancing the development and evaluation of LLM agents in real-world productivity scenarios. In addition, we release OdysseyBench and HomerAgents to foster research along this line. 1 1 1 https://github.com/microsoft/OdysseyBench.git

## 1 Introduction

Autonomous agents powered by large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, including reasoning (Lin et al., 2024; Boisvert et al., 2024; Yao et al., 2024; Wang et al., 2024a), software development (Qian et al., 2023; Yang et al., 2024; Murty et al., 2024; Zhou et al., 2023; Xie et al., 2025), and scientific research (Drouin et al., 2024; Wu et al., 2025; Zheng et al., 2025). As these agents increasingly transition from research settings to real-world applications, they are expected to handle complex, multi-step tasks such as drafting professional emails, updating documents, and managing personal calendars (Yao et al., 2024; Wang et al., 2024d; Xu et al., 2024a). This shift underscores the need for the development of comprehensive benchmarks that accurately reflect real-world scenarios and rigorously evaluate agent performance in complex, contextual task environments.

However, existing benchmarks for agents predominantly focus on atomic tasks that are self-contained and independent of previous interactions or accumulated context (Zhou et al., 2023; Paranjape et al., 2023; Bonatti et al., 2024; Wang et al., 2024d; Xu et al., 2024a), as illustrated in LABEL:fig:example-others. While these benchmarks serve as valuable initial assessments, they fundamentally misrepresent the nature of real-world workflows, which typically unfold across extended periods and encompass various agent-user interactions and require agents to systematically curate, integrate, and leverage information accumulated over extended periods (Schick et al., 2023; Wang et al., 2024c; Hu et al., 2024; Erdogan et al., 2025). Agents that perform well on atomic task benchmarks may struggle with the contextual dependencies, information persistence, and collaborative workflow management required in real-world scenarios.

In this work, we address these challenges by introducing a novel benchmark OdysseyBench designed to evaluate agents on complex, long-horizon workflows spanning diverse office applications, including Word, Excel, PDF, Email, and Calendar. Our benchmark includes two splits: OdysseyBench+, which consists of 300 long-horizon tasks originated from real-world use cases in OfficeBench (Wang et al., 2024d), and OdysseyBench-Neo, which contains 302 newly generated tasks that are more complex and diverse.
Each task, as illustrated in LABEL:fig:example-ours, is designed to require the agent to reason about the task and extract essential information from long-horizon dialogue histories between the user and agent. This enables the construction of feasible workflows and supports multi-step reasoning across various applications. The tasks are structured to reflect the complexities of agent-user interactions, emphasizing the need for agents to maintain context, synthesize information from prior exchanges, and coordinate actions across diverse tools and environments.

Furthermore, many benchmarks rely on costly human annotation, limiting scalability and constraining the diversity of evaluation scenarios (Zhou et al., 2023; Xu et al., 2024a; Yao et al., 2024). While recent efforts have explored synthetic data generation with LLMs (Ou et al., 2024; Xu et al., 2024b; Xie et al., 2025), these approaches typically yield atomic tasks, lacking the sustained interactions and long-term context essential for realistic workflows. These limitations highlight the urgent need for systematic, automated benchmarks that accurately reflect the challenges of real-world, long-horizon tasks.

To address these challenges, we propose HomerAgents, a multi-agent framework that automates the generation of long-horizon workflow benchmarks. Our framework consists of two complementary components: HomerAgents+, which leverages existing benchmarks from OfficeBench (Wang et al., 2024d) and employs a two-agent iterative refinement process to transform atomic tasks into contextually rich, multi-interaction scenarios, thereby creating OdysseyBench+; and HomerAgents-Neo, which utilizes a multi-agent system operating within realistic application environments to generate entirely new long-horizon tasks from scratch, producing OdysseyBench-Neo. Through systematic environment exploration, task generation, and dialogue creation, HomerAgents enables scalable production of diverse, contextually grounded benchmark tasks that reflect the complexity of real-world productivity scenarios while maintaining the quality standards necessary for rigorous agent evaluation.

We conduct extensive evaluations of OdysseyBench using state-of-the-art agents, demonstrating that these benchmarks effectively challenge current models and provide a more accurate assessment of their capabilities in real-world contexts.

In summary, our contributions are as follows:

- •
We introduce OdysseyBench, a comprehensive benchmark for evaluating agents on long-horizon workflows across multiple office applications, consisting of OdysseyBench+ and OdysseyBench-Neo.
- •
We propose HomerAgents, a multi-agent framework that automates the generation of long-horizon tasks, enabling scalable and diverse benchmark creation.
- •
We demonstrate the effectiveness of OdysseyBench in challenging state-of-the-art language agents, providing insights into their performance in complex, real-world scenarios.
- •
We analyze the impact of dialogue storage formats within OdysseyBench, demonstrating that semantic compression and coherent aggregation are essential for effective multi-step reasoning and agent performance.

## 2 Related Work

##### Evaluating LLMs in Executive Environments

As LLMs advance in tackling real-world challenges (Hurst et al., 2024; Jaech et al., 2024; OpenAI, 2025; Anthropic, 2025b; a; Comanici et al., 2025), there is a growing shift toward evaluating their capabilities in dynamic, executive environments rather than static datasets. Beyond text-based games (Côté et al., 2018; Shridhar et al., 2020), recent research increasingly simulates realistic scenarios to assess agents’ proficiency in tool use (Deng et al., 2023; Qin et al., 2023a; Zhuang et al., 2023; Qin et al., 2023b; Lù et al., 2024; Wang et al., 2024b; Shen et al., 2024; Xu et al., 2024a; Sutela & Lindström, 2024). Current benchmarks, such as WebArena (Zhou et al., 2023), AgentBench (Paranjape et al., 2023), WindowsArena (Bonatti et al., 2024), and OfficeBench (Wang et al., 2024d), provide valuable evaluation settings focused on web and office environments. However, these platforms primarily measure atomic performance in self-contained contexts and lack mechanisms to evaluate LLM agents’ interactions with complex environments over extended periods. This limitation is significant, as robust assessment of planning, long-term information retrieval, and execution is essential for understanding agents’ true capabilities in real-world tasks.

##### Synthetic Benchmark Generation

Existing agent datasets and benchmarks largely rely on human annotators for task creation, demonstrations, and evaluation metric design (Zhou et al., 2023; Xu et al., 2024a; Yao et al., 2024), resulting in high costs and limited diversity. Recent studies try to leverage LLMs to automatically generate agent tasks and trajectories (Ou et al., 2024; Xu et al., 2024b; Xie et al., 2025). For instance, Murty et al. (2024); Pahuja et al. (2025); Trabucco et al. (2025); Gandhi & Neubig (2025) employ LLMs as web agents to synthesize web-based interactions in semi-realistic environments. Moreover, composing atomic tasks is another method to construct more challenging tasks (Boisvert et al., 2024; Drouin et al., 2024). Li et al. (2024) iteratively propose and refine dataset descriptions to generate topic-specific problems. However, these approaches predominantly focus on web-based activities and are generally limited to simple interactions, lacking the complexity of multi-step reasoning and extensive tool use required for robust agent evaluation.

##### Ours

Distinct from previous approaches, we introduces a multi-agent framework HomerAgents to automatically construct the long-term workflow benchmark OdysseyBench, enabling a more rigorous assessment of agents’ abilities to curate context to handle complex tasks. OdysseyBench is specifically designed to evaluate agent performance in realistic office scenarios, where agents must interact with multiple applications to accomplish intricate objectives. This benchmark challenges agents to reason about rgb(0bp)=(0,0,0); rgb(25bp)=(0,0,0); rgb(75bp)=(0.5,0.5,0.5); rgb(100bp)=(0.5,0.5,0.5)task intent, extract critical information from dialogue history, and assemble feasible workflows, thereby providing a comprehensive evaluation of their capabilities in dynamic, multi-step environments.

Figure: Figure 2: HomerAgents Framework Overview. HomerAgents consists of two components: HomerAgents+ and HomerAgents-Neo. HomerAgents+ builds upon the task descriptions from OfficeBench to generate long-horizon dialogues, while HomerAgents-Neo creates entirely new tasks and corresponding dialogues from scratch by employing a multi-agent system that operates within realistic application environments.
Refer to caption: x1.png

Figure: Algorithm 1 HomerAgents+

Figure: Algorithm 2 HomerAgents-Neo

## 3 Methodology

In this section, we firstly introduce HomerAgents, a multi-agent framework that automatically generates the long-horizon workflow benchmark OdysseyBench in [Section 3.1](https://arxiv.org/html/2508.09124v1#S3.SS1), including two components: HomerAgents+ ([Section 3.1.1](https://arxiv.org/html/2508.09124v1#S3.SS1.SSS1)) and HomerAgents-Neo ([Section 3.1.2](https://arxiv.org/html/2508.09124v1#S3.SS1.SSS2)). We then describe the long-horizon workflow benchmark OdysseyBench in [Section 3.2](https://arxiv.org/html/2508.09124v1#S3.SS2), including the dataset analysis ([Section 3.2.2](https://arxiv.org/html/2508.09124v1#S3.SS2.SSS2)), quality control measures ([Section 3.2.3](https://arxiv.org/html/2508.09124v1#S3.SS2.SSS3)), and human evaluation ([Section 3.2.4](https://arxiv.org/html/2508.09124v1#S3.SS2.SSS4)).

### 3.1 HomerAgents : Automating Benchmark Creation

It is highly challenging to create OdysseyBench in a scalable and reliable manner, as it requires generating realistic user–assistant interaction histories and the context-dependent multi-step tasks that reflect the complexity and ambiguity of real-world productivity scenarios. To facilitate this process, we propose a multi-agent framework HomerAgents that automates the generation of OdysseyBench benchmark tasks, including HomerAgents+ (see [Section 3.1.1](https://arxiv.org/html/2508.09124v1#S3.SS1.SSS1)) and HomerAgents-Neo (see [Section 3.1.2](https://arxiv.org/html/2508.09124v1#S3.SS1.SSS2)).

#### 3.1.1 HomerAgents+ : Standing on the Shoulders of OfficeBench

HomerAgents+ builds upon the task descriptions from OfficeBench (Wang et al., 2024d) to generate long-horizon dialogue scenarios that more closely mirror real-world productivity workflows. Starting from a given task description $\mathcal{T}$, HomerAgents+ employs a two-agent iterative refinement framework to produce task intents $\mathbb{I}$ and corresponding long-horizon user-assistant dialogues $\mathbb{D}$, thereby contextualizing and enriching the original task.

The framework comprises two core components: a generator ($\mathcal{G}$) and a verifier ($\mathcal{V}$), as depicted in [Figure 2](https://arxiv.org/html/2508.09124v1#S1.F2). The generator $\mathcal{G}$ receives the task description $\mathcal{T}$ and any feedback from previous iterations $\mathbb{F}_{i-1}$, and outputs a task intent $\mathbb{I}_{i}$ along with a corresponding dialogue $\mathbb{D}_{i}$. Here, the task intent $\mathbb{I}$ succinctly captures the user’s goal without specific details, while the dialogue $\mathbb{D}$ provides the natural conversational context leading to the task. The verifier $\mathcal{V}$ then evaluates the generated content against criteria such as dialogue realism, task alignment, and contextual coherence, returning structured feedback $\mathbb{F}_{i}$.

This process is executed iteratively, as outlined in [Algorithm 1](https://arxiv.org/html/2508.09124v1#algorithm1), with a maximum of $N_{\text{max}}$ iterations. In each iteration $i$, the generator $\mathcal{G}$ refines its output based on the original task and accumulated feedback, while the verifier $\mathcal{V}$ either approves the result (“pass”) or provides actionable feedback for further improvement. The cycle continues until the verifier approves the content or the iteration limit is reached. By leveraging established benchmarks and introducing an iterative, feedback-driven process, HomerAgents+ enables the creation of contextually grounded, long-horizon tasks that are both practically relevant and sufficiently complex to rigorously evaluate long-horizon workflow task understanding in productivity settings.

#### 3.1.2 HomerAgents-Neo : Scaling up the Benchmark Creation

While HomerAgents+ effectively leverages existing benchmarks, HomerAgents-Neo addresses the need for more diverse and scalable task generation by creating entirely new long-horizon tasks from scratch. HomerAgents-Neo employs a multi-agent system that operates within realistic application environments to generate authentic productivity scenarios, as shown in [Figure 2](https://arxiv.org/html/2508.09124v1#S1.F2).

HomerAgents-Neo consists of productivity applications $\mathcal{A}=\{a_{k}\}_{k=0}^{K}$, environment $\mathcal{E}$, orchestrator $\mathcal{O}$, surfers $\mathcal{S}=\{S_{k}\}_{k=0}^{K}$, task generator $\mathcal{G}_{\text{task}}$, and dialogue generator $\mathcal{G}_{\text{dial}}$. Orchestrator $\mathcal{O}$ manages planning, progress tracking, and coordinates the entire generation process by orchestrating each stage of data generation, ensuring coherence in both task and dialogue creation. Surfers $\mathcal{S}$ gather information from environment by interacting with a diverse set of simulated productivity applications. Task generator $\mathcal{G}_{\text{task}}$ synthesizes the tasks and corresponding evaluation criteria. Dialogue generator $\mathcal{G}_{\text{dial}}$ then creates multi-day dialogues simulating realistic user-assistant interactions.

The framework consists of four distinct phases, as outlined in [Algorithm 2](https://arxiv.org/html/2508.09124v1#algorithm2):

##### Phase 1: Planning

The orchestrator $\mathcal{O}$ receives a set of applications $\mathcal{A}=\{a_{k}\}_{k=0}^{K}$ and environment $\mathcal{E}$, then formulates a comprehensive generation plan $\mathbb{P}=\{\mathbb{P}_{\text{surf}},\mathbb{P}_{\text{task}},\mathbb{P}_{\text{dial}}\}$. This plan specifies how the subsequent phases should explore the environment $\mathbb{P}_{\text{surf}}$, generate tasks $\mathbb{P}_{\text{task}}$, and create dialogues $\mathbb{P}_{\text{dial}}$.

##### Phase 2: Environment Exploration

A collection of specialized surfers $\mathcal{S}=\{S_{k}\}_{k=0}^{K}$ systematically explore the application environment. Each surfer $S_{k}$ follows the surfing plan $\mathbb{P}_{\text{surf}}$ to interact with application $a_{k}$ within environment $\mathcal{E}$, collecting contextual information $\mathbb{C}$. This exploration phase ensures that generated tasks are grounded in realistic application capabilities and user workflows.

##### Phase 3: Task Generation

The task generator $\mathcal{G}_{\text{task}}$ utilizes the collected contextual information $\mathbb{C}$ and the task generation plan $\mathbb{P}_{\text{task}}$ to create comprehensive task specifications $\tau=\{\mathbb{T},\mathbb{I},\mathbb{K},\mathbb{E}\}$. This includes the task description $\mathbb{T}$, the task intent $\mathbb{I}$, detailed subtask instructions $\mathbb{K}$, and evaluation criteria $\mathbb{E}$. The task description $\mathbb{T}$ outlines the specific goals and requirements of the task, the task intent $\mathbb{I}$ conveys the high-level overall goal but omits specific details of the task, $\mathbb{K}=\{k_{1},\ldots,k_{t}\}$ provides instructions for completing the task, and the evaluation criteria $\mathbb{E}$ define how the task’s success will be measured.

##### Phase 4: Dialogue Generation

Finally, the dialogue generator $\mathcal{G}_{\text{dial}}$ creates natural user-assistant conversations $\mathbb{D}$ that lead to the generated task. This process incorporates the dialogue plan $\mathbb{P}_{\text{dial}}$, contextual information $\mathbb{C}$, task intent $\mathbb{I}$, and subtask instructions $\mathbb{K}$ to produce realistic long-horizon dialogues that capture the gradual evolution of user requirements. For each subtask instruction $k_{i}\in\mathbb{K}$, the dialogue generator $\mathcal{G}_{\text{dial}}$ produces a corresponding dialogue $\mathbb{D}_{i}$ that simulates the interaction between the user and the assistant, reflecting how the task is approached over multiple days. Combining these dialogues, we obtain a comprehensive dialogue history $\mathbb{D}=\{\mathbb{D}_{1},\ldots,\mathbb{D}_{t}\}$ that illustrates the user’s journey through the task. Additionally, we also include task-irrelevant content (e.g. chitchat) in the generated dialogues $\mathbb{D}$ to make the generated content align better with the real-world scenarios.

By decomposing the generation process into these four phases, HomerAgents-Neo ensures systematic exploration of application environments while maintaining coherence between the generated tasks and dialogues. This approach enables scalable creation of diverse, contextually grounded benchmark tasks that reflect the complexity of real-world productivity scenarios.

#### 3.1.3 Implementation Details

Considering the trade-off between performance and cost, we implement all agents in HomerAgents using the GPT-4.1 model, which offers strong capabilities for complex reasoning tasks while remaining cost-effective. The maximum iterations $N_{\text{max}}$ in [Algorithm 1](https://arxiv.org/html/2508.09124v1#algorithm1) are set to 5, allowing for sufficient exploration of the task space while managing computational resources effectively. The $T$ in [Algorithm 2](https://arxiv.org/html/2508.09124v1#algorithm2) is set to 5, representing generating at least five days of dialogues, which is sufficient to capture the complexity of long-term workflows. Additionally, we implement HomerAgents-Neo based on the Magentic-One framework (Fourney et al., 2024).

During dialogue generation, when the user assigns subtasks to the assistant, the assistant does not actually execute the tasks but instead simulates execution by generating responses based on the task descriptions and dialogue context. This approach enables us to focus on generating diverse and realistic dialogues at scale, without the need for real task execution. By deferring execution, our benchmark evaluates agents’ abilities to curate and integrate information distributed across multiple dialogue turns and days, an essential aspect for assessing long-horizon comprehension and planning.

### 3.2 OdysseyBench : Long-Horizon Workflow Benchmark

#### 3.2.1 Evaluation

In OdysseyBench, LLM agents are required to interact with multiple applications to complete complex tasks. This process demands that agents reason about task intent and extract essential information from the dialogue history to construct feasible workflows. We construct OdysseyBench within a Docker environment containing pre-installed applications and automate operations using Python libraries. We set up a file system to manage documents, emails, and calendar events required for the tasks. After the agents complete each task, we save the entire file system and perform customized evaluations to verify correctness.

Our evaluation integrates exact matching, fuzzy matching, and execution-based methods. Exact and fuzzy matching assess whether the agent’s task output aligns with the expected (e.g., keyword matching for generated documents and calendar events), while the execution-based method verifies if the agent’s task outputs can be successfully evaluated via code snippets (e.g., checking calendar conflicts). The output is considered to be successful if all the evaluation criteria are satisfied. We report the pass rate as the measure of model performance, where the pass rate is defined as the percentage of tasks completed successfully: $\frac{\#\text{successful tasks}}{\#\text{total tasks}}$.

**Table 1: Data statistics of OdysseyBench+ and OdysseyBench-Neo.**
|  | OdysseyBench+ | OdysseyBench-Neo |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | single<br>apps | two<br>apps | three<br>apps | overall | single<br>apps | two<br>apps | three<br>apps | overall |
| Total # conversation $h.$ | 00093 | 00095 | 00112 | 00300 | 00060 | 00071 | 00171 | 000302 |
| Avg. # session $k.$ in conversation $h$ | 0027.8 | 0024.7 | 0030.6 | 0027.9 | 0005.0 | 0005.0 | 0005.1 | 0005.0 |
| Avg. # utterance $j.$ in session $k$ | 0010.8 | 0012.1 | 0011.4 | 0011.4 | 0072.3 | 0073.5 | 0073.3 | 0073.2 |
| Avg. # tokens. conversation $h$ | 3323.2 | 3209.6 | 3809.9 | 3468.9 | 5031.6 | 5223.1 | 5196.4 | 5169.9 |
| Avg. # tokens. sessions $k$ | 0119.7 | 0130.1 | 0124.4 | 0124.6 | 1006.3 | 1041.7 | 1026.1 | 1025.8 |
| Avg. # tokens. utterance $j$ | 0011.1 | 0010.8 | 0010.9 | 0010.9 | 0013.9 | 0014.2 | 0014.0 | 0014.0 |

#### 3.2.2 Dataset Analysis

As shown in [Table 1](https://arxiv.org/html/2508.09124v1#S3.T1), our dataset comprises 602 tasks, categorized by the number of applications involved: Single App (153 tasks), Two Apps (166 tasks), and Three Apps (283 tasks). Each task is documented through multi-day dialogues, with at least five days per task. Dialogues occurring within the same day are grouped into a single session, and every dialogue contains a minimum of 10 utterances, ensuring rich interaction data. OdysseyBench+ contains 300 conversation histories with an average of 27.9 sessions per conversation and 11.4 utterances per session, resulting in relatively short sessions with an average of 124.6 tokens per session. In contrast, OdysseyBench-Neo comprises 302 conversations with a more structured format of exactly 5 sessions per conversation (corresponding to the 5-day dialogue design) but significantly longer sessions, averaging 1025.8 tokens each and 73.2 utterances per session. This design difference reflects OdysseyBench-Neo’s focus on creating more comprehensive daily interactions, while OdysseyBench+ maintains the original fragmented conversation structure from OfficeBench. Overall, OdysseyBench-Neo generates richer conversational content with approximately 49% more tokens per conversation (5169.9 vs. 3468.9 tokens), demonstrating the enhanced depth and complexity of the newly generated tasks.

We further analyze the distribution of execution steps in OdysseyBench+ and OdysseyBench-Neo, as illustrated in LABEL:fig:execution-turns. The number of execution steps required to complete a task is consistent across both datasets, with the majority of tasks requiring 3-15 execution turns. This demonstrates that tasks in OdysseyBench are sufficiently complex, mirroring real-world scenarios where users must navigate multi-step workflows across multiple applications. Furthermore, we also analyze the diversity of the tasks in OdysseyBench, an overview of actions, objects, and applications in OdysseyBench is provided in LABEL:fig:verb-noun. Our OdysseyBench benchmark encompasses a wide range of actions, objects, and applications, ensuring that it captures the complexity and variety of real-world productivity tasks. This diversity enhances the benchmark’s applicability to various productivity scenarios, making it a valuable resource for evaluating long-horizon workflow understanding in LLMs.

#### 3.2.3 Quality Control

**Table 2: Quality verification performance of generated rgb(0bp)=(0,0,0); rgb(25bp)=(0,0,0); rgb(75bp)=(0.5,0.5,0.5); rgb(100bp)=(0.5,0.5,0.5)task intentand dialogues for OdysseyBench.**
| Metric | OdysseyBench+ | OdysseyBench-Neo |
| --- | --- | --- |
| Completeness | 81.33 | 93.71 |
| Appropriateness | 88.33 | 88.08 |
| Both | 72.67 | 83.77 |

##### Automated Validation

To ensure the generated tasks are both high-quality and solvable, we implement a systematic automated validation pipeline. Our approach consists of two primary validation stages. First, we verify task solvability by filtering out tasks whose evaluation criteria $\mathbb{E}$ fall outside our predefined evaluation function library. This ensures that each task has well-defined, measurable success criteria. Second, we conduct a consistency check between the task description $\mathbb{T}$ and the information available to agents during evaluation. Specifically, we test whether a powerful LLM (o3) can solve the task when provided with: (1) the task description $\mathbb{T}$, and (2) only the task intent $\mathbb{I}$ and subtask instructions $\mathbb{K}$. Tasks are considered valid only if the LLM succeeds in both scenarios, ensuring that the dialogue contains sufficient information for task completion while the intent appropriately abstracts the core objective. This cross-validation eliminates under-specified tasks and confirms that essential information is properly embedded within the conversational context. To complement this automated filtering, we employ an LLM-as-a-judge method where five independent GPT-4.1 agents evaluate the generated task intent $\mathbb{I}$ and dialogues $\mathbb{D}$ across two key dimensions:

- 1.
Completeness: For any given task description, the combination of rgb(0bp)=(0,0,0); rgb(25bp)=(0,0,0); rgb(75bp)=(0.5,0.5,0.5); rgb(100bp)=(0.5,0.5,0.5)task intentand dialogues should provide sufficient information for an agent to solve the task without omitting any necessary details.
- 2.
Soundness: The rgb(0bp)=(0,0,0); rgb(25bp)=(0,0,0); rgb(75bp)=(0.5,0.5,0.5); rgb(100bp)=(0.5,0.5,0.5)task intentshould not leak any specific information from the task description; all essential details must be conveyed through the dialogues.

Each agent provides an independent judgment, and a majority voting mechanism aggregates these assessments to determine the overall data quality. The results of this evaluation are presented in [Table 2](https://arxiv.org/html/2508.09124v1#S3.T2). A minimum pass rate of 70% reflects the quality of the data, indicating the effectiveness of the automatic generation of HomerAgents.

##### Human Verification and Post-Editing

In addition to automated validation, we implement human curation to further enhance the quality of the generated rgb(0bp)=(0,0,0); rgb(25bp)=(0,0,0); rgb(75bp)=(0.5,0.5,0.5); rgb(100bp)=(0.5,0.5,0.5)task intentand dialogues. A team of three native English-speaking annotators manually reviews the generated rgb(0bp)=(0,0,0); rgb(25bp)=(0,0,0); rgb(75bp)=(0.5,0.5,0.5); rgb(100bp)=(0.5,0.5,0.5)task intentand dialogues, assessing them for completeness, appropriateness, and logical coherence. During this process, curators remove any tasks that fail to meet established quality standards. While our cross-checking mechanism filters out tasks deemed unsolvable by the LLM, this method is inherently limited by the LLM’s problem-solving capabilities. Consequently, human curation intentionally includes tasks that satisfy quality standards yet remain unsolvable by the LLM. This human-in-the-loop approach ensures that the resulting dataset is both challenging and reflective of real-world use cases.

#### 3.2.4 Human Evaluation

**Table 3: Human performance of HomerAgents-Neo**
| Task | 1-apps | 2-apps | 3-apps | overall |
| --- | --- | --- | --- | --- |
| Human | 92.31 | 90.00 | 91.67 | 91.43 |

We also ask two human annotators to perform a randomly sampled subset of the tasks and report the human performance in [Table 3](https://arxiv.org/html/2508.09124v1#S3.T3). The human annotators are asked to complete the tasks using the same productivity applications as those provided to the agents. The human performance is over 90%, indicating that the tasks are solvable and coherent.

## 4 Experimental Setup

##### Long-Context Evaluation

We evaluate the agent performance on OdysseyBench using the long-context setting, where the entire dialogue history is provided to the agent.

##### RAG Evaluation

We also evaluate agent performance on OdysseyBench under the Retrieval-Augmented Generation (RAG) setting, where the agent retrieves relevant context from the dialogue history using embedding models to generate responses. We conduct experiments with two types of stored context: (1) raw context and (2) summarized context. Furthermore, each type of context is organized into two levels of granularity. For raw context: (a) *session-level:* the entire dialogue of each session is stored and embedded as a single document; (b) *utterance-level:* each user/assistant turn is treated as a separate document and embedded independently. For summarized context: (a) *session-level:* the full session is summarized and stored as a single document; (b) *chunk-level:* multiple sessions are concatenated and segmented into coherent chunks, with each chunk summarized independently.

##### Evaluation Metrics

As mentioned in [Section 3.2.1](https://arxiv.org/html/2508.09124v1#S3.SS2.SSS1), we measure the agent performance using the pass rate, which is the percentage of successful task completions out of the total number of tasks.

##### Models

We evaluate the long-horizon workflow automation capabilities of the agents of the proprietary LLMs, including o3, o3-mini, GPT-4o, GPT-4o-mini, GPT-4.1, GPT-5, and GPT-5-chat, and the open-weight LLMs, including DeepSeek-R1, DeepSeek-R1-Distill-Qwen-32b, and Qwen3-32b, as these models are among the highest-ranking LLMs available. And the embedding model used for RAG is OpenAI text-embedding-3-large.

**Table 4: Main results given by multiple proprietary models or open-weight models on OdysseyBench+ and OdysseyBench-Neo tasks under the long-context configuration. We divide the tasks into “1/2/3-apps”, specifying the number of applications required by the tasks. The overall performance is reported as the macro-average across all tasks.**
|  | OdysseyBench+ | OdysseyBench-Neo |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 1-apps | 2-apps | 3-apps | overall | 1-apps | 2-apps | 3-apps | overall |
| Proprietary Models |  |  |  |  |  |  |  |  |
| o3 | 72.83 | 70.53 | 30.36 | 56.19 | 68.33 | 60.56 | 59.06 | 61.26 |
| o3-mini | 38.04 | 20.00 | 15.18 | 23.75 | 71.67 | 39.44 | 45.61 | 49.34 |
| GPT-4o-mini | 30.11 | 22.11 | 07.14 | 19.00 | 65.00 | 33.80 | 29.83 | 37.75 |
| GPT-4o | 47.31 | 42.11 | 15.18 | 33.67 | 75.00 | 47.89 | 45.61 | 51.99 |
| GPT-4.1 | 55.91 | 43.16 | 12.50 | 35.67 | 75.00 | 63.38 | 47.37 | 56.62 |
| GPT-5-chat | 55.91 | 48.42 | 20.54 | 40.33 | 75.00 | 57.75 | 51.46 | 57.62 |
| GPT-5 | 75.27 | 66.32 | 25.89 | 54.00 | 61.67 | 56.34 | 53.80 | 55.96 |
| Open-weight Models |  |  |  |  |  |  |  |  |
| DeepSeek-R1 | 53.76 | 47.37 | 20.54 | 39.33 | 78.33 | 60.56 | 44.44 | 54.97 |
| DS.-Distill-Qwen-32b | 30.11 | 16.84 | 01.79 | 15.33 | 40.00 | 22.54 | 10.53 | 19.21 |
| Qwen-3-32b | 38.71 | 33.68 | 11.61 | 27.00 | 41.67 | 22.54 | 21.05 | 25.50 |

**Table 5: Performance of RAG-based GPT-4o on the OdysseyBench+. “Long-context prompting baseline” represents the results evaluated in the long-context setting. “top-k” means the top-k retrieved documents used as the context, and “tokens” indicates the number of tokens in the retrieved documents.**
| Storage | granularity | top-k | tokens | 1-apps | 2-apps | 3-apps | overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Long-context prompting baseline | 8000 | 47.31 | 42.11 | 15.18 | 33.67 |  |  |
| raw | session | 5 | 750 | 40.86 | 40.00 | 11.61 | 29.67 |
| 10 | 1500 | 39.79 | 40.00 | 14.29 | 30.33 |  |  |
| utterance | 5 | 80 | 29.03 | 35.79 | 08.04 | 23.33 |  |
| 10 | 155 | 27.96 | 33.68 | 08.93 | 22.67 |  |  |
| 25 | 370 | 39.79 | 35.79 | 12.50 | 28.33 |  |  |
| 50 | 730 | 57.69 | 40.00 | 17.17 | 29.41 |  |  |
| summary | session | 5 | 290 | 29.03 | 35.79 | 09.82 | 24.00 |
| 10 | 650 | 33.33 | 36.84 | 09.82 | 25.67 |  |  |
| chunk | 5 | 290 | 30.11 | 29.47 | 12.50 | 23.33 |  |
| 10 | 380 | 40.86 | 34.74 | 16.96 | 30.00 |  |  |
| 25 | 600 | 46.24 | 36.84 | 19.64 | 33.33 |  |  |
| 50 | 670 | 44.09 | 40.00 | 16.96 | 32.67 |  |  |

**Table 6: Performance of RAG-based GPT-4o on the OdysseyBench-Neo. “Long-context prompting baseline” represents the results evaluated in the long-context setting. “top-k” means the top-k retrieved documents used as the context, and “tokens” indicates the number of tokens in the retrieved documents.**
| Storage | granularity | top-k | tokens | 1-apps | 2-apps | 3-apps | overall |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Long-context prompting baseline | 6700 | 75.00 | 47.89 | 45.61 | 51.99 |  |  |
| raw | utterance | 5 | 90 | 30.00 | 16.90 | 08.19 | 14.57 |
| 10 | 180 | 31.67 | 16.90 | 11.11 | 16.56 |  |  |
| 25 | 450 | 35.00 | 32.39 | 21.05 | 26.49 |  |  |
| 50 | 915 | 56.67 | 40.85 | 31.58 | 38.74 |  |  |
| summary | session | 5 | 2200 | 75.00 | 46.48 | 49.12 | 53.64 |
| chunk | 5 | 1200 | 30.11 | 29.47 | 12.50 | 23.33 |  |
| 10 | 1260 | 40.86 | 34.74 | 16.96 | 30.00 |  |  |
| 25 | 1360 | 68.33 | 59.16 | 50.88 | 56.29 |  |  |
| 50 | 1460 | 68.33 | 59.16 | 48.54 | 54.97 |  |  |

**Table 7: The number of execution steps of the task in OdysseyBench+ and OdysseyBench-Neo under different configurations indicates how many steps are required to successfully execute the task. “configuration” represents the experimental setup used for evaluation.**
|  | configuration | 1-apps | 2-apps | 3-apps | overall |
| --- | --- | --- | --- | --- | --- |
| OdysseyBench+ | long-context | 6.31 | 11.61 | 12.70 | 10.25 |
| RAG-utterance | 6.85 | 11.48 | 14.70 | 11.05 |  |
| RAG-chunk | 7.25 | 8.28 | 14.86 | 10.10 |  |
| OdysseyBench-Neo | long-context | 7.81 | 9.63 | 11.74 | 10.46 |
| RAG-utterance | 8.17 | 9.66 | 12.52 | 10.92 |  |
| RAG-chunk | 7.93 | 9.92 | 12.54 | 10.95 |  |

## 5 Experimental Results

In this section, we evaluate several LLMs and RAG-based approaches on the OdysseyBench+ and OdysseyBench-Neo tasks, focusing on their performance across different configurations. We analyze how task complexity, context length, and retrieval strategies impact model effectiveness.

##### Tasks get increasingly complex with more applications involved, leading to a performance drop.

As shown in [Table 4](https://arxiv.org/html/2508.09124v1#S4.T4), we observe a consistent performance degradation across all models as the number of applications in a task increases. For OdysseyBench+ tasks, the average performance drops from single-app scenarios to three-app scenarios across all models: o3 drops from 72.83 to 30.36, GPT-4.1 from 55.91 to 12.50, and DeepSeek-R1 from 53.76 to 20.54. This trend is also evident in OdysseyBench-Neo tasks, though the degradation is less severe. For instance, o3 maintains relatively stable performance (68.33 to 59.06), while GPT-4o shows a decline from 75.00 to 45.61. These findings suggest that coordinating information across multiple applications presents significant challenges for current LLMs, requiring sophisticated reasoning about inter-application dependencies and state management.

[Table 5](https://arxiv.org/html/2508.09124v1#S4.T5) and [Table 6](https://arxiv.org/html/2508.09124v1#S4.T6) present comprehensive results for RAG-based GPT-4o across different storage formats, retrieval granularities, and top-k configurations on OdysseyBench+ and OdysseyBench-Neo, respectively.

##### More context typically leads to better performance, but at a cost.

Storing raw data without retrieval context (long-context baseline) yields the highest performance (33.67 on OdysseyBench+ with 8000 tokens; 51.99 on OdysseyBench-Neo with 6700 tokens), but incurs substantial token consumption. Within RAG approaches using raw storage, utterance-level retrieval demonstrates a nuanced balance between performance and efficiency, peaking at 29.41 with 730 tokens on OdysseyBench+ and 38.74 with 915 tokens on OdysseyBench-Neo. Notably, utterance-level retrieval outperforms the long-context baseline for single app and three apps tasks in OdysseyBench+, yet underperforms the baseline in OdysseyBench-Neo. This discrepancy likely stems from the shorter dialogues in OdysseyBench+, which make utterance-level retrieval more effective. In contrast, the performance drop in OdysseyBench-Neo suggests that excessive fragmentation of information undermines context integrity. These results underscore the importance of maintaining coherent conversational boundaries, as fragmented utterances fail to preserve essential dialogue context.

##### Summary storage effectively captures task essence.

Summarization condenses information while retaining key context, consistently improving performance across configurations. For instance, session-level summaries surpass the long-context baseline, achieving 53.64 on OdysseyBench-Neo with only one third of the token usage. Chunk-level summaries further excel, reaching 56.29 on OdysseyBench-Neo with less than 20% of the tokens.
The superior performance of summarized context can be attributed to its ability to distill key information while removing redundant details, allowing models to focus on essential task-relevant content. Additionally, summarized chunks provide better semantic density, enabling more effective retrieval of contextually relevant information within the same token budget.
By aggregating information across sessions and maintaining semantic coherence, chunk-level summaries balance context breadth with retrieval precision. Furthermore, analysis of execution steps in [Table 7](https://arxiv.org/html/2508.09124v1#S4.T7) reveals that chunk-level summaries introduce negligible computational overhead, and in some cases, even reduce the number of steps required to complete tasks. This indicates that summarization not only boosts performance, but also streamlines the reasoning process by providing relevant context efficiently, without overwhelming the model. These findings underscore the critical role of semantic compression and coherent aggregation in enabling effective multi-step reasoning.

##### Retrieval volume yields diminishing returns.

Increasing top-k from 25 to 50 results in a slight performance decrease (from 56.29 to 54.97 on OdysseyBench-Neo), suggesting that longer contexts introduce more noise and irrelevant information. This demonstrates that the quality of retrieved content is more important than sheer volume, as excessive retrieval can dilute contextual relevance. Furthermore, as task complexity increases, the performance gap between storage types widens, with summaries maintaining 2-3x higher performance on three-app tasks compared to raw utterances. Overall, these results advocate for memory architectures that prioritize semantic aggregation and context continuity, which are essential for long-term, multi-step workflow tasks.

## 6 Case Study

Figure: Figure 8: Error analysis of various file types under three configurations: long-C. (long-context) , RAG-U. (Rag-utterance), and RAG-C. (RAG-chunk).
Refer to caption: x2.png

To elucidate the failure patterns of LLM agents in OdysseyBench, we manually examined execution traces and systematically categorized failure modes based on agent behaviors and outcome status. Our analysis identifies four primary sources of failure:
(1) Missing required files: Agents overlook references to input sources mentioned in the dialogues. For example, in LABEL:fig:eg-file, agents missed the information about “EmployeeTrainingManuals.docx” and were unable to locate the file for reading.
(2) Missing required actions: Agents fail to generate or modify files as specified in the dialogues. As shown in LABEL:fig:eg-act, agents missed the instruction to “analyze the relationship,” resulting in no content for the save action.
(3) Incorrect tool calls: Agents invoke the wrong function or use incorrect arguments. In LABEL:fig:eg-tool, agents used the PDF tool to create PDF files, which should have been created with Word and then converted to PDF.
(4) Inaccurate planning: Agents do not formulate a coherent plan to complete the task. For instance, in LABEL:fig:eg-plan, agents should first read the content in PDF files and then write in the Word document, rather than writing directly in the Word document.
Further quantitative analysis based on the file types involved in failed executions ([Figure 8](https://arxiv.org/html/2508.09124v1#S5.F8)) reveals that most errors are associated with file creation or writing tasks, particularly for formats such as “docx” and “xlsx”. This suggests that agents frequently struggle to execute complex, multi-step workflows that demand precise coordination across time, tools, and reasoning.

## 7 Conclusion

In this work, we addressed the critical limitation of existing atomic task benchmarks by introducing OdysseyBench, a comprehensive benchmark for evaluating language agents on long-horizon workflows across diverse office applications. Our key contribution, HomerAgents, provides a scalable multi-agent framework that automates benchmark generation through two complementary approaches: HomerAgents+ transforms existing atomic tasks into contextually rich scenarios to create OdysseyBench+, while HomerAgents-Neo generates entirely new complex tasks from scratch to produce OdysseyBench-Neo. Extensive evaluation revealed substantial performance gaps between state-of-the-art agents on our benchmark compared to atomic tasks, demonstrating the importance of contextual dependencies and multi-interaction coordination in realistic scenarios. This work establishes a foundation for more rigorous agent evaluation and provides valuable insights into the challenges agents face when deployed in complex, real-world productivity environments.

## 8 Acknowledgements

The authors would like to thank Robert Sim for his contributions in enhancing the tooling and infrastructure that supported this work.

## References

- Anthropic (2025a)
Anthropic.
Claude 3.7 Sonnet and Claude Code.
Online, February 2025a.
URL [https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet).
5 min read.
- Anthropic (2025b)
Anthropic.
Introducing Claude 4: Claude Opus 4 and Claude Sonnet 4.
Online, May 2025b.
URL [https://www.anthropic.com/news/claude-4](https://www.anthropic.com/news/claude-4).
5 min read.
- Boisvert et al. (2024)
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault de Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin.
Workarena++: Towards compositional planning and reasoning-based common knowledge work tasks.
*Advances in Neural Information Processing Systems*, 37:5996–6051, 2024.
- Bonatti et al. (2024)
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al.
Windows agent arena: Evaluating multi-modal os agents at scale.
*arXiv preprint arXiv:2409.08264*, 2024.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Côté et al. (2018)
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al.
Textworld: A learning environment for text-based games.
In *Workshop on Computer Games*, pp.  41–75. Springer, 2018.
- Deng et al. (2023)
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su.
Mind2web: Towards a generalist agent for the web.
*Advances in Neural Information Processing Systems*, 36:28091–28114, 2023.
- Drouin et al. (2024)
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al.
Workarena: How capable are web agents at solving common knowledge work tasks?
*arXiv preprint arXiv:2403.07718*, 2024.
- Erdogan et al. (2025)
Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami.
Plan-and-act: Improving planning of agents for long-horizon tasks.
*arXiv preprint arXiv:2503.09572*, 2025.
- Fourney et al. (2024)
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, et al.
Magentic-one: A generalist multi-agent system for solving complex tasks.
*arXiv preprint arXiv:2411.04468*, 2024.
- Gandhi & Neubig (2025)
Apurva Gandhi and Graham Neubig.
Go-browse: Training web agents with structured exploration.
*arXiv preprint arXiv:2506.03533*, 2025.
- Hu et al. (2024)
Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo.
Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model.
*arXiv preprint arXiv:2408.09559*, 2024.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Jaech et al. (2024)
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.
Openai o1 system card.
*arXiv preprint arXiv:2412.16720*, 2024.
- Li et al. (2024)
Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto.
Autobencher: Towards declarative benchmark construction.
*arXiv preprint arXiv:2407.08351*, 2024.
- Lin et al. (2024)
Haohan Lin, Zhiqing Sun, Sean Welleck, and Yiming Yang.
Lean-star: Learning to interleave thinking and proving.
*arXiv preprint arXiv:2407.10040*, 2024.
- Lù et al. (2024)
Xing Han Lù, Zdeněk Kasner, and Siva Reddy.
Weblinx: Real-world website navigation with multi-turn dialogue.
*arXiv preprint arXiv:2402.05930*, 2024.
- Murty et al. (2024)
Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D Manning.
Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild.
*arXiv preprint arXiv:2410.02907*, 2024.
- OpenAI (2025)
OpenAI.
OpenAI o3 and o4‑mini System Card.
System card, OpenAI, San Francisco, CA, April 2025.
URL [https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf](https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf).
Accessed July 16, 2025.
- Ou et al. (2024)
Tianyue Ou, Frank F Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou.
Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale.
*Advances in Neural Information Processing Systems*, 37:91618–91652, 2024.
- Pahuja et al. (2025)
Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Awadallah.
Explorer: Scaling exploration-driven web trajectory synthesis for multimodal web agents.
*arXiv preprint arXiv:2502.11357*, 2025.
- Paranjape et al. (2023)
Jay N Paranjape, Shameema Sikder, Vishal M Patel, and S Swaroop Vedula.
Cross-dataset adaptation for instrument classification in cataract surgery videos.
In *International Conference on Medical Image Computing and Computer-Assisted Intervention*, pp.  739–748. Springer, 2023.
- Qian et al. (2023)
Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al.
Chatdev: Communicative agents for software development.
*arXiv preprint arXiv:2307.07924*, 2023.
- Qin et al. (2023a)
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al.
Webcpm: Interactive web search for chinese long-form question answering.
*arXiv preprint arXiv:2305.06849*, 2023a.
- Qin et al. (2023b)
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al.
Toolllm: Facilitating large language models to master 16000+ real-world apis.
*arXiv preprint arXiv:2307.16789*, 2023b.
- Schick et al. (2023)
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools.
*Advances in Neural Information Processing Systems*, 36:68539–68551, 2023.
- Shen et al. (2024)
Haiyang Shen, Yue Li, Desong Meng, Dongqi Cai, Sheng Qi, Li Zhang, Mengwei Xu, and Yun Ma.
Shortcutsbench: A large-scale real-world benchmark for api-based agents.
*arXiv preprint arXiv:2407.00132*, 2024.
- Shridhar et al. (2020)
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht.
Alfworld: Aligning text and embodied environments for interactive learning.
*arXiv preprint arXiv:2010.03768*, 2020.
- Sutela & Lindström (2024)
Mika Sutela and Nino Lindström.
A game theoretic approach to lowering incentives to violate speed limits in finland.
*arXiv preprint arXiv:2402.09556*, 2024.
- Trabucco et al. (2025)
Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov.
Insta: Towards internet-scale training for agents.
*arXiv preprint arXiv:2502.06776*, 2025.
- Wang et al. (2024a)
Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song.
Rethinking the bounds of llm reasoning: Are multi-agent discussions the key?
*arXiv preprint arXiv:2402.18272*, 2024a.
- Wang et al. (2024b)
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji.
Executable code actions elicit better llm agents.
In *Forty-first International Conference on Machine Learning*, 2024b.
- Wang et al. (2024c)
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al.
Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2024c.
- Wang et al. (2024d)
Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, and Jingbo Shang.
Officebench: Benchmarking language agents across multiple applications for office automation.
*arXiv preprint arXiv:2407.19056*, 2024d.
- Wu et al. (2025)
Junde Wu, Jiayuan Zhu, and Yuyuan Liu.
Agentic reasoning: Reasoning llms with tools for the deep research.
*arXiv preprint arXiv:2502.04644*, 2025.
- Xie et al. (2025)
Jingxu Xie, Dylan Xu, Xuandong Zhao, and Dawn Song.
Agentsynth: Scalable task generation for generalist computer-use agents.
*arXiv preprint arXiv:2506.14205*, 2025.
- Xu et al. (2024a)
Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al.
Theagentcompany: benchmarking llm agents on consequential real world tasks.
*arXiv preprint arXiv:2412.14161*, 2024a.
- Xu et al. (2024b)
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu.
Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials.
*arXiv preprint arXiv:2412.09605*, 2024b.
- Yang et al. (2024)
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press.
Swe-agent: Agent-computer interfaces enable automated software engineering.
*Advances in Neural Information Processing Systems*, 37:50528–50652, 2024.
- Yao et al. (2024)
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.
$\tau$-bench: A benchmark for tool-agent-user interaction in real-world domains.
*arXiv preprint arXiv:2406.12045*, 2024.
- Zheng et al. (2025)
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu.
Deepresearcher: Scaling deep research via reinforcement learning in real-world environments.
*arXiv preprint arXiv:2504.03160*, 2025.
- Zhou et al. (2023)
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al.
Webarena: A realistic web environment for building autonomous agents.
*arXiv preprint arXiv:2307.13854*, 2023.
- Zhuang et al. (2023)
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang.
Toolqa: A dataset for llm question answering with external tools.
*Advances in Neural Information Processing Systems*, 36:50117–50143, 2023.

## Appendix A Criteria of Verifier Agent

We provide the criteria used by the verifier agent in HomerAgents+ to ensure the quality and realism of the generated dialogues. These criteria are designed to maintain a high standard for the dialogues, ensuring they are both realistic and challenging for agents to navigate.

## Appendix B Prompts for Agents

In this section, we separately provide the illustrations of the prompts used in the HomerAgents+ and HomerAgents-Neo.

### B.1 Prompts for HomerAgents+

### B.2 Prompts for HomerAgents-Neo

#### B.2.1 Task Generator Prompt

#### B.2.2 Orchestrator Prompt

#### B.2.3 Chat Generator Prompt

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: See weaknesses.
1. Ablation studies for the OdysseyBench-Neo construction process are lacking.
2. The case study is underdeveloped. More insightful guidelines are needed to steer future research toward bridging the performance gap between humans and LLMs on long-horizon office tasks.
soundness: 3
strengths: 1. The benchmark construction approach is comprehensive: it not only derives tasks from existing task intents (via HOMERAGENTS+ transforming atomic tasks from OfficeBench into context-rich scenarios) but also generates long-horizon tasks entirely from scratch (via HOMERAGENTS-NEO). Notably, both subsets (OdysseyBench+ and OdysseyBench-Neo) are accompanied by multi-day dialogue histories, which better simulate real-world user-agent interactions.
2. Testing human performance is well done. Human annotators achieved an overall accuracy of 91.5% on OdysseyBench, which effectively validates that the benchmark is well-formed, solvable, and accurate.
3. Experiments on context management within OdysseyBench provide valuable insights for agent design. For instance, findings on the superiority of summarized context (over raw dialogue data) in enhancing reasoning efficiency, and the performance decline of agents when tasks involve more applications, offer valuable guidance for designing agents.
summary: This paper introduces OdysseyBench, a comprehensive benchmark for evaluating LLM agents on long-horizon workflows across 5 office applications (Word, Excel, PDF, Email, Calendar). Compared with previous works, which overreliance on self-contained atomic tasks, this paper proposes HOMERAGENTS, a multi-agent framework that automates the generation of long-horizon workflow benchmarks. OdysseyBench includes 300 real-use-case-derived tasks (OdysseyBench+) and 302 newly synthesized complex tasks (OdysseyBench-Neo). Experimental results show humans achieve ~91.5% accuracy on OdysseyBench, while top LLMs (e.g., o3, GPT-5) only reach ~55%, with agent performance declining as tasks involve more applications.
weaknesses: 1. Ablation studies for the OdysseyBench-Neo construction process are lacking, and key concepts (e.g., "surfers") lack concrete case illustrations. The paper proposes HOMERAGENTS-NEO to generate long-horizon tasks from scratch via four phases , but it does not validate the necessity of each component or compare its performance against simpler generation methods. Additionally, the role of "surfers" in collecting contextual information from office applications is described in abstract terms, and it would be better to provide some specific cases.
2. The case study is underdeveloped. While the study identifies four agent failure modes (missing files, incorrect tool calls, etc.), it does not analyze whether these issues can be addressed by optimizing agent frameworks or by injecting office workflow knowledge into LLMs. Without comparing these potential solutions, the paper misses an opportunity to guide future research on bridging the performance gap between humans and LLMs on long-horizon office tasks.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: Please solve the weakness above
soundness: 2
strengths: 1.  Addresses a Good question: The paper identifies a good limitation of current agent benchmarks: their focus on "atomic" tasks. It rightly argues for the need to evaluate agents on long-horizon workflows that require reasoning over accumulated
2.  Good Evaluation: The paper provides an evaluation of numerous models. The inclusion of both long-context and various RAG strategies (raw vs. summary, session-level vs. chunk-level) provides a valuable analysis of how agents cope with long context.
4.  Human-in-the-Loop Validation: The use of human annotators to verify task solvability (Table 3) is a strong point, ensuring that the OdysseyBench-Neo tasks are coherent and achievable.
summary: This paper introduces OdysseyBench, a new benchmark designed to evaluate the capabilities of LLM agents on complex, long-horizon tasks within office application environments (Word, Excel, PDF, Email, Calendar). The authors argue that existing benchmarks (e.g., OfficeBench, AgentBench) primarily focus on "atomic", self-contained tasks, failing to test an agent's ability to maintain context, reason over long interaction histories, and handle dependencies across multiple steps.

The paper presents two main contributions:
1.  OdysseyBench: A benchmark with two splits: OdysseyBench+ (300 tasks) which transforms atomic tasks from the existing OfficeBench into long-horizon workflows by synthetically generating dialogue histories, and OdysseyBench-Neo (302 tasks) which are entirely new, complex tasks generated from scratch.
2.  HomerAgents: A multi-agent framework created to automate the generation of these long-horizon benchmarks. HomerAgents+ uses a generator-verifier setup for refinement, while HomerAgents-Neo uses a multi-phase (Planning, Exploration, Task Generation, Dialogue Generation) system to create new tasks.

The authors evaluate several proprietary and open-weight models on OdysseyBench, using both full long-context and various RAG settings. Their results show that these long-horizon tasks are significantly more challenging for current agents than atomic tasks, and they analyze the effectiveness of different context retrieval strategies.
weaknesses: 1.  Overclaiming and Imprecise Language: The paper's repeated claim that `OdysseyBench+` is derived from "real-world use cases" is an overstatement. It is a synthetic transformation of another benchmark (OfficeBench). This linguistic imprecision ("real-world" vs. "realistic simulation") undermines the paper's claims about the benchmark's grounding.
2.  Misplaced Focus on Method over Benchmark: This is a benchmark paper, but the majority of the methodology (Section 3) is devoted to the HomerAgents generation framework. The analysis of the OdysseyBench benchmark itself (Section 3.2.2) is brief. The paper would be much stronger if it relegated the complex details of HomerAgents to an appendix and used the main body to provide a much deeper analysis of the benchmark's properties: e.g., a taxonomy of the long-horizon reasoning types required (temporal, causal, informational dependency), a deeper dive into the failure modes (Section 6 is a good start but brief), and a qualitative comparison of the task structures in OdysseyBench+ vs. OdysseyBench-Neo.
3.  Experimental Insights are Not Novel: The main conclusions from the experiments (Section 5) are not surprising: multi-app tasks are harder, full context beats RAG, and summarization helps RAG. These are all known findings. While they validate the benchmark, they don't offer new insights into agent behavior that one might hope for from a new benchmark paper.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: * Some references are not valid (e.g., line 148-149).
* How to interpret Table 1? Is the minimal difference between 1-apps and 3-apps in human performance implying that the collected tasks are of low discriminability?
* What is the relevance between the proposed benchmark and existing ones? In other word, it is interesting to see if the conclusions from the proposed benchmark are consistent/contrasted with previous ones (e.g., officebench).
* The evaluation (long-context and rag-evaluation) is missing details. For examples, what is the effect embedding/summarization models (performance themselves) on the entire benchmark? What is summarization technique/prompt?
* If any agent-framework is involved in the evaluation?
* What is the representation of environmental feedback and how this might get improved for performance of agents?
soundness: 3
strengths: * The benchmarks contain both the manually-curated and synthesized queries, which is good for the evaluation of LLMs nowadays.
* The paper is generally well-written and easy to follow.
summary: This paper proposes a new benchmark (OdysseyBench) for evaluation of LLM-driven agents on complex, long-horizon workflows that involve multiple office applications. In addition, it proposes a multi-agent framework named HomerAgents for construction of benchmarks generally.
weaknesses: * The benchmark is built upon/inspired from the existing OfficeBench, but there exists no detailed analysis on: 1) the differences between the previous work and the current study; 2) the limitations of the previous benchmark and how they are handled in the current study; 3) the environment (infra) scaling issue long horizon, complex office task evaluation.
* There exists no discussions on the performance/accuracy of the generator and verifier itself for synthesis/extension of office tasks, similar for the homeagents-neo pipeline (planning/generation).
* The evaluation metrics are limited where mostly the pass rate/success rate is reported and such coarse-grained metric cannot indicate the future improvement for LLMs.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. Figure 1 presents a comparison; however, if the focus is on the shortcomings of complex tasks, I suggest differentiating between complex and long-horizon (memory-dependent) tasks within these benchmarks, rather than contrasting them with simple tasks.
2. OdysseyBench+ and OdysseyBench-Neo together form the entire dataset, but readers may wonder why the authors chose to distinguish between these two splits. My understanding is that they were constructed using different methods; however, the paper does not clearly explain why two separate data construction approaches were necessary.
3. Section 3.1 has already explained the overall process clearly, and the additional Algorithm 1 and Algorithm 2 do not help clarify the information. Instead, they make the paper look more like a method-oriented work. Overall, the writing of this paper has considerable room for improvement.
4. It may be necessary to consider more recent works. Given the applications chosen by the authors, this benchmark is more closely related to web-based environments. Therefore, at a minimum, the evaluation should include three types of web agents: web agent frameworks, RL-based agents, and the thinking models already covered in the paper. 
5. For baselines, for example, [1] WebAgent-R1 and [2] WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Exploration should be considered as representative web-agent frameworks.
As for thinking models, it would be valuable to include recent ones such as doubao-1.5-thinking-pro, Claude-3.7-Sonnet-20250219, and Computer-Use-Preview for a more comprehensive comparison.
6. line 227: The writing does not conform to standard algorithmic conventions.
7. My understanding is that long-horizon tasks are closer to a combination of long-memory and informative search. Is that correct?
soundness: 2
strengths: 1. This paper proposes a new benchmark, OdysseyBench, to evaluate the comprehensive capabilities of LLM agents in long-term office tasks, and covers a variety of real-world office applications: Word, Excel, PDF, Email, and Calendar. 
2. The authors also propose HOMERAGENTS, a multi-agent automatic generation framework for scalably building long-term task benchmarks. Through environment exploration, the task generation and dialogue synthesis stages are automated.
3. Long-Horizon Complex tasks assessment has some prospects.
summary: This paper introduces OdysseyBench, a comprehensive benchmark designed to evaluate large language model (LLM) agents on long-horizon, multi-application workflows in realistic office scenarios such as Word, Excel, PDF, Email, and Calendar. It consists of two complementary splits—OdysseyBench+ (300 real-world tasks) and OdysseyBench-Neo (302 synthesized complex tasks). To enable scalable benchmark construction, the authors propose HOMERAGENTS, a multi-agent framework that automates environment exploration, task generation, and dialogue synthesis.
weaknesses: 1. The HOMERAGENTS framework is not particularly novel — similar approaches that combine information collection and data generation have already been employed in several prior works, such as AppAgent and others.
2. Long-horizon complex tasks are indeed underexplored in previous datasets and benchmarks. However, as a benchmark, it should be more comprehensive; the current limited amount of data is insufficient to support a thorough evaluation of agents’ capabilities in complex task scenarios.
3. The connection between HOMERAGENTS+ and HOMERAGENTS-NEO is not clearly explained, making the overall narrative behind the construction of these two splits somewhat confusing.
4. The experimental design is relatively weak, as many appropriate baselines were not included in the evaluation. Despite experimental analysis, no interesting conclusions were drawn.

More weaknesses can be found in the questions section for reference.