Title: ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Agents

Proactive Agent

Human-centric AI

Personal Assistant
Abstract: Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. 
While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts surrounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at https://github.com/openaiotlab/ContextAgent.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ContextAgent, a context-aware proactive agent designed to operate under open-world sensory conditions. The authors propose two new benchmark, contextagentbenchmark and lite version, which combines LLM-assisted generation with human curation. The benchmark includes rich multimodal information such as egocentric video, audio, reasoning traces, and user personas. Experimental results demonstrate that ContextAgent achieves SOTA performance compared to existing LLMs on this benchmark.\n\nHowever, the paper suffers from unclear descriptions in the task definition, task construction, and experimental comparisons, which significantly affects readability and makes it difficult to fully follow the methodology.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed ContextAgentBench dataset is well-motivated and addresses a practical need for evaluating proactive LLM agents in open-world, multimodal settings.\n2. The authors conduct extensive experiments across 10 different LLMs, including both proprietary and open-source models.\n3. The paper includes clear and informative figures that effectively illustrate the system architecture, data structure, and experimental results. \n\nWeakness：\n1. In Section 3.1 Task Definition and Task Construction, many variables are introduced but inconsistently notated (e.g., $P_S$  vs. PS) and defined out of order across paragraphs. Key terms such as the trigger threshold TR are underexplained, and the overall formulation lacks a clear separation between inputs, outputs, and constants—making the method harder to follow and reducing its interpretability.\n2. Although the authors conduct extensive experiments, I find the experimental setup somewhat problematic. Specifically, ContextAgent is trained via supervised fine-tuning, yet it is compared against several training-free baselines (e.g., ICL variants), where the control over input information appears loosely related to the actual task setting (sft). Additionally, the vanilla SFT baseline does not include persona information as input, which results in an unfair comparison due to missing critical context.\n3. The proposed agent does not demonstrate clear structural or architectural novelty. It primarily builds on existing LLMs and applies domain-specific supervised fine-tuning.\n4. While the paper mentions that ContextAgent is fine-tuned using supervised learning, it does not provide sufficient details about the training data $D_SFT$.\n5. The paper does not provide concrete examples from the dataset, nor does it release the dataset itself. As a result, it is unclear what the persona annotations look like, or how representative they are.\n\nI may have missed certain details, and I am open to adjusting my score based on the authors’ response and clarifications.\n\nQUESTIONS:\n1. In line 133, what exactly is the definition of $TR$?\n2. Why does the vanilla SFT baseline not include persona information during training? This seems to create an imbalance in the comparison against ContextAgent.\n3. What is the composition of the D_SFT training data? Please provide more details on the dataset size, construction. etc.\n4. What does the persona data look like in practice? Can you provide concrete examples to clarify the structure and content of persona annotations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

comments: SUMMARY:
This paper proposed a benchmark for evaluating context-aware proactive LLM agents and an author-designed agent as baseline model. The baseline agent is a context-aware proactive agent that fuses rich sensory data from video, audio with a user’s historical persona to infer intentions and decide when help is needed, then autonomously invokes the right tools to assist unobtrusively. The proposed ContextAgentBench contain nine everyday scenarios and twenty tools. On this benchmark, ContextAgent surpasses prior baselines.

STRENGTHS AND WEAKNESSES:
S1. The authors proposed a personalized agentic framework, which match well with the community's interests.    

S2. They tested the ContextAgent with open-sourced LLMs (including small LLMs), which is useful for re-implementation.     

S3. The main idea is simple and clear and also contain OOD experiment.           

W1: Comparison might be unfair in Table 1 because some of other ideas are zero-shot method. Only Vanilla SFT and your proposed idea utilize the training data.      

W2: Do not contain missing sensor situation. e.g., the input data of some sensor is not aviliable, what might be the context output?    

W3: For 5.4 Out-of-Domain Evaluation, it would be good to compare your method with the latest GPT/Claude series like o3.

QUESTIONS:
Q1. Why don't you use VLM as the main agent directly?     

Q2. About W2. Discuss the situation when some external tools are not aviliable, can your agent still work?        

Q3: ContextAgent is a multi-tool agent in this stage, why don't you consider code agent for this problem.

LIMITATIONS:
They discuss limitations in the supp.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a context-aware proactive LLM agent, ContextAgent, along with a benchmark, ContextAgentBench, to evaluate context-aware proactive LLM agents. While prior work have created proactive LLM agents leverage observations from computer interfaces or repeatedly reask questions. ContextAgent leverages sensory information (from wearable devices) and makes uses of user personas to proactively call tools and provide users information. ContextAgent is evaluated using ContextAgentBench by the accuracy of proactive predictions and the tools called to provide proactive feedback.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality:** Extensive experiments and ablations are conducted to demonstrate the usefulness of ContextAgent making use of sensory information and persona data.\n\n**Clarity:** The paper is fairly easy to understand but there are some improvements that can be made in clarity in the Task Definition section (some terms like TR are mentioned but not defined until later sections). It is also not obvious what a proactive score of 2-4 means.\n\n**Significance:** This is an important problem to study and the paper provides both a solution and benchmark for evaluation.\n\n**Originality:** This paper proposes the first benchmark for context-aware proactive LLM agents.\n\nQUESTIONS:\n- What do proactive task with a score of 2, 3, and 4 look like?\n  - How were these threshold of proactivity decided for the tasks? In Figure 8 some tasks involve a user working on a computer and the proactiveness score is 1 but I could imagine the user being stuck with an error and requiring some proactive assistance. Similarly for the case where the user is walking by some dried out plants which could prompt looking for water fountains nearby to water the plants.\n- Is this benchmark already saturated given that ContextAgent achieves nearly 90~ accuracy on proactive predictions? What are failure modes and insights in closing the gap?\n- Similar question for tool calling except that it seems there is room for improvement here.\n- For ProactiveBench, why is it necessary to leverage contextual information from wearable devices on computer tasks? For a human using a computer, wouldn't a wearable device need to focus on the task at hand?\n  - If this is the case, would it be possible to evaluate ContextAgent on ProactiveBench? Essentially the sensory information is distilled through the desktop UI and the persona information is distilled into the task instruction\n\nLIMITATIONS:\nThere is a discussion to be had on being context-aware for dangerous situations where the tool calls can contact emergency services for scenarios like (1) witnessing crimes or (2) health-related problems (e.g. fall detection on Apple watches is mentioned in the introduction).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ContextAgent for proactive LLM-based agents that leverage rich sensory and persona contexts to provide tool-augmented services without explicit user instructions. By integrating data from wearable devices, ContextAgent autonomously infers user intentions and determines when and how to assist, using external tools accordingly. This paper further propose a benchmark named ContextAgentBench with 1,000 samples across nine daily scenarios to evaluate such agents. Experiments show that ContextAgent significantly outperforms baselines in proactive prediction and tool-calling accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper presents a compelling approach by leveraging sensory data from wearable devices to enable context-aware, proactive LLM agents. This direction is highly relevant for advancing AI assistants toward more natural, human-centric interactions in real-world environments.\n2. The introduction of ContextAgentBench, a comprehensive benchmark designed specifically for evaluating proactive LLM agents in open-world settings, is a valuable contribution. It provides a structured and reproducible way to assess future systems and can foster further research in proactive agent design.\n\n**Weaknesses**\n1. While the proposed system is practically useful, the core technical components—such as using VLMs for visual understanding and in-context learning for context extraction—largely rely on existing methodologies. The innovation primarily lies in system integration rather than algorithmic advances, which limits the novelty from a technical standpoint.\n2. The paper would benefit from improved clarity and organization. Certain sections are dense and difficult to follow, which may hinder reader comprehension. Enhancing the exposition, especially around key concepts and system design choices, would strengthen the overall readability and impact of the work.\n\nQUESTIONS:\nSee the weaknesses part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}