ContextAgent: Context-Aware Proactive LLM Agents with Open-world Sensory Perceptions
Authors: Bufang Yang
Lilin Xu
Liekang Zeng
Kaiwei Liu
Siyang Jiang
Wenrui Lu
Hongkai Chen
Xiaofan Jiang
Guoliang Xing
Zhenyu Yan
Venue: NeurIPS 2025 poster ()
Keywords: LLM Agents
Proactive Agent
Human-centric AI
Personal Assistant
Primary area: applications

Abstract: Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. 
While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts surrounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at https://github.com/openaiotlab/ContextAgent.

ContextAgents Context-Aware Proactive LLM Agents with Open-Worle Sensory Perceptions
Bufang Yang Lilin Xu
Liekang Zeng' Kaiwei Liu'
Siyang Jiang' Wenrui Lu'
Hongkai Chen' Xiaofan Jiang Guoliang Xing' Zhenyu Yan
TTe Chinese University of Hong Kong CColumbia University
(bfyang, l 1,022,
s0002,,syJi syjiang, wrlu, hkchen, www.gh.gl..iiiggg........ g zyyan]eie cuhk. 1x23310coll umbiaa edu, jiangsee. columbia. edu
Abstract
Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively observations from enclosed environ- ments (e.g., desktop Uls) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited fune- tionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts sur- rounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimenan dimensiaianii contexts from massive sensory perceptions on wearables (c.g,, video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services, When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBence aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments ContextAgentBencc show that ContextAgent out- performs baselines by achieving up to 8.5% 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at http:///hhub.ccooopceaaaaalll/oa
the first benchmark for evaluating context-
Introduction
Large Language Model (LLM) agents are revolutionizing our daily life [16], assisting users with complex tasks such automated web navigation [12, 56, 9], software engineering [46, 36), and healthcare services [4, 43, 25]. While LLM agents are receiving growing attention and adoption, most of them still function in reactive paradigm: They initiate tasks only upon explicit user instructions and yet lack the autonomy to perceive environments and offer proactive support for users.
To further reduce reliance on instructions and alleviate human cognitive workload, proactive agents emerge, which are capable of initiating tasks without explicit user queries [55, 24, 23, 52, 50]. For example, research on proactive agents have explored coding assistance [55, 24], conversation participation [23, 44], re-asking strategies to reduce ambiguity in user instructions [52], and multi- agent cooperation scenarios [50, 39]. However, their limited ability in open-world perceptions and restricted functionality for proactive service hinders their potential as personal companions.
Equal Contribution. C *Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Environmental Perception. When explicit user instructions are absent, environment perception is crucial for proactive LLM agents. Recent studies [24, 55] proposed proactive agents for programming assistance, while they require access to specific inputs such as computer screenshots or keyboard inputs. We argue that an ideal proactive agent should able to perceive open- open-world environments in the user's daily life, utilizing wearable devices such as smart glasses and earphones. By sharing the same perception as the user, the agent can understand the user's s intention and provide services automatically, Besides, the hands-free nature of these ubiquitous wearable devices aligns well with the mission proactive agents, freeing both the user's hands and mind from additional workload.
Functionality for Proactive Services. Current personal assistants can de- liver proactive notifications via wear- ables, yet remain limited by static, rule-hased pipelines (e.g., alerts when rapid falling is detected [3]). Re- cent studies [24, 55, 44, 52] propose to build proactive agents with LLMs. However, these agents only provide di-
PV Earphone
CO
rS Exycoement
Conterts The context shows the user near bus slop and may reeed real-time bus arrivals ito cal bus soned, lo creaker). contimn isur's location GPS), tho alteraatvve transport call Ubeer....
Smart Glasses
#n Evrioement
(c
datedime).
Smartwatch
ContextAgont
Seaar Glasne Screet
Proactive
11:05 of University Stop
Smartphone
Assistance
Massive Sensory Contexts Open-world
nead No. bus:
Uber for you? { (4min $10)
rect answers during user interactions, without leveraging external tools. and remain limited to enclosed environ- ments (e.g., desktop and keyboard in- puts [24, 55]). Therefore, there re- mains research gap in developing context-aware proactive LLM agent that can exploit extensive sensory contexts to comprehensively understand user intentions, predict the necessity of proactive services, and automatically integrate external tools to deliver unobtrusive services as a personal companion.
ContextAgent is proactive Al assistant free of user explicit instructions. ContextAgent can continuously perceive environmental contexts (e.g.. image and audio) to detect the necessity of proactive services, and provide tool- augmented assistance based on LLM reasoning.
Figure 1:
Manual Instruction Required
Free Manual Instructions
Automation Tasks
Automatic Trigger Sensory Context
Explicit User Instructions
Programming
Tool-augmented Proactive Services
Remind me.... on Augusi 17 Write a code to process ECG...
LLMs
Senser Cocrdiation
Reactive LLM Agents
Earphones...
Sensory Context from Wearables
Enclosed Enviroment
Rule-based, Static Workflow
Lack of Tool-Calling
Intortion Understending
Highh Warning! Arrhythmia Detect.
O Prouctive Prodiction
Fall Detection
Pointer
Keyboard
Desktop
Rule-based
Proactive LLM Agents
Proactive Assistants
Trigger by Operation
Context-aware Proactive Agent (Ours)
Comparison with existing works. Reactive LLM agents require explicit user instructions to initiate tasks. Prior proactive LLM agents focus on perceiving enclosed environments (e.g., desktop Uls) and may still require user operations (e.g., keyboard inputs) alongside direct LLM inference. In contrast, ContextAgent requires no manual instructions, harnesses massive sensory contexts from the open world, and employs LLM reasoning for aoo-anu proactive services.
Figure 2:
In this paper, introduce ContextAgent, the first context-aware proactive LLM agent that harnesses extensive sensory contexts for enhanced proactive services. ContextAgent first employs a proactive- oriented context extraction approach derive both sensory and persona contexts from massive sensory perceptions such as egocentric videos and audio. We then develop a context-aware reasoner that integrates both sensory and persona contexts for reasoning, predicts the necessity of proactive services, and calls external tools when necessary. This reasoner is fine-tuned with reasoning traces distilled from advanced reasoning LLMs, enabling it to think before acting. Fig. shows an example scenario where the user arrives at bus stop just after the bus has left. ContextAgent can leverage this sensory context to proactively deliver useful services, such as real-time bus schedules, and determine whether alternative transportation is needed. By harnessing sensor perceptions from hands-free, egocentric wearables (e.g., smart glasses and earphones), along with I.I. reasoning, ContextAgent moves closer toward more ubiquitous and proactive AI assistant.

---

To better examine ContextAgent, we further introduce new benchmark, ContextAgemtBench for the comprehensive evaluation of context-aware proactive LLM agents. ContextAgemtBench contains 1 1,000 samples covering nine daily life scenarios, such as working and chitchat, and includes twenty external tools. We conduct comprehensive evaluations, comparing ContextAgent to six baselines and testing on 13 LLMs. Results show that ContextAgent achieves the state-of-the-at performance in proactive predictions and tool calling. We summarize the contributions of this work as follows.
We raise the research problem of context-aware proactive agents that harness extensive sensory contexts surrounding humans to enhance the proactivity of the LLM agents and deliver tool- augmented proactive services, moving toward the vision of proactive personal assistants.
We propose ContextAgent, the first framework for context-aware proactive LLM agents. ContextA- gent employs novel context extraction method that derives sensory and persona contexts from massive sensor perceptions, Additionally, we design a context-aware reasoner with think before action capabilities that can integrate both sensory and persona contexts for reasoning, predict the necessity of proactive services, and call external tools when necessary to assist the user.
We introduce ContextAgentBench, the first benchmark for the comprehensive evaluation of context- aware proactive LLM agents. Extensive evaluation on ContextAgentBench shows that ContextA- gent outperforms six baselines by achieving up to 8.5% higher accuracy for proactive predictions. 7. 7.0% higher Fl Fl-score for tool calling, and 6.0% higher accuracy for tool arguments.
2 Related Works
Reactive LLM-based Agents. Recent studies have proposed various LLM agents to perform complex
tasks, such as automated web navigation [12, 56], software engineering [46], personal assistant [44, 42], and household robotics [6]. Additionally, prior research has primarily focused on enhancing the
core capabilities of LLM agents, including task planning [47], function calling [29, 21, 32], experience
reflection [54, 33], generalization abilities [38, 27J, and multi-agent collaboration [34, 22, 53]. Other studies have explored the LLM agents in mobile systems, such as autonomous UI operations on smartphones [37, 18, 49] and embedded programming [31, 13, 45J. However, although numerous frameworks and optimizations have been proposed, prior research has primarily focused on reactive LLM agents that require explicit textual instructions from users and cannot utilize the extensive contextual information from sensor perceptions on wearable devices to enable proactive assistance.
Proactive agents aim to autonomously initiate services based environ- mental observations, without requiring explicit user instructions, evolving from early rule-based or periodic triggers [3] to recently proposed LLM-based approaches [55,24]. Ask-before-plan [52] employs re-asking strategies to proactively reduce ambiguity in user's instructions and enhance subsequent planning, although still requires an initial user query. ProAgent [50] is proactive cooperation framework among multiple robot agents, while its proactive design primarily focuses on predicting teammates' actions in multi-agent systems rather than the user's intention. Recent studies, such as Proactive Agent [24] and CodingGenie [55], also propose proactive LLM agents that monitor the user interface environment on computer systems and proactively assist with tasks such as coding and writing. However, existing work either leverages observations on computer interfaces or employs a re-asking strategy to gather more information, without utilizing the rich sensory contexts to proactively initiate services. Moreover, prior works primarily use L.MMs for direct inference rather than integrating external tools, resulting in limited proactive service functionality.
Proactive LLM Agents.
LLM Agent Benchmark. A diverse and large-scale benchmark is essential for the comprehensive evaluation of LLM agents. However, existing benchmarks primarily focus on reactive LLM agents [46, 29, 26, 11, 211, where the agent needs to take user instructions as inputs and perform task planning and tool calling. Although a recent work [24] proposes ProactiveBench, it is limited to an enclosed environment, i.e., desktop UI, and does not leverage the rich contextual information from multi- modal sensors on wearable devices. Additionally, Proactivelencch relies on direct LLM inference for responses, instead of calling diverse external tools. Therefore, a research gap remains in developing comprehensive benchmark for evaluating proactive I.IM agents that incorporate the rich contextual information from wearable devices for proactive reasoning with tool-calling capabilities.
3

---

3 Context-aware Proactive Agent Task
3.1 Task Definition
In contrast with existing reactive LLM agents and proactive agents that rely solely observations from desktop interfaces or direct inference, we formalize context-aware proactive LLM agents as: (T. Ps, Te, R) = A(S, P), where A is the LLM agent, which integrates the sensory perceptions S and persona context as input. Here S contains sensor perceptions from multi-modal wearables such as smart glasses and carphones, including egocentrie video Sy, audio SA, and smartphone notification N. We denote the sensory context C' as the implicit cues within the raw sensory perceptions that help determine the need for proactive services. We also formalize that the agent should consider user personas P for proactive reasoning, including a person's 's identity, preferences, and historical behaviors. Using these contexts, the agent generates Ps. Te. R), where T denotes the explicit thought traces. Ps denotes the proactive score, which triggers proactive services when Ps 2 A. Here, denotes the threshold for initiating proactive services and is a user-adjustable parameter reflecting the user's sensitivity to such services. Te is the planned tool chains that L.LM agents should call in sequence, where Te (f4, d t; E T, with & as each tool to be called and a; as the corresponding arguments. T is the tool set that the agent can use. is the agent's 's final response, summarizing the sensory context, persona context, reasoning traces, and tool results. Note that proactive assistance is only initiated when Ps 2 0, otherwise the agent does not disturb the user.
3.2 Task Construction
Recognizing the shortcomings of existing LLM agent benchmarks, we present ContextAgentBench, the first benchmark designed to evaluate context-aware proactive LLM agents.
Design Choices. Our dataset includes the following key features: 1) Sensory Context. Our dataset contains sensory context obtained from wearables (e.g., smart glasses and earphones), which capture shared perceptions of the user ubiquitously. This hands-free captured sensory context more suitable for proactive agents as it can reduce the user's physical and cognitive workload, aligning with the mission of proactive agents. 2). Persona Context. We incorporate diverse personas to support more comprehensive and personalized scenarios for proactive services. 3) Proactive Assistance with Tool Calling. The dataset targets tool-using LLM agents that map the contexts to proactive assistance by utilizing multiple external tools to generate more informative responses, rather than direct inference.
Formulation and Exemplar Design. Each sample in our dataset contains seven parts: (S,C,P,T, T,Ps, Tc, R). Next, we introduce the design of initial exemplars.
Multi-dimensional Context Information. Annotators first write textual descriptions of their egocentric perceptions, including what they see, hear, and any mobile device notifications, for both proactive and non- non- proactive scenarios that they encounter in daily life. This sensory perception can be captured from an egocentric perspective using various wearable devices. The context information contains the visual context Cy. the acoustic context C,, and the notifications on the smartphone N. Annotators also summarize them into contextual information C, providing a comprehensive description of the user's 's current conditions. Annotators write the user personas P for the sample if necessary. The persona can include any information about a person's 's preferences or identity.
Proactive Score with Planned Tool Chains. Next, annotators are instructed to analyze the current context and assign a proactive score Ps. We define Ps on scale from 1 to 5, where means that no proactivity is required and 5 means a high level of proactivity. Annotators also receive a tool set T that includes the usable tools, tool names, tool descriptions, arguments, and formats predefined by the developers. Details are in the Appendix D. For samples identified requiring proactivity, request annotators to further label the planned tool chains Tc, specifying the external tools that agents should use. If Ps- 1 or 2, both Te and R are None, as there is no need for proactivity.
We instruct the annotators create samples spanning nine everyday scenarios, ranging from work to chitchat. We ask annotators to document their thought processes, including their analysis of the current context, their rationale for assigning the proactive score, and the planned tool chains. Each annotator also cross-reviewed the samples produced by others, evaluating both the format and plausibility to avoid overproactivity and ensure the correctness of annotations. Through this process, acquire 200 human-created exemplars to serve as the seed dataset.

---

Automated Diversification Pipeline. Relying solely on manual efforts to scale the dataset presents challenges, as scenarios and contextual information created by humans may lack diversity and gener- alizability. Moreover, human fatigue during annotation can introduce bias, potentially compromising the dataset's quality. Therefore, we develop an automated diversification pipeline use L.LMs for data generation, producing large-scale dataset with diverse samples.
Information Source. We first prepare several resources to help LLMs generate synthetic data, including the tool set (Appendix D), an extensive persona pool, and the initial exemplars. The personas in our pool are sourced from [14], which includes one billion individual identities and preferences.
Generation with Verification. Next, we prompt LLMs to generate diverse samples by utilizing the initial exemplars, tool set, and persona pool for reference. We employ two strategies during generation: acenario-aware and proactive score-aware. In the first strategy, we group the seed dataset by scenarios and instruct LLMs generate samples based
specific scenarios within the nine categories. In the second strategy, LLMs are prompted to generate samples based on specific proactive score. Details in the Appendix B. After generation, annotators first evaluate the context and annotations for rationality, Next, we execute a script to verify the correctness of the data format and tool arguments. We perform several iterations of the above process obtain ContextAgentBench.
Outdoors
Cooking
Others
Paa Serditi Scort
Proactive
Munn di Tools
Proactive
120
100
Health
80
Travel
Number
Shopping
Non preactive
40
Chitchat
Work
20
50 100
100 200 300
100200 300-400
(a) Scenarios.
(b) Proactive score.
Tool number.
(d) Tool distribution.
Figure 3: Statistics of ContextAgentBende scenarios, proactive scores, and the number and types of tools. In subfigures (c)-(c), the x-axis shows the number of samples, whereas in (d) it denotes the tool index.
including the sample distribution across different
Consideration of Sensor Data. We also collect sensor data from copyright-free internet platforms [1] to pair with the textual contextual information ContextAgentBenech Specifically, we first randomly select samples from ContextAgentBentt and scrape the videos from Pexels [1] based on the textual descriptions of the visual context information. Note that we exclusively collect videos captured from egocentric perspective. Additionally, for samples with audio conversations, self-collect both video and audio to align with the textual context information. Finally, we obtain the ContextAgentBench-Liie consisting of 300 human-verified samples with raw sensor data.
Dataset Statistics. Fig. 3 shows the statistics our dataset. We collected 1,000 samples for ContextAgentBench and 300 samples for ContextAgenttennh--Litee Our dataset covers 9 daily life scenarios and includes 20 tool types, with each sample potentially involving the use of up to five tools. We provide more details the dataset and tool definitions in the Appendix C.
ContextAgent Framework
This section presents the framework of ContextAgent, introducing how it utilizes the massive sensory contexts for tool-augmented proactive LLM agent services. Fig. 4 shows the overview of ContextAgent. First, ContextAgent extracts proactive-oriented contexts from multi-modal sensory perceptions. Next, ContextAgent integrates these contexts for tool-augmented proactive services.
4.1 Proactive-oriented Context Extraction
Previous studies focus on extracting sensory contexts and use LLMs to summarize insights [28]. However, relying solely on these sensory contexts can lead to inferior proactive predictions. Therefore, ContextAgent employs a proactive-oriented context extraction method. ContextAgent, contexts comprise two types: sensory context and persona context. The sensory context includes insights for the user's surroundings and actions, which are crucial for inferring user intent. The persona context encompasses user personal information, including past behaviors, preferences, and identity,
5

---

Sensory Context Extraction
Though
Equewerts
Score
mo righ sed red-ttss
Signas
store
Conkol
Sensory
Context
uars curur bome
Disturb
4.0 Buck 200
Persona Context Extraction
ther robaa
Execution Intogratinn
Pensona
Context
Prachha some Ae
Paraption
Proactive-oriented
LUN
Contect
Context-aware Reasoner
Context-aware Reasoning
Proactive
sthack cerive like te bonk Uber for yerveru?
Context Perception
Context Extraction
Services
Figure 4: Overview of ContextAgent. ContextAgent extracts sensory context from massive sensor perceptions. Then it integrates both sensory and persona contexts into LLM reasoning, generating thought traces, proactive predictions, and calling external tools for proactive services when necessary.
which helps LLM agents to determine the need and urgency for proactive assistance. This subsection provides details on how ContextAgent extracts these contexts from extensive sensory perceptions.
Sensory Context. ContextAgent first employs Vision Language Models (VLMs) [19] to transform egocentric videos into visual contexts Cy. Although existing VLMs can generate video descrip- tions, they often produce overly simplistic descriptions that overlook crucial cues for understanding user intent or overly detailed, redundant insights, both of which can hinder the LLM agent's 'ss proactive predictions. Thus, instead of using zero-shot VLMs, ContextAgent employs in-context learning (ICL) to generate proactive-oriented visual contexts Cy. ContextAgent also employs the speech recognition model to generate audio contexts CA- Details and prompts for sensory context extraction are in the Appendix B. Finally, ContextAgent integrate these contexts into the final context information C == [vv, Cv. which includes visual contexts Cy, audio contexts CA. and textual information from smartphone notifications N, such calendar events and hotel reservations.
Persona Context. Since the need for assistance highly depends on the user's personal preferences, ContextAgent also integrates persona context into its reasoning. In this work, we use persona contexts within ContextAgentBentt for experiments. In practice, these contexts can be continuously updated by utilizing LLMs to extract insights from historical sensory data like daily conversations [44].
4.2 Context-aware Proactive Reasoning
While existing L.LM agents can handle complex tasks based on explicit user instructions [20, 56, 9], they face challenges when processing sensory contexts and correctly mapping them to the appropriate tools for proactive services. Next, we will introduce the context-awar reasoner in ContextAgent.
Context-aware Reasoner. ContextAgent employs context reasoner As to reason over the generated contexts and provide proactive services: Ps, Te) As(C, P). The context reasoner is an LLM that integrates both sensory context C and personas P as input to generate thought traces T, proactive scores Ps. and tool chains Te. We enable ContextAgent to perform think-before-ace reasoning by distilling traces from advanced LLMs (e.g., Claude 3.7le-33...0 5.7-Sonnt (2]) and constructing CoT- based [35] fine-tuning dataset. During inference, once Ps 2 9, ContextAgent will initiate the proactive services. Additionally, ContextAgent generates tool chains Te for enhanced proactive services. ContextAgent will execute the planned tools sequentially and integrate their results with the sensory context, persona context, and thought traces into the L.M to generate final responses.
Training Scheme. We use supervised fine-tuning (SFT) with CoT to train the context reasoner in ContextAgent. Specifically, we construct the SFT dataset DSFT = {(x, T,))). Here, X contains the sensory context C and persona context P. The thought traces T divided by <think> and </think>, are distilled from advanced LLMs [2], enabling ContextAgent to "think before acting". generating explicit thought traces before proactive predictions and tool calls. The output y contains proactive scores Ps and planned tool chains Tc.
5 Experiments
5.1 Experimental Setup
Implementation Details. Our experiments are conducted using 8 A6000 GPUs. For SFT, we use the AdamW optimizer with a learning rate of 0.0001 and apply LoRA techniques during model training. We set the LoRA rank to 8 and use cosine scheduler with 10% warmup ratio, training for 5 epochs.

---

For ICL-based baselines, we randomly select 10 samples from the dataset as demonstrations included in the prompt. We randomly split the dataset into 60% training and 40% testing our experiments.
Metries. We employ two categories of metrics to evaluate the performance of context-aware proactive L.M agents, including proactive prediction and tool calling. Details each metric are follows.
Proactive Prediction. We first evaluate the agent's ability accurately determine the need for initiating proactive services. Specifically, we use four metrics to assess proactive prediction performance, including the accuracy of proactive predictions (Ace-P), missed detections (MD), false detections (FD), and the root mean square error (RMSE) between predicted proactive scores and ground-truth. Ace-P, MD, and FD are commonly used in the existing work [24], while RMSE provides finer-grained evaluation of the performance of predicted proactive scores.
Tool Calling. To evaluate the agent's tool calling performance, we follow existing works [5, 8] and use standard metrics such as Precision, Recall, and Fi-score to compare the tool names in the predicted tool set with those in the ground-trute -rutt tool set. We also use Acc-Args evaluate whether the proactive agent can correctly generate the structured data for tool calls, including the tool names and arguments. If an argument of any tool is incorrect, the entire sample is considered incorrect. For Acc-Args, we calculate the accuracy only for the correctly predicted tools to ensure a fair comparison of different approaches.
Baselines. We compare ContextAgent with several baselines, including Proactive Agent [24], vanilla ICL, CoT, ICL-P, ICL-All, vanilla SFT. and SFT-P. For the Proactive Agent, we follow [24] and modify the task instructions in the system prompt to adapt to the proactive agent task. For the vanilla ICL, we use few-shot demonstrations with only sensory contexts. For CoT, we include both sensory contexts and thought traces. For ICL-P, we include sensory contexts and personas, and for ICL-All, we incorporate sensory contexts, thought traces, and personas into the prompt. Vanilla SFT uses sensory contexts for fine-tuning. SFT-P uses both sensory and persona contexts. We conduct experiments on 13 LLMs, comprising (1) proprietary LLMs including GPT-4o [17] GPT-3.5 [48], GPT-o3, GPT-o4-mini, and Claude Sonnet 4, (2) open-source LLMs including 1lama-3.1-770-Imm and S0ven2.5-2BB-Instruue [40], and (3) small LLMs including Llama-3.1 1-8.1--88-nnttte Qwen2.5- 7B-Instruct, --nn.---tB- Qwen2.5- 1..52.55--.--nsttett Instruct DeepSeek-R I-Distill -weill [10], and ----ssitl---- Swen--wennnll Details of the baselines and implementation are in the Appendix B.
[15]
5.2 Results on Benchmarks
Quantitative Results on ContextAgentBench. Tab. 1 shows the overall performance of ContextA- gent on ContexLAgenBeench. Results show that when using 1lama3.1-8P 1-B--Inttutt as the base LLM. ContextAgent consistently achieves the highest performance across all metrics, with increases of 8.5% in Acc-P, 7.0% in Fl-score, and 6.0% in Acc-Args. Fig. shows that ContextAgent can achieve performance comparable to or even exceeding baselines that employ 70B-scale LLMS and proprietary LLMs, with metrics such as Acc-P (-1.5% F1-score (-3.0%), and Acc-Args (+6.6%). Due to space constraints, Fig. 5 shows only three key metrics. See Tab. 6 in Appendix E for full comparison.
Proactive Agant C Vanilla ICL
CoT
ICL-P O ICL-AII Z Vanilla SFT Z ContextAgant
1.00
Acc-P1
1.00
Fl-score
0.8
Acc-Argst
0.75
0.75
0.6
0.50
0.4
0.0
GPT GPT QwenLlama DS Qwenklama
GPT
GPT Owerd hame
DS QwanLlama
GPT GPT
QwenLiama
DS
QwenLiama
44 -3.5
-728
-708
-7B
-78
-40
-3.5
-728
-705
-75
-7B
40
-708
.75
-78
-78
Figure 5: Main results on ContextAgemtBench. 'DS' refers to 'DeepSeek..
Quantitative Results on ContextAgentBench-LLite Fig. 6 and Tab. 2 show the performance of ContextAgent on ContextAgemtBench-Je
The results indicate that both ContextAgent and the baselines exhibit slight performance degradation. However, ContextAgent still achieves the highest performance across all metrics compared to the baselines. When using Qwen2.5-7B-Insttee base LLM,, ContextAgent achieves improvements of 6.2% Acc-P, 3.0% F1-score, and 7.6% Acc-Args, over the best baseline. It can even achieve comparable and even higher performance than baselines using 70B-scale and proprictary LLMs. Complete results are provided in Tab. 7 within Appendix E.
as the
7

---

Table
Main results on ContextAgemtBench
Pronctive Predictions
Tool Calling
Model
Method
Acc-P"
MD.
FD.
RMSE
Precision
Recall
Fl-score
Acc-Args
Proactive Agent
0.676
0.306
Vanilla ICT.
0.742
0.224
0.033
1.853
0.608
0.533
0.552
0.269
CoT
0.699
0.278
0.023
1.960
0.590
0.539
0.551
0.209
HJ 1--8--s
ICL-P 0.742
0.242
0.015
1.922
0.608
0.553
0.567
0.262
ICL-AII
0.757
0.229
0.012
1.872
0.631
0.565
0.582
0.270
Vanilla
0.813
0.117
0.405
ContexAAeen
0,874
0.030
0.095
1.408
0.660
0.627
0.626
0.448
Proactive Agent
0.544
0.411
0.044
3.093
0.467
0.454
0:457
0.019
Vanilla ICL
0.646
0.248
0.105
2.568
0.457
0.433
0.437
0.078
CoT
0.653
0.027
2.760
0.501
0.507
0.109
DeepSeck-R1-77
ICL-P
0.690
0.227
0
2.466
0.518
0.479
0.486
0.126
ICL.AII
0.704
0.103
Vanilla SPT
0.823
0.068
0.108
1.630
0.621
0.570
0.579
D. 0.386
ContextAgent
0.888
0.027
0.085
1,319
0.676
0.648
0.647
0,468
Proactive Agent
0.799
0.136
0.064
2,038
U.578
0.536
0.546
0,038
Vanilla ICL
0.326
CoT
0,833
0.085
0.081
1.790
0.585
0.527
0.541
0.272
Qwen2.5-7B-Ins
ICL-P
0.833
0.091
0.074
1.819
0.610
0,556
0.303
ICT-AII
0.867
0.088
0.044
1.721
0.635
0.577
0.591
0.301
Vanilla SFT
0.775
0.088
0.136
1.774
0.589
0.551
0.558
0.398
ContextAgent
0.894
0.459
Proactive Agent
Vanilla ICL = CoT
ICL-P O
ICL-AII
Vanille SFT
ContextAgent Acc-Argst
1.00
Acc-P1
0.6
0.25
0.25
0.2
0.00
GPT GPT QwenLJama
DS
OwanLlama
0.00
GPT
GPT
Owerd Liama DS
Qwenklama
0.0
GPT
GPT Qwandaama
DS
QwenLiama
-6 -3.5
7728
-708
-76
-78
-78
-40
-3.5
-728
-708
-78
-78
-78
40
-3.5
-728
-708
-78
-78
-75
Figure 6: Main results on Contex-AgentBench-LLicce
Qualitative Results. Fig. 7 and Fig. 8 show the qualitative results of ContextAgemn including both proactive and non-proactive cases. First, ContextAgent integrates both sensory and persona contexts for reasoning and offers appropriate proactive services. Second, it can further map these contexts to planned tool chains such as the weather and agenda checker, and integrate external knowledge for enhanced proactive service. Fig. 7 shows that ContextAgent can employ GPS, datetime tools, bus schedule checkers, and ride-booking apps like Uber when the user approaching a bus station. Furthermore, during casual conversations involving proposed outdoor activities, ContextAgent uses tools such as a weather checker, a datetime tool, and an agenda checker to proactively assist the user in evaluating feasibility and making informed decisions. Third, ContextAgent can leverage persona context to generate more personalized proactive predictions. For instance, Fig. 7 shows that for health-conscious individuals deciding what order restaurant, ContextAgent proactively offers food-related health information and suggestions. Fig. shows that the persona contexts also help ContextAgent determine not to disrupt the users. See Appendix E for more qualitative results.
5.3 Ablation Study and Discussion
Impact Modalities. We evaluate ContextAgent on ContextAgentBench--Liin to assess sensitivity to missing modalities. Tab. 3 shows that when vision or audio is missing, Acc-P decreases by up to 17.9% and F1-score decreases by up to 23.3%. The results show that both modalities are critical for the context-aware proactive agent, with missing vision having larger impact than missing audio. Sensory Context Perception. We conduct experiments using the zero-shot Qwen-2.5-VI. as the VLM for sensory context extraction in ContextAgent. Tab. 10 shows that this causes ContextAgent to decrease Acc-P, P, F1-score, and Acc-Args by 3.0%, 3.3%, and 1.9%, respectively. We observe that the context generated by zero-shot VLM lacks key proactive-oriented cues, such as simply describing the user tying their shoe while sitting on the floor. In contrast, our sensory context extraction module captures detailed scenario information about the gym and specific fitness equipment, providing deeper insight into the user's s conditions and intents and resulting in higher performance.
Persona Context. We also conduct experiments study the impact of user personas by removing them during both the training and testing phases. Tab. 10 shows that removing personas consistently
8

---

Table 2: Main results on ContextAgemuennch-Le
Pronctive Predictions
Tool Calling
Model
Method
Acc-PP
MD.
FD.
RMSE
Precision
Recall
Fl-score"
Acc-Args
Pronctive Agent
0.490
2.469
0.088
0.082
Vanilla ICL
0.580
0.320
0.100
2.623
0.500
0.450
0.466
0.136
Col
0.630
0.360
0.010
2.306
0.595
0.553
0.564
0.166
1.ama3.1-8P. I-8-Iss
ICL-P
0.710
0.210
0.080
2.315
0.535
0.495
0.506
0.155
ICL-AII
0.770
0.170
0.000
1.757
0.598
0.526
0,543
0.177
Vanilla SFT
0.720
0.160
1.959
0,497
0.362
SFT-P
0.734
0.115
0.151
1,980
0.555
0.582
0.552
0.353
Agnt
0,830
0.070
0.637
0.486
Proactive Agent
0,710
0.210
0.080
2.328
0.575
0.515
0,532
0.064
Vanilla ICL
0.700
0.280
0.020
2.596
0.595
0.533
0.107
CoT
0.750
0.230
0.020
2.306
0.630
0.541
0.564
0.163
Qwen2.5.7B
ICL.P
0.840
0.080
0.080
1.783
0.570
0.224
ICL-AII
0,850
0.100
0.050
1,780
0.615
0,565
0,573
0.183
Vanilla SFT
0.640
0.190
0.170
2.206
0.520
0.457
0.476
0,342
SFT-P
0.774
0,083
0.143
1.790
0.481
0.495
0.473
0.374
ContextAgeur
0.850
0.050
0.100
1.403
0.667
0.615
0.624
0.479
CC The user connentig wit a persor or the read..
uas is rea drrat loding at tu Hap rured Suarro 1hete wee
C
The usr snting al lible in mashhuu
Alsk taked
o9 vilkas? We have had ben do ward u
(dbb The anlly saice
and fov wot
Bell The user a tealh constous and wantl to nake
home
<picate Cortedual llo showe user engigad coneention
chiego at shop samed Squar
<hiik ihese usar
should
arial
solle aogessionn.
cona the westher whither
settobile offircionn perk like curserly lcoaed readis ase DP% io cenmare the Luerk bcation.
raet tnat rarty eaer's current date, and This
recemmentaion for popsier fame 2Md helrdd Beaal retasay usures and cestiiunll intomaation to rhis food to haiedd
cntvel Honnmnrd ale tie toragi to on setreci such Libe..
chanenne Gallage
X siveettn Gallige
X Pirecton Callege
<Tapper
spec caty wetteer
<tapper Serviee Trve
citec Jecc J0e7s
-Trgger Service Tree
aptiin conficcts
neccal Unouledge
cheepaneen Core delin th weather beaced, how York weskend reght exi be Midil tram-buliiiio tike
#iaspeensn i-s canenity 11:85 on Maxna 16, 2005, and ben achouded antie
deapesson 1a bel you valiioal Morsadoon cahas
city
rears terre (hvnn lubo hou right you
caderie aeriing
wit 400 lng anto Steah
Adsi whee the soater & vas bocentl and your sched de caer
contact
dras Uhei for you?
Weaad yeu lie rour socal ci adhic on mandin Eakolied cressnn
Figure 7: Qualitative results of ContextAgent in proactive cases, showing sensory and persona contexts, and ContextAgent's thought traces, proactive predictions, tool calls, and final responses.
leads to significant performance drops, with 1.ttna--..-BB-lle experiencing decreases of up to 9.0% in Acc-P, 12.3% in Fl-score, and 12.6% in Acc-Args. Results show that personas are crucial for the proactive agent task, impacting both proactive predictions and tool-calling capabilities.
Thought Traces. We also investigate the impact of thought traces the context-aware proactive agent. We observe that integrating those thought traces can significantly improve ICL performance. Tab. shows that ICL-All achieves up to 20.1% 1% improvement in Acc-P compared to ICL-P, which does not utilize thought traces. Results validate the effectiveness of thought traces for this task. Additionally, as shown in Tab. 10, we also observe that their benefits reduced under SFT.
Different Base LLMS and Tool Chain Lengths. We conduct experiments using different base LLMs in ContextAgent. Tab. 8 shows that Llama-3, 1--B--Instrue and Qwen2.5-7B -Instruct achieve comparable performance and outperform DeepSeek-RI- RI-7B.. We also test LLMs in 1.5B to 3B sizes. More details are in Appendix E. In addition, Tab. 11 11-Tab. 13 show the performance of ContextAgent across samples with varying tool chain lengths. We observe that most approaches achieve higher MD but lower FD, as the prompt used encourages more conservative initiation of proactive services, leading to less intrusive assistance. See Appendix E for more details.
CC2 vaklig
boing a seytiar aadk ne appeet with dua tnds and sumsnnded codee bepbound mouse
The pick satee paras, some beathy Natar Crihve
Nea
Prtdyea
Co.
C caw sorkigg scmntyy viyy otte with Indiction volcatng np i trnn. Theyfon Isoou predivey cooid inferation Wut's moti souft wopprtatte of
other Shon saoe kooe anghen ocke inspepiias stachs eagarddess Withou soa onal wirrss nanaad
p <ppats Diried Homaatcc sbove the use lewirg
humker vor cfickie and sieid be way lander io PRo wladera aodiina coneeraaaaaaa anplandden sinato
right vitaee
codl
sProeedive Sccnp-
Calling
<Proestiee doseeo
ofraacivv Spare-
PPanciee Calligr Neee
<Triper Sendice Polze
sTroper Service Fetre
shtepoonee Rarme
cingger Seriisio Fales
oieecsssao Hits
Figure& 8: Qualitative results of ContextAgent in non- con-proactive cases.
9

---

Table Performance with missing sensors. "wo vision" and "w/o audio" denote inference without visual or audio contexts in ContextAgent. The base model is DeepSeek-R1-72
Proactive Predictions
Tool Calling
Settings
Ace-p²
MD.
FD.
RMSE
Precision
Recall
Fi-seore'
Acc-Args'
w/o vision
0.709
0.129
0.163
2,070
0.424
0.422
0.414
0.163
wro audio
0.720
0.173
0.106
2,020
0.501
0.501
0.493
0.212
Full
0.888
0.027
0.085
1,310
0.676
0.648
0.647
0.468
5.4
Out-of-Domain Evaluation
We also evaluate ContextAgent under an out-of-distribution (OOD) setting. We randomly split ContextAgentBench based scenarios. Samples from six scenarios are used for training, while those from the remaining three scenarios are used for evaluation. Fig. 9 shows that ContextAgent achieves up 90.9% Acc-P, 68.9% FI-score, and 51.6% Acc-Args under OOD settings. Furthermore, ContextAgent outperforms the best baseline by 1.9% in Acc-Args, 10.7% in Fl-score, and 8.3% in Acc-P, validating its generalization capabilities. Tab, Appendix E presents the complete results.
Besides, Tab. shows the performance of ContextAgent compared with proprietary and advanced reasoning LLMS under OOD settings. Results demonstrate that ContextAgent achieves comparable performance to these proprietary LLMs. Additionally, enhancing the reasoning capabilities of LLMss can further improve both the accuracy of proactive predictions and tool-calling performance.
Proactive Agent I Vanilia ICL = CoT
ICL-P O
ICL-AII za Vanilla SFT ? ContextAgent
1.00
Acc-P1
1.00
F1-scoret
0.6
Acc-Argst
0.75
0.75
0.4
GPT GPT OwenLJama DS OwenLlama
GPT GPT Owerd Jama
DS QwanLlama
GPT GPT QweriLama
DS QwenLlama
-3.5
-708
-40
-4
-78 -78
Figure 9: Results on out-of-domain experiments.
Table 4: Performance of ContextAgent compared with proprietary and advanced reasoning LLMs on BentextAgemBennch. ContextAgent employs DeepSeck-RA. R1--BB as base model.
Prouctive Predictions
Tool Calling
Settings
Acc-P!
MD.
FD.
RMSE
Precision! !
Recall
F1-score"
Acc-Args
G.T-3.5-
0.879
0,020
0.300
1.452
0.657
0.521
0.555
0.235
GPT-4a
0.886
0.100
0.013
1.701
0.719
0.611
0.639
0.397
GPT-o4-mini
0.861
0.034
0.103
1,240
0.726
0.668
0.682
0.538
GPT-o3
0.069
0.062
0.755
0.711
0,563
Claude Sonnet
0.913
0.069
0.017
1,010
0.775
0.799
0.773
0.480
ContextAgent
1.249
6
Conclusion
This paper introduces ContextAgent, the first framework for context-aware proactive LLM agents. ContextAgent can harness the context information from extensive sensory perceptions and tool- augmented LLM reasoning for enhanced proactive services. To evaluate this new task, we further introduce ContextAgenlBench, the first benchmark for evaluating context-aware proactive LLM agents. Our research takes step towards further aligning with the vision of proactive AI assistants by leveraging rich context from hands-free wearable sensors to enhance proactive LLM reasoning.
Acknowledgments
The research reported in this paper was partially supported by Research Grants Council of Hong Kong under grants 14207123, STGI/E-403/24-N, and National Science Foundation under Grant Number 1943999 The views and conclusions contained here are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of The Chinese University of Hong Kong, Columbia University, NSF, or the U.S. Government any of its agencies.
10

---

References
[1] Pexels. http:///////aaaaaarrrrpa.o/o///a 2025.
2024
[2] Claude-3.7-soomet.
[3] Use fall detection with apple watch. https:://apportt.apppppp httppppp..cc//etttt//eee-/-/---/ 2025.
[4] M. Abbasian, I. Azimi, M. Rahmani, and Jain. Conversational health agents: personalized llm-powered agent framework. arXiv preprint arXiv:2310.02374, 2023.
[5] Abdelaziz, K. Basu, M. Agarwal, S. Kumaravel, M. Stallone, R. Panda, Y. Rizk, G. Bhargav, M. Crouse, C. Gunasekara, et al. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. arXiv preprint arXiv:2407.00121 2024.
[6] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes. B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al. Do can, as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
[7] S. Bai, Chen. Liu, J. Wang. Ge. Song, K. Dang. P. Wang, S. Wang, Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1300 2025.
[8] K. Basu, I. Abdelaziz, S. Chaudhury, S. Dan, M. Crouse, A. Munawar, S. Kumaravel, V. Muthusamy, P. Kapanipathi, and L. A. Lastras. Api-blend: A comprehensive corpora for training and benchmarking api Ilms. arXiv preprint arXiv:2402. 15491, 2024.
[9] K. Cheng, Q. Sun, Y. Chu, F. Xu, Y. Li. J. Zhang, and Z. Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXi::2401. 10935, 2024.
[10] DeepScek-Al. Deepseck-rl: Incentivizing reasoning capability in lims via reinforcement lcarning, 2025. URL puctps://ariis org/ wrg.abs/////aeeee 12948.
[11] S. Deng. W. Xu, H. Sun, W. Liu, T. Tan, J. Liu, A. Li, J. Luan, B. Wang. R. Yan, et al. Mobile-bench: An evaluation henchmark for lIm-based mobile agents. arXiv preprint arXiv:2407.000) 2024.
[12] X. Deng, Y. Gu, B. Zheng, S, Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2web; Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-281 14, 2023.
[13] Z. Englhardt, R. Li, D. Nissanka, Z. Zhang, G. Narayanswamy, Breda, X. Liu, S. Patel, and V. Iyer. Exploring and characterizing large language models for emhedded system development and debugging. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1-9, 2024.
[14] T. Ge, X. Chan, X. Wang. D. Yu, H. Mi, and D. Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv preprint arXiv:2406.2000 240.20094, 2024.
[15] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint 2rXiv:2407.217R8 2024.
[16] D. Guo, D. Yang, H. Zhang, J. Song. R. Zhang, R. Xu, Q Zhu, S. Ma, P. Wang, X. Bi, et al. Deepscek-rl: Incentivizing reasoning capability Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[17] A. Hurst, Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Go system card. arXiv preprint arXiv:2410.21276, 2024.
[18] S. Lee, J. Choi, J. Lee, M. H. Wasi, H. Choi, S. Ko, S. Oh, and I. Shin. Mobilegpt: Augmenting 1lm with human-like app memory for mobile task automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 1119 1133,111,,,,,0 2024.
[19] H. Liu, C. Li. Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 349116, 2023.
[20] K. Liu, B. Yang. Xu, Guo. G. Xing, Shuai, X. Ren, Jiang, and Z. Yan. Tasksense: A translation- like approach for tasking heterogencous sensor systems with llms. In Proceedings of the 23rd ACM Conference on Embedded Networked Sensor Systems, pages 213-225, 2025.
[21] W. Liu, X. Huang, X. Zeng, X. Hao, S. Yu, L. S. Wang, W. Gan, Z. Liu, Y. Yu, et al. Toolace: Winning the points of Im function calling, arXiv preprint 20X9::2409.0000 00920, 2024.
11

---

[22] W. Liu, C. Wang, Y. Wang, Z. Xie, R. Qiu, Y. Dang, Z. Du, W. Chen, C. Yang, and C. Qian. Autonomous agents for collaborative task under information asymmetry. arXiv preprint 24Xiv:2406.14925 2024.
[23] X. B. Liu, S. Fang, W. Shi, C.S. Wu, T. Igarashi, and X. A. Chen. Proactive conversational agents with inner thoughts. arXiv preprint arXiv:2501.000 00003, 2024.
[24] Y. Lu, S. Yang. C. Qian, G. Chen, Q. Luo, Y. Wu, H. Wang. X. Cong, Z. Zhang, Y. Lin, et al. Proactive agent: Shifting Ilm agents from reactive responses to active assistance. arXiv preprint arXiv:2400. 12361, 2024.
[25] M. A. Merrill, A. Paruchuri, N. Rezaci, G. Kovacs, J. Perez, Y. Liu, E. Schenck, N. Hammerquist, J. Sunshine, S. Tailor, et al. Transforming wearable data into health insights using large language model agents, arXiv preprint arXiv:2406.0644
2024.
[26] G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
[27] B. Paranjape, Lundberg, S. Singh, H. Hajishirzi, I.. Zettlemoyer, and M. Ribeiro, Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014 2023.
[28] K. Post, R. Kuchida, M. Olapade, Z. Yin, P. Nurmi, and H. Flores. Contextllm: Meaningful context reasoning from multi-sensor and multi-device data using llms. In Proceedings of ACM HOTMOBILE 25. Association for Computing Machinery (ACM), 2025.
[29] Y. Qin, S. Liang, Y. Yc, Zhu, L. Yan, Y. Lu, Lin, X. Cong, X. Tang, B. Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307116789 2023.
[30] A. Radford, W. Kim, T. Xu, G. Brockman, C. MeLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 24992-8518.... PMLR, 2023.
[31] L. Shen, Q. Yang. Y. Zheng, and M. Li. Autoiot: LIm-driven automated natural language programming for aiot applications. arXiv preprint or Xix::2003005344 2025.
[32] 7. Shi, S. Gao, X. Chen, Y. Feng, L. Yan, H. Shi, D. Yin, Z. Chen, S. Verberne, and 7. Ren. Chain of tools: Large language model an automatic multi-tool leamnrr arXiv preprint 155. 2024.
[33] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8652, 2023.
[34] P. Trirat, W. Jeong, and S. J. Hwang. Automl-agent, A multi-agent Ilm framework for full-pipeline automl. arXiv preprint arXiv:2410.02958. 2024.
[35] J. Wei, X. Wang. D. Schuurmans, M. Bosma, F. Xia, E. Chi. Q V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems. 35:24824-2485 24837, 2022.
[36] Y. Wei, O. Duchenne, J. Copet. Q. Carbonneaux, L. Zhang. D. Fried, G. Synnaeve, R. Singh, and S. L. Wang. Swe-rl: Advancing I1m reasoning via reinforcement learning on open software evolution. arXiv preprint 2Xiv:2502.18844 2025.
[37] H. Wen, Y. Li, G. Liu, S. Zhao, T. Yu, J.J. Li, S. Jiang, Y. Liu, Y. Zhang, and Y. Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 543 $57, 2024.
[38] S. Wu, S. Zhao, Q. Huang, K. Huang, M. Yasunaga, K. Cao, V. Ioannidis, K. Subbian, J. Leskovec, and J. Y. Zou. Avatar: Optimizing 1lm agents for tool usage via contrastive reasoning. Advances in Neural Information Processing Systems, 37-25981-26010, 2024.
[39] Y. Wu, G. Wan, J. Li, S. Zhao, L. Ma, T. Ye, I. Pop. Y. Zhang, Jnd Chen, Proai: Proactive multi-agent con- versational ai with structured knowledge base for psychiatric diagnosis. arXiv preprint 25X2::2502.20689 2025.
[40] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint 2511::24/2..5111. 2024.
[41] B. Yang, L. He, N. Ling, Z. Yan, G. Xing, X. Shuai, X. Ren, and X. Jiang. Edgefm: Leveraging foundation model for open-set learning on the edge. In Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems, pages 111-124, 2023.
12

---

[42] B. Yang, L. He, K. Liu, and Z. Yan. Viassist: Adaptigg multi-modal large language models for users with visual impairments. In 2024 IEEE International Workshop Foundation Models for Cyber-Physicul Systems & Internet of Things (FMSys). pages 32-37. IEEE, 2024.
[43] B. Yang. S. Jiang, L. Xu, K. Liu, H. Li, G. Xing, H. Chen, X. Jiang, and Z. Yan. Drhouse: An ui--empowered diagnostic reasoning system through harnessing outcomes from sensor data and expert knowledge. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8(4); 1-29, 2024.
[44] B. Yang, Y. Guo, L. Xu, Z. Yan, H. Chen, G. Xing. and X. Jiang. Socialmind: Lim-based proactive ar social assistive system with human like perception for intu live interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(1):1 30, 2025.
[45] H. Yang, M. Li, M. Han, Z. Li, and W. Xu. Embedgenius: Towards automated software development for generic embedded iot systems, arXiv preprint arXiv:2412.0005: 2024.
[46] J. Yang. C. Jimenez, A. Wettig. K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent- computer interfaces enable automated software engincering. Advances in Neural Information Processing Systems, 37:50528-50652, 2024.
[47] S. Yao. J. Zhao, D. Yu, N. Du, Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
[48] J. Ye, X. Chen, N. Xu, C. Zu, Z. Shao, S. Liu, Y. Cui, Z. Zhou, C. Gong, Y. Shen, al. A comprchensive capability analysis gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303. 10420, 2023.
[49] C. Zhang. Z. Yang. J. Liu, Y. Han, X. Chen, Z. Huang. B. Fu, and G. Yu. Appagent: Multimodal agents smartphone users. arXiv preprint drXiv:3312.3377,, 2023.
[50] C. Zhang, K. Yang. S. Hu, Z. Wang, G. Li, Y. Sun, C. Zhang, Z. Zhang, A. Liu, S.-C. Zhu, et al. Proagent: building proactive cooperative agents with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17591 17599, 2024.
[51] K. Zhang, Li, G. Li, X. Shi, Z. Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07333 2024.
[52] X. Zhang, Y. Deng, Z. Ren. S.-K. Ng, and T.-S. Chua. aak-before-plae Proactive language agents for real-world planning. arXiv preprint arXiv:2406.1266 2024.
[53] Y. Zhang, R. Sun, Y. Chen, T. Pfister, R. Zhang, and S. Arik. Chain agents: Large language models collaborating on long-context tasks. Advances in Neural Information Processing Systems, 37:132208- 132237, 2024.
[54] A. Zhao, D. Huang, Q. Xu, M. Lin, Y.J. Liu, and G. Huang. Expel: Lim agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632-19642, 2024.
[55] S. Zhao, A. Zhu, H. Mozannar, D. Sontag, A. Talwalkar, and V. Chen. Codinggenie: A proactive Ilm powered programming assistant. arXiv preprint arXiv:2503.1472. 2025.
[56] B. Zheng, B. Gou, J. Kil, H. Sun, and Y. Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.016// 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly outline the main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work the Appendix F.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper does not include any theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide the implementation details of experiments in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We will provide open access to the data and code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.o cc/ httpc//guidss//oooeeee
oreiionnocccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct /ttp:///ggiiieeeeeeeuuuubiiiii ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the implementation details of our experiments in Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
We do not report error bars due to the extensive computation costs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes] We provide the implementation details of experiments in Sec. 5.1.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / /ttp///eeersp/eeeeee/eee cc wc/ppuulc///ttieeeeeeeeeeeeeeeee htt.lliicEtticcccudeeeieees
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics in this study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in the Appendix G.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose risks requiring such safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the used code, data and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification:
We explain the data generation process in Section 3.2.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We have discussed the dataset ethics in the Appendix C.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We have discussed the dataset ethics in the Appendix C.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
19

---

Justification:
We have provided details on how we use LLMs in Section 4.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https:/ for what should or should not be described.
20

---

Appendix
A
Data Examples
We first provide several examples in our dataset, including cases with different proactive scores.
Example
"Context information": "Visual information suggests that the user is conversing with a partner on the road. Audio information shows there is a conversation between the user and others. The partner says "Hi, how about organizing team building hike this weekend? haven' t had a team building event in a long time.""
"Personas": [
"A software engineer specializing document management systems, working closely with the graphic designer to implement the designed interfaces."
],
"Thoughts": "When planning hiking activities, various factors should be considered, including the weather, whether it is the off-season or peak season for tourism passenger flow, schedule, and more. Since the hiking location has not been specified, it may be in the city where the user is currently located. I need to use GPS to determine the user's location.",
"Proactive score": 5,
"Tools": "[("name":
"desc":
"Get the current GPS coordinates of the user", "params ": "None"}, {"name": "get_city_veather", the weather for specified city at a given time params": {"city": "SRESULT (get_ "BEesu"[gett.ureent..""" city)", "time": "this weekend")}, {"name":
"desc": "Get
"desc": "Get the current date and time", "params": "None"}, ("name":
get_current_dattiing
bleck_agend__1 conflictt,, "desc": "Check if there is a time conflict the user's agenda for a given datetime.", "params": "None"}]",
"Ressoose": "The weather for the hiking location this weekend looks Do you want help you pack suggest what bring?"
skies and mild temperatures
}
Example 2
"Context information"; "Visual information shows the user is sitting at dining table with a plate of food that includes pasta, grilled chicken, and a side of salad.",
21

---

"Personas": [
"The user has a history managing Type diabetes.", "The user is health -oatthhoonsiiuss and actively manages their diet to control blood sugar levels."
1,
"Thoughts": "The user's personas show his glucose level is slightly elevated, and the user might appreciate dietary advice help stabilize it. Providing this information could be helpful without being intrusive.",
"Proactive score": 4,
"Tools": "[m"name": feet_health_datee "desc": "Get health data from the user 's smart device.". " "params": "None ">mm"name": wikkipedi__eearch searches the Wikipedia API. Useful for when you need answer general questions about people, places, companies, facts, historical events, or other subjects .,, "params": ("query": "The effect of pasta, grilled chicken, and side salad on stabilizing blood sugar levels. "}]]"
"desc": "Tool that
Ressonse"" "Your glucose level appears slightly elevated. To help stabilize it, you might consider increasing the portion of salad reducing the amount of pasta. Grilled chicken is great choice for protein, as it has minimal impact
blood sugar levels. Would you like me to provide additional dietary tips or help track your meal for better management?"
Example 3
"Context information"; "Visual information suggests that the user is looking in the mirror while wearing summer clothes, such as short sleeves and shorts.",
"Personas": [
"A university psychology student who is currently studying creativity and personality."
},
"Thoughts"; "The user is looking the mirror and may be preparing to out, wearing minimal clothing such as short sleeves and shorts. need to check the weather for the user's location. If the temperature is expected to drop or rain forecasted for the day, can proactively remind the user to dress warmly to avoid catching cold.",
"Proactive score": 3,
"Tools": "[["name":
"desc":
"Get the current GPS coordinates of the user", "params ": "None"}, {" :: "get_city: "eet_ciity.vatthe" weather", "desc": "Get the weather for specified city at a given time.",
22

---

parans": ("city": "$RESULT (get.currei..gs.corriiaae city)", "time" "nww"""]]"""""""""""""""""""""""
"Response"; "I checked the weather forecast, and the local temperature is expected to drop sharply after 6 PM. Considering that you are only wearing sloet-dleeved shirts and shorts, it is recommended that you add more layers."
}
Example 4
"Context information"; "Visual information suggests the user is in gym, lifting weights.",
"Personas": [
"A computer science major interested developing software for audio manipulation and enhancement.""
1,
"Thoughts": "The user might need guidance on proper weightlifting techniques between sets. However, without detailed persona information, such their fitness level, I would rate this proactive behavior at 2 points. This rating indicates some potential for proactivity, but the necessity is not very high. After collecting more detailed persona information, I could provide more accurate proactive services.",
reminders to take breaks
Proactive score": 2, "Tools": "None", "Response"; "None"
Example
"Context information"; "Visual information shows the user is walking the road. There are many trees both sides of the road, and the road seems to be going uphill.",
" "Personas":
"A dog owner who wants the best medical treatment for their furry friend."
}.
"Thoughts"; "Contextual information shows user is walking on the road. There many trees on both sides of the road, and the road seems to be going uphill. Without explicit input or additional contextual information, no proactive actions are required. The proactive score is set to 1.",
"Proactive score": 1, "Tools": "None",
23

---

"Response": "None"
B Prompts
This section introduces the details of the prompts used in this work, including the system prompt used in ContextAgent and baselines, the prompt used in the data generation pipeline, and the prompt used in proietive-oriented context extraction.
System Prompt in ContextAgeett. Fig. 10 shows the prompt used in ContextAgent. It contains both the static prompt and the runtime prompt. The static prompt includes task instructions and toolset definitions that remain constant throughout. The task instructions guide the LLM in understanding its role as a context-aware proactive agent and highlight the key considerations for this task. The toolset definitions allow LLM agents to identify the available external tools and understand how use them, including their names, arguments, and formats (See Appendix D). The runtime prompt includes user personas and contextual information, which vary across different samples in the dataset.
Prompt Template for ContextAgent
## Task Instructions You proactive LLM agent. You will receive both sensory and persona contexts. The sensory contexts are captured by the multi-modal sensors the user's wearable devices, such smart glasses and earphone. Persona context is the personal information like identity and preference. Based on these contexts, you need to generate thought and determine whether to initiate proactive service for the user not. Only propose assistance when you fully understand the user's actions. When generating the response, consider whether you need to call extemal tools to complete the task. The following the tool names and their arguments. Please only select tools from the specified tool sets and make
the correct format for their arguments.
## Tool set GetCityWeather: (Description: Arguments: ...)
CheckkAgendaiiieeConiicc (Description: .... Arguments: ...)
GoogleSearch: (Description: .... Arguments: ...)
Now you will receive the sensory and persona contexts.
## Persona Context: Personas Context
## Sensory Context: Sensory Context
Figure 10: System prompt for ContextAgent.
System Prompt for Baselines. Fig. 1l shows the prompt template used in bascline approaches. To ensure fair comparison, we keep the task instructions and toolset definitions in the prompt the same as those used ContextAgent. For the baseline of Proactive Agent, we do not include any samples in the example part of the prompt template. For Vanilla ICL, COT. ICL-P, and ICL-AII, randomly select ten samples from the training set and incorporate them into the prompt. In Vanilla ICL, the few-shot examples contain only sensory context. In CoT, additionally include thought traces in the few-shot examples. For ICL-P, we incorporate persona context into the few-shot demonstrations, and for ICL-AII, integrate sensory context, persona context, and thought traces into the few-shot demonstrations. Additionally, for the Vanilla SFT bascline, we use the prompt template shown in Fig. 10, but without including the persona context.
Prompt for Data Generation Pipeline. Fig. 12 shows the prompt template used for data generation. We include the required format of data, examples of persona' and completed data samples, and detailed descriptions of the accessible tools. Besides, specifies the structure and formulation of each component within data sample to guide LLMs generate high-quality samples. During data generation, we use two strategies, including scenario-aware and proactive score-aware. LLMs guided to generate samples for different scenarios and target proactive scores separately, which enables L.LMs to more effectively learn scenario-specific and score-specific patterns from the provided examples. And when generating samples requiring proactive services, LLMs are instructed to consider
Phttps://gitee
http:///ggttbbb.ccoeeeeeettt com/tencent- wallab/perronaahhe
24

---

Prompt Template for ICL-based Baselines
## Task Instructions
You are a proactive LLM agent. You will receive both sensory and persona contexts. The sensory contexts are captured by multi-modal sensors on user's wearable devices, such as smart glasses and earphone. Persona context the personal information generate thought and determine whether initiate a proactive service for user or not. Only propose assistance when you fully understand the user's actions. When generating the response, consider whether you need to call external tools to complete the task. The following are the tool names and their arguments. Please only select tools from the specified tool sets and make sure to use the correct format for their arguments.
identity and preference. Based on these contexts, you need
## Tool set:
GetCitWeatter: (Description: Arguments: ...]
CheekAigendaTmmeCCCCii::::: (Description: Arguments: ...)
imeConflict:
GoogleSearch: (Description: Arguments: ...)
## Examples: (Examples)
Strictly follow the following format for your oulputs:
For proactive samples. the output format should be...
For not require proactive samples, the output format should be...
Now you will receive the sensory and persona contexts.
## Persona Context: ( Personas Context
## Sensory Context: { Sensory Context
Figure 11: Prompt template for baselines.
using tools from tool To further enhance the diversity of the generated samples, we employ various LLMs, including GPT-44o,,
and Gemini-2.0-Flashh for generation.
Prompt for Proactive-oriented Con- text Extraction. Fig. 13 shows the prompt template used for proactive- oriented context extraction. With this prompt, the VLM focuses objective, detailed scene understanding and key capturing (e.g., location, objects, and user actions) that are critical for de- termining whether proactive assistance is needed. To ensure consistency and output quality, we include five example descriptions within the prompt. These examples illustrate the expected level of detail, structure, and tone, enabling the model to align its output with the expected format. Based on the clear content and formatting guidelines, generated descriptions can provide high-quality context information for the following tasks, including predicting the necessity of proactive services and tool planning. In this study, we employ Qwen-2.5-VL [7] and Whit Whisper [30] to extract the visual and audio contexts, respectively.
Process
Raw Data Collection
CAB-Lite
Caecstion
Check
Initial Examples
Exper
CAB
LLM-based Synthetic
Validation
Data Generation
Figure 14: Flowchart illustrating the pipeline for construct- ing our benchmark. 'CAB' refers to 'ContextAgentBechh
C Dataset Details
As described in Sec. 3.2, the annotators first brainstorm and design initial exemplars to construct the seed dataset. We then utilize automated diversification pipeline to scale the dataset. Fig. 14 shows the flowchart of our pipeline to generate the ContextAgentBened (CAB) and ContextAgemt Lite (CAB-Lite). Additionally, descriptions of sensory contexts in the dataset. This raw data is sourced from both copyright-
also collect raw video and audio data based the textual
card/ com/ www.e///weeeeeeeeee/eee sonnet "nttps://deepminn htt.://mww....ooogg/////ooooeoee www.://ologsgggggiinnnmm/e gemini ffash/
25

---

Prompt Data Generation
## Task Instructions
You are professional data generation specialist responsible for creating high-quality training examples for the development of a proactive Al assistant system integrated with smart glasses. Your mission is to generate realistic data samples that accurately represent situations in which the Al assistant should proactively trigger functions or tools to assist users based on contextual perceptions. Contextual perceptions include text descriptions of the physical environment or natural conversations with other people. This information can be captured by sensor data, such as RGB data, audio data, and more.
## Task Objective:
Generate comprehensive, realistic training examples to determine when an Al assistant should proactively offer assistance and when it should wait for explicit user queries. Consider two important factors when initiating proactive service: contextual information from the physical environment and the user's persona.
## Data Format Requirements:
Each example must follow this exact structure and include all fields:
"example-fnumber"" {
"Category": [domain area such as Work, Travel, Health, and Others.).
"Response": (natural-sounding, helpful response the assistant should provide).
## Persona: When creating personas for examples, use the following samples as guide. Generated personas should follow this style. concise descriptions that capture key user characteristics, habits or preferences that would influence their interaction with proactive assistance: (Personas Examples)
When generating data examples, you should limit the tools used to the following toolset, which includes commonly used tooks, Al models, or apps on the user's mobile devices. The following dictionary provides the names of the tools and their functions. (Tool Set)
## Tool Set:
## Guidelines for Professional Data Generation:
First-Person Perspective: All "Vision" descriptions must be from the user's point of view as captured by smart glasses (what the user is looking at. not the user themselves).
Ensure all fields align logically (e.g., high proactive score should correspond with actions and responses that provide significant value).
Logical Consistency:
## Example Reference: (Examples)
## When generating examples, remember these key considerations:
Would users genuinely appreciate proactive assistance in this scenario?
Do not give strong assumption on "Mobile API data". only limited to information that can be obtained from smartphone apilapp notification
Please generate (Generated Number) diverse, high-quality examples with proactive score of (Proactive Score) for the (Category) category. Format your response as a valid JSON object with each example following the exact structure specified above.
Figure 12: Prompt template for the data generation pipeline.
internet platforms and our own collections. The self-collected samples are primarily used to gather chitchat scenarios. This study has received IRB approval. All participants offered informed consent before any data were collected. Additionally, we have blurred sensitive regions in the vidco, such as faces, to ensure participants' privacy.
D Tool Definition
ContextAgentBench contains 20 tools. Given that the main focus of this study is to bridge the sensory context with tool-based LLM agents, we adopt the definitions of tools from existing works, Tab. provides detailed information about the tools, including their names, tool descriptions, and input and output arguments.
hnttpa://www www.wpppeels..ommmmm com.slcccoo/cceee/// "https://gite http://aaaa.elaaaaaaeaeeaaaaaaaa bankk/k//////////aa/a bank/
26

---

Table 5: The definition of the 20 tools used in Bont
Index
Name
Description
Inpot
Outpot
GeCiiyWetther Weather
Get the weather for a specified city given time.
[eex]] The city to fetch weather for. [text] The time to fetch weather for.
[tett] Weather condition for
specified city given time.
DateTime
Gct the current date and time.
None.
[tex]] Carrent date and time.
Check there time conflict in the user's is agenda for given datetime and return all events as summarized string.
[ecx]] The clickk com- flicts.
[[ex]] summary events and whether there a conflict,
WikipediaSeurch
Search on Wikipedia.
[tex]] Search query.
[texi] Wikipediaa search resultt
GetCumenntiPS
GPS coordinates
None.
[jext] GPS coordinates user.
of the user.
Get the price a product from an online
[text] The name of the product for.
[eat]] The price of the product string.
SeurchRednote
A platform where people share tips on travel, fitness, cuoking. and more, allowing users to scarch for relevant strategics.
[text] The search query.
[tex]] The search resalts from rednote.
ViuulLLanganeMModel
Visual Language Model can answer the user's is questions based the image.
[image] Any image. [text] The prompt containing the user's VLLM. question.
respousee from
GoogleMap
Get the route and distance from the current location to the desti- nation using Google Maps API.
[[ex]] The starting location. [ext] The route and distarce in-
[tei]] The destination location.
formation.
10
BookUlber
Hook an Uher ride from the cur- rent location to the destination.
[tex]] The starting location,
[tex]] The therr ride booking confirmation.
[eett] The destination location.
11
GelHealllDaaa
Get health data from the user's smart devioe.
None.
[ext)] The bealth data as a string.
12
GetMeciial/Kowlldde
medical expert knowledge from the up-to-date medical
[[ex]] query string contain- ing the medical topic or symp. toens.
[ext]] medical expert know]- edge as string.
knowledge datahise.
13
PlayMusic
Play a song from the user's 's ma- sic library.
None.
[ext] The song playing confir-
mation.
14
AddioAgenda
Add an event agenda.
[tex]] name of event add. [texi] The time of the event.
[tex]] confirmation message.
15
CheckBusSchodule
Check the bus schedulle for a spe- cific bus stop.
[text]
the stop.
[[ex]] The bus schedule informa- tio..
16
GoogleSeurch
Scurch Google.
[tei]] Search query.
[ext Description of the search
17
SetTimer
Set a timer for specific dura- tion.
[texi] The duration of the timer.
[toat] The timer set confirmation.
18
QueryStock
This API queries the stock price of given stock code and date.
[texi] The stock code of the given slock. [test] The date of the stock price.
[text] The stock price of the given stock.
19
AddMeeting
This API allows users to make reservation for meeting store the moeting information (c.p. topis, time, location, stien- dees) in the database.
[text] The topic of the meeting. [text] start time the ing. [text] The location where
[text] Success or failed.
the meeting to be held.
20
SendEmail
This API for sending email. given the receiver, subject and content.
[text] The receiver address of the email. [sear] The subject address email. [tes]] The content the email.
[text] The status of the email.
27

---

Prompt for Proactive-oriented Context Extraction
Task Instructions
You are an advanced first-person vision description specialist. Your lask analyze frames and generate detailed, objective descriptions specifically for the "Vision" field proactive Al assistant training examples. These descriptions represent what the user's smart glasses other ego camera would capture from their perspective.
## Description Requirements Your vision descriptions must:
1. Be factual and detailed
Maintain first-person perspective
3. Balance detail with conciseness
Use clear, straightforward language
Follow the pattern of provided examples
## Example Style Guide
Your descriptions should follow style of these examples:
Example-1: "The user a room with multiple individuals, all seated around circular table, appearing to be gathered for a meeting."
Example-5: "The user walking near busy street and looking at bus stop named "University."
## Key Elements Include
When describing the frame, focus on:
1. Location and setting- - Where the user (affice, store, outdoors, restaurant, etc.)
Key objects in view What items are visible and relevant understanding the scene
People present others are visible, their general arrangement and actions
4. Observable user activity What user appears to doing based on what's view
5. Text or signage Any readable text that prominent in view
6. Spatial arrangement- The general layout important elements in scene
## Important Guidelines
Write in third person, referring to "the user" shown the examples
Keep descriptions to 1-2 concise sentences
Focus on what is directly observable, not what might be happening
Include specific named elements when visible (like "bus stop named "University"")
It's acceptable to mention what the camera/smart glasses "detects" as in Example-2
Provide enough context for the proactive system to understand the situation
Avoid interpretations about user's intentions thoughts
Please analyze the provided frame and generate "Visian" field description following these guidelines and matching the style of the examples.
Figure 13: Prompt template for the proactive-oriented context extraction.
E
More Results
Full Results on Benchmark. We compare the performance of ContextAgent to baselines using different base LLMs. Since our approach requires model fine-tuning, we implement it only on LLMs with fewer than 7B parameters. However, we also provide the performance of baselines using 70B parameter LLMs and other advanced commercial LLMs as reference. Tab. shows the full experiment results on ContextAgentBench Results show that ContextAgent consistently outperforms the basclines when using the same LLMs. We also observe that the baseline approach ICL-All using GPT-4o achieve the highest performance with 92.1% 1% Acc-P, 71.5% Fl-score, and 59.6% Acc-Args. Additionally, ContextAgent with a 7B parameter LLM achieves performance comparable to the best baseline using 70B LLM, with only 0.4% lower Acc-P, I. 1.5% lower Fl-score, and 0.4% higher Ace-Args, respectively, demonstrating the strong performance of ContextAgent.
Tab. shows the full results on ContextAgemteench-Liit Tab. 9 shows the full results on Contex- tAgentBench under OOD settings. Similarly, we observe that ContextAgent consistently achieves the highest performance when using the same base LIMSs as the baselines. In addition, Contex- tAgent can still achieve performance comparable to the baselines that use 70B scale L.MMs. For example, ContextAgent using Qwen2.5-7B-Ins can achieve 0.7% lower Acc-P, P, 1.9% higher Fl-score,
28

---

Table Main results on ContextAgemBBench
Pronctive Predictions
Tool Calling
Model
Method
Acc-Pt
MD.
FD.
RMSE
Precision
Recall"
Fl-acore
Ace-Args
Pronctive Agent
0.197
0.017
0.696
0.591
0.618
Vanilla ICL.
0.816
0.166
0.017
2,079
0.703
0.614
0.635
0.462
GPT-4o
CoT
0.835
0.142
0.023
2.016
0.732
0.645
0.666
0.529
ICI.P
0.921
0.061
0.017
1.507
0.797
0.685
0.714
0.596
ICL-AII
0,921
0.057
0.020
1.472
0.788
0.090
0.715
0.596
Pronctive Agent
0.673
0.010
0.316
1.815
0.493
0,380
0.410
0.065
Vanilla ICL
0.772
0.064
0.163
1,710
0.602
0.490
0.518
0.374
GPr.
CoT
0.836
0.088
0.074
1.534
0.662
0.584
0.604
0.433
ICL-P
0.806
0.040
0.153
1,713
0.634
0,517
0.547
0.400
ICL-AII
0.846
0.054
0,098
1.439
0.681
0.595
0.616
0.458
Pronctive Agent
0.064
0.091
1.717
0.670
0.555
0.585
0.144
Vanilla ICL.
0.867
0.078
0.054
1.324
0.695
0.625
0.642
0.475
Qwen2,5-72B-hl
CoT
0.884
0.071
0.044
1.254
0.708
0.643
0.660
0.468
ICL-P
0,915
0.040
0.044
1,078
0.774
0.687
0.711
0.495
ICL-AII
0.935
0.037
0,027
1,059
0.750
0.688
0.703
0.414
Prouctive Agent
0.850
0.061
0.088
1.643
0.642
0.554
0.578
0.314
Vanilla ICT.
0.799
0.013
0.187
1.442
0.582
0.501
0.524
0.374
Llam-3.1-70B-Ins
CoT
0,850
0.020
0.129
1.237
0.637
0.543
0.566
0.385
ICL-P 0.857
0.000
0.143
1.322
0.660
0.578
0.599
0.375
0.660
Pronctive Agent
0.544
0.411
0.044
3.093
0.467
0.454
0.457
0.019
Vanilla ICL
0.646
0.248
0.105
2.568
0.457
0.433
0.437
0.078
CoT
0.653
0.319
0.027
2,760
0.528
0.501
0.507
0.109
DeepSeek-RI
0.486
ICL-AII 0.704
0.268
0,027
2,540
0.545
0,510
0.518
0.103
Vanilla SFT
0.068
0.108
1.630
0.621
0.570
0.579
0.386
ConlextAgent
0.883
0.027
0.085
1,319
0.676
0.648
0.647
0.468
Proactive Agent
0.136
0.064
0.578
0.546
0.038
Vanilla ICL
0.816
0.088
0,095
1.752
0.590
0,545
0.553
0.326
1,790
0.585
0.527
0.541
Qwen2.5-7B
ICL-P
0.833
0.091
0.074
1,819
0.610
0.556
0.568
0.303
ICL-AII
0.867
0.088
0.044
1.721
0.635
0.577
0.591
0.301
Vanilla SFT
0,775
0.088
0.136
1.774
0.589
0.551
0.558
0.398
ContextAgeni
0.894
0.013
0,091
1,264
0.672
0.644
0.645
0.459
Projctive Agent
0.690
0.006
0.302
1,831
0.376
0,280
0.305
0.091
Vanilla ICI.
0.748
0.193
0.057
1.898
0.612
0.526
0.548
0.234
CoT
0.768
0.159
0.071
1,770
0.596
0.512
0.533
0.257
Llama3. 1-8B-Ins
ICL-P
0.724
0.268
0.006
2.207
0.658
0.563
0.587
0.251
0.027
0.662
0.596
Vanilla SFT
0.819
0.071
0,108
1,650
0.597
0.567
0.567
0.362
ContextAgent
0.030
0.095
1,408
0.660
0.627
0.626
and 16.9% higher Acc-Args compared to the best baseline with Llama3.1-70B- Results validate the effectiveness of ContextAgent.
Impact of Thought Traces on ICL. We also analyze the effectiveness of thought traces on ICL. First, we observe that CoT generally outperforms Vanilla ICL on ContextAgemtBench, both in in-domain and OOD settings. This indicates that incorporating thought traces into few-shot demonstrations can enhance the performance of ICL. Additionally, we observe that ICL-All also outperforms ICL-P most of the time. This suggests that even after integrating persona context, further incorporating thought traces can offer benefits to ICI approaches for this task. Furthermore, we observe that thought traces provide greater benefits for 7iB-sized LLMs compared to 7B LLMs. For instance, using Llama3. 1-mma..-170BBIns. ICL-All achieves a 20.1% higher Acc-P, 20.7% higher F1-score, and a 14.3% higher Acc-Args than ICL-P on ContexAAgentBench under OOD settings. This may be because the limited parameters in smaller LLMs result in inherently limited knowledge, making challenging for them to fully learn the distilled thought traces from more advanced LLMs.
Impact of Different Base Models. We test using different LLMs as the base model in ContextAgent. Table shows that 1.5B to 3B LLMS (e.g., Qwen2.5-1 1.B--ssuttB) perform only 2.9%, 3.9%, and 5.5% lower on Acc-P, F1-score, and Acc-Args, respectively, compared to 7B L.MMs. This reveals the opportunities to deploy ContextAgent on mobile devices without accessing the cloud, further reducing privacy concerns and system overhead [41].
29

---

Table 7: Main results on ContextAgemtBnnch-LZ
Pronctive Predictions
Tool Calling
Model
Method
Acc-Pl
MD.
FD,
RMSE
Precision
Recall"
Fl-acore
Ace-Args
Pronctive Agent
0.020
0.630
0.531
0.555
0.235
Vanilla ICL.
0,710
0.270
0.020
2,517
0.600
0.525
0.544
0.153
GPT-4o
CoT
0.680
0.310
0.010
2,630
0.607
0.557
0.571
0.207
ICI.P
0,850
0.130
0.020
1,770
0.755
0.635
0.667
0.352
ICL-AII
0.890
0.110
0.000
1.627
0.782
0.675
0,701
0.397
Pronctive Agent
0,510
0.010
0,480
2.174
0.220
0.128
0.152
0.114
Vanilla ICL
0.630
0.050
0,320
2,253
0,410
0.282
0.314
0.172
GPr.
CoT
0.660
0.190
0.150
2.186
0.531
0.439
0.464
0.126
ICL-P
0,710
0.020
0.270
1.854
0.545
0.415
0,450
0.213
ICL-AII
0.110
1.578
0.635
0.537
0.561
0.216
Pronctive Agent
0.780
0.180
0.040
2.258
0.670
0.550
0.582
0.143
Vanilla ICL.
0.770
0.120
0.110
2,000
0.595
0.481
0.512
0.136
Qwen2,5-72B-hl
CoT
0.770
0.120
0.110
1.786
0.575
0.467
0.494
0.158
ICL-P
0.760
0.030
0.210
1.612
0.605
0.471
0.506
0.205
ICL-AII
0.840
0.020
0.140
1.349
0.646
0.530
0.559
0.221
Prouctive Agent
0.760
0.150
0.090
2,076
0.620
0.529
0.554
0.253
Vanilla ICT.
0.740
0.050
0,210
1.889
0.530
0.421
0.449
0.222
Llama3.1-70B-Ins
CoT
0,800
0.060
0.140
1.649
0.585
0.491
0.515
0.222
ICL-P
0.720
0.010
0,270
1.841
0.546
0.426
0.457
0.255
0.107
0.581
0.605
Pronctive Agent
0.540
0.420
0.040
3.119
0.490
0.470
0.475
0.039
Vanilla ICL
0.540
0.310
0.150
2.849
0.455
0.411
0,423
0.103
CoT
0.560
0.320
0.120
2,796
0.465
0.447
0.452
0.070
DeepSeek-RI
0.150
ICL-AII 0.650
0.260
0,090
2.541
0.455
0.434
0.439
0.080
Vanilla SFT
0.120
0.120
1.786
0.581
0.561
0.559
0.406
ConlextAgent
0.840
0.050
0.110
1,510
0.678
0.641
0.648
0.487
Proactive Agent
0.080
2.328
0.575
0.064
Vanilla ICL
0.700
0.280
0,020
2.596
0.595
0,533
0.550
0.107
0.020
2.306
0.630
0.541
0.564
Qwen2.5-7B
ICL-P
0.840
0,080
0.080
1.783
0.656
0.570
0.595
0.224
ICL-AII
0.850
0.100
0.050
1.780
0.615
0.565
0.573
0.183
Vanilla SFT
0.640
0.190
0.170
2.206
0.520
0.457
0,476
0.342
ContextAgeni
0,850
0.050
0,100
1.403
0.667
0.615
0.624
0.479
Projctive Agent
0.490
0,000
0.510
2,469
0.200
0,088
0.117
0.082
Vanilla ICI.
0.580
0.320
0.100
2.623
0.500
0.450
0.466
0.136
CoT
0.630
0.360
0.010
2.306
0.595
0.553
0.564
0,166
Llama3. 1-8B-Ins
ICL-P
0.710
0.210
0.080
2,315
0.535
0.495
0.506
0,155
0.060
1.757
0.543
Vanilla SFT
0.720
0.120
0.160
1.959
0.536
0.497
0.508
0.362
ContextAgent
0.070
0.100
1,510
0.687
0.637
0.645
Table 8: Overall performance of ContextAgent using different LLMSs as base models.
Pronctive Predictions
Tool Calling
Category
Model
Size
Acc-P²
MD.
FD.1
RMSE!
Precision
Recall
Fl-score?
Acc-Args
LLaMA3
8B
0.898
0.028
0.074
1.254
0.685
0.653
0.652
0.465
7B 8B
Qwen2.5
7B
0.883
0.040
0.076
1,215
0.682
0.653
0.633
0.481
DeepSeek-R1
0,074
1.275
0.659
0.648
0.639
0.434
Qwen2.5
3B
0.869
0.043
0.086
1.336
0.652
0.610
0.615
0.421
1.5B 3B
DeepSeek-R1
1.5B
0.882
0.041
0.076
1.245
0.686
0.652
0.652
0.447
Qwen2.5
1.5B
0.869
0.038
0.091
1.312
0.642
0.612
0.613
0.410
Tab. shows the ablation study using different base LLMs. The results show that persona context is crucial for the task. Removing it from ContextAgent can lead to significant decreases across all metrics, with Ace-P and Acc-Args reductions of up to 12.0% and 14.3%, respectively. Additionally, sensory context perception and thought traces also bring positive benefits.
Performance Across Different Tool Chain Lengths. Although Tab. shows that ContextA- gent outperforms the baselines on ContextAgemtBench, we provide detailed analysis in this section. Specifically, we examine the performance of ContextAgent across samples with varying tool chain lengths, categorizing them into three groups: 0-1 tools (level 1), 2 tools (level 2), and 3-5 tools (level
30

---

Table Results on out-of of-dmmiin experiments.
Pronctive Predictions
Tool Calling
Model
Method
Ace-Pl
MD.
FD.
RMSE
Precision
Recall
Fl-acore
Ace-Args
Pronctive Agent
0.016
0.652
0.542
0.568
Vanilla ICL.
0.779
0.207
0,013
2.253
0.681
0.573
0.600
0.338
GPT-4o
CoT
0.756
0.234
0.010
2.407
0.659
0.565
0.587
0.306
ICL.P
0.909
0.076
0.013
1,514
0.735
0.619
0.648
0.391
ICL-AII
0.886
0.100
0.013
1,701
0.719
0.611
0.639
0.397
Pronctive Agent
0.682
0.010
0.307
1,757
0.478
0,352
0.384
0.126
Vanilla ICL
0.742
0.050
0,207
1.797
0.555
0,412
0.448
0.222
GPr.
CoT
0.802
0.183
0.013
2.152
0.658
0.549
0.576
0.206
ICL-P
0.702
0.000
0.297
1,589
0.521
0,376
0,412
0.207
ICL-AII
0.100
1.452
0.657
0.521
0.555
0.235
Pronctive Agent
0.083
1,806
0.657
0.545
0.573
0.240
Vanilla ICL
0.792
0.090
0.117
1,600
0.627
0.522
0.551
0.330
Qwen2,5-72B-hl
CoT
0.819
0.140
0.040
1.564
0.678
0.583
0.607
0.330
ICL-P
0.882
0.013
0.103
1.199
0.675
0.569
0.595
0.297
ICL-AII
0.929
0.020
0,050
1,036
0.717
0.618
0.642
0.355
Prouctive Agent
0.796
0.107
0.097
1,950
0.659
0.551
0.580
0.292
Vanilla ICT.
0.689
0.020
0.291
1.840
0.445
0.350
0.377
0.221
Llama3.1-70B-Ins
CoT
0.829
0.026
0.143
1,386
0.629
0,489
0.525
0.069
ICL-P
0.725
0.003
0
1.734
0.533
0.423
0.453
0,184
0.070
0.756
0.660
Pronctive Agent
0.571
0.391
0.036
3,040
0.475
0.453
0.459
0.018
Vanilla ICL
0.665
0.254
0.080
2.621
0,483
0.435
0.447
0.085
CoT
0.605
0.347
0,046
2.717
0.516
0.467
0.479
0.076
DeepSeek-RI-
0.489
0.450
ICL-AII
0.639
0.321
0,040
2.637
0.505
0.466
0.476
0.071
Vanilla SFT
0.782
0.087
0.130
1.743
0.585
0.565
0.564
0.418
ConlextAgent
0.893
0.026
0,080
1.249
0.681
0.645
0.648
0.489
Agent
0.147
0.046
0.605
0.556
0.567
0.050
Vanilla ICL.
0.829
0,120
0,050
1,898
0.597
0,532
0.546
0.207
1.836
0.545
0.562
Qwen2.5-7B.
ICL-P
0.886
0.050
0.063
1,584
0.569
0.517
0.530
0.192
ICL-AII
0.890
0.040
0.070
1.502
0.593
0.527
0.543
0.206
Vanilla SFT
0,799
0.077
0.123
1.685
0.607
0.585
0.582
0.433
ContextAgeni
0,909
0.020
0.070
1.172
0.711
0.699
0.689
0.516
Prouctive Agent
0.652
0.013
0.334
1,918
0.403
0.266
0.301
0.139
Vanilla ICI.
0.739
0.163
0.097
2.052
0.565
0.470
0.496
0.210
CoT
0.732
0,220
0.046
1,934
0.623
0.529
0.554
0.247
Llama3. 1-8B-Ins
ICL-P
0.839
0.137
0.023
1.907
0.659
0.558
0.585
0.320
0.553
Vanilla SFT
0.816
0.090
0,093
1.161
0.628
0.619
0.611
0.460
ContextAgent
1,369
0.661
0.641
3), as shown in Tab. 11, Tab. 12, and Tab. 13, respectively. Results show that ContextAgent achieves significantly higher Fi-score, and Acc-Args than the baselines on the level 2 and level 3 samples. For level samples, ContextAgent can achieve 30.3% and 16.0% higher Acc-P and Acc-Args, respectively, when using Qwen2.5-7B-Ing as the base I.IM, validating its strong performance in scenarios that require multiple external tools for proactive services.
We also provide more qualitative results of ContextAgent in Fig. 15 and Fig. 16. Results indicate that ContextAgent initiates the proactive support when it perceives contexts such as the user showing interest in product or putting clothes into a washing machine. Additionally, integrating knowledge from external tools, such online product prices and app-based recommendations, can further enhance the usefulness of these proactive services. Moreover, when ContextAgent detects contexts like leisure time enjoying the sunset or beach waves, or normal daily activities such as walking upstairs, it correctly identifies that no proactive support is needed, thus reducing unnecessary interruptions.
More Qualitative Results.
F
Limitations
Although ContextAgent demonstrates strong performance in the context-aware proactive agent task, we summarize the limitations and future directions for this work as follows. First, the current tool set in ContextAgent includes twenty tools with APIs. The recent emergence of the Model Context
31

---

CV The exchente leoding thetey vind horuing suy neekiion
T.P The user banchoeet ladie care
and
seciii and chsod the...
blus buchgruud and and parable aloveciing produd....
Cill deilgg burncty....
shoate usn mose ber edteosied rextors are
sieatt Due are stine bad caer has (ait sgakel
sdrve Ohen be covenuton cuelit why the diuse
hearphone model. ogoogle sea venn modeue fernin chick deriiledd moeils wntiee dee unciwerr cual inersc moveesantiiee rourpeed
twele bay gradurs who mudtaad, they aoad besoth
<ge AutRE cluc aadtaal
sears
then reralaaoo vncad halp fiom nuragee them oi
complca
dest Thes preddive
NdBd c2hmne can's caltully wouther and wtw.ll/llee/l/ppppp.
X =fanction Calline ww..orddittttt
orlaary ogogle pearctoy
% <hassiknn Callegg
<hitgerr Servicer True
<Triger Servicee Thoe
<Trgper Servicee True
cieanct redroces
oboppessio The die on rocia sa-4k ran crong slac current hmppraiedd Me1 adriebie
Utio ieis ass headitor es are highly thik praviues have wentre the online platorm Accrting is the inker dios vodie ariovsiiee coan for coorts vellool conpuiio
maciire, e. and yeetad ave you woald lik me set timerr reel fanakis hevee lines nan case
trandertions are dppected vir propuedd
ror's cutach right
Figure 15: Qualitative results of ContextAgent in proactive cases, showing sensory and persona contexts, and ContextAgent's thought traces, proactive predictions, tool calls, and final responses.
U Meaad information depita a samee sureet acens. piie iver
The Theee a caln beech sceee with pantle yevess
with a bancial let and whise vall acc par led light color. Rad handdall run stong
wndee whkrr cole frrss bengrued
swn the
tove chiwite lia of beath ans The uane dacing sling
wihee Car aud informtion shows
C shoul
keer entes perartu
state with Centrraul mormtton the cset welting up
thi novert Howetar the a riim Inferratos doet net suppot
poacci exggrriess Withou 250 anal awhral neceraay
white requined The positive eoore nut chiiee
tuat relouiiss Thenlow thoe aet bu proctive lo calte
manpp de fatrari mo vitia
X Wisecson Gaklego Roas.
X oisnccion Galling Noas
X .Passtoo Calling Neee
<Treguer Servicee retre
Service Fatre
Figure 16: Qualitative results of ContextAgent in non-proactive cases.
Protocol (MCP) aims to standardize these external tools, which can be integrated into ContextAgent to further enhance the standardization of tool calls and the diversity of tool types. Second, although ContextAgenlBench currently includes nine life scenarios, it can be further diversified to enhance the ContextAgent's 's capabilities and practicality for daily use.
G Broader Impacts
In this paper, we explore the context-aware proactive LLM agents for the first time, and propose a framework called ContextAgentt. By utilizing the rich contextual information from sensory percep- tions alongside tool-based LLM reasoning, ContextAgent significantly enhances both perception and functionality compared to existing approaches, resulting improved proactive service. Con- textAgent utilizes sensor data from wearable devices such as smart glasses and earphones. This hands-free, egocentric perception not only offers better understanding of the user's conditions and intentions but also reduces both cognitive and physical workload, perfectly aligning with the vision of a proactive assistant. In addition, to bridge the gap in evaluating this new task, we introduce Con- Bext,genteench, the first benchmark to evaluate context-aware proactive L.MM agents. Furthermore, ContextAgent serves as bridge between research on sensory context perception in ubiquitous mobile systems and the emerging LLM agents, thereby opening up new research perspectives and directions. We hope our research will help advance the development of proactive, human-centric Al assistants.
32

---

Table 10: Ablation study on ContextAgemBennch. "woo persona" means the agent does not use persona information during both the training and testing stages. "w/o think" means that the SFT training data does not include the thought process.
Proactive Predictions
Tool Calling
Model
Method
Acc-P
MD.
FD.
RMSE
Precision
Recall
Fl-soore'
Acc-Args'
w/o persona
0.775
0.078
0.146
1.799
0.571
0.531
0.532
0,364
Qwen2.5-7B
w/o think
0.857
0.030
0.112
1.415
0634
0.592
0.599
0.411
ContextAgent
0.894
0.013
0.091
1.264
0.672
0.644
0.645
0.459
w/o persona
0.806
0.081
0.112
1.639
0.611
0.574
0.579
0.405
Jlama3.1-88
w/o think
0.857
0.030
0.112
1.397
0.607
0.612
0.419
ContextAgent 0.874
0.030
0.095
1.408
0.660
0.627
0.626
0.448
persona w/o think ContextAgent
0.109
1.742
0.575
0.409
DeepSeck-RR 1--78
0.884
0.017
0.099
1.355
0.652
0.625
0.625
0.439
0.027
0.085
1,319
0.648
0.468
Table 11: Results for the Level-1 1 samples in ContextAgentBench.
Proactive Predictions
Tool Calling
Model
Method
Acc-pt
MD.
FD.
RMSE
Precision
Recall
Fl-score?
Ace-Args
Pronctive Agent
0.138
0.025
1,983
0.789
0.789
0.789
Vanilla ICL
0.835
0.138
0.025
1.997
0.776
0.784
0.778
0.462
GPT-4o
CoT
0.835
0.128
0.035
1,770
0.775
0.784
0.777
0.529
PCL.P
0,928
0.046
0,025
1.432
0.860
0.866
0.862
0.596
ICII
0.923
0.046
0.030
1.421
0.839
0.851
0.842
0.596
Proactive Agent
0,517
0.005
0.476
2,104
0.467
0.467
0.467
0.065
Vanilla ICL
0.702
0.051
0,246
1,820
0.617
0.620
0.617
0.374
/rurboo/Turb
COT
0.815
0.071
0.112
1.525
0.706
0.717
0.710
0.433
ICL.P
0.733
0.035
0,230
1,872
0.625
0.630
0.627
0.400
0.148
1.542
0.712
Proactive Agent
0.800
0.061
0.069
1.684
0.700
0.702
0.700
0.144
Vanilla ICL
0.835
0,082
0.082
1.353
0.729
0.743
0.734
0.475
Qwen2.5.72B. Ins
CoT
0.856
0.076
0.066
1.254
0.747
0.759
0.751
0.468
0.066
0,806
ICL-AII 0.923
0.035
0.041
0.981
0.806
0.815
0.808
0.474
Pronctive Agent
0.825
0.041
0.133
1.676
0.687
0.692
0.688
0.314
Vanilla ICI.
0.697
0.020
0,282
1.688
0.548
0.553
0.550
0.374
LLama3.1-70B-Ins
CoT
0.194
0.625
0.616
ICL-P
0.784
0.000
0,215
1.529
0.654
0.666
0.658
0.374
0.133
1.176
0.740
Pronctive Agent
0.707
0.225
0.066
2.444
0.671
0.671
0.671
0.019
Vanilla ICL
0.143
0.159
2.215
0.588
0.594
0.590
0.078
CoT'
0.759
0.200
0,041
2.328
0.705
0.707
0.700
0.109
DeepSeek: R1-7B
0.123
0.660
ICL-AII
0.825
0.133
0.041
1.978
0.714
0.717
0.715
0.103
Vanilla SFT
0.779
0.056
0.164
1.783
0.628
0.651
0.635
0.386
ConfexiAgent
0.856
0.015
0.128
1,423
0.712
0.748
0.723
0.468
Prouctive Agent
0.692
0.702
0.695
Vanilla ICI.
0.769
0.087
0.143
1,868
0.642
0.661
0.648
0.326
0.794
0.082
0.123
1,874
0.668
0.676
0.670
0.272
Qwen2.5-7B-Ins
ICL-P
0,805
0.082
0.112
1.845
0.683
0.697
0.688
0,303
ICL-AII
0.851
0.082
0.066
1.724
0.735
0.748
0.739
0.301
Vanilla SFT
0.205
1.905
0.588
0.605
0.593
ContextAgent
0.856
0.005
0.138
1,396
0.701
0.728
0.709
0.459
Proactive Agent
0.533
0.010
0.456
2.169
0.328
0.328
0.328
0.091
Vanilla ICL
0.794
0.117
0.087
1,593
0.707
0.707
0.707
0.234
CoT
0.774
0.117
0.107
1.637
0.659
0.661
0.659
0.257
Llama3. 1-8B-Ins
ICL-P
0.794
0.194
0,010
1,900
0.753
0.753
0.753
0.251
1.379
Vanilla SFT
0.774
0.061
0.164
1.778
0.635
0.671
0.647
0.362
ContextAgent
0.841
0.015
0.143
1,485
0.695
0.733
0.707
0.448
33

---

Table 12: Results for the Level-2 samples in ContextAgentBench.
Proactive Predictions
Tool Calling
Model
Method
Acc-Pt
MD.
FD.
RMSE
Precision
Recall
F1-score?
Acc-Args
Proactive Agent
0.651
0.348
0.000
2.761
0.500
0.255
0.337
0.256
Vanilla ICL
0.674
0.325
0,000
2.676
0.333
0.209
0.249
0.467
GPT44
CoT
0.720
0.279
0,000
2.645
0.472
0.325
0.370
0.534
ICI.P
0.883
0.116
0.000
1.855
0.511
0.290
0.364
0.606
ICL-AII
0.883
0.116
0.000
1.848
0.546
0.314
0.391
0.590
Proactive Agent
0.953
0.046
0.000
1.229
0.581
0.290
0.387
0.070
Vanilla ICL
0.814
0.186
0,000
1,867
0.407
0.220
0.282
0.366
GPu...5.Tuho
CaT
0.790
0.209
0.000
1.994
0.480
0.302
0.359
0.449
ICL-P
0.883
0.116
0.000
1,830
0.534
0,279
0.364
0.408
ICL-AII
0.883
0.116
0.000
1.532
0.492
0.325
0,379
0.472
Proactive Agent
0.930
0.069
0.000
1.758
0.534
0.302
0.379
0.150
Vanilla ICI.
0.883
0.116
0.000
1.509
0.484
0.372
0.405
0.484
Qwen2.5-72B-hh
CoT
0.907
0.093
0.000
1.430
0.511
0.372
0.418
0.478
ICL-P
0.930
0.069
0,000
1,355
0.639
0.476
0.527
0.506
0.479
Pronctive Agent
0.837
0.162
0.000
1,935
0.503
0.302
0.367
0.333
Vanilla ICT.
1.000
0.000
0.000
0.849
0.531
0.395
0.439
0.381
Llama3.1-70B. 1-70B-Ins
CoT
1.000
0.000
0.000
0.902
0.589
0,348
0.427
0.392
ICL-AII
1.000
0.000
0.000
0.876
0.662
0.476
0.532
0.465
Pronctive Agent
0.279
0.720
0.000
3,882
0.058
0.034
0.042
0.021
Vanilla ICL
0.465
0.534
0.000
3.348
0.129
0.
0.110
0.082
0.089
DeepSeck-R I-7B
ICL-P
0.465
0.534
0.000
3,306
0.089
0.058
0.067
0.129
ICL-AII
0.418
0.581
0.000
3,474
0.127
0.081
0.096
0.096
Vanilla SFT
0.860
0.139
0.000
1.486
0.472
0.337
0.377
0.404
ContextAgent
0.930
0.069
0.000
1.181
0.536
0.441
0.464
0.471
Proactive Agent
0.674
0.325
0.000
2.663
0.302
0.220
0.248
0.040
Vanilla ICL
0.093
0.000
1.372
0.461
0.383
0.408
0.324
CoT
0.930
0.069
0.000
1.422
0.418
0.279
0.325
0.270
Qwen2.5-7B. 7Bs
ICL.P 0.860
0.139
0.000
1,861
0.434
0.314
0.351
0.302
ICL-AII
0.883
0.116
0.000
1,758
0.476
0.314
0.368
0.316
Vanilla SFT
0.744
0.255
0.000
1,823
0.412
0.372
0.377
0.419
ContentAent
Projctive Agent
1.000
0.000
0.000
0.940
0.383
0.197
0.259
0.088
Vanilla ICI.
0.558
0.441
0.000
2.583
0.290
0.162
0.205
0.231
CoT
0.674
0.325
0.000
2.151
0.352
0.209
0.255
0.252
Llama3.1
0.294
ICI-AII
0.651
0.348
0.000
2.327
0.383
0.232
0.282
0.291
Vanilla SFT
0.139
0.000
1,599
0.395
0.325
0.348
0.371
ContexiAgeni
0.907
0.093
0.000
1.462
0.503
0.430
0.445
0.468
34

---

Table 13: Results for the Level-3 samples in ContextAgentBench.
Proactive Predictions
Tool Calling
Model
Method
Acc-Pt
MD.
FD.
RMSE
Precision
Recall
F1-score?
Acc-Args
Proactive Agent
0.714
0.285
0.000
2,604
0.523
0.158
0.239
0.253
Vanilla ICL
0.857
0.142
0,000
1.817
0.752
0.332
0.434
0.466
GPT44
CoT
0,910
0,089
0,000
1,564
0.784
0.406
0.504
0.534
ICI-P
0.928
0.071
0.000
1.463
0.794
0.357
0.467
0.598
ICL-AII
0,946
0.053
0.000
1.309
0.794
0.415
0.516
0.598
Proactive Agent
1.000
0.000
0.000
0.845
0.517
0.149
0.231
0.068
Vanilla ICL
0.982
0.017
0,000
1,069
0.702
0.244
0.351
0.377
GPuh..5.Tuho
CaT
0.946
0.053
0.000
1.093
0.647
0.337
0.422
0.440
ICL-P
1.000
0.000
0.000
0,790
0.741
0,305
0,408
0.408
ICL-AII
0.981
0.017
0.000
0.886
0.727
0.374
0.462
0.468
Proactive Agent
0.928
0.071
0.000
1.797
0.669
0.237
0.343
0.144
Vanilla ICI.
0.964
0.035
0.000
1.043
0.738
0.407
0.503
0.482
Qwen2.5-72B-hh
CoT
0.964
0.035
0.000
1,101
0.720
0,451
0.528
0.477
ICL-P
1.000
0.000
0.000
0.790
0.770
0.422
0,518
0.504
Pronctive Agent
0.946
0.053
0.000
1.232
0.595
0.268
0.356
0.318
Vanilla ICT.
1.000
0.000
0.000
0,668
0.738
0.398
0.497
0.381
Llama3.1-708-BB 1-70B-Ins
CoT
1.000
0.000
0.000
0.801
0.758
0.407
0.497
0.392
ICL-AII
1.000
0.000
0.000
0,707
0.668
0,410
0.482
0.460
Pronctive Agent
0.285
0.714
0.000
3,964
0.142
0.048
0.070
0.040
Vanilla ICL
0.607
0.392
0.000
2.991
0.251
0.525
0. 158
0.081
0.238
DeepSeck-R R-7B
ICL-P
0.625
0.375
0.000
2.945
0.363
0. 148
0.203
0.130
ICL-AII
0.500
0.000
3,313
0.279
0.117
0.158
0.098
Vanilla SFT
0.946
0.053
0.000
1,093
0.714
0.467
0.542
0.400
ContextAgent
1.000
0.000
0.000
0.755
0.721
0.444
0.529
0.455
Proactive Agent
0.821
0.178
0.000
2,129
0.392
0.199
0.255
0.039
Vanilla ICL
0.910
0.089
0.000
1,586
0.506
0.263
0.334
CoT
0.892
0.107
0.000
1.742
0.425
0.196
0.258
0.271
Qwen2.5-7B. IBs
ICL.P
0.910
0.089
0.000
1.690
0.488
0.251
0.316
0.306
ICL-AII
0.910
0.089
0.000
1.679
0.410
0. 184
0.246
0.305
Vanilla SFT
0.946
0.053
0.000
1.157
0.731
0.504
0.576
0.417
ContenAAent
Projctive Agent
1.000
0.000
0.000
0.731
0.538
0.177
0.266
0.090
Vanilla ICI.
0.732
0.267
0.000
2.228
0.526
0.174
0.257
0.235
CoT
0.821
0.178
0.000
1,889
0.567
0.225
0.308
0.264
Llama3.1
0.233
ICI-AII
0.875
0.125
0.000
1.752
0.574
0.222
0.306
0.299
Vanilla SFT
0.946
0.053
0.000
1.149
0.620
0.389
0.459
0.369
ConfetiAgent
0.964
0.035
0.000
1,043
0.657
0.406
0.477
0.466
35


Reviews:
[{"review_id": "wyhkEl9lAK", "reviewer": "Reviewer_BDQH", "summary": "This paper introduces ContextAgent for proactive LLM-based agents that leverage rich sensory and persona contexts to provide tool-augmented services without explicit user instructions. By integrating data from wearable devices, ContextAgent autonomously infers user intentions and determines when and how to assist, using external tools accordingly. This paper further propose a benchmark named ContextAgentBench with 1,000 samples across nine daily scenarios to evaluate such agents. Experiments show that ContextAgent significantly outperforms baselines in proactive prediction and tool-calling accuracy.", "questions": "See the weaknesses part.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper presents a compelling approach by leveraging sensory data from wearable devices to enable context-aware, proactive LLM agents. This direction is highly relevant for advancing AI assistants toward more natural, human-centric interactions in real-world environments.\n2. The introduction of ContextAgentBench, a comprehensive benchmark designed specifically for evaluating proactive LLM agents in open-world settings, is a valuable contribution. It provides a structured and reproducible way to assess future systems and can foster further research in proactive agent design.\n\n**Weaknesses**\n1. While the proposed system is practically useful, the core technical components—such as using VLMs for visual understanding and in-context learning for context extraction—largely rely on existing methodologies. The innovation primarily lies in system integration rather than algorithmic advances, which limits the novelty from a technical standpoint.\n2. The paper would benefit from improved clarity and organization. Certain sections are dense and difficult to follow, which may hinder reader comprehension. Enhancing the exposition, especially around key concepts and system design choices, would strengthen the overall readability and impact of the work.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on our submission. We sincerely appreciate the reviewer’s recognition of our contribution, which leverages sensory data from wearable devices to develop context-aware, proactive LLM agents, paving the way for more natural and human-centric AI interactions in real-world settings.\n\n# W1. Contribution of this work.\n# Response:\n- First, we would like to clarify that **our work is the first to study: context-aware proactive agents**. Existing studies on LLM agents primarily adopt a reactive paradigm, which requires explicit user instructions to activate their services. In contrast, we first propose a new research problem aimed at leveraging the **massive contexts surrounding humans, which can be captured through various wearable and mobile devices, to enhance the proactivity of LLM agents**.  The paradigm of context-aware proactive agent proposed in this work needs to map sensory contexts into intention understanding and tool calling, rather than relying on direct user language instructions. Effectively leveraging these contexts surrounding humans to develop more intelligent and **proactive** agents is a promising research direction. This advancement will not only benefit the community but also drive the development of advanced, human-centric, and proactive AI assistants.\n- To address this new research problem, we propose a framework called ContextAgent, where we innovatively model this new task by incorporating both sensory contexts and personas for enhanced proactive LLM reasoning. Additionally, we introduce a comprehensive dataset, ContextAgentBench, the first benchmark for context-aware proactive agent tasks, designed to validate this new research problem. Each sample in our dataset includes multimodal sensory perceptions, sensory contexts, personas, reasoning traces (indicating whether and what proactive services are provided), proactive scores, tool chains, and the final responses. To gain a comprehensive understanding of the rationale behind whether proactive services are needed and what proactive services can benefit the user in the current context, we extract reasoning traces distilled from the latest reasoning LLMs. These reasoning traces not only benefit the training of smaller LLMs but also enhance the explainability of this new task. **We believe that our pioneering work on context-aware proactive agents, along with the developed benchmark, will significantly benefit both the AI and human-centric communities.**\n\nWe hope this clarification accurately reflects the scope of our work and addresses your concerns.\n\n# W2. Concern about the clarity and organization.\n# Response:\nFirst, we will enhance the clarity of the main contributions of this work, as well as the motivation and significance of developing context-aware proactive agents. Second, we will enhance the clarity of the design choices for both the ContextAgent architecture and our benchmark, while refining the overall paper to enhance its exposition and readability. Besides, we will add more examples from our constructed dataset in the appendix to help readers better understand our dataset. We will clearly present each component of the dataset, including sensory contexts, personas, reasoning traces, proactive scores, tool usage, and final responses."}, {"review_id": "8BWPmURHEC", "reviewer": "Reviewer_EoSu", "summary": "This paper introduces a context-aware proactive LLM agent, ContextAgent, along with a benchmark, ContextAgentBench, to evaluate context-aware proactive LLM agents. While prior work have created proactive LLM agents leverage observations from computer interfaces or repeatedly reask questions. ContextAgent leverages sensory information (from wearable devices) and makes uses of user personas to proactively call tools and provide users information. ContextAgent is evaluated using ContextAgentBench by the accuracy of proactive predictions and the tools called to provide proactive feedback.", "questions": "- What do proactive task with a score of 2, 3, and 4 look like?\n  - How were these threshold of proactivity decided for the tasks? In Figure 8 some tasks involve a user working on a computer and the proactiveness score is 1 but I could imagine the user being stuck with an error and requiring some proactive assistance. Similarly for the case where the user is walking by some dried out plants which could prompt looking for water fountains nearby to water the plants.\n- Is this benchmark already saturated given that ContextAgent achieves nearly 90~ accuracy on proactive predictions? What are failure modes and insights in closing the gap?\n- Similar question for tool calling except that it seems there is room for improvement here.\n- For ProactiveBench, why is it necessary to leverage contextual information from wearable devices on computer tasks? For a human using a computer, wouldn't a wearable device need to focus on the task at hand?\n  - If this is the case, would it be possible to evaluate ContextAgent on ProactiveBench? Essentially the sensory information is distilled through the desktop UI and the persona information is distilled into the task instruction", "limitations": "There is a discussion to be had on being context-aware for dangerous situations where the tool calls can contact emergency services for scenarios like (1) witnessing crimes or (2) health-related problems (e.g. fall detection on Apple watches is mentioned in the introduction).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Quality:** Extensive experiments and ablations are conducted to demonstrate the usefulness of ContextAgent making use of sensory information and persona data.\n\n**Clarity:** The paper is fairly easy to understand but there are some improvements that can be made in clarity in the Task Definition section (some terms like TR are mentioned but not defined until later sections). It is also not obvious what a proactive score of 2-4 means.\n\n**Significance:** This is an important problem to study and the paper provides both a solution and benchmark for evaluation.\n\n**Originality:** This paper proposes the first benchmark for context-aware proactive LLM agents.", "was_revised": true, "final_justification": "The authors have addressed my main concern regarding benchmark saturation and minor clarifications on the proactiveness scoring and why evaluating ProactiveBench is out of scope for the paper. I have increased my rating and confidence score since there is still a significant performance gap to close on the Acc-Args metrics that cannot trivially be solved through ICL.", "rebuttal": "# Q1-1. Details about the dataset.\n# Response: \nFirst, we would like to clarify the rationale behind the annotation of proactive scores in our dataset: 5 means a clear need for proactive services, 1 indicates no need, 3 is neutral, and 2 and 4 represent slight tendencies. Below are three examples whose proactive scores are 2, 3, and 4 in our dataset, respectively. We hope they can help you understand our dataset (we will also update more examples in the appendix).\n```json\n# Proactive score of 4\n{\n\"Context information\": \"Visual information shows the user is sitting at a dining table with a plate of food that includes pasta, grilled chicken, and a side of salad.\",\n\"Personas\": [\n\"The user has a history of managing Type 2 diabetes.\",\n\"The user is health-conscious and actively manages their diet to control blood sugar levels.\"\n ],\n\"Thoughts\": \"The user's personas show his glucose level is slightly elevated, and the user might appreciate dietary advice to help stabilize it. Providing this information could be helpful without being intrusive.\",\n\"Proactive score\": 4,\n\"Tools\": \"[{\"name\": \"get_health_data\", \"desc\": \"Get health data from the user's smart device.\", \"params\": \"None\"},{\"name\": \"wikipedia_search\", \"desc\": \"Tool that searches the Wikipedia API. Useful for when you need to answer general questions about people, places, companies, facts, historical events, or other subjects.\", \"params\": {\"query\": \"The effect of pasta, grilled chicken, and a side of salad on stabilizing blood sugar levels.\"}}]\"\n\"Response\": \"Your glucose level appears slightly elevated. To help stabilize it, you might consider increasing the portion of salad or reducing the amount of pasta. Grilled chicken is a great choice for protein, as it has minimal impact on blood sugar levels. Would you like me to provide additional dietary tips or help track your meal for better management?\"\n}\n\n# Proactive score of 3\n{\n\"Context information\": \"Visual information suggests that the user is looking in the mirror while wearing summer clothes, such as short sleeves and shorts.\",\n\"Personas\": [\"A university psychology student who is currently studying creativity and personality.\"],\n\"Thoughts\": \"The user is looking in the mirror and may be preparing to go out, wearing minimal clothing such as short sleeves and shorts. I need to check the weather for the user's location. If the temperature is expected to drop or if rain is forecasted for the day, I can proactively remind the user to dress warmly to avoid catching a cold.\",\n\"Proactive score\": 3,\n\"Tools\": \"[{\"name\": \"get_current_gps_coordinates\", \"desc\": \"Get the current GPS coordinates of the user\", \"params\": \"None\"},{\"name\": \"get_city_weather\", \"desc\": \"Get the weather for a specified city at a given time.\", \"params\": {\"city\": \"results of get_current_gps_coordinates\", \"time\":\"now\"}}]\"\n\"Response\": \"I checked the weather forecast, and the local temperature is expected to drop sharply after 6 PM. Considering that you are only wearing short-sleeved shirts and shorts, it is recommended that you add more layers.\"\n}\n\n# Proactive score of 2\n{\n\"Context information\": \"Visual information suggests the user is in a gym, lifting weights.\",\n\"Personas\": [\"A computer science major interested in developing software for audio manipulation and enhancement.\"],\n\"Thoughts\": \"The user might need guidance on proper weightlifting techniques or reminders to take breaks between sets. However, without detailed persona information, such as their fitness level, I would rate this proactive behavior at 2 points. This rating indicates some potential for proactivity, but the necessity is not very high. After collecting more detailed persona information, I could provide more accurate proactive services.\",\n\"Proactive score\": 2,\n\"Tools\": \"None\",\n\"Response\": \"None\"\n}\n```\n# Q1-2. Discussion on the threshold for proactivity and examples in Fig.8.\n# Response: \n- First, we consistently set the proactivity threshold to 3 across all methods in our experiments. However, in practice, this threshold can be manually adjusted to different levels by users based on their preferences for proactivity. For example, users who prefer highly sensitive proactive services, like for visual impairments, can lower the threshold to 2, while those needing less frequent assistance can raise it to 4. However, in our experiments, we focus solely on evaluating common-sense sensitivity, so the threshold is set to 3 for all experiments. In addition, we will add the definition of the proactivity threshold in Section 3.1 to improve the clarity.\n- Second, we would like to clarify that the proactivity score in working scenarios is not always fixed (i.e., neither as low as 1 nor as high as 5) and varies depending on the specific context. In Fig. 8, the proactive score is 1 because the context (*a user typing on a keyboard while working on a computer in an office*) does not suggest a need for proactivity. However, the score may be higher in scenarios where the user appears confused.\n- Third, regarding the example of watering plants in Fig. 8. In this work, we model the context-aware proactive agent task as leveraging both sensory contexts and personas for reasoning. **This means that even with identical sensory contexts, different personas in the samples may result in varying proactive scores and tool usage**. \nFor example, in our developed ContextAgentBench, two samples share the same sensory context: *\"The user is in a gym, standing in front of a treadmill.\"* However, they differ in personas: *\"The user is new to fitness and does not exercise regularly,\"* and *\"The user exercises frequently.\"* For the first sample, the annotated output is: *\"Would you like a personalized treadmill workout plan? I can also suggest a motivational playlist to keep you energized. Just let me know!\"* In contrast, for the second sample, the proactive score is 1, indicating no need to disturb the user.\n# Q2 and Q3. Discussion on the potential saturation of the benchmark and strategies to close the performance gap.\n# Response:\n- First, our experimental results in Tab.4~6 show that increasing the model size of LLMs can lead to obvious performance improvement on both proactive predictions and tool calling. \n- Additionally, **we conducted new experiments using the latest reasoning LLMs** (e.g., Claude and GPT-o3 series) in both zero-shot (ZS) and in-context learning (ICL) settings. Results show that advanced reasoning LLMs like Claude Opus 4-ICL can lead to improvements in Acc-P and F1 scores. This shows that enhancing the reasoning capabilities of LLMs has the potential to close this gap. \nHere are the results:\n|  | Acc-P| MD | FD| RMSE    | Precision | Recall   | F1 | Acc-Args |  \n|-|-|--|-|--|--|--|-|-|  \n| o4-mini (ZS)| 70.5%   | 1.73% | 27.6%  | 1.57 | 63.1% | 53.5%  | 55.9% | 42.4% |  \n| o4-mini (ICL) | 86.1%   | 3.4% | 10.3%  | 1.24 | 72.6% | 66.8%  | 68.2% | 53.8% |\n| GPT-o3 (ZS) | 83.0%   | 9.0% | 7.9% | 1.34 | 71.6%  | 64.9%  | 66.7% | 39.7% |  \n| GPT-o3 (ICL)  | 86.8%   | 6.9%| 6.2% | 1.10 | 75.5% | 69.7%  | 71.1% | 56.3% |  \n| Claude Sonnet 4 (ZS)| 83.7% | 11.0%| 5.1% | 1.34  | 75.7%  | 69.0% | 70.8%  | 23.7% |  \n| Claude Sonnet 4 (ICL) | 91.3% | 6.9% | 1.7% | 1.01 | 77.5% | 79.9%| 77.3% | 48.0% |  \n| Claude Opus 4 (ZS) | 83.0%| 7.2% | 9.6% | 1.27| 73.8% | 66.0% | 68.0% | 24.4% |  \n| Claude Opus 4 (ICL) | 93.7% | 4.1% | 2.0% | 0.86 | 81.4% | 85.6% | 81.8% | 56.8% |    \n# Q4. Discussion on the necessity of leveraging contextual information from wearable devices for computer tasks.\n# Response:\n- First, **opportunities for proactive assistance in computer usage or working scenarios can arise from many different perspectives**. Previous work on Proactive Agents and their proposed benchmark primarily focuses on scenarios related to computer usage, using inputs such as mouse clicks and keyboard types. However, **our work primarily focuses on leveraging rich sensory contexts surrounding humans**, such as vision, audio, and IMU data, rather than relying on computer inputs. These richer contexts surrounding humans, captured through various wearable and mobile devices, create opportunities to enhance LLM agents' proactivity. For example, from a **health perspective**, users can be proactively alerted with health-related suggestions, such as reminders to take breaks or engage in physical activity after sitting for extended periods, based on data from IMU sensory contexts. Additionally, in a \"**computer usage scenario during an in-person meeting**,\" wearable devices can offer work-related tips, such as using audio sensory data to identify potential scheduling conflicts during discussions about future plans.\n- Second, the wearable devices in our benchmark are not designed to detect computer inputs, such as mouse clicks or keyboard strokes. Instead, they are used to capture sensory contexts surrounding humans, enabling more opportunities for proactive assistance during computer usage, such as health alerts or work-related tips during meetings. Furthermore, our proposed ContextAgent and benchmark enable the development of a proactive agent that extends beyond computer-centric scenarios, offering a wider range of opportunities for proactive assistance.\n\n# L1. Discussion on contacting emergency services.\n# Response: \nWe will add more discussion on context awareness in dangerous situations, where the tool calls can contact emergency services for specific scenarios."}, {"review_id": "tFJXBxneMQ", "reviewer": "Reviewer_HAmH", "summary": "This paper proposed a benchmark for evaluating context-aware proactive LLM agents and an author-designed agent as baseline model. The baseline agent is a context-aware proactive agent that fuses rich sensory data from video, audio with a user’s historical persona to infer intentions and decide when help is needed, then autonomously invokes the right tools to assist unobtrusively. The proposed ContextAgentBench contain nine everyday scenarios and twenty tools. On this benchmark, ContextAgent surpasses prior baselines.", "questions": "Q1. Why don't you use VLM as the main agent directly?     \n\nQ2. About W2. Discuss the situation when some external tools are not aviliable, can your agent still work?        \n\nQ3: ContextAgent is a multi-tool agent in this stage, why don't you consider code agent for this problem.", "limitations": "They discuss limitations in the supp.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "S1. The authors proposed a personalized agentic framework, which match well with the community's interests.    \n\nS2. They tested the ContextAgent with open-sourced LLMs (including small LLMs), which is useful for re-implementation.     \n\nS3. The main idea is simple and clear and also contain OOD experiment.           \n\nW1: Comparison might be unfair in Table 1 because some of other ideas are zero-shot method. Only Vanilla SFT and your proposed idea utilize the training data.      \n\nW2: Do not contain missing sensor situation. e.g., the input data of some sensor is not aviliable, what might be the context output?    \n\nW3: For 5.4 Out-of-Domain Evaluation, it would be good to compare your method with the latest GPT/Claude series like o3.", "was_revised": true, "final_justification": "Based on authors' response and our discussion, I update my score to borderline.", "rebuttal": "# W1:  Concern about the tested baselines in Table 1.\n# Response:   \n- First, we would like to clarify that **our work is the first to study: how to build and evaluate a context-aware proactive agent**. Existing studies on LLM agents primarily adopt a **reactive** paradigm, which requires explicit user instructions to activate their services. In contrast, **we first propose a new research problem aimed at leveraging the massive contexts surrounding humans, which can be captured through various wearable and mobile devices, to enhance the proactivity of LLM agents**.  The paradigm of context-aware proactive agent proposed in this work needs to map sensory contexts into intention understanding and tool calling, rather than relying on direct user language instructions. This advancement will not only **benefit the community** but also drive the **development of advanced, human-centric, and proactive AI assistants**.\n- Second, **to the best of our knowledge, no prior work can be directly used as a baseline for this new task**. The most closely related work is Proactive Agent, which focuses on enclosed computer usage scenarios and does not leverage open-world sensory contexts or perform tool calling. To ensure the fairness of comparison, besides Proactive Agent, ICL-based and vanilla SFT, **we add two baselines for comparison: SFT-Persona and SFT-PCE. Both methods use all the training data for SFT**. Beyond vanilla SFT, SFT-Persona employs persona information as input. SFT-PCE further adopts the proactive-oriented context extraction method used by ContextAgent. Here are the results:\nModel|Setting|Acc-P|MD|FD|RMSE|Precision|Recall|F1|Acc-Args|\n|-|-|-|-|-|-|-|-|-|-|\nQwen2.5-7B-Ins|SFT-Persona|77.4%|8.3%| 14.3%|1.79|48.1%|49.5%| 47.3%| 37.4%|\n||SFT-PCE|79.7%|6.6%|13.7%|1.72|59.3%|59.8%|57.2%|40.4%|\n||ContextAgent|85.0%| 5.0%|10.0%|1.40|66.7%|61.5%|62.4%|47.9%|\nLlama3.1-8B-Ins|SFT-Persona|73.4%|11.5%|15.1%|1.98|55.5%|58.2%|55.2%| 35.3%|\n||SFT-PCE|78.6%|3.6%|17.9%|1.83| 58.1%|56.3%|55.5% |41.8%|\n||ContextAgent|83.0%|7.0%|10.0%|1.51|68.7%|63.7%|64.5%|48.6%|\nDeepSeek-R1-7B|SFT-Persona|76.2%|13.1%|10.7%|1.93|46.3%|46.5%|45.6%|22.8%|\n||SFT-PCE|77.4%|16.7%|6.0%|1.81|54.6%|55.0%|53.5%| 28.0%|\n||ContextAgent|84.0%|5.0%|11.0%|1.51|67.8%|64.1%|64.8%|48.7%|\n\nContextAgent employs both proactive-oriented sensory contexts, personas, and reasoning traces distilled from advanced LLMs for enhanced SFT, **consistently achieving higher performance than other personas and SFT-based baselines**. We hope this clarification accurately reflects the scope of our work and addresses your concerns.\n# W2: Performance on missing sensor situation.\n# Response: \nHere is an example where the input data from certain sensors is unavailable. Consider a context that originally includes both vision and audio: *\"The person is engaging in a conversation with a colleague in an office. The content is: Hi, how about going to City A this weekend? I want to go to the beach.\"* When the audio sensor is missing, the context output is: *\"The person is engaging in a conversation with a colleague in an office.\"* When the visual sensor is missing, the context is reduced to: *\"Hi, how about going to City A this weekend? I want to go to the beach.\"*\n\nTo clearly show the sensor data's impact, we **added new experiments on missing different sensor modalities**. Results show that missing either the vision or audio leads to performance degradation for both proactive prediction and tool calling. The impact of missing vision is more significant than that of missing the audio. Here are the results.\n|| Acc-P|MD|FD|RMSE|Precision|Recall|F1|Acc-Args|\n|-|-|-|-|-|-|-|-|-|\n|Full|88.8%|2.7%|8.5%|1.31|67.6%|64.8%|64.7%|46.8%|\n|Miss vision|70.9%|12.9%|16.3%|2.07|42.4%|42.2%|41.4%|16.3%|\n|Miss audio|72.0%|17.3%|10.6%|2.02|50.1%|50.1%|49.3%|21.2%|\n# W3: Comparison with the latest reasoning LLMs.\n# Response: \nWe **added new experiments using advanced reasoning LLMs** in both zero-shot (ZS) and ICL settings. Results show that among these reasoning models, Claude Opus 4 achieves the highest scores in both proactive predictions and tool calling. In addition, ContextAgent shows comparable performance to advanced reasoning models for Acc-P, while its tool-calling performance, such as F1, is lower compared to Claude Opus 4. It may be due to the gap in model size, as ContextAgent uses a 7B-scale LLM.\n|Model|Acc-P|MD|FD| RMSE|Precision|Recall|F1| Acc-Args|\n|-|-|-|-|-|-|-|-|-|\n|o4-mini (ZS)|70.5%|1.7%|27.6%|1.57|63.1%|53.5%|55.9%|42.4%|\n|o4-mini (ICL)|86.1%|3.4%|10.3%|1.24|72.6%|66.8%|68.2%|53.8%|\n|GPT-o3 (ZS)|83.0%|9.0%|7.9%|1.34|71.6%|64.9%|66.7%|39.7%|\n|GPT-o3 (ICL)|86.8%|6.9%|6.2%|1.10|75.5%|69.7%|71.1%|56.3%|\n|Claude Sonnet 4 (ZS)|83.7%|11.0%|5.1%|1.34|75.7%|69.0%|70.8%|23.7%|\n|Claude Sonnet 4 (ICL)|91.3%|6.9%|1.7%|1.01|77.5%|79.9%|77.3%|48.0%|\n|Claude Opus 4 (ZS)|83.0%|7.2%|9.6%|1.27|73.8%|66.0%|68.0%| 24.4%|\n|Claude Opus 4 (ICL)|93.7%|4.1%|2.0%|0.86|81.4%|85.6%|81.8%| 56.8%|\n|ContextAgent (Qwen2.5-7B-Ins)|90.9%|2.0%|7.0%|1.17|71.1%|69.9%|68.9%|51.6%|\n# Q1. Performance when using VLM as the main agent.\n# Response:\nFirst, we would like to clarify that we chose LLMs over VLMs because they use a unified text format to represent contexts across all sensor modalities, making them more scalable to new types of sensors. This modular design enhances the framework’s scalability, enabling seamless integration with various sensor modalities (e.g., IMU, video, sound) and emerging MLLMs. However, the ContextAgent framework also supports using other MLLMs as the main agent, such as VLM. We **added new experiments using VLMs as the main agent** in ContextAgent, while keeping all other components of the framework exactly the same. We tested various VLMs across different model sizes. Here are the results.\n|Model|Acc-P|MD|FD|RMSE|Precision|Recall|F1|Acc-Args|  \n|-|-|-|-|-|-|-|-|-|\n|Qwen2.5-VL-7B-ICL|64.2%|8.3%|27.3%|2.18|35.9%|32.5%|32.3%|26.8%|\n|InternVL3-8B-ICL|60.7%|1.1%|38.1%|2.02|21.8%|24.3%|21.6%|19.4%|\n|LLaVA1.5-7B|69.0%|27.3%|3.5%|2.16|54.3%|54.0%|53.3%|29.7%|\n|Qwen2.5-VL-3B |83.3%|10.7%|5.9%|1.58|60.3%|59.5% |58.6%|36.7%|\n|Qwen2.5-VL-7B|84.5%|10.7%|4.7%|1.56|71.0%|68.0%|68.4%|51.1%|\n|InternVL3-2B|85.7%|14.2%|0.0%|1.44|67.8%|68.3%|67.1%|45.8%|\n|InternVL3-8B|84.5%|10.7%|4.7%|1.42|69.4%|71.0%|68.7%|54.3%|\n|Qwen2-VL-2B|75.0%|20.2%|4.7%|1.93|54.9%|55.1%|54.2%|36.0%|\n|Qwen2-VL-7B|80.9%|13.1%|5.9%|1.55|66.5%|67.0%|65.5%|47.0%|\n\nWe conducted VLM experiments under both ICL and SFT settings. Results show that using VLM as the main agent perform significantly worse than LLM-based agents in ICL settings. This is because providing examples of the entire process in the prompt is more challenging for VLMs to understand compared to using a two-stage pipeline. However, in SFT settings, VLM as the main agent performs comparably to LLM-based agents. We will add the results to the appendix.\n# Q2. Discussion on external tools unavailable.\n# Response:\nWe would like to clarify that ContextAgent can still work when certain tools are unavailable. Here are two examples.\n```json  \n# Example 1:\n{\n\"Contexts\":\"The user is browsing wireless earbuds in the electronics section.\",\n\"Thoughts\":\"The user might be interested in wireless earbuds. Providing price comparisons and review information could help with their purchase decision.\",\n\"Proactive score\":5,\n\"Tools\":[{\"name\":\"get_online_product_price\",\"desc\":\"Get the price of a product from an online store such as Amazon or eBay\", \"params\":{\"product_name\":\"wireless earbuds\"}},{\"name\":\"search_rednote\",\"desc\":\"A platform where people share tips on travel, fitness..\",\"params\": {\"query\":\"wireless earbuds reviews and recommendations\"}}] \n\"Response\":\"It looks like you're exploring wireless earbuds. Would you like help comparing prices and reviews to find the best deal?\"\n}\n# Example 2:\n{\n\"Contexts\":\"The user is sitting at a dining table with a plate of food that includes pasta, grilled chicken, and a side of salad.\",\n\"Personas\":[\n\"The user has a history of managing Type 2 diabetes.\",\n\"The user is health-conscious and actively manages their diet to control blood sugar levels.\"\n ],\n\"Thoughts\":\"The user is managing Type 2 diabetes and is health-conscious, providing nutritional insights about their meal could support better blood sugar control and dietary decisions.\",\n\"Proactive score\":4,\n\"Tools\":\"[{\"name\":\"get_health_data\",\"desc\":\"Get health data from the user's smart device.\",\"params\":\"None\"},{\"name\":\"wikipedia_search\", \"desc\":\"Tool that searches the Wikipedia API.\", \"params\": {\"query\":\"The effect of pasta, grilled chicken, and a side of salad on stabilizing blood sugar levels.\"}}]\"\n\"Response\":\"Your meal of pasta, grilled chicken, and salad is a solid choice. To support Type 2 diabetes management, control pasta portions to avoid blood sugar spikes, while the chicken’s protein and salad’s fiber help slow carbohydrate absorption.\"\n}\n```\nResults show that ContextAgent can still reason on the current contexts and identify the need for proactive assistance. When tools are unavailable, ContextAgent can either proactively offer users helpful suggestions and ask if they would like to use certain tools, or, in some cases, rely on LLM's intrinsic knowledge to assist users. \n# Q3. Discussion on the code agent.\n# Response: \nFirst, we would like to clarify that ContextAgent and its benchmark are implemented using an API-based framework, a widely adopted paradigm in prior research. However, ContextAgent is not limited to an API-based approach and can be extended to support other implementation methods, such as code-based frameworks. Many existing works on LLM agents primarily focus on pipeline, reasoning, or implementations (e.g., API-based, code-based, or MCP-based), but they are within a reactive paradigm. However, our work is **orthogonal** to theirs. Unlike existing reactive LLM agents that rely on explicit user instructions, we focus on designing context-aware proactive agents that leverage rich sensory data."}, {"review_id": "5Ya2x7xV1p", "reviewer": "Reviewer_D7kT", "summary": "This paper introduces ContextAgent, a context-aware proactive agent designed to operate under open-world sensory conditions. The authors propose two new benchmark, contextagentbenchmark and lite version, which combines LLM-assisted generation with human curation. The benchmark includes rich multimodal information such as egocentric video, audio, reasoning traces, and user personas. Experimental results demonstrate that ContextAgent achieves SOTA performance compared to existing LLMs on this benchmark.\n\nHowever, the paper suffers from unclear descriptions in the task definition, task construction, and experimental comparisons, which significantly affects readability and makes it difficult to fully follow the methodology.", "questions": "1. In line 133, what exactly is the definition of $TR$?\n2. Why does the vanilla SFT baseline not include persona information during training? This seems to create an imbalance in the comparison against ContextAgent.\n3. What is the composition of the D_SFT training data? Please provide more details on the dataset size, construction. etc.\n4. What does the persona data look like in practice? Can you provide concrete examples to clarify the structure and content of persona annotations?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed ContextAgentBench dataset is well-motivated and addresses a practical need for evaluating proactive LLM agents in open-world, multimodal settings.\n2. The authors conduct extensive experiments across 10 different LLMs, including both proprietary and open-source models.\n3. The paper includes clear and informative figures that effectively illustrate the system architecture, data structure, and experimental results. \n\nWeakness：\n1. In Section 3.1 Task Definition and Task Construction, many variables are introduced but inconsistently notated (e.g., $P_S$  vs. PS) and defined out of order across paragraphs. Key terms such as the trigger threshold TR are underexplained, and the overall formulation lacks a clear separation between inputs, outputs, and constants—making the method harder to follow and reducing its interpretability.\n2. Although the authors conduct extensive experiments, I find the experimental setup somewhat problematic. Specifically, ContextAgent is trained via supervised fine-tuning, yet it is compared against several training-free baselines (e.g., ICL variants), where the control over input information appears loosely related to the actual task setting (sft). Additionally, the vanilla SFT baseline does not include persona information as input, which results in an unfair comparison due to missing critical context.\n3. The proposed agent does not demonstrate clear structural or architectural novelty. It primarily builds on existing LLMs and applies domain-specific supervised fine-tuning.\n4. While the paper mentions that ContextAgent is fine-tuned using supervised learning, it does not provide sufficient details about the training data $D_SFT$.\n5. The paper does not provide concrete examples from the dataset, nor does it release the dataset itself. As a result, it is unclear what the persona annotations look like, or how representative they are.\n\nI may have missed certain details, and I am open to adjusting my score based on the authors’ response and clarifications.", "was_revised": true, "final_justification": "The author's reply answered some of my questions, so I raised the score. However, I think the author's revisions need to be added to the main text to make the article more complete.", "rebuttal": "# W1 and Q1. Concern about the task formulation and notations.\n# Response:\n- We would like to clarify that this is a typo, which should be $\\mathcal{P_S}$, i.e., the proactive score. We will correct this typo by changing $\\mathcal{PS}$ to $\\mathcal{P_S}$ and thoroughly proofread the entire paper to ensure consistency in notation and a logical order of variable definitions throughout.\n- $TR$ is the threshold for the proactive trigger, as defined in Sec.4.2. To improve the clarity, we use $\\theta$ instead of $TR$ to represent this threshold. When the agent's predicted proactive score $\\mathcal{P_S}$ exceeds $\\theta$ (i.e., $\\mathcal{P_S} > \\theta$), the agent proactively provides suggestions to the user. Otherwise, ContextAgent avoids interrupting the user. In practice, this threshold can be manually adjusted to different levels by users based on their preferences for proactivity. For example, users who prefer highly sensitive proactive services, like for visual impairments, can lower the threshold to 2, while those needing less frequent assistance can raise it to 4. In our experiments, since we focus on evaluating common-sense sensitivity, the threshold is set to 3. We will add more descriptions of the trigger threshold $\\theta$ in Sec.3.1.\n- To enhance the clarity of constants, we replace the threshold $TR$ with $\\theta$, and use $\\textbf{T}$ to represent the tool set that the agent can use. To better illustrate the inputs, outputs, and constants, we provide a table as follows:\n|Notation|Definition|Function|  \n|-|-|-|  \n|$\\mathcal{S}$|Sensory perceptions|Inputs|  \n|$\\mathcal{P}$|Personas|Inputs|  \n|$\\mathcal{T}$|Thought traces|Outputs|  \n|$\\mathcal{P_S}$|Proactive score|Outputs|  \n|$\\mathcal{T_C}$|Tool chains|Outputs|  \n|$\\mathcal{R}$|Response|Outputs| \n|$\\theta$|Proactive trigger threshold|Constants| \n|$\\mathbf{T}$|Tool set|Constants| \n\nWe hope this clarification can address your concerns. We will use clearer notations to enhance the overall formulation in Sec.3.1.\n# W2, W3 and Q2. Concerns about the experimental setups and the contribution of this work.\n# Response:\n- First, we would like to clarify that **our work is the first to study: context-aware proactive agents**. Existing studies on LLM agents primarily adopt a **reactive** paradigm, which requires explicit user instructions to activate their services. **In contrast, we first propose a new research problem aimed at leveraging the massive contexts surrounding humans, which can be captured through various wearable and mobile devices, to enhance the proactivity of LLM agents**.  The paradigm of context-aware proactive agent proposed in this work needs to map sensory contexts into intention understanding and tool calling, rather than relying on direct user language instructions. Effectively leveraging these contexts surrounding humans to develop more intelligent and **proactive agents** is a promising research direction. This advancement will not only **benefit the community** but also drive the **development of advanced, human-centric, and proactive AI assistants**.\n- Second, to address this new research problem, we propose a framework called ContextAgent, where we innovatively model this new task by incorporating sensory contexts, personas, and the reasoning traces distilled from advanced LLMs for enhanced proactive reasoning. Additionally, we introduce ContextAgentBench, the first benchmark for context-aware proactive agent tasks, designed for this new research problem. Each dataset sample includes multimodal sensory perceptions, sensory contexts, personas, reasoning traces (indicating whether and what proactive services are provided), proactive scores, tool chains, and final responses. To understand the rationale behind the need for proactive services and their potential benefits, we extract reasoning traces distilled from the latest reasoning LLMs. These reasoning traces not only benefit the training of smaller LLMs but also enhance the explainability of this new task. **We believe our pioneering work on context-aware proactive agents and the developed benchmark will greatly benefit the communities.**\n- Third, **to the best of our knowledge, no prior work can be directly used as a baseline for this new task**. The most closely related work is Proactive Agent, which focuses on enclosed computer usage scenarios and does not leverage open-world sensory contexts or perform tool calling. Besides Proactive Agent, ICL-based and vanilla SFT, **we add two baselines for comparison: SFT-Persona and SFT-PCE. Both methods use all the training data for SFT (including personas)**. Beyond vanilla SFT, SFT-Persona employs persona information as input. SFT-PCE further adopts the proactive-oriented context extraction method used by ContextAgent. Here are the results:\nModel|Setting|Acc-P|MD|FD|RMSE|Precision|Recall|F1|Acc-Args|\n|-|-|-|-|-|-|-|-|-|-|\nQwen2.5-7B-Ins|SFT-Persona|77.4%|8.3%|14.3%|1.79|48.1%|49.5%|47.3%|37.4%|\n||SFT-PCE|79.7%|6.6%|13.7%|1.72|59.3%|59.8%|57.2%|40.4%|\n||ContextAgent|85.0%|5.0%|10.0%|1.40|66.7%|61.5%|62.4%|47.9%|\nLlama3.1-8B-Ins|SFT-Persona|73.4%|11.5%|15.1%|1.98|55.5% |58.2%|55.2%|35.3%|\n||SFT-PCE| 78.6%|3.6%|17.9%|1.83|58.1%|56.3%|55.5%|41.8%|\n||ContextAgent|83.0%|7.0%|10.0%|1.51|68.7%|63.7%|64.5%|48.6%|\nDeepSeek-R1-7B|SFT-Persona|76.2%|13.1%|10.7%|1.93|46.3%|46.5%|45.6%|22.8%|\n||SFT-PCE|77.4%|16.7%|6.0%|1.81|54.6%|55.0%|53.5%|28.0%|\n||ContextAgent|84.0%|5.0%|11.0%|1.51|67.8%|64.1%|64.8%|48.7%|\n\nContextAgent employs both proactive-oriented sensory contexts, personas, and reasoning traces distilled from advanced LLMs for enhanced SFT, **consistently achieving higher performance than other personas and SFT-based baselines**. We hope this clarification accurately reflects the scope of our work and addresses your concerns.\n# W5 and Q4. Dataset details.\n# Response:\nHere are examples from our dataset, including sensory contexts, personas, thought traces, proactive scores, and tool calling. These personas enhance our benchmark's practicality, as individuals with varying personas may require different levels of proactivity, even experiencing the same sensor contexts. Reasoning traces (rationale behind the need for proactive services and their potential benefits) are distilled from the latest reasoning LLMs. They not only benefit the training of smaller LLMs but also enhance the explainability of this new task. We will release the code and the benchmark in the final version. We hope they can help you understand our dataset (we will update more examples in the appendix). \n```json\n{\n\"Context information\":\"Visual information shows the user is sitting at a dining table with a plate of food that includes pasta, grilled chicken, and a side of salad.\",\n\"Personas\":[\n\"The user has a history of managing Type 2 diabetes.\",\n\"The user is health-conscious and actively manages their diet to control blood sugar levels.\"\n ],\n\"Thoughts\":\"The user's personas show his glucose level is slightly elevated, and the user might appreciate dietary advice to help stabilize it. Providing this information could be helpful without being intrusive.\",\n\"Proactive score\":4,\n\"Tools\": \"[{\"name\":\"get_health_data\",\"desc\":\"Get health data from the user's smart device.\",\"params\":\"None\"},{\"name\":\"wikipedia_search\",\"desc\":\"Tool that searches the Wikipedia API.\",\"params\": {\"query\":\"The effect of pasta, grilled chicken, and a side of salad on stabilizing blood sugar levels.\"}}]\"\n\"Response\":\"Your glucose level appears slightly elevated. To help stabilize it, you might consider increasing the portion of salad or reducing the amount of pasta. Grilled chicken is a great choice for protein, as it has minimal impact on blood sugar levels. Would you like me to provide additional dietary tips or help track your meal for better management?\"\n}\n```\n# W4 and Q3. Details about the training data.\n# Response:\nWe construct $D_{SFT}=\\{(\\mathcal{X},\\mathcal{T},\\mathcal{Y})\\}$, where inputs $\\mathcal{X}$ include sensory context and personas, $\\mathcal{T}$ is thought traces, and outputs $\\mathcal{Y}$ include proactive scores and tool chains. Details can be seen in Sec.4.2. ContextAgentBench contains 1000 samples, with 60% of the samples used to build $D_{SFT}$. Here are examples from the SFT dataset.\n```json\n{\n\"Instruction\":\"You are a proactive LLM agent. You will receive both sensory and persona contexts. The sensory contexts are captured by the multi-modal sensors on the user's wearable devices. Persona is the personal information like identity and preference. Based on these contexts, you need to generate a thought and determine whether to initiate a proactive service for the user or not. When generating the response, consider whether you need to call external tools to complete the task. The following are the tool names and their arguments. Please only select tools from the specified tool sets and make sure to use the correct format for their arguments.## Tool set: {tool_set}\",\n\"input\":\"Sensory Context:Visual information shows the user is sitting at a dining table with a plate of food that includes pasta, grilled chicken, and a side of salad. Persona Context:The user has a history of managing Type 2 diabetes. The user is health-conscious and actively manages their diet to control blood sugar levels.\",\n\"output\":\"<think>The user's personas show his glucose level is slightly elevated, and the user might appreciate dietary advice to help stabilize it. Providing this information could be helpful without being intrusive.<\\think> ## Proactive Predictions\\n  \"Proactive score\":4  ## Tool Calling: [{\"name\":\"get_health_data\", \"desc\":\"Get health data from the user's smart device.\", \"params\":\"None\"},{\"name\":\"wikipedia_search\", \"desc\":\"Tool that searches the Wikipedia API.\",\"params\":{\"query\":\"The effect of pasta, grilled chicken, and a side of salad on stabilizing blood sugar levels.\"}}]\"\n}\n```\nWe hope they can help you understand our SFT dataset. We will add more details about its construction in Sec.4.2."}]

Author rebuttal:

Meta-review: This paper receives three positive scores and one negative score. Overally speaking, the positive points outweigh the negative points. Thus, I recommend accepting this paper.
Decision: Accept (poster)