Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance

## Abstract

Abstract Agents powered by large language models have shown remarkable abilities in solving complex tasks. However, most agent systems remain reactive, limiting their effectiveness in scenarios requiring foresight and autonomous decision-making. In this paper, we tackle the challenge of developing proactive agents capable of anticipating and initiating tasks without explicit human instructions.
We propose a novel data-driven approach for this problem. Firstly, we collect real-world human activities to generate proactive task predictions. These predictions are then labeled by human annotators as either accepted or rejected. The labeled data is used to train a reward model that simulates human judgment and serves as an automatic evaluator of the proactiveness of LLM agents.
Building on this, we develop a comprehensive data generation pipeline to create a diverse dataset, ProactiveBench, containing 6 , 790 6 790 6,790 events.
Finally, we demonstrate that fine-tuning models with the proposed ProactiveBench can significantly elicit the proactiveness of LLM agents.
Experimental results show that our fine-tuned model achieves an F1-Score of 66.47 % percent 66.47 66.47\% in proactively offering assistance, outperforming all open-source and close-source models.
These results highlight the potential of our method in creating more proactive and effective agent systems, paving the way for future advancements in human-agent collaboration. 1 1 1 Our code and data are available at https://github.com/thunlp/ProactiveAgent .

## 1 Introduction

Figure: Figure 1: Comparison of agent systems with two types of human-agent interaction. Reactive agents passively receive user queries and then generate responses. Proactive agents infer tasks based on environmental observations and propose possible assistance requests accordingly.
Refer to caption: /html/2410.12361/assets/x1.png

The emergence of large language models (LLMs) like ChatGPT [] has significantly advanced the development of autonomous agent [, , , ]. These LLM-based agents can understand human instructions, make plans, explore environments, and utilize tools to solve complex tasks [, ] and have shown substantial potential in various applications such as robotics [], personal assistants [], and process automation [].

Currently, most existing LLM-based agents predominantly work in the reactive paradigm: they require explicit human instructions to initiate task completion and remain dormant in terms of providing services until prompted by user instructions [].
This paradigm limits their capacity for proactive assistance and autonomous service provision in the absence of direct human instructions.
We argue that LLM-based agents should be proactive, capable of autonomously initiating tasks by understanding and responding to their environment.
For instance, as illustrated in [Figure 1](#S1.F1), the reactive agent should wait for explicit instructions from the user to execute tasks such as “show unread emails" or “schedule a meeting with John".
In contrast, a proactive agent would predict its task automatically by noticing an email from John suggesting a meeting and automatically offering to schedule it.
This ability of context awareness [] enables the proactive agent to interpret signals and proactively propose and execute tasks without explicit human instructions.
Thus, it not only significantly reduces the cognitive burden on the user but also identifies latent needs not explicitly articulated by humans.
Consequently, the proactive agent could provide more comprehensive and seamless services to the user.

In this work, we propose a novel data-driven formalization for developing a proactive agent that anticipates user needs and takes the initiative by suggesting tasks or providing information without explicit requests.
Our approach centers on constructing ProactiveBench, allowing us to evaluate and enhance the agent’s proactive behavior.
Firstly, we collect real-world human activity data in three settings: coding, writing, and daily life.
This includes but is not limited to, user keyboard and mouse inputs, clipboard content, browser activity, etc.
Then, we build an LLM-driven gym to generate events that reflect the raw real-world contexts we collected.
We obtain a total of $233$ events across $12$ scenarios as the test set of the ProactiveBench.
To further refine the proactive behavior of the LLM-based agent, we construct various events and proactive tasks under synthetic contexts with the gym.
By iterative generating more events and predictions, we obtain up to $6,790$ events as the train set of the ProactiveBench, as shown in [Table 1](#S1.T1).
We fine-tune the LLaMA-3.1-8B-Instruct [] and the Qwen2-7B-Instruct [] on this training set to refine their proactive behavior.

**Table 1: Statistics of the ProactiveBench. Our dataset contains three distinct settings: Coding, Writing, and Daily Life. The dataset contains a total of $6,790$ events in the training set and 233 events in the test set.**
| Settings | Scenario Count | Events Count |
| --- | --- | --- |
| Coding | 46 | 2,275 |
| Writing | 46 | 2,354 |
| Daily Life | 44 | 2,161 |
| Train Set | 136 | 6,790 |
| Test Set | 12 | 233 |

To automatically evaluate the proactiveness of LLMs, we train a reward model that achieves up to $91.80\%$ consistency with human judgments in terms of F1-Score, serving as an evaluator.
Using the reward model, we compared the performance of different language models on ProactiveBench. The results indicate that even the latest open-source models struggle to effectively predict proactive tasks. For instance, the LLaMA-3.1-8B-Instruct model only achieved a $44.78\%$ F1-Score on ProactiveBench.
In contrast, our fine-tuned model demonstrated significant improvements, achieving a $61.74\%$ F1-Score.
Beside, our fine-tuned Qwen2-7B-Instruct model achieves $66.47\%$ F1-Score, outperforming all existing open-source and closed-source LLMs.
This underscores the effectiveness of our data-driven approach in developing proactive agents, highlighting their potential to enhance user experiences across various applications.

## 2 Related Works

Recent advancement in large language models [, , , ] has shown great progress in complex reasoning, task planning [, , , , , , ], tool utilization [, , , ], etc.
Consequently, a growing number of agent systems have been developed to utilize these models to tackle diverse tasks like automatic web search [], software development [, ], behavior simulation [].

Despite these advancements, current agents predominantly operate reactively, meaning they passively accept human instructions and lack the context awareness [] to proactively satisfy user needs.
This limitation is particularly pronounced in scenarios that require intensive human-agent interactions, such as coding or writing.
These reactive agents typically wait for explicit user commands, which can lead to inefficiencies and increased cognitive load for the user as the task complexity grows.
In response to this challenge, we propose a novel dataset named ProactiveBench, designed to evaluate and improve the proactiveness of the language models.
Unlike existing works that enable multi-turn interactions to better understand ambiguous instructions in the context [, , , ], we focus on anticipating the possible tasks, which can reduce the user’s cognitive burden of providing explicit instructions and promote more effective assistance.

## 3 Methodology

### 3.1 Task Definition

In our proposed proactive agent, which is distinct from traditional agent systems powered by large language models that rely on explicit user instructions, we investigate a new scenario where the agent autonomously predicts tasks users might assign, aiming to offer assistance proactively, as depicted in  [Figure 1](#S1.F1).
The proactive agent’s mission is to give predictions based on the user’s activities $A_{t}$, environmental events $E_{t}$, and state $S_{t}$, which can be formalized as:

$$ $P_{t}=f_{\theta}(E_{t},A_{t},S_{t}),$ (1) $$

where $f_{\theta}$ represents the proactive agent, parameterized by $\theta$, and $P_{t}$ denotes the prediction about possible task at time $t$.
It should be noticed that the prediction $P_{t}$ can be the predicted task or nothing if the agent believes that no task is needed.
Specifically, user activities $A_{t}$ contain the user’s interactions with the environment and the agent, like keyboard input or chatting with the agent.
Environmental events $E_{t}$ contain the event that the proactive agent captured, ranging from receiving a new email to an application closed.
Environmental state $S_{t}$ represents the state of the current environment, like the file system state or the content of opened web pages.

In our proactive agent framework, the objective is to maximize the user’s acceptance rate of the proposed tasks.
Given the user’s historical activities $A_{t}$, current environmental state $S_{t}$, and the prediction proposed by the proactive agent $P_{t}$, the user makes a binary decision:

$$ $R_{t}=g(P_{t},A_{t},S_{t}),$ (2) $$

where $R_{t}$ is a binary variable indicating acceptance ($R_{t}=1$) or rejection ($R_{t}=0$) of the prediction.
To unify the handling of cases where the prediction $P_{t}$ contains no task and where it contains a task, we introduce an auxiliary variable $N_{t}$ that indicates whether the user needs assistance:

- •
$N_{t}=1$ if the user needs assistance.
- •
$N_{t}=0$ if the user does not need assistance.

The user’s acceptance $R_{t}$ is then defined as:

$$ $R_{t}=\begin{cases}1&\text{if }(P_{t}\neq\emptyset\text{ and user accepts }P_{t})\text{ or }(P_{t}=\emptyset\text{ and }N_{t}=0)\\ 0&\text{otherwise}\end{cases}.$ $$

In this way, if the prediction $P_{t}$ contains no task (i.e., the agent believes the user does not need assistance), we check the user’s actual need for assistance $N_{t}$. If the user indeed does not require assistance ($N_{t}=0$), this is marked as acceptance ($R_{t}=1$). Conversely, if the user requires assistance ($N_{t}=1$), this is marked as rejection ($R_{t}=0$).
Our proactive agent aims to maximize the expected acceptance rate of the proposed tasks:

$$ $\max_{\theta}\mathbb{E}[R_{t}].$ (3) $$

### 3.2 Pipeline Overview

Figure: Figure 2: Overview of the data generation process. Taking daily life as an example, this process includes modules such as the initial scenario and job setup, events generation, proactive prediction, user judgment, and action execution.
Refer to caption: /html/2410.12361/assets/x2.png

To enhance the proactive capabilities of our large language model-powered agent, we adopt a data-driven approach by building an automatic data generation pipeline.
This pipeline simulates user activities and responses to the tasks predicted by the proactive agent across various scenarios.
Once a prediction is accepted, we simulate the agent performing the task by interactively generating new events within the simulated environment.
Subsequently, new user activities are created based on historical events, allowing the proactive agent to generate further predictions.
Through this pipeline, models can learn when to generate predictions and which predictions are likely to be accepted by users.
Specifically, our pipeline consists of three components:

(1) Environment Gym: This component simulates events within a specified background setting and example events, providing a sandbox for proactive agents to interact. It has two key functionalities:
(i) Event Generation: creating potential sequences of environmental events tailored to specific scenarios;
(ii) State Maintenance: updating and maintaining the environment’s state when new user activities are generated or when the agent performs actions during task execution.

(2) Proactive Agent: This component is responsible for predicting tasks that the user might assign to the agent based on the user’s needs as inferred from the event history. It also interacts with tools to complete specific tasks assigned by the user.

(3) User Agent: This component simulates the user’s activities and responses based on predefined user characteristics. It decides whether to accept and execute the tasks proposed by the agent.

In the following sections, we introduce the details of each component.

### 3.3 Environment Gym

#### Event Collection

To improve the quality of events generated by the environment gym, we first collect real-world events as reference.
We developed a monitor software based on Activity Watcher(^2^22[https://github.com/ActivityWatch/activitywatch](https://github.com/ActivityWatch/activitywatch)), which allows us to capture the details of user interactions with computer systems, including keyboard and mouse operations, visited web pages, and used development tools.
To enhance the semantic richness of the collected data and facilitate parsing by large language models, we further merge the raw data into logically coherent segments.
Additionally, we utilize a language model to translate the structured data into more natural textual descriptions.
This process not only improves the interpretability of the data but also makes it more suitable for subsequent usage.

#### Scenario Generation

After collecting reference events, rather than directly generating specific events, we generate a realistic interactive scenario to provide sufficient background information first for further generation.
To build such scenarios, we first prompt GPT-4o [] with the seed jobs collected from human annotators to create various jobs the user might perform under a specific category, like coding, writing, or daily life.
Then, we generate all possible entities that the tasks might involve, e.g. browser, software, and tools for the agent to perform tasks.
Next, we refine the scenario by adding more details like entity status or date time to improve the details.
Finally, the collected events are also provided to generate example events under each particular context for future events generation.
This allows us to control the granularity of events that will be generated and maintain the diversity of the scenarios.
See [Appendix C](#A3) for the specific prompt used.

#### Event Generation

When it comes to specific event generation, we start with user activity generation.
For each scenario, the user agent is first requested to describe its activities and actions $A_{t}$ at time $t$ to complete the job in the simulated environment.
Then, the gym accepts the user’s activities and actions to generate detailed events one by one.
As depicted in  [Figure 2](#S3.F2), the gym is tasked to generate logically correct and fluent events according to historical events and the current environmental state.
The key to improving the realities of the events generated and adapting to different environments is utilizing the example events we generated based on collected events during scenario generation.
Before generating events, we randomly sample the generated example events for the specific scenario and request the gym to produce new events according to them.
Once a new event $E_{t+1}$ is generated, the gym updates the entities’ status in the environments and repeats the process until there are no events that can be generated with the provided user activities.
This comprehensive approach ensures that each subsequent event is not only appropriate but also contributes to a coherent and logical progression within the scenario.

#### State Maintenance

Another important functionality of the gym is maintaining the state of the environment $S_{t}$.
During the scenario generation, the gym creates entities like browsers or development kits in the simulated environment, where each entity has its state and properties like the application version or the specific browser name.
When a new event is generated, the gym should update the states and properties of them to provide feedback for further event generation.
Specifically, we first gather historical state changes of related entities and prompt the GPT-4o to generate new states of the entities $S_{t+1}$ with the new event.
During the process, the simulated time will also be updated according to the granularity of the event.
After that, the next event will be generated based on the latest environment state $S_{t+1}$.

### 3.4 Proactive Agent

Figure: Figure 3: Overview of the proactive agent framework. The agent monitors new events and updates its memory to predict potential tasks.
Refer to caption: /html/2410.12361/assets/x3.png

The second component in our data generation pipeline is the proactive agent that predicts tasks the user might assign.
As detailed in  [Figure 3](#S3.F3), upon the agent receiving new event $E_{t}$ at time $t$, it first updates its memory with the event.
To improve the quality of the prediction, it also accepts feedback from the user agent on its draft prediction.
Combining new events with historical ones and conversations with the user, the agent incorporates user characteristics to raise potential tasks.
If the agent detects potential tasks, it will raise the task as a new event and wait for the judgment of the user agent.
Otherwise, the proactive agent predicts no potential tasks and stays silent.
Once the predicted task is accepted, the agent will execute the task within the gym, which generates multiple events about the interaction between the agent and the environment.
During the data generation, the agent would constantly receive events from the Gym and predict potential tasks.

#### Task Execution

As mentioned before, the proactive agent executes the predicted task once the user accepts.
This process is mainly done through multi-turn interaction between the proactive agent and the gym.
Specifically, the proactive agent will be provided with the tools generated during the scenario generation, like file system tools in the computer or access to the smart light switch, to interact with the simulated environment.
Each time the proactive agent takes an action, the gym will generate a new event, which is further processed by the gym and the user agent to update the environment state.
After that, the proactive agent detects the new environment state $S_{t+1}$ and takes new actions according to the events generated by the gym.
This process ends when the user interrupts or the proactive agent finishes its tasks.

### 3.5 User Agent

The user agent is designed to emulate users’ activities $A_{t}$ and responses about the agent’s prediction $P_{t}$, as illustrated in  [Figure 2](#S3.F2).
We prompt GPT-4o to generate activities and actions for the provided task in the specific environment.
The gym further processes the activities and actions to generate a new event.
Then the proactive agent predicts potential tasks according to the events.
Upon receiving the predicted task, the user agent determines whether to accept or reject it.
If the user agent accepts the task, the proactive agent will set up and execute the accepted task within the environment gym.
Otherwise, if the user agent declines the suggested assistance, the environment gym generates new events autonomously without any interventions.
In our settings, we collect judgment from human annotators and train a reward model to simulate the judgment.

Specifically, to ensure the reward model aligns closely with human judgment, we generate and annotate a dedicated dataset to indicate whether humans would accept the predicted task or not.
We utilize 9 different language models to generate diverse predictions for each event.
Between these predictions, we select 5 predictions with minimum total cosine distance as our label target.
Each prediction is annotated with one of three options by three separate annotators: accept, reject, or reject all.
The reject all option is chosen when the annotator believes that the given events did not imply any possible tasks that the user might assign, aka $N_{t}=0$ in [Section 3.1](#S3.SS1).
Otherwise, if one prediction is labeled as accepted, we label the event $E_{t}$ with $N_{t}=1$.
We use majority voting to make the final decision on each prediction.
After all, the annotation results in a dataset of $1,760$ entries, each containing event traces, task predictions, and decisions on accepting the predicted task from three distinct annotators.
Notably, our annotators achieve an impressive consistency rate of over $91.67\%$ on the test set, underscoring the annotations’ reliability and the dataset’s robustness for further analysis.
To further facilitate automatic data generation, we also prompt the GPT-4o to produce a more detailed explanation of the user judgment.
Further details regarding the assessment of the reward model are available in [Section 4.1](#S4.SS1).

## 4 Experiments

### 4.1 Reward Model Assessment

To automatically evaluate whether the predicted tasks and their timing are appropriate, we seek to train a reward model capable of accurately imitating user judgments. To this end, we apply the user-annotated data to train LLaMA-3.1-8B-Instruct [] and compare it with several baselines to show its superiority.

#### Setting.

We use the $1,760$ entries with human annotations and randomly split them into a training set ($1,640$ entries) and a test set ($120$ entries).
We then train LLaMA-3.1-8B-Instruct on the training set to obtain our reward model.
We employ a total batch size of $32$, a learning rate of $1e-5$, and an Adam Optimizer with a $0.01$ warm-up ratio.
We train the reward model for $3$ epochs to prevent it from over-fitting. We use 8 A100 GPUs on one node to train for approximately 1.5 hours.
The detailed prompt template is listed in [Appendix A](#A1).
We use the test split to evaluate our adapted model and all the baselines.
To be noticed, our human annotators achieve up to $91.67\%$ agreement ratio on the test set, demonstrating the effectiveness of our evaluation.

#### Metrics.

We use the reward model to perform binary classification on whether to accept predicted tasks and compare its results with human-annotated results.
This assesses how well the reward model aligns with human judgment regarding the suitability of the predicted tasks.
We compare the judgments made by the reward models and humans to calculate the Recall, Precision, Accuracy, and F1-Score.
Additionally, we calculate the agreement ratio for the following cases:

- •
Missed-Needed: $N_{t}=1,P_{t}=\emptyset$, the user needs help, but the agent does not provide it.
- •
Non-Response: $N_{t}=0,P_{t}=\emptyset$, the user does not need help, the agent does not prompt.
- •
Correct-Detection: $N_{t}=1,P_{t}\neq\emptyset$, and the user accepts the task predicted by the agent.
- •
False-Detection: $N_{t}=0,P_{t}\neq\emptyset$, the user does not need help but agent prompts.

**Table 2: Evaluation results of different models on our test set as the reward model. For the Missed-Need (MN), Non-Response (NR), Correct-Detection (CD), and False-Detection (FD) scenarios, we present the agreement ratio between models and the major voting of our human annotators. Our model which is fine-tuned based on LLaMA-3.1-Instruct-8B achieves the best F1-Score of $91.80\%$.**
$$ $Recall=\frac{TP}{TP+FN}$ (4) $$

#### Results.

As [Section 4.1](#S4.SS1.SSS0.Px2) shows, all existing models perform well on correct detection, but perform badly in other scenarios, especially in the false alarm scenario.
After a deeper analysis, we find that existing models just can not infer what the user might need and tend to accept arbitrary help, even if it is very abstract or meaningless to current observation.
In contrast, our reward model achieves a $100\%$ agreement ratio on false alarm scenario and a solid $91.80\%$ F1-Score across all scenarios.
We select our reward model for further analysis across the ProactiveBench.

### 4.2 Proactive Agent Evaluation

**Table 3: Evaluation results of different models’ performance on the ProactiveBench. The GPT-4o stands out for close-sourced models, achieving over $64.60\%$ F1-Score. For open-sourced models, our fine-tuned Qwen2-7B model achieves best result, with a $66.47\%$ F1-Score.**
$$ $Recall=\frac{TP}{TP+FN}$ (4) $$

#### Setting.

We use the training set of ProactiveBench to obtain the Proactive Agent based on the two open-source models: LLaMA-3.1-8B-Instruct and Qwen2-7B-Instruct.
During training, we employ a total batch size of $32$, a learning rate of $1e-5$, and an Adam Optimizer with a $0.01$ warm-up ratio.
We train the model for $3$ epochs.
We use 8 A100 GPUs on one node to train for approximately 2 hours.
The detailed prompt can be found in [Appendix B](#A2).
The automatic evaluation of these metrics relies on the simulated judgment given by the reward model.
All models are evaluated in our test split of the ProactiveBench, which contains $233$ events collected in the real world.
We employ the same prompt template across all models. The temperature is set to 0 0 during testing.

#### Metrics.

We evaluate the performance of the Proactive Agent based on whether the user accepts its prediction.
As described in [Section 3.1](#S3.SS1), the user’s acceptance $R_{t}$ contains four conditions.
In our specific settings, Recall measures the proportion of actual needs for assistance that were correctly predicted by the agent, including cases where the agent predicts a task and the user accepts it, as well as cases where no task is predicted and the user does not need assistance.
Precision measures the proportion of predicted tasks that were actually accepted by the user.
Accuracy measures the overall correctness of the agent’s predictions.
The False-Alarm measures the proportion of incorrect task predictions, specifically when a task is predicted but not needed.
The F1-Score provides a balanced measure of the goodness of the agent’s proactive behavior.
We use the reward model during the evaluation to automatically generate the user’s judgment.
Based on the confusion matrix, we report Recall, Precision, Accuracy, False Alarm, and F1-Score across all settings. The detailed calculation method can be found in [Appendix B](#A2).

#### Result.

[Section 4.2](#S4.SS2) compares various models on the test set of the ProactiveBench, which contains $233$ events collected from the real world user.
Close-sourced models like GPT-4o [] or GPT-4o-mini all tend to predict proactive tasks actively.
Most of them succeed in assisting when the user needs but fail to stay silent when the user does not require any assistance, resulting in a relatively high false alarm ratio.
For example, the GPT-4o-mini provides unnecessary assistance even when the events provided do not contain meaningful operations, like switching between software but doing nothing else.
Another big issue is early assistance when no precise users’ intents can be found in the given observation.
This makes the proactive tasks proposed by the model seem too abstract or useless, resulting in a relatively high false alarm ratio.
The Claude-3-Sonnet [] shows a different example of failing to detect the user’s need and provide assistance that does not satisfy the user’s expectation.

For open-sourced models, we evaluate the performance of the LLaMA-3.1-Instruct-8B and Qwen2-Instruct-7B before and after fine-tuning based on our synthesized data.
As shown in [section 4.2](#S4.SS2), both models obtain an impressive improvement, especially for LLaMA-3.1-8B, which improves its F1-Score from $44.78\%$ to $61.74\%$.
The results demonstrate the effectiveness of our data synthesis pipelines.
As for the concern of being overly interrupted by the proactive agent, our fine-tuned models achieve solid progress in reducing the false alarm ratio, which is comparable to the performance of the GPT-4o.
Besides, the finetuned Qwen2-7B is also outperform the GPT-4o in terms of the F1-Score, resulting in the highest F1-Score of $66.07\%$
However, we also observed the same pattern of models tends to provide as much assistance as possible, instead of providing necessary assistance when the user needs it.

In short, while most models can assist when needed, they still frequently offer unnecessary help, even when instructed to provide only essential assistance.

### 4.3 Ablation Study

#### Predict Multiple Tasks.

When it comes to real-world applications, the proactive agent can provide multiple candidate tasks to improve overall performance.
To evaluate how models perform under this condition, we allow them to generate multiple candidate tasks at once, but no more than three to avoid a high cognition burden for the user.
In this setting, we let the reward model check the candidate tasks one by one.
We label the result as accepted if one of the candidate tasks is accepted, and rejected if only all the candidate tasks are rejected.

As shown in [Section 4.3](#S4.SS3.SSS0.Px1), all models obtain solid improvement across all metrics.
Take the GPT-4o as an example, it obtains higher accuracy and precision while reducing its false alarm by providing diverse candidate tasks.
The huge drops in the false alarm ratio, from $51.85\%$ to $36.44\%$ are mainly due to its improvement in providing proactive tasks.
However, when comparing GPT-4o-mini with LLaMA-3.1-8B, we observed different degrees of improvement.
These two models perform similarly when predicting only one proactive task at once, but show a nearly $9\%$ difference in terms of F1-Score when predicting multiple candidates at once.
We analyzed the result and found that the LLaMA-3.1-8B tends to provide unexpected assistance when the user’s need is unclear, which can not be improved by providing multiple candidates.

**Table 4: Comparison between baseline, predicting multiple candidates, and getting feedback from the reward model. The result shows a general improvement across all models. The model’s false alarm ratio drops due to a higher possibility of accepting the predicted tasks or being improved by the reward model.**
$$ $Recall=\frac{TP}{TP+FN}$ (4) $$

#### Feedback From the Reward Model.

We also investigate whether the feedback from our reward model could help models improve their performance on the ProactiveBench.
This is done with the same logic as described in [Figure 3](#S3.F3).
For each model, we first ask them to generate a draft prediction and obtain feedback from the user agent (which is built on the reward model in this case).
Then we let the model refine its prediction to obtain the final prediction.

As shown in [Section 4.3](#S4.SS3.SSS0.Px1), by adding the feedback from the reward model, models generally reduce their false alarm ratio and improve the accuracy, but drop dramatically in terms of the recall.
We observe that models stay silent once they receive feedback from the reward model.
However, doing nothing is not always the optimal solution.
The GPT-4o seems to fail to capture the possible user needs, leading to a performance drop in terms of F1-Score.
For other models like GPT-4o-mini and LLaMA-3.1-8B, they deed obtain a marked improvement in terms of the F1-Score.
By integrating the reward model into the Proactive Agent, we can make the Proactive Agent more smartly detect user needs and reduce the false alarm ratio even when we can not access the weight directly, which is good news for developing the Proactive Agent.

### 4.4 Case Study

In this section, we explore two prevalent types of failures encountered in predicting possible tasks: the inability to detect user needs and making predictions at inappropriate times.

Figure: Figure 4: Two types of failure: failing to detect user needs (left) and inappropriate proposing time (right). We compare the response between our fine-tuned LLaMA-3.1-Instruct-8B with other models to show the refined proactive behavior of the model.
Refer to caption: /html/2410.12361/assets/x4.png

As illustrated in  [Figure 4](#S4.F4) (left), a notable failure occurs when the GPT-4o model does not assist at crucial moments.
For instance, when a user is engaged in integrating a payment API and requires a tutorial for guidance, the model remains silent.
Instead, our model successfully detects human needs and offers assistance.
The underlying intention to minimize disruptions ironically leads to missed opportunities to offer timely help.

Conversely, the right side of  [Figure 4](#S4.F4) showcases an instance of ill-timed prediction. Here, the GPT-4o-mini suggests an action when there are no user needs showing in events.
This scenario underscores the possible unintended events existing in human activities.
The model should judge whether there are possible tasks smartly to avoid unnecessary actions.
These instances highlight the intricate nature of human activities and the sophisticated reasoning required for models to accurately predict human needs.
To navigate the delicate balance between being helpful and intrusive, models must develop a deeper understanding of user contexts and activities, ensuring their interventions are both timely and pertinent.

## 5 Conclusion

We present an innovative approach to human-agent interaction by leveraging proactive task predictions that anticipate human needs.
We introduce ProactiveBench, a comprehensive dataset comprising $6,790$ events, designed to refine the proactive behavior of LLM-based agents and establish an automatic benchmark for assessing model proactiveness.
By iteratively generating events in synthesized scenarios, we create training data that enhances the proactive capabilities of our models.
Our experiments demonstrate significant improvements in the agent’s performance on ProactiveBench, validating the effectiveness of our methods.
Despite these advancements, our findings underscore ongoing challenges, particularly in minimizing inappropriate task proposals and ensuring task predictions are contextually accurate.
Future research should focus on enhancing the precision and timeliness of task predictions to improve the efficacy of the proactive human-agent interaction.

## Limitations

While our method demonstrates that it can effectively and proactively predict possible tasks, the current research is constrained by several limitations.
Firstly, the environment settings we have explored are still limited.
The contexts in this paper provide a foundational understanding, but broader application areas need to be investigated to fully establish the versatility and robustness of the proactive agent.
Moreover, models still exhibit a relatively high ratio of false alarms, indicating that they cannot yet perfectly predict possible tasks.
This limitation highlights the need for further refinement of the model’s proactive behavior to avoid bothering the user.
The high rate of false alarm can lead to unnecessary or incorrect actions, which may reduce user trust and the overall efficiency of the system.
The dynamic adjustment of the proactiveness of the agent according to the specific context should be explored in more depth.
Future research should focus on several key areas to address these limitations:

- •
Expansion of Environment Settings: Research should explore a wider variety of scenarios and contexts to validate the model’s generalizability. This includes domains where proactive prediction of tasks can significantly enhance user experience and operational efficiency.
- •
Improvement in Prediction Accuracy: Efforts should be directed towards reducing the false alarm rate by enhancing the model’s understanding of context and user behavior.
- •
User-Centric Evaluation: Future studies should involve extensive user-centric evaluations to better understand how users interact with the proactive agent and to identify areas for improvement. User feedback and behavioral data can provide valuable insights into refining the prediction algorithms and making the system more intuitive and reliable.
- •
Ethical and Privacy Considerations: As the proactive agent needs the environment information for prediction tasks, it is crucial to address ethical and privacy concerns. Ensuring that user data is handled responsibly and that the agent operates transparently and within ethical guidelines will be critical for gaining user trust and acceptance.

## Appendix

## Appendix A Reward Model Training Setting

We use Llama-3.1-Instruct-8B as the base model for our training. The total dataset size is approximately $1,640$. Specifically, we employ a total batch size of $32$, a learning rate of $1e-5$, and an Adam Optimizer with a $0.01$ warm-up ratio. We train the reward model for 3 epochs to prevent it from over-fitting. We use 8 A100 GPUs on one node to train for approximately 1.5 hours.
Prompt Template

## Appendix B Agent Model Training Setting

Similarly, we use Llama-3-Instruct 8B and Qwen2-Instruct-7B as the base model for agent model training.
The total dataset size is approximately $6,790$. Specifically, we employ a total batch size of $32$, a learning rate of $1e-5$, and an Adam Optimizer with a $0.01$ warm-up ratio. We train the model for 3 epochs to prevent it from over-fitting. We use 8 A100 GPUs on one node to train for approximately 2 hours.

#### Template.

We apply the following prompt template to train the agent model:

Prompt Template

## Appendix C Prompt Template for Environment Gym

### C.1 Prompt for Scene Generation

Prompt Template

### C.2 Seed Jobs Data

Prompt Template

### C.3 Prompt for User Agent Generation

Prompt Template

### C.4 Prompt for Status Updating

Prompt Template

### C.5 Metrics Calculation

#### Definition

Here is how we define the label of each prediction.

- •
True Positive (TP): Agent predicts task, the user accepts.
- •
False Positive (FP): Agent predicts task, the user rejects.
- •
True Negative (TN): Agent does not predict a task, and the user does not need assistance.
- •
False Negative (FN): Agent does not predict the task, but the user needs assistance ($N_{t}=1$ in [Section 3.1](#S3.SS1)).

#### Recall

A high recall indicates that the agent frequently identifies situations where help is needed. This metric is crucial for assessing the agent’s ability to recognize and respond to user needs on time.

$$ $Recall=\frac{TP}{TP+FN}$ (4) $$

#### Precision

A high precision indicates that the agent proposes good tasks while not bothering the user too much. This metric is crucial when considering the annoying behavior of the proactive agent could greatly reduce user satisfaction.

$$ $Precision=\frac{TP}{TP+FP}$ (5) $$

#### Accuracy

High Accuracy demonstrates that the agent has a good understanding of user needs, as most of its predictions are accepted. This metric is essential for measuring the relevance and correctness of the agent’s proactive suggestions.

$$ $Accuracy=\frac{TP+TN}{P+N}$ (6) $$

#### F1-Score

High F1-Score means the proactive agent strikes a good balance between being helpful and proactive.

$$ $F_{1}=2*\frac{Recall*Precision}{Recall+Precision}$ (7) $$

## Appendix D Data Examples

### D.1 Event Samples

Collected Raw Data

Processed Event

### D.2 Proactive Agent’s Prediction

Example 1

Example 2

Example 3

### D.3 Reward Model Judgment Example

Input Prompt

Ouput Judgment

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above. I may also suggest incorporating more examples in the method part.
soundness: 3
strengths: I really appreciate the concept of proactive agents, as it addresses a crucial challenge in the planning field: actions should have long-term impacts—a factor often overlooked in prior research. This approach could inspire future agent research to prioritize anticipating future outcomes before choosing actions in the current step. I'm also impressed by the model's performance on ProActiveBench, where it significantly outperforms the GPT series models.
summary: This paper addresses the limitations of reactive agent systems by introducing a proactive approach to LLM agents. The authors develop a data-driven method where real-world human activities are collected to generate task predictions, which are then labeled as accepted or rejected by human annotators. These labels train a reward model that simulates human judgment, enabling the agent to anticipate and initiate actions without explicit instructions. The authors create ProactiveBench, a dataset of 6,790 events, and show that fine-tuning agents with this dataset significantly enhances their proactive capabilities. It marks a promising step forward in human-agent collaboration.
weaknesses: The main weaknesses lie in several points:

First, the training process for the proactive agent is not fully explained, particularly regarding the type of annotations used. Besides, while the introduction mentions human-annotated data, it may also cover some portions potentially synthesized automatically. It’s unclear what instructions human annotators followed, how annotation quality was ensured, and how quality control was managed for automatically generated data.

Additionally, the process of maintaining environment state requires further detail. Since this involves using GPT models to generate new states, it raises questions about the accuracy and reliability of GPT in simulating these states. More clarification here would be valuable.

I also have concerns about the comprehensiveness of the tested baselines. For instance, the GPT-based baselines might not fully leverage GPT's potential. Incorporating more chain-of-thought reasoning could lead to more grounded and reliable action predictions, similar to a world model that simulates multiple steps ahead. It would also be insightful to see a comparison showing performance without any proactive agents as a benchmark.

Additionally, the evaluation relies heavily on an in-house test set, raising questions about the broader applicability of the proactive agent approach. A more comprehensive evaluation, possibly across varied scenarios, would help clarify its generalizability and robustness.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: * It's unclear whether the environment gym relies on an interactive simulator or is simply text-based.
* The four stages used to describe the environment gym setup seem redundant and could be streamlined for clarity.
* More information is needed on the dataset used for reward model assessment, particularly whether the test data is out-of-domain or in-domain.
* The paper lacks information on the human annotators involved in determining the agreement ratio in the reward model assessment. It is better to provide details on their demographics or annotation guidelines.
* In Proactive Agent Evaluation, it is unclear why Qwen2-7B outperforms LLaMA-3.1-8B despite having fewer parameters. Additionally, the performance gains from fine-tuning on Qwen-7B appear relatively small, which should be further discussed.
* The term *ablation study* seems somewhat misleading here, as this section primarily compares public models in different settings rather than systematically removing or modifying components. Furthermore, it is unclear why the *Predict Multiple Tasks* setting improves overall performance, and additional explanation would be helpful.
soundness: 2
strengths: * The paper presents a well-structured framework for proactive agents, supported by robust experimental design.
* The creation of ProactiveBench provides a comprehensive benchmark dataset that will be highly valuable for future research in proactive language models and user-agent interaction studies.
* The paper employs reinforcement learning to simulate the user agent and model user preferences, enabling a realistic and dynamic approach to assess agent behaviors and refine their decision-making processes.
summary: This paper presents a proactive agent framework designed to predict human intentions based on real-world-inspired environmental observations. To support this, the authors introduce ProactiveBench, a benchmark dataset containing 6,790 events across coding, writing, and daily life activities, capturing user behaviors such as keyboard and mouse inputs, clipboard content, and browser activity.

The data generation process integrates three components: a proactive agent, a user agent, and an environment simulation to emulate realistic interactions. Leveraging human-annotated data, the authors train a reward model that simulates human judgment, achieving 91.8% consistency with human evaluations.

To further enhance the capability to proactively predict human intentions, the authors fine-tuned the LLaMA-3.1-8B and Qwen2-7B. Among these, Qwen2-7B-Proactive achieved the highest accuracy and F1-scores in anticipating user intentions, outperforming both open-source and closed-source baselines.
weaknesses: * This paper lacks sufficient detail in sections such as *Section 3.3: Environment Gym* and *Section 4.1: Reward Model Assessment*, which hinders a full understanding of the framework and evaluation process.
* The paper does not adequately discuss current research on real-world proactive agents or clarify how this work differs from existing approaches.
* The experimental analysis is limited, and the ablation study section is somewhat misleading.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - They use GPT-4o for almost the entirety of the pipeline, except for initial user-data, and initial human judgement collection. 
	- How do they ensure that the created scenarios/events and states make logical sense, if they are doing it at scale?

- Not relevant to the paper's strengths/weaknesses: What are the authors' thoughts on being able to tune reactiveness and proactivness? Each user and/or task may have different preferences, is there a straight forward way to control this?

- Line 408 - "In short, while most models can assist when needed, they still frequently offer unnecessary help, even when instructed to provide only essential assistance."
	- Do the authors have any thoughts on how this can be prevented in the future?

- How many events are collected in real-world events? How much is synthetically generated?

- How long are the context windows for the events i.e how many events are used for a single prediction?

- To clarify, the judgement is the only thing that is done from humans? And actions are produced by GPT-4o?
soundness: 3
strengths: - The authors attempt to improve on a novel perspective and quality of LLM agents - proactiveness. The way they model this quality is also simple yet interesting - using rewards from user agents. They build a benchmark around this and show effectiveness of their approach towards building proactive agents.
- The design of the reward model and human judgement collection is thorough - they include all possible scenarios and the logic is sound.
- The additional metrics for the agreement ratio are interesting and help provide a better understanding of the performance of various reward models.
- The authors show improved performance against both open-source and closed-source models, when the LLMs are fine-tuned on their ProactiveBench dataset.
- The figures are really good, and they help with the understanding of the approach. The paper is more or less clear (see weaknesses - clarifications).
summary: The paper provides a novel approach towards building proactive LLMs. To this end, the authors propose a data collection pipeline on three different types of tasks (coding, writing, daily life). The data pipeline uses Activity Watcher to collect human data (clicks, clipboard, etc.) on these different tasks.

GPT-4o is used to synthetically generate more events and create "ProactiveBench" dataset. They train a reward model to simulate human judgement from some human annotations and create an automatic evaluator of proactiveness. The results show that this reward model is better aligned in comparison to existing LLMs (LLaMa, GPT-4o). 

They fine-tune two models (LLaMA, Qwen) on their Proactive Bench training set and show that this strategy leads to better proactiveness against baselines - measured in terms of precision, recall, accuracy, False-Alarm and F1-Score against the test set. Qwen2-7B based LLM fine-tuned on ProactiveBench achieves 66.47% F1 score, in comparison to 64.60% from GPT-4o. 

They also perform ablation studies where they allow LLMs to output multiple candidates, and a dummy feedback  from their reward model to improve on its prediction - both strategies lead to some improvement in F1.
weaknesses: **General Weaknesses**
- GPT-4o already achieves 64.60% F1, while the author's best F1 score is 66.47%, which is a ~2% relative improvement. The test set (233 events) is not big enough for this difference to be statistically significant. Would be nice to have some discussion on why the authors feel that this improvement will have an impact on future research in this area, considering the fact LLMs keep getting better by the day.
- The experiment is performed against their reward-tuned model, and their own fine-tuned proactive agent, both on subsets of ProactiveBench (please correct me if I am wrong). This casts a doubt on whether actual users will benefit from the approach. Ideally, there should have been some user studies to support this, possibly with unseen events/scenarios.
- The authors do not discuss why they think Qwen performs better than LLaMa and GPT-4o.

**Limitations**
- The authors do not discuss the resource consumption for LLMs when they are proactively predicting tasks on existing systems. A good reason why most LLM agents today are reactive could be because of this issue - it is not possible to have an 7B parameter model constantly running on-device for proactive response generation.
- The authors discuss this in the limitations section - but the gym / agents are in a toy-setting with limited task types. The authors do not perform any real-user evaluations i.e. actually integrating the proactive agent into a system and getting an actual user to evaluate how helpful or disruptive the agent is.

**Missing clarifications/data**:
- Table 4: Can the proactive agents also be used with multiple candidates and with RM? If yes, then this table should contain results for those too.
-  The paragraph describing the human annotation in Section 3.5 could use improvement: 
    - It is not clear what is the "cosine distance" computed against.
    - It is also not clear what "consistency rate" means or how is it measured.
    - What is the annotator agreement on train set?
- How is the GPT-4o explanation for the user judgement used downstream? Is this actually used?
- Section 3.4 - "it first updates its memory with the event" - what kind of memory mechanism is being used?
- Section 4.1 - Setting - How is the LLaMa 3.1 8B-Instruct trained? Are all the layers unfrozen during the training?
- Line 356 - "The temperature is set to 0 during testing." Which temperature? The one used for decoding? This line disrupts the flow of the reading, might be better placed elsewhere.
- Minor:
    - Not sure if I missed this: would be nice to have a well-defined description for what events and scenarios mean.
    - Would be nice to have examples of failure modes (showing scenarios, events, states, predictions), even if in Appendix, for different LLMs in comparison to authors' fine-tuned variant to clarify the differences.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Would it be possible to turn text-based simulation into a real interactive environment? 

Can you propose more baselines or analyses to demonstrate the effectiveness of the proposed agent?
soundness: 2
strengths: 1.  The concept of creating proactive agents is important yet less explored in the field. This paper represents a significant early step in that direction. While proactive conversational models are not entirely new, applying this approach to the realm of agents should be regarded as a noteworthy contribution to the field.

2. The curated benchmark and the proposed proactive agent prompts will serve as valuable resources for the community.
summary: This paper develops proactive agents—agents capable of initiating tasks without explicit human instructions by simply observing the environmental state and historical actions. This direction is important as it represents a shift from reactive to proactive agents, a very useful characteristic for future agents. 

To train such a proactive agent, the authors propose a data synthesis framework. They utilize GPT-4o to construct a text-based simulation environment called Environment Gym. This 'gym' functions as a sandbox that employs GPT-4o to generate new environmental states whenever agents perform actions. Additionally, the authors prompt GPT-4o to predict proactive actions through a proactive agent module and simulate user interactions to determine whether to accept the predicted actions.

Through this data pipeline, combined with some minor human annotations, the authors curated 1,760 data entries, using 120 of them for testing. The trained proactive agent demonstrates a clear performance improvement over the vanilla LLaMA-3.1-8B model and GPT-4o, which, while notable, is not entirely surprising.
weaknesses: One major weakness I identified is the text-based simulation environments. In the current implementation, all environmental changes and updates are managed by GPT-4o. This raises concerns about whether LLMs can accurately reflect true environmental changes. Additionally, this reliance on LLMs could potentially make the benchmark too easy, as they may only be capable of predicting changes that they fully understand and are familiar with, which are also easily solvable by other LLMs. 

After reviewing the entire paper and appendix, I still find myself unclear about the specifics of the dataset. Including some statistics and examples in the main text would be very helpful.

As stated in the introduction, the main contributions of this paper are the data pipeline and the proactive agent. However, the method used for creating this agent is just simple fine-tuning. Without comparison with strong baselines, it's really hard to assess the true effectiveness of the proposed data pipeline.  I understand the difficulty of finding baselines for new problems, but the current evaluation results are not very convincing.

Meta-review: (a) Summary: This paper proposes a novel data-driven formalization for developing a proactive agent that anticipates user needs and takes the initiative by suggesting tasks or providing information without explicit requests.
(b) Strengths: The paper is generally well-written and easy to follow. The proposed approach seems interesting and novel. The experimental results seem to support the authors' claims.
(c) Weaknesses: The reviewers pointed out a few concerns and issues. Some technical details need more explanations.
(d) This paper is a borderline case. Some of the reviewers replied to the authors' rebuttal and appreciated their refinement and clarifications. Generally speaking, I think that the authors addressed the majority of the reviewers' concerns and comments, although some of the reviewers did not reply during the rebuttal phase.

Decision: Accept (Poster)