Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance

Authors: Yaxi Lu, Shenzhi Yang, Cheng Qian, Guirong Chen, Qinyu Luo, Yesai Wu, Huadong Wang, Xin Cong, Zhong Zhang, Yankai Lin, Weiwen Liu, Yasheng Wang, Zhiyuan Liu, Fangming Liu, Maosong Sun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Human-Centered NLP, Dialogue and Interactive Systems, Resources and Evaluation

Abstract:
Agents powered by large language models have shown remarkable abilities in solving complex tasks. However, most agent systems remain reactive, limiting their effectiveness in scenarios requiring foresight and autonomous decision-making. In this paper, we tackle the challenge of developing proactive agents capable of anticipating and initiating tasks without explicit human instructions. We propose a novel data-driven approach for this problem. Firstly, we collect real-world human activities to generate proactive task predictions. These predictions are then labeled by human annotators as either accepted or rejected. The labeled data is used to train a reward model that simulates human judgment and serves as an automatic evaluator of the proactiveness of LLM agents. Building on this, we develop a comprehensive data generation pipeline to create a diverse dataset, ProactiveBench, containing 6,790 events. Finally, we demonstrate that fine-tuning models with the proposed ProactiveBench can significantly elicit the proactiveness of LLM agents. Experimental results show that our fine-tuned model achieves an F1-Score of 66.47% in proactively offering assistance, outperforming all open-source and close-source models. These results highlight the potential of our method in creating more proactive and effective agent systems, paving the way for future advancements in human-agent collaboration.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729060936527

thread_type:
official_review

<thread_object>
creator: reviewer_fw7i

<content>
summary:
This paper develops proactive agents—agents capable of initiating tasks without explicit human instructions by simply observing the environmental state and historical actions. This direction is important as it represents a shift from reactive to proactive agents, a very useful characteristic for future agents. 

To train such a proactive agent, the authors propose a data synthesis framework. They utilize GPT-4o to construct a text-based simulation environment called Environment Gym. This 'gym' functions as a sandbox that employs GPT-4o to generate new environmental states whenever agents perform actions. Additionally, the authors prompt GPT-4o to predict proactive actions through a proactive agent module and simulate user interactions to determine whether to accept the predicted actions.

Through this data pipeline, combined with some minor human annotations, the authors curated 1,760 data entries, using 120 of them for testing. The trained proactive agent demonstrates a clear performance improvement over the vanilla LLaMA-3.1-8B model and GPT-4o, which, while notable, is not entirely surprising.

strengths:
1.  The concept of creating proactive agents is important yet less explored in the field. This paper represents a significant early step in that direction. While proactive conversational models are not entirely new, applying this approach to the realm of agents should be regarded as a noteworthy contribution to the field.

2. The curated benchmark and the proposed proactive agent prompts will serve as valuable resources for the community.

weaknesses:
One major weakness I identified is the text-based simulation environments. In the current implementation, all environmental changes and updates are managed by GPT-4o. This raises concerns about whether LLMs can accurately reflect true environmental changes. Additionally, this reliance on LLMs could potentially make the benchmark too easy, as they may only be capable of predicting changes that they fully understand and are familiar with, which are also easily solvable by other LLMs. 

After reviewing the entire paper and appendix, I still find myself unclear about the specifics of the dataset. Including some statistics and examples in the main text would be very helpful.

As stated in the introduction, the main contributions of this paper are the data pipeline and the proactive agent. However, the method used for creating this agent is just simple fine-tuning. Without comparison with strong baselines, it's really hard to assess the true effectiveness of the proposed data pipeline.  I understand the difficulty of finding baselines for new problems, but the current evaluation results are not very convincing.

questions:
Would it be possible to turn text-based simulation into a real interactive environment? 

Can you propose more baselines or analyses to demonstrate the effectiveness of the proposed agent?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730165518477

thread_type:
official_review

<thread_object>
creator: reviewer_ztib

<content>
summary:
The paper provides a novel approach towards building proactive LLMs. To this end, the authors propose a data collection pipeline on three different types of tasks (coding, writing, daily life). The data pipeline uses Activity Watcher to collect human data (clicks, clipboard, etc.) on these different tasks.

GPT-4o is used to synthetically generate more events and create "ProactiveBench" dataset. They train a reward model to simulate human judgement from some human annotations and create an automatic evaluator of proactiveness. The results show that this reward model is better aligned in comparison to existing LLMs (LLaMa, GPT-4o). 

They fine-tune two models (LLaMA, Qwen) on their Proactive Bench training set and show that this strategy leads to better proactiveness against baselines - measured in terms of precision, recall, accuracy, False-Alarm and F1-Score against the test set. Qwen2-7B based LLM fine-tuned on ProactiveBench achieves 66.47% F1 score, in comparison to 64.60% from GPT-4o. 

They also perform ablation studies where they allow LLMs to output multiple candidates, and a dummy feedback  from their reward model to improve on its prediction - both strategies lead to some improvement in F1.

strengths:
- The authors attempt to improve on a novel perspective and quality of LLM agents - proactiveness. The way they model this quality is also simple yet interesting - using rewards from user agents. They build a benchmark around this and show effectiveness of their approach towards building proactive agents.
- The design of the reward model and human judgement collection is thorough - they include all possible scenarios and the logic is sound.
- The additional metrics for the agreement ratio are interesting and help provide a better understanding of the performance of various reward models.
- The authors show improved performance against both open-source and closed-source models, when the LLMs are fine-tuned on their ProactiveBench dataset.
- The figures are really good, and they help with the understanding of the approach. The paper is more or less clear (see weaknesses - clarifications).

weaknesses:
**General Weaknesses**
- GPT-4o already achieves 64.60% F1, while the author's best F1 score is 66.47%, which is a ~2% relative improvement. The test set (233 events) is not big enough for this difference to be statistically significant. Would be nice to have some discussion on why the authors feel that this improvement will have an impact on future research in this area, considering the fact LLMs keep getting better by the day.
- The experiment is performed against their reward-tuned model, and their own fine-tuned proactive agent, both on subsets of ProactiveBench (please correct me if I am wrong). This casts a doubt on whether actual users will benefit from the approach. Ideally, there should have been some user studies to support this, possibly with unseen events/scenarios.
- The authors do not discuss why they think Qwen performs better than LLaMa and GPT-4o.

**Limitations**
- The authors do not discuss the resource consumption for LLMs when they are proactively predicting tasks on existing systems. A good reason why most LLM agents today are reactive could be because of this issue - it is not possible to have an 7B parameter model constantly running on-device for proactive response generation.
- The authors discuss this in the limitations section - but the gym / agents are in a toy-setting with limited task types. The authors do not perform any real-user evaluations i.e. actually integrating the proactive agent into a system and getting an actual user to evaluate how helpful or disruptive the agent is.

**Missing clarifications/data**:
- Table 4: Can the proactive agents also be used with multiple candidates and with RM? If yes, then this table should contain results for those too.
-  The paragraph describing the human annotation in Section 3.5 could use improvement: 
    - It is not clear what is the "cosine distance" computed against.
    - It is also not clear what "consistency rate" means or how is it measured.
    - What is the annotator agreement on train set?
- How is the GPT-4o explanation for the user judgement used downstream? Is this actually used?
- Section 3.4 - "it first updates its memory with the event" - what kind of memory mechanism is being used?
- Section 4.1 - Setting - How is the LLaMa 3.1 8B-Instruct trained? Are all the layers unfrozen during the training?
- Line 356 - "The temperature is set to 0 during testing." Which temperature? The one used for decoding? This line disrupts the flow of the reading, might be better placed elsewhere.
- Minor:
    - Not sure if I missed this: would be nice to have a well-defined description for what events and scenarios mean.
    - Would be nice to have examples of failure modes (showing scenarios, events, states, predictions), even if in Appendix, for different LLMs in comparison to authors' fine-tuned variant to clarify the differences.

questions:
- They use GPT-4o for almost the entirety of the pipeline, except for initial user-data, and initial human judgement collection. 
	- How do they ensure that the created scenarios/events and states make logical sense, if they are doing it at scale?

- Not relevant to the paper's strengths/weaknesses: What are the authors' thoughts on being able to tune reactiveness and proactivness? Each user and/or task may have different preferences, is there a straight forward way to control this?

- Line 408 - "In short, while most models can assist when needed, they still frequently offer unnecessary help, even when instructed to provide only essential assistance."
	- Do the authors have any thoughts on how this can be prevented in the future?

- How many events are collected in real-world events? How much is synthetically generated?

- How long are the context windows for the events i.e how many events are used for a single prediction?

- To clarify, the judgement is the only thing that is done from humans? And actions are produced by GPT-4o?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707387728

thread_type:
official_review

<thread_object>
creator: reviewer_tupl

<content>
summary:
This paper presents a proactive agent framework designed to predict human intentions based on real-world-inspired environmental observations. To support this, the authors introduce ProactiveBench, a benchmark dataset containing 6,790 events across coding, writing, and daily life activities, capturing user behaviors such as keyboard and mouse inputs, clipboard content, and browser activity.

The data generation process integrates three components: a proactive agent, a user agent, and an environment simulation to emulate realistic interactions. Leveraging human-annotated data, the authors train a reward model that simulates human judgment, achieving 91.8% consistency with human evaluations.

To further enhance the capability to proactively predict human intentions, the authors fine-tuned the LLaMA-3.1-8B and Qwen2-7B. Among these, Qwen2-7B-Proactive achieved the highest accuracy and F1-scores in anticipating user intentions, outperforming both open-source and closed-source baselines.

strengths:
* The paper presents a well-structured framework for proactive agents, supported by robust experimental design.
* The creation of ProactiveBench provides a comprehensive benchmark dataset that will be highly valuable for future research in proactive language models and user-agent interaction studies.
* The paper employs reinforcement learning to simulate the user agent and model user preferences, enabling a realistic and dynamic approach to assess agent behaviors and refine their decision-making processes.

weaknesses:
* This paper lacks sufficient detail in sections such as *Section 3.3: Environment Gym* and *Section 4.1: Reward Model Assessment*, which hinders a full understanding of the framework and evaluation process.
* The paper does not adequately discuss current research on real-world proactive agents or clarify how this work differs from existing approaches.
* The experimental analysis is limited, and the ablation study section is somewhat misleading.

questions:
* It's unclear whether the environment gym relies on an interactive simulator or is simply text-based.
* The four stages used to describe the environment gym setup seem redundant and could be streamlined for clarity.
* More information is needed on the dataset used for reward model assessment, particularly whether the test data is out-of-domain or in-domain.
* The paper lacks information on the human annotators involved in determining the agreement ratio in the reward model assessment. It is better to provide details on their demographics or annotation guidelines.
* In Proactive Agent Evaluation, it is unclear why Qwen2-7B outperforms LLaMA-3.1-8B despite having fewer parameters. Additionally, the performance gains from fine-tuning on Qwen-7B appear relatively small, which should be further discussed.
* The term *ablation study* seems somewhat misleading here, as this section primarily compares public models in different settings rather than systematically removing or modifying components. Furthermore, it is unclear why the *Predict Multiple Tasks* setting improves overall performance, and additional explanation would be helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731127416787

thread_type:
official_review

<thread_object>
creator: reviewer_9sae

<content>
summary:
This paper addresses the limitations of reactive agent systems by introducing a proactive approach to LLM agents. The authors develop a data-driven method where real-world human activities are collected to generate task predictions, which are then labeled as accepted or rejected by human annotators. These labels train a reward model that simulates human judgment, enabling the agent to anticipate and initiate actions without explicit instructions. The authors create ProactiveBench, a dataset of 6,790 events, and show that fine-tuning agents with this dataset significantly enhances their proactive capabilities. It marks a promising step forward in human-agent collaboration.

strengths:
I really appreciate the concept of proactive agents, as it addresses a crucial challenge in the planning field: actions should have long-term impacts—a factor often overlooked in prior research. This approach could inspire future agent research to prioritize anticipating future outcomes before choosing actions in the current step. I'm also impressed by the model's performance on ProActiveBench, where it significantly outperforms the GPT series models.

weaknesses:
The main weaknesses lie in several points:

First, the training process for the proactive agent is not fully explained, particularly regarding the type of annotations used. Besides, while the introduction mentions human-annotated data, it may also cover some portions potentially synthesized automatically. It’s unclear what instructions human annotators followed, how annotation quality was ensured, and how quality control was managed for automatically generated data.

Additionally, the process of maintaining environment state requires further detail. Since this involves using GPT models to generate new states, it raises questions about the accuracy and reliability of GPT in simulating these states. More clarification here would be valuable.

I also have concerns about the comprehensiveness of the tested baselines. For instance, the GPT-based baselines might not fully leverage GPT's potential. Incorporating more chain-of-thought reasoning could lead to more grounded and reliable action predictions, similar to a world model that simulates multiple steps ahead. It would also be insightful to see a comparison showing performance without any proactive agents as a benchmark.

Additionally, the evaluation relies heavily on an in-house test set, raising questions about the broader applicability of the proactive agent approach. A more comprehensive evaluation, possibly across varied scenarios, would help clarify its generalizability and robustness.

questions:
See above. I may also suggest incorporating more examples in the method part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732550019227

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Invitation for Further Discussion

comment:
Thank you for your valuable feedback on our submission. We have carefully addressed each of your comments and believe further discussion could enhance our work. We invite you to review our responses and share any additional thoughts. Your input is invaluable in refining our research.

We look forward to your feedback and a constructive dialogue.

Thank you for your time and consideration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733301916273

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Brief Summary to Rebuttal

comment:
Our work proposes a novel data-driven formalization for developing a proactive agent that anticipates user needs and takes the initiative by suggesting tasks or providing information without explicit requests.

Our contribution is as follows:
1. We designed an automatic data generation pipeline, which simulates user activities and responses to tasks predicted by proactive agents across various scenarios.
2. We proposed the benchmark ProactiveBench, which is used to evaluate and enhance the agent's proactive behavior. The dataset contains a total of 6790 events in the training set and 233 events in the test set.
3. We trained a reward model that achieved up to 91.80% consistency with human judgments in terms of F1-score to automatically evaluate the proactiveness of LLMs.

Final Score:
- Reviewer 9sAE: 6
- Reviewer ZTiB: 7
- Reviewer fw7i: 5
- Reviewer tUPL: 5

Reviewers Summary:
- Reviewer 9sAE, ZTiB, fw7i found the concept of **proactiveness** to be novel, interesting, important, and promising.
- Reviewer 9sAE, tUPL, fw7i recognized our work as **a valuable contribution** to the field, highlighting its potential to **benefit future research**.
- Reviewer tUPL, ZTiB regarded our framework as robust and logically sound, particularly emphasizing the **user agent design**, which effectively simulates real users to assess agent behavior and refine task propositions.

However, some reviewers raised concerns or sought clarification on specific aspects of our work:
- **Robustness of the benchmark**: Reviewer 9sAE, fw7i expressed concerns about the robustness of our benchmark due to its generation by GPT-4o. We argue that the reviewer fw7i might have some **misunderstanding** in our benchmark.  We clarified that our test data is a **replay of real-world events**, thoroughly explained the components of our dataset, and conducted additional experiments without Chain of Thought (CoT) to further validate the strength and robustness of our benchmark. Notably, GPT-4o improved its F1-score from **54.38% (w/o CoT)** to **64.60% (w/ CoT)**. Similarly, the fine-tuned model achieved the best overall performance, with an F1-score of **66.25%**.
- **Data annotation process**: Reviewer 9sAE, tUPL, ZTiB were interested in the details of the data annotation process. To address this, we added further explanations of human annotation procedures in our paper and provided additional clarifications in the rebuttal.
- **Model performance**: Reviewer tUPL, ZTiB sought more insights into the performance of different models. In response, we presented detailed case studies and highlighted **additional failure modes** to provide a comprehensive understanding of model performance. We also conducted new experiments for further analysis. Multi-rank predictions consistently outperform single-rank predictions. Specifically, GPT-4o improved its F1-score from **64.60% (pred@1)** to **77.72% (pred@3)**. Using a reward model will make the model propose less unnecessary predictions. E.g. GPT-4o improved its False-Alarm from **51.85% (pred@1)** to **44.74% (w/ RM)**. Combining both strategies will boost the model's performance. LLaMA-3.1-8B improved its F1-score from **44.74%(pred@1)** to **74.76%(pred@3 w/ RM)**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary: This paper proposes a novel data-driven formalization for developing a proactive agent that anticipates user needs and takes the initiative by suggesting tasks or providing information without explicit requests.
(b) Strengths: The paper is generally well-written and easy to follow. The proposed approach seems interesting and novel. The experimental results seem to support the authors' claims.
(c) Weaknesses: The reviewers pointed out a few concerns and issues. Some technical details need more explanations.
(d) This paper is a borderline case. Some of the reviewers replied to the authors' rebuttal and appreciated their refinement and clarifications. Generally speaking, I think that the authors addressed the majority of the reviewers' concerns and comments, although some of the reviewers did not reply during the rebuttal phase.