CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing

Authors: Chen Yang, Chenyang Zhao, Quanquan Gu, Dongruo Zhou
Primary area: foundation or frontier models, including LLMs
Keywords: Agent, LLM

Abstract:
Sequential reasoning in agent systems has been significantly advanced by large language models (LLMs), yet existing approaches face limitations. Reflection-driven reasoning relies solely on knowledge in pretrained models, limiting performance in novel scenarios, while experience-assisted reasoning often depends on external experiences and lacks clear principles for selecting representative experiences. We address these limitations by proposing CoPS (Cross-Task Experience Sharing), a generalizable algorithm that enhances sequential reasoning by cross-task experience sharing and selection. In detail, CoPS leverages agents' experiences on previous tasks, selecting distribution-matched experiences via a provable pessimism-based strategy to maximize utility while minimizing risks from distribution shifts. Extensive experimental results on benchmarks like Alfworld, Webshop, and HotPotQA demonstrate that CoPS consistently outperforms state-of-the-art baselines, with superior sample efficiency suitable for resource-constrained scenarios. Theoretically, we show that the performance of our algorithm depends on both the quality of the pretrained LLM and the matching between the agent's task-dependent trial distribution and that generated by the LLM. Our work bridges the gap between existing sequential reasoning paradigms and validates the effectiveness of leveraging cross-task experiences, shedding light on the potential to improve agents' generalization and adaptability across diverse tasks. Our codes are released at [this link](https://anonymous.4open.science/r/AlphaMemory-05CA).

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730704708260

thread_type:
official_review

<thread_object>
creator: reviewer_ct7e

<content>
summary:
This paper proposes CoPS (Cross-Task Experience Sharing), an algorithm that aims to enhance LLM-based agents’ sequential reasoning by leveraging experiences across different tasks. The method uses a pessimism-based strategy to select relevant experiences from a memory bank while minimizing distribution shift risks. The authors evaluate CoPS on three benchmarks (Alfworld, Webshop, HotPotQA) and claim superior performance compared to baselines like Reflexion and RAP.

strengths:
- The core idea of leveraging cross-task experiences for LLM agents is novel and potentially impactful. The pessimism-based selection strategy provides a theoretically grounded approach to experience sharing.
 - The implementation is relatively straightforward and generalizable across different environments, requiring minimal task-specific modifications.
 - The empirical results show promising performance improvements, particularly with smaller models like Llama 3.1 8b, suggesting potential resource efficiency benefits.

weaknesses:
1. The idea seems similar to the Retrieval-Augmented Generation (RAG) technique. The paper lacks a comparison and discussion with traditional RAG approaches that also leverage external knowledge for LLM enhancement. The pessimism-based selection strategy could be better positioned against existing RAG retrieval methods like hybrid search and recursive retrieval[1].
2. The relationship to LLM agent memory mechanisms is insufficiently explored. For instance, no comparison is made with memory bank approaches that handle both short-term and long-term memory [2,3,4]. The paper should discuss how CoPS differs from or improves upon existing memory management solutions in LLM agents.
3. The current implementation lacks consideration of hybrid memory architectures that combine both short-term and long-term memory components. The system could benefit from incorporating recent advances in memory management like episodic memory modules or hierarchical attention mechanisms. Also, the paper doesn't address how the experience selection strategy could be enhanced with modern RAG techniques like recursive retrieval or adaptive retrieval mechanisms.
4. No ablation studies comparing different memory retrieval strategies (e.g., semantic search vs. keyword-based vs. hybrid approaches). Missing evaluation of memory retention and recall over extended periods, which is crucial for long-term agent deployment. Limited analysis of how the system handles memory updates and forgetting mechanisms compared to other memory-augmented LLM approaches[3].
5. Presentation could be improved. For instance, Fig. 1 does not with adequate explanation and it is hard to understand what does the example task means.

Refs:
[1] Retrieval Augmented Generation (RAG) for LLMs https://www.promptingguide.ai/research/rag

[2] Zhang, Zeyu, et al. "A survey on the memory mechanism of large language model based agents." arXiv preprint arXiv:2404.13501 (2024).

[3] MemoryBank: Enhancing Large Language Models with Long-Term Memory, https://ojs.aaai.org/index.php/AAAI/article/view/29946

[4] Wang, Guanzhi, et al. "Voyager: An open-ended embodied agent with large language models." arXiv preprint arXiv:2305.16291 (2023).

[5] A Survey on Retrieval-Augmented Text Generation for Large Language Models - NASA ADS https://ui.adsabs.harvard.edu/abs/2024arXiv240410981H/abstract

questions:
Apart from the weakness section, I also have a few questions:

1. How is the task defined? It is a bit unclear to me in the experiments -- does the author use the experience from the same benchmark or from other benchmarks as well?

2. How is the sampled experience number selected? How would the number affect the performance? Note that more sampled experience will result in more context length.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718709623

thread_type:
official_review

<thread_object>
creator: reviewer_cysd

<content>
summary:
This paper proposes CoPS (Cross-Task Experience Sharing), a method that can improve LLM agents by sharing distribution-matched experiences stored in a memory bank. CoPS first generates a trial experience from a LLM by conditioning on an initial state. Next, it sets a probability distribution that can approximately maximize the expected total reward while keeping the distribution close to a task-dependent distribution of the LLM. Then, it repeatedly samples candidate experiences from the probability distribution, and uses them as few-shot examples to sample an action from the LLM. This paper evaluates CoPS on three representative benchmarks such as ALFWorld, WebShop, and HotPotQA. The experiment results show that CoPS can achieve higher success rates than recent advancements such as Reflexion, RAP, and LATS on the benchmarks.

strengths:
S1. It is interesting to propose an idea that selects distribution-matched experiences from a memory bank for improving the performance of LLM agents.

S2. This paper demonstrates that CoPS can achieve higher success rates than recent advancements such as Reflexion, RAP, and LATS on the representative benchmarks such as ALFWorld, WebShop, and HotPotQA.

weaknesses:
W1. This paper aims to share cross-task experiences from a memory bank. However, it is rather unclear how to construct cross-task experiences in the memory bank. More specifically, how is offline data collected? Which LLMs are used as a policy to generate offline data? How many experiences are required to achieve the performance provided in the paper? How much different tasks can be used together to take advantages of cross-task experience sharing?

W2. To enable cross-task experience sharing, this paper proposes to find a probability distribution (in Equation 2.2) that can maximize the expected reward while keeping the distribution close to a task-dependent distribution of a LLM. However, this paper approximates the probability distribution by using cosine similarity between experiences. This approximation seems to make CoPS too similar to RAP.

W3. I am not sure that it is a fair comparison to constraint LAST to have similar running time with CoPS. LAST aims to improve the performance by using inference-time compute.

questions:
Q1. Please see the questions in the first weakness above.

Q2. Regarding the second weakness, what is the main difference between CoPS and RAP? And, what makes CoPS to perform better than RAP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730744288420

thread_type:
official_review

<thread_object>
creator: reviewer_4n3a

<content>
summary:
The paper proposes a method for utilizing experience from cross-tasks in order to improve performance. The proposed method samples experience from a probability distribution that captures reward and distance between experiences. Results are provided for widely studied ALFWorld, Webshop and HotPotQA environments. Authors also propose a theoretical framework for agents who are utilizing prior experience for performance improvement.

strengths:
- The paper is well written and easy to follow
- The paper focuses on an important aspect of using cross-task experience to improve performance in sequential decision making.

weaknesses:
- The paper fails to discuss relevant literature that uses cross-task experience for improving performance. For example O3D paper by Xiao et al. https://openreview.net/pdf?id=bkY8zEDdH9 proposes a method that uses offline trajectories from all tasks to distill knowledge for performance improvement. 

- Authors only provide results on 2 models from Llama family. How well does this method work with SOTA models such from GPT or Claude family

- The main contribution of the paper is proposing a method that benefit from cross-task experience. However, authors fail to illustrate this in given experimental results. What is the contribution of cross-task experiences (compared to same task experience) in the provided performance improvement?

questions:
Please refer to the weaknesses section for main questions. More minor questions are given below

- How is the reward r calculated in this work?

- How is distance between experiences are calculated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730787630467

thread_type:
official_review

<thread_object>
creator: reviewer_qjkw

<content>
summary:
The authors propose COPS, a method which utilizes an offline demonstration dataset of “experiences” and study how to use these experiences to solve downstream embodied and question-answering tasks. The authors motivate the work theoretically, drawing parallels to works in distribution selection and retrieval. Subsequently, they demonstrate the performance of COPS on AlfWorld, HotPotQA and WebShop.

strengths:
Strengths:
1. The paper is well written and easy to understand
2. The theoretical section is clear with well-defined notations used consistently through the manuscript.

weaknesses:
Weaknesses:
1. **Differences between COT/Few shot prompting?** The method is very similar to any form of few shot prompting and similar retrieval augmented generation works. The authors utilize an embedding model to calculate similarity between a current starting state and one sampled from the experience dataset. Additionally, their measure function for selecting the distribution to sample experiences from is a combination of reward and the similarity between the current start state and those in the dataset as defined by the embedding model. This to me, feels like a simple extension of RAG style methods with the additional reward label. This is interesting, yet on its own is not novel enough. Especially because none of the necessary offline RL analysis is conducted on this optimization. For instance, does this result in trajectory stitching, i.e., does the actor combine multiple subtrajectories in a demonstration to yield “stitched behaviors”? Can the agent outperform the demonstrations provided in the dataset? 
2. **Unclear what the quality of the demonstration dataset is?:** This brings me to my second point, it’s unclear what the quality of demonstrations utilized is. Do you only have successful trajectories in your demonstration? What if you only used suboptimal demonstrations? No such analysis is conducted. 
3. **Small experimentation setup:** The experiments section currently reads like that of a paper written in 2022 (which in LLM-application research is a significant period). No error bars/multiple seed runs are reported. The analysis is performed on 3 somewhat outdated benchmarks. This should be expanded. Why are some baselines missing for some of the benchmarks? For example, Table 3 does not report RAP performance. LATS is missing in Table 1?
4. Why is the pretraining performance being considered in your analysis? My understanding was that no LLMs had been pretrained for this work.
5. Since this is such an activate area of research, it would good to show how baselines discussed in the related works section actually perform. It is less convincing to read, *“However, their approach demonstrated poor sample efficiency, making it less suited for real-world agent settings where opportunities for trial and error are limited”* instead of seeing a plot with number of samples on the x-axis and success rate on the y-axis for this method.

questions:
1. What encoders are used for generating embeddings of start states?
2. Have you tried utilizing both start state and resultant actions as input to this embedding model? This would encapsulate a demonstration policy instead of just the start state distribution.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732427789170

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Summary

comment:
# Revision Summary

We summarize the key revisions made in our paper to address the reviewers' feedback:

---

## **Expanded Related Work**

- Added comparisons between CoPS and the O3D method, emphasizing differences in knowledge distillation and experience selection strategies, addressing **Reviewer 4n3a's Comment 1**.

---

## **Appendix Changes**

### **Appendix B**: Hyperparameter Tuning

- **Purpose**: Addressed hyperparameter impacts on performance, focusing on the effects of \(k\) (sampled experiences) and \(c\) (scaling factor).

- **Reviewer Concern**: Tackles questions from **Reviewer ct7E (Q2)** about the number of sampled experiences and their impact on performance.

---

### **Appendix D**: Quality of Demonstrations

- **Purpose**: Included ablation study comparing top-\(k\) and bottom-\(k\) trajectories to highlight the significance of high-quality demonstrations.

- **Reviewer Concern**: Added in response to **Reviewer QJKw's Comment 4**, emphasizing how demonstration quality affects performance.

---

### **Appendix E**: Experimental Robustness

- **Purpose**: Reported error bars and standard deviations for all benchmarks and models to ensure result reliability.

- **Reviewer Concern**: Responds to **Reviewer QJKw's Comment 5** about the lack of error bars and repeated experiments.

---

### **Appendix F**: Close-Sourced Models

- **Purpose**: Evaluated *CoPS* using GPT and Claude models to show its generalizability.

- **Reviewer Concern**: Directly addresses **Reviewer 4n3a's Comment 2**, demonstrating performance on state-of-the-art close-sourced models.

---

### **Appendix G**: Cross-Task Experience Impact

- **Purpose**: Compared performance with and without cross-task experiences, showing substantial improvements from sharing experiences across tasks.

- **Reviewer Concern**: Added in response to **Reviewer 4n3a's Comment 3** about the contributions of cross-task experiences.

---

### **Appendix H**: Retrieval Strategies

- **Purpose**: Performed ablation on different retrieval methods (semantic, keyword-based, hybrid) to analyze their impact on performance.

- **Reviewer Concern**: Responds to **Reviewer ct7E's Comment 4** about the lack of comparisons among retrieval strategies.

---

### **Appendix I**: Memory Constraints and Forgetting

- **Purpose**: Evaluated the performance under varying memory sizes and forgetting mechanisms to test robustness in constrained scenarios.

- **Reviewer Concern**: Tackles **Reviewer ct7E's Comment 4** about handling memory updates and forgetting mechanisms.

---

## **Figure and Presentation Improvements**

- Revised **Figure 1** with a detailed explanation of the example task to improve clarity, addressing **Reviewer ct7E's Comment 5**.

- Enhanced table formatting and captions for improved readability, including hyperparameter settings, retrieval strategies, and memory size analyses, ensuring results are easily interpretable.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a RAG-like approach, with multiple tasks. Reviewers generally commented that the work lacked significant novelty compared to the original RAG, while the experiments lacked rigor. There was a healthy discussion which was not sufficient for the reviewers to increase their scores, as concerns remained. The paper therefore fails to meet the bar for acceptance.