Title: Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: In-Context Learning

Credit Assignment

Large Language Models

Reinforcement Learning
Abstract: Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on domain-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pre-trained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i.e., the advantage function) through retrospective in-context learning (RICL).  We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states for temporal credit assignment. Extended evaluation on the BabyAI benchmark shows that RICOL significantly improves sample efficiency compared to traditional online RL algorithms while achieving performance comparable to imitation learning from expert demonstartions.  Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThus uses a pretrained LLM to read a hindsight trajectory (future states, actions, rewards) and generate textual feedback, turning sparse, delayed rewards into dense advantage estimates without a separate value network. Building on this, “Retrospective In-Context Online Learning (RICOL)” alternates between data collection, RICL-based advantage estimation, and advantage-weighted policy updates. Empirically, RICL matches ground-truth advantages with ∼10 trajectories (∼100× more sample-efficient than Monte Carlo) in a toy task, and RICOL outperforms reflection-based and PPO baselines on BabyAI benchmarks, remaining robust even when the feedback is noisy.\n\nSTRENGTHS AND WEAKNESSES:\n- This paper introduce a novel framework RICL motived by the RL by understanding the improvement by the ICL context fromth view of implicit reward guidance from the view of one-step soft policy iteration.\n- The paper contains comprehensive comparison on BabyAI benchmark, which validates the superiority of the proposed method.\n- The RL motivation is not consistent with the proposed method, and I will defer the clarifying questions into later part.\n\nQUESTIONS:\n- I’m concerned about the consistency between the RL motivation and the algorithm. Section 4 treats the ICL‐based update as a one‐step soft policy update, but since the implicit reward is generated by the updated policy itself, the reward changes each iteration. In other words, ICL does not genuinely perform maximum‐entropy RL because the reward is not well‐defined. Can the authors clarify this?\n- Substituting (3) into (1) gives $\\exp(A)\\approx\\frac{\\pi'}{\\pi_0}$. To recover $\\pi'$ on the RHS, shouldn’t the update be $\\pi_{k+1}\\propto \\pi_k\\,\\exp(A)$, which is the closed‐form for KL‐regularized policy updates in standard RL rather than maximum‐entropy RL? Am I misunderstanding something?\n- Computing $Z(s)$ at every iteration seems costly. Is this necessary, or can it be avoided without imposing a significant implementation burden?\n\nLIMITATIONS:\nI would suggest the author to provide more detailed experimental setups, e.g., how the reward is chosen for PPO training in Section 6.4.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how to utilize the in-context learning ability of LLMs to do credit assignment in a more informative way when learning under sparse reward. It first updates per-step advatange function by utilizing the policy change before and after in-context learning, then updates the policy via AWR algorithm. Empirical results on four BabyAI tasks show that this method outperforms baseline methods w.r.t. both sample efficiency and final performance on most tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The paper is well-motivated and proposes a novel way to enable temporal credit assignment with in-context learning, which is well supported by theoretical derivation. \n2. The experiments align well with the methodology design, which validate the importance of the key design choices in the algorithm, and the improvement over the baselines is generally significant and well supports the motivation of the paper. \n\nWeaknesses: \n1. The evaluation tasks considered in the paper are relatively limited, as also discussed by the authors in the conclusion section. The effectiveness of the algorithm heavily depends on the in-context learning ability of the LLMs, so I think it's important to see if such ability can scale up to and benefit more complicated tasks like math reasoning.\n\nQUESTIONS:\n1. Theorem 4.1 proves the existence of a reward function that is proportional to the log probability change of the policy before and after in-context learning. But it does not mean that this reward function will be the same as the true reward function of the MDP right? Empirically they seem to align well, but logically how do you ensure this?\n2. The proposed method is more sample efficient than the baseline methods. But I was wondering that how do they compare w.r.t. the total computational cost? As your method needs to reflect on each step of each collected episode, I'll expect that it will consume more tokens than the baseline methods, which either reflect on the whole trajectory once or do no reflection right? As you are evaluating on language games in simulation, the total computational cost may be a more important metric to reduce compared to the sample cost, as they are just tokens and have no difference practically. \n3. In Section 6.3, how do you drawer the conclusion that RICL is safer than ICL based on its higher accuracy? I agree that it's more effective, but I'm not clear about the logic of how you can say that it's also safer based on a single accuracy score. \n4. In figure 4, there seems to be no result for PPO (3B) on the \"open\" task?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method called retrospective in-context learning (RICL) to address the temporal credit assignment problem in reinforcement learning with large language models (LLMs). It effectively tackles language-conditioned, sparse-reward decision-making tasks with high sample efficiency. The proposed method first transforms sparse reward signals into advantage functions and then leverages these advantage functions for weight regression to extract additional informative signals. Empirical results demonstrate that RICL achieves both efficacy and efficiency in solving decision-making tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. This paper demonstrates that the ratio of two policies can be used to estimate the advantage function and the advantage function can be used to solve the temporal credit assignment problem. \n2. The empirical results show that RIOCL outperforms the baselines in the BabyAI scenarios.  Experiments on comparing RICL with the base model and the ICL method show the effectiveness of the RICL. Compared with the MC method, RICL achieves an accurate advantage estimation with fewer samples.\n3. The paper is well-structured and clear enough.\n4. The paper provides a novel method to solve the temporal credit assignment problem in sparse-reward decision-making tasks with high sample efficiency. It can provide a new technique for LLM agents to solve the sparse-reward decision-making tasks.\n5. The idea of merging Retrospective In-Context Learning (RICL) with Advantage-Weighted Regression to form a full online RL loop is novel and interesting. \n\n\n**Weaknesses:**\n\n1. The empirical study demonstrates effectiveness in structured environments such as BabyAI but lacks evaluation in more complex, real-world scenarios.\n\n2. The proposed methods need an additional reflector LLM and a second forward pass of the policy, which may introduce significant computational overhead. \n\n3. The paper compares the RIOCL with the PPO(3B) in Figure 4, which seems to be unfair. The base model should have the same size as the RIOCL.\n\nQUESTIONS:\n1. Can you elaborate more on the computational overhead of the RICL? Will it take much longer inference time than the other methods mentioned in the paper?\n2. Can you elaborate more on the difference between the RICL and the previous methods mentioned in the paper, such as the Reflexion and RICO-GRPO?\n3. Can you provide the PPO(7B) model results to have a fairer comparison with the RICL in BabyAI?\n4. Can you explain why the open task performs worse than the GPT-4o-mini while the other three tasks perform better than the GPT-4o-mini in Figure 4? Is there any specific reason for this observation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel Retrospective In-Context Learning (RICL) method and an associated online reinforcement learning framework, Retrospective In-Context Online Learning (RICOL), aimed at improving temporal credit assignment for LLM-based agents in sparse reward environments. The key insight is to leverage a Reflector LLM to generate verbal feedback based on observed trajectories, which is retrospectively incorporated into the actor LLM's prompt to update its behavior in a trajectory-specific manner. By comparing the log-probabilities of actions before and after in-context updates, the method estimates advantage functions, converting sparse feedback into dense supervision signals.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed use of advantage estimation is grounded and offers an elegant alternative to traditional Monte Carlo or value-based methods, particularly under sparse rewards.\n2. The Reflector LLM provides fine-grained, state-specific verbal feedback, addressing limitations of prior trajectory-level reflection approaches like Reflexion.\n3. Experimental results are comprehensive, spanning interpretable synthetic tasks and more challenging BabyAI environments, with consistent improvements in sample efficiency and task success rates.\n\nWeaknesses:\n1. The approach depends on the Reflector LLM generating high-quality, task-relevant feedback, and may degrade under systematic biases or poorly designed prompts.\n2. Theoretical guarantees are contingent on idealized assumptions regarding environment stationarity and policy-reward consistency, which may not hold universally.\n\nQUESTIONS:\nThe effectiveness of RICL hinges on the quality of verbal feedback generated by the Reflector LLM, yet:\n1. Does the Reflector require task-specific prompt engineering or fine-tuning?\n2. Have the authors tested RICOL with intentionally flawed or biased Reflector outputs to assess worst-case robustness?\n\nLIMITATIONS:\nThe same as the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}