Title: ABBEL: LLM Agents Acting Through Belief Bottlenecks Expressed in Language
Primary area: foundation or frontier models, including LLMs
Keywords: multi-step, RL, exploration, memory, reasoning, beliefs, context management

Abstract: As the length of multi-step interactive language tasks increases, it becomes computationally impractical to keep full interaction histories in context.
We propose a general and interpretable approach: Acting through Belief Bottlenecks Expressed in Language (ABBEL), which replaces long multi-step interaction history by a belief state, i.e., a natural language summary of what has been discovered about task-relevant unknowns. 
Under ABBEL, at each step the agent first updates the prior belief with the most recent observation from the environment, then uses only the updated posterior belief to select an action.  
We systematically evaluate frontier models under ABBEL across six diverse multi-step environments, 
finding that they can generate interpretable beliefs while maintaining near-constant memory use over interaction steps. However, we observed inferior performance to the full context setting due to errors in belief updating. Therefore, we train LLMs to generate and act on beliefs within the ABBEL framework via reinforcement learning (RL). We devise a scheme for belief grading that associates rewards with the quality of the beliefs, and 
demonstrate its ability to improve ABBEL's performance beyond full context RL in a sandbox Combination Lock setting. In a realistic multi-objective QA environment, we highlight the strength of ABBEL's isolated belief states, allowing for the generation of compressed beliefs without degrading reasoning.
Our findings indicate that ABBEL opens avenues to study issues and potential remedies for agents in long context settings.