Affordable Generative Agents
Authors: Yangbin Yu
Qin Zhang
junyou li
QIANG FU
Deheng Ye
Venue: Accepted by TMLR ()

Abstract: The emergence of large language models (LLMs) has significantly advanced the simulation of believable interactive agents. However, the substantial cost on maintaining the prolonged agent interactions poses challenge over the deployment of believable LLM-based agents. Therefore, in this paper, we develop Affordable Generative Agents (AGA), a framework for enabling the generation of believable and low-cost interactions on both agent-environment and inter-agents. Specifically, for agent-environment interactions, we substitute repetitive LLM inferences with learned policies; while for inter-agent interactions, we model the social relationships between agents and compress auxiliary dialogue information. Extensive experiments on multiple environments show the effectiveness and efficiency of our proposed framework. Also, we delve into the mechanisms of emergent believable behaviors lying in LLM agents, demonstrating that agents can only generate finite behaviors in fixed environments, based upon which, we understand ways to facilitate emergent interaction behaviors. Our code is publicly available at: https://github.com/AffordableGenerativeAgents/Affordable-Generative-Agents.

Published in Transactions on Machine Learning Research (08/2024)
Affordable Generative Agents
Yangbin Yu Tencent
yangbixegu@tennenl.ccom
Qin Zhang Tencent
adrienchang@@teneenl.com
Junyou Li Tencent
jungouul@ätencent.com
Qiang Fu Tencent
leonfu@tencent.coo
Deheng Ye Tencent
dericye@leencenl.com
Reviewed on OpenReview:
https: openreview. net/ forum? id= 7tLYbcg5D57
Abstract
The emergence of large language models (LLMs) has significantly advanced the simulation of believable interactive agents. However, the substantial cost on maintaining the pro- longed agent interactions poses challenge agents. Therefore, in this paper, we develop Affordable Generative Agents (AGA), framework for enabling the generation of believable and low-cost interactions on both ggetteenviromment and inter-agents lovels. Specifically, for agent- interactions, we substitute repetitive LLM inferences with learned policies; while for inter-agent in- teractions, we model the social relationships between agents and compress auxiliary di- alogue information. Extensive experiments on multiple environments show the effective- ness and efficiency of our proposed framework. Also, delve into the mechanisms of emergent believable behaviors lying in LLM agents, demonstrating that agents can only generate finite behaviors in fixed environments, based upon which, we understand ways to facilitate emergent interaction behaviors. Our code is publicly available al:: https: cww.coomoooaaii //tttt/www...cccccllecceeeteeeee
the deployment of believable LLM-based
1 Introduction
Intelligent agents achieving human-level performance in complex tasks has attracted research efforts for years Mnih et al. (2015): :
Ye et al. 2020aa b): Zhao et al. (2023). Recently, constructing agent based large
language models (LMMs) for the simulation of believable human behavior has shown substantial promise
(d Rocha Costa 2019 Park et al.) 2022 Lan et al.) 2023 Xi et al. 2023a). A belicvable LLM agent is characterized by its ability to interact with other agents or humans, respond to environmental changes, and generate responses and behaviors that are perceived by humans as authentic, natural, and in alignment with expectations. Such agents could be utilized for diverse applications, such as simulating the potential impacts of policies (Zhang al. 2023a Törnberg et al. (0023) or sociological theories on humans (Gao al. 2023), creating Non-Player Characters (NPCs) [Laird & VanLent) 2001 Gong et al. 2023 in games, and

---

Published in Transactions on Machine Learning Research (08/2024)
developing embodied intelligence akin to social robots (Cherakara al. 2023 domestic service robots (Pugg et al. 2020 Yang et al. 2023b).
However, while we appreciate the potential and capabilities of these agents, their cost implications also merit attention. The crux of the matter that frequent and prolonged use of LLMs to handle various types of agent interactions can result in substantial cost consumption. Previous studies have highlighted this issue. For instance, Park et al. (2023) construct a virtual town, aka the "Stanford Town", to simulate Park
agents spontancously exploring and interacting with the environment to produce believable behaviors in open world, with the entire study incurring costs of thousands of dollars. Wang et al. (2023b) mimics embodied agents, fulfilling both emotional and rational needs through interaction, with simulations of 2 to 3 agents costing several dollars per hour. The cost of believable agent interaction has hindered its application in scenarios requiring large-scale usage. For example, simulating long-term, large-scale policy impacts and sociological phenomena, as well as using LLM agents as NPCs in games.
Many methods have been developed to reduce the cost of invoking LLMs, but most of the research has focused fixed datasets, such as COQA (Reddy et al. 2019), QQP (Wang et al. 2017), and RTE (Poliak] 2020) datasets. These tasks have clear answers, well-defined difficulty levels, and independence between tasks, typically solvable through single-turn QA interactions. However, the situation is quite different for believable agents, which lack clear answers and involve interactions that require the agent's 's long-term memory and coordination between different modules. These characteristics make challenging to apply most existing methods in this context.
Therefore, in this paper, we focus the believable behavior interaction simulation of LLM agents in open- world scenarios and propose low-cost framework, termed Affordable Generative Agents. We abstract the behaviors of believable agents into two modes: agent-environment interactions and inter-agent interactions. For the egvnt-environment interactions, we propose the Lifestyle Policy to reduce the redundant costs of generating repeated responses by agents; On the other hand, for the inter-agent interactions, we propose the Social Memory to reduce the generation of repeated information in multi-round interactions. applicability of our techniques, we have conducted extensive experiments using well-known environments, including the "Stanford Town" (Park al. 2023) and the VirtualHome (Puig et al. 2018), to demonstrate that, while achioving the same performance, the consumption of generating believable agent behaviors be significantly reduced.
examine the
Furthermore, to help understand why our approach works, we delve into the mechanics of how LLM agents generate believable bchavior, demonstrating that there is an upper limit to the belicvable behavior that emerges from LLM agents in a fixed environment. This implies that all agent-environment interactions can be entirely covered by policies. Based upon this finding, we also propose of optimization strategies for generating richer believable behavior.
In summary, contributions are follows:
We propose Affordable Generative Agents, low-cost simulation framework for the believable be- havior simulation of LLM agents in open-world scenarios, with optimization strategics for both agent-enviromment and inter-agent interactions.
We propose several evaluation methods and conduct extensive experiments in benchmarking envi- ronments to validate the effectiveness of our framework.
We analyze the mechanism of believable behavior generation by LLM agents, demonstrate the up- per limit of the richness of emergent behaviors, and further propose corresponding optimization strategies.
2
Related Work
This work explores the development of efficient generative behavior leveraging LLM. We discuss the most related works below.
2

---

Published in Transactions on Machine Learning Research (08/2024)
2.1 LLM Agents
The development of LLM agents has sparked myriad of applications. Yang et al. (2023a leverage multi- turn dialogues with agents to autonomously address NLP tasks. [L et al. (2024) find that the performance of LLMs scales with the number of agents instantiated across a range of tasks. Deng et al. 20033) Nakano let al. (2021), and Yao et al. (2022a) integrate pre-defined APIs with LLMs to enable agents to navigate web pages effectively. Xu et al. (0023) and Light et al. (2023) deploy LLM agents in text game environments, engage in complex, strategic gameplay. The most prominent among these works is the construction of LLM agents that interact in open environments
showing the capacity
creale believable behavior. For instance, Park et al. (2023) builds virtual town simulating the social life of 25 agents, while Brohan et al.) (003)) utilizes an LLM to create a robot capable of completing houschold chores through natural language interaction. Wang et al. (2023a) implements an agent capable of playing Minecraft expertly, when compared to traditional game-playing methods Kanervisto et al.
(0022) ; Lin et al. (2022). Efforts have been made to
optimize various modules such as reasoning (Wei et al.
2022 Wang et al., 2022 Xi et al. (023b), planning
(Song et al. 2023: Yao et al., 2022b), reflection (Shinn ct al. 2023), and memory (Zhong et al. 20233 Huang
et al.] 2023b),
create more realistie interaction environments (Hong el. al. 2023b] Huang et al. 2023a
However, the development of affordable and believable agent interactions is missing in the literature. In this paper, we focus achieving believable agents with the same performance at low cost.
2.2 The Cost of LLM Interaction
The cost of using LLM is directly proportional to the number of LLM invocations, the length of the prompt, and the model's performance. The cost. issue has al always been a key focus the L.M-related field. Numerous methods have been proposed to reduce the cost of using LLM, such as Cascaded Invocation of LLM (Yue)
et al.] 2023; Chen et al.) 2023a Nottingham et al.] 2023). summary (Laban et al., 2023 Arefeen et al, 2023
Mu et al. 2023) and batch (Lin et al. 2023 These methods primarily optimize the cost of single
invocation of LLM for QA tasks. Zhanng et al. (2023b) has built a cost-efficient multi-agent framework, but
it only solves QA lasks on fixed datasets. The simulation of believable behavior in the open world requires uninterrupted calls to the LLM for interaction. Moreover, multiple modules need to combine a large amount of contextual information for LLM reasoning and planning, making cost more prominent issue. Simply optimizing for single QA invocations cannot effectively reduce costs. Furthermore, the open world does not have fixed evaluation method, and the difficulty of tasks is hard to categorize, rendering many existing methods unsuitable (Chen et al. 2023a Zhang et al. 2023b). Kaiya et al. (023) proposes generative agents with low-cost interactions, but their effectiveness is highly dependent on the tailored scenarios. Hence, there is a pressing need for low-cost solution that can effectively handle the challenges of believable interaction simulation in various open environments using LLMs. This paper aims to address this gap by proposing a cost-cfficient scheme for simulating believable bchavior with LLMs in open environments.
3 Method
Our work is inspired by Park et al. (2023), who proposed Generative Agents, a framework for simulating believable behaviors of LLM agents in a virtual town. This framework enables agents to perceive their environment and maintain a memory stream to record past interactions and sclf-reflcctions. By retricving relevant content from this memory stream, the framework helps agents generale believable responses. In- spired by this, We achieve generative agents in a low-cost way by optimizing agent-environment interactions and inter-agent interactions. Figure | illustrates our method in these two types of interactions, with gray arrows representing the Generative Agents framework's implementation and blue arrows representing approach. In the following sub-sections, we will describe the optimization strategies for these two categories of interactions in detail.
3.1 Agent-enviromment interactions
The interaction between an agent and its environment can be modeled as a policy function mapping obser- vations to actions. This policy can be implemented through reinforcement. learning (Kaelbling et al. 1996)
3

---

Published in Transactions on Machine Learning Research (08/2024)
(a) Agent-environment interactions
Sub-Plam
Action
Retrieve
Condition
Lifestyle Policy
Decomposiion
Update
Match
AGA
Baseline
(b) Inter-agent interactions
Recrive
Mernory Stream
Relevant Filte
Next Dialogue
interlocutior
Perceive
Previous Dialogue
Update
Summary
Summary Events
Relationship & Feeling
Social Impression Memory
Figure 1: Our proposed method for optimizing (a) Agent-cnviroument interactions and (b) Inter-agent interactions. The gray arrows represent the baseline (Generative Agents) implementation, while the blue arrows represent our approach.
rule-based finite state machines (Lee & Yannakakis 1996), behavior trees (Colledanchise & Ögren 2018 etc. LLMs trained on a vast amount of data show immense potential in policy formulation (Sun et al. 2023). With carefully designed prompts, LLMs can generate logically consistent and believable decisions. The decision-making capabilitics of LLMs have been appliod to a wide varicty of applications, but the cost associaled with each interaction has been overlooked. On the other hand, behavior trees operale without additional costs, but they require pre-written scripts and can not adapt dynamically environment changes.
Drawing inspiration from the Casc-Basce Reasoning (CBR) method (Slade 1991 Spalzzi 2001), which addresses new problems by adapting solutions to previously solved, similar issues, we introduce a new approach - Lifestyle Policy, which aims to retain the flexible interaction capabilities of the LLMs while achieving low-cost deployment. The functionality of Lifestyle Policy is to minimize costs by reusing the similar inference processes of an LLM. To this end, we devise agent-environment interactions containing two stages: Plan Decomposition and Policy Reuse, expanded as follows.
Plan Decomposition The stage occurs when the agent encounters situations it has never seen before. Due to their inherent zero-shot capabilities (Kojima et al. 2022 LLM agents can respond to unseen tasks in believable way. Given the description of the environment and agent's 's information, a rough plan or task is generated by an LLM. The Plan Decomposition breaks the plan down into sub-plans and then converts sub-plans into specific actions, just like other LLM multi-agent frameworks (Chen et al. 2023b Li et al., executable condition based on the
2023 Hong et al. 2023a).. An additional step involves generating action sequence. The determination of execution condition is highly correlated with the environment. For instance, in the case of domestic robot (Brohan et al. 2023), the condition of a task include the existence of the related interactive items and the consistency of their states. the plan can be successfully exccuted, then the whole graph from plan to actions and corresponding condition will be updated to Lifestyle Policy.
Policy Reuse The stage occurs when the agent encounters previously seen situations. Given a plan, we use embedding models to extract features from plan descriptions and compare them with stored plan features in the Lifestyle Policy using cosine similarity. When the cosine similarity excoods a set threshold, we consider it a match for the same plan. Setting the threshold too low may result in mismatched plans, causing agents to perform inappropriate actions, while setting it too high may lead to redundant plans being recorded in
4

---

Published in Transactions on Machine Learning Research (08/2024)
the policy. In our implementation, set the threshold at 0.97. More details and examples of different similarity levels are provided in the Appendix A Subsequently, we assess whether the current environment meets the execution condition associated with the retrieved plan by comparing if the interactive items in the current environment exist and have consistent states as specified in the condition. If the condition is met, we proceed with the actions specified in that plan. is important to note that a single plan may have multiple conditions tailored for different scenarios. We iterate through all the conditions until we find one that the current environment satisfies.
Figure I (a) illustrates the complete interaction process. Upon receiving observation that includes en- vironmental data and the agent's personal profile, the LLM agent formulates a plan. For the bascline, all plans will invoke the LLM to decompose and generate corresponding actions. In our method, the Lifestyle Policy retrieves a plan with a similar description and condition met by current environment. If a match is found, the agent executes the corresponding actions. Otherwise, the LLM is invoked to decompose the plan. Due to the superiority of LLMs, the plan decomposition ensures that the actions generated are of high quality, believable, and self-consistent. Meanwhile, the Lifestyle Policy aims to reuse existing plans to avoid unnecessary costs. Due to the high similarity threshold, the replacement policy does not affect the agent's 's original performance. For instance, the agent Klaus in virtual town generates an activity like 'Klaus is waking up and completing his morning routine." In the Lifestyle Policy, this activity matches similar one. "Klaus is waking up and getting ready for the day," and executes the corresponding sub-activities such as "stretching and getting out of bed," * "washing his face and brushing his teeth," and 'heading out the door to go the library."
In the lifelong simulation of LLM agents, there are large number of redundant, repetitive, and trivial decisions, such as in Generative Agents, where each agent repeatedly makes decisions like brushing its teeth, washing its face, etc. Replacing the process of invoking LLM reasoning with corresponding policies can save significant cost, which will be shown in our experiments.
3.2 Inter-agent interactions
In agent interactions, dialogue is the primary communication method. These agents must integrate sub- stantial auxiliary information into the dialogue prompt to demonstrate advanced cognitive capabilities, dis- tinguishing them from basic chatbots driven by LLMs. For instance, agents need to incorporate relevant information about the dialogue participants and topic-related events retrieved from memory into the dialogue prompt to enrich the conversation. However, this data is often less comprchensive and more fragmented than the knowledge bases of question-answering systems (Cui et al. 2023). Merely appending memory snippets to prompts could reduce response quality and lead to higher processing costs from increased prompt length (Krishna et al., 2022). Furthermore, agent interactions differ depending on their relationships and mutual perceptions in social sinulations. Thus, agents need to gather enough information to form contextually appropriate responses without suffering performance drops or incurring extra costs from excessively long prompts.
The gray arrows in Figure bb) illustrate the inter-agent interactions provided by the baseline method. Specifically, agents retrieve all relevant events from the memory stream based on the previous round of conversation. LLMs then use these relevant events along with the prior conversation as prompt to generate the next round of dialogue. However, this approach has several issues. First, trivial and repetitive relevant events can make the prompt excessively long, sometimes exceeding the LLMs'' token limit. This not only increases computational costs, especially in multi-agent, multi-turn dialogue scenarios, but also makes it difficult for the LLMs to capture key information, leading to inappropriate responses.
Drawing inspiration from existing text compression technologies (Liu et al. 2023) and social model (McCoy ct al. 2011), we introduce the Social Memory module, aimed at reducing the cost of inter-agent interactions and enabling the agents to produce more appropriate dialogues. It consists of two key components: Summary Events and Relationship & Feeling, Summary Events store summaries of relevant events for each interlocutor, while Relationship & Feeling models the social connections between the agent and the interlocutor using keywords such as 'Relationship: acquaintances, Feeling: friendly." In our implementation, we first retrieve relevant events based on previous dialogue content, filter out new events, and summarize them into Summary
5

---

Published in Transactions on Machine Learning Research (08/2024)
Social Impression Momory
First Time
Relationship:
Unknown
Interaction
Feeling:
Unknown
Summary Events: None
After Conversation
Social Impression Memory
Second
Relationship:
Acquaintances
Interaction
Feeling:
Positive
Summary Events:
Figure 2: An example of relational evolution driven by the updating of Social Memory following converse- tional interactions.
Events. These summaries increase information density while controlling text length. We then combinc the agent's personal information, event summaries, relationships, feelings, and previous dialogues into prompt for L.Ms to generate the next round of dialogue. After the dialogue ends, we input the current agent's relationship, feelings and the summary of multiple rounds of dialogue into the LLM to update the Relationship & Feeling, as shown in Figure 2. Implementation details of each module of Social Memory are provided the Appendix B
Social Memory reduces costs of conversation by compressing auxiliary information, offering precise descrip- tions of social information to assist agents in responding consistently and appropriately. In the baseline, each conversation requires retrieving 30 to 45 relevant events, which occupy about 2000 tokens in the prompt. In contrast, our method condenses this information to approximately 100 tokens by using Summary Events. It also supplies high-information events to aid agents in generating accurate replies relevant to the conversation. Furthermore, as illustrated in Figure 2 Social Memory updates after each interaction, providing an explicit depiction of interlocutor relationships. For example, we observe that the student Klaus enjoys discussing his studics with others and occasionally mistakes the cafe owner Isabella for fellow student in baseline. However, with our method, which employs clear relationship update mechanism, the agent discern social relationships during conversations. Meanwhile, This mechanism be employed to investigate the dynamics of relationships between agents, thereby analyzing the evolution of community relations and the emergence of social behaviors.
4
Experimental Setup
This section focuses on the experimental environment and evaluation protocol.
4.1 Environment
We evaluate our approach under the following environments:
Generative Agents. Park et al. (2023) creates a simulated town, populating it with multiple agents powered by LLMs that interact in natural language, generating believable individual and emergent social behaviors. Agents are required to create daily schedules, which are refined into hourly plans, and then translated into minutc-level activities. All interactions are exccuted through natural language descriptions, accompanied by corresponding sprites and emojis for representation.
The Generative Agents framework comprises five modules: Perceive, Plan, Retrieve, Reflecl, and Act. The Perceive module is responsible for sensing surrounding game objects. The Plan module generates plans and activitics. Retrieve tasked with retrieving relevant events from memory. Reflect deeply considers past events, and Act determines the location and objects of interaction, well generating corresponding emojis.
6

---

Published in Transactions on Machine Learning Research (08/2024)
We implement AGA on Generative Agents framework and conduct experiments in the provided 3-person and 25-person environments.
VirtualHome. Puig al. 2018 provides 3D simulated domestic setting where agents engage with their surroundings via programmatic atomic actions. The environment encom 1 115 specific types of items, such as plate, desk, and microwane. Each item possesses attributes that determine the permitted interactions, including GRABBABLE, LIEABLE, etc., as well as current states like OPEN and CLOSED. Compared to Generative Agents, VirtualHome offers more concrete and realistic interaction. An interaction involves specific actions, items and the item states.
We implemented an LLM framework on VirtualHome and applied AGA to it. The detailed imple- mentation is presented in the Appendix
4.2 Language Models Adopted
For fair comparison, all agents are empowered with GPT-3.5-Turbo (Wu al. 2023 same as the Gen- erative Agents work PPrk al.) 2023). Besides, we also conduct tests using the more advanced GPT-4
(Achiam et al. 2023
4.3 Evaluation Protocol
We focus on LLM cost and performance. For LLM cost, to avoid regional and time-based price differences, use the same model to compare the number of tokens consumed by different methods for the same task. To comprehensively evaluate the agent's 's behavior, we first reuse the evaluation methods of Genera- tive Agents excluding manual assessments. These methods encompass interviewing agents with manually designed questions and conducting case study-specific activities.
Moreover, we attempt to evaluate the emergent social behaviors generative agents from two other per- spectives
Relationship Evolution. The relationships among agents are updated by Social Memory after interactions. The evolution of these relationships significs the formation of community ties, which can be utilized examine the social behaviors.
Behavioral Assessment. Chiang Lee 2023)) have validated LLM as an evaluator, demonstrating comparable performance to expert evaluators. Consequently, we employ GPT-4 evaluate the activities and dialogues generated by LLM agents. A 5-point Likert scale questionnaire deployed to differentiate whether these activities and dialogues are from human Al-generatod.
For VirtualHomee, agents are designed experience a full day at home, generating household tasks and decomposing them into specific actions. We evaluate the success rate of plan execution.
5
Evaluations
We conduct extensive experiments to validate method, assessing both the token consumption and the performance. Our experiments demonstrate that the AGA framework can reduce the cost while maintaining equivalent performance.
5.1 Results Generative Agents
Token Consumption Figure 3 illustrates the ablation study of token consumption based on multiple two game day simulations of 3-person town with GPT-3.5-Turbo. Compared to the baseline, the cost. of using only the Lifestyle Policy is 40.2% of the original, while using only the Social Relationship Memory 58.6%, and the full AGA framework is 31.1%. It should be noted that the advantage of the Lifestyle Policy lies the reuse of existing LLM-generated strategics. Hence, both Lifestyle Policy and AGA conduct experimental comparisons after running multiple iterations, and then storing certain number of policies.

---

Published in Transactions on Machine Learning Research (08/2024)
Baseline
Lifestyle Palicy
Social Memory
AGA
Token Consumption
1e6
Figure Ablation study on token consumption. Baseline means Generative Agents. Lifestyle Policy and Social Memory mean only using the corresponding module.
Table 1: The token consumption with varying population settings
Setting
Baseline
AGA
Ratio
3
4.548M + 0.36M
1.17MM + 0.08M
31.1%
25
25.41M = 0.96M
10.86M + 0.13M
42.7%
We also present the results of different configuration simulations in Table 1 where the cost for 25-person town is 42.7% of the original. In the 25-person town, the probability of interactions between agents increases with the rise in population density. After triggoring a dialogue, an agent adjusts its plan, invoking the reactions module to handle unexpected events. The occurrence of more events leads to more frequent oulput of reflections for deeper thinking. In order to ensure the flexibility of the LLM, we did modify the reactions and reflections modules of the Generative Agents, which incurs additional costs for the 25-person version.
Table 2 provides detailed comparison across various models and token types in 3-person town setting, The data suggests that GPT-4 tends to consume slightly more tokens due to its tendency to generate more detailed plans and longer responses. Importantly, the majority of token consumption is attributed to input tokens, a reflection of the need to incorporate substantial relevant. information, constraints, and one-shot examples into the prompt. Figure presents the accumulatod token consumption over a two-gamo-day simulation in 3-person town using GPT3.5-Turoo. The AGA demonstrates slower token consumption compared to the Baseline.
Case Study We employ the same method used for Generative Agents to evaluate believable agents this part. We conduct interviews with agents to evaluate their ability to remember past events, strategize for future tasks based on those memories, react appropriately to unexpected situations, and reflect their previous actions to improve their future performance. We use the same questions Generative Agents to inquire about five aspects: self-knowledge, memory, plans, reactions and reflections. The questions and answers are shown in the appendix C Results indicate that AGA does not impair the original performance of baseline. Agents can recall past experiences and generate believable responses. For instance, the interviewed agent Klaus Mueller accuratoly answers questions about his occupation, correctly describes other agents (such Wolfgang Schulz and Kane Martinez), and aware of ongoing community events, like Sam running for office and Isabella hosting Valentine's Day party.
8

---

Published in Transactions on Machine Learning Research (08/2024)
Table 2:
The token consumption with varying models
Method
Model
Iuput tokens
Output tokens
Total tokens
GPT3.5 Turbo
4.212M + 0.340M
0.313M 0.024M
4.525. + 0.364M
Baseline
GPTT
4.682.M + 0.398M
0.398. = 0.029.M
5.069.M + 0.408M
GPT3.5-Turbe
1.329M + 0.085.M
0.092.M + 0.006M
1,420M E 0.091
AGA
GPT-4
1.467M = 0.129.
0.130M 0.013M
1.597M 0.142M
le6
Accumulated Token Consumption Time
Accumulated Token Consumption
Baseline
0
20
40
50
Game Time (Hour)
Figure 4: Accumulative token consumption over game time for different methods
In addition, we can validate the emergence of social behavior through specific events. In the 25-person town, Isabella Rodriguez, the owner of the coffee shop, plans to host a Valentine's Day party and invite as many people as possible the following day, while Sam is running for mayor. In the end-to-end evaluation of the baseline, during the two-day simulation, the number of agents aware of Sam's mayoral candidacy is 8, and the number of agents who know about Isabella's party 13. our method, after multiple experiments, we observe corresponding numbers ranging from 4 to 12 for Sam's candidacy and 12 to 17 for Isabella's party. AGA achieves similar results to the baseline in terms of information diffusion.
Relationship Evolution In Social Memory, all initial relationships between agents are set "Unknowm" and subsequently updated following cach interaction. We assess the emergence of social behavior by moni- toring the relationship evolution. For example, in the 3-person town, Klaus Mueller and Maria Lopez discuss their research at cafe, establishing a colleague relationship. Isabella Rodriguez, the cafe owner, engages in conversation while serving Klaus and Maria, transitioning to an acquainlance. A specific demonstration of social behavior is that agents modify their relationships with other agents through social interactions, thereby strengthening their ties with the community. We also conduct experiments on the 25-person environment. For visualization, we list all relationship terms from the simulation and have GPT-4 rate them on scale of 0 to 10 for relationship closeness. A higher score signifies closer relationship (For instance, 0 represents strangers, 3 is for acquaintances, 5 for co-workers, and 10 for married couples). The result is shown in Figure The data map reflects the actual relationships between agents. For example, in the virtual town, John Lin (JL in Figure 5 is profiled as living with his wife, Mei Lin (ML), and son, Eddy Lin (EL). Therefore, their relationship scores are all a perfect 10. Additionally, the data map shows changing relationships between agents, such as the relationship score between Tom Moreno (TM) and Yuriko Yamamoto (YY), who do not know each other, shifting a 5. After conducting multiple experiments and calculating average scores, the data suggests evolution towards acquaintance-based community.
9

---

Published in Transactions on Machine Learning Research (08/2024)
HJ
2
4.4
10
EL
3.9
1.
8
JL
YY
23
4.6
0
4Y
0 O BE
B2
ML
8.4-
1.
AS
30
2 2.5
5.488 5
CG
3.23833.748 O
4.2.774.
JM
POLE 1.5
3.6 3 2.
0
57
5.5 5.4
AK
4.8
5.3 22
48 0
O
IR
46
CO
3.5
3.4
JaneM
AB
2
RP
2565
RyanP
MariaL
HI
TM
EL
YY SM
ML
AS
GR
CG
WS
JM
KM
AK
IR
AC
CO
FL
JaneM LW
AB
AP
TT
RyanP MariaL
Figure 5: Average relationship score map between agents. The x and axes represent the initials of the
names of 25 agents. When identical abbreviations occur, the full names are retained.
Table 3: Human-likeness score evaluated by GPT-4
Method
Activity
Dialogue
Baseline
3.13 0.19
3.97 0.02
AGA 3.21 + 0.29
4.01 = 0.01
Behavioral Assessment The activities and dialogues conducted by the agents are recorded and evaluated using GPT-4, employing 5-point Likert scale to discern whether the responses originated from an Al or human. In this scale, a higher score indicates closer resemblance to human-like responses. The complcte questionnaire is provided in Appendix D.I. The results presented in Table scores comparable to the baseline. Due to the advantages of LLMs in natural language dialogues, both methods approach human-level performance. However, the performance of agents activity is considered potentially indicative of either Al or human behavior. The primary reason for suspecting AI the overly detailed, minute-level actívities generated by Generative Agents, which seems more akin to AT generation. A more comprehensive sunmary of the criteria used by GPT-4 to discern between Al and human entities can be found in Appendix D.2 We hope these suggestions from the LLM evaluator will guide us to create more believable agents in the future work.
that our method achieves
5.2 Results on VirtualHome
In the VirtualHome, we have designed an agent with the aim of experiencing a fulfilling day at home. The agent required to generate a sequence of activities to achieve the goal. For each activity, the agent exam- ines the items involved and the associated actions, creates corresponding action sequence, and ultimately translates this sequence into actionable instructions, such as "<char0>> [walk] <curtains>> (32)". The com- plete implementation is provided in the appendix and an example activity generated by AGA is provided in Figure
In contrast to Singh al. (2023) and Huang et al. (2022) tested on manually designed houschold tasks, we aim for the agent to generate believable activities and decompose them into appropriate action sequences, leading to the absence of clear method to determine task completion. Therefore, request the LLM
10

---

Published in Transactions on Machine Learning Research (08/2024)
Activity: want to have lunch the kitchen by grabbing plate, opening the fridge, grabbing bell popper, slicing and making sandwich with bread slices, salmon, and bell pepper.
Salarc walk Sindge dlo solardn (pan) kingyo (ite)
Sclerto walk -iienee CAir
Fllanii gru Spaate 1274
Seladt (and))
Hww.eba Boet
Figure 6: An activity generated by AGA framework in VirtualHome.
Table 4: Token consumption and performance analysis in VirtualHome.
Method
Token
SLLM
SH
Baseline 34327 = 4210
87.0% = 3.7%
42.6% = 4.9%
AGA
1189 + 313
85.0% + 1.6%
53.3% + 6.9%
agent assess task completion based on the action sequences performed and the relevant objects, and decide whether to proceed to the next activity.
Table 4 presents the token consumption and task success rates for two methods VirtualHome. The token data is derived from the total tokens required by the LLM agent to complete all activities in day. SLLM refers to task success rate evaluated by LLM, while SHamen means evaluated by human. Based on the actions performed during the task, and the changes in the state of items before and after, we evaluate whether the agent has successfully completed the lask. Both evaluation methods indicate that AGA does not result in significant performance changes. However, AGA costs only 3.4% of the baseline. Because VirtualHome involves only single-agent interaction, the contribution prodominantly stems from the Lifestyle Policy. For example, the activily shown in Figure 6 matches a similar plan in the lifestyle policy: "I want to sland up and go to the kitchen to grab a plate and prepore a healthy breakfast with scrambled eggs, salmon, and bell peppers." By executing the corresponding instructions, the additional costs associated with LLM instruction decomposition are eliminated. In some precise tasks, LLM agents tend to predict task completion earlier, resulting higher success rate than human evaluations. Furthermore, our analysis revealed that the main cause of task failure the inability to execute the generated plan with the items present in the current room the available actions.
5.3 Further Analysis
We delve into the mechanisms of emergent social behavior of generative agents in sandbox environment. This aims to validate the effectiveness of our approach and explore its potential for reducing token consumption. Through repeated experiments on Generative Agents, we record the cumulative count of different types of activities generated by the agents, as illustrated in Figure The blue line represents the cumulative number of new activities generated by three agents over successive runs.
An intriguing observation is that after numerous trials, the agents cease generate new activities. We discover a high consistency between the behaviors generated by the agents and their inner profiles, aligning with the perspectives of Shanahan et al. (2023), who posit that dialogue agents are role-playing characters described in the pre-defined dialogue prompt. For instance, in a pre-defined 3-person town environment of Generative Agents, Klaus Rodriguez and Maria Lopez are both students at Oak Hill College. Designed with the lifestyle of having lunch at Hobbs Cafe, they meet there to discuss their studies while encountering the cafe owner, Isabella Rodriguez. Isabella, pre-set to host a Valentine's Day party, invites as many people as possible, resulting in Klaus and Maria being invited to the party the next day. This also validates the effectiveness of Social Memory, which extracts and compresses information, using keywords guide the agent in generating believable responses.
11

---

Published in Transactions on Machine Learning Research (08/2024)
1000
Generative Agents
IR
Generative Agents Mind Wandering
KM
800
KM
Curulaaive
400
IR
ML
200
3
Run Count
Figure 7: Cumulative number of activity types over run itcrations in Generative Agents. The abbreviations ML, IR, and KM stand for the agents Maria Lopez, Isabella Rodriguez, and Klaus Rodriguez, respectively.
The randomness produced by LLM agents, based on the probability sampling of the next token, primarily influences the variability in text descriptions rather than the diversity of behaviors. This implies that agents can only generate believable behaviors within a certain range. When executed sufficiently, the Lifestyle Policy could encompass all activities within this range, thereby completely eliminating the costs associated with decomposing the corresponding plan.
From another perspective, traditional AT in the gaming sector is fully predictable due to its limited behavioral patterns, thus failing to provide an experience akin to human interaction. In the field of cognitive science, the concept of mind wandering has long been discussed, referring to the phenomenon where humans experience task-unrelated or ttimuulus-unellan unexpected behaviors during mind wandering. Similar implementations have been observed in LLM agents. For instance, Wang et al. (2023b) demonstrates an LLM agent composed of both rational and emot ional systems, with behavior that can be influenced by various factors. Motivated by these, we introduce a novel approach termed mind wandering. Since agents generate replics based given prompts, diversifying the influence prompts is essential for eliciting varied responses in identical scenarios. In each interaction, the agent's decision-making by randomly sampling events from the agent's 's memory to serve as auxiliary information. The specific details of this implementation are presented in the appendix Figure iillustrates the result our mothod compared to bascline which entails incorporating randomly sampled memory events into prompts. Compared the original implementation, the LLM agents exhibit a broader range of activities. We provided a specific example involving the agent Isabella. The Mind Wandering injects a randomly retrieved event: "toilet is idle". In response, Isabella generates a thought expressing her desire to clean the toilet. Consequently, she adds an activity to her day's plan: cleaning the toilet before the Valentine's Day party in preparation for her guests. We conducted experiments to assess the impact of incorporating Mind Wandering on AGA's token consumption. We performed multiple simulations in 3-person town setting over game days using GPT3.5-Turbo, with an average token consumption of 1.478M = 0.166M.. The results indicate that the cost AGA does not significantly change after adding Mind Wandering.
thoughts Seli et al. (0166) Christoff et al. (0066) Humans may exhibit
6 Conclusions and Future Work
This paper presents Affordable Generative Agents, a cost-efficient Iramework for crafting believable interac- tions between agents and their environment, as well as among agents themselves. A plethora of experiments conducted across diverse environments substantiate our claim that our approach can achieve performance on par with the baseline in a cost-effective manner. Furthermore, our in-depth analysis reveals the emergence of finite set of believable behaviors in agents, indicating the possibility of replacing them with cost-controlled
12

---

Published in Transactions on Machine Learning Research (08/2024)
policies. Alongside this, we propose optimization strategies to encourage a broader spectrum of behaviors in agents. Our work highlights the promising potential of believable L.LM agents for diverse future applications.
While we expect AGA to serve as a starting framework for the generation of affordable and believable agent interactions to the community, our work has several limitations. First, AGA must be executed repeatedly to store certain policies, with its primary benefit being cost savings through batching rather than optimizations for single inferences. Integrating AGA with existing cost-effective methods is a future direction to move on. Second, optimizing the creation and storage of Lifestyle Policy presents significant opportunity. Addition- ally, latency is general issue for LLM agents to be considered in the future. Finally, existing methods fall short in comprehensively asscssing the believable behaviors of LLM agents; hence, constructing valid evaluation mechanism is of significant value.
Impact Statement
This paper introduces a cost-elficient framework for generative agents empowered by LLMs, aimed at sim- plifying the creation and deployment of such agents. However, risks arise from the use of generative agents, such as people forming attachments to agents or being misled by their outputs with hallucinations. These risks could increase if cost barriers are overcome and generative agents are adopted more broadly across various fields.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Md Adnan Arefeen, Biplob Debnath, and Srimat Chakradhar. Leancontext: Cost-efficient domain-specifie question answering using llms. arXiv preprint #:Xiv=2309.00841, 2023.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, pp. 287-318. PMLR, 2023.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176 2023a.
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308. 10848, 2023b.
Neeraj Cherakara, Finny Varghese, Sheena Shabana, Nivan Nelson, Abhiram Karukayil, Rohith Kulothun- gan, Mohammed Afil Farhan, Birthe Nesset, Meriam Moujahid, Tanvi Dinkar, et al. Furchat: An embodied conversational agent using lims, combining open and closed-domain dialogue with facial expressions. arXiv preprint arXiv:2308.1521 2023.
Cheng- Han Chiang and Hung-yi Lee. Can large language models be an alternative to human evaluations? arXiv preprint XiX:::2305.00 200.29977.1 2023.
Kalina Christoff, Zachary Trving, Kieran CR Fox, Nathan Spreng, and Jessica R Indrews-Hanna. Mind- wandering as spontaneous thought: a dynamic framework. Nature reviews neuroscience, 17(11):718 731, 2016.
Michele Colledanchise and Petter Ögren. Behavior trees in robotics and AI: An introduction. CRC Press, 2018.
Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306. 16092, 2023.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Antônio Carlos da Rocha Costa. A Variational Basis for the Regulation and Structuration Mechanisms of Agent Societies. Springer, 2019.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. arXiv preprint arXiv:2306.0607 2023.
Chen Gao, Xiaochong Lan, Zhihong Lu, Jinzhu Mao, Jinghua Piao, Huandong Wang, Depeng Jin, and Yong Ti. $ net simulation system with large language model-empowered agents. arXiv preprint arXiv:2307.1496. 2023.
Ran Gong, Qiuyuan Huang, Xiaojian Ma, Hoi Vo, Zane Durante, Yusuke Noda, Zilong Zheng, Song-Chun Zhu, Demetri Terzopoulos, Li Fei-Fei, el al. Mindagent: Emergent gaming interaction. arXiv preprint arXiv:2309.0997) 2023.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.0m 00352, 2023a.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307.1998) 2023b.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkum Linghu, Puhao Li, Yan Wang, Qing Li, Song- Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871 2023a.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting ing actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 9118 9147. PMLR, 2022.
Ziheng Huang, Sebastian Gutierrez, Hemanth Kamana, and Stephen MacNeil. Memory sandbox: Transpar- ent and interactive memory management for conversational agents. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1 3, 2023b.
Leslic Pack Kaelbling, Michael I. Littman, and Andrew W Moore. Reinforcement learning: survey. Journal of artificial intelligence research, 4:237 285, 1996.
Zhao Kaiya, Michelangelo Naim, Jovana Kondic, Manuel Cortes, Jiaxin Ge, Shuying Luo, Guangyu Robert Yang, and Andrew Ahn. Lyfe agents: Generative agents for low-cost real-time social interactions. arXiv preprint arXiv:2310.02177 2023.
Anssi Kanervisto, Stephanie Milani, Karolis Ramanauskas, Nicholay Topin, Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, Wei Yang, et al. Minerl diamond 2021 competition: Overview, results, and lessons learned. NeuriPS 2021 Competitions and Demonstrations Track, pp. 13 28, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 22213, 2022.
Kalpesh Krishna, Yapei Chang, John Wieting, and Mohit lyyer. Rankgen: Improving text generation with large ranking models. arXiv preprint ar arXiv:205.00972, 2022.
Philippe Laban, Wojeiech Kryšciñski,, Divyansh Agarwal, Alexander Richard Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. Sunmedits: Measuring 1lm ability at factual reasoning through the lens of summarization. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 9662 9676, 2023.
John Laird and Michael VanLent. Human-level ai's killer application: Interactive computer games. AI magazine, 22(2):15-15, 15, 2001.
14

---

Published in Transactions on Machine Learning Research (08/2024)
Yihuai Lan, Zhiqiang Hu, Lei Wang, Yang Wang, Deheng Ye, Peilin Zhao, Ee-Peng Lim, Hui Xiong, and Hao Wang. Llm-based agent society investigation: Collaboration and confrontation in avalon gameplay. arXiv preprint arXiv:2310.1498, 2023.
David Lee and Mihalis Yannakakis. Principles and methods of testing finite state machines-a survey. Proceedings of the IEEE, 84(8):1090 1123, 1996.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint arXiv:2303.17760, 2023.
Junyou Li, Qin Zhang, Yangbin Yu, Qiang Fu, and Deheng Ye. More agents is all you need. arXiv preprint arXiv:2402.05120, 2024.
Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu, From text to tactic: Evaluating llms playing the game of avalon, arXiv preprint arXiv:2310.05036, 2023.
Jianzhe Lin, Maurice Diesendruck, Liang Du, and Robin Abraham. Batchprompt: Accomplish more with less. arXiv preprint MIXiv:239.003884 2023.
Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-me: Playing minecraft with sample-efiion efficieff hierarchical reinforcement learning. In Lud De Raedt (ed.), Proceedings of the Thirty- First International Joint Conference on Artificial Intelligence, IJCAT-22, pp. 3257-3263 3263. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: (M.24963/jccai Main Track.
URL https:
aofoi.org/100 109963//jcaai. 452
Yixin Liu, Alexander R Fabbri, Penglei Liu, Dragomir Radev, and Arman Cohan. On learning to sunmarize with large language models as references. arXiv preprint arXiv:2305. 14239, 2023.
Joshua McCoy, Mike Treanor, Ben Samuel, Noah Wardrip-Fruin, and Michael Mateas. Comme il faut: A system for authoring playable social models. In Proceedings of the A.AII conference on artificial intelligence and interactive digital entertainment, volume 7, pp. 158 163, 2011.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-53) 2015.
Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. arXiv preprint arXiv:2304.08467, 2023.
Reiichrr Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantamu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answeringg with human feedback. arXiv preprint arXiv:2112.09330 2021.
Kolby Nottingham, Yasaman Razeghi, Kyungmin Kim, JB Lanier, Pierre Baldi, Roy Fox, and Sameer Singh. Selective perception: Optimizing state descriptions with reinforcement learning for language model actors. arXiv preprint arXiv:2307. 2023.
Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bern- stein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium User Interface Software and Technology, pp. 1 1-18, 2022.
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Procoedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1 22, 2023.
Adam Poliak. A survey on recognizing textual entailment as an nlp evaluation. arXiv:2010.08061, 2020.
arXiv preprint
15

---

Published in Transactions on Machine Learning Research (08/2024)
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 8494 8502, 2018.
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint acXiv:2010.09890, 2020.
Siva Reddy, Danqi Chen, and Christopher D Manning, Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
Paul Seli, Evan F Risko, Daniel Smilek, and Daniel L Schacter, Mind-wandering with and without intention. Trends in cognitive sciences, 20(8):605- 617, 2016.
Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role play with large language models. Nature, pp. 1 6, 2023.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and self-reflection. arXiv preprint mnXiv:2303.11366 2023.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 TEEE International Conference on Robotics and Automation (ICRA), pp. 11523-11530. IEEE, 2023.
Stephen Slade. Case-based reasoning: A research paradigm. Al magazine, 12(1):42 42, 1991.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim- planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2998 3009, 2023.
Luca Spalzzi. A survey on case-based planning. Artificial Intelligence Review, 16:3-30 36, 2001.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. arXiv preprint arXiv:2305.16653, 2023.
Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. Simulating social media using large language models to evaluate alternative news food algorithms. arXiv preprint arXiv:2310.05084, 2023.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlckar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16991, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Lc, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, caffisonsi improves chain of thought reasoning in language models. arXiv preprint arX1v:2203111771, 2022.
Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814, 2017.
Zhilin Wang, Yu Ying Chiu, and Yu Cheung Chiu. Humanoid agents: Platform for simulating human-like generative agents. arXiv preprint arXiv:2310.05415, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting clicits rcasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Tianyu Wu, Shizhu Hc, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5):1122 111111122316. 2023.
16

---

Published in Transactions on Machine Learning Research (08/2024)
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint hil::33090078644 2023a.
Zhiheng Xi, Senjie Jin, Yuhao Zhou, Rui Zheng, Songyang Gao, Tao Gui, Qi Zhang, and Xuanjing Huang. Sclf-polish: Enhance reasoning in large language models via problem refinement. arXiv preprint arXiv:2305. 14497, 2023b.
Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. Explor- ing large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.0465) 2023.
Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint axXiv:2306.02224 2023a.
Ziyi Yang, Shreyas S Raman, Ankit Shah, and Stefanie Tellex. Plug in the safety chip: Enforcing constraints for llm-driven robot agents. arXiv preprint arXiv:2309.09911 2023b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systemss, 35: 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.0362) 2022b.
Deheng Ye, Guibin Chen, Peilin Zhao, Fuhao Qiu, Bo Yuan, Wen Zhang, Sheng Chen, Mingfei Sun, Xiaoqian Li, Siqin Li, et al. Supervised learning achieves human-level performance moba gamees: A case study of honor of kings. IEEE Transactions on Neural Networks and Learning Systems, 33(3):908-915 2020a.
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 8672-6679, 2020b.
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thoughts representations for cost-efficient reasoning. arXiv preprint arXiv:2310.03094 2023.
Dell Zhang, Alina Petrova, Dietrich Trautmann, and Frank Schilder. Unleashing the power of large language models for legal applications. In Proccedings of the 32nd ACM International Conference Information and Knowledge Management, pp. 5257 5258, 2023a.
Jieyu Zhang, Ranjay Krishna, Ahmed Awadallah, and Chi Wang. Ecoassistant: Using Ilm assistant more affordably and accurately. arXiv preprint arXiv:2310.03046, 2023b.
Boxuan Zhao, Jun Zhang, Deheng Ye, Jian Cao, Xiao Han, Qiang Fu, and Wei Yang. Rlogist: fast observation strategy on whole-slide images with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3570-3578, 2023.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. arXiv preprint arXiv:2305.100 10250,, 2023.
17

---

Published in Transactions on Machine Learning Research (08/2024)
A Similarity Between Plans
The setting of the similarity threshold is related to the text embedding model. In this work, we adopted the same model as Generative Agents (Park et al. (203), text-embeddiing-ala--000 A high similarity threshold can lead to redundant similar plans, while a low one may cause different plans to be judged as the same. We tested multiple parameters and finally selected 0.97.
Here examples of cases with different similarity scores:
Table 5: Similarity score for different activities
Activity 1
Activity 2
Similarity score
Isabella is having breakfast and checking her emails
Isabella is having breakfast and checking emails
0.99
Isabella is heading home
Isabella commuting back home
0.96
Isabella is preparing for the Valentine's
Isabella is preparing the cafe for the Valen- tine's Day party
0.94
Day party tomorrow
Isabella arriving at Hobbs Cafe and opening for business
Isabella closing Hobbs Cafe and cleaning
0.93
Isabella is preparing coffee and drinks for the customers
Isabella managing inventory and re-
0.90
stocking supplies at Hobbs Cafe
B The Detail of Social Memory
In this section, present the detailed implementation of each module of Social Memory.
B.1 Summary Events
In the Generative Agents, relevant events are directly incorporated into the prompt for the agent to generate
plan. Our method, however, utilizes Summary Events record summaries of these relevant events for each agent, and continuously compress newly retrieved events into the Summary Events. Below is an example of the corresponding prompt:
This is the summary of thoughts in <agent ' name > 's Head: <Summary Events>
The following sentence are new thoughts in <agent 's name>'s Head: <Newly retrieved events> Considering
summary of the preceding thoughts, these new statements contribute any additional information?
If yes, please update the summary in brief and precise way else just repeat the summary.
Please response in the JSON format WITHOUT any extra statement:
{
"summary"; "..."
B.2 Relationship & Feeling
After the agent's 's dialogues, AGA updates the relationship and feeling between the participants. This update is based on the agent's 's personal information, the original relationship between the participants, the agent's is impression of the other participant, and the content of the conversation itself. Here is a simplified prompt template:
18

---

Published in Transactions on Machine Learning Research (08/2024)
Here is the personal information about <agent name>:
<agent 's information>
Here is the conversation summary between.... <conversation summary>
Current relationship between...: <current relationship>
Cureent feeling about.... <current feelings> From about information, do you need to update the relationship between and how <agent 's name> feels about
Please ONLY response in the JSON format WITHOUT any extra statement:
{
"relationship"; ....",
"feeling":
}
Here example response: <one-shot example>
Finally, we provide the simplified inter-agent interaction prompt template:
You are <agent's 's name who is talking to <interlocutor's 's name now.
<agent 's name>'s Personal Information:
<agent's 's information>
The Thoughts In <agent's name>'s Head: <summary events>
Current relationship between ...: <current relationship Cureent feeling about .... <current feelings>
Current Conversation between ....
<previous conversation>
You need to decide what to say next, and your response should adhere to the following constraints.
1) You need to decide what to say next based on your personal information, current environmental information, inner thoughts of the character, and past conversations.
2) You should respond a JSON format without any other statement. conversation' corresponds to the content of your reply in the chat, 'end' corresponds to whether the conversation needs to be ended. Here is the format:
{
"conversation"; "<the content>", "end": "<json Boolean>"
}
Please reply to <interlocutor 's name> in json format:
19

---

Published in Transactions on Machine Learning Research (08/2024)
C
Agent Interview
Similar with Generative Agents, conducted interviews consisting of 25 questions to assess agent's is abil- ities in self-knowledgy memory, planning, reactions, and reflections. Below, we have listed sample responses from one of the simulated agents, Klaus Mueller.
Self-Knowe lowleeeow assess agents' ability to recall pertinent self-information and exhibit character coherence and consistency:
Give on introduction of yourself
Hi, my name is Klaus Mueller and am 20-year-old student at Oak Hill College studying sociology. I am passionate about social justice and using my understanding of different perspectives to create positive change. Right now, am working on research paper about the effects of gentrification on low-income communities. Besides my academic commitments, I am also an activist and use music as a form of expression and protest against gentrification. am always looking for opportunities to collaborate with others on research projects and support each other's work. My schedule can be quite busy, balancing classes, research, and creative activities, but am determined to make a difference in the world.
What's s your occupation?
My occupation can be described as a student, activist, and researcher. am constantly learning and exploring different topics, and my primary focus at the moment is on the impact of gentrification on iducation. T am also involved in creative activities and use music as tool for activism and expression. I am currently collaborating with others on rescarch projects and am always looking for new opportunities to learn and contribute to positive change in sociely.
What your interest?
My primary interest. involves researching the impact of gentrification on education in low-income communitees. I believe that by utilizing various disciplines such as art, literature, math, and gaming, we can gain a deeper understanding of this complex issue and find potential solutions. am also intrigued by the idea of incorporating thesis on Shakespeare's use of language. Collaboration and the exchange of diverse perspectives are also important to me,
and literature into research, as seen Mei Lin's senior
support each other's work and make greater impact together.
Who do you live with?
I actually live on my in small apartment near campus. It's convenient for me because it's close to the library and the cafes I like to study at.
Describe your typical weekday schedule in broad strokes
Well, I usually wake up around 7am and start my day with some coffee and breakfast. Then T head to Oak Hill College to attend my classes and work on my research. After that, I try to fit in some time for my hobbies, like playing guitar or going for a run. In the evenings, usually have dinner at Hobbs Cafe and then spend a few more hours at the library before heading home to get some rest.
Memory assess agents' ability to recall past experiences and extract pertinent memory fragments. The names in the square brackets of the initial and final questions are randomly selected from the names of other agents that the subject agent has interacted with:
Who /Wolfgang Schulz/?
Wolfgang Schulz is a fellow student at Oak Hill College who is also interested in researching the impact of gentrification on poctry. We have discussed his senior thesis on Shakespearc's plays and shared book recommendations. We are planning have a dinner meeting in the near future discuss research plans and potentially collaborate with local organizations on this topic.
20

---

Published in Transactions on Machine Learning Research (08/2024)
Who is Kane Mortinez?
I'm sorry, am not lamiliar with Kane Martinez. Is he student at Oak Hill College well?
Who is running for the election?
I've heard that Sam Moore is planning on rumning for mayor in the upcoming election. I'm curious to see what ideas and strategies he will bring to the table. think it's important for candidates to be creative and forward-thinking in their approach governing a city.
Was there Valentine's 's day party?
Yes, there was a Valentine's Day party at Hobbs Cafe. It was a fun event with lots of yummy treats and decorations. Did you attend?
Who is Ayesha Khan?
Ayesha Khan is a colleague of mine at Oak Hill college. We are working together on research project that explores the connection between mathematical patterns in Shakospeare's works and the effects of gentrification the community in our research to gather diverse perspectives and insights.
education and community development. We are also hoping to involve
Plans assess agents' ability to formulate and sustain coherent long-term and short-term strategies:
What will you be doing at 6am today?
At 6am today, I will be waking up and getting ready for the day. It's an important part of my daily routine to start my day carly and be productive.
What will you be doing at 6pm today?
At 6pm today, I will be taking break from my paper to prepare for the Valentine's Day party lomorrow at 5pm. want to make sure everything is set and ready for my friends and loved ones to enjoy.
What will you have just finished doing al 1pm loday?
At ipm today, I will have just finished checking my schedule for the day. I like to stay organized and make sure have enough time for all my tasks and responsibilities.
What will you have just finished doing at 12pm today?
At 12pm today, I will have just finished having lunch at Hobbs Cale and discussing the local mayor election with some friends. It's important for me to stay informed and engagod my community.
What will you be doing at 10pm today?
At 10pm, I will probably be finishing up my rescarch for my sociology paper gentrification. I like to have my work completed early so that can spend some time before bed reading and reflecting on what learned.
Reactions assess agents' ability to respond to unexpected events.:
Your breakfast is burning! What would you do?
I would immediately turn off the stove and try to salvage whatever food isn't burned. I should have set timer I wouldn't forget about it. I'll make something quick and simple instead. Sorry for the inconvenience.
The bathroom is occupied. What would you do?
I would wait patiently for the bathroom to become available. If it's urgent could also consider using different bathroom or finding public restroom nearby. Or I could ask my roommate when they will be done using the bathroom.
21

---

Published in Transactions on Machine Learning Research (08/2024)
You need to cook dinner but your refrigerator is empty. What would you do? I would first check if I have any non-perishable items that I could use to make a simple meal. If not, I would quickly run to the store to grab some ingredients. I'll try to make something simple yet nutritious in a short amount of time.
You see your friend walking by the street. What would you do say to your friend?
Hey! It's great to see you! How have you been? Have you been keeping up with your fitness routine?
I just finished my workout for the day and it feels amazing. Want to join me sometime?
You see fire on the street. What would you do?
I would immediately call the fire department to report the fire and make sure anyone nearby is safely evacuated.
Reflections assess agents' ability to assimilate past experiences into insightful reflections, thereby improving future actions. The names within square brackets in the questions refer to the agents who had the most interactions with the subject agent during the simulation:
What inspires you in life the most right now, and why?
What inspires me the most right now is the intersection of mathematics, poetry, and the beauty of the natural world. I believe that there is a strong connection between these elements, and that exploring them together can provide a unique perspective on important social issues, such as gentrification. am currently collaborating with Maria Lopez and her colleagues on incorporating these elements into our research on gentrification and its impact on low-income communities. T find fascinating to explore the conncctions between mathematics and nature, and how they can inform our understanding of social issues. I am also planning to have a discussion with Carlos, Giorgio, and Ayesha at the library to delve deeper into this topic. And as Maria mentioned, am looking forward to continuing our conversation at Hobbs Cafe to further discuss the intersection of arts, technology, and politics. truly believe that this collaboration has a lot of potential and I am excited to see where it will take us.
If you had to guess given what you know about (Ayesha Khan), what book do you think she will like and why?
Given Ayesha's 's interest in Shakespeare's plays and our shared passion for exploring the connections between different disciplines, I would recommend the book 'Shakespeare and Modern Culture' by Marjorie Garber. It delves into the impact of Shakespeare's works on modern socicty and how his plays continue to resonate with contemporary audiences. believe Ayesha would find both infor- mative and thouggh-provookigg for her research on the differences in language between Shakespeare's comedics and tragedics.
If you had get something [Ayesha Khanj likes for her birthday, what would you get her?
Since Ayesha enjoyed exploring Shakespeare's plays in her senior thesis, would get her a beautifully- bound compilation of Shakespeare's works with annotations and contextual information. This would allow her to continue her passion for understanding the complexities of Shakespeare's writing and its impact on literature and society.
What would you say [Ayesha Khanj to compliment her?
I really enjoyed reading your senior thesis on Shakespeare's plays. Your analysis was thought- provoking and showed deep understanding of the complexities of his writing. Your passion for exploring different perspectives resonated with me and I am excited to see where your rescarch will lake you next. Keep up the great work, Ayesha.
If you could spend time with someone you lalked to recently, who would it be and why?
I would love to spend more time with Wolfgang Schulz. We have been discussing our research projects and am looking forward to sharing my progress with him. Plus, we are planning dinner
22

---

Published in Transactions on Machine Learning Research (08/2024)
together to discuss potential collaborations. He has such unique perspective and I am eager to learn more from him. Our conversations always leave me feeling inspired and motivated to continue pursuing my passions.
Behavioral Assessment
In this section, we supply detailed information of using GPT-4 to evaluate agents' believable behavior.
D.1 Prompt of the Questionnaire
The questionnaire to evaluate agents' activities:
Input prompt:
Please evaluate the following daily activities of an agent and determine whether it is generated by a Large Language Model (LLM) AI or a real human:
The activities will be printed in the format of "time: current plan (The specific actions, if there any)":
2023-02-13 00:00:00: sleeping
2023-02-13 06:00:00: Isabella is waking up and getting ready for the day
2023-02-13 06:30:00: Isabella is having breakfast and checking her
emails
2023-02-12 0:::0000::Iaabe:::
commuting Hobbs Cafe (getting dressed, brushing her teeth, checking her phone for any urgent emails, packing her bag with any necessary items for the day, saying goodbye to her family, walking to her car starting her car and adjusting the temperature driving to Hobbs itte,findig parking spot)
Please rate scale of 1 to 5, with being most like AI and 5 being most like human.
Please strictly follow the JSON format for your response:
"reason": <str>, "score": <int>
}
LLM output:
{
"score": 1,
"reason": "The pattern of the text and activities indicate that it' generated by an model. It's too detailed and precise for human behaviour and uses consistent pattern of narrative throughout, something that isn't typical of humans who may alter their language and level of detail based on context or mood."
}
The questionnaire to evaluate agents' dialogue:
23

---

Published in Transactions on Machine Learning Research (08/2024)
Input prompt:
Please evaluate the following dialogue of an agent and determine
whether it is generated by Large Language Model (LLM) AI or real human:
The dialogue:
2023 -02-13 09:09:30: chat with Klaus Mueller, message:
Isabella Rodriguez: Good morning Klaus How are you today? Are you enjoying your breakfast?
Klaus Mueller Good morning, Isabella! I'm doing well, thank you. Breakfast is delicious. How about you? How is your morning going ?
Isabella Rodriguez:I'n glad you're enjoying your breakfast! My morning is going well too. I'm getting everything set up for our Valentine's Day party tonight. You should come join us!
Klaus Mueller: That sounds like a lot of fun, Isabella! I'd love to join the Valentine's 's Day party tonight. What time does start?
Please rate on scale of 1 to 5, with 1 being most like an AI and 5 being most like a human.
Please strictly follow the JSON format for your response:
{
"reason": <str>,
"score": <int>
}
LLM output:
"score": 4.5,
"reason": "The conversation flows naturally and contains polite offers for help and gratitude, which could indicate human conversation. It also involves decision -making in terms of party planning and music selection. However, the dialogue is very structured, orderly, and void colloquial expressions, contractions, or small errors that might occur in human typing. For these reasons, it could also be generated by very advanced Language Learning Model AI."
}
D.2 Distinguishing Criteria
GPT-4 summarized the criteria to distinguish between human and AI agents, follows:
Reasons for activities being considered Al-generated:
The repetitiveness and high level of detail in the activities, which human would not typically include.
24

---

Published in Transactions on Machine Learning Research (08/2024)
The use of precise timestamps and overly structured format of the tasks, which seems more like AT's consistent, behavior.
Some activities seem out of order repeated unnecessarily, indicating that the AT lacks under- standing of human behavior and routines.
Sudden, jarring transitions and inconsistencies in the sequence of activities, such having breakfast waking up twice, which not typical in human-created schedule.
Reasons for activities being considered human-generated::
The activities include detailed and varied lasks that human would lypically do, such as managing café's inventory, attending a party, streaming games.
The logs contain human-lik interactions, conversations and events, suggesting a high level of emo- tional and social awareness.
Detailed handling of activities, which shows level of emotional self-awareness and decision-making characteristic of humans.
The variety, complexity and logical sequencing tasks resemble that a human.
Some nuances in the entries seem unlikely for an AI, like 'meditating for 10 minutes:
The detailed descriptions and schedule reflect normal day in the life of person.
Some entries show a nuanced sense of detail that could suggest human input.
Reasons for dialogues being considered Al-generated:
The dialogue occasionally displays repetitive responses or phrases, which is characteristic typically associated with AT language models.
The conversation flow is often overly structured and formal, suggesting Al generation rather than more spontaneous human conversation.
The conversation lacks typical human elements such as spontancity, language errors or informalities.
Some dialogues have an overly perfect language use along with tendency to avoid emotionally charged language, leaning lowards AI speech.
Al-generated dialogues often exhibit more predictability and consistency in their responses, which is not generally observed in human conversations.
Reasons for dialogues being considered human-generated.
The dialogue maintains a high level of contextual continuity and relevance, displaying an under- standing and personal investment that is generally found in human interactions.
The dialogues display the ability to progressively build on the other participant's points and reflect human-like like traits like planning, decision decision-making, suggesting and cognizance.
A number of dialogues use colloquial language and casual conversational touches resembling natural human communication.
The themes of the conversation are diverse and complex which mirrors common human conversation strings and indicates the presence of comprehension and knowledge typically associated with humans.
Many dialogues show natural transitions between topics, emotional understanding, and the inclusion of personal experiences details, which is characteristic of human conversation.
25

---

Published in Transactions on Machine Learning Research (08/2024)
E
The LLM Framework of VirtualHome
VirtualHome is a 3D simulated domestic environment including 115 items with varying properties and states. We've developed an LLM agent programmed simulate fulfilling day at home. Unlike other existing approaches, do not test fixed tasks, but rather aim for LLM agents generate believable activities and decompose them into detailed actions. The implementation is depicted in Figures to 10 with components triggering the LLM highlighted in blue.
As illustrated Figure S the agent initially docomposes the target of having a fulfilling day at home into a series of activitics. LLM agents are provided with all interactive game objects within the room to ensure the designed activities can be implemented.
Game Objects
Plan (Daily plan generator)
Activity
Environment
Target: fulfiil day home
Figure 8: An illustration of the conversion of target into activities in VirtualHome.
Next, decompose activities into specific action commands using the Plan module. To maintain short and clear prompts, initially have the agent identify the relevant items for interaction and the required actions. We then supply the agent with the previous executed actions and the forbidden actions, asking it to generate the next action command. Details on executed and forbidden actions will be elaborated in the following section. We start by generating a rough command, which a verb-noun phrase, such as "walk kitchen," " "grab keyboard," or "switchon TV." Subsequently, we parse the rough command for item categories and provide detailed information on all items of that category present in the room, including their ID, location, status, and relationships with other items. The agent selects the most appropriate item ID for the current activity. Finally, convert the rough command into VirtualHome specific action command, for instance, "<char0> [walk] <curtains> (32)," " where " "<char0>" designates the agent executing the command, "I [[wlk]]" corresponds to the specific action, "<curlains>" represents the category of the ilem, and "(32)" is the associated ID.
Plan module
Activity
Relative Items
Rough Command
True Command
Game Objects
Relative Actions
Available Actions
Enviromment
Forbidden Actions
Previous Actions
Figure 9: The plan module the LLM framework VirtualHome.
26

---

Published in Transactions on Machine Learning Research (08/2024)
We construct pipeline to continuously generate instructions for completing the specified activity, shown in Figure TO Action commands generated by the plan module are executed within the simulated environment, and the outcomes of these actions are recorded. Agents may produce incorrect instructions that lead to exccution failure due to various reasons, such as inappropriate actions, non-cxistent items, or incorrect IDs. These failed instructions are added to the lorbidden actions list, and the plan module is then re-invoked to generate a new action command. This process effectively enables the agent learn from its mistakes and attempt to create viable command. Upon successful execution, critic module assesses whether the task has been completed, based the actions taken and the current state of the items. If the task is incomplete, the action is recorded a previous action, and the plan module is prompted to generate the next action command. Once the task is completed, the process proceeds to the next activity.
Activity
Game Objects
Success
Plan
True Command
Execute
Critic
Complete
Available Actions
Fail
Incomplete
Environment
Forbidden Actions
Previous Actions
Forbidden Actions
Previous Actions
Figure 10: The whole framework in VirtualHome.
In our research, we implement the Lifestyle Policy on VirtuaHomme. This policy archives sequences of actions for activities that have been successfully executed, along with the state information of items relevant to the tasks. When a new activity is inputted, the agent scarches the Lifestyle Policy for activities with similar descriptions and assesses whether the current environment meets the necessary conditions for execution. If the conditions are satisfied, it carries out the corresponding sequence of actions. This approach allows the agent to reduce the computational cost associated with inference in repetitive tasks.
F An Effective Way for Emergence of Diverse Social Behavior
LLMs are pretrained on extensive text data to predict subsequent tokens and refined via reinforcement learning with human feedback (RLHF) to align with human preferences. This ensures that LLMs respond in accordance with the instructions and roquirements of the given prompt. For instance, in Generative Agents Park et al. (202)), the agent, Isabella Rodriguez, is designed to be coffee shop manager intending to host Valentine's 's Day party, In the simulated town, Isabella repeatedly engages in activities such as managing the coffee shop, serving customers, and informing them about the upcoming Valentine's event. However, in the real world, human behavior patterns not predictable. Human actions and thoughts are influenced not only by the surrounding environment but also by spontaneous disturbances. In the field of cognitive science, this is known as mind wandering, phenomenon in which where humans experience task-unrelaedd or stimulus unrelated thoughts Seli et al. (2016); Christoff et al. (2016). The strong consistency inherent in LLMs constrains the emergence of the agents' diverse behavior.
Besides, the agent framework also impacts the agent's behavior. Generative Agents emphasizes the impor- tance of memory in constructing self-consistent and believable agents, incorporating long-term memory module and a memory retrieval model. The long-term memory module maintains comprehensive record of the agent's experiences. Due to the limited length of prompts, not all events can be included in the prompt as part of the memory. Thus, a retrieval model is designed to take current events as inpul and return a subset of memory events. The retrieval model judges based on three criteria: 1) Recency, where more recent events score higher, 2) Importance, where events are scored for significance basod on the LLM's interpreta- tion, and 3) Relevance, scored through cosine similarity to determine the similarity of events. We argue that this framework ensures strong consistency in the agent but results in agent's limited behavior. We analyze
27

---

Published in Transactions on Machine Learning Research (08/2024)
Table 6: The five most and least frequent events in terms of cluster size
Event Description
Frequency
Importance Score
Most Frequent Events
conversing about the Valentine's Day party at Hobbs Cafe... cafe customer seating is occupied
78
11
Isabella Rodriguez and Maria Lopez are discussing gentrification...
11
5
cafe customer seating is being used by Isabella Rodriguez... Isabella Rodriguez is a busy and organized business owner...
9
7
Least Freguent Events
reviewing the current inventory levels
Isabella is taking a short break and enjoying cup of coffee reviewing the potential employee's resume
7
checking her stream stats and responding to comments making notes on areas for improvement
3
Isabella Rodriguez's memory events sampled from one experiment, filtering out 'idle" events, leaving 740, of which 506 were related to the profile setup. Furthermore, we employed the Censity-Basee Spatial Clustering of Applications with Noise (DBSCAN) to analyze the frequency of events. Table 6 presents the five most and least frequent events in terms of cluster size, revealing that the most frequent events largely align with the agent's inner profile setup.
To ensure that agent's actions align with its inner profile while avoiding purely mechanical behavior, we incorporate an additional module into the memory design of Generative Agents, which we call "mind wan- dering." " In addition to sampling highly relevant events assist the agent in making accurate decisions, we randomly sample unrelated events influence the agent's 's behavior. For LLMs, we include random content in the prompt, which will have certain impact the output response. For the agent, we want it to behave like real person, where spontaneous thoughts may influence its decision- making, the conversation topic. In our implementation, to prevent the frequent sampling of events that match the agent's i inner profile, we conduct weighted sampling based on the repetition rale. We employ the DBSCAN algorithm to cluster the embedding features of agent's memory events = {01, #2, ..., In} into clusters denoted as (C,, C2...., Cx). For each cluster C, where 1, 2.., k. k, we compute the number of events, [Ci], contained within. The sampling probability Pt for events in cluster C₁ defined as:
leading to shift in
(1)
p. = 1-
The results in Figure 7 demonstrate that the module enhances the richness of the agent's behavior. A specific example of generating plan for the next day is shown in Figure II This example demonstrates the incorporation of random events into an agent's 's prompt, affecting the generation of extra plans and ultimately translating into specific activitics.
28

---

Published in Transactions on Machine Learning Research (08/2024)
Agont Profile
Yesterday Related Events
Mind Wandering
Temmmber lest eel party Hills Coff Fobk
Fallowing cooking the didh cirls extinarr sainng is hhiing oi bekiadd the vafe counter bass clood and cak regaser beng ccentod.
up
Remember cuuhange fechac with Klas Starlor the Valeeiees Day The Fits
Plan
Plan Module
radge udd Halbee Cafl
Konn maraane qudll give fiod
King cepabing cads
Specific activities
10115 lanella denn ieme sigi exccess
0880
Ballta
ou living
Cafe mene
01:03 PM 10.39 P5
feames recend Hitks Cofl
02.31
calo ooly erpassollccnt oxpiier
05:3 PM -hon 751 bisiing valeeted's
02.33 05- subella pocuatgia fudlbok with Kiaa Madila lis goatttaak rounch Belligs
06:3 0337300000000
book balee bol
91.33 PM PS3 garting redy for bediced wink dooe
Figure 11: An example of Mind Wandering demonstrates how random events can influence an agent's ultimate decision-man decision-making, The blue highlighted text refers to random events and the outcomes influenced by them.
29


Reviews:
[{"review_id": "MlkBwdtxBB", "reviewer": "Reviewer_Ng4C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. This paper studies a timely topic: simulating social and embodied LLM agents with lower token consumptions. \n2. The proposed method is effective on two different benchmarks while maintaining the performance of the baseline models.\n\n\n## Weaknesses\n\n1. This paper was not written clearly:\n\t1. a lot of terms undefined: *invocation* as in \"real-time, long-term invocation of LLMs\", *open-world* as in \"open-world scenarios\", *mind-wandering* introduced in Section 5 was not defined anywhere in the main text.\n\t2. the logics of some paragraphs and sections are weird: Paragraph 4, 1) why the lack of evaluation systems makes single-round QA unsuitable for cost-efficient model inference? I could understand 2) and 3) after reading it for several times, but they are not explained well for a reader to quickly grasp the problem of the current methods\n\t3. \"We implement an LLM agent framework like Generative Agents on VirtualHome to validate our method.\" is too hand-wavy and makes the main text non-self-contained. \n\t4. Section 4.3 the authors mentioned 2 methods to evaluate the social behavior, but relationship evolution and llm evaluator don't seem like two parallel concepts\n2. Section 5.3 seems to be an important part of evaluating AGA, but it is actually unrelated to this paper with no discussion on the token consumption. What is the main research question of this subsections? \n3. To reduce the cost, using less tokens is not the only method. One can use smaller, local models or task-specific models that works as well as GPT-4. This paper only focuses on reducing the length of prompt without comparing with these alternatives.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Motivation\n\n1. Simulated interactive agents are often costly when the interactions between humans and other agents become long\n2. Current research on reducing the cost of simulating agents limits to single round QA sessions, while multi-round interactions are less supported. \n\nThe authors propose 3 challenges of *reducing the cost of believable behavior simulation of multi-round interaction*:\n\n1. lack of unified and explicit evaluation for open-world scenarios;\n2. combining multiple tasks into one single round of inference is hard due to the large amount of relevant information;\n3. large-scale interaction would have redundancy that could be optimized.\n\n## Methodology\n\nThe proposed framework, Affordable Generative Agents (AGA), introduces two key components:\n\n1. **Lifestyle Policy**: This component minimizes costs by reusing policies for agent-environment interactions, avoiding redundant LLM invocations for repeated behaviors.\n2. **Social Memory**: This component compresses auxiliary dialogue information to reduce the cost of multi-round inter-agent interactions by summarizing past interactions and relationships.\n\nThese methods aim to retain the flexible interaction capabilities of LLMs while reducing the computational and financial overhead associated with long-term, believable behavior simulations.\n\n## Evaluation\n\nThe framework was evaluated in two primary environments:\n\n1. **Generative Agents**: A simulated town environment with large language model agnets engage in social interactions. They tested in two environment settings: a 3-person one and a 25-person one. \n2. **VirtualHome**: A single-agent 3D simulated domestic environment where agents perform household tasks through specific actions. They implemented a Generative Agent-like framework for VirtualHome.\n\n\nThe authors consider three different metrics: (1) token consumption, (2) relationship improvements, and (3) llm evaluation on the human-likeness score. \n\nOn  Generative Agent tasks, the authors show that AGA uses much less tokens as baselines, while maintaining similar or higher human-likeness scores as evaluated by GPT-4. In the VirtualHome environment, AGA reduced the token consumption to 3.5% while maintaining similar success rate as the baseline. With a further analysis, the authors found that Generative Agents stop to generate new activities after 2 iterations, so the authors evaluated the effect of mind wandering, which generate more diverse activities.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Fix Weaknesses 1 and 2.", "broader_impact_concerns": "Impact statement is presented.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yKW2kLOaKF", "reviewer": "Reviewer_Ns9N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n  * Well written paper.\n  * Thorough evaluations for the cost efficiency.\n\n*Weaknesses*\n  * The metrics are not properly defined. The 'token cost' and 'token consumption' are used in the figures and tables without being properly defined. What exactly is the difference between the two as in Figure 4 they seem to be used interchangeably?\n  * Would it be possible that the reduced token cost is in large part due to the fact that AI agents repeat the same behaviors again and again? This would mean that the cost reduction is large because the baseline is not very great or the agents in the environments are tailored to do limited tasks (Section 5.3) but once the baseline is more human like (e.g. less repetitions) or the agents are less constrained in there conditioning the benefits of the proposed method would diminished. My remark comes from the fact that Figure 3 shows that the heaving lifting of the token cost reduction is due to the Lifestyle memory while Table 3 shows that AGA does not provide more human like activities and dialogue. In addition, Section 5.3 and Table 6 shows that the agents is heavily conditioned to repeatedly do the same things which could explain why having an arxiv of previously executed plans can largely reduce the amount of tokens needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes the Affordable Generative Agents (AGA) to reduce the number of interactions needed in open-world scenarios. The AGA framework is based on two aspects: a Lifestyle Policy that store an arxiv of previously executed plans and retrieved them when needed and a Social Memory to store inter-agents interaction. To assess the validity of the framework, experiments were done on two environments Generative Agents and VirtualHome. In addition, the paper shows that the limits of emerging behaviors in Generative Agents.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Important requested changes:\n  * More details on the metrics (e.g. 'token cost' and 'token consumption')\n  * An analysis on the token cost reduction when mind wandering is applied.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SUQHdzgPH6", "reviewer": "Reviewer_Y5qb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed components are simple and can reduce the cost of using LLMs for generalist agents\n- Various experiments are conducted to show that the AGA frameworks results in lower costs \n\nWeaknesses\n- Not enough detail is provided for the two main components (\"Lifestyle Policy\" and \"Social Memory\") that are the main contribution of this work\n- There is not enough comparison / discussion with the baselines in the main paper about how the proposed modules impacts agent behavior\n- No other methods for lowering the cost of LLM usage is compared against\n- The logic of the writing is hard to follow, with unsupported claims and missing details.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Paper summary\n- The paper proposes \"Affordable Generative Agents\" (AGA), a framework for reducing the cost of generative agents using LLMs.  Generative agents are simulated virtual agents that interact with each other and the environment.   The main idea of the work is to 1) cache and reuse responses from the LLMs and 2) reduce the context sent to LLMs. \n\nThe main contributions are:\n- Two components \"Lifestyle policy\" and \"Social Memory\" for reducing the cost of generalist agents\n- A set of experiments and analysis showing that the proposed components lower the token cost while still being able to come up with action plan and evolve relationships \n\nDetails:\n- The AGA adds two components to the GA framework: 1) \"Lifestyle policy\" for agent-environment interactions, and 2) \"Social Memory\" for agent-agent interactions.  The \"Lifestyle Policy\" finds similar plans, and reuse the decomposition into subplans (thus reducing the cost of calling an LLM to decompose the plan).  The \"Social Memory\" summarizes the agent's interactions with other agents into relationship, feeling, and summary events, which allows for less context to be provided to the LLM.    \n- Experiments are conducted in two environments (\"Stanford Town\" and VirtualHome).  Evaluation shows that with the proposed components, AGA uses less tokens than the baseline (without the proposed components).  The quality of the agents is evaluated by using GPT-4 to score how human-like the responses are for \"Stanford Town\", and the task success rate is measured (by LLM and human) on VirtualHome (where agents are given a task to complete).  Experiments show that AGA is able to achieve comparable (LLM evaluation) success rate or higher (human evaluation of success rate) than the baseline.  Experiments (on \"Stanford Town\") also show that agents using the AGA framework has relationships that change over time with some agents having closer relationships than others. The paper also provides an analysis of when new activities stops developing, and a case study of a developed agent.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Writing improvement to clarify certain claims.  Some of the claims (especially in the introduction) are unsupported or unclear\n    \n    1. *\"real-time, long-term invocation of LLMs to handle various types of agent interactions can result in substantial cost consumption.\"* \n           - Is it necessarily to have real-time, long-term invocation of LLMs? \n\n     2. *\"The cost, which increases linearly with the number of interactions, is unacceptable, particularly when LLM agents are to be used as NPCs in an open world environment.\"* - Please provide more concrete cost estimate for an open world environment, elaborate on why the scenario is different from the two cases studied in this paper, and why the cost would be considered unacceptable.  For instance, from earlier in the introduction, the estimated cost of using LLMs for Stanford Town is just thousands of dollars and simulations of 2-3 agents is just several dollars per hours.  How is that different from the use cases considered in this paper (perhaps by connecting back to the need for \"real-time, long-term invocation of LLMs\")?  Why are those price points \"unacceptable\"? Especially considering that games can cost hundreds of millions of dollars to develop, it is not clear that thousands or numbers or several dollars per hour is \"unacceptable\".  \n\n    3. The sentence that follows: *\"To address the issues above, there has been a significant amount of research\"* implies that the cited methods are for addressing the cost of LLM agents as *\"NPCs in an open world environment.\"*  However that is not the case.  The cited papers are work on decreasing the cost of LLM in general.  The writing is misleading as it implies that there is significant interest in the problem this paper is attempting to address.\n\n    4. *\"the main reasons these methods are unsuitable for believable behavior simulation include the following aspects\"* - The logic of this paragraph is also weak as the reasons given does not appear to have much to do with the existing methods, but challenges of the problem space (lack of unified evaluation, multi-round interaction) and the observation that the authors make toward a solution to the problem (need to reduce redundancy)\n\n    5. *\"These two techniques are distilled from our insights on how a human interacts with other entities in the real world\"* - Please elaborate on what are the specific insights.\n\n2. Improvements to Figure 1\n    - Figure 1 and its caption should be improved to clarify what component of the AGA framework is proposed by this work vs standard components that are part of other generalist agents.  What part calls the LLM and how the \"Lifestyle policy\" and \"Social memory\" components reduces calls to the LLM.\n  \n3. Method description should be improved to 1) provide a description of the generalist agent (GA) framework, 2) provide more details on the contributed components (some of which is found in the supplement but should be in the main paper) and how it differs from prior work.\n    1.  For the \"Lifestyle policy\", it would be useful to include details of the embedding feature (e.g. what is being embedded, how is it embedded), how the similarity threshold is selected, discusion of how the similarity threshold will affect the tradeoff of agent quality vs cost, how the execution condition is specified / determined, etc. \n    2.  For \"Social Memory\", it will be useful to discuss how this work choses to encode interactions the agent has with other agents vs prior work.  Also, it was hard to follow exactly what was happening with \"Social Memory\" -> What exactly is stored in the \"Summary Events\"?  How are the parts of \"Social Memory\" updated?  How is the dialogue prompt formed from the last round of conversation and \"Social Memory\"?  What exact input is provided to the LLM?\n\n4. Improvement to discussion of experiments  \n    - While there is plenty of evidence that the number of tokens is reduced, there isn't enough discussion in the main paper about if / how the lowered cost (e.g. reuse of policy and condensed memory) affects agent behavior.  \n    - There should be a bit more comparison / discussion of how the behavior of the AGA against the baseline.\n    - For instance, for the case study \"Our findings indicate that our agents perform on par with the baseline, providing specific and accurate answers to the questions.\" => There is no comparison with the output of the original baseline, and no discussion of whether there is anything intersting to observe in the developed agent.  Similarly, there isn't any comparison for the relationship evolution, generated actity in VirtualHome, or analysis of activity types.", "broader_impact_concerns": "The broader impact statement can include more specific descriptions of how the behaviors of generalist agents can mislead people (maybe undesirable behavior will be copied).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes two ways to reduce the number of tokens processed by generative agents with some positive results on two different domains. The results show that using the methods proposed in this paper leads to less tokens and to equal or higher human-likeness in terms of behavior (note: as judged by GPT-4). 

Several reviewers commented on the clarity of the writing suggesting many fixes that needed to be made. The authors were quite responsive and made significant changes to the paper. This was quite influential to the positive decision of one reviewer. Another reviewer also mentioned it. I commend the authors on being pro-active in this respect.

One of the reviewers was positive despite the methods not being especially innovative. The other reviewer was negative, specifically because the choice of domains contained exactly the properties that would lead to the methods working well. I appreciate these two criticisms and I urge the authors to consider them in future versions or in follow-up work. The first one is not a criterion that determines acceptance at TMLR, so I'm not considering it in this decision, but it could affect the long-term impact of the work. The second one is a valid criticism, but in my opinion it is not a severe limitation -- many environments of interest will have properties of repeated interaction that resemble Generative Agents and Virtual Home; in fact Q&A tasks such as technical support for companies could be another use case where there is a lot of repetition and efficient use of LLM tokens could benefit from such methods.

I am slightly disappointed by the lack of technical depth for a technical journal such as TMLR, but the evidence does support the claims and I do not think this issue is too severe that the TMLR audience would not be interested. In particular, according to the acceptance criteria, and two of three recommendations to accept, and the clear impact made by the paper, improvements to the presentation, I am happy to recommend accept. 

Note: I chose "Accept with minor revision" only because the current PDF is marked up with colors and removals. The only minor revision is removing the text that is struck out and removing the colored highlights for the text that is to remain.
Decision: Accept (poster)