# Multi-Agent System for Emulating Personality Traits Using Deep Reinforcement Learning +
## 2 Materials and Methods
### Background  
#### 2.1.1 Reinforcement Learning  
A Reinforcement Learning (RL) model is typically formalized as a Markov Decision Process (MDP) represented by the 5-tuple \(M=(S,A,p,\gamma,R)\). In this tuple, \(S\) denotes the state space (all possible states that the environment can be in at any given time), A is the action space (all possible actions the agent can take in the environment), p is the environment dynamics function (rules or probabilities that define how the environment transitions between states), \(\gamma\) is the discount factor (weighing future rewards relative to immediate rewards), and R is the reward function (feedback mechanism that the environment uses to indicate the success or failure of the agent's actions) [11]. As an environment, we define the world context in which our RL agents operate.  
In our agent system, we use a centralized critic for the environment dynamics function p and incorporate a discount factor \(\gamma\) of 0.99, as they are set by the ml agents package and our configuration files. The observation space, action space, and rewards are discussed in detail in the next sections.  
Training in RL occurs through iterative interactions between the agent and the environment. The agent learns to maximize cumulative rewards by selecting actions, receiving feedback, and adjusting its policy, which is the strategy it uses. Each step provides rewards, guiding the agent to refine its decision-making strategy. The process unfolds across multiple episodes, which are sequences of interactions that start from an initial state and end when a specific goal or condition is reached. Over time, the agent balances exploration (trying new actions) and exploitation (leveraging learned strategies) to optimize long-term outcomes [11].  
It is important to note that multiple autonomous cooperating agents, collectively referred to as a \"multi-agent system\", operate within a shared environment. These agents observe the environment using sensors and interact with it through actuators. While  
Figure 1: The workflow of the MindEscape environment (the colored entities are the ones analyzed in this paper).  
pre-designed behaviors can be embedded in such agents, they often need to learn new behaviors and actions online. This continuous learning process leads to improvements in the performance of individual agents, as well as the overall multi-agent system [12].  
We employed the MA-POCA (Multi-Agent Posthumous Credit Assignment) algorithm from the ML-agents package [13]. The algorithm learns a centralized value function to estimate the expected discounted returns of the group of agents and a centralized agent-centric counterfactual baseline to facilitate credit assignment [8]. Essentially, this allows for the agents that have escaped from the environment to continue training and learn quicker and more effectively.  
This was the main reason we chose MA-POCA against others like Independent Q-Learning that can be limited to complicated and dynamic environments [14] or the Multi-Agent Advantage Actor-Critic (MA-A2C) that struggles in synchronization across agents and may struggle in high-dimensional environments [15].