## 3 System Design and the Proposed SEAC-Based Resource Allocation Methodology
### Multi-Agent MDP Model  
The SEAC algorithm builds on the actor-critic framework in a multi-agent MDP model to address the limited information in decentralized systems. The MDP components include the state \(S\) representing the shared environment where all agents operate, actions \(A_{i}\) taken by each agent that together form a joint action space \(A\), and rewards \(R_{i}\) given to each agent based on the state and actions. The system then transitions to the next state, which is influenced by the collective actions of all the agents. In SEAC, agents share their experiences--comprising state, action, and reward information--with other agents. This allows each agent to update its actor-critic model using both local and shared experiences, leading to improved decision making and more efficient resource allocation in future states [23]. The SEAC framework integrates actor and critic networks for effective learning. The actor network, parameterized by \(\phi\), selects actions based on the current state, while the critic network, parameterized by \(\theta\), evaluates these actions by estimating the value function. Agents leverage both their own and others' experiences through a shared experience pool. This off-policy learning approach adjusts the actor and critic loss functions to incorporate diverse interactions, enhancing the adaptability and resource allocation efficiency in dynamic environments. In SEAC, agents share their experiences through a common experience pool, enhancing the learning efficiency. The goal is to determine policies \(\pi=(\pi_{1},\pi_{2},\dots,\pi_{N})\) that maximize the total reward over time \(T\):  
\[R^{i}=\sum_{t=0}^{T-1}\beta^{t}r_{t}^{i} \tag{10}\]  
where \(\beta\) is the discount factor, and \(r_{t}^{i}\) is the reward obtained by agent \(i\) at time \(t\). The main objective for all agents is to maximize the expected reward:  
\[\underset{\pi_{i}^{i}}{max}E\Big{[}R^{i}\Big{|}\pi^{\prime},\pi_{-i}\Big{]} \ \forall i\in\pi_{i} \tag{11}\]  
where \(\pi_{-i}\) denotes the policies of all agents except \(i\). Each agent adjusts its strategy by minimizing losses in both the actor and critic networks, taking into account the partial observability of the environment. The actor network's loss, which guides how it changes its policy, is defined as  
\[L(\varphi_{i})=-\log\pi(a_{i}^{i}\Big{|}o_{i}^{i},\varphi_{i})(r_{t}^{i}+\beta Q ^{\prime}\Big{(}s_{t+1},\ a^{\prime},\ \theta Q^{\prime}\Big{)}-Q\ \Big{(}s,\ a,\ \theta Q \Big{)}) \tag{12}\]  
where  
* \(\varphi_{i}\) are the actor's network parameters;
* \(Q\ \big{(}s,\ a,\ \theta Q\big{)}\) is the value function computed by the critic network, and \(Q^{\prime}\Big{(}s_{t+1},\ a^{\prime},\ \theta Q^{\prime}\Big{)}\) is the target value function. The critic network's loss function, which helps in evaluating actions, is  
\[L(\theta_{i})=\left||Q\ (s,\ a,\ \theta^{Q}-y_{t}||\right|^{2} \tag{13}\]  
Here, \(Q\ \big{(}s,\ a,\ \theta^{Q}\big{)}\) is the online value network, and \(y_{t}\) is the target value network. The target value \(y_{t}\) is calculated using  
\[y_{t}=r_{t}^{i}+\beta Q^{\prime}\Big{(}s_{t+1},\ a^{\prime},\ \theta^{Q^{\prime}} \Big{)} \tag{14}\]  
where  
* \(r_{t}^{i}\) is the reward obtained by agent i at time t;
* \(\beta\) is the discount factor;
* \(Q^{\prime}\Big{(}s_{t+1},\ a^{\prime},\theta^{Q^{\prime}}\Big{)}\) is the target value for the next state-action pair. In SEAC, each agent not only learns from its own experiences but also incorporates experiences from other agents. This shared experience is considered \"off-policy\" data.