## 4 Proposed Algorithm for Multi-Agent Configuration Change
### Improved Multi-Agent Proximal Policy Optimization  
The research task is modeled as decentralized partially observable Markov Decision Processes (DEC-POMDP) with shared rewards. A DEC-POMDP in multi-agent systems is defined as \(M=\langle S,\{\mathbf{\mathcal{A}}\},O,\{\Omega_{i}\},r,\rho,N,\gamma\rangle\), where \(s\in S\) is the state space of the environment, \(\mathbf{\mathcal{A}}\) is the joint action space with each agent's individual action space as \(\mathbf{\mathcal{A}}=\prod_{i\in N}a_{i}\). \(O(o|s,a)\) are the observation probabilities for each agent with a given state \(p_{i}s\). \(\rho(s^{t}|s,\mathbf{\mathcal{A}})\) (\(S\times\mathbf{\mathcal{A}}\times S^{\prime}\to[0,1]\)) represents the transition probabilities function when \(s(s_{t}\in S)\to s^{\prime}(s_{t+1}\in S^{\prime})\). \(r_{i}(s,a_{i})\) denotes the agent's reward function. \(\gamma\in[0,1)\) is the discount factor for calculating the future rewards. The fundamental process is as follows: agents use joint policy \(\mathbf{\pi}(\mathbf{a}_{t}|o_{t})\equiv\{\tau_{1},\tau_{2},\dots,\tau_{N}\}\) parameterized by \(\theta\) to execute the joint action \(\mathbf{\mathcal{A}}\) to the local observation. Then, updating each agent's policy based on the received reward and the observed state transition \(s(s_{t}\in S)\to s^{\prime}(s_{t+1}\in S^{\prime})\). The final objective is to maximize the expected cumulative return of each agent at each time step by optimizing their strategies during the iterated interactions with the environment, as \(J(\theta)=\mathbb{E}_{a,s}[\sum_{t}\gamma^{t}R(s_{t},a_{t})]\). Algorithm which is described in the end of Section 4. As for the MAPPO algorithm, it is a policy-based gradient-type method with two separate networks (policy/value) that extends the PPO algorithm from a single-agent to multi-agent system. It utilizes the centralized training with decentralized execution (CTDE) framework, shown in Figure 6. Thus, each agent owns its own actor/critic networks and updates optimal policy, which is typically parameterized with parameters \(\theta\) in the actor network, based on the commonly used formulation:  
\[\Delta\theta=\hat{\mathbb{E}}_{t}[\
abla_{\theta}\log\pi_{\theta}(a_{n}^{t}|s_ {n}^{t})\hat{A}(s_{n}^{t},a_{n}^{t})], \tag{7}\]  
where \(\hat{\mathbb{E}}_{t}[\cdots]=1/N\sum_{n=1}^{N}\sum_{t=1}^{Tn}\left(\cdots\right)\) is the average sample of \(\mathbb{E}_{\tau}[\cdots]\). \(\hat{A}(s_{n}^{t},a_{n}^{t})\) is the estimation of advantage function \(A_{\pi}(s_{n}^{t},a_{n}^{t})\), which can be estimated by the state value function \(V_{\pi}(s_{t})\) in the critic network with parameters \(\omega\), calculated as follows:  
\[A(s_{t},a_{t})\approx r_{t}+\gamma*V_{\omega}(s_{t+1})-V_{\omega}(s_{t}), \tag{8}\]  
where \(V_{\omega}(s_{t})\) is the parameterized critic of the \(V_{\pi}\) estimation, and the parameter \(\omega\) is updated by minimizing the loss function as follows:  
\[\underset{\omega}{\text{argmin}}L(\omega)=\frac{1}{N}\sum_{n=1}^{N}\sum_{t=1} ^{Tn}\left[V_{\omega}^{\pi}(s_{n}^{t})-\delta_{n}^{t}\right]^{2}, \tag{9}\]  
where \(\delta_{n}^{t}=r_{t}+\gamma*V_{\overline{\omega}}^{\pi}(s_{t+1})\), and \(\overline{\omega}\) is updated with the recent \(\omega\) to stabilize the learning process in target network. Thus, two parameters \(\theta\) and \(\omega\) are updated iteratively by interacting with the environment. Normally, the value function \(V_{\pi}(s_{t})\) is often designed as the baseline to be used in estimating \(A_{\pi}(s_{n}^{t},a_{n}^{t})\) as follows:  
\[\hat{A}(s_{n}^{t},a_{n}^{t})=A(s_{n}^{t},a_{n}^{t})-b(s_{n}^{t}), \tag{10}\]  
where \(b(s_{n}^{t})=V_{\omega}(s_{t})\) is the baseline to reduce the variance of the gradient by providing constant values for the state, making the policy update smoother. However, using statewhere \(\beta\) and \(\alpha\) are the hyperparameters.