## 3 Methodology
### PG-DMPC Algorithm  
This section details the PG-DMPC algorithm implemented as a negotiating agent required for the upper layer of a distributed MPC control system. The aim of the upper layer PG-DMPC algorithm is to achieve a consensus among all candidate control sequences for each agent to obtain the final control action implemented. In order to reach consensus, the deep neural network receives the available control sequences for each agent as _state_\(s_{t}\),  
\[s_{t}=[U_{1}(t),U_{2}(t),\ldots,U_{N}(t)],\]  
Figure 2: PG optimization scheme for the required negotiation agent in the upper-level control layer in the distributed MPC-based control system. where \(U_{i}=\left[U_{i}^{f_{1}}(t),U_{i}^{f_{2}}(t),\ldots,U_{i}^{f_{M_{i}}}(t)\right]\), \(N\) is the number of agents, \(i\in\{1,2,\ldots,N\}\), and \(M_{i}\) is the total number of pairwise fuzzy negotiations in the lower layer performed by agent \(i\). The total number of control sequences received by the PG-DMPC upper level is defined as \(M=\sum_{i=1}^{N}M_{i}\). The states \(s_{t}\in S\), where \(S\) is a compact subspace \(S\subseteq\mathbb{R}^{N\cdot M\cdot H_{p}}\). The action \(a_{t}\in A\) where \(A\subseteq\mathbb{R}^{N\cdot M}\) is a compact set, \(a_{t}\) are some coefficients weighting the control sequences to compute a suitable final control sequence \(U_{i}^{f}\) to be applied in the plant for agent \(i\), this procedure being the way to achieve consensus among agents in the upper layer. Particularly, the action vector is defined as \(a_{t}=\{\phi_{i},\ldots,\phi_{N}\}\), where \(\phi_{i}=\left[a_{i}^{1},a_{i}^{2},\ldots,a_{i}^{M_{i-1}}\right]\), \(i\in\{1,\ldots,N\}\). Note that for agent \(i\), the number of coefficients provided is \(M_{i-1}\) because the last one is obtained as the complement to \(1\) in order to reduce the computational load, and provided that \(\sum_{m=1}^{M}a_{i}^{m}<1\) due to normalization when training the deep neural network,  
\[\alpha_{i}^{M_{i}}=1-\sum_{m=1}^{M_{i}-1}\alpha_{i}^{m}. \tag{6}\]  
Finally, the reward provided by the environment at each time step is defined in Section 4.2.1 because it depends on the particular case study and its global control objective. Considering states and actions defined above, Algorithm 1 developed negotiation in this paper is described:  
At each time step \(t\) for each agent \(i\):  
1. The states \(s_{t}\) are taken from the environment (low-level control layer) by the RL upper layer (deep neural network) defined by non-linear function \(\zeta\). 2. The deep neural network takes the states as inputs in order to provide the actions \(a_{t}\) which are the negotiation coefficients \(\alpha_{i}^{m}\), for \(i\in\{1,\ldots,N\}\), \(m\in\{1,\ldots,M_{i-1}\}\), as outputs, \[a_{t}=\zeta(s_{t}).\] The deep neural network \(\zeta\) outputs satisfy \(\sum_{m=1}^{M_{i-1}}\leq 1\) because it is a constraint considered at training for normalization. 3. The final control sequence \(U_{i}^{f}\) is obtained, considering the actions provided by \(\zeta\), \(U_{i}^{f}(t)=\sum_{m=1}^{M_{i}}U_{i}^{f_{m}}(t)\cdot a_{i}^{m}\). 4. The control sequences for each agent are aggregated to \(U_{N}=\left[U_{1}^{f},U_{2}^{f},\ldots,U_{N}^{f}\right]\). \(U_{N}^{f}(t)\) is computed and compared with the global cost \(f_{N}^{f}(x_{N}(t))\) from the previous time \(t\). Otherwise, the previous \(t\) control sequences that check stability, \(U_{N}^{s}(t+1)\), are applied. Hence, the global cost function is \[ \begin{split}& f_{N}^{f}\Big{(}x_{N}(t),U_{N}^{f}(t)\Big{)}= \sum_{k=1}^{H_{p}-1}\Big{(}\|x_{N}(t+k)-x_{r_{N}}(t+k)\|_{\mathbb{Q}_{N}}^{2}+ \\ &\|u_{N}(t+k)-u_{r_{N}}(t+k)\|_{R_{N}}^{2}\Big{)}+\|x_{N}(t+H_{p} )-x_{r_{N}}(t+H_{p})\|_{P_{N}}^{2},\end{split} \] (7) with the weighting matrices \(Q_{N}=\left[Q_{i}\right]_{i\in N}\) and \(R_{N}=\left[R_{i}\right]_{i\in N}\) and the terminal cost matrix \(P_{N}=\left[P_{i}\right]_{i\in N}\), \(x_{r_{N}}\) and \(u_{r_{N}}\) the global state and input references is calculated by a procedure to remove offset based on [30].