## 3 Methodology  
Given a problem to be solved by the RL method, the environment is modelled as a Markov decision process with state space \(S\), action space \(A\), state transition function \(P(s_{t+1}|s_{t},a_{t})\), with \(s\in S\) and \(a\in A\), \(t\) as the time step, and a reward function \(r(s_{t},a_{t})\). The training of an agent consists of the interaction of the _agent_ with the _environment_, for which the _agent_ sends an _action_\(a_{t}\) to the _environment_, and it sends back a _state_\(s_{t+1}\) and a _reward_\(r_{t+1}\) as a qualifier of the \(a_{t}\) according to the _environment_ objectives. Consequently, each episode training generates an episode experience composed of the sequence \(\{s_{t},a_{t},r_{t+1},s_{t+1},\ldots,s_{T-1},a_{T-1},s_{T},r_{T}\}\) with \(t\in\{0,1,\ldots,T\}\) where T is the time horizon.  
The PG algorithm selects \(a_{t}\) based on a stochastic policy \(\pi_{\theta}(a_{t}|s_{t})\) that assigns states to actions \((\pi:S\to P(A))\). For this task, the algorithm uses a deep neural network as an approximation function of the stochastic policy \(\pi_{\theta}(a_{t},s_{t})\). The parameter of the policy optimization \(\theta\) is performed through the ascent optimization method of the gradient. This optimization aims to maximize for each state in the episode training for \(t=1,2,\ldots,T-1\), the discounted future reward,  
\[G_{t}=\left\{\sum_{t=0}^{T}\gamma^{t}r_{t}\right\} \tag{2}\]  
where \(\gamma\) is a discount factor, and \(r_{t}\) is the reward received at \(t\) time step. Therefore, the objective function is  
\[G_{t}\cdot\
abla_{\theta}log\pi_{\theta}(a_{t}\mid s_{t}). \tag{3}\]  
Figure 1: PG-DMPC algorithm proposed in the upper-level PG negotiation control layer within the control architecture.  
The score function \(\
abla_{\theta}\,log\,\pi_{\theta}(a_{t}\mid s_{t})\) allows optimization to be achieved without requiring the dynamic model of the system, optimizing the gradient of the logarithm of \(\pi_{\theta}(a_{t}|s_{t})\) which expresses the probability that the agent selects \(a_{t}\) given a \(s_{t}\).  
The estimate of the discounted future reward \(G_{t}\) made by the Monte Carlo method has a high variance, leading to unstable learning updates, slow convergence, and thus slow learning of the optimal policy. The baseline method takes from \(G_{t}\) a value given from a baseline function \(b(s;\varphi)\) to address the variance in the estimate,  
\[\hat{A}_{t}=G_{t}-b_{\varphi}(s), \tag{4}\]  
where \(\hat{A}_{t}\) is called the advantage function. Consequently, the objective function becomes,  
\[\hat{A}_{t}\cdot\
abla_{\theta}\,log\,\pi_{\theta}(a_{t}\mid s_{t}). \tag{5}\]  
The exploration policy used by PG is based on a categorical distribution, where \(\pi_{\theta}(a\mid s)=\mathbb{P}(a\mid s;\theta)\) is a discrete probability distribution that describes the probability \(\pi(s,a\,;\theta)\) that the policy can take action \(a\) of a set \(k\) actions, with \(i\in\{1,\ldots,k\}\), given a state \(s\), with the probability of each action listed separately and is the sum of the probabilities of all the actions equal to zero, \(\sum_{t=1}^{k}p_{i}=1\).  
Figure 2 shows the policy gradient algorithm for the negotiating agent training. The optimization of deep neural networks is based on the input data collected at training by a categorical distribution exploration policy. Model knowledge is not necessary, only local decisions as the _state_ and control objectives as the _reward_, a consequence of the _action_ of the previous \(t\) time step employed to agree among the local decisions and obtain the global decisions of the agents.