# A Multi-Agent Based Intelligent Training System for Unmanned Surface Vehicles
## 2 Multi-Agent-Based USVs' Training Problem  
We first discuss the multi-agent-based intelligent training system learning problem of this study.  
In general, a stochastic game is an extension of the Markov decision process (MDP) to the multi-agent context. In such a game, agents may have conflicting goals, and their joint actions determine rewards and transition between states. Stochastic games usually assume agents as having a complete view of their states, whereas the partial observation case is discussed under the more general partially-observable stochastic games (POSGs) domain. Our study is performed under the POSG case.  
**Theorem 1**.: _A POSG is a tuple \(<X,U_{1}...U_{n},O_{1}...O_{n},f,R_{1}...R_{n}>\) where:_  
* \(n\) _is the number of agents_
* \(X\) _is the finite set of states_
* \(U_{i},i=1,...,n\) _is the finite set of actions available to the agents, which form the joint action set_ \(\mathbf{U}=U_{1}\times...\times U_{n}\)__
* \(O_{i},i=1,...,n\) _is the finite set of observations of the agents. The joint observation is denoted as_ \(\boldsymbol{o}=\{o_{1},...,o_{n}\}\)__
* \(f:X\times\mathbf{U}\times X\rightarrow[0,1]\) _is the Markovian state transition function where_ \(f(\boldsymbol{x}^{\prime}|x,\boldsymbol{u})\) _denotes the probability of reaching state_ \(\boldsymbol{x}^{\prime}\) _after the joint action_ \(\boldsymbol{u}\) _in state_ \(x\)_. The joint action of all agents_ \(\boldsymbol{u}\) _at a stage_ \(t\) _is denoted as_ \(\boldsymbol{u}_{t}=[u_{1,t},...,u_{n,t}]^{\top}\)__\(u_{t}\in\mathbf{U},u_{i,t}\in\mathbf{U}_{i}\)__
* \(R_{i}:X\times\mathbf{U}\times X\rightarrow\mathbb{R}\)_,_ \(i=1,...,n\) _are the reward functions of the agents._In this study, we consider a team of multiple unmanned surface vehicles that must be trained to perform a given mission \(m\) in a mission space \(\mathcal{D}\subset\mathbb{R}^{n}\) consisting of opposite forces. Let \(H(m)\) be the objective function dependent on the mission. Given that a team \(T\) consists of \(N\) USVs, the dynamics of the \(i^{\text{th}}\) USV is given by:  
\[\begin{split}\dot{x}_{i}&=f_{i}(x_{i}(t),u_{i}(t)) \\ i&=1,2,3...,N\end{split} \tag{1}\]  
where \(x_{i}(t)\in\mathbb{R}^{n}\) and \(u_{i}(t)\in\mathbb{R}^{m}\) are the state and control inputs (available actions) of the USV, respectively, at time \(t\). Depending on the USV model and mission type, a set of constraints (such as obstacle avoidance and the physical limitations of the USV) on the state \(x_{i}(t)\) of a USV are imposed. The control inputs of a USV are bounded according to the limits of the USV's actuators \(|u_{i}(t)|<u_{max}\).  
The reward of an USV \(i\) after taking action \(u_{i,t}\in U_{i}\) in stage \(t\) is denoted as \(r_{i,t+1}\). The individual policies \(h_{i}:X\times U_{i}\rightarrow[0,1]\) of the USVs in a team form the joint policy \(\mathbf{h}\). In the case of team learning, the joint policy is defined by the single learner. Since the reward of each USV in a team depends on the joint action, the respective USV rewards depend on the joint policy:  
\[R_{i}^{\text{h}}(x)=\mathbf{E}\bigg{\{}\sum_{t=0}^{T}\gamma^{t}r_{i,t+1}|x_{0 }=x,\mathbf{h}\bigg{\}} \tag{2}\]  
where \(\gamma\) is the discounting factor and \(T\) is the length of the horizon. The training objective for the intelligent training system is to find a policy that maximizes Equation (2) such that the trained team of USVs can outperform its rival USVs, while adapting to the changing dynamics of it's operating environment.