# A Multi-Agent Based Intelligent Training System for Unmanned Surface Vehicles
## 3 Multi-Agent-Based Training System Design
### Multi-Agent-Based USV Training Algorithm Design
After all these KBs have taken turns performing the control task and have been evaluated, they are then subjected to the GA operators (Lines 6-17). This operation continues till the termination condition is reached. The resulting candidate KBs are returned as the best performing or most suitable of the KBs. ```
0: GA hyperparameters
0: Best set of GFT KBs
1: Initialize \(t=0\)
2: Generate initial population as \(\mathcal{P}_{0}\)
3: Set current population \(\mathcal{C}_{t}:=\mathcal{P}_{0}\)
4: Run the fuzzy control system using \(\mathcal{C}_{t}\)
5: Evaluate the members of \(\mathcal{C}_{t}\)
6:while Not termination conditiondo
7:\(\mathbb{P}_{t}:=selection(\mathcal{C}_{t})\)
8:\(\mathcal{O}_{t}:=crossover(\mathbb{P}_{t})\)
9:\(\mathcal{C}_{temp}:=mutate(\mathcal{O}_{t})\)
10:if elitismthen
11:\(\mathcal{C}_{temp}:=applyEitism(\mathcal{C}_{temp})\) //keeps a percentage of previous chromosomes
12:endif
13:\(t:=t+1\)
14:\(\mathcal{C}_{t}:=\mathcal{C}_{temp}\)
15: Run the fuzzy control system using \(\mathcal{C}_{t}\)
16: Evaluate the members of \(\mathcal{C}_{t}\)
17:endwhile
18:Return best-performing candidate solutions
```  
**Algorithm 1** Procedure: GFS procedure. Non-stationarity is an inherent problem in the multiple learning agents' environment. In this section, we present an approach for incorporating agent-induced non-stationarity awareness into a GFT based on the framework of [32]. Since the root of the GFT has the strongest impact on the learning process or action selection, we consider it to be the main component for addressing this problem. Unlike the other GFSs in the GFT, which can be modeled to address specific aspects of a control problem (domain dependent), we consider the root GFS to be a special case. We regard the sub-trees that are formed under the root GFS as the elements of the BR function co-domain. Thus, a BR function, in the GFT case, selects a sub-policy to be used for selecting an action for the agent. The influence function is also represented as an FIS (or GFS in our case, since GA is applied to learn the KB), which takes PGFsas input variables and a BF as output the variable. This is motivated by the idea that an influence function maps beliefs to possible best responses and can be  
Figure 5: The knowledge base genetic structure of a genetic fuzzy tree. Each chromosome or member of the GA population takes this form. designed to use deductive reasoning. These are properties that an FIS exhibits due to fuzzy logic. Thus, the root of the GFT takes incomplete/partial observations and produces its belief of another agent's policy. This connotes that the terms of the output variables then serve as possible opponent strategies as perceived by the reasoning agent. In the case of the PGFs that serve as input variables to the GFT root FIS, the history of observations concerning the opponent or another agent is maintained and used by each PGF variable to produce an input value. This PGF input value can then be fuzzified and used in the inference process. The problem that arises is then how to design a PGF to capture the adaptation dynamics of another agent. In the work of [33], an agent \(i\) estimated the model of agent \(j\) as:  
\[\partial_{j}^{i}(u_{j})=\frac{C_{j}^{i}(u_{j})}{\sum_{a\in U_{j}}C_{j}^{i}( \bar{a}_{j})} \tag{7}\]  
where \(C_{j}^{i}(u_{j})\) is the frequency that agent \(i\) observed agent \(j\) taking action \(u_{j}\). Therefore, given the history of observations, Equation (7) can be extended to include the observation of agent \(i\) as:  
\[\partial_{j}^{i}(o_{i},u_{j})=\frac{C_{j}^{i}(o_{i},u_{j})}{\sum_{a\in U_{j}}C _{j}^{i}(o_{i},\bar{a}_{j})} \tag{8}\]  
where \(C_{j}^{i}(o_{i},u_{j})\) is the frequency that agent \(i\) observed agent \(j\) taking action \(u_{j}\) when its local observation was \(o_{i}\). \(\partial_{j}^{i}(o_{i},u_{j})\) becomes the input value of agent \(i\)'s PGF for monitoring a specific action \(u_{j}\) of agent \(j\).