# A Period Training Method for Heterogeneous UUV Dynamic Task Allocation
## 1 Introduction  
With the rapid development of UUV technologies, multi-UUVs can cooperatively perform various complicated tasks, such as target localization, photogrammetry, and cooperative search coverage [1; 2; 3]. Usually, multiple heterogeneous UUVs equipped with various sensors are allocated to search irregular task areas, where the UUVs may be broken due to unpredictable threats, and many new search tasks need to be conducted. To deal with emergencies, each UUV is required to adaptively reallocate its task sets, resulting in multiple UUV search coverage task allocation problems in dynamic environments.  
To solve the UUV task allocation problem, many heuristic algorithms have been researched in recent years. To deal with large-scale geospatial search problems, [PERSON] et al. use a novel iterative greedy method to plan the search paths of UUVs, which the algorithm scales well to the deployment of large groups of agents [3]. In [4], a new technology based on receiver operating characteristic (ROC) analysis is used to centrally handle the UUV task allocation problem, and the path planning problem is solved using a distributed model. To address the dynamic UUV task allocation problem, [PERSON] et al. design a tri-level optimization method to plan safe paths in severe ocean environments, in which UUVs have limited detection ranges and are required to respond to emergencies in real time [5]. To guarantee the effectiveness of scheme allocation, heuristic algorithms commonly require a certain number of optimization iteration processes to generate, update, and select solutions, leading to high time consumption in dynamic task environments.  
Multi-agent reinforcement learning (MARL) provides a novel fast-solving framework to solve multi-agent task planning problems [6; 7]. MARL formulates problems as a sequential Markov decision process, where intelligent agents collaboratively complete tasks in a shared environment [8; 9]. In terms of the control and obstacle avoidance of UUVs, [PERSON] et al. use the decentralized MARL training framework and improve the original multi-agent generative adversarial imitation learning method by introducing a new randomly selecting and updating method [10]. In [11], the researchers construct multiple reward functions to adjust the action distribution of UUVs, which quickly allocate task sequences in ocean current environments. MARL has achieved many significant results in different UUV planning problems. Therefore, it has great potential in UUV coordination task allocation. However, real task scenarios involve dynamic changes due to emergencies. To ensure the completion of tasks, it is inevitable for MARL to reallocate task schemes in different task environments. The changing numbers of UUVs reduce the generalization performance of the MARL algorithm and may lead to model invalidation [11; 12].  
In this paper, we propose a novel MARL with an attention mechanism (MARLAM) to quickly assign tasks to heterogeneous UUVs. Herein, we propose a novel task allocation algorithm based on multi-agent reinforcement learning (MARL) and a period training method (PTM). The designed MARL model employs an encoder-decoder framework with an attention mechanism to adaptively allocate many irregular task areas for heterogeneous UUVs. In addition, PTM uses periodically changing training conditions to optimize the model parameters, which enables the MARL model to achieve better scalability after the training process. The experiments demonstrate that the proposed MARLAM with the designed period training method (MARLAM-PTM) can use a trained MARL model to quickly allocate task schemes in different task environments.