# Revising the Observation Satellite Scheduling Problem Based on Deep Reinforcement Learning
## 1 Introduction
As a significant research domain in machine learning, DRL has achieved success in game playing and robot control. Moreover, recently, DRL has also gained more and more attention in optimization domains. [PERSON] et al. [27] presented a DRL framework using neural networks and a policy gradient algorithm for solving problems modeled as the traveling salesman problem (TSP). Additionally, for the TSP, [PERSON] et al. [28] embedded a graph in DRL networks and it learnt effective policies. Furthermore, [PERSON] et al. [29] gave an end-to-end framework with a parameterized stochastic policy for solving the vehicle routing problem (VRP), which is an expanded problem based on the TSP. [PERSON] et al. [30] presented a dynamic attention model with dynamic encoder-decoder architecture and obtained a good generalization performance in the VRP. Besides the TSP and VRP, another type of optimization problem, resource allocation [25], also has been solved by RL. [PERSON] et al. [31] adopted RL to schedule time resourced for a railway system, and it was found that the Q-learning algorithm is superior to heuristic approaches in effectiveness. [PERSON] et al. [32] proposed a decentralized resource allocation mechanism for vehicle-to-vehicle communications based on DRL and improved the communication resource allocation. Inspired by the above applications of DRL, solving the EOSSP by using DRL has become a feasible solution. [PERSON] et al. [33] adopted A2C to handle the EOSSP in order to reduce the time to completion of large-area requests. [PERSON] et al. [34] proposed a real-time online scheduling method for image satellites by importing A3C into satellite scheduling. [PERSON] et al. [35] developed a two-phase neural combinatorial optimization RL method to address the EOSSP with the consideration of the transition time constraint and image quality criteria. [PERSON] et al. [36] proposed a training system based on RL which is fast enough to generate decisions in near real time. In the present paper, the EOS scheduling problems as a time-continuous model with multiple constraints are revised by adopting the deep deterministic policy gradient (DDPG) algorithm, and comparisons with the traditional metaheuristic methods are conducted with an increase in the task scale. The major highlights are summarized as follows:  
1. Aiming to enhance the task scheduling efficiency further, an improved graph-based minimum clique partition algorithm is introduced as a task clustering preprocess to decrease the task scale and improve the scheduling algorithm's effect. Different from previous studies, the EOSSP was considered as a time-discrete model when solving by RL algorithms. In this paper, a time-continuous model is established for the EOSSP, which could make accurate observation time decisions for each task by the DDPG algorithm. 3. Considering practical engineering constraints, comparison experiments were implemented between the RL method and some metaheuristic methods, such as the GA, SA and GA-SA hybrid algorithm, to validate the feasibility of the DDPG algorithm.