# Revising the Observation Satellite Scheduling Problem Based on Deep Reinforcement Learning
## 1 Introduction  
Earth observation satellites (EOSs) are platforms equipped with optical instruments in order to take photographs of specific areas at the request of users [1]. Currently, EOSs have been extensively employed in scientific research, mainly in environment and disaster surveillance [2], ocean monitoring [3], agricultural harvesting [4], etc. However, with the increase in multi-user and multi-satellite space application scenarios [5, 6], it is becoming more difficult to meet various observation requirements under the limitation of satellite resources. Therefore, an effective EOS scheduling algorithm plays an important role to improve high-quality space-based information services, and not only guides the corresponding EOSs on how to perform the following actions, but also controls the time to start the observations [5]. The main purpose is to maximize the observation profit within the limited observation time window and with other resources (for example, the available energy, the remaining data storage, etc.) [7, 8]. The EOS scheduling problem (EOSSP) is well known as a complex non-deterministic polynomial (NP) hard problem and multiple objective combinational optimization problem [9, 10]. Currently, inspired by increasing demands on scheduling EOSs effectively and efficiently, the study of the EOSSP has gained more and more attention. [PERSON] et al. [7] summarized that the EOSSP could be divided into time-continuous and time-discrete problems. In time-continuous models [11, 12, 13], a continuous decision variable is introduced to represent the observation start time for each visible time window (VTW), which is defined to check whether tasks are scheduled or not. On the other hand, for time-discrete models [14, 15, 16], each time window generates multiple observation tasks for the same target. In this way, each candidate task has a determined observation time, and binary decision variables are introduced to represent whether a task is operated in a specific time slice. General solving algorithms are usually classified into exact method, heuristic, metaheuristic [17; 18] and machine learning [7]. Exact methods, such as branch and bound (BB) [19] and mixed integer linear programming (MILP) [15], have deterministic solving steps and could solve problems in polynomial time, but it is nearly impossible to build a deterministic model for a larger scale problem. Heuristic methods can be used to speed up the process by finding a satisfactory solution, but this method depends on a specific heuristic policy and the policy is not always feasible. [PERSON] et al. [20] proposed a heuristic solution approach to solve the image collection planning problem of the KOMPSAT-2 satellite. [PERSON] et al. [21] combined the neural network method and heuristic search algorithm and the result was superior to the existing heuristic search algorithm in terms of the overall profit. Alternatively, without relying on a specific heuristic policy, metaheuristic methods could provide a sufficiently high-quality and universal solution to an optimization problem. For example, [PERSON] et al. [22] proposed an optimal algorithm based on a genetic algorithm for synthetic aperture radar (SAR) imaging satellite constellation scheduling. [PERSON] et al. [23] presented a multi-objective genetic algorithm to solve the problem of satellite areal task scheduling during disaster emergency responses. [PERSON] et al. [24] proposed a two-phase GA-SA hybrid algorithm for the EOSSP which is superior to the GA or SA algorithm alone. Although metaheuristic algorithms could gain better operation results and have been widely adopted in the EOSSP, they easily fall into local optimum [25] due to the dependence on one certain mathematical model. Consequently, this makes us turn to deep reinforcement learning (DRL), which is known as a model-free solution and can autonomously build a general task scheduling strategy by training [26]. It has the promising potential to be applied in combinatorial optimization problems [27; 28; 29].