# A Reinforcement Learning Method for a Hybrid Flow-Shop Scheduling Problem
## 2 Literature Review  
For solving HFSP, a lot of precious works have been done. On exact algorithms, [PERSON] et al. [20] proposed a branch and bound method (B&B) based on an improved search strategy. When the maximal number of workpieces reaches 1000, the deviation from the lower limit remains within 0.1%, but its performance remains to be improved for medium-scale problems with workpieces from 20 to 50. [PERSON] et al. [21] studied real-time and wait-free HFSP in the background of steel production, established an integer programming model, and proposed an improved Lagrangian relaxation algorithm, which can achieve satisfactory optimization performance in fewer iterations, especially for large-scale problems.  
Many scholars have devoted themselves to the research of heuristic algorithms for the rapidity of solution. To solve the two-stage unrelated parallel machine problem, [PERSON] et al. [22] proposed a heuristic method based on dynamic programming and [PERSON] et al. [23] put forward a heuristic method based on an improved Johnson rule, which effectively solves two-stage HFSP with irrelevant parallel computers. In multi-stage HFSP, due to the complexity of the problem, the study of heuristic methods is rare. [PERSON] et al. [24] proposed an improved fuzzy heuristic rule and studied the problem including fuzzy data. [PERSON] et al. [25] researched the multi-stage HFSP with multiprocessors applying heuristic algorithm.  
Recently, various intelligent algorithms have been being constantly proposed and effectively solve the HFSP, including the genetic algorithm (GA) [9], the simulated annealing algorithm [10], the tabu search algorithm [26], the ant colony optimization algorithm [27], the particle swarm optimization algorithm [28], the grey wolf optimization algorithm [29], the artificial bee colony algorithm [30], the artificial immune system (AIS) [31], the agent-based method [32], and so on. [PERSON] et al. [9] proposed a GA based on heuristic rules. To generate feasible scheduling, GA is used to allocate and sort machines in the first stage, while, in the second stage, FCFS is used to sort machines. [PERSON] et al. [31] established a mixed-integer nonlinear programming model of HFSP with the minimum of makespan as the objective function. [PERSON] et al. (2019) studied a multi-agent based hybrid flow shop dynamic scheduling system, in which the coordination mechanism between the various agents was designed.  
Overall, a number of algorithms have been proposed to address HFSP. To the best of the authors' knowledge, nevertheless, there is still no existing research about the RL method for HFSP.  
Being always the focus of academic research, RL has achieved remarkable development till now and the performance of RL algorithm has been continuously improved. [PERSON] et al. (2019) proposed soft actor-critic and an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework where the actor aims to maximize the expected reward while also maximizing entropy. Their method achieves state-of-the-art performance on a range of continuous control benchmark tasks. [PERSON] et al. (2019) studied how maximum entropy policies, which are trained using soft Q-learning, can be applied to real-world robotic manipulation. [PERSON] et al. (2019) proposed a unified RL algorithm, Normalized Actor-Critic, that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. [PERSON] et al. (2019) explored algorithms and representations to reduce the sample complexity of deep reinforcement learning for continuous control tasks. Nevertheless, existing literature about the RL for HFSP is still nowhere to be found.  
The HFSP is researched in this paper and attributed to MDP for which the special states, actions and reward function are designed, on whose basis the Q-learning method is adopted to find the optimal policy.