# Multi-Objective Optimization of Cascade Blade Profile Based on Reinforcement Learning
Recently, Google Deepmind developed Deep Reinforcement Learning (DRL). Deep Q-learning (DQN) [30] is one such example in that it is the improvement of Q-learning with three innovations: (1) DQN uses deep neural networks to approximate the value function; (2) DQN uses a replay buffer to reduce the correlation between samples; and (3) DQN sets up an independent target network to handle the error in the temporal difference. However, DQN is restricted to cases of the discrete and low-dimensional state and action. For high-dimensional continuous action problems, Deterministic Policy Gradient (DPG) was proposed. Different from the value-based learning in DQN, DPG learning is policy-based in that directly optimizes deterministic policies in form of parameters of the neural networks via gradient-based methods [31]. Combined with Deep Learning, Deep Deterministic Policy Gradient (DDPG) can thus be utilized in cases of more complicated action [32]. In its applications on aerodynamic optimizations, [PERSON] et al. [33] proposed a DDPG-based aerodynamic shape optimization. They optimized a preliminary missile shape design involving lift coefficient and drag coefficient, where the DDPG network was applied to guide the shape parameters variation by interacting with the environment based on CFD and semi-empirical method DATCOM. In this paper, DDPG is applied in a multi-objective optimization of compressor cascade blade profile. The geometric parameters of blade profile are taken as State, and the modification of parameters are taken as Action. From interaction with the simulation-based environment, the aerodynamic parameters corresponding to a certain State (profile geometry) can be achieved. The DDPG network can learn the design experience of cascade blade with the feedback according to the aerodynamic parameters, which can guide the modification of the profile optimization. The existing computational data is now utilized to build the policy to guide the optimization direction in the design space, which saves many calculations. With the learnt policy, the DDPG network can optimize the blade profile as an experienced designer to achieve the good-performance profile. In order to reduce the computation and shorten the optimization period, an ANN (artificial neural network)-based surrogate model is used in this paper to approximate high-fidelity CFD simulations [34]. The model is trained with profile geometric parameters and aerodynamic parameters by numerical calculation. The surrogate model is utilized to feed back the aerodynamic information to the DDPG network as the environment in the optimization. To achieve the multiple objectives interconnection, as well as an appropriate objective function, analysis on the objective space, composed of optimization aerodynamic parameters, is done with Principal Component Analysis (PCA) [35]. The flowchart of the optimization is illustrated in Figure 1.  
Figure 1: Flowchart of multi-objective optimization of cascade blade profile. This paper is organized as follows: In Section 2, the parameterization of the blade profile is introduced. In Section 3, the cascade flow field calculation is validated. Section 4 introduces the surrogate model based on ANN. Section 5 introduces the DDPG algorithm, and the optimization application is presented. Section 6 shows the optimization results with some discussions, and the paper is concluded in Section 7.