# Multi-Objective Optimization of Cascade Blade Profile Based on Reinforcement Learning
## 5 DDPG-Based Optimization for Blade Profile  
### Deep Deterministic Policy Gradient (DDPG)  
In this paper, DDPG [32] is used as the intelligent designer to find the optimal policy for the multi-objective design of cascade blade. DDPG is one reinforcement learning algorithm which can choose the appropriate action according to specific state. Different from the value-based learning methods, e.g., deep Q-learning (DQN), which is restricted to discrete and low-dimensional state and action space, DDPG can handle the high-dimensional complicated action problems with the continuity of state and action space.  
DDPG is developed from DPG, by using deterministic policy instead of random policy [31]. Since the policy is deterministic, the output action is determined according to the input state and does not need to be sampled from a probability distribution. Therefore, the calculation of internal expectation for policies is avoided, and external expectation only needs to be calculated according to the environment. The computational resources are saved, and the variance becomes small, improving the convergence.  
DDPG is a policy gradient method with an actor-critic structure [59]. The actor network approximates to the deterministic policy function \(a=\mu(x)\), which learns a policy that yields the highest possible reward so that a definite action from the current state can be given. The critic network approximates the action-state value function \(Q^{\mu}(s,a)\) to criticize the given action, according to the action \(a_{t}\) in state \(s_{t}\) and following policy \(\mu\). With the network parameter \(\Theta^{Q}\), the critic network can be written as \(Q  
Drawing on the idea of DQN, DDPG has two frameworks of actor-critic structure: main framework and target framework. Each framework contains one actor network and one critic network. The main actor network gives the action \(a_{t}\). The main critic network outputs the value function \(Q(s_{t},a_{t})\) and updates the actor network. The action \(a_{t}\) is performed in the environment, to achieve the next moment state \(s_{t+1}\) and the calculated reward \(r_{t}\). The target framework stores the fixed target networks. The target actor network gives \(a_{t+1}\) according to the state \(s_{t+1}\) and the policy \(\mu^{\prime}(s_{t+1}|\Theta^{\mu^{\prime}})\). The target critic network outputs the value \(Q^{\prime}(s_{t+1},a_{t+1})\). When the target networks are updated, soft update is used: \(\Theta^{Q^{\prime}}\leftarrow\tau\Theta^{Q}+(1-\tau)\Theta^{Q^{\prime}},\Theta ^{\mu^{\prime}}\leftarrow\tau\Theta^{\mu}+(1-\tau)\Theta^{\mu^{\prime}}\), where \(\tau\) is the soft-update parameter. Therefore, the target network parameters can be updated slowly, with improved learning stability. The main critic network is updated to minimize the loss function:  
\[L=E[(r_{t}+\gamma Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t}))^{2}]. \tag{13}\]  
DDPG uses an experience replay memory to store transitions \((s_{t},a_{t},r_{t},s_{t+1})\) from interacting with the environment. The experience transitions are stored in the memory and then fed to update the networks in batches with a random sample strategy. Random sampling can eliminate the correlation between transitions, improving the stability of network training. The structure of DDPG is shown in Figure 13.  
Based on DPG, DDPG uses deep neural networks. A multilayer feed-forward network with enough neurons in hidden layers can approximate any continuous function with arbitrary accuracy [60]. With the deeper network structure, the actor and the critic network can approximate the policy and the value function more accurately. The deeper network structure also increases the dimension of action that can be handled. But, for problems with complex action space, increasing the number of hidden layers and the number of nodes of each layer may increases the training time of the network, and even lead to the tendency of over-fitting.  
Figure 13: Deep Deterministic Policy Gradient (DDPG) algorithm structure.