Achieving Safe Deep Reinforcement Learning via Environment Comprehension Mechanism

Chinese Journal of ElectronicsVolume 30, Issue 6 p. 1049-1058 Original Research PaperFree Access Achieving Safe Deep Reinforcement Learning via Environment Comprehension Mechanism PENG Pai, Corresponding Author PENG Pai 20185227062@stu.suda.edu.cn School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this authorZHU Fei, Corresponding Author ZHU Fei zhufei@suda.edu.cn School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this authorLIU Quan, Corresponding Author LIU Quan quanliu@suda.edu.cn School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this authorZHAO Peiyao, Corresponding Author ZHAO Peiyao 1097988869@qq.com School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this authorWU Wen, Corresponding Author WU Wen 717955768@qq.com School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this author PENG Pai, Corresponding Author PENG Pai 20185227062@stu.suda.edu.cn School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this authorZHU Fei, Corresponding Author ZHU Fei zhufei@suda.edu.cn School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this authorLIU Quan, Corresponding Author LIU Quan quanliu@suda.edu.cn School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this authorZHAO Peiyao, Corresponding Author ZHAO Peiyao 1097988869@qq.com School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this authorWU Wen, Corresponding Author WU Wen 717955768@qq.com School of Computer Science and Technology, Soochow University, Suzhou, 215006 ChinaSearch for more papers by this author First published: 01 November 2021 https://doi.org/10.1049/cje.2021.07.025 This work is supported by the National Natural Science Foundation of China (No.61303108), Suzhou Key Industries Technological Innovation-Prospective Applied Research Project (No.SYG201804), and a project funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions. AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Deep reinforcement learning (DRL), which combines deep learning with reinforcement learning, has achieved great success recently. In some cases, however, during the learning process agents may reach states that are worthless and dangerous where the task fails. To address the problem, we propose an algorithm, referred as Environment comprehension mechanism (ECM) for deep reinforcement learning to attain safer decisions. ECM perceives hidden dangerous situations by analyzing object and comprehending the environment, such that the agent bypasses inappropriate actions systematically by setting up constraints dynamically according to states. ECM, which calculates the gradient of the states in Markov tuple, sets up boundary conditions and generates a rule to control the direction of the agent to skip unsafe states. ECM is able to be applied to basic deep reinforcement learning algorithms to guide the selection of actions. The experiment results show that the algorithm promoted safety and stability of the control tasks. I. Introduction As a category of machine learning, reinforcement learning aims to attain an optimal policy by maximizing long-term reward when interacting continuously with the environment[1]. Reinforcement learning has been widely studied and applied in many fields, such as healthcare[2], automation control[3], and gaming[4]. Deep learning[5] incorporates the idea of extracting higher level features from raw input by multi-layer networks and nonlinear transformations[6], which has achieved remarkable success in many fields, such as computer vision[7] and speech recognition[8]. Deep reinforcement learning, which combines deep learning with reinforcement learning, uses neural networks to extract features from raw visual data, and takes advantage of reinforcement learning to directly learn optimal control strategies. Ensuring safe decisions made by the agent is always a challenging goal when artificial intelligence algorithms, e . g . deep reinforcement learning, is applied in practical problems. Most of existing reinforcement learning algorithms are bothered with safety problems, even for high performance algorithms[9] and rainbow[10]. Agent interacts with the environment and obtains the feedback, in the form of a reward to seek an optimal strategy and selects the action with the highest reward. In order to obtain a better solution space, it introduces exploration random strategy, such as ε-greedy. Although it extends solution space, random exploration behaviors are risky[11], in that they are likely to result in unexpected catastrophic consequences. For instance, when using reinforcement learning to allocate network resources, taking random actions may overflow the buffer. Although reinforcement learning emphasizes interactions with the environment, the only feedback from the environment is a reward item in the form of a numeric value. In fact, it can be argued that complex environments should be described from a multi-dimensional, multi-angle and dynamic perspective[12], while a reward denoting by a single value may let slip valuable information. Therefore, traditional reinforcement learning should be improved with an environment comprehension mechanism by setting up rules concerning safety, as to enhance the semantics of environmental feedback. Hence, to solve the problem, a general mechanism for the environment comprehension, referred as Environment comprehension mechanism (ECM), is proposed in this paper, which analyzes the environment and establishes constraints for the agent to prevent it from choosing inappropriate actions. ECM can be integrated with many basic deep reinforcement learning algorithms to help them to take safer decisions, and improve the stability of the performance as well by establishing rules for the agent to decide whether a certain state is safe or not such that the agent could avoid dangerous actions. II. Preliminaries 1. Reinforcement learning Reinforcement learning belongs to machine learning that maps the given states of the environment to actions, with the goal of maximizing long-term cumulative reward. The Markov decision process (MDP)[13] is often used in reinforcement learning modeled by defining a four tuple < S , A , P , R > , where S is the set of all environmental states, and s t ∈ S denotes the state of the agent at time t; A is the set of all executable actions of the agent, and a t ∈ A denotes actions taken by the agent at time t; P is the transition probability representing the likelihood of the agent taking action from the state st at time t to move to the next state s t + 1 , and is usually formalized as P ( s t + 1 | s t , a t ) ; R is the immediate reward, which is given by the environment after the agent moves from the state st to the next state s t + 1 , usually formalized as R ( s t , a t , s t + 1 ) . The policy π maps from current state s t to the action a t , i . e . , a t = π ( s t ) . The overall reward from time t to T[14] is defined as R t = ∑ t ' = t T γ t ' - t r t ' (1)where γ ∈ [ 0 , 1 ] is the discount rate to weigh the influence of future rewards on cumulative rewards. The state action value function Q ( s , a ) is usually defined as the cumulative return for executing action a under given state s t using policy π until the end of the iteration, defined as Q π ( s , a ) = E R t | s t = s , a t = a ; π (2)where E denotes expectation. For all states, if the expected return of a policy is greater than those of all other policies, that policy π * , is called an optimal policy Q π * ( s , a ) = max π E R t | s t = s , a t = a ; π (3) Eq.(3) is called the optimal state-action value function, which has been proved to follow the optimal Bellman equation[15]. More generally, the optimal strategy is to select the action that obtains the greatest expected return r + γ Q * ( s ' , a ' ) . Q π * ( s , a ) = E [ r + γ max a ' Q ( s ' , a ' ) | s , a ] (4) 2. Deep reinforcement learning Deep reinforcement learning is a combination of reinforcement learning and deep learning. The most classic model of deep reinforcement learning is Deep Q-network (DQN) which combines the traditional Q-learning algorithm with Convolutional neural network (CNN). The standard DQN uses two CNN models to approximate the given value function and the target value function of the state-action pair, where Q ( s , a | θ i ) represents the output of the given value network[16], which is used to calculate the value function and Q ( s , a | θ i - ) represents the output of the target value network. In general, the optimal value is approximated by % Y i = r + γ max a ' Q ( s ' , a ' | θ i - ) .% Y i = r + γ max a ' Q ( s ' , a ' | θ i - ) (5) The weights of the network are updated by using the mean-squared error between the target function and the given value function. The loss function is L i ( θ i ) = E [ ( Y i - Q ( s , a | θ i ) ) 2 ] (6) The DQN takes advantage of the experience replay mechanism[17] to process the transferred samples during the training process. At time t, the sample obtained by the interaction between the agent and the environment is stored in the experience replay memory pool, and the Stochastic gradient descent (SGD)[18] is applied to update the parameters of networks as ∇ θ L i ( θ i ) = ( Y i - Q ( s , a | θ i ) ) ∇ Q ( s , a | θ i ) (7) As a famous deep reinforcement learning approach, Dueling deep Q-Network (DuDQN)[19] uses CNN to extract features from the original input, which is divided into advantage function and value function to generate action-value function. In many DRL tasks based on visual perception, the value functions of different state action pairs are different, but in some states, the value function is independent of the action. DuDQN is particularly outstanding in this kind of tasks. Its competitive structure can learn the values of environmental states without the influence of actions. It can ensure that the relative order of the dominance function of each action in this state remains unchanged, and can narrow the range of Q value. Therefore, the redundant degrees of freedom can be removed and the stability of the algorithm can be improved. 3. Safe reinforcement learning The notion of safety is additionally considered when evaluating decisions made by agents in reinforcement learning. Unrestricted exploratory behaviors may cause security problems that trigger permanent damages. Therefore, it is necessary to ensure safety of the agent even at the cost of a lower convergence rate. We propose a safe reinforcement learning based on environment comprehension mechanism which sets a safety boundary around the agent where comprehension of the environment is realized by detecting objects in the system which improves the agent's ability to maintain a safe state. Continuous trial-and-error and exploration behaviors in model-free reinforcement learning may lead to some irreversible consequences which can be greatly reduced by adding some human interventions. Delage et al.[18] proposed an idea whereby the agent receives observations from other than explorations to reduce uncertainty about the parameters. However, the method is not well suited with safe exploration as it cannot guarantee safety for similar criteria and variance in the expected value. Hans et al.[20] presented a safety function learned during exploration by collecting samples of the minimum reward obtained for executing state-action pairs. Their work showed that being close to a known safe space is always better than not. However, some route planning is needed to ensure that the agent continues to explore. Many researchers have carried out study on safe reinforcement learning. Heger et al.[21] took advantage of the worst-case criterion used in control theory domain to propose a variant of Q-learning, which calculated the maximization of the minimum possible rewards rather than the maximization of long-term rewards. By their work, it is feasible to find policies that minimize the worst-case total discounted cost. Zhu et al.[22] proposed Dual Deep Network (DDN) to use the original network objective function and penalty term to calculate the objective function, which improved the security of the agent. Berkenkamp et al.[23] extended control-theoretic results on Lyapunov stability verification and used statistical models of the dynamics to obtain high-performance control policies with provable stability certificates. Saunders et al.[24] formalized human intervention for reinforcement learning and reduced the human labor required by training a supervised learner to imitate the human's intervention decisions. Achiam et al.[25] introduced constrained policy optimization for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Geibel et al.[26] turned reinforcement learning problems into finding good policies whose risk is smaller than some user-specified thresholds. Inspired by previous works, we formalize the risk as the degree of danger of actions performed by agents. III. Algorithm Description 1. Neural network structure The model of neural network used in this paper applies three CNN layers. The CNN uses rectifier nonlinearity for each hidden layer and extracts features[27] from the image. The first hidden layer convolves 32 filters ( 8 × 8 ) with a stride of 4; The second hidden layer convolves 64 filters ( 4 × 4 ) with a stride of 2; The third hidden layer convolves 64 filters ( 3 × 3 ) with stride of 1. Two fully connected layers are used to complete the mapping from state to action. The output layer is a fully connected layer with a single output for each action. The illustration of the architecture of network is shown in Fig.1: Fig. 1Open in figure viewerPowerPoint The illustration of the architecture of network 2. Environment comprehension mechanism In this work, safe deep reinforcement learning is based on environment comprehension mechanism. For most of the state-of-the-art deep reinforcement learning methods, Red-green-blue (RGB) images are usually used as input states. The environment comprehension mechanism refers to analyzing the environment and setting up relevant constraints for safety of the agent according to states. In order to better comprehend the environment security boundaries of agents should be detected. Since the difference between choosing grayscaling image and choosing RGB image as input has little influence on network training results, the computational cost generated by gray scale image will be greatly reduced. Therefore, our algorithm first converts RGB image into grayscaling image. A RGB image is transformed into a gray scale image[28] by G r a y = R × 0 . 299 + G × 0 . 587 + B × 0 . 114 (8)where R , G and B denote the gray value of red, green and blue channels respectively. The image gradient makes great contributions to environment comprehension. In order to obtain a smoother and wider boundary, the gray level transformation is carried out by Gaussian filtering. The security boundary is defined as the agent staying in a security state within the safety boundary. For a point ( u , v ) , its gray value is f ( u , v ) . The conversion equation[29] is g σ ( u , v ) = 1 2 π σ 2 e - u 2 + v 2 2 σ 2 · f ( u , v ) (9)where σ denotes the Gaussian standard deviation of point. By calculating the gradient of each point, the maximum gradient is used as a candidate for the safety boundary with sobel operator. The gradient equation[30] of the point ( u , v ) is G ( u , v ) = g x ( u , v ) 2 + g y ( u , v ) 2 (10) After obtaining the basic security edge, we expand the safety boundary. For example, in the experiment, we expand the security boundary by 150%, so that the agent acts in a larger range. After processing, the safety boundary of all objects will be highlighted. The gradient direction is determined by the safety boundary, presented as λ = arctan g y ( u , v ) g x ( u , v ) (11) 2. Safe deep reinforcement learning with ECM In the process of exploring the environment, the agent may get great negative feedback after taking some actions due to encountering obstacles, which we define as negative reward objects. We define the risk degree of encountering negative reward when taking action a in state s as D a | s , which is determined by the difference between the direction of the action and the safe edge gradient. The smaller the D a | s is, the more dangerous the action is. D a | s depends on the difference between the gradient direction of the boundary λ and the direction of executable action λ a . Through D a | s , the agent judges if the safety of the selected action is safe and decides whether to take different strategies. We define the approach as environmental comprehension mechanism by which the MDP model is constrained. Under the constraint of the minimum risk of each action state pair, the goal is to maximize the cumulative reward. More details can be found in Algorithm 1. By the comprehension of environment, we establish safety boundaries for agents and evaluate the danger degree of an action at each state. Constraints are then added to the process of selecting actions. In turn, a safe strategy can be obtained through equipping deep reinforcement learning with the environment comprehension mechanism that enables detecting dangerous states. Safe deep reinforcement learning based on environment comprehension mechanism is shown in Algorithm 2. Line 1 of Algorithm 2 initializes the replay memory D with capacity N to store the training samples. Line 3 initializes the Q-convolution neural network of action-value function and randomly initializes the weight parameter θ. When initializing the convolution neural network of target action-value function, the structure and initialization weight θ are the same as that of Q. Line 9 to line 13 chooses the appropriate action from Algorithm 1. Line 15 gets post-execution rewards and input for the next network. Line 18 stores the four parameters as a state at the moment into the replay pool. Line 19 randomly retrieves the minibatch state from the replay pool to calculate the target value of each state and update the Q value as the target value by performing the reward. Line 21 update the weight θ through stochastic gradient descent. The parameter of the target action-value function network is replaced with the parameter of the current action-value function after a few times. In Algorithm 2, the fixed number of step μ, which means the requirement of influence degree that is used to ensure the safe exploration, determines how often the danger is checked. With fix number of step, it is able to avoid frequently invoking ECM functions that may cause excessive computational cost. By using the probability of different action directions in Algorithm 1 relative to whether the agent is safe in line 10 of Algotithm 2, we made corresponding changes to the action selection probability through environment comprehension. IV. Experiment and Analysis In this section, to evaluate the performance of SDRL-ECM, we carried out four experiments of Atari 2600 tasks to compare the proposed algorithm with DQN and DuDQN. Among the four games, three are shooting games, including Assault, Demon Attack and Space Invaders, and one is a difficult strategy game called Seaquest. The proposed algorithm is then proven to have certain applicability, which can be used not only for shooting games, but also for strategy games. 1. Settings of experiment The experiment environment used in this paper is Atari 2600 based on the Gym toolkit developed by OpenAI, which provides many experimental platforms for researchers[31]. The CPU used is Intel i7-8700k, with 16GB of primary memory and the GTX1080Ti graphics processor for acceleration. In addition, some common techniques are used to improve the stability of the model in the training process. First, due to the difference of the score intervals of different tasks, all reward values greater than 1 are set to 1, reward values less than –1 are set to –1, and the rest remain unchanged to narrow the range of values of Q. Second, the agent selects the next action at certain points according to the Q value of the network's output, which might take longer time than needed for the forward propagation of the network. Frame skipping is thus used to ease the difference in computation speed between the processes in the experiment. The selects chose the subsequent action based on the ε-greedy policy every four frames, and only repeated the action at the given time before choosing the next one. Third, the mean square error calculated by Eq.(5) is limited to [ - 1 , 1 ] because cutting the Q value and mean square error to specific intervals is conducive for providing the same learning rate for different tasks, avoiding falling into a local optimal solution caused by the large fluctuation in strategy, as well as improving the stability of training. In our experience, stochastic gradient descent based on root mean square[32] is used to update the parameters, the momentum coefficient is set to 0.95 and the discount factor is set to 0.99. The learning rate drops from 0.005 to 0.00025, and ε-greedy strategy is dynamically assigned values as factor ε decreasing from 1.0 to 0.1. It should be noted that there are not enough transfer samples in sample pool at the beginning stage of training. In the experiment, the agent used a random strategy to store enough transfer samples in sample pool which can prevent deviations in learning. All models carried out 2000 iterations, with 5000 steps in each training iteration in the experiment. In order to compare the reward value more intuitively, we normalize the reward of different games by r ' = ( r - r min ) / ( r max - r min ) (12) 2. Assault In Assault, the agent needs to keep moving to avoid enemy bullets and shoot other enemies to obtain higher scores. We obtain the location of the agent by object detection and perceived bullets around itself through our environment comprehension algorithm, as in Fig.2. Fig. 2Open in figure viewerPowerPoint The illustration of Assault in SDRL-ECM where a square is displayed around the agent in the image After a certain number of steps, the degree of danger in the agent's state is checked. If there are more than one bullet, the one closest to the agent is considered the most dangerous and the probability of different actions is determined by the strategy obtained by ECM algorithm. Otherwise, if the bullet is near the security line, the relevant state is referred as critical. We separate the different states based on the position relationship between danger (bullets) and attention area around the agent and chose the appropriate actions according to the state instead of using the neural network. For example, as shown in the image on the left in Fig.3, there are no bullets or enemies within the defined safe area. The state is regard as safe, in which case we select a random action with probability ε otherwise for an unsafe state, we select an action that follows a t = arg max a Q ( s t , a ; θ ) . While in the right part of Fig.3, there are two bullets. The upper-right one is of no threat and the lower-left one is dangerous. The agent thus judges the state to be dangerous. The action, moving right to be far away from the dangerous bullet, is less harmful according to the relative position of the bullet and agent and thus is regarded as an optimal. Fig. 3Open in figure viewerPowerPoint The illustration of environment comprehension mechanism in the ECM where a square displayed around the agent in the image In general, the environment comprehension mechanism can locate the agent and enemies around it to judge the safety of the given states and can take suitable actions to improve the performance of the agent on such strategic tasks. Since the goal of the reinforcement learning task is to obtain the highest reward possible, it is reasonable for us to use the cumulative reward as a comparison between different algorithms. The average values of cumulative rewards in each iteration of the DQN, DuDQN, and ECM-DQN are shown in Fig.4 where the effect of SDRL-ECM is the best among three methods. The DQN can perceive only four continuous image frames nearest to the given state at each time, the value function estimated by the DuDQN is more precise than that by the DQN. As a result, the cumulative reward of the DuDQN is larger than that of DQN. Take the SDRL-ECM into consideration, it begins to explore at a good starting position. SDRL-ECM can achieve a better result in a short time and tends to be stable which can be seen from Fig.4. However, DQN and DuDQN do not take these into account, resulting larger fluctuation. In the task, the agent has seven available actions, which includes four flight directions, firing bullets and avoiding objects. The plane must move around to avoid attacks from enemies. Most actions influence only one-time step ahead. For example, by moving leftward in the next four continuous image frames, the agent avoids most bullets in the next few steps. Positional information of the plane is critical in this situation. DQN and DuDQN are incapable of choosing appropriate actions based on the past four images more quickly and accurately than SDRL-ECM. The model of SDRL-ECM can detect the plane controlled by the agent. We only need to provide a safe direction once in a limited number of steps. Fig. 4Open in figure viewerPowerPoint Average cumulative reward per iteration with different methods compared in Assault Meanwhile, the average reward for each iteration in the training process of the SDRL-ECM still fluctuates slightly because the parameters of the network are constantly updated during the training of the model, which leads to significant changes in the distribution of the policy in next period. To further illustrate the applicability of the SDRL-ECM model to other kinds of strategic tasks, it is also assessed on Demon Attack, Seaquest, and Space Invaders. The average cumulative rewards per iteration obtained by different methods on the four tasks are discussed in the next section. 3. Demon attack In the task of Demon Attack, the aircraft must keep moving to avoid enemy bullets. We obtain the location of the aircraft and the bullets are preceived around the agent through environment comprehension, as shown in Fig.5. Although Demon Attack is a more complex task than Assault, bullets can still be identified by ECM. One can therefore judge states using several frames in succession. However, we need to set the parameters more carefully to perceive the position relationship between the aircraft and the bullets, regardless of the velocity of the latter. Fig. 5Open in figure viewerPowerPoint The illustration of Demon Attack in the SDRL-ECM where a square displayed around the aircraft in the image Fig. 6Open in figure viewerPowerPoint The average cumulative reward per iteration with different methods compared in Demon Attack The average cumulative rewards in each iteration of the DQN, DuDQN, and SDRL-ECM are shown in Fig.6. As indicated in Fig.6, the training effect of SDRL-ECM is the best of the three methods. The cumulative rewards of DQN and DuDQN are similar and are lower than that of the SDRL-ECM. As the complexity of the task increases, so does the difficulty of the environment comprehension. Moreover, the growth rate at the beginning stage of the SDRL-ECM is also larger than those of DQN and DuDQN which means that ECM algorithm makes it faster for the agent to learn through experience in the beginning by the environment comprehension. The Fig.6 shows that ECM grows faster than all other algorithms. For models that are increasingly effective, we need to make more exploitation and less exploration. As a result, the advantage of SDRL-ECM is not as obvious as in Demon Attack. 4. Seaquest In the task of Seaquest, a submarine is supposed to avoid sea creatures and meanwhile attends to the remaining amount of oxygen and needs to reach the surface of the sea to supply oxygen. We visualize this task through visual tasks in the process, as shown in Fig.7. Unlike other shooting tasks, Seaquest can be classified as a tactical task. Compared with tasks of the shooting class, Seaquest needs to perform more actions (18 available actions). Unsafe stat