SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After training, we can extract  $|S_{\phi}|$  LCUs based on the learned coefficients of linear control policies for the abstract states in  $S_{\phi}$ . Specifically, we choose an arbitrary actual state  $s \in \phi^{-1}(s_{\phi})$  for each abstract state  $s_{\phi}$  and feed it to a PLDNN to obtain the coefficients defined on the abstract state. For instance, we can feed  $(-1, -1)$  to the PLDNN in Example 1 and obtain the coefficients  $(-0.16610657, -1.7437580, -1.8227874)$  of the linear control policy for the region  $[-2,0) \times [-2,0)$ . Figure 3 shows the LCUs extracted from a trained PLDNN in Example 1. They are depicted by planes with different colors in Figure 3. These four planes denote the following linear control functions:
$$
\pi \left(x _ {1}, x _ {2}\right) = - 0. 1 6 6 1 0 6 5 7 - 1. 7 4 3 7 5 8 0 x _ {1} - 1. 8 2 2 7 8 7 4 x _ {2}, \quad x _ {1} \in [ - 2, 0), \quad x _ {2} \in [ - 2, 0) (\mathrm {L C U} _ {1})
$$
$$
\pi \left(x _ {1}, x _ {2}\right) = - 0. 2 0 4 0 0 0 3 5 - 1. 8 0 0 6 0 3 7 x _ {1} - 1. 8 6 7 9 8 8 5 x _ {2}, \quad x _ {1} \in [ - 2, 0), \quad x _ {2} \in [ 0, 2) \quad \left(\mathrm {L C U} _ {2}\right)
$$
$$
\pi \left(x _ {1}, x _ {2}\right) = - 0. 2 7 5 4 7 9 3 0 - 1. 8 8 8 4 7 2 2 x _ {1} - 1. 9 3 4 2 2 6 8 x _ {2}, \quad x _ {1} \in [ 0, 2), \quad x _ {2} \in [ - 2, 0) (\mathrm {L C U} _ {3})
$$
$$
\pi \left(x _ {1}, x _ {2}\right) = - 0. 2 9 5 4 9 8 9 7 - 1. 9 0 2 2 3 3 8 x _ {1} - 1. 9 4 3 6 3 4 6 x _ {2}, \quad x _ {1} \in [ 0, 2), \quad x _ {2} \in [ 0, 2) \quad \left(\mathrm {L C U} _ {4}\right)
$$
The underlying  $x_{1} \times x_{2}$  plane in Figure 3 is the projection of the four LCUs. Under the control of these four LCUs, the agent can reach the goal region (orange box). The two sequences of purple boxes represent the range of reachable states from corresponding initial regions to the goal region.
With exacted LCUs from a PLDNN, we can build a verifiable hybrid automaton for the system by substituting equivalently the neural networks using corresponding LCUs. Theorem 2 formulates the hybrid automaton after a decision network  $\pi$  is substituted by LCUs. The differences from Theorem 1 include the definitions of transitions  $T$ , guard condition  $G$  and reset formula  $R$ . For the PLDNN controlled systems, We use  $|S_{\phi}|$  transitions each of which contains a guard condition and a reset formula to update the action  $a$  to  $\pi_c(s)\cdot [1,s]$  at each  $\delta$ . Notice that,  $\pi_c(s)$  is a determined vector since  $\forall s\in \phi^{-1}(s_i^i)$ , the outputs of  $\pi_c$  are the same according to the dedicated structure of  $\pi_c$ . Thus, the reset formula for  $a$  is simplified to an affine mapping.
Figure 3: The LCUs extracted from a trained PLDNN for Example 1.
Theorem 2. Given a DRL system with environment dynamics  $f$ , PLDNN  $\pi$ , time step size  $\delta$  and initial state set  $S_0$ , it can be equivalently modeled as a hybrid automaton defined as follows:
- Var: state variable  $s$ , action  $a$ , clock variable  $t_c$  -  $I_0$ :  $\{(l_0, (s \in S_0, a = 0, t_c = \delta))\}$
-  $L \colon  \left\{  {l}_{0}\right\}$  -  $\bullet  \operatorname{Inv} : \operatorname{Inv}\left( {l}_{0}\right)  = \left\{  {s \in  S,{t}_{c} \leq  \delta }\right\}$ $\bullet  F$  :  $F\left( {l}_{0}\right)  = \left\{  {\dot{s} = f\left( {s,a}\right) ,\dot{a} = 0,{\dot{t}}_{c} = 1}\right\}$
-  $T\colon \{(l_0,l_0),\dots ,(l_0,l_0)\}$  where  $|T| = |S_{\phi}|$
-  $G\colon G(T[i]) = \{t_c = \delta, s \in \phi^{-1}(s_\phi^i)\}$  where  $0 \leq i < |T| \land s_\phi^i \in S_\phi$
-  $R\colon R(T[i]) = \{t_c = 0,a = \pi_c(s)\cdot [1,s]\}$  where  $0\leq i < |T|\land s\in \phi^{-1}(s_{\phi}^{i})$
Proof. We prove that the path of hybrid automaton  $H_{1}$  in Theorem 1 is the same as the corresponding hybrid automaton  $H_{2}$  in Theorem 2 for an arbitrary initial state  $(l_0, [s_0, 0, \delta])$ . The path of  $H_{2}$  can be described by the following sequence:
where  $s_{i+1} = s_i + \int_0^\delta f(s, a_i) dt$  and  $a_i = \pi_c(s_i) \cdot [1, s_i]$ . With the dedicated structure of PLDNN  $\pi$ , we have  $\pi(s) = \pi_c(s) \cdot [1, s]$ . Thus, at each  $\delta$ , the discrete transitions of  $H_1$  and  $H_2$  change both location
and continuous variables in the same way. We can conclude that the paths of  $H_{1}$  and  $H_{2}$  are exactly the same. By Theorem 1, it is obvious that  $\mathcal{D}$ ,  $H_{1}$  and  $H_{2}$  produce the same state-action value for the same initial state.
According to Theorem 2, we can build a hybrid automaton that is equivalent to the DRL system in Example 1. Assuming the trained system has four linear control units as shown in Formulas (LCU $_1$ -LCU $_4$ ) and  $\delta = 0.2$ , we construct the corresponding hybrid automaton as depicted in Figure 4. The four transitions in the automaton correspond to the four LCUs, respectively. The guard of each transition represents the condition of triggering the corresponding policy.
Thanks to the linearity of control policies in  $R$ , a hybrid automaton built for a PLDNN-controlled system can be efficiently verified by state-of-the-art tools. For instance, Flow* [24] is a representative tool for the reachability analysis of hybrid systems. In this paper we are focused on the verification of both goal-reach and reach-avoid properties. The former means that given a set of initial states, a system must eventually reach the goal region from any initial state. The latter means that the system never enters unsafe regions within a specific time horizon. Both properties can be verified via reachability analysis.
Figure 4: The hybrid automaton of the DRL system with the trained piece-wise linear controllers in Figure 3.
5 Experimental Evaluations
We prototype our approach into a tool called LINCON, with DDPG as the backend DRL algorithm and Flow* as the verification engine. We extensively assess it, along with the state-of-the-art tools. Our goal is to show, for the PLDNN-based training, (i) the reduction in the number of partitions with comparable cumulative rewards, robustness, and time overhead with respect to conventional DNN-based training; (ii) its high verification performance including the tightness of over-approximation sets and the efficiency of verification; and (iii) its scalability for large-sized neural networks and systems with complex dynamics and high-dimensional state space.
Experimental Setup. All experiments were conducted on a workstation equipped with a 32-core AMD Ryzen Threadripper CPU @ 3.6GHz and 256GB RAM, running Ubuntu 22.04.
**Benchmarks.** We choose eight benchmarks, including six regular benchmarks from Verisig 2.0 [9] (B1-B5 and Tora) and two complex benchmarks (CartPole with extreme complex dynamics from OpenAI Gym [41] and quadrotor (QUAD) with 12-dimensional state space and 3-dimensional action space from [10]). For fair comparisons, we use the same training configuration and guarantee that all trained systems reach the specified reward threshold. See Appendix A.1 for the detailed setting. For B1-B5 and Tora the environment dynamics are the same with [9]. The dynamics of QUAD are the same with [10]. We omit their detailed definition here. However, the dynamics of CartPole are originally represented as a discrete-time model in [41] by difference equations. In our experiment, we consider its dynamics as a continuous-time model and formalize it with the following ODEs:
$$
\begin{array}{l} \dot {x _ {1}} = x _ {2}, \quad \dot {x _ {3}} = x _ {4}, \\ \dot {x _ {2}} = (a + 0. 0 5 x _ {4} ^ {2} \sin (x _ {3})) / 1. 1 - (0. 0 5 ((9. 8 \sin (x _ {3}) - \cos (x _ {3})) ((a + 0. 0 5 x _ {4} ^ {2} \sin (x _ {3})) / 1. 1)) / \\ (0. 5 (4. 0 / 3. 0 - (0. 1 \cos^ {2} (x _ {3}) / 1. 1))) \cos (x _ {3})) / 1. 1, \\ \dot {x _ {4}} = 9. 8 \sin (x _ {3}) - \cos (x _ {3}) ((a + 0. 0 5 x _ {4} ^ {2} \sin (x _ {3})) / 1. 1) / 0. 5 (4. 0 / 3. 0 - (0. 1 \cos^ {2} (x _ {3}) / 1. 1)). \\ \end{array}
$$
5.1 Performance Evaluation
We assess the performance of PLDNN, together with the conventional DNNs, in terms of cumulative reward, robustness, and training time under the same training configuration. We also measure the number of abstracted states required for training linear control policies and constant policies [21]. Due to space limitations, we present the experimental results only for B1, B2, and two complex cases
Figure 5: Performance and robustness comparison between PLDNNs and DNNs. The number in the parentheses is the base of  $\sigma$ , e.g., when the abscissa is 50 in B1, we have  $\sigma = 50 \times 0.0004 = 0.02$ .
(i.e., CartPole and QUAD). The associated conclusions (from Table 1 and Figure 5) also apply to the other four cases; see Appendix A.2 for the detailed experimental results.
Cumulative Reward. Figure 5 plots the system cumulative reward (the average of five trials) during the training process. The solid lines and shadows refer to the average reward and  $95\%$  confidence interval, respectively. Apparently, the trends of accumulative rewards by PLDNNs and DNNs
Table 1: Training time and number of partitions
Task	B1	B2	CartPole	QUAD
Training Time	PLDNN	14.3	7.9	428.2	871.1
DNN	11.0	6.6	403.6	781.5
#partitions	LCU	1	4	16	1
Const.	4	100	254	412
are comparable. Despite slightly more time required for training with PLDNN due to its additional conversion from actual state to abstract state (see Table 1, "Training Time"), we can, however, obtain a significant advance in the efficiency and tightness of verification (see Section 5.2).
Robustness. We also evaluate the robustness of PLDNNs as training the linear controller on each partition may lead to discontinuity of control decisions in the boundaries of partitions. For a current state  $s = (x_{1},\ldots ,x_{n})$ , we add a Gaussian noise  $X_{1},\ldots X_{n}$  to  $s$  and obtain a perturbed state  $s^{\prime} = (x_{1} + X_{1},\dots,x_{n} + X_{n})$  for calculating the control action, where  $X_{i}\sim \mathbf{N}(\mu_{i},\sigma_{i}^{2})$  with  $1\leq i\leq n$  and  $\mu_{i} = 0$ . For each benchmark, we train 10 different policies and evaluate their robustness under 100 different perturbation levels to obtain the average and  $95\%$  confidence interval of the cumulative reward. Figure 5 depicts the reward trend with the increasing perturbation level. As  $\sigma$  increases, the decline ratio of the system with PLDNNs is comparable to that with DNNs, which implies that both achieve similar robustness.
Reduction in the Number of Partitions. We measure the effect of reducing the number of partitions by utilizing a linear policy, instead of the constant action on each partition, which is adopted by the work [21]. In both cases, we start training from a coarse-grained abstraction granularity (with only one partition) and gradually increase the number of partitions until the preset reward threshold can be reached by the trained PLDNN. As shown in Table 1, linear policies significantly reduce the number of partitions required for reaching the reward threshold, which benefits both the verification efficiency and accuracy as we will see in the next section.
5.2 Verification Efficiency and Tightness
We evaluate the verification efficiency and tightness for the PLDNN-based and DNN-based DRL systems, respectively. Regarding tightness, we choose two state-of-the-art tools, i.e., Polar [10] and Verisig 2.0 [9], for the reachability analysis of DNN-controlled systems. We do not consider ReachNN* which both Polar and Verisig 2.0 have been demonstrated to outperform [9, 10]. For efficiency, we employ Flow* to perform reachability analysis, which is used by both Polar and Verisig 2.0 as the backend reachability analysis tool.
Efficiency. Table 2 presents the comparison results for the verification efficiency. For each regular case, we choose four different network configurations: two smaller networks (e.g.,  $\mathrm{Tanh}_{2\times 20}$ ) from [10] and two larger networks (e.g.,  $\mathrm{Tanh}_{3\times 100}$ ). Our approach LinCon can handle all 26 instances including the two complex instances, while Polar succeeds only in 20 cases. Verisig 2.0 is not applicable to
Table 2: Verification results and time in seconds.
Task	Dim	Network	LINCon	Polar	Verisig 2.0
1 Core	V.R.	1 Core	Impr.	V.R.	1 Core	Impr.	20 Cores	Impr.	V.R.
B1	2	Tanh2×20	2.31	✓	17	7.4×	✓	45	19.5×	38	16.5×	✓
Tanh3×100	2.28	✓	125	54.8×	✓	413	181.1×	123	53.9×	✓
ReLU2×20	2.11	✓	3	1.4×	✓	-	-	-	-	Xc
ReLU3×100	2.59	✓	-	-	Xb	-	-	-	-	Xc
B2	2	Tanh2×20	0.57	✓	5	8.8×	✓	5	8.8×	4	7.0×	Xa
Tanh3×100	0.56	✓	-	-	Xb	-	-	-	-	Xb
ReLU2×20	0.64	✓	3	4.7×	✓	-	-	-	-	Xc
ReLU3×100	0.60	✓	-	-	Xb	-	-	-	-	Xc
B3	2	Tanh2×20	2.69	✓	18	6.7×	✓	36	13.4×	28	10.4×	✓
Tanh3×100	3.57	✓	91	25.5×	✓	357	100.0×	88	24.6×	✓
ReLU2×20	3.05	✓	8	2.6×	✓	-	-	-	-	Xc
ReLU3×100	2.92	✓	14	4.8×	✓	-	-	-	-	Xc
B4	3	Tanh2×20	1.44	✓	5	3.5×	✓	7	4.9×	5	3.5×	✓
Tanh3×100	1.45	✓	27	18.6×	✓	114	78.6×	31	21.4×	✓
ReLU2×20	1.43	✓	2	1.4×	✓	-	-	-	-	Xc
ReLU3×100	1.43	✓	5	3.5×	✓	-	-	-	-	Xc
B5	3	Tanh3×100	3.24	✓	38	11.7×	✓	157	48.5×	44	13.4×	✓
Tanh4×200	3.29	✓	157	47.7×	✓	1443	438.6×	191	58.1×	✓
ReLU3×100	3.28	✓	7	2.1×	✓	-	-	-	-	Xc
ReLU4×200	3.29	✓	49	14.9×	✓	-	-	-	-	Xc
Tora	4	Tanh3×20	1.57	✓	45	28.7×	✓	69	43.9×	46	29.3×	✓
Tanh4×100	1.75	✓	-	-	Xb	-	-	-	-	Xb
ReLU3×20	1.58	✓	30	19.0×	✓	-	-	-	-	Xc
ReLU4×100	1.62	✓	53	32.7×	✓	-	-	-	-	Xc
CartPole	4	Tanh3×64	151	✓	-	-	Xb	-	-	-	-	Xb
QUAD	12	Tanh3×64	1054	✓	-	-	Xb	-	-	-	-	Xb
Remarks. Impr.: time speedup of LinCon compared to Verisig or Polar (Verisig or Polar/LinCon).  $\mathrm{Tanh} / \mathrm{ReLU}_{n\times k}$ : a DNN with the activation function  $\mathrm{Tanh} / \mathrm{ReLU}$ ,  $n$  hidden layers, and  $k$  neurons per hidden layer. VR: verification result.  $\checkmark$ : the reachability problem is successfully verified. Xtype: the reachability problem cannot be verified due to type: (a) large over-approximation error, (b) the calculation did not finish, (c) not applicable. —: no data available due to  $X^b$  or  $X^c$ .
ReLU networks (marked by  $\chi^c$ ) and succeeds only in 9 instances. Overall, LinCon outperforms both Polar (up to  $47.7 \times$  speedup) and Verisig 2.0 (up to  $438.6 \times$  speedup). In particular, LinCon achieves even up to  $58.1 \times$  speedup compared to Verisig 2.0 accelerated by 20-core parallelization. For our approach LinCon, the only time overhead for encoding networks stems from extracting LCUs from PLDNNs, which is polynomial and negligible (less than 0.05s). Consequently, LinCon can scale up to large-sized neural networks.
Tightness. We compare the tightness of the over-approximation sets computed by different approaches. Figure 6 plots the experimental results, along with the corresponding simulation trajectories. For B2 and Tora, LINCON has a significant tightness improvement over Polar: the range of the over-approximation sets (red boxes) calculated by Polar far outreaches the range of the reachable states obtained from the simulation. Verisig 2.0 terminates prematurely in Tora since the range of the action reaches  $10^{7}$  during the calculation which is too large to proceed. We defer to Appendix A.3 the experimental results for the remaining four cases where all three tools obtain similar tightness results.
Discussion on CartPole and QUAD. We discuss the verification results of the two complex cases, namely CartPole and QUAD. Both Polar and Verisig 2.0 fail to verify them. As shown in Figure 6, Polar and Verisig 2.0 abort after 20 steps due to the huge over-approximation error in CartPole. In contrast, such a complex policy trained with our approach can be efficiently and tightly verified. In particular, the trajectories diverge first and finally merge. The computed reachable states tightly over-approximate these trajectories, and the computation takes only 151 seconds. Regarding the 12-dimensional QUAD case, Polar and Verisig 2.0 time out (two hours) after only two steps, while LINCON produces a very tight set of reachable states in 1054 seconds even after 15 steps, which is 7 times deeper than Polar. To the best of our knowledge, this is the first time that QUAD can be formally verified more than ten steps under various decision networks. Note that the trained policies used in the comparison differ due to different decision networks, and thus agents may follow different paths to the goal region (see the simulated trajectories in Figure 6). Hence, for a fair comparison, we conduct more evaluations, from which we can draw the same conclusion as from Figure 6. The results are given in Appendix A.3.
Figure 6: Tightness comparison with respect to the DRL systems with larger decision networks (red box: over-approximation sets; green lines: simulation trajectories; blue box: goal region).
Policy Synthesis. Several works adopt programmatic policies (e.g., decision trees and program controllers) which are more interpretable and amenable to formal verification than neural policies. Bastani et al. [15] construct a decision tree to represent a DNN policy based on imitation learning [42]. Verma et al. [43, 44] follow a similar routine, distilling neural network policies into predefined program templates. Trivedi et al. [45] use a two-stage learning scheme to synthesize programmatic policies. Some efforts are dedicated to exploring the combination of training and verification. Zhu et al. [30] propose an inductive framework for synthesizing a deterministic policy program from neural policies. Wang et al. [31] learn programmatic controllers based on verification feedback to avoid safety violations. Our proposed PLDNN is essentially a DNN-based implementation of programmatic controllers, which could be integrated with these verification-guided synthesis approaches.
Reachability Analysis. Our work is also built atop the approaches for reachability analysis of neural-network-controlled systems. NNV [46] utilizes star set [47] to perform range analysis of decision networks. JuliaReach [48] uses zonotope propagation to cover the output of a decision network. Verisig [49, 9] models a decision network with differentiable activation functions (e.g., Tanh) as a hybrid system and analyzes its reachability for over-approximating the network. ReachNN* [28, 8] abstracts the input-output mapping of a decision network with a Bernstein polynomial, together with an error bound on the approximation. Sherlock [7] focuses on ReLU-based networks and computes tight Taylor models via rule generation. Polar [10] integrates the Taylor and Bernstein approximation techniques for building a Taylor model which over-approximates decision networks. All these efforts over-approximate the embedded DNNs, which limit their scalability and verification accuracy.
We have presented PLDNN that seamlessly integrates DNN and programmatic controls via state abstraction for boosting the reachability analysis of DRL systems. Unlike traditional train-then-transform approaches, PLDNN accompanies a novel inverse training and verification method, in which a DNN is first transformed into an equivalent set of linear control policies and then trained to optimize them. Experimental results have shown that PLDNN-controlled systems can be more efficiently and tightly verified than DNN-based systems, with up to 438 times speedup and 7 times deeper computation steps for a 12-dimensional control task.
Our work sheds light on a promising direction towards developing dependable DRL systems: learning easy-to-verify and high-performance control policies via DRL and abstraction techniques. Given the encouraging results of linear control policies, our work would also stimulate a passion for substituting them with, e.g., polynomial control policies, for training and verifying more complex DRL systems.
We thank all the anonymous reviewers for their valuable feedback on this work. The work has been supported by the National Key Research Program (2020AAA0107800), NSFC-ISF Joint Program (62161146001,3420/21), Huawei Technologies Co., Ltd., the Shanghai International Joint Lab of Trustworthy Intelligent Software (Grant No. 22510750100), the Shanghai Trusted Industry Internet Software Collaborative Innovation Center, the Engineering and Physical Sciences Research Council (EP/T006579/1), the National Research Foundation (NRF-RSS2022-009), Singapore, and the Shanghai Jiao Tong University Postdoc Scholarship.
[37] T. P. Lillicrap, J. J. Hunt, A. Pritzel et al., "Continuous control with deep reinforcement learning," in International Conference on Learning Representations (ICLR'16). OpenReview.net, 2016.
[38] S. Fujimoto, H. Hoof, and D. Meger, "Addressing function approximation error in actor-critic methods," in International Conference on Machine Learning (ICML'18). PMLR, 2018, pp. 1587-1596.
[39] D. E. Rumelhart, R. Durbin, R. Golden, and Y. Chauvin, "Backpropagation: The basic theory," Backpropagation: Theory, architectures and applications, pp. 1-34, 1995.
[40] S.-i. Amari, "Backpropagation and stochastic gradient descent method," Neurocomputing, vol. 5, no. 4-5, pp. 185-196, 1993.
[41] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “OpenAI Gym,” 2016, arXiv:1606.01540.
[42] P. Abbeel and A. Y. Ng, "Apprenticeship learning via inverse reinforcement learning," in Proceedings of the twenty-first International Conference on Machine Learning (ICML'04), 2004, p. 1.
[43] A. Verma, V. Murali, R. Singh, P. Kohli, and S. Chaudhuri, “Programmatically interpretable reinforcement learning,” in International Conference on Machine Learning. PMLR, 2018, pp. 5045–5054.
[44] A. Verma, H. Le, Y. Yue, and S. Chaudhuri, "Imitation-projected programmatic reinforcement learning," Advances in Neural Information Processing Systems, vol. 32, 2019.
[45] D. Trivedi, J. Zhang, S.-H. Sun, and J. J. Lim, "Learning to synthesize programs as interpretable and generalizable policies," Advances in Neural Information Processing Systems (NeurIPS'21), vol. 34, pp. 25-146-25-163, 2021.
[46] H.-D. Tran, X. Yang, D. Manzanas Lopez, P. Musau, L. V. Nguyen et al., “NNV: the neural network verification tool for deep neural networks and learning-enabled cyber-physical systems,” in International Conference on Computer Aided Verification (CAV'20). Springer, 2020, pp. 3–17.
[47] H.-D. Tran, S. Bak, W. Xiang, and T. T. Johnson, “Verification of deep convolutional neural networks using imagestars,” in International Conference on Computer Aided Verification (CAV'20). Springer, 2020, pp. 18–42.
[48] C. Schilling, M. Forets, and S. Guadalupe, “Verification of neural-network control systems by integrating taylor models and zonotopes,” in AAAI Conference on Artificial Intelligence (AAAI'22), 2022, pp. 8169–8177.
[49] R. Ivanov, T. J. Carpenter, J. Weimer, R. Alur, G. J. Pappas, and I. Lee, “Verifying the safety of autonomous systems with neural network controllers,” ACM Transactions on Embedded Computing Systems, vol. 20, no. 1, pp. 1–26, 2020.
A.1 Benchmarks Setting
We first present the goal region and unsafe region of each benchmark in Table 3.
As for the training settings, for the 6 regular benchmarks, the target is training a policy to guide the agent to reach the goal region. Therefore, we set a negative reward when the agent is not in the goal region. Once the agent reaches the goal region, it will be awarded a positive reward. In CartPole, the target of training is to prevent the pole from falling over, namely the pole angle  $-0.2 \leq x_{3} \leq 0.2$ . We modify the original discrete reward function to a continuous reward function  $-|x_{3}|$  to try to balance the pole to stay upright. As for QUAD, we aim to control the altitude of the quadrotor above 0.6 ( $x_{3} \geq 0.6$ ). Thus we set a continuous reward function  $-|x_{3} - 0.8|$  for training a policy that drives the quadrotor to ascend above 0.6.