General solutions for nonlinear differential equations: a deep reinforcement learning approach

s Physicists use differential equations to describe the physical dynamical world, and the solutions of these equations constitute our understanding of the world. During the hundreds of years, scientists developed several ways to solve these equations, i.e., the analytical solutions and the numerical solutions. However, for some complex equations, there may be no analytical solutions, and the numerical solutions may encounter the curse of the extreme computational cost if the accuracy is the first consideration. Solving equations is a high-level human intelligence work and a crucial step towards general artificial intelligence (AI), where deep reinforcement learning (DRL) may contribute. This work makes the first attempt of applying (DRL) to solve nonlinear differential equations both in discretized and continuous format with the governing equations (physical laws) embedded in the DRL network, including ordinary differential equations (ODEs) and partial differential equations (PDEs). The DRL network consists of an actor that outputs solution approximations policy and a critic that outputs the critic of the actor's output solution. Deterministic policy network is employed as the actor, and governing equations are embedded in the critic. The effectiveness of the DRL solver in Schrödinger equation, Navier-Stocks, Van der Pol equation, Burgers' equation and the equation of motion are discussed.  These authors contributed to the work equally and should be regarded as co-first authors. † Corresponding author. E-mail address: lihui@hit.edu.cn. 2 Introduction Differential equations, including ordinary differential equations (ODEs) and partial differential equations (PDEs), formalize the description of the dynamical nature of the world around us. However, solving these equations is a challenge due to extreme computational cost, although limited cases have analytical or numerical solutions1-3. Solving equations is a high-level human intelligence work and a crucial step towards general artificial intelligence. Therefore, the obstacle of extreme computational cost in numerical solution may be bypassed by using general AI techniques, such as deep learning and reinforcement learning4, 5, which are rapidly developed during the last decades. Recent years such efforts have been made, and three main kinds of the existed efforts using deep learning can be categorized into: 1) directly map to the solution represented by the deep neural network in the continuous manner as in the analytical solution6, data used to train the network is randomly sampled within the entire solution domain in each training batch, including initial conditions and boundary conditions; 2) directly map to the solution in the discretized manner as in the numerical solution7-9; and 3) indirectly map to the internal results or parameters of the numerical solutions, and use the internal results to derive the numerical solutions6, 10. The essence is to take advantage of the nonlinear representing ability of deep neural networks. The solutions are either directly output by the network or numerically derived from the outputs of the neural network, and the solution task is regarded as a weak-label task while the governing equation is treated as the weak-label to calculate the loss function of the network. The term ‘weaklabel’ is emphasized to make difference with the label in supervised learning, i.e., the true solutions are not known in these tasks, however, when we get a candidate solution by the neural network output, we can tell how far the output solution is to the true solution by the imbalance of the physical law. Because of the weak-label property, the solution using deep learning may be unstable for highdimensional ODEs/PDEs tasks. Hence, we propose a deep reinforcement learning (DRL) paradigm for the ODEs/PDEs solution. DRL is naturally suitable for weak-label tasks by the trial-error learning mechanism5, 11. Take the game of Go for example12, the only prior information about the task is the playing rules that defines win or lose, the label (or score) of each step is whether win or lose after the whole episode of playing rather than an exact score. 3 While employing reinforcement learning, we are essentially treating the solving of differential equations as a control task. The state is the known current-step solution (either the given initial condition or the intermediate DRL solution) of the differential equations, the action is the solution of the task, and the goal is to find a proper action to balance the governing equation with an acceptable error. A deep deterministic policy network is used to output action policy given a state, and the governing equation is used as the critic, gradients of the policy network is calculated based on the critic.