Optimal Control Via Neural Networks: A Convex Approach

1 INTRODUCTION . Decisions on how to best operate and control complex physical systems such as the power grid , commercial and industrial buildings , transportation networks and robotic systems are of critical societal importance . These systems are often challenging to control because they tend to have complicated and poorly understood dynamics , sometimes with legacy components are built over a long period of time ( Wolf , 2009 ) . Therefore detailed models for these systems may not be available or may be intractable to construct . For instance , since buildings account for 40 % of the global energy consumption ( Cheng et al. , 2008 ) , many approaches have been proposed to operate buildings more efficiently by controlling their heating , ventilation , and air conditioning ( HVAC ) systems ( Zhang et al. , 2017 ) . Most of these methods , however , suffer from two drawbacks . On one hand , a detailed physics model of a building can be used to accurately describe its behavior , but this model can take years to develop . On the other hand , simple control algorithms have been developed by using linear ( RC circuit ) models ( Ma et al. , 2012 ) to represent buildings , but the performance of these models may be poor since the building dynamics can be far from linear ( Shaikh et al. , 2014 ) . In this paper , we leverage the availability of data to strike a balance between requiring painstaking manual construction of physics based models and the risk of not capturing rich and complex system dynamics through models that are too simplistic . In recent years—with the growing deployment of sensors in physical and robotics systems—large amount of operational data have been collected , such as in smart buildings ( Suryadevara et al. , 2015 ) , legged robotics ( Meger et al. , 2015 ) and manipulators ( Deisenroth et al. , 2011 ) . Using these data , the system dynamics can be learned directly and then automatically updated at periodic intervals . One popular method is to parameterize these ∗Authors contribute equally . complex system dynamics using deep neural networks to capturing complex relationships ( He et al. , 2016 ; Vaswani et al. , 2017 ) , yet few research investigated how to integrate deep learning models into real-time closed-loop control of physical systems . A key reason that deep neural networks have not been directly applied in control is that even though they provide good performances in learning system behaviors , optimization on top of these networks is challenging ( Kawaguchi , 2016 ) . Neural networks , because of their structures , are generally not convex from input to output . Therefore , many control applications ( e.g. , where real-time decisions need to be made ) choose to favor the computational tractability offered by linear models despite their poor fitting performances . In this paper we tackle the modeling accuracy and control tractability tradeoff by building on the input convex neural networks ( ICNN ) in ( Amos et al. , 2017 ) to both represent system dynamics and to find optimal control policies . By making the neural network convex from input to output , we are able to obtain both good predictive accuracies and tractable computational optimization problems . The overall methodology is shown in Fig . 1 . Our proposed method ( shown in Fig . 1 ( b ) ) firstly utilizes an input convex network model to learn the system dynamics and then computes the best control decisions via solving a convex model predictive control ( MPC ) problem , which is tractable and has optimality guarantees . This is different from existing methods that uses model-free end-to-end controller which directly maps input to output ( shown in Fig . 1 ( a ) ) . Another major contribution of our work is that we explicitly prove that ICNN can represent all convex functions and systems dynamics , and is exponentially more efficient than widely used convex piecewise linear approximations ( Magnani & Boyd , 2009 ) . 1.1 RELATED WORK . The work in ( Amos et al. , 2017 ) was an impetus for this paper . The key differences are that the goal in ( Amos et al. , 2017 ) is to show that ICNN can achieve similar classification performances as conventional neural networks and how the former can be used in inference and prediction problems . Our goal is to use these networks for optimization and closed-loop control , and in a sense that we are more interested in the overall system performances and not directly the performance of the networks . We also extend the class of networks to include RNNs to capture dynamical systems . Control and decision-making have used deep learning mainly in model-free end-to-end controller settings ( shown in Fig . 1 ( a ) ) , such as sequential decision making in game ( Mnih et al. , 2013 ) , robotics manipulation ( Levine & Koltun , 2014 ; Levine et al. , 2016 ) , and control of cyber-physical systems ( Wei et al. , 2017 ; O ’ Neill et al. , 2010 ) . However , much of the success relies heavily on a reinforcement learning setup where the optimal state-action relationship can be learned via a large number of samples . However , many physical systems do not fit into the reinforcement learning process , where both the sample collection is limited by real-time operations , and there are physical model constraints hard to represent efficiently . To address the above sample efficiency , safety and model constraints incompatibility concerns faced by model-free reinforcement learning algorithms in physical system control , we consider a model-based control approach in this work . Model-based control algorithms often involve two stages – system identification and controller design . For the system identification stage , the goal is to learn a fixed form of system model to minimize some prediction error ( Ljung , 1998 ) . Most efficient model-based control algorithms have used a relatively simple function estimator for the system dynamics identification ( Nagabandi et al. , 2018 ) , such as linear model ( Ma et al. , 2012 ) and Gaussian processes ( Meger et al. , 2015 ; Deisenroth et al. , 2011 ) . These simplified models are sample-efficient to learn , and can be nicely incorporated in the sub-sequent optimal control problems . However , such simple models may not have enough representation capacity in modeling large-scale or high-dimension systems with nonlinear dynamics . Deep neural networks ( DNNs ) feature powerful representation capability , while the main challenge of using DNNs for system identification is that such models are typically highly non-linear and non-convex ( Kawaguchi , 2016 ) , which causes great difficulty for following decision making . A recent work from ( Nagabandi et al. , 2018 ) is close in spirit as our proposed method . Similarly , the authors use a model-based approach for robotics control , where they first fit a neural network for the system dynamics and then use the fitted network in an MPC loop . However , since ( Nagabandi et al. , 2018 ) use conventional NN for system identification , they can not solve the MPC problem to global optimality . Our work shows how the proposed ICNN control algorithm achieves the benefits from both sides of the world . The optimization with respect to inputs can be implemented using off-the-shelf deep learning optimizers , while we are able to obtain good identification accuracies and tractable computational optimization problems by using proposed method at the same time . 2 CLOSED-LOOP CONTROL WITH INPUT CONVEX NEURAL NETWORKS . In this paper , we consider the settings where a neural network is used in a closed-loop system . The fundamental goal is to optimize system performance which is beyond the learning performance of network on its own . In this section we describe how input convex neural networks ( ICNN ) can be extremely useful in these systems by considering two related problems . First , we show how ICNN perform in single-shot optimization problems . Then we extend the results to an input convex recurrent neural networks ( ICRNN ) , which allows us to both capture systems ’ complex dynamics and make time-series decisions . 2.1 SINGLE-SHOT PROBLEM . The following proposition states a simple sufficient condition for a neural network to be input convex : Proposition 1 . The feedforward neural network in Fig . 2 ( a ) is convex from input to output given that all weights between layers W1 : k and weights in the “ passthrough ” layers D2 : k are non-negative , and all of the activation functions are convex and nondecreasing ( e.g . ReLU ) . The structure of the input convex neural network ( ICNN ) structure in Proposition 1 is motivated by the structure in ( Amos et al. , 2017 ) but modified to be more suitable to control of dynamical systems . In ( Amos et al. , 2017 ) it only requires W2 : k to be non-negative while having no restrictions on weights W1 and D2 : k. Our construction achieves the exact representation by expanding the inputs to include both u ( ∈ Rd ) and −u . Then any negative weights in W1 and D2 : k in ( Amos et al. , 2017 ) ’ s ICNN structure is set to zero and its negation ( which is positive ) is added as the weight for corresponding −u . The reason for our construction is to allow the network to be “ rolled out in time ” when we are dealing with dynamical systems and multiple networks need to be composed together . An simple example that demonstrates how the proposed ICNN can be used to fit a convex function comes form fitting the |u| function . This function is convex and both decreasing and increasing . Let the activation function be ReLU ( · ) = max ( ·,0 ) . We can write |u| = −u+ 2ReLU ( u ) ( Amos et al. , 2017 ) . However , in this representation , we need a negative weight , the −1 in front of u , and this would be troublesome if we compose several networks together . In our proposed ICNN structure with all positive weights and input negation duplicates , we can write |u| = v+ 2ReLU ( u ) , where we impose a constraint v = −u . Such doubline on the number of input variables may potentially make the network harder to train . Yet during control , having all of the weights positive maintains the convexity between inputs and outputs even if multiple steps are considered which will be discussed in Section 2.2 . The constraint v =−u is linear and can be easily included in any convex optimization . This proposition follows directly from composition of convex functions ( Boyd & Vandenberghe , 2004 ) . Although it allows for any increasing convex activation functions , in this paper we work with the popular ReLU activation function . Two notable additions in ICNN compared with conventional feedforward neural networks are : 1 ) Addition of the direct “ passthrough ” layers connecting inputs to hidden layers and conventional feedforward layers connecting hidden layers for better representation power . 2 ) the expanded inputs that include both u and −u . The proposed ICNN structure is shown in Fig . 2 ( a ) . Note that such construction guarantees that the network is convex and non-decreasing with respect to the expanded inputs û = [ u −u ] , while the output can achieve either decreasing or non-decreasing functions over u . Fundamentally , ICNN allows us to use neural networks in decision making processes by guaranteeing the solution is unique and globally optimal . Since many complex input and output relationships can be learned through deep neural networks , it is natural to consider using the learned network in an optimization problem in the form of min u f ( u ; W ) ( 1a ) s.t . u ∈U , ( 1b ) where U is a convex feasible space . Then if f is an ICNN , optimizing over u is a convex problem , which can be solved efficiently to global optimality . Note that we will always duplicate the variables by introducing v = −u , but again this does not change the convexity of the problem . Of course , since the weights of the network are restricted to be nonnegative , the performance of the network ( e.g. , classification ) may be worse . A common thread we observe in this paper is that trading off classification performance with tractability can be preferable .

Summary: The paper proposes neural networks which are convex on inputs data to control problems. These types of networks, constructed based on either MLP or RNN, are shown to have similar representation power as their non-convex versions, thus are potentially able to better capture the dynamics behind complex systems compared with linear models. On the other hand, convexity on inputs brings much convenience to the later optimization part, because there are no worries on global/local minimum or escaping saddle points. In other words, convex but nonlinear provides not only enough search space, but also fast and tractable optimization. The compromise here is the size of memory, since 1) more weights and biases are needed to connect inputs and hidden layers in such nets and 2) we need to store also the negative parts on a portion of weights.