Patent Publication Number: US-11663370-B2

Title: Automated system for generating approximate safety conditions for monitoring and verification

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of and is a non-provisional patent application of U.S. Provisional Application No. 62/953,088, filed on Dec. 23, 2019, the entirety of which is hereby incorporated by reference. 
    
    
     BACKGROUND OF INVENTION 
     (1) Field of Invention 
     The present invention relates to a machine learning system and, more specifically, to an automated system for generating safety conditions for monitoring and verification. 
     (2) Description of Related Art 
     This disclosure is directed to a process for yielding what are known as barrier functions. The notion of a barrier function in its most recent form began with the work of Prajna et al. (see the List of Incorporated Literature References, Literature Reference Nos. 6 and 7), although the notion is implicit in much earlier work. Barrier functions were introduced for proving the safety of hybrid and nonlinear dynamical systems. The concept was introduced in control systems by Wieland et al. (see Literature Reference No. 8), and later reformulated in a number of ways, notably in the recent works of Ames et al. (see Literature Reference Nos. 2 and 3). See the work of Ames et al (2019) (see Literature Reference No. 1) for an overview of barrier functions and related formulations. For the latter, see also the work of Ohnishi et al. (see Literature Reference No. 5). 
     Control barrier functions have been used in ensuring safety in the control of a number of different systems. It is often the case that barrier functions are identified analytically (and, in particular, manually) according to the particular application, and there do not appear to be many versatile methods of constructing barrier functions for a general class of control systems. One exception is given in the work of Wu et al. (see Literature Reference No. 9), where a general method for constructing barrier functions is given for fully-actuated simple mechanical systems. 
     One relevant work is that of Fisac et al. (see Literature Reference No. 4). In that work, the authors formulated a method of approximating the globally safe region of the state space of a given control system. Their method does this by learning a fixed function of a contraction operator whose support approximates the globally safe region. While the method has a way of improving the approximation of the region, the weakness of the method is that it does not provide an absolute guarantee that the function&#39;s support indicates a truly safe region. 
     In contrast, the process as provided for in the present disclosure directly aims to learn barrier functions, which are guaranteed to be supported in the globally safe region of control systems. Thus the approximate barrier function yielded by the present approach is more likely to be supported in the globally safe region, and the method can readily be used in tandem with a satisfiability modulo theories solver to obtain genuine (not just approximate) barrier functions, from which one can identify a provably safe region of the state space of a system. On a more general level, another advantage of the approach provided by the present disclosure is that it naturally encompasses (and is more general than) that of Fisac et al. In other words, the approach of Fisac et al. can be viewed as a particular step of the present invention. So, for example, a function yielded by their approach can be found in the present, and then can be improved further in subsequent learning steps as described in further detail below. 
     The approach as provided for in the present disclosure is an unexpected improvement over the prior art because when researchers in control theory have considered barrier functions, they have focused on barrier functions that are explicitly identified and described rather than on learning barrier functions from data. For example, the closest work (i.e., Fisac et al.) is informed by the theory of contraction operators in connection with reinforcement learning. By contrast, because the approach of the present disclosure is explicitly directed instead toward trying to learn a barrier function, the approach is more likely to yield high-quality approximations. In particular, the functions learned by the present approach are more likely to be actual barrier functions (as one could potentially prove using a satisfiability modulo theories solver) and therefore any safety guarantees with respect to the outputs of the process will likely be of a higher quality. Likewise, in contrast with the manual and analytical approach taken in the literature on control theory, the present approach can potentially learn barrier functions for significantly more complex dynamical systems than could be feasibly handled manually. 
     Other work on using machine learning to learn barrier functions has been done, one by Zhao et al. (2020) (see Literature Reference No. 10), and another by Chang et al. (2020) (see Literature Reference No. 11). However, there are some important differences between these works and the present invention. First, the work of Zhao et al. applies to non-controlled dynamical systems, i.e., systems that do not have control variables, or alternatively systems for which such control variables are governed by an already set control policy. Additionally, the work of Zhao et al. requires a specific set of initial conditions which the learned barrier function aims to ensure the safety of. Also, the work of Chang et al. learns barrier functions together with a control policy as a way of helping the policy learn to be safe. In particular, the method aims only to learn barrier functions that certify the safety of the particular learned control policy. 
     Unlike these works, the system of the present disclosure learns barrier functions that are not restricted to certifying the safety of a particular set of initial conditions or control policy. Instead, the system of the present disclosure seeks to ensure the safety of the state space of a control system in general, and can potentially be used to find as much of the full safe region of the state space as possible, subject to the limitations due to data, dimensionality, and other aspects of a given application. Thus, given the issues with the prior art, the system of the present disclosure solves a continuing need for an improved and automated system for generating approximate safety conditions for monitoring and verification. 
     SUMMARY OF INVENTION 
     Described is a system and method for generating safety conditions for a cyber-physical system with state space S, action space A and trajectory data labelled as either safe or unsafe. In one aspect, the system comprises one or more processors and a memory. The memory is a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform several operations, such as (1) receiving inputs, the inputs being at least one of a discrete dynamics model, a continuous dynamics model, a set of state space data points, a neural network, an initial safety function, and a method of determining an action at a given dynamical system state that is optimal according to the neural network; (2) if a discrete dynamics model is part of the input, then minimizing first and second loss functions then proceeding, otherwise proceeding; (3) causing the neural network h θ  to become the barrier function by minimizing a distinct loss function; and (4) outputting the barrier function. 
     In another aspect, the system causes the cyber-physical system to initiate a physical operation based on the barrier function. 
     In another aspect, the system performs an operation of determining if the cyber-physical system is entering an usafe state based on the barrier function, such that if the cyber-physical system is entering the usafe state, then causing the cyber-physical system to initiate a maneuver to position the cyber-physical system into a safe state. 
     In yet another aspect, minimizing the first and second loss functions includes operations of: 
     minimizing the first loss function that causes the neural network h θ  to learn an approximation of an initial safety function; 
     minimizing the second loss function that causes the neural network h θ  to satisfy a Bellman-like equation, which in turn modifies the neural network h θ , causing the neural network h θ  to be closer to being a barrier function than a previous version of itself; 
     performing an iteration of minimizing the second loss function to cause the neural network h θ  to satisfy a Bellman-like equation, which in turn further modifies the neural network h θ , causing the neural network h θ  to be closer to being a barrier function than a previous version of itself; and iterating minimization of the second loss function a predetermined number of times. 
     Further, in minimizing the distinct loss function, a loss function of    η,α,δ     1     ,δ     2   (θ) is minimized for the discrete dynamics model and a loss function of    κ,α,δ     1     ,δ     2   (θ) is minimized for the continuous dynamics model. 
     In yet another aspect, the discrete dynamics model is τ: S×A→S, where S denotes a state space of the dynamical system, A denotes action space, and τ denotes a transition function that yields a state of the cyber-physical system following from an inputted state-action pair. 
     Further, the continuous dynamics model is {dot over (s)}=f (s,a), where s denotes a state of the system, and α denotes a control action being taken, and {dot over (s)} denotes a time derivative of s, and f denotes a vector field function that yields a time derivative of the state resulting from an inputted state-action pair. 
     Finally, the present invention also includes a computer program product and a computer implemented method. The computer program product includes computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having one or more processors, such that upon execution of the instructions, the one or more processors perform the operations listed herein. Alternatively, the computer implemented method includes an act of causing a computer to execute such instructions and perform the resulting operations. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The objects, features and advantages of the present invention will be apparent from the following detailed descriptions of the various aspects of the invention in conjunction with reference to the following drawings, where: 
         FIG.  1    is a block diagram depicting the components of a system according to various embodiments of the present invention; 
         FIG.  2    is an illustration of a computer program product embodying an aspect of the present invention; 
         FIG.  3    is a graph depicting a function h θ  resulting from a learning process according to various aspects of the present invention; 
         FIG.  4    is a table providing statistics from an example evaluation of the system according to various aspects of the present invention; 
         FIG.  5    is a graph depicting the function h θ  from a learning process according to various aspects of the present invention; 
         FIG.  6    is a graph depicting the value of h θ  (x, θ, s) with respect to (x, θ) under the assumption that 
       
         
           
             
               
                 s 
                 = 
                 
                   10 
                   ⁢ 
                   
                     m 
                     s 
                   
                 
               
               ; 
             
           
         
       
         FIG.  7    is a diagram outlining the barrier function learning process; and 
         FIG.  8    is a block diagram depicting control of a device according to various embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     The present invention relates to a machine learning system and, more specifically, to an automated system for generating safety conditions for monitoring and verification. The following description is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications, will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. 
     In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without necessarily being limited to these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring the present invention. 
     The reader&#39;s attention is directed to all papers and documents which are filed concurrently with this specification and which are open to public inspection with this specification, and the contents of all such papers and documents are incorporated herein by reference. All the features disclosed in this specification (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features. 
     Furthermore, any element in a claim that does not explicitly state “means for” performing a specified function, or “step for” performing a specific function, is not to be interpreted as a “means” or “step” clause as specified in 35 U.S.C. Section 112(f). In particular, the use of “step of” or “act of” in the claims herein is not intended to invoke the provisions of 35 U.S.C. 112(f). 
     Before describing the invention in detail, first a list of cited references is provided. Next, a description of the various principal aspects of the present invention is provided. Subsequently, an introduction provides the reader with a general understanding of the present invention. Finally, specific details of various embodiment of the present invention are provided to give an understanding of the specific aspects. 
     (1) List of Incorporated Literature References 
     The following references are cited throughout this application. For clarity and convenience, the references are listed herein as a central resource for the reader. The following references are hereby incorporated by reference as though fully set forth herein. The references are cited in the application by referring to the corresponding literature reference number, as follows:
     1. Ames, A. D., Coogan, S., Egerstedt, M., Notomista, G., Sreenath, K., &amp; Tabuada, P. (2019). Control Barrier Functions: Theory and Applications.  IEEE European Control Conference  (pp. 3420-3431). Naples, Italy: IEEE.   2. Ames, A. D., Grizzle, J. W., &amp; Tabuada, P. (2014). Control Barrier Function based Quadratic Programs with Application to Adaptive Cruize Control.  IEEE Conference on Decision and Control  (pp. 6271-6278). Los Angeles, Calif.: IEEE.   3. Ames, A. D., Xu, X., Grizzle, J. W., &amp; Tabuada, P. (2017). Control Barrier Function Based Quadratic Programs for Safety Critical Systems.  IEEE Transactions on Automatic Control,  62(8), 3861-3876.   4. Fisac, J. F., Lugovoy, N. F., Rubies-Royo, V., Ghosh, S., &amp; Tomlin, C. J. (2019).  Bridging Hamilton - Jacobi Safety Analysis and Reinforcement Learning . Retrieved from UC Berkeley: Jaime Fisac Papers: found at people.eecs.berkeley.edu/˜jfisac/papers/Bridging_Safety_and_RL.pdf   5. Ohnishi, M., Wang, L., Notomista, G., &amp; Egerstedt, M. (2019). Barrier-Certified Adaptive Reinforcement Learning With Applications to Brushbot Navigation.  IEEE Transactions on Robotics,  35(5), 1186-1205.   6. Prajna, S. (2006). Barrier certificates for nonlinear model validation.  Automatica,  42(1), 117-126.   7. Prajna, S., &amp; Jadbabaie, A. (2004). Safety Verification of Hybrid Systems Using Barrier Certificates.  Hybrid Systems: Computation and Control  (pp. 477-492). Philadelphia, Pa.: Springer.   8. Wieland, P., &amp; Allgöwer, F. (2007). Constructive safety using control barrier functions.  IFAC Symposium on Nonlinear Control Systems  (pp. 462-467). Pretoria, South Africa: Elsevier.   9. Wu, G., &amp; Sreenath, K. (2016). Safety-Critical Geometric Control for Systems on Manifolds Subject to Time-Varying Constraints.  IEEE Transactions on Automatic Control, in review.      10. Zhao, H., Zeng, X., Chen, T., &amp; Liu, Z. (2020). Synthesizing barrier certificates using neural networks . ACM International Conference on Hybrid Systems: Computation and Control . Sydney, Australia: ACM.   11. Chang, Y.-C., &amp; Gao, S. (2020). Stabilizing Neural Control Policies with Invariance Guarantee Using Self-Learned Lyapunov Critics. Unpublished.   

     (2) Principal Aspects 
     Various embodiments of the invention include three “principal” aspects. The first is a system for generating safety conditions for monitoring and verification. The system is typically in the form of a computer system operating software or in the form of a “hard-coded” instruction set. This system may be incorporated into a wide variety of devices that provide different functionalities. The second principal aspect is a method, typically in the form of software, operated using a data processing system (computer). The third principal aspect is a computer program product. The computer program product generally represents computer-readable instructions stored on a non-transitory computer-readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape. Other, non-limiting examples of computer-readable media include hard disks, read-only memory (ROM), and flash-type memories. These aspects will be described in more detail below. 
     A block diagram depicting an example of a system (i.e., computer system  100 ) of the present invention is provided in  FIG.  1   . The computer system  100  is configured to perform calculations, processes, operations, and/or functions associated with a program or algorithm. In one aspect, certain processes and steps discussed herein are realized as a series of instructions (e.g., software program) that reside within computer readable memory units and are executed by one or more processors of the computer system  100 . When executed, the instructions cause the computer system  100  to perform specific actions and exhibit specific behavior, such as described herein. In various aspects, the computer system  100  can be embodied in any device(s) that operates to perform the functions as described herein as applicable to the particular application, such as a desktop computer, a mobile or smart phone, a tablet computer, a computer embodied in a mobile platform, or any other device or devices that can individually and/or collectively execute the instructions to perform the related operations/processes. 
     The computer system  100  may include an address/data bus  102  that is configured to communicate information. Additionally, one or more data processing units, such as a processor  104  (or processors), are coupled with the address/data bus  102  and/or associated memories. The processor  104  is configured to process information and instructions. In an aspect, the processor  104  is a microprocessor. Alternatively, the processor  104  may be a different type of processor such as a parallel processor, application-specific integrated circuit (ASIC), programmable logic array (PLA), complex programmable logic device (CPLD), or a field programmable gate array (FPGA) or any other processing component operable for performing the relevant operations. 
     The computer system  100  is configured to utilize one or more data storage units (memories). The computer system  100  may include a volatile memory unit  106  (e.g., random access memory (“RAM”), static RAM, dynamic RAM, etc.) coupled with the address/data bus  102 , wherein a volatile memory unit  106  is configured to store information and instructions for the processor  104 . The computer system  100  further may include a non-volatile memory unit  108  (e.g., read-only memory (“ROM”), programmable ROM (“PROM”), erasable programmable ROM (“EPROM”), electrically erasable programmable ROM “EEPROM”), flash memory, etc.) coupled with the address/data bus  102 , wherein the non-volatile memory unit  108  is configured to store static information and instructions for the processor  104 . Alternatively, the computer system  100  may execute instructions retrieved from an online data storage unit such as in “Cloud” computing. In an aspect, the computer system  100  also may include one or more interfaces, such as an interface  110 , coupled with the address/data bus  102 . The one or more interfaces are configured to enable the computer system  100  to interface with other electronic devices and computer systems. The communication interfaces implemented by the one or more interfaces may include wireline (e.g., serial cables, modems, network adaptors, etc.) and/or wireless (e.g., wireless modems, wireless network adaptors, etc.) communication technology. 
     In one aspect, the computer system  100  may include an input device  112  coupled with the address/data bus  102 , wherein the input device  112  is configured to communicate information and command selections to the processor  104 . In accordance with one aspect, the input device  112  is an alphanumeric input device, such as a keyboard, that may include alphanumeric and/or function keys. Alternatively, the input device  112  may be an input device other than an alphanumeric input device. In an aspect, the computer system  100  may include a cursor control device  114  coupled with the address/data bus  102 , wherein the cursor control device  114  is configured to communicate user input information and/or command selections to the processor  104 . In an aspect, the cursor control device  114  is implemented using a device such as a mouse, a track-ball, a track-pad, an optical tracking device, or a touch screen. The foregoing notwithstanding, in an aspect, the cursor control device  114  is directed and/or activated via input from the input device  112 , such as in response to the use of special keys and key sequence commands associated with the input device  112 . In an alternative aspect, the cursor control device  114  is configured to be directed or guided by voice commands. 
     In an aspect, the computer system  100  further may include one or more optional computer usable data storage devices, such as a storage device  116 , coupled with the address/data bus  102 . The storage device  116  is configured to store information and/or computer executable instructions. In one aspect, the storage device  116  is a storage device such as a magnetic or optical disk drive (e.g., hard disk drive (“HDD”), floppy diskette, compact disk read only memory (“CD-ROM”), digital versatile disk (“DVD”)). Pursuant to one aspect, a display device  118  is coupled with the address/data bus  102 , wherein the display device  118  is configured to display video and/or graphics. In an aspect, the display device  118  may include a cathode ray tube (“CRT”), liquid crystal display (“LCD”), field emission display (“FED”), plasma display, or any other display device suitable for displaying video and/or graphic images and alphanumeric characters recognizable to a user. 
     The computer system  100  presented herein is an example computing environment in accordance with an aspect. However, the non-limiting example of the computer system  100  is not strictly limited to being a computer system. For example, an aspect provides that the computer system  100  represents a type of data processing analysis that may be used in accordance with various aspects described herein. Moreover, other computing systems may also be implemented. Indeed, the spirit and scope of the present technology is not limited to any single data processing environment. Thus, in an aspect, one or more operations of various aspects of the present technology are controlled or implemented using computer-executable instructions, such as program modules, being executed by a computer. In one implementation, such program modules include routines, programs, objects, components and/or data structures that are configured to perform particular tasks or implement particular abstract data types. In addition, an aspect provides that one or more aspects of the present technology are implemented by utilizing one or more distributed computing environments, such as where tasks are performed by remote processing devices that are linked through a communications network, or such as where various program modules are located in both local and remote computer-storage media including memory-storage devices. 
     An illustrative diagram of a computer program product (i.e., storage device) embodying the present invention is depicted in  FIG.  2   . The computer program product is depicted as floppy disk  200  or an optical disk  202  such as a CD or DVD. However, as mentioned previously, the computer program product generally represents computer-readable instructions stored on any compatible non-transitory computer-readable medium. The term “instructions” as used with respect to this invention generally indicates a set of operations to be performed on a computer, and may represent pieces of a whole program or individual, separable, software modules. Non-limiting examples of “instruction” include computer program code (source or object code) and “hard-coded” electronics (i.e. computer operations coded into a computer chip). The “instruction” is stored on any non-transitory computer-readable medium, such as in the memory of a computer or on a floppy disk, a CD-ROM, and a flash drive. In either event, the instructions are encoded on a non-transitory computer-readable medium. 
     (3) Introduction 
     The present disclosure provides a novel machine learning technique to learn a function that effectively monitors the safety of a system in time. The function can be used to markedly improve a variety of systems. For example, the function can be used (1) to provide inputs to runtime monitoring systems for evaluating the safety of the system and determining whether or not to intervene, (2) for formal verification procedures that attempt to prove (or disprove) the safety of the algorithms being used to control the system, or (3) with systems for control or decision making. The kinds of systems that we consider are cyber-physical systems (CPS), including autonomous systems such as self-driving cars, autopilot systems, and so forth. Where the state space of the system is the set S, the machine learning approach yields as output a mathematical function h: S→ . Intuitively, this function measures, for each state s∈S, the degree h(s) to which it is a globally safe state. Here, being a globally safe state s means that there exists a safe trajectory starting at s. This is in contrast with instantaneously safe states for which there may be no such safe trajectory (e.g., if the car is driving too fast in the direction of an obstacle, it may be safe now, but will not be so for long). In the intended semantics, those states s such that h(s)&gt;0 are estimated to be globally safe. The procedure requires a sequence of labelled states where the labels indicate whether or not the state is considered as being instantaneously safe. These constitute the data required to use machine learning to obtain h. 
     The central premise of the invention is specifically a method of learning a type of functions called barrier functions for a range of control systems. Once found, barrier functions characterize globally safe regions of their respective state spaces and can therefore be used practically for safety monitoring, informing control/decision making, and for formal verification of system safety. There are not many general methods for finding barrier functions, with the traditional approach being a manual one, and the present procedure is the first to use machine learning to do so. 
     The system and method of the present disclosure invention has several distinct uses and advantages. In terms of usage, there are three primary modes in which the invention can be employed: (1) for (runtime) safety monitoring, (2) for informing a control/planning module, and (3) for formal verification. 
     First, the goal of safety monitoring is to provide an indication of the safety of a system at a given time. In particular, it is often employed at runtime and when a state is encountered that is deemed to be too close to being unsafe, the monitor raises a flag and allows a redundancy/backup/failsafe system to take over control of the system. This operation can be implemented into a variety of applications, a non-limiting example of which includes autonomous vehicles. As a specific example scenario, for an autonomous car, if the safety monitor determines that the vehicle is too close to driving out of the prescribed driving lane, it might take over steering to move the vehicle away from the lane marker. In the notation above, given a threshold value δ&gt;0, the invention can be used for runtime monitoring by raising the monitoring flag whenever a state s is encountered such that h(s)&lt;δ. This can be employed on autonomous systems, as well as on human-operated systems. Thus, in some aspects, the invention can be used as a safety monitoring tool for cyber-physical systems, such as an automobile, air or spacecraft, etc., such that it is employed either on the platform or else on a remote location in order to actively monitor the safety of the platform. 
     Second, the invention can be employed in connection with automatic control or planning functions. In these cases, at each time t the system is in a state s t  and an action a t  is selected from a set A of actions. For example, in an autonomous car, an action could be to adjust the steering angle or throttle and in an autonomous aircraft it might be to adjust the ailerons. In this setting the function h above can be used to filter acceptable actions as follows. Assume that the system is provided an estimate τ(s,a) of the state that will be reached by taking action a from s. Then, given a mechanism for sampling candidate actions a, the system can be operated by first evaluating whether or not h(τ(s,a))&gt;δ, for a fixed threshold value δ. If the inequality holds, then select action a, otherwise sample a new candidate action a′ and repeat this process. Thus, the system can be implemented as an automatic control or decision making module on a cyber-physical system, wherein it consumes state information obtained from sensor or other modules and produces outputs which are routed either to actuators or else to further modules. 
     Finally, the condition that h(s)&gt;δ, for a threshold value δ, can be employed as an approximate definition of what it means for a state s to be (globally) safe. Therefore, this condition can be employed as a safety specification for formal verification to obtain mathematical guarantees on system behavior with respect to this condition, or examples where the condition is violated that might indicate a safety failure. Thus, in some aspects, the invention operates to produce a function (the neural network used to approximate the barrier function) that can be used by an external tool, such as a satisfiability modulo theories solver or a computer proof assistant as a specification against which to carry out formal verification of a cyber-physical system control or decision making module. 
     As can be appreciated by those skilled in the art, the process as described herein can be incorporated into a variety of systems, including applications for advanced driver assistance, flight management, flight control, autonomous driving, and others. For each of these domain areas, the invention can be employed to provide safety monitoring capabilities, improved control capabilities, or for purposes of formal verification (e.g., in connection with a more general verification and validation regime). Specific details are provided below. 
     (4) Specific Details of Various Embodiments 
     As noted above, the present disclosure provides an automated system for identifying a function that effectively monitors the safety of a system in time. As a preliminary matter, the notion of barrier functions for both discrete and continuous time control systems must be defined. Suppose that there is first a discrete time control system defined by a transition function τ:S×A→S, where S and A are the state and action spaces of the system, respectively. So, taking an action a∈A at state s∈S leads to the subsequent state τ(s,a). The transition function can be known or learned from system trajectories. Assume also that there is a subset K⊆S which contains all of the states that are considered instantaneously safe. It is desired that the state of the system never exits K. A barrier function is a function h:S→  such that there exists a constant η∈[0,1) in which (1−η)h(s)≤max a∈A  h(τ(s,a)) for all s∈K satisfying h(s)&gt;0, and h(s)≤0 if s∉K. For such a function, the states s∈S such that h(s)&gt;0 are known to be globally safe; that is, starting at s, it is possible to keep the system state in K indefinitely in the future. 
     To define barrier functions for continuous time control systems, suppose as before that S and A are the state and action spaces, and K⊆S is the set of safe states. Assume that S⊆   n  and A⊆   m , and that the dynamics of the system are defined by the system of differential equations {dot over (s)}=f(s,a), where f: S×A→   n  is a Lipschitz continuous function. Then a barrier function for this system is a function h: S→  such that there exists a constant κ&gt;0 in which ∇h(s)·f(s,a)+κh(s)≥0 for all s∈K satisfying h(s)&gt;0, and h(s)≤0 if s∉K. Then as before, the states s∈S such that h(s)&gt;0 are known to be globally safe. 
     Next, described are the approaches to learning barrier functions from data in both the discrete and continuous settings. Begin with the discrete setting. Assume the control system is defined by the transition function τ: S×A→S and safe set K as described above. Next, suppose that there exists system state data D={x j :j=1, . . . , N}⊆S obtained in some manner, e.g., obtained from previous system simulations, random state space sampling, or real world data collection. Then D can be divided into the sets K′=D∩K and U=D∩(S\K), so that K′ consists of the safe data points and U consists of the unsafe data points. Next, let h θ :S→  be a neural network with which it is desired to learn a barrier function for the system. 
     As shown in  FIG.  7   , the learning process in the discrete setting is made up of two fundamentally different learning stages; Learning Stage One  700  and Learning Stage Two  702 . Learning Stage One  700  involves learning with a contractive Bellman operator, similar to Q-learning and the work of Fisac et al. (see Literature Reference No. 4). In that work, the goal is to learn a function V:S→  that satisfies the Bellman equation: 
                 V   ⁡     (   s   )       =         (     1   -   γ     )     ⁢     L   ⁡     (   s   )         +     γ   ⁢   min   ⁢     {       L   ⁡     (   s   )       ,           ⁢       max     a   ∈   A       ⁢     V   ⁡     (     τ   ⁡     (     s   ,   a     )       )           }           ,         
where γ∈(0,1) is a discount factor and L:S→  is a differentiable function that satisfies L(s)≥0 if and only if s∈K. For γ≈1, the set {s∈S:V(s)&gt;0} gives an approximation of the globally safe set, which improves as γ→1. The first stage of the learning process is a generalization of this basic idea. Briefly, the above Bellman equation is generalized to relate it to the barrier function conditions, and allowing for replacements of the function L in the equation. More details are in the following paragraphs.
 
     It should be noted that several example inputs  704  are provided in  FIG.  7   . The nature of the inputs  704  will vary depending on the particular application in which the system is employed, although each of these inputs must be provided. For example, in some cases the maximum of h θ (r(x,a)) with respect to actions a can be calculated directly (i.e., when the set of actions is sufficiently small), but in other cases it may be estimated using a machine learning algorithm. 
     As in Fisac et al., the present process utilizes a differentiable function L:S→  that satisfies the weaker condition that L(s)&gt;0 only if s∈K. This function can be learned or given, and is considered as an upper bound and a first approximation to the barrier function. Also, fixed are a barrier function constant η∈[0,1), a discount factor γ∈(0,1), and other constants δ 1 , δ 2 &gt;0. With these constants in place, the learning process can be described. The first step is to let h θ  learn an approximation of the function L; to do so, minimize the loss function 
     
       
         
           
             
               
                 
                   
                     
                       ℓ 
                       
                         
                           δ 
                           1 
                         
                         , 
                         
                           δ 
                           2 
                         
                       
                       
                         ( 
                         L 
                         ) 
                       
                     
                     ⁡ 
                     
                       ( 
                       θ 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         ∑ 
                         
                           
                             x 
                             j 
                           
                           ∈ 
                           U 
                         
                       
                       ⁢ 
                       
                         max 
                         ⁢ 
                         
                           { 
                           
                             0 
                             , 
                             
                               
                                 
                                   h 
                                   θ 
                                 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     x 
                                     j 
                                   
                                   ) 
                                 
                               
                               + 
                               
                                 δ 
                                 1 
                               
                             
                             , 
                             
                               
                                 - 
                                 
                                   
                                     h 
                                     θ 
                                   
                                   ⁡ 
                                   
                                     ( 
                                     
                                       x 
                                       j 
                                     
                                     ) 
                                   
                                 
                               
                               - 
                               
                                 δ 
                                 2 
                               
                             
                           
                           } 
                         
                       
                     
                     + 
                     
                       
                         ∑ 
                         
                           
                             x 
                             j 
                           
                           ∈ 
                           
                             K 
                             ′ 
                           
                         
                       
                       ⁢ 
                       
                         
                           
                             ( 
                             
                               
                                 
                                   h 
                                   θ 
                                 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     x 
                                     j 
                                   
                                   ) 
                                 
                               
                               - 
                               
                                 L 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     x 
                                     j 
                                   
                                   ) 
                                 
                               
                             
                             ) 
                           
                           2 
                         
                         . 
                       
                     
                   
                 
               
               
                 
                     
                 
               
             
           
         
       
     
     Note that the purpose of the first sum is to ensure that the values of h θ  at unsafe points are in the interval [−δ 2 , −δ 1 ]⊆(−∞, 0]. The manner in which this minimization is carried out will be clear to one skilled in the art. 
     The goal of the subsequent learning steps is to have h θ  learn to satisfy 
                 h   θ     ⁡     (   s   )       ≈         (     1   -   γ     )     ⁢     L   ⁡     (   s   )         +     γ   ⁢   min   ⁢     {       L   ⁡     (   s   )       ,       η   ⁢            h   θ     ⁡     (   s   )              +       max     a   ∈   A       ⁢       h   θ     ⁡     (     τ   ⁡     (     s   ,   a     )       )             }               
for all s∈K. This expression corresponds to a Bellman operator, which is contractive if
 
               γ   ⁢           ∈     (     0   ,     1     1   +   η         )       ,         
of which h θ  seeks to learn a fixed function. Note that in most cases, equality in this expression does not imply that h θ  is a barrier function. Intuitively, h θ  provides a function that is closer to being a barrier function than L. For this reason, periodically replace L in the above equation by the current status of h θ . Assuming equality in the above expression can always be attained, this process amounts to finding a sequence of functions L≥H 1 ≥H 2 ≥ . . . , where H 1  satisfies
 
                 H   1     ⁡     (   s   )       =         (     1   -   γ     )     ⁢     L   ⁡     (   s   )         +     γ   ⁢           ⁢   min   ⁢     {       L   ⁡     (   s   )       ,           ⁢       η   ⁢            H   1     ⁡     (   s   )              +       max     a   ∈   A       ⁢       H   1     ⁡     (     τ   ⁡     (     s   ,           ⁢   a     )       )             }               
and the subsequent functions H k , k≥2, satisfy
 
     
       
         
           
             
               
                 H 
                 k 
               
               ⁡ 
               
                 ( 
                 s 
                 ) 
               
             
             = 
             
               
                 
                   ( 
                   
                     1 
                     - 
                     γ 
                   
                   ) 
                 
                 ⁢ 
                 
                   
                     H 
                     
                       k 
                       - 
                       1 
                     
                   
                   ⁡ 
                   
                     ( 
                     s 
                     ) 
                   
                 
               
               + 
               
                 γ 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 min 
                 ⁢ 
                 
                   
                     { 
                     
                       
                         
                           H 
                           
                             k 
                             - 
                             1 
                           
                         
                         ⁡ 
                         
                           ( 
                           s 
                           ) 
                         
                       
                       , 
                       
                           
                       
                       ⁢ 
                       
                         
                           η 
                           ⁢ 
                           
                              
                             
                               
                                 H 
                                 k 
                               
                               ⁡ 
                               
                                 ( 
                                 s 
                                 ) 
                               
                             
                              
                           
                         
                         + 
                         
                           
                             max 
                             
                               a 
                               ∈ 
                               A 
                             
                           
                           ⁢ 
                           
                             
                               H 
                               k 
                             
                             ⁡ 
                             
                               ( 
                               
                                 τ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     s 
                                     , 
                                     
                                         
                                     
                                     ⁢ 
                                     a 
                                   
                                   ) 
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                     } 
                   
                   . 
                 
               
             
           
         
       
     
     Optionally, one may let γ approach 1, as in the work of Fisac et al., while learning any of the functions H k . 
     In theory, when all functions H k  are bounded below by a single constant, the limit lim k→∞ H k  converges to a barrier function. Practically, exact equality in the above expression cannot be ensured; thus, the process only seeks to minimize the difference between the two sides of the equality. So a goal is to learn a sequence of functions L=H 0 , H 1 , H 2 , . . . that satisfy 
     
       
         
           
             
               
                 H 
                 k 
               
               ⁡ 
               
                 ( 
                 s 
                 ) 
               
             
             ≈ 
             
               
                 
                   ( 
                   
                     1 
                     - 
                     γ 
                   
                   ) 
                 
                 ⁢ 
                 
                   
                     H 
                     
                       k 
                       - 
                       1 
                     
                   
                   ⁡ 
                   
                     ( 
                     s 
                     ) 
                   
                 
               
               + 
               
                 γ 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 min 
                 ⁢ 
                 
                   
                     { 
                     
                       
                         
                           H 
                           
                             k 
                             - 
                             1 
                           
                         
                         ⁡ 
                         
                           ( 
                           s 
                           ) 
                         
                       
                       , 
                       
                           
                       
                       ⁢ 
                       
                         
                           η 
                           ⁢ 
                           
                              
                             
                               
                                 H 
                                 k 
                               
                               ⁡ 
                               
                                 ( 
                                 s 
                                 ) 
                               
                             
                              
                           
                         
                         + 
                         
                           
                             max 
                             
                               a 
                               ∈ 
                               A 
                             
                           
                           ⁢ 
                           
                             
                               H 
                               k 
                             
                             ⁡ 
                             
                               ( 
                               
                                 τ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     s 
                                     , 
                                     
                                         
                                     
                                     ⁢ 
                                     a 
                                   
                                   ) 
                                 
                               
                               ) 
                             
                           
                         
                       
                     
                     } 
                   
                   . 
                 
               
             
           
         
       
     
     More precisely, at step k (where H k−1  has just been learned), minimize the loss function 
     
       
         
           
             
               
                 ℓ 
                 
                   η 
                   , 
                   
                     δ 
                     1 
                   
                   , 
                   
                     δ 
                     2 
                   
                 
                 
                   ( 
                   
                     H 
                     
                       k 
                       - 
                       1 
                     
                   
                   ) 
                 
               
               ⁡ 
               
                 ( 
                 θ 
                 ) 
               
             
             = 
               
             ⁢ 
             
               
                 
                   ∑ 
                   
                     
                       x 
                       j 
                     
                     ∈ 
                     U 
                   
                 
                 ⁢ 
                 
                   max 
                   ⁢ 
                   
                     { 
                     
                       0 
                       , 
                       
                           
                       
                       ⁢ 
                       
                         
                           
                             h 
                             θ 
                           
                           ⁡ 
                           
                             ( 
                             
                               x 
                               j 
                             
                             ) 
                           
                         
                         + 
                         
                           δ 
                           1 
                         
                       
                       , 
                       
                         
                           - 
                           
                             
                               h 
                               θ 
                             
                             ⁡ 
                             
                               ( 
                               
                                 x 
                                 j 
                               
                               ) 
                             
                           
                         
                         - 
                         
                           δ 
                           2 
                         
                       
                     
                     } 
                   
                 
               
               + 
               
                 
                   ∑ 
                   
                     
                       x 
                       j 
                     
                     ∈ 
                     
                       K 
                       ′ 
                     
                   
                 
                 ⁢ 
                 
                   ( 
                   
                     
                       
                         h 
                         θ 
                       
                       ⁡ 
                       
                         ( 
                         
                           x 
                           j 
                         
                         ) 
                       
                     
                     - 
                     
                       
                         ( 
                         
                           1 
                           - 
                           γ 
                         
                         ) 
                       
                       ⁢ 
                       
                         
                           H 
                           
                             k 
                             - 
                             1 
                           
                         
                         ⁡ 
                         
                           ( 
                           
                             x 
                             j 
                           
                           ) 
                         
                       
                     
                     - 
                     
                       γ 
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       min 
                       ⁢ 
                       
                         
                           
                             { 
                             
                               
                                 
                                   H 
                                   
                                     k 
                                     - 
                                     1 
                                   
                                 
                                 ⁡ 
                                 
                                   ( 
                                   s 
                                   ) 
                                 
                               
                               , 
                               
                                   
                               
                               ⁢ 
                               
                                   
                                 
                                   
                                     η 
                                     ⁢ 
                                     
                                        
                                       
                                         
                                           h 
                                           θ 
                                         
                                         ⁡ 
                                         
                                           ( 
                                           
                                             x 
                                             j 
                                           
                                           ) 
                                         
                                       
                                        
                                     
                                   
                                   + 
                                   
                                     
                                       max 
                                       
                                         a 
                                         ∈ 
                                         A 
                                       
                                     
                                     ⁢ 
                                     
                                       
                                         h 
                                         θ 
                                       
                                       ⁡ 
                                       
                                         ( 
                                         
                                           τ 
                                           ⁡ 
                                           
                                             ( 
                                             
                                               
                                                 x 
                                                 j 
                                               
                                               , 
                                               
                                                   
                                               
                                               ⁢ 
                                               a 
                                             
                                             ) 
                                           
                                         
                                         ) 
                                       
                                     
                                   
                                 
                                 } 
                               
                             
                             ) 
                           
                           2 
                         
                         . 
                       
                     
                   
                 
               
             
           
         
       
     
     At a certain number of learning epochs as determined by the user and based on performance or other criteria, stop and define the current status of h θ  to be H k . After a number of iterations k=n of replacing L in the Bellman equation, this stage in the learning process is terminated. One thing to note is that one of the inputs to the procedure is an algorithm for computing exactly or else approximating the maximum over A in the above loss function (as mentioned above, this could be calculated directly when the set A is sufficiently small, but can also be approximated using a separate machine learning procedure in other cases). In the reduction to practice described below, A in the maximum is replaced by a fixed finite subset of A. However, note that the calculation of this maximum can be formulated itself as a problem of reinforcement learning and the techniques of that field can be applied to this end. Other approximations are also possible. The manner in which the above learning stage is carried out in practice will be evident to one skilled in the art. 
     The second and final stage in the learning process (i.e., Learning Stage Two  702 ) is used to refine h θ  to satisfy the barrier function properties more precisely by changing the loss function to 
     
       
         
           
             
               
                 ℓ 
                 
                   η 
                   , 
                   α 
                   , 
                   
                     δ 
                     1 
                   
                   , 
                   
                     δ 
                     2 
                   
                 
               
               ⁡ 
               
                 ( 
                 θ 
                 ) 
               
             
             = 
               
             ⁢ 
             
               
                 
                   ∑ 
                   
                     
                       x 
                       j 
                     
                     ∈ 
                     U 
                   
                 
                 ⁢ 
                 
                   max 
                   ⁢ 
                   
                     { 
                     
                       0 
                       , 
                       
                           
                       
                       ⁢ 
                       
                         
                           
                             h 
                             θ 
                           
                           ⁡ 
                           
                             ( 
                             
                               x 
                               j 
                             
                             ) 
                           
                         
                         + 
                         
                           δ 
                           1 
                         
                       
                       , 
                       
                         
                           - 
                           
                             
                               h 
                               θ 
                             
                             ⁡ 
                             
                               ( 
                               
                                 x 
                                 j 
                               
                               ) 
                             
                           
                         
                         - 
                         
                           δ 
                           2 
                         
                       
                     
                     } 
                   
                 
               
               + 
               
                 
                   ∑ 
                   
                     
                       x 
                       j 
                     
                     ∈ 
                     
                       K 
                       ′ 
                     
                   
                 
                 ⁢ 
                 
                   
                     ( 
                     
                       
                         ReLU 
                         ⁡ 
                         
                           ( 
                           
                             
                               - 
                               
                                 
                                   h 
                                   θ 
                                 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     x 
                                     j 
                                   
                                   ) 
                                 
                               
                             
                             - 
                             
                               δ 
                               2 
                             
                           
                           ) 
                         
                       
                       - 
                       
                         α 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           
                             h 
                             θ 
                           
                           ⁡ 
                           
                             ( 
                             
                               x 
                               j 
                             
                             ) 
                           
                         
                       
                       + 
                       
                         ReLU 
                         ⁡ 
                         
                           ( 
                           
                             
                               
                                 h 
                                 θ 
                               
                               ⁡ 
                               
                                 ( 
                                 
                                   x 
                                   j 
                                 
                                 ) 
                               
                             
                             - 
                             
                               η 
                               ⁢ 
                               
                                  
                                 
                                   
                                     h 
                                     θ 
                                   
                                   ⁡ 
                                   
                                     ( 
                                     
                                       x 
                                       j 
                                     
                                     ) 
                                   
                                 
                                  
                               
                             
                             - 
                             
                               
                                 
                                     
                                   max 
                                 
                                 
                                   α 
                                   ∈ 
                                   A 
                                 
                               
                               ⁢ 
                               
                                 
                                   h 
                                   θ 
                                 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     τ 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         
                                           x 
                                           j 
                                         
                                         , 
                                         a 
                                       
                                       ) 
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                           ) 
                         
                       
                     
                     ) 
                   
                   . 
                 
               
             
           
         
       
     
     This function more directly pushes h θ  toward satisfying the barrier function properties and generates as an output the approximate barrier function  706 . Note that the constant α is positive and close to 0, and the term −αh θ (x j ) encourages larger support {s∈S:h θ (s)&gt;0} with the idea of finding a larger globally safe set. Generally, it may be necessary to change a during the stage. In particular, a typically starts out at a larger value and decreases as time goes on. This completes the description of the learning process for discrete control systems. The manner in which the above learning stage is carried out in practice will be evident to one skilled in the art. 
     Described below is the barrier function learning process for continuous control systems. Assume the control system is defined by the system of differential equations {dot over (s)}=f(s,a) and the safe set K is as described above. Note that by defining a discrete approximation for the system, one can use either of the learning stages in the discrete case in the continuous case as well. Indeed, the transition function τ: S×A→S can be defined so that if one chooses action a∈A at state s∈S, then τ(s,a) is an approximation of the state of the system (which can be obtained by, e.g., the Euler or Runge-Kutta methods) after some fixed time gap. 
     After these stages have been utilized, the main learning stage is as follows. As before, let D={x j : j=1, . . . , N}⊆S be a data set, which is divided into instantaneously safe K′=D n K and unsafe U=D∩(S\K) data points; let h θ :S→  be a neural network with which to learn a barrier function; and let α,δ 1 , δ 2 &gt;0 be constants. Additionally, fix the barrier function constant κ&gt;0. The neural network h θ :S→  then learns according to the loss function 
     
       
         
           
             
               
                 ℓ 
                 
                   κ 
                   , 
                   α 
                   , 
                   
                     δ 
                     1 
                   
                   , 
                   
                     δ 
                     2 
                   
                 
                 cts 
               
               ⁡ 
               
                 ( 
                 θ 
                 ) 
               
             
             = 
             
               
                 
                   ∑ 
                   
                     
                       x 
                       j 
                     
                     ∈ 
                     U 
                   
                 
                 ⁢ 
                 
                   max 
                   ⁢ 
                   
                     { 
                     
                       0 
                       , 
                       
                           
                       
                       ⁢ 
                       
                         
                           
                             h 
                             θ 
                           
                           ⁡ 
                           
                             ( 
                             
                               x 
                               j 
                             
                             ) 
                           
                         
                         + 
                         
                           δ 
                           1 
                         
                       
                       , 
                       
                         
                           - 
                           
                             
                               h 
                               θ 
                             
                             ⁡ 
                             
                               ( 
                               
                                 x 
                                 j 
                               
                               ) 
                             
                           
                         
                         - 
                         
                           δ 
                           2 
                         
                       
                     
                     } 
                   
                 
               
               + 
               
                 
                   ∑ 
                   
                     
                       x 
                       j 
                     
                     ∈ 
                     
                       K 
                       ′ 
                     
                   
                 
                 ⁢ 
                 
                   
                     ( 
                     
                       
                         ReLU 
                         ⁡ 
                         
                           ( 
                           
                             
                               - 
                               
                                 max 
                                 
                                   a 
                                   ∈ 
                                   A 
                                 
                               
                             
                             ⁢ 
                             
                               { 
                               
                                 
                                   
                                     ( 
                                     
                                       
                                         ∇ 
                                         x 
                                       
                                       ⁢ 
                                       
                                         h 
                                         θ 
                                       
                                     
                                     ) 
                                   
                                   ⁢ 
                                   
                                     
                                       ( 
                                       
                                         x 
                                         j 
                                       
                                       ) 
                                     
                                     · 
                                     
                                       f 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           
                                             x 
                                             j 
                                           
                                           , 
                                           a 
                                         
                                         ) 
                                       
                                     
                                   
                                 
                                 + 
                                 
                                   κ 
                                   ⁢ 
                                   
                                      
                                     
                                       
                                         h 
                                         θ 
                                       
                                       ⁡ 
                                       
                                         ( 
                                         
                                           x 
                                           j 
                                         
                                         ) 
                                       
                                     
                                      
                                   
                                 
                               
                               } 
                             
                           
                           ) 
                         
                       
                       - 
                       
                         α 
                         ⁢ 
                         
                           
                             h 
                             θ 
                           
                           ⁡ 
                           
                             ( 
                             
                               x 
                               j 
                             
                             ) 
                           
                         
                       
                     
                     ) 
                   
                   . 
                 
               
             
           
         
       
     
     Similar to the second stage in the discrete case, this loss function directly pushes h θ  to satisfy the barrier function properties while the term −αh θ (x j ) encourages larger support. Also, the constant α may start out at a larger value and decrease as time goes on. This completes the description of the learning process in the continuous case. The manner in which the above learning stage is carried out in practice will be evident to one skilled in the art. 
     Next, described below are several example applications in which the system has been reduced to practice. In a first example, consider a control system (i.e., controlled dynamical system) with state space S=   2 , action space A=[−½, ½], transition function T:S×A→S defined by T((x,v), a)=(x+v, x+v+a), and instantaneously safe set K={(x,v)∈   2 :|x|≤1}. Intuitively, the triple (x,v,a) can be thought of as (position, velocity, acceleration) of an object, where the acceleration is controlled; and the point (x,v)=(0,0) is an unstable fixed point. The ground truth globally safe set can be shown to be {(x,v)∈   2 :|x+v|≤½}, and one can therefore accurately assess the effectiveness of the approach in this example. 
     The data set D used to learn an approximate barrier function consists of states from 10,000 trajectories of the system, each of which were terminated at the first state outside of K, and the initial point and all actions were chosen uniformly at random from {(x,v)∈   2 :|≤1, |v|≤1} and respectively. The transition function τ: S×A→S used in the loss function in the learning process was a learned approximation of T. The neural network h θ :   2 →  used to learn the approximate barrier function was fully connected, had hidden 2 layers with 20 nodes each, and had hyperbolic tangent nonlinearities. In this case, only the second stage of the discrete time barrier function learning process was used. In particular, the experiment minimized the loss function    η, α,δ     1     , δ     2   (θ), with η=0.5, δ 1 =0.01, δ 2 =0.1, and various values for α. The initial value for a was chosen to be sufficiently large so that h θ  learned to be generally positive in the instantaneously safe region of the state space. Subsequently a was periodically decreased in order for h θ  to better conform to the barrier function properties until    0.5,0,0,0.01,0.1 (θ) was brought to 0. For the purpose of computing the maximum in the loss function, A was restricted to the subset {−1,0,1}. The function h θ  resulting from the learning process is depicted in  FIG.  3   . Specifically,  FIG.  3    depicts the values of the learned barrier function  300  over the state space of the system together with the (analytically known) ground truth safety boundary  300  (i.e., the boundary between globally safe and non-globally safe states). In this case, the learned barrier function correctly under-approximates the actual safe set. 
     The process evaluated the accuracy of how well h θ  indicates what the globally safe region is since it is known. Specifically, the value of h θ  was computed over a 200×200 grid of points over the set of instantaneously safe states {(x,v)∈   2 :|x|≤1, |v|≤1}, and evaluated whether h θ  correctly identified the points as globally safe or unsafe, corresponding to h θ (s)&gt;0 and h θ (s)≤0, respectively. The statistics of this evaluation are given in the table  400  depicted in  FIG.  4   . 
     Notably, the false safe rate for h θ   402 , which is the percentage of globally unsafe points that are incorrectly indicated by h θ  as being globally safe, was 0%. This means that h θ  is particularly cautious and cannot mislead the control system into moving to an unsafe state. On the other hand, the false unsafe rate for h θ   404 , that is, the percentage of globally safe points that are incorrectly indicated by the barrier function as being globally unsafe, is 8.7%. This shows that h θ  is somewhat conservative. However, the table  400  compares the results of h θ  with some baseline methods of determining the safety of states. These methods take a given state, and generate some number of random simulations of a particular length starting at that state, and label a state as being globally safe if and only if at least one of the simulations did not terminate in an unsafe state. As the table  400  indicates, h θ  was more accurate at determining the globally safe states, while at the same time correctly identifying all unsafe states. The procedure was implemented in the Python programming language and the manner of implementation will be familiar to one skilled in the art. 
     For a second example, consider a double integrator, a continuous time dynamical system having state space S=   2 , action space A=[−1,1], has function f:S×A→   2  defined by f((x,v), a)=(v,a) so that the dynamics are given by the system of differential equations ({dot over (x)}, {dot over (v)})=(v,a), and instantaneously safe set K={(x,v)∈   2 :|x|≤1}. As in the previous example, (x,v,a) can be thought of as (position, velocity, acceleration). The ground truth globally safe set is known in this example as well; it can be shown to be {(x,v)∈   2 :|x|≤1, −√{square root over (2(x+1))}≤v≤√{square root over (2(1−x))}}. The data set D used to learn an approximate barrier function consists of a 240×800 grid of points that uniformly spans the set {(x,v)∈   2 :|x|≤1.2, |v|≤4}. The neural network h θ :   2 →  used was fully connected, had hidden 2 layers with 20 nodes each, and had hyperbolic tangent nonlinearities. For this case, the learning process consisted in minimizing the loss function    κ,α,δ     1     , δ     2     cts (θ) for θ=4, δ 1 =0.01, δ 2 =0.1, and various values for α. As in the previous example, a was initially set at a sufficiently large positive number, and was decreased throughout the process until    4,0,0.01,0.1   cts  (θ) was equal to 0. For the purpose of computing the maximum in the loss function, A was again restricted to the subset {−1,0,1}. The resulting function h θ  is depicted in  FIG.  5   . Specifically,  FIG.  5    depicts the values of the learned barrier function over the state space of the system and the corresponding ground truth safety boundary  500 . Note that the ground truth globally safe region is known in this case as well, and it is shown qualitatively that h θ  accurately identifies that region. 
     For a third and final example, consider the setting in which a vehicle is being driven in a straight road, with the goal of learning a barrier function indicating whether the vehicle will be able to remain in the lane. This situation was modeled as a discrete time control system as follows. Let the state space be 
               S   =     {         (     x   ,           ⁢   θ   ,           ⁢   s     )     ∈       ℝ   3     :          x        ≤   W         ,           ⁢          θ        ≤     π   2       ,     0   ≤   s   ≤     s   max         }       ,         
where W=4.2 m, and
 
                 s   max     =     6   ⁢   5   ⁢     m   s         .         
The variable x represents the location of the center of the vehicle on the lane; the span of the lane with respect to x is [−w, w], where w=3.7 m. Also, the variable θ is the angle between the direction the vehicle is facing and the direction parallel to the road; and s is the speed of the vehicle, which is assumed to be nonnegative and have maximum value s max . Additionally, let the vehicle&#39;s length and width be 4.12379394 m and 1.94305298 m; and for a given vehicle state (x, θ, s), one can easily compute the x-coordinates of the 4 corners of the vehicle, which are denoted as x ur , x ul , x lr , x ll . Next, let the action space be A={(ϕ, a)∈   2 :|ϕ|≤ϕ max , a min ≤a≤a max }, where
 
                 ϕ   max     =       50   ⁢           ⁢   π       1   ⁢   8   ⁢   0         ,       a   min     =       -     4   .   5       ⁢     m     s   2           ,       and   ⁢           ⁢     a   max       =       2   .   2     ⁢   5   ⁢       m     s   2       .               
The variable ϕ denotes the steering angle of the vehicle and a denotes the acceleration. Note that the steering angle must be restricted in such a way that the centripetal acceleration
 
               s   2     r         
(r being the radius of the vehicle path resulting from the steering angle), is bounded by the product μg of the coefficient of friction μ=0.7 between the vehicle and the road and gravitational acceleration
 
     
       
         
           
             g 
             = 
             
               
                 9 
                 . 
                 8 
               
               ⁢ 
               
                 
                   m 
                   
                     s 
                     2 
                   
                 
                 . 
               
             
           
         
       
     
     Let the instantaneously safe region be the set K={(x, θ, s)∈S:w≤x ur , x ul , x lr , x ll ≤w} of states where all four corners of the vehicle are in the lane. 
     Next, define the transition function τ:S×A→S so that at a given state (x, θ, s)∈S with a chosen action (ϕ, a)∈A, τ((x, θ, s), (ϕ, a)) is the state of the vehicle after a time gap, which was chosen to be ¼ s. Note that x can be analytically computed. The data set D used to learn an approximate barrier function was obtained by sampling 100,000 states uniformly at random over S. As in the other examples, the neural network h θ :   2 →  used was fully connected, had hidden 2 layers with 20 nodes each, and had hyperbolic tangent nonlinearities. For this example, both stages of the discrete time learning process were used. Let γ=0.75, η=0.5, δ 1 =0.01, and δ 2 =0.4, and define the initial function L:S→  by 
     
       
         
           
             
               L 
               ⁡ 
               
                 ( 
                 
                   x 
                   , 
                   
                       
                   
                   ⁢ 
                   θ 
                   , 
                   s 
                 
                 ) 
               
             
             = 
             
               1 
               - 
               
                 
                   max 
                   ⁢ 
                   
                     { 
                     
                       
                         x 
                         ur 
                         2 
                       
                       , 
                       
                         x 
                         ul 
                         2 
                       
                       , 
                       
                         x 
                         lr 
                         2 
                       
                       , 
                       
                         x 
                         ll 
                         2 
                       
                     
                     } 
                   
                 
                 
                   w 
                   2 
                 
               
               - 
               
                 
                   δ 
                   1 
                 
                 . 
               
             
           
         
       
     
     The learning process is started by minimizing the loss function    0.01,0.4   (L) (θ) so that h θ ≈L. The process was continued by performing several iterations of minimizing    0.5,0.01,0.4   (H     k−1     ) (θ) to push h θ  to be closer to satisfying the barrier function properties. Lastly, the process performed a number of steps of minimizing the loss function    0.5,α,0.01,0.4 (θ) (with α≈θ) until    0.5,0,0.01,0.4 (θ)=0 to push h θ  to satisfy the barrier function properties more precisely. For the purpose of computing the maximum in the loss function, ϕ was restricted to the set {0, ±3 −k ϕ max :k=0,1,2,3,4} and α to 
               {       a   min     ,       a   min     2     ,           ⁢   0   ,           ⁢     a   max       }     .         
Based on the results of a vehicle simulation, the resulting function h θ  empirically appears to successfully indicate all globally unsafe states, though is particularly cautious.  FIG.  6    depicts the value of h θ  (x, θ, s) with respect to (x, θ) under the assumption that
 
             s   =     1   ⁢   0   ⁢       m   s     .             
Specifically,  FIG.  6    depicts the values of the learned barrier function over a portion of the state space of the system.
 
     The neural network architecture used in this case is the same as in the foregoing examples. As a general rule of thumb one should regard the task of learning a barrier function for a particular dynamical system as being bounded below by reinforcement learning tasks for the same system. As such, architectures that are sufficient, for a fixed dynamical system, for reinforcement learning are a good starting point for barrier function learning architectures. Similar remarks apply to the question of data volume sufficient for learning barrier functions. 
     There are several additional options for the loss functions that can be used in the process. In particular, loss functions can be made that are alternatives to    η,α,δ     1     ,δ     2   (θ) and    κ, α,δ     1     , δ     2     cts (θ), which weight the terms that penalize violations of the barrier function condition by the value of h θ . This causes the learning to exhibit more stability in some cases. In addition, loss functions can be created for continuous time control systems that are formed from Bellman-like operators, analogously to    η,δ     1     ,δ     2     (H     k−1     ) (θ), that work directly with the continuous dynamics and do not require a discrete dynamics model for the system. These loss functions are to be used prior to the stage where    κ, α,δ     1     , δ     2     cts (θ) is used, and are better able to direct a neural network to learn an approximate barrier function at the beginning of the process. 
     (4.1) Control of a Device 
     As shown in  FIG.  8   , the system  100  may be used to control a device  800  (e.g., a mobile device display, a virtual reality display, an augmented reality display, a computer monitor, a motor, a machine, a drone, a camera, an autonomous vehicle, etc.) based on the barrier function. For example, the device  800  may be controlled to cause the device to move or otherwise initiate a physical action based on the barrier function. By way of example, if the barrier function determines, through being evaluated on the present (estimated) system state based on sensor readings and outputs of sensor fusion/state estimation algorithms employed on the platform, that the autonomous vehicle is projected to move out of a driving lane (state is unsafe), the system  100  will cause the vehicle to alter course (steer, brake, accelerate, etc.) to move and maintain the vehicle within the driving lane or otherwise safe state. Analogous examples can be imagined in cases where the barrier functions correspond to different safety properties. Additionally, the outputs of the barrier function could be used together in the loop with the control system as part of a learning process to better regulate or optimize the control of the system for safe performance. 
     Finally, while this invention has been described in terms of several embodiments, one of ordinary skill in the art will readily recognize that the invention may have other applications in other environments. It should be noted that many embodiments and implementations are possible. Further, the following claims are in no way intended to limit the scope of the present invention to the specific embodiments described above. In addition, any recitation of “means for” is intended to evoke a means-plus-function reading of an element and a claim, whereas, any elements that do not specifically use the recitation “means for”, are not intended to be read as means-plus-function elements, even if the claim otherwise includes the word “means”. Further, while particular method steps have been recited in a particular order, the method steps may occur in any desired order and fall within the scope of the present invention.