Patent Publication Number: US-2023142461-A1

Title: Tactical decision-making through reinforcement learning with uncertainty estimation

Description:
TECHNICAL FIELD 
     The present disclosure relates to the field of autonomous vehicles and in particular to a method of controlling an autonomous vehicle using a reinforcement learning agent. 
     BACKGROUND 
     The decision-making task of an autonomous vehicle is commonly divided into strategic, tactical, and operational decision-making, also called navigation, guidance and stabilization. In short, tactical decisions refer to high-level, often discrete, decisions, such as when to change lanes on a highway, or whether to stop or go at an intersection. This invention primarily targets the tactical decision-making field. 
     Tactical decision-making is challenging due to the diverse set of environments the vehicle needs to handle, the interaction with other road users, and the uncertainty associated with sensor information. To manually predict all situations that can occur and code a suitable behavior is infeasible. Therefore, it is an attractive option to consider methods that are based on machine learning to train a decision-making agent. 
     Conventional decision-making methods are based on predefined rules and implemented as hand-crafted state machines. Other classical methods treat the decision-making task as a motion planning problem. Although these methods are successful in many cases, one drawback is that they are designed for specific driving situations, which makes it hard to scale them to the complexity of real-world driving. 
     Reinforcement learning (RL) has previously been applied to decision-making for autonomous driving in simulated environments. See for example C. J. Hoel, K. Wolff and L. Laine, “Automated speed and lane change decision making using deep reinforcement learning”, Proceedings of the 21 st  International Conference on Intelligent Transportation Systems (ITSC), 4-7 Nov. 2018, pp. 2148-2155 [doi:10.1109/ITSC.2018.8569568]. However, the agents that were trained by RL in previous works can only be expected to output rational decisions in situations that are close to the training distribution. Indeed, a fundamental problem with these methods is that no matter what situation the agents are facing, they will always output a decision, with no suggestion or indication about the uncertainty of the decision or whether the agent has experienced anything similar during its training. If, for example, an agent that was trained for one-way highway driving was deployed in a scenario with oncoming traffic, it would still output decisions, without any warning that these are presumably of a much lower quality. A more subtle case of insufficient training is one where the agent that has been exposed to a nominal or normal highway driving environment is suddenly facing a speeding driver or an accident that creates standstill traffic. 
     A precaution that has been taken in view of such shortcomings is comprehensive real-world testing in confined environments, combined with successive refinements. Testing and refinement are iterated until the decision-making agent is seen to make an acceptably low level of observed errors and thus fit for use outside the testing environment. This is onerous, time-consuming and drains resources from other aspects of research and development. 
     SUMMARY 
     One objective of the present invention is to make available methods and devices for assessing the uncertainty of outputs of a decision-making agent, such as an RL agent. A particular objective would be to provide methods and devices by which the decision-making agent does not just output a recommended decision, but also estimates an uncertainty of this decision. Such methods and devices may preferably include a safety criterion that determines whether the trained decision-making agent is confident enough about a particular decision, so that - in the negative case - the agent can be overridden by a safety-oriented fallback decision. 
     These and other objectives are achieved by the invention according to the independent claims. The dependent claims define example embodiments of the invention. 
     In a first aspect of the invention, there is provided a method of controlling an autonomous vehicle using an RL agent. The method begins with a plurality of K training sessions in which the RL agent interacts with an environment including the autonomous vehicle, each training session having a different initial value and yielding a state-action value function Q k (s,a) (k = 1, ..., K) dependent on state s and action a. The training sessions can be performed sequentially with respect to time, or in parallel with each other. Next follows a decision-making stage, in which the RL agent outputs at least one tentative decision relating to the control of the autonomous vehicle. A tentative decision to perform action  
     
       
         
           
             a 
             ^ 
           
         
       
     
     in state  
     
       
         
           
             s 
             ^ 
           
         
       
     
     can be represented as state-action pair  
     
       
         
           
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             . 
           
         
       
     
     According to an embodiment, there follows an uncertainty estimation, which is performed on the basis of a variability measure for the K state-action value functions evaluated for the state-action pair  
     
       
         
           
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             , 
           
         
       
     
     that is, the variability of the K numbers  
     
       
         
           
             
               Q 
               1 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             , 
             
               Q 
               2 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             , 
             … 
             , 
             
               Q 
               K 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             . 
           
         
       
     
     Vehicle control is then based on this estimation, namely by executing the at least one tentative decision in dependence of the estimated uncertainty. 
     Generalizing this to the case where the RL agent outputs an arbitrary number L ≥ 1 of tentative decisions corresponding to possible actions  
     
       
         
           
             
               
                 a 
                 ^ 
               
               1 
             
               
             
               
                 a 
                 ^ 
               
               2 
             
             , 
             … 
             , 
               
             
               
                 a 
                 ^ 
               
               L 
             
           
         
       
     
     to be taken in a state  
     
       
         
           
             s 
             ^ 
           
         
       
     
     the K different state-action value functions will be evaluated for each pair  
     
       
         
           
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   
                     a 
                     ^ 
                   
                   l 
                 
               
             
           
         
       
     
     with 1 ≤ l ≤ L. 
     Accordingly, the uncertainty of a tentative decision to take action  
     
       
         
           
             a 
             ^ 
           
         
       
     
     in state  
     
       
         
           
             s 
             ^ 
           
         
       
     
     is assessed on the basis of a measure of the statistic variability of K observations being the K state-action value functions evaluated for this state-action pair:  
     
       
         
           
             
               Q 
               1 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             , 
             
               Q 
               2 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             , 
             … 
             , 
             
               Q 
               K 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             . 
           
         
       
     
     The uncertainty is assessed on the basis of the variability measure, that is, either by considering its value without processing or by considering a quantity derived from the variability measure, e.g., after normalization, scaling, combination with other relevant factors etc. Where there are multiple tentative decisions (L ≥ 2), multiple values of the variability measure are computed. Then, the tentative decision or decisions can be executed in dependence of the estimated uncertainty. When execution of the tentative decision is made dependent on the uncertainty - wherein possible outcomes may be non-execution or execution with additional safety-oriented restrictions - a desired safety level can be sustained. 
     As used herein, an “RL agent” may be understood as software instructions implementing a mapping from a state s to an action a. The term “environment” refers to a simulated or real-world environment, in which the autonomous vehicle - or its model/avatar in the case of a simulated environment - operates. A mathematical model of the RL agent’s interaction with an “environment” in this sense is given below. A “variability measure” includes any suitable measure for quantifying statistic dispersion, such as a variance, a range of variation, a deviation, a variation coefficient, an entropy etc. Generally, all terms used in the claims are to be construed according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to “a/an/the element, apparatus, component, means, step, etc.” are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated. 
     In one embodiment, a tentative decision is executed only if the estimated uncertainty is less than a predefined threshold. This embodiment may impose a condition requiring the uncertainty to be below a tolerable threshold in order for the tentative decision to be executed. This serves to inhibit execution of uncertain decisions, which tend to be unsafe decisions, and is therefore in the interest of road safety. 
     As noted previously, while the present embodiment quantizes the estimated uncertainty into a binary variable, other embodiments may treat the estimated uncertainty as a continuous variable, which may guide the quantity of additional safety measures necessary to achieve a desired safety standard, e.g., a maximum speed or traffic density at which the tentative decision shall be considered safe to execute. 
     In one embodiment, where multiple tentative decisions by the RL agent are available (L ≥ 2), the tentative decisions are ordered sequentially and evaluated with respect to their estimated uncertainties. The method may apply the rule that the first tentative decision in the sequence which is found to have an estimated uncertainty below the predefined threshold shall be executed. While this may imply that a tentative decision which is located late in the sequence is not executed even though its estimated uncertainty is below the predefined threshold, this remains one of several possible ways in which the tentative decisions can be “executed in dependence of the estimated uncertainty” in the sense of the claims. An advantage with this embodiment is that an executable tentative decision is found without having to evaluate all available tentative decision with respect to uncertainty. 
     In a further development of the preceding embodiment, a fallback decision is executed if the sequential evaluation does not return a tentative decision to be executed. For example, if the last tentative decision in the sequence is found to have too large uncertainty, the fallback decision is executed. The fallback decision may be safety-oriented, which benefits road safety. At least in tactical decision-making, the fallback decision may include taking no action. To illustrate, if all tentative decisions achieving an overtaking of a slow vehicle ahead are found to be too uncertain, the fallback decision may be to not overtake the slow vehicle. 
     In various embodiments, the RL agent may be implemented by at least one neural network. In particular, K neural networks may be utilized to perform the K training sessions. Each of the K neural networks may be initialized with an independently sampled set of weights. 
     The invention is not dependent on the specific type of RL agent but can be embodied using a policy-based or value-based RL agent. Specifically, the RL agent may include a policy network and a value network. The RL agent may be obtained by a policy-gradient algorithm, such as an actor-critic algorithm. As another example, the RL agent is a Q-learning agent, such as a deep Q network (DQN). 
     In a second aspect of the invention, there is provided an arrangement for controlling an autonomous vehicle. The arrangement, which may correspond to functional or physical components of a computer or distributed computing system, includes processing circuitry and memory which implement an RL agent configured to interact with an environment including the autonomous vehicle in a plurality of training sessions, each training session having a different initial value and yielding a state-action value function Q k  (s, a) dependent on state and action. The RL agent is further configured to output at least one tentative decision relating to control of the autonomous vehicle. The processing circuitry and memory further implement an uncertainty estimator configured to estimate an uncertainty on the basis of a variability measure for the plurality of state-action value functions evaluated for a state-action pair corresponding to each of the tentative decisions by the RL agent. The arrangement further comprises a vehicle control interface, which is configured to control the autonomous vehicle by executing the at least one tentative decision in dependence of the estimated uncertainty. 
     In a third aspect, the invention provides a computer program for executing the vehicle control method on an arrangement with these characteristics. The computer program may be stored or distributed on a data carrier. As used herein, a “data carrier” may be a transitory data carrier, such as modulated electromagnetic or optical waves, or a non-transitory data carrier. Non-transitory data carriers include volatile and non-volatile memories, such as permanent and non-permanent storages of the magnetic, optical or solid-state type. Still within the scope of “data carrier”, such memories may be fixedly mounted or portable. 
     The arrangement according to the second aspect of the invention and the computer program according to the third aspect have same or similar effects and advantages as the method according to the first aspect. The embodiments and further developments described above in method terms are equally applicable to the second and third aspects. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Embodiments of the invention are described, by way of example, with reference to the accompanying drawings, on which: 
         FIG.  1    is a flowchart of a method according to an embodiment of the invention; 
         FIG.  2    is a block diagram of an arrangement for controlling an autonomous vehicle according to another embodiment of the invention; 
         FIG.  3    shows an architecture of a neural network of an RL agent; and 
         FIG.  4    is a plot of the mean uncertainty of a chosen action over 5 million training steps in an example. 
     
    
    
     DETAILED DESCRIPTION 
     The aspects of the present invention will now be described more fully with reference to the accompanying drawings, on which certain embodiments of the invention are shown. These aspects may, however, be embodied in many different forms and the described embodiments should not be construed as limiting; rather, they are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of invention to those skilled in the art. 
     Reinforcement learning is a subfield of machine learning, where an agent interacts with some environment to learn a policy π(s) that maximizes the future expected return. Reference is made to the textbook R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2 nd  ed., MIT Press (2018). 
     The policy π(s) defines which action a to take in each state s. When an action is taken, the environment transitions to a new state s′ and the agent receives a reward r. The reinforcement learning problem can be modeled as a Markov Decision Process (MDP), which is defined by the tuple (S; A; T; R; γ), where S is the state space, A is the action space, T is a state transition model (or evolution operator), R is a reward model, and y is a discount factor. This model can also be considered to represent the RL agent’s interaction with the training environment. At every time step t, the goal of the agent is to choose an action a that maximizes the discounted return,  
     
       
         
           
             
               R 
               t 
             
             = 
             
               
                 ∑ 
                 
                   k 
                   = 
                   0 
                 
                 ∞ 
               
               
                 
                   γ 
                   k 
                 
                 
                   r 
                   
                     t 
                     + 
                     k 
                   
                 
                 . 
               
             
           
         
       
     
      In Q-learning, the agent tries to learn the optimal action-value function Q*(s, a), which is defined as   
     
       
         
           
             
               Q 
               ∗ 
             
             
               
                 s 
                 , 
                 a 
               
             
             = 
               
             
               
                 max 
               
               π 
             
             E 
             
               
                 
                   R 
                   t 
                 
                 | 
                 
                   s 
                   t 
                 
                 = 
                 s 
                 , 
                 
                   a 
                   t 
                 
                 = 
                 a 
                 , 
                 π 
               
             
             . 
           
         
       
     
      From the optimal action-value function, the policy is derived as per 
     
       
         
           
             π 
             
               s 
             
             = 
               
               
             
               
                 arg 
                 max 
               
               a 
             
               
             
               Q 
               ∗ 
             
             ( 
             s 
             , 
             a 
             ) 
             . 
           
         
       
     
     An embodiment of the invention is illustrated by  FIG.  1   , which is a flowchart of a method  100  for controlling an autonomous vehicle by means of an RL agent. In the embodiment illustrated, the method begins with a plurality of training sessions  110 - 1 ,  110 - 2 , ...,  110 -K (K ≥ 2), which may be carried out in a simultaneous or at least time-overlapping fashion. In each training session, the RL agent interacts with an environment which has its own initial value and includes the autonomous vehicle (or, if the environment is simulated, a model of the vehicle). The k th  training session returns a state-action value function Q k  (s, a), for any 1 ≤ k ≤ K, from which a decision-making policy may be derived in the manner described above. Preferably, all K state-action value functions are combined into a common state-action value function  
     
       
         
           
             
               Q 
               ¯ 
             
             
               
                 s 
                 , 
                 a 
               
             
             , 
           
         
       
     
      which may represent a central tendency of the state-action value functions, such as a mean value, of the K state-action values: 
     
       
         
           
             Q 
             
               
                 s 
                 , 
                 a 
               
             
             = 
             
               1 
               K 
             
             
               
                 ∑ 
                 
                   k 
                   = 
                   1 
                 
                 K 
               
               
                 
                   Q 
                   k 
                 
                 
                   
                     s 
                     , 
                     a 
                   
                 
                 . 
               
             
           
         
       
     
     The inventors have realized that the uncertainty of a tentative decision corresponding to the state-action pair  
     
       
         
           
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
           
         
       
     
      can be estimated on the basis of a variability of the numbers  
     
       
         
           
             
               Q 
               1 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             , 
             
               Q 
               2 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             , 
             … 
             , 
             
               Q 
               K 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
             . 
           
         
       
     
      The variability may be measured as the standard deviation, coefficient of variation (i.e., standard deviation normalized by mean), variance, range, mean absolute difference or the like. The variability measure is denoted  
     
       
         
           
             
               c 
               v 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   a 
                   ^ 
                 
               
             
           
         
       
     
      in this disclosure, whichever definition is used. 
     In the illustrated method  100 , therefore, a decision-making step  112 , in which the RL agent outputs at least one tentative decision  
     
       
         
           
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   
                     a 
                     ^ 
                   
                   l 
                 
               
             
             , 
               
             1 
             ≤ 
             l 
             ≤ 
             L 
           
         
       
     
      with L ≥ 1, relating to control of the autonomous vehicle, is followed by a third step  114  of estimating the uncertainty of this tentative decision or decisions on the basis of the variability measure 
     
       
         
           
             
               c 
               v 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   
                     a 
                     ^ 
                   
                   l 
                 
               
             
             . 
           
         
       
     
     In a fourth step  116 , the L ≥ 1 tentative decisions are put to use in dependence of their respective estimated uncertainties, i.e., on the basis of  
     
       
         
           
             
               c 
               v 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   
                     a 
                     ^ 
                   
                   l 
                 
               
             
             
               
                 1 
                 ≤ 
                 l 
                 ≤ 
                 L 
               
             
           
         
       
     
      for the purpose of controlling the autonomous vehicle. A relatively high value of the variability measure  
     
       
         
           
             
               c 
               v 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   
                     a 
                     ^ 
                   
                   l 
                 
               
             
           
         
       
     
      indicates that the RL agent is far from the training distribution and, thus, that the tentative decision  
     
       
         
           
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   
                     a 
                     ^ 
                   
                   l 
                 
               
             
           
         
       
     
      may be regarded as relatively unsafe. For example, one may choose to execute the l th  decision only if the variability is less than a threshold C v , that is, only if 
     
       
         
           
             
               c 
               v 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 
                   
                     a 
                     ^ 
                   
                   l 
                 
               
             
             &lt; 
             
               C 
               v 
             
             . 
           
         
       
     
     To find a single executable action  
     
       
         
           
             a 
             ^ 
           
         
       
     
      for state  
     
       
         
           
             
               s 
               ^ 
             
             , 
           
         
       
     
      one may maximize the mean Q value subject to the above threshold condition: 
     
       
         
           
             
               
                 arg 
                 ​ 
                 max 
               
               a 
             
             
               Q 
               ¯ 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 a 
               
             
             , 
           
         
       
     
     
       
         
           
             subject 
               
             to 
               
             
               c 
               v 
             
             
               
                 
                   s 
                   ^ 
                 
                 , 
                 a 
               
             
             &lt; 
             
               C 
               v 
             
             . 
           
         
       
     
     The threshold C v  may represent a desired safety level at which the autonomous vehicle is operated. It may have been determined or calibrated by traffic testing and may be based on the frequency of decisions deemed erroneous, collisions, near-collisions, road departures and the like. 
     It is noted that the variability measure need not be actually computed between the second  112  and fourth  116  steps of the method  100 . A feasible alternative would be to pre-compute the variability for all possible state-action pairs, or for a subset of all possible state-action, and store this. Example subsets include the state-action pairs that a programmer expects may be relevant during driving or the state-action pairs recorded during simulated or real-world test drives. When variability measures have been pre-computed for merely a subset of the possible state-action values, the set of pre-computed values may need to be supplemented by processing capacity allowing to add a missing variability measure value during operation. The set of pre-computed values need not be updated as long as the agent is unchanged; this may be the case for as long as the agent does not undergo additional training and/or is reconfigured. 
       FIG.  2    illustrates an arrangement  200  for controlling an autonomous vehicle  299  according to another embodiment of the invention. The autonomous vehicle  299  may be any road vehicle or vehicle combination, including trucks, buses, construction equipment, mining equipment and other heavy equipment operating in public or non-public traffic. The arrangement  200  maybe provided, at last partially, in the autonomous vehicle  299 . The arrangement  200 , or portions thereof, may alternatively be provided as part of a stationary or mobile controller (not shown), which communicates with the vehicle  299  wirelessly. 
     The arrangement  200  includes processing circuitry  210 , a memory  212  and a vehicle control interface  214 . The vehicle control interface  214  is configured to control the autonomous vehicle  299  by transmitting wired or wireless signals, directly or via intermediary components, to actuators (not shown) in the vehicle. In a similar fashion, the vehicle control interface  214  may receive signals from physical sensors (not shown) in the vehicle so as to detect current conditions of the driving environment or internal states prevailing in the vehicle  299 . The processing circuitry  210  implements an RL agent  220  and an uncertainty estimator  222  to be described next. 
     The RL agent  220  interacts with an environment including the autonomous vehicle in a plurality of training sessions, each training session having a different initial value and yielding a state-action value function dependent on state and action. The RL agent  220  then outputs at least one tentative decision relating to control of the autonomous vehicle. The RL agent  220  may, at least during the training phase, comprise as many sub-agents as there are training sessions, each sub-agent corresponding to a state-action value function Qk(s, a). The sub-agents maybe combined into a joint RL agent, corresponding to a common state-action value function  
     
       
         
           
             
               Q 
               ¯ 
             
             
               
                 s 
                 , 
                 a 
               
             
             , 
           
         
       
     
      for the purpose of the decision-making. 
     The uncertainty estimator  222  is configured to estimate an uncertainty on the basis of a variability measure for the plurality of state-action value functions evaluated for a state-action pair corresponding to each of the tentative decisions by the RL agent. The outcome is utilized by the vehicle control interface  214  which, in this embodiment, is configured to control the autonomous vehicle  299  by executing the at least one tentative decision in dependence of the estimated uncertainty. 
     Returning to the description of the invention from a mathematical viewpoint, an embodiment relies on the DQN algorithm. This algorithm uses a neural network with weights θ to approximate the optimal action-value function as Q*(s, a) ≈Q(s, a; θ); see further V. Mnih et al., “Human-level control through deep reinforcement learning”,  Nature , vol. 518, pp. 529-533 (2015) [doi:10.1038/nature14236.]. Since the action-value function follows the Bellman equation, the weights can be optimized by minimizing the loss function  
     
       
         
           
             L 
             
               θ 
             
             = 
             E 
             
               
                 
                   
                     
                       
                         r 
                         + 
                         γ 
                         
                           
                             max 
                           
                           a 
                         
                         Q 
                         
                           
                             
                               s 
                               ′ 
                             
                             , 
                             
                               a 
                               ′ 
                             
                             : 
                             
                               θ 
                               − 
                             
                           
                         
                         − 
                         Q 
                         
                           
                             s 
                             , 
                             a 
                             ; 
                             θ 
                           
                         
                       
                     
                   
                   2 
                 
               
             
           
         
       
     
      As explained in Mnih, the loss is calculated for a minibatch M and the weights θ  -  of a target network are updated repeatedly. 
     The DQN algorithm returns a maximum likelihood estimate of the Q values but gives no information about the uncertainty of the estimation. The risk of an action could be represented as the variance of the return when taking that action. One line of RL research focuses on obtaining an estimate of the uncertainty by statistical bootstrap; an ensemble of models is then trained on different subsets of the available data and the distribution that is given by the ensemble is used to approximate the uncertainty. A sometimes better-performing Bayesian posterior is obtained if a randomized prior function (RPF) is added to each ensemble member; see for example I. Osband, J. Aslanides and A. Cassirer, “Randomized prior functions for deep reinforcement learning,” in: S. Bengjo et al. (eds.), Adv. in Neural Inf. Process. Syst. 31 (2018), pp. 8617-8629. When RPF is used, each individual ensemble member, here indexed by k, estimates the Q values as the sum  
     
       
         
           
             
               Q 
               k 
             
             
               
                 s 
                 , 
                 a 
               
             
             = 
             f 
             
               
                 s 
                 , 
                 a 
                 ; 
                 
                   θ 
                   k 
                 
               
             
             + 
             β 
             p 
             
               
                 s 
                 , 
                 a 
                 ; 
                 
                   
                     θ 
                     ^ 
                   
                   k 
                 
               
             
             , 
           
         
       
     
      where f, p are neural networks, with parameters θ k  that can be trained and further parameters  θ   k  that are kept fixed. The factor β can be used to tune the importance of the prior function. When adding the prior, the loss function L(θ) defined above changes into 
     
       
         
           
             L 
             ​ 
             
               
                 
                   θ 
                   k 
                 
                 ​ 
               
             
             = 
             
               E 
               M 
             
             ​ 
             
               
                 
                   
                     
                       
                         r 
                         + 
                         γ 
                         
                           
                             max 
                           
                           
                             a 
                             ′ 
                           
                         
                         
                           
                             
                               f 
                               
                                 
                                   θ 
                                   k 
                                   − 
                                 
                               
                             
                             + 
                             β 
                             
                               p 
                               
                                 
                                   
                                     θ 
                                     ^ 
                                   
                                   k 
                                 
                               
                             
                           
                         
                         
                           
                             
                               s 
                               ′ 
                             
                             , 
                             
                               a 
                               ′ 
                             
                           
                         
                         − 
                         
                           
                             
                               f 
                               
                                 
                                   θ 
                                   k 
                                   − 
                                 
                               
                             
                             + 
                             β 
                             
                               p 
                               
                                 
                                   
                                     θ 
                                     ^ 
                                   
                                   k 
                                 
                               
                             
                           
                         
                         
                           
                             s 
                             , 
                             a 
                           
                         
                       
                     
                   
                   2 
                 
               
             
             . 
           
         
       
     
     The full ensemble RPF method, which was used in this implement-tation, may be represented in pseudo-code as Algorithm 1: 
     
       
         
           
               
               
             
               
                 Algorithm 1 Ensemble RPF training process 
               
             
            
               
                 1: 
                 for k ← 1 to K 
               
               
                 2: 
                  Initialize θ k  and  θ   k  randomly 
               
               
                 3: 
                  m k  ←{} 
               
               
                 4: 
                 i ← 0 
               
               
                 5: 
                 while networks not converged 
               
               
                 6: 
                  s i  ←initial random state 
               
               
                 7: 
                  k~U{1, K} 
               
               
                 8: 
                  while episode not finished 
               
               
                 9: 
                   a i  ←arg max a  Q  k ( s i,  a) 
               
               
                 10: 
                   s i +1,r i ←STEPENVIRONMENT(s i, a i ) 
               
               
                 11: 
                   for k ←1 to K 
               
               
                 12: 
                    if p ~ U(0, 1) &lt; Padd 
               
               
                 13: 
                     m k ←m k ∪ {(s i , a i , r i ,s i+1 )} 
               
               
                 14: 
                    M ←sample from m k 
 
               
               
                 15: 
                    update θ k  with SGD and loss L(θ k ) 
               
               
                 16: 
                   i ← i + 1 
               
            
           
         
       
     
      In the pseudo-code, the function StepEnvironment corresponds to a combination of the reward model R and state transition model T discussed above. The notation k ~ U{1, K} refers to sampling of an integer k from a uniform distribution over the integer range [1, K], and p ~ U (0, 1) denotes sampling of a real number from a uniform distribution over the open interval (0,1). 
     Here, an ensemble of K trainable neural networks and K fixed prior networks are first initialized randomly. A replay memory is divided into K parallel buffers m k , for the individual ensemble members (although in practice, this can be implemented in a memory-efficient way that uses a negligible amount of more memory than a single replay memory). To handle exploration, a random ensemble member is chosen for each training episode. Actions are then taken by greedily maximizing the Q value of the chosen ensemble member, which corresponds to a form of approximate Thompson sampling. The new experience (s i , a i , r i , s i+1 ) is then added to each ensemble buffer with probability Padd•Finally, a minibatch M of experiences is sampled from each ensemble buffer and the trainable network parameters of the corresponding ensemble member are updated by stochastic gradient descent (SGD), using the second definition of the loss function given above. 
     The presented ensemble RPF algorithm was trained in a one-way, three-lane highway driving scenario using the Simulation of Urban Mobility (SUMO) traffic simulator. The vehicle to be controlled (ego vehicle) was a 16 m long truck-trailer combination with a maximum speed of 25 m/s. In the beginning of each episode, 25 passenger cars were inserted into the simulation, with a random desired speed in the range 15 to 35 m/s. In order to create interesting traffic situations, slower vehicles were positioned in front of the ego vehicle, and faster vehicles were placed behind the ego vehicle. Each episode was terminated after N = 100 timesteps, or earlier if a collision occurred or the ego vehicle drove off the road. The simulation time step was set to Δt = 1 s. The passenger vehicles were controlled by the standard SUMO driver model, which consists of an adaptive cruise controller for the longitudinal motion and a lane-change model that makes tactical decisions to overtake slower vehicles. In the scenarios considered here, no strategical decisions were necessary, so the strategical part of the lane-changing model was turned off. Furthermore, in order to make the traffic situations more demanding, the cooperation level of the lane changing model was set to zero. Overtaking was allowed both on the left and right side of another vehicle, and each lane change took 4 s to complete. This environment was modeled by defining a corresponding state space S, action space A, state transition model T, and reward R. 
       FIG.  3    illustrates the architecture of the neural network is used in this embodiment. The architecture includes a temporal convolutional neural network (CNN) architecture, which makes the training faster and, at least in some use cases, gives better results than a standard fully connected (FC) architecture. By applying CNN layers and a max pooling layer to the part of the input that describes the surrounding vehicles, the output of the network becomes independent of the ordering of the surrounding vehicles in the input vector, and the architecture allows a varying input vector size. Rectified linear units (ReLUs) are used as activation functions for all layers, except the last, which has a linear activation function. The architecture also includes a dueling structure that separates the state value V(s) and action advantage A(s, a) estimation. 
     In an example, the RL agent was trained in the simulated environment described above. After every 50000 added training samples, henceforth called training steps, the agent was evaluated on 100 different test episodes. These test episodes were randomly generated in the same way as the training episodes, but not present during the training. The test episodes were also kept identical for all the test phases. The safety criterion c v (s, a) &lt; C v  was not active in the test episodes but used when the fully trained agent was exposed to unseen scenarios. 
     To gain insight into how the uncertainty estimation evolves during the training process, and to illustrate how to set the uncertainty threshold parameter C v ,  FIG.  4    shows the coefficient of variation C v  for the chosen action during the test episodes as a function of the number of training steps (scale in millions of steps). Each plotted value is an average over the 100 test episodes of that test phase.  FIG.  4    shows the uncertainty of the chosen action, whereas the uncertainty for not-chosen actions may be higher. After around four million training steps, the coefficient of variation settles at around 0.01, with a small spread in values, which may justify setting the threshold C v  = 0.02. 
     To assess the ability of the RPF ensemble agent to cope with unseen situations, the agent obtained after five million training steps was deployed in scenarios that had not been included in the training episodes. In various situations that involved an oncoming vehicle, the uncertainty estimate was consistently high, c v  ≈ 0.2. The fact that this value is one level of magnitude above the proposed value of the threshold C v  = 0.02, along with several further examples, suggests that the safety criterion c v (s, a) &lt; C v  is a robust and reliable safeguard against decision-making for which the agent has not been adequately trained. 
     The aspects of the present disclosure have mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims. In particular, the disclosed approach to estimating the uncertainty of a decision by an RL agent is applicable in machine learning more generally, also outside of the field of autonomous vehicles, and may be advantageous wherever the reliability of a decision is expected to influence personal safety, material values, information quality, user experience and the like.