Patent Publication Number: US-2020302322-A1

Title: Machine learning system

Description:
TECHNICAL FIELD 
     This invention is in the field of machine learning systems. One aspect of the invention has particular applicability to decision making utilising reinforcement learning algorithms. Another aspect of the invention concerns improving a probabilistic model utilised when simulating an environment for a reinforcement learning system. 
     BACKGROUND 
     Machine learning involves a computer system learning what to do by analysing data, rather than being explicitly programmed what to do. While machine learning has been investigated for over fifty years, in recent years research into machine learning has intensified. Much of this research has concentrated on what are essentially pattern recognition systems. 
     In addition to pattern recognition, machine learning can be utilised for decision making. Many uses of such decision making have been put forward, from managing a fleet of taxis to controlling non-playable characters in a computer game. The practical implementation of such decision making presents many technical problems. 
     SUMMARY 
     According to one aspect, there is provided a machine learning system comprising a first subsystem and a second subsystem remote from the first subsystem. The first subsystem comprises an environment having multiple possible states and a decision making subsystem comprising one or more agents. Each agent is arranged to receive state information indicative of a current state of the environment and to generate an action signal dependent on the received state information and a policy associated with that agent, the action signal being operable to cause a change in a state of the environment. Each agent is further arranged to generate experience data dependent on the received state information and information conveyed by the action signal. The first subsystem includes a first network interface configured to send said experience data to the second subsystem and to receive policy data from the second subsystem. The second subsystem comprises: a second network interface configured to receive experience data from the first subsystem and send policy data to the first subsystem; and a policy learner configured to process said received experience data to generate said policy data, dependent on the experience data, for updating one or more policies associated with the one or more agents. The decision making subsystem is operable to update the one or more policies associated with the one or more agents in accordance with policy data received from the second subsystem. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Various embodiments of the invention will now be described with reference to the accompanying figures, in which: 
         FIG. 1  schematically shows a process in which a single agent interacts with an environment in a reinforcement learning problem; 
         FIG. 2  schematically shows a process in which two autonomous agents interact with an environment in a reinforcement learning problem; 
         FIG. 3  is a schematic diagram showing examples of policy updates for three different configurations of agents; 
         FIG. 4A  is a schematic diagram showing the main components of a data processing system according to an embodiment of the invention; 
         FIG. 4B  is a schematic diagram showing the policy learning subsystem of the system of  FIG. 4A ; 
         FIG. 4C  is a schematic diagram showing the model input subsystem of the system of  FIG. 4A ; 
         FIG. 4D  is a schematic diagram showing the model learning subsystem of the system of  FIG. 4A ;  FIG. 5  is a flow diagram representing operations of the data processing system of  FIG. 4A ; 
         FIG. 6  is a schematic diagram of a deep neural network (DNN) used by in the data processing system of  FIG. 4A ; 
         FIG. 7  is a flow diagram showing operations of the DNN of  FIG. 6  to learn an approximate state value function; 
         FIG. 8  shows graphs of a prior distribution and a posterior distribution for a one-dimensional function; 
         FIG. 9  is a flow diagram representing operations to generate a probabilistic model according to an embodiment of the invention; 
         FIG. 10  is a schematic diagram showing an example of a transition system; 
         FIG. 11  is a schematic diagram of a server used to implement a learning subsystem for a correctness by learning algorithm. 
         FIG. 12  is a schematic diagram of a deep neural network (DNN) configured for use in a correctness by learning algorithm; 
         FIG. 13  is a schematic diagram of an alternative deep neural network (DNN) configured for use in a correctness by learning algorithm; 
         FIG. 14  is a schematic diagram of a user device used to implement an interaction subsystem for a correctness by learning algorithm. 
         FIG. 15  is a flow diagram representing a routine performed by a data processing system to implement a correctness by learning algorithm 
     
    
    
     DETAILED DESCRIPTION 
     Reinforcement Learning: Definitions and Formulation 
     For the purposes of the following description and accompanying drawings, a reinforcement learning problem is definable by specifying the characteristics of one or more agents and an environment. The methods and systems described herein are applicable to a wide range of reinforcement learning problems, including both continuous and discrete high-dimensional state and action spaces. However, an example of a specific problem, namely managing a fleet of taxis in a city, is referred to frequently for illustrative purposes and by way of example only. 
     A software agent, referred to hereafter as an agent, is a computer program component that makes decisions based on a set of input signals and performs actions based on these decisions. In some applications of reinforcement learning, each agent represents a real-world entity. In a first example of managing a fleet of taxis in a city, an agent is be assigned to represent each individual taxi in the fleet. In a second example of managing a fleet of taxis, an agent is assigned to each of several subsets of taxis in the fleet. In other applications of reinforcement learning, an agent does not represent a real-world entity. For example, an agent can be assigned to a non-playable character (NPC) in a video game. In another example, an agent is used to make trading decisions based on financial input data. Furthermore, in some examples agents send control signals to real world entities. In some examples, an agent is implemented in software or hardware that is part of the real world entity (for example, within an autonomous robot). In other examples, an agent is implemented by a computer system that is remote from the real world entity. 
     An environment is a virtual system with which agents interact, and a complete specification of an environment is referred to as a task. In many practical examples of reinforcement learning, the environment simulates a real-world system, defined in terms of information deemed relevant to the specific problem being posed. In the example of managing a fleet of taxis in a city, the environment is a simulated model of the city, defined in terms of information relevant to the problem of managing a fleet of taxis, including for example at least some of: a detailed map of the city; the location of each taxi in the fleet; information representing variations in time of day, weather, and season; the mean income of households in different areas of the city; the opening times of shops, restaurants and bars; and information about traffic. 
     It is assumed that interactions between an agent and an environment occur at discrete time steps n=0, 1, 2, 3, . . . . The discrete time steps do not necessarily correspond to times separated by fixed intervals. At each time step, the agent receives data corresponding to an observation of the environment and data corresponding to a reward. The data corresponding to an observation of the environment may also include data indicative of probable future states, and the sent data is referred to as a state signal and the observation of the environment is referred to as a state. The state perceived by the agent at time step n is labelled S n . The state observed by the agent may depend on variables associated with the agent itself. For example, in the taxi fleet management problem, the state observed by an agent representing a taxi can depend on the location of the taxi. 
     In response to receiving a state signal indicating a state S n  at a time step n, an agent is able to select and perform an action A n  from a set of available actions in accordance with a Markov Decision Process (MDP). In some examples, the true state of the environment cannot be ascertained from the state signal, in which case the agent selects and performs the Action A n  in accordance with a Partially-Observable Markov Decision Process (PO-MDP). Performing a selected action generally has an effect on the environment. Data sent from an agent to the environment as an agent performs an action is referred to as an action signal. At a later time step n+1 , the agent receives a new state signal from the environment indicating a new state S n+1 . The new state signal may either be initiated by the agent completing the action A n , or in response to a change in the environment. In the example of managing a fleet of taxis, an agent representing a particular taxi may receive a state signal indicating that the taxi has just dropped a passenger at a point A in the city. Examples of available actions are then: to wait for passengers at A; to drive to a different point B; and to drive continuously around a closed loop C of the map. Depending on the configuration of the agents and the environment, the set of states, as well as the set of actions available in each state, may be finite or infinite. The methods and systems described herein are applicable in any of these cases. 
     Having performed an action A n , an agent receives a reward signal corresponding to a numerical reward R n+1 , where the reward R n+1  depends on the state S n , the action A n  and the state S n+1 . The agent is thereby associated with a sequence of states, actions and rewards (S n , A n , R n+1 , S n+1 , . . . ) referred to as a trajectory T. The reward is a real number that may be positive, negative, or zero. In the example of managing a fleet of taxis in a city, a possible strategy for rewards to be assigned is for an agent representing a taxi to receive a positive reward each time a customer pays a fare, the reward being proportional to the fare. Another possible strategy is for the agent to receive a reward each time a customer is picked up, the value of the reward being dependent on the amount of time that elapses between the customer calling the taxi company and the customer being picked up. An agent in a reinforcement learning problem has an objective of maximising the expectation value of a return, where the value of a return G n  at a time step n depends on the rewards received by the agent at future time steps. For some reinforcement learning problems, the trajectory T is finite, indicating a finite sequence of time steps, and the agent eventually encounters a terminal state S T  from which no further actions are available. In a problem for which T is finite, the finite sequence of time steps refers to an episode and the associated task is referred to as an episodic task. For other reinforcement learning problems, the trajectory T is infinite, and there are no terminal states. A problem for which T is infinite is referred to as an infinite horizon task. Managing a fleet of taxis in a city is an example of a problem having a continuing task. An example of a reinforcement learning problem having an episodic task is an agent learning to play the card game blackjack, in which each round of play is an episode. As an example, a possible definition of the return is given by Equation (1) below: 
     
       
         
           
             
               
                 
                   
                     
                       G 
                       n 
                     
                     = 
                     
                       
                         ∑ 
                         
                           j 
                           = 
                           0 
                         
                         
                           T 
                           - 
                           n 
                           - 
                           1 
                         
                       
                        
                       
                         
                           γ 
                           j 
                         
                          
                         
                           R 
                           
                             n 
                             + 
                             j 
                             + 
                             1 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     in which γ is a parameter called the discount factor, which satisfies 0≤y≤1, with γ=1 only being permitted if T is finite. Equation (1) states that the return assigned to an agent at time step n is the sum of a series of future rewards received by the agent, where terms in the series are multiplied by increasing powers of the discount factor. Choosing a value for the discount factor affects how much an agent takes into account likely future states when making decisions, relative to the state perceived at the time that the decision is made. Assuming the sequence of rewards {R j } is bounded, the series in Equation (1) is guaranteed to converge. A skilled person will appreciate that this is not the only possible definition of a return. For example, in R-learning algorithms, the return given by Equation (1) is replaced with an infinite sum over undiscounted rewards minus an average expected reward. The applicability of the methods and systems described herein is not limited to the definition of return given by Equation (1). 
     In response to an agent receiving a state signal, the agent selects an action to perform based on a policy. A policy is a stochastic mapping from states to actions. If an agent follows a policy π, and receives a state signal at time step n indicating a specific state S n  =s, the probability of the agent selecting a specific action A n  =a is denoted by π (a|s). A policy for which π n (a|s) takes values of either 0 or 1 for all possible combinations of a and s is a deterministic policy. Reinforcement learning algorithms specify how the policy of an agent is altered in response to sequences of states, actions, and rewards that the agent experiences. 
     The objective of a reinforcement learning algorithm is to find a policy that maximises the expectation value of a return. Two different expectation values are often referred to: the state value and the action value respectively. For a given policy π, the state value function v π (s) is defined for each states by the equation v π (s)=   π (G n |S n =s), which states that the state value of states given policy π is the expectation value of the return at time step n, given that at time step n the agent receives a state signal indicating a state S n =s. Similarly, for a given policy π, the action value function q π (s, a) is defined for each possible state-action pair (s, a) by the equation q π (s, a)=   π (G n |S n =s, A n =a), which states that the action value of a state-action pair (s, a) given policy π is the expectation value of the return at time step t, given that at time step n the agent receives a state signal indicating a state S n =s, and selects an action A n =a. A computation that results in a calculation or approximation of a state value or an action value for a given state or state-action pair is referred to as a backup. A reinforcement learning algorithm generally seeks a policy that maximises either the state value function or the action value function for all possible states or state-action pairs. In many practical applications of reinforcement learning, the number of possible states or state-action pairs is very large or infinite, in which case it is necessary to approximate the state value function or the action value function based on sequences of states, actions, and rewards experienced by the agent. For such cases, approximate value functions {circumflex over (v)}(s, w) and {circumflex over (q)}(s, a, w) are introduced to approximate the value functions v π (s) and q π (s, a) respectively, in which w is a vector of parameters defining the approximate functions. Reinforcement learning algorithms then adjust the parameter vector w in order to minimise an error (for example a root-mean-square error) between the approximate value functions {circumflex over (v)}(s, w) or {circumflex over (q)}(s, a, w) and the value functions v π (s) or q π (s, a). 
     In many reinforcement learning algorithms (referred to as action-value methods), a policy is defined in terms of approximate value functions. For example, an agent following a greedy policy always selects an action that maximises an approximate value function. An agent following an ε-greedy policy instead selects, with probability 1−ε, an action that maximises an approximate value function, and otherwise selects an action randomly, where ε is a parameter satisfying 0&lt;ε&lt;1. Other reinforcement learning algorithms (for example actor-critic methods) represent the policy π without explicit reference to an approximate value function. In such methods, the policy π is represented by a separate data structure. It will be appreciated that many further techniques can be implemented in reinforcement learning algorithms, for example bounded rationality or count-based exploration. 
       FIG. 1  illustrates an example of a single agent interacting with an environment. The horizontal axis  101  represents increasing time, the dashed line  103  above the axis  101  represents the agent, and the dashed line  105  below the axis  101  represents the environment. At time step n, the agent receives a first state signal  107  from the environment, indicating a state S n , and the agent receives an associated reward R n  associated with the state S n . In response to receiving the first state signal  107 , the agent selects an action A n  in accordance with a policy π, and performs the action A n . The action A n  has an effect on the environment, and is completed at time step n+1. Immediately after the action A n  has been performed, the environment sends a new state signal  109  to the agent, indicating a new state S n+1 . The new state S n+1  is associated with a reward R n+1 . The agent then performs an action A n+1 , leading to a state S n+2  associated with a reward R n+2 . As shown in  FIG. 1 , the interval between time steps n+1 and n+2 does not need to be the same as the interval between time steps n and n+1, and the reward R n+2  does not need to be the same as the rewards R n+1  or R n . 
     A range of reinforcement learning algorithms are well-known, and different algorithms may be suitable depending on characteristics of the environment and the agents that define a reinforcement learning problem. Examples of reinforcement learning algorithms include dynamic programming methods, Monte Carlo methods, and temporal difference learning methods, including actor-critic methods. The present application introduces systems and methods that facilitate the implementation of both existing and future reinforcement learning algorithms in cases of problems involving large or infinite numbers of states, and/or having multiple agents, that would otherwise be intractable using existing computing hardware. 
     Multi-Agent Systems 
     Systems and methods in accordance with the present invention are particularly advantageous in cases in which more than one agent interacts with an environment. The example of managing a fleet of taxis in a city is likely to involve many agents.  FIG. 2  illustrates an example in which two agents interact with an environment. As in  FIG. 1 , time increases from left to right. The top dashed line  201  represents Agent  1 , the bottom dashed line  203  represents Agent  2 , and the middle dashed line  205  represents the environment. Agent  1  has a trajectory (S n   (1) , A n   (1) , R n+1   (1) , . . . ), and Agent  2  has a trajectory (S m   (2) , A m   (2) , R m+1   (2) , . . . ), and in this example the set of time steps at which Agent  1  receives state signals is different from the set of time steps at which Agent  2  receives state signals. In this example, the agents do not send signals directly to each other, but instead interact indirectly via the environment (although in other examples signals can be sent directly between agents). For example, the action A n   (1) , performed by Agent  1  and represented by arrow  207  has an effect on the environment that alters the information conveyed by state signal  209 , indicating a state S m+1   (2)  to Agent  2 . In an example in which  FIG. 2  represents two competing taxis in a small town, a first taxi being represented by Agent  1  and a second taxi being represented by Agent  2 , the action A n   (1)  may represent the first taxi driving to a taxi rank in the town in order to seek customers. The action A m   (2)  may represent the second taxi driving to the taxi rank from a different part of the town. When the second taxi reaches the taxi rank, Agent  2  receives a state signal indicating a state S m+1   (2)  in which the first taxi is already waiting at the taxi rank. Agent  2  receives a negative reward R m+2   (2)  because a state in which the first taxi is already waiting at the taxi rank is not a favourable result for Agent  2 . Agent  2  then makes a decision to take action S m+1   (2) , causing the second taxi to drive to a different taxi rank. 
     In the example of  FIG. 2 , the two agents Agent  1  and Agent  2  act autonomously such that each agent makes decisions independently of the other agent, and the agents interact indirectly via the effect each agent has on the environment. Each agent selects actions according a policy that is distinct from the policy of each other agent. In the example of  FIG. 3   a,  four autonomous agents, referred to collectively as agents  301 , receive policy data from a data processing component referred to as policy source  303 . At a first time, policy source  303  sends policy data  305  to agent  301   a,  causing the policy of agent  301   a  to be updated. Similarly, policy source  303  sends policy data to each of the agents  301 , causing the policies of the agents  301  to be updated. In some examples, the policies of the agents  301  are updated simultaneously. In other examples, the policies of the agents  301  are updated at different times. The configuration of  FIG. 3 a    is referred to as a decentralised configuration. A skilled person will appreciate that in the case of a decentralised configuration of agents such as that of  FIG. 3   a,  the computing resources necessary to apply a particular reinforcement learning algorithm for each agent, including memory, processing power, and storage, can be arranged to scale proportionally with the number of neighbouring agents. Furthermore, a separate reinforcement learning algorithm can be applied to each agent using a separate processor or processor core, leading to parallelised reinforcement learning. 
     In the example of  FIG. 3   b,  policy source  313  sends policy data  315  to co-ordinator  317 . Co-ordinator  317  is an agent that receives state signals from agents  311  and sends instructions to agents  311  to perform actions. The union of agents  311  and co-ordinator  317  is an example of a composite agent, and the configuration of agents in  FIG. 3 b    is referred to as a centralised configuration. In cases where several agents work together to learn a solution to a problem or to achieve a shared objective (referred to as co-operative problem solving), centralised configurations such as that of  FIG. 3 b    typically achieve better coherence and co-ordination than autonomous agents such as those of  FIG. 3   a.  Coherence describes the quality of a solution to the problem, including the efficiency with which agents use resources in implementing the solution. Co-ordination describes the degree to which agents avoid extraneous activity. For a composite agent having a single co-ordinator, the computational expense of implementing a learning algorithm typically scales exponentially with the number of agents receiving instructions from the co-ordinator. For centralised configurations of agents, a co-ordinator selects actions for each agent included in a corresponding composite agent. In some specific examples, particular states are specified to be “bad states” and it is an objective of the co-ordinator to select combinations of actions to avoid bad states. An example of a bad state in co-operative problem solving is a deadlock state, in which no combination of possible actions exists that advances agents towards a shared objective. In the present application, a novel method, referred to as “correctness by learning”, is provided for composite agents, in which a co-ordinator learns to avoid bad states in a particular class of problem. 
     The example of  FIG. 3 c    includes two composite components, each having a co-ordinator and two agents. Co-ordinator  327  and agents  321  form a first composite agent, and co-ordinator  337  and agents  331  form a second composite agent. The configuration of agents in  FIG. 3 c    is referred to as locally centralised. For co-operative problem solving, a locally centralised configuration typically provides a compromise between: relatively good coherence, relatively good co-ordination, and high computational expense, associated with a centralised configuration; and relatively poor coherence, relatively poor co-ordination, and relatively low computational expense, associated with a decentralised co-ordination. The applicability of methods and systems described herein is not limited to the configurations of agents described above and it is not a concern of the present application to propose novel configurations of agents for reinforcement learning problems. Instead, the present application introduces systems and methods that facilitate the flexible implementation of reinforcement learning algorithms for a wide range of configurations of agents. 
     In some examples, agents are provided with a capability to send messages to one another. Examples of types of messages that a first agent may send to a second agent are “inform” messages, in which the first agent provides information to the second agent, and “request” messages, in which the first agent requests the second agent to perform an action. A message sent from a first agent to a second agent becomes part of a state signal received by the second agent and, depending on a policy of the second agent, a subsequent action performed by the second agent may depend on information received in the message. For examples in which agents are provided with a capability to send messages to each other, an agent communication language (ACL) is required. An ACL is a standard format for exchange of messages between agents. An example of an ACL is knowledge query and manipulation language (KQML). 
     For examples in which agents are used for co-operative problem solving, various problem-sharing protocols may be implemented, leading to co-operative distributed problem solving. An example of a well-known problem-sharing protocol is the Contract Net, which includes a process of recognising, announcing, bidding for, awarding, and expediting problems. It is not a concern of the present application to develop problem-sharing protocols. 
     Agents in a decision-making system may be benevolent, such that all of the agents in the decision-making system share a common objective, or may be fully self-interested where each agent has a dedicated objective, or different groups of autonomous agents may exist with each group of autonomous agents sharing a common objective. For a particular example in which agents are used to model two taxi companies operating in a city, some of the agents represent taxis operated by a first taxi company and other agents represent taxis operated by a second taxi company. In this example, all of the agents are autonomous agents, and agents representing taxis operated by the same taxi company have the capability to send messages to one another. In this example, conflict may arise between agents representing taxis operated by the first taxi company and agents representing taxis operated by the second taxi company. 
     Different agents may be designed and programmed by different programmers/vendors. In such an arrangement, can learn how to interact with other agents through learning from experience by interacting with these “foreign” agents. 
     System Architecture 
     The data processing system of  FIG. 4A  includes interaction subsystem  401 , learning subsystem  403 , and problem system  415 . Learning subsystem  403  includes policy learning subsystem  435 , model input subsystem  437 , and model learning subsystem  439 . Data is sent between interaction subsystem  401  and learning subsystem  403  via communication module  429  and communication module  431 . It is noted that the present arrangement of subsystems is only an example, and other arrangements are possible without departing from the scope of the invention. For example, model input subsystem  437  and model learning subsystem  439  may be combined and/or may be remote from the policy learning subsystem  435 . Alternatively, model input subsystem  437  may be incorporated within a subsystem including interaction subsystem  401 . Furthermore, any of the subsystems shown in  FIG. 4A  may be implemented as distributed systems. 
     Interaction subsystem  401  includes decision making system  405 , which comprises N agents, collectively referred to as agents  407 , of which only three agents are shown for ease of illustration. Agents  407  perform actions on environment  409  depending on state signals received from environment  409 , with the performed actions selected in accordance with policies received from policy source  411 . In this example, each of agents  407  represents an entity  413  in problem system  415 . Specifically, in this example problem system  415  is a fleet management system for a fleet of taxis in a city, and each entity  413  is a taxi in the fleet. For example, agent  407   a  represents entity  413   a.  In this example environment  409  is a dynamic model of the city, defined in terms of information deemed relevant to the problem of managing the fleet of taxis. Specifically, environment  409  is a probabilistic model of the city, as will be described herein. Interaction subsystem  401  also includes experience sink  417 , which sends experience data to policy learning subsystem  435 . Interaction subsystem  401  further includes model source  433 , which provides models to environment  409  and policy source  411 . 
     As shown in  FIG. 4B , policy learning subsystem  435  includes policy learner  419 , which implements one or more learning algorithms to learn policies for agents  407  in the decision making system  405 . In a specific example, policy learner  419  includes several deep neural networks (DNNs), as will be described herein. However, the policy learner  419  may implement alternative learning algorithms which do not involve DNNs. Policy learning subsystem  435  also includes two databases: experience database  421  and skill database  423 . Experience database  421  stores experience data generated by interaction system  401 , referred to as an experience record. Skill database  423  stores policy data generated by policy learner  419 . Policy learning subsystem  435  also includes experience buffer  425 , which processes experience data in preparation for the experience data to be sent to policy learner  419 , and policy sink  427 , which sends policy data generated by policy learner  419  to policy source  411  of interaction subsystem  401 . 
     As shown in  FIG. 4C , model input subsystem  437  includes data ingesting module  441 , which receives model input data related to problem system  415 . Model input data is data input to model learning subsystem  439  in order to generate models of problem system  415 . Model input data is distinct from experience data in that model input data is not experienced by agents  407 , and is used for learning models of problem system  415 , as opposed to being used to learn policies for agents  407 . A specific example of learning a model will be described in detail hereafter. In the example of taxi fleet management, model input data may include historic traffic data or historic records of taxi journeys. Model input data may include data indicative of measurements taken by sensors in the problem system  415 , for example measurements of weather. Model input subsystem further includes model input data pipeline  443 . Model input data pipeline  443  processes model input data and passes the processed model input data to model learning system  439 . Model input data pipeline  443  includes data cleaning module  445 , data transformation module  447 , and data validation module  449 . Data cleaning module  445  removes any model input data that cannot be further processed, for example because the model input data includes records that are in a format that is not recognised. Data transformation module  447  transforms or reformats data into a standardised format for further processing. For example, model input data containing dates may be reformatted such that the dates are transformed into a standard format (such as ISO 8601). Data validation module  449  performs a validation process to ensure the data is valid and therefore able to be further processed. For example, if model input data is expected to contain certain fields, or a certain number of fields, the data validation module  449  may check whether the expected fields and/or number of fields appear in the model input data. In some configurations, data validation module  449  may disregard model input data that fails the validation process. In some configurations, data validation module  449  may generate an alert for a human user if model input data fails the validation process. 
     As shown in  FIG. 4D , model learning subsystem  439  includes model learner  451 , which implements one or more learning algorithms to learn models to be incorporated into environment  409  and/or provided to agents  407  within decision making system  405 . Model learner  451  is arranged to receive model input data from model input subsystem  437 , and further to receive experience data from experience buffer  425 . An example of learning a model from model input data for incorporation into an environment will be described in detail hereafter with reference to  FIGS. 8 and 9 . Model learner  451  may additionally or alternatively learn models for providing to agents  407 . For example, model learner  451  may process experience data to learn a model for predicting subsequent states of an environment, given a current state signal and a proposed action. Providing such a model to agents  407  may allow agents  407  to make better decisions. In the example of taxi fleet management, a model may be provided to agents  407  that predicts journey times of taxi trips based on model input data comprising historic taxi records and/or traffic data. 
     Model learning subsystem  439  includes two databases: model input database  453  and model database  455 . Model input database  453  stores model input data received from model input subsystem  437 . Model input database  421  may store a large volume of model input data, for example model input data collected from problem system  415  over several months or several years. Model database  455  stores models generated by model learner  451 , which may be made available at later times, for example for incorporation into environment  409  or to be provided to agents  407 . Model learning subsystem  439  also includes model input data buffer  457 , which processes model input data in preparation for the model input data to be sent to model learner  451 . In certain configurations, model input data buffer  457  splits model input data into training data which model learner  451  uses to learn models, and testing data which is used to verify that models learned by model learner  451  make accurate predictions. Model learning subsystem also includes model sink  459 , which sends models generated by model learner  451  to model source  433  of interaction subsystem  401 . 
     In the example of the problem system  415  being a fleet management system, interaction subsystem  401  is a connected to the fleet management system and learning subsystem  403  is remote from the fleet management system and from interaction subsystem  401 . Communication module  429  and communication module  431  are interconnected via network interfaces to a communications network (not shown). More specifically, in this example the network is the Internet, learning subsystem  403  includes several remote servers connected to the Internet, and interaction subsystem  401  includes a local server. Learning subsystem  403  and interaction subsystem  401  interact via an application programming interface (API). 
     As shown in  FIG. 5 , during a reinforcement learning operation, each of the agents  407  generates, at S 501 , experience data corresponding to an associated trajectory consisting of successive triplets of state-action pairs and rewards. For example, agent  407   a,  which is labelled i=1, generates experience data corresponding to a trajectory including a sequence of tuples (S n   (1) , A n   (1) , R n+1   (1) , S n+1   (1) ) for n=1, 2, . . . , ∞ as the data processing system is in operation. Agents  407  send, at S 503 , experience data corresponding to sequentially generated tuples (S n   (i) , A n   (i) , R n+1   (i) , S n+1   (1) ) for n=1, 2, . . . , ∞; i=1, 2, . . . , N, to experience sink  417 . Experience sink  417  transmits, at S 505 , the experience data to experience database  421  via a communications network. Depending on configuration, experience sink  417  may transmit experience data in response to receiving data from one of the agents  407 , or may instead transmit batches of experience data corresponding to several successive state-action-reward tuples. Experience sink  417  may transmit batches of experience data corresponding to each of the agents  407  separately. In the present example, experience sink  417  transmits batches of experience data, each batch corresponding to several state-action-reward tuples corresponding to one of the agents  407 . Experience database  421  stores the experience data received from experience sink  417 . 
     Experience database  421  sends, at S 509 , the experience data to experience buffer  425 , which arranges the experience data into an appropriate data stream for processing by policy learner  419 . In this example, experience database  421  only stores the experience data until it has been sent to experience buffer  421 . Experience buffer  421  sends, at S 511 , the experience data to policy learner  419 . Depending on the configuration of policy learner  419 , the experience data may be sent to policy learner  419  as a continuous stream, or may instead be sent to policy learner  419  in batches. For a specific example in which the agents are arranged in a decentralised configuration similar to that shown in  FIG. 3   a,  the policy learner  419  may include a separate DNN for each of the agents  407 . Accordingly, in that specific example, experience buffer  425  sends experience data corresponding to each of the agents  407  to a separate DNN. 
     Policy learner  419  receives experience data from experience buffer  425  and implements, at S 513 , a reinforcement learning algorithm. The specific choice of reinforcement learning algorithms implemented by policy learner  419  is selected by a user and may be chosen depending on the nature of a specific reinforcement learning problem. In a specific example, policy learner  419  implements a temporal-difference learning algorithm, and uses supervised-learning function approximation to frame the reinforcement learning problem as a supervised learning problem, in which each backup plays the role of a training example. Supervised-learning function approximation allows a range of well-known gradient descent methods to be utilised by a learner in order to learn approximate value functions {circumflex over (v)}(s, w) or {circumflex over (q)}(s, a, w). The policy learner  419  may use the backpropagation algorithm for DNNs, in which case the vector of weights w for each DNN is a vector of connection weights in the DNN. 
     By way of example only, a DNN  601 , which can be used by policy learner  419  to learn approximate value functions, will now be described with reference to  FIGS. 6 and 7 . It is, however, emphasised that other algorithms could be used to generate policy data. 
     DNN  601  consists of input layer  603 , two hidden layers: first hidden layer  605  and second hidden layer  607 , and output layer  609 . Input layer  603 , first hidden layer  605  and second hidden layer  607  each has M neurons and each neuron of input layer  603 , first hidden layer  605  and second hidden layer  607  is connected with each neuron in the subsequent layer. The specific arrangement of hidden layers, neurons, and connections is referred to as the architecture of the network. A DNN is any artificial neural network with multiple hidden layers, though the methods described herein may also be implemented using artificial neural networks with one or zero hidden layers. Different architectures may lead to different performance levels for a given task depending on the complexity and nature of the approximate state value function to be learnt. Associated with each set of connections between successive layers is a matrix Θ (j)  for j=1, 2, 3 and for each of these matrices the elements are the connection weights between the neurons in the preceding layer and subsequent layer. 
       FIG. 7  describes how policy learner  419  uses DNN  601  to learn an approximate state value function {circumflex over (v)}(s, w) in accordance with a temporal difference learning algorithm, given a sequence of backups corresponding to a sequence of states S n , S n+1 , S n+2 , . . . observed by an agent. In this example, the return is given by Equation (1). Policy learner  419  randomly initialises the elements of the matrices Θ (j)  for j=1, 2, 3, at S 701 , to values in an interval [−δ, δ], where δ is a small user-definable parameter. The vector of parameters w contains all of the elements of the matrices Θ (j)  for j=1, 2, 3, unrolled into a single vector. 
     Policy learner  419  receives, at S 703 , experience data from experience buffer  425  corresponding to a state S n =s received by an agent at a time step n. The) experience data takes the form of a feature vector q(s)=(q 1 (s), q 2 (s), . . . , q M (s)) T  with M components (where T denotes the transpose). Each of the M components of the feature vector q(s) is a real number representing an aspect of the state s. In this example, the components of the feature vector q(s) are normalised and scaled as is typical in supervised learning algorithms in order to eliminate spurious effects caused to the output of the learning algorithm by different features inherently varying on different length scales, or being distributed around different mean values. Policy learner  419  supplies, at S 705 , the M components of q(s) to the M neurons of the input layer  603  of DNN  601 . 
     DNN  601  implements forward propagation, at S 707 , to calculate an approximate state value function. The components of q(s) are multiplied by the components of the matrix Θ (1)  corresponding to the connections between input layer  603  and first hidden layer  605 . Each neuron of first hidden layer  605  computes a real number A k   (2) (s)=g(z), referred to as the activation of the neuron, in which z=Σ m Θ km   (1) q m (s) is the weighted input of the neuron. The function g is generally nonlinear with respect to its argument and is referred to as the activation function. In this example, g is the sigmoid function. The same process of is repeated for second hidden layer  607  and for output layer  609 , where the activations of the neurons in each layer are used as inputs to the activation function to compute the activations of neurons in the subsequent layer. The activation of output neuron  611  is the approximate state value function {circumflex over (v)}(S n , w n ) for state S n =s, given a vector of parameters w n  evaluated at time step n. 
     Having calculated {circumflex over (v)}(S n , w n ), DNN  601  implements, at S 709 , the backpropagation algorithm to calculate gradients ∇ w     n   {circumflex over (v)}(S n , w n ) with respect to the parameter vector w n . DNN  601  then implements gradient descent, at S 711 , in parameter space to update the parameters. Gradient descent is implemented in this example by equation (2): 
         w   n+1   =w   n −½α∇ w     n   [ V   n ( s )− {circumflex over (v)} ( S   n   ,w   n )] 2   =w   n +α[ V   n   −{circumflex over (v)} ( S   n   ,w   n )]∇ w     n     {circumflex over (v)} ( S   n   ,w   n ),  (2)
 
     in which α is a parameter referred to as the learning rate, V n (s) is an estimate of the state value function v π (s). In this example, the estimate V n (s) is given by V n (s)=R n+1 +γ{circumflex over (v)}(S n+1 , w n ), and the gradient ∇{circumflex over (v)} 3 (S n , w n ) is augmented using a vector of eligibility traces, as is well-known in temporal difference learning methods. In some examples, other optimisation algorithms are used instead of the gradient descent algorithm given by Equation (2). In some examples, each layer in a neural network include an extra neuron called a bias unit that is not connected to any neuron in the previous layer and has an activation that does not vary during the learning process (for example, bias unit activations may be set to 1). In some examples of reinforcement learning algorithms, a learner computes approximate action value functions {circumflex over (q)}(s, a, w), instead of state value functions {circumflex over (v)}(s, w). Analogous methods to that described above may be used to compute action value functions. 
     Referring again to  FIG. 5 , policy learner  419  sends, at S 515 , policy data to policy sink  427 . Policy sink  427  sends, at S 517 , the policy data to policy source  411  via the network. Policy source  411  then sends, at S 519 , the policy data to agents  407 , causing the policies of agents  407  to be updated at S 521 . Depending on the reinforcement learning algorithm used by policy learner  419 , the policy data may either cause approximate value functions {circumflex over (v)}(s, w) or {circumflex over (q)}(s, a, w) stored by agents  407  to be updated (for action-value methods), or may instead cause separate data structures representing polices of agents  407  to be updated (for actor-critic methods and other methods in which the policy is stored as a separate data structure). In the example of  FIG. 4 , an actor-critic method is employed, and therefore agents use the policy data to update data structures that explicitly represent policies. At certain time steps (for example, a time step after which a policy is measured to satisfy a given performance metric), policy learner  419  also sends policy data to skill database  423 . Skill database  423  stores a skill library including approximate value functions and/or policies learned during the operation of the data processing system, which can later be provided to agents and/or learners in order to negate the need to relearn the same or similar approximate value functions and/or policies from scratch. 
     The architecture shown in  FIG. 4 , in which the learning subsystem  403  is remotely hosted and the interaction subsystem  401  is locally hosted, is designed to provide flexibility and scalability for a wide variety of reinforcement learning systems. In many reinforcement learning systems, data is frequently provided to the environment, causing the task to change. In the example of managing a fleet of taxis in a city, data corresponding to an event such as a change in weather may be provided to environment  409 , causing a probabilistic model of environment  409  to be altered and therefore causing the task to change. Furthermore, the task associated with environment  409  is dependent on action signals received from agents  407 . The architecture of  FIG. 4  decouples on the one hand the sending of experience data and policy data between the interaction subsystem and the learning subsystem and on the other hand the sending of data between the agents and the environment. In the system of  FIG. 4 , only experience data and policy data are required to be transferred over the network between learning subsystem  403  and the interaction subsystem  401 . Experience data corresponding to states and actions experienced by agents  407  is relatively compact, with state information capable of being reduced to feature vectors (although in some examples all information about a state is included in the experience data so as to be available for analysis by the learning subsystem). Further, the format of experience data is independent on the nature of environment  409  and is specified by the API through which interaction system  401  and learning system  403  interact. It is therefore possible for policy learning subsystem  435  to be agnostic with respect to details of environment  409 , which allows flexibility as a range of interaction systems are able to be connected with learning system  403  over the network without making substantial alterations within learning subsystem  403 . Policy data is also relatively compact. For example, in the case of an actor-critic method, a scalar signal could be used to transfer policy data from policy learner  419  to each of the agents  407  in order for agents  407  to update policies, although a vector signal or a matrix of values may be used in some examples. The frequency at which experience data and policy data are transferred between policy learning subsystem  435  and interaction subsystem  401  is configurable. For example, in some examples experience data is sent in batches corresponding to a configurable number of time steps. Similarly, in some examples the reinforcement learning algorithm implemented by policy learner  419  works in a batch configuration, such that policy data is sent to interaction system  401  after policy learner  419  has processed experience data corresponding to a configurable number of time steps. In some examples, configuring the batch size as described above is manual, in which case a user selects the size of the batches of experience data. In other examples configuring the batch size is automatic, in which case an optimal batch size is calculated and selected depending on the specific reinforcement learning algorithm and the specific configuration of agents  407  and environment  409 . Configuring the batch size provides flexibility and scalability regarding the number of agents  407  and the complexity of environment  409 , because in doing so the time scale associated with the learning process performed by policy learning subsystem  435  is decoupled from the time scale associated with time steps in interaction subsystem  401 . For large numbers of agents and/or complex environments, the time scale associated with each time step is typically much shorter than the time scale associated with the reinforcement learning process, so configuring an appropriate batch size means that interaction subsystem  403  is able to operate without being slowed down by the reinforcement learning algorithm implemented by learning system  401 . 
     Distributing the processing between a local interaction subsystem and a remote learning subsystem has further advantages. For example, the data processing subsystem can be deployed with the local interaction subsystem utilising the computer hardware of a customer and the learning subsystem utilising hardware of a service provider (which could be located in the “cloud”). In this way, the service provider can make hardware and software upgrades without interrupting the operation of the local interaction subsystem by the customer. 
     As described herein, reinforcement learning algorithms may be parallelised for autonomous agents, with separate learning processes being carried out by policy learner  419  for each of the agents  407 . For systems with large numbers of agents, the system of  FIG. 4  allows for policy learning subsystem  435  to be implemented by a distributed computing system. Further, for composite agents such as those described in  FIG. 3 b    or  3   c,  in which the computational expense of learning algorithms typically scale exponentially with the number of component agents, servers having powerful processors, along with large memory and storage, may be provided. Implementing the learning subsystem using a remote, possibly distributed, system of servers, allows the necessary computing resources to be calculated depending on the configuration of the agents and the complexity of the environment, and for appropriate resources to be allocated to policy learning subsystem  435 . Computing resources are thereby allocated efficiently. 
     Probabilistic Modelling 
     As stated above, an environment is a virtual system with which agents interact, and the complete specification of the environment is referred to as a task. In some examples, an environment simulates a real-world system, defined in terms of information deemed relevant to the specific problem being posed. Some examples of environments in accordance with the present invention include a probabilistic model which can be used to predict future conditions of the environment. In the example architecture of  FIG. 4A , the model learner  451  may be arranged to process model input data received from the model input data subsystem  437 , and/or experience data received from the experience buffer  425 , in order to generate a probabilistic model. The model learner  451  may send the generated probabilistic model to the model source  433  for incorporation into the environment  409 . Incorporating a probabilistic model into an environment allows state signals sent from the environment to agents to include information corresponding not only to a prevailing condition of the environment, but also to likely future conditions of the environment. In an example of managing a fleet of taxis in a city in which a probabilistic model is included in the environment, an agent representing a taxi may receive a state signal indicating that an increase in demand for taxis is likely to occur in a certain region of the city at a given point in the future. In this example, the probabilistic model is used to generate a probability distribution for taxi demand in the city. This allows agents to predict variations in demand and to select actions according to these predictions, rather than simply reacting to observed variations in demand. Further to providing additional state information to agents, in some examples a probabilistic model is used to generate simulation data for use in reinforcement learning. In such examples, the simulation data may be used to simulate states of an environment. Agents may then interact with the simulated states of the environment in order to generate experience data for use by a policy learner to perform reinforcement learning. Such examples make efficient use of data corresponding to observed states of an environment, because a large volume of simulation data can be generated from a limited volume of observed data. In particular, data corresponding to observed states of an environment is likely to be limited in cases where the environment corresponds to a physical system. 
     It is an objective of the present application to provide a computer-implemented method for implementing a particular type of probabilistic model of a system. The probabilistic model is suitable for incorporation into an environment in a reinforcement learning problem, and therefore the described method further provides a method for implementing a probabilistic model within a reinforcement learning environment for a data processing system such as that shown in  FIG. 4 . Novel techniques are provided that significantly decrease the computational cost of implementing the discussed probabilistic model, thereby allowing larger scale models and more complex environments to be realised. A formal definition of the probabilistic model will be described hereafter. 
     The present method relates to a type of inhomogeneous Poisson process referred to as a Cox process. For a D-dimensional domain χ⊂   d , a Cox process is defined by a stochastic intensity function λ: χ→   + , such that for each point x in the domain χ, λ(x) is a non-negative real number. A number N p (τ) of points found in a sub-region τ⊂χ is assumed to be Poisson distributed such that N p (τ)˜Poisson(λ τ ) for λ τ =∫ τ λ(x)dx. The interpretation of the domain χ and the Poisson-distributed points depends on the system that the model corresponds to. In the example of managing a fleet of taxis in a city, the domain χ is three-dimensional, with first and second dimensions corresponding to co-ordinates on a map of the city, and a third dimension corresponding to time. N p (τ) then refers to the number of taxi requests received over a given time interval in a given region of the map. The stochastic intensity function λ(x) therefore gives a probabilistic model of taxi demand as a function of time and location in the city. An aim of the present disclosure is to provide a computationally-tractable technique for inferring the stochastic intensity function λ(x), given model input data comprising a set of discrete data X N ={x (n)}   n=1   N  corresponding to observed points in a sub-region τ of domain χ, which does not require the domain χ to be discretised, and accordingly does not suffer from problems associated with discretisation of the domain χ. In the example of managing a fleet of taxis in a city, each data point x (n)  for n=1, . . . , N corresponds to the location and time of an observed taxi request in the city during a fixed interval. In some examples, the data X N  may further include experience data, for example including locations and times of taxi pickups corresponding to actions by the agents  407 . The model learner  451  may process this experience data to update the probabilistic model as the experience data is generated. For example, the model learner  451  may update the probabilistic model after a batch of experience data of a predetermined size has been generated by the agents  407 . 
     The present method is an example of a Bayesian inference scheme. Such schemes are based on the application of Bayes&#39; theorem in a form such as that of Equation (3): 
     
       
         
           
             
               
                 
                   
                     
                       p 
                        
                       
                         ( 
                         
                           
                             λ 
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           | 
                           
                             X 
                             N 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         
                           p 
                            
                           
                             ( 
                             
                               
                                 X 
                                 N 
                               
                               | 
                               
                                 λ 
                                  
                                 
                                   ( 
                                   x 
                                   ) 
                                 
                               
                             
                             ) 
                           
                         
                          
                         
                           p 
                            
                           
                             ( 
                             
                               λ 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                             ) 
                           
                         
                       
                       
                         p 
                          
                         
                           ( 
                           
                             X 
                             N 
                           
                           ) 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     in which: 
     p(λ(x)|X N ) is a posterior probability distribution of the function λ(x) conditioned on the data X N ; 
     p(X N |λ(x)) is a probability distribution of the data X N  conditioned on the function λ(x), referred to as the likelihood of λ(x) given the data X N ; 
     p(λ(x)) is a prior probability distribution of functions λ(x) assumed in the model, also referred to simply as a prior; and 
     p(X N ) is the marginal likelihood, which is calculated by marginalising the likelihood over functions λ in the prior distribution, such that p(X N )=∫p(X N |λ(x))p(λ(x))df. 
     For the Cox process described above, the likelihood of λ(x) given the data X N  is given by Equation (4): 
     
       
         
           
             
               
                 
                   
                     
                       p 
                        
                       
                         ( 
                         
                           
                             X 
                             N 
                           
                           | 
                           
                             λ 
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         exp 
                          
                         
                           ( 
                           
                             - 
                             
                               
                                 ∫ 
                                 τ 
                               
                                
                               
                                 
                                   λ 
                                    
                                   
                                     ( 
                                     x 
                                     ) 
                                   
                                 
                                  
                                 d 
                                  
                                 x 
                               
                             
                           
                           ) 
                         
                       
                        
                       
                         
                           ∏ 
                           
                             n 
                             = 
                             1 
                           
                           N 
                         
                          
                         
                           λ 
                            
                           
                             ( 
                             
                               x 
                               
                                 ( 
                                 n 
                                 ) 
                               
                             
                             ) 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
     which is substituted into Equation (3) to give Equation (5): 
     
       
         
           
             
               
                 
                   
                     
                       p 
                        
                       
                         ( 
                         
                           
                             λ 
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                           | 
                           
                             X 
                             N 
                           
                         
                         ) 
                       
                     
                     = 
                     
                       
                         
                           exp 
                            
                           
                             ( 
                             
                               - 
                               
                                 
                                   ∫ 
                                   τ 
                                 
                                  
                                 
                                   
                                     λ 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                    
                                   d 
                                    
                                   x 
                                 
                               
                             
                             ) 
                           
                         
                          
                         
                           
                             ∏ 
                             
                               n 
                               = 
                               1 
                             
                             N 
                           
                            
                           
                             
                               λ 
                                
                               
                                 ( 
                                 
                                   x 
                                   
                                     ( 
                                     n 
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                              
                             
                               p 
                                
                               
                                 ( 
                                 
                                   λ 
                                    
                                   
                                     ( 
                                     x 
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                       
                       
                         ∫ 
                         
                           
                             p 
                              
                             
                               ( 
                               
                                 λ 
                                  
                                 
                                   ( 
                                   x 
                                   ) 
                                 
                               
                               ) 
                             
                           
                            
                           
                             exp 
                              
                             
                               ( 
                               
                                 - 
                                 
                                   
                                     ∫ 
                                     τ 
                                   
                                    
                                   
                                     
                                       λ 
                                        
                                       
                                         ( 
                                         x 
                                         ) 
                                       
                                     
                                      
                                     d 
                                      
                                     x 
                                   
                                 
                               
                               ) 
                             
                           
                            
                           
                             
                               ∏ 
                               
                                 n 
                                 = 
                                 1 
                               
                               N 
                             
                              
                             
                               
                                 λ 
                                  
                                 
                                   ( 
                                   
                                     x 
                                     
                                       ( 
                                       n 
                                       ) 
                                     
                                   
                                   ) 
                                 
                               
                                
                               d 
                                
                               λ 
                             
                           
                         
                       
                     
                   
                   . 
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     In principle, the inference problem is solved by calculating the posterior probability distribution using Equation (5). In practice, calculating the posterior probability distribution using Equation (5) is not straightforward. First, it is necessary to provide information about the prior p(λ(x)). This is a feature of all Bayesian inference schemes and various methods have been developed for providing such information. For example, some methods include specifying a form of the function to be inferred (λ(x) in the case of Equation (5)), which includes a number of parameters to be determined. For such methods, Equation (5) then results in a probability distribution over the parameters of the function to be inferred. Other methods do not include explicitly specifying a form for the function to be inferred, and instead assumptions are made directly about the prior (p(λ(x)) in the case of Equation (5)). A second reason that calculating the posterior probability distribution using Equation (5) is not straightforward is that computing the nested integral in the denominator of Equation (5) is computationally very expensive, and the time taken for the inference problem to be solved for many methods therefore becomes prohibitive if the number of dimensions D and/or the number of data points N is large (the nested integral is said to be doubly-intractable). 
     The doubly-intractable integral of Equation (5) is particularly problematic for cases in which the probabilistic model is incorporated into an environment for a reinforcement learning problem, in which one of the dimensions is typically time, and therefore the integral over the region τ involves an integral over a history of the environment. Known methods for approaching problems involving doubly-intractable integrals of the kind appearing in Equation (5) typically involve discretising the domain τ, for example using a regular grid, in order to pose a tractable approximate problem. Such methods thereby circumvent the double intractability of the underlying problem, but suffer from sensitivity to the choice of discretisation, particularly in cases where the data points are not located on the discretising grid. It is noted that, for high-dimensional examples, or examples with large numbers of data points, the computational cost associated with a fine discretisation of the domain quickly becomes prohibitive, preventing such methods from being applicable in many practical situations. 
     The present method provides a novel approach to address the difficulties mentioned above such that the posterior p(λ(x)|X N ) given above by Equation (5) is approximated with a relatively low computational cost, even for large values of N. Furthermore, the present method does not involve any discretisation of the domain τ, and therefore does not suffer from the associated sensitivity to the choice of grid or prohibitive computational cost. The method therefore provides a tractable method for providing a probabilistic model for incorporation into an environment for a reinforcement learning problem. Broadly, the method involves two steps: first, the stochastic intensity function λ(x) is assumed to be related to a random latent function f(x) that is distributed according to a Gaussian process. Second, a variational approach is applied to construct a Gaussian process q(f(x)) that approximates the posterior distribution p(f(x)|X N ). The posterior Gaussian process is chosen to have a convenient form based on a set of M Fourier components, where the parameter M is used to control a bias related to a characteristic length scale of inferred functions in the posterior Gaussian process. The form chosen for the posterior Gaussian process results in the variational approach being implemented with a relatively low computational cost. 
     In the present method, the latent function f is assumed to be related to the stochastic intensity function λ by the simple identity λ(x)≡[f(x)] 2 . The posterior distribution of λ conditioned on the data X N  is readily computed if the posterior distribution of f conditioned on the data X N  is known (or approximated). Defining the latent function f in this way permits a Gaussian process approximation to be applied, in which a prior p(f(x)) is constructed by assuming that f(x) is a random function distributed according to a Gaussian process. In the following section, the present method will be described for the one-dimensional case D=1, and extensions to D&gt;1, which are straightforward extensions of the D=1 case, will be described thereafter. 
     Variational Gaussian Process Method in One Dimension 
     The following section describes in some mathematical detail a method of providing a probabilistic model in accordance with an aspect of the present invention. 
     For illustrative purposes,  FIG. 8 a    shows an example of a prior constructed for a one-dimensional latent function f(x), for which f(x) is assumed to be distributed according to a Gaussian process having a mean function of zero. Dashed lines  801  and  803  are each separated from the mean function by twice the standard deviation of the distribution, and solid curves  805 ,  807 , and  809  are sample functions taken from the prior distribution.  FIG. 8 b    illustrates a posterior distribution f(x)|X 2 , conditioned on two data points  811  and  813 . Although in this example the observations of the function are made directly, in an inhomogeneous Poisson process model the data is related indirectly to the function through a likelihood equation. Solid line  815  shows the mean function of the posterior distribution and dashed lines  817  and  819  are each separated from the mean function by twice the standard deviation of the posterior distribution. In this example, the mean function represented by solid line  815  passes through both of the data points, and the standard deviation of the posterior distribution is zero at these points. This is not necessarily the case for all posterior distributions conditioned on a set of points. 
     Returning to the present method, a prior is constructed by assuming f(x) is distributed as a Gaussian process: f(x)˜GP(0, k(x, x′)), which has a mean function of zero and a covariance function k(x, x′) having a specific form as will be described hereafter. In one specific example, k(x, x′) that is a member of the Matérn family with half-integer order. It is further assumed that f(x) depends on an 2M+1-dimensional vector u of inducing variables u m  for m=1, . . . ,2M+1, where 2M+1&lt;N. The idea is to select the inducing variables such that the variational method used for approximating the posterior p(f(x)|X N ) is implemented at a relatively low computational cost. 
     Any conditional distribution of a Gaussian process is also a Gaussian process. In this case, the distribution of f(x)|u conditioned on the inducing variables u is written in a form given by Equation (6): 
       f(x)|u˜GP(k u (x) T K uu   −1 u,k(x,x′)−k u (x) T K uu   −1 k u (x′)),  (6)
 
     in which the m th a component of the vector function k u (x) is defined as k u (x)[m]≡cov(u m , f(x)), and the (m, m′) element of the matrix K uu  is defined as K uu [m, m′]≡cov(u m , u m ,), with cov denoting the covariance cov(X, Y)≡ ((X− (X))(Y− (Y))), and   denoting the expectation. The posterior distribution is approximated by marginalising the distribution of Equation (6) over a variational distribution q(u)˜Normal(m, Σ), which is assumed to be a multivariate Gaussian distribution with mean m and covariance Σ, in which the form of Σ is restricted for convenience, as will be described hereafter. The resulting approximation is a variational Gaussian process, given by Equation (7): 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           q 
                            
                           
                             ( 
                             
                               f 
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                             ) 
                           
                         
                         = 
                           
                          
                         
                           ∫ 
                           
                             
                               q 
                                
                               
                                 ( 
                                 u 
                                 ) 
                               
                             
                              
                             
                               q 
                                
                               
                                 ( 
                                 
                                   
                                     f 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                   | 
                                   u 
                                 
                                 ) 
                               
                             
                              
                             du 
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                           
                          
                         
                           G 
                            
                           
                             
                               P 
                                
                               
                                 ( 
                                 
                                   
                                     
                                       
                                         
                                           k 
                                           u 
                                         
                                          
                                         
                                           ( 
                                           x 
                                           ) 
                                         
                                       
                                       T 
                                     
                                      
                                     
                                       K 
                                       uu 
                                       
                                         - 
                                         1 
                                       
                                     
                                      
                                     m 
                                   
                                   , 
                                   
                                       
                                   
                                    
                                   
                                     
                                       k 
                                        
                                       
                                         ( 
                                         
                                           x 
                                           , 
                                           
                                             x 
                                             ′ 
                                           
                                         
                                         ) 
                                       
                                     
                                     + 
                                     
                                       
                                         
                                           
                                             k 
                                             u 
                                           
                                            
                                           
                                             ( 
                                             x 
                                             ) 
                                           
                                         
                                         T 
                                       
                                        
                                       
                                         ( 
                                         
                                           
                                             
                                               K 
                                               
                                                 u 
                                                  
                                                 u 
                                               
                                               
                                                 - 
                                                 1 
                                               
                                             
                                              
                                             Σ 
                                              
                                             
                                               K 
                                               
                                                 u 
                                                  
                                                 u 
                                               
                                               
                                                 - 
                                                 1 
                                               
                                             
                                           
                                           - 
                                           
                                             K 
                                             
                                               u 
                                                
                                               u 
                                             
                                             
                                               - 
                                               1 
                                             
                                           
                                         
                                         ) 
                                       
                                        
                                       
                                         
                                           k 
                                           u 
                                         
                                          
                                         
                                           ( 
                                           
                                             x 
                                             ′ 
                                           
                                           ) 
                                         
                                       
                                     
                                   
                                 
                                 ) 
                               
                             
                             . 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     The method proceeds with the objective of minimising a Kuller-Leibler divergence (referred to hereafter as the KL divergence), which quantifies how much the Gaussian process q(f(x)) used to approximate the posterior distribution diverges from the actual posterior distribution p (f(x)|X N ). The KL divergence is given by equation (8): 
         KL [ q ( f )∥ p ( f|X   N )]=   q(f(xx)) [log  q ( f ( x ))−log  p ( f ( x )| X   N )],  (8)
 
     In which    q(f(x))  denotes the expectation under the distribution q(f(x)). Equation (8) is written using Bayes&#39; theorem in the form of Equation (9): 
     
       
         
           
             
               
                 
                   
                     K 
                      
                     
                       L 
                        
                       
                         [ 
                         
                           
                             q 
                              
                             
                               ( 
                               f 
                               ) 
                             
                           
                           || 
                           
                             p 
                              
                             
                               ( 
                               
                                 f 
                                 | 
                                 
                                   X 
                                   N 
                                 
                               
                               ) 
                             
                           
                         
                         ] 
                       
                     
                   
                   = 
                   
                     
                       log 
                        
                       
                         p 
                          
                         
                           ( 
                           
                             X 
                             N 
                           
                           ) 
                         
                       
                     
                     - 
                     
                       
                         
                            
                           
                             q 
                              
                             
                               ( 
                               
                                 f 
                                  
                                 
                                   ( 
                                   x 
                                   ) 
                                 
                               
                               ) 
                             
                           
                         
                          
                         
                           [ 
                           
                             log 
                              
                             
                               
                                 
                                   p 
                                    
                                   
                                     ( 
                                     
                                       
                                         X 
                                         N 
                                       
                                       | 
                                       
                                         f 
                                          
                                         
                                           ( 
                                           x 
                                           ) 
                                         
                                       
                                     
                                     ) 
                                   
                                 
                                  
                                 
                                   p 
                                    
                                   
                                     ( 
                                     
                                       f 
                                        
                                       
                                         ( 
                                         x 
                                         ) 
                                       
                                     
                                     ) 
                                   
                                 
                               
                               
                                 q 
                                  
                                 
                                   ( 
                                   
                                     f 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                           ] 
                         
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     The subtracted term on the right hand side of Equation (9) is referred to as the Evidence Lower Bound (ELBO), which is simplified by factorising the distributions p(f(x)) and q(f(x)), resulting in Equation (10): 
     
       
         
           
             
               
                 
                   
                     ELBO 
                     = 
                     
                       
                         
                            
                           
                             
                               q 
                                
                               
                                 ( 
                                 u 
                                 ) 
                               
                             
                              
                             
                               q 
                                
                               
                                 ( 
                                 
                                   
                                     f 
                                     N 
                                   
                                   | 
                                   u 
                                 
                                 ) 
                               
                             
                           
                         
                          
                         
                           [ 
                           
                             log 
                              
                             
                               p 
                                
                               
                                 ( 
                                 
                                   
                                     X 
                                     N 
                                   
                                   | 
                                   
                                     f 
                                     N 
                                   
                                 
                                 ) 
                               
                             
                           
                           ] 
                         
                       
                       - 
                       
                         
                            
                           
                             q 
                              
                             
                               ( 
                               u 
                               ) 
                             
                           
                         
                          
                         
                           [ 
                           
                             log 
                              
                             
                               
                                 q 
                                  
                                 
                                   ( 
                                   u 
                                   ) 
                                 
                               
                               
                                 p 
                                  
                                 
                                   ( 
                                   u 
                                   ) 
                                 
                               
                             
                           
                           ] 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
     in which f N ={f(x (n) )} n=1   N , p(u)˜Normal(0, K uu ) and q(f N |u)˜Normal(K fu K uu   −1 u, K ff −K fu K uu   −1 K fu   T ), in which K fu [m, m′]≡cov(f(x (m) ), u m′ ) and K ff [m, m′]≡cov(f(x (m) ),f(x (m′) )). Minimising the KL divergence with respect to the parameters of the variational distribution q(u) is achieved by maximising the ELBO with respect to the parameters of the variational distribution q(u). For cases in which the ELBO is tractable, any suitable nonlinear optimisation algorithm may be applied to maximise the ELBO. In this example, a gradient-based optimisation algorithm is used. 
     A specific choice of inducing variables u is chosen in order to achieve tractability of the ELBO given by Equation (10). In the particular, the inducing variables u are assumed to lie in an interval [a, b], and are related to components of a truncated Fourier basis on the interval [a, b], the basis defined by entries of the vector ϕ(x)=[1, cos(ω 1 (x−a)), . . . , cos(ω M (x−a)), sin(ω 1 (x−a)), . . . , sin(ω M (x−a))] T , in which ω m =2πm/(b−a). The interval [a, b] should be chosen such that all of the data X N  lie on the interior of the interval. It can be shown that increasing the value of M necessarily improves the approximation in the KL sense, though increases the computational cost of implementing the method. The inducing variables are given by u m =P ϕ     m   (f), where the operator P ϕ     m    in denotes the Reproducing Kernel Hilbert Space (RKHS) inner product, i.e. P ϕ     m   (h)≡ ϕ m , h   H . The components of the resulting vector function k u (x) are given by Equation (11): 
     
       
         
           
             
               
                 
                   
                     
                       
                         k 
                         u 
                       
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       [ 
                       m 
                       ] 
                     
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             
                               
                                 φ 
                                 m 
                               
                                
                               
                                 ( 
                                 x 
                                 ) 
                               
                             
                           
                           
                             
                               
                                 for 
                                  
                                 
                                     
                                 
                                  
                                 x 
                               
                               ∈ 
                               
                                 [ 
                                 
                                   a 
                                   , 
                                   b 
                                 
                                 ] 
                               
                             
                           
                         
                         
                           
                             
                               cov 
                                
                               
                                 ( 
                                 
                                   
                                     
                                       P 
                                       
                                         φ 
                                         m 
                                       
                                     
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                   , 
                                   
                                     f 
                                      
                                     
                                       ( 
                                       x 
                                       ) 
                                     
                                   
                                 
                                 ) 
                               
                             
                           
                           
                             
                               
                                 for 
                                  
                                 
                                     
                                 
                                  
                                 x 
                               
                               ∉ 
                               
                                 [ 
                                 
                                   a 
                                   , 
                                   b 
                                 
                                 ] 
                               
                             
                           
                         
                       
                       , 
                     
                   
                 
               
               
                 
                   ( 
                   11 
                   ) 
                 
               
             
           
         
       
     
     In the cases of Matérn kernels of orders 1/2, 3/2, and 5/2, simple closed-form expressions are known for the RKHS inner product (see, for example, Durrande et al, “Detecting periodicities within Gaussian processes”,  Peer J Computer Science , (2016)), leading to closed-form expressions for k u (x)[m] both inside and outside of the interval [a, b]. Using the chosen inducing variables, elements of the matrix K uu  are given by K uu [m, m′]= ϕ m , ϕ m′     H , and in the case of Matérn kernels of orders 1/2, 3/2, and 5/2, are readily calculated, leading to a diagonal matrix plus a sum of rank one matrices, as shown by Equation (12): 
     
       
         
           
             
               
                 
                   
                     
                       K 
                       
                         u 
                          
                         u 
                       
                     
                     = 
                     
                       
                         diag 
                          
                         
                           ( 
                           α 
                           ) 
                         
                       
                       + 
                       
                         
                           ∑ 
                           
                             j 
                             = 
                             1 
                           
                           J 
                         
                          
                         
                           
                             β 
                             j 
                           
                            
                           
                             γ 
                             j 
                             T 
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   12 
                   ) 
                 
               
             
           
         
       
     
     where α, β j  and γ j  for j=1, . . . ,J are vectors of length 2M+1. In this example, the covariance matrix Σ is restricted to having the same form as that given in Equation (12) for K uu , though in other examples, other restrictions may be applied to the form of Σ. In some examples, no restrictions are applied to the form of Σ. The closed-form expressions associated with Equation (11), along with the specific form of the matrix given by Equation (12), lead directly to the tractability of the ELBO given by Equation (10), as will be demonstrated hereafter. The tractability of the ELBO overcomes the problem of double-intractability that prevents other methods of evaluating the posterior distribution in Equation (3) from being applicable in many probabilistic modelling contexts. As mentioned above, some known methods circumvent the doubly-intractable problem by posing an approximate discretised problem (see, for example, Rue et al, “Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations”,  J. R. Statist. Soc. B  (2009)). 
     The present method is applicable to any kernel for which the RHKS associated with the kernel contains the span of the Fourier basis ϕ(x), and in which the RKHS inner products are known (for example, in which the RHKS inner products have known closed-form expressions). By way of example, in the case of a Matérn kernel of order 1/2 with variance σ 2  and characteristic length scale l, defined by k 1/2 (x, x′)≡σ 2  exp(−|x−x′|/l), the matrix K uu  is given by Equation (12) with J=1, and in this case α, β 1 , and γ 1  are given by Equation (13): 
     
       
         
           
             
               
                 
                   
                     α 
                     = 
                     
                       
                         
                           
                             b 
                             - 
                             a 
                           
                           2 
                         
                          
                         
                           [ 
                           
                             
                               2 
                                
                               
                                 
                                   s 
                                    
                                   
                                     ( 
                                     0 
                                     ) 
                                   
                                 
                                 
                                   - 
                                   1 
                                 
                               
                             
                             , 
                             
                               
                                 s 
                                  
                                 
                                   ( 
                                   
                                     ω 
                                     1 
                                   
                                   ) 
                                 
                               
                               
                                 - 
                                 1 
                               
                             
                             , 
                             … 
                              
                             
                                 
                             
                             , 
                             
                               
                                 s 
                                  
                                 
                                   ( 
                                   
                                     ω 
                                     M 
                                   
                                   ) 
                                 
                               
                               
                                 - 
                                 1 
                               
                             
                             , 
                             
                               
                                 s 
                                  
                                 
                                   ( 
                                   
                                     ω 
                                     1 
                                   
                                   ) 
                                 
                               
                               
                                 - 
                                 1 
                               
                             
                             , 
                             … 
                              
                             
                                 
                             
                             , 
                             
                               
                                 s 
                                  
                                 
                                   ( 
                                   
                                     ω 
                                     M 
                                   
                                   ) 
                                 
                               
                               
                                 - 
                                 1 
                               
                             
                           
                           ] 
                         
                       
                       T 
                     
                   
                   , 
                   
                     
 
                   
                    
                   
                       
                   
                    
                   
                     
                       β 
                       1 
                     
                     = 
                     
                       
                         γ 
                         1 
                       
                       = 
                       
                         
                           [ 
                           
                             
                               σ 
                               
                                 - 
                                 1 
                               
                             
                             , 
                             
                               σ 
                               
                                 - 
                                 1 
                               
                             
                             , 
                             … 
                              
                             
                                 
                             
                             , 
                             
                               σ 
                               
                                 - 
                                 1 
                               
                             
                             , 
                             0 
                             , 
                             
                               … 
                                
                               
                                   
                               
                                
                               0 
                             
                           
                           ] 
                         
                         T 
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   13 
                   ) 
                 
               
             
           
         
       
     
     with s(ω)=2σ 2 λ 2 (λ 2 +ω 2 ) −1  and λ=l −1 . The components of vector function k u (x) for x∉[a, b] are given by Equation (14): 
     
       
         
           
             
               
                 
                   
                     
                       
                         k 
                         u 
                       
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                      
                     
                       [ 
                       m 
                       ] 
                     
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             
                               exp 
                               ( 
                               
                                 - 
                                 
                                   λ 
                                    
                                   
                                     ( 
                                     
                                        
                                       
                                         x 
                                         - 
                                         c 
                                       
                                        
                                     
                                     ) 
                                   
                                 
                               
                             
                           
                           
                             
                               
                                 
                                   for 
                                    
                                   
                                       
                                   
                                    
                                   m 
                                 
                                 = 
                                 1 
                               
                               , 
                               … 
                                
                               
                                   
                               
                               , 
                               
                                 M 
                                 + 
                                 1 
                               
                             
                           
                         
                         
                           
                             0 
                           
                           
                             
                               
                                 
                                   for 
                                    
                                   
                                       
                                   
                                    
                                   m 
                                 
                                 = 
                                 
                                   M 
                                   + 
                                   2 
                                 
                               
                               , 
                               … 
                                
                               
                                   
                               
                               , 
                               
                                 
                                   2 
                                    
                                   M 
                                 
                                 + 
                                 1 
                               
                             
                           
                         
                       
                       , 
                     
                   
                 
               
               
                 
                   ( 
                   14 
                   ) 
                 
               
             
           
         
       
     
     where c is whichever of a or b is closest to x. In order to evaluate the ELBO, the first term on the right hand side of Equation (10) is expanded as in Equation (15): 
     
       
         
           
             
               
                 
                   
                     
                        
                       
                         
                           q 
                            
                           
                             ( 
                             u 
                             ) 
                           
                         
                          
                         
                           q 
                            
                           
                             ( 
                             
                               
                                 f 
                                 N 
                               
                               | 
                               u 
                             
                             ) 
                           
                         
                       
                     
                      
                     
                       [ 
                       
                         log 
                          
                         
                           p 
                            
                           
                             ( 
                             
                               
                                 X 
                                 N 
                               
                               | 
                               
                                 f 
                                 N 
                               
                             
                             ) 
                           
                         
                       
                       ] 
                     
                   
                   = 
                   
                     
                       
                          
                         
                           
                             q 
                              
                             
                               ( 
                               u 
                               ) 
                             
                           
                            
                           
                             q 
                              
                             
                               ( 
                               
                                 
                                   f 
                                   N 
                                 
                                 | 
                                 u 
                               
                               ) 
                             
                           
                         
                       
                        
                       
                         [ 
                         
                           
                             
                               ∑ 
                               
                                 n 
                                 = 
                                 1 
                               
                               N 
                             
                              
                             
                               
                                 f 
                                 2 
                               
                                
                               
                                 ( 
                                 
                                   x 
                                   
                                     ( 
                                     n 
                                     ) 
                                   
                                 
                                 ) 
                               
                             
                           
                           - 
                           
                             
                               ∫ 
                               τ 
                             
                              
                             
                               
                                 
                                   f 
                                   2 
                                 
                                  
                                 
                                   ( 
                                   x 
                                   ) 
                                 
                               
                                
                               d 
                                
                               x 
                             
                           
                         
                         ] 
                       
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   15 
                   ) 
                 
               
             
           
         
       
     
     Substituting Equation (7) into Equation (15), the first term on the right hand side of Equation (15) results in a sum of one-dimensional integrals that are straightforward to perform using any well-known numerical integration scheme (for example, adaptive quadrature), and the computational cost of evaluating this term is therefore proportional to N, the number of data points. The second term involves a nested integral that is prima facie doubly intractable. However, the outer integral is able to be performed explicitly, leading to the second term being given by a one-dimensional integral −˜ τ {(k u (x) T K uu   −1 m) 2 +k u (x) T [K uu   −1 ΣK uu   −1 −K uu   −1 ]k u (x)}dx. Due to the form of K uu  given by Equation (12), the number of operations necessary to calculate the inverse K uu   −1  is proportional to M, as opposed to being proportional to M 3  as would be the case for a general matrix of size (2M+1)×(2M+1). The integrals involving k u (x) are calculated in closed form using the calculus of elementary functions, and therefore the right hand side of Equation (15) is tractable. 
     The second term on the right hand side of Equation (10) is evaluated as in Equation (16) to give 
     
       
         
           
             
               
                 
                   
                     - 
                     
                       
                          
                         
                           q 
                            
                           
                             ( 
                             u 
                             ) 
                           
                         
                       
                        
                       
                         [ 
                         
                           log 
                            
                           
                             
                               q 
                                
                               
                                 ( 
                                 u 
                                 ) 
                               
                             
                             
                               p 
                                
                               
                                 ( 
                                 u 
                                 ) 
                               
                             
                           
                         
                         ] 
                       
                     
                   
                   = 
                   
                     
                       1 
                       2 
                     
                      
                     
                       
                         ( 
                         
                           M 
                           - 
                           
                             log 
                              
                             
                                
                               
                                 K 
                                 
                                   u 
                                    
                                   u 
                                 
                               
                                
                             
                           
                           - 
                           
                             log 
                              
                             
                                
                               Σ 
                                
                             
                           
                           - 
                           
                             tr 
                              
                             
                               [ 
                               
                                 
                                   K 
                                   
                                     u 
                                      
                                     u 
                                   
                                   
                                     - 
                                     1 
                                   
                                 
                                  
                                 
                                   ( 
                                   
                                     
                                       mm 
                                       T 
                                     
                                     + 
                                     Σ 
                                   
                                   ) 
                                 
                               
                               ] 
                             
                           
                         
                         ) 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   16 
                   ) 
                 
               
             
           
         
       
     
     As discussed above, the number of operations required to calculate the inverse K uu   −1  is proportional to M. Similarly, the number of operations required to calculate the determinants |K uu | and |Σ| are proportional to M. The computational complexity of evaluating the ELBO is therefore O(N+M), where O denotes the asymptotic order as N,M→∞. 
     The operations discussed above will now be summarised with reference to  FIG. 9 . As shown, data is received, at S 901 , corresponding to a discrete set of points. A variational distribution is then generated, at S 903 , depending on a predetermined prior distribution, the variational distribution comprising a plurality of Fourier components. Next, a set of parameters is determined, at S 905 , such that the variational distribution approximates a posterior distribution conditioned on the data. The variational distribution is then squared, at S 907 , to determine a stochastic intensity function. 
     Extension of Variational Gaussian Process Method to D Dimensions 
     The method of generating a probabilistic model described in the previous section is straightforwardly extended to multiple dimensions. Extending the method to multiple dimensions is necessary for many applications in which a probabilistic model is generated to be incorporated into a reinforcement learning environment. In an example of managing a fleet of taxis in a city, the domain over which a probabilistic model is generated includes one temporal dimension and two spatial dimensions corresponding to a two-dimensional representation of the city, and therefore D=3. 
     Two ways of extending the method described above to multiple dimensions are discussed below. 
     Method 1: Additive Kernels 
     The simplest way to extend the method above to multiple dimensions is to use a prior that is a sum of independent Gaussian processes corresponding to the D dimensions of the domain, as shown in equation (17): 
     
       
         
           
             
               
                 
                   
                     
                       f 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                     = 
                     
                       
                         ∑ 
                         
                           d 
                           = 
                           1 
                         
                         D 
                       
                        
                       
                         
                           f 
                           d 
                         
                          
                         
                           ( 
                           
                             x 
                             d 
                           
                           ) 
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   17 
                   ) 
                 
               
             
           
         
       
     
     in which f d ˜GP(0, k d (x d , x d ′)). For each dimension, the kernel k d (x d , x d ′) has a form compatible with the one-dimensional method described above (for example, each may be a Matérn kernel of half-integer order). This leads to a prior having an additive Kernel, as shown in Equation (18): 
     
       
         
           
             
               
                 
                   
                     f 
                      
                     
                       ( 
                       x 
                       ) 
                     
                   
                   ~ 
                   
                     GP 
                      
                     
                       ( 
                       
                         0 
                         , 
                         
                           
                             ∑ 
                             
                               d 
                               = 
                               1 
                             
                             D 
                           
                            
                           
                             
                               k 
                               d 
                             
                              
                             
                               ( 
                               
                                 
                                   x 
                                   d 
                                 
                                 , 
                                 
                                   x 
                                   d 
                                   ′ 
                                 
                               
                               ) 
                             
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   18 
                   ) 
                 
               
             
           
         
       
     
     A matrix of features is constructed in analogy to the inducing variables of the one-dimensional case, such that u m,d =P ϕ     m   (f d ), resulting in DM features. It is straightforward to show that cov(u m,d , u m,d′ )=0 for d≠d′, and hence the K uu  matrix in the additive case is of block-diagonal form K uu =diag(K uu   (1) , . . . , K uu   (D) ), where each of the matrices K uu   (d)  for d=1, . . . , D takes the convenient form given by Equation (12). 
     For the additive kernel case, the ELBO is tractable analogously to the one-dimensional case above, and the method proceeds with analogy to the one-dimensional case. The computational complexity increases linearly with the number of dimensions, making the additive kernel particularly suitable for high-dimensional problems. 
     Method 2: Separable Kernels 
     A second way to extend the method above to multiple dimensions is to use a prior with a separable kernel, as shown in Equation (19): 
     
       
         
           
             
               
                 
                   
                     
                       f 
                        
                       
                         ( 
                         x 
                         ) 
                       
                     
                     ~ 
                     
                       GP 
                        
                       
                         ( 
                         
                           0 
                           , 
                           
                             
                               ∑ 
                               
                                 d 
                                 = 
                                 1 
                               
                               D 
                             
                              
                             
                               
                                 k 
                                 d 
                               
                                
                               
                                 ( 
                                 
                                   
                                     x 
                                     d 
                                   
                                   , 
                                   
                                     x 
                                     d 
                                     ′ 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                         ) 
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   19 
                   ) 
                 
               
             
           
         
       
     
     where each kernel factor k d (x d , x d ′) has a form compatible with the one-dimensional method described above. A vector of features of length M D  is constructed as the Kronecker product of truncated Fourier bases over [a d , b d ] for each dimension, as shown in Equation (20): 
       ϕ( x )=⊗ d [ϕ 1 ( x   d ), . . . ,ϕ M ( x   d )] T .  (20)
 
     Inducing variables u are defined analogously to the one-dimensional case, with u m =P ϕ     m   (f). The resulting K uu  matrix in the separable case is given by the Kronecker product of K uu =⊗ d K uu   (d) , where each of the matrices K uu   (d)  for d=1, . . . , D takes the convenient form given by Equation (18). 
     For the separable kernel case, the number of inducing variables grows exponentially with the number of dimensions, allowing for very detailed representations with many basis functions. The ELBO is still tractable and the required integrals can still be calculated in closed form. However, the computational complexity is proportional to M D , and therefore the separable kernel case may require more computational resources than the additive kernel case for cases of high dimensions. 
     Correctness by Learning 
     In the following section, a novel method is discussed for avoiding bad states in a system referred to as a transition system. In such a system, at a discrete set of time steps, a collaborative group of agents (referred to collectively as a composite agent) perform actions simultaneously on an environment, causing the environment to transition from one state to another. A wide variety of complex software systems can be described as transition systems and the algorithm described hereafter is applicable to any of these, leading to runtime enforcement of correct behaviour in such software systems. In some examples, the agents correspond to real-world entities . . . 
     At a given time step, a co-ordinator receives state signals from N A  agents, each state signal indicating a component state s i ∈Q i  experienced by one of the agents, where Q i  is the set of all possible component states that the i th  agent can experience. Each set Q i  for i=1, . . . , N A  may be finite or infinite, depending on the specific transition system. A composite state s∈Q, referred to hereafter as a state s, where Q⊗ i=1   N     A   Q i , is a tuple of all of the component states s i  experienced by the N A  agents. A subset {tilde over (Q)}⊂Q of states are defined as bad states. 
     The co-ordinator receives state signals in the form of feature vectors q i (s) for i=1, 2, . . . , N A . In response to receiving state signals indicating a state s, the co-ordinator selects and performs an interaction a from a set Γ s ⊆Γ of available interactions in the state s, based on a policy π, where Γ is the set of all possible interactions in the transition system. Performing an interaction means instructing each of the N A  agents to perform an action from a set of actions that are available to that agent, given the state of the agent. In some interactions, the co-ordinator may instruct one or more of the agents not to perform any action. For some states, several interactions will be possible. The objective of the present method (referred to as the correctness by learning method) is to learn a policy for the co-ordinator such that choosing interactions in accordance with the policy leads to the reliable avoidance of bad states. 
       FIG. 10  shows an example of a simple transition system. In this example, the problem system includes 9×9 grid  1001 , and four robots, referred to collectively as robots  1003  and labelled Robot  0 ,  1 ,  2 , and  3  respectively. Robots  1003  are synchronised such that for i=0, 1, 2, 3, Robot i must move simultaneously with, in the same direction as, either Robot i−1 (modulo 4) or Robot i+1 (modulo 4). Furthermore, robots  1003  are only permitted to move one square at a time, and only in the right or upwards directions. For example, given the state shown in  FIG. 10 , one possible interaction is for Robot  1  and Robot  2  both to move one square to the right, as indicated by the solid arrows. Another possible interaction is for Robot  0  and Robot  1  both to move one square upwards, as indicated by the dashed arrow. Grid  1001  includes exit square  1005  in the centre of the right hand column, labelled E. The remaining robots continue, with Robot i moving simultaneously with either Robot i−1 (modulo 3) or Robot i+1 (modulo 3). The squares in the upper row and the squares above the exit square in the right hand column are bad squares  1007 , labelled B. For a single episode, robots  1003  are assigned starting locations within dashed box  1009  (level with, or below, exit square  1005 , and not including the right-hand column). The aim of the problem is to learn, for any given starting locations of robots  1003 , a policy that guides all of the robots  1003  to exit square  1005 , without any of the robots  1003  landing on a bad square  1007 . In other examples, the problem is extended straightforwardly to other N s ×N s  grids for which N s  is an odd number, and for other integer numbers N R  of robots. 
     The present problem illustrates an advantage of the present method over known runtime-enforcement tool sets such as Runtime-Enforcement Behaviour Interaction Priority, referred to hereafter as RE-BIP, and previous game-theoretic methods. In contrast with the present method, these methods are all limited to one-step recovery, meaning that if the transition system enters a correct state from which all reachable states are bad states, the method fails. For example, in the state shown in  FIG. 10 , if Robot  1  moves upward, it will enter a region of grid  1001  which is not a bad state, but for which it will eventually always reach a state for which it is only possible to reach a bad state. As a results, any method that is limited to one-step recovery will fail if such a state is encountered. Methods limited to one-step recovery therefore cannot be used to solve the present problem. 
     In order for the data processing system of  FIG. 4  to apply the correctness by learning method to the present problem, an agent is assigned to each of the four robots  1003 , along with a co-ordinator that receives state signals from the agents and sends instructions to the agents, causing the robots to move in accordance with the instructions. In this example, robots  1003  and grid  1001  are both virtual entities (the problem system is virtual), but in another embodiment, the robots are physical entities moving on a physical grid (in which case, the problem system is physical) and the agents send control signals to the robots, causing them to move. In either case, the environment is a virtual representation of the grid, indicating the locations of each of the robots. At each time step, for i=0,1, 2, 3, Agent i assigned to Robot i sends a state signal to the co-ordinator in the form of a 2-component vector q i (s i )=(x, y) T , where (x, y)∈[0,8] 2  encodes integer Cartesian co-ordinates of the robot. 
     Returning to the case of a general transition system, at time step n the co-ordinator receives state signals indicating a state S n , performs an interaction A n , and receives updated state signals indicating a new state S n+1 . As described above with regard to reinforcement learning algorithms, a reward function R(s) is associated with the each state encountered. In this example, the reward function is given by Equation (21): 
     
       
         
           
             
               
                 
                   
                     R 
                      
                     
                       ( 
                       s 
                       ) 
                     
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             
                               
                                 
                                   
                                     R 
                                     + 
                                   
                                    
                                   
                                       
                                   
                                    
                                   for 
                                    
                                   
                                       
                                   
                                    
                                   s 
                                 
                                 ∉ 
                                 
                                   Q 
                                   ~ 
                                 
                               
                               , 
                             
                           
                         
                         
                           
                             
                               
                                 
                                   R 
                                   - 
                                 
                                  
                                 
                                     
                                 
                                  
                                 for 
                                  
                                 
                                     
                                 
                                  
                                 s 
                               
                               ∈ 
                               
                                 Q 
                                 ~ 
                               
                             
                           
                         
                       
                       , 
                     
                   
                 
               
               
                 
                   ( 
                   21 
                   ) 
                 
               
             
           
         
       
     
     where R + &gt;R − . In a specific example, R + =1 and R − =−1. 
     In this example, the task associated with the problem is treated as being episodic (as is the case in the example problem illustrated by  FIG. 10 ), although it is also straightforward to apply the method described hereafter to problems having continuous tasks by breaking the continuous task into episodes with a predetermined number of time steps. The return associated with an initial state S 0 =s is given by substituting n=0 into Equation (1). The state value function for the state s is therefore given by Equation (22): 
     
       
         
           
             
               
                 
                   
                     
                       
                         v 
                         π 
                       
                        
                       
                         ( 
                         s 
                         ) 
                       
                     
                     = 
                     
                       
                          
                         π 
                       
                        
                       
                         ( 
                         
                           
                             ∑ 
                             
                               j 
                               = 
                               0 
                             
                             
                               T 
                               - 
                               1 
                             
                           
                            
                           
                             
                               γ 
                               j 
                             
                              
                             
                               R 
                                
                               
                                 ( 
                                 
                                   S 
                                   
                                     j 
                                     + 
                                     1 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                         ) 
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   22 
                   ) 
                 
               
             
           
         
       
     
     where for each episode, T is the number of time steps in the episode. The method proceeds with the objective of finding an optimal policy π* such that the state value function v 90  (s) is maximised for all states s∈Q. 
       FIG. 11  shows server  1101  configured to implement a learning subsystem in accordance with the present invention in order to implement the correctness by learning algorithm described hereafter. In this example, the learning subsystem is implemented using a single server, though in other examples the learning subsystem is distributed over several servers as described elsewhere in the present application. Server  1101  includes power supply  1103  and system bus  1105 . System bus  1105  is connected to: CPU  1107 ; communication module  1109 ; memory  1111 ; and storage  1113 . Memory  1111  stores program code  1115 ; DNN code  1117 ; experience buffer  1121 ; and replay memory  1123 . Storage  1113  stores skill database  1125 . Communication module  1109  receives experience data from an interaction subsystem and sends policy data to the interaction subsystem (thus implementing a policy sink). 
       FIG. 12  shows DNN  1201  used by server  1101  to implement the correctness by learning algorithm. DNN  1201  is similar to DNN  601  of  FIG. 6 , but in contrast to DNN  601 , DNN  1201  is used to estimate action value functions, rather than state value functions. The approximate action value functions learned are denoted {circumflex over (q)}(s, a, w), which depend on: (composite) state s; interaction a; and weight vector w, where weight vector w contains the elements of the connection weight matrices Θ (j)  of DNN  1201 . The specific architecture of DNN  1201  is illustrative, and different architectures will be suitable for different transition systems, depending on the complexity and nature of the approximate action value function to be learnt. In contrast to output layer  609  of DNN  601 , which had only one node, output layer  1209  of DNN  1101  has |Γ| nodes, where |Γ| denotes the number of elements in the set Γ of possible interactions. Input layer  1203  of DNN  1201  has M=N A ×N F  nodes, where N A  is the number of agents and N F  is the number of features in each feature vector q i (s). For example, in the problem of  FIG. 10 , M=4×2=8. Data associated with DNN  1201 , including data corresponding to the network architecture and the connection weights, is stored as DNN data  1117  in memory  1111 . 
     As shown in  FIG. 13 , alternative DNN  1301  has the same architecture as DNN  1101 , but the connection weights are given by alternative weight vector {tilde over (w)}, corresponding to alternative weight matrices {tilde over (Θ)} (j)  of DNN  1301 . 
       FIG. 14  shows local computing device  1401  configured to implement an interaction subsystem in accordance with the present invention in order to implement the correctness by learning algorithm described hereafter. Local computing device  1401  includes power supply  1403  and system bus  1405 . System bus  1405  is connected to: CPU  1407 ; communication module  1409 ; memory  1411 ; storage  1413 ; and input/output (I/O) devices  1415 . Memory  1411  stores program code  1417 ; environment  1419 ; agent data  1421 ; and policy data  1423 . In this example, I/O devices  1415  include a monitor, a keyboard, and a mouse. Communication module  1409  receives policy data from server  1101  (thus implementing a policy source) and sends experience data to server  1101  (thus implementing an experience sink). 
     In order to implement the correctness be learning algorithm, server  1101  and local computing device  1401  execute program code  1115  and program code  1417  respectively, causing the routine of  FIG. 15  to be implemented. The routine begins with server  1101  randomly initialising, at S 1501 , the connection weights of DNN  1201  in an interval [−δ, δ], where δ is a small positive parameter. Server  1101  transfers copies of the randomly initialised connection weights of DNN  1201  to local computing device  1401 , where they are saved as policy data  1423 . Server  1101  also updates alternative DNN  1301  to have the same connection weights as DNN  1201 . 
     Server  1101  then initialises, at S 1503 , replay memory  1123  to store experience data corresponding to a number N T  of transitions. 
     The routine now enters an outer loop corresponding to episodes of the transition system task. For each of a total number M of episodes, local computing device  1401  sets, at S 1505 , an initial state S 0  of the transition system. In some examples, the initial state is selected randomly. In other examples, the initial state is selected as a state from which all other states in the system can be reached. In the example of  FIG. 10 , the initial state is set with all four of robots  1003  at the bottom left square, so that q i (S 0 )=(0,0) T  for i=1,2,3,4, which is the only state for which all other possible states of the system are able to be reached. For transition systems in which it is not clear to a user which states are able to be reached from which other states, the initial state should be selected randomly. This may be the case, for example, in transition systems for which the set Q of states is infinite. 
     After the initial state has been set, the routine enters an inner loop corresponding to the T time steps in the episode. 
     For each time step in the episode, computing device  1401  calculates, at S 1507 , approximate action values {circumflex over (q)}(S j , a, w) by inputting the feature vectors q i (S j ) for i=1, . . . , N A  into the copy of DNN  1201  saved in policy data  1323 , and applying forward propagation. The approximate action values are given by the activations of the nodes in the output layer of the copy of DNN  1201 . 
     Next, the co-ordinator selects and performs, at S 1509 , an interaction A j =a from a set Γ s ⊆Γ of available interactions in the state S j =s. Specifically, the co-ordinator stochastically selects either an optimal interaction (at S 1511 ) or a random interaction (at S 1513 ). The probability of selecting a random interaction is given by ε, where ε is a parameter satisfying 0&lt;ε&lt;1, and accordingly the probability of selecting an optimal interaction is 1−ϵ. In this example, selecting a random interaction means selecting any interaction from the set Γ s  of available interactions, with each interaction in Γ s  having an equal probability of being selected. Selecting an optimal interaction, on the other hand, means selecting an interaction according to a greedy policy TE defined by Equation (23): 
       π( s )=argmax{ {circumflex over (q)} ( s,a,w )| a∈Γ   s },  (23)
 
     which states that the policy π selects the interaction a from the set Γ s  that has the highest approximate action value function {circumflex over (q)}(s, a, w), as calculated at S 1507 . According to the above rule, the co-ordinator follows an ε-greedy policy. 
     After the co-ordinator performs an interaction according to the rule above, the agents send a new set of state signals to the co-ordinator, indicating a new state S j+1  along with a reward R(S j+1 ), calculated in this example using Equation (21). Local computing device  1401  sends experience data corresponding to the transition to server  1101 . Server  1101  stores, at S 1515 , the transition in the form of a tuple (S j , A j , S j+1 , R(S +1 )), in replay memory  1123 . Server  1101  samples, at S 1517 , a mini-batch of transitions from replay memory  1123  consisting of N 2  tuples of the form (S k , A k , S k+1 , R(S k+1 )), where N 2 ≤N T . 
     For each of the transitions in the sampled mini-batch, server  1101  assigns, at S 1519 , an output label y k  using the rule of Equation (24) below: 
     
       
         
           
             
               
                 
                   
                     y 
                     k 
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             
                               
                                 
                                   R 
                                    
                                   
                                     ( 
                                     
                                       S 
                                       
                                         k 
                                         + 
                                         1 
                                       
                                     
                                     ) 
                                   
                                 
                                  
                                 
                                     
                                 
                                  
                                 if 
                                  
                                 
                                     
                                 
                                  
                                 
                                   S 
                                   
                                     k 
                                     + 
                                     1 
                                   
                                 
                                  
                                 
                                     
                                 
                                  
                                 is 
                                  
                                 
                                     
                                 
                                  
                                 a 
                                  
                                 
                                     
                                 
                                  
                                 bad 
                                  
                                 
                                     
                                 
                                  
                                 state 
                               
                               , 
                             
                           
                         
                         
                           
                             
                               
                                 R 
                                  
                                 
                                   ( 
                                   
                                     S 
                                     
                                       k 
                                       + 
                                       1 
                                     
                                   
                                   ) 
                                 
                               
                               + 
                               
                                 γ 
                                  
                                 
                                     
                                 
                                  
                                 max 
                                  
                                 
                                   { 
                                   
                                     
                                       
                                         q 
                                         ^ 
                                       
                                        
                                       
                                         ( 
                                         
                                           
                                             S 
                                             
                                               k 
                                               + 
                                               1 
                                             
                                           
                                           , 
                                           a 
                                           , 
                                           
                                             w 
                                             ~ 
                                           
                                         
                                         ) 
                                       
                                     
                                     | 
                                     
                                       a 
                                       ∈ 
                                       
                                         Γ 
                                         s 
                                       
                                     
                                   
                                   } 
                                 
                                  
                                 
                                     
                                 
                                  
                                 otherwise 
                               
                             
                           
                         
                       
                       , 
                     
                   
                 
               
               
                 
                   ( 
                   24 
                   ) 
                 
               
             
           
         
       
     
     which states that if S k+1  is a bad state, y k  is given by the evaluation of the reward function associated with S k+1 , and if S k+1  is not a bad state, y k  is given by the evaluation of the reward function associated with S k+1 , added to the product of a discount factor γ and the highest approximate action value from the state S k+1 , as calculated using alternative DNN  1301 . 
     Server  1101  retrains DNN  1201  by treating (S k ,y k ) for k=1, . . . , N 2  as labelled training examples. Training DNN  1201  in this example includes inputting the feature vectors q i (S k ) for i=1, . . . , N A  into DNN  1201  and applying the well-known supervised learning technique of forward propagation, backpropagation, and gradient descent, to update the connection weights of DNN  1201 . 
     The method of retraining DNN  1201  using a randomly sampled mini-batch of transitions is referred to as experience replay. Compared with the nave alternative of retraining DNN  1201  using a chronological sequence of transitions, experience replay ensures that data used in retraining DNN  1201  is uncorrelated (as opposed to training a DNN using successive transitions, which are highly correlated), which reduces the probability of the gradient descent algorithm leading to a set of connection weights corresponding to a local minimum. Furthermore, experience replay allows the same transitions to be used multiple times in retraining DNN  1201 , thereby improving the efficiency of the training with respect to the number of transitions experienced. 
     At the end of every K episodes, where K&lt;M, server  1101  updates, at S 1523 , alternative DNN  1301  to have the same connection weights as DNN  1201 . 
     After the outer loop has executed M times, server  1101  saves the connection weights of DNN  1201  in skill database  1125 . 
     Fairness in Correctness by Learning 
     In the correctness by learning algorithm described above, the co-ordinator follows an ε-greedy policy, meaning that the co-ordinator selects a greedy interaction according to Equation (23) with probability 1−ε. In another example, the greedy policy of Equation (23) is replaced with the fair policy of Equation (25): 
       π( s )={ a|a∈Γ   s   Λ{circumflex over (q)} ( s,a,w )&gt;max{ {circumflex over (q)} ( s,a,w )}− F},   (25)
 
     which states that the co-ordinator randomly selects an interaction a from all of the interactions in the set Γ s  that are within a tolerance F&gt;0 of the interaction having the maximum estimated action value function. The value of the tolerance parameter F is configurable and a higher value of F leads to more deviation from the optimal policy. The policy of Equation (25) allows the transition system to learn traces that are different from the optimal trace (corresponding to the policy of Equation (21)) but which also avoid bad states. 
     The above embodiments are to be understood as illustrative examples of the invention. Further embodiments of the invention are envisaged. For example, a range of well-known reinforcement learning algorithms may be applied by a learner, depending on the nature of a reinforcement learning problem. For example, for problems having tasks with a relatively small number of states, in which all of the possible states are provided, synchronous or asynchronous dynamic programming methods may be implemented. For tasks having larger or infinite numbers of states, Monte Carlo methods or temporal-difference learning may be implemented. Reinforcement learning methods using on-policy approximation or off-policy approximation of state value functions or action value functions may be implemented. Supervised-learning function approximation may be used in conjunction with reinforcement learning algorithms to learn approximate value functions. A wide range of linear and nonlinear gradient descent methods are well-known and may be used in the context of supervised-learning function approximation for learning approximate value functions. 
     It is to be understood that any feature described in relation to any one embodiment may be used alone, or in combination with other features described, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the invention, which is defined in the accompanying claims. 
     Modifications and Further Embodiments 
     In some examples, the invention can incorporate Mechanism Design, which is a field in economics and game theory that takes an engineering approach to designing incentives, toward desired objectives, in strategic settings, assuming players act rationally. For example, in a ridesharing company or a fleet management problem as the one previously described, in order to arrive to a solution that is good for the parties to the system (i.e. city council, taxi company, passengers and drivers), their preferences among different alternative results is considered (e.g. a specific task allocation) using mechanism design principles together with learning techniques to assess preferences of the parties in such a way that the parties willingly share this information and have no incentive to lie about it.