Neural network reinforcement learning

A computer-implement method and an apparatus are provided for neural network reinforcement learning. The method includes obtaining, by a processor, an action and observation sequence. The method further includes inputting, by the processor, each of a plurality of time frames of the action and observation sequence sequentially into a plurality of input nodes of a neural network. The method also includes updating, by the processor, a plurality of parameters of the neural network by using the neural network to approximate an action-value function of the action and observation sequence.

BACKGROUND

Technical Field

The present invention relates to reinforcement learning with a neural network, and, in particular, to reinforcement learning with a neural network modelling a Partially Observable Markov Decision Process (POMDP).

Description of the Related Art

One of the major challenges for Reinforcement Learning (RL) is to learn near optimal policies in high-dimensional state or action spaces, especially when there is non-Markovian or partially observable state space. There has been recent progress in learning human level control policies on different Atari® games or even tackle the high-dimensional state, action space for the game of Go. However, most of these are suitable for Markovian environments and have very limited memory unless coupled with additional recurrent networks.

Previous work on energy-based RL has been mainly focused on Restricted Boltzmann Machines (RBMs), where the action-value function is approximated by the negative free energy of an RBM and trained using TD-learning. However, due, to the hidden layer of RBMs, this amounts to TD-learning with a non-linear value function. Non-linear TD learning, however, is known to diverge in theory and is highly unstable in practice. Furthermore, these methods cannot directly deal with POMDP problems requiring memory of past actions and observations.

SUMMARY

According to an aspect of the present invention, a computer program product is provided for neural network reinforcement learning. The computer program product includes a non-transitory computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the computer to perform a method. The method includes obtaining, by a processor, an action and observation sequence. The method further includes inputting, by the processor, each of a plurality of time frames of the action and observation sequence sequentially into a plurality of input nodes of a neural network. The method also includes updating, by the processor, a plurality of parameters of the neural network by using the neural network to approximate an action-value function of the action and observation sequence.

According to another aspect of the present invention, a computer-implemented method is provided. The method includes obtaining, by a processor, an action and observation sequence. The method further includes inputting, by the processor, each of a plurality of time frames of the action and observation sequence sequentially into a plurality of input nodes of a neural network. The method also includes updating, by the processor, a plurality of parameters of the neural network by using the neural network to approximate a function for determining a subsequent action based on the action and observation sequence.

According to yet another aspect of the present invention, an apparatus is provided. The apparatus includes a processor. The processor is configured to obtain an action and observation sequence. The processor is further configured to input each of a plurality of time frames of the action and observation sequence sequentially into a plurality of input nodes of a neural network. The processor is also configured to update a plurality of parameters of the neural network by using the neural network to approximate a function for determining a subsequent action based on the action and observation sequence.

DETAILED DESCRIPTION

Hereinafter, example embodiments of the present invention will be described. The example embodiments shall not limit the invention according to the claims, and the combinations of the features described in the embodiments are not necessarily essential to the invention.

Embodiments of the present invention may perform reinforcement learning on a neural network, such as neural networks adapted for a POMDP.

The recently introduced Dynamic Boltzmann Machine (DyBM) provides a particularly structured Boltzmann Machine (BM), as a generative model of multi-dimensional time-series. This BM can have infinitely many layers of units but allows exact interference and learning based on Spike Timing Dependent Plasticity (STDP). Embodiments of the present invention may extend the original DyBM to reinforcement learning problems by including a novel Temporal-Difference (TD) modulated STDP rule for learning with DyBMs that may effectively deal with high dimensional action spaces, and may also solve Partially Observable Markov Decision Process (POMDP) tasks. Using the energy of a DyBM in order to approximate an action-value Q-function, near optimal policy control may be achieved. Value functions may be parameterized using free-energy-based models, trained using non-linear TD-learning. While previous methods were prone to divergence due to non-linear TD, the energy function in a DyBM is linear with respect to its parameters and may theoretically guarantee convergence in the limit of a sufficiently large exploration. Algorithms using energy-based spike timing TD-learning may converge to near-optimal solutions, and may outperform previous energy-based methods.

Embodiments for RL using energy-based policies may utilize TD-learning with a linear value function, thus not suffering from divergence issues. Specific embodiments may employ TD-learning called DySARSA using the architecture of DyBM, which was proposed as a generative model of a high-dimensional time-series. Embodiments may use the energy of a DyBM to approximate an action-value function Q, and learn near-optimal policies with Boltzmann exploration. DyBM may be made very deep by unfolding through time, allowing infinitely many layers. In embodiments having a DyBM with no hidden units, the energy function of a DyBM may be linear in its parameters, and DySARSA may not suffer from divergence issues. In some embodiments, DyBM may be viewed as a fully connected Recurrent Neural Network (RNN) with memory units and with conduction delays between units in the form of First-in First-Out (FIFO) queues that can store long temporal history of inputs. This architecture may enable the DySARSA algorithm to make use of the long memory of prior actions and observations in order to learn optimal policies in POMDP scenarios.

A standard DyBM may be motivated by postulates and observations from biological neural networks, such that it may allow exact inference and spike timing dependent learning of its parameters. In some embodiments, using FIFO queues and a formulation of neural, and synaptic eligibility traces, spike timing information may be propagated between units (neurons) in the network. The precise spike timing information may be crucial for learning in biological systems, and this forms one of the motivations for RL inspired by, reward-modulated spiking timing dependent plasticity. Therefore, embodiments may use an energy-based linear TD learning algorithm that can utilize such spike timing information available in a DyBM to solve memory intensive POMDP tasks or with high-dimensional action spaces. Embodiments with DySARSA may converge to near optimal solutions in a reasonably fast time, while outperforming previous RBM based energy methods and RNN-based RL models dealing specifically for POMDPs.

FIG. 1shows an apparatus100for neural network reinforcement learning, according to an embodiment of the present invention. Apparatus100may a host computer such as a server computer or a mainframe computer that executes an on-premise application and hosts client computers that use it. Apparatus100may be a computer system that includes two or more computers. Alternatively, apparatus100may be a personal computer that executes an application for a user of apparatus100. Apparatus100may perform reinforcement learning on a neural network adapted for an action and observation sequence by using the neural network to approximate an action-value function of the action and observation sequence, and updating the parameters of the neural network based on an action-value determined from the action-value function.

Apparatus100may include an obtaining section101, which may include a selecting section102including a probability evaluating section103, and a causing section104, an inputting section105, and an updating section106, which may include an action-value evaluating section107, a caching section108, and a calculating section109. Apparatus100may be a computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform the operations of the various sections. Apparatus100may alternatively be analog or digital programmable circuitry, or any combination thereof. Apparatus100may alternatively be a computer on which the computer program product installed. Apparatus100may be composed of physically separated storage or circuitry that interacts through communication.

Apparatus100may interact with action and observation sequence110, which may be a person, a machine, or other object subject to modelling as a POMDP. The observations may be observed through sensors, and actions may be caused through instructions or physical interaction. Action and observation sequence110may be represented by a computer program, such as a game, which is bound by a digitally created environment. Such a computer program may be observed, by receiving data output from the program, and actions may be caused by issuing commands to be executed by the computer program.

Obtaining section101may receive data from data storage in communication with apparatus100. For example, obtaining section101may be operable to obtain an action and observation sequence, such as action and observation sequence110. Action and observation sequence110may be obtained sequentially as the actions are performed and the observations are observed. For example, obtaining section101may be operable to obtain an observation of a subsequent time frame of action and observation sequence110. Alternatively, obtaining section101may be operable to obtain an entire action and observation sequence for a set of time frames, such as a training sequence, complete with actions and observations at each time frame. Obtaining section101may communicate directly with such data stores, or may utilize a transceiver to communicate with a computer through wired or wireless communication across a network.

Selecting section102may select an action. For example, selecting section102may be operable to select an action with which to proceed from a current time frame of the action and observation sequence to a subsequent time frame of action and observation sequence110.

Probability evaluation section103may evaluate a reward probability of a possible action. For example probability evaluating section103may be operable to evaluate each reward probability of a plurality of possible actions according to a probability function based on action-value function, such as action-value function112. In many embodiments, selected section102may select the possible action that yields the largest reward probability from the probability function.

Causing section104may cause an action to be performed. For example, causing section104may be operable to cause the action selected by selecting section102to be performed in the subsequent time frame of action and observation sequence110.

Inputting section105may input values into input nodes of a neural network. For example, inputting section105may be operable to input each time frame of action and observation sequence110sequentially into a plurality of input nodes of a neural network, such as neural network120.

Updating section106may update the parameters of a neural network, such as neural network120. For example, updating section106may update a plurality of parameters a neural network120by using neural network120to approximate action-value function112based on action and observation sequence110.

Action-value determining section107may determine an action-value. For example, action-value determining section107may be operable to determine a current action-value from an evaluation of action-value function112in consideration of an actual reward.

Caching section108may cache values and parameters for functions and neural networks. For example, caching section108may be operable to cache a previous action-value determined for a previous time frame from action-value function112. Caching section108may also be operable to cache parameters of neural network120, such as eligibility traces, weights, biases, and function parameters for determining such parameters of neural network120.

Calculating section109may calculate parameters. For example, calculating section109may be operable to calculate a temporal difference error based on the previous action-value, the current action-value, and the plurality of parameters of neural network120.

An apparatus, such as apparatus100, may be useful for DyBM. Apparatus100can also be used for any neural network model adapted for an action and observation sequence. A DyBM may be defined from BM having multiple layers of units, where one layer represents the most recent values of a time-series, and the remaining layers represent the historical values of the time-series. The most recent values are conditionally independent of each other given the historical values. It may be equivalent to such a BM having an infinite number of layers, so that the most recent values can depend on the whole history of the time series (see supplementary for DyBM∞). For unsupervised learning, a DyBM may be trained in such a way that the likelihood of a given time-series is maximized with respect to the conditional distribution of the next values given the historical values. Similar to a BM, a DyBM may consist of a network of artificial neurons. In some embodiments using a DyBM, each neuron ay take a binary value, 0 or 1, following a probability distribution that depends on the parameters of the DyBM. In other embodiments using a DyBM, each neuron may take a real value, an integer value, or a multi-value. Unlike the BM, the values of the DyBM can change over time in a way that depends on its previous values. That is, the DyBM may stochastically generate a multi-dimensional series of binary values.

Learning in conventional BMs may be based on a Hebbian formulation, but is often approximated with a sampling based strategy like contrastive divergence. In this formulation, the concept of time is largely missing. In DyBM, like biological networks, learning may be dependent on the timing of spikes. This is called spike-timing; dependent plasticity, or STDP, which means that a synapse is strengthened if the spike of a pre-synaptic neuron precedes the spike of a post-synaptic neuron (Long Term Potentiation—LTP), and the synapse is weakened if the temporal order is reversed (Long Term Depression—LTD). The conventional DyBM may use an exact online learning rule that has the properties of LTP and LTD.

In embodiments of an apparatus in which entire action and observation sequences are obtained at once, such as training sequences, the apparatus may not require a selecting section or a causing section, because the actions are already determined as part of the sequence.

FIG. 2shows a dynamic Boltzmann machine (DyBM) as an example of a neural network, according to an embodiment of the present invention. DyBM220may include a plurality of layers of nodes (e.g. layers221A,222A1,222A2,222Z1, and222Z2) among a plurality of nodes (e.g.224A,226A1,226A2,226Z1, and226Z2). Each layer sequentially forwards input values of a time frame of the action and observation sequence to a subsequent layer among the plurality of layers. The plurality of layers of nodes includes a first layer221A of input nodes, such as input node224A, and a plurality of intermediate layers, such as intermediate layer222A/222Z. In the first layer221A, the input nodes224A receive input values representing an action of a current time frame of the action and observation sequence. The plurality of layers of nodes may also include another first layer of other input nodes that receive input values representing an observation of a current time frame of the action and observation sequence.

Each node, such as action node226A and observation node226Z, in each intermediate layer forwards a value representing an action or an observation to a node in a subsequent or shared layer.FIG. 2shows three time frames, t, t−1, and t−2. Each time frame is associated with an action, A, and an observation, Z. The action at time t is represented as At. The action at time t−1 is represented as At−1, and the action at time t−2 is represented as At−2. The observation at time t−1 is represented as Zt−1, and the action at time t−2 is represented as t−2.FIG. 2does not show an observation at time t, because DyBM220is shown at a moment in which action Atis being determined, but has not been caused. Thus, in this moment, each other node is presynaptic to the nodes of action At221A. Once an action has been selected and caused, DyBM220will create input nodes for the observation at time t, Zt, for storing binary numbers representing. In other implementations, observation Z1at time t can be input to Zt−1after the current values of Zt−1, Zt−2, . . . are forwarded to Zt−2, Zt−3, . . . and the current values of At−1, At−2, . . . are forwarded to At−2, At−3, . . .

InFIG. 2, values representing an action A at time t, t−1, t−2, . . . are denoted xj[t], xj[t−1], and xj[t−2], where j(1≤j≤Nd) represents a node number relating to an action and Ndrepresents a number of values (or nodes) in an action. Values representing an observation Z at time t, t−1, and t−2, . . . are denoted xi[t], xi[t−1], xi[t−2], where i(1≤i≤Nb) represents a node number relating to an observation and Nbrepresents a number of values (or nodes in an observation.

Each action, A, and each observation, Z, at each time frame of DyBM220may be represented as a plurality of binary numbers. For example, if there are 256 possible actions, then each action can be represented as a permutation of 8 binary numerals. Input node224A is a binary numeral representing the action at time t, and is represented as xj[t]. Action node226A is a binary numeral representing the action at time t−2, and is represented as xj[t−2]. The action node representing the action at time t−1 is represented as xj[t−1]. Observation node226Z is a binary numeral representing the observation at time t−2, and is represented as xi[t−2]. The observation node representing the observation at time is represented as xi[t−1].

DyBM220may also include a plurality of weight values among the plurality of parameters of the neural network. Each weight value is to be applied to each value in the corresponding node to obtain a value propagating from a pre-synaptic node to a post-synaptic node,

FIG. 3shows a connection between a presynaptic neuron326, which has a neural eligibility trace328, and a post-synaptic neuron324via a FIFO queue325, which has a synaptic eligibility trace329, according to an embodiment of the present invention. Although the diagram of DyBM220shown inFIG. 3looks different from the diagram of DyBM220shown inFIG. 2, these diagrams represent a same or similar structure of DyBM. InFIG. 3, values from nodes xj[t−1], xj[t−2], . . . of the same j inFIG. 2are sequentially stored in a FIFO queue325(shown as xj[t−1], xj[t−2], . . . ) as an implementation of for example, node226A forwarding a value from the action and observation sequence. InFIG. 3, values from nodes xi[t−1], xi[t−2], . . . of the same i inFIG. 2are also sequentially stored in a FIFO queue325assigned to another i inFIG. 3corresponding to an i inFIG. 3.

InFIG. 3, DyBM220may consist of a set of neurons having memory units and FIFO queues. Let N be the number of neurons. Each neuron may take a binary value at each moment. For j ∈ [1, N], let xj[t]be the value of the j-th neuron at time t.

A neuron, i ∈ [1, N], may be connected to another neuron, j ∈ [1, N], with a FIFO queue of length di,j−1, where di,jis the axonal or synaptic delay of conductance, or conduction delay, from the pre-synaptic neuron, i, to the post-synaptic neuron, j. Please note that the usage of i and j inFIG. 3is different from that ofFIG. 2, since the above usage is more convenient to explain the diagram ofFIG. 3. We assume di,j≥1. At each moment t, the tail of the FIFO queue holds xi[t−1], the head of the FIFO queue holds xi[t−di,j+1]. A simile increment in time causes the value at the head of the FIFO queue to be removed, and the remaining values in the FIFO queues are pushed toward the head by one position. A new value is then inserted at the tail of the FIFO queue. Self-connections via a FIFO queue are permitted.

Each neuron stores a fixed number, L, of neural eligibility traces. For l ∈ [1, L] and j ∈ [1, N], let γj,l[t−1]be the l-th neural eligibility trace of the j-th neuron immediately before time t:
γj,l[t−1]≡Σs=−∞t−1μlt−sxj[s],  Eq. (1)
where μl∈ (0,1) is the decay rate for the l-th neural elegibility trace, i.e. the neural eligibility trace is the weighted sum of the past values of that neuron, where the recent values have greater weight than the others.

Each neuron may also store synaptic eligibility traces, where the number of the synaptic eligibility traces depends on the number of the neurons that are connected to that neuron. Namely, for each of the (pre-synaptic) neurons that are connected to a (post-synaptic) neuron j, the neuron j stores a fixed number, K, of synaptic elegibility traces. For k ∈ [1,K], let αi,j,k[t−1]be the k-th synaptic eligibility trace of the neuron j for the pre-synaptic neuron i immediately before time t:
αi,j,k[t−1]≡Σs=−∞t−di,jλkt−s−di,jxi[s],  Eq. (2)
where λk∈ (0,1) is the decay rate for the k-th synaptic eligibility traces, i.e. the synaptic eligibility trace is the weighted sum of the values that has reached that neuron, j, from a pre-synaptic neuron, i, after the conduction delay, di,j.

The values of the eligibility traces stored at a neuron, j, are updated locally at time i based on the value of that neuron, j, at time t and the values that have reached that neuron, j, at time t from its pre-synaptic neurons. Specifically,
γj,l[t]←μl(γj,l[t−1]+xj[t]),  Eq. (3)
αi,j,k[t]←λk(αi,j,k[t−1]+xit−di,j),  Eq. (4)
for l ∈ [1, L] and k ∈ [1, K], and for neurons i that are connected to j.

The learnable parameters of DyBM220are bias and weight. Specifically, each neuron, j, is associated with bias, k. Each synapse, or each pair of neurons that are connected via a FIFO queue, is associated with the weight of long term potentiation (LTP weight) and the weight of long term depression (LTD weight). The LTP weight from a (pre-synaptic) neuron, i, to a (post-synaptic) neuron, j, is characterized with K parameters, ui,j,kfor k ∈ [1, K]. The k-th LTP weight corresponds to the k-th synaptic eligibility trace for k ∈ [1, K]. The LTD weight from a (pre-synaptic) neuron, i, to a (post-synaptic) neuron, j, is characterized with L parameters, υi,j,lfor l ∈ [1, L]. The l-th LTD weight corresponds to the l-th neural eligibility trace for l ∈[1, L]. The learnable parameters are collectively denoted with θ.

Similar to the conventional BM, the energy of DyBM220determines what patterns of the values that DyBM220is more likely to generate than others. Contrary to the conventional BM, the energy associated with a pattern at a moment depends on the patterns that DyBM220has previously generated. Let x[t]=(xj[t])j∈[1,N]be the vector of the values of the neurons at time t. Let x[:t−1]=(x[s])s<tbe the sequence of the values of DyBM220before time t. The energy of DyBM220at time t depends not only on x[t]but also on x[:t−1], which is stored as eligibility traces in DyBM220. Let Eθ(x[t]|x[:t−1]) be the energy of DyBM220at time t. The lower the energy of DyBM220with particular values x[t], the more likely DyBM220takes those values. The energy of DyBM220can be decomposed into the energy of each neuron at time t:
Eθ(x[t]|x[:t−1])=Σj=1NEθ(x[t]|x[:t−1]),  Eq. (5)

The energy of the neuron j at time i depends on the value it takes as follows (see supplementary for explanation of the individual components):
Eθ(x[t]|x[:t−1])=−bjxj[t]−Σi=1NΣk=1Kui,j,kαi,j,k[t−1]xj[t]+Σi=1NΣl=1Lυi,j,lβi,j,l[t−]xj[t]+Σi=1NΣl=1Lυj,i,lγi,l[t−1]xi[s],   Eq. (6)
where ui,j,kand υi,j,lare weights, and
βi,j,l[t−1]xj[t]≡Σs=t−di,j+1t−1μls−txi[s].  Eq. (7)

To perform reinforcement learning with SARSA for a POMDP using DyBM220, we divide the set of nodes (neurons) into two groups. One group represents actions and is denoted by A. The other represents observations and is denoted by Z. That is, an action that we take at time t is denoted by a vector xA[t]≡(xj[t])j∈A, and the observation that we make immediately after we take that action is analogously denoted by xZ[t]. The pair of the action and the observation at time t is denoted by x≡(xj[t])j∈A∪Z. Here, an observation can include the information about the reward that we receive, if the past reward affect What actions will be optimal in the fixture. The actions that we take are certainly observable, but we separate the action from observation for convenience.

In some embodiments, it is also possible to predict values of an observation Z1once an action Athas been fixed in the neural network. In this case, values xi[t]in Ztcan also be predicted, and works as an input layer including input nodes xi[t]. In further embodiments, all of the values xi[t]and xj[t]of both Ztand Atmay be predicted.

DyBM exhibits some of the key properties of STDP due to its structure consisting, of conduction delays, such as pre-synaptic neuron326, and memory units, such as FIFO queue325. A neuron may be connected to another in a way that a spike from pre-synaptic neuron326, i, travels along an axon and reaches pest-synaptic neuron324, j, via a synapse after a delay consisting of a constant period, di,j. FIFO queue325causes the conduction delay. FIFO queue325may store the values of pre-synaptic neuron326for the last di,j−1 units of time. Each stored value may be pushed one position toward the head of the queue when the time is incremented by one unit. The value of pre-synaptic neuron326is thus given to post-synaptic neuron324after the conduction delay. Moreover, the DyBM aggregates information about the spikes in the past into neural eligibility trace328and synaptic eligibility trace329, which are stored in the memory units. Each neuron is associated with a learnable parameter called bias. The strength of the synapse between pre-synaptic neuron326and post-synaptic neuron324is represented by learnable parameters called weights, which may be further divided into LTP and LTD components.

FIG. 4shows an operational flow for neural network reinforcement learning, according to an embodiment of the present invention. The operational flow may provide a method of perform reinforcement learning on a neural network adapted for an action and observation sequence, such as a DyBM. The operations may be performed by an apparatus, such as apparatus100.

At S430, an obtaining section, such as obtaining section101, may obtain an action and observation sequence. More specifically, as the operational flow ofFIG. 4is iteratively performed, the iterations of the operations of S430collectively amount to an operation of obtaining the action and observation sequence. Operation S430may include operations S440, S432, and S434. Alternatively at S430, the obtaining, section may obtain an entire action and observation sequence for a set of time frames, such as a training sequence, complete with actions and observations at each time frame.

At S440, a selecting section, such as selecting section102, may select an action according to a probability function. For example, the selecting section may select an action with which to proceed from a current time frame of the action and observation sequence to a subsequent time frame of the action and observation sequence.

At S432, a causing section, such as causing section104, may cause the selected action to be performed. For example, the causing section may cause the action selected at S440to be performed in the subsequent time frame of the action and observation sequence. Depending on the nature of the action and observation sequence, actions may be caused through instructions or physical interaction, such as in the case of a human or machine, in which case the actions may be performed by the human or the machine, or caused by issuing commands to be executed by the computer program, in which case the actions are performed by the computer program.

At S434, the obtaining section may obtain an observation. For example, the obtaining section may obtain an observation of the subsequent time frame of the action and observation sequence. Once the selected action has been performed, certain observations can be sensed, detected, measured, or otherwise received by the obtaining section. The setting of reinforcement learning may be where a (Markovian) state cannot be observed (i.e., our setting is modeled as a partially observable Markov decision process or POMDP). If such a state was observable, a policy that maps a state to an action could be sought, because the future would become conditionally independent of the past given the state. In a partially observable state setting, the optimal policy may depend on the entire history of prior observations and actions, which are represented as xi[t−n]inFIG. 2. In some embodiments, the observation obtained may also include or be accompanied by an actual reward, which may reduce the number of time frames required for convergence, but may also require more computational resources. The actual reward may be a supplied through conscious feedback, such as in indication by a person, or calculated from, for example, a final state, and is therefore assumed to be factual.

At S436an input section, such as input section105, may input values corresponding to the current time frame into a neural network. As the operational flow ofFIG. 4is iteratively performed, the iterations of the operations of S436collectively amount to, the input section inputting each time frame of the action and observation sequence sequentially into a plurality of input nodes of a neural network, such as the DyBM.

At S460, an updating section, such as updating section106, may update parameters of a neural network. For example, the updating section may update a plurality of parameters of the neural network hey using the neural network to approximate an action-value function based the action and observation sequence. By updating the parameters of the neural network, the approximation of the action-value function may become more accurate, which may in turn improve the accuracy of the probability function. Inch may result in the selection of actions that more efficiently achieve goals.

At S438, the apparatus may determine whether a stopping condition is met. If the stopping condition is met, such as if a maximum number of iterations have been performed, then the operational flow is discontinued. If the stopping condition is not met, such as if a maximum number of iterations have not yet been performed, then the operational flow proceeds to S439.

At S439, the apparatus proceeds to the next time frame, and the operational flow returns to operation S430to perform the next iteration. In the next iteration, the current time frame becomes a previous time frame, and the subsequent time frame becomes the current time frame.

In other embodiments of an operational flow for neural network reinforcement learning, the updating section may update the par meters of the neural network every other iteration, every third iteration, and so on. The number of iterations before performing an update may change, and/or may depend on the rewards.

In embodiments of operational flow for neural network reinforcement learning in which entire action and observation sequences are obtained at once, such as training sequences, the operational flow may not require a selecting operation or a causing operation, because the actions are already determined as part of the sequence. In further embodiments, such training sequences may be run through the operational flow multiple times and combined with different training sequences to train a neural network.

FIG. 5shows an operational flow for selecting a possible action, according to an embodiment of the present invention. The operational flow ma provide a method of selecting an action according to a probability function. The operations may be performed by an apparatus, such as apparatus100.

At S542, a selecting section, such as selecting section102, may input a possible action into, a probability function. For example, out of all possible actions, a single possible action is input into the probability function. Once the possible action is input into the probability function, the selecting section may make an indication, such as by updating a pointer, so that the same possible action is not input into the probability function twice in a single time frame. In embodiments where the neural network is a DyBM, as shown inFIG. 2, each permutation of binary action input nodes xj[t]may represent a possible action.

At S544, a probability evaluating, section, such as probability evaluating section103, may evaluate the probability function to yield a reward probability, or the probability that a possible action will result in receiving a reward. As operations S542and S544are iteratively performed, the selecting section evaluates each reward probability of a plurality of possible actions according to the probability function based on the action-value function.

At S546, the selecting section may determine whether any unevaluated possible actions remain. If the last possible action has not yet been evaluated, then the operational flow returns to S542. If the last possible action has been evaluated, then the operational flow proceeds to S548.

At S548, the selecting section may determine the highest reward probability that was yielded from the evaluations performed by the probability evaluating section at S544.

At S549, the selecting section may select the possible action that is associated with the highest reward probability determined at S548. In other words, the selected action among the plurality of possible actions yields the largest reward probability from the probability function. Once the possible action has been selected, the operational flow proceeds to cause the selected action, such as S432inFIG. 4, to be performed.

In alternative embodiments of an operational flow for selecting a possible action, each node of the action may be evaluated individually. Because the value of each node is not affected by the values of other nodes, an operation can determine each Action node individually. When all nodes have been determined individually, the action represented by result of each node is the selected action.

FIG. 6shows an operational flow for updating the parameters of a neural network, according to an embodiment of the present invention. The operational flow may provide a method of updating parameters of a neural network. The operations may be performed by an apparatus, such as apparatus100. Before showing the operational flow shown inFIG. 6, underlying theory is explained below.

An approach for reinforcement learning is called SARSA, which refers to a general class of on-policy TD-learning methods for RL. SARSA stand for State-Action-Reward-State-Action, as a representation of its formula. SARSA updates an action-value function Q according to
Q(St,At)←Q(St,At)+η(Rt+1+γQ(St+1,At+1)−Q(St,At)),  Eq. (8)
where Stis the (Markovian and observable) state at time t, Atis the action that we take at time t, Rt+1is the reward that we receive after taking At, γ is the discount factor for future reward, and t is the learning rate. In our case, the Markovian state is not observable, and Strefers to the entire history of observations and actions before t (i.e., St=X[:t−1]).

In some embodiments, the action-value function may be an energy function of the neural network. By Eq. (5), the energy of a DyBM having the structure inFIG. 2can be decomposed into a sum of the energy associated with its individual nodes as follows:
Eθ(x[t]|x[:t−1])=Σj∈A∪ZEθ(xj[t]|x[:t−1]),  Eq. (9)

Here, the energy associated with the nodes is used for the action to approximate the Q-function:
Q(x[:t−1],xA[t])=−Σj∈AEθ(xj[t]|x[:t−1]),  Eq. (10)
where Eθ(xj[t]|x[:t−1]) is given by Eq. (6). Recall that αi,j,k[t−1], βi,j,l[t−1], and γi,l[t−1]in Eq. (6) are updated at each time step using Eqs. (3), (4), and (7).

In other embodiments, the action-value function is a linear function. In many embodiments, such as embodiments where the neural network is a DyBM, the action-value function is a linear energy function of the neural network. The approximate Q-function Eq. (10) is lineal, with respect to the parameters of the DyBM. This is in contrast to ESARSA where the free-energy of a Restricted Boltzmann Machine (RBM) is used to approximate the Q-function. Due to the hidden nodes in an RBM, this is a non-linear function approximation method, which may diverge in theory and practice. However, convergence of SARSA with a linear function approximation is guaranteed under suitable conditions.

When the Q-function is approximated with a linear function of parameters, θ, such that:
Qθ(S,A)=ϕ(S,A)Tθ,  Eq. (11)
SARSA learning rule is given by
θt+1=θt+ηtΔtϕ(St,At),  Eq. (12)
where ηtis a learning rate, and Δtis a TD error:
Δt=Rt+1+γϕ(St+1,At+1)Tθt−ϕ(St,At)Tθt,  Eq. (13)

Each is duplicated in Eq. (16) and Eq. (17) and thus updated twice in each step. This is just for notational convenience, and the two could be merged.

SARSA allows selection of a subsequent action on, the basis of the values of Q for candidate actions. Therefore, actions are selected based on the policy with Boltzmann exploration. Boltzmann exploration is particularly suitable for DyBM because Eq. (10) allows us to sample each bit of an action (i.e., xj[t]j ∈ A) independently of each other according to the following:

Overall the DySARSA learning algorithm proceeds as in Algorithm 1, where we use vector notations: α[t]≡(αi,j,k[t])i,j∈A∪Z,k∈[1,k]; β[t]and γ[t]are defined analogously.

In some embodiments where the neural network is a DyBM as shown inFIG. 2, the action-value function may be evaluated with respect to nodes of the neural network associated with actions of the action and observation sequence. In other embodiments where the neural network is a DyBM as shown inFIG. 2, the action-value function may be evaluated with respect to nodes of the neural network associated with actions and observations of the action and observation sequence.

The operational flow may begin after an inputting section, such as inputting section105, inputs values into a neural network.

At S651, an action-value determining section, such as action-value determining section107, may evaluate an action-value function in consideration of an actual reward to determine an action-value. In other words, the updating the plurality of parameters of the neural network may further include determining a current action-value from an evaluation of the action-value function in consideration of an actual reward. In some embodiments, the previously cached action-value, such as from a time frame t−2, may be deleted.

At S652, a caching section, such as caching, section108, may cache the action-value determined at a previous iteration of S651. In other words, the updating the plurality of parameters of the neural network may further include caching a previous action-value determined from a previous time frame from the action-value function.

At S654, a calculating section, such as calculating section109, may calculate a temporal difference (TD) error, which may be based on the action-value determined at S651and the plurality of parameters of neural network. In other words, the updating the plurality of parameters of the neural network may further include calculating a temporal difference error based on the previous action-value, the current action-value, and the plurality of parameters. The TD-error may be calculated using Eq. (13).

At S656, the updating section may update a plurality of function parameters based on the temporal difference error calculated at S654and a learning rate. In other words, the updating the plurality of parameters of the neural network includes updating a plurality of function parameters based on the temporal difference error and a learning rate. These function parameters may be updated using Eqs. (14-17).

At S658, the caching section may cache the plurality of function parameters updated at S656, which may be used to determine and update eligibility traces of the neural network. The values of x[t+1], α[t], β[t], and γ[t]may be updated. In some embodiments, the previous values of x[t+1], α[t], β[t], and γ[t]may be deleted.

At S659, the updating section may update the eligibility traces and any FIFO queues of the neural network. In other words the updating the plurality of parameters of the neural network includes updating a plurality of eligibility traces and a plurality of first-in-first-out (FIFO) queues. The eligibility traces and FIFO queues may be updated with Eqs. (3, 4, and 7)

FIG. 7shows an exemplary hardware configuration of a computer configured to perform the foregoing operations, according to an embodiment of the present invention. A program that is installed in the computer700can cause the computer700to function as or perform operations associated with apparatuses of the embodiments of the present invention or one or more sections (including modules, components, elements, etc.) thereof, and/or cause the computer700to perform processes of the embodiments of the present invention or steps thereof. Such a program may be executed by the CPU700-12to cause the computer700to perform certain operations associated with some or all of the blocks of flowcharts and block diagrams described herein.

The computer700according to the present embodiment includes a CPU700-12, a RAM700-14, a graphics controller700-16, and a display device700-18, which are mutually connected by a host controller700-10. The computer700also includes input/output units such as a communication interface700-22, a hard disk drive700-24, a DVD-ROM drive700-26and an IC card drive, which are connected to the host controller700-10via an input/output controller700-20. The computer also includes legacy input/output units such as a ROM700-30and a keyboard700-42, which are connected to the input/output controller700-20through an input/output chip700-40.

The CPU700-12operates according to programs stored in the ROM700-30and the RAM700-14, thereby controlling each unit. The graphics controller700-16obtains image data generated by the CPU700-12on a frame buffer or the like provided in the RAM700-14or in itself, and causes the image data to be displayed on the display device700-18.

The communication interface700-22communicates with other electronic devices via a network700-50. The hard disk drive700-24stores programs and data used by the CPU700-12within the computer700. The DVD-ROM drive700-26reads the programs or the data from the DVD-ROM700-01, and provides the hard disk drive700-24with the programs or the data via the RAM700-14. The IC card drive reads programs and data from an IC card, and/or writes programs and data into the IC card.

The ROM700-30stores therein a boot program or the like executed by the computer700at the time of activation, and/or a program depending on the hardware of the computer700. The input/output chip700-40may also connect various input/output units via a parallel port, a serial port, a keyboard port, a mouse port, and the like to the input/output controller700-20.

A program is provided by computer readable media such as the DVD-ROM700-01or the IC card. The program is read from the computer readable media, installed into the hard disk drive700-24, RAM700-14, or ROM700-30, which are also examples of computer readable media, and executed by the CPU700-12. The information processing described in these programs is read into the computer700, resulting in cooperation between a program and the above-mentioned various types of hardware resources. An apparatus or method may be constituted by realizing the operation or processing of information in accordance with the usage of the computer700.

For example, when communication is performed between the computer700and an external device, the CPU700-12may execute a communication program loaded onto the RAM700-14to instruct communication processing to the communication interface700-22, based on the processing described in the communication program. The communication interface700-22, under control of the CPU700-12, reads transmission data stored on a transmission buffering region provided in a recording medium such as the RAM700-14, the hard disk drive700-24, the DVD-ROM700-01, or the IC card, and transmits the read transmission data to network700-50or writes reception data received from network700-50to a reception buffering region or the like provided on the recording medium.

In addition, the CPU700-12may cause all or a necessary portion of a file or a database to be read into the RAM700-14, the file or the database having been stored in an external recording medium such as the hard disk drive700-24, the DVD-ROM drive700-26(DVD-ROM700-01), the IC card, etc., and perform various types of processing on the data on the RAM700-14. The CPU700-12may then write back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium to undergo information processing. The CPU700-12may perform various types of processing on the data read from the RAM700-14, which includes various types of operations, processing of information, condition judging, conditional branch, unconditional branch, search/replace of information, etc., as described throughout this disclosure and designated by an instruction sequence of programs, and writes the result back to the RAM700-14. In addition, the CPU700-12may search for information in a file, a database, etc., in the recording medium. For example, when a plurality of entries, each having an attribute value of a first attribute is associated with an attribute value of a second attribute, are stored in the recording medium, the CPU700-12may search for an entry matching the condition whose attribute value of the first attribute is designated, from among the plurality of entries, and reads the attribute value of the second attribute stored in the entry, thereby obtaining the attribute value of the second attribute associated with the first attribute satisfying the predetermined condition.

The above-explained program or software modules mar be stored in the computer readable media on or near the computer700. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as the computer readable media, hereby providing the program to the computer700via the network.

A neural network in accordance with the present invention can be used for a myriad of applications including, but not limited to, classification, recognition e.g., speech recognition, speaker recognition, pattern recognition, etc.), robotics (e.g., robotic control, robotic decision making), machine control (e.g., controlling a movement of a machine e.g., an assembly line machine), or powering down the machine, or changing the operational state of the machine (e.g., faster, slower, on, off, full-power, low-power, etc.) based on an output of the neural network, and so forth. Thus, such a neural network can be used within systems including, but not limited to, classification systems (e.g., speech recognition systems, speaker recognition systems, pattern recognition systems, etc.), machine control systems (or machine controllers), etc. These and other applications to which the present invention can be applied are readily determined by one of ordinary skill in the art, given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

As made clear from the above, the embodiments of the present invention can be used to realize cloud service utilization.