METHOD OF TRAINING A NEURON DEVICE

Abstract: A method is described of training a pRAM device to produce a desired output behaviour in response to real-valued inputs. In a simple form thereof, each address input of the device is supplied with a pulse train having a length R equal to a first length R1, and the device computes an output as a function of the pulse outputs produced by the device in response the pulses of the input pulse trains. The behaviour of the device is assessed after carrying out this procedure at least once, and a performance measure produced therefrom. A new value of R = R2 </= R1 is then selected in dependence on the performance measure, and the process repeated. The contents of the memory locations in the pRAM device are updated in dependence on the performance measure just derived. The method may be applied to a network comprising a plurality of modules each having a pRAM or a plurality of pRAMs.

Description: Method of training a neuron device This invention relates to a method of training a neuron device. Most neural network research proceeds via simulations on conventional serial computers; most applications also use software packages. However, simulation may be inappropriate in application areas requiring not just a system with a fast response but ideally also real-time learning. The probabilistic RAM (pRAM) uses inexpensive and readily-available hardware to construct neural systems with the benefits of fast processing, real-time on-chip learning, non-linearity and stochasticity. A description of a pRAM can be found, for example, in our International Patent Application published under No. WO92/00573 and in a paper by T.G. Clarkson, D. Gorse and J.G. Taylor entitled "From wetware to hardware: reverse engineering using probabilistic RAMs", Journal of Intelligent Systems, 2, 11-30 (1992). In brief, a pRAM is a device having a plurality of storage locations at each of which a number representing a probability is stored. The output of the pRAM is a given one of two values, conventionally representing 0 and 1, and the probability of the output being 1 when a given storage location is addressed is equal to the probability represented by the number stored at the location. In hardware terms this can be achieved by causing the output of an addressed storage location to be read to an input of a comparator. A noise generator inputs to the comparator a random number representing noise, and the output of the pRAM is the result of the comparing operation performed by the comparator. An N-input pRAM has 2 N memory locations, indexed by the binary vector u, containing probabilities u : if the pRAM is addressed by the binary vector u then u is the probability that the binary output a will be equal to 1. The variable α u may be represented by a binary spike train (a stream of binary digits) of length R. Accessing location u R times results in a stream of output bits a(r) , r = 1..R; ά u , a stochastic approximation to α u , is then given by R & a (i?> = " ∑ a(r) (1) 1=1 and has a variance inversely proportional to R σ- i a^, R) = tt„(l - α„) ^2 ) R This process is illustrated in Figure l. The pRAM A has 2 N memory contents, but in this case only one of these is accessed, the one indexed by the binary input vector u=(u 1; u 2 ,....,u N ). Each of the r = 1..R accesses to this location u results in an output spike a(r) which is 1 with probability u ; these R spikes are summed at the output and divided by R to produce the stochastic approximation ά u . The variance σ 2 of a real-valued quantity (in this case α u ) is a mathematical expression of the degree of variation about the mean (α u ) expected when this quantity is measured. A large variance means that the results of measurements can be expected to be spread widely about the mean; a small variance means that the values of the measured quantity are expected to be more tightly clustered about the mean. Equation (2) shows that the amount of 'spread' expected when a memory content α u is measured is maximal for a given R when u = 0.5, and is inversely proportional to R. Figures 2a-2c show the distribution of measured values of a memory content with mean value u = 0.5 when spike trains of increasing lengths (a) R=10, (b) R=100, (c) R=1000 are used; it can be clearly seen that as R increases the measured values of α u , the ά u , are closer to the mean value of 0.5. If the input to the pRAM is not a binary vector, but a real vector x e[0,l] N then the output y has potential contributions from all 2 N memory locations N X a = E[ [ u j x j + ( l-"j ) (l -*j ) ] O b) 3-1 where X u is the probability that u is addressed. In a hardware realisation y would be approximated by the random variable y, which would be computed in an R-step sampling process such that R ■ - (R) = ±£ a (r) (4) 1 = 1 In this sampling process, each of the N address lines has supplied thereto a binary spike train of length R in which the probability of any given digit in the train being 1 is equal to the real number which the train represents. The difference between this and (1) is that in this case the output stream contains contributions from a range of memory locations. The process is illustrated in Figure 3. At each step r = 1..R each of the j = 1..N input lines carries a binary signal i j (r) , where Prob(i j (r)=l )=X j . The input vector A(r) addresses the memory location u for which u = i.(r) . A binary output a(r) is then added to the output pulse stream with probability α i(r ,. A range of learning algorithms have been described for the pRAM, including supervised, reinforced and self- organising techniques. The reinforcement method has the advantage of being implementable on-chip, but has previously been restricted to the learning of binary outputs (inputs were allowed to be real-valued) ; although the method could be used very successfully for pattern classification it therefore could not be used for learning arbitrary functions. The general problem we wish to solve is how to train a pRAM net, of suitable architecture with N input lines and M output lines, to approximate an arbitrary continuous function from the domain [0,1] N to the range [0,l] M . Most problems to which neural computing techniques can be applied require a learning capability of this kind (although, as remarked above, pattern classification problems can be framed in such a way as to require only binary outputs - i.e. the range is {0,1} M for some suitable M) . Figures 4a and 4b show two examples of continuous functions into the range [0,1] (M=l) . Figure 4a shows a hypothetical input/output relationship that might be observed in a control problem. In this case the objective would be to train the net to produce the correct control signal y e [0,1] given the input signal x e [0,1] (N=l) . Figure 4b is a hypothetical example of a time series: the value of the variable x(t) e [0,1] (for example the price of some commodity) at time t is assumed to depend on its previous values x(t-l) , x(t-2)... The objective here would be to train the net to predict the value x(t) given a set of T previous values (N=T) . The present invention provides a new reinforcement training algorithm, suitable for the learning of continuous functions, and imple entable in hardware using pRAM technology. As pointed out by Gullapalli (V. Gallapalli, "A stochastic reinforcement learning algorithm for learning real-valued functions", Neural Networks, 3, 671-692 (1990)) reinforcement learning of real-valued quantities requires the ability to manipulate both the mean and the variance; conventional A RP learning, which adjusts only the mean, results in inevitable saturation (variables are driven to their limiting binary values) . Gullapalli used a normal distribution ψ(μ,σ) and provided mathematical algorithms for the adjustment of mean μ and variance σ ; we make use of the inherent noise associated with the use of finite-length spike trains, manipulating R to provide the necessary changes in variance. The pRAM device could consist of a single pRAM. Normally, however, it would comprise a network of a plurality of pRAMs. In that case, the output of the device could be a vector [0,1] M . The network could further be divided into a set of > 1 modules, each such subnetwork learning to perform a distinct task which contributes to a desired output behaviour. At any given time each module could independently be active (computing an output in response to an input) or not. Each active module may independently be in learning phase (the contents of its memory locations being available for update) or not. According to the present invention there is provided a method of training a pRAM device comprising one or more modules to produce a desired output behaviour in response to real-valued inputs, comprising: (a) selecting the module, or one or more of the modules, of the network to be active; (b) selecting each active module to be in either learning or assessment phase; (c) supplying to the or each address input of each active module i of the device a pulse train in which the probability of any given pulse being a 1 is egual to the value of the real-valued number to which the pulse train corresponds, the pulse train having a length R=R £ for active module i; (d) computing an output for each active module as a function of the pulse outputs produced by the device in response to the pulses of the said input pulse train(s); (e) zero or more repetitions of (a) - (d) ; (f) assessing the behaviour of the device in the light of some performance criterion, and producing a performance measure for each active module therefrom; (g) for each active module i in assessment phase, selecting a value for R=R i (where preferably R i / < R i ) in dependence on the performance measure for module i produced in (f); (h) for each active module i in learning phase, computing values for the reinforcement signals r i f p L in dependence on the performance measure for module i produced in (f) ; and (i) for each active module i in learning phase, updating the contents of the memory locations in this module in dependence on the reinforcement signals for module i computed in (h) , and the actions taken in (d) and/or any repetition of (d) ; (j) zero or more repetitions of (a) - (i), until a desired performance level is reached. It should be noted that alternatively, steps (c) and (d) could be replaced by an appropriate calculation delivering a mean output value as a function of the probabilities associated with each of the input lines. As an example of network training according to the invention, there will now be described the problem of reaching a network to balance a pole on top of a moving cart, whilst keeping the cart position (as well as the pole angle) within prescribed bounds. The network has two modules, a movement generator and state value assessor. Both may receive real-valued input signals which represent the positions and velocities of the pole and the cart. These signals are converted into pulse trains of lengths ^ (for the movement generator) R., (for the state value assessor) . The output of the movement generator is thresholded to result in a binary output which is interpreted as providing a force to the cart of ±F ("bang-bang" control). The output of the state value assessor is not thresholded; this continuous quantity represents the value or "safety" (distance from a failure position) ascribed to the currently input state by this network. The training process consists of multiple trials each usually ending with pole, cart or both in positions outside their prescribed range. Training is terminated when a trial exceeds a certain length of simulated balancing time, typically 1 - 2 hours (or equivalently a maximum number t maχ of discrete simulation time steps, typically of length 0.02s) or when the number of trials exceeds some fixed number. In one approach to this task, trials are divided into two alternating types: one in which the balancing strategy embodied by the network's current set of weight parameters is assessed, and ones in which this strategy is updated by changing the values of the memory contents in the movement generator during the trial. The state value assessor is only needed during this latter type of "learning" trial; its memory contents are also updated during the trial. The training process for the cart-pole system can be described by the following algorithm: Table 1 I . Assessment trial T e -T O . l T roiit l , time step t £ / 0 , 1 , . . . , t maχ l (a) select movement generator to be active; (b) select movement generator to be in assessment phase; (c) present input representing the current dynamical state x(t) (positions and velocities of the cart and pole at time step t) to the movement generator, using spike trains of length R^ = R^; (d) compute an output at time step t for the movement generator (which will result in a change of dynamical state to x(t+l) at time step t+1) ; (e) repeat (a) - (d) until new state x(t+l) is out of range or t+l=t max ; (f) equate performance measure for the movement generator to the number of repetitions of (a) - (d) executed (time steps in range) ; (g) select a value R^ ) < ^^ for the movement generator, in dependence on the number of executions of (a) - (d) . 11 • Learning trial T e-TO.l τ ma_ ' time step t £; °' 1 r -- τ t rπaχ I (1) Parameter update at time step t for movement generator (a) select movement generator and state value assessor to be active; (b) select movement generator to be in learning phase, state value assessor to be in assessment phase; (c) present input representing the current dynamical state x(t) to the movement generator, using spike trains of length ^ (T) , and to the state value assessor, using spike trains of length R v = R j ,,-^; (d) compute an output at time step t for the movement generator (which will result in a change of dynamical state to x (t+1) at the next time step) and for the state value assessor (which will give the 'value' of state x (t) , v t (x (t)); (e) repeat (a) - (d) in the new state x (t+l) but with only the state value assessor active (and in assessment phase, R^ = R^^) - this will result in a 'value' being assigned to state x (t+l) at time t, v t (x (t+l)); (f) calculate a performance measure (reflecting the change in state value produced by the action of (d) in dependence on v t (x (t) , v t (x (t+1) ) ; (h) compute values for the reinforcement signals r m (t) , p m (t) to the movement generator, as functions of the performance measure of (f) ; (i) update the movement generator memory contents in dependence on r m (t) , p m (t) . 2. Parameter update at time step t, sub-step s 6/0,1 s ma x^ (denoted by time step (t.s. (2.1) (a) select state value assessor to be active; (b) select state value assessor to be in assessment phase; (c) present state x (t) to the state value assessor, using spike trains of lengths R^_ = R m&x j (d) compute an output at time step (t,s) for the state value assessor, giving a value v t s (x (t) ) ; (f) calculate a performance measure, known as the temporal difference (TD) error in dependence on v t s (x (t) , v t (x (t+1)); (g) select a value y . (t,s) < R.,^ for the state value assessor, in dependence on the TD error of (f) ; (2.2) (a) select state value assessor to be active; (b) select state value assessor to be in learning phase; (c) present state x (t) to the state value assessor, using spike trains of length ^ (t,s); (d) compute an output at time step (t,s) for the state value assessor, giving a value v t>s (x (t) ) ; (f) calculate TD error in dependence on v t s (x(t)), v t (x (t+1); (h) compute values for the reinforcement signals r v (t,s) , p v (t,s) to the state value assessor, as functions of the TD error of (f) ; (i) update the state value assessor memory contents in dependence on r v (t,s), p v (t,s). Repeat (II.2.1) + (II.2.2) until either the TD error calculated in (II.2.1 (f)) < a desired minimum, or s = s max . Repeat (II.1) + (II.2) until either the new state computed in (11.1(d)) is out of range, or t+1 in (11.1(d)) = t maχ . Repeat (I) + (II) until t+1 in (1(d)) or (11.1(d)) = t maχ or until T = T maχ . Although the training process set out above in Table 1 may appear complex, it may be seen to consist of no more than repetitions of steps (a) - (i) , using different combinations of active modules in different (assessment or learning) phases, with repetition until a desired performance criterion is reached. The invention will now be described, referring, inter alia to the accompanying drawings, in which, as already indicated: Figure 1 is a flow diagram illustrating a known process involved in generating a value ά u from a pRAM; Figure 2 is a graph showing the amount of spread when a pRAM memory content α u is measured; Figures 2a - 2c show the distribution of measured values of a memory content when spike trains of various lengths are used; Figure 3 is a diagram corresponding to Figure 1, but for the case where the input to the pRAM is a real vector; Figure 4 are graphs showing two examples of continuous functions which a pRAM net might be required to map from the domain [0,1] N to the range [0,1] M ; and, as described further below; Figure 5 is a block diagram illustrating the use of a parameter β to transfer a pRAM output; and Figure 6 is a set of graphs showing how, for a particular function involving β, the transformation varies with the value of β . The simplest application of the method of the invention requires a single N-input pRAM to learn the function implicitly specified by a set of P training patterns x p e [0,l] N and their desired outputs d p e [0,1]. A broad overview of the learning algorithm for this application is set out below in Table 2. Table 2 (1) (a) select function approximator (the only module in the network) to be active; (b) select function approximator to be in assessment phase; (c) present input pattern x , chosen from the set p = 1..P, using maximum length spike trains R = m^; (d) calculate y p ≡ y p (R^a*) as a function of x p ; (f) calculate pattern error E p (d p , y p ) ; (g) select a value for R as a function of E (2) (a) select function approximator to be active; (b) select function approximator to be in learning phase; (c) present the same input pattern x as was chosen in (1(c)) , this time using spike trains with the length R < ^ma selected in (1(g)); (d) calculate y ≡ y (R) as a function of x ; Λ (f) calculate pattern error E p (d , y D ) ; (h) compute values for the reinforcement signals r_, p in dependence on E ; (i) update the memory contents of the function approximator in dependence on r_, p p . Repeat (1) + (2) until each pattern x in the training set has been presented. Repeat the presentation of the entire training set (P iterations of (1) + (2)) until performance is satisfactory. A number of points are to be noted about the algorithm set out in Table 2. Firstly in repeating the steps (l) + (2) to cover the training set each pattern may be selected in turn or the patterns may be selected randomly or according to some other criterion. Secondly, training continues until the error over the training set is within the desired tolerance. For this purpose, the error over the training set will not normally be taken to be the last value of E p . A more appropriate measure of the error over the training set is the average of a given number of the preceding values of E p , for which purpose it is necessary to keep a record of the running average over whatever the selected given number of values may be. It should be noted, however, that other measures of the error may be used instead, for example some function of the preceding values of E p other than their average. R should normally be adjusted so that a smaller, error E results in a longer estimation spike train (and thus less sampling noise for this pattern at this stage of training) . R may be calculated, for example, by taking the integer part of R (E p ) = -£ - + ^ in (5) E k * (clipped if necessary so that R < R^x ) . R ι -, ax is a constant related to the hardware implementation of the algorithm. It should be chosen so that (R πιax ) is a close approximation to the theoretically predicted (using mean-valued quantities) output given by equations (3a) , (3b) . c R ,k R and R- nin are constants, for which, by way of example, values of 0.25, 1.5 and 1 have been found to be operable. The values of R(E p ) could conveniently be stored as a look-up table rather than calculated afresh each time they are needed. The function E(d,y) may be taken, for example, to be the absolute value of the difference between the desired output d and the actual output y E(d, y) = |d - y| (6) The update to memory location u is given by Δ^u = P [ά jj - ά r p + A(l - &U - α P p l^u ( 7 > where ά u , X u are stochastic approximations to the memory content u and the access probability X u (equation (3b)) . The use of a spike train of restricted length (R < ^x) in effect creates the opportunity for a stochastic 'experiment' during which the pRAM can try the effect of using the new value ά u in place of α u . If performance improves, the learning rule (7) moves α u closer to this value; if it declines, the rule moves α u away from it. The reward r p and penalty p should be, respectively, decreasing and increasing functions of E p , the error measured after this stochastic procedure. If R(E D ) is given by f(5 R) . , . +thh*es PerTTr-novr ffnunnrctti i nonnss V Σ J P', E r given by (6) , and - - ~ ~ E P (8a) k P P P ~ Ep (8b) (k„, k D are constants > 0) it can be shown that the algorithm of Table 2 is guaranteed to converge to a stable solution provided that (i) k R + k r > 2 (ii) k R + k p > 3 (iii) penalty parameter λ > 0 (9) The learning rule can also be written in the form Δ« a = -£[ (A - - ) ?, - λ (A u - n- u ) S u ) p ] (10) where (11) and A.. α,. = - (12) T. S u is the number of times during the R-step sampling process location u is accessed R , S ~ £ -u.± ) (13) r=ι (where <? Ufi(r) = 0 unless u = i(r)) and A y is the number of l's emitted by u during this sampling process R ~ a ( l (ι ) r Σ r) δ u. (14) =l Since there are S u opportunities for u to emit a 1, this results in the expression (12) for ά u , the stochastic approximation to the memory contest α u (if S u = * = 0, ά u is undefined according to (12) and therefore might give rise to an error condition in (7) ; however, in the update rule (10) S u = A u = 0 just results in a zero update, ΔC_ U = 0) . It is to be noted here and throughout that the learning parameter p may be time (training step) dependent; this dependence will not be explicitly shown. Using equation (5) to select the value of R < R^x . R will always increase for decreasing values of E D and vice versa. This is what one would normally want, whether or not the particular formula of equation (5) was employed. It is conceivable, however, that under some circumstances one might wish to decrease R when E_ decreased, for example if one had reason to believe that the values of α u were converging to false minima. The function learning algorithm outlined above in Table 2 assumes that (a) It is possible to 're-present' the same pattern p (two output calculations, one with R - R- n ax' one with R < R jj ^x are used) . (b) A desired value d p for the pRAM output in the context of pattern p can be specified. There will be some learning situations where one or both of these conditions fails to hold. In real-time learning situations the network may not have control over the presentation of input patterns, and may be required to respond continuously to a changing environment whilst learning. In certain applications the output of the net may drive a further complex system whose behaviour is then assessed by the environment, resulting in global reinforcement signals - but not the delivery of specified values for the network outputs y D , which are not directly accessible. These situations may be handled by a more general algorithm which appropriately interleaves 'assessment phase' and 'learning phase' procedures. The pole balancing system described above is an example of this. The learning rule (7) (or equivalently (10)) may still be used in such situations, with R as calculated in (5) replaced by a spike train length reflecting the performance measure of step (f) of the invention whilst in assessment phase and r and p as calculated in (8a) and (8b) replaced by reward and penalty signals that are calculated from the performance measure of step (f) of the invention whilst in learning phase. Under certain circumstances, in particular when multi- pRAM networks with hidden units are used (a situation to be described in more detail below) , it may be desirable to replace the mean-valued output function (3a) by the more general expression and the stochastic output function (4) by where the function s would ideally be an arbitrary non¬ linear function of its argument. However, it is desirable to achieve this extension to pRAM functionality without disrupting the input-output bit stream, and in addition to allow the degree of non-linearity produced by this output transformation to be learned using similar techniques to those described in the previous section. A method will now be described which transforms a pRAM output with mean value y into a new output with mean value f {β , γ) , the amount of non-linearity being controlled by the learnable parameter β . β interpolates between the identity function y (when β = 0) and a non-linear function s(y) (when β = 1) . An output computation (for evaluation (R = R^x) or learning (R < R^x) ) consists of the following procedure, illustrated in Figure 5. The output stream of the pRAM, with mean output y, is tapped and n+1 output spikes diverted to a buffer. Each location contains a value a i e {0,1}, i=0..n, generated with probability y. Location 0 in this buffer forms one of the two inputs to the interpolation pRAM (the other is from the recurrent pRAM to be described below); locations l..n in this buffer are input to the 1st to nth address lines of a recurrent n+l-pRAM, with non-learning memory contents α u which might for example be given by « a = d-"o ) ττ -j + u o ( !- rr ( 1-Uj- )) (17) (the Oth input to this pRAM is a self-feedback line) . The mean output of the recurrent pRAM is a non-linear function of y; for the memory contents (17) this would be the sigmoid-like function for which s(0)=0, s(0.5)=0.5, s(l)=l and the derivative at y = 0.5 is equal to n (so that the function becomes progressively steeper at y = 0.5 as n increases, achieving a greater degree of 'squashing' of the input toward the limiting values of 0,1). Interpolation is performed between the identity function and s(y) using an interpolation pRAM with memory contents (0, β, 1-3, 1) , giving a resultant output with mean value f(/3,y) = β s(y) + (1-3) y (19) An example of interpolation with s (y) = — y 4 yA ( l-y) i is shown in Figure 6. The parameter β may be learned in a similar way to any other pRAM memory content, using the techniques of the previous section (the values of 0 and 1 respectively contained in locations u = 00, u = 11 of the interpolation pRAM are not adapted. The only extra feature connected with the learning of β relates to the question of averaging over the updates to the locations containing β (location u = 01) and 1-/3 (location u = 10) . Satisfactory performance may still be achieved if this) is not done, but it is more desirable to retain the condition that the sum of the memory contents in locations 01 and 10 is equal to 1: β 01 + β 1 Q = 1. To achieve this, updates to β Ql and /3 10 are calculated according to (7) (or equivalently (10)) but where in this case (13) and (14) are replaced by *• ~ S u - f ∑ (δcι ^<" ' δ :o, i( ) ) 1. = 01,10 c20 ) r=l (access to either location interpreted as "half access" to both) ι R Λ - = 1 a(r) δ -"<- (r) u =01 ' 10 < 21) (access to u = 01 (10) and emission of a(r) = l interpreted as "half an access to u = 01 (10) and emission of 1" and "half an access to neighbouring location u = 10 (01) and emission of 0") . If the interpolation pRAM memory contents are originally set so that β Q1 + ø 10 = 1 the updates to these memory contents will retain this condition throughout training. As far as the assessment of a pRAM device's output is concerned (for example for the purposes of establishing a value for R, or stochastic reward and penalty signals r, p) we simply note that it is the interpolated outputs f(2.y(R)-) (with mean value (19)) that are supplied to the external world or further pRAM devices, not the y(R) (with mean value 3a)). So for example, y 0 ( max ) . y p (R) n Table 2 should be replaced by f(/3, y p (R^) ) , f (β, y p (R) ) • Finally, it must be noted that under some circumstances it may be important to allow sufficient time for transient effects to die down before any stochastic quantities are evaluated; the network should ideally be allowed to stabilise before any measurements are taken. The techniques described above can be extended to multi-pRAM nets and multi-pRAM modular components of nets. The most general device of this type might have N real- valued (but spike-train encoded) inputs, M pRAMs producing spike-encoded real-valued outputs, and a further H pRAMs acting as 'hidden units'. 'Hidden units' are units whose outputs are passed on exclusively to other pRAMs in the device rather than to the external environment. They are important in the recording of input data (learning to act as 'feature detectors') so that a device can learn to perform the most general types of mapping. In the most general case the connectivity of the device would not be restricted in any way (any pRAM might pass on its input to any other, including itself) and output transforms might be applied to any of the pRAMs in the device. In practice the architectures used in neural computing applications tend to be more regular than the very general structure suggested above. The most commonly used architecture arranges the neural processors in a set of layers 1..L (layer L being the output layer, containing M units which pass on their outputs to the external environment) . The units within a given layer 2<L normally pass on their outputs only to units in layers 1 ' >1 (very frequently just to layer 1+1) ; feedback within layers (1' = 1) or to lower-indexed layers (1'<J) is not used. This kind of structure is referred to as a feedforward architecture. In addition the use of non-linear output transforms is usually restricted to the hidden layer (KL) units; in fact it may be shown that a device with a single hidden layer (with output transforms) can act as a universal approximator, so the choice of a feedforward architecture of this kind is not unduly restrictive. The extension of the learning algorithms in the multi- pRAM case is very straightforward, and devices of arbitrary connectivity may be used. It should be noted that (i) In the function learning example of Table 2 the network output y and desired output d are now M- dimensional vectors. The pattern error E 0 will therefore now be a function of y and d. (ii) If feedback among the pRAM units (in addition to the possible use of output transform circuits) is allowed it may be necessary to allow transients to die down before evaluating the outputs by averaging pulses over a set of R consecutive time steps. (iii) In the case that the output evaluation is to be followed by parameter update, values for α u , X u (if equation (7) is used) or S u (if equation (10) is used) are computed for each location u in each pRAM. (iv) The values of the training parameters p and λ may be different for each pRAM, or even each storage location within each pRAM, and if p is time- dependent the functional form of this dependence may also differ for each pRAM, or even for each storage location. However, it is usual in a feedforward architecture for the values of p and λ, and time-dependence of p, to be the same for all pRAMs in a given layer, with differences between one layer and another. Under some circumstances it may be necessary to use traces in pRAM devices, and the learning techniques described above must then be adapted accordingly. Two types of traces must be considered. Type 1 - access/activity traces These traces are able to take into account the build up of momentum in a system interacting with an environment with non-trivial dynamical behaviour. In such a situation it may not be reasonable to associate the performance criteria of step (f) of the definition of the invention (later used to calculate r \ - and p i for the learning modules i in step (h) ) , solely with those actions just taken by the learning modules in the preceding step (d) . An example of this is provided by the movement generator of the cart-pole system when it is in learning phase (step (II.1) of Table 1) . Suppose that the pole is falling to the right, and that the action taken in (11.1(d)) is a push +F on the cart to the right. This is a correct action in these circumstances. However, if the falling pole has built up significant momentum, this correct action will still be associated with a decrease in state value, and a penalty according to (11.1(h)). What is needed here is a means to associate the reinforcement calculated in (h) with a history of recent actions taken by the learning system, not just the most recent (step (d) ) . In the cart-pole situation used as an example here, the improved strategy would result in not just the most recent, rightward push being (incorrectly) penalised, but in those recent leftward pushes which caused the current rightward fall of the pole being (correctly) penalised also. It is proposed to extend steps (d) and (i) in the definition of the invention to read: (d) computing an output for each active module as a function of the pulse outputs produced by the device in response to the pulses of the said input pulse train(s) , and updating access/activity traces if currently used; (i) for each active module i in learning phase, updating the contents of the memory locations in this module in dependence on the reinforcement signals for module i computed in (h) , and the actions taken in (d) or the access/activity traces updated in (d) if currently used. There are two components to the access/activity traces. The first, S u , keeps a record of the frequency with which location u has been accessed. It is updated according to s u - δs M + 1-£L s u (22) where S u is the number of times during the preceding R-step sampling process (contained in steps (c) to (d) in the definition of the invention) location u has been accessed (as defined in equation (13) or (20)). Preferably the initial values S u (0) of the S u should be chosen so that Σ 5, (0 ) The second component A^, keeps a record of the frequency with which location u has been accessed and has emitted a 1. It is updated according to - * X u + -^ir 1 . < 23) where A u is the number of times during the preceding R-step sampling process (contained in steps (c) to (d) in the definition of the invention) location u has been accessed and has emitted a 1 (as defined in equation (14) or 21)). Preferably the initial values * (0) of the A y should be chosen so that 0 < Σ A u (0) < 1. The parameter δ in (22) and (23) is a trace decay parameter, 0 < δ < 1. As an example, the value δ = 0.9 is found operable for the movement generator module of the cart-pole system; learning in this system is found very poor when δ = 0 (corresponding to parameter updates based on the most recent actions alone) . The memory content update rule used in step (i) of the definition of the invention is, when access/activity traces are used, given by A a u = p[tf a - J M ) x - λ (A u - (1-a S u ) p (24) Equation 24 is very similar to the no-trace formula (10) , except that A u and S u are replaced by their traces A y and S u , and the new rule does not require p to be divided by R since > and S u are already scaled into [0,1] by their update equations (22) and (23) (notice that when 5 = 0 equation (24) reduces to the previous memory content update rule (10)). Type 2 - access history traces These traces, X u , also keep a record of the frequency with which a location u is accessed, with an update rule similar to (22) X u " λ _ + - L -r I S U ( 25 ) (with decay parameter 0 < A < 1) . Preferably the initial values of the X u should be chosen so that However, unlike the S u of (22) , which when 5 - 0 (along with their partner A u ) only affect the memory content update rule, the X u play a role in the spike-generation process (steps (c) to (d) ) itself: when λ - 0 spike output depends not only on the currently presented input but*also on other inputs seen previously. The mean-valued output expression (3a) is in the A - 0 case replaced by y - J -u (26) (with a corresponding change to the interpolated version (19), y in this latter equation being replaced by (26)) where X u is in the mean updated by X u - . - (l-λ) X u (27) (X u ~ S u /R rnaχ being the probability of access of location u) . The question remains as to how to get an R-step, spike-based approximation to (26) , given a current input x and memory location access traces X u . There are two methods for doing this. Both of them begin by registering the presence of the current input x by presenting this to the pRAM device and either using the mean-valued update (27) or using a large R=R rnaχ updating the X u according to (25) (which for large R will approximate the mean-valued update (27), with X u (x) given by (3b)). The methods then proceed differently. Method 1 (for an N-inout pRAM For r = 1 to R (1.1) Present that input pattern consisting of 0.5 on each of the N input lines. (1.2) Generate spikes from this input pattern (every address u will be addressed with. equal probability 1/2 N ) and allow each accessed location to emit a 1 with probability X u until some location v emits a 1, for a maximum of T opportunities. (1.3) Generate an output spike a(r) with probability α v of being 1 if any y was accessed, otherwise (this will be unlikely if T is large) generate an output spike randomly, preferably with probability 0.5. Method 2 (for an N-input pRAM) For r = 1 to R (2.1) Present that input pattern consisting of 0.5 on each input line. (2.2) For t = l to T = S x 2 N (preferably S is an integer > 1) (i) Generate spikes from this input pattern (every address u will be addressed with equal probability 1/2 N ) and allow that location v accessed by the input spike pattern to emit a spike s 2 (r,t) with probability X v . (ii) Allow location v to emit a spike s 2 (r,t) with probability α v . (iii)Increment the output stream by s 1 (r,t) x s 2 (r,t). (2.3) Generate an output spike a(r) with accumulated probability 1 T=Sx2 K , . — ^ S χ (X , t) X S 2 (X , t) t=l The use of Type 2 traces, which allow an output behaviour to depend on a previous input history, is independent of the use of Type 1 traces, which record a history of input/output behaviour for use in a modified memory content update rule (equation (24)). The memory content update rule for nets with Type 2 traces is formally identical to (10) or (24) (depending on whether Type 1 traces are used also) . However, the A y and S u used directly in (10) , or contributing to the traces A y , S u in (24) may be accumulated differently if using Type 2 traces. For Method 1, accumulation will be according to the conventional (13), (14) , (20) , (21) where an address u will only make an increment to S u or A y if X u previously emitted a 1 in (1.2) , allowing accessing of α u in (1.3). For Method 2 (13) is replaced by S» - ∑ ∑ 6» -«*-« r=l t=l and (14) by ι R T A„ = — ∑ ∑ S 2 (r ' U) -U.Xa.-r ) r=l t=l (20) is replaced by 1 R T r=l t=l and (2l) is replaced by 1 R T ~ 2T ∑∑ s 2 (x, t) δ ϋ, ( , 0 r=l t=l Type 2 traces may be used in any pRAM output calculation. Therefore, it is proposed to further extend step (d) in the definition of the invention to read (d) computing an output for each active module as a function of the pulse outputs produced by the device in response to the pulses of the said input pulse trains and to updated access history traces if currently used, and updating access/activity traces if currently used; In a modular net some modules may use Type 2 traces when active, some not. Those that do use Type 2 traces may not use them at every opportunity. As an example, the state value assessor benefits from the use of Type 2 traces (A=0.8 is found to be operable) within procedure (II.2) (its own memory content update procedure) but not within procedure (II.l) (the movement generator memory content update procedure) . In the pole balancing example the state value assessor is not found to benefit from the use of Type 1 traces at any point, and conversely the movement generator, which uses Type 1 traces in (II.1), is not found to benefit from the use of Type 2 traces. To summarise, the algorithm described herein uses the noise associated with the representation of real-valued quantities by finite spike trains to drive a reinforcement algorithm which can enable a probabilistic RAM (pRAM) device to learn to approximate any function from [0,1] N to [0,1] M . Because the algorithm can be directly implemented in hardware it is potentially extremely fast, and could therefore find many applications in areas requiring a real¬ time learning capability.

Claims: CLAIMS : 1. A method of training a PRAM device to produce a desired output behaviour in response to real-valued inputs, comprising: (a) supplying to the or each address input of the device a pulse train in which the probability of any given pulse being a pulse representing a 1 is equal to the value of the real-valued number to which the pulse train corresponds, the pulse train having a length R equal to a first length R,; (b) computing an output as a function of the pulse outputs produced by the device in response the pulses of the said input pulse train(s); (c) assessing the behaviour of the device after one or more applications of steps (a) to (b) in the light of some performance criterion, and producing a performance measure therefrom; (d) selecting a value for R = R 2 , in dependence on the performance measure; (e) repeating steps (a) to (b) one or more times with R = R 2 ; and (f) repeating step (c) with R = R 2 , using the same performance measure or a different performance measure; (g) updating the contents of the memory locations in the pRAM device in dependence on the performance measure of step (f) ; and (h) repeating steps (a) to (g) zero or more times. 2. A method according to claim 1, wherein R 2 <R 2 . 3. A method of training a pRAM device to produce a desired output behaviour in response to real-valued inputs, comprising: (a) selecting the module one or more modules, of the network to be active; (b) selecting each active module to be in either learning or assessment phase; (c) supplying to the or each address input of each active module i of the device a pulse train in which the probability of any given pulse being a 1 is equal to the value of the real-valued number to which the pulse train corresponds, the pulse train having a length R=R i for active module i; (d) computing an output for each active module as a function of the pulse outputs produced by the device in response to the pulses of the said input pulse train(s); (e) zero or more repetitions of (a) - (d) ; (f) assessing the behaviour of the device in the light of some performance criterion, and producing a performance measure for each active module therefrom; (g) for each active module i in assessment phase, selecting a value for R=R i 'in dependence on the performance measure for module i produced in (f) ; (h) for each active module i in learning phase, computing values for the reinforcement signals r i , p i in dependence on the performance measure for module i produced in (f) ; and (i) for each active module i in learning phase, updating the contents of the memory locations in this module in dependence on the reinforcement signals for module i computed in (h) , and the actions taken in (d) ; (j) zero or more repetitions of (a) - (i) , until a desired performance level is reached. 4. A method according to claim 3, wherein R i '<R i . 5. A method according to claim 3 or 4, wherein traces are stored representing the frequency with which individual pRAM memory locations have been accessed and/or the frequency with which individual pRAM memory locations have been accessed and have produced a 1 in response thereto, and wherein the said trace or traces of at least one module are updated in step (d) on at least one occasion, and wherein the updating in step (i) is carried out on at least one occasion in dependence on the reinforcement signals for module i computed in step (h) , and the actions taken in step (d) or the trace or traces updated in step (d) . 6. A modification of the method according to any one of claims 3 to 5, wherein steps (c) and (d) are replaced by a calculation delivering a mean output value as a function of the probabilities associated with each of the inputs. 7. A method according to any preceding claim, wherein the output of the pRAM, or at least one of the pRAMs, is supplied to the external environment, or to the input of at least one other pRAM, via a transformation device which generates at its output a function of the said pRAM output. 8. A method according to claim 7, wherein the said function is a squashing function. 9. A method according to claim 7 or 8, wherein the said transformation device comprises at least one pRAM. 10. A method according to 7, 8 or 9, wherein the said transformation device has at least one memory location containing a learnable parameter.

Inventors: GORSE DENISE, TAYLOR JOHN GERALD, GORSE, DENISE, TAYLOR, JOHN, GERALD
Applicants: GORSE DENISE, KING S COLLEGE LONDON, TAYLOR JOHN GERALD, UNIV LONDON, GORSE, DENISE, KING'S COLLEGE LONDON, TAYLOR, JOHN, GERALD, UNIVERSITY COLLEGE LONDON
Application year: 1994
Main classification: G06F15
IPC codes: G06F15, G06N3