Patent GB-1011685-A (GB)
Title: Learning machine
Inventor: ANDREAE JOHN HUGH
IPC: G06N20/00
Filed: 1962-07-12
Abstract: 1,011,685. Learning machines. STANDARD TELEPHONES & CABLES Ltd. July 5, 1963 [July 12, 1962], No. 26835/62. Heading G4R. An electrical learning machine comprises output means able to alter the machine's environment or its relationship with it, input means which obtains data words relating to the environment, a store with a compartment for each of the conditions of the output means, each of which can contain a data word, a comparator for comparing a received data word with those stored and for selecting the one least like the received data word, control means responsive to the selection to cause the output means to generate a corresponding output condition and means for altering the stored word to make it less unlike the input data word whereby the machine is caused to explore its environment. The machine may be applied to character readers or to control a process, e.g. a chemical process by receiving data from certain positions in the processing plant and exercising appropriate controls. As described, the machine is a movable device having six photo-cells designed to " see " different patterns in different positions and orientations. It has controls to perform movements, forward, backwards, right or left, one of which is selected at a time. The machine operates at a "search" level, a " reward " level and a " correlation " level. In the " search " level eye 1, Fig. 1, is used and memory 2, a dissimilarity detector 4 and a memory improver 3. In memory 2 there is a row for each of the four actions that can be performed by the motor control 5. Each row contains a pattern which is compared with patterns from the eye by detector 4 which selects the pattern least similar and causes the motor to perform that action. While the machine is carrying out the action, or trying to, the improver 3 causes the selected stored pattern to be made more similar to the eye pattern. As the machine moves the pattern received by the eye changes and, since the improver changes the pattern for greater similarity, the row selected by detector 4 ceases to have the least similar pattern and the detector 4 selects another row so that the action changes and the machine thereby explores its environment. In the " reward " operation a longterm step memory 6 is used together with a threshold detector 7, a sequence store 10, Fig. 2 (connected as shown in Fig. 4), and a random selector 9. Each row of memory 6 can hold an eye pattern and an action of the motor control. This is termed a " step ". The sequence store holds information in each row that the pattern and action in the corresponding row of memory 6 were followed by the pattern and action of another row, or were followed by a reward. Each row of memory 6 has a " weight " which increases with use and decreases with disuse. Memory 6 and store 10 therefore provide a record of paths which the system has previously followed and these are paths which led to a reward because steps are only stored when a reward is received. The detector 7 selects those rows of the step memory 6 having more than a certain degree of match with the eye pattern and one of them is chosen at random by selector 9 having regard to the weight of the row and the number of steps between the row and a reward. Store 10 also holds a record of those steps which led to " pain " warnings. The match threshold of detector 7 varies according to rules of which two are that the threshold rises to 1 : 1 match on receipt of each reward and thereafter decreases slowly exponentially and that the threshold is normally at a maximum and is reduced when none of the patterns in the memory 6 matches the pattern seen by the eye. In any case there is a minimum for the threshold level. The first rule causes the system to follow accurately paths which frequently lead to reward but when there has been no reward for some time it tries new paths until it discovers more rewarding paths, the threshold then becoming high again. The second rule causes the system to follow old, learned paths precisely as long as the patterns seen are the same as those stored but when they are not, the threshold is reduced and allows less complete matches to be made. In the " correlation level " of operation use is made of four correlation matrices one for each of the motor actions. A certain action may tend to change the eye pattern from pattern 1 to pattern 2. A matrix is formed by multiplying the first pattern by each term of the second. If the operation is repeated many times the matrix tends asymptotically to a form which indicates the invariant aspects of the way the corresponding action transforms pattern 1 to pattern 2. These matrices Fig. 3, can therefore be used to make a guess as to what pattern will follow a given action if the initial pattern is known. As they are used in correlation level operation they enable system to choose paths which have a higher than chance probability of leading it to a reward. Generally they enable the system to explore the consequences of future action and to take anticipatory precautions. The eye pattern received is transformed in those by the matrices and ifone transformation corresponds to a reward pattern in store 6 the corresponding action is performed. If not the system reverts to the search operation. The correlation matrices help the device if it loses its way on a remembered path, e.g. by the sequence store anticipating a certain pattern which fails to appear. The eye.-The eye consists of 6 photo-cells with suitable masking so that each looks in a different direction forwards, sideways and backwards. The photo-cells can consist of a photo transistor and a transistor output amplifier operates a slow-to-release relay. The relay is operated only when the light exceeds a predetermined value. The six relays give the eye pattern which is applied to memory 2 and store 6 in Fig. 1. Memory 2.-The short term memory 2, consists of an array of triode triggers arranged in columns and rows. A two-position relay contact applies positive voltage to one or the other tubes of the trigger, the contacts for each column being contacts of the relay of one of the eye photo-cells. The patterns stored in the rows of triggers are thereby continuously compared with the eye pattern. If the relay contact corresponds with the trigger setting positive voltage is applied to the conducting tube and a voltage drop appears across the common cathode resistor. The cathode resistors for each row are connected in parallel to the cathodes of trigger tubes forming the dissimilarity detector 4. The tube receiving the lowest signal fires and prevents the others firing. A relay in series with the conducting tube is energized and causes energization of a corresponding control coil in the motor control 5. The motor control governs the direction of the drive motor and the engagement of steering clutches. The motor also drives a random improver 3, Fig. 1, which consists of a device operating like a roulette table, a ball-bearing moving about on a rotating metal disc constrained by an insulating wall having terminals which are contacted by the ball in a random manner. There are six such terminals connected to the column conductors of matrix 2 to apply random pulses. The arrangement is such that these pulses are allowed to reach and switch the triggers only of that row which has been selected by the dissimilarity detector 4, the other rows having short-circuit paths through contacts of the relays of detector 4. While each action thus selected is being carried out the eye pattern is temporarily entered in the first empty row of store 6 and the associated action is stored in the corresponding row of matrix 8. These patterns are held until, in the next cycle the eye has seen a new pattern and the detector 7 has determined whether it has any similar patterns already in the store 6. Store 6 is similar in general principle to memory 2, triggers being connected with the eye relays so that each row gives a match signal. A " reward " button causes a capacitor to charge, the charge decaying thereafter. This signal is applied to one side of a valve-relay circuit acting as a threshold device the capacitor providing a falling threshold. Initially the level is high and only perfectly matching patterns in the store give an output. Later partial matches are accepted. The action, one of the four possibles, is stored in matrix 8 in the form of a combinational setting of two triggers. A " weight " store has a row for each row of store 6 having inputs to receive positive and negative weight pulses which are stored in capacitors each providing the bias for a triode. The currents in the triodes are determined by the weight stored in the corresponding capacitor. The anode potential of each valve is compared with that of an associated triode also controlled by a storage capacitor. These capacitors are discharged by a random selector, a relay coil between the anodes being operated when the voltage is great enough. Another relay is connected in the anode circuit of each of the first triodes which releases when the weight charge falls to zero. There is also a sequence store 10, Fig. 2, consisting of an array of relays each at the cross point of vertical and horizontal conductors. Each relay when energized establishes a connection between a pair of rows. It enables the control circuit to determine the relationship or sequence between various stored patterns. Operation.-In the " search " operation described above, supposing that no " reward ' has been received, the pattern seen by the eye is temporarily held in the first vacant row (i.e. the first) of the store 6 and the resulting action is stored in the same row of matrix 8. The patterns stored are all removed at the end of the cycle until the reward button is pressed by an operator to indicate that the device is now in a reward area. The pattern and action producing the reward are stored i