Patent ID: 12242947

DETAILED DESCRIPTION

FIG.1shows a schematic of a neural network system1during inference according to an implementation. The neural network system1may be implemented as one or more computer programs on one or more computers in one or more locations.

The neural network system1comprises an embedding network2, a memory3and an output network4. The embedding network2and the output network4form a parametric component5of the system and the memory3forms a non-parametric component6of the system. The neural network system1is used during inference to process input data7to generate output data10. In particular, input data7is processed by the embedding network2to generate a query data item8and the query data item8is used to query the memory3. A returned query9from the memory3is used to modify the output network4. In particular, such modification may relate to modifying parameters of the output network4. The output data10may then be generated based on the modifications made to the output network4. For example, the query data item8, output by the embedding network2, may be processed by the output network4having the modified parameters so as to output the output data10.

FIG.2shows a specific implementation of the neural network system1during training andFIG.4shows the specific implementation of the neural network system1during inference. In the implementation shown, the embedding network2(which notationally may be written as fγ), and the output network4(which notationally may be written as gθ), are standard parametric (feed forward or recurrent) neural networks with parameters, γ and θ, respectively. That is, γ and θ are the weights of the neural networks fγand gθ. The memory3is a dynamically-sized memory module that stores key, hi, and value, vi, pairs, M={(hi; vi)}, where each pair relates to a specific input xi. Thus the memory may store episodic memories, i.e. data relating to specific training examples which have previously been presented to the system.

The memory3may store the key and value pairs in an append only fashion, and thus the memory size may increase (up to a limit) as new data items are stored. The keys {hi} are given by the embedding network2and the values {vi} correspond to a desired output yi. For classification, yimay be the true class label, whereas for regression, yimay be the true regression target. The size of the memory may depend on the number of training examples, for example it may be capable of storing 1% or more of the training examples.

The neural network system1is used differently depending on whether the system1is being trained or used in inference. During training of the neural network system1, for a given input x, the conditional likelihood of an output, y, given an input, x, may be parameterized with a deep neural network given by the composition of the embedding network2, fγ, and the output network4, gθ. Thus the conditional likelihood may be given by:
ptrain(y|x,γ,θ)=gθ(fγ(x))

In the case of classification, the last layer of the output network4is configured to be a softmax layer. The parameters γ and θ may be estimated by maximum likelihood estimation, using for example back propagation.

During training of the neural network system1, data derived from observed data, x, which is used to train the system1is also stored to the memory3. For example, upon observing a j-th example, xj, the memory3is updated by appending11the corresponding pair (hi, vi) to the memory3, where:
hj←fγ(xj)
vj←yj

The memory3may have a fixed size and act as a circular buffer, such that when it is full, the oldest data is overwritten first. Subsequent retrieval from the memory3, described later, may comprise performing a K-nearest neighbor search on the keys {hi} with e.g. Euclidean distance to obtain the K most similar keys and associated values.

With reference toFIG.3, there is now described a method for training the neural network system1.At step S1, a mini-batch of one or more training examples B={(xb, yb)}bis sampled from training data.At step S2, an embedded mini-batch B′={(fγ(xb), yb): xb, yb∈B} is determined using the embedding network2, fγ.At step S3, the parameters γ and θ are updated by maximizing the likelihood of γ and θ with respect to the mini-batch of examples B.At step S4, the embedded mini-batch examples B′ are added to the memory3, M:M←M∪B′.

As can be seen fromFIG.2, in implementations the memory3stores data from the examples during training but at training time it is not used in generating an output from the system, nor is the stored data used for training. Thus in implementations it is not necessary for the memory to be differentiable.

FIG.4shows an example of the neural network system1during use in inference.

During use in inference the embedding network2receives an input x and the embedding network2processes x to generate a key embedding h for the input x. The key embedding h output from the embedding network2, which may be written as query q=fγ(x)=h, is used to query the memory3and obtain the returned query9. The returned query9from the memory3is used to modify the parameters θ of the output network4. The returned query9may, in an implementation, indicate the K nearest key embeddings13in the memory3to the current key embedding h according to a distance measure, such as Euclidean distance using, for example, a kernel function. Optionally an approximate nearest neighbor search, such as a k-NN search, may be used. In the schematic representation show inFIG.4, five nearest neighbors are retrieved from the memory. However, it will be appreciated that any suitable number of nearest neighbors may be retrieved. The nearest neighbors are combined in a weighted sum to provide the returned query. The weights of the weights sum may correspond to the distance measure i.e. to the kernel function for the corresponding value.

In an implementation, the returned query9defines a context C of an input x, where the context C input x is defined as the keys hi, values viand associated weights wiof the K nearest neighbors13to the query q=fγ(x). The associated weights wimay be determined by the system1from distances between the K nearest key embeddings13and the current key embedding h. Thus the context C may be written as:
C={(hk(x),vk(x),wk(x))}k=1K

In an implementation, the weights are dependent upon a similarity between the query and the stored keys. For example in some implementations wk(x)∝kern(hk(x), q), where kern(hk(x), q) is a kernel function dependent upon the closeness of the query to each key hkin the set of nearest neighbor keys to (x). The weights may be normalized, i.e. divided by the sum of kern(hk(x), q) over the nearest neighbors. An example kernel function is:

kern⁡(hk(x),q)=1ϵ+h-q22

where ε is a predetermined constant to avoid a division by zero. The neural network system1can therefore select K key embeddings in the memory3that have the shortest distances to the current key embedding h according to the kernel function.

Using the context C described above, the parametrization of the likelihood of an output, y, given an input, x, takes the form:
p(y|x,θx)=p(y|x,θx,C)=gθx(fγ(x))

where θxare the weights of the output network4as modified by the weighted values read from the memory3, i.e. as modified by a soft read from the memory3.

Thus the parameterization of the likelihood differs from the standard parametric approach gθ(fγ(x)), used during training and described above, in that θ has been replaced with θx. In implementations θx=θ+ΔM(x, θ), with ΔM(x, θ) being a contextual update of the parameters of the output network4, e.g. being based upon the input x. Updating of parameters according to the p(y|x, θx) corresponds to decreasing the weighted average negative log-likelihood over the retrieved neighbors in C.

In implementations the parameters of the output network4are only modified temporarily during inference. The soft read from the memory applies a correction ΔM(x, θ) to the parameters of the output network which is dependent upon the specific input presented to the system at that time. However this “local adaptation” is afterwards discarded, allowing the weights of the parametric part of the system, that is of the embedding and output networks, to learn slowly thus facilitating generalization and long-term improved performance. As the model becomes better at fitting the training data the correction diminishes.

Given input x and context C, the maximum a posteriori likelihood of θxover the context C, given the parameters θ obtained after training, can be written as:

maxθx⁢log⁢⁢p⁡(θx❘θ)+∑k=1K⁢wk(x)⁢log⁢p⁡(vk(x)❘hk(x),θx,x)

The second term is a weighted likelihood of the data in C; the superscript (x) on the weights denotes that these depend on the particular input x. The first term can be thought of as a regularization term. For example in some implementations

log⁢⁢p⁡(θx❘θ)∝-θx-θ222⁢αM

i.e. a Gaussian prior on θxcentered at θ, where αMis the learning rate, which inhibits θxfrom moving too far away from θ to help prevent overfitting.

A fixed number of gradient descent steps, including just one, may be carried out to minimize the maximum a posteriori over the context C, where the fixed number of steps is represented by loop12inFIG.4. One step of gradient descent to the loss in the maximum a posteriori over the context C with respect to θxyields:

ΔM⁡(x,θ)=-αM⁢∇θ⁢∑k=1K⁢wk(x)⁢log⁢⁢p⁡(vk(x)❘hk(x)⁢θx,x)-β⁡(θ-θx)

where αMis the learning rate and β is a scalar hyper-parameter which defines the relative weight of the second, regularization term; the superscript (x) denotes that the nearest neighbor weights and values depend on x. Other forms of regularization term may be used, e.g. an L2 regularization, or this term may be omitted. The probability of value vkmay be given, for example, by the output of the parametric model, gθ(hk); e.g. in the case of a classification system, from a logit vector prior to a final softmax layer. Thus a correction to the output may be made by determining a gradient with respect to the parameters of the output network4of a loss function, which gradient depends upon an output of the parametric system, for each of a set of nearest neighbors to the query, weighted by the similarity to the query in the embedding space. The gradient is multiplied by a learning rate to determine an adjustment to the parameters of the output network. An example number of gradient adjustment steps is in the range 1-20; an example learning rate is in the range 0.1-1.0.

The updated parameters θxof the output network4are then used when processing the current embedding h, where h is the output fγ(x) obtained from the embedding network2for given input x. The updated parameters θxare discarded after use. That is, when a subsequent input is queried during inference, the contextual update ΔM(x, θ) is recalculated for the subsequent input and the recalculated values are used by the output network4to process the output fγ(x) of the embedding network2.

Modifying the output network4in the way described herein allows the system1to output an improved prediction to an unseen example, when the unseen example closely matches examples that have previously been seen during training. For example, the system1may be used as a classifier to classify numbers, and may have seen a number of examples of each character from 1 to 9. When the system1receives an input, such as the number 7, the memory3is queried using the embedding h of the input. The system1determines entries in the memory3which closely match the input, e.g. other examples of 7. The other examples of 7 are then used to modify the output network4such that the output network4is more likely to correctly classify the input as the number 7. The use of a memory3as described is particular advantageous when there is limited training data available for training the system1. The contextual update ΔM(x, θ) is such that, as the parametric model becomes better at fitting the training data (and consequently the episodic memories), it self-regulates and diminishes. Therefore, the system1is able to generate more accurate outputs when training data is limited, with the effect of the memory3gradually reducing as more training data becomes available and is used to train the system1.

In some implementations the output network4may be simpler than the embedding network2; this can facilitate adapting the output of the neural network system by adapting the parameters of the output network. For example the output network may comprise a last fully-connected layer and/or a softmax output layer of the neural network system. In some implementations the memory may store input data e.g. images from pixels such as raw pixel values, but look-ups may still be based on distance in the embedding space and the embeddings may be periodically re-computed to refresh them.

With reference toFIG.5, there is now described a method for use of the neural network system1in inference.At step S5, an input x is processed by the embedding network2to generate query q=fγ(x).At step S6, the K-nearest neighbors to the query q are retrieved from the memory3, and context C is determined, i.e. the keys, values and weights for the K-nearest neighbors.At step S7, a contextual update ΔM(x, θ+Δtotal) is calculated for one or more steps, for example using the formula given above, or a variant thereof. Initially Δtotal←0 and at each step Δtotal←Δtotal+ΔM(x).At step S8, prediction ŷ is output, where ŷ=gθ+Δtotal(h)=gθx(fγ(x)).

With reference toFIG.6, there will now be described a method of processing an input data item using a parametric model to generate output data, where the parametric model comprises a first sub-model and a second sub-model. The first sub model may be the embedding model2and the second sub-model may be the output network4as described above.At step S9the input data is processed by the first sub-model to generate a query data item. For example, the query data item may be query data item8shown inFIG.1.At step S10at least one data point-value pair is retrieved from a memory storing data point-value pairs, the at least one data point-value pair being based upon the query data item. The memory may form part of a non-parametric model, and may be the memory3described above.At step S11, weights of the second sub-model are modified based upon the retrieved at least one data point-value pair.At step S12, the output data is generated based upon the modified second sub-model to generate the output data. The output data may be output data10shown inFIG.1.

FIG.7conceptually illustrates operation of the neural network system1on a regression task: A query15has 4 nearest neighbors h1. . . h4with corresponding values. The parameters of the output network4are adjusted from those of the trained model, gθ16, to parameters gθxdefining curve17. This allows a more accurate fit to the data in the episodic memory than, for example, an attention-based model as represented by curve18.

The neural network system1may, for example, be used for any suitable purpose, such as in a language modelling system, an image processing system, or an action selection system. The neural network system1may be used for supervised and unsupervised learning tasks. For example, the supervised learning tasks may include classification tasks, such as image processing tasks, speech recognition tasks, natural language processing tasks, word recognition tasks, or optical character recognition tasks. The unsupervised learning tasks may include reinforcement learning tasks where an agent interacts with one or more real or simulated environments to achieve one or more goals.

The input data x may comprise, for example, one or more of: image data, moving image/video data, motion data, speech data, audio data, an electronic document, data representing a state of an environment, and/or data representing an action. For example, the image data may comprise color or monochrome pixel value data. Such image data may be captured from an image sensor such as a camera or LIDAR sensor. The audio data may comprise data defining an audio waveform such as a series of values in the time and/or frequency domain defining the waveform; the waveform may represent speech in a natural language. The electronic document data may comprise text data representing words in a natural language. The data representing a state of an environment may comprise any sort of sensor data including, for example: data characterizing a state of a robot or vehicle, such as pose data and/or position/velocity/acceleration data; or data characterizing a state of an industrial plant or data center such as sensed electronic signals such as sensed current and/or temperature signals. The data representing an action may comprise, for example, position, velocity, acceleration, and/or torque control data or data for controlling the operation of one or more items of apparatus in an industrial plant or data center. These data may, generally, relate to a real or virtual, e.g. simulated, environment.

The output data may similarly comprise any sort of data. For example in a classification system the output data may comprise class labels for input data items. In a regression task the output data may predict the value of a continuous variable, for example a control variable for controlling an electronic or electromechanical system such as a robot, vehicle, data center or plant. In another example of a regression task operating on image or audio data the output data may define one or more locations in the data, for example the location of an object or of one or more corners of a bounding box of an object or the time location of a sound feature in an audio waveform. In a reinforcement learning system the output data may comprise, for example, data representing an action, as described above, the action to be performed by an agent operating an in environment, for example a mechanical agent such as a robot or vehicle.

The data representing an action may comprise, for example, data defining an action-value (Q-value) for the action, or data parameterizing a probability distribution where the probability distribution is sampled to determine the action, or data directly defining the action, for example in a continuous action space. Thus in a reinforcement learning system the neural network system1may directly parameterize a probability distribution for an action-selection policy or it may learn to estimate values of an action-value function (Q-values). In the latter case multiple memories and respective output networks may share a common embedding network, to provide a Q-value for each available action.

The system and methods described herein may also be used as a component or module within a larger machine learning system, such as a component or module of a reinforcement learning system.

This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, a Microsoft Cognitive Toolkit framework, an Apache Singa framework, or an Apache MXNet framework.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.