Patent Description:
Artificial intelligence (AI) applications involve machines or software that are made to exhibit intelligent behavior such as learning, communication, perception, motion and manipulation, and even creativity. The machines or software can achieve this intelligent behavior through a variety of methodologies such as search and optimization, logic, probabilistic methods, statistical learning, and neural networks. Along these lines, various deep learning architectures such as deep neural networks (deep NN) including deep multi-layer perceptrons (MLPs) (often referred to as a DNN), convolutional deep neural networks, deep belief networks, recurrent neural networks (RNN), and long-short-term memory (LSTM) RNNs, have gained interest for their application to fields like computer vision, image processing/recognition, speech processing/recognition, natural language processing, audio recognition, and bioinformatics.

A deep NN generally consists of an input layer, an arbitrary number of hidden layers, and an output layer. Each layer contains a certain amount of units, which may follow the neuron model, and each unit corresponds to an element in a feature vector (such as an observation vector of an input dataset). Each unit typically uses a weighted function (e.g., a logistic function) to map its total input from the layer below to a scalar state that is sent to the layer above. The layers of the neural network are trained (usually via unsupervised machine learning) and the units of that layer assigned weights. Depending on the depth of the neural network layers, the total number of weights used in the system can be massive.

Many computer vision, image processing/recognition, speech processing/recognition, natural language processing, audio recognition, and bioinformatics are executed and managed at data centers supporting services available to large numbers of consumer and enterprise clients. Data centers are designed to run and operate computer systems (servers, storage devices, and other computers), communication equipment, and power systems in a modular and flexible manner. Data center workloads demand high computational capabilities, flexibility, power efficiency, and low costs. Being able to accelerate at least some portions of large-scale software services can achieve desired throughputs and enable these data centers to meet the demands of their resource consumers. However, the increasing complexity and scalability of deep learning applications can aggravate problems with memory bandwidth.

<NPL> * sections II-IV, discloses in the abstract that many applications which take advantage of accelerators are amenable to approximate execution. Past work has shown that neural acceleration is a viable way to accelerate approximate code. In light of the growing availability of on-chip field-programmable gate arrays (FPGAs), this paper explores neural acceleration on off-the-shelf programmable SoCs. We describe the design and implementation of SNNAP, a flexible FPGA-based neural accelerator for approximate programs. SNNAP is designed to work with a compiler workflow that configures the neural network's topology and weights instead of the programmable logic of the FPGA itself. This approach enables effective use of neural acceleration in commercially available devices and accelerates different applications without costly FPGA reconfigurations. No hardware expertise is required to accelerate software with SNNAP, so the effort required can be substantially lower than custom hardware design for an FPGA fabric and possibly even lower than current "C-to-gates" high-level synthesis (HLS) tools. Our measurements on a Xilinx Zynq FPGA show that SNNAP yields a geometric mean of <NUM>× speedup (as high as <NUM>×) and <NUM>× energy savings (as high as <NUM> x) with less than <NUM>% quality loss across all applications but one. We also compare SNNAP with designs generated by commercial HLS tools and show that SNNAP has similar performance overall, with better resource-normalized throughput on <NUM> out of <NUM> benchmarks.

<NPL> discloses in the abstract how artificial neural networks (ANNs) are a natural target for hardware acceleration by FPGAs and GPGPUs because commercial-scale applications can require days to weeks to train using CPUs, and the algorithms are highly parallelizable. Previous work on FPGAs has shown how hardware parallelism can be used to accelerate a "Restricted Boltzmann Machine" (RBM) ANN algorithm, and how to distribute computation across multiple FPGAs. KIM ET AL describe a fully pipelined parallel architecture that exploits "mini-batch" training (combining many input cases to compute each set of weight updates) to further accelerate ANN training, and implement on an FPGA, a more powerful variant of the basic RBM, the "Factored RBM" (fRBM). The fRBM has proved valuable in learning transformations and in discovering features that are present across multiple types of input. A <NUM>-fold acceleration (vs. CPU software) for an fRBM having N = <NUM> units in each of its four groups (two input, one output, one intermediate group of units) running on a Virtex-<NUM> LX760 FGPA is obtained (in simulation) Many of the architectural features are applicable not only to fRBMs, but to basic RBMs and other ANN algorithms more broadly.

United States Patent Application <CIT> in the abstract discloses the use of a pipelined algorithm that performs parallelized computations to train deep neural networks (DNNs) for performing data analysis may reduce training time. The DNNs may be one of context-independent DNNs or context-dependent DNNs. The training may include partitioning training data into sample batches of a specific batch size. The partitioning may be performed based on rates of data transfers between processors that execute the pipelined algorithm, considerations of accuracy and convergence, and the execution speed of each processor. Other techniques for training may include grouping layers of the DNNs for processing on a single processor, distributing a layer of the DNNs to multiple processors for processing, or modifying an execution order of steps in the pipelined algorithm.

Memory bandwidth management techniques and systems for accelerating neural network evaluations are described.

In a data center, a neural network evaluation accelerator includes a processor that supports parallel processing ("parallel processor"), such as a field programmable gate array (FPGA). This processor, which is separate from the general computer processing units (CPUs) at the data center, performs a process using a weight dataset loaded from external memory after at least two observation vectors from a same or different data streams (from the cores of the CPUs). By queuing up input data of at least two streams or at least two observation vectors before applying the weighted dataset, the memory bandwidth requirement for the neural network weight loading can be reduced by a factor of K, where K is the number of input datasets in a batch. In addition, by using a processor that supports parallel processing, N simultaneous streams can be processed in parallel lock step to ensure that the memory bandwidth requirement for N parallel streams remains the same as it is for a single stream. This enables a throughput of N*K input datasets for each loading of a weight dataset.

Memory bandwidth management techniques and systems are described that accelerate neural network evaluations.

Due to the computation pattern of many neural network evaluations, general purpose processors and pure software based solutions tend to be inefficient and, in some cases, unable to meet performance requirements for the applications that they form a part. Furthermore, these evaluations tend to be limited by the resources available at the data centers performing the computations. By including FPGAs in data centers, and leveraging these FPGAs in the manners described herein, it is possible to perform the complex deep neural network evaluations within the processor-to-memory bandwidth constraints as well as the machine-to-machine networking bandwidth constraints of current data centers. In some cases, particularly where power consumption efficiency is not a priority or fewer parallel computation streams are needed, graphics processing units (GPUs) are used to perform the neural network evaluations.

<FIG> illustrates an example operating environment providing memory bandwidth management for deep learning applications. Referring to <FIG>, a data center <NUM> can include a number of resources <NUM> - physical and virtual - on which applications and services can be hosted. Routing server(s) <NUM> can facilitate the directing of a request to the appropriate resource. One or more of the routing server(s) <NUM> may be physically present in a particular data center <NUM>. The service(s) hosted at the data center <NUM> may service many clients such as Client0 <NUM>, Client1 <NUM>, Client2 <NUM>, Client3 <NUM>, Client4 <NUM>, Client5 <NUM>, and Client6 <NUM>, that communicate with the service(s) (and access the data center resource(s) <NUM>) over the Internet <NUM>.

Various implementations of the described techniques are suitable for use as part of a process for services involving computer vision, image processing/recognition, speech processing/recognition, natural language processing, audio recognition, bioinformatics, weather prediction, stock forecasting, control systems and any other application where neural networks may be applied.

As an example scenario, the operating environment supports a translation service for an audio or video call. The translation service involves a deep learning application to recognize words from a conversation. The clients (e.g., <NUM>, <NUM>, <NUM>,. ) can enable a user to elect to participate in the translation service so that the audio of the user's conversation can be sent to the translation service. For example, the audio from a conversation at a device running Client0 <NUM> can be sent as Input0 to the translation service, the audio from a conversation at a device running Client1 <NUM> can be sent as Input1 to the translation service, the audio from a conversation at a device running Client2 <NUM> can be sent as Input2 to the translation service, and the audio from a conversation at a device running Client3 <NUM> can be sent as Input3 to the translation service.

These independent conversations can be processed at the data center <NUM> and have an output sent to a same client or different client (that may or may not be participating in sending audio to the service). For example the translated conversation from Input0 can be sent to Client4 <NUM> as Output0, the translated conversation from Input1 can be sent to Client5 <NUM> as Output1, the translated conversation from Input2 can be sent to Client6 <NUM> as Output2, and the translated conversation from Input3 can be sent back to Client3 <NUM> as Output3. Accelerating one or more of the processes associated with the translation services can help the real-time functionality of such a service. However, any acceleration - or even just actual computation - is constrained, at least in part, by the physical limitations of the systems - the data center resources <NUM> - at the data center <NUM>.

<FIG> illustrates an architecture for managing and accelerating at least a component of a deep learning application hosted by resources of a data center. In a data center <NUM> housing numerous servers, switches, and other equipment for a variety of services and applications, such as described with respect to data center <NUM> of <FIG>, deep learning applications, and particularly a neural network evaluation where the weight dataset is in the megabytes (tens of megabytes, hundreds of megabytes or even more), are accelerated and its memory bandwidth requirements reduced by batch processing of the input data.

A server at the data center <NUM> includes a processor with two or more cores. Each core tends to handle a single thread, or stream of data. A parallel processor <NUM> is used to accelerate a neural network evaluation. The parallel processor <NUM> is an FPGA. The FPGA can show improved power savings over the use of a GPU.

The parallel processor <NUM> includes input buffers <NUM>, which have a specified queue depth (of K=<NUM> or more), and output buffers <NUM> for holding data before the data is further processed and/or output to another component. In some cases, the parallel processor can include logic <NUM> for processing data from one or more of the input buffers <NUM>. The logic <NUM> may be programmable and reconfigurable between and/or during operations depending on the type of parallel processor <NUM>.

An "observation vector" refers to the initial dataset (or feature vector) that is input to the neural network and is used to start or trigger the recognition or classification process. These could include data representing colors, prices, sound amplitudes or any other quantifiable value that may have been observed in the subject of interest. The observation vectors input to the parallel processor <NUM> is generated by a core of a CPU (discussed in more detail in the example below), by another computational unit such as an FPGA, a GPU, or other logic on the parallel processor <NUM> that is performing the neural network evaluation.

The "intermediate outputs" or "intermediates" refer to the internal state values of the neural network that are used to track progress of data through the network for either the current evaluation, or across multiple evaluations of the network (in the case of RNNs).

The intermediate values may correlate with features of an observation vector, but typically they represent some abstracted form of the original observation data as the network algorithm "reasons" about the inputs that it was given. Logically, intermediate values represent the network's attempt to categorize data based in a hyper dimensional decision line between competing concepts. Mathematically, intermediate values represent the nearness that the observation vector, prior intermediate values, or a combination of both, appear to the dividing line between competing concepts that one or more neural network nodes represent.

As illustrated in <FIG>, a single parallel processor <NUM> receives input data from multiple cores such as Core0 <NUM>, Core1 <NUM>, Core2 <NUM>, and Core3 <NUM>, which may be provided in one or more processing units housed in one or more servers. One server contains a processing unit with, as is common today, <NUM>-<NUM> cores. The input data from these cores is loaded into corresponding ones of the input buffers <NUM>. In other applications, input data from one of these cores may be loaded into more than one input buffer.

In another implementation, not covered by the subject-matter of the claims, instead of many separate queues (provided by the input buffers <NUM>), one for each of the N cores, the parallel processor can have a single queue where the different cores add their datasets to that single queue as they become available. The parallel processor could periodically poll that queue (after each complete evaluation of the deep NN) and read a new batch of datasets from the queue for parallel processing. The new batch of datasets would be processed through the deep NN in parallel and then a decoder process, either on the parallel processor or on one of the CPU cores, would send them back to the appropriate core. Each dataset would be tagged by the core that sent it to facilitate this de-multiplexing operation. This type of implementation is suitable for cases where the parallel processor such as the FPGA can handle the computational load such as the addition of the decoding process.

As mentioned above, the observation vectors are generated by the cores and provided directly as the input data to the parallel processor <NUM>; however, in some cases, the observation vectors are be output by the cores. In some of those cases, the parallel processor <NUM> generates the observation vectors (using separate logic to do so) or other computational units are generating the observation vectors from data output from the cores or in systems fully implemented using other computational units. Thus when a core and data stream is described herein, the use of other computational units can be considered as other implementations that may be suitable as the processing unit for a particular recognition process (or other application that benefits from deep learning).

When the parallel processor performs a weighted function for the neural network evaluation, the weight dataset is generally too large to be stored on-chip with the processor. Instead, the weight dataset is stored in off-chip storage <NUM> and loaded, in partitions small enough for on-chip storage, onto the parallel processor <NUM> each time the particular weighted function is carried out. For maximum efficiency, weights must be loaded at the speed the processor can consume them, which requires a significant amount of memory bandwidth. Off-chip storage <NUM> includes memory modules (e.g., DDR, SDRAM DIMMs), hard drives (solid state, hard disk, magnetic, optical, etc.), CDs, DVDs, and other removable storage devices. It should be understood that the storage <NUM> does not consist of propagating signals.

According to various techniques described herein, the memory bandwidth is managed by processing parallel streams of data (e.g., from the cores Core0 <NUM>, Core1 <NUM>, Core2 <NUM>, and Core3 <NUM>) in batches at the parallel processor <NUM> so at least two feature vectors are processed for each layer's set of weight data input to the parallel processor <NUM> from the off-chip storage <NUM>. Although the described techniques are useful for two feature vectors (from a same or different stream of data), when at least four feature vectors are processed in parallel, a noticeable effect on bandwidth and/or power efficiency can be seen. For example, doubling the number of items processed in parallel will roughly halve the memory bandwidth.

In some cases, acceleration of deep NN evaluation can be managed by a manager agent <NUM>. The manager agent <NUM> can be implemented in software executable by any suitable computing system, including physical servers and virtual servers, and any combination thereof. The manager agent <NUM> may be installed on and run in the context of a virtual machine in some scenarios or may be directly installed and executed on a computing system in a non-virtualized implementation. In some cases, the manager agent <NUM> may be implemented in whole or in part in hardware.

The manager agent <NUM>, when used, can coordinate the timing of communicating data between various components at the data center, between the off-chip storage <NUM> that stores weights for the deep NN evaluation and the parallel processor <NUM>. Accordingly, in certain embodiments, the manager agent <NUM> and/or the bus/data routing configuration for the data center <NUM> enables datasets (e.g., from the cores Core0 <NUM>, Core1 <NUM>, Core2 <NUM>, Core3 <NUM>) to be communicated to a single parallel processor <NUM> for processing in batches.

<FIG> illustrate a comparison of bandwidth management in neural network evaluations. In the scenario illustrated in <FIG>, the deep NN evaluation involves a software or hardware evaluation and does not include the acceleration or memory management as described herein. Instead, the weight datasets stored in the external memory <NUM> are applied separately to each dataset (e.g., from the cores Core0 <NUM> and Core1 <NUM>) during the corresponding deep NN evaluations (e.g., DNN evaluation(<NUM>) <NUM> and DNN evaluation(<NUM>) <NUM>, respectively). In particular, a first layer weight set <NUM> from external memory <NUM> and a first feature vector Vector0 <NUM> from a first stream Stream0 from Core0 <NUM> are retrieved/received (<NUM>) for the DNN evaluation(<NUM>) <NUM>; and a first layer process is performed (<NUM>), generating intermediates <NUM>.

A second layer weight set <NUM> from the external memory <NUM> is then retrieved/received <NUM> in order to perform the second layer process (<NUM>) on the intermediates <NUM>. This evaluation process, where the weights are retrieved/received from external memory <NUM>, continues for each layer until the entire process is completed for a particular input vector. The process can repeat for each input feature vector (e.g., Vector01 of Vector0). If there are multiple cores running the deep NN evaluations, then multiple deep NN evaluations could be performed in parallel, but, the without memory management as described herein, each evaluation requires retrieving/receiving the weight dataset from external memory <NUM> as an independent request, and use of an additional processor/core.

For example, the first layer weight set <NUM> from external memory <NUM> and a second feature vector Vector1 <NUM> from a second stream Stream1 from Core1 <NUM> are retrieved/received (<NUM>) for the DNN evaluation(<NUM>) <NUM>; and a first layer process is performed (<NUM>) to generate intermediates <NUM> for the second stream. The second layer weight set <NUM> from the external memory <NUM> is then retrieved/received <NUM> in order to perform the second layer process (<NUM>) on the intermediates <NUM>. As with the DNN evaluation(<NUM>) <NUM>, the DNN Evaluation(<NUM>) <NUM> continues for each layer until the entire process is completed for a particular input feature vector, and repeated for each input feature vector (e.g., Vector11 of Stream1). As can be seen, this is not an efficient mechanism for performing the deep NN evaluations and requires considerable memory bandwidth to perform since each evaluation requires a retrieval of the weighted datasets and its own processor/core(s) to perform the evaluation.

In the scenario illustrated in <FIG>, a memory managed, accelerated deep NN process, not covered by the subject-matter of the claims, is using an FPGA <NUM> to perform the deep NN evaluation on two feature vectors from a same stream of data. Here, two feature vectors, observation vectors Vector0 and Vector01 <NUM> is loaded onto the FPGA <NUM> from Core0 <NUM> and evaluated as a batch. The first layer weight set <NUM> is received/retrieved for loading (<NUM>) on the FPGA <NUM> and the first layer process performed (<NUM>) at the FPGA <NUM> on both Vector0 and Vector01 <NUM>. Intermediates <NUM> from the first layer process can be available when the first layer process is complete for both observation vectors; and the second layer weight set <NUM> loaded (<NUM>) onto the FGPA <NUM>.

The intermediates <NUM> can be loaded into a buffer (such as input buffer <NUM> of <FIG>) for the next layer of processing and the second layer process performed (<NUM>). The deep NN evaluation continues at the FPGA <NUM> for each layer until the entire process is completed for the two observation vectors Vector0 and Vector01 <NUM>. This process can then be repeated for the next pair of observation vectors (e.g., Vector02 and Vector03). It should be understood that although only two feature vectors are described, more than two feature vectors/observation vectors can be evaluated in parallel using the FPGA. This will further reduce the required memory bandwidth, but will find its limit in the latency of buffering the batch from the single stream of data, because in real-time applications, buffering N observation vectors before commencing computation causes the system's output to be delayed by N vectors, and in some applications, acceptable delays are not large enough to allow efficient operation of the parallel processor. Further, batching of vectors from a same stream is not applicable to recurrent networks (RNNs) because there, the vectors within a batch are not independent (computation at time step t requires the output of time step t-<NUM>).

In the scenario illustrated in <FIG>, a memory managed, accelerated deep NN process using the FPGA <NUM> entails performing the DNN evaluation on two feature vectors from two different streams of data. That is, two feature vectors, one observation vector Vector0 <NUM> from Core0 <NUM> and one observation vector Vector1 <NUM> from Core1 <NUM> are evaluated as a batch. The first layer weight set <NUM> can be received/retrieved for loading (<NUM>) on the FPGA <NUM> and the first layer process performed (<NUM>) at the FPGA <NUM> on both Vector0 <NUM> and Vector1 <NUM>, generating intermediates <NUM>. The intermediates <NUM> (the data output of the first layer process) can be loaded into input buffers for the next layer of processing. Unlike the scenario illustrated in <FIG>, which is suitable for evaluating a DNN for MLPs but not suitable for evaluating the deep NN of a RNN, this scenario is suitable for evaluating a RNN (recurrent neural network) because all observations within the batch are from different streams and thus independent.

After the first layer process is complete for both observation vectors <NUM>, <NUM>, the second layer weight set <NUM> is loaded (<NUM>) onto the FGPA <NUM> and the second layer process performed (<NUM>). The deep NN evaluation continues at the FPGA <NUM> for each layer until the entire process is completed for the two observation vectors Vector0 <NUM> and Vector1 <NUM>. This process can then be repeated for the next observation vectors for these two streams (e.g., Vector01 and Vector11). Of course, although only two streams and cores are shown for simplicity, more than two can be evaluated in parallel at the FPGA <NUM>.

In the scenario illustrated in <FIG>, a memory managed, accelerated deep NN process using the FPGA <NUM> entails performing the deep NN evaluation on four feature vectors - two feature vectors (observation vectors) each from two streams of data. That is, two observation vectors <NUM> (Vector0, Vector01) from Core0 <NUM> and two observation vectors <NUM> (Vector1, Vector11) from Core1 <NUM> are loaded and evaluated as a batch. The two observation vectors from each stream can be loaded by having a queue depth of two for the input buffers of the FPGA (see e.g., input buffers <NUM> <FIG>). Accordingly, with a single loading (<NUM>) of the first layer set of weights <NUM>, the two observation vectors <NUM> (Vector0, Vector01) from Core0 <NUM> and the two observation vectors <NUM> (Vector1, Vector11) from Core1 <NUM>, the first layer process can be performed.

As described above with respect to the scenario illustrated in <FIG>, although this scenario is suitable for various deep NN architectures evaluations, it is not as suitable to RNNs due to the dependent nature of the vectors in a batch as described with respect to <FIG>.

The intermediates <NUM> (the data output of the first layer process) can be loaded into input buffers for the next layer of processing, the second layer weight set <NUM> can be retrieved/received for loading (<NUM>) onto the FPGA <NUM> and then the second layer process performed (<NUM>). The DNN evaluation continues at the FPGA <NUM> for each layer until the entire process is completed for at least the four observation vectors Vector0, Vector01, Vector1, and Vector11. This process can then be repeated for the next pair of observation vectors for these two streams (e.g., Vector02, Vector03 and Vector12, Vector13). Of course, this scenario is also scalable to additional cores being handled in parallel by the FPGA <NUM>.

As can be seen from the illustrated scenarios, the configurations shown in <FIG> and <FIG> reduce the memory bandwidth needed to evaluate the same amount of data as the configuration shown in <FIG>. In addition, the configuration shown in <FIG> can even further reduce the necessary memory bandwidth. For the configurations shown in <FIG> and <FIG>, there is a latency cost for the time to queue the multiple observation vectors from a single data stream. In addition, there may further be some latency cost for evaluating twice (or more) the amount of data through each line of the available parallel processes.

Accordingly, the input data of at least two streams and/or at least two observation vectors are queued for processing at an FPGA to reduce the memory bandwidth requirement for neural network weight loading by a factor of K, where K is the number of input datasets in a batch (and can also be considered the queue depth for the FPGA). For optimum bandwidth efficiency, processing occurs when a batch of K input datasets are accumulated in the on-chip FPGA memory. By queueing the inputs in this manner, any I/O bound problem where the bandwidth of the database (weights) required for processing the input dataset is prohibitive can be handled. Thus, in general, for a required bandwidth B, the average effective bandwidth needed using the queuing method is B/K. N simultaneous streams can be processed in parallel lock step to ensure that the memory bandwidth requirement for N parallel streams remains the same as it is for a single stream.

For an Internet translator, conversations may arrive at the data center after being input via a microphone at a client and be transformed (e.g., via a Fast Fourier Transform) to establish power bands based on frequency (from which the observation vectors can be obtained). The deep NN evaluation for the scenario involves a DNN (for MPL) performing eight layers of matrix multiplication, adding a bias vector and, for all but the top layer, applying a non-linearity function. The output of the DNN evaluations can establish probability scores, indicating what the probability that the slice being looked at belongs to a unit of speech, for example that the slice being looked at belongs to a middle part of an "ah" pronounced in left context of "t" and right context of "sh". Additional processing tasks performed by the processor cores can involve identifying the words based on the probability scores and applying against certain dictionaries to perform a translation between languages.

<FIG> illustrates an implementation of accelerating a deep NN process using an FPGA. Referring to <FIG>, a server blade <NUM> at a data center can include external storage <NUM> storing the weight datasets. A single FPGA <NUM> is capable of performing all of the deep NN evaluation for an entire server blade containing <NUM> CPU cores <NUM> (N=<NUM>); leaving those cores <NUM> to handle the other processing tasks required for <NUM> simultaneous conversations while the deep NN evaluation is being carried out.

In the scenario, input datasets from each of the <NUM> cores can be loaded on to the input buffers <NUM> having a queue depth of K=<NUM>. That way two observation vectors from each conversation/core's data stream can undergo processing through the layer logic <NUM>, for example a matrix multiply (such as for deep MLP) or multiple parallel matrix multiplies and non-linearity's (such as for LSTM RNN), as a single batch. The intermediates <NUM> from a layer of the layer logic <NUM> can be routed back (<NUM>) to undergo another process with a new weighted function when a new weight data set is loaded from the storage <NUM>. This process can be repeated until all the layers have been processed, at which time the output is then sent back to the processing cores, which may demultiplex (demux) the data at some point.

Live translation of speech requires careful attention to latency, power, and accuracy. FPGAs typically have relatively low power requirements (10W) and yet can still deliver high compute performance. Since using only CPU cores (a pure software approach) for performing speech recognition using a deep NN evaluation typically requires at least <NUM> CPU cores per conversation, where at least two are consumed for deep NN evaluation, the FPGA <NUM> is able to effectively remove the need for <NUM>*<NUM> = <NUM> CPU cores, which translates to high power savings. For example, assuming a bloated estimate of 25W for FPGA power consumption, and a reasonable average power consumption for a CPU core of 100W/<NUM> = <NUM>. 33W, the net power savings would be on the order of <NUM>*<NUM>. 33W - 25W = 375W per server blade. Calculated another way, without an FPGA, the power use would be <NUM>*<NUM>. 33W = 25W per conversation, while with the FPGA the power per conversation would be <NUM>. 33W + 25W/<NUM> = <NUM>.

When scaled to large numbers of users, the ~3x increase in compute power needed by the pure software deep NN approach as opposed to just using a single CPU core for a conversation (and a single FPGA for easily <NUM> conversations) makes a pure software approach cost prohibitive, even though deep NNs provide greater recognition accuracy and hence a better user experience when incorporated in speech recognition and translation services. Therefore, the inclusion of the FPGA <NUM> enables deep NNs to be incorporated into speech recognition and translation services.

Usually, performing the deep NN evaluation on the FPGA would entail a very high bandwidth requirement. One of the primary difficulties with the FPGA implementation is the management of memory bandwidth. For one exemplary Internet translator, there are approximately <NUM> million <NUM>-bit neural network weights that must be processed for each complete evaluation of the neural network. For every evaluation of the neural network, the FPGA must load <NUM>*<NUM> bytes = <NUM> bytes of data from memory. In order to meet performance specifications, the neural network must be evaluated <NUM> times per second; per conversation. For even one conversation, this means that the memory bandwidth requirement for the FPGA is <NUM>* 100MB = 10GB/Sec. The absolute peak memory bandwidth of the typical FPGA memory interface is about <NUM> GB/Sec, but this is rarely achieved and assumes perfect operating conditions with no other activity being present in the system. If one considers that the task of the FPGA is to process N=<NUM> such conversations simultaneously, the problem appears intractable. However, the techniques illustrated in <FIG> (and reflected in <FIG> for the specific implementation of N=<NUM>) can address this problem.

First considering the single conversation case, which is not covered by the scope of the claims and can be considered illustrated in <FIG>, the memory bandwidth requirement may be lowered by batching the input data in groups of K input datasets (of observation vectors). By delaying processing until K input datasets have been accumulated, and then loading the neural network weight data once for all K inputs, the effective memory bandwidth required goes down by a factor of K (while delaying speech recognition output by the duration corresponding to K-<NUM> vectors, for example (K-<NUM>) * <NUM>). For example, if the memory bandwidth requirement is 10GB/Sec, and K=<NUM>, the effective memory bandwidth required is 10GB/Sec / <NUM> = 5GB/Sec, which is a much more manageable figure. Greater values of K result in lower effective memory bandwidth and can be chosen to reduce the memory bandwidth requirement to a manageable number for the application. This comes at the cost of added computational latency, as input datasets are delayed until K have been accumulated, but since maintaining throughput can be more important than latency in certain situations, it is a good tradeoff in those certain situations.

In the case of processing N simultaneous conversations, such as illustrated in <FIG> with N=<NUM> and K=<NUM> and K=<NUM> respectively, each conversation uses a queue of K input datasets and N such queues in use simultaneously (N=<NUM> and K=<NUM> for the example illustrated in <FIG>). The input datasets are scheduled so that all N queues are processed in lock step, using exactly the same weights simultaneously across all queues during a layer of the layer logic <NUM> (such as a matrix multiply or other weighted processing step, which can be carried out on intermediates <NUM> that are then re-queued <NUM> for the next layer processing with new weights). Thus, the neural network weight data is only loaded a single time for all N conversations (for each layer of the process), and the memory bandwidth requirement for the FPGA remains the same as if only a single conversation were being processed.

<FIG> is a block diagram illustrating components of a computing device or system that may be used to carry out some of the processes described herein. Referring to <FIG>, system <NUM> can include one or more blade server devices, standalone server devices, personal computers, routers, hubs, switches, bridges, firewall devices, intrusion detection devices, mainframe computers, network-attached storage devices, and other types of computing devices. The hardware can be configured according to any suitable computer architectures such as a Symmetric Multi-Processing (SMP) architecture or a Non-Uniform Memory Access (NUMA) architecture. Accordingly, more or fewer elements described with respect to system <NUM> may be incorporated to implement a particular computing system.

The system <NUM> includes a processing system <NUM>, which includes one or more processing devices such as a central processing unit (CPU) with one or more CPU cores, a microprocessor or other circuitry that retrieves and executes software <NUM> from storage system <NUM>. Processing system <NUM> may be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions.

The one or more processing devices of processing system <NUM> includes multiprocessors or multi-core processors and may operate according to one or more suitable instruction sets including, but not limited to, a Reduced Instruction Set Computing (RISC) instruction set, a Complex Instruction Set Computing (CISC) instruction set, or a combination thereof. In certain embodiments, one or more digital signal processors (DSPs) may be included as part of the computer hardware of the system in place of or in addition to a general purpose CPU.

Storage system <NUM> comprises any computer readable storage media readable by processing system <NUM> and capable of storing software <NUM> including instructions for performing various processes in which the neural network evaluation performed on an FPGA forms a part. Software <NUM> may also include additional processes, programs, or components, such as operating system software, database management software, or other application software. Software <NUM> may also include firmware or some other form of machine-readable processing instructions executable by processing system <NUM>. In addition to storing software <NUM>, storage system <NUM> may store matrix weights and other datasets used to perform neural network evaluations. In some cases, the manager agent <NUM> is stored, at least in part, on a computer-readable storage medium forming part of the storage system <NUM> and implementing virtual and/or non-virtual memory.

Storage system <NUM> may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data.

Although storage system <NUM> is shown as one block, storage system <NUM> represents the on-chip and external storage available to computing system <NUM>. Storage system <NUM> can include various storage media such as random access memory (RAM), read only memory (ROM), magnetic disks, optical disks, CDs, DVDs, flash memory, solid state memory, phase change memory, or any other suitable storage media. Certain implementations may involve either or both virtual memory and non-virtual memory. In no case do storage media consist of a propagated signal or carrier wave. In addition to storage media, in some implementations, storage system <NUM> may also include communication media over which software and data may be communicated internally or externally.

In some cases, the processing system <NUM> can access the storage system <NUM> (or parts of the storage system <NUM>) by system bus. Storage system <NUM> may include additional elements, such as a controller, capable of communicating with processing system <NUM>.

Computing system <NUM> further includes an FPGA <NUM> for performing neural network evaluations. Multiple FPGAs may be available in a data center. In some cases, a plurality of FPGAs can be incorporated into a daughter card and housed with a subset of the servers. Alternatively, a single FPGA may be housed in a single server, where services requiring more than one FPGA are mapped across FPGAs residing in multiple servers and/or services requiring more than one server accesses a single FPGA residing at one of the servers. In some cases, one or more FPGAs may be housed separately from the servers. When incorporated in a same server, the FPGA(s) are coupled to the processing system <NUM> on a same board or on separate boards interfaced with a communications interface technology such as PCIe (PCI express).

A communication interface <NUM> is included, providing communication connections and devices that allow for communication between device <NUM> and other computing systems (not shown) over a communication network or collection of networks (not shown) or the air. Examples of connections and devices that together allow for intersystem communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices may communicate over communication media to exchange communications with other computing systems or networks of systems, such as metal, glass, air, or any other suitable communication media. The aforementioned communication media, network, connections, and devices are well known and need not be discussed at length here.

It should be noted that many elements of device <NUM> may be included in a system-on-a-chip (SoC) device. These elements may include, but are not limited to, the processing system <NUM>, elements of the storage system <NUM>, and even elements of the communications interface <NUM>.

It should be understood that the examples and embodiments described herein are for illustrative purposes only.

Claim 1:
A method of performing neural network processes using a field programmable gate array, FPGA (<NUM>), the method comprising:
receiving at a plurality of N input buffers (<NUM>) included in the field programmable gate array (<NUM>), a batch of input data for accelerated processing of a neural network evaluation for a multi-layer neural network comprising a first layer, the batch of input data comprising N observation vectors, wherein each respective observation vector of the N observation vectors belongs to a corresponding respective one of N data streams, wherein each input buffer (<NUM>) of the N input buffers (<NUM>) provides a respective queue of a plurality of N queues for receiving a respective observation vector of the N observation vectors of the batch of input data, each queue having a queue depth of at least two, where N is the number of parallel streams of the field programmable gate array (<NUM>) that can be processed in parallel lock step;
loading (<NUM>, <NUM>, <NUM>) the field programmable gate array (<NUM>) with a first layer set of weights (<NUM>) for the neural network evaluation from an external memory (<NUM>); and
performing a first layer process by applying (<NUM>, <NUM>, <NUM>), within the field programmable gate array (<NUM>), the first layer set of weights to the N observation vectors to generate intermediate outputs (<NUM>, <NUM>, <NUM>) for the neural network evaluation; wherein the N observation vectors are scheduled so that all the plurality of N queues are processed in parallel lock step, using the same weights simultaneously across all of the N queues during the first layer process, wherein the intermediate outputs generated from the N observation vectors being processed are re-queued in the input buffers for subsequent processing using a second layer set of weights of the multi-layer neural network.