Patent Description:
Various mobile device applications (e.g., camera application, social media application, or the like) may each need to use a corresponding neural network model on a hardware accelerator. Conventionally, the architecture of the hardware accelerator permits storage of parameters of a single neural network model in the memory of the hardware accelerator, and thus a compiler for the hardware accelerator is able to compile only one neural network model at a time. When another neural network model (e.g., another model for another application) needs to be executed, the parameters of that neural network model replace the parameters of the previous neural network model. Accordingly, when the previous neural network model needs to be processed by the hardware accelerator, parameters of that model are again loaded into one or more memories of the hardware accelerator. Such reloading of the parameters into the memories can consume significant amounts of memory and power, thereby causing latency.

<CIT> discloses technology related to hardware accelerated neural network subgraphs.

<CIT> relates to memory management in gaming rendering.

A compiler of a computing device is described that identifies a sequence of neural network models frequently invoked by an application of the computing device, compiles the models in that sequence, and loads a static random access memory (SRAM) of a hardware accelerator with the compiled models only when the same compiled models from another, but same sequence that was previously invoked are not already present in the SRAM.

According to the invention, a method performed by a compiler is described. The compiler identifies a first set of neural network models that have been executed on a hardware accelerator of a computing device more than a threshold number of times in a preset amount of time in the past. The compiler identifies a sequence in which the first set of models are executed on the hardware accelerator. Each neural network model of the first set of neural network models is compiled for execution by the hardware accelerator. For each neural network model of the first set of neural network models, the compiler outputs the compiled model to the hardware accelerator for storage according to the sequence in one or more memories of the hardware accelerator. The storage, according to the sequence, of the compiled model for each neural network model of the first set of neural network models in the one or more memories prevents a need for reloading a compiled result recompilation of the sequence of the first set of neural network models into the one or more memories when the first set of neural network models are to be executed again on the hardware accelerator.

For each neural network model of the first set of neural network models, a data structure including parameters of the neural network model is received. The compiling further includes compiling the data structure for each neural network model of the first set of neural network models to generate a compiled data structure for each neural network model of the first set of neural network models, wherein the compiled data structure is the compiled model.

A same first hash can be assigned to each compiled model in the sequence. The first hash can be output along with each compiled model in the sequence to the hardware accelerator for storage in the one or more memories of the hardware accelerator. A same second hash can be assigned to each compiled model in a second sequence of models. The second sequence can be subsequent to the first sequence. The second hash can be same as the first hash when the second sequence is same as the first sequence. The second hash can be different from the first hash when the second sequence is different from the first sequence. If the second hash is different from the first hash, the hardware accelerator is configured to replace each compiled model in the first sequence in the one or more memories with each compiled model in the second sequence in the one or more memories. If the second hash is same as the first hash, the hardware accelerator is configured to prevent erasing each compiled model in the first sequence from the one or more memories.

Each of the first set of neural network models has been processed on the hardware accelerator more than a preset number of times (e.g. five times) in the past. The compiler compiles the first set of neural network models while the hardware accelerator simultaneously performs neural network computations of other one or more neural network models. The identification of the first set of models and the identification of the sequence can be updated after preset intervals of time. The compiler can abstain, for a preset time, from the updating in response to a failure of the compilation of the first set of neural network models. The abstaining can include abstaining, for <NUM> milliseconds, from the updating in response to the failure of the compilation of the first set of neural network models.

The compiling of the first set of neural network models can include determining that each neural network model within the first set of neural network models has a particular size that is compatible for the compiling. The compiling of the first set of neural network models can include compiling only a single neural network model at any time. The sequence can include a face recognition neural network model and one or more dependent neural network models that are to be processed after processing the face recognition neural network model.

In another aspect, a system is described according to claim <NUM>.

In some implementations, one or more of the following can additionally be implemented either individually or in any feasible combination. The one or more memories can be static random access memory (SRAM). The hardware accelerator can further include a plurality of computing units configured to process the first set of neural network models. Each computing unit of the plurality of computing units can include at least one processor and a memory. The plurality of computing units can be coupled serially via at least one bus. The first set of neural network models can include at least one face recognition neural network model. The at least one face recognition neural network model can be activated in response to the controller receiving an instruction from the computing device to execute the at least one face recognition neural network model.

The computing device can include an application and an application programming interface (API). The application can generate the instruction to be sent via the API. The application can be a camera application. The API can be a Neural Networks API (NNAPI). The computing device can be an Android device.

The subject matter described herein provides many advantages. For example, the simultaneous storage of several compiled models in a sequence in the SRAM of the hardware accelerator prevents a redundant removal from and reloading into the SRAM of same, previously loaded, compiled models of the same sequence that was invoked earlier as well. This avoidance of unnecessary clearing of the SRAM and reloading of compiled data in the SRAM can substantially reduce latency and improve processing speed. Furthermore, the storage of parameters in, and retrieval of those parameters from, the SRAM is significantly faster and more energy efficient than storage in and retrieval from the main memory-i.e., dynamic random access memory (DRAM). Additionally, in the unlikely but possible event that the compilation process fails, the compiler can prevent repeated compilation failures by pausing, for some time, identification of frequently invoked neural network models or the sequence thereof. Furthermore, the compiler can attempt compilation of frequently occurring neural network models only when each of those models has a size lower than a preset amount (e.g., <NUM> megabytes), which can increase (e.g., maximize) the number of models for which compiled models are simultaneously stored in the SRAM.

Other features and advantages of the subject matter described herein will be apparent from the description, the drawings, and the claims.

<FIG> illustrates a computing device <NUM> that has a hardware accelerator <NUM> that processes neural network models with a reduced (e.g., low) latency. The computing device <NUM> can be a mobile device, such as a phone, a phablet computer, a tablet computer, a laptop, and/or any other mobile device. While the computing device <NUM> is described as a mobile device, in some implementations, the computing device <NUM> can be a desktop computer or a cluster or network of computers. The hardware accelerator <NUM> refers to computer hardware that is specially made to perform some functions more efficiently than possible in software running on a general-purpose processor. For example, the hardware accelerator <NUM> can perform specified operations using special-purpose hardware, e.g., matrix multiplication, that allow the hardware accelerator to execute deep feed-forward neural networks such as convolutional neural networks (CNNs) much more efficiently than general-purpose processers. In order for the hardware accelerator <NUM> to execute a neural network model, the neural network model is compiled specifically for the accelerator <NUM>.

The hardware accelerator <NUM> can be a tensor processing unit (TPU). Although a TPU is described, in other implementations, the hardware accelerator <NUM> can be any other hardware accelerator <NUM>, such as a graphics processing unit (GPU), digital signal processor (DSP), field-programmable analog array (FPAA), sound card, network processor, cryptographic accelerator, artificial intelligence accelerator, physics processing unit (PPU), data compression accelerator, network on a chip, field programmable gate arrays (FPGA), application specific integrated circuit (ASIC), complex programmable logic device, and/or a system on chip.

The computing device <NUM> further includes "N" software applications <NUM>, such as a camera application, a social networking application, and any other software application. N can be any integer, such as <NUM>, <NUM>, <NUM>, <NUM> or any other integer. For these applications <NUM> to communicate with (e.g., provide inputs to and receive outputs from) the hardware accelerator <NUM>, the mobile computing device <NUM> further employs an application programming interface (API) <NUM> that outputs specific data structures to be processed in response to execution of the application <NUM>, a processor <NUM> to perform quantization (also referred to as quantizer <NUM>) on the specific data structures, and a processor <NUM> that implements a compiler configured to use the quantized data structures.

The API <NUM> enables communication between the applications <NUM> and the processor <NUM>. The API <NUM> can be a communication protocol (e.g., syntax, semantics and synchronization of communication and possible error recovery methods) that facilitates such communication. The API <NUM> can, for example, be a neural network API (NN API). The API <NUM> can allow the applications <NUM> to generate a data structure, which includes mathematical operations that constitute a neural network model to be processed in response to execution of the application <NUM>. For example, in response to acquisition of images by a camera application <NUM>, the API <NUM> can allow the camera application <NUM> to generate a data structure (e.g., TensorFlow data structure) <NUM> that indicates mathematical operations that constitute a face recognition model that is to be implemented on the accelerator. The data structure <NUM> can have parameter data (e.g., weights and input data for a neural network) represented as floating-point numbers having a preset number of bits (e.g., <NUM>-bit floating-point numbers).

The processor/quantizer <NUM> can receive, from the API <NUM> (or the application <NUM> in some implementations), the data-structure <NUM> and convert it into a smaller data structure (e.g., TensorFlowLite data structure) <NUM> that has the same parameter data (e.g., weights and input data for the neural network model, as in the data structure <NUM>) represented as fixed-point numbers having a lower number of preset number of bits (e.g., <NUM>-bit fixed-point numbers). Converting all the <NUM>-bit floating-point numbers in the data structure <NUM> to the nearest <NUM>-bit fixed-point numbers in the data structure <NUM> is referred to as quantization. Quantization advantageously makes the data structure smaller, and thus makes the operations performed by the hardware accelerator <NUM> on the data faster and less compute intensive. Further, although these lower bit (e.g., <NUM>-bit) representations can possibly be less precise than corresponding higher-bit (e.g., <NUM>-bit) representations in data structure <NUM>, the inference accuracy of the neural network is not significantly (i.e., not noticeably) affected. While quantization is described here as occurring during the API call, in some implementations the quantization can be performed at any time prior to the compilation of the quantized data, as described below. Furthermore, while quantization is described as an automated process in which the quantizer <NUM> automatically receives data and performs quantization based on that data, in some implementations at least some of the values in the data structure <NUM> have already been quantized, e.g., at an external system.

The processor <NUM> implements the compiler <NUM>. The compiler <NUM> compiles the quantized data structure <NUM> into a compiled data structure <NUM>, which is compatible with the hardware accelerator <NUM>. In addition to the quantized data structure <NUM>, the compiled data structure <NUM> can include machine-level code, which includes low level instructions that are to be executed by the hardware accelerator <NUM>. Generally, the compiler <NUM> can be run on any appropriate operating system e.g., an Android system or Debian-based Linux system.

Further, although quantization and compilation of quantized data is described, in some implementations the quantization is not performed because quantization may not be necessary (e.g. if the hardware accelerator <NUM> such as a GPU or a TPU is capable of floating point operations, then the compiler <NUM> can work directly on floating point models without requiring quantization).

The hardware accelerator <NUM> can perform various neural network computations to process the neural network model (e.g., face recognition model) based on the compiled data structure <NUM>. Every time the hardware accelerator <NUM> processes the neural network model (e.g., face recognition model), the hardware accelerator <NUM> needs to access the parameters of that neural network model. For such access, the hardware accelerator <NUM> includes one or more memories-specifically a static random access memory (SRAM)-that stores such data structure <NUM>, which includes parameters of the neural network model (e.g., face detection neural network model). The parameters are stored in the SRAM rather than a main memory of the computing device <NUM>, because the SRAM allows faster access to data stored therein by the hardware accelerator <NUM>, thereby increasing processing speed and energy efficiency and decreasing latency.

The SRAM on the hardware accelerator <NUM> has a limited amount of memory space (e.g., up to <NUM> megabytes) that can store the compiled data structure <NUM> (which includes parameters) of a model. When data structures with parameters are compiled individually, the compiler <NUM> gives each compiled data structure a unique hash (e.g., a <NUM>-bit number) for unique identification. When the neural network model is executed at runtime, the hardware accelerator compares that hash to the hash of previously compiled data structures stored in SRAM. If the tokens match, the hardware accelerator uses the stored previously compiled data structure, thereby avoiding the need to reload the compiled data structure <NUM> into the SRAM. If the tokens do not match, the hardware accelerator <NUM> wipes/erases the stored data structure (i.e. previously compiled data structure) and writes the compiled data structure <NUM> instead into the SRAM (i.e. clears and reloads the SRAM) so as to increase (e.g., maximize in some implementations) efficiency for using that limited memory space (e.g., up to <NUM> megabytes) in the SRAM. However, the hardware accelerator <NUM> is configured to store data (including parameters) corresponding to a single hash. Therefore, when all compiled data structures <NUM> have different hashes, after every individual compilation by the compiler <NUM>, the SRAM is reloaded with new compiled data <NUM>, which can cause latency.

The latency caused, and power requirement, due to this reloading of the SRAM after every individual compilation is significantly reduced by the compiler <NUM> by:.

Whenever the same frequently occurring sequence of a set of models is invoked again (e.g., in response to the user again clicking multiple images using the camera application <NUM> on the computing device <NUM>), the hardware accelerator <NUM> can quickly access the compiled data structures <NUM> directly from the SRAM, thereby avoiding the need for clearing (i.e. wiping/erasing) the SRAM and reloading it with the same compiled data structures. Avoidance of clearing and reloading of the SRAM can advantageously converse processing resources and power, and improve processing speed, thereby substantially decrease latency.

The simultaneous storage of compiled data structures <NUM> for all the models in the sequence in the SRAM of the hardware accelerator <NUM> is performed as follows. Every time the compiler <NUM> outputs compiled data structures to the hardware accelerator <NUM>, the compiler <NUM> computes and sends a separate unique hash (e.g., a <NUM>-bit number) for unique identification of the compiled data structure <NUM>. However when multiple compiled data structures <NUM> are determined for neural network models in a sequence, the compiler <NUM> allocates a single hash (e.g., a <NUM>-bit number) to identify all those compiled data structures. For example, the compiled data structures <NUM> for all models in the sequence are assigned the same hash, and a compiled data structure for any model that is not within that sequence would have a different hash. The compiler <NUM> computes a same hash for same model, and thus a same hash for a same sequence of models. Therefore, if the hash is same as the hash for models for which compiled data structures were previously stored in the SRAM, this indicates that the current sequence of models is same as a previous sequence for which compiled data structures were previously stored, thereby avoiding the need to clear the SRAM and then reload the SRAM with the same compiled data structures. Such prevention of clearing and then unnecessarily reloading the SRAM significantly reduces latency.

The amount of SRAM allocated to each model is fixed at compile-time, and is prioritized based on the order the data structures are compiled by the compiler. For example, when two models A and B within a sequence, where A is invoked before B, are compiled and corresponding data structures <NUM> are assigned the same hash, as much SRAM space as needed is first allocated to model A's data structure <NUM>, and if SRAM space remains after that, SRAM space is given to model B's data structure <NUM>. If some of the model data structure <NUM> cannot fit into the SRAM, then it is instead stored in and fetched from an external memory (e.g., main memory of the computing device <NUM>) at run time. If the entire model does not fit in the SRAM, the compiler <NUM> generates appropriate instructions for the accelerator <NUM> to fetch the data from the dynamic random access memory (DRAM). Maximal utilization of the SRAM in this manner can advantageously improve (e.g. maximize) processing speed.

In some implementations, if several models are compiled, some models may possibly not be allocated space in SRAM, so those models must load all data from an external memory (e.g., main memory of the computing device <NUM>). Loading from the external memory is slower than loading from the SRAM, but when running the models in a sequence of models that is frequently invoked, this can still be faster than swapping (i.e., clearing and reloading) the SRAM every time any model is run. As noted above, if the entire model does not fit in the SRAM, the compiler <NUM> generates appropriate instructions that the accelerator <NUM> executes to fetch the data from the dynamic random access memory (DRAM). Note this interaction between the compiler <NUM> and the accelerator <NUM> is different from the conventional central processing units (CPUs) that usually have hardware caches that automatically store the most frequently used data in the SRAM.

The compiler <NUM> can continue to compile data structures for neural network models while the hardware accelerator <NUM> simultaneously performs neural network computations of other one or more neural network models. Such simultaneous functionality prevents a pause of processing of neural network models during compilation activity by the compiler, thereby advantageously improving speed and reducing latency.

The compiler <NUM> can update the identification of the first set of models and the identification of the sequence (i.e., again identify the first set of models and again identify the sequence) after preset intervals of time (e.g., <NUM> second, <NUM> seconds, <NUM> minute, <NUM> minutes, <NUM> minutes, <NUM> minutes, <NUM> minutes, <NUM> hour, <NUM> hours, <NUM> days, or any other suitable time). Such update ensures that the SRAM is being used optimally to simultaneously store parameters of most relevant models (i.e. models currently or recently determined to be most frequently invoked, as opposed to such determination having been done a long time ago (e.g., more than a threshold time ago)). The compiler can abstain, for a preset time (e.g., <NUM> milliseconds), from the updating in response to a failure of the compilation of the first set of neural network models. Abstaining for a preset time can provide protection against transient failures that might otherwise trigger a continuous loop of compilation, leading to significantly increased power consumption. Suspending co-compilation for the preset time after a failure increases the chances that the set of active models would change and thus avoid a recurrence of the compilation failure. In some implementations, the preset time can have another value such as <NUM> second, <NUM> seconds, <NUM> seconds, <NUM> seconds, <NUM> seconds, <NUM> minute, or any other value that can avoid a recurrence of the compilation failure. Such abstention can conserve compilation resources, as immediate compilation using such resources may result in another compilation failure.

The compiling of the first set of neural network models can include determining, prior to the compiling, that each data structure to be compiled is compatible (e.g., in terms of size in megabytes) with the compiler <NUM>. For example, the compiler <NUM> may not compile data structures having a size greater than a preset amount (e.g., <NUM> megabytes), which ensures that parameters of that model can be simultaneously stored in the SRAM with parameters of other neural network models.

<FIG> illustrates a method performed by the compiler <NUM> to prevent redundant clearing and reloading of the SRAM. The compiler <NUM> can identify, at <NUM>, a frequently occurring sequence of neural network models. Frequently occurring sequence of models can be a set of neural network models that are invoked together in a particular sequence by an application <NUM> and processed by the hardware accelerator <NUM> more than a threshold number of times in a preset amount of time in the past. For example, users may click multiple images on the phone using the camera application <NUM> thereon frequently (e.g. more than a threshold number of times), and in such case the frequently occurring models and corresponding sequence can be: (a) first a face detection model to detect faces in each image, (b) then an orientation detection model to detect orientations in each image, (c) then a blur-detection model to detect a blur in each image, (d) then another neural network model to suggest the best image based on the detections of (a), (b) and (c). The threshold number of times for which a sequence needs to be repeated for the sequence to qualify as a frequently occurring sequence can be <NUM> times, and in other implementations <NUM> times, <NUM> times, <NUM> times, <NUM> times, <NUM> times, or any other integer (which is greater than <NUM>) number of times. The preset amount of time in the past that may be considered for this determination may be since the time the hardware accelerator <NUM> is deployed to perform neural network computations for the corresponding application <NUM>. In another implementation, such preset amount of time in the past may be <NUM> minute, <NUM> minutes, <NUM> minutes, <NUM> minutes, <NUM> minutes, <NUM> hour, <NUM> hours, <NUM> days, or any other suitable time. In some implementations, the preset amount of time in the past to be used for determining frequently occurring sequence may be dynamically computed based on usage by a user of one or more applications <NUM>.

The compiler <NUM> can receive, at <NUM> for each neural network model within the frequently occurring sequence of neural network models, data structure including model parameters. In some examples, the data structure received by the compiler <NUM> can have <NUM>-bit fixed-point numbers in a data structure <NUM>, which can be obtained by quantizing, for example, a data structure <NUM> that has <NUM>-bit floating-point numbers. The model parameters for each neural network model can include weights and data for that neural network model.

The compiler <NUM> can compile, at <NUM>, data structure for each neural network model in the sequence to generate compiled data structure <NUM> for each neural network model. Such compilation can be performed in an order of the sequence. For example, a first model in the sequence is compiled first, a second model in the sequence is compiled second, a third model in the sequence is compiled third, and so on until the last model in the sequence is compiled last. Individual models can be compiled using any appropriate technique (including any conventional technique) that performs compilation, by generating machine level code to be accessed and executed by the hardware accelerator <NUM>.

The compiler <NUM> can assign, at <NUM>, a hash to compiled data structure <NUM> for each neural network model in set of neural network models. The hash can be a unique <NUM>-bit number for uniquely identifying the compiled data structure <NUM>. While the hash is described as a <NUM>-bit number, in other implementations, it can have any other number of bits. Generally, a hash function receives, as input, the compiled data structure and outputs a hash (e.g., <NUM>-bit number) that can be used, for example, as an index in a hash table. The hash can also be referred to as a hash value, a hash code, or a digest in various implementations. The hash function can be MD5, SHA-<NUM>, CRC32, any other one or more hash functions, and/or any combination thereof.

The compiler can output, at <NUM>, compiled data structures, according to the sequence, and the hash to the hardware accelerator <NUM> for storage in the SRAM (i.e., reload the SRAM) when the hash (e.g., a first <NUM>-bit number) is different from another hash (e.g., a second <NUM>-bit number) of another previously identified sequence in the SRAM. Such reloading of the SRAM is unnecessary when the two hashes are the same, and the compiler <NUM> thus prevents clearing and reloading of the SRAM in such case. In other words, clearing and reloading of the SRAM is done if a hash of all models in a first sequence is different from another hash of all models in a previous sequence (which indicates that the two sequences are different). If the two hashes are the same, this indicates that the two sequences are the same, and accordingly the SRAM does not need to be cleared and reloaded, thereby reducing latency.

<FIG> illustrates two different sequences of models, for which data structures are compiled, along with corresponding hashes, which are compared to determine whether the SRAM needs to be cleared and reloaded with compiled data. Each model in a frequently occurring sequence is assigned a same hash (<NUM>-bit unique number). For example, each model of the first sequence shown is assigned a first hash (computed by the compiler), and each model of the second sequence is assigned a second hash (computed by the compiler). The compiler <NUM> computes a same hash for same model, and thus a same has for a same sequence of models. Therefore, if the second hash is same as the first hash, this indicates that the second sequence is same as the first sequence, and thus the compiled data structures <NUM> for the second sequence of models do not need to be reloaded in the SRAM of the hardware accelerator <NUM>, thereby reducing latency. In such case, the compiler <NUM> thus prevents clearing and reloading of the SRAM.

<FIG> illustrates steps performed by the hardware accelerator <NUM> to store compiled data in the SRAM and act in response to a determination of whether the SRAM needs to be cleared and reloaded. The hardware accelerator <NUM> can allocate, at <NUM>, SRAM space for storage of compiled data structure <NUM> and first hash of <FIG>. If second hash is different from first hash, the hardware accelerator <NUM> can, at <NUM>, erase the SRAM space for compiled data structure <NUM> and replace it with a result of compilation of the second sequence. If second hash is same as the first hash, the hardware accelerator <NUM> can, at <NUM>, avoid erasing compiled data structure <NUM> from the SRAM, which advantageously reduces latency.

<FIG> illustrates aspects of the hardware accelerator <NUM> that includes the SRAM <NUM> for storing compiled data structures <NUM>, which include parameters for processing neural network models. The hardware accelerator <NUM> communicates with the application <NUM> via the application programming interface (API) <NUM>. The API <NUM> sends data to the hardware accelerator <NUM> via the compiler, and can output data from the hardware accelerator <NUM> (e.g., via a decompiler, which is not shown). For example, the API <NUM> can send to the compiler <NUM> specific data structures to be processed in response to execution of the application <NUM>. The API <NUM> may need to send data to the compiler via a quantizer, depending on the configuration of the hardware accelerator <NUM>, as described by <FIG>.

The hardware accelerator <NUM> is configured to perform neural network computations in response to instructions and input data received from applications running on a computing device <NUM>. The accelerator <NUM> can have a controller <NUM> and multiple separate computing units <NUM>. While eight computing units <NUM> are shown, in alternate implementations the hardware accelerator <NUM> can have any other number of computing units <NUM>, such as any number between two and sixteen. Each computing unit <NUM> can have at least one programmable processor <NUM> and at least one memory <NUM>. In some implementations, the parameters for processing neural network models, as indicated by the compiled data structures <NUM>, may be distributed across one or more (e.g., all) of the memories <NUM>.

The computing units <NUM> can accelerate machine learning inference workloads of a neural network layer. Each computing unit <NUM> is self-contained and can independently execute computations required by a given layer of a multi-layer neural network. The hardware accelerator <NUM> can perform the computation of a neural network layer by distributing tensor computations across the computing units <NUM>. The computation process performed within a neural network layer may include a multiplication of an input tensor including input activations with a parameter tensor including weights. The computation can include multiplying an input activation with a weight on one or more cycles and performing an accumulation of a products over many cycles. The term tensor as used herein refers to a multi-dimensional geometric object, which can be a matrix or a data array.

Each computing unit <NUM> can implement a software algorithm to perform tensor computations by processing a nested loop to traverse an N-dimensional tensor (where N can be any integer). In one example computational process, each loop can be responsible for traversing a particular dimension of the N-dimensional tensor. For a given tensor construct, a computing unit <NUM> can require access to an element of a particular tensor to execute a plurality of dot product computations associated with the tensor. Computation occurs when an input activation is multiplied with a parameter or weight. Tensor computations end when multiplication results are written to an output bus, which serially connects the computing units <NUM> and over which data is passed between the computing units, and stored in memory.

The hardware accelerator <NUM> can support specific types of data structures (e.g., structures <NUM> with <NUM>-bit floating point numbers) that are quantized (e.g., to obtain structures <NUM> with <NUM>-bit floating point numbers) and then compiled (e.g., to obtain compiled structures <NUM>) specifically for the hardware accelerator <NUM>.

The hardware accelerator <NUM> can perform various neural network computations to process the neural network model (e.g., face recognition model) based on the compiled data structure <NUM> generated by the compiler <NUM>. Every time the hardware accelerator <NUM> processes the neural network model (e.g., face recognition model), the hardware accelerator <NUM> needs to access the parameters within the compiled data structure <NUM> of that neural network model. To store the data received from the compiler <NUM> and the API <NUM> (including the parameters in the compiled data structure <NUM>), the hardware accelerator <NUM> further includes an instruction memory <NUM>, the SRAM <NUM>, and a data memory <NUM>.

The SRAM <NUM> has a limited amount of memory space (e.g., up to <NUM> megabytes) that can store compiled data structures <NUM> of a model. To optimally use the SRAM <NUM>, the compiler <NUM> can identify a sequence in which frequently occurring neural network models are executed, compile all the data structures in that sequence together to generate compiled data structures <NUM> that are assigned the same identification (e.g. hash), and output the compiled data structures <NUM> to the SRAM <NUM> in a selective manner (specifically, only when a different sequence of models is invoked). This prevents redundant reloading of the SRAM <NUM>.

The amount of SRAM <NUM> allocated to each model is fixed at compile-time, and is prioritized based on the order the data structures are compiled by the compiler. For example, if two models A and B are compiled with the same hash for the compiled models, as much SRAM <NUM> space as needed is first allocated to model A's data structure, and if SRAM <NUM> space remains after that, SRAM <NUM> space is given to model B's data structure. If the data structure of one of the models A or B cannot fit into the SRAM <NUM>, then it is instead stored in and fetched from an external memory (e.g., main memory of the computing device <NUM>) at run time.

If several models are compiled, some models may possibly not be allocated space in SRAM <NUM>, so those models must load all data from external memory. Loading from external loading is slower than loading from the SRAM <NUM>, but when running the models in a frequent sequence, this could still be faster than swapping the SRAM <NUM> every time any model is run.

Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output(s). The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), or a GPGPU (General purpose graphics processing unit).

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular implementations of particular inventions. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

Moreover, the separation of various system modules and components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Claim 1:
A method performed by a compiler (<NUM>) and a hardware accelerator (<NUM>), the method comprising:
identifying, by the compiler, a first set of neural network models that have been executed on a hardware accelerator of a computing device (<NUM>) more than a threshold number of times in a preset amount of time in the past;
identifying, by the compiler, a sequence in which the first set of neural network models are executed on the hardware accelerator;
compiling, by the compiler, each neural network model of the first set of neural network models for execution by the hardware accelerator to generate a respective compiled data structure (<NUM>) for the neural network model; and
outputting, by the compiler, for each neural network model of the first set of neural network models, the respective compiled data structure to the hardware accelerator for storage according to the sequence in one or more memories of the hardware accelerator, wherein the outputting comprises:
determining that a portion of a compiled data structure in the respective compiled data structures cannot be allocated in the one or more memories of the hardware accelerator;
in response, outputting the compiled data structure for storage in another memory external to the one or more memories of the hardware accelerator;
generating instructions that, once executed by the hardware accelerator, cause fetching the compiled data structure stored in the other memory; and
storing the rest of the respective compiled data structures other than the compiled data structure stored in the other memory according to the sequence in the one or more memories;
executing, by the hardware accelerator, the sequence of the first set of neural network models, wherein the executing comprises executing the generated instructions;
determining, by the compiler, a next sequence of neural network models to be executed and
if the next sequence of neural network models differs from the sequence of the first set of neural network models, erasing the rest of the compiled data structures stored in the first memory; and
if the next sequence of neural network models is the sequence of the first set of neural network models, executing the compiled data structures in the one or more memories again on the hardware accelerator.