Patent Description:
The present disclosure relates to memories, and, more particularly, to an apparatus and a method that can dynamically load a neural network inference stored in a memory and manage power supplied to the memory.

Neural networks (NNs) can learn from various examples of a certain task during a process called training. After learning, the task can be performed on new data during a process called inference. An NN inference can have a huge amount of weights and activations and have to be stored in a sufficiently large memory, such as a dynamic random access memory (DRAM). During the execution of the entire NN inference, the DRAM, when powered on, will consume power.

<NPL>, pertains to an open, generic, and customizable deep learning accelerator, the Versatile Tensor Accelerator (VTA), for cross-stack deep learning research and optimization.

<NPL>, pertains to hardware implementations of convolutional neural networks, ConvNets, deployed in low-power embedded systems and Internet of Everything, IoE, platforms.

<NPL> discloses reducing the energy consumption and increasing the performance of deep neural network inference systems by using approximate DRAM devices.

The present invention is defined by the subject matter of the appended claims.

Aspects of the disclosure provide an apparatus for executing a program. The apparatus can include an executor and a dynamic loading agent. The executor can be coupled to a second memory, and configured to execute a portion of the program loaded on the second memory from a first memory that is configured to store the program, and to generate a signal based on a progress of the execution of the program. The dynamic loading agent can be coupled to the executor, the first memory and the second memory, and configured to load a next portion of the program stored in the first memory to the second memory and to manage power supplied to the first memory based on the signal from the executor and an executing scheme stored in the second memory.

For example, the dynamic loading agent can manage the power supplied to the first memory by powering on/off the first memory, configuring an operation mode of the first memory, or scaling a voltage and/or a frequency applied to the first memory. As another example, the executing scheme can include a script, a rule, or a model. For example, the rule can generate a script for a certain layer of the program based on input tensors from a previous layer of the program. As yet another example, the signal can include an operation, a node identification (ID), a tensor, a kernel, and/or time of the program.

According to the invention, the second memory is a tightly-coupled memory (TCM) or a static random access memory (SRAM). According to the invention, the first memory is a dynamic random access memory (DRAM), an on-bus SRAM, or a serial flash memory. For example, the executor can be a central processing unit (CPU), and the dynamic loading agent can be a microcontroller unit (MCU). For another example, the executor can be a CPU, and the dynamic loading agent can be CPU exception. For yet another example, the executor can be a deep learning accelerator (DLA), and the dynamic loading agent can be an MCU.

In an embodiment, the dynamic loading agent and the executor can be included on a single chip. In another embodiment, the apparatus can further include a direct memory access (DMA) controller coupled to the dynamic loading agent, the first memory, and the second memory, and the dynamic loading agent can be further configured to instruct the DMA controller to load the next portion of the program stored in the first memory to the second memory.

Aspects of the disclosure also provide a method for executing a program. The method can include loading a portion of the program from a first memory that stores the program to a second memory. The method can further include executing the portion of the program loaded on the second memory, and generating a signal based on a progress of the execution of the program. The method can also include loading a next portion of the program stored in the first memory to the second memory and managing power supplied to the first memory based on the signal and an executing scheme stored in the second memory.

Various embodiments of this disclosure that are proposed as examples will be described in detail with reference to the following figures, wherein like numerals reference like elements, and wherein:.

Neural networks (NNs) can be used to extract meaningful information or features out of a huge amount of data. Without being specifically programmed, the NNs can learn from various examples of a certain task during a process called training. After learning, the task can be performed on new data during a process called inference. During training, a best set of weights that optimize the accuracy of NN inference can be estimated. An NN inference, which is a type of programs, can include a huge amount of weights and activations, and, accordingly, be too large to be stored in, for example, a tightly-coupled memory (TCM), which can have a limited bandwidth. For example, the NN inference can be stored in a dynamic random access memory (DRAM), and the NN inference will be loaded from the DRAM to the TCM portion by portion for an inference executor to execute. During the loading and execution of the entire NN inference, the DRAM is always powered on, which can consume a great amount of power.

According to some aspects of the disclosure, a dynamic loading agent and an executing scheme, e.g., an inference executing scheme, can be introduced. In an embodiment, the inference executing scheme can be used to tell the dynamic loading agent about when and how to load a portion of the program, e.g., the NN inference, from the DRAM to the TCM and manage power supplied to the DRAM. For example, when the dynamic loading agent looks up the inference executing scheme and learns that a portion of the NN inference needs to be loaded from the DRAM to the TCM, the dynamic loading agent can turn on the DRAM, load the portion of the NN inference from the DRAM to the TCM, and turn off the DRAM after the portion of the NN inference is loaded to the TCM. According to some aspects of the disclosure, the DRAM will be powered off unless the dynamic loading agent for a great portion of the time when the NN inference is executed.

Smart devices, wearables, and other Internet of things (IoT) devices can include always-on continuous sensing apps and a variety of sensors, such as accelerometers, microphones, ambient light sensors, biometric sensors, etc., and use data gathered by these sensors to enable gesture or voice based control, monitor user activity and safety, and provide interpretation of a user's context. These IoT devices can perform analysis and interpretation of the gathered data on a main processor, and, accordingly, the main processor needs to be powered on frequently to address these continuous sensing tasks. These IoT devices may also utilize a dedicated low-power microprocessor, referred to as a sensor hub, to process the data, to offload the vast processing of the data from the main processor and reduce power consumption.

Tightly-coupled memories (TCMs) are one of a variety of on-chip memories that can be typically embedded in a low power processor to accelerate the data processing in the processor. TCMs are area expensive, so it is not cost-effective to embed a TCM of a large size in a processor. A TCM is implemented very close to a central processing unit (CPU) with access latency and size close to those of an L1 cache.

Neural networks (NNs), which are loosely modeled on the biology of human brain, can be used to extract meaningful information or features out of a huge amount of data. Without being specifically programmed, the NNs can learn from various examples of a certain task during a process called training. After learning, the task can be performed on new data during a process called inference. During training, a best set of weights that optimize the accuracy of NN inference can be estimated.

Artificial neural networks (ANNs) are one type of neural networks in deep learning (DNNs). An ANN can include multiple interconnected neurons (or nodes or units) arranged on various layers. Nodes from adjacent layers have connections or edges therebetween, and the connections have weights associated therewith. Each node can receive input(s) from some other nodes connected thereto at an adjacent layer, or outside world, and compute an output. Each input can have an associated weight, which is assigned based on its relative importance to other inputs. The node weighs the inputs with their associated weights, sums up the weighted inputs and a trainable constant value, referred to as a bias, and applies a non-linear activation function to the sum, so as to compute an output. The activation function can be a Sigmoid function, a hyperbolic tangent (tanh) function or a rectified linear unit (ReLU). The nodes of the ANN are categorized into input nodes, hidden nodes and output nodes, which are arranged in different layers (or called operations or operators) including an input layer, a hidden layer, and an output layer, respectively. The input nodes can receive data from the outside world, and pass on the data, without any computation, to the hidden nodes. The hidden nodes, as the name implies, have no direct connection with the outside world. The hidden nodes compute the data transferred from the input nodes, and transfer the computation results to the output nodes. The output nodes can transfer the computation results to the outside world.

Convolutional neural networks (CNNs) are one of the most popular NNs in deep leaning. Compared with an ANN, which treats input pixels of an image which are far apart and close together on exactly the same footing, and has its layers fully connected, a CNN takes into consideration the spatial structure of the image by using three basic ideas, including local receptive fields, shared weights, and pooling. Each neuron in the first hidden layer can be connected to a local receptive field of the input neurons corresponding to some input pixels of an image. The local receptive field can slide across the entire image by a certain stride length (e.g., one or two pixels) at a time, and build up the first hidden layer. The hidden neutrons in thus built first hidden layer can have the same weights and biases, and detect exactly the same feature just at different locations in the image. The map from the input layer to the hidden layer can be called a feature map. Weights and biases defining the feature map are called shared weighted and shared bias, respectively, which in turn define a kernel or filter. The shared weights and biases can greatly reduce the number of parameters involved in a convolutional network.

A CNN can be composed of a variety of different layers, including convolutional layers, pooling layers and fully-connected layers. Each of the layers can be defined by a weight matrix learned via a one-time training process that is executed before the CNN is ready for inference. Each of the convolutional layers can have kernels (or filters), and convolutes inputs with the kernels to get the outputs. For example, an operation (also called a layer or an operator), e.g., 2D convolution or addition, takes one or more input tensors and produces a single output. The pooling layers can be usually used immediately after the convolutional layers, and pool the convolutional layer to reduce the number of parameters of the input tensors. Each neuron in the pooling layer can summary a region of <NUM> × <NUM> neurons, for example, in the previous convolutional layer. For example, a pooling neutron can only output the maximum or average activation in the <NUM> × <NUM> input region. The final convolutional or pooling layers are flattened and output to the fully-connected layers, which form the last few layers in the CNN.

An NN inference may include a huge amount of weights and activations. As a model, e.g., the NN inference, becomes larger, it becomes more difficult for the entire NN inference to be deployed in the TCM. In practice, the NN inference may be stored in an external memory, e.g., a DRAM, and a portion of the NN inference will be moved to the TCM at a time for execution.

<FIG> shows a functional block of an exemplary apparatus <NUM> for executing a program, e.g., an NN inference, according to some embodiments of the disclosure. For example, the apparatus <NUM> can be a smartphone, a wearable, or any IoT device. In an embodiment, the apparatus <NUM> can include a controller <NUM> and an executor, e.g., an inference executor <NUM>. For example, the inference executor <NUM> can be a central processing unit (CPU) or a microprocessor unit (MCU). The MCU is an attractive platform for building smartphones due to their low cost, wide availability, and modest power usage. In an embodiment, the controller <NUM> can be coupled between a first memory <NUM> and a second memory <NUM>, and the inference executor <NUM> can be coupled to the second memory <NUM>. For example, the first memory <NUM> can be a dynamic random access memory (DRAM), an on-bus static random access memory (SRAM), or a serial flash memory. As another example, the second memory <NUM> can be a cache, and the controller <NUM> can be a cache controller accordingly.

In an embodiment, an NN inference <NUM> can be stored in the DRAM <NUM>. The cache controller <NUM> can load a portion <NUM> of the NN inference <NUM> stored in the DRAM <NUM> to the cache <NUM> at a time. The inference executor <NUM> can execute the portion <NUM> of the NN inference loaded on the cache <NUM>. For example, when the cache <NUM> is hit, the inference executor <NUM> can execute the portion <NUM> of the NN inference <NUM> loaded on the cache <NUM>. As another example, when the cache <NUM> is missed, which can indicate that the inference executor <NUM> is going to execute a next portion <NUM> of the NN inference <NUM>, the cache controller <NUM> can then access the DRAM <NUM> and load the next portion <NUM> of the NN inference <NUM> stored in the DRAM <NUM> to the cache <NUM>, and the inference executor <NUM> can then execute the next portion <NUM> of the NN inference <NUM> loaded on the cache <NUM>.

<FIG> shows a timing diagram of a current flowing through the DRAM <NUM> of <FIG> during the execution of the entire NN inference <NUM> according to some embodiments of the disclosure. As shown in <FIG>, when the cache <NUM> is missed, the cache controller <NUM> needs to access the DRAM <NUM>. For example, the cache controller <NUM> can power on the DRAM <NUM>, and then load the next portion <NUM> of the NN inference <NUM> to the cache <NUM>. Once the DRAM <NUM> is powered on, it will not be powered off until the entire NN inference <NUM> is executed completely. Accordingly, the DRAM <NUM> can be always on during the loading and execution of the entire NN inference <NUM>. <FIG> shows that the average current flowing through the DRAM <NUM> is approximately <NUM> mA.

<FIG> shows a functional block diagram of another apparatus <NUM> for executing an NN inference according to some embodiments of the disclosure. The apparatus <NUM> can also be a smartphone, a wearable, or any IoT device. The apparatus <NUM> can include a dynamic loading agent <NUM> and an inference executor <NUM> coupled to the dynamic loading agent <NUM>. For example, the dynamic loading agent <NUM> and the inference executor <NUM> can be hardware or software. In an embodiment, the dynamic loading agent <NUM> and the inference executor <NUM> can be included on a single chip. In another embodiment, the dynamic loading agent <NUM> and the inference executor <NUM> can be two distinct chips. For example, the inference executor <NUM> can be a CPU, and the dynamic loading agent <NUM> can be an MCU. As another example, the inference executor <NUM> can be a deep learning accelerator (DLA), and the dynamic loading agent <NUM> can be an MCU. As yet another example, the inference executor <NUM> can be a CPU, and the dynamic loading agent <NUM> can be CPU exception. As still another example, the inference executor <NUM> can be a CPU that is set with an interrupt descriptor table, and the dynamic loading agent <NUM> can perform a specific exception handler function listed in the interrupt descriptor table when an exception occurs and the CPU interrupts its current works. In an embodiment, the inference executor <NUM> can be coupled to a second memory <NUM>. For example, the second memory <NUM> can be a TCM or a static random access memory (SRAM). In another embodiment, the dynamic loading agent <NUM> can be coupled between the first memory <NUM> and the second memory <NUM>. In yet another embodiment, the dynamic loading agent <NUM> can also be coupled to a power controller <NUM>, and the power controller <NUM> can be coupled to the DRAM <NUM>.

The TCM <NUM> can be configured to store the portion <NUM> of the NN inference <NUM> loaded from the DRAM <NUM>. The inference executor <NUM> can execute the portion <NUM> of the NN inference <NUM> loaded on the TCM <NUM>, and generate a signal based on a progress of the execution of the NN inference <NUM>. For example, the signal can include a node identification (ID) of the NN inference <NUM>. In an embodiment, an inference executing scheme <NUM> can also be stored in the TCM <NUM>. The dynamic loading agent <NUM> can load the next portion <NUM> of the NN inference <NUM> from the DRAM <NUM> to the TCM <NUM> and manage, via the power controller <NUM>, power supplied to the DRAM <NUM> based on the signal and the inference executing scheme <NUM>. For example, the dynamic loading agent <NUM> can manage the power supplied to the DRAM <NUM> by powering on/off the DRAM <NUM>. In an embodiment, the dynamic loading agent <NUM> can be a script player, and the inference executing scheme <NUM> can be a script, which can recite the following contents:.

<FIG> shows a timing diagram of a current flowing through the DRAM <NUM> of <FIG> during the execution of the entire NN inference <NUM> with the above script <NUM> stored in the TCM <NUM> according to some embodiments of the disclosure. Initially, the inference executor <NUM> needs a portion of an NN inference to execute, and generates a signal, e.g., Node <NUM>. Upon receiving the signal of Node <NUM>, the dynamic loading agent <NUM> can look up the script <NUM> stored in the TCM <NUM>, and power on the DRAM <NUM> based on the script content "Node <NUM>: DRAM powered on," load data <NUM>-<NUM> stored in the DRAM <NUM> to the TCM <NUM> based on the script content "Node <NUM>: data <NUM>-<NUM> moved to TCM," and power off the DRAM <NUM> based on the script content "Node <NUM>: DRAM powered off. " Accordingly, the current flowing through the DRAM <NUM> can have a spike at node <NUM>, as indicated by "NODE <NUM>" in <FIG>, since the DRAM <NUM> is powered on, and become very small after node <NUM>, since the DRAM <NUM> is powered off. Then, the inference executor <NUM> can execute the portion <NUM> of the NN inference <NUM> loaded on the TCM, e.g., the data <NUM>-<NUM>.

When executing the portion <NUM> of the NN inference <NUM> loaded on the TCM <NUM>, the inference executor <NUM> can generate the signals of node <NUM> to node <NUM> sequentially based on a progress of the execution of the NN inference <NUM>. As the script <NUM> does not recite any content indexed by node <NUM> to node <NUM>, the dynamic loading agent <NUM> does nothing when receiving the signals of node <NUM> to node <NUM>, and the DRAM <NUM> can be kept powered off, as indicated by "NODEs <NUM>-<NUM>" in <FIG>.

Then, the inference executor <NUM> can generate the signal of Node <NUM>, as the inference executor <NUM> is executing at a node <NUM> and needs the next portion <NUM> of the NN inference <NUM> to execute, for example. Upon receiving the signal of Node <NUM>, the dynamic loading agent <NUM> can look up the script <NUM>, and power on the DRAM <NUM> based on the script content "Node <NUM>: DRAM powered on," load data <NUM> and <NUM> stored in the DRAM <NUM> to the TCM <NUM> based on the script content "Node <NUM>: data <NUM> and <NUM> moved to TCM," and power off the DRAM <NUM> based on the script content "Node <NUM>: DRAM powered off" sequentially. Accordingly, the current flowing through the DRAM <NUM> can have a spike at node <NUM>, as indicated by "NODE <NUM>" in <FIG>, since the DRAM <NUM> is powered on again, and become very small after node <NUM>, since the DRAM <NUM> is powered off again. Then, the inference executor <NUM> can execute the next portion <NUM> of the NN inference <NUM> loaded on the TCM <NUM>, e.g., the data <NUM> and <NUM>.

Then, the inference executor <NUM> can generate the signal of Node <NUM>, as the inference executor <NUM> is executing at a node <NUM> and needs another next portion of the NN inference <NUM> to execute, for example. Upon receiving the signal of Node <NUM>, the dynamic loading agent <NUM> can look up the script <NUM>, and power on the DRAM <NUM><NUM> after the reception of the signal of Node <NUM> based on the script content "<NUM> after Node <NUM>: DRAM powered on. " Accordingly, the current flowing through the DRAM <NUM> can have a spike at a position <NUM> after node <NUM>, as indicated by "<NUM> after node <NUM>" in <FIG>.

Then, the inference executor <NUM> can generate the signal of Node <NUM>, as the inference executor <NUM> is executing at a node <NUM> and needs yet another next portion of the NN inference <NUM> to execute, for example. Upon receiving the signal of Node <NUM>, the dynamic loading agent <NUM> can look up the script <NUM>, and load data <NUM> stored in the DRAM <NUM> to the TCM <NUM> based on the script content "Node <NUM>: data <NUM> moved to TCM," and power off the DRAM <NUM> based on the script content "Node <NUM>: DRAM powered off" sequentially. In this scenario, the DRAM <NUM> can be powered on already before the inference executor <NUM> generates the signal of Node <NUM>, and the dynamic loading agent <NUM> can load the data <NUM> from the DRAM <NUM> to the TCM <NUM> right after the reception of the signal of Node <NUM>. In an embodiment in which the dynamic loading agent <NUM> and the inference executor <NUM> are two distinct chips, the dynamic loading agent <NUM>, after receiving the signal of Node <NUM>, can power on the DRAM <NUM> while the inference executor <NUM> is executing at the node <NUM>.

Then, the inference executor <NUM> can generate the signal of Node <NUM>, as the inference executor <NUM> is executing at a node <NUM> and an operation associated with node <NUM> needs to be performed, for example. Upon receiving the signal of Node <NUM>, the dynamic loading agent <NUM> can look up the script <NUM>, and power on the DRAM <NUM> based on the script content "Node <NUM>: DRAM powered on," pass setting or configurations to a convolution operator (e.g., indicated by a channel number, a data address, a buffer size etc.) and do some special operations (e.g., fine-grained operator, including loading small amount of data and executing fine-grained convolution operations) based on the script content "Node <NUM> & Convolution OP: do special op," and power off the DRAM <NUM> based on the script content "Node <NUM>: DRAM powered off" sequentially.

Compared to the average current of <NUM> mA flowing through the DRAM <NUM> of <FIG>, the average current flowing through the DRAM <NUM> of <FIG> is approximately <NUM> mA, as the DRAM <NUM> is powered off for a great portion of the time when the NN inference <NUM> is executed.

In an embodiment, the signal can further include time of the NN inference <NUM>. For example, the signal of Node <NUM> can further include the time of <NUM>. In another embodiment, the signal can further include operations of the NN inference <NUM>. For example, the signal of Node <NUM> can further include a convolution operation, which can include a convolution channel number. In yet another embodiment, the signal can further include tensors of the NN inference <NUM>. For example, if there are some weighted data need to be processed, the signal can include a "tensor," and the dynamic loading agent <NUM> can, after receiving the tensor, perform the processing of the weighted data. In still another embodiment, the signal can further include kernels of the NN inference <NUM>. For example, the signal can include a kernel and a tensor, and the dynamic loading agent <NUM> can, after receiving the kernel and the tensor, execute the tensor by using the kernel.

In an embodiment, the dynamic loading agent <NUM> can further manage the power supplied to the DRAM <NUM> by configuring an operation mode of the DRAM <NUM>. For example, the dynamic loading agent <NUM> can configure the DRAM <NUM> to operate at a high-performance mode by scaling the data rate and row cycle time of the DRAM <NUM>. In another embodiment, the dynamic loading agent <NUM> can further manage the power supplied to the DRAM <NUM> by dynamically scaling a voltage and/or a frequency applied to the DRAM <NUM> (DVFS), as the power consumption is proportional to V<NUM> × f. For example, the dynamic loading agent <NUM> can manage the power supplied to the DRAM <NUM> by reducing the voltage applied to the DRAM <NUM>. Although reducing voltage may increase the propagation delay of signals, which can cause errors when using unmodified timing parameters, e.g., time required for the data to be reliably sensed and amplified in the row buffer, neutral networks are universal approximators and can still work very well if a system that is used to model the neutral networks has a high tolerance to errors.

In an embodiment, the inference executing scheme <NUM> can further include a rule. In an embodiment, the rule <NUM> can generate a script for a certain layer of the NN inference <NUM> based on the input tensors from a previous layer of the NN inference <NUM>. For example, as shown in <FIG>, which is a timing diagram of a current flowing through the DRAM <NUM> of <FIG> when the inference executing scheme <NUM> is a rule, the rule <NUM> can tell the dynamic loading agent <NUM> when (e.g., which layer or which node) and how to manage the power supplied to the DRAM <NUM> and load the data stored in the DRAM <NUM> to the TCM <NUM>. As another example, the dynamic loading agent <NUM> can be a policy controller that is controlled by the rule <NUM>. In another embodiment, the inference executing scheme <NUM> can further include a model. For example, the model <NUM> can be well trained to generate a rule or even a script for the dynamic loading agent <NUM> to look up. As another example, the dynamic loading agent <NUM> can be an NN executor of model for policy. In yet another embodiment, the inference executing scheme <NUM> can be man-made, such as the rule or script <NUM>, or created by offline optimizers or online/runtime optimizers. Such optimizers can analyze a model and find out one of the best rules. For example, the online/runtime optimizers can create the rule <NUM> during runtime. As another example, the offline optimizers can create the rule <NUM> first, and the dynamic loading agent <NUM> can apply the rule <NUM> to know when and how to load a portion of the NN inference <NUM> from the DRAM <NUM> to the TCM <NUM> and manage power supplied to the DRAM <NUM>.

<FIG> shows a functional block diagram of yet another exemplary apparatus <NUM> for executing an NN inference according to some embodiments of the disclosure. The apparatus <NUM> differs from the apparatus <NUM> of <FIG> in that the apparatus <NUM> can further include a direct memory access (DMA) controller <NUM> that is coupled among the DRAM <NUM>, the TCM <NUM> and the dynamic loading agent <NUM>. In an embodiment, the dynamic loading agent <NUM> can instruct the DMA controller <NUM> to load the portion <NUM> and the next portion <NUM> of the NN inference <NUM> directly to the TCM <NUM>, without the involvement of the dynamic loading agent <NUM>. For example, the dynamic loading agent <NUM>, after receiving the signal from the inference executor <NUM>, looking up the inference executing scheme <NUM> stored in the TCM <NUM> and learning that some data stored in the DRAM <NUM> need to be loaded to the TCM <NUM>, can issue a command to the DMA controller <NUM> by sending some information to the DMA controller <NUM>, including a read command, the number of words to be read, a starting location in the DRAM <NUM> to read from, and the address of the TCM <NUM>, and the DMA controller <NUM> can then load the data from the DRAM <NUM> directly to the TCM <NUM> based on the information without going through the dynamic loading agent <NUM>. After the loading is complete, the DMA controller <NUM> can send an interrupt signal to the dynamic loading agent <NUM> to inform that the DMA controller <NUM> has finished using the system bus. Compared with the apparatus <NUM>, the apparatus <NUM> can have better speed performance. Accordingly, the DRAM <NUM> of <FIG> can consume less power than the DRAM <NUM> of <FIG> does.

<FIG> shows a functional block diagram of yet another exemplary apparatus <NUM> for executing an NN inference according to some embodiments of the disclosure. The apparatus <NUM> differs from the apparatus <NUM> of <FIG> in that the apparatus <NUM> can further include a cache controller <NUM> that is coupled to the dynamic loading agent <NUM>. In an embodiment, the cache controller <NUM> can be further coupled between the DRAM <NUM> and a cache <NUM>. In another embodiment, the cache <NUM> can be further coupled to the inference executor <NUM>. In yet another embodiment, the cache controller <NUM> can load another portion <NUM> of the NN inference <NUM> stored in the DRAM <NUM> to the cache <NUM>, and the inference executor <NUM> can execute the another portion <NUM> of the NN inference <NUM> loaded from the DRAM <NUM> on the cache <NUM>.

In an embodiment, the apparatuses <NUM>, <NUM>, <NUM> and <NUM> can further be configured to perform various functions in accordance with one or more embodiments or examples described herein. Thus, the apparatuses <NUM>, <NUM>, <NUM> and <NUM> can provide means for implementation of techniques, processes, functions, components, systems described herein. The <NUM>, <NUM>, <NUM> and <NUM> can be a general purpose computer in some embodiments, and can be a device including specially designed circuits to implement various functions, components, or processes described herein in other embodiments. The apparatuses <NUM>, <NUM>, <NUM> and <NUM> can optionally include other components, such as input and output devices, additional or signal processing circuitry, and the like. Accordingly, the apparatuses <NUM>, <NUM>, <NUM> and <NUM> may be capable of performing other additional functions, such as executing application programs, and processing alternative communication protocols.

The processes and functions described herein can be implemented as a computer program which, when executed by one or more processors, can cause the one or more processors to perform the respective processes and functions. The computer program may be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with, or as part of, other hardware. The computer program may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. For example, the computer program can be obtained and loaded into an apparatus, including obtaining the computer program through physical medium or distributed system, including, for example, from a server connected to the Internet.

The computer program may be accessible from a computer-readable medium providing program instructions for use by or in connection with a computer or any instruction execution system. The computer readable medium may include any apparatus that stores, communicates, propagates, or transports the computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable medium can be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. The computer-readable medium may include a computer-readable non-transitory storage medium such as a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a magnetic disk and an optical disk, and the like. The computer-readable non-transitory storage medium can include all types of computer readable medium, including magnetic storage medium, optical storage medium, flash medium, and solid state storage medium.

<FIG> shows a flow chart of an exemplary method <NUM> for executing an NN inference according to some embodiments of the disclosure. Aspects of the method <NUM> can be performed at smartphones, wearables and other IoT devices, such as the apparatuses <NUM>, <NUM>, <NUM> and <NUM> illustrated in and describe with respect to the preceding figures. In various embodiments, some of the steps of the method <NUM> shown can be performed concurrently, in a different order than shown, can be substituted for by other method step, or can be omitted. Additional method steps can also be performed as desired. The method <NUM> can include steps <NUM>, <NUM> and <NUM>.

At step <NUM>, a portion of an NN inference can be loaded from a first memory that stores the NN inference to a second memory. For example, the second memory can be a TCM or an SRAM. For another example, the first memory can be a DRAM, an on-bus SRAM, or a serial flash memory. In an embodiment, the portion <NUM> of the NN inference <NUM> can be loaded from the DRAM <NUM> to the TCM <NUM>.

At step <NUM>, the portion of the NN inference loaded on the second memory can be executed, and a signal can be generated based on a progress of the execution of the NN inference. In an embodiment, the portion <NUM> of the NN inference <NUM> loaded on the TCM <NUM> can be executed by the inference executor <NUM>, and the signal, e.g., the signals of Node <NUM> and Node <NUM>, can be generated by the inference executor <NUM> based on a progress of the execution of the NN inference <NUM>.

At step <NUM>, a next portion of the NN inference stored in the first memory can be loaded to the second memory and power supplied to the first memory can be managed based on the signal and an inference executing scheme stored in the second memory. For example, the power suppled to the first memory can be managed by powering on/off the first memory, configuring an operation mode of the first memory, or scaling a voltage and/or a frequency applied to the first memory. As another example, the inference executing scheme can be a script, a rule or a model. As yet another example, the signal can include an operation, a node ID, a tensor, a kernel and/or time of the NN inference. In an embodiment, the next portion <NUM> of the NN inference <NUM> stored in the DRAM <NUM> can be loaded to the TCM <NUM> and the power supplied to the DRAM <NUM> can be managed by powering on/off the DRAM <NUM> based on the script <NUM>, e.g., DRAM powered on, DRAM powered off, data <NUM>-<NUM> moved to TCM and data <NUM> and <NUM> moved to TCM, and the signals, e.g., the signals of Node <NUM>, Node <NUM>, Node <NUM>, Node <NUM>, Node <NUM>, <NUM> and convolution op. The method <NUM> can keep executing steps <NUM> and <NUM>, till the complete of the execution of the entire NN inference <NUM>.

<FIG> shows a flow chart of another exemplary method <NUM> for executing an NN inference according to some embodiments of the disclosure. Aspects of the method <NUM> can be performed at smartphones, wearables and other IoT devices, such as the apparatuses <NUM>, <NUM>, <NUM> and <NUM> illustrated in and describe with respect to the preceding figures. In various embodiments, some of the steps of the method <NUM> shown can be performed concurrently, in a different order than shown, can be substituted for by other method step, or can be omitted. Additional method steps can also be performed as desired. The method <NUM> can include steps <NUM>, <NUM>, <NUM>, <NUM> and <NUM>.

At step <NUM>, an inference executor, e.g., the inference executor <NUM>, can execute a portion of an NN inference that is loaded on a second memory, e.g., the portion <NUM> of the NN inference <NUM> loaded on the TCM <NUM>. At step <NUM>, the inference executor <NUM> can send a signal based on a progress of the execution of the NN inference <NUM>, e.g., node ID, time, operation, tensor, kernel etc. At step <NUM>, a dynamic loading agent, e.g., the dynamic loading agent <NUM>, can receive the signal. At step <NUM>, the dynamic loading agent <NUM>, upon receiving the signal, can look up an inference executing scheme stored in the second memory <NUM> indexed by the signal, e.g., the inference executing scheme <NUM> stored in the TCM <NUM> index by node ID. At step <NUM>, the dynamic loading agent <NUM> can perform corresponding actions based on the inference executing scheme <NUM> indexed by the signal, such as driving a DMA controller, e.g., the DMA controller <NUM>, to load data, e.g., the next portion <NUM> of the NN inference <NUM>, at specified time (preload) or to load multiple data (batch) from a first memory <NUM>, e.g., the DRAM <NUM>, to the TCM <NUM>, enabling or disabling AP/DRAM/DMA power (schedule on/off), and/or passing configurations or settings to operator and do special operations (fine-grained operator). Steps <NUM>, <NUM>, <NUM>, <NUM> and <NUM> of the method <NUM> can be kept executed, until the entire NN inference <NUM> has been loaded from the DRAM <NUM> to the TCM <NUM>. In an embodiment, step <NUM> and steps <NUM> and <NUM> can be performed synchronously. For example, the dynamic loading agent <NUM> and the inference executor <NUM> can be included on a single chip, and the single chip can perform the steps <NUM>, <NUM> and <NUM> sequentially. In another embodiments, step <NUM> and steps <NUM> and <NUM> can be performed asynchronously. For example, the dynamic loading agent <NUM> and the inference executor <NUM> can be two distinct chips, and the two distinct chips can perform step <NUM> and steps <NUM> and <NUM>, respectively, in parallel.

Claim 1:
An apparatus (<NUM>) for executing a neural network, NN, inference program, comprising:
an inference executor (<NUM>) that is configured to execute a portion (<NUM>) of the NN inference program loaded on a second memory (<NUM>) from a first memory (<NUM>) that stores the NN inference program, wherein the second memory is a tightly-coupled memory, TCM, or a static random access memory, SRAM, and wherein the first memory is a dynamic random access memory, DRAM, an on-bus SRAM, or a serial flash memory; and
a dynamic loading agent (<NUM>) coupled to the inference executor (<NUM>), the first memory (<NUM>), and the second memory (<NUM>), the dynamic loading agent (<NUM>) being configured to load a next portion (<NUM>) of the NN inference program stored in the first memory (<NUM>) to the second memory (<NUM>),
characterized in that
the inference executor (<NUM>) is further configured to generate a signal based on an execution of the portion (<NUM>) of the NN inference program loaded in the second memory (<NUM>), and in that
the dynamic loading agent (<NUM>) is further configured to obtain information about when and how to load a portion of the NN inference program from the first memory (<NUM>) to the second memory (<NUM>) from an inference executing scheme (<NUM>) stored in the second memory (<NUM>), and
to manage power supplied to the first memory (<NUM>) based on the signal from the inference executor (<NUM>) and the inference executing scheme (<NUM>), by scaling a voltage and/or a frequency applied to the first memory (<NUM>).