Scalable neural network processing engine

Embodiments relate to a neural processor circuit with scalable architecture for instantiating one or more neural networks. The neural processor circuit includes a data buffer coupled to a memory external to the neural processor circuit, and a plurality of neural engine circuits. To execute tasks that instantiate the neural networks, each neural engine circuit generates output data using input data and kernel coefficients. A neural processor circuit may include multiple neural engine circuits that are selectively activated or deactivated according to configuration data of the tasks. Furthermore, an electronic device may include multiple neural processor circuits that are selectively activated or deactivated to execute the tasks.

BACKGROUND

1. Field of the Disclosure

The present disclosure relates to a circuit for implementing a neural network and more specifically to scalable processing of neural network tasks.

2. Description of the Related Arts

An artificial neural network (ANN) is a computing system or model that uses a collection of connected nodes to process input data. The ANN is typically organized into layers where different layers perform different types of transformation on their input. Extensions or variants of ANN such as convolution neural network (CNN), deep neural networks (DNN), recurrent neural networks (RNN) and deep belief networks (DBN) have come to receive much attention. These computing systems or models often involve extensive computing operations including multiplication and accumulation. For example, CNN is a class of machine learning technique that primarily uses convolution between input data and kernel data, which can be decomposed into multiplication and accumulation operations.

Depending on the types of input data and operations to be performed, these machine learning systems or models can be configured differently. Such varying configuration would include, for example, pre-processing operations, number of channels in input data, kernel data to be used, non-linear function to be applied to convolution result, and applying of various post processing operations. Using a central processing unit (CPU) and its main memory to instantiate and execute machine learning systems or models of various configuration is relatively easy because such systems or models can be instantiated with mere updates to code. However, relying solely on the CPU for various operations of these machine learning systems or models would consume significant bandwidth of a central processing unit (CPU) as well as increase the overall power consumption.

SUMMARY

Embodiments relate to a neural processor circuit with scalable architecture for instantiating one or more neural networks. The neural processor circuit may include multiple engine circuits that are selectively activated or deactivated to support parallel processing of input data according to configuration data that programs the neural processor circuit to instantiate or implement a neural network. The neural processor circuit includes a plurality of neural engine circuits, a data buffer, and a kernel extract circuit. The neural engine circuits are selectively activated, and each of the neural engine circuits are configured to perform convolution operations on input data and kernel coefficients to generate output data. The data buffer is between the plurality of neural engine circuits and a memory external to the neural processor circuit. The data buffer stores the input data from the memory for sending to the neural engine circuits and the output data received from the neural engine circuits. The kernel extract circuit receives kernel data from the memory external to the neural processor circuit, and sends a corresponding kernel coefficient extracted from the kernel data to neural engine circuits selected for activation.

In some embodiments, a neural task manager circuit configured to provide configuration data to the neural engine circuits to activate or deactivate one or more of the neural engine circuits for a task.

In some embodiments, the data buffer allocates memory space for storing the input data and the output data responsive to receiving the configuration data from the neural task manager circuit.

In some embodiments, the kernel extract circuit allocates memory space for storing the kernel data responsive to receiving the configuration data from the neural task manager.

In some embodiments, the neural task manager circuit configured to provide configuration data to the neural processor circuit to activate or deactivate the neural processor circuit for a task. An electronic device may include multiple neural processor circuits that are selectively activated or deactivated.

In some embodiments, a plurality of neural engine circuits of a neural processor circuit are selectively activated. Each of the neural engine circuits is configured to perform convolution operations on input data and kernel coefficients to generate output data. The input data is stored in a data buffer of the neural engine circuit for sending to the neural engine circuits and the output data received from the neural engine circuits. A corresponding kernel coefficient extracted from kernel data is sent to neural engine circuits selected for activation.

The figures depict, and the detail description describes, various non-limiting embodiments for purposes of illustration only.

DETAILED DESCRIPTION

Embodiments of the present disclosure relate to a scalable neural processor circuit that includes a plurality of neural engines that can be selectively activated to instantiate neural networks. For a processing cycle, a data buffer of the neural processor circuit (e.g., coupled to a system memory and the first and second neural engine circuits) provides input data to each of the neural engine circuits that are active. Each of the active neural engine circuits may apply different kernel coefficients to the input data to generate different output data. The size of the data buffer and kernel memories that store kernel data may also be modified.

Exemplary Electronic Device

Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described. In some embodiments, the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as personal digital assistant (PDA) and/or music player functions. Exemplary embodiments of portable multifunction devices include, without limitation, the iPhone®, iPod Touch®, Apple Watch®, and iPad® devices from Apple Inc. of Cupertino, Calif. Other portable electronic devices, such as wearables, laptops or tablet computers, are optionally used. In some embodiments, the device is not a portable communications device, but is a desktop computer or other computing device that is not designed for portable use. In some embodiments, the disclosed electronic device may include a touch sensitive surface (e.g., a touch screen display and/or a touch pad). An example electronic device described below in conjunction withFIG.1(e.g., device100) may include a touch-sensitive surface for receiving user input. The electronic device may also include one or more other physical user-interface devices, such as a physical keyboard, a mouse and/or a joystick.

Figure (FIG.)1is a high-level diagram of an electronic device100, according to one embodiment. Device100may include one or more physical buttons, such as a “home” or menu button104. Menu button104is, for example, used to navigate to any application in a set of applications that are executed on device100. In some embodiments, menu button104includes a fingerprint sensor that identifies a fingerprint on menu button104. The fingerprint sensor may be used to determine whether a finger on menu button104has a fingerprint that matches a fingerprint stored for unlocking device100. Alternatively, in some embodiments, menu button104is implemented as a soft key in a graphical user interface (GUI) displayed on a touch screen.

In some embodiments, device100includes touch screen150, menu button104, push button106for powering the device on/off and locking the device, volume adjustment buttons108, Subscriber Identity Module (SIM) card slot110, head set jack112, and docking/charging external port124. Push button106may be used to turn the power on/off on the device by depressing the button and holding the button in the depressed state for a predefined time interval; to lock the device by depressing the button and releasing the button before the predefined time interval has elapsed; and/or to unlock the device or initiate an unlock process. In an alternative embodiment, device100also accepts verbal input for activation or deactivation of some functions through microphone113. The device100includes various components including, but not limited to, a memory (which may include one or more computer readable storage mediums), a memory controller, one or more central processing units (CPUs), a peripherals interface, an RF circuitry, an audio circuitry, speaker111, microphone113, input/output (I/O) subsystem, and other input or control devices. Device100may include one or more image sensors164, one or more proximity sensors166, and one or more accelerometers168. The device100may include components not shown inFIG.1.

Device100is only one example of an electronic device, and device100may have more or fewer components than listed above, some of which may be combined into a components or have a different configuration or arrangement. The various components of device100listed above are embodied in hardware, software, firmware or a combination thereof, including one or more signal processing and/or application specific integrated circuits (ASICs).

FIG.2is a block diagram illustrating components in device100, according to one embodiment. Device100may perform various operations including image processing. For this and other purposes, the device100may include, among other components, image sensor202, system-on-a chip (SOC) component204, system memory230, persistent storage (e.g., flash memory)228, motion (orientation) sensor234, and display216. The components as illustrated inFIG.2are merely illustrative. For example, device100may include other components (such as speaker or microphone) that are not illustrated inFIG.2. Further, some components (such as motion sensor234) may be omitted from device100.

Image sensor202is a component for capturing image data and may be embodied, for example, as a complementary metal-oxide-semiconductor (CMOS) active-pixel sensor a camera, video camera, or other devices. Image sensor202generates raw image data that is sent to SOC component204for further processing. In some embodiments, the image data processed by SOC component204is displayed on display216, stored in system memory230, persistent storage228or sent to a remote computing device via network connection. The raw image data generated by image sensor202may be in a Bayer color kernel array (CFA) pattern (hereinafter also referred to as “Bayer pattern”).

Motion sensor234is a component or a set of components for sensing motion of device100. Motion sensor234may generate sensor signals indicative of orientation and/or acceleration of device100. The sensor signals are sent to SOC component204for various operations such as turning on device100or rotating images displayed on display216.

Display216is a component for displaying images as generated by SOC component204. Display216may include, for example, liquid crystal display (LCD) device or an organic light emitting diode (OLED) device. Based on data received from SOC component204, display216may display various images, such as menus, selected operating parameters, images captured by image sensor202and processed by SOC component204, and/or other information received from a user interface of device100(not shown).

System memory230is a component for storing instructions for execution by SOC component204and for storing data processed by SOC component204. System memory230may be embodied as any type of memory including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM) or a combination thereof. In some embodiments, system memory230may store pixel data or other image data or statistics in various formats.

Persistent storage228is a component for storing data in a non-volatile manner. Persistent storage228retains data even when power is not available. Persistent storage228may be embodied as read-only memory (ROM), flash memory or other non-volatile random access memory devices.

SOC component204is embodied as one or more integrated circuit (IC) chip and performs various data processing processes. SOC component204may include, among other subcomponents, image signal processor (ISP)206, a central processor unit (CPU)208, a network interface210, sensor interface212, display controller214, neural processor circuit218, graphics processor (GPU)220, memory controller222, video encoder224, storage controller226, and bus232connecting these subcomponents. SOC component204may include more or fewer subcomponents than those shown inFIG.2.

ISP206is hardware that performs various stages of an image processing pipeline. In some embodiments, ISP206may receive raw image data from image sensor202, and process the raw image data into a form that is usable by other subcomponents of SOC component204or components of device100. ISP206may perform various image-manipulation operations such as image translation operations, horizontal and vertical scaling, color space conversion and/or image stabilization transformations, as described below in detail with reference toFIG.3.

CPU208may be embodied using any suitable instruction set architecture, and may be configured to execute instructions defined in that instruction set architecture. CPU208may be general-purpose or embedded processors using any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, ARM or MIPS ISAs, or any other suitable ISA. Although a single CPU is illustrated inFIG.2, SOC component204may include multiple CPUs. In multiprocessor systems, each of the CPUs may commonly, but not necessarily, implement the same ISA.

Graphics processing unit (GPU)220is graphics processing circuitry for performing graphical data. For example, GPU220may render objects to be displayed into a frame buffer (e.g., one that includes pixel data for an entire frame). GPU220may include one or more graphics processors that may execute graphics software to perform a part or all of the graphics operation, or hardware acceleration of certain graphics operations.

Neural processor circuit218is a circuit that performs various machine learning operations based on computations including multiplication, addition and accumulation. Such computations may be arranged to perform, for example, convolution of input data and kernel data. Neural processor circuit218is a configurable circuit that performs these operations in a fast and power-efficient manner while relieving CPU208of resource-intensive operations associated with neural network operations. Neural processor circuit218may receive the input data from sensor interface212, the image signal processor206, system memory230or other sources such as network interface210or GPU220. The output of neural processor circuit218may be provided to various components of device100such as the image signal processor206, system memory230or CPU208for various operations. The structure and operation of neural processor circuit218is described below in detail with reference toFIG.3.

Network interface210is a subcomponent that enables data to be exchanged between devices100and other devices via one or more networks (e.g., carrier or agent devices). For example, video or other image data may be received from other devices via network interface210and be stored in system memory230for subsequent processing (e.g., via a back-end interface to image signal processor206, such as discussed below inFIG.3) and display. The networks may include, but are not limited to, Local Area Networks (LANs) (e.g., an Ethernet or corporate network) and Wide Area Networks (WANs). The image data received via network interface210may undergo image processing processes by ISP206.

Sensor interface212is circuitry for interfacing with motion sensor234. Sensor interface212receives sensor information from motion sensor234and processes the sensor information to determine the orientation or movement of the device100.

Display controller214is circuitry for sending image data to be displayed on display216. Display controller214receives the image data from ISP206, CPU208, graphic processor or system memory230and processes the image data into a format suitable for display on display216.

Memory controller222is circuitry for communicating with system memory230. Memory controller222may read data from system memory230for processing by ISP206, CPU208, GPU220or other subcomponents of SOC component204. Memory controller222may also write data to system memory230received from various subcomponents of SOC component204.

Video encoder224is hardware, software, firmware or a combination thereof for encoding video data into a format suitable for storing in persistent storage128or for passing the data to network interface210for transmission over a network to another device.

In some embodiments, one or more subcomponents of SOC component204or some functionality of these subcomponents may be performed by software components executed on ISP206, CPU208or GPU220. Such software components may be stored in system memory230, persistent storage228or another device communicating with device100via network interface210.

Image data or video data may flow through various data paths within SOC component204. In one example, raw image data may be generated from the image sensor202and processed by ISP206, and then sent to system memory230via bus232and memory controller222. After the image data is stored in system memory230, it may be accessed by video encoder224for encoding or by display116for displaying via bus232.

Example Neural Processor Circuit

Neural processor circuit218is a configurable circuit that performs neural network operations on the input data based at least on kernel data340. For this purpose, neural processor circuit218may include, among other components, neural task manager310, a plurality of neural engines314A through314N (hereinafter collectively referred as “neural engines314” and individually also referred to as “neural engine314”), kernel direct memory access (DMA)324, data buffer318and buffer DMA320. Neural processor circuit218may include other components not illustrated inFIG.3.

Each of neural engines314performs computing operations for neural network operations in parallel. Depending on the load of operation, entire set of neural engines314may be operated or only a subset of the neural engines314may be operated while the remaining neural engines314are placed in a power save mode to conserve power. Each of neural engines314includes components for storing one or more kernels, for performing multiply-accumulate operations, and for post-processing to generate an output data328, as described below in detail with reference toFIG.4. One example of a neural network operation is a convolution operation.

Neural task manager310manages the overall operation of neural processor circuit218. Neural task manager310may receive a task list from a compiler executed by CPU208, store tasks in its task queues, choose a task to perform, and send instructions to other components of the neural processor circuit218for performing the chosen task. Neural task manager310may also perform switching of tasks on detection of events such as receiving instructions from CPU208. In one or more embodiments, the neural task manager310sends rasterizer information to the components of the neural processor circuit218to enable each of the components to track, retrieve or process appropriate portions of the input data and kernel data, as described below in detail with reference toFIGS.5through7. Although neural task manager310is illustrated inFIG.3as part of neural processor circuit218, neural task manager310may be a component outside the neural processor circuit218.

Kernel DMA324is a read circuit that fetches kernel data from a source (e.g., system memory230) and sends kernel data326A through326N to each of the neural engines314. Kernel data represents information from which kernel elements can be extracted. In one embodiment, the kernel data may be in a compressed format which is decompressed at each of neural engines314. Although kernel data provided to each of neural engines314may be the same in some instances, the kernel data provided to each of neural engines314is different in most instances.

Data buffer318is a temporary storage for storing data associated with the neural network operations. In one embodiment, data buffer318is embodied as a memory that can be accessed by all of the neural engines314. Data buffer318may store input data322A through322N for feeding to corresponding neural engines314A through314N, as well as output from each of neural engines314A through314N for feeding back into neural engines314or sending to a target circuit (e.g., system memory230). The operations of data buffer318and other components of the neural processor circuit218are coordinated so that the input data and intermediate data stored in the data buffer318is reused across multiple operations at the neural engines314, and thereby reduce data transfer to and from system memory230. Data buffer318may be operated in a broadcast mode where input data of all input channels are fed to all neural engines314or in a unicast mode where input data of a subset of input channels are fed to each neural engine314.

The input data322stored in data buffer318can be part of, among others, image data, histogram of oriented gradients (HOG) data, audio data, meta data, output data328of a previous cycle of the neural engine314, and other processed data received from other components of the SOC component204.

Buffer DMA320includes a read circuit that receives a portion (e.g., tile) of the input data from a source (e.g., system memory230) for storing in data buffer318, and a write circuit that forwards data from data buffer318to a target (e.g., system memory).

Example Neural Engine Architecture

FIG.4is a block diagram of the neural engine314, according to one embodiment. The neural engine314performs various operations to facilitate neural network operations such as convolution, spatial pooling and local response normalization. The neural engine314receives the input data322, performs multiply-accumulate operations (e.g., convolution operations) on the input data322based on stored kernel data, performs further post-processing operations on the result of the multiply-accumulate operations, and generates the output data328. The input data322and/or the output data328of the neural engine314may be of a single channel or multiple channels.

Input buffer circuit402is a circuit that stores a portion of the input data322as it is received from the data buffer318and sends an appropriate portion408of input data for a current task or process loop to computation core416for processing. Input buffer circuit402includes a shifter410that shifts read locations of input buffer circuit402to change the portion408of input data sent to computation core416. By changing portions of input data provided to the computation core416via shifting, neural engine314can perform multiply-accumulate for different portions of input data based on fewer number of read operations. In one or more embodiments, the input data322includes data of difference convolution groups and/or input channels.

Kernel extract circuit432is a circuit that receives kernel data326from kernel DMA324and extracts kernel coefficients422. In one embodiment, the kernel extract circuit432references a look up table (LUT) and uses a mask to reconstruct a kernel from compressed kernel data326. The mask indicates locations in the reconstructed kernel to be padded with zero and remaining locations to be filled with numbers. The kernel coefficients422of the reconstructed kernel are sent to computation core416to populate register in multiply-add (MAD) circuits of computation core416. In other embodiments, the kernel extract circuit432receives kernel data in an uncompressed format and the kernel coefficients are determined without referencing a LUT or using a mask.

Computation core416is a programmable circuit that performs computation operations. For this purpose, the computation core416may include MAD circuits MADO through MADN and a post-processor428. Each of MAD circuits MADO through MADN may store an input value in the portion408of the input data and a corresponding kernel coefficient in the kernel coefficients422. The input value and the corresponding kernel coefficient are multiplied in each of MAD circuits to generate a processed value412.

Accumulator414is a memory circuit that receives and stores processed values412from MAD circuits. The processed values stored in accumulator414may be sent back as feedback information419for further multiply and add operations at MAD circuits or sent to post-processor428for post-processing. Accumulator414in combination with MAD circuits form a multiply-accumulator (MAC)404. In one or more embodiments, accumulator414may have subunits where each subunit sends data to different components of neural engine314. For example, during a processing cycle, data stored in a first subunit of accumulator414is sent to MAC circuit while data stored in a second subunit of accumulator414is sent to post-processor428.

Post-processor428is a circuit that performs further processing of values412received from accumulator414. The post-processor428may perform operations including, but not limited to, applying nonlinear functions (e.g., Rectified Linear Unit (ReLU)), normalized cross-correlation (NCC), merging the results of performing neural operations on 8-bit data into 16-bit data, and local response normalization (LRN). The result of such operations is output from the post-processor428as processed values417to output circuit424.

NE control418controls operations of other components of the neural engine314based on the operation modes and parameters of neural processor circuit218. Depending on different modes of operation (e.g., group convolution mode or non-group convolution mode) or parameters (e.g., the number of input channels and the number of output channels), neural engine314may operate on different input data in different sequences, return different values from accumulator414to MAC circuits, and perform different types of post-processing operations at post processor428. To configure components of the neural engine314to operate in a desired manner, the NE control418sends a control signal to components of the neural engine. NE control418may also include rasterizer430that tracks the current task or process loop being processed at neural engine314, as described below in detail with reference toFIG.5through7.

Output circuit424receives processed values417from the post-processor428and interfaces with data buffer318to store processed values417in data buffer318. For this purpose, output circuit424may send out as output data328in a sequence or a format that is different from the sequence or format in which the processed values417are processed in post-processor428.

The components in the neural engine314may be configured during a configuration period by the NE control418and the neural task manager310. For this purpose, the neural task manager310sends configuration information to the neural engine314during the configuration period. The configurable parameters and modes may include, but are not limited to, mapping between input data elements and kernel elements, the number of input channels, the number of output channels, performing of output strides, and enabling/selection of post-processing operations at the post processor428.

Operation of Segmenting of Data for Processing at Neural Processor Circuit

Input data is typically split into smaller pieces of data for parallel processing at multiple neural engines314. Often multiple cycles of operations are performed to generate output for a task associated with a neural network. A compiler executed by CPU208analyzes the hierarchy and nodes of the neural network and determines how the input data is to be segmented based on the hardware constraints of the neural processor circuit218. One of functions of the compiler is to determine how input data is to be split into smaller data units for processing at the neural engines314, and how the processing is to be iterated in loops to produce the result for tasks.

FIG.5is a conceptual diagram illustrating loops for processing the input data at neural processor circuit218, according to one embodiment. The outermost loop represents processing for a convolution group, if group convolution involving multiple convolution group is used. Group convolutions are convolutions where input data of the input channels in each group are used only for generating output data of output channels of each group but are not used for generating output data for output channels of other groups. Hence, each group of the group convolution can be treated as a separate convolution operation.

In the loop for each convolution group is a processing loop for a slice of the input data. The entire input data for a convolution operation is segmented into multiple strips of slices in an overlapping manner, as shown inFIG.6. The overlapping portions602,604,606are parts of the input data that are overfetched in two adjacent slices to provide spatial support for a corresponding kernel. The second outermost loop performs convolution operation for each slice in the input data. Within the loop for a slice is a processing loop for a tile of the slice. Each slice is segmented into a plurality of tiles, as shown inFIG.6. The overlapping portions608,610,612,614are parts of the input data in slice4that are overfetched in two adjacent tiles to provide spatial support for a corresponding kernel. The rightmost tile will typically have a width smaller than other tiles of the slice. In one embodiment, input data for each tile is loaded onto data buffer318in a read cycle and reused for operations in processing loops for the tile. In the processing loop for the tile is a processing loop for a work unit. Each tile is segmented into multiple work units as shown inFIG.6. A work unit is a portion of the input data having a size that produces output values that fit into accumulator414of neural engine314during a single cycle of the computation core416. Although the shape of each work unit is shown as a horizontal strip inFIG.6, the shape of the work unit can be different depending on the shape and size of the tile. The work units also have overlapping parts that represent overfetched data to provide support for a corresponding kernel. Especially, work units for the last tile of a slice may have a shape of a vertical strip if the tile is tall. In one or more embodiments, the size of each work unit is 256 bytes. In such embodiments, for example, work units be shaped to one of 16×16, 32×8, 64×4, 128×2 or 256×1 dimension.

For each work unit, an internal processing loop may be provided for an output channel group (OCG). The number of output channels produced for a given work unit by a single cycle of the computation core416is referred to as an OCG. Depending on operation modes, each neural engine314may process output data of different numbers of output channels (e.g., 8 channels, 32 channels) for a single load of input data into its input buffer circuit402.

For each output channel group, an internal processing loop may be provided for an input channel (Cin). If an input stride is implemented to skip certain input data, loops for sub-input channels (Sub-Cin) may be provided within the processing loop for the input channel (Cin).

For each input channel or each sub-input channel, internal loops are provided for processing horizontal spatial support for a kernel and the vertical support within each horizontal spatial support. The spatial support refers to the input data for convolution with the kernel, and includes overfetched input data for performing convolution at the edges of the input data.

Overfetch refers to fetching additional input data in current slice, tile or work unit so that proper dimension of input data can be provided for convolution with a kernel. In one or more embodiments, overfetch is performed vertically between slices to obtain additional rows of input data (shown as overlapping portions602,604,606inFIG.6), horizontally between tiles to obtain additional columns of input data (shown as overlapping portions608,610,612,614inFIG.6), and vertically between work units within a tile to obtain additional rows of input data.

For each spatial support for the kernel, an internal processing loop for an output channel (OC) is provided to generate output data for each output channel (Cout). In cases where output stride implements a spatial upsampling, an additional inner loop for processing each sub-output channel is provided. Loading of kernel coefficients and MAC operations are performed within the loop for the output channel (OC) or sub-output channel if an output stride is implemented, to generate output data for the output channel (OC) or sub-output channel.

The nested loop structure ofFIG.5is merely illustrative. Loops may be omitted, added or structured differently depending on various factors. For example, if only a single convolution group is used, the outermost loop may be removed. Further, the loop structure for the horizontal spatial support and the vertical spatial support may be reversed.

In one or more embodiments, the operations associated dividing the input space into smaller units and processing these smaller units as described above with reference toFIGS.5and6are performed by rasterizers714,718,720,722in various components of neural processor circuit218. A rasterizer is a circuit in various components of neural processor circuit218that keeps track of the segment of the input/output data (e.g., group, work unit, input channel, output channel) and instructs the components of neural processor circuit for proper handling of the segment of the input data. For example, rasterizer720in buffer DMA320tracks tiles and slices received from system memory230while rasterizer718in data buffer318broadcasts in sequence work units for processing by the neural engines314. Rasterizer722in kernel DMA322determines which kernels are to be received and distributed to neural engines314, while rasterizers714in neural engines314operate shifters410in input buffer circuits402to forward correct portions408of input data to MAC404, and send the finished output data328to the data buffer318.

FIG.7is a diagram illustrating programming of rasterizers714,718,720,722in components314,318,320,322of the neural processor circuit218, according to one embodiment. To perform their functions, each of rasterizers714,718,720,722receives task information710(e.g., configuration data) indicating how the input data and/or kernel data are to be segmented and to be handled by each component of the neural processor circuit218. The task information includes information about particulars of the current layer (e.g., dimensions of input and output data, dimension of an associated kernel, types of padding at the boundaries of input data). Rasterizers714,718,720, and722may also receive constraints on their operations (e.g., whether to allow or disallow tile width over a threshold).

By providing rasterizers in different components of neural processor circuit218, overhead in data transmitted between the components of the neural processor circuit218may be reduced. If a single central rasterizer is provided to control different components of the neural processor circuit218, kernel data, input data, and output data transmitted between the components may be needed in these data to identify associated position in the loops of the task such as convolution group, tile, slice, work unit, input channel and output channel. By using distributed rasterizers, no separate metadata is needed to transmit the kernel data, input data and output data among components of the neural processor circuit218.

Example Process at Neural Engine Architecture

FIG.8is a flowchart illustrating a method of processing input data in neural processor circuit218, according to one embodiment. The method may include different and/or additional steps, or the steps may be in different orders.

After neural task manager310programs rasterizers714,718,720,722, the process of operating buffer DMA320is initiated by rasterizer720instructing804buffer DMA320to cause buffer DMA320to receive a tile of input data from system memory230. The tile received by buffer DMA320is stored806in data buffer318.

Rasterizer718in data buffer318then instructs808data buffer318to send a work unit to one or more neural engines314. The work unit is then stored in input buffer circuits402of the one or more neural engines314.

In one or more embodiments, input buffer circuit402selects816a portion of work unit to be sent to MAC404to perform multiply-accumulate operation. Then MAC404performs820multiply-accumulate operations on the selected portion of the work unit using a corresponding kernel. Then it is determined824if the entire work unit is processed at one or more neural engines314. If not, the selected portion of the work unit is shifted828by shifter410and returns to perform820another round of multiply-accumulate operations.

If it is determined824that the entire work unit was processed, then it proceeds to determine832if all work units in the tile was processed. If not, then the process proceeds836to the next work unit by having data buffer318send808a next work unit to one or more neural engines314, and repeats the subsequent processes.

If it is determined832that all work units in the tile was processed by the neural engines314, the process proceeds to determine840whether all tiles for the input data were processed. If not, the process proceeds844to a next tile by having rasterizer720instructs804buffer DMA320to receive a next tile from system memory230and repeats the subsequent processes.

If it is determined840that all tiles of the input data are processed, then the process ends for the current input data. Then, the process may be repeated to process the next input data or proceed to the next task.

Embodiments of the process as described above with reference toFIG.8are merely illustrative. Further loops may be embodied, as described above with reference toFIG.5. Moreover, sequence of the process may be modified or omitted.

A neural network may include network layers or sub-layers that are instantiated or implemented as a series of tasks executed by the neural processor circuit218.FIG.9is a schematic block diagram illustrating a neural network900represented by a list904of tasks, according to one embodiment. The neural network900includes network layers (or sub-layers) including convolution layers C1, C2, C3 (including sub-layers C300, C310, C311, C320, and C321), C5, and C6, and pooling layers P2 and P4. The neural network900is an example of a neural network architecture that may be instantiated by the neural processor circuit218. That is, when the neural network900is converted into the task list904to become executable by the neural processor circuit218. Other types of neural network architectures with different types of network layers or orders of network layers may also be instantiated by the neural processor circuit218.

The neural network900is converted, such as by the CPU208, to the task list904. The task list904includes a linear link-list defining a sequence of tasks including task C1, task C2+P2, task C300+P4, task C310, task C311+P4, task C320, task C321+P4, task C5a, task C5b, and task C6. Each task is associated with a task descriptor that defines a configuration of the neural processor circuit218to execute the task. Each task may correspond with a single network layer of the neural network900, a portion of a network layer of the neural network900, or multiple network layers of the neural network900. For example, the task C1 corresponds with a single network layer C1, the task C2+P2 corresponds with multiple network layers C2 and P2, and the tasks C5aand C5beach correspond with a portion of the network layer C5. The neural processor circuit218instantiates the neural network900by executing the tasks of the task list904under the control of the neural task manager310.

FIG.10is a block diagram illustrating the neural task manager310, according to one embodiment. The neural task manager310manages the execution of tasks for one or more neural networks900by the neural processor circuit218. The neural task manager310may include, among other components, a task arbiter1002, task queues1004A through1004N (hereinafter collectively referred as “task queues1004” and individually also referred to as “task queue1004”), a task manager direct memory access (DMA)1006, a fetch queue1008, and a configuration queue1010. The neural task manager310may include other components not illustrated inFIG.10.

The task arbiter1002is a circuit or a combination of circuit and firmware that selects tasks from the task queues1004for execution by the neural processor circuit218. The task arbiter1002dequeues tasks from the task queues1004, and places tasks in the configuration queue1010. While a task is in a configuration queue, it is committed to execution and the neural processor circuit performs a prefetch for input data and kernel data before the task is executed by other components of the neural processor circuit218. For example, the task arbiter1002may perform fixed-priority arbitration between multiple task queues1004, and select the task from the task queues1004with the highest priority for retrieval of a task descriptor1012from the system memory230by the task manager DMA1006.

The neural task manager310may include one or more task queues1004. Each task queue1004is coupled to the CPU208and the task arbiter1002. Each task queue1004receives from the CPU208a reference to a task list904of tasks that when executed by the neural processor circuit218instantiates a neural network900. The reference stored in each task queue1004may include a set of pointers and counters pointing to the task list904of the task descriptors1012in the system memory230. Each task queue1004may be further associated with a priority parameter that defines the relative priority of the task queues1004. The task descriptor1012of a task specifies a configuration of the neural processor circuit218for executing the task.

The task manager DMA1006is coupled to task arbiter1002, the system memory230, and the fetch queue1008. The task manager DMA1006includes a read circuit that receives task descriptors1012of tasks from a source (e.g., system memory230) for storing in the fetch queue1008. For example, the task arbiter1002selects a task queue1004according to the priorities of the task queues1004, and uses the task list904referenced by the selected task queue1004to control the task manager DMA1006to select the task descriptor1012of a task.

The fetch queue1008is a single entry queue that stores a task descriptor1012of a task that is pending to commit for execution. The fetch queue1008is coupled to the task manager DMA1006to receive the task descriptor1012from the system memory230, and provides the task descriptor1012to the configuration queue1010, or the configuration data1014extracted from the task descriptor1012to the configuration queue1010.

The configuration queue1010holds configuration data1014of multiple tasks that have been committed for execution. When a task is in the configuration queue1010, the kernel DMA324may fetch kernel data from the system memory230to store in the kernel extract circuit432of neural engines314, and the buffer DMA320may fetch input data from the system memory230to store in the data buffer318. To execute the task, the kernel extract circuit432provides the prefetched kernel data to the MAC404of the neural engine314, and the data buffer318provides the prefetched input data to the MAC404of the neural engine314. In some embodiments, the configuration queue1010may include multiple queues that holds configuration data1014extracted from the committed task descriptors1012. As discussed in greater detail in connection withFIG.13, the configuration queue1010is further coupled to other components of the neural processor circuit218to configure the neural processor circuit218according to the configuration data1014.

FIG.11is a diagram illustrating retrieval of task descriptors1012using a task queue1004, according to one embodiment. The task queue1004includes a reference, such as a set of pointers, to the task descriptors1012A through1012N stored in the system memory230. To that end, the task queue1004may include a memory storing a head parameter1102, a network identifier (ID)1104, a base address index1106, a tail parameter1108, a count parameter1110, and a priority parameter1112. The head parameter1102is a pointer to a location of the system memory230storing the task descriptor1012A at the head of the task queue1004. The network ID1104identifies the neural network900of the task descriptor1012at the head of the task queue1004, and the base address index1106is an index to a base-address table1114tagged with the network ID1104of the task descriptor1012A at the head of the task queue1004. The count parameter1110defines the number of task descriptors1012in the task queue1004. The priority parameter1112defines the priority of the task queue1004, which is used by the task arbiter1002to select between multiple task queues1004.

When a particular task queue1004is selected (e.g., according to the priority parameter1112), the task arbiter1002references the head parameter1102, the network ID1104, the base address index1106, and the base address table1114to retrieve a task descriptor1012from the system memory230, and places the task descriptor1012into the fetch queue1008to initiate commitment of the task for execution. In each configuration period, the task arbiter1002may continue to place a task descriptor1012into the fetch queue1008according to the order of tasks defined by the task list904of the task queue1004, such as by retrieving the next task descriptor1012B, and so forth.

FIG.12is a diagram illustrating a task descriptor1012, according to one embodiment. The task arbiter1002places the task descriptor1012in the fetch queue1008from system memory230, which is then transferred to the configuration queue1010. The highest priority (e.g., first in) task descriptor1012in the configuration queue1010is used to configure the neural processor circuit218for execution during the configuration period. The task descriptor1012includes configuration data1014including a task descriptor header1202and address data1204A through1204N (hereinafter referred as “address data1204”). The task descriptor header1202includes configuration data1014that configures various operations of the neural task manager310, including operations related to task selection and task switching. For example, the task descriptor header1202may be parsed by the task arbiter1002to extract configuration data1014that programs the neural task manager310and other components of the neural processing circuit218. The task descriptor header1202may include a task identifier (ID)1206that identifies the task, a neural network identifier (ID)1208that identifies a neural network900instantiated by the task, a task switch parameter1210defining whether the neural task manager310should initiate a task switch (e.g., to execute a task of a different task queue1004) after execution of the task, an input surface parameter1212defining whether the input data for the task should be retrieved from the system memory230or the data buffer318, an output surface parameter1214defining whether the output data of the task should be stored in the system memory230or the data buffer318, various (e.g., base address) pointers1216to facilitate the programming of the neural processor circuit218, and one or more debug/exception parameters1218that control event, exception, or debug logging.

Each instance of address data1204A through1204N (collectively or individually referred to as “address data1204”) defines an address and data payload pair used to program the components of the neural processor circuit218. The data payload may include input data and kernel data used to execute the task. For example, each instance of address data1204includes register data defining the data payload, a register address defining a destination memory location of the neural processing circuit218for receiving the register data, and a register count defining a number of consecutive memory locations (e.g., registers) to be written with the register data. In some embodiments, the register address is combined with the base address stored in the header1202to define the full address of each memory location. If the task descriptor1116is generated at compile time, then the actual run time addresses may not be known. The base address table1114is used avoid duplicating or updating all task descriptors with dynamically assigned addresses.

FIG.13is a block diagram illustrating the fetch queue1008and configuration queue1010, according to one embodiment. The configuration queue1010is coupled to the fetch queue1008, which is coupled to the system memory230via the task manager DMA1006. The configuration queue1010is further coupled to the rasterizer714of one or more neural engines314, the rasterizer718of the data buffer318, the rasterizer720of the buffer DMA320, and the rasterizer722of the kernel DMA322. The fetch queue1008stores a task descriptor1012(e.g., including the task descriptor header1202and the address data1204A through1204N) for a task that is pending and not committed to execution. The fetch queue1008reduces the latency of reading the next task descriptor1012into the configuration queue1010from the system memory230. The fetch queue1008stores the highest priority task descriptor1012as determined by the task arbiter1002. The task arbiter1002may replace the task descriptor1012stored in the fetch queue1008if a higher priority task descriptor1012has been enqueued (e.g., from a higher priority task queue1004). The task descriptor1012in the fetch queue1008does not initiate an input data or kernel prefetch, and does not affect task queue priorities, pointers, or counters. As such, a task descriptor1012in the fetch queue1008may be readily replaced by a higher priority task descriptor1012by writing the higher priority task descriptor1012into the fetch queue1008. When a task descriptor1012stored in the configuration queue1010is executed by the neural processor circuit218, the task descriptor1012stored in the fetch queue1008is transferred to the configuration queue1010, and another task descriptor1012of a subsequent task may be stored in the fetch queue1008

The configuration queue1010stores task descriptors1012of tasks committed for execution by the neural processor circuit218. In some embodiments, the configuration queue1010includes multiple separate queues1310that each store a portion of the configuration data1014(including configuration data1014A through1014E) extracted from the task descriptor1012. Furthermore, the queues1310are each coupled to a respective component of the neural processor circuit218for programming the component with the configuration data1014. Through operation of the configuration queue1010, the neural task manager310programs the rasterizers714,718,720,722to perform the functionality discussed above inFIGS.7and8. For example, a queue1310A is coupled to the rasterizers714of the neural engines314to provide configuration data1014A that controls the operations of the shifters410in the input buffer circuits402to forward correct portions408of input data to MAC404, and send the finished output data328to the data buffer318. The queue1310B is coupled to the rasterizer718of the data buffer318to provide configuration data1014B that controls the broadcasting of input data (e.g., work units) by the data buffer318for processing by the neural engines314. The queue1310C is a read queue that is coupled to the rasterizer720of the buffer DMA320to provide configuration data1014C that controls the buffer DMA320to retrieve input data (e.g., a tile) from system memory230and store the input data in the data buffer318. The queue1310D is a write queue that is coupled to the rasterizer720of the buffer DMA320to provide configuration data1014D that controls the buffer DMA320to store output data in the system memory230. The queue1310E is coupled to the rasterizer722of the kernel DMA322to provide configuration data1014E that controls which kernels are to be received and distributed to neural engines314. In some embodiments, a task descriptor1012or configuration data1014stored in the configuration queue1010cannot be replaced and will be executed in a first in, first out (FIFO) order.

Example Process at Neural Task Manager Architecture

FIG.14is a flowchart illustrating a method of managing tasks in the neural processor circuit218, according to one embodiment. The method may include different and/or additional steps, or the steps may be in different orders.

The CPU208generates1402a task list904of task descriptors1012of tasks that when executed by the neural processor circuit218instantiates a neural network900. For example, the CPU208may receive input data for a machine learning operation from the image sensor202, the system memory230, the persistent storage228, the network interface210, or some other components. The machine learning operation may include an inferencing operation, or a training operation. The neural network900may include kernel data and a neural network architecture including network layers. The input data is applied to the neural network900to perform the machine learning operation. The kernel data and network layers of the neural network900may be computed in a machine learning training process. The CPU208performs a compile operation (offline or on-the-fly) to turn a neural network description900into a linked list of task descriptors1012referred to herein as the task list904. Each task is defined by a task descriptor1012, and when executed by the neural processor circuit218instantiates a single network layer, multiple network layers, or a portion of a network layer. Each task descriptor1012of a task includes configuration data1014, such as the task descriptor header1202and the address data1204defining neural processor circuit218address and data payload pairs. The data payload may include the kernel data of the neural network900and the input data. The configuration data1014further includes instructions that configure operation of the rasterizers714,718,720, and722to execute the task.

The CPU208stores1404the task list904of task descriptors1012in the system memory230. In some embodiments, the CPU208or another CPU external to the electronic device100generates the task descriptors1012and stores the task descriptors1012in the persistent storage228or some other non-volatile memory. The task descriptors1012are loaded along with the kernel data and input data in the system memory230for use by the neural processor circuit218. The CPU208may be coupled to the system memory230via the bus232and the memory controller222.

The task list904may include a set of pointers that reference the locations of the task descriptors1012in the system memory230. Furthermore, the CPU208may update parameters of the task descriptors1102such as memory addresses or the network ID. For example, the task list904may include a head parameter1102and a tail parameter1108respectively defining a beginning register address and an end register addresses of the system memory230where multiple tasks descriptors1012are stored. In some embodiments, the references to the register addresses in the task list904may be partial addresses, and a base address table1114is used to define the full reference addresses to the system memory230. In some embodiments, the CPU208may patch absolute addresses as necessary. The CPU208may further configure the neural processor circuit218by setting its base address registers.

The task arbiter1002of the neural task manager310enqueues1406the task list904for execution by the neural processor circuit218. For example, the neural task manager310includes multiple task queues1004A through1004N. The task queues1004may each store a reference to a task list904. Furthermore, the task queues1004are prioritized for execution according to the priority parameters1112of the task lists904referenced by the task queues1004.

In some embodiments, the CPU208performs a general configuration of the neural processor circuit218to execute the task. The CPU208may further start the neural processor circuit218if the neural processor circuit218is not already running.

The task arbiter1002selects1408a task queue with highest priority for execution. For example, the task arbiter1002in each programming period selects a task queue1004based a comparison of priority parameters of the task queues1004or the task lists904of the task queues1004, and executes tasks from the task list904from the highest priority task queue1004.

The neural processor circuit218executes1410a task from the selected task queue. For example, the neural processor circuit218performs the method ofFIG.8to execute the task. The configuration queue1010of the neural task manager310may provide the configuration data1014(or task information710) to the rasterizers714,718,720,722of the neural processor circuit218to program the neural processor circuit218to execute the task, as shown inFIG.7. Furthermore, the execution of the task may include processing of the prefetched kernel data and input data while the task was in the configuration queue1010. The execution of each task may include multiple processing loops for handling the input data as shown inFIGS.5and6. The task arbiter1002may dequeue the task from the configuration queue1010after execution of the task.

In some embodiments, to execute the task, the task arbiter1002controls the task manager DMA1006to retrieve the task descriptor1012of the task of the task list904, and store the task descriptor1012in the fetch queue1008. After execution of a previously committed task, the task arbiter1002may dequeue the executed task by removing the task descriptor or configuration data of the task from the configuration queue1010. The task descriptor1012or extracted configuration data1014of the current task is then placed in the configuration queue1010from the fetch queue1008. When the task is in the configuration queue1010, the neural processor circuit218may initiate a prefetch operation by the kernel DMA324for kernel data from the system memory230to the kernel extract circuit432, and a prefetch operation by the buffer DMA320for input data from the system memory230to the data buffer318.

If a different task queue has a higher priority, a task switch to the task queue with the higher priority may be performed when possible. To perform a task switch (e.g., to another machine learning operation), the task arbiter1002replaces a task in the fetch queue1008with another task descriptor referenced in a different task queue1004. The task arbiter1002may resume the interrupted machine learning operation defined by the task list904after completion of the task switch by storing the replaced task into the fetch queue1008.

In some embodiments, the CPU208determines1408a priority parameter1110of the task list904to select a task queue. The priority parameter1110defines the priority of the tasks of the machine learning operation relative to other tasks of other machine learning operations executed by the neural processor circuit218. The CPU208may facilitate the execution of multiple machine learning operations. The machine learning operations may be different, such as by using different input data or different neural network architectures and kernel data. The CPU208may determine the priority parameter of the task list904programmatically, or based on user input. The task descriptor1012referenced by the task queue1004with the highest priority may be selected for execution. The priority of a task queue is either determined by the CPU208, or dynamically by information from a previously executed task descriptor.

The task arbiter1002determines1412whether all tasks of the task list904have been executed. If a task of the task list904has not been executed, the process returns to step1410, where the task arbiter1002executes the unexecuted task of the task list904.

If each task of the task list904has been executed, the task arbiter1002removes1414the task list904from the task queue1004. In another example, a reference to a task descriptor1012of a task may be removed from the task queue1004subsequent to execution of the task. The process may end. In another example, the process may return to1402, where the CPU208may continue to generate task descriptors of tasks for other machine learning operations. In that sense, the process may be repeated to place tasks into task queues for execution of machine learning operations by the neural processor circuit218according to specified task priorities.

Scalable Neural Engines

Various components of the electronic device100and the neural processor circuit218may be scalable to facilitate parallel processing of machine learning operations. Configuration data for tasks may define the selective usage of hardware components.

The neural engines314of a neural processor circuit218may be selectively activated or deactivated for particular tasks. With reference toFIG.3, for example, the neural processor circuit218includes the neural engines314A through314N. In some examples, the neural processor circuit218may include two, eight, or sixteen neural engines314. Each of the neural engines314is coupled to the data buffer318. In some embodiments, the data buffer318broadcasts the same input data to each activated neural engines314in a processing cycle. In the processing cycle, each of the neural engines apply different or the same kernel coefficients to the input data to generate different output values. For example, output data from different neural engines314in a processing cycle may include output data associated with different channels. One or more neural engines314of the neural processor circuit218may be deactivated for the processing cycle. Deactivation may result in the neural engine314being turned off or placed in a power saving mode (e.g., using less power than when activated).

In some embodiments, the data buffer318stores a tile of input data, and provides a work unit of the tile, or a portion of a work unit of the tile, to the neural engines314in a processing cycle. The neural engines314may apply different kernel coefficients so that different operations on the work unit of the tile may be processed in parallel by multiple neural engines314. As such, the number of processing cycles used to process input data by a neural processor circuit218can be decreased by using additional neural engines314in the neural processor circuit218. In another example, a neural engine314may be deactivated if not used in processing a task to reduce power consumption.

Scalable Neural Processor Circuits

An electronic device may include different numbers of neural processor circuits218, with each neural processor circuit218being selectively activated or deactivated for particular tasks.FIG.15is a block diagram illustrating multiple neural processor circuits218in an electronic device1500, according to one embodiment. The electronic device1500includes the bus232, the CPU208, the system memory230, and may further include other components of the electronic device100as discussed above in connection withFIG.2. The electronic device1500further includes a neural interconnect circuit1502coupled to the system memory230and neural processor circuits218A through218N.

The neural interconnect circuit1502includes a neural interconnect buffer direct memory access (DMA)1504and a neural interconnect data buffer1506. The neural interconnect buffer DMA1504retrieves configuration data for tasks from the system memory230and stores the configuration data in the neural interconnect data buffer1506. The neural interconnect data buffer1506is coupled to the buffer DMAs320and the data buffers318of each of the neural processor circuits218A through218N. The neural interconnect data buffer1506distributes configuration data to neural processor circuits218A through218N, such as one or more of the neural processor circuits218A through218N which are selectively activated for processing tasks.

In some embodiments, the neural interconnect data buffer1506stores a slice or work group of input data, and provides different tiles of a slice to one or more of the neural processor circuits218A through218N. The data buffers318of different neural processor circuits218may store a different tile. Each data buffer318of a neural processor circuit218may provide a work unit or a portion of a work unit of the tile stored in the data buffer318to each of the neural engines314of the neural processor circuits218in a processing cycle. The neural processor circuits314may use different input data or different kernel coefficients so that different operations on different tiles may be processed in parallel by multiple neural processor circuits218. In some embodiments, multiple neural processor circuits218may be used to process the different tasks of a neural network in parallel. In some embodiments, different neural processor circuits218may process tasks for different neural networks in parallel.

The number of processing cycles used to process input data by an electronic device1500can be decreased by using additional neural processor circuits218. If fewer than all of the neural processor circuits218of an electronic device are committed to task processing, one or more neural processor circuits218may operate in a power saving mode or be turned off while one or more other neural processor circuits218processes input data.

In some embodiments, multiple neural processor circuits218may each be coupled to the system memory230. For example, the data buffers318of each neural processor circuit218may retrieve configuration data, input data, and kernel data directly from the system memory230. Here, the neural interconnect circuit1502may be omitted from the electronic device.

In some embodiments, barriers may be used to support multiple neural processor circuits218to avoid data hazards. Data hazards occur when input data or kernel coefficients haven't arrived in the neural interconnect data buffer1506or system memory230when a neural processor circuit218tries to consume it. Examples of data hazards may include (1) kernel prefetch from system memory230to the neural interconnect data buffer1506followed by the neural processor circuit218consuming the kernel coefficients, (2) input data load from the system memory230to the neural interconnect data buffer1506followed by the neural processor circuit218consuming the input data, (3) neural processor circuit218producing output data followed by a different neural processor circuit consuming the output data as input data, (4) neural processor circuit218producing output data followed by the same neural processor circuit218consuming the output data as kernel coefficients, or (5) a neural engine314producing data output data followed by a neural processor circuit218consuming the output data.

In some embodiments, prefetching is performed for kernel data or kernel coefficients from system memory230into the neural interconnect data buffer1506. The neural interconnect buffer DMA1504needs to complete the transfer of coefficients of a given kernel before the neural processor circuit218consumes it. In case the neural processor circuit218is faster than the data transfer by the neural interconnect buffer DMA1504, to avoid the data hazard, the CPU220may configure upfront a breakpoint register inside the neural task manager310which stops the execution of the neural processor circuit218directly after a provided task ID has been reached. The breakpoint register contains this ask ID bit field and an enable bit. When the neural interconnect buffer DMA1504completes a configurable amount of kernel coefficients, it sends an interrupt to the CPU220which then updates the breakpoint register to the task ID of the most recently transferred kernel coefficients. If the data transfer by the neural interconnect buffer DMA1504is ahead of the kernel read from the neural processor circuit218, everything is fine and the neural processor circuit218keeps running. In case the neural processor circuit218is faster than the data transfer by the neural interconnect buffer DMA1504, the neural processor circuit218will reach the breakpoint and stops until the neural interconnect buffer DMA1504has completed and the CPU220re-activates the neural engine314of the neural processor circuit218.

In the case of (4) neural processor circuit218producing output data followed by the same neural processor circuit218consuming the output data as kernel coefficients, the compiler provides the depending task ID in the task descriptor of the consuming task. The neural processor circuit218then avoids the hazard in hardware by making sure that the current task will not execute before the task with the depending task ID finishes execution including all DMA writes.

In some embodiments, a neural processor circuit218or other component of the electronic device100(e.g., CPU22ISP206, etc.) may include a memory scale and rotate (MSR) circuit. The MSR circuit crops input patches of variable size from an input image and scales the input image to a different resolution before the input image as input data the neural processor circuit218. For example, the neural processor circuit218may instantiate a neural network that performs object detection by providing bounding boxes of potential objects of different sixes. The MSR circuit then crops these objects and scales it to the same size such that another consuming neural network performing object classification can classify these objects.

In some embodiments, a neural processor circuit218may be located in the CPU208, the GPU220, the image signal processor206, or other component of the electronic device100. The neural processor circuit218in another component may operate as discussed herein for a neural processor circuit218. For example, a neural processor circuit218may operate with a neural processor circuit in the CPU208, the GPU220, or the image signal processor206to process different tiles of a slice of input data, or otherwise process input data in parallel.

In some embodiments, multiple neural processor circuits218can either run separate neural networks, or speed up the execution of a single neural network by using the compiler to slice a network spatially (width or height), channel-wise and/or into separate branches so that these slices can be scheduled across neural processor circuits218.

Scalable Data Buffer and Kernel Extract Circuit

The size of the data buffer318of each neural processor circuit218may be scalable according to the number of activated neural engines314that are activated for a task. For example, a portion of the memory space of the data buffer318may be allocated to storing input data and kernel data for each neural engine314that is activated for performing a task. Furthermore, the total size of the data buffer318may correspond with the number of neural engines314in the neural processor circuit218, and the size of the MAC circuit404of each neural engine314. In some embodiments, the data buffer318is sufficiently sized to store the input data for a tile including work units that are broadcast to each of the neural engines314for each processing cycle, the kernel data to be applied to the work units in each processing cycle, and any other configuration data for each processing cycle (e.g., instructions for rasterizers). The data buffer318may also be sufficiently sized to store output data generated by the MAC circuits404of the neural engines314from the input data and kernel data. The output data may replace the input data for a previous processing cycle in the data buffer318, or may be stored in a different location of the data buffer318. In that sense, the size of the data buffer318or allocation of data to the data buffer318may vary to provide scalability of neural engine314use in a neural processor circuit218.

The size of the kernel extract circuit432of each neural engine314may be scalable to support different types of neural operations with input data. Furthermore, a kernel extract circuit432of a neural engine314may be selectively activated or deactivated with the neural engine314.FIG.16is a block diagram illustrating a kernel extract circuit432of a neural engine314, according to one embodiment. The kernel extract circuit432is a component of a neural processor circuit218, and includes a kernel memory1604and a kernel decompress circuit1606. The kernel memory1604receives the kernel data326from the kernel DMA324of the neural processor circuit218, and stores the kernel data326. The kernel decompress circuit1606generates the kernel coefficients422using the kernel data326, and provides the kernel coefficients422to the MAC circuit404of the neural processor circuit218. The kernel decompress circuit1606converts the kernel data326in a compressed format into the kernel coefficients422. The size of the kernel memory1604, or the allocation of data to the kernel memory1604, may vary to support scalable kernel sizes.

In some embodiments, the number of MAD circuits in the MAC404of the neural engine314may also be scaled to support the kernel size. For example, the number of MAD circuits located or activated in a neural engine314may correspond with the number of kernel coefficients422to facilitate processing of each of the kernel coefficients422provided to the MAD circuits MADO through MADN in a processing cycle.

In some embodiments, each neural engine314includes X number of MAD circuits. As such, N neural engines314in a neural processor circuit218provides for up to N*X operations in each processing cycle. Furthermore, M neural processor circuits218in an electronic device provides for up to N*M*X operations in each processing cycle. In an example, an electronic device may include a single neural processor circuit218(M=1) each having eight neural engines314(N=8) with each neural engine314including 256 MAD circuits (X=256) to provide for up to 2,048 operations in each processing cycle. In another example, an electronic device may include a two neural processor circuit218(M=2) each having sixteen neural engines314(N=16) with each neural engine314including 256 MAD circuits (X=256) to provide for up to 8,192 operations in each processing cycle.

In addition to the components of the electronic device100/1500and neural processing circuit218being scalable, various fabrics, buses, and/or interfaces that connect these components may also be scalable in size to facilitate data transfers between the components.

Example Scalable Process at Neural Processor Architecture

FIG.17is a flowchart illustrating a method of processing input data with a scalable neural processor circuit, according to one embodiment. The method may include different and/or additional steps, or the steps may be in different orders.

The CPU208generates1702configuration data of tasks according to computing resources of an electronic device. The configuration data may define the activation and deactivation of components such as neural processor circuits218and neural engines318for the task. The configuration data may also define allocation of input data into memory space of data buffers318of activated neural processor circuits, and allocation of kernel data into memory space of kernel memories1604of activated neural engines314.

The configuration data for each task may be defined by a task descriptor1012. The CPU208performs a compile operation to generate task descriptors1012of the tasks based on input data and the network layers of a neural network900, and the available computing resources. The configuration data may include the address data1204defining neural processor circuit218address and data payload pairs for input data and kernel data. In some embodiments, the CPU208generates configuration data of tasks for multiple neural networks that are implemented in parallel. The tasks for each neural network may be generated in a separate compile operation.

As discussed above, the configuration data may be generated based on the resources of the electronic device and neural processor circuit(s)218. For example, the CPU208may optimize the processing of the network layers of the neural network into tasks according to the computing resources of the electronic device including the number of neural processor circuits218, the number of neural engines314in the neural processor circuits218, the size of the data buffers318of the neural processor circuits, the number of MAD circuits in the neural engines314, and the size of the kernel extract circuits432in the neural engines314. The configuration data may define behavior of rasterizers to control how kernel data and input data is provided to the data buffer318of each activated neural processor circuit218. For each neural processor circuit218, the configuration data may define how portions of the input data and kernel coefficients extracted from the kernel data are provided to each activated neural engine314of the neural processor circuit218. If a neural processor circuit218or neural engine314is not used for a task, the configuration data may specify a deactivation of the unused component.

The system memory230stores1704the configuration data of the tasks. For example, the CPU208provides the configuration data of the tasks to the system memory230. Furthermore, the CPU208may enqueue task lists in task queues1004of a neural task manager310.

In some embodiments, the system memory230stores multiple tiles of input data. For example, the system memory230may store input data for a convolution group, each convolution group including multiple slices, and each slice including the multiple tiles. In some embodiments, the system memory230stores multiple tiles belonging to different convolution groups or different neural networks.

The buffer DMA320of a neural processor circuit218retrieves1706input data from the system memory230to the data buffer318of the neural processor circuit218, and the kernel DMA324of the neural processor circuit218retrieves1706kernel data from the system memory230to the buffer extract circuit432. For example, when the configuration data for a task is placed in the configuration queue1010of the neural task manager310of a neural processor circuit218, the buffer DMA320and kernel DMA324performs a prefetch for the input data and kernel data, respectively. The prefetch results in the input data being stored in the data buffer318and the kernel data being stored in the kernel extract circuits432of each neural engine314prior to the processing cycle for the input data and the kernel data. The size of the data buffer318or the kernel extract circuit432(e.g., the kernel memory1604) may be selected to support the utilization of the neural engines314in the neural processor circuit218, and a subset of the available memory space may be allocated according to the configuration data and the activation and deactivation of the neural engines314.

In some embodiments, multiple activated neural processor circuits218perform the step1706in parallel. The buffer DMA320of each neural processor circuit218may retrieve one of the multiple tiles of input data stored in the system memory230according to the configuration data. Furthermore, the kernel DMA324of each neural processor circuit218retrieves kernel data to be processed with the input data retrieved by the buffer DMA320of the neural processor circuit218.

In some embodiments, one or more neural processor circuits218may be deactivated when not used in a processing cycle. The deactivation may include placing a neural processor circuit218in a power saving mode or powering off the neural processor circuit218. Here, the deactivated neural processor circuit218does not retrieve input data or kernel data from the system memory230.

The data buffer318of the neural processor circuit218provides1708a portion of the input data stored in the data buffer318to MAC circuits404of the neural engines314of the neural processor circuit218, and the kernel extract circuit432of the neural processor circuit218provides1708kernel coefficients generated from the kernel data to the MAC circuits404of the neural engines314of the neural processor circuit218. The neural engines314that receive the input data and kernel data may be activated neural engines314. One or more neural engines314of the neural processor circuit218may be deactivated, such as by being placed in a power saving mode or powered off, when not used in a processing cycle.

The portion of input data provided to each of the activated neural engines314of the neural processor circuit218may include a work unit of the tile of input data stored in the data buffer318of the neural processor circuit218. In some embodiments, the activated neural engines314of the neural processor circuit218receives the same (e.g., work unit or portion of a work unit) input data, and applies different kernel coefficients to the input data in a processing cycle. Multiple neural processor circuits218may perform step1708in parallel so that the neural processor circuits218process different input data in the processing cycle. In the same processing cycle, multiple neural engines314of a neural processor circuit218may be tasked with applying different kernel coefficients to the same input data. Here, multiple neural engines314may operate in parallel to generate different channels of output data in the processing cycle. In a subsequent processing cycle, the input data provided to the neural engines314may be different, or the kernel data used by the neural engines314may be different, or both.

The neural engines314of the neural processor circuit218generate1710output data using the portion of input data and the kernel coefficients. For example, each neural processing circuit218may generate different output data in a processing cycle based on the input data and kernel data provided to the MAC circuits404of the activated neural engines314in the processing cycle.

In some embodiments, the neural engines314perform multiple processing cycles to implement loops for processing the input data as shown inFIGS.5through8. For example, the neural engine314of a neural processor circuit218may process different work units of a tile of input data in different processing cycles. In another example, different neural processor circuits218may process different tiles of a common slice of input data in a processing cycle. In another example, different neural processor circuits218may process different neural networks in a processing cycle, such as different tiles of different input data.

The neural engines314of the neural processor circuit218store1712the output data in the data buffer318of the neural processor circuit218. After performing one or more processing cycles, the output data from each neural engine314of the neural processor circuit218is stored in the data buffer318of the neural processor circuit218.

The neural task manager310determines1714for the neural processor circuit218whether the output data stored in the data buffer318of the neural processor circuit218is input data for a subsequent task executed by the neural processor circuit218. The determination may be based on the configuration data for the task, such as the output surface parameter1214of the task descriptor1012of the task.

If the output data stored in the data buffer318is input data for a subsequent task, the method returns to1408, where the input data and corresponding kernel coefficients are provided to MAC circuits404of the neural engines314of the neural processor circuit218. In the subsequent task, different components such as neural processor circuits218or neural engines314may be selectively activated or deactivated according to the configuration data of the subsequent.

If the output data stored in the data buffer318is not input data for the subsequent task, the buffer DMA320of the neural processor circuit218stores the output data from the data buffer318to the system memory230. The method may end.

In some embodiments, the neural processor circuit218may perform a prefetch operation in the processing cycle that generates the output data for subsequent input data from the system memory230to be processed by the neural processor circuit218in a subsequent processing cycle.

In some embodiments, the output data from a first neural processor circuit218is used as input data for a second neural processor circuit218. For example, the output data from the first neural processor circuit218may be stored in the system memory230, and the system memory230may provide the output data to the second neural processor circuit218as input data.

The steps1706through1716may be performed by multiple neural processor circuits218of the electronic device, such as in parallel to improve the speed of machine learning operations executed by the electronic device. The output data from each neural processor circuit218may be collected by the neural interconnect data buffer1506or the system memory230. Depending on the configuration data of tasks, the output data may be provided to one or more neural processor circuits218as input data for a subsequent processing cycle.