Patent Description:
Various computer vision, speech recognition, and signal processing applications may benefit from the use of learning/inference machines, which may quickly perform hundreds, thousands, or even millions of concurrent operations. Learning/inference machines, as discussed in this disclosure, may fall under the technological titles of machine learning, artificial intelligence, neural networks, probabilistic inference engines, accelerators, and the like. Conventional learning/inference machines can deliver hundreds of teraflops (e.g., one million millions - <NUM><NUM> - floating-point operations per second) of computing power.

Such learning/inference machines may include or otherwise utilize CNNs, such as deep convolutional neural networks (briefly, DCNN). A DCNN is a computer-based tool that processes large quantities of data and adaptively "learns" by conflating proximally related features within the data, making broad predictions about the data, and refining the predictions based on reliable conclusions and new conflations. The DCNN is arranged in a plurality of "layers," and different types of predictions are made at each layer. Hardware accelerators including convolutional accelerators are often employed to accelerate the processing of large amounts of data by a DCNN.

Further aspects and examples are provided for facilitating the understanding of the invention.

In an embodiment, a convolutional accelerator comprises a feature line buffer, a kernel buffer separate from the feature line buffer, a Multiply-ACcumulate (briefly, MAC) cluster, a configuration register and mode control circuitry coupled to the feature line buffer, the kernel buffer, and the MAC cluster. The mode control circuitry, in a first mode of operation of the convolutional accelerator, stores feature data in the feature line buffer, stores kernel data in the kernel buffer, transfers feature data from the feature line buffer to the MAC cluster, and transfers kernel data from the kernel buffer to the MAC cluster. In a second mode of operation of the convolutional accelerator, the mode control circuitry stores feature data in the kernel buffer, stores kernel data in the feature line buffer, transfers feature data from the kernel buffer to the MAC cluster, and transfers kernel data from the feature line buffer to the MAC cluster. The feature line buffer is a single port memory; the kernel buffer comprises a plurality of dual-port buffers, and the mode control circuitry, in operation, determines whether to operate in the first mode of operation or the second mode of operation based on a configuration parameter stored in the configuration register. The second mode of operation may be employed to efficiently process 1xN kernels, where N is an integer greater than or equal to <NUM> (one).

In an embodiment, a system comprises a stream engine, which, in operation, streams feature and kernel data, and a convolutional accelerator coupled to the stream engine, wherein the convolutional accelerator, in operation, receives streams of feature and kernel data from the stream engine. The convolutional accelerator includes a feature line buffer, a kernel buffer, a configuration register and a multiply-accumulate cluster coupled to the feature line buffer and the kernel buffer, mode control circuitry coupled to the feature line buffer, the kernel buffer, and the MAC cluster. The mode control circuitry, in a first mode of operation of the convolutional accelerator, stores feature data in the feature line buffer, stores kernel data in the kernel buffer, transfers feature data from the feature line buffer to the MAC cluster, and transfers kernel data from the kernel buffer to the MAC cluster. In a second mode of operation of the convolutional accelerator, the mode control circuitry stores feature data in the kernel buffer, stores kernel data in the feature line buffer, transfers feature data from the kernel buffer to the MAC cluster, and transfers kernel data from the feature line buffer to the MAC cluster. The feature line buffer is a single port memory; the kernel buffer comprises a plurality of dual-port buffers, and the mode control circuitry, in operation, determines whether to operate in the first mode of operation or the second mode of operation based on a configuration parameter stored in the configuration register. The second mode of operation may be employed to efficiently process 1xN kernels, where N is an integer greater than or equal to <NUM> (one or unity).

In an embodiment, a method comprises streaming feature data and kernel data to a convolutional accelerator, and convolving streamed kernel data with streamed feature data. The convolving includes, in a first mode of operation of the convolutional accelerator, storing feature data in a feature line buffer of the convolutional accelerator, storing kernel data in a kernel buffer of the convolutional accelerator, transferring feature data from the feature line buffer to a MAC cluster of the convolutional accelerator, and transferring kernel data from the kernel buffer to the MAC cluster. In a second mode of operation of the convolutional accelerator the convolving includes storing feature data in the kernel buffer, storing kernel data in the feature line buffer, transferring feature data from the kernel buffer to the MAC cluster, and transferring kernel data from the feature line buffer to the MAC cluster. The feature line buffer is a single port memory; the kernel buffer comprises a plurality of dual-port buffers, and the mode control circuitry, in operation, determines whether to operate in the first mode of operation or the second mode of operation based on a configuration parameter stored in the configuration register. The second mode of operation may be employed to efficiently process 1xN kernels, where N is an integer greater than or equal to <NUM> (one or unity).

In an embodiment, a non-transitory computer-readable medium's instructions configure a convolutional accelerator having a plurality of modes of operation to convolve streamed kernel data with streamed feature data. The convolving includes, in a first mode of operation of the plurality of modes of operation of the convolutional accelerator, storing feature data in a feature line buffer of the convolutional accelerator, storing kernel data in a kernel buffer of the convolutional accelerator, transferring feature data from the feature line buffer to a MAC cluster of the convolutional accelerator, and transferring kernel data from the kernel buffer to the MAC cluster. In a second mode of operation of the plurality of modes of operation of the convolutional accelerator, the convolving includes storing feature data in the kernel buffer, storing kernel data in the feature line buffer, transferring feature data from the kernel buffer to the MAC cluster, and transferring kernel data from the feature line buffer to the MAC cluster. The feature line buffer is a single port memory; the kernel buffer comprises a plurality of dual-port buffers, and the mode control circuitry, in operation, determines whether to operate in the first mode of operation or the second mode of operation based on a configuration parameter stored in the configuration register. The second mode of operation may be employed to efficiently process 1xN kernels, where N is an integer greater than or equal to <NUM> (one or unity). In an embodiment, the contents comprise instructions executed by the convolutional accelerator.

Document <CIT> discloses techniques and systems for implementing a convolutional neural network, where one or more convolution accelerators are provided that each include a feature line buffer memory, a kernel buffer memory, and a plurality of multiply-accumulate (MAC) circuits arranged to multiply and accumulate data.

Document <CIT> is also part of the background.

One or more embodiments are described hereinafter with reference to the accompanying drawings.

The following description, along with the accompanying drawings, sets forth certain specific details in order to provide a thorough understanding of various disclosed embodiments. However, one skilled in the relevant art will recognize that the disclosed embodiments may be practiced in various combinations, with or without one or more of these specific details, or with other methods, components, devices, materials, etc. In other instances, well-known structures or components that are associated with the environment of the present disclosure, including but not limited to interfaces, power supplies, physical component layout, convolutional accelerators, Multiply-ACcumulate (MAC) circuitry, etc., in a hardware accelerator environment, have not been shown or described in order to avoid unnecessarily obscuring descriptions of the embodiments. Additionally, the various embodiments may be methods, systems, devices, computer program products, etc..

Throughout the specification, claims, and drawings, the following terms take the meaning associated herein, unless the context indicates otherwise. The term "herein" refers to the specification, claims, and drawings associated with the current application. The phrases "in one embodiment," "in another embodiment," "in various embodiments," "in some embodiments," "in other embodiments," and other variations thereof refer to one or more features, structures, functions, limitations, or characteristics of the present disclosure, and are not limited to the same or different embodiments unless the context indicates otherwise. As used herein, the term "or" is an inclusive "or" operator, and is equivalent to the phrases "A or B, or both" or "A or B or C, or any combination thereof," and lists with additional elements are similarly treated. The term "based on" is not exclusive and allows for being based on additional features, functions, aspects, or limitations not described, unless the context indicates otherwise. In addition, throughout the specification, the meaning of "a," "an," and "the" include singular and plural references.

CNNs are particularly suitable for recognition tasks, such as recognition of numbers or objects in images, and may provide highly accurate results. <FIG> is a diagram illustrating a digit recognition task concept and <FIG> is a diagram illustrating an image recognition task concept.

CNNs are specific types of deep neural networks (briefly, DNNs) with one or multiple layers which perform a convolution on a three-dimensional feature data tensor (width by height by depth). The first layer is an input layer and the last layer is an output layer. The intermediate layers may be referred to as hidden layers. The most used layers are convolutional layers, fully connected or dense layers, and pooling layers (max pooling, average pooling, etc.). Data exchanged between layers are called features or activations. Each layer also has a set of learnable parameters typically referred to as weights or kernels. <FIG> is a diagram illustrating concepts of an example of a CNN, that is AlexNet. The illustrated CNN has a set of convolutional layers interleaved with max pooling layers, followed by a set of fully connected or dense layers.

The parameters of a convolutional layer include a set of learnable filters referred to as kernels. Each kernel has three dimensions: height, width and depth. The height and width are typically limited in range (e.g., [<NUM>, <NUM>] - one to eleven). The depth typically extends to the full depth of an input feature data. Each kernel slides across the width and the height of the input features and a dot product is computed. At the end of the process a result is obtained as a set of two-dimensional feature maps. In a convolutional layer, many kernels are applied to an input feature map, each of which produces a different feature map as a result. The depth of the output feature tensors is also referred to the number of output channels. <FIG> is a diagram illustrating the application concept of a kernel to a feature map, producing a two-dimensional feature map having a height of <NUM> (four) and a width of <NUM>.

Convolutional layers also may have other parameters, which may be defined for the convolutional layer, rather than learned parameters. Such parameters may be referred to as hyper-parameters. For example, a convolutional layer may have hyper-parameters including stride and padding hyper-parameters.

The stride hyper-parameter indicates a step-size used to slide kernels across an input feature map. <FIG> is a diagram illustrating a concept of comparing a stride of <NUM> (one or unity) and a stride of <NUM>. When the stride is greater than <NUM> (one or unity), the output feature map will be smaller than the input feature map.

The padding hyper-parameter indicate a number of zeros to be added along the height, the width or the height and width of the input feature map. The padding parameters may be used to control a size of an output feature map generated by the convolution.

<FIG> is a diagram illustrating application concepts of padding to an input feature map. The padding preserves the input feature size along the height and width of the feature map.

The feature data of a convolutional layer may have hundreds or even thousands of channels, with the number of channels corresponding to the depth of the feature data and of the kernel data. For this reason, feature and kernel data are often loaded into memory in batches. <FIG> is a conceptual diagram illustrating the concept of loading feature data in batches. The feature data is split along the depth dimension into batches, with each batch of feature data having the same height, width and depth. The kernel depth is generally the same as the depth of the input feature map, so similar issues are addressed by batching.

As illustrated, the batches have a height of <NUM>, a width of <NUM>, and a depth of <NUM>. Batches are typically written into memory sequentially, with writing of a first batch being completed before beginning the writing of a second batch.

The arrows in <FIG> illustrate an example order in which data of a batch is written into memory. A similar batching process is typically applied to the kernel data, with each batch of the kernel data having a same kernel height and kernel width, and the same depth as the batches of feature data. Each batch of feature data is convolved with a related batch of kernel data, and a feedback mechanism is employed to accumulate the results of the batches. The conceptual diagram of <FIG> illustrates the concept of batch processing of a convolution.

As can be seen, the computations performed by a CNN, or by other neural networks, often include repetitive computations over large amounts of data. For this reason, computing systems having hardware accelerators may be employed to increase the efficiency of performing operations associated with the CNN.

<FIG> is a functional block diagram of an embodiment of an electronic device or system <NUM> of the type to which described embodiments may apply. The system <NUM> comprises one or more processing cores or circuits <NUM>. The processing cores <NUM> may comprise, for example, one or more processors, a state machine, a microprocessor, a programmable logic circuit, discrete circuitry, logic gates, registers, etc., and various combinations thereof. The processing cores may control overall operation of the system <NUM>, execution of application programs by the system <NUM> (e.g., programs which classify images using CNNs), etc..

The system <NUM> includes one or more memories <NUM>, such as one or more volatile and/or non-volatile memories which may store, for example, all or part of instructions and data related to control of the system <NUM>, applications and operations performed by the system <NUM>, etc. One or more of the memories <NUM> may include a memory array, which, in operation, may be shared by one or more processes executed by the system <NUM>.

The system <NUM> may include one or more sensors <NUM> (e.g., image sensors, audio sensors, accelerometers, pressure sensors, temperature sensors, etc.), one or more interfaces <NUM> (e.g., wireless communication interfaces, wired communication interfaces, etc.), and other circuits <NUM>, which may include antennas, power supplies, one or more built-in self-test (briefly, BIST) circuits, etc., and a main bus system <NUM>. The main bus system <NUM> may include one or more data, address, power and/or control buses coupled to the various components of the system <NUM>.

The system <NUM> also includes one or more hardware accelerators <NUM> which, in operation, accelerate the performance of one or more operations associated with implementing a CNN. The hardware accelerator <NUM> as illustrated includes one or more convolutional accelerators <NUM> to facilitate efficient performance of convolutions associated with convolutional layers of a CNN. The convolutional accelerators include a feature line buffer memory <NUM>, which, in operation, conventionally stores lines of feature data, a kernel buffer memory <NUM>, which, in operation, conventionally stores kernel data, and one or more clusters of Multiply-Accumulate (briefly, MAC) units or circuits <NUM>, which in operation perform convolution operations using the buffered kernel and feature data. The hardware accelerator <NUM> as illustrated also includes a stream engine <NUM> and a stream switch <NUM>. The stream engine <NUM>, in operation, transmits data streams. For example, the stream engine <NUM> may stream data, such as feature data or kernel data stored in memory <NUM>, to a convolutional accelerator <NUM> via the stream switch <NUM>.

The kernel dimensions may vary between CNNs, and between convolutional layers of a single CNN. For example, in <FIG> convolutional layers with kernels having height and width dimensions of 11x11, 5x5 and 3x3 are illustrated. Historically, 3x3 kernels have been the most common. Thus, convolutional accelerators may typically be designed to efficiently support kernel computations within defined kernel height and width sizes, such as 3x3 kernels.

For example, the feature line buffer memory <NUM> is typically a deep single port memory storing a few lines of feature data (e.g., three lines) and having a large number of storage elements per line (e.g., <NUM> elements per line). Feature line data is often reused with multiple kernels, and a single port is generally sufficient for the feature line buffer memory <NUM> because the computations may typically be started once the feature line buffer is full.

The kernel buffer memory <NUM> may typically be a small, wide and shallow dual-port memory (e.g., <NUM> bits of width by <NUM> memory locations). Because the kernel buffer is wide, it is relatively expensive, and for this reason is often shallow to reduce the costs of the memory (e.g., to reduce the area costs). To provide for an efficient bandwidth in view of the shallow depth of the memory, the kernel buffer <NUM> is a dual-port memory to facilitate simultaneous reading of stored kernel data for use in current convolutional operations and loading of kernel data into the kernel buffer for use in subsequent convolutional operations.

<FIG> is a diagram illustrating processing concepts of convolutional operations having a 3x3 kernel using an embodiment of a convolutional accelerator, such as the convolutional accelerator <NUM> of <FIG>. Feature data is stored in the feature line buffer <NUM> (e.g., three lines of feature data which may be processed together), and kernel data is stored in the kernel buffer <NUM> (e.g., supporting kernels of various sizes up to a threshold size, with the storage in the buffer optimized, for example, for 3x3 kernel data). Convolutional operations are performed using the MAC clusters <NUM> (e.g., <NUM> MACs organized into <NUM> clusters <NUM> of <NUM> MACs each - three sets of four MAC units <NUM>) to operate on data received from the feature line buffer and the kernel buffer. <FIG> illustrates examples of other typical components of a convolutional accelerator (e.g., streaming interfaces <NUM>, streaming buffers <NUM>, an adder tree <NUM>, configuration registers <NUM>, etc.), which are not described in detail with reference to <FIG>. Larger size kernels (e.g., 4x4, 11x11, etc.) may typically be handled using, for example, software kernel decomposition techniques.

However, many CNNs have one or more layers with kernels having smaller height and width dimensions. For example, hidden or inner layers, especially dense or deep layers, may have a large number of channels of feature data and may frequently employ kernels having height and width dimensions smaller than 3x3, such as 1x1, 1x2, or 1x3, 1xN kernels. In particular, deep or dense convolutional layers having feature data with a larger number of channels convolved with 1x1 deep (or unitary) kernels are becoming increasingly popular in CNNs. <FIG> is a diagram illustrating concepts of convolutional layers having 3x3 and 1x1 kernels respectively.

Kernels having smaller height and width dimensions (e.g., 1x1 or unitary kernels) may be processed by switching off some of the components of a convolutional accelerator. This can be inefficient in a number of ways, as discussed with reference to the diagram of <FIG>, which illustrates concepts of an example of processing a convolutional layer with a 1x1 (or unitary) kernel by switching off components of a convolutional accelerator optimized for use with larger size kernels. Feature data is stored in the feature line buffer <NUM> and kernel data is stored in the kernel buffer <NUM>.

To avoid congestion due to bandwidth saturation at the output, the number of input channels to the MAC units (which is limited by the depth of the kernel data that can be stored in the kernel buffer) is maintained to be greater than or equal to the number of output channels of the MAC units. Typical kernel buffer depth limitations may limit the number of input channels to the MAC clusters <NUM> to, for example, <NUM> channels for kernels having a height and width dimensions of 1x1. Thus, in the example considered the number of output channels of the MAC clusters would be limited to <NUM> channels, due to the bottleneck created by the size of kernel buffer memory, to avoid bandwidth saturation at the output.

In an example configuration of <NUM> MAC units organized into <NUM> clusters C0-C5 of <NUM> MACs <NUM> each discussed above, only four MAC units <NUM> of the twelve MAC units of each cluster may be active, with the rest of the MAC units remaining idle, and only one output pixel per kernel may be processed per cycle. As illustrated, kernels <NUM>-<NUM> are provided to the first cluster of <NUM> MAC units with only <NUM> of the <NUM> MAC units being used, kernels <NUM>-<NUM> are provided to the second cluster of <NUM> MAC units with only <NUM> of the <NUM> MAC units of the second cluster being used, etc..

In addition, the ratio between fetched kernel data and the number of operations performed using the fetched kernel data drops dramatically as compared to processing of a 3x3 kernel, etc. using the same or a similar convolutional accelerator configuration. Thus, a significant number of additional fetch operations must be performed to process the convolutions of a convolutional layer having a 1x1 kernel when a solution of switching off components of the convolutional accelerator is employed.

Another alternative is to use fine-grained accelerator hardware. This approach, however, involves a substantial overhead in terms of control circuitry and software processing to process larger kernels using the fine-grained hardware. Using fine-grained hardware is not as efficient in processing larger kernels as coarse-grained accelerators. Such larger kernels, such as 3x3 kernels, also are frequently employed in CNNs as noted above.

The Inventors have realized that hardware resources of a coarse-grained convolutional accelerator (e.g., a convolutional accelerator designed to efficiently process kernels having height and width dimensions of 3x3) may be reused or reconfigured to more efficiently perform convolutions using kernels having smaller height and width dimensions (e.g., 1x1, 1x2,. 1xN dimensions) by employing two different modes of operation instead of switching off components.

When a convolutional layer having a 1x1 kernel is being processed, only one line of feature data needs to be stored at a time for convolving with kernels of the 1x1 kernels, so a deep three-line feature line buffer is not needed for the feature data. In addition, the size characteristics of the relatively deep feature line buffer <NUM> are more suited to storing kernel data associated with a 1x1 deep (or unitary) kernel than the size characteristics of the shallower kernel buffer <NUM>.

The inventors realized the types of data stored in the two buffers may be switched when convolutional layers having kernels with smaller height and width dimensions are being processed. A first mode of operation is employed to process kernels having larger height and width dimensions (e.g., 3x3 kernels), and a second mode of operation is employed to process kernels having smaller height and width dimensions (e.g., 1x1, 1x2,. Two concept example modes of operation are conceptually illustrated by the diagram of <FIG>.

In a first mode of operation, labeled normal mode in <FIG>, feature data is stored in the feature line buffer <NUM> and kernel data is stored in the kernel buffer <NUM>. The first mode of operation may be employed, for example, to process typical 3x3 kernels in a generally conventional manner. See the discussion of <FIG>, above.

In a second mode of operation, labeled deep mode in <FIG>, feature data is stored in the kernel buffer <NUM> instead of in the feature line buffer, and kernel data is stored in the feature line buffer <NUM>. For example, in the second mode of operation, two sets of <NUM> different kernels of 1x1x128 may be stored in the feature line buffer <NUM>, and <NUM> lines of feature data having a depth of <NUM> pixels may be stored in the kernel data buffer <NUM>. The second mode of operation may be employed, for example, to process 1x1 kernels, such as 1x1 kernels having a depth for which using the second mode of operation instead of the first mode of operation may provide increased efficiencies (e.g., a depth larger than a number of MAC units of the MAC cluster, in the example discussed, a kernel depth of <NUM>, which is larger than <NUM>).

To facilitate the first and second modes of operation, the system <NUM> as illustrated in <FIG> comprises mode control circuitry <NUM>. In the first mode of operation, the mode control circuitry <NUM> controls the storage of feature data in the feature line buffer <NUM> and the transfer of feature data from the feature line buffer <NUM> to the MAC clusters <NUM>, and the storage of kernel data in the kernel buffer <NUM>, and the transfer of kernel data from the kernel buffer <NUM> to the MAC clusters <NUM>. Operation of the mode control circuitry in the first mode of operation may be performed to control the storage and transfer of feature and kernel data in a generally conventional manner. For example, the concept is illustrated in the diagram of <FIG>. While the mode control circuitry <NUM> is illustrated as a block in <FIG> and <FIG>, the mode control circuitry may be implemented, for example, using control logic and discrete components (e.g., multiplexers, existing buffers, etc.) distributed through the convolutional accelerator <NUM>.

In the second mode of operation, the mode control circuitry <NUM> controls the storage of kernel data in the feature line buffer <NUM> and transmission of kernel data from the feature line buffer <NUM> to the MAC clusters <NUM>, and the storage of feature data in the kernel buffer <NUM>, and the transfer of feature data from the kernel buffer <NUM> to the MAC clusters <NUM>. Examples of the operation and configurations of embodiments of the mode control circuitry <NUM> in the second mode of operation are discussed herein with reference to <FIG>.

Embodiments of the system <NUM> of <FIG> may include more components than illustrated, may include fewer components than illustrated, may combine components, may separate components into sub-components, and various combination thereof. For example, the hardware accelerator <NUM> may include DMA controllers, the mode control circuitry <NUM> may be split into input mode control circuitry to control the storage of input feature data and kernel data into the buffers <NUM>, <NUM>, and output mode control circuitry to control the transfer of stored input feature data from the buffers <NUM>, <NUM> to the MAC clusters <NUM>, all or part of the mode control circuitry <NUM> may be shared by one or more convolutional accelerators <NUM>, the mode control circuitry <NUM> may be integrated into other control circuitry of the convolutional accelerator, etc., and various combinations thereof.

<FIG> is a diagram illustrating concepts of an example of processing a convolutional layer with a 1x1 (or unitary) kernel by operating the convolutional accelerator in a second mode of operation in which kernel data is stored in a feature line buffer (e.g., feature line buffer <NUM> of <FIG>) and pixels of feature data are stored in a kernel buffer (e.g., kernel buffer <NUM> of <FIG>).

A large single-port feature line buffer <NUM>, which may normally be used to store <NUM> lines of feature data having a depth of <NUM> elements, is instead controlled to store <NUM> kernels having a depth of up to <NUM> kernel values. Thus, <NUM> input channels are available, which is greater than the number of output channels of <NUM> available in the example configuration discussed above with reference to <FIG>. In the second mode of operation, the kernel buffer <NUM> is controlled to store three lines of feature data, and thus three lines of feature data may be fed to clusters of MAC units <NUM>.

As shown in <FIG>, kernels <NUM>-<NUM> are provided to a set of three clusters of four MAC units each. One of the three clusters of four MAC units computes the output for line <NUM> of the feature data and kernels <NUM>-<NUM>, a second one of the three clusters of <NUM> MAC units computes the output for line <NUM> of the feature data and kernels <NUM>-<NUM>, and a third one of the three clusters computes the output for line <NUM> of the feature data and kernels <NUM>-<NUM>. Similarly, kernels <NUM>-<NUM> are provided to another set of three clusters of four MAC units each, each cluster computing the output for a respective line of the feature data and kernels <NUM>-<NUM>. Kernels <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM> and <NUM>-<NUM> are similarly provided to respective clusters of four MAC units each to compute outputs for respective lines of the feature data and the respective kernels.

The throughput is improved from <NUM> pixel depth per kernel in a cycle to <NUM> pixel depths per kernel in a cycle when a convolutional layer having a 1x1 kernel is processed using the deep mode of operation as compared to the normal mode of operation. In addition, each kernel value is used in three MAC operations in a cycle, significantly improving the ratio of the number of fetch operations to the number of MAC operations performed as compared to the use of each kernel value in one MAC operation per cycle when the normal mode of operation is employed to process a 1x1 kernel (see <FIG>).

<FIG> is a diagram illustrating concepts of an embodiment of a configuration of a feature data path <NUM> and a kernel data path <NUM> of a convolutional accelerator <NUM> operating in a mode of operation, such as a deep mode of operation, in which feature line data is stored in a kernel buffer <NUM>, and kernel data is stored in a feature line buffer <NUM>. An embodiment of the feature data path <NUM> is shown in more detail in <FIG>, and an embodiment of the kernel data path <NUM> is shown in more detail in <FIG>.

The kernel buffer <NUM> in the embodiment of <FIG> comprises six dual-port memories <NUM>, as illustrated six 16x96 bit dual-port buffers. In the normal mode of operation, the six dual port memories <NUM> of the kernel buffer <NUM> store kernel data, which may be done in a generally conventional manner. In the deep mode of operation, four of the six dual-port memories <NUM> are included in the feature data path <NUM> and controlled by (deep-mode) feature load control circuitry 130f of the mode control circuitry <NUM> to store feature line data.

As illustrated the four dual-port memories are organized into six virtual buffers 116v of 128x8 bits in the feature line path <NUM>, each virtual buffer 116v storing a line of feature data. Three lines of feature data may be fed to the MAC clusters <NUM> at a time, each line being provided by the respective virtual buffer to two of the six clusters of MAC units <NUM>. While feature data from three of the virtual buffers 116v is being fed to the MAC clusters <NUM>, feature data for a next set of computations may be loaded into the other three virtual buffers 116v. Thus, three lines of feature data may be stored into the virtual buffers 116v of the kernel buffer memory <NUM> at a time in a ping-pong fashion. The (deep-mode) feature data load control circuitry 130f of the mode control circuitry <NUM> is included in the feature data path <NUM>, and in operation the (deep-mode) feature data load control circuitry 130f controls the order in which pixels of feature data are stored into the virtual buffers 116v of the kernel buffer <NUM>. As illustrated, the feature line path <NUM> includes a FIFO buffer and optional data manipulation circuitry.

In the second or deep mode of operation, the feature line buffer <NUM> is included in the kernel data path <NUM> and controlled to store kernel data by (deep-mode) kernel load control circuitry <NUM> of the mode control circuitry <NUM>. The two remaining dual-port buffers <NUM> of the kernel buffer <NUM> may advantageously be used in the kernel data path <NUM> to facilitate the storage in and retrieval from the feature line buffer <NUM> of kernel data stored therein. As illustrated, one of the dual-port buffers 216a may be used to reduce the number of cycles needed to fill the feature line buffer <NUM> with kernel data, and one of the dual-port buffers 216b may be used to mirror the output of the feature line buffer <NUM> in order to emulate dual-port memory performance by the single port feature line buffer <NUM> (e.g., to facilitate use of kernel data stored in the dual-port buffer 216b by the MAC clusters <NUM> while subsequent kernel data is being loaded into the feature line buffer <NUM> from the dual-port buffer 216a).

As illustrated, the (deep-mode) kernel load control circuitry <NUM> includes filtering circuitry 130kfilt, which in operation selects kernels from kernel data path <NUM> to load into the feature line buffer <NUM> via the dual-port buffer 216a, and multiplexer circuitry <NUM>, which in operation, controls the transmission of kernel data from the dual port buffer 216b to the MAC clusters <NUM>. The mode control circuitry <NUM> of <FIG> as illustrated also includes (deep-mode) serialization circuitry <NUM>, which in operation serializes the outputs of the MAC clusters to account for the order in which the MAC operations are performed.

<FIG> is a diagram illustrating concepts of feature and kernel data flow paths in a deep mode of operation of a convolutional accelerator according to an embodiment. Features are streamed depth-wise and a whole pixel is stored in each virtual buffer 116v of the kernel buffer <NUM>. Three pixels at a time are sent to the MAC clusters <NUM>, with each pixel being provided to a pair of the six MAC clusters <NUM>. For example, multiplexers 130fm of the mode control circuitry <NUM> may be controlled to provide either pixels <NUM>-<NUM> or pixels <NUM>-<NUM> to respective pairs of MAC clusters of the MAC clusters <NUM>.

Kernels <NUM>-<NUM> are stored in the feature line buffer <NUM>, and all <NUM> kernels are sent to respective MAC clusters in a cycle (see <FIG>), generating three lines of an output feature map. <FIG> is a diagram illustrating concepts of the transmission of three pixels per cycle for a given kernel value in the example configuration of <FIG>. The same kernel is strided three times in a cycle, obtaining three output values for each kernel value, or <NUM> output values for <NUM> kernels in a cycle. For example, the term "strided" refers to an object to which application of a stride is performed one or multiple times.

<FIG> is a diagram illustrating a concept of data path <NUM> of the feature data in a convolutional accelerator operating in a deep mode of operation according to an embodiment. Streaming engine <NUM> may be capable of streaming, for example, <NUM> bytes of data or more in a stream engine transaction. As illustrated, the streaming engine streams pixel <NUM>, depth <NUM> of feature data (<NUM> in <FIG>), pixel <NUM>, depth <NUM> (<NUM> in <FIG>) and pixel <NUM>, depth <NUM> (<NUM> in <FIG>) in a first transaction, T1. Transaction T2 sends the next three depths of pixel <NUM>, shown as <NUM>, <NUM> and <NUM> in <FIG>. The transmission continues in successive transmissions until the transmission of a set of pixels is complete, as illustrated until pixel W, depth <NUM> is transmitted. Other pixel depths may be employed. The pixels are stored in a FIFO buffer, and may be manipulated using data manipulation circuitry, before being stored in respective virtual buffers 116v by the (deep-mode) feature load control circuitry 130f. As illustrated, three pixels at a time are provided by respective virtual buffers 116v to respective pairs of clusters of four MAC units. The availability of three pixel-values (e.g., pixel <NUM>: depth <NUM>, pixel <NUM>: depth <NUM>, and pixel <NUM>: depth <NUM>) for processing with each kernel value available (e.g., kernels <NUM>-<NUM>) in a cycle provides a three-to-one improvement in the throughput and improves the ratio of MAC calculations per retrieved kernel value.

As mentioned above, while feature data from three of the virtual buffers 116v is being fed to the MAC clusters <NUM>, feature data for a next set of computations may be loaded into the other three virtual buffers 116v. Thus, three lines of feature data may be stored into the virtual buffers 116v of the kernel buffer memory <NUM> at a time in a ping-pong fashion.

<FIG> is a diagram illustrating concepts of a data path of the kernel data <NUM> in a convolutional accelerator operating in a deep mode of operation according to an embodiment. The operation of the elements of the kernel data path is described above with reference to <FIG>. In each cycle, all <NUM> kernel values stored in the dual-port memory 216b are supplied to respective MAC units of the <NUM> MAC units, with each kernel value being supplied to <NUM> MAC units.

<FIG> is a diagram illustrating concepts of the sharing of each kernel value, twenty-four in the illustrated embodiment, with three MAC units and the sharing of each pixel of feature data with a pair of MAC clusters in a convolutional accelerator operating in a deep mode of operation according to an embodiment, to produce <NUM> (seventy-two) output pixels per cycle.

As exemplified in <FIG>:
block <NUM>: each pixel is shared between a pair of clusters, and/or block <NUM>: each kernel is shared between <NUM> MACs of separate clusters.

As compared to the embodiment of <FIG> in processing a convolutional layer having 1x1 deep (or unitary) kernel, the performance per cycle is tripled and each retrieved kernel value is used on three MAC operations instead of one MAC operation. The additional control logic employed may add a small amount of costs in terms of area (e.g., <NUM>%).

<FIG> illustrate an embodiment of a deep mode of operation in the context of convolutional processing using a 1x1 (or unitary) kernel. <FIG> is a conceptual diagram illustrating convolutional operations associated with a 1x2 kernel in a convolutional accelerator operating in a deep mode of operation according to an embodiment. The 1x2 kernel is decomposed into two 1x1 sub-kernels and the pixel values are strided. The first 1x1 sub-kernel is processed in a first deep mode cycle producing a first partial sum using pixel values <NUM>-<NUM>, and the second 1x1 (or unitary) sub-kernel is processed in a second deep mode cycle using pixel values <NUM>-<NUM> producing an accumulated output for the 1x2 kernel.

<FIG> (which is used to refer to <FIG> and/or <FIG>) is a diagram illustrating convolutional operations concepts associated with a 1x3 kernel in a convolutional accelerator operating in a deep mode of operation according to an embodiment. The 1x2 kernel is decomposed into three 1x1 (or unitary) sub-kernels and the pixel values are strided. The first 1x1 sub-kernel is processed in a first deep mode cycle producing a first partial sum using pixel values <NUM>-<NUM>, the second 1x1 sub-kernel is processed in a second deep mode cycle using pixel values <NUM>-<NUM> producing a second partial sum, and the third 1x1 sub-kernel is processed in a third deep mode cycle using pixel values <NUM>-<NUM> producing an accumulated output for the 1x3 kernel. The concept may be extended to convolutional processing with 1xN kernels (e.g., up to the available memory in the feature line buffer <NUM> for storing kernel depths). Multiple convolutional accelerators may be employed in some embodiments, for example, to process the sub-kernels.

<FIG> is a conceptual diagram illustrating convolutional operations associated with a 1x2 kernel in a convolutional accelerator operating in a deep mode of operation using an adder tree according to an embodiment. As illustrated, two sub-kernels are each convolved with two pixel values in a cycle and the results are combined using an adder tree, which may reduce latency times associated with accumulating the results of processing of the sub-kernels in sequential cycles.

<FIG> illustrates a logical flow diagram generally showing an embodiment of a method <NUM> for controlling a convolutional accelerator in multiple modes of operation, which may be performed, for example, by the convolutional accelerator <NUM> using the mode control circuitry <NUM> as discussed above with reference to <FIG>. For convenience, the method <NUM> will be described with reference to <FIG>.

The method <NUM> starts at <NUM>, and proceeds to <NUM>. At <NUM>, the method <NUM> determines or selects a mode of operation of a convolutional accelerator <NUM> in processing a convolutional layer. This may be done, for example, based on the size of a kernel to be processed by a convolutional accelerator <NUM> in the convolutional layer, the configuration of the convolutional accelerator <NUM> (e.g., the characteristics of the feature line buffer <NUM>, the kernel buffer <NUM>, etc.), and various combinations thereof. For example, if the convolutional accelerator <NUM> is optimized to process 3x3 kernels (e.g., based on the size and configuration of the kernel buffer <NUM> and the clusters of MAC circuits <NUM>), and the kernel to be processed is a 2x2 or larger kernel, a first mode of operation, such as a normal mode of operation, may be selected. On the other hand, if the convolutional accelerator <NUM> is optimized to process 3x3 kernels (e.g., based on the size and configuration of the kernel buffer <NUM>), and the kernel to be processed is a 1x1 (or unitary) to 1xN kernel, where N is an integer greater than or equal to <NUM> (one or unit), a second mode of operation may be selected, such as a deep mode of operation. Other factors may be considered as well, such as the depth of the kernel. The determination of the mode of operation may be made by a host processor, such as a process <NUM> of <FIG>, and stored in a configuration register, such as the configuration register <NUM> of <FIG>. The method <NUM> proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> determines whether the selected mode of operation is a first mode of operation or a second mode of operation, for example, a normal mode of operation or a deep mode of operation, based on the determination at <NUM>. When it is determined at <NUM> that the mode of operation is a first mode of operation, the method <NUM> proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> stores feature data in the feature line buffer <NUM>. This may be done in a generally conventional manner, such as discussed above with reference to <FIG> and <FIG>. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> stores kernel data in the kernel buffer <NUM>. This may be done in a generally conventional manner, such as discussed above with reference to <FIG> and <FIG>. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> transfers feature data from the feature line buffer <NUM> to the MAC clusters <NUM>, and transfers kernel data from the kernel buffer <NUM> to the MAC clusters <NUM>. In other words, the method comprises an operation <NUM> to transfer data to clusters of MAC circuits. This may be done in a generally conventional manner, such as discussed above with reference to <FIG> and <FIG>. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> performs MAC operations using the MAC clusters <NUM> with the feature data and kernel data transferred at <NUM>. This may be done in a generally conventional manner, such as discussed above with reference to <FIG> and <FIG>. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> performs other processing operations associated with the convolutional layer, such as accumulating results of batches of data, serializing output data, returning to <NUM> to process a subsequent batch of data of the convolutional layer, transferring data to and from external memory, etc., and various combinations thereof. This may be done in a generally conventional manner, such as discussed above with reference to <FIG>, <FIG> and <FIG>. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> performs other processing operations associated with the CNN that includes the convolutional layer, such as returning to <NUM> to determine a mode of operation for a next convolutional layer of the CNN, performing pooling operations of a next layer of the CNN, transferring data to and from external memory, etc., and various combinations thereof.

When it is determined at <NUM> that the mode of operation is a second mode of operation, for example, a deep kernel mode of operation, the method <NUM> proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> stores feature data in the kernel buffer <NUM>. This may be done, for example, as discussed above with reference to <FIG> and <FIG>. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> stores kernel data in the feature line buffer <NUM>. This may be done, for example, as discussed above with reference to <FIG>, <FIG> and <FIG>. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> transfers feature data from the kernel buffer <NUM> to the MAC clusters <NUM>, and transfers kernel data from the feature line buffer <NUM> to the MAC clusters <NUM>. In other words, the method comprises an operation <NUM> to transfer data to clusters of MAC circuits. This may be done, for example, as discussed above with reference to <FIG>. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> performs MAC operations using the MAC clusters <NUM> with the feature data and kernel data transferred at <NUM>. This may be done in a generally conventional manner. The method proceeds from <NUM> to <NUM>.

At <NUM>, the method <NUM> performs other processing operations associated with the convolutional layer, such as accumulating results of batches of data, serializing output data (e.g., to account for the order in which feature line and kernel data are processed in <NUM>), returning to <NUM> to process a subsequent batch of data of the convolutional layer (see <FIG>), transferring data to and from external memory, etc., and various combinations thereof. The method proceeds from <NUM> to <NUM>, where, as discussed above, other processing operations associated with the CNN may be performed.

Embodiments of the foregoing processes and methods may contain additional acts not shown in <FIG>, may not contain all of the acts shown in <FIG>, may perform acts shown in <FIG> in various orders, may combine acts, may split acts into separate acts, and may be otherwise modified in various respects. For example, <FIG> may be modified to combine acts <NUM> and <NUM>, to perform acts <NUM> and <NUM> in parallel, to perform acts <NUM> and <NUM> in parallel, to include processing acts to facilitate processing of 1xN kernels in the second mode of operation, where N is an integer greater than <NUM> (see, e.g., <FIG>), etc., and various combinations thereof.

Some embodiments may take the form of or comprise computer program products. For example, according to one embodiment there is provided a computer readable medium comprising a computer program adapted to perform one or more of the methods or functions described above. The medium may be a physical storage medium, such as for example a Read Only Memory (briefly, ROM) chip, or a disk such as a Digital Versatile Disk (briefly, DVD-ROM), Compact Disk (briefly, CD-ROM), a hard disk, a memory, a network, or a portable media article to be read by an appropriate drive or via an appropriate connection, including as encoded in one or more barcodes or other related codes stored on one or more such computer-readable mediums and being readable by an appropriate reader device.

Claim 1:
A convolutional accelerator (<NUM>), comprising:
a feature line buffer (<NUM>);
a kernel buffer (<NUM>) separate from the feature line buffer (<NUM>);
a Multiply-Accumulate, MAC cluster (<NUM>);
a configuration register (<NUM>), and
mode control circuitry (<NUM>) coupled to the feature line buffer (<NUM>), the kernel buffer (<NUM>), and the MAC cluster (<NUM>). wherein the mode control circuitry (<NUM>):
in a first mode of operation (<NUM>, <NUM>, <NUM>, <NUM>) of the convolutional accelerator (<NUM>):
stores feature data (<NUM>) in the feature line buffer (<NUM>);
stores kernel data (<NUM>) in the kernel buffer (<NUM>);
transfers feature data (<NUM>) from the feature line buffer (<NUM>) to the MAC cluster (<NUM>); and
transfers kernel data (<NUM>) from the kernel buffer (<NUM>) to the MAC cluster (<NUM>); and
in a second mode of operation (<NUM>, <NUM>, <NUM>, <NUM>) of the convolutional accelerator (<NUM>):
stores feature data (<NUM>) in the kernel buffer (<NUM>);
stores kernel data (<NUM>) in the feature line buffer (<NUM>);
transfers feature data (<NUM>) from the kernel buffer (<NUM>) to the MAC cluster (<NUM>); and
transfers kernel data (<NUM>) from the feature line buffer (<NUM>) to the MAC cluster (<NUM>),
wherein:
the feature line buffer (<NUM>) is a single port memory;
the kernel buffer (<NUM>) comprises a plurality of dual-port buffers, and
the mode control circuitry (<NUM>), in operation, determines whether to operate in the first mode of operation (<NUM>, <NUM>, <NUM>, <NUM>) or the second mode of operation (<NUM>, <NUM>, <NUM>, <NUM>) based on a configuration parameter stored in the configuration register (<NUM>).