Patent ID: 12236336

Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order. Also, descriptions of features that are known after an understanding of the present disclosure may be omitted for increased clarity and conciseness.

The features described herein may be embodied in different forms, and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.

The terminology used herein is for the purpose of describing particular examples only, and is not to be used to limit the disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term “and/or” includes any one and any combination of any two or more of the associated listed items. As used herein, the terms “include,” “comprise,” and “have” specify the presence of stated features, numbers, operations, elements, components, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, elements, components, and/or combinations thereof. The use of the term “may” herein with respect to an example or embodiment (for example, as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

In addition, terms such as first, second, A, B, (a), (b), and the like may be used herein to describe components. Each of these terminologies is not used to define an essence, order, or sequence of a corresponding component but used merely to distinguish the corresponding component from other component(s).

Throughout the specification, when an element, such as a layer, region, or substrate, is described as being “on,” “connected to,” or “coupled to” another element, it may be directly “on,” “connected to,” or “coupled to” the other element, or there may be one or more other elements intervening therebetween. In contrast, when an element is described as being “directly on,” “directly connected to,” or “directly coupled to” another element, there can be no other elements intervening therebetween. Likewise, expressions, for example, “between”, “directly between,” and “immediately between” and “adjacent to” and “immediately adjacent to” may also be construed as described in the foregoing.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains consistent with and after an understanding of the present disclosure. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Also, in the description of example embodiments, detailed description of structures or functions that are thereby known after an understanding of the disclosure of the present application will be omitted when it is deemed that such description could cause ambiguous interpretation of the example embodiments. Examples will be described in detail with reference to the accompanying drawings, and like reference numerals in the drawings refer to like elements throughout.

Example apparatuses include various types of products or electronic devices such as, for example, a data center, a server, a personal computer, a laptop computer, a tablet computer, a smart phone, a television, a smart home appliance, an intelligent vehicle, a kiosk, and a wearable device, as apparatuses configured to perform deep learning operations.

FIG.1Aillustrates an example of an artificial neural network,FIG.1Billustrates an example of a method of performing deep learning operations using an adder tree structure, andFIG.10illustrates an example of a method of performing deep learning operations using a single instruction multiple data (SIMD) structure including a plurality of multiplier accumulator (MAC) units. The term “unit” described herein references a hardware component or a combination of the hardware component and instructions.

An artificial intelligence (AI) algorithm implementing the deep learning technique may input training data to an artificial neural network to train the artificial neural network with output data and potentially labeled information, for operations of the artificial neural network such as convolution. The trained artificial network may thereafter be used, e.g., to implement such a convolution or other operation(s), to extract features for input information. In the artificial neural network, nodes are connected, e.g., through weighted connections, to each other and collectively operate to process the input data. There are various types of artificial neural networks, for example, a feed-forward artificial neural network, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), restricted Boltzman machine (RBM) method, and the like and as non-limiting examples, and any combination of any two more of such types of artificial neural networks. However, examples are not limited thereto. In a feed-forward artificial neural network, for example, nodes of the artificial neural network have weighted connections or links to other nodes of the artificial neural network. Such links may extend in one direction, for example, in a forward direction, through plural layers of the artificial neural network, where each layer includes multiple nodes and the weighted connections or links are between nodes of different layers. In an RNN example, the artificial neural network may further include such weighted connection or links to the same nodes of the same layer at a different time.

Among such various types of artificial neural networks, a CNN may be used to extract features from input data. For example, the CNN may extract visual features such as edges, lines, colors, and the like from an input image. The CNN may include a plurality of layers, and each layer may receive and process respective input data to generate data to be output. For example, the data output from a layer may be a feature map generated by performing a convolution operation between an image or input feature map and a trained weight of a filter, also referred to as a kernel. In an example, initial layers of the CNN may extract simple features such as edges or gradients from the input, and subsequent layers of the CNN may extract progressively more complex features such as eyes, nose, and the like from the image.

Referring toFIG.1A, a convolution operation110may include a process of generating a 6×6 single-channel output feature map115by performing multiply and add operations between an 8×8 three-channel input feature map and a 3×3 three-channel filter113. The size of data may be defined by (width, height) and the number of channels. For example, this size of the output feature map115may also be referred to as a volume.

A depth-wise convolution operation120may perform a convolution operation only within the same channel, and extract a spatial feature of each channel through this. The depth-wise convolution operation120may include a process of generating a 6×6 three-channel output feature map130including output feature maps127,128, and129corresponding to respective input channels, by performing, for each input channel, a convolution operation between the 8×8 three-channel input feature map111and each of three 3×3 filters124,125, and126. In detail, the output feature map127of the first output channel may be generated by performing multiply and add operations between an input feature map121of the first input channel and the first filter124. Similarly, the output feature map128of the second output channel may be generated by performing multiply and add operations between an input feature map122of the second input channel and the second filter125, and the output feature map129of the third output channel may be generated by performing multiply and add operations between an input feature map123of the third input channel and the third filter126.

Referring toFIG.1B, an apparatus, configured to perform deep learning operations, may use an adder tree structure that can be driven with low power when performing a convolution operation. The adder tree structure may include a plurality of multipliers configured to calculate a product of two items of data, adders configured to calculate the sum of outputs of two adjacent multipliers or the sum of two adjacent adders, and an accumulator configured to calculate the cumulative sum of final output data. The adder tree structure using only a small number of accumulators (for example, a single accumulator) may perform a convolution operation with low power.

For example, the adder tree structure may perform a convolution operation between a 5×5 four-channel input feature map140and four 3×3 filters150, e.g., specifically between nine items of data 0, 1, 2, 5, 6, 7, 10, 11, and 12 of an input feature map141and weights 0 to 8 of a first filter151. The multipliers of the adder tree structure may calculate products of the data of the input feature map141of the first input channel and the weights of the first filter151, and the adders of the adder tree structure may accumulate and add the output values of the multipliers, that is, 0×0, 1×1, 2×2, 5×3, 6×4, 7×5, 10×6, 11×7, and 12×8.

Since a general convolution operation accumulates and adds output values of input feature maps of each input channel, the multipliers160that are not used for the convolution operation between the input feature map141and the filter151may be used to perform a convolution operation on an input feature map of another input channel. However, the depth-wise convolution operation performs a convolution operation for each input feature map of each input channel. Thus, when a depth-wise convolution operation is performed using the adder tree structure, the multipliers160may not all be used, resulting in a decrease in resource utilization.

In addition, since the convolution operation using the adder tree structure has a long data path, the apparatus may operate at a low clock frequency. Accordingly, the adder tree structure is suitable for performing general convolution operations but may not typically be suitable for operations for parallel processing of data such as depth-wise convolution operations. Furthermore, when the adder tree structure is used to perform an asymmetric convolution operation with an asymmetric filter such as a 7×1, 1×7, 3×1, or 1×3 filter, the resource utilization efficiency may also further decrease.

Referring toFIG.10, an apparatus, configured to perform deep learning operations, may perform a depth-wise convolution operation using an included SIMD structure including a plurality of MAC units.

The SIMD structure includes many processing elements (PEs)191,192,193, and194configured to perform the same operation, for example, and thus many operations may be performed simultaneously by inputting data into each of the PEs. Each PE of the SIMD structure may be configured as a MAC unit to perform an operation of ACC=ACC+(A×B).

In the SIMD structure, each MAC unit includes an ACC accumulator. Thus, it may be suitable for performing a depth-wise convolution operation of performing a convolution operation for each channel. For example, each PE191,192,193,194of the SIMD structure may perform a depth-wise convolution operation respectively between a 5×5 four-channel input feature map171,172,173,174and a 3×3 filter181,182,183,184. In detail, the PE191may perform a convolution operation between the input feature map171and the filter181, the PE192may perform a convolution operation between the input feature map172and the filter182, the PE193may perform a convolution operation between the input feature map173and the filter183, and the PE194may perform a convolution operation between the input feature map174and the filter184. In addition, the SIMD structure may have a short data path for operation and thus, may operate at a high clock frequency.

However, such a SIMD structure may require an accumulator for every MAC unit and thus, result in greater power consumption. For example, while a single accumulator is sufficient to perform a general convolution operation using the adder tree structure, when performing a general convolution operation using the SIMD structure all accumulators may operate, resulting in a decrease in power efficiency with the SIMD structure compared to the adder tree structure. Accordingly, while the SIMD structure is suitable for operations for parallel processing of data such as depth-wise convolution operations, the SIMD structure may not be as suitable for performing general convolution operations.

As described above, the adder tree structure may be more suitable for performing general convolution operations, but may not be as suitable for asymmetric convolution operations using asymmetric filters and operations for parallel processing of data such as depth-wise convolution operations, and conversely, the SIMD structure may be more suitable for operations for parallel processing of data such as depth-wise convolution operations, but may not be as suitable for performing general convolution operations.

FIG.2illustrates an example of a method of performing deep learning operations according to one or more embodiments.

Referring toFIG.2, operations210and220may be performed by the apparatus configured to perform deep learning operations as described above and below. The apparatus may be implemented by one or more hardware modules, though examples also include the apparatus implementing the deep learning operations using various combinations of hardware and instructions implemented or executed by the hardware.

In operation210, the apparatus receives an operation mode and input data. The apparatus may be implemented to select an operation mode, where the selectable operation modes may include an adder tree mode, a systolic adder tree mode, a SIMD mode, and/or a systolic mode.

In operation220, the apparatus may control operations of MAC units included in the systolic array and data movements between the MAC units in response to the selected operation mode. The apparatus may perform an operation corresponding to the received/selected operation mode. For example, the apparatus may be selected to operate in an adder tree mode to perform a general convolution operation and may alternatively be selected to operate in a SIMD mode to perform a depth-wise convolution operation.

Specifically, the apparatus may control the operations of the MAC units included in the systolic array so as to operate in a mode determined most suitable for a predetermined operation. For example, the apparatus may control the operations of the MAC units so as to use only an accumulator of any one of the MAC units included in the systolic array and perform a general convolution operation. In the SIMD mode, the apparatus may control the operations of the MAC units so as to respectively use accumulators of all the MAC units included in the systolic array and perform an operation for parallel processing of data such as a depth-wise convolution operation.

In addition, the apparatus may control the data movements between the MAC units included in the systolic array. For example, in the adder tree mode, the apparatus may control data movements between MAC units so as to receive an operation result from an adjacent MAC unit.

Although it will be described in greater detail below, in a systolic adder tree mode, the apparatus may perform an operation of the adder tree mode, while controlling the data movements between the MAC units such that input data of the systolic array are transferred to MAC units arranged along a column in a pipelining manner. In addition, in a systolic mode, the apparatus may control the data movements between the MAC units so as to transfer the input data of the systolic array to the MAC units along a row or column.

As described herein, the apparatus may support a plurality of operation modes and perform an operation determined or predetermined suitable for a corresponding operation mode.

For example, as described in greater detail below, the apparatus may support a plurality of operation modes using a systolic array of a structure in which inputs and/or outputs of adjacent MAC units are pipelined. Thus, such an apparatus may include technology for achieving both high operation processing speed and high power efficiency.

FIG.3illustrates an example of a structure of an apparatus configured to perform deep learning operations.

Referring toFIG.3, an apparatus, configured to perform deep learning operations, includes a processor300configured to support a plurality of operation modes. The processor300may include a plurality of PEs, and the PEs may be respectively configured as MAC units310. The processor300may have a cell array structure in which cells corresponding to the MAC units310are arranged in an array structure, and in particular, may have a systolic array structure in which adjacent MAC units310are connected to each other.

The processor300may include a control circuit320configured to control operations of the MAC units310included in the systolic array and data movements between the MAC units310in response to the plurality of operation modes. The control circuit320may control the MAC units310through a control path. However, the apparatus shown inFIG.3is only described as an example, and specific structures such as the number of MAC units310and the control path may vary in different examples.

The apparatus may support the plurality of operation modes using the control circuit320. The control circuit320may control operations of accumulators of the MAC units310included in the systolic array in response to the plurality of operation modes. For example, the control circuit320may control the operations of the MAC units so as to operate only an accumulator of one of the MAC units310in response to an adder tree mode. The operation of the apparatus in the adder tree mode will be described in detail below with reference to an apparatus410ofFIG.4A. In addition, the control circuit320may control the operations of the MAC units so as to operate accumulators of all the MAC units310included in the systolic array in response to a SIMD mode. The operation of the apparatus in the SIMD mode will be described in detail below with reference to an apparatus420ofFIG.4A.

FIGS.4A to4Cillustrate examples of apparatuses, configured to perform deep learning operations, with an adder tree mode and a SIMD mode.

Referring toFIG.4A, an apparatus400, configured to perform deep learning operations, may operate in an adder tree mode or a SIMD mode. For example, the apparatus400may operate in the adder tree mode when a control signal “0” is input and operate in the SIMD mode when a control signal “1” is input.

A control circuit may include multiplexers (MUX) disposed in MAC units to determine inputs of adders. The apparatus400may control data movements between the MAC units using the multiplexers. For example, the multiplexers may be respectively disposed in the MAC units, or may be disposed only in a portion of the MAC units. In examples where the multiplexers are disposed only in a portion of the MAC units, a predetermined multiplexer may control a plurality of MAC units together.

The apparatus400ofFIG.4Amay include a plurality of PEs401to408. Each of the PEs401to408may include a MAC unit and a control circuit and have a different control circuit depending on the type of the PE. Types of processing arrays that may make up the array ofFIG.4Awill be described with reference toFIG.4B.

Referring toFIG.4B, the apparatus400may include a combination of a first PE430, a second PE440, and/or a third PE450.

The first PE430may include a first multiplier431, a first adder432, and a first accumulator435making up a MAC unit, and a first multiplexer433and a first AND gate434making up a control circuit.

The first multiplexer433may determine one of an output of the first accumulator435or an output of a neighboring PE to be an input of the first adder432. In an adder tree mode (for example, when a control signal “0” is input), the first multiplexer433may determine the output of the neighboring PE to be the input of the first adder432. In a SIMD mode (for example, when a control signal “1” is input), the first multiplexer433may determine the output of the first accumulator435to be the input of the first adder432.

The second PE440may be a PE positioned at the edge, e.g., an edge of the example array of PEs, and may include a second multiplier441, a second adder442, and a second accumulator445making up a MAC unit, and a second multiplexer443and a second AND gate444making up a control circuit.

The second multiplexer443may determine one between a predetermined value (for example, “0”) and an output of the second multiplier441, to be an input of the second adder442. In the adder tree mode (for example, when the control signal “0” is input), the second multiplexer443may determine the predetermined value (for example, “0”) to be the input of the second adder442. In the SIMD mode (for example, when the control signal “1” is input), the second multiplexer443may determine the output of the second multiplier441to be the input of the second adder442.

The third PE450is a PE configured to output a final output value in the adder tree mode, and may include a third multiplier451, a third adder457, and a third accumulator456making up a MAC unit, and a fourth adder452, a third multiplexer453, a fourth multiplexer454, a third AND gate455, a fifth adder458, and a fifth multiplexer459making up a control circuit.

The fifth multiplexer459may determine one between an output of the third adder457and the sum of the output of the third adder457and outputs of neighboring PEs, to be an input of the third accumulator456.

In the adder tree mode (for example, when the control signal “0” is input), the outputs of the respectively neighboring PEs may be input through the third multiplexer453and the fourth multiplexer454, and the fourth adder452may add the outputs of the neighboring PEs. In addition, the third adder457may add the output of the third multiplier451and the output of the third accumulator456. The fifth multiplexer459may determine an output of the fifth adder458, which adds an output of the third adder457and an output of the fourth adder452, to be an input of the third accumulator456. In the SIMD mode (for example, when the control signal “1” is input), the fifth multiplexer459may determine an output of the third adder457to be the input of the third accumulator456.

Referring back toFIG.4A, the PEs401and408of the apparatus400may have a structure of the second PE440ofFIG.4B, the PEs402,403,405,406, and407may have a structure of the first PE430ofFIG.4B, and the PE404may have a structure of the third PE440ofFIG.4B.

The apparatus410shows an example of operating in an adder tree mode when the control signal “0” is input. As described above, when the control signal “0” is input, only an accumulator of a MAC unit included in the PE404may operate, and accumulators of MAC units included in the remaining PEs may not operate. Adders of MAC units included in the PEs401and408may also not operate. By limiting the operations of the accumulators and the adders as described above, the adder tree mode may reduce power consumption. Also, multiplexers included in control circuits of the PEs402to407may receive data from neighboring MAC units and perform operations as an adder tree. For example, the apparatus410in the adder tree mode may perform an operation the same as that represented by the adder tree ofFIG.4C.

The apparatus420ofFIG.4Ashows an example of operating in a SIMD mode when the control signal “1” is input. As described above, when the control signal “1” is input, the accumulators of the respective MAC units included in all of the PEs401to408may operate. Meanwhile, adders of the control circuit included in the PE404may not operate. For example, the operations of adders not in use may be limited by controlling an output value of a multiplexer connected to inputs of the adders not in use to be “0”, and thereby may reduce power consumption.

FIGS.5A to5Cillustrate examples of apparatuses configured to perform deep learning operations.

When the apparatuses described with reference toFIGS.4A to4Coperate in an adder tree mode, the distances from the PE404to the PEs401and408positioned at the edges may be relatively far, and thus the data path may pass through many adders and multiplexers. Thus, it may be difficult to achieve a high timing.

The apparatuses shown inFIGS.5A to5Cmay perform an adder tree mode operation at a high speed by separately designing an adder tree not to pass through a multiplexer.

For example, the apparatus ofFIG.5Amay include a systolic array and an adder tree520configured to share multipliers of MAC units included in the systolic array.

For ease of description, an operation of a PE510among a plurality of PEs will be described. When a control signal “0” is input, an output of a multiplier511may be transferred to an adder515of the adder tree520through a second multiplexer514, rather than being transferred to an adder513of the MAC unit through a first multiplexer512. Through this, when the control signal “0” is input, the apparatus may operate in an adder tree mode.

If a control signal “1” is input, the output of the multiplier511may be transferred to the adder513of the MAC unit through the first multiplexer512, and may not be transferred to the adder515of the adder tree520through the second multiplexer514. Through this, when the control signal “1” is input, the apparatus may operate in a SIMD mode.

Referring toFIG.5B, the apparatus may include a systolic array, and an adder tree550configured to share multipliers and accumulators of MAC units included in the systolic array.

For ease of description, an operation of a PE530among a plurality of PEs will be described. When a control signal “0” is input, an output of a multiplier531may be transferred to an adder535of the adder tree550through a second multiplexer534, rather than being transferred to an adder533of the MAC unit through a first multiplexer532. The final output of the adder tree550may be transferred to an accumulator542of the MAC unit through the first multiplexer541of the PE540. Through this, when the control signal “0” is input, the apparatus may operate in an adder tree mode.

If a control signal “1” is input, the output of the multiplier531may be transferred to the adder533of the MAC unit through the first multiplexer532, and may not be transferred to the adder535of the adder tree550through the second multiplexer534. Through this, when the control signal “1” is input, the apparatus may operate in a SIMD mode.

Referring toFIG.5C, the apparatus may have a structure of an adder tree further including flip-flops560and565. The flip-flops560and565may be disposed between multipliers included in the systolic array and an accumulator for the adder tree. The apparatus may configure a synchronous circuit by adding the flip-flops560and565, thereby increasing an operating frequency in an adder tree mode. A flip-flop continuously outputs information input at an edge (for example, a rising edge or a falling edge) of a clock signal until an edge of a subsequent clock signal. When the same clock signal is applied to the flip-flops, the apparatus may operate in synchronization with the clock signal. A synchronous circuit is generally robust against a transfer delay or a circuit delay when compared to an asynchronous circuit and thus, may operate at a higher operating frequency.

FIGS.6A and6Billustrate an example of an apparatus, configured to perform deep learning operations, with a systolic adder tree mode and a SIMD mode. The description ofFIGS.1to5Cis also applicable to the following drawings, and thus a duplicated description will be omitted.

Referring toFIG.6A, an apparatus may selectively operate in a systolic adder tree mode or a SIMD mode. For example, the apparatus may operate in the systolic adder tree mode when a control signal “0” is input and operate in the SIMD mode when a control signal “1” is input.

The apparatus shown inFIG.6Afurther includes a predetermined configuration based on the apparatus shown inFIG.4Aand may provide a systolic adder tree mode through the additional configuration. A control circuit may thus further include a plurality of multiplexers611and621and data paths612to transfer input data of an upper-end systolic array600to a lower-end systolic array650as operating in the systolic adder tree mode. The control circuit may control operations of accumulators of MAC units included in the systolic array and data movements between the MAC units in respective responses to a plurality of operation modes. For example, the control circuit may control the operations of the MAC units so as to operate only an accumulator of one of the MAC units included in the systolic array in response to a systolic adder tree mode. Further, the apparatus may multiplex outputs of accumulators and output the multiplexed outputs through a shift register (SFT). Through this structure, the number of output ports may be reduced.

The systolic adder tree mode may perform the same operation as an adder tree mode, but differs in a data input method. In the systolic adder tree mode, the systolic array is arranged in the form of a 2D array, such that input data (for example, weights) of the upper-end systolic array may be transferred to the lower-end systolic array along respective columns of the arranged PE units, e.g., using the respective data paths112. For ease of description, PEs610and620among a plurality of PEs will be described. The PE620may further include a multiplexer621compared to the PE401ofFIG.4A. When the control signal “0” is input, input data B of the PE610are shared or transferred to the PE620through the data path612, and the multiplexer621may apply the input data B received through the data path612to a multiplier of the PE620. The PEs other than the PEs610and620may also operate as described above.

For example, it may be understood that in the systolic adder tree mode, the apparatus may operate as in the structure ofFIG.6B. Referring toFIG.6B, it may be learned that input data of the upper-end systolic array600are transferred to the lower-end systolic array650when a control signal “0” is input.

Conversely, when a control signal “1” is input, the multiplexer621of the PE620may select separate input data (e.g., other weights) rather than selecting the input data (the same weights) received through the data path612as an input, and thus the input data B of the PE610may not be transferred to the PE620. Accordingly, when the control signal “1” is input, the apparatus may operate in a SIMD mode in which data are not moved between systolic arrays. The PEs other than the PEs610and620may also operate as described above.

FIGS.7A and7Billustrate an example of an apparatus, configured to perform deep learning operations, with a systolic adder tree mode and a systolic mode. The description ofFIGS.1to6Bis also applicable to the following drawings, and thus a duplicated description will be omitted.

The apparatus ofFIG.7Amay selectively operate in a systolic adder tree mode or a systolic mode. For example, the apparatus may operate in the systolic adder tree mode when a control signal “0” is input and operate in the systolic mode when a control signal “1” is input.

The apparatus shown inFIG.7Afurther includes a predetermined configuration based on the apparatus shown inFIG.6Aand may provide a systolic mode through an additional configuration, e.g., rather than the SIMD mode ofFIG.6A. For example, the control circuit may further include a plurality of multiplexers and data paths to transfer input data of a systolic array in the illustrated row direction as operating in the systolic mode.

The control circuit may control operations of accumulators of MAC units included in the systolic array and data movements between the MAC units in select response to a plurality of operation modes. For example, the control circuit may control the operations of the MAC units so as to operate accumulators of all the MAC units included in the systolic array in response to the systolic mode. Further, the apparatus may multiplex outputs of accumulators and output the multiplexed outputs through an SFT. Through this structure, the number of output ports may be reduced.

The systolic mode ofFIG.7Adiffers from a SIMD mode ofFIG.6Ain a data input method. In the systolic mode, the input data of the systolic array may be transferred to the MAC units along rows and/or columns. For example, when the systolic array is arranged in the form of a 2D array, first input data (for example, weights) may be transferred from the upper-end systolic array to the lower-end systolic array along the columns, and second input data (for example, data values of an input feature map) may be transferred to neighboring MAC units of each of the upper-end and lower-end systolic arrays along the rows. For ease of description, PEs710and720among a plurality of PEs will be described. The PE720may further include a multiplexer712compared to the PE610ofFIG.6A. When the control signal “1” is input, input data A of the PE710are shared or transferred to the PE720through the data path711, and the multiplexer712may apply the input data A received through the data path711as an input of the multiplier of PE720. The PEs other than the PEs710and720may also operate as described above.

Referring toFIG.7B, the apparatus may perform a matrix-vector multiply operation using the systolic mode. For example, in the systolic mode, the MAC units may respectively receive matrix data730at shown timings (for example, t1 to t8) through input terminals B. In addition, the control circuit may control vector data740to be transferred to neighboring MAC units of each of the upper-end and lower-end systolic arrays along their respective rows at shown timings (for example, t1 to t8) in response to the systolic mode. Through this, the MAC units may respectively receive the vector data at the shown timings (for example, t1 to t8) through input terminals A.

The control circuit may control operations of the MAC units so as to operate accumulators of all the MAC units included in the systolic array in response to the systolic mode. Through this, the MAC units may respectively perform matrix-vector multiply operations by accumulating the product of corresponding matrix data730and vector data740each time.

FIG.8illustrates an example of an apparatus, configured to perform deep learning operations, with a systolic adder tree mode, a SIMD mode, and a systolic mode. The description ofFIGS.1to7Bis also applicable to the following drawings, and thus a duplicated description will be omitted.

The apparatus ofFIG.8may operate in a select one of a systolic adder tree mode, a SIMD mode, and a systolic mode. For example, the apparatus may operate in the systolic adder tree mode when a control signal “0” is input, operate in the SIMD mode when a control signal “1” is input, and operate in the systolic mode when a control signal “2” is input.

For ease of description, PEs810,820, and830among a plurality of PEs will be described. For example, as demonstrated inFIG.8, the control signal may be input to multiplexers811and816of PE810, multiplexer813of PE820, and multiplexer815of PE830.

When the control signal “0” is input, input data entering through an input terminal B of the PE810may be shared or transferred to the PE830through a data path814, with the multiplexer815of the PE830selecting (according to the control signal “0”) the input data received through the data path814as an input of an input terminal B of the PE830. Also when the control signal “0” is input, the multiplexer816of the PE810may not share or transfer an output of the multiplier of the PE810to the adder of the PE810and thus, may not operate in the systolic mode. Therefore, when the control signal “0” is input, the apparatus may operate in the systolic adder tree mode in which input data (for example, weights) of an upper-end systolic array are transferred to a lower-end systolic array along columns. The PEs other than the PEs810and830may also operate as described above.

When the control signal “2” is input, the input data entering through an input terminal A of the PE810are shared or transferred to the PE820through a data path812, with the multiplexer813of the PE820selecting (according to the control signal “2”) to select the input data received through the data path812as an input of an input terminal A of the PE820. Accordingly, the apparatus may operate in the systolic mode when the control signal “2” is input.

Conversely, when the control signal “1” is input, the multiplexer813of the PE820may select separate input data for the input terminal A of the PE820, rather than selecting the input data received through the data path812as an input, and thus the input data of the input terminal A of the PE810may not be transferred to be the input data of the input terminal A of the PE820. Accordingly, when the control signal “1” is input, the apparatus may operate in the SIMD mode in which data are not moved between systolic arrays. The PEs other than the PEs810and820may also operate as described above.

FIGS.9A and9Billustrate an example of an apparatus, configured to perform deep learning operations, with a systolic adder tree mode and a plurality of SIMD modes. The description ofFIGS.1to8is also applicable to the following drawings, and thus a duplicated description will be omitted.

The apparatus ofFIG.9Amay operate in one of a systolic adder tree mode and a plurality of SIMD modes. For example, the apparatus may operate in the systolic adder tree mode when a control signal “0” is input, operate in a SIMD mode to perform an elementwise add operation when a control signal “1” is input, operate in a SIMD mode to perform a matrix-vector multiply operation when a control signal “2” is input, and operate in a SIMD mode to perform a depth-wise convolution operation when a control signal “3” is input.

The apparatus shown inFIG.9Afurther includes a predetermined configuration based on the apparatus shown inFIG.6Aand may provide a plurality of SIMD modes, rather than a single SIMD mode, through the additional configuration.

When the control signal “2” is input, the apparatus may perform the matrix-vector multiply operation as shown inFIG.9B. The apparatus may also be suitable for a recurrent neural network (RNN) where matrix-vector operations are frequently used, and may process data while minimizing time delay even when the batch size increases.

Referring toFIG.9A, a multiplexer911may output a select one of a weight and a weight received from a systolic array of another row based on the control signal. In response to the control signal, a multiplexer912may selectively output one of “0”, an output of the multiplier of the PE910, and input data received through an input terminal B of the PE910. The multiplexer912may operate to perform an elementwise add operation of the input data by outputting the input data received through the input terminal B of the PE910to the adder of the PE910. In response to the control signal, a multiplexer913may selectively output one of input data received through the input terminal A of PE910, a “0”, and an output of the accumulator (ACC) of the PE910. In response to the control signal, a multiplexer914may selectively output one of the output of the ACC of the PE910, the output of the adder of the PE910, and the output of the multiplier of the PE910.

FIG.10illustrates an example of an apparatus, configured to perform deep learning operations, with a systolic adder tree mode, a plurality of SIMD modes, and a systolic mode. The description ofFIGS.1to9Bis also applicable to the following drawings, and thus a duplicated description will be omitted.

The apparatus ofFIG.10may operate in a select one of a systolic adder tree mode, a plurality of SIMD modes, and a systolic mode. For example, the apparatus may operate in the systolic adder tree mode when a control signal “0” is input, operate in a SIMD mode to perform an elementwise add operation when a control signal “1” is input, operate in a SIMD mode to perform a matrix-vector multiply operation when a control signal “2” is input, operate in a SIMD mode to perform a depth-wise convolution operation when a control signal “3” is input, and operate in a systolic mode when a control signal “4” is input.

The apparatus shown inFIG.10further includes a predetermined configuration based on the apparatus shown inFIG.9Aand may thus further provide a systolic mode through the additional configuration.

The apparatus may use multiplexers to reduce shift registers at an output. For example, in the element add operation mode and the matrix-vector multiply operation mode, one output may need to be extracted in one cycle. If a clock frequency applied to the shift registers is increased to N times a frequency applied to the MAC units, and N:1 multiplexers are used, one output may be moved in one cycle.

The systolic mode may be used for applications desiring input-stationary, weight-stationary, output-stationary, or various combinations thereof. When a depth-wise convolution operation is processed in the systolic mode, high MAC utilization may also be achieved.

The systolic adder tree mode may be driven with lower power compared to the systolic mode. In addition, the SIMD mode has higher MAC utilization compared to the systolic mode, and may achieve relatively high MAC utilization in a depth-wise convolution operation.

When the size of a filter (for example, width (w)*height (h)*the number of channels (c)) is larger than the horizontal/vertical length of MAC units, it may be effective to operate in the systolic mode. On the other hand, a time of h*w*c*2 may be consumed to move the output to the shift register, and thus it may not be as effective when the size of MAC units is relatively large. Accordingly, it may be effective for the apparatus shown inFIG.10to operate in the systolic mode at a first layer of a neural network. However, power efficiency may rather decrease at the remaining layers due to greater power consumption of the MAC array.

The apparatus shown inFIG.9Amay have better power efficiency than the apparatus shown inFIG.10.

The apparatus shown inFIG.9Amay have a relatively high overall processing rate since the MAC units may process elementwise add and matrix-vector multiply operations. However, since the elementwise add operation may desire a higher bandwidth, a high clock frequency may be desired to extract an output, and thus, an example may be provided with an additional multiplexer inserted for the elementwise add operation. Accordingly, the apparatus shown inFIG.9Amay thus have increased power consumption due to the inserted multiplexer in such an additional example.

The apparatus shown inFIG.7Amay have better power efficiency than the apparatus shown inFIG.9Asince the configuration for providing a plurality of SIMD modes is not illustrated in the apparatus shown inFIG.9A. The apparatus shown inFIG.7Amay receive input data differently for each MAC unit and thus, may have relatively higher MAC utilization compared to a pure adder tree structure. In addition, the apparatus shown inFIG.7Amay reduce the time for filling the MAC units with data compared to the systolic mode and thus, may have higher MAC utilization.

FIG.11Aillustrates an example of a systolic array arranged in the form of a three-dimensional (3D) array, andFIG.11Billustrates an example of an apparatus configured to perform deep learning operations, as a convolution operation, a matrix-vector multiply operation, and a matrix-matrix multiply operation, for example. InFIGS.11A and11B, locations of an input feature map (or Activation, ACT, IFM) memory and a weight memory may vary in different examples.

The apparatus ofFIG.11Amay improve processing rates of a convolution operation, a matrix-vector multiply operation, and a matrix-matrix multiply operation by arranging a systolic array in the example 3D form.

The apparatus ofFIG.11Bmay further include a direct memory access (DMA)1110, a controller1120, an SRAM cluster1130, and a normalized lattice filter (NLF)1140, for example. The apparatus may apply data to a desired row or column using a device capable of moving data such as the DMA1110or a central processing unit (CPU), and read output results. The processor ofFIG.11Cbelow may be an example of the CPU. Furthermore, the apparatus may connect outputs of two or more rows or columns to add or accumulate result values of several rows or columns.

FIG.11Cis a diagram illustrating an example electronic apparatus. Herein, any of the apparatuses ofFIGS.1A to11Cmay also be referred to as a deep learning apparatus or devices, with respective configurations for deep learning capabilities, of training and/or inference operations.

An electronic apparatus1100may be representative of any, any combination, or all of the apparatuses, configured for deep learning operations, described above with respect toFIG.1AthroughFIG.11B. In another example, a neural processor1150ofFIG.11Cmay represent any, any combination, or all of the apparatuses described above with respect toFIG.1AthroughFIG.11B. As non-limiting examples, the electronic apparatus1100may be any of a data center, a server, a personal computer, a laptop computer, a tablet computer, a smart phone, a television, a smart home appliance, an intelligent vehicle, a kiosk, or a wearable device, in various respective examples.

Referring toFIG.11, the electronic apparatus1100may include a processor1160, the neural processor unit (NPU)1150, a memory1165, a communication device1170, a storage device1175, a communication bus1180, an input device1185, and an output device1190.

The processor1160may control an overall operation of the electronic apparatus1100, and execute functions and instructions in the electronic apparatus1100. For example, the processor1160may be or include a CPU. The processor1160may be configured to interact with the NPU1150to perform one or more operations or methods described above with reference toFIGS.1through11B, for example. In an example, the NPU1150may be configured to perform one or more, or all, of the operations or methods described above with reference toFIGS.1through11Bbased on input/activation information from the processor1160and kernel weights from the memory1165, as a non-limiting example. Another example exists without the NPU1150, and the processor1160may correspond to the processors or apparatuses described herein and be configured to perform one or more operations or methods described above with reference toFIGS.1through11B, for example.

The memory1165may store information for the processor1160and/or the NPU1150to perform various training or trained operational objectives, i.e., the deep learning operations described herein refer to examples of inference operations using trained weights and/or examples of training operations that generate one or more or all of the trained weights through iterative operation. The memory1165may also store instructions to be executed by the processor1160and/or NPU1150, and store related information during the execution of software or an application in the electronic apparatus1100. The memory1165may include, for example, a random-access memory (RAM), a dynamic RAM (DRAM), a static RAM (SRAM), or other types of nonvolatile memory that are well-known in the related technical field.

The storage device1175may include a computer-readable storage medium or a computer-readable storage device. The storage device1175may store a greater amount of information for a longer period of time compared to the memory1165. The storage device1175may include, for example, a magnetic hard disk, an optical disc, a flash memory, a floppy disk, an electrically erasable programmable read-only memory (EEPROM), and other types of nonvolatile memory that are well-known in the related technical field.

The input device1185may receive an input from a user, for example. The input device1185may include, for example, a keyboard, a mouse, a touchscreen, a camera, a microphone, and other devices that may detect the input from the user.

The output device1190may provide an output of the electronic apparatus1100, e.g., to a user through a visual, auditory, or tactile channel based on output of any of the apparatuses described herein with respect toFIGS.1A to11B. The output device1190may include a display, a touchscreen, a speaker, and other devices that may provide the output to the user.

The communication device1170may communicate with an external device through a wired or wireless network. The communication device1170may receive and transmit data or information from and to an external device. The communication bus1180may provide communication between such components of the electronic apparatus1100.

The DMAs1110, the controller1120, the SRAM clusters1130, the NLF1140, processors, the PE units, the MAC units, accumulators, the multiplexers, the adders, the multipliers, the flip flops, the gates, the shift registers, timing clock, two-dimensional arrays, three-dimensional arrays, the electronic apparatus1100, the processor1160, the NPU1150, the memory1165, the storage device1175, the communication device1170, the communication bus1180, the input device1185, and the output device1190, and other apparatuses, devices, units, modules, and components described herein with respect toFIGS.1A through11Care implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller or unit, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term “processor” or “computer” may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated inFIGS.1A through11Cthat perform the operations described in this application may be performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

Instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler. In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computer using an interpreter. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions used herein, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.

The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computers.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents.