Patent Description:
Neural networks refer to computational architectures modeling biological brains. With the development of neural network technology, neural networks are used in various types of electronic systems to analyze input data and extract valid information.

Research has been actively conducted into a hardware accelerator for efficiently using a deep neural network (DNN) at low power. A neural network processing apparatus requires a large amount of operations on complex input data.

Particularly, in a device implemented with low power and low performance, a technology capable of efficiently processing an operation on a neural network is required to extract desired information by analyzing a large amount of input data in real time by using the neural network. <NPL>, is a paper on multihead attention introducing a model architecture, called Transformer, which relies entirely on an attention mechanism to draw global dependencies between input and output. The Transformer provides an alternative to recurrent layers most commonly used in encoder-decoder architectures with multiheaded self-attention. The multihead attention is computed using a number of matrix multiplications and transpose operations. <NPL>, proposes a method to improve the information aggregation for multihead attention using a routing-by-agreement algorithm. Specifically, the routing algorithm iteratively updates the proportion of how much a part (i.e. the distinct information learned from a specific subspace) should be assigned to a whole (i.e. the final output representation), based on the agreement between parts and wholes.

It is the object of the present invention to provide an improved solution for obtaining multihead self-attention.

Apparatuses and methods of performing a matrix multiplication operation of a neural network. Computer-readable recording mediums that store a program that, when executed by a computer, performs the methods.

In one general aspect, a neural network apparatus includes: a memory having at least one program stored therein; and a processor to perform one or more operations by executing the at least one program, wherein the processor acquires an input feature map and an initial weight from the memory, determines whether to divide the initial weight in a column direction or a row direction according to whether a reshape operation and a transpose operation are performed before or after a matrix multiplication operation, generates division weights by dividing the initial weight by a head count in the determined column direction or row direction, generates intermediate feature maps by performing a matrix multiplication operation between the input feature map and the division weights, and generates a final feature map based on the intermediate feature maps.

The processor may generate the division weights by dividing the initial weight by a head count in the column direction of the initial weight when the reshape operation and the transpose operation are performed after the matrix multiplication operation, and generate the final feature map by concatenating the intermediate feature maps.

The processor may generate the division weights by dividing the initial weight by a head count in the row direction of the initial weight when the reshape operation and the transpose operation are performed before the matrix multiplication operation, and generate the final feature map through an element-wise sum of the intermediate feature maps.

The matrix multiplication operation between the input feature map and the plurality of division weights may be one of a one-dimensional convolution operation and a two-dimensional convolution operation.

The processor may include a weight divider, and the weight divider may divide the initial weight by the head count in one of the column direction and the row direction.

In another general aspect, a method includes: acquiring an input feature map and an initial weight from a memory; determining whether to divide the initial weight in one of a column direction or a row direction according to whether a reshape operation and a transpose operation are performed before or after a matrix multiplication operation; generating division weights by dividing the initial weight by a head count in the determined column direction or row direction; generating intermediate feature maps by performing the matrix multiplication operation between the input feature map and the division weights; and generating a final feature map based on the intermediate feature maps.

In another general aspect, a method includes: receiving an initial feature map and an initial weight; dividing the initial weight into division weights; performing a matrix multiplication operation between the input feature map and each of the division weights to generate intermediate feature maps; and manipulating the intermediate feature maps to generate an output feature map.

The method may include determining whether the input feature map has been subjected to a reshape operation and a transpose operation.

In a case in which the input feature map has been subjected to the reshape operation and the transpose operation, the initial weight may be divided into the division weights based on a head count of the initial weight in a row direction.

The method may include generating the output feature map as an element-wise sum of the intermediate feature maps.

In a case in which the input feature map has not been subjected to the reshape operation and the transpose operation, the initial weight may be divided into the division weights based on a head count of the initial weight in a column direction.

The method may include generating the output feature map by concatenating the intermediate feature maps.

In another general aspect, a computer-readable recording medium stores a program that, when executed by a computer, performs one or more of the methods.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein. The use of the term "may" herein with respect to an example or embodiment (e.g., as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

Some examples may be represented in terms of functional block components and various processing operations. Some or all of these functional blocks may be implemented by any number of hardware and/or software components that execute particular functions. For example, the functional blocks may be implemented by one or more microprocessors or may be implemented by circuit components for a certain function. Also, for example, the functional blocks may be implemented in various programming or scripting languages. The functional blocks may be implemented by an algorithm that is executed in one or more processors. Terms such as "mechanism," "element," "unit," and "configuration" may be used in a broad sense and are not limited to mechanical and physical configurations.

Also, connection members or connection lines between elements illustrated in the drawings merely represent examples of functional connections and/or physical or logical connections. In actual apparatuses, the connection between elements may be represented by various alternative or additional functional connections, physical connections, or logical connections.

Hereinafter, examples will be described in detail with reference to the accompanying drawings.

<FIG> is a diagram illustrating an operation performed in a deep neural network (DNN) according to an example.

Referring to <FIG>, a DNN <NUM> may have a structure including an input layer, hidden layers, and an output layer, and may perform an operation based on received input data (e.g., I<NUM> and I<NUM>) and generate output data (e.g., O<NUM> and O<NUM>) based on the operation performance result.

For example, as illustrated in <FIG>, the DNN <NUM> may include an input layer (Layer <NUM>), two hidden layers (Layer <NUM> and Layer <NUM>), and an output layer (Layer <NUM>). Because the DNN <NUM> may include more layers capable of processing valid information, the DNN <NUM> may process more complex data sets than a neural network having a single layer. Although the DNN <NUM> is illustrated as including four layers, this is merely an example, and the DNN <NUM> may include less or more layers or may include less or more channels. That is, the DNN <NUM> may include layers of various structures different from those illustrated in <FIG>.

Each of the layers included in the DNN <NUM> may include a plurality of channels. The channel may correspond to a plurality of artificial nodes known as neurons, processing elements (PEs), units, or similar terms. For example, as illustrated in <FIG>, Layer <NUM> may include two channels (nodes), and each of Layer <NUM> and Layer <NUM> may include three channels. However, this is merely an example, and each of the layers included in the DNN <NUM> may include various numbers of channels (nodes).

The channels included in each of the layers of the neural DNN <NUM> may be connected to each other to process data. For example, one channel may receive and operate data from other channels and output the operation result to other channels.

Each of the input and output of each of the channels may be referred to as input activation and output activation. That is, the activation may be an output of one channel and a parameter corresponding to the input of the channels included in the next layer. Moreover, each of the channels may determine its own activation based on the activations and weights received from the channels included in the previous layer. The weight may be a parameter used to calculate the output activation in each channel and may be a value allocated to the connection relationship between the channels.

Each of the channels may be processed by a computational unit or a PE that receives an input and outputs an output activation, and the input-output of each of the channels may be mapped. For example, when σ is an activation function, <MAT> is a weight from the kth channel included in the (i-<NUM>)th layer to the jth channel included in the ith layer, <MAT> is a bias of the jth channel included in the ith layer, and <MAT> is an activation of the jth channel of the ith layer, the activation <MAT> may be calculated by using Equation <NUM> below.

As illustrated in <FIG>, the activation of the first channel (CH <NUM>) of the second layer (Layer <NUM>) may be represented as <MAT>. Also, <MAT> may have a value of <MAT> <MAT> according to Equation <NUM>. However, Equation <NUM> is merely an example for describing the activation and the weight used to process data in the DNN <NUM>, and the examples not limited thereto. The activation may be a value obtained by passing a value obtained by applying an activation function to the sum of the activations received from the previous layer through a rectified linear unit (ReLU).

In an example, the DNN <NUM> may determine a factor defining the relationship between a descriptor and a property through learning based on the descriptor and a property value. That is, among Layer <NUM> to Layer <NUM> constituting the DNN <NUM>, the descriptor may be Layer <NUM> that is the input layer, the property value may be Layer <NUM> that is the output layer, and the factor may be at least one hidden layer (Layer <NUM> and/or Layer <NUM>).

The DNN <NUM> may perform an operation by using a descriptor as input data in the input layer and generate a property value as output data based on the operation performance result.

<FIG> is a diagram for describing the relationship between an input feature map and an output feature map in a neural network according to an example.

Referring to <FIG>, in a layer <NUM> of the neural network, a first feature map FM1 may correspond to an input feature map and a second feature map FM2 may correspond to an output feature map. The feature map may refer to a data set representing various features of the input data. The feature maps FM1 and FM2 may have elements of a two-dimensional matrix or elements of a three-dimensional matrix, and a pixel value may be defined in each element. The feature maps FM1 and FM2 may have a width W (or a column), a height H (or a row), and a depth D. In this case, the depth D may correspond to the number of channels.

A convolution operation may be performed on the first feature map FM1 and a weight WM and as a result, the second feature map FM2 may be generated. The weight may be a weight defined in each element and may filter the features of the first feature map FM1 by performing a convolution operation with the first feature map FM1. The weight may perform a convolution operation with the windows (or tiles) of the first feature map FM1 while shifting the first feature map FM1 in a sliding window manner. During each shift, each of the elements included in the weight may be multiplied and added with each of the pixel values of an overlapped window in the first feature map FM1. As the first feature map FM1 and the weight are convoluted (convolved) together, one channel of the second feature map FM2 may be generated. Although one weight is illustrated in <FIG>, a plurality of weights may each be convoluted with the first feature map FM1 to generate the second feature map FM2 of a plurality of channels.

Moreover, the second feature map FM2 may correspond to an input feature map of the next layer. For example, the second feature map FM2 may be an input feature map of a pooling (or subsampling) layer.

In <FIG>, only a schematic architecture of the neural network is illustrated for convenience of description. However, those of ordinary skill in the art may understand that the neural network may be implemented by more or less layers, feature maps, weights, or the like, unlike the illustration, and the sizes thereof may also be modified in various ways.

<FIG> and <FIG> are example diagrams for describing a reshape operation and a transpose operation according to an example.

<FIG> illustrates a reshape operation process. The reshape operation may be a process of changing the structure of particular data. When particular data is reshaped, the data structure thereof may be changed but the data order thereof may not be changed.

For example, it is assumed that one-dimensional data <NUM> [<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>] includes eight elements. When the one-dimensional data <NUM> is reshaped into a data structure of (<NUM>, <NUM>), the structure of the one-dimensional data <NUM> may be changed into two-dimensional data <NUM> [[<NUM>, <NUM>, <NUM>, <NUM>], [<NUM>, <NUM>, <NUM>, <NUM>]].

Alternatively, when the one-dimensional data <NUM> is reshaped into a data structure of (<NUM>, <NUM>, <NUM>), the structure of the one-dimensional data <NUM> may be changed into three-dimensional data <NUM> [[[<NUM>, <NUM>], [<NUM>, <NUM>]], [[<NUM>, <NUM>], [<NUM>, <NUM>]]].

<FIG> illustrates a transpose operation process. The transpose operation may be a process of obtaining a new matrix by exchanging row and column values with each other. When particular data is transposed, only the data order thereof may be changed or both the data order and the data structure thereof may be changed depending on the initial structure of the particular data.

For example, when two-dimensional data <NUM> [[<NUM>, <NUM>], [<NUM>, <NUM>]] is transposed, it may be changed into two-dimensional data <NUM> [[<NUM>, <NUM>], [<NUM>,<NUM>]]. In this case, the data order may be changed but the data structure may be the same as before.

Alternatively, when two-dimensional data <NUM> [[<NUM>, <NUM>], [<NUM>, <NUM>], [<NUM>, <NUM>]] is transposed, it may be changed into two-dimensional data <NUM> [[<NUM>, <NUM>, <NUM>], [<NUM>, <NUM>, <NUM>]]. In this case, not only the data order thereof but also the data structure thereof may be changed. That is, the two-dimensional data <NUM> of a (<NUM>, <NUM>) structure may be changed into the two-dimensional data <NUM> of a (<NUM>, <NUM>) structure.

<FIG> is an example diagram for describing a self-attention according to an example.

The self-attention may be used to measure the relationship between words in a sentence. In this case, the relationship value of each word with other words may be calculated. This value may be referred to as an attention score. The attention score between highly related words may be high. A table of attention scores may be referred to as an attention map.

In a transformer model, an attention score may be obtained by performing a dot-product operation between word vectors. After an attention score of a particular word with respect to other words is obtained, a softmax function may be applied to an attention map where attention scores are collected. As a result, in the attention map, the correlation value of a particular word with respect to another word may appear in probability.

The probability value of the attention map and each existing word vector may be referred to as a weighted sum. The weighted sum may be an operation of multiplying and then adding each probability value and each word vector. The weighted sum result may be used as a vector value for a particular word.

<FIG> is an example diagram for describing a multihead self-attention according to an example.

The multihead self-attention may be a method of identifying attention on various feature values by generating a plurality of attention maps. The multihead self-attention may be used in a transformer model, an automatic speech recognition (ASR) model, or the like but is not limited thereto.

The multihead self-attention may have a form in which a scaled dot-product attention structure is overlapped. The input of a dot-product attention may include a query, a key, and/or a value. For example, in the case of finding the meaning of a particular word in an English dictionary, the particular word may correspond to the query, the word registered in the dictionary may correspond to the key, and the meaning of a key word may correspond to the value.

In order obtain the multihead self-attention, each of feature values for a value V, a key K, and a query Q may be divided by a head count h and then concatenated through a first linear layer <NUM> and dot-product attentions. Thereafter, when the concatenated value is finally output through a second linear layer <NUM>, the multihead self-attention may be obtained.

In an example, a matrix multiplication operation, a reshape operation, and a transpose operation may be performed in the first linear layer <NUM> and the second linear layer <NUM>. Particularly, in the first linear layer <NUM>, the reshape operation and the transpose operation may be performed after the matrix multiplication operation, and in the second linear layer <NUM>, the reshape operation and the transpose operation may be performed before the matrix multiplication operation.

Hereinafter, a method of obtaining a multihead self-attention by using a matrix multiplication operation without performing a reshape operation and a transpose operation in the first linear layer <NUM> and the second linear layer <NUM> will be described.

<FIG> is a diagram for describing the number of times accessing a memory in the case of performing a reshape operation and a transpose operation, according to an example.

<FIG> may correspond to a portion of the process of obtaining a multihead self-attention and may correspond to a process performed in the first linear layer <NUM> of <FIG>.

Referring to <FIG>, an input feature map may have a (B, L, H) structure. In the (B, L, H) structure, B denotes a batch size, L denotes a row of the input feature map, and H denotes a column of the input feature map.

A matrix multiplication operation may be performed between the input feature map of the (B, L, H) structure and a weight of an (H, H) structure. The weight may be a structure having H rows and H columns. As a result of the performance of the matrix multiplication operation, a first intermediate feature map of a (B, L, H) structure may be generated. A reshape operation may be performed on the first intermediate feature map to generate a second intermediate feature map of a (B, L, S, H/S) structure. A transpose operation may be performed on the second intermediate feature map to generate an output feature map of a (B, H/S, L, S) structure.

In the process of generating the output feature map of the (B, S, L, H/S) structure from the input feature map of the (B, L, H) structure of <FIG>, a total of four read/write processes may be required on the memory (e.g., SRAM or DRAM).

<FIG> is a diagram for describing the number of times accessing a memory in the case of not performing a reshape operation and a transpose operation, according to an example.

<FIG> may also correspond to a portion of the process of obtaining a multihead self-attention and may correspond to a process performed in the first linear layer <NUM> of <FIG>.

In <FIG>, instead of performing a matrix multiplication operation between the input feature map of the (B, L, H) structure and an initial weight of an (H, H) structure, the initial weight of the (H, H) structure may be divided into S division weights having an (H, H/S) structure.

A matrix multiplication operation may be performed between the input feature map of the (B, L, H) structure and the S division weights of the (H, H/S) structure. As a result of the performance of the matrix multiplication operation, S intermediate feature maps of a (B, L, H/S) structure may be generated. The S intermediate feature maps of the (B, L, H/S) structure may be concatenated to finally generate an output feature map of a (B, S, L, H/S) structure.

In the process of generating the output feature map of the (B, S, L, H/S) structure from the input feature map of the (B, L, H) structure of <FIG>, a total of two read/write processes may be required on the memory (e.g., SRAM or DRAM).

The structure of the input feature map and the output feature map of <FIG> may be the same as the structure of the input feature map and the output feature map of <FIG>. As for the comparison between the memory access counts of <FIG> and <FIG>, because the transpose operation is not performed in <FIG>, the memory access count in <FIG> may decrease in comparison with that in <FIG>. In <FIG>, the memory access count may decrease twice in comparison with that in <FIG>, and accordingly, the read/write data may decrease by <NUM>*BLH (= B*L*S*H/S). Moreover, as the batch size increases, the effect of reducing the memory access count in the method according to <FIG> may be greater in comparison with <FIG>.

<FIG> is an example diagram for describing a process of performing matrix multiplication by using a weight divider, according to an example.

Referring to <FIG>, an initial weight and an input feature map may be stored in a memory <NUM>. Hereinafter, it is assumed that the initial weight is a (<NUM>, <NUM>) structure having <NUM> rows and <NUM> columns and the input feature map is a (T, <NUM>) structure having T rows (T is a natural number) and <NUM> columns.

The initial weight of the (<NUM>, <NUM>) structure stored in the memory <NUM> may be input to a weight divider <NUM>. The weight divider <NUM> may divide the initial weight into a plurality of division weights. The weight divider <NUM> may divide the initial weight in any one of the column direction and the row direction.

Particularly, the weight divider <NUM> may determine whether to divide the initial weight of the (<NUM>, <NUM>) structure in any one of the column direction and the row direction according to whether the reshape operation and the transpose operation are performed after or before the matrix multiplication operation.

For example, when the reshape operation and the transpose operation are performed after the matrix multiplication operation, the weight divider <NUM> may divide the initial weight of the (<NUM>, <NUM>) structures in the column direction to generate a plurality of division weights. Alternatively, when the reshape operation and the transpose operation are performed before the matrix multiplication operation, the weight divider <NUM> may divide the initial weight of the (<NUM>, <NUM>) structures in the row direction to generate a plurality of division weights.

Also, the weight divider <NUM> may divide the initial weight of the (<NUM>, <NUM>) structure by the head count in the determined direction. For example, when the head count is <NUM>, <NUM> division weights having a (<NUM>, <NUM>) structure (column-direction division) or <NUM> division weights having a (<NUM>, <NUM>) structure (row-direction division) may be generated depending on the division direction of the initial weight.

Each of the division weights generated by the weight divider <NUM> may be transmitted to a PE. The PE may perform a matrix multiplication operation on the division weight received from the weight divider <NUM> and the input feature map received from the memory <NUM>. As a result of the performance of the matrix multiplication operation, an intermediate feature map may be output from the PE.

For example, when the weight divider <NUM> divides the initial weight of the (<NUM>, <NUM>) structure in the column direction, the weight divider <NUM> may transmit the first to 16th division weights having the (<NUM>, <NUM>) structure to first to 16th PEs <NUM>. , <NUM>, respectively. Also, the first to 16th PEs <NUM> to <NUM> may receive the input feature map from the memory <NUM>.

The first PE <NUM> may perform a matrix multiplication operation between the first division weight of the (<NUM>, <NUM>) structure and the input feature map of the (T, <NUM>) structure and output the first intermediate feature map of the (T, <NUM>) structure. In the same way, the second to 16th PEs <NUM> to <NUM> may output the second to 16th intermediate feature maps of the (T, <NUM>) structure, respectively.

Although <FIG> illustrates the case where the weight divider <NUM> divides the initial weight in the column direction, the above description may also be applied to the case where the weight divider <NUM> divides the initial weight in the row direction.

When the initial weight is divided in the column direction, the first to 16th intermediate feature maps of the (T, <NUM>) structure output from the first to 16th PEs <NUM> to <NUM> may be concatenated to generate a final feature map. The final feature map may have a (<NUM>, T, <NUM>) structure. In an example, the first to 16th intermediate feature maps may be sequentially stored at consecutive positions of the memory <NUM> and thus the first to 16th intermediate feature maps may be concatenated.

When the initial weight is divided in the row direction, a final feature map may be generated through the element-wise sum of the first to 16th intermediate feature maps output from the first to 16th PEs <NUM> to <NUM>. The element-wise sum may be performed in the PE.

Although <FIG> illustrates that <NUM> PEs are used assuming that the head count is <NUM>, the number of PEs used may be smaller or larger depending on the head count. Also, at least one PE may be used several times when a matrix multiplication operation is performed on one input feature map.

<FIG> is an example diagram illustrating the result of performance of matrix multiplication in the case of a weight being divided in a column direction, according to an example.

<FIG> is a diagram illustrating the case where the reshape operation and the transpose operation are performed after the matrix multiplication operation.

Referring to <FIG>, an input feature map <NUM> may have a (<NUM>, <NUM>, <NUM>) structure. That is, the input feature map <NUM> may have a structure of a batch size <NUM>, <NUM> rows, and <NUM> columns. An initial weight <NUM> may have a (<NUM>, <NUM>, <NUM>) structure. That is, the initial weight <NUM> may have a structure of a batch size <NUM>, <NUM> rows, and <NUM> columns.

A matrix multiplication operation may be performed on the input feature map <NUM> of the (<NUM>, <NUM>, <NUM>) structure and the initial weight <NUM> of the (<NUM>, <NUM>, <NUM>) structure, and then a reshape operation and a transpose operation may be performed on the operation result. As a result, a final feature map <NUM> of a (<NUM>, <NUM>, <NUM>, <NUM>) structure may be generated. That is, the output feature map <NUM> may have a structure of a batch size <NUM>, <NUM> channels, <NUM> rows, and <NUM> columns.

Moreover, the initial weight <NUM> may be divided in the column direction to generate a plurality of division weights. In this case, the number of division weights generated may be determined according to the head count. <FIG> is an example of the case where the head count is <NUM>, and the initial weight <NUM> of the (<NUM>, <NUM>, <NUM>) structure may be divided into first, second, and third division weights <NUM>, <NUM>, and <NUM> of a (<NUM>, <NUM>, <NUM>) structure.

A matrix multiplication operation may be performed between the input feature map <NUM> of the (<NUM>, <NUM>, <NUM>) structure and the first to third division weights <NUM> to <NUM> of the (<NUM>, <NUM>, <NUM>) structure, and the operation results may be concatenated to generate the final feature map <NUM> of the (<NUM>, <NUM>, <NUM>, <NUM>) structure.

In the example, when the reshape operation and the transpose operation are performed after the matrix multiplication operation, the initial weight <NUM> may be divided in the column direction to generate the first to third division weights <NUM> to <NUM> and the matrix multiplication operation may be performed between the input feature map <NUM> and the first to third division weights <NUM> to <NUM> to generate the same final feature map <NUM> as the case of having performed the reshape operation and the transpose operation.

<FIG> is an example diagram illustrating the result of performance of matrix multiplication in the case of a weight being divided in a row direction, according to an example.

Referring to <FIG>, an input feature map <NUM> may have a (<NUM>, <NUM>, <NUM>, <NUM>) structure. That is, the input feature map <NUM> may have a structure of a batch size <NUM>, <NUM> channels, <NUM> rows, and <NUM> columns. When a reshape operation and a transpose operations are performed on the input feature map <NUM>, an input feature map <NUM> may have a (<NUM>, <NUM>, <NUM>) structure. That is, the input feature map <NUM> may have a structure of a batch size <NUM>, <NUM> rows, and <NUM> columns. An initial weight <NUM> may have a (<NUM>, <NUM>, <NUM>) structure. That is, the initial weight <NUM> may have a structure of a batch size <NUM>, <NUM> rows, and <NUM> columns.

When a matrix multiplication operation is performed on the input feature map <NUM> of the (<NUM>, <NUM>, <NUM>) structure and the initial weight <NUM> of the (<NUM>, <NUM>, <NUM>) structure, a final feature map <NUM> of a (<NUM>, <NUM>, <NUM>) structure may be generated as a result thereof. That is, the final feature map <NUM> may have a structure of a batch size <NUM>, <NUM> rows, and <NUM> columns.

A matrix multiplication operation may be performed between the input feature map <NUM> of the (<NUM>, <NUM>, <NUM>, <NUM>) structure and the first to third division weights <NUM> to <NUM> of the (<NUM>, <NUM>, <NUM>) structure, and first, second, and third intermediate feature maps <NUM>, <NUM>, and <NUM> of a (<NUM>, <NUM>, <NUM>) structure may be generated as a result thereof. A final feature map <NUM> of a (<NUM>, <NUM>, <NUM>) structure may be generated through the element-wise sum of the first to third intermediate feature maps <NUM> to <NUM>.

In the example, when the reshape operation and the transpose operation are performed before the matrix multiplication operation, the initial weight <NUM> may be divided in the row direction to generate the first to third division weights <NUM> to <NUM> and the matrix multiplication operation may be performed between the input feature map <NUM> and the first to third division weights <NUM> to <NUM> to generate the same final feature map <NUM> as the case of having performed the reshape operation and the transpose operation.

<FIG> is a block diagram illustrating a hardware configuration of a neural network apparatus according to an example.

A neural network apparatus <NUM> may be implemented by various types of devices such as personal computers (PCs), server devices, mobile devices, or embedded devices, and as a particular example, the neural network apparatus <NUM> may correspond to a smart phone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, an autonomous car, robotics, a medical apparatus, or the like performing voice recognition, image recognition, image classification, or the like by using a neural network, but is not limited thereto. In addition, the neural network apparatus <NUM> may correspond to a dedicated hardware (HW) accelerator mounted on the above device, and the neural network apparatus <NUM> may be a hardware accelerator such as a neural processing unit (NPU), a tensor processing unit (TPU), or a neural engine, which is a dedicated module for neural network driving, but is not limited thereto.

Referring to <FIG>, the neural network apparatus <NUM> may include a processor <NUM> and a memory <NUM>. In the neural network apparatus <NUM> illustrated in <FIG>, only the components related to the various examples are illustrated. Thus, it will be apparent to those of ordinary skill in the art that the neural network apparatus <NUM> may further include other general-purpose components in addition to the components illustrated in <FIG>.

The processor <NUM> may control overall functions for executing the neural network apparatus <NUM>. For example, the processor <NUM> may generally control the neural network apparatus <NUM> by executing programs stored in the memory <NUM> in the neural network apparatus <NUM>. The processor <NUM> may be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), or the like provided in the neural network apparatus <NUM>, but is not limited thereto.

The memory <NUM> may be hardware for storing various data processed in the neural network apparatus <NUM>, and for example, the memory <NUM> may store data processed or to be processed in the neural network apparatus <NUM>. Also, the memory <NUM> may store applications, drivers, or the like to be driven by the neural network apparatus <NUM>. The memory <NUM> may include random access memory (RAM) such as dynamic random access memory (DRAM) or static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, Blu-ray, other optical disk storages, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory.

The processor <NUM> may read/write neural network data, for example, image data, feature map data, weight data, or the like from/to the memory <NUM> and execute the neural network by using the read/written data. When the neural network is executed, the processor <NUM> may repeatedly perform a convolution operation between an input feature map and a weight in order to generate data about an output feature map. In this case, the amount of the convolution operations may be determined depending on various factors such as the number of channels of the input feature map, the number of channels of the weight, the size of the input feature map, the size of the weight, and the precision of a value. Unlike the DNN <NUM> illustrated in <FIG>, an actual neural network driven in the neural network apparatus <NUM> may be implemented in a more complex architecture. Accordingly, the processor <NUM> may perform a very large amount of operations (operation count), ranging from hundreds of millions to tens of billions, and the frequency with which the processor <NUM> accesses the memory <NUM> for an operation may also increase rapidly. Due to this operation load, the neural network may not be smoothly processed in a mobile device such as a smart phone, a tablet device, or a wearable device, an embedded device, or the like having a relatively low processing performance.

The processor <NUM> may perform a matrix multiplication operation, a reshape operation, and a transpose operation. In an example, the processor <NUM> may perform a matrix multiplication operation, a reshape operation, and a transpose operation in order to obtain a multihead self-attention. In the process of obtaining the multihead self-attention, the reshape operation and the transpose operation may be performed after or before the matrix multiplication operation.

The processor <NUM> may perform a portion of the process of obtaining the multihead self-attention even without performing the reshape operation and the transpose operation. The processor <NUM> may determine whether to divide an initial weight in any one of a column direction and a row direction according to whether the reshape operation and the transpose operation are performed after or before the matrix multiplication operation and generate a plurality of division weights by dividing the initial weight by a head count in the determined direction. The processor <NUM> may generate a plurality of intermediate feature maps by performing a matrix multiplication operation between the input feature map and the plurality of division weights and generate a final feature map based on the plurality of intermediate feature maps. In the above way, the processor <NUM> may reduce the number of times accessing the memory <NUM>, by performing a portion of the process of obtaining the multihead self-attention even without performing the reshape operation and the transpose operation.

<FIG> is a flowchart of a method of performing a matrix multiplication operation in a neural network apparatus, according to an example. Because the method of performing matrix multiplication operation in the neural network apparatus illustrated in <FIG> relates to the examples described above with reference to the above drawings, the descriptions given above with reference to the above drawings may also be applied to the method of <FIG> even though omitted below.

Referring to <FIG>, in operation <NUM>, the neural network apparatus may obtain an input feature map and an initial weight from a memory.

In operation <NUM>, the neural network apparatus may determine whether to divide the initial weight in any one of a column direction and a row direction according to whether a reshape operation and a transpose operation are performed after or before the matrix multiplication operation.

When the reshape operation and the transpose operation are performed after the matrix multiplication operation, the neural network apparatus may divide the initial weight in the column direction of the initial weight. Alternatively, when the reshape operation and the transpose operation are performed before the matrix multiplication operation, the neural network apparatus may divide the initial weight in the row direction of the initial weight.

In operation <NUM>, the neural network apparatus may generate a plurality of division weights by dividing the initial weight by a head count in the direction determined in operation <NUM>.

For example, when the structure of the initial weight is (<NUM>, <NUM>) and the head count is <NUM>, <NUM> division weights having a (<NUM>, <NUM>) structure (column-direction division) or <NUM> division weights having a (<NUM>, <NUM>) structure (row-direction division) may be generated depending on the division direction of the initial weight.

In operation <NUM>, the neural network apparatus may generate a plurality of intermediate feature maps by performing a matrix multiplication operation between the input feature map and the plurality of division weights.

The matrix multiplication operation between the input feature map and the plurality of division weights may be any one of one-dimensional convolution and two-dimensional convolution operations.

In operation <NUM>, the neural network apparatus may generate a final feature map based on the plurality of intermediate feature maps.

When the initial weight is divided in the column direction in operation <NUM>, the neural network apparatus may generate the final feature map by concatenating the plurality of intermediate feature maps.

When the initial weight is divided in the row direction in operation <NUM>, the neural network apparatus may generate the final feature map through the element-wise sum of the plurality of intermediate feature maps.

The various examples may also be implemented in the form of a computer-readable recording medium including instructions executable by a computer, such as program modules executed by a computer. The computer-readable recording medium may be any available medium accessible by a computer and may include all of volatile or non-volatile mediums and removable or non-removable mediums. Also, the computer-readable recording medium may include all of computer storage mediums and communication mediums. The computer storage mediums may include all of volatile or non-volatile mediums and removable or non-removable mediums that are implemented by any method or technology to store information such as computer-readable instructions, data structures, program modules, or other data. For example, the communication mediums may include any information transmission medium and may include other transmission mechanisms or other data of modulated data signals such as computer-readable instructions, data structures, or program modules.

Also, herein, a "unit" may include a hardware component such as a processor or a circuit, and/or a software component executed by a hardware component such as a processor.

The foregoing is illustrative of various examples, and those of ordinary skill in the art will readily understand that various modifications may be made therein without materially departing from the features of the various examples. Therefore, it is to be understood that the examples described above should be considered in a descriptive sense only and not for purposes of limitation. For example, elements described as being combined may also be implemented in a distributed manner, and elements described as being distributed may also be implemented in a combined manner.

The scope of the various examples is defined not by the above detailed descriptions but by the following claims, and all modifications or differences within the scope of the claims should be construed as being included in the various examples.

According to the various examples, the same result may be obtained by repeating the matrix multiplication operation several times without performing the reshape operation and the transpose operation, and accordingly, the memory access count may be reduced and thus the memory power amount may be reduced.

Claim 1:
A neural network apparatus (<NUM>) for obtaining a multihead self-attention, comprising:
a memory (<NUM>) having at least one program stored therein; and
a processor (<NUM>) configured to perform one or more operations by executing the at least one program, wherein the processor is configured to:
acquire an input feature map (<NUM>) to a first linear layer (<NUM>) of a multihead self-attention and an initial weight from the memory,
perform at least one of a first and a second matrix multiplication, respectively in the first linear layer (<NUM>) and a second linear layer (<NUM>) of the multihead self-attention,
wherein a first matrix multiplication in the first linear layer of the multihead self-attention comprises the following steps:
divide the initial weight in a column direction of the initial weight by a head count defined by the multihead self-attention to generate division weights;
generate, using one or more computational units, intermediate feature maps by performing the matrix multiplication operation between the input feature map and the generated division weights, wherein each computational unit performs a matrix multiplication operation between each division weight and the input feature map, and outputs an intermediate feature map; and
generate a final feature map by concatenating the intermediate feature maps; and
wherein a second matrix multiplication in the second linear layer of the multihead self-attention comprises the following steps:
divide the initial weight in a row direction of the initial weight by the head count defined by the multihead self-attention to generate division weights;
generate, using one or more computational units, intermediate feature maps by performing the matrix multiplication operation between an input feature map generated based on the final feature map of the first linear layer and the generated division weights, wherein each computational unit performs a matrix multiplication operation between each division weight and the input feature map, and outputs an intermediate feature map; and
generate a final feature map through an element-wise sum of the intermediate feature maps.