Patent Description:
A neural network may be a computing system implemented with reference to a computational architecture.

A typical neural network device may require a large amount of operations for complex input data. Moreover, the typical neural network device may not efficiently process operations and therefore the typical neural network device may not efficiently analyze input data in real time and extract information. Accordingly, as the typical neural network device may require a large amount of operations and/or inefficiently process the operations, the performance of the typical neural network device may be inefficient when implemented by low-power, high-performance embedded systems such as smartphones having limited resources.

<CIT> relates to a computer-implemented method which includes receiving input activations and determining whether each of the input activations has either a zero value or a non-zero value. The method further includes storing at least one of the input activations. Storing the at least one input activation includes generating an index comprising one or more memory address locations that have input activation values that are non-zero values. The method still further includes providing at least one input activation onto a data bus that is accessible by one or more units of a computational array. The activations are provided, at least in part, from a memory address location associated with the index.

<NPL>, relates to accelerate convolutional neural networks (CNNs) due to their ever-widening application areas from server, mobile to loT devices. Based on the fact that CNNs can be characterized by a significant amount of zero values in both kernel weights and activations, a novel hardware accelerator for CNNs is proposed, which exploits zero weights and activations. Zero-aware parallel CNN hardware architectures suffer from a zero-induced load imbalance problem. A zero-aware kernel allocation is proposed as a solution.

It is the object of the present invention to enhance the efficiency of performing convolutional operations in a neural network.

In this regard, the one or more embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and after an understanding of the disclosure of this application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of this application, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Throughout the specification, when an element is referred to as being "connected to" another element, it may be directly connected to the other element or electrically connected to the other element with an intervening element disposed therebetween. Also, the term "including" an element does not preclude the other elements but further includes an element unless otherwise stated.

<FIG> illustrates a neural network architecture according to one or more embodiments.

A neural network <NUM> illustrated in <FIG> may be a deep neural network (hereinafter, referred to as "DNN") architecture. The DNN may be a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network, a restricted Boltzmann machine, but is not limited thereto. The neural network <NUM> may be a DNN including an input layer (Layer <NUM>), four hidden layers (Layer <NUM>, Layer <NUM>, Layer <NUM>, and Layer <NUM>), and an output layer (Layer <NUM>). For example, when the neural network <NUM> is a convolutional neural network (CNN), the layers (Layer <NUM> to Layer <NUM>) may correspond to some layers of the CNN, such as a convolutional layer, a pooling layer, and a fully connected layer.

Each of the layers included in the neural network <NUM> may include a plurality of artificial nodes also known as "neurons", "processing elements (PEs), or similar terms thereto. While the nodes may be referred to as "artificial nodes" or "neurons," such reference is not intended to impart any relatedness with respect to how the neural network architecture computationally maps or thereby intuitively recognizes information and how a human's neurons operate. , the terms "artificial nodes" or "neurons" are merely terms of art referring to the hardware implemented nodes of a neural network. As shown in <FIG>, the input layer (Layer <NUM>) may include five nodes and the hidden layer (Layer <NUM>) may include seven nodes, for example. However, this is only an example, and each of the layers included in the neural network <NUM> may include various numbers of nodes.

The nodes included in each of the layers included in the neural network <NUM> may be connected to one another to exchange data. For example, one node may receive data from other nodes and perform operations thereon and output operation results to other nodes.

The neural network <NUM> implemented by a DNN architecture may include a plurality of layers that process effective information. Thus, the neural network <NUM> may process more complex data sets than neural networks including a single layer. Although <FIG> illustrates that the neural network <NUM> includes six layers, this is only an example and the neural network <NUM> may include less or more layers. That is, the neural network <NUM> may include various structures of layers different from that illustrated in <FIG>.

<FIG> illustrates an operation performed by a neural network according to one or more embodiments.

Thus, as illustrated in <FIG>, a neural network <NUM> may have a structure including an input layer, hidden layers, and an output layer, may perform operations based on received input data (e.g., I<NUM> and I<NUM>), and may generate output data (e.g., O<NUM> and O<NUM>) based on a result of the operations.

The neural network <NUM> may be a DNN or an n-layer neural network including two or more hidden layers as described above. For example, as illustrated in <FIG>, the neural network <NUM> may be a DNN including an input layer (Layer <NUM>), two hidden layers (Layer <NUM> and Layer <NUM>), and an output layer (Layer <NUM>). Because the neural network <NUM> implemented by a DNN architecture includes more layers capable of processing effective information, the neural network <NUM> may process more complex data sets than neural networks having a single layer. The neural network <NUM> may include less or more layers or less or more channels. That is, the neural network <NUM> may include various structures of layers different from that illustrated in <FIG>.

The neural network <NUM> may be configured to perform, as non-limiting examples, object classification, object recognition, voice recognition, and image recognition by mutually mapping input data and output data in a nonlinear relationship based on deep learning. Such deep learning is indicative of processor implemented machine learning schemes for solving issues, such as issues related to automated image or speech recognition from a big data set, as non-limiting examples. The deep learning may be implemented by mapping of input data and the output data through supervised or unsupervised learning or training, such that when trained the resultant machine learning model, engine, or example NN may intuitively map further input data to output data with a desired accuracy or reliability. Herein, it is noted that use of the term "may" with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

Each of the layers included in the neural network <NUM> may include a plurality of channels. The channels may correspond to a plurality of artificial nodes also known as neurons, PEs, units, or similar terms thereto. For example, as illustrated in <FIG>, the input layer (Layer <NUM>) may include two channels (nodes) and each of the hidden layers (Layer <NUM> and Layer <NUM>) may include three channels. However, this is only an example, and each of the layers included in the neural network <NUM> may include various numbers of channels (nodes).

The channels included in each of the layers included in the neural network <NUM> may be connected to one another to process data. For example, one channel may receive data from other channels and perform operation thereon and output operation results to other channels.

An output value of a channel may be referred to as an activation, or a value which results from such a predetermined activation function of the corresponding channel. Input and output of each channel may be referred to as input activation and output activation, respectively. That is, an activation may be a parameter corresponding to an output of one channel as well as input of another channel included in a next layer, due to corresponding connection(s) with the next layer. Each channel may determine its own activation based on resultant activations received from channels included in a previous layer and a weight and a bias. A weight may be a parameter used to calculate an output activation in each channel, and may be a value assigned to a connection relationship between the channels. For example, an output from a previous layer's channel may be provided to as an input to a channel of a next or subsequent layer through a weighted connection between the previous layer's channel and the channel of the next layer, with the weight of the weighted connection being variously adjusted during the training of the neural network until the neural network is trained for a desired objective. There may be additional connections to the channel of the next layer, such as for providing a bias connection value through a connection that may or may not be weighted and/or for providing the above example recurrent connection which may be weighted. During training and implementation such connections and connection weights may be selectively implemented, removed, and varied to generate or obtain a resultant neural network that is thereby trained and that may be correspondingly implemented for the trained objective, such as for any of the above example recognition objectives.

Accordingly, returning to <FIG>, each channel may be processed by a computational unit or processing element that receives an input (e.g., through such weighted connections) and outputs an output activation and inputs-outputs of each channel may be mapped. The computational unit may correspond to the activation function for a channel. As a non-limiting example, when σ is an activation function, <MAT> is a weight from a kth channel included in an (i-<NUM>)th layer to a jth channel included in an ith layer, <MAT> is a bias of the jth channel included in the ith layer, and <MAT> is an activation of the jth channel of the ith layer, the activation <MAT> may be calculated using Equation <NUM> below.

As illustrated in <FIG>, the activation of a first channel CH <NUM> of a second layer (Layer <NUM>) may be expressed by <MAT>. In addition, according to Equation <NUM>, the <MAT> may have a value of <MAT>. However, the above-described Equation <NUM> is only an example for describing the activation, the weight, and the bias used to process data in the neural network <NUM>, and the embodiment is not limited thereto. The activation may be a value obtained by processing a weighted sum of activations received from the previous layer by using an activation function such as a sigmoid function or a Rectified Linear Unit (ReLU) function.

In a typical neural network implementing the structure of the neural network <NUM> as described above, numerous data sets may be exchanged between a plurality of channels interconnected to one another, and may undergo numerous operation processes while passing through layers. Thus, a neural network of one or more embodiments may reduce a loss of precision while advantageously decreasing an amount of operations used to process complex input data, thereby increasing a processing speed and/or reducing a used processing power of processor-implemented devices on which the neural network of one or more embodiments may be implemented.

<FIG> illustrates a neural network device according to one or more embodiments.

A neural network device <NUM> may be a device in which a neural network (e.g., the neural network <NUM> of <FIG> or the neural network <NUM> of <FIG>) implemented as, or using, various types of devices such as personal computers (PCs), server devices, mobile devices, and embedded devices. For example, the neural network device <NUM> may be implemented as, or using, various devices that perform speech recognition, image recognition, image classification, and the like using any one or any combination of the neural network layers and/or neural networks made up of one or more of the layers of nodal convolutional interactions discussed herein, such as smart phones, tablet devices, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous driving vehicles, robot devices, and medical devices, without being limited thereto. Furthermore, the neural network device <NUM> may be a hardware (HW) accelerator dedicated for implementing or driving the above-described devices or a hardware accelerator dedicated for implementing or driving a neural network, such as a neural processing unit (NPU), a tensor processing unit (TPU), and a neural engine, without being limited thereto.

Referring to <FIG>, the neural network device <NUM> may include a memory <NUM> and a processor <NUM>. It will be understood to one of ordinary skill in the art after an understanding of the present disclosure that the neural network device <NUM> may further include general-use components in addition to the components illustrated in <FIG>.

The memory <NUM> may store data processed and to be processed by the neural network device <NUM>. For example, the memory <NUM> may store input activation data, weight data, and the like which are processed in layers of the neural network. The memory <NUM> may be random access memory (RAM) such as dynamic random access memory (DRAM) and static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, blue-ray disk, or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory, without being limited thereto.

Referring to <FIG>, the memory <NUM> may include an input activation memory <NUM> and a weight memory <NUM>. The input activation memory <NUM> and the weight memory <NUM> may store input activation data and weight data, respectively. The activation memory <NUM> may be physically separated from the weight memory <NUM>, but the embodiment is not limited thereto. For example,.

<FIG> exemplarily shows that one memory <NUM> stores input activation data and weight data, but the activation memory <NUM> and the weight memory <NUM> may share one physically connected memory <NUM>.

The processor <NUM> may read/write data processed in the neural network device <NUM>, e.g., input activation data or weight data, from the memory <NUM> and execute the neural network device <NUM> by using the read/written data. For example, the processor <NUM> may perform operations between input activation data and weight data.

In addition, the processor <NUM> may include a multiplier <NUM> and an adder <NUM>.

The multiplier <NUM> may include an array of a plurality of multipliers. For example, the multiplier <NUM> may include k multipliers. Multiplication operations between input activations and weights may be carried out respectively in the k multipliers in parallel. For example, in the multiplier <NUM>, point multiplication operations between k input activations and k weights respectively corresponding to the input activations may be performed. Thus, k may be a basic unit of operations performed by the processor <NUM>.

The adder <NUM> may receive results of multiplication operations between the k input activations and the k weights respectively corresponding to the input activations from the multiplier <NUM>. The adder <NUM> may sum up all results of k multiplication operations to output a result.

The processor <NUM> may be a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), or the like included in the neural network device <NUM>, but is not limited thereto.

When a value of at least one of the input activation and the weight is <NUM>, a result of the multiplication operation therebetween is <NUM> which therefore may not affect the result. Therefore, when the value of at least one of the input activation and the weight is <NUM>, the operation between the input activation and the weight may be skipped (e.g., determined not to be performed and/or not performed).

As described above, the processor <NUM> may perform operations between the k input activations and the k weights respectively corresponding to the input activations, as a basic unit. Thus, when the processor <NUM> is able to group k input activations whose value is <NUM> among the input activation data received from the memory <NUM>, operations between the k input activations having the value <NUM> and the k weights respectively corresponding thereto are skipped. Thus, by skipping such operations, the processor <NUM> of one or more embodiments may decrease an amount of operations used to process complex input data, thereby advantageously increasing a processing speed and/or reducing a used processing power of the processor <NUM> on which the operations may be implemented.

Accordingly, the processor <NUM> groups input activations whose value is <NUM> before processing the input activation data received from the memory <NUM> in the multiplier <NUM> and determine whether to skip the operations of the grouped input activations. Hereinafter, embodiments will be further described in detail with reference to <FIG>, as non-limiting examples.

<FIG> illustrates a method of performing convolutional operations between input activations and weights in a neural network according to one or more embodiments.

Referring to <FIG>, in operation <NUM>, the neural network device <NUM> generates a bit vector based on whether each of the input activations is <NUM> or not. The generating of the bit vector will be further described with reference to <FIG>, as a non-limiting example.

<FIG> illustrates generating a bit vector according to one or more embodiments.

First, the processor <NUM> may obtain k+n input activations <NUM> and k+n weights <NUM> corresponding thereto from the memory <NUM>.

The k+n input activations <NUM> may include input activations a<NUM> to ak+n. Each of the input activations is represented by multi bits. For example, each of the input activations may be represented by <NUM> bits, but is not limited thereto and may be represented by various numbers of bits. In a similar manner, the k+n weights <NUM> may include weights w<NUM> to wk+n. Each of the weights is represented by multi bits. For example, each of the weights may be represented by <NUM> bits, but is not limited thereto.

The processor <NUM> generates a first bit value for an input activation whose value is not <NUM> among the k+n input activations <NUM>, and generate a second bit value for an input activation whose value is <NUM> among the k+n input activations <NUM>. The first bit value may be <NUM> and the second bit value may be <NUM>, without being limited thereto. For example, when the input activation a<NUM> is not <NUM>, the first bit value of <NUM> may be generated to correspond to the input activation a<NUM>. When the input activation a<NUM> is <NUM>, the second bit value of <NUM> may be generated to correspond to the input activation a<NUM>. In such a manner, a bit vector <NUM> may be generated based on whether each of the k+n input activations <NUM> is <NUM>. Thus, the bit vector <NUM> may include k+n bit values respectively corresponding to the k+n input activations <NUM>.

Referring back to <FIG>, in operation <NUM>, the neural network device <NUM> merges the bit vector <NUM> into the input activations <NUM> such that each of the bit values included in the bit vector <NUM> is to be a most significant bit (MSB) of the multi bit expression corresponding to each input activation. The merging of the bit vector <NUM> into the input activations will be further described with reference to <FIG>, as a non-limiting example.

<FIG> illustrates merging a bit vector to input activations according to one or more embodiments.

Referring to <FIG>, the processor <NUM> may merge each of the bit values of the bit vector <NUM> into each of the input activations corresponding thereto among the k+n input activations <NUM>. For example, the first bit value of <NUM> generated for the input activation a1 may be merged into the input activation a1, and the second bit value of <NUM> generated for the input activation a2 may be merged into the input activation a2. In this regard, the first bit value of <NUM> added to the input activation a1 may be an MSB of the multi bit expression corresponding to the input activation a1, and the second bit value of <NUM> added to the input activation a2 may also be an MSB of the multi bit expression corresponding to the input activation a<NUM>. Therefore, each of k+n input activations <NUM> generated by adding the bit vector <NUM> may be represented by <NUM> bits.

Referring back to <FIG>, in operation <NUM>, the neural network device <NUM> merges the bit vector <NUM> into the weights such that each of the bit values included in the bit vector <NUM> is to be an MSB of a multi bit expression corresponding each weight. The merging of the bit vector <NUM> to the weights will be further described with reference to <FIG>, as a non-limiting example.

<FIG> illustrates merging a bit vector into weights according to one or more embodiments.

Referring to <FIG>, the processor <NUM> may merge the bit vector <NUM> generated based on the k+n input activations <NUM> into the k+n weights <NUM>. A bit value merged into the MSB of the multi bit expression corresponding to a first input activation among the input activations may be merged into the MSB of the multi bit expression corresponding to a first weight on which an operation is to be performed with the first input activation among the weights.

For example, the first bit value of <NUM> generated for the input activation a1 may be merged into the weight w1 on which an operation is to be performed with the input activation a1, and the second bit value of <NUM> generated for the input activation a2 may be merged into the weight w2 on which an operation is to be performed with the input activation a2. Here, the first bit value of <NUM> merged into the input activation a1 may be the MSB of the multi bit expression corresponding to the weight w1, and the second bit value of <NUM> merged into the input activation a2 may also be the MSB of the multi bit expression corresponding to the weight w2. Thus, each of k+n weights <NUM> generated by merging the bit vector <NUM> thereinto may be represented by <NUM> bits.

Referring back to <FIG>, in operation <NUM>, the neural network device <NUM> sorts the input activations and the weights based on bits corresponding to the MSBs. The sorting of the input activations and the weight will be further described with reference to <FIG>, as a non-limiting example.

<FIG> illustrates sorting input activations and weights according to one or more embodiments.

Referring to <FIG>, the processor <NUM> applies a sorting algorithm respectively to the k+n input activations <NUM> generated by merging the bit vector <NUM> thereinto and the k+n weights <NUM> generated by merging the bit vector <NUM> thereinto. In this case, the processor <NUM> may sort the k+n input activations <NUM> and the k+n weights <NUM> based on the bits corresponding to the MSBs.

For example, the processor <NUM> may sort the k+n input activations <NUM> generated by merging the bit vector <NUM> thereinto in the order of a first group including input activations in which the first bit value of <NUM> is merged into the MSB and a second group including input activation in which the second bit value of <NUM> is merged into the MSB.

In a similar manner, the processor <NUM> may sort the k+n weights <NUM> generated by merging the bit vector <NUM> thereinto in the order of a third group including weights in which the first bit value of <NUM> is merged into the MSB and a fourth group including weights in which the second bit value of <NUM> is merged into the MSB.

For example, the input activations including the first bit value of <NUM> merged into the MSB may include input activations a<NUM>, a<NUM>, and ak+n which constitute the first group.

In addition, the input activations including the second bit value of <NUM> merged into the MSB may include input activations a<NUM> and ak+n-<NUM> which constitute the second group.

In a similar manner, the weights including the first bit value of <NUM> merged into the MSB may include weights w<NUM>, w<NUM> and wk+n which constitute the third group. In addition, the weights including the second bit value of <NUM> merged into the MSB may include weights w<NUM> and wk+n-<NUM> which constitute the fourth group.

As described above, a same bit vector may be merged into each of the input activations and the weights and a same sorting algorithm may be applied thereto. Thus, an nth input activation among sorted input activations <NUM> and an nth weight among sorted weights <NUM> may be a pair for an operation to be performed by the pointwise multiplication operation (where n is a natural number).

Meanwhile, the sorting algorithm may be a bubble sort algorithm, shell sort algorithm, bitonic sort algorithm, comb sort algorithm, cycle sort algorithm, or even-odd mergesort algorithm, without being limited thereto. The sorting algorithm may be the even-odd mergesort. The Merge Sort is a sorting method including splitting an array of numbers into individual numbers and merging the numbers. The even-odd mergesort algorithm may sort items via comparison of odd and even positions.

Referring back to <FIG>, in operation <NUM>, the neural network device <NUM> performs operations between the sorted input activations and the sorted weights. The operations between the sorted input activations and the sorted weights will be further described with reference to <FIG>, as a non-limiting example.

<FIG> illustrates performing operations between sorted input activations and sorted weights according to one or more embodiments.

When the number of input activations belonging to the second group among the sorted input activations <NUM> is less than a preset value, the processor <NUM> may perform operations between an nth input activation among the sorted input activations <NUM> and an nth weight among the sorted weights <NUM>, respectively (where n is a natural number). The preset value may correspond to a total number of input activations on which operations may simultaneously be performed (e.g., performed in parallel). For example, when pointwise multiplication operations between k input activations and k weights respectively corresponding to the input activations are able to be performed simultaneously in the multiplier <NUM>, the preset value may be k.

Referring to <FIG>, an operation result between the first input activation a1 among the sorted input activations <NUM> and the first weight w1 among the sorted weights <NUM> is S1. In a similar manner, an operation result between the second input activation a3 among the sorted input activations <NUM> and the second weight w3 among the sorted weights <NUM> is S3. The processor <NUM> may then sum up all operation results obtained from the operations between the sorted input activations <NUM> and the sorted weights <NUM> to obtain an operation result of S.

In <FIG>, when each the operation results between the input activations belonging to the second group among the sorted input activations <NUM> and the weights belonging to the fourth group among the sorted weights <NUM> is <NUM>, the operation results do not affect the operation result of S. However, when the number of input activations belonging to the second group is less than w (e.g., a row size), the operations between the input activations belonging to the second group and the weights belonging to the fourth group among the sorted weights <NUM> are performed and are not skipped.

When the number of input activations belonging to the second group among sorted input activations <NUM> is greater than or equal to the preset value, the processor <NUM> may perform operations between input activations belonging to the first group and weights belonging to the third group and skip (e.g., determine not to perform and/or not perform) one or more of the operations between the input activations belonging to the second group and the weights belonging to the fourth group.

For example, the processor <NUM> may skip operations between the number of input activations corresponding (or equal) to the preset value from among the input activations belonging to the second group and the number of weights corresponding (or equal) to the preset value from among the weights belonging to the fourth group. For example, the neural network device <NUM> may skip a number of operations corresponding to the preset value from among operations between the input activations belonging to the second group and the weights belonging to the fourth group.

Referring to <FIG>, when the number of input activations belonging to the second group among the sorted input activations <NUM> is k, the operations between the input activations belonging to the second group and the weights belonging to the fourth group may be skipped. Therefore, all operation results between the input activations belonging to the first group and the weights belonging to the third group are summed up to obtain an operation result of S.

Also referring to <FIG>, when the number of input activations belonging to the second group among the sorted input activations <NUM> is greater than k, operations between k input activations belonging to the second group and k weights belonging to the fourth group may be skipped.

As described above with reference to <FIG>, the neural network device <NUM> may generate a bit vector based on whether each of the input activations is <NUM> and perform operations between sorted input activations and sorted weights based thereon. In this regard, when the number of input activations of which a value is <NUM> among the sorted input activations is less than a preset value, one or more of the operations between the input activations belonging to the second group and the weights belonging to the fourth group are not skipped.

As well as the input activations, when a weight is <NUM>, a result of a multiplication operation between the input activation and the weight is <NUM>, which does not affect a result. Thus, the bit vector may be generated based on whether at least one of the input activation and the weight is <NUM>.

For example, as described above with reference to <FIG>, the processor <NUM> may generate a first bit value for an input activation whose value is not <NUM> among the k+n input activations <NUM> and generate a second bit value for an input activation whose value is <NUM> among the k+n input activations <NUM>. The first bit value may be <NUM> and the second bit value may be <NUM>, without being limited thereto. For example, when the input activation a1 is not <NUM>, the first bit value of <NUM> may be generated to correspond to the input activation a1. When the input activation a2 is <NUM>, the second bit value of <NUM> may be generated to correspond to the input activation a2. In such a manner, a first vector <NUM> may be generated based on whether each of the k+n input activations <NUM> is <NUM>.

In a similar manner, as shown in <FIG>, the processor <NUM> may generate the first bit value for a weight whose value is not <NUM> among the k+n weights and generate the second bit value for a weight whose value is <NUM> among the k+n weights <NUM>. For example, when the weight w1 is <NUM>, the second bit value of <NUM> may be generated to correspond to the weight w1. When the weight w2 is not <NUM>, the first bit value of <NUM> may be generated to correspond to the weight w2. In such a manner, a second vector <NUM> may be generated based on whether each of the k+n weights <NUM> is <NUM>.

Then, the processor <NUM> may generate a bit vector <NUM> by performing an AND logic operation between the first vector <NUM> and the second vector <NUM>. Accordingly, in an example, a bit value of the bit vector <NUM> (corresponding to an input activation and a weight) may be generated to be <NUM> when either one of the input activation and the weight has a value of zero, and may be generated to be <NUM> when both of the input activation and the weight has a value of zero.

The bit vector <NUM> generated as described above may include more bit values corresponding to <NUM> than the bit vector generated based on whether each of the input activations is <NUM>.

Referring to <FIG>, an nth input activation among sorted input activations <NUM> and an nth weight among the sorted weights <NUM> may constitute one pair (where n is a natural number).

In this regard, the processor <NUM> may merge each of the bit values of the bit vector <NUM> into a corresponding pair among the pairs of the k+n input activations <NUM> and the k+n weights <NUM>. For example, the second bit value of <NUM> generated to correspond to the input activation a1 and the weight w1 may be merged into a pair of the input activation a1 and the weight w1. The second bit value of <NUM> merged into the pair of the input activation a1 and weight w1 may be an MSB of the multi bit expression corresponding to the pair. Therefore, each of k+n pairs <NUM> generated by merging the bit vector <NUM> thereto may be represented by <NUM> bits.

Then, the processor <NUM> may apply a sorting algorithm to each of the k+n pairs <NUM> generated by merging the bit vector <NUM> thereinto. The neural network device <NUM> may sort the k+n pairs <NUM> generated by merging the bit vector <NUM> thereinto based on the bits corresponding to the MSBs. For example, the neural network device <NUM> may sort the k+n pairs <NUM> in the order of a first group including pairs in which the first bit value is merged into the MSB and a second group including pairs in which the second bit value is merged into the MSB.

When the number of pairs belonging to the second group among sorted pairs <NUM> is less than a preset value, the neural network device <NUM> may perform operations between input activations and weights respectively included in the sorted k+n pairs <NUM>.

Alternatively, when the number of pairs belonging to the second group among the sorted pairs <NUM> is greater than or equal to the preset value, the neural network device <NUM> may skip at least some of the operations between input activations and weights respectively included in the pairs belonging to the second group. For example, the neural network device <NUM> may skip operations between input activations and weights included in each of the number of pairs corresponding to the preset value from among the pairs belonging to the second group.

The present value may be a total number of input activations on which operations may simultaneously be performed.

<FIG> illustrates an electronic system according to one or more embodiments.

Referring to <FIG>, an electronic system <NUM> may extract effective information by analyzing input data in real time, based on a neural network, and may determine a situation or control elements of an electronic device on which the electronic system <NUM> is mounted, based on the extracted information. For example, the electronic system <NUM> may be applied to or mounted on robot devices such as drones and advanced drivers assistance systems (ADAS), smart TVs, smart phones, medical devices, mobile devices, image display devices, measuring devices, loT devices, and any other various types of electronic devices. In another example, the electronic system <NUM> includes the electronic device, or the electronic system <NUM> is external to the electronic device.

The electronic system <NUM> may include a CPU <NUM>, RAM <NUM>, a neural network device <NUM>, a memory <NUM>, a sensor module <NUM>, and a communication module <NUM>. The electronic system <NUM> may further include an input/output module, a security module, a power control device, and the like. Some hardware components of the electronic system <NUM> may be mounted on at least one semiconductor chip. The neural network device <NUM> may be the above-described hardware accelerator dedicated for implementing or driving the neural network or a device including the same.

The CPU <NUM> may control the overall operation of the electronic system <NUM>. The CPU <NUM> may include one processor core (Single Core) or a plurality of processor cores (Multi-Core). The CPU <NUM> may process or execute programs and/or data stored in the memory <NUM>. In an embodiment, the CPU <NUM> may control functions of the neural network device <NUM> by executing the programs stored in the memory <NUM>. The CPU <NUM> may be implemented by a CPU, GPU, AP, or the like.

The RAM <NUM> may temporarily store programs, data, or instructions. For example, the programs and/or data stored in the memory <NUM> may be temporarily stored in the RAM <NUM> according to the control of the CPU <NUM> or booting code. The RAM <NUM> may be implemented by memory such as dynamic RAM (DRAM) or static RAM (SRAM).

The neural network device <NUM> may perform an operation of a neural network based on received input data and generate an information signal based on a result of the operation. The neural network may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks, restricted Boltzmann machines, and the like, but is not limited thereto. The neural network device <NUM> may correspond to the hardware accelerator dedicated for implementing or driving the neural network described above.

The information signal may include one of various types of recognition signals such as a speech recognition signal, an object recognition signal, an image recognition signal, and a biometric information recognition signal. For example, the neural network device <NUM> may receive frame data included in a video stream as input data and generate, from the frame data, a recognition signal for an object included in an image indicated by the frame data. However, the present disclosure is not limited thereto, and the neural network device <NUM> may receive various types of input data and generate a recognition signal according to the input data, according to type or function of an electronic device on which the electronic system <NUM> is mounted.

The memory <NUM> is a storage for storing data, such as an operating system (OS), various programs, and various pieces of data. In an embodiment, the memory <NUM> may store intermediate results generated in an operation performing process of the neural network device <NUM>, such as an output feature map, as an output feature list or an output feature matrix. In an embodiment, the memory <NUM> may store a compressed output feature map. In addition, the memory <NUM> may store quantized neural network data, such as, parameters, weight maps, or weight lists which are used by the neural network device <NUM>.

The memory <NUM> may be DRAM, but is not limited thereto. The memory <NUM> may include at least one of a volatile memory and a nonvolatile memory. The nonvolatile memory may include ROM, PROM, EPROM, EEPROM, flash memory, PRAM, MRAM, RRAM, FRAM, and the like. The volatile memory may include DRAM, SRAM, SDRAM, PRAM, MRAM, RRAM, FeRAM, and the like. In one or more embodiments, the memory <NUM> may include at least one of HDD, SSD, CF, SD, Micro-SD, Mini-SD, xD, or Memory Stick.

The sensor module <NUM> may collect information about periphery of the electronic device on which the electronic system <NUM> is mounted. The sensor module <NUM> may sense or receive a signal (such as an image signal, a speech signal, a magnetic signal, a biometric signal, and a touch signal) from the outside of the electronic device, and convert a sensed or received signal to data. To this end, the sensor module <NUM> may include at least one of various types of sensing devices, such as microphones, imaging devices, image sensors, light detection and ranging (LIDAR) sensors, ultrasonic sensors, infrared sensors, biosensors, or touch sensors.

The sensor module <NUM> may provide the neural network device <NUM> with the converted data as input data. For example, the sensor module <NUM> may include an image sensor, and may generate a video stream by photographing the external environment of the electronic device and provide the neural network device <NUM> with consecutive data frames of the video stream in the order of input data. However, the present disclosure is not limited thereto, and the sensor module <NUM> may provide various types of data to the neural network device <NUM>.

The communication module <NUM> may include various wired or wireless interfaces capable of communicating with external devices. For example, the communication module <NUM> may include local area network (LAN), wireless local area network (WLAN) such as wireless fidelity (Wi-Fi), wireless personal area network (WPAN) such as Bluetooth, wireless universal serial bus (USB), Zigbee, near field communication (NFC), radio-frequency identification (RFID), power line communication (PLC), or a communication interface capable of connecting to a mobile cellular network such as <NUM>rd generation (<NUM>), <NUM>th generation (<NUM>), long-term evolution (LTE), and <NUM>th generation (<NUM>).

Hereinafter, the neural network device <NUM> will be further described in more detail with reference to <FIG>, as a non-limiting example.

Referring to <FIG>, a neural network device <NUM> according to an embodiment includes a memory <NUM> (including an input activation memory <NUM> and a weight memory <NUM>) and a processor <NUM>.

The neural network device <NUM> illustrated in <FIG> includes elements used to perform the method of convolutional operations between input activations and weights in the neural network described above with reference to <FIG>. Thus, descriptions given above with reference to <FIG> may also be applied to the neural network device <NUM> illustrated in <FIG>.

Because the neural network device <NUM> of <FIG> may be similar or identical to the neural network device <NUM> of <FIG>, descriptions for the neural network device <NUM> of <FIG> may also be applied to those of <FIG>.

In addition, because the memory <NUM> of <FIG> may be similar or identical to the memory <NUM> of <FIG>, descriptions for the memory <NUM> of <FIG> may also be applied to those of <FIG>.

The processor <NUM> of <FIG> may include a bit vector generator <NUM>, a bit vector merger <NUM>, a sorting unit <NUM>, a buffer <NUM>, and a processing unit <NUM>. In addition, the processing unit <NUM> may include a multiplier <NUM> and an adder <NUM>.

The bit vector generator <NUM> may generate a bit vector based on whether each of the input activations is <NUM>. For example, the bit vector generator <NUM> may generate the first bit value for an input activation whose value is not <NUM> among the input activations and generate the second bit value for an input activation whose value is <NUM> among the input activations.

The bit vector merger <NUM> may merge the bit vector into the input activations such that each of the bit values included in the bit vector is to be an MSB of the multi bit expression corresponding to each input activation. In addition, the bit vector merger <NUM> may merge the bit vector into the weights such that each of the bit values included in the bit vector is to be an MSB of the multi bit expression corresponding to each weight. For example, a bit value merged into the MSB of the multi bit expression corresponding to the first input activation among the input activations may be merged into the MSB of the multi bit expression corresponding to the first weight on which an operation is to be performed with the first input activation among the weights.

The sorting unit <NUM> may sort input activations and weights based on the bits corresponding to the MSBs. For example, the sorting unit <NUM> may sort the input activations in the order of a first group including input activations in which the first bit value is merged into the MSB and a second group including input activations in which the second bit value is merged into the MSB. In addition, the sorting unit <NUM> may sort the weights in the order of a third group including weights in which the first bit value is merged into the MSB and a fourth group including weights in which the second bit value is merged into the MSB.

The buffer <NUM> may store k+n input activations and k+n weights obtained from the memory <NUM>. The buffer <NUM> may be included in the processor <NUM> for a pipeline.

The processing unit <NUM> may perform operations between the sorted input activations and the sorted weights. For example, when the number of input activations belonging to the second group is less than a preset value, the processing unit <NUM> may perform operations between an nth input activation among the sorted input activations (n is a natural number) and an nth weight among the sorted weights, respectively.

In addition, when the number of input activations belonging to the second group is the preset value or greater, the processing unit <NUM> may perform the operations between the input activations belonging to the first group and the weights belonging to the third group and skip at least some of the operations between the input activations belonging to the second group and the weights belonging to the fourth group. In this regard, the preset value corresponds to the total number of input activations on which operations may simultaneously be performed, and operations between the number of input activations corresponding to the preset value from among the input activations belonging to the second group and the number of weights corresponding to the preset value from among the weights belonging to the fourth groups may be skipped.

Because the multiplier <NUM> and the adder <NUM> of <FIG> may be similar or identical to the multiplier <NUM> and the adder <NUM> of <FIG> respectively, descriptions of the multiplier <NUM> and the adder <NUM> of <FIG> may also be applied to those of <FIG>.

The neural network devices, memories, input activation memories, weight memories, processors, multipliers, adders, electronic systems, CPUs, RAMs, sensor modules, communication modules, bit vector generators, bit vector mergers, sorting units, buffers, processing units, neural network device <NUM>, memory <NUM>, input activation memory <NUM>, weight memory <NUM>, processor <NUM>, multiplier <NUM>, adder <NUM>, electronic system <NUM>, CPU <NUM>, RAM <NUM>, neural network device <NUM>, memory <NUM>, sensor module <NUM>, communication module <NUM>, neural network device <NUM>, memory <NUM>, input activation memory <NUM>, weight memory <NUM>, processor <NUM>, bit vector generator <NUM>, bit vector merger <NUM>, sorting unit <NUM>, buffer <NUM>, processing unit <NUM>, multiplier <NUM>, adder <NUM>, and other apparatuses, units, modules, devices, and other components described herein with respect to <FIG> are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic modules, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic module, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions used herein, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.

Claim 1:
A processor-implemented method of performing convolutional operations between a plurality of input activations and a plurality of weights in a neural network, each input activation of the plurality of input activations and each weight of the plurality of weights being represented by a respective multi bit expression, the method comprising:
generating (<NUM>) a bit vector having a plurality of bit values, each bit value corresponding to a respective input activation of the plurality of input activations and having either a first value or a second value, wherein generating the bit vector comprises generating bit values having the first value for input activations among the plurality of input activations whose values are not <NUM> and generating bit values having the second value for input activations among the input activations whose values are <NUM>;
merging (<NUM>) the bit vector into the input activations such that each bit value of a plurality of bit values included in the bit vector is merged with a respective input activation by generating a merged multi bit expression for the respective input activation including the bit value as the most significant bit, MSB, and the multi bit expression of the respective input activation;
merging (<NUM>) the bit vector into the weights such that each bit value of the plurality of bit values included in the bit vector is merged with a respective weight by generating a merged multi bit expression for the respective weight including the bit value as the MSB and the multi bit expressions of the respective weight;
applying a same sorting algorithm respectively to the merged multi bit expressions of the input activations and the weights to sort (<NUM>) the input activations and the weights based on bits corresponding to the MSBs, wherein applying the sorting algorithm to the merged multi bit expressions of the input activations comprises grouping input activations whose value is <NUM>; and
implementing (<NUM>) the neural network, including performing operations between the sorted input activations and the sorted weights, wherein the performing of the operations comprises skipping one or more of those operations for which the input activation is among the grouped input activations.