Apparatus and method for performing a recognition operation in a neural network

A method of performing an operation in a neural network apparatus may include acquiring operation groups each comprising at least one input feature map and at least one kernel, and tag information corresponding to each of the operation groups, determining an operating unit in an idle state from among operating units, performing, at the operating unit in the idle state, a convolution operation between an input feature map and a kernel included in a operation group from among the operation groups to create an intermediate feature map, determining, based on tag information corresponding to the operation group, a post-processing unit from among post-processing units, and creating, at the post-processing unit, an output feature map using the intermediate feature map.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2018-0160341, filed on Dec. 12, 2018, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.

BACKGROUND

The following description relates to a method and apparatus for load balancing in a neural network.

2. Description of Related Art

Neural networks are specialized computational architecture, which after substantial training may provide computationally intuitive mappings between input patterns and output patterns. Along with the development of neural network technology, various kinds of electronic systems have been developed that use a neural network to analyze input data to extract valid data.

Recently, hardware has been developed to accelerator the use of a Deep Neural Network (DNN) efficiently with low power. However, apparatuses for processing the neural network require a large amount of computations for complex input data.

Particularly, in devices implemented with low power and low performance, technology for efficiently processing operations for a neural network is needed in order to analyze a large amount of input data in real time to extract desired information using the neural network.

SUMMARY

In one general aspect, there is provided a method of performing an operation in a neural network apparatus, the method including acquiring operation groups each comprising at least one input feature map and at least one kernel, and tag information corresponding to each of the operation groups, determining an operating unit in an idle state from among operating units, performing, at the operating unit in the idle state, a convolution operation between an input feature map and a kernel included in a operation group from among the operation groups to create an intermediate feature map, determining, based on tag information corresponding to the operation group, a post-processing unit from among post-processing units, and creating, at the post-processing unit, an output feature map using the intermediate feature map.

The determining of the operating unit in the idle state may include determining queue units to receive the operation groups and the tag information, receiving, at a load balancing unit, a signal that a workload is to be processed from the queue units, and a signal representing that the operating unit is in the idle state from the operating unit, matching, at a load balancing unit, the queue units with the operating unit in the idle state, and transferring the operation groups and the tag information from the queue units to the operating unit in the idle state.

The operating units are connected to the post-processing units by a connecting unit, and the determining of the post-processing unit further may include receiving, at a connecting unit, the intermediate feature map and tag information mapped to the operation group from the operating units, and transferring, at the connecting unit, the intermediate feature map to the post-processing unit based on the tag information.

The creating of the output feature map may include determining, at the post-processing unit, a partial sum of the intermediate feature map to create the output feature map.

The creating of the output feature map may include performing, at the post-processing unit, at least one of pooling or an activation function operation using the intermediate feature map to create the output feature map.

The queue units may be connected to some of the operating units by a fully-connected method, and some of the operating units may be connected to some of the post-processing units by a fully-connected method.

The operating units may include multiply and accumulate (MAC) units.

The load balancing unit may include an arbiter.

The connecting unit may include at least one of a multiplexer or a bus.

In another general aspect, there is provided a neural network apparatus for performing an operation, the neural network apparatus including a processor configured to acquire operation groups each comprising at least one input feature map and at least one kernel, and tag information corresponding to each of the operation groups, determine an operating unit in an idle state from among operating units, control the operating unit in the idle state to perform a convolution operation between an input feature map and a kernel included in a operation group from among the operation groups to create an intermediate feature map, determine, based on tag information corresponding to the operation group, a post-processing unit from among post-processing units, and control the post-processing unit to create an output feature map using the intermediate feature map.

The processor may be configured to determine queue units to receive the tag information and the operation groups, control a load balancing unit to receive a signal that a workload is to be processed from the queue units, to receive a signal representing that the operating unit is in the idle state from the operating unit, and to match the queue units with the operating unit in the idle state, and control the queue units to transfer the operation groups and the tag information to the operating unit in the idle state.

The operating units may be connected to the post-processing units by a connecting unit, and the processor may be configured to control the connecting unit to receive the intermediate feature map and tag information mapped to the operation group from the operating units, and to transfer the intermediate feature map to the post-processing unit based on the tag information.

The processor may be configured to control the post-processing unit to determine a partial sum of the intermediate feature map to create the output feature map.

The processor may be configured to control the post-processing unit to perform at least one of pooling or an activation function operation using the intermediate feature map to create the output feature map.

Some of the queue units may be connected to some of the operating units by a fully-connected method, and some of the operating units may be connected to some of the post-processing units by a fully-connected method.

The operating units may include multiply and accumulate (MAC) units.

The load balancing unit may include an arbiter.

The connecting unit may include at least one of a multiplexer or a bus.

The apparatus may include a memory storing instructions that, when executed, configures the processor to acquire the operation groups, to determine the operating unit in the idle state, to control the operating unit in the idle state, to determine the post-processing unit, and to control the post-processing unit to create the output feature map.

DETAILED DESCRIPTION

Throughout the specification, when a component is described as being “connected to,” or “coupled to” another component, it may be directly “connected to,” or “coupled to” the other component, or there may be one or more other components intervening therebetween. In contrast, when an element is described as being “directly connected to,” or “directly coupled to” another element, there can be no other elements intervening therebetween. Likewise, expressions, for example, “between” and “immediately between,” and “adjacent to” and “immediately adjacent to” may also be construed as described in the foregoing. As used herein, the term “and/or” includes any one and any combination of any two or more of the associated listed items.

The singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises/comprising” and/or “includes/including” when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof. The expression “at least one of a, b or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof. The use of the term ‘may’ herein with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

FIG.1is a diagram illustrating an example of an architecture of a neural network.

Referring toFIG.1, a neural network1may be an architecture of a Deep Neural Network (DNN) or n-layer neural networks. The DNN or the n-layer neural networks may correspond to neural networks such as, for example, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Deep Belief Networks (DBNs), Restricted Boltzmann Machines (RBMs), fully-connected network (FCN), a deep convolutional network (DCN), a long-short term memory (LSTM) network, and a grated recurrent units (GRUs). For example, the neural network1may be a CNN, although not limited thereto. InFIG.1, a CNN corresponding to an example of the neural network1may further include a sub-sampling layer (or called a pooling layer) and a fully connected layer, in addition to a convolution layer.

The neural network1may be implemented as an architecture having a plurality of layers including an input image, feature maps, and an output. In the neural network1, the input image may be convoluted with a filter called weights, and as a result, a plurality of feature maps may be output. The output feature maps may be again convoluted as input feature maps with the weights, and a plurality of new feature maps may be output. After the convolution operations are repeatedly performed, the recognition results of features of the input image through the neural network1may be finally output.

For example, when an image of a 24×24 pixel size is input to the neural network1ofFIG.1, the input image may be output as feature maps of 4 channels each having a 20×20 size through a convolution operation with weights. Also, some of the pixel values of the feature maps of 4 channels each having the 20×20 size may be subject to a sub-sampling operation, such as, for example, max-pooling and average-pooling, to output feature maps of 4 channels each having a 10×10 size.

In an example, the 10×10 feature maps may be repeatedly subject to convolution operations and sub-sampling operations with weights so that the sizes of the 10×10 feature maps may be reduced and global features may be output. The neural network1may repeatedly perform convolution operations and sub-sampling (or pooling) operations on the several layers to filter robust features, i.e., global features that are capable of representing the input image from the input image, to output the global features, and to input the global features to the fully connected layer, thereby recognizing the input image.

In another example, the neural network may receive an input source sentence, (e.g., voice entry) instead of an input image. In such an example, a convolution operation is performed on the input source sentence with a kernel, and as a result, the feature maps are output. The convolution operation is performed again on the output feature maps as input feature maps, with a kernel, and new feature maps are output. When the convolution operation is repeatedly performed as such, a recognition result with respect to features of the input source sentence may be output through the neural network.

FIG.2is a diagram illustrating an example of a relation between an input feature map and an output feature map in a neural network.

Referring toFIG.2, in an example, in a certain layer2of a neural network, a first feature map FM1may correspond to an input feature map and a second feature map FM2may correspond to an output feature map. A feature map may mean a data set representing various features of input data. The first and second feature maps FM1and FM2may have elements of a 2-Dimensional (2D) matrix or elements of a 3-Dimensional (3D) matrix, wherein each element defines a pixel value. The first and second feature maps FM1and FM2may have a width (or called a column) W, a height (or called a row) H, and a depth D, wherein the depth D may correspond to the number of channels.

The first feature map FM1may be convoluted with kernels to create the second feature map FM2. The kernels may be defined for individual elements and convoluted with the first feature map FM1to filter features of the first feature map FM1. The kernels may be respectively convoluted with windows, also called tiles, of the first feature map FM1, while shifting on the first feature map FM1by a sliding window method. During each shifting, pixel values of the kernels may be respectively multiplied by and added to pixel values of windows overlapping with the kernels in the first feature map FM1. As the first feature map FM1is convoluted with the kernels, a channel of the second feature map FM2may be created. InFIG.1, a single kernel is shown. However, a plurality of kernels may be convoluted with the first feature map FM1to create the second feature map FM2of a plurality of channels.

Meanwhile, the second feature map FM2may correspond to an input feature map of the next layer. For example, the second feature map FM2may become an input feature map of a pooling (or sub-sampling) layer.

InFIGS.1and2, for convenience of description, a schematic architecture of the neural network1is shown. However, the neural network1may be implemented with a larger or smaller number of layers, feature maps, kernels, etc., than those shown inFIGS.1and2, and sizes of the layers, feature maps, weights, etc. may also be modified variously.

FIG.3is a diagram illustrating examples of devices used to perform operations in a neural network apparatus.

Referring toFIG.3, a neural network apparatus3may include a controller30, a memory300, queue units310, operating units320, and post-processing units330.

The controller30may control all operations for driving the neural network apparatus3.

For example, the controller30may execute programs stored in the memory300of the neural network apparatus3to control all operations of the neural network apparatus3. The controller30includes at least one of the apparatuses described with reference toFIGS.3-6and8-10or performs at least one of the methods described with reference toFIGS.1-11. The controller30refers to a data processing device configured as hardware with a circuitry in a physical structure to execute desired operations. For example, the desired operations may include codes or instructions included in a program. For example, the controller30may be implemented as a microprocessor, a processor core, a multicore processor, a multiprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA) included in the neural network apparatus3, although not limited thereto. Also, the controller30may execute programs stored in the controller30to control all operations of the neural network apparatus3. For example, the controller30may be implemented as a microprocessor (MCU) in which a CPU, a memory (Read Only Memory (ROM) or Radom Access Memory (RAM)), etc. are installed in a single chip, although not limited thereto. Further details regarding the controller30is provided below.

According to an embodiment of the disclosure, the controller30may control the memory300, the queue units310, the operating units320, and the plurality of post-processing units330.

The memory300may be hardware storing various data that is processed on the neural network apparatus3, and for example, the memory300may store data processed on the neural network apparatus3and data that is to be processed on the neural network apparatus3. Also, the memory300may store applications that are to be driven by the neural network apparatus3, drivers, etc. The memory300may include a mass storage medium, such as, for example, random access memory (RAM) (for example, dynamic random access memory (DRAM) or static random access memory (SRAM)), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc-read only memory (CD-ROM), Blue-ray or another optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. Further details regarding the memory300is provided below.

The controller30may read/write neural network data, for example, image data, voice data, feature map data, weight data, etc. from/in the memory300, and execute the neural network using the read/written data.

When the neural network is executed, the controller30may repeatedly perform convolution operations between an input feature map and weights to create data for an output feature map. In an example, an operation count of the convolution operations may be determined depending on various factors, such as, for example, the number of channels of the input feature map, the number of channels of the weights, a size of the input feature map, a size of the weights, and precisions of values.

An input feature map that is input to the neural network may be output as an output feature map through operations that use the queue units310, the operating units320, and the post-processing units330may be used.

Referring toFIG.3, the neural network apparatus3may include the queue units310. The queue units310may receive various operands from the memory300. The queue units310may receive operands, such as an input feature map and a kernel, from the memory300.

In an example, each of the queue units310may receive an operation group including at least one input feature map and at least one kernel from the memory300. The operation group may include at least one input feature map and at least one kernel that are used to create an output feature map.

The number of input feature maps included in an operation group may be determined depending on the number of input channels. For example, when input channels are RGB channels, an operation group may include three input feature maps respectively corresponding to an R channel, a G channel, and a B channel.

Also, the number of kernels included in the operation group may be determined depending on the number of input channels. For example, when the number of input channels is three, an operation group may include three kernels respectively corresponding to three input feature maps.

In an example, when a plurality of kinds of kernels are applied to an input feature map, each operation group including the input feature map may include different kinds of kernels. For example, the kinds of kernels may include an edge detection kernel, a sharpen kernel, a Gaussian blur kernel, etc., and at least one kernel that is included in an operation group may be determined depending on the kind of the kernel. Meanwhile, the kinds of kernels are not limited to the above-mentioned kinds of kernels.

Referring toFIG.3, in an example, the queue units310may be one-to-one connected to the operating units320, respectively.

In an example, the controller30may determine whether a first operating unit connected to a first queue unit is in an idle state. The first operating unit may be in an operating state in which an operation is being performed, or in an idle state in which no operation is being performed.

When the first operating unit is in the idle state, the first queue unit may transfer an input feature map and a kernel to the first operating unit. In an example, the first queue unit may transfer an input feature map included in an operation group and a kernel corresponding to the input feature map to the first operating unit. In an example, when the first operating unit terminates an operation, the first queue unit may transfer another input feature map included in the operation group and a kernel corresponding to the other input feature map to the first operating unit. That is, the first queue unit may transfer input feature maps and kernels corresponding to the input feature maps sequentially to the first operating unit.

Likewise, second through N-th queue units may transfer at least one input feature map and at least one kernel included in an operation group received from the memory300to second through N-th operating units, respectively.

Meanwhile, referring toFIG.3, the queue units310may be one-to-one connected to the operating units320, respectively, so that each of the queue units310may transfer an operation group to the corresponding one of the operating units320. For example, the first queue unit may transfer an operation group to the first operating unit, without transferring the operation group to the other operating units included in the operating units.

In the operating units320, convolution operations between input feature maps and kernels received from the queue units310may be performed. As the result of the operations between the input feature maps and the kernels in the operating units320, an intermediate feature map may be created.

The operating units320may be one-to-one connected to the post-processing units330, respectively, so that each of the operating units320transfers the intermediate feature map to the corresponding one of the post-processing units330. For example, the first operating unit may transfer an intermediate feature map to the first post-processing unit, without transferring the intermediate feature map to the other post-processing units included in the post-processing units330.

The post-processing units330may perform post-processing on the intermediate feature maps received from the operating units320. For example, the post-processing units330may perform pooling or an activation function operation on the intermediate feature maps to create an output feature map. The activation function may be a function such as, for example, ReLu function, a Sigmoid function, and a Softmax function.

A plurality of output feature maps created by the post-processing units330may be stored in the memory300. The output feature maps stored in the memory300may be used as input feature maps, or the controller30may make a determination based on the output feature maps.

Meanwhile, a count of the number of convolution operations that are performed by the operating units320may be determined according to various factors, such as, for example, the size of an input feature map, the size of a kernel, and a kind of a kernel. For example, the operation count may be determined according to a proportion of 0 (zero) included in a kernel.

Accordingly, some of the operating units320may complete operations earlier than the other operating units320. However, when the queue units310, the operating units320, and the post-processing units330are one-to-one connected to each other, as shown inFIG.3, some operating units that have completed operations earlier than the other operating units may receive no operation group from queue units connected thereto although the operating units enter an idle state.

For example, the first operating unit may complete an operation earlier than the second operating unit. In this case, when the first operating unit is connected only to the first queue unit, as shown inFIG.3, the first operating unit may receive no operation group from the second queue unit although the first operating unit enters an idle state.

FIG.4is a diagram illustrating examples of devices used to perform operations in a neural network apparatus.

Referring toFIG.4, a neural network apparatus4may include a controller40, a memory400, a plurality of queue units410, a plurality of operating units420, and a plurality of post-processing units430.

In addition to the description ofFIG.4below, the descriptions ofFIGS.1-3are also applicable toFIG.4, and are incorporated herein by reference. Thus, the above description may not be repeated for convenience of description.

In an example, at least some of the queue units410may be connected to at least some of the operating units420by a fully-connected method. Referring toFIG.4, a first queue unit and a second queue unit may be connected to a first operating unit and a second operating unit by a 2:2 fully-connected method.

The controller40may determine whether the first operating unit and the second operating unit connected to the first queue unit are in an idle state. When the first operating unit is in an idle state, the first queue unit may transfer an input feature map and a kernel to the first operating unit. In an example, when the first operating unit is performing an operation and the second operating unit is in an idle state, the first queue unit may transfer an input feature map and a kernel to the second operating unit. Also, when the first operating unit is performing an operation, an operation for an operation group of the second queue unit has been completed, and the second operating unit is in an idle state, the first queue unit may transfer an input feature map and a kernel to the second operating unit.

In an example, the controller40may determine whether the first operating unit and the second operating unit connected to the second queue unit are in an idle state. When the second operating unit is in an idle state, the second queue unit may transfer an input feature map and a kernel to the second operating unit. Also, when the second operating unit is performing an operation and the first operating unit is in an idle state, the second queue unit may transfer an input feature map and a kernel to the first operating unit. Also, when the second operating unit is performing an operation, an operation of an operation group of the first queue unit has been completed, and when the first operating unit is in an idle state, the second queue unit may transfer an input feature map and a kernel to the first operating unit.

Meanwhile, referring toFIG.4, the third to N-th queue units may be one-to-one connected to the third to N-th operating units. Each of the third to N-th queue units may transfer an operation group only to the corresponding one of the third to N-th operating units.

In an example, some of the operating units420may be connected to some of the post-processing units430by a fully-connected method. Referring toFIG.4, the first operating unit and the second operating unit may be connected to the first post-processing unit and the second post-processing unit by a 2:2 fully-connected method.

In an example, the result of an operation between at least one input feature map and at least one kernel included in the operation group of the first queue unit is transferred to a post-processing unit corresponding to the first queue unit. For this, when the first queue unit transfers an input feature map and a kernel to the first operating unit or the second operating unit, the first queue unit may transfer tag information to the first operating unit or the second operating unit, together with the input feature map and the kernel. The tag information may include information about a position of a post-processing unit to which an intermediate feature map as the result of an operation between the input feature map and the kernel is to be transferred. For example, tag information that is transferred from the first queue unit to the first operating unit or the second operating unit may include information about a position of the first post-processing unit.

The controller40may determine which one of the first post-processing unit and the second post-processing unit the intermediate feature map created by the first operating unit is transferred to. Based on the tag information acquired from the first operating unit, the controller40transfers the intermediate feature map to the first post-processing unit or the second post-processing unit.

For example, when an intermediate feature map created by the first operating unit is the result of an operation between an input feature map and a kernel transferred from the first queue unit, tag information about a position of the first post-processing unit, transferred from the first queue unit, may exist in the first operating unit. The controller40may transfer the intermediate feature map created by the first operating unit to the first post-processing unit based on the tag information. Likewise, when an intermediate feature map created by the first operating unit is the result of an operation between an input feature map and a kernel transferred from the second queue unit, the first operating unit may transfer the intermediate feature map to the second post-processing unit based on the tag information transferred from the second queue unit.

An operation count of convolution operations that are performed by the operating units420may be determined according to various factors, such as, for example, the size of an input feature map, the size of a kernel, and a kind of a kernel. Accordingly, some of the operating units420may complete an operation earlier than the other operating units420.

For example, when the first operating unit completes an operation for the first queue unit, an input feature map and a kernel that have not yet been operated may remain in the second queue unit. In this case, the controller40may transfer the input feature map and the kernel remaining in the second queue unit to the first operating unit because the first queue unit and the second queue unit are connected to the first operating unit and the second operating unit by the fully-connected method, as shown inFIG.4. Because at least some of the queue units410included in the neural network apparatus4are connected to at least some of the operating units420by the fully-connected method, the number of operating units420that are in an idle state may be reduced, thereby achieving load balancing. Also, latency that is caused when convolution operations are performed may be reduced.

Meanwhile, in the operating units420, matrix multiplication operations may be performed on a fully-connected layer, although the kinds of operations that are performed in the operating units420are not limited thereto.

Because at least some of the queue units410are connected to at least some of the operating units420by the fully-connected method, a plurality of input feature maps and a plurality of kernels included in a queue unit may be transferred to different operating units. However, a plurality of intermediate feature maps created from the input feature maps and the kernels included in the queue unit may be sent at a post-processing unit corresponding to the queue unit. For this, when the input feature maps and the kernels included in the queue unit are transferred to different operating units, the queue unit may transfer tag information including information about a position of a post-processing unit corresponding to the queue unit to the different operating units, together with the input feature maps and the kernels.

For example, an input feature map and a kernel included in the first queue unit may be transferred to the first operating unit or the second operating unit. When the first queue unit transfers the input feature map and the kernel to the first operating unit or the second operating unit, the first queue unit may transfer tag information including information about a position of the first post-processing unit to the first operating unit or the second operating unit, together with the input feature map and the kernel, so that an intermediate feature map created by the first operating unit or the second operating unit arrives at the first post-processing unit.

FIGS.5A and5Bare diagrams illustrating examples for describing a load balancing unit and a connecting unit.

Referring toFIG.5A, a neural network apparatus51may include a plurality of queue units511and512(i.e., a first queue unit511and a second queue unit512), a plurality of operating units521and522, a plurality of post-processing units531and532, a load balancing unit540and a plurality of MUXs551and552. Meanwhile,FIG.5Ashows a certain number of units, however, more or less units may be included in the neural network apparatus51.

In an example, the queue units511and512may be connected to the operating units521and522by a fully-connected method. Referring toFIG.5A, the load balancing unit540may connect the queue units511and512to the operating units521and522.

The load balancing unit540may receive an input feature map, a kernel and tag information from the queue units511and512, and transfer the input feature map, the kernel and the tag information to an operating unit that is in an idle state among the operating units521and522.

For example, the load balancing unit540may receive an input feature map, a kernel and tag information from the first queue unit511. In this case, when a first operating unit521is performing an operation and a second operating unit522has completed an operation, the load balancing unit540may transfer the input feature map, the kernel and the tag information received from the first queue unit511to the second operating unit522.

In an example, the load balancing unit540may be an arbiter, although not limited thereto.

In an example, the operating units521and522may be connected to the post-processing units531and532by a 2:2 fully-connected method. Referring toFIG.5A, the MUXs551and552, which are connecting units, may connect the operating units521and522to the post-processing units531and532.

The MUXs551and552may receive an intermediate feature map and tag information from the operating units521and522. The MUXs551and552may determine a post-processing unit to which the intermediate feature map needs to be transferred, based on the tag information.

For example, the MUX551may receive an intermediate feature map and tag information from the second operating unit522, wherein the tag information includes information about a position of the first post-processing unit531. The MUX551may transfer the intermediate feature map to the first post-processing unit531based on the tag information including information about the position of the first post-processing unit531.

Referring toFIG.5B, a neural network apparatus52may include the queue units511and512, the operating units521and522, the post-processing units531and532, the load balancing unit540, and a bus560. In addition to the description ofFIG.5Bbelow, the descriptions ofFIG.5Aare also applicable toFIG.5B, and are incorporated herein by reference. Thus, the above description may be omitted for convenience of description.

ComparingFIG.5BtoFIG.5A, the neural network apparatus52ofFIG.5Bmay use the bus560as a connecting unit, instead of the MUXs551and552.

The BUS560may be 1:N (N is a natural number that is equal to or greater than 2) connected to the operating units521and522and the post-processing units531and532. When the BUS560is used as a connecting unit, the degree of freedom of data movements may increase.

FIG.6is a diagram illustrating an example of an operation method of a load balancing apparatus.

Referring toFIG.6, a neural network apparatus60may include a plurality of queue units611and612(i.e., a first queue unit611and a second queue unit612), a plurality of operating units621and622(i.e., a first operating unit621and a second operating unit622) and a load balancing unit640.

In an example, the queue units611and612may be connected to the operating units621and622by a fully-connected method. Referring toFIG.6, the load balancing unit640may connect the queue units611and612to the operating units621and622.

The load balancing unit640may receive an input feature map, a kernel and tag information from the queue units611and612, and transfer the input feature map, the kernel and the tag information to an operating unit that is in an idle state from among the operating units621and622.

In an example, the load balancing unit640may receive a signal661representing that there is a workload that is to be processed, from the first queue unit611. The signal661may be a signal representing that there is data (an input feature map, a kernel, etc.) that has not yet been processed. Also, the load balancing unit640may receive signals662and672representing a current state of the first operating unit621and a current state of the second operating unit622from the first operating unit621and the second operating unit622, respectively. In an example, the current states of the first operating unit621and the second operating unit622may be an operating state or an idle state.

After the load balancing unit640receives the signal661from the first queue unit611, the load balancing unit640may determine which operating unit from among of the first operating unit621and the second operating unit622is in an idle state. The load balancing unit640transfers an input feature map, a kernel, and tag information received from the first queue unit611to the operating unit in the idle state based on the signals662and672received from the first operating unit621and the second operating unit622, respectively.

For example, the signal662received from the first operating unit621may represent that the first operating unit621is operating, and the signal672received from the second operating unit622may represent that the second operating unit622is in an idle state. In this case, the load balancing unit640may transfer the input feature map, the kernel and the tag information received from the first queue unit611to the second operating unit622.

FIG.7is a diagram illustrating an example of creating output feature maps as the results of convolution operations between input feature maps and kernels.

In an example, the number of input feature maps701to703may be determined according to the number of input channels. For example, when input channels are RGB channels, the number of input feature maps701to703may be three. Referring toFIG.7, the size of each of the input feature maps701to703may be 5×4. In an example, the input feature maps701to703shown inFIG.7may be sub input feature maps. For efficiency of operations, the input feature maps701to703may be a plurality of sub input feature maps split from all input feature maps.

Kernels may be divided into a first kernel set711to713, a second kernel set721to723, and a third kernel set731to733. In an example, the number of kernels included in each kernel set may correspond to the number of input feature maps701to703. Also, the respective kernel sets may include different kinds of kernels. For example, each kernel set may include any one of edge detection kernels, sharpen kernels, and Gaussian blur kernels. However, other kinds and number of the kernels are are considered to be well within the scope of the present disclosure.

Convolution operations between the input feature maps701to703and the first to third kernel sets711to713,721to723, and731to733may be performed to create a plurality of output feature maps741to743.

When convolution operations between the input feature maps701to703and the first kernel set711to713are performed, a first output feature map741may be created. More specifically, a convolution operation between a first input feature map701and a 1-1 kernel711may be performed to create a first intermediate feature map (not shown), a convolution operation between a second input feature map702and a 1-2 kernel712may be performed to create a second intermediate feature map (not shown), and a convolution operation between a third input feature map703and a 1-3 kernel713may be performed to create a third intermediate feature map (not shown). An output feature map may be a sum of the first to third intermediate feature maps.

Likewise, convolution operations between the input feature maps701to703and the second kernel set721to723may be performed to create a second output feature map742, and convolution operations between the input feature maps701to703and the third kernel set731to733may be performed to create a third output feature map743.

In an example, the input feature maps701to703and the kernel set, which are used to create each of the output feature maps741to743, may be included in an operation group. More specifically, the input feature maps701to703and the first kernel set711to713may be included in an operation group, and the input feature maps701to703and the second kernel set721to723and the input feature maps701to703and the third kernel set731to733may also be included in different operation groups.

FIG.8is a diagram illustrating an example of a process in which convolution operations between the input feature maps and the kernels ofFIG.7are performed in a neural network apparatus.

Referring toFIG.8, a neural network apparatus80may include a memory800, a plurality of queue units811to814, a plurality of operating units821to824, and a plurality of post-processing units831to834.

In an example, each of the queue units811to814may be connected to the operating units821to824by a fully-connected method. A load balancing unit may be positioned between the queue units811to814and the operating units821to824to connect the queue units811to814to the operating units821to824by a fully-connected method851. For example, the load balancing unit may be an arbiter, although not limited thereto.

In an example, each of the operating units821to824may be connected to the post-processing units831to834by a fully-connected method. A connecting unit may be positioned between the operating units821to824and the post-processing units831to834to connect the operating units821to824to the post-processing units831to834by a fully-connected method852. For example, the connecting unit may be a MUX or a BUS, although not limited thereto.

In an example, a first queue unit811may receive a first operation group including the input feature maps701to703and the first kernel set711to713from the memory800. In an example, a second queue unit812may receive a second operation group including the input feature maps701to703and the second kernel set721to723from the memory800. In an example, the third queue unit813may receive a third operation group including the input feature maps701to703and the third kernel set731to733from the memory800.

The operation groups received by the queue units811to814may be processed in the operating units821to824and the post-processing units831to834to create the output feature maps741to743. The output feature maps741to743may be stored in the memory800. The output feature maps741to743stored in the memory800may be used as input feature maps, or the neural network apparatus80may make one or more determinations based on the output feature maps741to743.

FIGS.9A to9Care diagrams illustrating examples of describing a process in which convolution operations between the input feature maps and the kernels ofFIG.7are performed in a neural network apparatus.

In addition to the description ofFIGS.9A to9Cbelow, the descriptions ofFIG.8are also applicable toFIGS.9A to9C, and are incorporated herein by reference. Thus, the above description may be omitted for convenience of description.

The queue units811to814may transfer an input feature map, a kernel and tag information to an operating unit that is in an idle state from among the operating units821to824. The queue units811to814, the operating units821to824, and the post-processing units831to834ofFIG.9Amay also be connected by a fully-connected method, like the corresponding ones ofFIG.8. Hereinafter, a path through which data is transferred will be described for convenience of description.

Referring toFIG.9A, when all of the operating units821to824are in an idle state, the first queue unit811may transfer the first input feature map701and the 1-1 kernel711from among the input feature maps701to703and the first kernel set711to713included in the first operation group to the first operating unit821. Also, the first queue unit811may transfer tag information including information about a position of the first post-processing unit831to the first operating unit821, together with the first input feature map701and the 1-1 kernel711.

The second queue unit812may transfer the first input feature map701and the 2-1 kernel721from among the input feature maps701to703and the second kernel set721to723included in the second operation group to the second operating unit822. Also, the second queue unit812may transfer tag information including information about a position of the second post-processing unit832to the second operating unit822, together with the first input feature map701and the 2-1 kernel721.

The third queue unit813may transfer the first input feature map701and the 3-1 kernel731from among the input feature maps701to703and the third kernel set731to733included in the third operation group to the third operating unit823. Also, the third queue unit813may transfer tag information including information about a position of the third post-processing unit833to the third operating unit823, together with the first input feature map701and the 3-1 kernel731.

Because the fourth operating unit824is in an idle state and the first queue unit811is connected to the fourth operating unit824by a fully-connected method, the first queue unit811may transfer the second input feature map702and the 1-2 kernel712from among the input feature maps701to703and the first kernel set711to713included in the first operation group to the fourth operating unit824. In this case, the first queue unit811may transfer tag information including information about a position of the first post-processing unit831to the fourth operating unit824, together with the second input feature map702and the 1-2 kernel712. In another example, the second queue unit812or the third queue unit813in which a workload that is to be processed is included, instead of the first queue unit811, and may transfer an input feature map, a kernel and tag information to the fourth operating unit824.

InFIG.9A, the queue units811to814may be directly connected to the operating units821to824. However, in other example, the queue units811to814may be connected to the operating units821to824through a load balancing unit (not shown), such as, for example, an arbiter, by a fully-connected method.

In the operating units821to824, convolution operations may be performed based on input feature maps and kernels received from the queue units811to814. As the results of the convolution operations by the operating units821to824, a plurality of intermediate feature maps may be created.

The first operating unit821may transfer an intermediate feature map911created as the result of a convolution operation between the first input feature map701and the 1-1 kernel711to the first post-processing unit831, based on the tag information representing the information about the position of the first post-processing unit831.

The second operating unit822may transfer an intermediate feature map921created as the result of a convolution operation between the first input feature map701and the 2-1 kernel721to the second post-processing unit832, based on the tag information representing the information about the position of the second post-processing unit832.

The third operating unit823may transfer an intermediate feature map931created as the result of a convolution operation between the first input feature map701and the 3-1 kernel731to the third post-processing unit833, based on the tag information representing the information about the position of the third post-processing unit833.

The fourth operating unit824receives tag information representing information about the position of the first post-processing unit831from the first queue unit811, the fourth operating unit824may transfer an intermediate feature map created as the result of a convolution operation between the second input feature map702and the 1-2 kernel712to the first post-processing unit831.

InFIG.9A, the operating units821to824are directly connected to the post-processing units831to834. However, as described above, the operating units821to824may be respectively connected to the post-processing units831to834through a connecting unit (not shown), such as, for example, a MUX and/or a BUS, by a fully-connected method.

The first post-processing unit831may receive the intermediate feature map created by the first operating unit821and the intermediate feature map created by the fourth operating unit824, and calculate a partial sum of the intermediate feature maps to create an intermediate feature map911. The post-processing units831to834may include an accumulator for performing a partial sum.

The first to third post-processing units831to833may reserve post-processing until receiving the results of convolution operations between input feature maps and kernels included in all operation groups.

Referring toFIG.9B, after the operating units821to824complete convolution operations as described above with reference toFIG.9A, the operating units821to824may again enter an idle state.

The first queue unit811may transfer the third input feature map703and the 1-3 kernel713to the first operating unit821. Also, the first queue unit811may transfer tag information including information about the position of the first post-processing unit831to the first operating unit821, together with the third input feature map703and the 1-3 kernel713.

The second queue unit812may transfer the second input feature map702and the 2-2 kernel722to the second operating unit822. Also, the second queue unit812may transfer tag information including information about the position of the second post-processing unit832to the second operating unit822, together with the second input feature map702and the 2-2 kernel722.

The third queue unit813may transfer the second input feature map702and the 3-2 kernel732to the third operating unit823. Also, the third queue unit813may transfer tag information including information about the position of the third post-processing unit833to the third operating unit823, together with the second input feature map702and the 3-2 kernel732.

Because the fourth operating unit824is in an idle state, and the second queue unit812is connected to the fourth operating unit824by a fully-connected method, the second queue unit812may transfer the third input feature map703and the 2-3 kernel723to the fourth operating unit824. In this case, the second queue unit812may transfer tag information including information about the position of the second post-processing unit832to the fourth operating unit824, together with the third input feature map703and the 2-3 kernel723.

The first operating unit821may transfer an intermediate feature map911created as the result of a convolution operation between the third input feature map703and the 1-3 kernel713to the first post-processing unit831, based on tag information representing information about the position of the first post-processing unit831.

The second operating unit822may transfer an intermediate feature map921created as the result of a convolution operation between the second input feature map702and the 2-2 kernel722to the second post-processing unit832, based on tag information representing information about the position of the second post-processing unit832.

The third operating unit823may transfer the intermediate feature map931created as the result of a convolution operation between the second input feature map702and the 3-2 kernel732to the third post-processing unit833, based on tag information representing information about the position of the third post-processing unit833.

Because the fourth operating unit824receives tag information representing information about the position of the second post-processing unit832from the second queue unit812, the fourth operating unit824may transfer an intermediate feature map created as the result of a convolution operation between the third input feature map703and the 2-3 kernel723to the second post-processing unit832.

The first post-processing unit831may calculate a partial sum of the pre-stored intermediate feature map911and the intermediate feature map912received from the first operating unit821to create the first output feature map741.

The second post-processing unit832may receive the intermediate feature map created by the second operating unit822and the intermediate feature map created by the fourth operating unit824, and calculate a partial sum of the intermediate feature maps to create an intermediate feature map922. Also, the second post-processing unit832may calculate a partial sum of the pre-stored intermediate feature map921and the newly created intermediate feature map922to create the second output feature map742.

The third post-processing unit833may calculate a partial sum of the pre-stored intermediate feature map931and an intermediate feature map932received from the third operating unit823.

In an example, the first and second output feature maps741and742created by the first and second post-processing units831and832may be stored in the memory800.

Referring toFIG.9C, the third queue unit813may transfer the third input feature map703and the 3-3 kernel733to the third operating unit823. Also, the third queue unit813may transfer tag information including information about the position of the third post-processing unit833to the third operating unit823, together with the third input feature map703and the 3-3 kernel733.

The third post-processing unit833may calculate a partial sum of a pre-stored intermediate feature map933and an intermediate feature map934received from the third operating unit823to create the third output feature map743. The third output feature map743created by the third post-processing unit833may be stored in the memory800.

FIG.10is a diagram illustrating an example of a hardware configuration of a neural network apparatus.

A neural network apparatus1000may be implemented as various kinds of devices, such as, for example, a server, a mobile device, a smart phone an embedded device, a wearable smart device (such as, a ring, a watch, a pair of glasses, glasses-type device, a bracelet, an ankle bracket, a belt, a necklace, an earring, a headband, a helmet, a device embedded in the cloths, or an eye glass display (EGD)), a computing device, for example, a server, a laptop, a notebook, a subnotebook, a netbook, an ultra-mobile PC (UMPC), a tablet personal computer (tablet), a phablet, a mobile internet device (MID), a personal digital assistant (PDA), an enterprise digital assistant (EDA), an ultra mobile personal computer (UMPC), a portable lab-top PC, electronic product, for example, a robot, a digital camera, a digital video camera, a portable game console, an MP3 player, a portable/personal multimedia player (PMP), a handheld e-book, a global positioning system (GPS) navigation, a personal navigation device, portable navigation device (PND), a handheld game console, an e-book, a television (TV), a high definition television (HDTV), a smart TV, a smart appliance, a smart home device, or a security device for gate control, voice authentication systems, an Augmented Reality (AR) device, various Internet of Things (IoT) devices, robotics, medical equipment, which perform voice recognition, image recognition, image classification, through a neural network, although not limited thereto. The examples described herein may be applicable to vehicles and vehicle management systems such as, for example, an autonomous vehicle, an automatic or autonomous driving system, an intelligent vehicle, an advanced driver assistance system (ADAS), a navigation system to assist a vehicle with safely maintaining a lane on which the vehicle is travelling. The examples described herein may be used for road guidance information in a navigation device of a vehicle, such as, for example, an augmented reality head-up display (AR 3D HUD). Furthermore, the neural network apparatus1000may be a dedicated hardware accelerator mounted in the above-mentioned devices, and the neural network apparatus1000may be a hardware accelerator, such as, for example, a neural processing unit (NPU), a tensor processing unit (TPU), a neural engine, which is a dedicated module for driving a neural network, although not limited thereto. The examples described above are non-limiting, and other examples such as, for example, training, gaming, applications in healthcare, public safety, tourism, and marketing are considered to be well within the scope of the present disclosure.

Referring toFIG.10, the neural network apparatus1000may include a processor1010, a memory1020, and a user interface or a display (not shown). In the neural network apparatus1000shown inFIG.10, components related to the current embodiments are shown. The neural network apparatus1000may further include other general-purpose components, in addition to the components shown inFIG.10.

The processor1010may control all functions for executing the neural network apparatus1000. For example, the processor1010may execute programs stored in the memory1020of the neural network apparatus1000to thereby control all operations of the neural network apparatus1000. The processor1010refers to a data processing device configured as hardware with a circuitry in a physical structure to execute desired operations. For example, the desired operations may include codes or instructions included in a program. For example, the processor1010may be the data processing device configured as hardware may include a microprocessor, a central processing unit (CPU), a processor core, a multicore processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA), a GPU, and an AP included in the neural network apparatus1000, although not limited thereto. In addition to the description of processor1010, the processor1010has also been described inFIG.3, which description is also applicable to processor1010, and are incorporated herein by reference. Thus, the above description may be omitted for convenience of description.

The memory1020may be hardware storing various data processed in the neural network apparatus1000, and for example, the memory1020may store data processed in the neural network apparatus1000and data that is to be processed in the neural network apparatus1000. Also, the memory1020may store applications, drivers, etc. that are to be driven by the neural network apparatus1000. The memory1020may include RAM (for example, DRAM or SRAM), ROM, EEPROM, CD-ROM, Blue-ray or another optical disk storage, HDD, SSD, or a flash memory. In addition to the description of memory1020, the memory1020has also been described inFIG.3, which description is also applicable to memory1020, and are incorporated herein by reference. Thus, the above description may be omitted for convenience of description.

The user interface is a physical structure that includes one or more hardware components that provide the ability to render a user interface, render a display, outputs information, and/or receive user input. The user interface outputs the result that it receives from the neural network apparatus1000. However, the user interface is not limited to the example described above, and in an example, any displays, such as, for example, computer monitor and eye glass display (EGD) that are operatively connected to the apparatus1000may be used without departing from the spirit and scope of the illustrative examples described.

The processor1010may read/write neural network data, for example, image data, feature map data, weight data, etc., from/in the memory1020, and execute the neural network using the read/written data. When the neural network is executed, the processor1010may repeatedly perform convolution operations between an input feature map and weights to create data for an output feature map. At this time, an operation count of the convolution operations may be determined depending on various factors, such as the number of channels of the input feature map, the number of channels of the weights, a size of the input feature map, a size of the weights, precisions of values, etc. Unlike the neural network1shown inFIG.1, an actual neural network that is driven on the neural network apparatus1000may be implemented with a more complicated architecture. Accordingly, the processor1010may perform convolution operations by a great operation count that may reach hundreds of millions to tens of billions of convolution operations, so that the frequency of access to the memory1020by the processor1010for the convolution operations may also increase rapidly.

Meanwhile, in the neural network, weights may be floating point-type weights, fixed point-type weights, binary weights, or ternary weights. That is, in the neural network, the weights may be defined variously considering various factors, such as the purpose of use of the neural network, device performance, etc.

FIG.11is a diagram illustrating an example of a method of performing convolution operations in a neural network apparatus. The operations inFIG.11may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the spirit and scope of the illustrative examples described. Many of the operations shown inFIG.11may be performed in parallel or concurrently. One or more blocks ofFIG.11, and combinations of the blocks, can be implemented by special purpose hardware-based computer, such as a processor, that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description ofFIG.11below, the descriptions ofFIGS.1-10are also applicable toFIG.11, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring toFIG.11, in operation1110, a neural network apparatus may acquire a plurality of operation groups each including at least one input feature map and at least one kernel, and tag information corresponding to the respective operation groups.

The neural network apparatus may acquire at least one input feature map and at least one kernel from a memory or from the outside.

Each operation group may include at least one input feature map and at least one kernel that are used to create an output feature map. The tag information may include information about a position of a post-processing unit to which an intermediate feature map that results from an operation between the input feature map and the kernel needs to be transferred.

In an example, the neural network apparatus may include a plurality of queue units, and each of the queue units may acquire (or receive) a operation group and tag information corresponding to the operation group from the memory.

In an example, the neural network apparatus may determine an operating unit that is in an idle state among the operating units, in operation1120.

When an operating unit is performing an operation, the operating unit may be in an operating state, and when the operating unit is performing no operation, the operation unit may be in an idle state.

In an example, the neural network apparatus may include a load balancing unit. The load balancing unit may connect at least some of the queue units to at least one of the operating units by a fully-connected method.

The load balancing unit may receive a signal representing that there is a workload that is to be processed, from the queue units, and a signal representing that an operating unit is in an idle state, from the operating unit, and match the queue units with the operating unit that is in an idle state. Thereafter, the load balancing unit may transfer an input feature map, a kernel and tag information received from each of the queue units to the operating unit that is in the idle state.

In an example, the load balancing unit may be an arbiter, although not limited thereto.

In operation1130, the neural network apparatus may perform a convolution operation between an input feature map and a kernel included in the operation group from among the operation groups in the operating unit that is in the idle state, thereby creating an intermediate feature map. The operating unit may be a multiply and accumulate (MAC) unit.

In operation1140, the neural network apparatus may determine, based on tag information corresponding to the operation group, a post-processing unit corresponding to the tag information from among the post-processing units.

In an example, the neural network apparatus may include a connecting unit. The connecting unit may connect at least some of the operating units to at least some of the post-processing units by a fully-connected method.

The connecting unit may be at least one of a multiplexer or a bus.

In operation1150, the neural network apparatus may create an output feature map by using at least one intermediate feature map, in the post-processing unit.

In an example, the post-processing unit may calculate a partial sum of a plurality of intermediate feature maps. Also, the post-processing unit may create an output feature map by performing at least one of pooling or an activation function operation by using the at least one intermediate feature map.

According to the embodiments of the disclosure, by performing dilated convolutions using hardware including a splitter, a convolution operator, and a merger, computing speed and power efficiency may be improved.

The neural network apparatus3, neural network apparatus4, neural network apparatus51, queue units511and512, operating units521and522, post-processing units531and532, load balancing unit540, MUXs551and552, neural network apparatus60, queue units611and612, operating units621and622, load balancing unit640, neural network apparatus80, queue units811to814, operating units821to824, post-processing units831to834, neural network apparatus1000and other apparatuses, units, modules, devices, and other components described herein are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term “processor” or “computer” may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.