Neural network method and apparatus

A processor-implemented method of performing a convolution operation is provided. The method includes obtaining input feature map data and kernel data, determine the kernel data based on a number of input channels of the input feature map, a number of output channels of an output feature map, and a number of groups of the input feature map data and a number of groups of the kernel data related to the convolution operation, and performing the convolution operation based on the input feature map data and the determined kernel data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2019-0175484, filed on Dec. 26, 2019, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.

BACKGROUND

The present disclosure relates to a neural network method and apparatus.

2. Description of Related Art

A neural network is a processor-implemented computing system which is implemented by referring to a computational architecture. Neural networks have an operation structure in which a large number of processing devices that implement simple functions may be connected in parallel, and are widely used as a new technique to address issues that may have been difficult to solve by implementing typical techniques. The neural network may have a generalization ability to generate a relatively correct output for an input pattern that has not be used for training, based on a result of training.

Neural network devices perform a large amount of computations on input data.

SUMMARY

In a general aspect, a processor-implemented method of performing a convolution operation includes obtaining input feature map data and kernel data, manipulating the kernel data based on a number of input channels of the input feature map, a number of output channels of an output feature map, and a number of groups of the input feature map data and a number of groups of the kernel data related to the convolution operation, and performing the convolution operation based on the input feature map data and the manipulated kernel data.

The manipulating of the kernel data may include generating a default tensor, and replacing one or more elements of the default tensor with the kernel data.

The default tensor may include a tensor comprising zeros.

A size of the default tensor may be determined based on the number of input channels and the number of output channels.

The replacing of the one or more elements of the default tensor with the kernel data may include sequentially replacing the one or more elements of the default tensor with the kernel data in a first direction.

The first direction may be a direction corresponding to the output channels.

Each of the input feature map data and the kernel data may include data on which a group convolution operation is performed.

In a general aspect, a neural network apparatus includes one or more processors configured to obtain input feature map data and kernel data, manipulate the kernel data based on a number of input channels of the input feature map, a number of output channels of an output feature map, and a number of groups of the input feature map data and a number of groups of the kernel data related to a convolution operation, and perform the convolution operation based on the input feature map data and the manipulated kernel data.

The processor may be further configured to generate a default tensor, and replace one or more elements of the default tensor with the kernel data.

The default tensor may include a tensor comprising zeros.

A size of the default tensor may be determined based on the number of input channels and the number of output channels.

The processor may be further configured to sequentially replace the one or more elements of the default tensor with the kernel data in a first direction.

The first direction may be a direction corresponding to the output channels.

Each of the input feature map data and the kernel data may include data on which a group convolution operation is performed.

The apparatus may include a memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform the obtaining of the input feature map data and kernel data, the manipulating of the kernel data, and the performing of the convolution operation.

DETAILED DESCRIPTION

The terminology used herein is for the purpose of describing various examples only, and is not to be limiting of the examples. The articles “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises/comprising” and/or “includes/including,” when used herein, specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements, and/or combinations thereof.

FIG.1illustrates an example architecture of a neural network1, in accordance with one or more embodiments.

Referring toFIG.1, the neural network1may have an architecture of a deep neural network (DNN) or n-layer neural network, as a non-limiting example. The DNN or n-layer neural network may include a plurality of layers. For example, the deep neural network may include an input layer to which input data is applied, an output layer for outputting a result derived through prediction based on training and the input data, and a plurality of hidden layers for performing a neural network operation between the input layer and the output layer. In such an example, the DNN may be, or correspond to, one or more of a fully connected network, a convolution neural network (CNN), a recurrent neural network (RNN), deep belief networks, restricted boltzman machines and the like, or may include different or overlapping neural network portions respectively with such full, convolutional, or recurrent connections, according to an algorithm used to process information. The neural network may be configured to perform, as non-limiting examples, speech recognition and voice recognition by mutually mapping input speech or voice data and output speech or voice data, e.g., in a nonlinear relationship based on deep learning. Such deep learning is indicative of processor implemented machine learning schemes for solving issues, such as issues related to automated image or speech recognition from a data set, as non-limiting examples.

The neural network1may be configured to perform, as non-limiting examples, object classification, object recognition, voice recognition, and image recognition by mutually mapping input data and output data in a nonlinear relationship based on deep learning. Such deep learning is indicative of processor implemented machine learning schemes for solving issues, such as issues related to automated image or speech recognition from a data set, as non-limiting examples. Herein, it is noted that use of the term ‘may’ with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

Referring toFIG.1, in an example, some convolutional layers are depicted in a convolutional neural network corresponding to an example of the neural network1, but the convolutional neural network may further include a pooling layer or a pulley connected layer in addition to the depicted convolutional layers.

The neural network1may have an architecture having a plurality of layers including input images, feature maps, and outputs. In the neural network1, the input image is subject to a convolution operation with a filter referred to as a kernel, and as a result, feature maps are output. The generated output feature maps at this time are input feature maps, and a convolution operation with the kernel is performed again, and as a result, new feature maps are output. As a result of the convolution operations being repeatedly performed, a result of recognition with respect to features of an input image may be finally output through the neural network1.

For example, when an image having a pixel size of 24*24 is input to the neural network1ofFIG.1, the input image may be output as four channel feature maps having a pixel size of 20*20 through a convolution operation with a kernel. Subsequently, the feature maps having a pixel size of 20*20 are reduced in size through repeated convolution operations with the kernel, and finally features having a pixel size of 1*1 may be output. The neural network1filters and outputs robust features that may represent a whole image from an input image by repeatedly performing convolutional operations and sub-sampling (or pooling) operations on the various layers, and a recognition result of the input image may be obtained through finally outputted features.

FIGS.2A-2C and3illustrate examples of a convolution operation of a neural network, in accordance with one or more embodiments.

Referring toFIG.2A, in an example, an input feature map data210has a pixel size of 6*6, the kernel data220has a pixel size of 3*3, and the output feature map data230has a pixel size of 4*4, but the example is not limited thereto. The neural network may be implemented with feature maps and kernels of various different sizes. Also, values defined in the input feature map data210, the kernel data220, and the output feature map data230are all exemplary values, and the present embodiments are not limited thereto.

A convolution operation is performed while the kernel data220slides in a region (or tile) unit having a pixel size of 3*3 in the input feature map data210. The convolution operation denotes an operation of outputting each pixel value of the feature map data230by performing a multiplication between each pixel value of a region of the input feature map data210and a corresponding weight which is an element of kernel data220and adding all the values obtained by the multiplication.

First, the kernel data220may be subjected to a convolution operation with the first region211of the input feature map data210. That is, pixel values 1, 2, 3, 4, 5, 6, 7, 8, and 9 of the first region211are multiplied by weights −1, −3, +4, +7, −2, −1, −5, +3, and +1, which respectively are elements of the kernel data220, and as a result −1, −6, 12, 28, −10, −6, −35, 24, and 9 are obtained. Next, 15, which is a result of adding the obtained values 1, −6, 12, 28, −10, −6, −35, 24, and 9, is calculated, and a pixel value231of the first row and the first column of the output feature map data230is determined to be 15. In an example, the pixel value231of the first row and the first column of the output feature map data230corresponds to the first region211.

Referring toFIG.2B, similar to the discussion above, a convolution operation between a second region212of the input feature map data210and the kernel data220is performed, and a result of 4, which is a pixel value232of the first row and second column of the output feature map data230, is determined.

Referring toFIG.2C, a convolution operation between the sixteenth region213, which is the last window of the input feature map data210, and the kernel data220is performed, and thus, a result of 11, which is a pixel value233of the fourth row and fourth column of the output feature map data230, is determined.

InFIGS.2A-2C, examples of a two-dimensional convolution operation have been described. However, the convolution operation may correspond to a three-dimensional convolution operation in which input feature map data, kernel data, and output feature map data of a plurality of channels exist. The three-dimensional convolution operation will be described with reference toFIG.3.

Referring toFIG.3, an input feature map data201may have a three-dimensional size, and may include X input channels, and a two-dimensional input feature map data of each input channel may have a size of H rows and W columns (where X, W, and H are natural numbers). A kernel data202may have a 4-dimensional size, and a 2-dimensional kernel having a size of an R row S column may exist as many as X input channels and Y output channels (where R, S, and Y are natural numbers). In other words, the kernel data202may have a number of channels corresponding to the number of input channels X of the input feature map data201and the number of output channels Y of the output feature map data203, and a two-dimensional kernel of each channel may have a size of R rows and S columns. The output feature map data203may be generated through a three-dimensional convolution operation between the three-dimensional input feature map data201and the four-dimensional kernel data202, and Y channels may exist according to the three-dimensional convolution operation result.

An example process of generating output feature map data through a convolution operation between one two-dimensional input feature map data and one two-dimensional kernel data may be implemented as described above with reference toFIGS.2A-2C. The output feature map data203of the Y channels may be generated by repeatedly performing the two-dimensional convolution operation between the input feature map data201of the X input channels and the kernel data202of the X input channels and the Y output channels described with reference toFIGS.2A-2C.

FIG.4illustrates an example data processing apparatus400.

Referring toFIG.4, the data processing apparatus400includes a memory410and a processor420. In an example, the data processing apparatus400may further store instructions, e.g., in memory410, which when executed by the processor420configure the processor4200to implement one or more or any combination of operations herein. The processor420and the memory410may be respectively representative of one or more processors420and one or more memories410. Also, although not shown inFIG.4, the data processing apparatus400may be connected to an external memory. In the data processing apparatus400ofFIG.4, only components related to the present embodiment are depicted. Accordingly, it will be apparent to those skilled in the art that other general-purpose components may further be included in the data processing apparatus400in addition to the components shown inFIG.4.

The data processing apparatus400may be a device in which the neural network described above with reference toFIGS.1to3is implemented. For example, the data processing apparatus400may be implemented with various kinds of devices, such as, but not limited to, a personal computer (PC), a server device, a mobile device, and an embedded device. As a specific example, the data processing apparatus400may correspond to, or be an apparatus provided in, as non-limiting examples, a smart phone, a tablet device, an AR (Augmented Reality) device, an Internet of Things (IoT) device, autonomous driving robotics, medical devices, etc. that perform processes such as, but not limited to voice recognition, image recognition, image classification, etc. by implementing a neural network, but is not limited thereto. Additionally, the data processing apparatus400may correspond to a dedicated hardware accelerator (HW accelerator) mounted on the above-described device, and may be a hardware accelerator, such as a neural processing unit (NPU), a tensor processing unit (TPU), a neural engine, or the like, which is a dedicated module for driving a neural network.

The memory410is hardware for storing various data processed in the data processing apparatus400. For example, the memory410may store data processed by the data processing apparatus400and data to be processed. Also, the memory410may store applications, drivers, and the like to be driven by the data processing apparatus400.

The memory410may include at least one of volatile memory or nonvolatile memory. The nonvolatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), and the like. The volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FeRAM), and the like. Furthermore, the memory420may include at least one of hard disk drives (HDDs), solid state drive (SSDs), compact flash (CF) cards, secure digital (SD) cards, micro secure digital (Micro-SD) cards, mini secure digital (Mini-SD) cards, extreme digital (xD) cards, CD-ROM, Blu-ray or other optical disk storage, hard disk drive (HDD), solid state drive (SSD), or flash memory or Memory Sticks.

The processor420is a hardware configuration that controls overall functions for driving the neural network in the data processing apparatus400. For example, the processor420generally controls the data processing apparatus400by executing programs stored in the memory410. The processor420may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc. provided in the data processing apparatus400, but is not limited thereto.

The processor420reads/writes data (for example, image data, feature map data, kernel data, etc.) from the memory410, and executes a neural network with the read/write data. When the neural network is executed, the processor420drives a processing unit included therein to repeatedly perform a convolution operation between input feature map data and kernel data to generate output feature map data. In this example, an amount of calculation of the convolution operation may be determined depending on various factors, such as the number of channels of the input feature map data, the number of channels of the kernel data, the size of the input feature map data, the size of the kernel data, and the precision of the output value.

In an example, the processing unit may include a logic circuit for convolutional operations. Specifically, the processing unit may include an operator implemented by a combination of a multiplier, an adder, and an accumulator. In addition, a multiplier may be implemented in a combination of a plurality of sub-multipliers, and an adder may be implemented in a combination of a plurality of sub-adders.

The processor420may further include an on-chip memory that is responsible for caching to perform convolution operations and a dispatcher for dispatching various operands, such as pixel values of input feature map data or weight values of kernel data. For example, the dispatcher dispatches operands, such as pixel values and weight values required for an operation to be performed by the processing unit from data stored in the memory410to the on-chip memory. Afterwards, the dispatcher re-dispatches the operands dispatched to the on-chip memory back to the processing unit for convolution operation.

The processor420may generate the same result as the result of the group convolution operation. Here, the group convolution may denote a method of performing independent convolution by dividing channels of input feature map data into a plurality of groups. Hereinafter, the group convolution will be described with reference toFIG.5.

FIG.5illustrates an example of group convolution, in accordance with one or more embodiments.

FIG.5illustrates input feature map data511and512, kernel data521and522, and output feature map data531and532in a single layer of a neural network. In an example, it is assumed that the output feature map data531and532are generated by a convolution operation of the input feature map data511and512and the kernel data521and522.

In an example, when there are a large number of channels of an input image that are input to the neural network, a large number of multiplier-accumulators (MAC) are required to generate an output. For example, if the size of input feature map data is W*H, the size of kernel data is kx*ky, the number of input channels is #InC, and the number of output channels is #OtC, in order to complete the operation of a single layer of the neural network, MACs as much as W*H*kx*ky*#InC*#OtC may be needed.

Group convolution is a of neural network lightweighting technique. Specifically, the group convolution technique may independently perform convolution by dividing channels of the input feature map data into several groups. Therefore, in contrast to performing a normal convolution operation, the group convolution may need a small number of MACs needed for an operation, and a parallel operation is possible.

Referring toFIG.5, the input feature map data511and512divided into two groups is illustrated. The input feature map data511may undergo a convolution operation with the kernel data521, and as a result, the output feature map data531is generated. Similarly, the input feature map data512may undergo a convolution operation with the kernel data522, and as a result, the output feature map data532may be generated. A final output feature map data may be generated in a single layer of the neural network by combining the output feature map data531with the output feature map data532.

However, it may be inefficient to perform group convolution operations through general hardware that implements a convolutional neural network. For example, if a group convolution operation is performed by using general processing units, it may be inefficient since a single processing unit per single layer must perform multiple convolution operations.

The data processing apparatus400may manipulate data used for the group convolution operation. For example, the data processing apparatus400may manipulate kernel data based on the number of input channels, the number of output channels, and the number of groups. As the data processing apparatus400may manipulate the kernel data, a result (that is, output feature map data) generated by the data processing apparatus400may be the same as the result generated as the group convolution operation is performed. Accordingly, even if the data processing apparatus400is implemented with a normal processing unit, the group convolution operation may be efficiently performed.

Hereinafter, an example of a convolution operation by the data processing apparatus400will be described in detail with reference toFIGS.6to8.

FIG.6is a flowchart illustrating an example method of performing a convolution operation, in accordance with one or more embodiments. The operations inFIG.6may be performed in the sequence and manner as shown. Many of the operations shown inFIG.6may be performed in parallel or concurrently. One or more blocks ofFIG.6, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description ofFIG.6below, the descriptions ofFIGS.1-5are also applicable toFIG.6, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring toFIG.6, a method of performing a convolution operation is composed of operations that may be processed in time series in the data processing apparatus400depicted inFIG.4. Therefore, even though omitted below, the descriptions given with respect to the data processing apparatus400shown inFIG.4may be applied to the method of performing a convolution operation ofFIG.6.

In operation610, the processor420may obtain input feature map data and kernel data.

The processor420may read input feature map data and kernel data stored in the memory410. For example, the input feature map data and the kernel data may be data on which group convolution is performed. The group convolution is as described above with reference toFIG.5. In other words, the processor420may obtain input feature map data that may be divided into a plurality of groups and kernel data divided into a plurality of groups.

In operation620, the processor420manipulates or determines kernel data based on the number of input channels, the number of output channels, and the number of groups related to the convolution operation.

For example, processor420may create a default tensor, and replace some of elements of the default tensor with kernel data. Here, the default tensor may be a tensor consisting of zeros. In other words, the elements of the default tensor may be zero.

The processor420may determine the size of the default tensor based on the number of input channels and the number of output channels. Also, the processor420may determine the size of a sub-tensor included in the default tensor as the size of the kernel data.

An example of manipulating the kernel data by the processor420will be described below with reference toFIGS.7and8.

In operation630, the processor420performs a convolution operation by using the input feature map data and the manipulated kernel data.

The method of performing a convolution operation by the processor420is as described above with reference toFIGS.2and3. Therefore, hereinafter, a detailed description of the method of performing a convolution operation by the processor420will be omitted.

Although not shown inFIG.6, the processor420may perform zero-skipping on the input feature map data and/or the manipulated kernel data and may perform a convolution operation by using the zero-skipping data. Accordingly, the processor420may prevent an unnecessary increase in the amount of calculation during the convolution operation.

FIG.7is a flowchart illustrating an example of manipulating kernel data by the processor420, in accordance with one or more embodiments. The operations inFIG.7may be performed in the sequence and manner as shown. Many of the operations shown inFIG.7may be performed in parallel or concurrently. One or more blocks ofFIG.7, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description ofFIG.7below, the descriptions ofFIGS.1-6are also applicable toFIG.7, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring toFIG.7, in operation710, the processor420may generate a default tensor.

In an example, the default tensor may denote tensors that contain zero as an element.

The processor420may determine the size of the default tensor based on the number of input channels and the number of output channels. For example, assuming that the number of input channels is #InC, the number of output channels is #OtC, and the size of the kernel data is kx*ky, the processor420determines the size of the default tensor as (#InC, #OtC, kx, and ky). Also, the processor420may determine that each element included in the tensor having the size of (#InC, #OtC, kx, and ky) is 0.

In operation720, the processor420may replace some of elements of the default tensor with kernel data.

The processor420may divide the default tensor into a plurality of sub-tensors. In this example, the processor420may determine the size of each of the sub-tensors as the size of kernel data.

The processor420may replace at least one of the plurality of sub-tensors with kernel data. At this point, the processor420may sequentially replace the sub-tensor with kernel data in a first direction. For example, the first direction may be a direction corresponding to the output channels, but is not limited thereto.

Hereinafter, an example of manipulating kernel data by the processor420according to the flowchart ofFIG.7will be described with reference toFIG.8.

FIG.8illustrates an example of manipulating kernel data by the processor420, in accordance with one or more embodiments.

FIG.8illustrates input feature map data811and812, and output feature map data831and832in a single layer of the neural network. Additionally, a default tensor820and a manipulated default tensor840are illustrated inFIG.8.

For convenience of description, an assumption may be made that the input feature map data811and812ofFIG.8may be the same as the input feature map data511and512ofFIG.5. Additionally, although not shown inFIG.8, it is assumed that kernel data is the same as the kernel data521and522ofFIG.5.

The processor420generates the default tensor820. For example, assuming that the number of input channels is 2, the number of output channels is 4, and the size of kernel data is 2*2, the processor420may determine the size of the default tensor820to be (2, 4, 2, 2). In addition, the processor420may set elements of the default tensor820to zero.

The processor420may generate manipulated kernel data840by replacing some of the elements of default tensor820with kernel data. First, the processor420may divide the default tensor820into a plurality of sub-tensors. In the example described above, the size of kernel data521and522is 2*2. Accordingly, the processor420may determine the size of the sub-tensor to be 2*2.

The processor420may replace some of the plurality of sub-tensors with kernel data. At this time, the processor420may determine a location of the sub-tensor at which the replacement of the kernel data begins according to the number of input channels, the number of output channels, and the number of groups. For example, assuming that the number of input channels is #InC, the number of output channels is #OtC, the number of groups is #Group, and the index of kernel data is g, the processor420may determine a location of a sub-tensor at which the replacement of the kernel data begins to be (#InC/#Group*g, #OtC/#Group*g).

Then, the processor420may sequentially replace the sub-tensor with kernel data in the first direction from the sub-tensor indicating the location of (#InC/#Group*g, #OtC/#Group*g). For example, the first direction may be a direction corresponding to the output channels (the ‘output channel direction’ as illustrated inFIG.8).

Referring to the example depicted inFIGS.5and8, in a non-limiting example, the number of input channels is 2, the number of output channels is 4, and the number of groups is 2. Also, the index of the kernel data521is 0, and the index of the kernel data522is 1.

In this example, the processor420may determine the location of the sub-tensor at which the replacement with the kernel data521begins to be (2/2*0, 4/2*0)=(0,0). In other words, referring toFIG.8, the processor420may determine a sub-tensor841indicating the location of (0,0) in the default tensor820as a starting point for replacing the kernel data521. Additionally, the processor420may also replace the sub-tensor842with kernel data in a direction corresponding to the output channels.

Additionally, the processor420may determine the location of the sub-tensor at which the replacement with the kernel data522begins to be (2/2*1, 4/2*1)=(1,2). In other words, referring toFIG.8, the processor420may determine a sub-tensor843indicating the location of (1,2) in the default tensor820as a starting point to be replaced with the kernel data522. Additionally, the processor420may replace the sub-tensor844with kernel data in a direction corresponding to the output channels.

Accordingly, a manipulated kernel data840may be generated by replacing the sub-tensors841,842,843, and844with the kernel data521and522in the default tensor820.

The processor420may generate output feature map data831and832by performing a convolution operation on the input feature map data811and812and the manipulated kernel data840. Additionally, the processor420may generate a final output feature map data in a single layer of the neural network by combining the output feature map data831and832.

At this time, the output feature map data831and832ofFIG.8may be generated in a same manner as the output feature map data531and532ofFIG.5. In other words, the processor420may generate the same result as that of the group convolution operation even when performing the normal convolution operation.

The processor420may perform zero-skipping on the input feature map data811and812and/or the manipulated kernel data840and may perform a convolution operation by using data on which the zero-skipping is performed. Accordingly, the processor420may prevent an unnecessary increase in the amount of calculation during the convolution operation.

As described above, the data processing apparatus400may generate the same result as the group convolution operation even by performing a normal convolution operation by manipulating kernel data. Therefore, a group convolution operation may be efficiently performed even through general hardware that implements a convolutional neural network.