Patent Description:
A neural network may be a computing system implemented by referring to a set machine learning that analyzes input data and extracts valid information, for example, as various types of electronic systems.

The publication of <NPL>) refers to object detection with mask-based feature encoding. Region-based Convolutional Neural Networks, R-CNNs, usually divide a Region-of-Interest, ROI, into grids, and then localize objects by utilizing the spatial information reflected by the relative position of each grid in the ROI. An encoding approach is proposed, where spatial information is represented through the spatial distributions of visual patterns. In particular, a Mask Weight Network, MWN, learns a set of masks and then applies channel-wise masking operations to ROI feature map, followed by a global pooling and a cheap fully-connected layer. The resulting R-CNNs can preserve the object-detection accuracy of the standard R-CNNs by using substantially fewer parameters.

<CIT> refers to a pooling operation device and method for conventional neural network. A pooling operation method for a convolutional neural network includes the steps of reading multiple new data in at least one column of a pooling window, performing a first pooling operation with the new data to generate at least a pooling result column, storing the pooling result column in a buffer, and performing a second pooling operation with the pooling result column and at least a preceding pooling result column stored in the buffer to generate a pooling result of the pooling window. The first pooling operation and the second pooling operation are max pooling operations.

The publication titled "<NPL> relates to NVDLA documentation. Planar Data Processor, POP, executes operations along the width × height plane. In NVDLA version <NUM>, PDPD is designed to accomplish pooling layers. Max, min, and mean pooling methods are supported. Several neighboring input elements within a plane will be sent to a non-linear function to compute one output element. The maximum value among 3x2 neighboring elements is the pooling result value.

It is the object of the present invention to provide an improved processor implemented method of a neural network, an improved neural processing apparatus, and a related computer-readable storage medium.

According to one aspect of the present invention, there is provided a processor-implemented method of performing image recognition by using a neural network, the method comprising: obtaining intermediate pooling results, respectively corresponding to sub-pooling kernels obtained by decomposing an original pooling kernel, by performing a pooling operation on pixels of an input image included in a current window in an input feature map using the sub-pooling kernels; obtaining a final pooling result corresponding to the current window by post-processing the intermediate pooling results; and determining an output pixel value of an output feature map, based on the final pooling result, wherein the current window is determined according to the original pooling kernel having been slid, according to a raster scan order, in the input feature map, wherein each of the sub-pooling kernels is a <NUM>-dimensional, 1D, kernel that has a height of <NUM>, respectively comprising only row elements of the original pooling kernel, and a total number of the <NUM> height sub-pooling kernels obtained by decomposing from the original pooling kernel corresponds to a height of the original pooling kernel, and wherein an intermediate pooling result obtained by a sub-pooling kernel from among the sub-pooling kernels with respect to the current window is shared with at least one other window in the input feature map (<NUM>) by respectively storing the intermediate pooling results that correspond to a same window in memory cells comprising memory addresses of a same column and different rows in a share line buffer.

The final pooling result may be obtained in response to all of the intermediate pooling results being obtained for the current window.

The method may further include receiving a value of a current input pixel included in the current window according to the raster scan order for the input feature map, wherein the obtaining of the intermediate pooling results includes updating at least one partial pooling result stored in at least one memory cell affected by the received value of the current input pixel, based on the received value of the current input pixel.

The obtaining of the final pooling result corresponding to the current window may include reading the intermediate pooling results for the current window from the memory cells of the share line buffer, and obtaining the final pooling result corresponding to the output pixel value by performing, on the read intermediate pooling results, a post-processing operation according to a pre-set pooling type.

The share line buffer may store, in memory lines of a total number of rows corresponding to a height of the original pooling kernel, intermediate pooling results obtained for other windows in the input feature map, in a circular manner.

An intermediate pooling result stored in one memory cell of the share line buffer may be re-used for a subsequent intermediate pooling result obtained by another sub-pooling kernel to be stored, in response to the intermediate pooling result stored in the one memory cell no longer being shared, to obtain a final pooling result corresponding to another window.

The method of may further include obtaining a hyper-parameter, of the neural network, comprising information about any one or any combination of any two or more of a size of the original pooling kernel, a stride size, and a pooling type, wherein a share line buffer storing the obtained intermediate pooling results may be addressed based on the obtained hyper-parameter.

The pooling operation may be an operation based on a pooling type of max pooling, wherein each of the intermediate pooling results is a maximum value from among values of input pixels mapped to a corresponding sub-pooling kernel and the final pooling result is a maximum value among the intermediate pooling results, or the pooling operation may be an operation based on a pooling type of average pooling, wherein each of the intermediate pooling results is a sum of the values of input pixels mapped to the corresponding sub-pooling kernel and the final pooling result is a value obtained by dividing a sum of the intermediate pooling results by a size of the original pooling kernel.

According to a further aspect of the present invention, there is provided a neural processing apparatus for performing image recognition, comprising: one or more processors configured to: obtain intermediate pooling results respectively corresponding to sub-pooling kernels obtained by decomposing an original pooling kernel, by performing a pooling operation on pixels of an input image included in a current window in an input feature map with the sub-pooling kernels, obtain a final pooling result corresponding to the current window by post-processing the intermediate pooling results, and determine an output pixel value of an output feature map, based on the final pooling result, wherein the current window is determined according to the original pooling kernel (<NUM>) having been slid, according to a raster scan order, in the input feature map, wherein each of the sub-pooling kernels is a <NUM>-dimensional, 1D, kernel that has a height of <NUM>, respectively comprising only row elements of the original pooling kernel, and a total number of the <NUM> height sub-pooling kernels obtained by decomposing from the original pooling kernel corresponds to a height of the original pooling kernel, and wherein an intermediate pooling result obtained by a sub-pooling kernel from among the sub-pooling kernels with respect to the current window is shared with at least one other window in the input feature map by respectively storing the intermediate pooling results that correspond to a same window in memory cells comprising memory addresses of a same column and different rows in a share line buffer.

The neural processing apparatus may further include a memory configured to store instructions, that when executed by the one or more processors configure the one or more processors to perform the obtaining of the intermediate pooling results, the obtaining of the a final pooling result, and the determining of the output pixel value.

The one or more processors may be further configured to read the intermediate pooling results for the current window from the memory cells of the share line buffer to obtain the final pooling result corresponding to the current window and obtain the final pooling result corresponding to the output pixel value by performing, on the read intermediate pooling results, a post-processing operation according to a pre-set pooling type.

An intermediate pooling result stored in one memory cell of the share line buffer is re-used for a subsequent intermediate pooling result obtained by another sub-pooling kernel to be stored, in response to the intermediate pooling result stored in the one memory cell no longer being shared, to obtain a final pooling result corresponding to another window.

The one or more processors may be further configured to obtain a hyper-parameter of the neural network, comprising information about any one or any combination of any two or more of a size of the original pooling kernel, a stride size, and a pooling type, wherein a share line buffer included in the memory to store the obtained intermediate pooling results may be addressed based on the hyper-parameter.

According to another aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a processor-implemented method for performing image recognition by using a neural network, the method comprising: obtaining intermediate pooling results, respectively corresponding to sub-pooling kernels obtained by decomposing an original pooling kernel, by performing a pooling operation on pixels of an input image included in a current window in an input feature map with the sub-pooling kernels using the sub-pooling kernels; obtaining a final pooling result corresponding to the current window by post-processing the intermediate pooling results; and determining an output pixel value of an output feature map, based on the final pooling result, wherein the current window is determined according to the original pooling kernel having been slid, according to a raster scan order, in the input feature map, wherein each of the sub-pooling kernels is a <NUM>-dimensional, 1D, kernel that has a height of <NUM>, respectively comprising only row elements of the original pooling kernel, and a total number of the <NUM> height sub-pooling kernels obtained by decomposing from the original pooling kernel corresponds to a height of the original pooling kernel, and wherein an intermediate pooling result obtained by a sub-pooling kernel from among the sub-pooling kernels with respect to the current window is shared with at least one other window in the input feature map by respectively storing the intermediate pooling results that correspond to a same window in memory cells comprising memory addresses of a same column and different rows in a share line buffer.

Herein, it is noted that use of the term "may" with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists where such a feature is included or implemented while all examples and embodiments are not limited thereto.

Hereinafter, embodiments will be described in detail with reference to accompanying drawings.

An apparatus implementing the neural network may use a large quantity of calculations with respect to complex input data to the neural network. As data of the neural network increases and connectivity of an architecture constituting the neural network becomes complex, excessive increases in the quantity of calculations of the apparatus and in memory access frequency may occur, resulting in inefficient performance with respect to miniaturization and commercialization issues.

<FIG> is a diagram for describing an architecture of a neural network, according to an example.

Referring to the example of <FIG>, the neural network <NUM> may be an architecture of a deep neural network (DNN) or an n-layers neural network, as non-limiting examples. The DNN or n-layers neural network may correspond to a convolution neural network (CNN), a recurrent neural network (RNN), a deep belief network, or a restricted Boltzmann machine, and any combinations thereof, as non-limiting examples. For example, the neural network <NUM> may be implemented as a CNN, but a type of the neural network <NUM> is not limited to a CNN, but may instead further include be another type of neural network.

The neural network <NUM> may be implemented as a computing architecture having a plurality of layers including an input image provided to an input layer, feature maps generated by hidden or intervening layers, and an output layer. The input image in the neural network <NUM> may be subjected to a convolution operation with a filter referred to as a weight kernel. As a result of the convolution operation, output feature maps may be output. The output feature maps generated at this time may be used as input feature maps of a next layer where they may be subjected to another convolution operation with another kernel again, and thus further output feature maps are output. As a result of iteratively performing such a convolution operation, where the input of a subsequent convolution operation is dependent on an output of a previous convolution operation, a recognition result regarding features of the input image may be finally output through the neural network <NUM>.

For example, when an image of a <NUM> × <NUM> pixel size is input to the neural network <NUM> of the example of <FIG>, the input image may be output as feature maps of four channels, each having a <NUM> × <NUM> pixel size, via the convolution operation with the first kernel by a first hidden layer. Subsequently, the respective sizes of the generated feature maps may be progressively reduced by the iterative convolution operations with each subsequent convolution operating hidden layer and the corresponding kernels, and with features of a <NUM> × <NUM> pixel size being output by a last convolution operating hidden layer. In such an example, the neural network <NUM> may filter and output robust features that may represent the entire image from the input image by iteratively performing the convolution operation along with a pooling operation or a sub-sampling operation in several layers, and may derive the recognition result of the input image by finally outputting features, as illustrated in <FIG>.

In addition, a pooling layer performing the pooling operation may be arranged to occur subsequent to a convolution layer. The pooling operation of the pooling layer may be a process for reducing a computational overhead for a next convolution layer by reducing a size, for example, rows by columns, of an output feature map of a previous convolution layer that is input to the next convolution layer.

In subsequent examples, a method of effectively performing a pooling operation on an input feature map in a pooling layer will be described in further detail.

<FIG> is a diagram for describing a concept of a pooling operation performed in a pooling layer of a neural network.

Referring to the example of <FIG>, an input feature map <NUM> may have a <NUM> × <NUM> size, where sizes are provided as rows by columns and a pooling kernel <NUM>, also referred to as a pooling window. The pooling kernel <NUM> performing a pooling operation on the input feature map <NUM> may have a <NUM> × <NUM> size. Also, a stride indicating a degree of the pooling kernel <NUM> sliding on the input feature map <NUM> may be <NUM>. However, a hyper-parameter, such as the size of the pooling kernel <NUM>, the stride, or a pooling type, of the pooling operation is not limited to those described with reference to <FIG> and may vary in other examples.

The pooling kernel <NUM> may perform the pooling operation while being slid in units of windows, also referred to as a pixel group, block, or scan window, of a <NUM> × <NUM> size, with respect to the <NUM> × <NUM> pixels of the input feature map <NUM>. The sliding order depends on a raster scan order for the input feature map <NUM>. The raster scan order for the input feature map <NUM> may denote an order in which pixels of a first row are sequentially processed from a pixel of the first row and a first column of the input feature map <NUM>, pixels from a second row to the tenth row are processed subsequently, and lastly, a pixel of the tenth row and a ninth column is processed.

When pixels corresponding to the pooling kernel <NUM> mapped to a window of a current order in the input feature map <NUM> are all scanned according to the raster scan order, the pooling kernel <NUM> may perform the pooling operation on the pixels. For example, the pooling operation by the pooling kernel <NUM> mapped to a window <NUM> in the input feature map <NUM> may be performed by using values of pixels included in the window <NUM> when the pixels are all scanned according to the raster scan order. Also, the pooling operation regarding each of a window <NUM> and a window <NUM> may be performed when all of the pixels included in each of the window <NUM> and the window <NUM> are all scanned according to the raster scan order.

In the example of <FIG>, only some windows <NUM>, <NUM>, and <NUM> are illustrated for convenience of description. The sliding of the pooling kernel <NUM> regarding the example windows <NUM> to <NUM> and the remaining windows may be performed based on the size of the pooling kernel <NUM> and the stride.

In addition, pooling results may be obtained when the pooling operation is performed on the input feature map <NUM> by the pooling kernel <NUM> in such a manner, wherein the pooling results may respectively correspond to output pixels of an output feature map.

<FIG> is a diagram for describing some example different pooling types for performing a pooling operation.

Referring to the example of <FIG>, an input feature map <NUM> may have a <NUM> × <NUM> size, a pooling kernel <NUM> may have a <NUM> × <NUM> size. In such an example, a stride may be <NUM> for convenience of description about a pooling type. However, these values are only non-limiting examples.

A pooling operation for pixels of a window mapped to the pooling kernel <NUM> may be an operation based on a pooling type of max pooling or a pooling type of average pooling. However, the pooling operation is not limited to these types of pooling, and the pooling operation may be performed according to a pooling type other than those described with reference to the example of <FIG>, in other examples.

First, the max pooling will be described in further detail. The max pooling indicates that the pooling operation may be performed by using maximum values of pixels of the input feature map <NUM> mapped to the pooling kernel <NUM> as a result of the respective pooling operations.

As a particular example, when the pooling kernel <NUM> is mapped to four pixels included in a <NUM> × <NUM> upper left window of the input feature map <NUM>, the pooling kernel <NUM> may obtain, as a pooling result corresponding to the upper left window, a value of "<NUM>" that is a maximum value from among pixel values "<NUM>," "<NUM>," "<NUM>," and "<NUM>" included in the upper left window. Max pooling is also performed on a <NUM> × <NUM> upper right window, a <NUM> × <NUM> lower left window, and a <NUM> × <NUM> lower right window in the input feature map <NUM>, using the pooling kernel <NUM>, as shown in the first example of <FIG>, respective pooling results of which would be "<NUM>," "<NUM>," and "<NUM>" as the maximum values in the respective windows. The pooling results "<NUM>," "<NUM>," "<NUM>," and "<NUM>" of the max pooling for the input feature map <NUM> may correspond to pixel values of output pixels of an output feature map <NUM> that may then be input to a next convolutional layer, for example.

Next, the average pooling will be described in further detail. The average pooling indicates that the pooling operation may be performed by using the respective averages of pixels of the input feature map <NUM> mapped to the pooling kernel <NUM>.

As a particular example, when the pooling kernel <NUM> is mapped to the four pixels included in the <NUM> × <NUM> upper left window of the input feature map <NUM>, the pooling kernel <NUM> obtains, as the pooling result corresponding to the upper left window, "<NUM>" that is an average value of the pixel values "<NUM>," "<NUM>," "<NUM>," and "<NUM>" included in the upper left window. Here, the average value used is a mean value. Average pooling is also performed on the <NUM> × <NUM> upper right window, the <NUM> × <NUM> lower left window, and the <NUM> × <NUM> lower right window in the input feature map <NUM>, as shown in the second example of <FIG>, respective pooling results of which would be "<NUM>," "<NUM>," and "<NUM>" as the averages using the pooling kernel <NUM> of the respective windows. The pooling results "<NUM>," "<NUM>," "<NUM>," and "<NUM>" of the average pooling for the input feature map <NUM> may correspond to pixel values of output pixels of an output feature map <NUM>, that may then be input to a next convolutional layer, for example.

In other words, even when the pooling operation is performed by the same pooling kernel <NUM>, generated output feature maps may be different based on a pooling type used in the pooling operation. Non-limiting examples of pooling types of max pooling type and average pooling type are discussed in greater detail above, but other pooling types are available in other examples. The pooling type may be a hyper-parameter as described above and may be predefined with respect to a neural network.

<FIG> is a block diagram of a hardware configuration of a neural processing apparatus <NUM> processing pooling of a neural network, according to an example.

Referring to the example of <FIG>, the neural processing apparatus <NUM> may include a processor <NUM> and a memory <NUM>. Components of the neural processing apparatus <NUM> related to the current example are shown in the example of <FIG>, noting that the illustrated neural processing apparatus <NUM> is representative of alternate and/or including additions in various examples of <FIG>. Thus, the neural processing apparatus <NUM> may further include components other than those shown in the example of <FIG>.

The neural processing apparatus <NUM> may correspond to a computing device. For example, as non-limiting examples, the neural processing apparatus <NUM> may correspond to a personal computer (PC), a server, or a mobile device, or may correspond to an accelerator for performing a neural network operation in such a device. In addition, the neural processing apparatus <NUM> may be representative of, or an apparatus included in, an autonomous vehicle, robotics, a smart phone, a tablet device, an augmented reality (AR) device, or an Internet of Things (IoT) device, e.g., which may perform voice recognition, image recognition, and similar tasks using the neural networks, as non-limiting examples. However, the neural processing apparatus <NUM> is not limited to these non-limiting examples, and may correspond to various types of devices or a processing apparatus that perform a neural network operation in such devices.

The processor <NUM> may be a hardware component that performs overall control functions for controlling operations of the neural processing apparatus <NUM>. For example, the processor <NUM> may control the neural processing apparatus <NUM> in general by processing or executing instructions and/or data stored in the memory <NUM> in the neural processing apparatus <NUM>. In examples, the processor <NUM> may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a neural processing unit (NPU), or a tensor processing unit (TPU) included in the neural processing apparatus <NUM>, but is not limited to these enumerated non-limiting examples, and the processor <NUM> may be implemented as other types of processor or as multiple processors or combinations of processors.

The memory <NUM> may be hardware storing hyper-parameters, e.g., including trained parameters, of various received network examples herein and various types of neural network data processed or to be processed by the processor <NUM>. For example, the memory <NUM> may store input/output feature map data, convolution data, and pooling data processed in the neural network, as discussed in further detail, above. Also, the memory <NUM> may store various applications to be driven by the processor <NUM>, for example, a convolution process application, pooling process application, and other similar applications.

The memory <NUM> may correspond to a memory device, such as random access memory (RAM), read-only memory (ROM), a hard disk drive (HDD), a solid state drive (SSD), a compact flash (CF) card, a secure digital (SD) card, a micro-SD card, a mini-SD card, an extreme digital (xD) card, or a memory stick, but a type of the memory <NUM> is not limited to these non-limiting examples and may include other types of memory devices in other examples.

As shown in the example of <FIG>, the memory <NUM> may include a share line buffer <NUM> as an on-chip memory for a neural network process by the processor <NUM>. The share line buffer <NUM> may be implemented as dynamic RAM (DRAM) or static RAM (SRAM) for a high speed access with the processor <NUM>, but the share line buffer <NUM> is not limited to these non-limiting examples and may include other types of memory technologies in other examples. In this example, the term "share line buffer" may be variously modified and referred by another term that also refers to a similar portion of the memory <NUM> used for buffering. In addition, the memory <NUM> may additionally include other buffers for buffering for other purposes, in addition to the share line buffer <NUM>.

The processor <NUM> may be implemented to have at least one processor core for processing various operations for performing convolution and pooling of the neural network. In particular, the processor <NUM> may generate an output feature map by performing respective convolution operations between input feature maps and corresponding kernel weights in layers of the neural network, each of which may generate a feature map for a next convolution layer by performing a pooling operation on the generated output feature map, as described in further detail throughout this disclosure. In addition, the processor <NUM> may perform various operations for implementing or executing the neural network.

When performing the pooling operation, the processor <NUM> may read, from the share line buffer <NUM> of the memory <NUM>, pooling data such as input pixel values of the input feature map or a processed pooling result, may update the pooling data already stored in the share line buffer <NUM>, or may write the pooling data to the share line buffer <NUM>. In other words, the share line buffer <NUM> may operate as a memory for temporarily storing the pooling data of the processor <NUM>.

<FIG> is a diagram for describing a pooling operation performed using sub-pooling kernels decomposed from a pooling kernel, according to an example.

Referring to the example of <FIG>, an original pooling kernel <NUM> may have a <NUM> × <NUM> size, noting that a size of the original pooling kernel <NUM> is not limited to a <NUM> × <NUM> size and may vary from such a size. Even when an original pooling kernel of a different size than <NUM> × <NUM> is used, a decomposing method as described below may be similarly applied.

Herein, prior to the kernel being decomposed, the kernel will be referred to by the term "original," e.g. the "original" pooling kernel <NUM>.

The processor <NUM> decomposes the original pooling kernel <NUM> into a plurality of sub-pooling kernels <NUM> to <NUM>, in order to perform a pooling operation. The sub-pooling kernels <NUM> to <NUM> are one dimensional (1D) kernels, respectively including row elements of the original pooling kernel <NUM>. In the examples, the sub-pooling kernels <NUM> to <NUM> may be obtained by decomposing the row elements of the original pooling kernel <NUM> according to the raster scan order described above, but when pixels in an input feature map are scanned in a raster scan order different from that described above, sub-pooling kernels may be obtained by decomposing elements of the original pooling kernel <NUM> in another direction. For example, sub-pooling kernels may be obtained by decomposing elements of the original pooling kernel <NUM> in a column direction. In other words, a decomposing direction may vary based on a raster scan order, in different examples.

The number of sub-pooling kernels <NUM> to <NUM> obtained by decomposing the original pooling kernel <NUM> correspond to a height of the original pooling kernel <NUM>. For example because the height of the original pooling kernel <NUM> of <FIG> is <NUM>, the number of sub-pooling kernels <NUM> to <NUM> is also <NUM>.

Elements of the sub-pooling kernel <NUM> may correspond to elements of a first row of the original pooling kernel <NUM>, elements of the sub-pooling kernel <NUM> may correspond to elements of a second row of the original pooling kernel <NUM>, and elements of the sub-pooling kernel <NUM> may correspond to elements of a third row of the original pooling kernel <NUM>.

The processor <NUM> may individually obtain intermediate pooling results by individually performing pooling operations on the sub-pooling kernels <NUM> to <NUM>. Then, the processor <NUM> may merge the intermediate pooling results in order to output a final pooling result. In other words, according to the examples, the pooling operation may be performed in units of sub-pooling kernels instead of units of original pooling kernels. In the examples, the term "intermediate pooling result" refers to partial pooling data obtained by a sub-pooling kernel, and the term "final pooling result" refers to final pooling data corresponding to a window obtained from a plurality of intermediate pooling results.

<FIG> is a diagram for describing a method of performing max pooling (MaxPool) by using sub-pooling kernels <NUM> to <NUM> decomposed from an original pooling kernel <NUM>, according to an example.

Referring to the example of <FIG>, a MaxPool operation may be performed on an input feature map <NUM> of a <NUM> × <NUM> size by using the three <NUM> × <NUM> sub-pooling kernels <NUM> to <NUM> decomposed from the <NUM> × <NUM> original pooling kernel <NUM>. In a pooling operation of the example of <FIG>, a stride may be <NUM>.

The processor <NUM> obtains intermediate pooling results <NUM> respectively corresponding to the sub-pooling kernels <NUM> to <NUM> by performing a pooling operation on input pixels included in a current window to be pooled in the input feature map <NUM> by using the sub-pooling kernels <NUM> to <NUM> decomposed from the original pooling kernel <NUM>.

When all of the intermediate pooling results <NUM> are obtained for the current window, the processor <NUM> may obtain a final pooling result corresponding to the current window by post-processing the intermediate pooling results <NUM>. The processor <NUM> may then determine an output pixel value of an output feature map <NUM>, based on the final pooling result.

In a particular example as shown in the example of <FIG>, when the current window is a first window of the input feature map <NUM>, the sub-pooling kernel <NUM> may be mapped to input pixels of a first row included in the first window. The processor <NUM> may determine a maximum value "<NUM>" chosen from among values "<NUM>," "<NUM>," and "<NUM>" of the input pixels of the first row and may obtain the maximum value "<NUM>" as an intermediate pooling result corresponding to the sub-pooling kernel <NUM>. Also, the processor <NUM> may determine a maximum value "<NUM>" from among values "<NUM>," "<NUM>," and "<NUM>" of input pixels of a second row and a maximum value "<NUM>" from among values "<NUM>," "<NUM>," and "<NUM>" of input pixels of a third row, and obtains the maximum values "<NUM>" and "<NUM>" as intermediate pooling results corresponding to the sub-pooling kernels <NUM> and <NUM>. In other words, intermediate pooling results <NUM> obtained by the sub-pooling kernels <NUM> to <NUM> as mapped to the first window may be elements of ("<NUM>," "<NUM>," "<NUM>").

When all of the intermediate pooling results <NUM> for the current window that is the example first window are obtained, the processor <NUM> may obtain the final pooling result corresponding to the current window that is the first window by post-processing the intermediate pooling results <NUM>. Because the pooling type described with reference to the example of <FIG> is MaxPool, the processor <NUM> may perform a post-processing by determining a maximum value "<NUM>," e.g., by implementing, "Max (<NUM>, <NUM>, <NUM>)," from the intermediate pooling results <NUM>. There, the processor <NUM> may thus determine the output pixel value "<NUM>" of the output feature map <NUM> corresponding to the current window or first window, based on the final pooling result "<NUM>.

Then, when the current window corresponds to a third window of the input feature map <NUM> according to a raster scan order, the sub-pooling kernels <NUM> to <NUM> may be respectively mapped to input pixels of first through third rows included in the third window. The processor <NUM> may determine maximum values ("<NUM>," "<NUM>," and "<NUM>") regarding the rows and may obtain elements ("<NUM>," "<NUM>," and "<NUM>") of intermediate pooling results <NUM> that correspond to the sub-pooling kernels <NUM> to <NUM>.

When all of the intermediate pooling results <NUM> for the current window that is the third window are obtained, the processor <NUM> may perform a post-processing of determining a maximum value "<NUM>" which is chosen as the "Max(<NUM>, <NUM>, <NUM>)" from the intermediate pooling results <NUM> as a final pooling result, and may determine an output pixel value "<NUM>" of the output feature map <NUM> corresponding to the current window that is the third window, based on the final pooling result "<NUM>.

In the example of <FIG>, the pooling operation may be performed on some windows, such as the first and third windows, of the input feature map <NUM>, but the processor <NUM> may perform the pooling operation on the remaining windows in a similar manner and may finally obtain values of output pixels of the output feature map <NUM>.

In other words, according to the pooling operation of the examples, the processor <NUM> may perform the pooling operation by using Equation <NUM> below when a <NUM> × <NUM> original pooling kernel includes, for example, nine elements designated as (a, b, c, d, e, f, g, h, i).

In other words, the processor <NUM> may perform intermediate pooling operations for each sub-pooling kernel, i.e., a sub-pooling kernel including elements of (a, b, c), a sub-pooling kernel including elements of (d, e, f), and a sub-pooling kernel including elements of (g, h, i) and may perform a post-processing pooling operation on intermediate pooling results, thereby obtaining a final pooling result. According to an example of MaxPool described with reference to the example of <FIG>, a pooling operator may be a max operator.

Meanwhile, the sub-pooling kernels <NUM> and <NUM> when the current window is the first window of the input feature map <NUM> and the sub-pooling kernels <NUM> and <NUM> when the current window is the third window of the input feature map <NUM> may perform the pooling operation base on using the same input pixels of the input feature map <NUM>. Accordingly, the intermediate pooling results obtained by the sub-pooling kernels <NUM> and <NUM> mapped to the first window may be reused as the intermediate pooling results of the sub-pooling kernels <NUM> and <NUM> mapped to the third window, in that they use some of the same information to perform their calculations in a similar manner. A pooling operation performed on input pixels that overlap in different windows of an input feature map are described in further detail in corresponding drawings below.

Even when a pooling type is not separately described in the description about the embodiments below, the examples below may be realized by being applied to a pooling type of MaxPool, a pooling type of average pooling (AvgPool), and other alternative pooling types available in other examples.

<FIG> is a diagram for describing a method of performing AvgPool by using sub-pooling kernels <NUM> to <NUM> decomposed from an original pooling kernel <NUM>, according to an example.

Referring to the example of <FIG>, an AvgPool operation may be performed on an input feature map <NUM> of a <NUM> × <NUM> size by the three <NUM> × <NUM> sub-pooling kernels <NUM> to <NUM> decomposed from the 3x3 original pooling kernel <NUM>. In a pooling operation of the example of <FIG>, a stride may be <NUM>.

The pooling operation performed by the sub-pooling kernels <NUM> to <NUM> decomposed from the original pooling kernel <NUM> with respect to the input feature map <NUM> as shown in the example of <FIG> may be similar to that described with reference to the example of <FIG>. However, the AvgPool pooling operation of the example of <FIG> is different from the MaxPool pooling operation of the example of <FIG> in a method of obtaining an intermediate pooling result and a method of obtaining a final pooling result.

In particular, according to the example of <FIG>, when a current window is a first window of the input feature map <NUM>, the processor <NUM> may add values ("<NUM>," "<NUM>," and "<NUM>") of input pixel values of a first row included in the first window mapped to the sub-pooling kernel <NUM>, and may obtains their sum "<NUM>," or (<NUM>+<NUM>+<NUM>) as an intermediate pooling result corresponding to the sub-pooling kernel <NUM>. In other words, a method of calculating an intermediate pooling result according to the AvgPool approach is different from a method of calculating an intermediate pooling result according to the MaxPool approach.

Similarly, the processor <NUM> may obtains a sum "<NUM>," or (<NUM>+<NUM>+<NUM>) of values ("<NUM>", "<NUM>", and "<NUM>') of input pixels of a second row mapped to the sub-pooling kernel <NUM> and a sum "<NUM>," or (<NUM>+<NUM>+<NUM>) of values ("<NUM>," "<NUM>," and "<NUM>") of input pixels of a third row mapped to the sub-pooling kernel <NUM>, respectively, as intermediate pooling values corresponding to the sub-pooling kernels <NUM> and <NUM>. In other words, intermediate pooling results <NUM> obtained by the sub-pooling kernels <NUM> to <NUM>, as mapped to the first window, are elements of the group ("<NUM>," "<NUM>," and "<NUM>").

When all of the intermediate pooling results <NUM> for the current window that is the first window are obtained, the processor <NUM> may obtain the final pooling result corresponding to the current window that is first window by post-processing the intermediate pooling results <NUM>. Because the pooling type described with reference to the example of <FIG> is AvgPool, the processor <NUM> may add the intermediate pooling results <NUM> "<NUM>," "<NUM>," and "<NUM>," and may perform a post-processing of dividing the sum "<NUM>," or (<NUM>+<NUM>+<NUM>) by the size <NUM> × <NUM>, or <NUM>, of the original pooling kernel <NUM>. Accordingly, the processor <NUM> may determine an output pixel value "<NUM>" of an output feature map <NUM> corresponding to the current window that is the first window, based on a final pooling result "<NUM>," or (<NUM>/<NUM>).

Similarly, when the current window is a third window of the input feature map <NUM>, the processor <NUM> may perform an intermediate pooling operation on input pixels mapped to each of the sub-pooling kernels <NUM> to <NUM> to obtain elements of intermediate pooling results <NUM> ("<NUM>," "<NUM>," and "<NUM>").

When the intermediate pooling results <NUM> for the current window that is the third window are all obtained, the processor <NUM> may add the intermediate pooling results <NUM> "<NUM>," "<NUM>," and "<NUM>" and may perform a post-process of dividing the sum "<NUM>," or (<NUM>+<NUM>+<NUM>) by a size <NUM> × <NUM>, or <NUM>, of the original pooling kernel <NUM>. Accordingly, the processor <NUM> may determine an output pixel value "<NUM>" of an output feature map <NUM> corresponding to the current window that is the third window, based on a final pooling result "<NUM>," or (<NUM>/<NUM>).

In the example of <FIG>, the pooling operation may be performed on some windows, including the first and third windows, of the input feature map <NUM>, but the processor <NUM> may perform the pooling operation on the remaining windows in a similar manner and may finally obtain values of output pixels of the output feature map <NUM>.

In other words, according to the pooling operation of the examples, the processor <NUM> may perform the pooling operation by using Equation <NUM> above when a <NUM> × <NUM> original pooling kernel includes, for example, elements of (a, b, c, d, e, f, g, h, i). However, unlike the example of <FIG>, a pool operator calculating an intermediate pooling result may be a sum/add operator, and a pool operator calculating a final pooling result may be an averaging operator.

As in the example of <FIG>, the sub-pooling kernels <NUM> and <NUM> when the current window is the first window of the input feature map <NUM> and the sub-pooling kernels <NUM> and <NUM> when the current window is the third window of the input feature map <NUM> may perform the pooling operation on the same input pixels of the input feature map <NUM>. Accordingly, the intermediate pooling results obtained by the sub-pooling kernels <NUM> and <NUM> mapped to the first window may be reused as the intermediate pooling results of the sub-pooling kernels <NUM> and <NUM> mapped to the third window. As discussed above, certain information may be reused in a similar manner, avoiding redundant calculation. A pooling operation performed on input pixels that overlap in different windows of an input feature map are described in further detail in the corresponding drawings, below.

In the examples of <FIG> and <FIG>, for convenience of description, a <NUM> × <NUM> input feature map, a <NUM> × <NUM> original pooling kernel, and a stride <NUM> have been described as a non-limiting example, but the pooling operations described with reference to the examples <FIG> and <FIG> may be readily applied in the same manner to an input feature map of another size, an original pooling kernel of another size, and a stride of another value, in other examples.

<FIG> is a diagram for describing a method, performed by a processor and a share line buffer of a memory, of processing a pooling operation using sub-pooling kernels <NUM> to <NUM>, according to an example. As a non-limiting example and for convenience of explanation, below, the processor, share line buffer, and memory may correspond to processor <NUM>, share line buffer <NUM>, and memory <NUM> of <FIG>.

Referring to the example of <FIG>, a <NUM> × <NUM> original pooling kernel <NUM> may be decomposed into the <NUM> × <NUM> sub-pooling kernels <NUM> to <NUM>, and a pooling operation may be performed on an input feature map <NUM> by using the sub-pooling kernels <NUM> to <NUM>. However, in the examples, sizes of the original pooling kernel <NUM> and the sub-pooling kernels <NUM> to <NUM> and a size of the input feature map <NUM> are not limited to the non-limiting examples shown in the example of <FIG> and may vary and have other values.

The example processor <NUM> may include at least one arithmetic logic unit (ALU). As a non-limiting example in <FIG>, the processor <NUM> includes ALU <NUM>-<NUM> and ALU <NUM>-<NUM>, each of which performs an arithmetic operation related to the pooling operation. For convenience of description, two ALUs <NUM>-<NUM> and <NUM>-<NUM> are discussed in the example of <FIG>, but ALUs in the processor <NUM> may be the same single ALU and/or the processor <NUM> may include three or more ALUs.

The processor <NUM> may receive a value of a current input pixel <NUM> included in a current window of the input feature map <NUM>, according to a raster scan order for the input feature map <NUM>. The processor <NUM> may update at least one partial pooling result stored in at least one memory cell in the share line buffer <NUM>, where the partial pooling result is affected by the value of the current input pixel <NUM>, in accordance with the value of the current input pixel <NUM>. In such an example, the ALU <NUM>-<NUM> of the processor <NUM> may perform an arithmetic operation for updating the partial pooling result.

In such an example, the partial pooling result may be an intermediate value for obtaining an intermediate pooling result for all input pixels mapped to one sub-pooling kernel. For example, the sub-pooling kernel <NUM> may be mapped to <NUM> input pixels total in one window, and an intermediate pooling result for the <NUM> input pixels may be obtained only when all data of the mapped <NUM> input pixels have been received. However, because the input pixels of the input feature map <NUM> may be sequentially input to the processor <NUM> according to the raster scan order, it may be difficult to obtain the intermediate pooling result for the <NUM> input pixels at the same time.

Accordingly, in the case of a MaxPool approach, when a value of a first input pixel mapped to the sub-pooling kernel <NUM> is received, the processor <NUM> may store the value of the first input pixel as a maximum value in one memory cell of the share line buffer <NUM>. When a value of a second input pixel is received, the processor <NUM> may compare the already stored maximum value, that is, the value of the first input pixel, to the value of the second input pixel and may update data of the memory cell to a maximum value from among these values, as necessary. In such an example, the data stored in the memory cell may correspond to the partial pooling result. Finally, when a value of a last third input pixel mapped to the sub-pooling kernel <NUM> is received, the processor <NUM> may compare the already stored maximum value, that is, the maximum value from among the value of the first input pixel and the value of the second input pixel, to the value of the third input pixel and finally updates the data of the memory cell to a maximum value from among all three of the input pixels. Because the sub-pooling kernel <NUM> may have the <NUM> × <NUM> size, the data of the memory cell finally updated by the value of the third input pixel may correspond to an intermediate pooling result corresponding to the sub-pooling kernel <NUM>, in that each input pixel may have had an opportunity to affect the maximum pixel value.

Similarly, in the case of AvgPool, the processor <NUM> updates the partial pooling result by adding a received value of an input pixel and data already stored in the memory cell of the share line buffer <NUM>.

In other words, when performing the pooling operation, the processor <NUM> may read pooling data from the share line buffer <NUM> of the memory <NUM>, may update a partial pooling result already stored in the share line buffer <NUM> by using the ALU <NUM>-<NUM>, and may write the partial pooling result to the share line buffer <NUM>. By storing the accumulated partial pooling results, the information from the input pixels is able to be stored temporarily, thus avoiding the need for all pooling data to be available before the pooling process can begin.

The ALU <NUM>-<NUM> of the processor <NUM> may read intermediate pooling results for a current window from memory cells of the share line buffer <NUM> and may perform a post-processing according to a pre-set pooling type on the read intermediate pooling results to obtain a final pooling result corresponding to an input pixel value. In the example of <FIG>, because the <NUM> × <NUM> original pooling kernel <NUM> may be decomposed into the three <NUM> × <NUM> sub-pooling kernels <NUM> to <NUM>, the ALU <NUM>-<NUM> may read and post-process three intermediate pooling results retrieved from the share line buffer <NUM>.

When the <NUM> × <NUM> original pooling kernel <NUM> is decomposed into the three <NUM> × <NUM> sub-pooling kernels <NUM> to <NUM> as in the example of <FIG>, the share line buffer <NUM> may store pooling data in memory cells of <NUM> rows total. Alternatively put, the share line buffer <NUM> may store the intermediate pooling results obtained for the windows in the input feature map <NUM> in memory lines, shown as Row k*i, Row k*i+<NUM>, and Row k*i+<NUM>, of the number of rows corresponding to the height of the original pooling kernel <NUM>, in a circular manner. In such an example, k refers to the height of the original pooling kernel <NUM>, and according to the example of <FIG>, k=<NUM>. However, <FIG> is a non-limiting example, and k may take on other values in other examples.

In particular, the intermediate pooling results corresponding to the same window in the input feature map <NUM> may be respectively stored in memory cells having memory addresses of the same column and different rows in the share line buffer <NUM>. For example, when three intermediate pooling results respectively corresponding to the three sub-pooling kernels <NUM> to <NUM> are obtained for a window <NUM> of the input feature map <NUM>, the three intermediate pooling results may be respectively stored in memory cells <NUM>, <NUM>, and <NUM> of a first row and a first column, a second row and the first column, and a third row and the first column of the share line buffer <NUM>, according to this approach.

When the stride is <NUM> and a window <NUM> to be pooled from the input feature map <NUM> has the same columns as the window <NUM> previously pooled, the window <NUM> and the window <NUM> may share input pixels of two rows. In other words, at least one of sub-pooling kernels regarding the current window <NUM> may be sharable as a sub-pooling kernel regarding at least one other window <NUM> in the input feature map <NUM>. Accordingly, an intermediate pooling result obtained by a sharable sub-pooling kernel among the sub-pooling kernels mapped to the current window <NUM> may be shared with respect to the at least one other window <NUM>.

When the intermediate pooling results for the window <NUM> are all stored in the memory cells <NUM>, <NUM>, and <NUM>, the ALU <NUM>-<NUM> of the processor <NUM> may post-process the intermediate pooling results stored in the memory cells <NUM>, <NUM>, and <NUM> so as to output a final pooling result for the window <NUM>. According to the raster scan order, at a time at which pooling is performed on the window <NUM> afterwards, data stored in the memory cell <NUM> corresponding to the intermediate pooling result of the sub-pooling kernel <NUM> for the window <NUM> may no longer be used for the pooling operation. However, because the window <NUM> shares two rows with the window <NUM>, the intermediate pooling results of the sub-pooling kernels <NUM> and <NUM> for the window <NUM> may be reused for the window <NUM>. Accordingly, the intermediate pooling results stored in the memory cells <NUM> and <NUM> may not be deleted and may instead be reused to pool the window <NUM>.

In other words, an intermediate pooling result stored in one memory cell of the share line buffer <NUM> may be deleted or over-written for an intermediate pooling result obtained by another sub-pooling kernel to be stored, when the intermediate pooling result stored in the one memory cell is no longer shared to obtain a final pooling result corresponding to another window. In this manner, the intermediate pooling results may be stored in the memory cells of the same column of the share line buffer <NUM>, in a circular manner.

The examples described with reference to the example of <FIG> may be applied to a pooling operation based on an input feature map of another size, an original pooling kernel of another size, and a stride of another value, in the same manner, in various examples, and the particular details of the example of <FIG> are to be taken as non-limiting.

<FIG> is a diagram for describing processes of updating pooling data and generating an output feature map <NUM> from the pooling data, according to an example.

Referring to the example of <FIG>, an input pixel IFM_4-<NUM> of an input feature map <NUM> may correspond to a current input pixel overlapped or shared in different sub-pooling kernels. In such an example, the different sub-pooling kernels may be indicated by dotted lines and broken lines as indicated in the example of <FIG>. For example, the different sub-pooling kernels may correspond to a third sub-pooling kernel of a <NUM> × <NUM> original pooling kernel mapped to a window A, a third sub-pooling kernel of a <NUM> × <NUM> original pooling kernel mapped to a window B, and a third sub-pooling kernel of a <NUM> × <NUM> original pooling kernel mapped to a window C.

When the current input pixel is received, the processor <NUM> may update at least one partial pooling result affected by a value of the current input pixel. For example, when the input pixel IFM_4-<NUM> is received according to a raster scan order, the processor <NUM> may update partial pooling results IPR_4-<NUM>, IPR_4-<NUM>, and IPR_4-<NUM> based on the way in which they are affected by the input pixel IFM_4-<NUM>. When the updating of the partial pooling result IPR_4-<NUM> is complete, the finally updated partial pooling result IPR_4-<NUM> may correspond to an intermediate pooling result.

The processor <NUM> may obtain a final pooling result by performing a post-processing on intermediate pooling results <NUM>, i.e., IPR_2-<NUM>, IPR_3-<NUM>, and IPR_4-<NUM>. The obtained final pooling result may correspond to an output pixel OFM_2-<NUM> of the output feature map <NUM>.

As such, when the processor <NUM> receives any one input pixel value, at least one partial pooling result stored in the share line buffer <NUM> and affected by the input pixel value may also be updated.

Generally, without the sub-pooling kernel approach discussed herein, when a value of any single input pixel is received, all pooling results for several windows including the input pixel may be repeatedly read/updated/written. For example, when a sufficiently large input feature map is pooled based on a <NUM> × <NUM> pooling kernel of a stride of <NUM>, a processor and a share line buffer may read/update/write all pieces of pooling data corresponding to <NUM> pixel values of each of maximum <NUM> windows sharing the input pixel whenever the input pixel is received. Accordingly, a throughput of the processor and an access frequency between the processor and a memory, such as the share line buffer, is relatively high. However, when a pooling method according to the embodiments herein with sub-pooling approaches, e.g., employing 1D sub-pooling kernels decomposed from an original pooling kernel, less than all pieces of pooling data of the assigned kernel pieces of pooling data of the 1D sub-pooling kernels sharing an input pixel may be read/updated/written, which may therefore effectively reduce overhead of a processor and a memory, such as the shared line buffer.

<FIG> is a flowchart of a neural network method, according to an example. Referring to the example of <FIG>, the neural network method may be performed or implemented in a time-based series by the neural processing apparatus <NUM>, such as in the example of <FIG>. As a hyper-parameter of the pooling operation of the neural network method of the example of <FIG>, a size of an original pooling kernel may be k × k and a stride may s, wherein k and s are natural numbers and s < k.

In addition, the pooling operations shown in the example of <FIG> may merely be an operation in which one input pixel of an input feature map is received. Accordingly, the neural network method of the example of <FIG> may be repeatedly performed on all pixels of the input feature map, and a current input pixel (X,Y) may be sequentially received according to a raster scan order of the input feature map in operation <NUM> of the example of <FIG>, such that all of the input pixels are processed.

In operation <NUM>, the processor <NUM> may receive the current input pixel (X,Y) in the input feature map, the input feature map having a width of W and a height of H. In such an example, <NUM> ≤ X < Wand <NUM> ≤ Y< H.

In operation <NUM>, the processor <NUM> may set a memory cell of an address (xp, yp) among memory cells of the share line buffer <NUM> to correspond to an update pivot and may set an update size of the memory cell of the share line buffer <NUM> to correspond to t, wherein t is a natural number. Then, the processor <NUM> may initialize i to <NUM>.

In such an example, in the update pivot (xp, yp), xp = X % k and yp = Y % k, and in the update size t, t is assigned the value of the update size.

In operation <NUM>, the processor <NUM> may determine whether a condition of i < t is satisfied. When the condition of i < t is satisfied, operation <NUM> may be performed. However, when the condition of i < t is not satisfied, operation <NUM> may be performed instead.

In operation <NUM>, the processor <NUM> may set a target address from among the memory cells of the share line buffer <NUM> to be (xp + i, yp).

In operation <NUM>, the processor <NUM> may increase i by <NUM>.

The processor <NUM> may perform operations <NUM> to <NUM> until the condition of i < t is no longer satisfied.

In operation <NUM>, in order to obtain a final pooling result, the processor <NUM> may determine whether intermediate pooling results capable of being post-processed, for example, a <NUM> × k kernel such as the reference numeral <NUM> of the example of <FIG>, may be present on the share line buffer <NUM>. When there are no intermediate pooling results capable of being post-processed on the share line buffer <NUM>, the neural network method regarding the current input pixel (X,Y) is ended and operation <NUM> is initiated/restarted for a next input pixel.

In such an example, the processor <NUM> may determine whether an update is completed in the updated pivot regarding the current input pixel (X,Y) by using Equation <NUM> below as a conditional test and may determine whether the <NUM> × k kernel capable of performing post-process is present by using Equation <NUM> below as a conditional test. <MAT><MAT>.

In operation <NUM>, the processor <NUM> may access an xp column on the share line buffer <NUM> as a target column.

In operation <NUM>, the processor <NUM> may read intermediate pooling results (<NUM> × k size) stored in the xp column on the share line buffer <NUM>.

In operation <NUM>, the processor <NUM> may obtains the final pooling result by performing a post-processing on the read intermediate pooling results, according to a pre-set pooling type. The processor <NUM> determines that the obtained final pooling result corresponds to a value of an output pixel at a location ((X-k+<NUM>)/<NUM>, (Y-k+<NUM>)/<NUM>) of an output feature map.

The processor <NUM> may repeatedly perform the neural network method described in further detail, above, until the output feature map is completed as final pooling results are obtained for all intermediate pooling results obtained from the input feature map.

<FIG> is a diagram for describing a process of a processor of updating pooling data <NUM> on a share line buffer by receiving a current input pixel, according to an example. For convenience of explanation, the operations of <FIG> will be descried with reference to the processor <NUM> and share line buffer of <NUM> of <FIG>, noting that examples are not limited thereto.

Referring to the example of <FIG>, when the ALU <NUM>-<NUM> of the processor <NUM> receives a current input pixel in an input feature map <NUM>, the ALU <NUM>-<NUM> reads the pooling data <NUM> that is, a partial pooling result, stored in the share line buffer <NUM> affected by the current input pixel. As described in further detail above, the read pooling data <NUM> may be data stored in memory cells of the same row, but different columns.

The ALU <NUM>-<NUM> may update the pooling data <NUM> based on the pooling data <NUM> and a value of the current input pixel. For example, the ALU <NUM>-<NUM> may determine a maximum value according to a MaxPool technique as described in further detail, above, or perform an adding operation according to AvgPool as described in further detail, above.

In other words, according to the examples employing the 1D sub-pooling kernel, as described in further detail, above, only the pooling data <NUM> of 1D sub-pooling kernels sharing the current input pixel is read/updated/written at a given time, and thus, overheads of the processor <NUM> and the share line buffer <NUM> may be effectively reduced by managing data in this manner, compared to a general process without sub-pooling.

<FIG> is a diagram for describing a process of a processor of obtaining a final pooling result by post-processing intermediate pooling results <NUM> stored in a share line buffer, according to an example. For convenience of explanation, the operations of <FIG> will be descried with reference to the processor <NUM> and share line buffer of <NUM> of <FIG>, noting that examples are not limited thereto.

Referring to the example of <FIG>, when a partial pooling result stored in a memory cell <NUM> of the share line buffer <NUM> is finally updated, the finally updated partial pooling result corresponds to an intermediate pooling result.

The processor <NUM> may read intermediate pooling results <NUM> stored in all memory cells of the same column as the memory cell <NUM> in which the intermediate pooling result is stored. In such an example, it is assumed that the intermediate pooling results <NUM> are all finally updated results, as discussed in further detail, previously.

The processor <NUM> may obtain a final pooling result corresponding to the intermediate pooling results <NUM> by performing a post-processing on the intermediate pooling results <NUM>, according to a pre-set pooling type. Various non-limiting examples of pooling have been discussed in greater detail, above. As a result, the processor <NUM> may generate an output feature map based on a value of an output pixel <NUM> corresponding to the final pooling result.

Intermediate pooling results stored in the same column of the share line buffer <NUM> may be, as described above, pre-processed results of a pooling operation performed by each of sub-pooling kernels mapped to a window. Accordingly, the processor <NUM> may complete pooling on the corresponding window by performing a post-processing of a pooling operation of merging the intermediate pooling results stored in the same column.

<FIG> is a flowchart of a neural network method, performed by an apparatus, including processing pooling of a neural network, according to an example. Referring to the example of <FIG>, because the neural network method of <FIG> is related to the examples described above with reference to the drawings, the descriptions above are also applicable to the operations of <FIG>.

In operation <NUM>, the processor <NUM> obtains intermediate pooling results respectively corresponding to sub-pooling kernels by performing a pooling operation on input pixels included in a current window to be pooled in an input feature map by using a plurality of sub-pooling kernels decomposed from an original pooling kernel. The current window is determined as the original pooling kernel is slid according to a raster scan order in the input feature map.

In operation <NUM>, when all of the intermediate pooling results are obtained for the current window, the processor <NUM> may obtain a final pooling result corresponding to the current window by post-processing the intermediate pooling results.

In operation <NUM>, the processor <NUM> determines an output pixel value of an output feature map based on the final pooling result.

The pooling of the neural network according to the examples is processed based on a hyper-parameter including information about a size of the original pooling kernel, a stride size, and a pooling type. Alternatively put, the number of sub-pooling kernels to be decomposed, the minimum required number of memory lines of the share line buffer <NUM>, and an update pivot and update size of the share line buffer <NUM> described with reference to the example of <FIG> may all be set based on the hyper-parameter. In addition, the share line buffer <NUM> storing the obtained intermediate pooling results may be addressed based on the hyper-parameter.

The neural processing apparatuses, neural processing apparatus <NUM>, processors, processor <NUM>, ALUS, ALUs <NUM>-<NUM>, <NUM>-<NUM>, memories, memory <NUM>, share line buffers, and share line buffer <NUM>, in <FIG> that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
A processor-implemented method of performing image recognition by using a neural network, the method comprising:
obtaining (<NUM>) intermediate pooling results, respectively corresponding to sub-pooling kernels (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) obtained by decomposing an original pooling kernel (<NUM>, <NUM>, <NUM>), by performing a pooling operation on pixels of an input image included in a current window in an input feature map (<NUM>) using the sub-pooling kernels (<NUM>, <NUM>, <NUM>);
obtaining (<NUM>) a final pooling result corresponding to the current window by post-processing the intermediate pooling results; and
determining (<NUM>) an output pixel value of an output feature map, based on the final pooling result,
wherein the current window is determined according to the original pooling kernel (<NUM>) having been slid, according to a raster scan order, in the input feature map (<NUM>),
wherein each of the sub-pooling kernels (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) is a <NUM>-dimensional, 1D, kernel that has a height of <NUM>, respectively comprising only row elements of the original pooling kernel (<NUM>, <NUM>, <NUM>), and a total number of the <NUM> height sub-pooling kernels (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) obtained by decomposing from the original pooling kernel (<NUM>, <NUM>, <NUM>) corresponds to a height of the original pooling kernel (<NUM>), and
wherein an intermediate pooling result obtained by a sub-pooling kernel from among the sub-pooling kernels (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) with respect to the current window is shared with at least one other window in the input feature map (<NUM>) by respectively storing the intermediate pooling results that correspond to a same window in memory cells comprising memory addresses (<NUM>, <NUM>, <NUM>) of a same column and different rows in a share line buffer (<NUM>).