IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND NON-TRANSITORY COMPUTER-READABLE STORAGE MEDIUM

An image processing apparatus comprises a first obtaining unit configured to obtain a pixel from a feature image, and a calculating unit configured to perform a convolution operation based on a pixel obtained by the first obtaining unit. The first obtaining unit is capable of obtaining non-adjacent pixels from the feature image.

BACKGROUND

Field of the Disclosure

The present disclosure relates to convolution processing.

Description of the Related Art

With recent advancements in deep learning, the accuracy of image recognition is also improving. A convolutional neural network (CNN) is a known technique used in deep learning.

In a CNN, a plurality of layers are connected in a hierarchical manner, with a plurality of feature image existing in each layer. In the example of a CNN configuration illustrated inFIG. 2, the number of layers is four and the number of feature images in each layer is four. In a CNN, convolution processing is executed using a learned coefficient (weighting coefficient) and the pixels (feature data) of the feature images. Convolution processing is a multiply-accumulate operation including a plurality of multiplications and a cumulative sum. Each arrow illustrated inFIG. 2indicates a multiply-accumulate operation.

Feature images of a current layer are calculated using feature images of a preceding layer and filter coefficients corresponding to the preceding layer. To calculate one feature image of the current layer, the information of a plurality of feature images of the preceding layer is required. The multiply-accumulate operation for calculating a feature image of the current layer is performed as follows (Formula 1).

Herein, n is an index of a feature image in the current layer, and m (m=1 to M) is an index of a feature image in the preceding layer. Oi,j(n) represents feature data (a multiply-accumulate operation result) corresponding to a position (i,j) in a feature image with an index of n in the current layer. Ii,j(m) represents feature data corresponding to a position (i,j) in a feature image I(m) with an index of m in the preceding layer. Cx,y(m,n) represents coefficients between a feature image with an index of n in the current layer and feature data corresponding to a position (x,y) in a feature image with an index of m in the preceding layer. In Formula 1, there are (X×Y) coefficients (C0,0(m,n) to CX−1,Y−1(m,n)), and the coefficients differ for each feature image. X and Y are variables representing a reference range. The multiply-accumulate operation for calculating the feature data of the current layer is performed (M×X×Y) times.

After the multiply-accumulate operation (filter processing) described above has been executed, based on the network structure of a CNN, the feature images of the current layer are calculated by executing processing such as activation processing and pooling using a multiply-accumulate operation result Oi,j(n).

CNNs are also used in applications such as image segmentation and the like. Dilated convolution described in Huikai Wu, Junge Zhang, Kaiqi Huang, Kongming Liang, and Yizhou Yu, “FastFCN: Rethinking dilated convolution in the backbone for semantic segmentation,” 2019, CoRR, abs/1903.11816 is a technique for improving the accuracy of image segmentation with a lost calculation cost. When performing dilated convolution, a multiply-accumulate operation is performed as follows (Formula 2).

Herein, the variable R is the dilation rate of the dilated convolution processing. When the variable R is 1, Formula 2 is the same as Formula 1. The larger the value of the variable R, the wider the reference range in the feature image of the preceding layer. After dilation, the reference range changes from (X×Y) to [R×(X−1)+1]×[R×(Y−1)+1]. In this operation, the processing is performed without skipping coefficients, and to process feature data of a feature image at intervals of (R−1) data, however, feature data in the horizontal direction or the vertical direction are referred to as they are skipped.

A multiply-accumulate operation in which a coefficient (weighting coefficient) C used in the multiply-accumulate operation (Formula 2) in the filter processing is placed with a coefficient C′ in an extended (dilated) filter as described below (in Formula 4) is performed as follows (Formula 3).

Formula (3) is as described above in terms of the variables in common with Formula (1) and Formula (2), and thus a description thereof is omitted. The coefficient C′x,y(m,n) is represented by Formula 4 below.

When x and y are multiples of R, the value of the coefficient C′x,y(m,n) is the same as that of the coefficient Cx/R,y/R(m,n) and is a significant value (effective coefficient). On the other hand, if the values of x and y are not multiples of R, the value of the coefficient C′x,y(m, n) is 0, and this means that the calculation will be omitted. In this case,

is a floor function that outputs a maximum integer equal to or less than X. In a CNN, multiply-accumulate operations are performed many times. Thus, in a case where the CNN is applied to a portable terminal or an embedded system such as an in-vehicle device, it is necessary to reduce the transfer amounts of feature data and coefficients, efficiently perform multiply-accumulate operations, and shorten the overall processing time. In US 2020/0410036 and U.S. Ser. No. 10/861,123, a configuration is described that processes a plurality of feature data in parallel.

Dilated convolution can achieve high recognition accuracy with a low calculation cost, but because the coordinates of the reference data and the address of the memory or the like are not continuous, the processing efficiency of the hardware may be decreased.

In the technique described in US 2020/0410036, output data is calculated in parallel using a systolic array. The dilated convolution described in Huikai Wu, Junge Zhang, Kaiqi Huang, Kongming Liang, and Yizhou Yu, “FastFCN: Rethinking dilated convolution in the backbone for semantic segmentation,” 2019, CoRR, abs/1903.11816 can be performed referencing a network parameter and using a zero coefficient represented by Formula 4, however referencing while skipping feature data of feature image I(m) cannot be performed.

In the technique described in U.S. Ser. No. 10/861,123, output data is calculated in parallel using a coefficient common among different feature data. In a case where the dilated convolution described in Huikai Wu, Junge Zhang, Kaiqi Huang, Kongming Liang, and Yizhou Yu, “FastFCN: Rethinking dilated convolution in the backbone for semantic segmentation,” 2019, CoRR, abs/1903.11816 is performed, a zero coefficient represented by Formula 4 can be used. In a case where each pixel and each coefficient are read out in one action, dilated convolution processing can be efficiently performed. However, because referencing cannot be performed while skipping feature data of a feature image I(m), all of the zero coefficient processing cannot be omitted.

SUMMARY

The present disclosure provides technology for reducing the processing time by skipping pixels in a feature image when referencing when performing a convolution operation.

According to the first aspect of the present disclosure, there is provided an image processing apparatus, comprising: a first obtaining unit configured to obtain a pixel from a feature image; and a calculating unit configured to perform a convolution operation based on a pixel obtained by the first obtaining unit, wherein the first obtaining unit is capable of obtaining non-adjacent pixels from the feature image.

According to the second aspect of the present disclosure, there is provided an image processing method, comprising: obtaining a pixel from a feature image; and performing a convolution operation based on the obtained pixel, wherein in the obtaining, non-adjacent pixels are able to be obtained from the feature image.

According to the third aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing a computer program for causing a computer to function as a first obtaining unit configured to obtain a pixel from a feature image; and a calculating unit configured to perform a convolution operation based on a pixel obtained by the first obtaining unit, wherein the first obtaining unit is capable of obtaining non-adjacent pixels from the feature image.

DESCRIPTION OF THE EMBODIMENTS

First Embodiment

First, a hardware configuration example of an image processing apparatus that performs a convolution operation in a hierarchical neural network (such as the CNN described above) will be described using the block diagram ofFIG. 13. The image processing apparatus according to the present embodiment is applicable to a personal computer (PC), a tablet terminal apparatus, a smart phone, or other similar computer apparatus. Also, as long as the processing described below as being executed by the image processing apparatus can be executed, the configuration of the image processing apparatus is not limited to that illustrated inFIG. 13, and another configuration may be used.

An input unit1301is a user interface such as a keyboard, a mouse, or a touch panel. The user can input various instructions to a CPU1306by operating the input unit1301. Note that the input unit1301is not limited to being included in the image processing apparatus.

A data storage unit1302is a large capacity information storage apparatus such as a hard disk drive. An operating system (OS), computer programs for executing or controlling, via the CPU1306, processing described as being executed by the image processing apparatus, and data are stored in the data storage unit1302. The computer programs and data stored in the data storage unit1302are loaded as appropriate on a RAM1308in accordance with control by the CPU1306and are the targets for processing by the CPU1306.

Examples of the data storage unit1302include a flexible disk, a CD-ROM, a CD-R, a DVD, a memory card, a CF card, a smart medium, an SD card, a memory stick, a xD picture card, a USB memory, and other similar memory apparatuses.

Also, the data storage unit1302is not limited to being included in the image processing apparatus and may be provided on a network using a wired or wireless connection via a LAN, the Internet, or the like. In this case, the image processing apparatus accesses the data storage unit1302via a communication unit1303and reads and writes computer programs and data from/to the data storage unit1302. The communication unit1303communicates data with an external apparatus via the network described above.

A display unit1304includes a liquid crystal screen or a touch panel screen and displays results of the processing by the CPU1306using images, characters, and the like. Note that the display unit1304is not limited to being included in the image processing apparatus. Also, the display unit1304may be a projecting apparatus such as a projector that projects images or characters. Furthermore, the input unit1301and the display unit1304may be integrally formed, forming a touch panel screen.

A CNN processing unit1305inputs a processing target image (input image) stored in the RAM1308into an input layer of the CNN, executes calculation processing at each layer of the CNN, and stores the calculation processing result in the RAM1308or the data storage unit1302.

The CPU1306executes various processing using computer programs and data stored in the RAM1308and a ROM1307. Accordingly, the CPU1306performs operation control of the entire image processing apparatus and executes or controls various processing described below as processing executed by the image processing apparatus.

Note that the method of storing the computer programs and data in the RAM1308is not limited to a specific method, and, for example, the computer programs and data received from an external apparatus via the communication unit1303may be stored in the RAM1308after being temporarily stored in the data storage unit1302. Also, the computer programs and data received from an external apparatus via the communication unit1303may be directly stored in the RAM1308.

The CPU1306can execute various processing associated with the input image using the result of calculation processing executed on the input image stored in the RAM1308or the data storage unit1302by the CNN processing unit1305. The various processing associated with the input image includes, for example, face detection from the input image, facial recognition using the input image, counting the number of faces in the input image, and the like. The CPU1306stores the results of various processing including the various processing associated with the input image in the RAM1308or the data storage unit1302.

Note that inFIG. 13, there is one CPU1306, however there may be two or more.

Setting data of the image processing apparatus, computer programs and data associated with activating the image processing apparatus, computer programs and data associated with basic operations of the image processing apparatus, and the like are stored in the ROM1307.

The RAM1308includes an area for storing computer programs and data loaded from the data storage unit1302or the ROM1307and an area for storing the calculation processing result output from the CNN processing unit1305. Also, the RAM1308includes an area for storing computer programs and data received from an external apparatus via the communication unit1303and a working area used when the CPU1306or an image processing unit1309executes one of the various items of processing. The RAM1308of such a configuration can provide various areas as appropriate.

When the image processing unit1309receives an image processing instruction from the CPU1306, the image processing unit1309executes image processing such as range adjustment of the pixel values of the pixels in the image on the image stored in the RAM1308or the data storage unit1302.

The input unit1301, the data storage unit1302, the communication unit1303, the display unit1304, CNN processing unit1305, the CPU1306, the ROM1307, the RAM1308, and the image processing unit1309are all connected to a system bus1310.

Next, a configuration example of the CNN will be described usingFIG. 2. The CNN illustrated inFIG. 2includes four layers (layer 1, layer 2, layer 3, and layer 4). The layer 1 includes four feature images (feature image (1,1), (1,2), (1,3), (1,4)), and the layer 2 includes four feature images (feature image (2,1), (2,2), (2,3), (2,4)). The layer 3 includes four feature images (feature image (3,1), (3,2), (3,3), (3,4)), and the layer 4 includes four feature images (feature image (4,1), (4,2), (4,3), (4,4)).

In the layer 1, the multiply-accumulate operation of Formula 2 is performed using a coefficient filter for the feature images (1,1), (1,2), (1,3), (1,4), generating the feature images (2,1), (2,2), (2,3), (2,4) of the layer 2 as output feature images.

In the layer 2, the multiply-accumulate operation of Formula 2 is performed using a coefficient filter for the feature images (2,1), (2,2), (2,3), (2,4), generating the feature images (3,1), (3,2), (3,3), (3,4) of the layer 3 as output feature images.

In the layer 3, the multiply-accumulate operation of Formula 2 is performed using a coefficient filter for the feature images (3,1), (3,2), (3,3), (3,4), generating the feature images (4,1), (4,2), (4,3), (4,4) of the layer 4 as output feature images.

Also, a dilated convolution processing dilation rate R of the layer 1 is set to 1, the dilated convolution processing dilation rate R of the layer 2 is set to 2, and the dilated convolution processing dilation rate R of the layer 3 is set to 4. The size of the coefficient filter prior to dilation is 3×3, and in each layer, a multiply-accumulate operation is performed with the feature image using the post-dilation coefficient filter obtained by dilation of the pre-dilation coefficient filter according to a dilated convolution processing dilation rate corresponding to the layer.

An example of dilation using a coefficient filter for each layer is illustrated on the left side ofFIG. 17. In the layer 1, as illustrated in frame1701, the multiply-accumulate operation of Formula 2 is performed with the feature image using a post-dilation coefficient filter (coefficient filter C′x, y on the right side) obtained by dilation of the pre-dilation coefficient filter (coefficient filter Cx, y on the left side) according to the dilated convolution processing dilation rate R of 1. In a case where R is 1, the size does not change from the pre-dilation coefficient filter to the post-dilation coefficient filter.

In the layer 2, as illustrated in frame1702, the multiply-accumulate operation of Formula 2 is performed with the feature image using a post-dilation coefficient filter (coefficient filter C′x, y on the right side) obtained by dilation of the pre-dilation coefficient filter (coefficient filter Cx, y on the left side) according to the dilated convolution processing dilation rate R of 2. The post-dilation coefficient filter is a 5×5 coefficient filter obtained by inserting a single inactive coefficient (for example, a coefficient of 0) between the coefficients of the pre-dilation coefficient filter.

In the layer 3, as illustrated in frame1703, the multiply-accumulate operation of Formula 2 is performed with the feature image using a post-dilation coefficient filter (coefficient filter C′x, y on the right side) obtained by dilation of the pre-dilation coefficient filter (coefficient filter Cx, y on the left side) according to the dilated convolution processing dilation rate R of 4. The post-dilation coefficient filter is a 9×9 coefficient filter obtained by inserting three inactive coefficients between the coefficients of the pre-dilation coefficient filter. In this manner, the dilated convolution processing dilation rate R is different for each layer in the CNN.

FIG. 12is a diagram illustrating an example of the CNN and convolution processing. Feature data is extracted from the same position of four feature images1201in the layer 1, and a result of the multiply-accumulate operation using the coefficient filter is calculated. The result corresponds to feature data of the same position in the next layer, the layer 2, and a feature image1202including the feature data is obtained.

FIG. 4is a block diagram illustrating a configuration example of the CNN processing unit1305described above. The functional units illustrated inFIG. 4may be implemented via hardware or implemented via software (a computer program). In the case of the latter, the computer program is stored in the data storage unit1302. The operations of the CNN processing unit1305will now be described according to the flowchart ofFIG. 1.

In step S101, a control unit401reads out information (structural information) relating to the pre-dilation coefficient filter, the input feature image (the feature image of the layer 1 in the example ofFIG. 2), and the structure of the CNN from the data storage unit1302or the RAM1308and stores this in a data holding unit408. The structural information includes, for example, the multiply-accumulate operation calculation amount, the number of layers, the size of the feature image, the number of feature images in each layer, and the like.

Also, under control by the control unit401, the processing of steps S103to S111is executed for each layer in the CNN. In the example ofFIG. 2, first, the layer 1 is set as the target layer, and the processing of steps S103to S111is executed on the layer 1. Next, the layer 2 is set as the target layer, and the processing of steps S103to S111is executed on the layer 2. Then, the layer 3 is set as the target layer, and the processing of steps S103to Sill is executed on the layer 3.

In step S103, a read out unit405, under control by the control unit401, obtained the dilated convolution processing dilation rate R corresponding to the target layer from the data holding unit408. The read out unit405obtains 1 as the dilated convolution processing dilation rate R in a case where the target layer is the layer 1, obtains 2 as the dilated convolution processing dilation rate R in a case where the target layer is the layer 2, and obtains 4 as the dilated convolution processing dilation rate R in a case where the target layer is the layer 3.

In this manner, the dilated convolution processing dilation rate R can be set for each layer. Note that the dilated convolution processing dilation rate is not limited to being set for each layer and may be for each output feature image or set for each group of output feature images, for example.

Also, under control by the control unit401, the processing of steps S105to S111is executed for each output feature image (feature image of the next layer (next layer to be set as the target layer) subsequent to the target layer) generated at the target layer. In the example ofFIG. 2, in a case where the target layer is the layer 1, first, the processing of steps S105to S111to generate a feature image (2,1) is executed, and, next, the processing of steps S105to S111to generate a feature image (2,2) is executed. In a similar manner, next, the processing of steps S105to S111to generate a feature image (2,3) is executed, and, next, the processing of steps S105to S111to generate a feature image (2,4) is executed. In step S105, the control unit401initializes to 0 the convolution operation processing result held in a storage unit503(FIG. 5) of a processing unit406.

Also, under control by the control unit401, the processing of steps S107and S108is executed for each feature image in the target layer. In the example ofFIG. 2, in a case where the target layer is the layer 1, first, the feature image (1,1) is set to the target feature image, and the processing of steps S107and S108is executed for the target feature image. Next, the feature image (1,2) is set to the target feature image, and the processing of steps S107and S108is executed for the target feature image. Then, the feature image (1,3) is set to the target feature image, and the processing of steps S107and S108is executed for the target feature image. Next, the feature image (1,4) is set to the target feature image, and the processing of steps S107and S108is executed for the target feature image.

In step S107, in a case where the target layer is the layer 1, the control unit401transfers the input feature image stored in the data holding unit408to a holding unit402as the target feature image and transfers the pre-dilation coefficient filter stored in the data holding unit408to a holding unit404. In a case where the target layer is a layer A (A being an integer of 2 or greater), for the layer A, the feature image generated by a processing unit407is transferred and stored in the holding unit402as the target feature image. Because the pre-dilation coefficient filter has already been transferred to the holding unit404, step S107may be omitted.

In step S108, under control by the control unit401, the processing unit406executes convolution operation processing using the target feature image held in the holding unit402, the dilated convolution processing dilation rate R read out by the read out unit405, and the coefficient filter held in the holding unit404. The processing in step S108will be described in detail below usingFIG. 15.

At the point in time when the processing has advanced to step S110, convolution operation processing has been executed for all of the feature images in the processing target layer and the results (convolution operation processing results) of the convolution operation processing has been stored in the storage unit503(FIG. 5). However, in step S110, under control by the control unit401, the processing unit407executes activation processing by executing calculation processing based on Formula 5 below on the basis of each of the convolution operation processing results stored in the storage unit503.

Herein, f(x) is an activation function, and x is a convolution operation processing result. Also, in this example, the activation function is implemented using a rectified linear unit (ReLU). However, the activation function is not limited to the ReLU and may be implemented using another nonlinear function or a quantization function. Then, in accordance with the information of the layer, the processing unit407executes pooling processing on the basis of the activation processing result and adjusts the size of the output feature image as necessary.

In step S111, the control unit401stores the result (output feature image) of the activation and pooling processing obtained in step S110in the holding unit402as the feature image of the next layer (next layer to be set as the target layer) subsequent to the target layer.

Next, the processing in step S108described above will be described in detail with reference to the flowchart ofFIG. 15.FIG. 3is a block diagram illustrating a configuration example of the processing unit406. In this example, the holding unit402is configured to readably store the target feature image in units of pixel blocks (pixel sets) with a size of 6 pixels×6 pixels. For example, the control unit401divides the target feature image into a plurality of pixel sets (with there being an overlapping portion between adjacent pixel sets) and stores each pixel set in the holding unit402. In this example, as illustrated inFIG. 14A, the pixel (feature data) at position (i,j) (1≤i,j≤6) in a pixel set1401is represented by Di,j.

Also, in this example, the holding unit404is configured to store a coefficient filter with a 3×3 size. In this example, as illustrated inFIG. 14B, the coefficient at (i,j) (1≤i,j≤3) in the coefficient filter is represented by Ci,j, which is equivalent to Cx,y,(0≤x,y≤2) in Formula 1, 2.

In the present embodiment, there are four calculation devices, and the pixel values of different pixels are multiplied in parallel using a common coefficient and, as illustrated inFIG. 14C, four convolution processing results (O1,1, O1,2, O2,1, O2,2) of different positions are calculated in parallel.

In this example, the processing of steps S1503to S1512is executed for each pixel set in the target feature image. In step S1503, a control unit311sets, in setting unit301, a function (Formula 8) defining a pixel transfer pattern, a function (Formula 9) defining a coefficient transfer pattern, a function (Formula 10) defining the interval of the pixel to be transferred, and a function (Formula 11) defining the interval of the coefficient to be transferred. Formulas 8 to 11 will be described below.

In step S1504, the control unit311selects, as a selected pixel set, a single pixel set from among unselected pixel sets in the target feature image stored in the holding unit402, reads out the selected pixel set from the holding unit402, and stores the selected pixel set in a storage unit306. Also, the control unit311selects, as a selected coefficient filter, a corresponding coefficient filter from among the coefficient filters held in the holding unit404, reads of the selected coefficient filter from the holding unit404, and stores the selected coefficient filter in a storage unit305.

In step S1505, the control unit311initializes the processing unit406by setting an initial value for the result of convolution processing and initializing to 1 a variable m (the variable m used from the present embodiment onward is different to the variable m used prior to Formula 4) representing a processing order for the coefficient. The initial value of the result of the convolution processing is related to the input feature image, and in the case of processing the first input feature image, the initial value of the result of the convolution processing is set to 0 and in the case of processing a feature image after the first one, the initial value of the result of the convolution processing corresponds to the processing result of the previous feature image.

In step S1506, a coefficient filter kernel is scanned and convolution processing (dilated convolution processing) is performed via a multiply-accumulate operation using the selected pixel set and the selected coefficient filter. In the present embodiment, the convolution processing result is obtained by calculating Formula 6 and Formula 7 below.

Herein, i is an index representing the position in the horizontal direction, and j is an index representing the position in the vertical direction. Dx(m) is a function that returns the horizontal position corresponding to the variable m in the pixel set, and Dy(m) is a function that returns the vertical position corresponding the variable m in the pixel set. Cx(m) is a function that returns the horizontal position corresponding to the variable m in the selected coefficient filter, and Cy(m) is a function that returns the vertical position corresponding the variable m in the selected coefficient filter. The processing in step S1506will be described in detail below.

In step S1512, a processing unit309stores, in a memory310, four convolution processing results (O1,1, O1,2, O2,1, O2,2) obtained via a multiply-accumulate operation performed by the processing unit309in step S1506. The processing unit407executes activation processing and pooling processing using the convolution processing results stored in the memory310.

Next, the processing in step S1506described above will be described in detail. In step S1506, the processing of steps S1508to S1510is repeated a number of times corresponding to the number of coefficients in the selected coefficient filter.

In step S1508, a transfer control unit307transfers the function of Formula 8 and the function of Formula 10 set by the setting unit301to the storage unit306. Also, the transfer control unit307obtains a pixel DDx(m),Dy(m)at a position (Dx(m),Dy(m)) in the selected pixel set using the functions transferred to the storage unit306. A transfer control unit304transfers the function of Formula 9 and the function of Formula 11 set by the setting unit301to the storage unit305. Also, the transfer control unit304obtains a coefficient CCx(m),Cy(m)at a position (Cx(m),Cy(m)) in the selected coefficient filter using the functions transferred to the storage unit305.

An example configuration of the storage unit306will now be described usingFIG. 6. The storage unit306includes a plurality of pixel storage units. There are two types (pixel storage unit A and pixel storage unit B) of the pixel storage units, with each type being connected to the surrounding pixel storage units and disposed in two-dimensional arrangement. The pixel storage unit A is connected to the processing unit309, and the pixel storage unit B is not connected to the processing unit309. The pixel storage unit A and the pixel storage unit B each include a pixel storage memory for storing pixels and a multiplexer.

InFIG. 6, storage unit306includes 36 pixel storage units and can support processing in which the dilated convolution processing dilation rate R is 1 or 2. The configuration of the pixel storage units B other than the pixel storage units A in a block601is illustrated within a dashed line603. A single pixel storage unit B can store a single pixel. The pixel storage unit B selects the pixels held in the eight pixel storage units surrounding itself (one above, one below, one to the left, one to the right, two above, two below, two to the left, and two to the right) on the basis of the function of Formula 8 and the function of Formula 10 held in the setting unit301and stores these in its own pixel storage unit. Also, a single pixel storage unit B can supply a pixel to the eight surrounding pixel storage units in parallel. There are four parallel calculation devices, and, because the four pixels to be processed in parallel are different, there are four pixel storage units A (pixel storage units1-1,1-2,2-1,2-2) within the block601. The configuration of the pixel storage unit A is illustrated within a dashed line602. The pixel storage unit A and the pixel storage unit B have the same configuration, and a single pixel storage unit A can store a single pixel. The pixel storage unit A selects the pixels held in the eight pixel storage units surrounding itself (one above, one below, one to the left, one to the right, two above, two below, two to the left, and two to the right) on the basis of the function of Formula 8 and the function of Formula 10 held in the setting unit301and stores these in its own pixel storage memory. Also, a single pixel storage unit can supply a pixel to the eight surrounding pixel storage units in parallel and the output destination of the storage unit306. In a case where the dilated convolution processing dilation rate R is 1, “surrounding itself” means one above, one down, one to the left, and one to the right are selected, and in a case where the dilated convolution processing dilation rate R is 2, “surrounding itself” means two above, two below, two to the left, and two to the right are selected.

An example configuration of the storage unit305will now be described usingFIG. 7. The storage unit305includes a plurality of coefficient storage units. There are two types (coefficient storage unit A and coefficient storage unit B) of the coefficient storage units, with each type being connected to the surrounding coefficient storage units and disposed in two-dimensional arrangement. The coefficient storage unit A is connected to the processing unit309, and the coefficient storage unit B is not connected to the processing unit309. The coefficient storage unit A and the coefficient storage unit B each include a coefficient storage memory for storing coefficients and a multiplexer.

InFIG. 7, the storage unit305includes nine coefficient storage units, with the configuration of the coefficient storage units B other than the coefficient storage unit A within a dashed line701being illustrated within a dashed line703. A single coefficient storage unit B can store a single coefficient. The coefficient storage unit B selects the coefficient held in the four coefficient storage units surrounding itself (one above, one below, one to the left, one to the right) on the basis of the function of Formula 9 and the function of Formula 11 set by the setting unit301and stores these in its own coefficient storage memory. Also, a single coefficient storage unit B can supply a coefficient to the four surrounding coefficient storage units in parallel. There are four parallel calculation devices, and, because the coefficients to be processed in parallel are the same, there is a single coefficient storage unit A (coefficient storage unit1-1) within the dashed line701. The configuration of the coefficient storage unit A is illustrated within a dashed line702. The coefficient storage unit A and the coefficient storage unit B have the same configuration, and a single coefficient storage unit A can store a single coefficient. The coefficient storage unit A selects the coefficient held in the four coefficient storage units surrounding itself (one above, one below, one to the left, one to the right) on the basis of the function of Formula 9 and the function of Formula 11 held in the setting unit301and stores these in its own coefficient storage memory. Also, a single coefficient storage unit can supply a coefficient to the four surrounding coefficient storage units in parallel and the output destination of the storage unit305. Formulas 8 to 11 are as follows.

Dx(1), Dy(1), Cx(1), Cy(1) are each set with an initial value. In a case where the value of the variable m is 1, Dx(1), Dy(1), Cx(1), and Cy(1) are used for Dx(m), Dy(m), Cx(m), and Cy(m), respectively. In a case where the value of the variable m is 2, Dx(m), Dy(m), Cx(m), Cy(m) are obtained in accordance with Formulas 8 to 11. Ix( ) and Jx( ) are shift functions with respect to the horizontal direction, and Iy( ) and Jy( ) are shift functions with respect to the vertical direction. PD( ) and PC( ) will be described below.

In step S1509, the transfer control unit307transfers the four pixels (in the present embodiment, there are four parallel calculation devices and thus four pixels obtained in parallel and transferred) obtained in step S1508to the processing unit309. Also, the transfer control unit304transfers the single coefficient obtained in step S1508to the processing unit309.

An example configuration of the processing unit309will now be described using the block diagram ofFIG. 5. As illustrated inFIG. 5, the processing unit309includes (M×N) individual sets including a multiplier501, an adder502, and the storage unit503, allowing multiply-accumulate operations using (M×N) number of pixels and a single coefficient to be processed in parallel. In the present embodiment, there are four calculation devices and (M×N) is four. In a case where the variable m is 1 due to the initial value of the processing result being set in step S1505, this initial value is stored in the storage unit503.

In step S1510, the processing unit309performs multiplications and cumulative arithmetic operations of the multiplication results using the multipliers501and the adders502to perform the multiply-accumulate operations (convolution processing) of Formula 6 and Formula 7 described above. Also, the processing unit309stores the result of the convolution processing in the storage unit503. In the present embodiment, there are four parallel calculation devices. Thus, the products (Pi,j,m, i=1,2, j=1,2) of the pixels and the coefficients indicated in Formula 6 can be calculated in parallel, and the calculated products can be added to the processing results (Oi,j, i=1,2, j=1,2).

Also, in a case where, at the time the step S1510ends, the value of the variable m has not reached the number (X×Y=9) of coefficients in the coefficient filter, the control unit311advances the value of the variable m one increment and repeats the processing of steps S1508to S1510. On the other hand, in a case where, at the time the step S1510ends, the value of the variable m has reached the number (X×Y=9) of coefficients in the coefficient filter, the process proceeds to step S1512.

Processing Example of Dilated Convolution Processing Herein, a processing example of the multiply-accumulate operation (step S1506) of a single pixel set and a single coefficient filter will be described. Before the multiply-accumulate operation is started, because the processing unit406has been initialized in step S1505, the convolution processing results O1,1to O2,2correspond to the initial value. The value of the variable m is set to 1, and then the process proceeds to step S1508.

Regarding PD(m) in Formula 8 and PC(m) in Formula 9, in a case where the value of the variable m is 1, PC(m), PD(m), Ix( ), and Iy( ) are not calculated, and (Dx(1), Dy(1))=(Cx(1), Cy(l))=(1,1) is set to.

In step S1508, the transfer control unit307obtains the pixel D1,1and the coefficient C1,1. As illustrated inFIGS. 8A and 8B, in a case where the value of the variable m is 1, four of the pixels (D1,1to D2,2) in a pixel set801are obtained. A dashed line802indicates an example of a pixel stored in the storage unit306illustrated inFIG. 6, and the 36 pixel storage units store 36 pixels (D1,1to D6,6).

As illustrated inFIG. 9, in a case where the value of the variable m is 1, a single coefficient (C1,1) is obtained from a coefficient filter901. A dashed line902indicates an example of a coefficient stored in the storage unit305illustrated inFIG. 6, and the nine coefficient storage units store nine coefficients (C1,1to C3,3).

Examples of the pixel transfer pattern (m, PD(m), Ix(PD(m)), Iy(PD(m)) and direction, Dx(m), Dy(m)), the coefficient transfer pattern (m, PC(m), Jx(PC(m)), Jy(PC(m)) and direction, Cx(m), Cy(m)) and the direction will be described usingFIG. 10. As illustrated inFIG. 6, there are four pixel storage units linked to output. However, here only the upper left pixel storage unit1-1will be described. The pixel storage unit1-1corresponds to the output result (O1,1), and the start position of the pixel transfer is the upper left. As illustrated inFIG. 7, the coefficient storage unit linked to output is the coefficient storage unit1-1. The start position of the coefficient transfer is the upper left.

An example (in which the convolution processing dilation rate R is 2) of pixel transfer and coefficient transfer will be described usingFIGS. 11A and 11B. In a case where the value of the variable m is 1, the product of D1,1and C1,1is added to the initial value, and O1,1is calculated using Formula 6. O1,2, O2,1, and O2,2illustrated inFIGS. 11A and 11Bcan be obtained by a method similar to the method for obtaining O1,1, and thus the description thereof is omitted. In the present embodiment, four convolution processing results can be calculated in parallel. Then, the value of the variable m is changed to 2, and the process proceeds to step S1508.

In a case where the value of the variable m is 2 or greater, PD(2) is 0, and Ix(PD(2)) and Iy(PD(2)) is (2,0). Also, (Dx(2), Dy(2)) is obtained as (3,1) from Formula 8 and Formula 10 described above. As illustrated inFIG. 6, there are eight types of shift function values, and an input pixel can be directly selected with the configuration in the dashed line602or the dashed line603. Thus, the processing speed is fast.

PC(2) is 0, and Ix(PC(2)) and Iy(PC(2)) are (1,0). Also, (Cx(2), Cy(2)) is obtained as (2,1) from Formula 9 and Formula 11 described above. As illustrated inFIG. 7, there are four types of shift function values, and coefficient can be directly selected with the configuration in the dashed line702or the dashed line703. Thus, the processing speed is fast.

In step S1508, the transfer control unit307obtains the pixel D3,1from the selected pixel set. The transfer control unit304obtains the coefficient C2,1from the coefficient filter. As illustrated inFIGS. 8A and 8B, in a case where the value of the variable m is 2, four of the pixels (D2,1to D3,2) in a pixel set803are obtained as output pixels. The value of the shift function is different depending on the dilated convolution processing dilation rate R.

The dashed line802indicates the pixel held by the storage unit306ofFIG. 6in a case where the dilated convolution processing dilation rate R is 1. Because the shift direction of the output pixel is to the right, the pixel storage unit1-1in the storage unit306selects a pixel from the pixel storage unit2-1and holds it in the pixel storage memory ofFIG. 6. The pixel in the pixel storage unit1-1is replaced with the pixel in the pixel storage unit2-1, and the state indicated by the dashed line802is changed to the state indicated by a dashed line804.

A dashed line808indicates the pixel held by the storage unit306ofFIG. 6in a case where the dilated convolution processing dilation rate R is 2. As illustrated by805and806, because the shift direction of the output pixel is to the right, the pixel storage unit1-1in the storage unit306selects a pixel from the pixel storage unit3-1and holds it in the pixel storage memory ofFIG. 6. The pixel in the pixel storage unit1-1is replaced with the pixel in the pixel storage unit3-1, and the state indicated by a dashed line807is changed to the state indicated by the dashed line808.

In this manner, according to the present embodiment, data (data separated from the read out data in the row direction by two or more) other than the data adjacent to the read out data can be read out next. In other words, a non-adjacent pixel can be obtained from the feature image.

As illustrated inFIG. 9, in a case where the value of the variable m is 2, a single coefficient (C2,1) is obtained from a coefficient filter903as an output coefficient. The processing using the shift function is not related to the dilated convolution processing dilation rate R.

The dashed line902(corresponding to901) indicates the coefficient held by the storage unit305illustrated inFIG. 7. Because the shift direction of the output coefficient is to the right, the coefficient storage unit1-1in the storage unit305selects a coefficient from the coefficient storage unit2-1and holds it in the coefficient storage memory ofFIGS. 8A and 8B. The coefficient in the coefficient storage unit1-1is replaced with the coefficient in the coefficient storage unit2-1, and the state indicated by the dashed line902is changed to the state indicated by a dashed line904.

Examples of the coefficient transfer pattern (m, PD(m), Ix(PD(m)), Iy(PD(m)) and direction, Dx(m), Dy(m)), the coefficient transfer pattern (m, PC(m), Jx(PC(m)), Jy(PC(m)) and direction, Cx(m), Cy(m)) and the direction will be described usingFIG. 10. As illustrated inFIG. 10, the storage unit1-1corresponds to the output result (O1,1), and the pixel shift direction is to the right. Also, the coefficient shift direction is to the right.

An example (in which the convolution processing dilation rate R is 2) of pixel transfer and coefficient transfer will be described usingFIGS. 11A and 11B. In a case where the value of the variable m is 2, the product of D3,1and C2,1is added to the product of D1,1and C1,1calculated before, and O1,1is calculated using Formula 6. O1,2, O2,1, and O2,2illustrated inFIGS. 11A and 11Bcan be obtained by a method similar to the method for obtaining O1,1, and thus the description thereof is omitted.

Then, the value of the variable m is changed to 3, and the process proceeds to step S1508. As illustrated inFIGS. 10, 11, and 12, the output results O1,1to O2,2are calculated using Formulas 6 to 11 on the basis of the transfer pattern. In this manner, multiplication and cumulative sum are repeated. When the value of the variable m reaches 9, the output result (O1,1, O1,2, O2,1, O2,2) is stored in the memory310as the dilated convolution result.

In this manner, in the present embodiment, the shift amount and direction of the pixel and the coefficient are selected on the basis of the transfer pattern, the dilated convolution processing dilation rate, and the like, allowing for efficient transfer. Also, by data being received from eight adjacent units by a single pixel storage unit and coefficient storage unit, unnecessary calculations using a coefficient of zero are omitted and the pixels and coefficients required in the multiply-accumulate operation can all be output. Thus, dilated convolution processing can be efficiently performed.

Second Embodiment

In the present embodiment, the differences between the first embodiment will be described, and unless particularly mentioned, the other components are the same as in the first embodiment. In the present embodiment, processing is executed with a plurality of dilated convolution processing dilation rates R using hierarchical pixel storage units.

An example configuration of the storage unit306constituted by hierarchical pixel storage units will be described using the block diagram ofFIG. 16. In the example ofFIG. 16, there are three hierarchical levels, and there are three variables, X0, X1, X2, of the hierarchical levels. A pixel is selected from the three hierarchical level pixel storage units, and a pixel can be output via the same throughput as in the configuration ofFIG. 6described in relation to the first embodiment. The plurality of pixel storage units include a plurality of multiplexers. Here, the hierarchical levels are not related to the CNN hierarchical layers.

In a first hierarchical level1601, there are two types of shift amounts (1 or 0). In a pixel storage unit AA1604, a pixel selection unit AB1605at the same position in a second hierarchical level1602and the pixel storage units adjacent above, below, to the left, and to the right of the pixel storage unit AA1604are connected. In a case where the pixel selection unit AB1605at the same position of the second hierarchical level1602is selected, the value of a variable X0of the hierarchical level corresponds to zero, and if not, then the value of the variable X0corresponds to 1.

In the second hierarchical level1602, there are two types of shift amounts (2 or 0). In the pixel selection unit AB1605, a pixel selection unit AC1606at the same position in a third hierarchical level1603and the pixel selection units not adjacent above, below, to the left, and to the right of the pixel selection unit AB1605are connected. In a case where the pixel selection unit AC1606at the same position of the third hierarchical level1603is selected, the value of a variable X1of the hierarchical level corresponds to zero, and if not, then the value of the variable X1corresponds to 1.

In the third hierarchical level1603, there are two types of shift amounts (4 or 0). In the pixel selection unit AC1606, the pixel storage unit AA1604at the same position in the first hierarchical level1601and the pixel selection units not adjacent above, below, to the left, and to the right of the pixel selection unit AC1606are connected. In a case where the pixel storage unit AA1604at the same position in the first hierarchical level1601is selected, the value of a variable X2of the hierarchical level corresponds to zero, and if not, then the value of the variable X2corresponds to 1.

When calculating Ix( ) and Iy( ) using Formula 10, the dilated convolution processing dilation rate R is calculated using the following Formula 12.

Accordingly, the maximum value of the dilated convolution processing dilation rate R corresponds to seven, and seven types, from one to seven, of the dilated convolution processing dilation rate R can be supported. The interval in the horizontal direction and the interval in the vertical direction of the read out data can be determined on the basis of the dilated convolution processing dilation rate R.

In a case where the number of hierarchical levels is H, the maximum value of the dilated convolution processing dilation rate R corresponds to (2H−1), and 1 to (2H−1) of (2H−1) types of the dilated convolution processing dilation rate R can be supported.FIG. 21is a block diagram illustrating a configuration example of the CNN processing unit1305according to the present embodiment. In this example, the variable representing the hierarchical level is Xi(i=1, . . . , H). In this case, the dilated convolution processing dilation rate R can be determined on the basis of the following Formula 13.

In this manner, according to the present embodiment, the hierarchical pixel storage units can support (2H−1) types of the dilated convolution processing dilation rate R at H times the circuit cost. This gives that advantage of high flexibility at a low hardware cost.

Third Embodiment

The first embodiment and the second embodiment are examples applied to a CNN. However, no such limitation is intended, and an example may be applied to an application (image processing application or the like) other than the CNN.

The first embodiment and the second embodiment are examples applied to a CNN that deals with two-dimensional data. However, no such limitation is intended, and an example may be applied to a CNN that deals with one-dimensional data or three- or more dimensional data.

In the first embodiment, there are 36 pixel storage units. However, depending on the application, the number of pixel storage units in the horizontal direction and the vertical direction may be determined and various input data sizes can be accommodated.

In the first embodiment, processing with the dilated convolution processing dilation rate R of 1 or 2 can be supported. However, input of the pixel storage units illustrated inFIG. 6may be increased and processing with the dilated convolution processing dilation rate R of 3 or greater may be supported.

In the first embodiment and the second embodiment, pixels (data) are transferred in the horizontal direction and the vertical direction. However, data transfer is not limited to being two-dimensional, and simplified one-dimensional data transfer or three- or more dimensional data transfer may be implemented.

In the second embodiment, the results selected from the above, below, left, and right pixel storage units in the first hierarchical level1601and the second hierarchical level1602are transferred to the next hierarchical level. However, a hierarchical pixel storage unit may be provided for each of the four directions (above, below, to the left, and to the right).

In the first embodiment, the convolution processing is executed using the pre-dilation coefficient filter size of 3×3. However, the size of the pre-dilation coefficient filter is not limited to 3×3, and a discretionary size such as 2×2 may be used.

On the right side ofFIG. 17, an example of coefficient filter dilation is illustrated in which the pre-dilation coefficient filter size is 2×2. As illustrated in frame1704, a post-dilation coefficient filter (coefficient filter C′x, y on the right side) is obtained by dilation of the pre-dilation coefficient filter (coefficient filter Cx, y on the left side) according to the dilated convolution processing dilation rate R of 1. In a case where R is 1, the size does not change from the pre-dilation coefficient filter to the post-dilation coefficient filter.

As illustrated in frame1705, a post-dilation coefficient filter (coefficient filter C′x, y on the right side) is obtained by dilation of the pre-dilation coefficient filter (coefficient filter Cx, y on the left side) according to the dilated convolution processing dilation rate R of 2. The post-dilation coefficient filter is a 3×3 coefficient filter obtained by inserting one inactive coefficients between the coefficients of the pre-dilation coefficient filter.

As illustrated in frame1706, a post-dilation coefficient filter (coefficient filter C′x, y on the right side) is obtained by dilation of the pre-dilation coefficient filter (coefficient filter Cx, y on the left side) according to the dilated convolution processing dilation rate R of 4. The post-dilation coefficient filter is a 5×5 coefficient filter obtained by inserting three inactive coefficients between the coefficients of the pre-dilation coefficient filter.

Also, in a case where the pre-dilation coefficient filter size is 2×2, the index of the coordinates may be corrected, and Oi,j(n) may be determined using the following Formula 14. In the Formula, the dilated convolution processing dilation rate R is a multiple of 2.

In a case where the pre-dilation coefficient filter size is 2×2, the size can be made similar to the 3×3 coefficient filter using two cascade-connected coefficient filters.FIG. 18is a diagram illustrating an example of cascade-connected dilated convolution processing coefficient filters.

In a case where the dilated convolution processing dilation rate R is 2, the size of the coefficient filters expands from 2×2 to 3×3. Because the position where the coefficient is zero is the same, the cascade-connected coefficient filters can be made similar to the coefficient filter with a size expanded from 3×3 to 5×5.

In a case where the dilated convolution processing dilation rate R is 4, the size of the coefficient filters expands from 2×2 to 5×5. Because the position where the coefficient is zero is the same, the cascade-connected coefficient filters can be made similar to the coefficient filter with a size expanded from 5×5 to 9×9.

FIG. 19is a diagram illustrating an example of cascade-connected dilated convolution processing applied to a convolutional neural network. There are two convolution processing layers and three sets (C1, C1, and C2 or C1, C2, and C2) of the feature images. When used in combination with depthwise convolution processing, the number of coefficients in the coefficient filter of the cascade-connected dilated convolution processing is 8×C1, 4×C1×(C2+1), or 4×(C1+1) C2. In a case where the depthwise convolution processing is not used in combination, the number of coefficients in the coefficient filter of the cascade-connected dilated convolution processing is 4×(C1+C2)×C2 or 4×(C1+C2)×C1. The number of coefficients in the coefficient filters with size expanded from 3×3 to 5×5 is 9×C1×C2. Because the number C1 and C2 of the feature images can be adjusted and made less than a target with a similar number of coefficients of the cascade-connected dilated convolution processing, an effect of coefficient reduction is achieved.

FIG. 20is a diagram illustrating an example of a convolutional neural network for image segmentation processing. The convolutional neural network include cascade-connected dilated convolution processing. Dilated convolution processing with a pre-dilation filter size of 2×2 is executed using the configuration of the first embodiment or the second embodiment, and the convolutional neural network illustrated inFIG. 20is implemented.

Note that in the first embodiment and the second embodiment, the feature image is processed per pixel set. However, the processing described above may be executed treating a feature image as one pixel set.

The numerical values, processing timing, processing order, processing subject, configuration, transmission destination, transmission source, storage location of data (information), and the like used in the description of the embodiments are examples for facilitating a detailed description, and no such limitation to the examples is intended.

Also, a part or all of the embodiments described above may be combined as appropriate. Furthermore, a part or all of the embodiments described above may be selectively used.

OTHER EMBODIMENTS

While the present disclosure has been described with reference to exemplary embodiments, the scope of the following claims are to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2021-046240, filed Mar. 19, 2021, which is hereby incorporated by reference herein in its entirety.