Patent Description:
<CIT> discloses an image processing apparatus that performs processing of personal recognition of a face from an image including the face. In order to detect a face area included in the image, an edge forming a local feature of the face, an edge part forming a local feature of the face obtained by integrating edges, a local feature of the face obtained by integrating edge parts, and a face area obtained by integrating local features are detected. Setting means sets a first local area in the face area such that, of features detected by the detection means, edges forming local features in terms of shapes of parts forming the face are included in the first local area. Furthermore, the setting means sets a second local area in the face area such that the second local area includes an edge part forming a local feature in terms of a positional relationship between parts forming the face. Based on feature groups in the first and second local areas set in the face area by the setting means, identifying means identifies whose face of different individuals the face area belongs to.

In <CIT>, the detection means detects each feature in the face area included in the image using a hierarchical neural network for detecting the features of the face included in the image. The technique disclosed in Patent Document <NUM> requires a large-capacity memory to store the features detected by the detection means. In particular, to realize an image processing apparatus using a large-scale hierarchical neural network based on recent techniques, an increase in memory cost occurs which may result in a problem.

The prior art also comprises the following document from <NPL>.

An object of the present invention is to provide a discrimination calculation apparatus, a discrimination calculation method, and a program, capable of performing a discrimination calculation using a small-capacity storage unit. Solution to Problem.

According to an aspect of the present invention, there is provided a discrimination calculation apparatus as specified in claims <NUM>-<NUM>. According to another aspect of the present invention, there is provided a discrimination calculation method as specified in claim <NUM>. According to a further aspect of the present invention, there is provided a program as specified in claim <NUM>.

According to the present invention, it is possible to perform a discrimination calculation using a small-capacity storage unit.

<FIG> is a diagram showing an example of a configuration of an image processing system <NUM> according to a first embodiment of the present invention. The image processing system <NUM> includes an image input unit <NUM>, a discrimination calculation apparatus <NUM>, an image bus <NUM>, a bridge <NUM>, a preprocessing unit <NUM>, a DMAC <NUM>, a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, and a CPU bus <NUM>. The discrimination calculation apparatus <NUM> includes a RAM <NUM> and a pattern recognition process unit <NUM>. The image processing system <NUM> has a function of detecting an area of a specific object from image data.

The image input unit <NUM> includes an optical system, a photoelectric conversion device, a driver circuit that controls the photoelectric conversion device, an analog-digital converter, a signal processing circuit that performs an image correction, a frame buffer, and/or the like. The photoelectric conversion device is a CCD (Charge-Coupled Devices) or CMOS (Complimentary Metal Oxide Semiconductor) sensor that generates an image signal by photoelectric conversion. The image input unit <NUM> outputs digital image data to the image bus <NUM>. The RAM (Random Access Memory) <NUM> is a calculation work buffer used by the pattern recognition process unit <NUM>. The pattern recognition process unit <NUM> recognizes a specific pattern from the image using the RAM <NUM>.

The DMAC (Direct Memory Access Controller) <NUM> transfers data between each processing unit on the image bus <NUM> and the CPU bus <NUM>. The bridge <NUM> provides a bridge function between the image bus <NUM> and the CPU bus <NUM>. The preprocessing unit <NUM> performs various kinds of preprocessing to make it possible for the pattern recognition process unit <NUM> to effectively perform the pattern recognition process. More specifically, the preprocessing unit <NUM> performs, by hardware, an image data conversion process such as a color conversion process, a contrast correction process, and/or the like. The ROM (Read Only Memory) <NUM> stores a program that defines an operation of the CPU <NUM> and parameter data. The CPU <NUM> controls an operation of the image processing system <NUM> by executing the program stored in the ROM <NUM>. The RAM <NUM> is a memory necessary for the operation of the CPU <NUM>. The CPU <NUM> can also access the RAM <NUM> on the image bus <NUM> via the bridge <NUM>.

The image input unit <NUM> outputs the image data to the preprocessing unit <NUM>. The preprocessing unit <NUM> performs preprocessing on the image data, and outputs the preprocessed image data to the pattern recognition process unit <NUM>. The pattern recognition process unit <NUM> performs a predetermined discrimination process on the input preprocessed image data in units of pixels, determines an area of a predetermined object in the input image, and stores a determination result as a discrimination map in an image format in the RAM <NUM>.

<FIG> is a diagram schematically illustrating an example of a determination result for an image to be discriminated by the pattern recognition process unit <NUM>. A person area <NUM> (indicated by a solid area) in the image <NUM> is an area obtained as a result of the determination in pixel units. The pattern recognition process unit <NUM> stores the determination result shown in <FIG> in the RAM <NUM>. The DMAC <NUM> transfers the determination result stored in the RAM <NUM> to the RAM <NUM>. The CPU <NUM> executes various application processes using the determination result stored in the RAM <NUM>. For example, the CPU <NUM> executes processing such as optimum image processing depending on an area of an object.

<FIG> is a diagram illustrating an example of a configuration of the discrimination calculation apparatus <NUM> shown in <FIG>. The discrimination calculation apparatus <NUM> includes a CNN calculation process unit <NUM>, a hierarchical data storage unit <NUM>, a partial discrimination calculation process unit <NUM>, a discrimination result storage unit <NUM>, and a control unit <NUM>. CNN (Convolutional Neural Network) stands for a convolutional neural network.

The CNN calculation process unit <NUM> performs a plurality of types of convolution operations on previous-layer data stored in the hierarchical data storage unit <NUM>, and stores the calculation results in the hierarchical data storage unit <NUM>. The partial discrimination calculation process unit <NUM> partially performs the calculation of the discrimination process using the calculation result of an intermediate hierarchical layer obtained as the result of the CNN calculation process unit <NUM>, and the partial discrimination calculation process unit <NUM> stores the partial calculation result of the discrimination process in the discrimination result storage unit <NUM>. The discrimination process performed here is a linear discrimination process generated by a learning method such as a support vector machine or logistic regression. The hierarchical data storage unit <NUM> and the discrimination result storage unit <NUM> correspond to the RAM <NUM> in <FIG>. The control unit <NUM> is a processing unit configured to control a processing sequence performed by the CNN calculation process unit <NUM> and a processing sequence performed by the partial discrimination calculation process unit <NUM>. The control unit <NUM> is configured by a small-scale CPU or hardware sequencer.

<FIG> is a diagram illustrating an example of an algorithm of a process performed by the CNN calculation process unit <NUM> and that by the partial discrimination calculation process unit <NUM> shown in <FIG>. In the following description, a discrimination process using a CNN and feature plane data will be explained for a case where the CNN used is small in scale and has only three layers <NUM> to <NUM>. Note that many practical CNNs are configured using a larger number of pieces of feature plane data and a larger number of hierarchical layers.

In a case where a CNN calculation process is performed on image data, an input layer <NUM> is given by raster-scanned image data with a predetermined size. Note that the image data in the input layer <NUM> is discrimination target data. Pieces of feature plane data 403a to 403c are feature plane data in a first hierarchical layer <NUM>. The feature plane data is obtained as a processing result of a predetermined feature extraction calculation (a convolution calculation and nonlinear processing). Since the feature plane data is a processing result for raster-scanned image data, the processing result is also given by a plane. The pattern recognition process unit <NUM> uses the pieces of feature plane data 403a to 403c as data in an intermediate hierarchical layer for discrimination. The pieces of feature plane data 403a to 403c are calculated by performing a convolution operation and nonlinear processing on the input layer <NUM>. The processing unit 4021a of the CNN calculation process unit <NUM> calculates the feature plane data 403a by performing a convolution operation using a two-dimensional convolution kernel on the input layer <NUM>, and further performing a non-linear transformation on a calculation result thereof. The processing unit 4021b of the CNN calculation process unit <NUM> calculates the feature plane data 403b by performing a convolution operation using a two-dimensional convolution kernel on the input layer <NUM> and further performing a nonlinear transformation on the calculation result thereof. The processing unit 4021c of the CNN calculation process unit <NUM> calculates the feature plane data 403c by performing a convolution operation using a two-dimensional convolution kernel on the input layer <NUM> and further performing a nonlinear transformation on the calculation result thereof. The processing units 4021a to 4021c respectively calculate different pieces of feature plane data 403a to 403c.

For example, each of the processing units 4021a to 4021c performs a convolution operation with a kernel (a coefficient matrix) of a size given by columnSize × rowSize by performing a product-sum operation such as that shown in formula (<NUM>). <NUM>] <MAT>.

In this formula (<NUM>), input(x, y) represents a reference pixel value at two-dimensional coordinates (x, y) in the input layer <NUM>, output(x, y) represents a calculation result in two-dimensional coordinates(x, y), weight(column, row) represents a weighting coefficient at coordinates (x + column, y + row), and columnSize and rowSize are horizontal and vertical sizes of the two-dimensional convolution kernel.

The CNN calculation process unit <NUM> repeats the product-sum operation on the input layer <NUM> while scanning a plurality of convolution kernels in units of pixels, and performs the nonlinear transformation on the final product-sum result thereby obtaining the pieces of feature plane data 403a to 403c. Note that when the feature plane data 403a is calculated, the number of connections with the previous layer is <NUM>, and thus there is one convolution kernel.

<FIG> is a diagram illustrating main calculation units of the CNN calculation process unit <NUM> shown in <FIG>. Refering to <FIG>, a basic operation of the main calculation units will be described below for an example case where the feature plane data 405a shown in <FIG> is calculated. The feature plane data 405a is calculated by performing a convolution operation on the three pieces of feature plane data 403a to 403c in the previous hierarchical layer <NUM>. The CNN calculation process unit <NUM> includes a convolution operation process unit <NUM>, a cumulative adder <NUM>, and a nonlinear conversion process unit <NUM>. The convolution operation process unit <NUM> performs a convolution operation 4041a using a kernel on the feature plane data 403a, and stores the operation result in the cumulative adder <NUM>. Similarly, the convolution operation process unit <NUM> performs convolution operations 4041a and 4041c using kernels on the pieces of feature plane data 403b and 403c, respectively, and stores the operation results in the cumulative adder <NUM>. Note that the convolution operation process unit <NUM> may process the three kernels sequentially or in parallel. The cumulative adder <NUM> cumulatively adds the operation results and outputs the cumulative addition result to the nonlinear conversion process unit <NUM>. The nonlinear conversion process unit <NUM> performs a nonlinear conversion process using a ReLU (Rectified Linear Unit) function or a logistic function on the cumulative addition result, and outputs the feature plane data 405a. The CNN calculation process unit <NUM> calculates the feature plane data 405a by performing the above-described processing while scanning the entire feature plane data 403a to 403c pixel by pixel.

Similarly, as shown in <FIG>, the CNN calculation process unit <NUM> performs three convolution operations 4041b, 4042b, and 4043b respectively on the plurality of pieces of feature plane data 403a to 403c in the previous hierarchical layer <NUM> thereby calculating the feature plane data 405b. The CNN calculation process unit <NUM> further performs two convolution operations <NUM> and <NUM> on the pieces of feature plane data 405a and 405b in the second hierarchical layer <NUM> thereby calculating the feature plane data <NUM> in the third hierarchical layer <NUM>. As described above, the CNN calculation process unit <NUM> functions as feature calculation means configured to sequentially calculate feature plane data layer by layer. Note that the coefficients of each convolution operation are determined in advance by learning using a general method such as back-propagation learning.

The partial discrimination calculation process unit <NUM> concatenates pieces of data located at pixel positions of each piece of feature plane data 403a to 403c, 405a to 405b and <NUM> such that a feature vector is given by a vector formed by respective pieces of concatenated data of each piece of feature plane data. On these feature vectors, the discrimination calculation partial process unit <NUM> performs a linear discrimination calculation process, and outputs a discrimination map <NUM>. Since the feature vector is data obtained by concatenating the feature plane data corresponding to the pixel positions, in the case of the network shown in <FIG>, a <NUM>-dimensional feature vector is obtained for corresponding six pieces of feature plane data 403a to 403c, 405a to 405b, and <NUM>. The partial discrimination calculation process unit <NUM> performs a linear discrimination calculation for each pixel position in accordance with formula (<NUM>). <NUM>] <MAT>.

In this formula (<NUM>), featuremap(n, f, x, y) is a value of f-th feature plane data in an n-th hierarchical layer at coordinates (x, y), result(x, y) is a discrimination result for data at coordinates (x, y), weight(n, f) is a coefficient for the f-th feature plane data in the n-th hierarchical layer, LayerNumber is the number of hierarchical layers, and FeaturemapNumber is the number of pieces of feature plane data included in the n-th hierarchical layer.

Furthermore, weight (n, f) is a coefficient learned in advance by logistic regression, a support vector machine, or the like. The partial discrimination calculation process unit <NUM> outputs the result of processing for each pixel position of the feature plane data by the calculation of formula (<NUM>) as the discrimination map <NUM>. Since the partial discrimination calculation process unit <NUM> performs a discrimination process for each pixel position, the discrimination result is also generated as a discrimination map <NUM> in an image format. In the discrimination map <NUM>, a value corresponding to each pixel position indicates a probability for a target object area.

<FIG> is a flow chart illustrating a discrimination calculation method by the discrimination calculation apparatus <NUM> shown in <FIG>. In step S601, the control unit <NUM> sets parameters for specifying an operation of the CNN calculation process unit <NUM> and an operation of the partial discrimination calculation process unit <NUM>. Next, in step S602, the CNN calculation process unit <NUM> performs a CNN calculation for one hierarchical layer and calculates feature plane data for the one hierarchical layer. For example, the CNN calculation process unit <NUM> calculates feature plane data in a hierarchical layer <NUM> in <FIG> and calculates pieces of feature plane data 405a and 405b. Next, in step S603, the CNN calculation process unit <NUM> stores the feature plane data calculated in step S602, as intermediate hierarchical data, in the hierarchical data storage unit <NUM>. The CNN calculation process unit <NUM> sequentially performs CNN calculations in step S602, and stores the feature plane data in the hierarchical data storage unit <NUM> in step S603.

Next, in step S604, the partial discrimination calculation process unit <NUM> reads a partial calculation result of the discrimination process based on the result of the feature extraction process for the previous hierarchical layer stored in the discrimination result storage unit <NUM>. Next, in step S605, based on the calculation result read out in step S604 and the feature plane data in terms of the hierarchical layer output via the processing by the CNN calculation process unit <NUM>, the partial discrimination calculation process unit <NUM> performs a partial discrimination process according to formula (<NUM>). <NUM>] <MAT>.

In this formula (<NUM>), featuremap(n, f, x, y) is a value of f-th feature plane data in an n-th hierarchical layer at coordinates (x, y), weight(n, f) is a coefficient for the f-th feature plane data in the n-th hierarchical layer, featuremap_number is the number of pieces of feature plane data in the n-th hierarchical layer, part_result(n-<NUM>, x, y) is a result of a partial discrimination calculation performed for feature plane data up to the (n-<NUM>)th hierarchical layer, and part_result(n, x, y) is a result of a partial discrimination calculation performed for feature plane data up to the n-th hierarchical layer.

The partial discrimination calculation process unit <NUM> calculates a sum of part_result(n-<NUM>, x, y) given as the result of the discrimination calculation process performed for up to (n-<NUM>)th hierarchical layer in formula (<NUM>) and an inner product of feature plane data featuremap(n, f, x, y) and weighting factors weight(n, f). Thus, the partial discrimination calculation process unit <NUM> obtains part_result(n, x, y) as a result of a partial discrimination calculation for the n-th hierarchical layer.

An overall result of the linear discrimination calculation result(x, y) is obtained by calculating a sum of inner products of featuremap(n, f, x, y) of the entire feature plane data and corresponding weighting factors weight(n, f) as shown in formula (<NUM>). In the present embodiment, in step S604, the partial discrimination calculation process unit <NUM> reads out part_result(n-<NUM>, x, y) indicating the result of the partial discrimination calculation performed for up to the previous hierarchical layer from the discrimination result storage unit <NUM>. Next, in step S605, the partial discrimination calculation process unit <NUM> calculates the second term on the right side of formula (<NUM>), and calculates a sum of the second term on the right side of formula (<NUM>) and the calculation result part_result (n-<NUM>, x, y) read in step S604. Thus, the partial discrimination calculation process unit <NUM> obtains a partial discrimination calculation result part_result (n, x, y) calculated for up to the current hierarchical layer. That is, the partial discrimination calculation process unit <NUM> executes a calculation of a part of hierarchical layers in the discrimination process calculation. Therefore, in step S603, the CNN calculation process unit <NUM> may store only the feature plane data of hierarchical layers necessary for a next CNN calculation in the hierarchical data storage unit <NUM>.

<FIG> is a diagram illustrating an example of a configuration of the partial discrimination calculation process unit <NUM> shown in <FIG>. The partial discrimination calculation process unit <NUM> includes a multiplier <NUM>, a cumulative adder <NUM>, and an adder <NUM>. The multiplier <NUM> performs a multiplication of an# f-th feature plane data featuremap (n, f, x, y) in the n-th hierarchical layer by the weight coefficients weight(n, f). The cumulative adder <NUM> cumulatively adds the multiplication results output by the multiplier <NUM> for respective pieces of feature plane data, and outputs the data of the second term on the right side of formula (<NUM>).

The multiplier <NUM> may read out the feature plane data in the n-th hierarchical layer being processed such that feature plane data is read out each time the process of the CNN calculation in units of pixels by the CNN calculation process unit <NUM> is completed, or such that feature plane data is buffered in the hierarchical data storage unit <NUM> and feature plane data is read out at a time in particular units. For example, <FIG> illustrates an example of a modification of the flow chart shown in <FIG>. In this modification, the position of step S603 is different from that shown in <FIG>. In <FIG>, step S603 is processed in parallel to steps S604 to S606. In a case where the hierarchical data storage unit <NUM> and the discrimination result storage unit <NUM> are configured using physically different memories or the like, it is expected that the parallel processing described above result in an increase in the processing speed.

The adder <NUM> shown in <FIG> calculates the sum of partial determination operation result part_result (n-<NUM>, x, y) for up to the previous hierarchical layer stored in the discrimination result storage unit <NUM> and the data of the second term on the right side of formula (<NUM>) output by cumulative adder <NUM>. The adder <NUM> then outputs a partial discrimination calculation result part_result (n, x, y) for up to the current hierarchical layer on the left side of formula (<NUM>).

As described above, the multiplier <NUM> and the cumulative adder <NUM> calculate the sum of products of the feature plane data featuremap (n, f, x, y) in the present hierarchical layer and the weight coefficients weight(n, f). The adder <NUM> adds the result of the sum of products calculated for the current hierarchical layer output by the cumulative adder <NUM> and the result, part_result (n-<NUM>, x, y), of the sum of products calculated for up to the previous hierarchical layer stored in the discrimination result storage unit <NUM> and stores a resultant sum in the discrimination result storage unit <NUM>.

Next, in step S606 in <FIG>, the partial discrimination calculation process unit <NUM> stores, in the discrimination result storage unit <NUM>, the partial discrimination calculation result part_result (n, x, y) calculated for up to the current hierarchical layer obtained in step S605. This discrimination calculation result part_result (n, x, y) is used, in the processing for a next hierarchical layer, as a partial discrimination calculation result part_result (n-<NUM>, x, y) up to the previous hierarchical layer. Next, in step S607, the control unit <NUM> determines whether or not processing has been completed for all hierarchical layers. In a case where the control unit <NUM> determines that the processing has not been completed for all hierarchical layers, the control unit <NUM> returns the processing to step S602 and repeats the processing for a next hierarchical layer. In a case where the control unit <NUM> determines that the processing has been completed for all hierarchical layers, the control unit <NUM> ends the processing. Thus, by performing the processing described above, the discrimination calculation apparatus <NUM> obtains a discrimination map <NUM> as the result of the discrimination calculation process for all hierarchical layers.

As described above, the CNN calculation process unit <NUM> functions as a feature calculation unit configured to sequentially calculate feature plane data on discrimination target data given in the input layer <NUM> layer by layer. More specifically, the CNN calculation process unit <NUM> calculates pieces of feature plane data 403a to 403c in a first hierarchical layer <NUM> based on discrimination target data given in the input layer <NUM>, and stores the pieces of feature plane data 403a to 403c in the first hierarchical layer into the hierarchical data storage unit (the feature storage unit) <NUM>. Next, the CNN calculation process unit <NUM> calculates pieces of feature plane data 405a and 405b in a second hierarchical layer <NUM> based on the pieces of feature plane data 403a to 403c in the first hierarchical layer <NUM> stored in the hierarchy data storage unit <NUM>, and stores the calculated pieces of feature plane data 405a and 405b into the hierarchical data storage unit <NUM>. Next, the CNN calculation process unit <NUM> calculates feature plane data <NUM> in a third hierarchical layer <NUM> based on the pieces of feature plane data 405a and 405b in the second hierarchical layer <NUM> stored in the hierarchy data storage unit <NUM>, and stores the calculated feature plane data <NUM> into the hierarchical data storage unit <NUM>.

The partial discrimination calculation process unit <NUM> sequentially performs partial discrimination calculations on the discrimination target data given in the input layer <NUM> using the feature plane data sequentially calculated by the CNN calculation process unit <NUM>, and stores the result of the partial discrimination calculations in the discrimination result storage unit (the discrimination result storage unit) <NUM>. The partial discrimination calculation process unit <NUM> performs a next partial discrimination calculation using the feature plane data sequentially calculated by the CNN calculation process unit <NUM> and the result of the partial discrimination calculation stored in the discrimination result holding unit <NUM>, and stores a result of the next partial discrimination calculation in the discrimination result storage unit <NUM>.

Next, features of the discrimination calculation apparatus <NUM> according to the present embodiment shown in <FIG> and <FIG> are described in comparison with a discrimination calculation apparatus <NUM> according to a basic technology shown in <FIG> and <FIG>. <FIG> is a diagram illustrating an example of a configuration of the discrimination calculation apparatus <NUM> according to the basic technique. The discrimination calculation apparatus <NUM> includes a CNN calculation process unit <NUM>, a feature plane data storage unit <NUM>, a discrimination calculation process unit <NUM>, a discrimination result storage unit <NUM>, and a control unit <NUM>. The CNN calculation process unit <NUM> performs a convolution operation on the feature plane data stored in the feature plane data storage unit <NUM> and stores a result of the calculation in the feature plane data storage unit <NUM>. The CNN calculation process unit <NUM> stores feature plane data of all hierarchical layers in the feature plane data storage unit <NUM>. The discrimination calculation process unit <NUM> refers to the feature plane data stored in the feature plane data storage unit <NUM> and executes a discrimination calculation process. The discrimination calculation process unit <NUM> stores a result thereof in the discrimination result storage unit <NUM>. The control unit <NUM> is a processing unit configured to control a processing sequence performed by the CNN calculation process unit <NUM> and a processing sequence performed by the discrimination calculation process unit <NUM>. The control unit <NUM> is configured by a small-scale CPU or hardware sequencer.

<FIG> is a flow chart illustrating a discrimination calculation method by the discrimination calculation apparatus <NUM> shown in <FIG>. In step S901, the control unit <NUM> sets parameters for specifying an operation of the CNN calculation process unit <NUM> and an operation of the discrimination calculation process unit <NUM>. Next, in step S902, the CNN calculation process unit <NUM> performs a CNN calculation for one hierarchical layer thereby obtaining feature plane data for the one hierarchical layer. Next, in step S903, the CNN calculation process unit <NUM> stores the feature plane data calculated in step S902 in the feature plane data storage unit <NUM>. Next, in step S904, in a case where the control unit <NUM> determines that the processing has not been completed for all hierarchical layers, the control unit <NUM> returns the processing to step S902 and repeats the processing for a next hierarchical layer. In this case, in step S903, the CNN calculation process unit <NUM> stores pieces of feature plane data of respective hierarchical layers in different areas of the feature plane data storage unit <NUM>. In a case where the control unit <NUM> determines that the processing has been completed for all hierarchical layers, the control unit <NUM> advances the processing to step S905. In step S905, the discrimination calculation process unit <NUM> refers to the feature plane data of each hierarchical layer stored in the feature plane data storage unit <NUM>, and performs a discrimination calculation process according to formula (<NUM>). Next, in step S906, the discrimination calculation process unit <NUM> stores a discrimination result in the discrimination result storage unit <NUM>.

In the discrimination calculation apparatus <NUM> according to the basic technology, the CNN calculation process unit <NUM> stores the feature plane data of all hierarchical layers in the feature plane data storage unit <NUM>. For example, when a network shown in <FIG> is processed, the feature plane data storage unit <NUM> needs to store feature plane data of <NUM> × feature plane data size (it is assumed that the feature plane data size is equal for all hierarchical layers).

On the other hand, in the discrimination calculation apparatus <NUM> according to the present embodiment, the hierarchical data storage unit <NUM> only needs to be able to hold the feature plane data for two layers necessary for processing the next CNN calculation of the CNN calculation process unit <NUM>. In the network in <FIG>, in a case where feature plane data of a hierarchical layer <NUM> is calculated, the hierarchical data storage unit <NUM> stores a maximum amount of feature plane data. In this case, the hierarchical data storage unit <NUM> holds feature plane data of <NUM> × feature plane data size. That is, in the case of the present embodiment, the discrimination calculation apparatus <NUM> can reduce the capacity of the hierarchical data storage unit <NUM> for storing the feature plane data.

In the example shown in <FIG>, the CNN has a small size for simplicity of explanation. However, in practical CNNs, CNNs include a large number of hierarchical layers. Therefore, the effect of reducing the capacity of the hierarchical data storage unit <NUM> is extremely large. For example, an explanation is given below for a case in which each hierarchical layer has a <NUM> feature planes (it is assumed that the number of feature planes is equal for all hierarchical layers), and the number of hierarchical layers is <NUM>. In this case, the discrimination calculation apparatus <NUM> according to the basic technique needs the feature plane data storage unit <NUM> with a capacity for <NUM> × <NUM> pieces of feature plane data. On the other hand, in the discrimination calculation apparatus <NUM> according to the present embodiment, the hierarchical data storage unit <NUM> needs a capacity for only <NUM> × <NUM> pieces of feature plane data. In particular, when the present embodiment is applied to a method for calculating CNN with a memory having a small capacity such as that disclosed in <CIT>, a further enhanced effect can be achieved.

As described above, in the discrimination calculation apparatus <NUM> according to the present embodiment, storing intermediate hierarchical data (feature plane data) of CNN in the hierarchical data storage unit <NUM> makes it possible to reduce the capacity of the hierarchical data storage unit <NUM>, which allows a reduction in cost.

<FIG> is a diagram illustrating an example of a configuration of the discrimination calculation apparatus <NUM> according to a second embodiment of the present invention. The discrimination calculation apparatus <NUM> shown in <FIG> is obtained by adding a data reduction process unit <NUM> to the discrimination calculation apparatus <NUM> shown in <FIG>. The data reduction process unit <NUM> is provided between the partial discrimination calculation process unit <NUM> and the discrimination result storage unit <NUM>. Note that the data reduction process unit <NUM> may be provided in the partial discrimination calculation process unit <NUM>. Differences of the present embodiment from the first embodiment are described below.

In the first embodiment, the partial discrimination calculation process unit <NUM> stores the result of the cumulative addition of the partial calculation result of each hierarchical layer in the discrimination result storage unit <NUM>. In this case, the partial discrimination calculation process unit <NUM> needs an increasing number of calculation digits with proceeding of the cumulative addition process. Therefore, in a case where the processing is performed using integer fixed-point calculations, it is required to increase the memory data width of the discrimination result storage unit <NUM>. The increase in the memory data width directly leads to an increase in the capacity of the discrimination result storage unit <NUM> and causes an increase in cost of the discrimination calculation apparatus <NUM>. Therefore, it is desirable to perform processing with a memory data width as small as possible. In the present embodiment, the data reduction process unit <NUM> and the partial discrimination calculation process unit <NUM> reduce the number of digits of the discrimination calculation result output by the partial discrimination calculation process unit <NUM> and store the discrimination calculation result with the reduced number of digits in the discrimination result storage unit <NUM>.

<FIG> is a flow chart showing a discrimination calculation method by the discrimination calculation apparatus <NUM> shown in <FIG>. In the flow chart shown in <FIG>, steps S1106 to S1108 are provided instead of steps S606 and S607 in the flow chart shown in <FIG>.

First, the discrimination calculation apparatus <NUM> performs processing in steps S601 to S605 as with the processing shown in <FIG>. Next, in step S1106, the data reduction process unit <NUM> and the partial discrimination calculation process unit <NUM> reduce the number of digits of the partial calculation result in the discrimination process in step S605 according to formula (<NUM>). More specifically, the data reduction process unit <NUM> and the partial discrimination calculation process unit <NUM> extract data of a particular number of digits at a particular digit position from the result of the sum of products on the second term on the right side of formula (<NUM>), and add a value of the first term on the right side of formula (<NUM>) to the extracted data. That is, the data reduction process unit <NUM> and the partial discrimination calculation process unit <NUM> reduce the number of digits of the result of the sum products of the feature plane data of the current hierarchical layer and the weighting coefficient, and add the result of the sum of products with the reduced number of digits to the result of the sum of products calculated for up to the previous hierarchical layer. The addition result is stored in the discrimination result storage unit <NUM>.

<FIG> is a diagram illustrating a specific example of the digit number reduction processing by the data reduction process unit <NUM>. A binary data bit string <NUM> is a data bit string of a partial calculation result output from by the partial discrimination calculation process unit <NUM>, and includes <NUM>-bit data Dn (n: <NUM> to <NUM>). The bit data Dn is an element of the binary data bit string <NUM> and indicates a <NUM>-bit data string. In step S1106, the data reduction process unit <NUM> extracts data <NUM> of a required number of significant digits at a predetermined data extraction digit position <NUM> in the binary data bit string <NUM>. For example, the data <NUM> may be <NUM>-bit data. The data extraction digit position <NUM> is a parameter given in advance in step S1106. The data extraction digit position <NUM> is specified by a value determined based on the decimal point position of the feature plane data in step S602, the decimal point position of the weighting coefficient used in the discrimination process, and the decimal point position of the data stored in the discrimination result storage unit <NUM>. That is, the data extraction digit position <NUM> is a parameter necessary for matching the digit positions of the first term on the right side of formula (<NUM>) and the second term on the right side of formula (<NUM>).

The data extraction digit position <NUM> may be different for each hierarchical layer or for each category of the discrimination target data. The result of step S602 and the data distribution of the corresponding weighting coefficients of the discrimination process may differ depending on the hierarchy and category. In such a case, each decimal point position is different. In step S1106, taking into account each decimal point position, the data reduction process unit <NUM> extracts the data <NUM> with the required number of digits such that the number of digits matches that of the data stored in the discrimination result storage unit <NUM>. Each decimal point position is determined in advance using test data, and correspondingly, a data extraction digit position <NUM> is determined in advance. In the determination of the number of digits of the data <NUM>, there is a trade-off between memory cost and calculation accuracy, which is also determined in advance using test data.

In the present embodiment, as can be seen from formula (<NUM>), a calculation is first performed to determine the sum of the inner product of the feature plane data of all hierarchical layer and the weighting coefficients and the partial calculation for the previous hierarchical layer stored in the discrimination result storage unit <NUM>, and then the reduction of the number of digits is performed. However, the processing is not limited to this. For example, the processing may be performed such that the reduction of the number of digits is performed for each predetermined number of pieces of feature plane data in the hierarchical layer, and data is accumulated in the discrimination result storage unit <NUM> due to mounting restrictions or the like. From the viewpoint of discrimination accuracy, it is desirable to perform the reduction of the number of digits after the inner product operation is performed for as many pieces of feature plane data as possible using the cumulative adder <NUM> (<FIG>) having a sufficiently large number of digits. However, the processing is not limited to such a method.

Next, in step S1107 in <FIG>, the partial discrimination calculation process unit <NUM> writes the partial discrimination calculation result for up to the current hierarchy in which the number of digits is reduced in step S1106 in the discrimination result storage unit <NUM>. Next, in step S1108, the control unit <NUM> determines whether or not processing has been completed for all hierarchical layers. In a case where the control unit <NUM> determines that the processing has not been completed for all hierarchical layers, the control unit <NUM> returns the processing to step S602 and repeats the processing for a next hierarchical layer. In a case where the control unit <NUM> determines that the processing has been completed for all hierarchical layers, the control unit <NUM> ends the processing. By performing the processing described above, the discrimination calculation apparatus <NUM> obtains discrimination calculation results of all hierarchical layers as the discrimination map <NUM>.

As described above, according to the present embodiment, it is possible to realize the discrimination process using the intermediate layer data of the CNN using the layer data storage unit <NUM> having a small capacity. Furthermore, the data reduction process unit <NUM> appropriately reduces the number of digits of the partial discrimination calculation and thus the capacity of the discrimination result storage unit <NUM> can be reduced. As a result, further capacity reduction of the RAM <NUM> is expected.

<FIG> is a diagram illustrating an example of a configuration of the discrimination calculation apparatus <NUM> according to a third embodiment of the present invention. The discrimination calculation apparatus <NUM> shown in <FIG> is obtained by adding a first data reduction process unit <NUM> to the discrimination calculation apparatus <NUM> shown in <FIG>. A second data reduction process unit <NUM> in <FIG> corresponds to the data reduction process unit <NUM> in <FIG>. The first data reduction process unit <NUM> is provided between the CNN calculation process unit <NUM> and the hierarchical data storage unit <NUM>. The first data reduction process unit <NUM> may be provided in the CNN calculation process unit <NUM>. Differences of the present embodiment from the second embodiment are described below. In a case where the discrimination calculation apparatus <NUM> is an embedded apparatus, in general, all processing is performed using integer fixed-point calculations. The CNN calculation process unit <NUM> generates feature plane data by performing a CNN calculation process using an integer type. The first data reduction process unit <NUM> and the CNN calculation process unit <NUM> reduce the number of digits of the feature plane data, and stores the feature plane data with the reduced number of digits into the hierarchical data storage unit <NUM>.

<FIG> is a flow chart illustrating a discrimination calculation method by the discrimination calculation apparatus <NUM> shown in <FIG>. The flow chart in <FIG> is obtained by adding step S1509 to the flow chart in <FIG>. Step S1509 is provided between steps S602 and S603. In step S1106, the second data reduction process unit <NUM> performs processing similar to that in <FIG>.

In step S1509, the first data reduction process unit <NUM> reduces the number of digits of the feature plane data calculated in step S602. Next, in step S603, the CNN calculation process unit <NUM> stores the feature plane data with the reduced number of digits in the hierarchical data storage unit <NUM>. In step S1106, the second data reduction process unit <NUM> reduces the number of digits to <NUM>-bit data, as described in the second embodiment. On the other hand, in step S1509, the first data reduction process unit <NUM> reduces the number of digits to data having a data width of <NUM> bits. The number of digits reduced by the first data reduction process unit <NUM> and the number of digits reduced by the second data reduction process unit <NUM> are different from each other. In general, in the feature extraction processing by the CNN calculation process unit <NUM>, the reduction in the number of digits results in a small reduction in discrimination accuracy. In contrast, in the discrimination process calculation by the partial discrimination calculation process unit <NUM>, the reduction in the number of calculation digits often has a large influence on the discrimination accuracy.

In the present embodiment, taking into consideration the fact that the amount of data stored in the discrimination result storage unit <NUM> is smaller than the amount of data (feature plane data of one plane in the example shown in <FIG>) necessary to be stored in the hierarchical data storage unit <NUM>, the number of digits of data stored in the discrimination result storage unit <NUM> is set to be larger than that of other processing units. This makes it possible to realize the discrimination process using CNN intermediate hierarchical data and using the RAM <NUM> with a small capacity without resulting a significant reduction in discrimination accuracy.

In the first to third embodiments, an explanation has been given, as an example, for the case in which the partial calculation of the discrimination process is executed layer by layer. However, the embodiments are not limited to this example. The partial calculation may be performed in units of data determined depending on the number of pieces of feature plane data in the hierarchical network, the size of feature plane data, and the memory size of the hierarchical data storage unit <NUM>. For example, the partial calculation of the discrimination process may be performed in units of a plurality of layers. As described in the second embodiment, when the number of digits of data stored in the discrimination result storage unit <NUM>, if the number of times of storing the partial calculation result in the determination result holding unit <NUM> is reduced as much as possible, it is possible to reduce the influence of the reduction in the number of digits on the performance.

In the first to third embodiments, an explanation has been given for an example case in which the CNN calculation process is performed on two-dimensional image data. However, the CNN calculation process may also be applied to one-dimensional data such as audio data, three-dimensional data which may change with time, and/or the like.

In the first to third embodiments, an explanation has been given, as an example, for the case in which the CNN calculation process is performed as the feature extraction process. However, the embodiments are not limited to this example. Various other hierarchical processes such as Restricted Boltzmann Machines, Recursive NeuralNetwork, or the like may be employed.

In the first to third embodiments, an explanation has been given, as an example, for the case where a CPU is used as the control unit <NUM>. However, the entire control unit <NUM> may be realized using hardware for controlling sequences.

In the first to third embodiments, an explanation has been given, as an example, for the case where the convolution operation and the inner product operation are performed by hardware. However, the embodiments are not limited to this example. All processes by the pattern recognition process unit <NUM> may be realized by executing a program by a general-purpose CPU. Even in this case, it becomes possible to greatly reduce the amount of working memory required by the CPU. The embodiments are also effective when processing is performed by a general-purpose computer system or a cloud system.

The present invention may also be realized by providing to a system or an apparatus a storage medium including a program stored therein for implementing one or more functions disclosed in the embodiments described above and by reading and executing the program on one or more processor in a computer disposed in the system or the apparatus. The present invention may also be realized by a circuit (such as an ASIC) configured to realize one or more functions described above.

The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

Claim 1:
A discrimination calculation apparatus (<NUM>) configured to discriminate an object (<NUM>) in discrimination target image data (<NUM>), comprising:
feature calculation means (<NUM>) configured to sequentially perform a product-sum operation on features (403a-403c, 405a-405b, <NUM>) of discrimination target image data (<NUM>), for determining an object (<NUM>), for respective hierarchical layers of a hierarchical neural network; the apparatus (<NUM>) further comprises:
discrimination calculation means (<NUM>) configured to sequentially perform a linear discrimination calculation at a target layer on the discrimination target image data (<NUM>) using a feature (403a-403c, 405a-405b, <NUM>) of a target layer that has been processed through the product-sum operation (S602) by the feature calculation means (<NUM>), and a result of the linear discrimination calculation from the preceding layer of the target layer;
holding means (<NUM>, <NUM>, <NUM>) configured to hold (S603, S606) the result of the linear discrimination calculation from the preceding layer of the target layer and the feature (403a-403c, 405a-405b, <NUM>) of the target layer that has been processed through the product-sum operation by the feature calculation means (<NUM>); and
control means (<NUM>) configured to control the discrimination calculation means (<NUM>) to perform (S605) the linear discrimination calculation.