Patent Description:
In recent years, technologies for recognizing a specific object from an image captured by a camera and identifying the specific object have been rapidly developed, and have been used various fields such as, for example, driving support of vehicles, diagnosis support of medical treatment.

In these image recognition technologies, a feature amount is extracted from image data by a certain method, and is compared with a feature amount of an identification object (e.g., a person) to determine whether or not the identification object is present in the aforementioned image data.

The technologies for performing such image recognition include technologies disclosed in Non-Patent Literatures <NUM> to <NUM>.

Such technologies are to detect an identification object by extracting a feature amount called a HOG feature amount from an image and being compared with the HOG feature amount which is learned from the image in which the identification object is captured in advance.

In addition, there are a CoHOG feature amount having more robustness than the HOG feature amount, a MRCoHOG feature amount having further more robustness, and the like, in the technologies of extracting the feature amount from the image.

By the way, since such image recognition technologies use high-dimensional feature amounts, when being implemented in hardware, a circuit becomes complicated and large-scale. Therefore, it has been a problem how to reduce a processing cost and realize the circuit with small resources.

If this image recognition technologies can be implemented on semiconductor chips, it is expected to be used in various situations, e.g., used in mobile devices such as vehicles and airplanes, or used in mobile terminals and wearable terminals, and used in various situations.

On the other hand, there has been rapidly developed a technology of making a neural network to learn an object, recognizing the object by input data using the learned result (neural network) thereof to identify the recognized object.

However, although the learning is performed by a back propagation or the like using a teacher signal in the neural network, there is a problem that an enormous amount of calculation is required for this learning processing, and an enormous amount of calculation is further required if the number of input data (number of dimensions of feature amount) increases.

Also, when the neural network is implemented in hardware, the increase in the number of input data causes a problem in that a circuit becomes complicated and large-scale.

The object of the present invention is to reduce a processing cost.

According to the present invention, there is provided an information processing device, as defined by the claims.

The present invention can reduce the processing cost by selecting the feature amount used for the identification.

The image recognition device <NUM> (<FIG>) includes an image processing device <NUM> configured to acquire a feature amount from an image captured by a camera <NUM>, and an identification device <NUM> configured to determine whether a predetermined identification object is present in the image using the acquired feature amount to identify this identification object.

The image processing device <NUM> acquires various feature amounts, such as a HOG feature amount, a CoHOG feature amount, a MRCoHOG feature amount, and a Haar-like feature amount, from the image as the feature amount.

The identification device <NUM> includes a binary neural network (BNN) that has learned the identification object in advance, and performs an identification processing by performing a binary calculation with the BNN on the feature amount acquired by the image processing device <NUM>. The learning of the identification object by the BNN is performed by optimizing weighting of a network by using the feature amount acquired from the image by the image processing device <NUM> as input data and a result to be recognized from the aforementioned image as a teacher signal.

In the image captured by the camera <NUM>, a high-dimensional feature amount is acquired from the aforementioned image by the image processing device <NUM> and is input into the learned BNN, and thereby a recognized result for the aforementioned image is output.

In the first embodiment, instead of inputting all the high-dimensional feature amounts output from the image processing device <NUM> with respect to this learned BNN, a portion effective for identification among the high-dimensional feature amounts is selected, thereby reducing dimensions (the number of input object data) used for the identification processing. Moreover, in the second embodiment, a low-dimensional feature amount output from the image processing device <NUM> is duplicated, thereby increasing the dimensions (the number of input object data). In the third embodiment in which the first embodiment and the second embodiment are combined with each other, a portion advantageous for identification is selected among the high-dimensional feature amounts output from the image processing device <NUM> and the selected feature amount is duplicated, thereby increasing the dimensions.

Compared with general neural networks which require multiplication using a floating point, etc., the BNN perform calculation by binary addition. Moreover, the dimension of the feature amount used for identification can be appropriately adjusted while ensuring the required identification accuracy, by selecting and duplicating the dimension of the feature amount. Accordingly, the identification device <NUM> can be implemented on a small-scale and low-power-consumption hardware circuit.

<FIG> is a drawing showing an example of a configuration of a computer <NUM> on which an image recognition device <NUM> according to the present embodiment is implemented.

The computer <NUM> is mounted in vehicles and used for driving support, such as automatic driving and navigation, for example.

Although a case where the computer <NUM> performs image recognition processing will be described in the following, this technology can be widely applied to identification processing performed by artificial intelligence, such as voice recognition and medical diagnosis.

The computer <NUM> is composed of a Central Processing Unit (CPU) <NUM>, a Read Only Memory (ROM) <NUM>, a Random Access Memory (RAM) <NUM>, a camera <NUM>, an image recognition device <NUM>, a storage device <NUM>, an input device <NUM>, an output device <NUM>, and the like.

The CPU <NUM> performs desired processing in accordance with an application program stored in the storage device <NUM>, and also performs control of each unit of the computer <NUM>, for example.

The ROM <NUM> is a read only memory which stores basic programs, parameters, and the like to operate the computer <NUM> by the CPU <NUM>.

The RAM <NUM> is a readable/writable memory which provides a working memory for the CPU <NUM> to exhibit an application function.

An identification result of the image recognition performed by the image recognition device <NUM> is stored in the RAM <NUM> and is used in accordance with an application program.

The camera <NUM> is a moving image capturing camera, and is configured to capture a moving image of a subject and to output the image data composed of moving image frames to the image recognition device <NUM> in accordance with a time series. The aforementioned image data is functioned as recording data in which the identification object is recorded.

The image recognition device <NUM> is an information processing device composed of a hardware device configured to identify a person who is an image recognition object (not a specific person but a general pedestrian or the like is meant herein) from image data, and to output an identification result thereof. The image recognition device <NUM> includes an image processing device <NUM> configured to extract and acquire a feature amount from the image data, and an identification device <NUM> configured to recognize and identify an identification object from the aforementioned extracted feature amount. The image processing device <NUM> is functioned as a feature description unit.

Generally, the image recognition system is composed as one set of a module configured to acquire the feature amount and a module configured to identify the feature amount.

The image processing device <NUM> is formed into a semiconductor chip (IC chip) with a semiconductor device <NUM>, and the aforementioned IC chip is implemented on a video capture board <NUM>. The details of a circuit configuration of the image processing device <NUM> will be described below (refer to <FIG> and the like).

By the way, there has been an image recognition technology for recognizing an object by extracting a luminance gradient distribution as a feature amount from an image, and comparing the extracted luminance gradient distribution with a luminance gradient distribution of a previously learned image.

As a feature amount according to the luminance gradient distribution, a Histograms of Oriented Gradients (HOG) feature amount has been well known and has been actively studied.

A Co-occurrence HOG (CoHOG) feature amount is one of a feature amount into which the HOG feature amount is developed, and has more robustness than that of the HOG feature amount.

Moreover, a Multi Resolution CoHOG (MRCoHOG) feature amount having further more robustness than that of the CoHOG feature amount has been proposed in recent years.

It has been clear by experiments that the MRCoHOG feature amount has extremely high robustness.

Further, a Haar-like feature amount is also present as another feature.

Such feature amounts can be applied to the image processing device <NUM>. The latter half of this specification will describe a hardware configuration example of the image processing device <NUM> using the MRCoHOG feature amount, as an example.

The identification device <NUM> is Binarized Neural Networks (BNN, it is also called a binary neural network in Japan) that has learned an identification object in advance, and is configured to receive an input of the feature amount output from the image processing device <NUM> and to identify whether or not the identification object is present in the image data.

The identification device <NUM> is also formed into an IC chip. Although not illustrated, the identification device <NUM> with the semiconductor device <NUM> can also be implemented on the video capture board <NUM>, and image recognition device <NUM> can also be realized by the integrated video capture board <NUM>.

Alternatively, the image processing device <NUM> and the identification device <NUM> can be formed to be integrated with in the semiconductor device <NUM>, and then can also be implemented on the video capture board <NUM>.

The storage device <NUM> is a storage device using, e.g., a storage medium, such as a hard disk or a semiconductor memory, and is configured to store an application program which allows the CPU <NUM> to perform application processing using the identification result of the image recognition.

Moreover, the storage device <NUM> also includes a data storage unit storing an operation setting of the application program, and the like.

In this operation setting, when the image recognition device <NUM> detects a person, a content whether or not to issue an alarm to a driver is set by the user, for example.

The input device <NUM> is a device through which various kinds of information are input to the computer <NUM>, and is composed of input devices such as operation buttons which allow a user to operate the computer <NUM>.

The output device <NUM> is a device for outputting various information from the computer <NUM>, for example, and is composed of output devices such as a liquid crystal display for display an operation screen or displaying a person detected by the image recognition device <NUM> on a moving image captured by the camera <NUM> to be surrounded with a rectangle frame.

Next, the identification device <NUM> will now be described.

The identification device <NUM> is configured to identify a feature amount with the BNN, i.e., binarized neural networks. As will be described later, the identification device <NUM> includes an identification unit <NUM> (<FIG>, <FIG>, and <FIG>) functioned as an identification means which has learned an identification object using multiple-valued weighting.

The reason why the identification device <NUM> uses the BNN is that a hardware circuit for performing multiplication or the like has a large area, and is difficult to be implemented on a semiconductor device, in the neural network using a general floating point.

As will be described later, since the weight of the BNN is a binary value of <NUM> and -<NUM> and the BNN can be configured using an adder, a counter, and the like, a circuit area can be reduced to approximately <NUM>/<NUM> in the case of using the floating point, for example, and it is easy to be implemented on hardware and power consumption is also reduced. Furthermore, as will be described later, identification performance which can be sufficient to practical use can be exhibited in spite of the small-scaled circuit configuration.

<FIG> is a drawing for describing a scheme of the BNN used by the identification device <NUM>.

The BNN <NUM> includes an input layer composed of input units <NUM>-i (i = <NUM>, <NUM>, <NUM>), an intermediate layer (hidden layer) composed of intermediate units <NUM>-j (j = <NUM>, <NUM>) constituting a hidden unit, and an output layer composed of output units <NUM>-k (k = <NUM>, <NUM>, <NUM>).

It is to be noted that the number of the units which constitute the input layer, the intermediate layer, and the output layer is an example, and may be any number.

These units are calculation units (perceptron) which constitute nodes of the neural network, and form the neural network by fully coupling the respective units between respective phases.

Hereinafter, when the input unit <NUM>-i is not particularly distinguished, it is simply referred to as an input unit <NUM>, and the same applies to the intermediate unit <NUM> and the output unit <NUM>.

For an output from the input unit <NUM>-i to the intermediate unit <NUM>-j, a calculation weight Wji which takes any one value of the binary value of {-<NUM>, <NUM>} is set.

Also for an output from the intermediate unit <NUM>-j to the output unit <NUM>-k, a calculation weight Wkj which takes any one value of the binary value of {-<NUM>, <NUM>} is set.

Although i, j, and k are expressed by subscripts in the diagrams, they are expressed in a normal size in the specification in order to prevent garbled characters. The same applies to other elements.

Moreover, although the variables x, y, z, and w are written in lower-case letters in the diagrams, these are written in upper-case letters of X, Y, Z, and W in the specification in order to improve visibility of the variables and the subscripts.

An Input Xi to the input unit <NUM>-i is a component of a feature amount output from the image processing device <NUM>.

An activating function of the intermediate unit <NUM> is binarized to {-<NUM>, <NUM>}, and an output Yj of the intermediate unit <NUM>-j takes any one of the binary value of {-<NUM>, <NUM>}.

The output unit <NUM>-k sums up the output of the intermediate unit <NUM> and outputs the positive/negative sign thereof as a binary value of {-<NUM>, <NUM>}.

An output Zk of the output unit <NUM>-k corresponds to a k-th identification object. For example, the output unit <NUM>-<NUM> corresponds to a person, outputs Z1=<NUM> when a person is identified, and outputs Z1=-<NUM> when no person is detected. The same applies to the other output units <NUM>. Hereinafter, these calculations will now be described.

<FIG> is a drawing showing a portion <NUM> (a portion of the input layer and the intermediate layer) of <FIG>.

The input unit <NUM>-i performs an operation f based on Wji (Xi, Wji) to the input Xi and outputs the result thereof to the intermediate unit <NUM>-j. This operation is an operation for equalizing the positive/negative sign of Xi to the positive/negative sign of Wji; and if Wji is <NUM>, f(Xi, Wji)=Xi, and if Wji is -<NUM>, f(Xi, Wji)=-Xi.

In the example of the drawing, the input units <NUM>-<NUM> and <NUM>-<NUM> respectively calculate f(X2, W22) and f(X3, W23) and output the results thereof to the intermediate unit <NUM>-<NUM>.

On the other hand, the intermediate unit <NUM>-j adds a value output from each input unit <NUM>-i to the intermediate unit <NUM>-j in accordance with the equation <NUM>, and outputs the positive/negative sign by outputting Yj=<NUM> if the total value is zero or more but outputting Yj=-<NUM> otherwise. Thus, the intermediate unit <NUM> is functioned as an adder for the input unit <NUM>.

In the example of the drawing, the intermediate unit <NUM>-<NUM> adds output values from the input units <NUM>-<NUM> and <NUM>-<NUM>.

<FIG> is a drawing showing a portion <NUM> of <FIG>.

The intermediate unit <NUM>-j takes the exclusive NOR between Yj and Wkj in accordance with Xnor(Yj, Wkj) in the equation <NUM> and outputs the result thereof to the output unit <NUM>-k.

More specifically, the intermediate unit <NUM>-j outputs <NUM> to the output unit <NUM>-k if (Yj, Wkj) is (<NUM>, <NUM>) and (-<NUM>, -<NUM>) but outputs -<NUM> otherwise.

On the other hand, the output unit <NUM>-k is composed using a counter. The output unit <NUM>-k adds a binary value sent from each intermediate unit <NUM>-j in accordance with the equation <NUM> and outputs the positive/negative sign by outputting Zk=<NUM> if it is zero or more but outputting Zk=-<NUM> otherwise. The activating function is not applied to the output layer.

In the example of the drawing, the output unit <NUM>-<NUM> calculates the output values of the intermediate units <NUM>-<NUM> and <NUM>-<NUM> in accordance with the equation <NUM>.

As described above, although the BNN <NUM> has been described with reference to <FIG>, these weights Wji and Wkj are set by learning.

For example, when the feature amount input from the input layer corresponds to a person, the output unit <NUM>-<NUM> is set as <NUM> and other output units <NUM> are set as -<NUM>; when the feature amount corresponds to a background, the output unit <NUM>-<NUM> is set as <NUM> and other output units <NUM> are set as -<NUM>; and when the feature amount corresponds to another object (e.g., a cat), the output unit <NUM>-<NUM> is set as <NUM> and other output units <NUM> are set as -<NUM>.

As previously described, since the weight etc. are real numbers in the case of the neural network using the general floating point, it is necessary to calculate floating point multiplication. However, the BNN <NUM> can be composed of the adding circuit using the adder and the counter (subtraction is also kind of addition).

Therefore, since the BNN <NUM> does not need to perform multiplication using a floating point and only needs to perform addition, the circuit configuration thereof can be simple and the circuit area can be reduced.

Thus, the binary neural networks (BNN <NUM>) are composed using the adder for binarizing and adding the feature amount, and the counter for calculating the output of the aforementioned adder.

The BNN <NUM> described above has a single intermediate layer, but may have a multilayer structure. In this case, all intermediate layers perform calculation with the binary activating function in the similar manner to the intermediate unit <NUM>.

Moreover, although the number of units in the intermediate layer is set less than that in the input layer or the output layer, it can be also larger than that in the input layer or the output layer. When the number of the intermediate layers is smaller, the input feature amount can be more narrowed down, and when the number of the intermediate layers is larger, the dimension of the feature amount increases and identification object can be easily separated. Since the number of units in the intermediate layer has such a property, an appropriate number thereof can be obtained by trial and error.

Moreover, although the BNN <NUM> is calculated using the binary value, the BNN <NUM> can also be configured so as to be calculated using three or more discrete values.

<FIG> is a drawing for describing an identification device <NUM> according to the present embodiment.

The identification device <NUM> includes a selection unit <NUM> and an identification unit <NUM>.

The image processing device <NUM> extracts a high-dimensional feature amount from image data of a moving image frame, and outputs the extract feature amount to the selection unit <NUM>.

Thus, the image processing device <NUM> is functioning as a feature amount acquiring means for acquiring the feature amount of the identification object data (image data).

Herein, the feature amount is a MRCoHOG feature amount, as an example. The MR-CoHOG feature amount is high-dimensional vector data having <NUM>,<NUM> dimensions (in which components are arranged in a predetermined order, specifically a histograms as described later), and is composed of <NUM>,<NUM> components.

The reason why the feature amount is set to the high dimension is that the image recognition device <NUM> is particularly effective in the case of such a high dimension, but it is also possible to use a feature amount that is not a high dimension.

The selection unit <NUM> selects a component composed of a predetermined portion specified in advance from a before-selection feature amount <NUM> input from the image processing device <NUM> and inputs a selected after-selection feature amount <NUM> into the identification unit <NUM>.

Thus, the selection unit <NUM> is functioning as a selecting means for selecting a portion which is used for the identification specified in advance from the feature amount acquired by the extraction.

By selecting and culling the high-dimensional feature amount, the components of the feature amount used for the identification can be reduced. Thereby, a circuit of the identification device <NUM> can be miniaturized and a circuit area can also be reduced. Moreover, the power consumption can be reduced accordingly.

Although the portion of the feature amount to be selected may be specified at random, a portion effective in improvement in identification accuracy (detection accuracy for correctly detecting an object) is specified for selecting in order to improve identification performance, in the present embodiment.

Accordingly, in the present embodiment, a portion of the feature amount which contributes to improvement in the identification accuracy is determined using an algorithm of an identification instrument called Real AdaBoost (hereinafter RAdB).

Although the RAdB is an algorithm widely used for identification instruments, the present embodiment uses the RAdB for selecting the feature amount instead of for identification.

In the RAdB, when the number of components of the feature amount to be selected is specified, the components corresponding to the number thereof are automatically specified to be output. As described above, the inventors of the present application have reclaimed such a novel method for use of the RAdB.

Here, the selecting means selects, from the feature amount, a portion specified in advance by the identification algorithm, such as RAdB.

Moreover, the selecting means selects, from the feature amount, a portion, in which the identification accuracy by the identification means becomes high, specified in advance by the identification algorithm.

It is to be noted that the setting means of the portion to be selected is not be limited to this example, and may be determined from the characteristics of each feature description.

Since it is also possible to consider that an input terminal sequence of the before-selection feature amount <NUM> is an input layer, and to consider that an output terminal sequence of the after-selection feature amount <NUM> and the input layer composed of the input unit <NUM> are an intermediate layer composed of two layers, a part where the identification accuracy is increased may be searched while changing the component to be selected, as a part of the learning.

Although the RAdB is used for the selection unit <NUM> in a stage of specifying the component to be selected, this is used to be fixed, after once specified. Accordingly, the select function can be realized by connecting between the terminal of the before-selection feature amount <NUM> and the terminal of the after-selection feature amount <NUM> and terminating other terminals of the before-selection feature amount <NUM> instead of connecting.

Although the portion selected is determined from the viewpoint of specifying the component having a large effect at the time of identifying in the above-described example, the component to be selected can also be determined from the viewpoint of simplifying the circuit of the image processing device <NUM>.

In other words, when a set of certain components of the feature amount is dependent on a certain circuit of the image processing device <NUM>, and an influence on the identification performance is small without selecting the set of the components, and when the image processing device <NUM> can calculate another feature amount even if the circuit is omitted, the components belonging to the set together with the circuit for calculating the components can be omitted from the image processing device <NUM>.

For example, in the MR-CoHOG feature amount, the histogram of co-occurrence of the luminance gradient is used as the feature amount among a low-resolution image, a medium-resolution image, and a high-resolution image. However, when the desired identification accuracy can be obtained if taking the co-occurrence between the low-resolution image and the high-resolution image, the medium-resolution image is unnecessary. Accordingly, it becomes possible to omit, from the image processing device <NUM>, a circuit configuration for generating the medium-resolution image, for calculating the luminance gradient of the generated medium-resolution image, or for measuring the co-occurrence of the luminance gradient among the medium-resolution image, the low-resolution image, and the high-resolution image.

A portion of the feature amount from which desired identification accuracy is obtained from both of a viewpoint of selecting the component which contributes to the identification accuracy and a viewpoint of simplifying the circuit configuration of the image processing device <NUM> is driven therein, and thereby it is also possible to set the component to be selected.

In this case, the feature amount acquiring means acquires a feature amount based on distribution of co-occurrence of the luminance gradient extracted by the feature amount extraction means from the image data which is identification object data, and the selection means selects from the feature amount portion in which the extraction processing or the extraction circuit configuration by the feature amount extraction means is simplified, specified in advance by the identification algorithm.

The identification unit <NUM> uses the BNN <NUM>, and performs identification processing using the after-selection feature amount <NUM> selected from the terminal sequence of the before-selection feature amount <NUM>.

Thus, the identification unit <NUM> includes an input means for inputting the selected portion into the identification means, and is composed the binary neural network to which learning of the identification object (a person and a background in this example) is already conducted by binarized weighting.

When an object is identified from the person by the identification processing, the identification unit <NUM> sets the output unit <NUM>-<NUM> as <NUM> and sets the output unit <NUM>-<NUM> as -<NUM>, and outputs an identification result thereof, and when an object is identified from the background (no person is captured = background), the identification unit <NUM> sets the output unit <NUM>-<NUM> as -<NUM> and sets the output unit <NUM>-<NUM> as <NUM>, and outputs an identification result thereof.

Thus, the identification unit <NUM> includes an output means for outputting the identification result of being identified by the identification means using the input portion.

After composing such an image recognition device <NUM>, the inventors of the present application have performed various experiments verify how much the number of the components of the feature amount can be narrowed down by selection and whether or not the circuit configuration of the identification unit <NUM> can be simplified.

These experiments will now be described. Each of the experiments have been performed using the MR-CoHOG feature amount.

<FIG> shows an experimental result showing change of identification accuracy in the case of reducing the number of intermediate units <NUM> in the intermediate layer while the number of input dimensions remains <NUM>,<NUM> (i.e., there are <NUM>,<NUM> input units <NUM>) without selecting the feature amount.

For comparison, identification accuracy when the identification device <NUM> is composed of the RAdB is also written.

As shown in the drawing, the identification accuracy in the case of the RAdB is <NUM>%.

On the other hand, the identification accuracy when sequentially reducing the number of the units in the intermediate layer to <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> is <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, and <NUM>%.

As can be seen from the experimental results, even if the number of the intermediate units <NUM> is one, the identification accuracy is <NUM>% or more, and therefore it can sufficiently withstand to practical use.

<FIG> shows an experimental result showing change of identification accuracy in the case of reducing the number of intermediate units <NUM> in a state where the input dimensions is reduced to <NUM> dimensions (i.e., there are <NUM> input units <NUM>) by selecting the feature amount.

<FIG> shows an experimental result showing change of identification accuracy when the number of intermediate units <NUM> is reduced to one and the input dimensions of the feature amount to be selected are sequentially reduced from <NUM> (i.e., when the input units <NUM> are reduced from <NUM>).

As shown in the drawing, the identification accuracy in the case of reducing the input dimensions to <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> is <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, and <NUM>%.

As can be seen from the experimental results, even if the input dimension is four-dimensional and the number of the intermediate units <NUM> is one, the identification accuracy is <NUM>% or more, and therefore it can withstand to practical use depending on an application purpose. Moreover, when the input dimension is <NUM> dimensions or more, the identification accuracy is <NUM>% or more, and therefore it can sufficiently withstand to practical use.

As described above, although the experimental results have been described with reference to <FIG>, the learning is performed whenever the input dimension or the number of the units are changed.

<FIG> shows a table comparing between a circuit scale when the identification device is composed of the RAdB and a circuit scale when the identification device is composed of the identification device <NUM>. With regard to identification device <NUM>, a case where the number of the intermediate units <NUM> is one is shown.

The resister is a memory having a small capacitor, and <NUM> resisters are required in the case of the RAdB, whereas only <NUM> resisters are sufficient in the case of the identification device <NUM>.

The LUTs are lookup tables used in order to replace complicated computation by reference process of simple array.

<NUM>,<NUM> LUTs are required in the case of the RAdB, whereas only <NUM> LUTs are sufficient in the case of the identification device <NUM>.

The DSP is a digital signal processor, and <NUM> DSPs are required in the case of the RAdB, whereas no DSP is required in the case of the identification device <NUM>.

The block RAM is a large-capacity memory, and two block RAMs are required in the case of the RAdB, whereas no block RAM is required in the case of the identification device <NUM>.

As described above, the identification device <NUM> can be composed of a small-scaled circuit as compared with RAdB conventionally used as an identification instrument, and is suitable for being formed into a semiconductor device, i.e., an IC chip.

<FIG> is a table comparing between a memory capacity required when the identification device is composed by the RAdB and a memory capacity required when the identification device is composed of the identification device <NUM> having one intermediate unit.

As shown in the table, <NUM> kilobits are required for the RAdB, whereas only <NUM> kilobit is required for the identification device <NUM> (when the feature amount to be selected is <NUM> dimensions).

<FIG> is a flow chart for describing an operation of the image recognition device <NUM> of the present embodiment.

The following processing is performed by hardware circuits of the image processing device <NUM> and the identification device <NUM>.

First, the image processing device <NUM> receives an input of a moving image frame output from the camera <NUM> (Step <NUM>).

Next, the image processing device <NUM> processes the moving image frame in accordance with the circuit, extracts before-selection feature amount <NUM> of the moving image frame to be output to the identification device <NUM> (Step <NUM>).

On the other hand, the identification device <NUM> selects the before-selection feature amount <NUM> received from the image processing device <NUM> in the selection unit <NUM> (Step <NUM>), and inputs the after-selection feature amount <NUM> into the identification unit <NUM> (Step <NUM>).

Next, the identification device <NUM> performs identification processing by calculating the after-selection feature amount <NUM> using the BNN <NUM>, and outputs an identification result obtained as a result of the calculation (Step <NUM>).

Next, the image recognition device <NUM> determines whether or not to terminate the processing, and if terminating the processing (Step <NUM>; Y), the image recognition device <NUM> terminates the image recognition processing, whereas if not terminating the processing (Step <NUM>; N), the image recognition device <NUM> returns to Step <NUM> to perform the image recognition processing for the next moving image frame.

The determination whether or not to terminate the processing is on the basis of determining whether or not a user has instructed the termination from a menu screen which is not illustrated, for example.

According to the first embodiment described above, the following effects can be obtained.

A low-dimensional feature amount may be used depending on the image processing device <NUM>.

For example, in the technology of Non-Patent Literature <NUM>, since the identification is performed from a low-dimensional feature amount (i.e., approximately <NUM> dimensions), detection accuracy of a person is limited.

When performing more highly accurate detection, it is necessary to calculate a high-dimensional feature amount, but the calculation cost is increased if all the feature amounts are calculated as they are.

Moreover, studies have been conducted to further multivalue the BNN, in order to ensure required identification accuracy.

However, if the feature amount is formed into high dimensions or the neural network is multiple-valued, a circuit is complicated and the circuit area is also increased.

For that reason, the inventors of the present application have succeeded in improving the identification accuracy by duplicating a low-dimensional feature amount while the neural network remains binarized.

Hereinafter, image recognition processing according to the aforementioned duplication will now be described.

The image recognition device <NUM> is composed of an image processing device <NUM> and an identification device <NUM>, and the identification device <NUM> includes a duplication unit <NUM> and an identification unit <NUM>.

The image processing device <NUM> outputs a feature amount extracted from a moving image frame to the duplication unit <NUM>.

As an example, the image processing device <NUM> is configured to extract from the moving image frame a low-dimensional approximately <NUM>-dimensional HOG feature amount (i.e., approximately <NUM> components are present), and to output the extracted HOG feature amount to the duplication unit <NUM>.

Here, the image processing device <NUM> is functioning as a feature amount acquiring means for acquiring the feature amount of an identification object from recording data (image data of the moving image frame) in which the aforementioned identification object is recorded. The identification device <NUM> includes an identification object data acquiring means for acquiring the aforementioned feature amount as the identification object data.

Moreover, the feature amount acquiring means acquires distribution of co-occurrence of the luminance gradient according to the HOG feature amount, in the aforementioned image data, as the feature amount.

The duplication unit <NUM> duplicates a before-duplication feature amount <NUM> input from the image processing device <NUM> by a predetermined number, and generates an after-duplication feature amount <NUM> (duplicated to twice in the example of the drawing) to be input into the identification unit <NUM>.

Thus, the duplication unit <NUM> includes the duplication means for duplicating the identification object data.

The duplication is performed by connecting an output terminal of the before-duplication feature amount <NUM> to input terminals of a plurality of the after-duplication feature amounts <NUM> in parallel, for example.

By redirecting an output destination of the before-duplication feature amount <NUM> to the terminals of the plurality of the after-duplication feature amount <NUM>, each component may be input into the identification unit <NUM> multiple times by sequentially outputting the before-duplication feature amount <NUM>, and such a case is also included in the duplication.

The identification unit <NUM> uses the BNN <NUM> performs identification processing using the after-duplication feature amount <NUM> selected from the terminal sequence of the before-duplication feature amount <NUM>.

Thus, the identification unit <NUM> includes an input means for inputting the duplicated identification object data into the identification means, and is composed the binary neural network to which learning of the identification object (a person and a background in this example, as described in the following) is already conducted by binarized weighting.

The BNN <NUM> which constitutes the binary neural network is composed using an adder configured to multiple-value and add the duplicated identification object data, and a counter configured to count the output of the adder.

Thus, the identification unit <NUM> includes an output means for outputting the identification result of being identified using the input identification object data.

<FIG> is a drawing for considering an improvement in identification accuracy by duplicating a feature amount.

As shown in the subsequent experimental results, when the dimensions are increased by duplicating the feature amount to be input into the identification unit <NUM>, the identification accuracy can be improved.

This is for the following reasons. Since a weight and an activating function are binarized for calculation in the intermediate layer as shown in the left diagram of <FIG> when not duplicating, the value which can be expressed inside the networking system from one component of the feature amount is limited to {-X, X}. In contrast, when being duplicated to twice, as shown in the right diagram thereof, the value which can be expressed is increased to {-2X, <NUM>, 2X}. When being duplicated to three times or more, the value which can be expressed is further increased.

<FIG> shows experimental results showing change of identification accuracy due to the duplication.

The identification accuracy when a feature amount acquired from the original image data is <NUM>-dimensional and is not duplicated, the identification accuracy when this is duplicated to twice (2x magnification) (duplicated once) to be <NUM> dimensions, the identification accuracy when further being duplicated to three times (3x magnification) (duplicated twice) to be <NUM> dimensions, the identification accuracy when being duplicated to four times (4x magnification) (duplicated <NUM> times) to be <NUM> dimensions, the identification accuracy when being duplicated to five times (5x magnification)(duplicated four times) to be <NUM> dimensions, and the identification accuracy when the feature amount is <NUM>-dimensional and is not duplicated are respectively <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, and <NUM>%.

Thus, since the identification accuracy is improved whenever increasing the number of the duplications, and the identification accuracy of approximately <NUM>% to approximately <NUM>% can be ensured, it can sufficiently withstand to practical use.

As proved from this experiment, the identification accuracy can be improved by a simple process of duplicating low-dimensional feature amounts, without using high-dimensional feature amounts or multi-valuing the neural networks to three or more values.

Next, the image processing device <NUM> processes the moving image frame in accordance with the circuit, extracts the feature amount (before-duplication feature amount <NUM>) of the moving image frame to be output to the identification device <NUM> (Step <NUM>).

On the other hand, the identification device <NUM> duplicates the before-duplication feature amount <NUM> received from the image processing device <NUM> (Step <NUM>), and inputs the generated after-duplication feature amount <NUM> into the identification unit <NUM> (Step <NUM>).

Next, the identification unit <NUM> in the identification device <NUM> calculates the input after-duplication feature amount <NUM>, and the identification device <NUM> outputs the identification result obtained as a result of the calculation (Step <NUM>).

It is to be noted that, in the present embodiment, although the feature amount is extracted from the image data of the moving image frame, the image data of the moving image frame can be directly input to the identification device <NUM> without providing the image processing device <NUM> in the image recognition device <NUM>.

In this case, the identification object data acquired by the identification data acquiring means is image data (corresponding to recording data) of the moving image frame.

Moreover, the number of duplications may be changed for each component, for example, a first component of the feature amount is duplicated to two pieces, a second component is duplicated to four pieces, and so on.

According to the second embodiment described above, the following effects can be obtained.

The identification device <NUM> of the present embodiment is composed by combining the first embodiment and the second embodiment with each other.

The identification device <NUM> includes a selection unit <NUM>, a duplication unit <NUM>, and an identification unit <NUM>; and these configurations are the same as those described in the above embodiments.

The image processing device <NUM> outputs a feature amount to the selection unit <NUM>, and the selection unit <NUM> selects a component used for identification and inputs the selected component into the duplication unit <NUM>.

In response, the duplication unit <NUM> duplicates the feature amount input from the selection unit <NUM> to be input into the identification unit <NUM>.

Then, the identification unit <NUM> calculates the duplicated feature amount, to identify the image recognition object.

It is to be noted that wiring may be provided in the identification device <NUM> so that the input means of identification device <NUM> may perform the duplication, without providing the after-duplication feature amount <NUM>.

Thus, the image recognition device <NUM> of the present embodiment includes a selection means for selecting, from feature amount, a portion which is used for the identification which is specified in advance, a duplication means for duplicating the portion selected by the aforementioned selection means, and an input means for inputting the portion selected by the selection means and the portion duplicated by the duplication means into the identification means.

Next, the image processing device <NUM> processes the moving image frame in accordance with the circuit, extracts the feature amount (before-selection feature amount <NUM>) of the moving image frame to be output to the identification device <NUM> (Step <NUM>).

On the other hand, the identification device <NUM> selects the before-selection feature amount <NUM> received from the image processing device <NUM> in the selection unit <NUM> (Step <NUM>), and inputs the after-selection feature amount <NUM> into the duplication unit <NUM>.

The duplication unit <NUM> receives the after-selection feature amount <NUM> input from the selection unit <NUM> as a before-duplication feature amount <NUM> to be duplicated (Step <NUM>), and inputs the after-duplication feature amount <NUM> into the identification unit <NUM> (Step <NUM>).

Next, the identification device <NUM> performs identification processing by calculating the duplicated feature amount using the BNN <NUM> in the identification unit <NUM>, and outputs an identification result obtained as a result of the calculation (Step <NUM>).

As described above, although the duplication is performed after the selection, the order of the duplication unit <NUM> and the selection unit <NUM> may be replaced with each other to perform the selection after the duplication of the feature amount.

The first to third embodiments described above can be configured as follows.

An information processing device comprising: a feature amount acquiring means configured to acquire a feature amount of identification object data; a selection means configured to select a feature amount of a portion, used for identification, specified in advance from the feature amount acquired by the feature amount acquiring means; an identification means configured to have learned an identification object using multiple-valued weighting; an input means configured to input the feature amount of the portion selected by the selection means into the identification means; and an output means configured to output an identification result of being identified by the identification means using the feature amount of the portion input by the input means.

The information processing device according to the eleventh configuration, wherein in the identification means, the learning of the identification object is conducted by binarized weighting.

The information processing device according to the eleventh configuration or twelfth configuration, wherein the selection means selects a feature amount of a portion specified by an identification algorithms, such as RAdB, in advance, from the feature amount acquired by the feature amount acquiring means.

The information processing device according to the thirteenth configuration, wherein the selection means selects a feature amount of a portion, in which identification accuracy by the identification means becomes high, specified by the identification algorithm in advance, from the feature amount acquired by the feature amount acquiring means.

The information processing device according to the thirteenth configuration, wherein the feature amount acquiring means acquires a feature amount based on distribution of co-occurrence of a luminance gradient extracted by the feature amount extraction means from the image data which is identification object data, and the selection means selects a feature amount of a portion in which extraction processing or an extraction circuit configuration by the feature amount extraction means is simplified, specified by the identification algorithm in advance, from the feature amount acquired by the feature amount acquiring means.

The information processing device according to any one of the eleventh to fifteenth configurations, wherein the identification means is a binary neural network.

The information processing device according to the sixteenth configuration, wherein the binary neural network is composed using an adder for binarizing and adding the feature amount of the portion, and a counter for calculating an output of the adder.

The information processing device according to any one of the eleventh to sixteenth configurations, further comprising a duplication means configured to duplicate the feature amount of the portion selected by the selection means, wherein the input means inputs into the identification means the feature amount of the portion selected by the selection means and the feature amount of the portion duplicated by the duplication means.

An information processing device comprising: an identification object data acquiring means configured to acquire identification object data; an identification means configured to have learned an identification object using multiple-valued weighting; a duplication means configured to duplicate the identification object data acquired by the identification object data acquiring means; an input means configured to input the identification object data duplicated by the duplication means into the identification means; and an output means configured to output an identification result of being identified by the identification means using the feature object data of the portion input by the input means.

The information processing device according to the twenty-first configuration, wherein in the identification means, the learning of the identification object is conducted by binarized weighting.

The information processing device according to the twenty-first configuration or twenty-second configuration, wherein the identification means is a binary neural network.

The information processing device according to the twenty-first configuration, the twenty-second configuration, or the twenty-third configuration, further comprising: a feature amount acquiring means configured to acquire a feature amount of an identification object from recording data in which the aforementioned identification object is recorded, wherein the identification object data acquiring means acquires the feature amount acquired by the feature amount acquiring means as an identification object data.

The information processing device according to the twenty-fourth configuration, wherein the recording data are image data, and the feature amount acquiring means acquires distribution of co-occurrence of a luminance gradient in the image data as the feature amount.

The information processing device according to the twenty-third configuration, wherein the binary neural network is composed using an adder for multiple-valuing and adding the identification object data duplicated by the duplication means, and a counter for calculating an output of the adder.

Although three embodiments with regard to the identification device <NUM> have been described above, the image processing device <NUM> which is another element constituting the image recognition device <NUM> will now be described hereinafter.

The image processing device <NUM> (<FIG>) arranges: in parallel, a processing line for high-resolution images composed from a three lines buffer 25a to a buffer 28a for extracting a luminance gradient direction from a high-resolution image; a processing line for medium-resolution images composed from a medium-resolution unit 24b to a buffer 28b for extracting a luminance gradient direction from a medium-resolution image; and a processing line for low-resolution images composed from a low-resolution unit 24c to a buffer 28c for extracting a luminance gradient direction from a low-resolution image. The image processing device <NUM> simultaneously extracts the luminance gradient direction for every pixel from the three resolution images in parallel.

Each of co-occurrence-matrix creation units 30a, 30b, and 30c is configured to create a co-occurrence matrix using the luminance gradient direction extracted from the three resolution images, and a histogram creating unit <NUM> outputs a histogram as a MRCoHOG feature amount using this co-occurrence matrix.

Since three resolution images are simultaneously processed, high-speed processing can be realized, and moving images output from a camera can be processed in real time.

First, the HOG feature amount, the CoHOG feature amount, and the MRCoHOG feature amount will now be briefly described.

<FIG> are drawings for illustrating a concept of the HOG feature amount.

The HOG feature amount is extracted from an image by the following procedure.

An image <NUM> shown in a left drawing of <FIG> is assumed to be image-of-interest regions extracted by an observation window etc. for observing an object.

First, the image <NUM> is divided into rectangular cells 102a, 102b,.

Next, as shown in a right drawing of <FIG>, luminance gradient directions (directions from a low luminance toward a high luminance) of respective pixels are quantized into, e.g., eight directions in accordance with each cell <NUM>.

Subsequently, as shown in <FIG>, the quantized directions of the luminance gradients are determined as classes, and a histogram showing the number of occurrence as a frequency is produced, whereby the histogram <NUM> of the luminance gradients included in the cell <NUM> is produced in accordance with each cell <NUM>.

Further, normalization is performed in such a manner that a total frequency of the histograms <NUM> becomes <NUM> in blocks each forming a group of several cells <NUM>.

In the example shown in the left drawing of <FIG>, the cells 102a, 102b, 102c, and 102d form one block.

A histogram in which the histograms 106a, 106b,. normalized in this manner are arranged in a line as shown in <FIG> becomes a HOG feature amount <NUM> of the image <NUM>.

<FIG> are drawings for describing the CoHOG feature amount.

The CoHOG feature amount is the feature amount focusing on a gradient pair between two pixels in a local region, and is extracted from an image by the following procedure.

As shown in <FIG>, an image <NUM> is divided into rectangular cells 102a, 102b,. The cell is also called a block.

In the CoHOG feature amount, a pixel of interest <NUM> is set to the cells 102a, 102b,. , and a co-occurrence matrix (histogram with regard to the pixel of interest <NUM>) is created with a combination of the luminance gradient direction of the pixel of interest <NUM> and the luminance gradient direction of pixels which are at distances <NUM> to <NUM> from the pixel of interest <NUM>. The pixel related to the combination with the pixel of interest <NUM> is called offset.

For example, the distance from the pixel of interest <NUM> is expressed by expression, and when the aforementioned expression is applied, pixels 1a to 1d which are adjacent to the pixel of interest <NUM> are obtained as a pixel at the distance <NUM>, as shown in <FIG>.

It is to be noted that the reason why the upper and left pixels of the pixel of interest <NUM> are not comprised in the combination is that the pixel of interest <NUM> is set and processed in order from the left end of the top pixel row toward the right; and therefore the processing has been already completed.

Next, the luminance gradient directions of the pixel of interest <NUM> and the pixel 1a are observed. The luminance gradient direction is quantized into, for example, eight directions, and the directions are shown by the arrows in the drawing.

The luminance gradient direction of the pixel of interest <NUM> is a right direction, and the luminance gradient direction of the pixel 1a is an upper right direction.

Therefore, one vote is cast for an element of (row number, column number)=(right direction, upper right direction), in the co-occurrence matrix <NUM> shown in <FIG>.

In the example of <FIG>, as a set of the luminance gradient directions of the pixel of interest <NUM> and the pixel 1a, as a result of adding <NUM> to an element of a row in which the arrow in the right direction is described as a row number and a column in which the arrow in the upper right direction is described as a column number, the value of the aforementioned element is <NUM>.

It is to be noted that the co-occurrence matrix <NUM> should be fundamentally drawn with a three-dimensional histogram and the number of votes should be fundamentally expressed by a bar graph in a height direction, but the number of votes is expressed by a numerical value in order to simplify the drawing.

Hereinafter, voting (counting) according to the combination of the pixel of interest <NUM> and the pixels 1b, 1c, and 1d is similarly performed.

As shown in <FIG>, centered on the pixel of interest <NUM>, the pixels of the distance <NUM> is specified to the pixels 2a to 2f of the outer periphery of the pixels 1a to 1d, the pixels of the distance <NUM> is specified to the pixels 3a to <NUM> of the further outer periphery thereof, and the pixels of the distance <NUM> is specified to the pixels 4a to <NUM> of the further outer periphery thereof.

These are similarly voted for the co-occurrence matrix <NUM> in combination with the pixel of interest <NUM>.

The above-described voting processing is performed with respect to all the pixels that constitute the cell <NUM>, and the co-occurrence matrix for every pixel is obtained.

Furthermore, a histogram in which this processing is performed in all the cells <NUM> and all the components of the co-occurrence matrix are arranged in a line as shown in <FIG> is the CoHOG feature amount <NUM> of the image <NUM>.

<FIG> are drawings for describing the MRCoHOG feature amount.

The MRCoHOG feature amount significantly reduces the offset number by co-occurring between different resolutions of the same image.

First, as shown in <FIG>, a high-resolution image <NUM> (original image), a medium-resolution image <NUM>, and a low-resolution image <NUM> are obtained by generating images having different resolutions (image sizes) from an original image. The grid in the image represents the pixel. Although not illustrated, a cell (also called a block) is set also to each of the resolution images.

Then, the luminance gradient direction quantized with respect to each pixel of the high-resolution image <NUM>, the medium-resolution image <NUM>, and the low-resolution image <NUM> is calculated.

Although the medium-resolution image <NUM> and the low-resolution image <NUM> are used for extraction of the MRCoHOG feature amount, the medium-resolution image <NUM> and the low-resolution image <NUM> are extended to the medium-resolution image 121a and the low-resolution image 122a so as to have the same size as that of the high-resolution image <NUM>, as shown in <FIG>, in order to make it easy to understand.

Next, as shown in <FIG>, in the similar manner to the CoHOG feature amount, co-occurrence (combination of the luminance gradient directions) between the luminance gradient direction in a pixel of interest <NUM> of the high-resolution image <NUM> and the luminance gradient direction in surrounding pixels 1a to 1d of the high-resolution image <NUM> is taken, and a vote is cast for a co-occurrence matrix (not illustrated).

Next, a vote is cast for a co-occurrence matrix in accordance with co-occurrence between the pixel of interest <NUM> of the high-resolution image <NUM> and pixels 2a to 2d of the medium-resolution image 121a on the outer periphery of the pixels 1a to 1d, and a vote is further cast for a co-occurrence matrix in accordance with co-occurrence between the pixel of interest <NUM> and pixels 3a to 3d of the low-resolution image 122a on the outer periphery of the pixels 2a to 2d.

In this manner, for the pixel of interest <NUM> of the high-resolution image <NUM>, the co-occurrence matrixes obtained by taking the co-occurrence with the combination in the high-resolution image <NUM>, the combination with the medium-resolution image 121a, and the combination with the low-resolution image 122a are obtained.

This processing is performed for each pixel in the cells of the high-resolution image <NUM>, and is further performed for all cells.

Thereby, the co-occurrence matrix for every pixel of the high-resolution image <NUM> is obtained.

Similarly, a co-occurrence matrix with each resolution image in the case of setting a pixel of interest to the medium-resolution image 121a and a co-occurrence matrix with each resolution image in the case of setting a pixel of interest to the low-resolution image 122a are further calculated. A histogram in which the components of all co-occurrence matrices are arranged in a line as shown in <FIG> is the MRCoHOG feature amount <NUM> of the high-resolution image <NUM>.

In this example, although the histogram obtained by connecting the co-occurrence matrix in the case of setting the pixel of interest to the high-resolution image <NUM>, the co-occurrence matrix in the case of setting the pixel of interest to the medium-resolution image 121a, and the co-occurrence matrix in the case of setting the pixel of interest to the low-resolution image 122a is used as the MRCoHOG feature amount, any one of a histogram according to a co-occurrence matrix in the case of setting a pixel of interest to the high-resolution image <NUM> can be used as the MRCoHOG feature amount, for example.

Alternatively, any two co-occurrence matrices may be combined, or the co-occurrence may be obtained for four or more types of resolution images by further increasing the resolutions.

Experiments conducted by the inventors have revealed that the MRCoHOG feature amount can significantly reduce the feature amount compared with the CoHOG, but the robustness is more effective than that of the CoHOG.

It is assumed that this is because a noise is reduced by lowering the resolution, and the co-occurrence with a part away from the pixel of interest is observed.

Next, an application form of the mathematical calculation formula to hardware will now be described.

In order to calculate the MRCoHOG feature amount, it is necessary to calculate the square root, division, and arc tangent.

However, since a computer performs various calculations, such as the square root etc. by addition, these operations have a large load.

Accordingly, in order to increase the calculation speed or to make a circuit scale appropriate so as to be formed into IC chip, it is necessary to design a calculation method suitable for hardware.

<FIG> are drawings for describing a calculation method used for the present embodiment.

As shown in <FIG>, (a) m(x, y) in the equation (<NUM>) shown in <FIG> indicates a calculation formula of a gradient strength of the luminance gradient of the pixel in the coordinate (x, y).

It is to be noted that, in order to prevent garbled characters, lowercase subscripts are represented by full-width characters.

fx(x, y) and fy(x, y) are respectively the gradient strengths of the luminance in the x direction (horizontal direction / lateral direction) and the y direction (vertical direction / lengthwise direction).

fx(x, y) and fy(x, y) respectively are mathematically obtained by partially differentiating the luminance in the x and y directions. However, in the present embodiment, fx(x, y) is expressed by the difference between the luminances of the pixels adjacent to each other in the horizontal direction (lateral direction) of the pixel of interest, and fy(x, y) is expressed by the difference between the luminances of the pixels adjacent to each other in the vertical direction (lengthwise direction) of the pixel of interest.

As expressed in the equation (<NUM>), although the gradient strength includes the square root, the equation (<NUM>) is approximated by the additive expression of the equation (<NUM>) by replacing Euclidean distance to Manhattan distance.

This replacement is performed by approximating the square root of the Euclidean distance between points TU (t square + u square) by t+u which is Manhattan distance, as expressed in the right diagram of <FIG>. The name of Manhattan is derived from the fact that the streets of Manhattan, a U. city, have a grid pattern.

The gradient strength is an amount that increases as the difference between the luminance levels of the luminance gradients increases, and is used for zero offset.

Although predetermined processing of not taking co-occurrence is performed with regard to that to which the gradient strength does not reach the predetermined threshold value, for example, since the influence exerted on the image identification accuracy is small, description of the aforementioned processing is omitted in the present embodiment.

As a result of the experiment, it is confirmed that replacing the Euclidean distance by the Manhattan distance hardly affected the image recognition capability.

The equation (<NUM>) shown in <FIG> expresses a calculation formula of the luminance gradient direction θ generally used.

Since the expression (<NUM>) includes the division of fy(x, y) by fx(x, y) and the calculation of arctangent, a processing load required for the calculation is increased.

Therefore, a present embodiment focuses attention on the fact that not the accurate value according to the equation (<NUM>) but the quantized luminance gradient direction is required for the calculation of the MRCoHOG feature amount, a correspondence table to which the set of fx(x, y) and fy(x, y) is associated with the luminance gradient direction is prepared without using the equation (<NUM>), and thereby the set of fx(x, y) and fy(x, y) is mapped in the quantized luminance gradient direction.

<FIG> shows a relationship between a range of angle θ and the quantized luminance gradient direction θ.

In the present embodiment, the luminance gradient direction is quantized in the eight directions, as an example.

In this case, as shown in <FIG>, when the luminance gradient direction θ is within a range of <NUM>° <= θ < <NUM>°, it is quantized to <NUM>°; when the luminance gradient direction θ is within a range of <NUM>° <= θ < <NUM>°, it is quantized to <NUM>°; and when the luminance gradient direction θ is within other angles, it is quantized to <NUM>°, <NUM>°, <NUM>°, <NUM>°, <NUM>°, and <NUM>°.

First, this method classifies a combination of positive and negative of fx(x, y) and fy(x, y) into classifications a to d in accordance with the classification <NUM>.

The classification a is a case where fx(x, y) and fy(x, y) are both positive, the classification b is a case where fx(x, y) and fy(x, y) are both negative, the classification c is a case where fy(x, y) is negative and fx(x, y) is positive, and the classification d is a case where fx(x, y) is negative and fy(x, y) is positive.

Next, the magnitude relationship between fx(x, y) and fy(x, y) is compared to be made to correspond to the quantized luminance gradient direction in accordance with the classification <NUM>.

If y of the classification a is equal to or less than x, it corresponds to <NUM>°, and if y of the classification a is greater than x, it corresponds to <NUM>°.

If -y of the classification b is equal to or less than x, it corresponds to <NUM>°, and if -y of the classification b is greater than x, it corresponds to <NUM>°.

If y of the classification c is equal to or greater than x, it corresponds to <NUM>°, and if y of the classification c is less than x, it corresponds to <NUM>°.

If -y of the classification d is equal to or greater than x, it corresponds to <NUM>°, and if -y of the classification d is less than x, it corresponds to <NUM>°.

As described above, in the present embodiment, the luminance gradient direction quantized at high speed can be obtained by referring the correspondence table configured by the classifications <NUM> and <NUM>, without using an arc tangent or division.

Thus, the image processing device of the present embodiment acquires the luminance gradient intensity fx(x, y) in the horizontal direction and the luminance gradient intensity fy(x, y) in the vertical direction of the pixel of interest using the luminance of the pixels adjacent to the pixel of interest, and outputs the quantized gradient direction by referring the acquired luminance gradient intensity in the horizontal direction and the acquired luminance gradient intensity in the vertical direction with the correspondence table to which the positive/negative relationship and the magnitude relationship between the luminance gradient intensity in the horizontal direction and the luminance gradient intensity in the vertical direction are associated with the quantized gradient direction.

<FIG> is a drawing showing a circuit configuration of the image processing device of the present embodiment.

The image processing device <NUM> is formed, as a semiconductor device, on a semiconductor chip, for example.

The image processing device <NUM> includes: a high-resolution image processing line composed from a three lines buffer 25a to a buffer 28a; a medium-resolution image processing line composed from a medium-resolution unit 24b to a buffer 28b; and a low-resolution image processing line composed from a low-resolution unit 24c to a buffer 28c.

Since these pipelines are arranged in parallel and simultaneously perform parallel processing of high-resolution image, medium-resolution image, and low-resolution image, thereby high-speed processing can be realized.

Here, the circuit composed of a combination of the high-resolution image processing line, the medium-resolution image processing line, and the low-resolution image processing line is functioned, using the luminance sequentially output, as a gradient direction output means for sequentially outputting the gradient direction of the luminance of each pixel in a plurality of resolutions in parallel to for every resolution.

Moreover, the high-resolution image processing line, the medium-resolution image processing line, and the low-resolution image processing line are arranged in parallel for each of the plurality of resolutions, and is functioned as a plurality of gradient direction output means for each resolution for outputting the gradient direction of the luminance in the pixel of the aforementioned resolution from the luminance sequentially outputted from the below-described image input unit <NUM>.

The image processing device <NUM> can sequentially output the gradient direction for each resolution in parallel by simultaneously operating these gradient direction output means for each resolution, in synchronization with a clock.

Hereinafter, each circuit which constitutes the image processing device <NUM> will now be described.

In order to calculate the MRCoHOG feature amount, it is only necessary to have luminance data of each pixel constituting the image.

Accordingly, in the present embodiment, Y (luminance) of the pixel is extracted from an image formed in the YUYV format and is input into the image processing device <NUM> as luminance data.

Hereinafter, the luminance data of the pixel on the i-th row and the j-th column of the image and gradient direction data described later are represented by the row number and the column number (e.g., (i-j)) of the corresponding pixel.

The image input unit <NUM> is a circuit configured to sequentially output luminance data of an image of a frame transmitted from a moving image camera on the basis of a pixel order (order that the aforementioned pixels are arranged in the image), and is functioned as a luminance output means for sequentially outputting the luminance of the pixel which constitutes the image on the basis of the order of the aforementioned pixels.

It is to be noted that although the luminance data Y is extracted in advance from the image in YUYV format and is input into the image input unit <NUM> as an image, in the present embodiment, it may be configured so that a luminance component may be extracted from pixel data by the image input unit <NUM> or by the gradient direction calculation units 26a, 26b, and 26c.

As shown in <FIG>, the image <NUM> is composed of the luminance data (<NUM>-<NUM>), (<NUM>-<NUM>), (<NUM>-<NUM>),. , (<NUM>-n) in the first row, luminance data (<NUM>-<NUM>), (<NUM>-<NUM>), (<NUM>-<NUM>),. , (<NUM>-n) in the second row, and luminance data (m-<NUM>), (m-<NUM>), (m-<NUM>),. , (m-n) in the m-th row.

The image input unit <NUM> reads the luminance data from the image <NUM> sent from the image input unit in order from the upper row to the right, and outputs the luminance data (<NUM>-<NUM>), (<NUM>-<NUM>), (<NUM>-<NUM>),. , (<NUM>-n), (<NUM>-<NUM>), (<NUM>-<NUM>),.

Returning to <FIG>, the output lines of the image input unit <NUM> are wired to the three lines buffer 25a, the medium-resolution unit 24b, and the low-resolution unit 24c, and the luminance data output from the image input unit <NUM> is simultaneously output to each of the three lines buffer 25a, the medium-resolution unit 24b, and the low-resolution unit 24c.

<FIG> represents the wiring of the high-resolution luminance data by the thick arrow, represents the wiring of the medium-resolution luminance data by the thin arrow, and represents the wiring of the low-resolution luminance data by the dotted line.

The medium-resolution unit 24b and the low-resolution unit 24c respectively are resolution conversion circuits configured to convert the resolution (size) of the image <NUM> into a half and a quarter.

These resolution conversion circuits respectively generate an image having a half of the resolution and an images having a quarter of the resolution, from the image <NUM>.

The image <NUM> is directly used also as the high-resolution image, without converting the resolution.

The methods of converting (resizing) the resolution include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation.

The nearest neighbor interpolation is a method of extracting pixels before resizing to be directly used. The bilinear interpolation is a method of weighting and averaging a <NUM>×<NUM> region centering on the object pixel. The bicubic interpolation is a method of interpolating a <NUM>×<NUM> region centering on the object pixel with a cubic function.

In the image processing device <NUM>, the nearest neighbor interpolation, which is simple in calculation and further improves detection accuracy (described later) is adopted.

<FIG> are drawings for describing the resolution conversion processing performed by the medium-resolution unit 24b and the low-resolution unit 24c.

As shown in image 40b of <FIG>, the medium-resolution unit 24b reads luminance data at every second frequency (alternately) represented by the hatching among the luminance data of the image <NUM> transmitted from the image input unit <NUM>, and skips the remaining luminance data, thereby generating image data having a resolution of <NUM>/<NUM>, in which luminance data are arranged at every second frequency (alternately) in the vertical direction and the horizontal direction.

As shown in image 40c, the low-resolution unit 24c reads luminance data at every third frequency represented by the hatching among the luminance data of the image <NUM> transmitted from the image input unit <NUM>, and skips the remaining luminance data, thereby generating image data having a resolution of <NUM>/<NUM>, in which luminance data are arranged at every third frequency in the vertical direction and the horizontal direction.

By culling out the luminance data in this manner, the medium-resolution unit 24b generates and outputs the medium-resolution image of which the resolution is one half, and the low-resolution unit 24c generates and outputs the low-resolution image of which the resolution is one fourth.

Since the nearest neighbor interpolation is adopted thereinto, the resolution can be changed by simple processing with a small calculation load of skipping unnecessary data and picking up necessary data.

As described above, the image processing device <NUM> sequentially outputs the luminance of the aforementioned resolution by selecting from a luminance output means (image input unit <NUM>) the luminance sequentially output with the frequency based on the aforementioned resolution.

In more detail, the high-resolution image processing line (from the three lines buffer 25a to the buffer 28a) select and output (the luminance of) the pixels according to the frequency (every time since all the luminance are selected) based on the high resolution. In the medium-resolution image processing line (from the medium-resolution unit 24b to the buffer 28b), the medium-resolution unit 24b selects and outputs (the luminance of) the pixels according to the frequency (at every second frequency) based on the medium resolution. In the low-resolution image processing line (from the low-resolution unit 24c to the buffer 28c), the low-resolution unit 24c selects and outputs (the luminance of) the pixels according to the frequency (at every fourth frequency) based on the low resolution.

These processing lines output the gradient direction in each resolution using the luminance data.

<FIG> is a Receiver Operating Characteristic (ROC) curve diagram showing experimental results of an identification rate in the case of using the nearest neighbor interpolation and an identification rate in the case of using the bilinear interpolation.

The vertical axis and the horizontal axis are respectively a recall rate and an erroneous detection rate, and indicate that the larger the area under the curve, the larger the identification rate.

As shown in the drawing, the identification rate in the case of using the nearest neighbor interpolation shows overwhelmingly effective performance more than the identification rate in the case of using the bilinear interpolation. This is probably because the nearest neighbor interpolation has sharper edges than the bilinear interpolation, and thus the accuracy has improved.

Thus, the nearest neighbor interpolation is suitable for hardware implementation since it is simple to process and also greatly improves the identification rate.

Returning to <FIG>, the three lines buffer 25a is a circuit configured to store the luminance data of the high-resolution image, and to outputs the stored luminance data arranges for three rows in parallel to the gradient direction calculation unit 26a.

The gradient direction calculation unit 26a is a circuit configured to output gradient direction data indicating the luminance gradient direction of the pixel of interest in the high-resolution image using the luminance data for three rows.

The three lines buffer 25b is a circuit configured to store the luminance data of the medium-resolution image, and to outputs the stored luminance data arranges for three rows in parallel to the gradient direction calculation unit 26b.

The gradient direction calculation unit 26b is a circuit configured to output gradient direction data indicating the luminance gradient direction of the pixel of interest in the medium-resolution image using the luminance data for three rows.

The three lines buffer 25c is a circuit configured to store the luminance data of the low-resolution image, and to outputs the stored luminance data arranges for three rows in parallel to the gradient direction calculation unit 26c.

The gradient direction calculation unit 26c is a circuit configured to output gradient direction data indicating the luminance gradient direction of the pixel of interest in the low-resolution image using the luminance data for three rows.

<FIG> are drawings for describing a detailed operation of the three lines buffer 25a and the gradient direction calculation unit 26a.

As previously described with reference to <FIG>, the luminance data of image <NUM> of the high-resolution image is output as (<NUM>-<NUM>), (<NUM>-<NUM>),. , from the image input unit <NUM>.

As shown in <FIG>, the three lines buffer 25a stores the luminance data for three rows for each row, outputs the three rows of data in parallel to the gradient direction calculation unit 26a.

In the example of <FIG>, the luminance data (<NUM>-<NUM>), (<NUM>-<NUM>), (<NUM>-<NUM>),. of the second line of the image <NUM>, the luminance data (<NUM>-<NUM>), (<NUM>-<NUM>), (<NUM>-<NUM>),. of the third line thereof, and the luminance data (<NUM>-<NUM>), (<NUM>-<NUM>), (<NUM>-<NUM>),. of the fourth line thereof are output in parallel to the gradient direction calculation unit 26a so as to align the pixel columns.

The gradient direction calculation unit 26a receives the input of three rows of luminance data which is output in parallel, and outputs quantized luminance gradient direction.

As shown in the drawing, the gradient direction calculation unit 26a includes an array of storage elements of three rows and three columns, and acquires luminance data of three rows and three columns in synchronization with the output of the three lines buffer 25a to read the luminance based on the luminance data.

As shown in the drawing, the gradient direction calculation unit <NUM> sets centered luminance data as the pixel of interest among the luminance data of three rows and three columns. In the example shown in the drawing, the luminance data (<NUM>-<NUM>) enclosed with the bold rectangle is the luminance data of the pixel of interest.

Then, the gradient direction calculation unit 26a calculates luminance gradient intensity fx(x, y) in the horizontal direction from the luminance difference between the luminance data (<NUM>-<NUM>) and the luminance data (<NUM>-<NUM>) adjacent to the pixel of interest in the horizontal direction, and calculates luminance gradient intensity fy(x, y) in the vertical direction from the luminance difference between the luminance data (<NUM>-<NUM>) and the luminance data (<NUM>-<NUM>) adjacent to the pixel of interest in the vertical direction.

The gradient direction calculation unit 26a calculates m(x, y) by substituting the obtained fx(x, y) and fy(x, y) into the equation (<NUM>) shown in <FIG>, and performs predetermined processing if the calculated m(x, y) is not reached at a threshold value.

If m(x, y) is reached at the threshold value, the gradient direction data (<NUM>-<NUM>) representing the quantized luminance gradient direction of the aforementioned pixel is output with reference to fx(x, y) and fy(x, y) with the correspondence table.

As described above, the gradient direction data is generated corresponding to the pixel similarly to the luminance data.

In the next clock, the column of the luminance data moves by one, and the next luminance data (<NUM>-<NUM>) becomes the pixel of interest, and the gradient direction data (<NUM>-<NUM>) is output, as shown in the gradient direction calculation unit 26a2.

As described above, the gradient direction calculation unit 26a sequentially outputs the gradient direction data for every clock.

When reaching the last column, the gradient direction calculation unit 26a proceed the row by one, and outputs the gradient direction data using the luminance data of the next row as a pixel of interest.

Similarly, the three lines buffer 25b and the gradient direction calculation unit 26b output gradient direction data of the medium-resolution image, and the three lines buffer 25c and the gradient direction calculation unit 26c output gradient direction data of the low-resolution image.

Thus, the position of a pixel of interest and the position of the adjacent pixels adjacent thereto are associated with the array of the storage elements of three rows and three columns arranged in the gradient direction calculation unit 26a, and the luminance data of the pixel of interest and the luminance data of the adjacent pixels are sequentially specified using the correspondence of the positions, in the luminance data sequentially transmitted in accordance with the clock.

The order of passing through the position of the pixel of interest and the position of the adjacent pixels is determined by the order to which the image input unit <NUM> outputs the luminance data.

This configuration is logically the same as a configuration in which the luminance data of the pixel of interest and the luminance data of the adjacent pixels are specified by observing a window for the pixel of interest and a window for the adjacent pixels provided on a path of the luminance data sequentially transmitted in a bucket brigade manner.

In image processing device <NUM>, since the processing of the edge and the circuit configuration are simplified, the luminance data is buffered for three rows and sequentially sent to the array of three rows and three columns, but this is merely an example. Various modification for specifying the luminance data of the pixel of interest and the adjacent pixels based on the order output by the image input unit <NUM> can be adopted.

Similarly, the gradient direction calculation units 26b and 26c respectively specify the luminance data of the pixel of interest and the adjacent pixels based on the order in which the medium-resolution unit 24b and the low-resolution unit 24c performed downsampling of the image <NUM> to output the luminance data.

Thus, the gradient direction output means for each resolution specifies the luminance of the adjacent pixels adjacent in the horizontal direction and the vertical direction of the pixel of interest based on the output order of the luminance of the aforementioned resolution, and outputs the gradient direction of the pixel of interest using the luminance of the specified aforementioned adjacent pixels.

Moreover, the gradient direction output means for each resolution specifies the luminance of the adjacent pixels by arranging the luminance in the array in which the positions of the adjacent pixels are associated with each other based on the output order of the luminance of the resolution.

The array is composed by three arrays consisting of a pixel row belonging to the pixel of interest and two vertical adjacent pixel rows. The gradient direction output means for each resolution arranges the luminances of the three pixel rows in three corresponding arrays and specifies the luminance of the adjacent pixels on the basis of the position where the luminance is arranged.

Returning to <FIG>, a vertical doubling unit 27b and a vertical quadruple unit 27c are respectively circuits configured to extend the gradient direction data for vertical direction twice (2x magnification) and four times (4x magnification) with regard to the vertical direction for the medium resolution image and the low resolution image.

This processing is for adjusting the timing at the time of reading later the co-occurrence by the co-occurrence-matrix creation unit 30a or the like.

<FIG> is a drawing for describing data extension processing in a vertical direction.

The data <NUM> shows a configuration of the gradient direction data before data extension. Each grid shows each gradient direction data, and the gradient direction data is arranged in order of the corresponding pixels.

When each row of the data <NUM> is duplicated to be arranged adjacent to the original row, the data <NUM> expanded twice in the vertical direction and the data <NUM> expanded four times are obtained.

According to this method, the vertical doubling unit 27b duplicates the gradient direction data of the medium-resolution image output from the gradient direction calculation unit 26b for every row, and extends the duplicated data twice in the lengthwise direction (vertical direction).

Moreover, the vertical quadruple unit 27c duplicates the gradient direction data of the low-resolution image output from the gradient direction calculation unit 26c for each row and extends the duplicated data by four times in the lengthwise direction.

Returning to <FIG>, the buffers 28a, 28b, and 28c are respectively buffers configured to temporarily store the gradient direction data of the high-resolution image, the gradient direction data of the medium-resolution image, and the gradient direction data of the low-resolution image output respectively from the gradient direction calculation unit 26a, the vertical doubling unit 27b, and the vertical quadruple unit 27c.

A timing controller <NUM> is a control circuit configured to control the timing of sending out the gradient direction data of the high-resolution image, the medium-resolution image, and the low-resolution image to the co-occurrence-matrix creation units 30a, 30b, and 30c.

The timing controller <NUM> waits until the gradient direction data of each of the resolution images is ready in the buffers 28a, 28b, and 28c, and outputs the gradient direction data when all the gradient direction data is ready.

Thereby, the output timing for every resolution image which has deviated by resolution change can be aligned.

The gradient direction data of the high-resolution image is output from the wiring shown by the thick line of the drawing, the gradient direction data of the medium-resolution image is output from the wiring shown by the thin line of the drawing, and the gradient direction data of the low-resolution image is output from the wiring shown by the dotted line of the drawing. Thus, the gradient direction data of each of the resolution images respectively is output from different wirings.

These wirings are respectively connected with the co-occurrence-matrix creation units 30a, 30b, and 30c, and thereby the gradient direction data for each resolution image is transmitted to the co-occurrence-matrix creation units 30a, 30b, and 30c.

Furthermore, the timing controller <NUM> extends the gradient direction data of the medium-resolution image and the low-resolution image by twice and four times in the horizontal (lateral) direction, respectively, in order to match the timing when the co-occurrence-matrix creation units 30a, 30b, and 30c take the co-occurrence.

<FIG> is a drawing for describing data extension processing in a horizontal direction.

Data sequences <NUM>, <NUM>, and <NUM> respectively show the timing when the timing controller <NUM> outputs the gradient direction data of the high-resolution image, the medium-resolution image, and the low-resolution image.

For example, as shown in the data sequence <NUM>, the timing controller <NUM> sequentially outputs the gradient direction data of the high-resolution image from the first data to the thirtieth data one by one.

On the other hand, for the gradient direction data of the medium-resolution image, as shown in the data sequence <NUM>, the first data is output once, the second to fifteenth data is respectively output twice each, and the sixteenth data is output once, in accordance with the output timing of the high-resolution image.

Moreover, for the gradient direction data of the low-resolution image, as shown in the data sequence <NUM>, the first data is output three times, the second to seventh data is respectively output four times each, and the eighth data is output three times, in accordance with the output timing of the high-resolution image.

It is to be noted that the reason why the numbers of outputs at the beginning and end of the data sequence <NUM> and the data sequence <NUM> respectively are not twice each and four times each is to adjust each width thereof to the same width as the data sequence <NUM>.

Consequently, the gradient direction data of the medium-resolution image and the gradient direction data of the low-resolution image are respectively extended twice and four times in the horizontal direction.

Returning to <FIG>, each of the co-occurrence-matrix creation units 30a, 30b, and 30c is a circuit configured to create a co-occurrence matrix by voting according to the co-occurrence using the gradient direction data output from the timing controller <NUM>.

The co-occurrence-matrix creation units 30a, 30b, and 30c respectively create the co-occurrence matrixes in which pixels of the high-resolution image, the medium-resolution image, and the low-resolution image as a pixel of interest.

The histogram creating unit <NUM> is a circuit configured to create the MRCoHOG feature amount from the co-occurrence matrix output from the co-occurrence-matrix creation units 30a, 30b, and 30c.

When the image processing device <NUM> is formed into an IC chip, the histogram creating unit <NUM> may be configured as an external circuit without being included in the image processing device <NUM> so that the histogram creating unit <NUM> may be connected to the IC chip.

Thereby, it is possible to realize more flexible operation, such as selecting the co-occurrence matrixes output from the co-occurrence-matrix creation units 30a, 30b, and 30c, and general-purpose properties can be improved.

<FIG> is a drawing for describing a scheme in which the co-occurrence-matrix creation unit 30a calculates the co-occurrence matrix.

The co-occurrence-matrix creation unit 30a includes a two lines buffer 61a for high-resolution images, a two lines buffer 61b for medium-resolution images, and a two lines buffer 61c for low-resolution images, each stores the gradient data transmitted from the timing controller <NUM> over two rows for each resolution.

The assignments of the gradient direction data stored in the two lines buffers 61a, 61b, and 61c is shown respectively at the right-hand side of the two lines buffers 61a, 61b, and 61c.

The reference signs indicating the positions of gradient direction data respectively corresponded to the reference signs of the positions shown in <FIG> (the gradient directions do not correspond thereto). The gradient direction data corresponding to the pixel of interest is surrounded by the bold rectangle, and the gradient direction data of the partner pixel to be combined with this gradient direction data for voting is surrounded by the white round mark.

As shown in the drawing, the gradient direction data of the high-resolution image, the gradient direction data of the medium-resolution image, and the gradient direction data of the low-resolution image for two rows and three columns are respectively arranged in the two lines buffers 61a, 61b, and 61c.

In order to arrange the data in the order of the luminance data output by the image input unit <NUM>, the arrangement in the two lines buffers 61a, 61b, and 61c is opposite to the arrangement of <FIG> in the left and right directions.

A co-occurrence-matrix storage unit <NUM> is a circuit configured to create the co-occurrence matrix for the pixel of interest <NUM> by receiving the voting by the co-occurrence and incrementing the frequency (number of the votes) of the co-occurrence matrix.

First, the co-occurrence-matrix creation unit 30a votes for the co-occurrence-matrix storage unit <NUM> on the basis of a combination of the gradient direction data of the pixel of interest <NUM>, and the gradient direction data of the pixels 1a to 1d.

Furthermore, the co-occurrence-matrix creation unit 30a votes for the co-occurrence-matrix storage unit <NUM> on the basis of a combination of the gradient direction data of the pixel of interest <NUM>, and the gradient direction data of the pixels 2a to 2d, and votes for the co-occurrence-matrix storage unit <NUM> on the basis of a combination of the gradient direction data of the pixel of interest <NUM>, and the gradient direction data of the pixels 3a to 3d.

When voting of the aforementioned pixel of interest <NUM> is completed, the co-occurrence-matrix creation unit 30a outputs it to the histogram creating unit <NUM>, resets the number of the votes of the co-occurrence matrix to <NUM>, and advances one column of the gradient direction data stored in the two lines buffers 61a, 61b, and 61c.

Consequently, the co-occurrence-matrix creation unit 30a arranges the gradient direction data corresponding to the pixel 1a in the position of the pixel of interest <NUM>, and performs voting using this for the co-occurrence-matrix storage unit <NUM>.

By repeating the above-described operation, the co-occurrence-matrix creation unit 30a completes the co-occurrence matrix for each pixel of the high-resolution image in the co-occurrence-matrix storage unit <NUM>, and outputs the completed co-occurrence matrix to the histogram creating unit <NUM>.

The output histogram based on the co-occurrence matrix is coupled to the histogram creating unit <NUM>, and becomes a MRCoHOG feature amount when the pixel of the high-resolution image is made as the pixel of interest.

Returning to <FIG>, similarly to the co-occurrence-matrix creation unit 30a, the co-occurrence-matrix creation units 30b and 30c also respectively output the co-occurrence matrix when the pixel of the medium-resolution image is made as the pixel of interest and the co-occurrence matrix when the pixel of the low-resolution image is made as the pixel of interest.

Consequently, the MRCoHOG feature amount when the pixel of the medium-resolution image is made as the pixel of interest and the MRCoHOG feature amount when the pixel of the low-resolution image is made as the pixel of interest are obtained, and the histogram creating unit <NUM> couples the three MRCoHOG feature amounts of the high, middle, and low images to one another to complete the MRCoHOG feature amounts.

The image processing device <NUM> is configured as described above, and each circuit simultaneously operates in synchronization with the clock to sequentially perform operation in an assembly-line method.

In this manner, the image output from the moving image camera can be processed in real time.

As described above, by sequentially combining the gradient direction for each resolution to be sequentially output, the co-occurrence-matrix creation units 30a, 30b, and 30c are functioned as a co-occurrence-matrix creation means for creating the co-occurrence matrix based on the co-occurrence of the gradient direction between different resolutions, and also a co-occurrence-matrix output means for outputting the created aforementioned co-occurrence matrix as image feature amount of the aforementioned image.

Moreover, the co-occurrence matrix creation units 30a, 30b, and 30c respectively arrange the gradient direction data in the two lines buffers 61a, 61b, and 61c in the order in which the gradient direction data is output, and thereby, in order to specify the combination for taking the co-occurrence, sequentially specify the gradient direction of the pixel of interest, and the gradient direction of the pixels to be combined with the aforementioned pixel of interest on the basis of the output order for each resolution of the gradient direction to be sequentially output from the gradient direction output means, and create the co-occurrence matrix by sequentially voting for the co-occurrence matrix on the basis of the combination of the specified aforementioned gradient directions.

Moreover, since the two lines buffers 61a, 61b, and 61c are functioned as the array for specifying the gradient direction data to be the object of the co-occurrence, the co-occurrence matrix creation units 30a, 30b, and 30c arrange the gradient direction for each resolution based on the output order for each resolution in the array in which the pixel of interest and the position of the pixel to be combined with the aforementioned pixel of interest are associated to be provided (divided) for each resolution, and thereby, the gradient direction to be combined as the object of the co-occurrence is specified.

Moreover, the aforementioned array is composed of six arrays corresponding to two pixel rows adjacent to in the vertical direction with regard to each resolution (total of six, two buffers for high-resolution images, two buffers for medium-resolution images, and two buffers for low-resolution images), and each of the co-occurrence matrix creation units 30a, 30b, and 30c arranges the gradient direction of the two pixel rows of each resolution in two corresponding arrays, and specifies the gradient direction to be combined on the basis of the position where the gradient direction is arranged.

<FIG> is a flow chart for describing an image processing procedure performed by the image processing device <NUM>.

First, the image input unit <NUM> outputs luminance data of the image <NUM>, and the medium-resolution unit 24b and the low-resolution unit 24c respectively output luminance data of which the resolutions are converted into the medium resolution and the low resolution (Step <NUM>).

Moreover, the three lines buffers 25a, 25b, and 25c respectively buffer the luminance data of the high-resolution image, the medium-resolution image, and the low-resolution image for three rows (Step <NUM>).

Moreover, the gradient direction calculation units 26a, 26b, and 26c respectively calculate the gradient directions of the pixels of the high-resolution image, the medium-resolution image, and the low-resolution image to output the gradient direction data (Step <NUM>).

The processings of Steps <NUM>, <NUM>, and <NUM> are simultaneously performed in parallel.

Next, the vertical doubling unit 27b and the vertical quadruple unit 27c respectively extend the gradient direction data of the medium-resolution image and the low-resolution image twice and four times in the vertical direction (Step <NUM>).

The gradient direction data of the high-resolution image, the gradient direction data of the medium-resolution image extended vertical twice, and the gradient direction data of the low-resolution image extended vertical for times are respectively buffered in the buffers 28a, 28b, and 28c.

Next, the timing controller <NUM> outputs the gradient direction data of each resolution at the same timing.

At this time, the timing controller <NUM> respectively extends and outputs the gradient direction data of the medium-resolution image and the low-resolution image twice and four times in the horizontal direction (Step <NUM>).

The co-occurrence matrix creation units 30a, 30b, and 30c calculate the element of a co-occurrence matrix using the gradient direction data of each resolution output from the timing controller <NUM>, and create the co-occurrence matrix (Step <NUM>).

Furthermore, the histogram creating unit <NUM> creates a histogram from the created co-occurrence matrix, and outputs the created histogram as a MRCoHOG feature amount (Step <NUM>).

As described above, the operation of each circuit has been described individually. However, each circuit simultaneously operates in synchronization with the clock, and simultaneously performs a flow operation in which data flowing from the left is sequentially (successively) processed and flown to the right.

<FIG> is a drawing showing an example of constituting a semiconductor device using the image processing device <NUM>.

The semiconductor device <NUM> is composed of an IC chip, for example, and internally includes a processor, a RAM, a MRCoHOG accelerator <NUM>, an affine accelerator, a histogram accelerator, a video input interface <NUM>, a video output interface <NUM>, an input/output interface <NUM>, and the like.

The MRCoHOG accelerator <NUM> incorporates the circuit configuration of the image processing device <NUM> and is configured to generate and output a co-occurrence matrix from an image. Alternatively, the MRCoHOG accelerator <NUM> may be configured to create the histogram and extract the MRCoHOG feature amount.

The semiconductor device <NUM> receives an input of moving image data from the video input interface <NUM>, extracts the MRCoHOG feature amount of each frame image by means of the MRCoHOG accelerator <NUM> or the like, and can recognize the object image by the processor using this MRCoHOG feature amount.

Claim 1:
An information processing device comprising:
a feature amount acquiring means (<NUM>) configured to acquire a feature amount of identification object data;
a selection means (<NUM>) configured to select a feature amount of a portion, used for identification, specified in advance from the feature amount acquired by the feature amount acquiring means;
a duplication means (<NUM>) configured to duplicate the feature amount of the portion selected by the selection means;
an identification means (<NUM>), comprising a binary neural network (<NUM>), configured to have learned an identification object using multiple-valued weighting;
an input means configured to input the feature amount of the portion selected by the selection means and the feature amount of the portion duplicated by the duplication means (<NUM>) into the identification means (<NUM>); and
an output means configured to output an identification result of being identified by the identification means (<NUM>) using the feature amounts input by the input means.