Information processing device that implements image processing, and image processing method

The present invention provides an information processing device in which a degradation process of an input image is accurately estimated and a dictionary necessary for generating a desired restored image from the input image can be obtained. The information processing device is provided with: an image acquisition means that acquires a plurality of study images and an input image; and an estimation means that, on the basis of similarity between an arbitrary region of the input image and each of a plurality of degradation images in a case where regions of the study images corresponding to the arbitrary region are degraded on the basis of each of the plurality of degradation processes, outputs an estimated degradation process corresponding to the degradation process corresponding to the region of the input image.

CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

This application is a National Stage Entry of International Application No. PCT/JP2014/003824, filed Jul. 18, 2014, which claims priority from Japanese Patent Application No. 2013-168793, filed Aug. 15, 2013. The entire contents of the above-referenced applications are expressly incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to an image processing technology, and in particular to a technology to generate a dictionary that is used in study based super-resolution processing.

BACKGROUND ART

In relation to image processing, various related technologies have been known.

For example, as an example of a technology to generate a restored image (for example, a high-resolution image) from an input image (for example, a low-resolution image), super-resolution technology is known. Among the super-resolution technologies, a technology to generate a high-resolution image using a dictionary is, in particular, referred to as a study based super-resolution technology. The dictionary mentioned above is a dictionary that are created through studying cases each of which includes a study image (in general, a high-quality image) and a degraded image corresponding to the study image (for example, an image created by reducing the resolution of the study image). The restored image generated by the super-resolution technology is also referred to as a super-resolution image.

PTL 1 discloses an example of a character recognition device. The character recognition device disclosed in PTL 1 performs super-resolution processing to recognize characters on a license plate or the like, which are included in an object image taken with a camera.

The character recognition device performs the super-resolution processing by using a database (dictionary) in which low-resolution dictionary images, feature values of the low-resolution dictionary images, and high-resolution dictionary images are associated with one another. The low-resolution dictionary images mentioned above are images of characters that have been taken with the camera with which the object image is taken. The feature values are feature values that are calculated on the basis of respective ones of the low-resolution dictionary images. The high-resolution dictionary images are images of characters that have been taken with a camera that has a higher resolution compared with the camera with which the object image is taken.

PTL 2 discloses an example of a super-resolution image processing device. The super-resolution image processing device disclosed in PTL 2 outputs a high-resolution image from a low-resolution original image (input image data).

The super-resolution image processing device uses a dictionary table and others, which have been generated by a dictionary creation device, to infer lost high frequency components in generating output image data through applying super-resolution image processing to the input image data. The dictionary table and others mentioned above are a dictionary table, a first principal component basis vector, and a second principal component basis vector. The dictionary creation device generates the dictionary table and others that are optimized for a specific scene by the following procedure.

First, the dictionary creation device acquires a sectioned bitmap, which is a processing object, from a sample image file, breaks down the acquired bitmap into a plurality of broken bitmaps, and stores the broken bitmaps in records in a temporary table.

Next, the dictionary creation device applies MP (Max-Plus) wavelet transformation processing, permutation processing, principal component analysis processing, inner product operation processing, and frequency partition processing to the broken bitmaps in order, and stores results of the processing in respective fields in the temporary table. In the principal component analysis processing, the dictionary creation device calculates the first principal component basis vector and the second principal component basis vector.

Last, the dictionary creation device creates the dictionary table, which has a smaller number of records compared with the temporary table, using a mean value operation unit. The dictionary table differs from the dictionary of the above-described study based super-resolution technology. That is, the dictionary table is not a dictionary that is created through studying cases in which study images are associated with degraded images.

PTL 3 discloses an example of an image super-resolution device. The image super-resolution device disclosed in PTL 3 generates a super-resolution image that is enlarged with a preset enlargement ratio from an input image degraded due to encoding and decoding. The encoding and decoding mentioned above are encoding and decoding by a preset encoding method. Specifically, the image super-resolution device generates a super-resolution image through the following processing.

First, the image super-resolution device partitions an input image into blocks of a prefixed size, and cuts out respective ones of the blocks as processing blocks. Next, the image super-resolution device generates enlarged processing blocks by enlarging the processing blocks with a prefixed enlargement ratio. The prefixed enlargement ratio is an enlargement ratio with which the image super-resolution device enlarges the input image when the image super-resolution device generates the super-resolution image.

Second, the image super-resolution device writes reference blocks and degraded reference blocks in association with each other in a block storage means. The reference blocks mentioned above are blocks that are cut out from the input image and have the same size as that of the processing blocks. The degraded reference blocks mentioned above are blocks into which the reference blocks are degraded by a specific degradation process. The specific degradation process is a degradation process when it is assumed that the input image is an image into which the to-be-generated super-resolution image has been degraded through the degradation process. Specifically, the image super-resolution device degrades the reference blocks using a degradation model based on an encoding method by which the input image has been degraded (a model that simulates predefined orthogonal transformation, quantization, and so on) to generate the degraded reference blocks.

Third, the image super-resolution device calculates similarities between the degraded reference blocks and the processing blocks.

Fourth, the image super-resolution device enlarges the degraded reference blocks with the prefixed enlargement ratio to generate restored reference blocks. Further, the image super-resolution device calculates differences between the restored reference blocks and the reference blocks as loss components.

Fifth, the image super-resolution device combines the enlarged processing blocks with the loss components on the basis of the similarities to generate super-resolution blocks. The image super-resolution device constructs the super-resolution blocks into an image to generate the super-resolution image into which the input image is enlarged.

CITATION LIST

Patent Literature

SUMMARY OF INVENTION

Technical Problem

However, the above-described technologies disclosed in the documents cited in the citation list have a problem in that there is a case in which it is impossible to obtain a dictionary that is required to generate a desired restored image (super-resolution image) from an input image and used in study based super-resolution processing.

That is because accurate estimation of a degradation process applied to an input image is difficult and complicated.

Specifically, the character recognition device disclosed in PTL 1 does not estimate a degradation process of the object image. In the character recognition device, the low-resolution dictionary images (equivalent to degraded images in the dictionary used in the study based super-resolution processing) are images of characters that have been taken with a camera with which the object image is taken. That is, the low-resolution dictionary images included in the database (dictionary) are not images that are obtained by estimating a degradation process of the object image.

The super-resolution image processing device in PTL 2 generates the dictionary table and others by operations using functions or the like on the basis of a sample image file (equivalent to study images in the dictionary used in the study based super-resolution processing). The dictionary table and others are a dictionary optimized for a specific scene but not a dictionary obtained by performing estimation of a degradation process.

The super-resolution processing performed by the image super-resolution device in PTL 3 is super-resolution processing when a degradation process is apparent beforehand. Thus, the image super-resolution device is incapable of processing an input image the degradation process of which is unclear.

Further, it is difficult to estimate an accurate degradation process by a technology like blind de-convolution or the like. The blind de-convolution mentioned above is a method, targeting natural images, to restore an original signal from a measured signal. Further, it is difficult and substantially complicated for a user (operator) to estimate an accurate degradation process on the basis of experience or the like.

An object of the present invention is to provide an information processing device, an image processing method, and a program or a non-transitory computer-readable recording medium recording a program that are capable of solving the above-described problem.

Solution to Problem

An information processing device according to an exemplary aspect of the present invention includes: image acquisition means for acquiring a plurality of first study images and an input image; and estimation means for outputting an estimated degradation process on a basis of first similarities between an arbitrary region in the input image and respective ones of a plurality of first degraded images when regions, in the first study images, corresponding to the region are degraded on a basis of respective ones of a plurality of degradation processes, wherein the estimated degradation process corresponds to a degradation process in the degradation processes, the degradation process being related to the region in the input image.

An image processing method according to an exemplary aspect of the present invention, using a computer implementing the image processing method, includes: acquiring a plurality of first study images and an input image; and outputting an estimated degradation process on a basis of first similarities between an arbitrary region in the input image and respective ones of a plurality of first degraded images when regions, in the first study images, corresponding to the region are degraded on a basis of respective ones of a plurality of degradation processes.

A non-transitory computer-readable recording medium according to an exemplary aspect of the present invention, recording a program that makes a computer execute processing of: acquiring a plurality of first study image and an input image; and outputting an estimated degradation process on a basis of first similarities between an arbitrary region in the input image and respective ones of a plurality of first degraded images when regions, in the first study images, corresponding to the region are degraded on a basis of respective ones of a plurality of degradation processes.

Advantageous Effects of Invention

The present invention has an advantageous effect in that it becomes possible to estimate a degradation process applied to an input image accurately and obtain a dictionary required to generate a desired restored image from the input image.

DESCRIPTION OF EMBODIMENTS

Exemplary embodiments to achieve the present invention will be described in detail with reference to the accompanying drawings. In the respective drawings and the exemplary embodiments described in the description, the same signs are assigned to the same components and descriptions thereof will be omitted appropriately.

First Exemplary Embodiment

FIG. 1is a block diagram illustrating a configuration of a degradation process estimation device (also referred to as an information processing device)100according to a first exemplary embodiment of the present invention.

As illustrated inFIG. 1, the degradation process estimation device100according to the present exemplary embodiment includes an image acquisition unit150and an estimation unit160.

FIG. 2is a block diagram illustrating a configuration of an image processing system101that includes the degradation process estimation device100according to the present exemplary embodiment.

As illustrated inFIG. 2, the image processing system101according to the present exemplary embodiment includes the degradation process estimation device100, a study unit102, a dictionary103, and a restoration unit104. The image processing system101is also referred to as an information processing device.

First, an overall operation of the image processing system101, which includes the degradation process estimation device100according to the present exemplary embodiment, will be described.

The degradation process estimation device100acquires study images411(first study images) and an input image430, which are, for example, input from the outside. The study images411are high-resolution images (high-quality images) that may correspond to the input image430and have been prepared in advance. The input image430is an image that is an object of restoration. In general, the input image430is a low-quality image, such as a low-resolution image.

The degradation process estimation device100outputs an estimated degradation process867to the study unit102on the basis of the study images411and the input image430. The estimated degradation process867is information of a degradation process of an image (information indicating degradation details of an image), which the study unit102uses to generate the dictionary103. The dictionary103is the dictionary103required for the restoration unit104to generate a desired restored image440from the input image430.

The study unit102acquires study images410(second study images), which are, for example, input from the outside, and the estimated degradation process867, which is input from the degradation process estimation device100. The study unit102generates the dictionary103on the basis of the study images410and the estimated degradation process867. The study images410are high-resolution images (high-quality images) that may correspond to the input image430and have been prepared in advance. A set of study images410and a set of study images411may overlap completely, overlap partially, or not overlap at all.

Specifically, first, on the basis of the estimated degradation process867, the study unit102generates degraded images420(second degraded images, which will be described later and are illustrated inFIG. 20) each of which corresponds to one of the study images410. Second, the study unit102generates the dictionary103that includes a patch in the study images410and a patch in the corresponding degraded images420in pairs. The patch mentioned above is one of small regions into which an image (a study image410, a degraded image420, or the like) is partitioned.

The restoration unit104acquires the input image430, which is input from the outside, and outputs a restored image440to the outside. The restoration unit104generates the restored image440corresponding to the input image430on the basis of entries in the dictionary103.

Next, the respective components (the image acquisition unit150and the estimation unit160) that the degradation process estimation device100in the first exemplary embodiment includes will be described. The components illustrated inFIG. 1may be either components corresponding to hardware units or components the division of which is done in accordance with functional units of a computer device. The components illustrated inFIG. 1will be described herein as components the division of which is done in accordance with functional units of a computer device.

FIG. 3is a diagram illustrating an example of correspondence information860, which is stored in a not-illustrated storage means in the degradation process estimation device100. As illustrated inFIG. 3, each record in the correspondence information860includes a degradation process861and a feature vector862(a first feature vector, which will be described late and is illustrated inFIG. 4). The correspondence information860illustrated inFIG. 3is used in after-mentioned processing that is performed by the estimation unit160in the degradation process estimation device100.

Each of the degradation processes861illustrated inFIG. 3is information that discriminates a degradation process (degradation details) of an image. Each of the degradation processes of an image is an arbitrary combination of, for example, intensity of blur, a compression ratio, luminance, a field in interlacing, and the like. For example, a degradation process861expresses degradation details in respective ones of intensity of blur, a compression ratio, luminance, and a field in interlacing as “B3”, “C2”, “L3”, and “F1”, respectively. In addition, posture, noise, or the like may be added to the degradation process.

Each feature vector862is, for example, a vector that has the absolute values of Fourier transformed quantities of a degraded image421(a first degraded image, which will be described later and is illustratedFIG. 4) or the logarithmic values of the absolute values as elements, which are arranged in raster scan order. Each feature vector862may have the absolute values of transformed quantities of a degraded image421by, without being limited to a Fourier transformation, an integral transform, such as a Laplace transform, that focuses on the frequency part, or the logarithmic values of the absolute values as elements. Each feature vector862may also be a vector that has respective pixel values of a degraded image421, which are normalized with respect to luminance, as elements, which are arranged in raster scan order.

The image acquisition unit150acquires a plurality of study images411and an input image430.

The estimation unit160outputs an estimated degradation process867on the basis of similarities851(first similarities, which will be described later and are illustrated inFIG. 4) between an arbitrary region in the input image430and respective ones of a plurality of degraded images421. The estimated degradation process867mentioned above corresponds to a degradation process861that corresponds to the region in the input image430.

The arbitrary region mentioned above is either an arbitrary local region in the image or the whole region of the image. That is, the arbitrary region in the input image430is an arbitrary partial image of the input image430or the whole of the input image430. The present exemplary embodiment is an exemplary embodiment when the arbitrary region is the whole of an image. The case in which an arbitrary region is a local region will be described in a second exemplary embodiment. Thus, since, in the present exemplary embodiment, “an arbitrary region in the input image430” is the whole of the input image430, that is, the input image430itself, hereinafter, “the whole of the input image430as an arbitrary region in the input image430” will be simply referred to as “input image430”.

The degraded images421are images when the study images411are degraded on the basis of respective ones of the plurality of degradation processes861.

The similarities851correspond to, for example, relations between the feature vectors862corresponding to the degraded images421and a feature vector864(a second feature vector, which will be described later and is illustrated inFIG. 4) corresponding to the input image430.

Each relation is, for example, a value based on a distance between two vectors (in the present exemplary embodiment, a feature vector862and the feature vector864). The relation may also be a value based on an angle between two feature vectors. Further, the relation may also be a value calculated by a normalized cross-correlation function, and is not limited to these values. The feature vector864is a vector that has the same structure as the feature vectors862.

FIG. 4is a diagram illustrating relations among the above-described study image411, degraded image421, input image430, feature vector862, feature vector864, and similarity851.

The estimated degradation process867is, for example, a degradation process861corresponding to the input image430. That is, the estimation unit160outputs a degradation process861corresponding to the input image430as the estimated degradation process867on the basis of the similarities851and the correspondence information860.

In this case, the correspondence information860indicates correspondence relations between the respective feature vectors862of the degraded images421and degradation details from the study images411(first study images) to respective ones of the degraded images421. In other words, the correspondence information860indicates relations between the degraded images421and the degradation processes861.

As described above, a degradation process861is discrimination information that specifies a degradation process from a study image411to a degraded image421. The degradation process861related to the input image430, that is, the estimated degradation process867, is a degradation process861that indicates degradation details in degradation processing when it is assumed that the input image430is an image created by applying the degradation processing to a specific image. The specific image is a restored image (also referred to as a super-resolution image)440that the restoration unit104generates.

In other words, the estimated degradation process867indicates degradation details in a degraded image421with respect to a study image411. At the same time, the estimated degradation process867indicates degradation details in a degraded image420with respect to a study image410. That is because both the study images410and the study images411are high-resolution images (high-quality images) that may correspond to the input image430.

The estimated degradation process867may be information from which the degradation process estimation device100and the study unit102are able to discriminate degradation details therein in synchronization with each other. For example, the estimated degradation process867may be information that specifically indicates degradation details therein as the degradation processes861, or may be a serial number.

As illustrated inFIG. 2, the estimation unit160includes, for example, a correspondence information generation unit110and a degradation process selection unit120.

The correspondence information generation unit110generates the correspondence information860.

For example, the correspondence information generation unit110generates the degraded images421from the study images411on the basis of the degradation processes861. For example, each of the degradation processes861indicates typical degradation details selected on the basis of, for example, empirical knowledge from among all degradation details that may correspond to the study images410. Next, the correspondence information generation unit110generates the feature vectors862of the generated degraded images421.

The number of study images411and the number of degradation processes861are arbitrary. The number of degraded images421is the number of study images411multiplied by the number of degradation processes861. For example, when the number of study images411is 10000 and the number of types of degradation processes861is 100, the number of degraded images421is 1000000.

In this case, the number of feature vectors862is 1000000, which is the same as the number of degraded images421. In other words, the correspondence information860includes 1000000 records, each of which includes a pair of a feature vector862and a degradation process861.

The degradation process selection unit120selects a degradation process861from the correspondence information860on the basis of the similarities851between the input image430and the degraded images421, and outputs the selected degradation process861as the estimated degradation process867.

Specifically, the degradation process selection unit120selects a degradation process861from the correspondence information860on the basis of relations between the feature vector864related to the input image430and the feature vectors862included in the correspondence information860. Next, the degradation process selection unit120outputs the selected degradation process861to the outside (for example, the study unit102) as the estimated degradation process867.

FIG. 5is a diagram for explaining selection of a degradation process861by the degradation process selection unit120.

It is assumed herein that feature vectors (feature vectors862and a feature vector864) are three-dimensional vectors. InFIG. 5, respective ones of dotted lines indicate respective axes in a three-dimensional space where the feature vectors exist.

InFIG. 5, squares, triangles, and circles indicate feature vectors862belonging to a square class, a triangle class, and a circle class, respectively. The classes mentioned above are related to types of degradation processes861. That is, the square class, the triangle class, and the circle class are related to respective ones of three types of degradation processes861.

InFIG. 5, a star shape indicates the feature vector864(the feature vector related to the input image430).

The degradation process selection unit120classifies the feature vector864(star shape) into any one of the square class, the triangle class, and the circle class. For example, the degradation process selection unit120classifies the feature vector864into any one of the classes on the basis of relations between the centroids of the feature vectors862in the respective classes and the feature vector864. For example, the degradation process selection unit120may classify the feature vector864into the class to which a feature vector862having the shortest distance from the feature vector864belongs.

Next, the degradation process selection unit120selects a degradation process861corresponding to the class into which the feature vector864(star shape) has been classified. Subsequently, the degradation process selection unit120outputs the selected degradation process861to the outside (for example, the study unit102) as the estimated degradation process867.

In the above description, the degradation process selection unit120classifies the feature vector864into a nearest neighbor class, and selects only one degradation process861related to the nearest neighbor class. However, the degradation process selection unit120may classify the feature vector864into a plurality of classes. For example, the degradation process selection unit120may classify the feature vector864into k-nearest neighbor (k is an arbitrary natural number equal to or greater than 1) classes. In this case, the degradation process selection unit120may output the degradation processes861that is related to respective ones of the k classes as estimated degradation processes867.

The above is a description of the respective components corresponding to functional units of the degradation process estimation device100.

Next, components corresponding to hardware units of the degradation process estimation device100will be described.

FIG. 6is a diagram illustrating a hardware configuration of a computer700that achieves the degradation process estimation device100in the present exemplary embodiment.

As illustrated inFIG. 6, the computer700includes a CPU (Central Processing Unit)701, a storage unit702, a storage device703, an input unit704, an output unit705, and a communication unit706. Further, the computer700includes a recording medium (or a storage medium)707, which is supplied from the outside. The recording medium707may be a nonvolatile recording medium that stores information non-transitorily.

The CPU701operates an operating system (not illustrated) and controls the whole operation of the computer700. The CPU701also reads a program and data from, for example, the recording medium707, which is mounted on the storage device703, and writes the read program and data in the storage unit702. The program mentioned above is, for example, a program that makes the computer700carry out an operation of a flowchart illustrated inFIG. 7, which will be described later.

The CPU701carries out various processing as the image acquisition unit150and the estimation unit160illustrated inFIG. 1in accordance with the read program and on the basis of the read data.

The CPU701may be configured to download the program and data from an external computer (not illustrated) connected to communication networks (not illustrated) into the storage unit702.

The storage unit702stores the program and data. The storage unit702stores, for example, the study images411, the degraded images421, the input image430, and the correspondence information860.

The storage device703is, for example, an optical disk, a flexible disk, a magneto-optical disk, an external hard disk, or a semiconductor memory, and includes the recording medium707. The storage device703(the recording medium707) stores the program in a computer-readable manner. The storage device703may also store the data. The storage device703stores, for example, the study images411, the degraded images421, the input image430, and the correspondence information860.

The input unit704is achieved by, for example, a mouse, a keyboard, built-in key buttons, or the like, and is used in input operations. The input unit704is not limited to a mouse, a keyboard, and built-in key buttons, and may also be, for example, a touch panel.

The output unit705is achieved by, for example, a display, and is used to confirm an output.

The communication unit706achieves an interface to the outside. The communication unit706is included in the image acquisition unit150and the estimation unit160as portions thereof. The degradation process estimation device100may be connected to the study unit102via the communication unit706.

As described above, the blocks corresponding to functional units of the degradation process estimation device100, illustrated inFIG. 1, are achieved by the computer700, which has the hardware configuration illustrated inFIG. 6. However, means for achieving the respective components included in the computer700are not limited to the above-described components. That is, the computer700may be achieved by a physically-connected single device or by a plurality of physically-separate devices that are interconnected by wires or radio waves.

The recording medium707recording codes of the above-described program may be supplied to the computer700, and the CPU701may read and execute the codes of the program recorded in the recording medium707. Alternatively, the CPU701may store the codes of the program recorded in the recording medium707in the storage unit702or the storage device703or both. That is, the present exemplary embodiment includes an exemplary embodiment of a recording medium707that stores, transitorily or non-transitorily, a program (software) that a computer700(a CPU701) executes. A storage medium that stores information non-transitorily is also referred to as a nonvolatile storage medium.

The computer700may achieve the image processing system101illustrated inFIG. 2. In this case, the CPU701carries out various processing, in accordance with a read program and on the basis of read data, as the degradation process estimation device100, the study unit102, and the restoration unit104, illustrated inFIG. 2. The storage unit702and the storage device703may include the dictionary103. The storage unit702and the storage device703may further store the study images410, the degraded images420, and the restored image440.

The above is a description of the respective components corresponding to hardware units of the computer700that achieves the degradation process estimation device100in the present exemplary embodiment.

Next, an operation of the present exemplary embodiment will be described in detail with reference toFIGS. 1 to 7.

FIG. 7is a flowchart illustrating an operation of the present exemplary embodiment. Processing in accordance with the flowchart may be performed on the basis of the afore-described program control by the CPU701. Step names of the processing are denoted by signs, such as S601.

The image acquisition unit150acquires study images411(S601). For example, the image acquisition unit150reads the study images411that have been stored in the storage unit702or the storage device703, illustrated inFIG. 6, in advance. The image acquisition unit150may acquire the study images411that are input by users through the input unit704illustrated inFIG. 6. The image acquisition unit150may receive the study images411from a not-illustrated device through the communication unit706illustrated inFIG. 6. The image acquisition unit150may acquire the study images411that are recorded in the recording medium707through the storage device703illustrated inFIG. 6.

Next, with respect to each of the acquired study images411, the correspondence information generation unit110in the estimation unit160generates degraded images421each of which is related to one of a plurality of degradation processes861(S602).

In the processing above, the estimation unit160reads degradation processes861, which have been stored in the storage unit702or the storage device703, illustrated inFIG. 6, in advance. The estimation unit160may acquire the degradation processes861that are input by users through the input unit704illustrated inFIG. 6. The estimation unit160may receive the degradation processes861from a not-illustrated device through the communication unit706illustrated inFIG. 6. The estimation unit160may acquire the degradation processes861that are recorded in the recording medium707through the storage device703illustrated inFIG. 6.

Next, the correspondence information generation unit110calculates feature vectors862, which correspond to the respective degraded images421(S603).

Next, the correspondence information generation unit110generates correspondence information860, which includes tuples of a degradation process861and a feature vector862, and outputs the generated correspondence information860to the degradation process selection unit120(S604).

Next, the image acquisition unit150acquires an input image430(S605). For example, the image acquisition unit150acquires the input image430that has been stored in the storage unit702or the storage device703, illustrated inFIG. 6, in advance. The image acquisition unit150may acquire the input image430that is input by a user through the input unit704illustrated inFIG. 6. The image acquisition unit150may receive the input image430from a not-illustrated device through the communication unit706illustrated inFIG. 6. The image acquisition unit150may acquire the input image430that is recorded in the recording medium707through the storage device703illustrated inFIG. 6.

Next, the degradation process selection unit120in the estimation unit160calculates a feature vector864corresponding to the input image430(S606).

Next, the degradation process selection unit120selects a degradation process861related to the input image430from the correspondence information860on the basis of relations between the feature vector864and the feature vectors862included in the correspondence information860(S607).

Next, the degradation process selection unit120outputs the selected degradation process861as an estimated degradation process867(S608). For example, the degradation process selection unit120transmits the estimated degradation process867to the study unit102through the communication unit706illustrated inFIG. 6. The degradation process selection unit120may output the estimated degradation process867through the output unit705illustrated inFIG. 6. The degradation process selection unit120may record the estimated degradation process867in the recording medium707through the storage device703illustrated inFIG. 6.

The above is a description of an operation of the present exemplary embodiment.

Next, a more specific configuration of the present exemplary embodiment will be described.

FIG. 8is a diagram illustrating an example of a detailed configuration of the degradation process estimation device100.

As illustrated inFIG. 8, the image acquisition unit150includes a study image acquisition unit151and an input image acquisition unit152. The estimation unit160includes the correspondence information generation unit110, the degradation process selection unit120, and a degradation process estimation dictionary165. A degraded image group generation unit161, a feature vector calculation unit162, and a degradation process estimation dictionary creation unit163constitute the correspondence information generation unit110. The feature vector calculation unit162and a feature vector classification unit164constitute the degradation process selection unit120.

Further, the degradation process estimation dictionary165corresponds to the correspondence information860. The degradation process estimation dictionary165is stored in, for example, the storage unit702, the storage device703, illustrated inFIG. 6, or the like.

An operation of the degradation process estimation device100having the configuration illustrated inFIG. 8will be described along the flowchart illustrated inFIG. 7.

The study image acquisition unit151acquires study images411(S601).

Next, with respect to each of the study images411that the study image acquisition unit151has acquired, the degraded image group generation unit161, on the basis of a plurality of degradation processes861, generates degraded images421each of which is related to one of the plurality of degradation processes861(S602).

Next, the feature vector calculation unit162calculates feature vectors862that is related to the degraded images421, which the degraded image group generation unit161has generated (S603).

Next, the degradation process estimation dictionary creation unit163generates correspondence information860that includes tuples of a feature vector862and a degradation process861, and writes the generated correspondence information860in the degradation process estimation dictionary165(S604). The feature vectors862mentioned above are the feature vectors862that the feature vector calculation unit162has created in S603. The degradation processes861mentioned above are the degradation processes861that the feature vector calculation unit162has used to create the feature vectors862in S603.

Next, the input image acquisition unit152acquires an input image430(S605).

Next, the feature vector calculation unit162calculates a feature vector864(S606).

Next, the feature vector classification unit164classifies the feature vector864calculated by the feature vector calculation unit162into any one of the afore-described classes on the basis of relations to the feature vectors862included in the correspondence information860, which is stored in the degradation process estimation dictionary165. Subsequently, the feature vector classification unit164selects a degradation process861that is related to the class into which the feature vector864has been classified (S607).

Next, the feature vector classification unit164outputs the selected degradation process861as an estimated degradation process867(S608).

A first advantageous effect in the above-described present exemplary embodiment is that it becomes possible to estimate a degradation process of the input image430accurately and obtain the dictionary103required to restore a desired restored image440from the input image430.

That is because the present exemplary embodiment includes the following configuration. That is, first, the image acquisition unit150acquires the study images411and the input image430. Second, the estimation unit160outputs a selected degradation process861as the estimated degradation process867on the basis of relations between the feature vector864and the feature vectors862.

A second advantageous effect in the above-described present exemplary embodiment is that it becomes possible to estimate a degradation process of the input image430accurately even for degradation including a blur.

That is because the estimation unit160is configured to generate the feature vectors862and the feature vector864that have a structure in which the absolute values of Fourier transformed quantities of an original degraded image421or the logarithmic values of the absolute values are arranged in raster scan order. Alternatively, that is because the estimation unit160is configured to generate the feature vectors862and the feature vector864that have a structure in which the respective pixel values of a degraded image421, which are normalized with respect to luminance, are arranged in raster scan order.

A third advantageous effect in the above-described present exemplary embodiment is that it becomes possible to increase a probability with which an estimated degradation process is accurate to a higher level.

That is because the degradation process selection unit120is configured to classify the feature vector864into a plurality of classes and output the estimated degradation processes867that are related to respective ones of the plurality of classes.

<<<First Variation of First Exemplary Embodiment>>>

FIG. 9is a diagram illustrating an example of a detailed configuration of a degradation process estimation device100in a first variation of the present exemplary embodiment.

As illustrated inFIG. 9, an estimation unit160of the present variation includes a degraded image estimation dictionary creation unit166in place of the degradation process estimation dictionary creation unit163. The estimation unit160of the present variation also includes a feature vector classification unit167in place of the feature vector classification unit164. The estimation unit160of the present variation also includes a degraded image estimation dictionary168in place of the degraded image estimation dictionary165.

The degradation process estimation device100of the present variation outputs a selected degraded image421as an estimated degradation process867in place of a degradation process861. That is, the estimation unit160of the present variation selects any one of degraded images421on the basis of similarities851, and outputs the selected degraded image421as the estimated degradation process867.

The degradation process estimation dictionary creation unit166generates correspondence information that includes tuples of a feature vector862created by a feature vector calculation unit162and a degraded image421related to the feature vector862, and writes the generated correspondence information in the degraded image estimation dictionary168.

The degraded image estimation dictionary168stores the correspondence information including tuples of a feature vector862and a degraded image421.

The feature vector classification unit167, for example, outputs a degraded image421related to a feature vector862that is the nearest neighbor of a feature vector864. The feature vector classification unit167may output k (k is an arbitrary natural number equal to or greater than 1) degraded images421each being related to feature vectors862that are k-nearest neighbors of the feature vector864.

Specifically, the feature vector classification unit167classifies the feature vector864into any one of the afore-described classes on the basis of relations to the feature vectors862that are included in the correspondence information, which is stored in the degraded image estimation dictionary168. Next, the feature vector classification unit167outputs a degraded image421that is related to the class into which the feature vector864has been classified (for example, the centroid of the class). Alternatively, the feature vector classification unit167may classify the feature vector864into k-nearest neighbor classes on the basis of relations to the feature vectors862. In this case, the feature vector classification unit167may output degraded images421each being related to the k classes.

For example, the degradation process estimation device100of the present variation outputs degraded images421with respect to study images411in an overlap of a set of study images411with a set of study images410.

The above-described present variation has an advantageous effect in that generation processing of degraded images420in the study unit102can be omitted.

<<<Second Variation of First Exemplary Embodiment>>>

FIG. 10is a diagram illustrating an example of a detailed configuration of a degradation process estimation device100in a second variation of the present exemplary embodiment.

As illustrated inFIG. 10, the degradation process estimation device100of the present variation further includes a degradation information input unit170.

The degradation information input unit170allows a user to input degradation information of an input image430.

Each ofFIGS. 11 and 12is a diagram illustrating an example of such degradation information.FIG. 11illustrates four vertices of a license plate, which are degradation information, with black dots.FIG. 12illustrates feature points of a face, which are degradation information, with black dots. The degradation information is not limited to the examples illustrated inFIGS. 11 and 12, and may be, for example, information specifying the outline of a specific region.

For example, the degradation information input unit170allows a user to input degradation information of the input image430by the following procedure. First, the degradation information input unit170displays the input image430on the output unit705illustrated inFIG. 6. Second, the user specifies vertex positions or the positions of feature points in the displayed input image430through the input unit704illustrated inFIG. 6. Third, the degradation information input unit170acquires the input positions through the input unit704.

Degradation information that a user inputs is not limited to vertex positions or feature points. For example, a user may, instead of specifying points, specifies degradation information by lines, or specifies degradation information by surfaces or regions.

With respect to each of study images411that a study image acquisition unit151has acquired, a degraded image group generation unit161in an estimation unit160of the present variation, on the basis of not only a plurality of degradation processes861but also the degradation information, generates degraded images421each of which is related to one of the plurality of degradation processes861.

The above-described present variation has an advantageous effect in that it is possible to reduce a load on generation processing of degraded images421in the estimation unit160.

That is, by allowing a user to input information of points to determine a posture as described above, it becomes unnecessary to solve problems of estimating a posture, determining a magnification, detecting an object (for example, a license plate or a face) of super-resolution processing, and so on.

Second Exemplary Embodiment

Next, a second exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, within a range not to obscure the description of the present exemplary embodiment, descriptions of portions overlapping the earlier description will be omitted.

The present exemplary embodiment is an exemplary embodiment when the arbitrary region is a local region. In the following description, an arbitrary region in an input image430will be referred to as an input image local region.

In the present exemplary embodiment, degraded images421are degraded local regions when local regions in study images411are degraded on the basis of respective ones of a plurality of degradation processes861. The local regions in study images411mentioned above correspond to local regions that are the input image local regions.

The degradation processes861of the present exemplary embodiment are degradation processes each of which discriminates a process of degradation from a local region in a study image411to a local region in a degraded image421.

Feature vectors862of the present exemplary embodiment are feature vectors of degraded images421, which are degraded local regions.

A feature vector864of the present exemplary embodiment is a feature vector of an input image local region.

Each of the input image local region, the degraded images421, which are degraded local regions, local regions in an after-mentioned study image410, and local regions in an after-mentioned input image430are local regions that have corresponding positions and shapes. The local regions are local regions specified in, for example, units of patches.

FIG. 13is a block diagram illustrating a configuration of a degradation process estimation device200according to the second exemplary embodiment of the present invention.

As illustrated inFIG. 13, the degradation process estimation device200in the present exemplary embodiment includes an estimation unit260in place of the estimation unit160.

The estimation unit260includes a correspondence information generation unit210and a degradation process selection unit220.

The correspondence information generation unit210generates degraded images421on the basis of a region specification871that includes, for example, an arbitrary number of patch identifiers (identifiers that identify individual patches). Next, the correspondence information generation unit210calculates feature vectors862that are related to respective ones of the degraded images421. The correspondence information generation unit210acquires the region specification871that has been stored in, for example, the storage unit702or the storage device703, illustrated inFIG. 6, in advance. The correspondence information generation unit210may acquire the region specification871that is input by a user through the input unit704illustrated inFIG. 6. The correspondence information generation unit210may receive the region specification871from a not-illustrated device through the communication unit706illustrated inFIG. 6. The correspondence information generation unit210may acquire the region specification871that is recorded in a recording medium707through the storage device703illustrated inFIG. 6.

The degradation process selection unit220generates the input image local regions on the basis of the region specification871that is acquired by the correspondence information generation unit210. Next, the degradation process selection unit220calculates feature vectors864that are related to the input image local regions.

The degradation process selection unit220selects a degradation process861from correspondence information860on the basis of similarities851between the feature vectors864and the feature vectors862included in the correspondence information860.

Next, the degradation process selection unit220outputs the selected degradation process861to the outside (for example, the after-mentioned study unit202) as an estimated degradation process867.

In other words, the degradation process estimation device200outputs an estimated degradation process867that is related to the local region.

FIG. 14is a block diagram illustrating a configuration of an image processing system201that includes the degradation process estimation device200according to the present exemplary embodiment. The image processing system201is also referred to as an information processing device.

As illustrated inFIG. 14, the image processing system101according to the present exemplary embodiment includes the degradation process estimation device200, the study unit202, the dictionary103, and the restoration unit104.

The study unit202acquires study images410, which are, for example, input from the outside, the estimated degradation process867input from the degradation process estimation device200, and the region specification871. The study unit202generates the dictionary103from the study images410on the basis of the estimated degradation process867and the region specification871. Specifically, first, the study unit202generates degraded images420that correspond to the local regions in respective ones of the study images410on the basis of the estimated degradation process867and the region specification871. Second, the study unit202generates the dictionary103that includes patches in the study images410and patches in the corresponding degraded images420in pairs.

A first advantageous effect of the above-described present exemplary embodiment is the same as the advantageous effect of the first exemplary embodiment. Further, a second advantageous effect of the present exemplary embodiment is that, even when degradations with different details occur with respect to each local region in the input image430, it becomes possible to estimate accurate degradation processes861and obtain the dictionary103that is required to restore an accurate super-resolution image (restored image440).

The reason for the second advantageous effect is that the degradation process estimation device200is configured to further output the estimated degradation processes867each of which is related to one of the input image local regions, which are local regions in the input image430, on the basis of the region specification871.

The present exemplary embodiment may be combined with the first exemplary embodiment. That is, the combined exemplary embodiment may have a configuration that includes both a means processing the whole of each image and a means processing respective local regions in each image.

Third Exemplary Embodiment

Next, a third exemplary embodiment of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, within a range not to obscure the description of the present exemplary embodiment, descriptions of portions overlapping the earlier description will be omitted.

FIG. 15is a block diagram illustrating a configuration of a degradation process estimation device300according to the third exemplary embodiment of the present invention.

As illustrated inFIG. 15, the degradation process estimation device300in the present exemplary embodiment further includes a study image selection unit340, compared with the degradation process estimation device100of the first exemplary embodiment.

FIG. 16is a block diagram illustrating a configuration of an image processing system301that includes the degradation process estimation device300according to the present exemplary embodiment. The image processing system301is also referred to as an information processing device.

As illustrated inFIG. 16, the image processing system301according to the present exemplary embodiment includes the degradation process estimation device300, the study unit102, the dictionary103, and the restoration unit104. The degradation process estimation device300includes the image acquisition unit150and an estimation unit360. The estimation unit360further includes the study image selection unit340, compared with the estimation unit160of the first exemplary embodiment.

The study image selection unit340selects a study image411on the basis of similarities (second similarities)852between respective ones of the study images411and a restored image440, and outputs the selected study image411to a correspondence information generation unit110. The restored image440mentioned above is an image generated by the restoration unit104.

Next, with reference toFIGS. 17 and 18, an operation of the study image selection unit340will be described.FIG. 17is a diagram illustrating relations among a study image411, a restored image440, and a similarity852.FIG. 18is a flowchart illustrating an operation of the present exemplary embodiment.

The study image selection unit340acquires a plurality of study images411(S630).

Next, the study image selection unit340outputs the acquired study images411to the correspondence information generation unit110(S631).

Next, the study image selection unit340determines whether or not the number of times in which the study image selection unit340detects the output of a restored image440(hereinafter, referred to as the number of times of detection) reaches a preset threshold value (S632). When the number of times of detection reaches the preset threshold value (YES in S632), the study image selection unit340ends the processing.

When the number of times of detection does not reach the preset threshold value (NO in S632), the study image selection unit340calculates feature vectors881that are related to respective ones of the study images411(S633).

Next, the study image selection unit340waits for the restoration unit104to output a restored image440(S634).

When the restoration unit104outputs a restored image440(YES in S634), the study image selection unit340updates the number of times of detection and acquires the restored image440(S635).

Next, the study image selection unit340calculates a feature vector884related to the restored image440(S636).

Next, the study image selection unit340selects a study image411on the basis of relations between the feature vectors881and the feature vector884(S637).

For example, when the number of restored images440is one, the study image selection unit340selects study images411each being related to a preset number of feature vectors881in ascending order of distances between respective ones of the feature vectors881and the feature vector884. When the number of restored images440is one, the study image selection unit340may select study image411each being related to feature vectors881the distances of which from the feature vector884are equal to or less than a preset threshold value.

For example, when the number of restored images440is plural, the study image selection unit340selects study images411each being related to a preset number of feature vectors881in ascending order of distances between respective ones of the feature vectors881and a vector corresponding to the centroid of a plurality of feature vectors884. When the number of restored images440is plural, the study image selection unit340may select study images411each being related to feature vectors881the distances of which from a vector corresponding to the centroid of a plurality of feature vectors884are equal to or less than a preset threshold value. When the number of restored images440is plural, the study image selection unit340may, with respect to each feature vector884, select study images411each being related to a preset number of feature vectors881in ascending order of distances between respective ones of the feature vectors881and the feature vector884. When the number of restored images440is plural, the study image selection unit340may, with respect to each feature vector884, select study images411each being related to feature vectors881the distances of which from the feature vector884are equal to or less than a preset threshold value.

Next, the study image selection unit340outputs the selected study images411to the correspondence information generation unit110(S638).

Next, the study image selection unit340determines whether or not the number of times of detection reaches the preset threshold value (S639). When the number of times of detection reaches the preset threshold value (YES in S639), the study image selection unit340ends the processing. When the number of times of detection does not reaches the predetermined threshold value (NO in S639), the study image selection unit340returns to the processing in S634.

The correspondence information generation unit110of the present exemplary embodiment generates correspondence information860on the basis of the study images411acquired from the study image selection unit340, and outputs the generated correspondence information860to the degradation process selection unit120.

The study unit102creates the dictionary103that stores a plurality of patch pairs each of which associates a degraded patch that is a patch in a degraded image420with a restored patch that is a patch in a study image410. Patches in each degraded image420are patches in the degraded image420into which a study image410is degraded on the basis of an estimated degradation process867, which is output by the degradation process estimation device300.

FIG. 19is a block diagram illustrating a structure of the study unit102. As illustrated inFIG. 19, the study unit102includes a reception unit1021, a degraded image generation unit1022, a patch pair generation unit1023, and a registration unit1024.

FIG. 20is a conceptual diagram for a description of a study phase. As illustrated inFIG. 20, the study unit102applies degradation processing to the study images410on the basis of the estimated degradation process867to generate the degraded images420. The study unit102registers, in the dictionary103, patch pairs403each of which includes patches at corresponding positions in a study image410and a degraded image420. Each patch pair403includes a restored patch401and a degraded patch402.

The reception unit1021receives the input of the study images410. The reception unit1021outputs the received study images410to the degraded image generation unit1022and the patch pair generation unit1023.

The degraded image generation unit1022applies degradation processing to the study images410output from the reception unit1021on the basis of the estimated degradation process867output from the degradation process estimation device200to generate the degraded images420.

When a plurality of estimated degradation processes867exist, the degraded image generation unit1022may apply degradation processing to each of the study images410output from the reception unit1021on the basis of the respective estimated degradation processes867to generate a plurality of degraded images420corresponding to each of the study image410.

The degraded image generation unit1022, for example, reduces a study image410to one N-th of the original size thereof on the basis of the estimated degradation process867to generate a degraded image420. As an algorithm to reduce an image, for example, a nearest neighbor method, in which image degradation is comparatively substantial, is used. As an algorithm to reduce an image, for example, a bi-linear method and a bi-cubic method may also be used.

The degraded image generation unit1022may increase the intensity of blurs of a study image410through, for example, removing high frequency components thereof on the basis of the estimated degradation process867to generate a degraded image420. The degraded image generation unit1022may change a posture of a study image410through, for example, tilting the study image410on the basis of the estimated degradation process867to generate a degraded image420. Alternatively, the degraded image generation unit1022may reduce the luminance of a study image410through, for example, decreasing the brightness thereof on the basis of the estimated degradation process867to generate a degraded image420. The degraded image generation unit1022may generate a degraded image420by various existing methods on the basis of the estimated degradation process867.

The patch pair generation unit1023receives the study images410from the reception unit1021and the degraded images420corresponding to the study images410from the degraded image generation unit1022. The patch pair generation unit1023generates a plurality of patch pairs403each of which includes patches at corresponding positions in the study image410and the degraded image420. The patch pair generation unit1023may generate a plurality of pairs (patch pairs403) of a restored patch401and a degraded patch402by an existing method. The patch pair generation unit1023outputs the plurality of generated patch pairs403to the registration unit1024.

The registration unit1024receives the plurality of patch pairs403from the patch pair generation unit1023. The registration unit1024registers the plurality of patch pairs403in the dictionary103.

The dictionary103stores the plurality of patch pairs generated by the study unit102.

The restoration unit104generates a restored image440from an input image430using the dictionary103, and outputs the generated restored image440to the degradation process estimation device300and the outside.

FIG. 21is a block diagram illustrating a configuration of the restoration unit104. As illustrated inFIG. 21, the restoration unit104includes a reception unit1041, a patch generation unit1042, and a restored image generation unit1043.

The reception unit1041receives the input image430that is an object of image processing from the outside. For example, the reception unit1041may connect to a network to receive the input image430, or read the input image430from a memory storing the input image430to receive the input image430. That is, the form of reception of the input image430by the reception unit1041is not limited to a specific form. The reception unit1041outputs the received input image430to the patch generation unit1042.

The patch generation unit1042generates a plurality of patches (input patches) from the input image430output from the reception unit1041, and outputs the generated patches to the restored image generation unit1043.

FIG. 22is a conceptual diagram for a description of a restoration phase. As illustrated inFIG. 22, the restoration unit104selects restored patches401on the basis of similarities between the input patches431in the input image430and the degraded patches402in the dictionary103.

The restored image generation unit1043selects a plurality of restored patches401which correspond to each of the input patches431from within the patch pairs403stored in the dictionary103on the basis of patch similarities, each of which is a value indicating a similarity between an input patch431and a degraded patch402. For example, the restored image generation unit1043selects restored patches401paired with degraded patches402the patch similarities of which to each input patch431are equal to or greater than a preset value. The restored image generation unit1043may select a preset number of restored patches401paired with degraded patches402in descending order of patch similarities to each input patch431.

The restored image generation unit1043composites a plurality of restored patches401to generate a restored image440. Each of the restored patches401is one of a plurality of restored patch401. The plurality of restored patches401correspond to each of the input patches431.

The restored image generation unit1043outputs the generated restored image440to the degradation process estimation device300. The restored image generation unit1043also outputs the restored image440to the outside. For example, the restoration unit104transmits the restored image440to the outside through the communication unit706illustrated inFIG. 6. The restoration unit104may output the restored image440through the output unit705illustrated inFIG. 6. The restoration unit104may record the restored image440in a recording medium707through the storage device703illustrated inFIG. 6.

The restoration unit104may select restored patches401on the basis of similarities between patches cut out from the restored image440and the restored patches401in addition to similarities between the input patches431and the degraded patches402.

FIG. 23is a diagram illustrating an example of a patch450. As illustrated inFIG. 23, a patch450, for example, includes a pixel group451that is a multidimensional vector having a plurality of pixel values of pixels452as elements. The patch450also includes, as meta-information, a patch identifier453that identifies the patch450individually. A patch450is a concept that includes a restored patch401, a degraded patch402, and an input patch431. A pixel value may be a brightness value, but is not limited thereto.

In this case, a value indicating a patch similarity between two patches may be a value based on differences in the brightness values of respective pixels452between the patches. For example, a value indicating a patch similarity may be a value based on an SSD (Sum of Square Distance), which is a sum of squares of differences in the brightness values of respective pixels452between patches. For example, a value indicating a patch similarity may be a value calculated by subtracting an SSD from a specific constant. In this case, the specific constant may, for example, be the SSD between a patch with the lowest brightness and a patch with the highest brightness. Alternatively, a value indicating a patch similarity may be a value based on an SAD (Sum of Absolute Distance), which is a sum of absolute values of differences in the brightness values of respective pixels452between patches. For example, a value indicating a patch similarity may be a value calculated by subtracting an SAD from a specific constant. In this case, the specific constant may, for example, be the SAD between a patch with the lowest brightness and a patch with the highest brightness.

In addition, a value indicating a patch similarity may, for example, be a value based on an angle between two feature vectors. Alternatively, a value indicating a patch similarity may be a value calculated by a normalized cross-correlation function, but is not limited thereto.

That is, the patch similarity is a similarity between images that are expressed by the respective pixel groups451of two patches. The above description is applied to not only the patch similarities but also the similarities851and the similarities852.

A first advantageous effect of the above-described present exemplary embodiment is the same as the advantageous effect of the first exemplary embodiment. Further, a second advantageous effect of the present exemplary embodiment is that it becomes possible to obtain the dictionary103that is required to estimate a degradation process of an input image430more accurately and restore a super-resolution image (restored image440) corresponding to the input image430with higher-resolution.

The reason for the second advantageous effect is that a configuration to repeat the following steps is applied. First, the degradation process estimation device300outputs an estimated degradation process867on the basis of study images411. Second, the study unit102generates the dictionary103on the basis of the estimated degradation process867. Third, the restoration unit104generates a restored image440on the basis of the dictionary103. Fourth, the study image selection unit340of the degradation process estimation device300selects study images411on the basis of the restored image440.

The present exemplary embodiment may be applied to the second exemplary embodiment. That is, local regions in respective images may be treated as processing units.

The respective components described in the above exemplary embodiments are not always required to be individually independent entities. For example, the respective components may be achieved so that a plurality of components is achieved as a single module. The respective components may be achieved so that a single component is achieved as a plurality of modules. The respective components may be configured in such a way that a component is a portion of another component. The respective components may be configured in such a way that a portion of a component overlaps a portion of another component.

The respective components and modules achieving the respective components in the above-described exemplary embodiments may be achieved, if possible, in a form of hardware in accordance with necessity. The respective components and modules achieving the respective components may be achieved by a computer and a program. The respective components and modules achieving the respective components may also be achieved by a mixture of modules in a form of hardware, and a computer and a program.

The program is, for example, provided being recorded in a nonvolatile computer-readable recording medium, such as a magnetic disk and a semiconductor memory, and read by a computer in activating the computer. The read program makes the computer function as the components in the afore-described exemplary embodiments by controlling operations of the computer.

Although, in the exemplary embodiments described above, a plurality of operations are described in sequence in a form of flowchart, the sequence of description does not limit a sequence in which the plurality of operations are performed. Thus, when the exemplary embodiments are carried out, the sequence of the plurality of operations can be changed within a range not affecting the content of the operations.

Further, in the exemplary embodiments described above, a plurality of operations are not limited to being performed at individually different timings. For example, an operation may be initiated while another operations is being performed, and execution timings of an operation and another operation may overlap each other partially or completely.

Further, although the exemplary embodiments described above were described in a manner in which an operation becomes a trigger of another operation, those descriptions do not limit all relations between an operation and another operation. Thus, when the exemplary embodiments are carried out, the relations between a plurality of operations can be changed within a range not affecting the content of the operations. Specific descriptions of the respective operations of the respective components do not limit the respective operations of the respective components. Thus, the respective specific operations of the respective components can be changed within a range not affecting functional, performance, and other features in carrying out the exemplary embodiments.

All or a part of the exemplary embodiments described above may be described as in the following Supplementary Notes, but the present invention is not limited thereto.

An information processing device comprising:

image acquisition means for acquiring a plurality of first study images and an input image; and

estimation means for outputting an estimated degradation process on a basis of first similarities between an arbitrary region in the input image and respective ones of a plurality of first degraded images when regions, in the first study images, corresponding to the region are degraded on a basis of respective ones of a plurality of degradation processes,

wherein the estimated degradation process corresponds to a degradation process in the degradation processes, the degradation process being related to the region in the input image.

The information processing device according to Supplementary Note 1,

wherein, further on a basis of correspondence relations between respective ones of the first degraded images and the degradation processes from the first study images to respective ones of the first degraded images, the estimation means outputs information discriminating the degradation process related to the region in the input image as the estimated degradation process.

The information processing device according to Supplementary Note 1 or 2,

wherein the first similarities correspond to relations between feature vectors corresponding to the first degraded images and a feature vector corresponding to the region in the input image.

The information processing device according to Supplementary Note 3,

wherein the estimation means outputs the estimated degradation process that corresponds to one of k (k is a natural number equal to or greater than 1) degradation processes of k-nearest neighbors, the k degradation processes being related to the region in the input image.

study means for creating a dictionary that stores a plurality of patch pairs each of which includes a degraded patch and a restored patch, the degraded patch being a patch in a second degraded image into which a second study image is degraded on a basis of the estimated degradation process, the restored patch being a patch in the second study image;

restoration means for generating a restored image from the input image by using the dictionary, and outputting the generated restored image; and

selection means for selecting the first study images on a basis of second similarities between the first study images and the restored image,

wherein, further on a basis of the first similarities between respective ones of a plurality of first degraded images into which the regions in the selected first study images are degraded and the region in the input image, the estimation means outputs the estimated degradation process.

The information processing device according to Supplementary Note 5,

wherein, further on a basis of correspondence relations between respective ones of the first degraded images and degradation processes from the selected first study images to respective ones of the first degraded images, the estimation means outputs information discriminating the degradation process related to the region in the input image as the estimated degradation process.

The information processing device according to Supplementary Note 5 or 6,

wherein the second similarities correspond to relations between feature vectors corresponding to the second study images and a feature vector corresponding to the restored image.

The information processing device according to any one of Supplementary Notes 1 to 7,

wherein the estimation means includes: degraded image generation means; feature vector calculation means; degradation process estimation dictionary creation means; degradation process estimation dictionary means; and feature vector classification means,

the degraded image generation means generates the first degraded images from the study images on a basis of the degradation processes, and outputs the generated first degraded images to the feature vector calculation means,

the feature vector calculation means generates the feature vectors corresponding to the first degraded images, and outputs the feature vectors corresponding to the first degraded images to the degradation process estimation dictionary creation means,

the degradation process estimation dictionary creation means generates correspondence information that indicates relations between the feature vectors corresponding to the first degraded images and the degradation processes corresponding to the feature vectors corresponding to the first degraded images, and outputs the generated correspondence information to the degradation process estimation dictionary means,

the degradation process estimation dictionary means stores the correspondence information,

the feature vector calculation means calculates the feature vector corresponding to the region in the input image, and outputs the feature vector corresponding to the region in the input image to the feature vector classification means, and

the feature vector classification means classifies the feature vector corresponding to the region in the input image into one of classes of the degradation processes on a basis of relations between the feature vector corresponding to the region in the input image and the feature vectors included in the correspondence information, and outputs the estimated degradation process corresponding to the class into which the feature vector corresponding to the local region in the input image is classified to the outside, the classes being of the feature vectors included in the correspondence information.

The information processing device according to Supplementary Note 8,

wherein the degradation process estimation dictionary creation means compresses the feature vectors corresponding to the first degraded images, and generates the correspondence information that indicates relations between the compressed feature vectors and the degradation processes.

wherein the estimation means outputs a first degraded image in the first degraded images as the estimated degradation process, the first degraded image being related to the degradation process corresponding to the region in the input image.

degradation information input means for allowing a user to input degradation information of the input image,

wherein, further on a basis of the degradation information, the estimation means outputs the estimated degradation process.

An image processing method using a computer implementing the image processing method, the image processing method comprising:

acquiring a plurality of first study images and an input image; and

outputting an estimated degradation process on a basis of first similarities between an arbitrary region in the input image and respective ones of a plurality of first degraded images when regions, in the first study images, corresponding to the region are degraded on a basis of respective ones of a plurality of degradation processes.

The information processing method according to Supplementary Note 12, using the computer,

wherein the outputting of the estimated degradation process includes outputting information discriminating the degradation process related to the region in the input image as the estimated degradation process further on a basis of correspondence relations between respective ones of the first degraded images and the degradation processes from the first study images to respective ones of the first degraded images.

The information processing method according to Supplementary Note 12 or 13,

wherein the first similarities correspond to relations between feature vectors corresponding to the first degraded images and a feature vector corresponding to the region in the input image.

The information processing method according to Supplementary Note 14, using the computer,

wherein the outputting of the estimated degradation process includes outputting the estimated degradation process that corresponds to one of k (k is a natural number equal to or greater than 1) degradation processes of k-nearest neighbors, the k degradation processes being related to the region in the input image.

The information processing method according to any one of Supplementary Notes 12 to 15, using the computer, further comprising:

creating a dictionary that stores a plurality of patch pairs each of which includes a degraded patch and a restored patch, the degraded patch being a patch in a second degraded image into which a second study image is degraded on a basis of the estimated degradation process, the restored patch being a patch in the second study image;

generating a restored image from the input image by using the dictionary, and outputting the generated restored image; and

selecting the first study images on a basis of second similarities between the first study images and the restored image,

wherein, the outputting of the estimated degradation process includes outputting the estimated degradation process further on a basis of the first similarities between respective ones of a plurality of first degraded images into which the regions in the selected first study images are degraded and the region in the input image.

The information processing method according to Supplementary Note 16, using the computer,

wherein the outputting of the estimated degradation process includes outputting information discriminating the degradation process related to the region in the input image as the estimated degradation process further on a basis of correspondence relations between respective ones of the first degraded images and degradation processes from the selected first study images to respective ones of the first degraded images.

The information processing method according to Supplementary Note 16 or 17,

wherein the second similarities correspond to relations between feature vectors corresponding to the second study images and a feature vector corresponding to the restored image.

The information processing method according to any one of Supplementary Notes 12 to 18, using the computer,

wherein the outputting of the estimated degradation process includes:

generating the first degraded images from the study images on a basis of the degradation processes, and outputting the generated first degraded images;

generating the feature vectors corresponding to the first degraded images, and outputting the feature vectors corresponding to the first degraded images;

generating correspondence information that indicates relations between the feature vectors corresponding to the first degraded images and the degradation processes corresponding to the feature vectors corresponding to the first degraded images, and outputting the generated correspondence information;

storing the correspondence information;

calculating the feature vector corresponding to the region in the input image, and outputting the feature vector corresponding to the region in the input image; and

classifying the feature vector corresponding to the region in the input image into one of classes of the degradation processes on a basis of relations between the feature vector corresponding to the region in the input image and the feature vectors included in the correspondence information, and outputing the estimated degradation process corresponding to the class into which the feature vector corresponding to the local region in the input image is classified to the outside, the classes being of the feature vectors included in the correspondence information.

The information processing method according to Supplementary Note 19, using the computer,

wherein the generating of correspondence information includes compressing the feature vectors corresponding to the first degraded images, and generates the correspondence information that indicates relations between the compressed feature vectors and the degradation processes.

The information processing method according to any one of Supplementary Notes 12 to 20, using the computer,

wherein the outputting of the estimated degradation process includes outputting a first degraded image in the first degraded images as the estimated degradation process, the first degraded image being related to the degradation process corresponding to the region in the input image.

A program causing a computer to execute processing of:

acquiring a plurality of first study image and an input image; and

outputting an estimated degradation process on a basis of first similarities between an arbitrary region in the input image and respective ones of a plurality of first degraded images when regions, in the first study images, corresponding to the region are degraded on a basis of respective ones of a plurality of degradation processes.

The program according to Supplementary Note 22, the program causing a computer to execute processing of:

further on a basis of correspondence relations between respective ones of the first degraded images and the degradation processes from the first study images to respective ones of the first degraded images, outputting information discriminating the degradation process related to the region in the input image as the estimated degradation process.

The program according to Supplementary Note 22 or 23,

wherein the first similarities correspond to relations between feature vectors corresponding to the first degraded images and a feature vector corresponding to the region in the input image.

The program according to Supplementary Note 24, the program causing a computer to execute processing of:

outputting the estimated degradation process that corresponds to one of k (k is a natural number equal to or greater than 1) degradation processes of k-nearest neighbors, the k degradation processes being related to the region in the input image.

The program according to any one of Supplementary Notes 22 to 25, the program further causing a computer to execute processing of:

creating a dictionary that stores a plurality of patch pairs each of which includes a degraded patch and a restored patch, the degraded patch being a patch in a second degraded image into which a second study image is degraded on a basis of the estimated degradation process, the restored patch being a patch in the second study image;

generating a restored image from the input image by using the dictionary, and outputting the generated restored image; and

selecting the first study images on a basis of second similarities between the first study images and the restored image,

further on a basis of the first similarities between respective ones of a plurality of first degraded images into which the regions in the selected first study images are degraded and the region in the input image, outputting the estimated degradation process.

The program according to Supplementary Note 26, the program causing a computer to execute processing of:

further on a basis of correspondence relations between respective ones of the first degraded images and degradation processes from the selected first study images to respective ones of the first degraded images, outputting information discriminating the degradation process related to the region in the input image as the estimated degradation process.

The program according to Supplementary Note 26 or 27, the program causing a computer to execute processing of:

wherein the second similarities correspond to relations between feature vectors corresponding to the second study images and a feature vector corresponding to the restored image.

The program according to any one of Supplementary Notes 22 to 28, the program causing a computer to execute processing of:

generating the first degraded images from the study images on a basis of the degradation processes, and outputting the generated first degraded images,

generating the feature vectors corresponding to the first degraded images, and outputting the feature vectors corresponding to the first degraded images,

generating correspondence information that indicates relations between the feature vectors corresponding to the first degraded images and the degradation processes corresponding to the feature vectors corresponding to the first degraded images, and outputting the generated correspondence information,

storing the correspondence information,

calculating the feature vector corresponding to the region in the input image, and outputting the feature vector corresponding to the region in the input image, and

classifying the feature vector corresponding to the region in the input image into one of classes of the degradation processes on a basis of relations between the feature vector corresponding to the region in the input image and the feature vectors included in the correspondence information, and outputting the estimated degradation process corresponding to the class into which the feature vector corresponding to the local region in the input image is classified to the outside, the classes being of the feature vectors included in the correspondence information.

The program according to Supplementary Note 29, the program causing a computer to execute processing of:

compressing the feature vectors corresponding to the first degraded images, and generates the correspondence information that indicates relations between the compressed feature vectors and the degradation processes.

The program according to any one of Supplementary Notes 22 to 30, the program causing a computer to execute processing of:

outputting a first degraded image in the first degraded images as the estimated degradation process, the first degraded image being related to the degradation process corresponding to the region in the input image.

(Supplementary Note 32) A non-transitory computer readable recording medium storing the program according to any one of Supplementary Notes 22 to 31.

An information processing device comprising:

a processor; and

a memory unit storing instructions for the processor to operate as image acquisition means, and estimation means, wherein

the image acquisition means acquires a plurality of first study images and an input image,

the estimation means outputs an estimated degradation process on a basis of first similarities between an arbitrary region in the input image and respective ones of a plurality of first degraded images when regions corresponding to the regions in the first study images are degraded on a basis of respective ones of a plurality of degradation processes, and

the estimated degradation process corresponds to a degradation process in the degradation processes, the degradation process being related to the region in the input image.

The present invention was described above through exemplary embodiments thereof, but the present invention is not limited to the above exemplary embodiments. Various modifications that could be understood by a person skilled in the art may be applied to the configurations and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2013-168793, filed on Aug. 15, 2013, the entire disclosure of which is incorporated herein by reference.

REFERENCE SIGNS LIST