Patent Description:
In order to understand the damage caused by disasters, such as floods, forest fires, volcanic eruptions, earthquakes, tsunamis, droughts, and the like, or urban development, a change detection technique for detecting an area in which the condition of the ground surface has changed, on the basis of an image photographed from a high place, such as a satellite image, has been developed.

Examples of the above change detection technique are disclosed in Non Patent Literature(NPL) <NUM> and NPL <NUM>. NPL <NUM> discloses a technique for individually correcting a photographed image as a preprocess. In addition, NPL <NUM> discloses a technique for masking (hiding), among detected areas in which the condition of the ground surface has changed, an area in which a change other than a change of a detection target has occurred.

In addition, NPL <NUM> discloses a method of computing a component of the sunlight spectrum from the solar zenith angle.

In addition, as networks usable for machine learning, a convolutional neural network (CNN) is disclosed in NPL <NUM>, a sparse auto encoder (SAE) is disclosed in NPL <NUM>, and a deep belief network (DBN) is disclosed in NPL <NUM>. Patent document <CIT> discloses to use a classifier for aerial images change detection, trained to discard irrelevant changes.

However, the above change detection technologies have a problem that changes of non-detection targets that are not related to damage or urban development, for example, changes due to sunshine conditions such as the presence/absence of shadow, changes of atmospheric conditions such as clouds and fog, and seasonal changes of plants is detected together with a change of a detection target.

The above problem will be described with reference to <FIG> is an explanatory diagram showing an example of generating a change map from two images. The upper of <FIG> shows an example in which the above change detection technique detects changes of non-detection targets together with a change of a detection target.

As shown in the upper of <FIG>, a change detection means <NUM> to which the above change detection technique is applied receives input of an image It-<NUM> photographed at a time (t-<NUM>) and an image It photographed at a time t. Note that, the image It-<NUM> and the image It are photographed images of the same area.

As shown in the upper of <FIG>, the image It-<NUM> shows a tree, a shadow of the tree, and a cloud. The image It shows a tree, a shadow of the tree, and buildings. Compared to the contents shown in the image It-<NUM>, the contents shown in the image It have differences that "the position of the shadow of the tree has changed", "the color of the leaves of the tree has changed", "there is no cloud", and "there are buildings".

Of the above differences, the only difference of the detection target is "there are buildings". However, if no settings for detecting changes are made, the change detection means <NUM> reflects all the differences between the image It-<NUM> and the image It in the change map.

In the change map shown in <FIG>, an area in which a change has detected is shown in white, and an area in which a change has not detected is shown in black. Thus, in a general change map shown in the upper of <FIG>, all the changes of not only a change of the buildings corresponding to "there are buildings" but also a change of the position of the shadow corresponding to "the position of the shadow of the tree has changed", a seasonal change of plants corresponding to "the color of leaves of the tree has changed", and a change of clouds corresponding to "there is no cloud" are reflected.

As described above, the change of the position of the shadow, the seasonal change of plants, and the change of clouds are unnecessary changes that should not be reflected in the change map. The lower of <FIG> shows an ideal change map with unnecessary changes removed from the general change map.

In the ideal change map shown in the lower of <FIG>, only a change of the buildings corresponding to "there are buildings" is reflected. That is, a change only of the detection target is reflected in the change map.

As described above, a technique for detecting, from a plurality of images with different photographing times, a change only of a detection target without detecting changes of non-detection targets, such as changes due to sunshine conditions, changes of atmospheric conditions, seasonal changes of forest, and the like is desired. NPL <NUM> to NPL <NUM> do not disclose techniques capable of detecting a change only of a detection target.

In view of the above, a purpose of the present invention is to provide a learning device, a learning method and a learning program that solve the above problem and are capable of detecting, among changes between a plurality of images with different photographing times, a change only of a detection target.

A learning device according to the present invention is specified in claim <NUM>.

A learning method according to the present invention is specified in claim <NUM>.

A learning program according to the present invention is specified in claim <NUM>.

According to the present invention, it is possible to detect, among changes between a plurality of images with different photographing times, a change only of a detection target.

First, the reason why it is difficult for the technique disclosed in each of NPL <NUM> and NPL <NUM> to detect a change only of a detection target will be described with reference to the drawings.

<FIG> is a block diagram showing a configuration example of a general image processing device <NUM>. The technique disclosed in NPL <NUM> is applied to the image processing device <NUM> shown in <FIG>. As shown in <FIG>, the image processing device <NUM> includes a first correction means <NUM>, a second correction means <NUM>, a feature-value computation means <NUM>, and a change-pixel detection means <NUM>.

The first correction means <NUM> has a function of correcting a shadow in an input observation image. The second correction means <NUM> has a function of correcting a shadow in an input reference image. The first correction means <NUM> and the second correction means <NUM> each correct a shadow in such a manner as to satisfy a hypothetical condition of "the reflectance of the shadow is <NUM>, and there is no water area".

The feature-value computation means <NUM> has a function of computing a feature value of a change. The feature value indicates the degree of a change between an observation image with corrected shadow and a reference image with corrected shadow. The change-pixel detection means <NUM> has a function of detecting a change pixel on the basis of the computed feature value of a change to generate a change map on the basis of the detected change pixel.

<FIG> is an explanatory diagram showing an example in which the image processing device <NUM> generates a change map. In the example shown in <FIG>, an image It-<NUM> photographed at a time (t-<NUM>) is firstly input to the first correction means <NUM>. In addition, an image It photographed at a time t is input to the second correction means <NUM>. Note that, the image It-<NUM> and the image It are similar to the image It-<NUM> and the image It shown in <FIG>, respectively.

As shown in <FIG>, the first correction means <NUM> performs a first correction process of erasing the shadow in the input image It-<NUM>. However, as shown in <FIG>, the cloud is corrected as well as the shadow in the image It-<NUM> that has been subjected to the first correction process. The correction of the cloud is a correction caused by a correction error by the first correction means <NUM>. The correction error is caused because the first correction means <NUM> has corrected the shadow in such a manner as to satisfy the hypothetical condition.

In addition, as shown in <FIG>, the second correction means <NUM> performs a second correction process of erasing the shadow in the input image It. However, as shown in <FIG>, the shadow is not completely erased, and a seasonal change of the plant is also corrected in the image It that has been subjected to the second correction process. Both corrections are caused by correction errors by the second correction means <NUM>. The correction error is caused because the second correction means <NUM> has corrected the shadow in such a manner as to satisfy the hypothetical condition.

The feature-value computation means <NUM> computes a feature value of a change between the image It-<NUM> with the correction error and the image It with the correction error. The change-pixel detection means <NUM> detects a change pixel on the basis of the computed feature value of the change to generate a change map on the basis of the detected change pixel.

In the change map generated through a change detection process by the image processing device <NUM>, unnecessary changes, such as the change of the position of the shadow, the seasonal change of the plant, and the change of the cloud, caused by the correction error are reflected as shown in <FIG>.

As described above, the image processing device <NUM> has a problem of limited conditions that can be satisfied without causing a correction error in a correction process. Furthermore, some conditions cannot be satisfied in a correction process, which is another problem that each correction means of the image processing device <NUM> cannot always correct shadows properly.

<FIG> is a block diagram showing a configuration example of a general image processing device <NUM>. The technique disclosed in NPL <NUM> is applied to the image processing device <NUM> shown in <FIG>. As shown in <FIG>, the image processing device <NUM> includes a feature-value computation means <NUM>, a change-pixel detection means <NUM>, an unnecessary-change-area detection means <NUM>, and an unnecessary-change removal means <NUM>.

The feature-value computation means <NUM> has a function of computing a feature value of a change between an observation image and a reference image. The change-pixel detection means <NUM> has a function of detecting a change pixel on the basis of the computed feature value of the change to generate a first change map on the basis of the detected change pixel.

The unnecessary-change-area detection means <NUM> has a function of detecting, as an unnecessary change area, an area in which a change of non-detection targets has occurred between the observation image and the reference image. The unnecessary-change-area detection means <NUM> generates an unnecessary-change map representing the detected unnecessary change area. The unnecessary-change removal means <NUM> has a function of detecting the difference between the first change map and the unnecessary-change map to generate a second change map.

<FIG> is an explanatory diagram showing an example in which the image processing device <NUM> generates a change map. In the example shown in <FIG>, an image It-<NUM> photographed at a time (t-<NUM>) and an image It photographed at a time t are firstly input to the feature-value computation means <NUM> and the unnecessary-change-area detection means <NUM>. Note that, the image It-<NUM> and the image It are similar to the image It-<NUM> and the image It shown in <FIG>, respectively.

The feature-value computation means <NUM> computes a feature value of a change between the image It-<NUM> and the image It. The change-pixel detection means <NUM> detects a change pixel on the basis of the computed feature value of the change to generate a first change map on the basis of the detected change pixel.

As shown in <FIG>, in the first change map generated through a change detection process by the change-pixel detection means <NUM>, all the changes are reflected similarly to those in the general change map shown in the upper of <FIG>.

In addition, as shown in <FIG>, the unnecessary-change-area detection means <NUM> detects an unnecessary change area between the image It-<NUM> and the image It and performs an unnecessary change detection process to generate an unnecessary-change map representing the detected unnecessary change area. As shown in <FIG>, in the unnecessary-change map generated through the unnecessary change detection process by the unnecessary-change-area detection means <NUM>, changes only of the non-detection target are reflected.

The unnecessary-change removal means <NUM> performs an unnecessary change removal process to generate a second change map by subtracting the unnecessary-change map from the first change map.

Theoretically, a change only of the detection target is to be reflected in the second change map generated after the unnecessary change removal process by the unnecessary-change removal means <NUM>. However, as shown in <FIG>, the change of the building that had occurred in the shadow of the tree is not reflected in the second change map.

This is because an algorithm that simply removes all the areas in which a change of a shadow occurs is applied to the image processing device <NUM>. That is, the image processing device <NUM> has a problem that a change of a shadow cannot be detected.

As described above, it is difficult for the technique disclosed in each of NPL <NUM> and NPL <NUM> to detect a change only of a detection target. For the above reason, the present invention is to provide a learning device and an image processing device that cause a detector to detect a change only of a detection target with high accuracy and also to detect a change of a shadow.

Hereinafter, exemplary embodiments of the present invention will be described with reference to the drawings. <FIG> is a block diagram showing a configuration example of an image processing device according to a first example useful for understanding but not pertaining to the invention as claimed.

An image processing device <NUM> according to the present example detects a change between images photographed at two different times and a change between metadata of the images. After detecting the change, the image processing device <NUM> generates a change map and a reliability map indicating the degree of reliability of each pixel.

Then, the image processing device <NUM> extracts, on the basis of the generated reliability map, an area corresponding to the periphery of a reliable pixel from each of the two images and the change map and combines the extracted areas with the metadata to generate a data set. The generated data set is used for learning to detect a change only of a detection target.

As shown in <FIG>, the image processing device <NUM> includes a satellite image database (DB) <NUM>, an earth observation means <NUM>, a change detection means <NUM>, a metadata extraction means <NUM>, and a data-set generation means <NUM>.

The satellite image DB <NUM> stores a reference image photographed by an artificial satellite and metadata of the reference image. The satellite image DB <NUM> outputs an image photographed at a reference time and the metadata of the image photographed at the reference time.

The earth observation means <NUM> has a function of photographing the condition of the ground surface of an observation target. The earth observation means <NUM> outputs an image photographed at an arbitrary time and the metadata of the image photographed at the arbitrary time.

The metadata of an image indicates the photographing condition when the image is photographed. The metadata of the image includes, for example, data indicating the position of the artificial satellite at the photographing time and data indicating the direction of the antenna used for photographing.

The change detection means <NUM> has a function of generating a change map and a reliability map on the basis of the image photographed at the reference time, the metadata of the image photographed at the reference time, the image photographed at the arbitrary time, and the metadata of the image photographed at the arbitrary time.

For example, the change detection means <NUM> limits using model parameters, the range of the spectrum that changes in accordance with conditions causing unnecessary changes. The model parameters, which will be described later, are computed from the metadata indicating the solar zenith angle, the date and time, and the like.

The unnecessary changes include changes due to sunshine conditions, changes of atmospheric conditions, seasonal changes of forests, and the like as described above. That is, it can be said that the unnecessary changes in the present exemplary embodiment are periodic changes in accordance with the photographing environment.

By limiting the range of the spectrum, the change detection means <NUM> computes a feature value of a change indicating the degree of a change with no unnecessary changes. Then, the change detection means <NUM> detects a change pixel on the basis of the computed feature value of the change. The change detection means <NUM> classifies the detected change pixel and also computes the reliability of the detection for each pixel.

The metadata extraction means <NUM> has a function of extracting metadata required for a data set from the metadata of the image photographed at the reference time and the metadata of the image photographed at the arbitrary time.

The data-set generation means <NUM> has a function of generating a data set to be used for learning, on the basis of the generated change map and reliability map, the image photographed at the reference time, and the image photographed at the arbitrary time.

<FIG> is a block diagram showing a configuration example of the change detection means <NUM>. As shown in <FIG>, the change detection means <NUM> includes a model-parameter computation means <NUM>, a feature-value computation means <NUM>, a change-pixel detection means <NUM>, and a reliability computation means <NUM>.

The model-parameter computation means <NUM> has a function of computing a model parameter at the arbitrary time on the basis of the metadata of the image photographed at the arbitrary time and computing a model parameter at the reference time on the basis of the metadata of the image photographed at the reference time.

The model parameters in the present example are environment data indicating the state of a periodic change at a photographing time and data about an object. That is, the model-parameter computation means <NUM> computes a model parameter representing the state of a periodic change on the basis of the metadata of an image.

The feature-value computation means <NUM> has a function of computing a feature value of the change with no unnecessary changes, on the basis of the image photographed at the reference time, the image photographed at the arbitrary time, and the computed model parameters.

The change-pixel detection means <NUM> has a function of generating a change map on the basis of the computed feature value of the change with no unnecessary changes. The reliability computation means <NUM> has a function of generating a reliability map on the basis of the computed feature value of the change with no unnecessary changes.

In the following, an example in which the image processing device <NUM> generates a data set will be described with reference to the drawings. <FIG> is an explanatory diagram showing an example in which the change detection means <NUM> computes a feature value of a change with no unnecessary changes.

As shown in <FIG>, in this example, the satellite image DB <NUM> outputs an image It-<NUM> photographed at a time (t-<NUM>) and the metadata of the image It-<NUM>. In addition, the earth observation means <NUM> outputs an image It photographed at a time t and the metadata of the image It. Note that, the image It-<NUM> and the image It are similar to the image It-<NUM> and the image It shown in <FIG>, respectively.

The model-parameter computation means <NUM> computes a model parameter at the time (t-<NUM>) on the basis of the metadata of the image It-<NUM>. The model-parameter computation means <NUM> further computes a model parameter at the time t on the basis of the metadata of the image It.

An example of computing model parameters by the model-parameter computation means <NUM> is described below. The model-parameter computation means <NUM> uses, for example, the solar zenith angle θ indicated by the metadata and the latitude and longitude of the point indicated by the image as input to compute, in accordance with a radiation transmission model of the atmosphere, a direct light component of the sunlight spectrum (hereinafter, also referred to as a direct component) sa and a scattered light component (hereinafter, also referred to as a scattered component) ss as follows.

The subscript t in Expression (<NUM>) indicates that the data is at the time t. Similarly, the subscript t-<NUM> indicates that the data is at the time (t-<NUM>). The function fBird in Expression (<NUM>) is the function disclosed in NPL <NUM>. In addition, the direct component sa and the scattering component ss are vectors.

The computed direct component sa and scattering component ss of the sunlight spectrum represent the state of sunlight at the photographing time. In addition, the direct component sd and scattering component ss of the sunlight spectrum suggest how the image changes due to shadows.

The model-parameter computation means <NUM> may further compute the solar zenith angle θ from, for example, the date and time, indicated by the metadata, when the image was photographed and from the latitude and longitude of the point indicated by the image. The model-parameter computation means <NUM> may further compute the solar azimuth angle together with the solar zenith angle θ.

The model-parameter computation means <NUM> may further compute, for example, the zenith angle of the artificial satellite having photographed the image. The model-parameter computation means <NUM> may further compute the azimuth angle of the artificial satellite together with the zenith angle of the artificial satellite.

The model-parameter computation means <NUM> may further use, for example, the date and time, indicated by the metadata, when the image was photographed and the latitude and longitude of the point indicated by the image as input to compute, in accordance with a model of a seasonal change of plants, the spectrum of vegetation in the season when the image was photographed. The model-parameter computation means <NUM> may further compute, together with the spectrum of vegetation, a normalized difference vegetation index (NDVI), which is a kind of vegetation index, and the CO<NUM> absorption amount.

Each computed information represents the state of vegetation at the photographing time. In addition, each computed information suggests how the forest changes seasonally. The model-parameter computation means <NUM> may compute each information for each pixel together with a map showing the plant community.

The model-parameter computation means <NUM> may further use, for example, the solar azimuth angle and observation azimuth angle indicated by the metadata as input to compute the solar azimuth angle relative to the image in accordance with a geometric model.

The solar azimuth angle relative to the image is information indicating the direction in which a shadow is formed at the photographing time. The model-parameter computation means <NUM> may use the solar azimuth angle relative to the image and the solar zenith angle as information suggesting the direction in which a shadow is formed and the length of the shadow.

<FIG> is an explanatory diagram showing examples of the model parameters computed by the model-parameter computation means <NUM>. The subscript t of each vector shown in <FIG> indicates that the data is at the time t. Similarly, the subscript t-<NUM> of each vector shown in <FIG> indicates that the data is at the time (t-<NUM>).

The upper of <FIG> shows vectors representing the state of the direct component sd and the state of the scattering component ss of the sunlight spectrum. When each condition shown in the upper of <FIG> is satisfied, each component of the vectors becomes <NUM>. The "band" in each condition shown in the upper of <FIG> means a band spectrum.

Alternatively, the model-parameter computation means <NUM> may directly compute a vector representing the intensity of each wavelength instead of the vector representing the state of a component of the sunlight spectrum.

The middle of <FIG> shows vectors representing the state of the NDVI of a plant. The components of the vectors to be <NUM> are determined according to which range shown in the middle of <FIG> the value of the NDVI falls into. The model-parameter computation means <NUM> may directly compute the scalar representing the value of the NDVI instead of the vector representing the state of the NDVI of the plant.

The lower of <FIG> shows vectors representing the state of the solar azimuth angle relative to the image at the photographing time. The components of the vectors to be <NUM> are determined according to which range shown in the lower of <FIG> the value of the solar azimuth angle falls into. The model-parameter computation means <NUM> may directly compute the scalar representing the solar azimuth angle instead of the vector representing the state of the relative solar azimuth angle.

As described above, the model-parameter computation means <NUM> computes, on the basis of the data indicating the photographing condition of each of a plurality of images, a parameter representing a periodic change of a predetermined object displayed in the plurality of images. The model-parameter computation means <NUM> inputs the computed model parameter at the time (t-<NUM>) and model parameter at the time t to the feature-value computation means <NUM>.

The feature-value computation means <NUM> computes, on the basis of the image It-<NUM>, the image It, and the computed model parameter at the time (t-<NUM>) and model parameter at the time t, a feature value of a change with no unnecessary changes.

The feature-value computation means <NUM> computes, for each pixel, a feature value of a change with no unnecessary changes on the basis of, for example, a physical model. Then, the feature-value computation means <NUM> generates a change map indicating the feature value of the change with no unnecessary changes.

Regarding the areas of the change map shown in <FIG>, an area where a change is larger has the color closer to white. The grid pattern area in the change map shown in <FIG> is an area where a change is not larger than the white area.

The white dots encircled by the broken-line ellipse in the change map shown in <FIG> are areas where changes have occurred due to noise. In addition, the horizontal-line-pattern area in the change map shown in <FIG> is an area where a change has occurred due to an error of the model itself (model error).

In the following, a computation example of a feature value of a change with no unnecessary changes will be described. <FIG> is an explanatory diagram showing a computation example of a feature value of a change not including a change of the position of a shadow. For example, the feature-value computation means <NUM> computes a change vector c of an arbitrary pixel in the spectral space having the same dimension as the observed wavelength number as shown in <FIG>.

As shown in <FIG>, the change vector c is computed using the direct component sd and scattering component ss of the sunlight spectrum computed by the model-parameter computation means <NUM> and a standard sunlight spectrum sstd. The slant-line-pattern area shown in <FIG> represents the possible range of the change vector c due to a change of the position of a shadow.

The shortest distance from the origin to the change vector c is computed by the Expression shown in <FIG>. The computed shortest distance corresponds to a feature value icf of the change not including a change of the position of the shadow.

As described above, the feature-value computation means <NUM> is capable of computing, using the model parameters computed by the model-parameter computation means <NUM> and a plurality of images, a feature value indicating the degree of a change in which a periodic change (for example, a change of the position of a shadow) is removed from changes between the plurality of images. Note that, the feature-value computation means <NUM> may compute a feature value of a change with no unnecessary changes by a method other than the method shown in <FIG>.

<FIG> is an explanatory diagram showing an example of generating a change map and a reliability map. The feature-value computation means <NUM> inputs the computed feature value of the change with no unnecessary changes to the change-pixel detection means <NUM> and the reliability computation means <NUM>.

The change-pixel detection means <NUM> generates a change map by reflecting only a feature value of a change equal to or greater than a predetermined threshold among input feature values of changes. For example, in the change map shown in <FIG>, a white area indicating a feature value of a change with no unnecessary changes in the change map and a horizontal-line-pattern area are represented as areas "with a change".

The reliability computation means <NUM> generates a reliability map by reflecting only the feature value of the change equal to or greater than the predetermined threshold among the input feature values of the changes. The reliability computation means <NUM> may further generate a reliability map by reflecting only a feature value of a change in which dispersion is equal to or less than a predetermined threshold among the input feature values of the changes. That is, the reliability computation means <NUM> computes the reliability of the feature value computed by the feature-value computation means <NUM>.

In the reliability map shown in <FIG>, an area with reliability is shown in white, and an area without reliability is shown in black. For example, in the reliability map shown in <FIG>, areas determined as "with noise" and as "with a model error" on the basis of the feature value of the change with no unnecessary changes are represented as areas "without reliability".

<FIG> is an explanatory diagram showing an example of generating a data set. The data-set generation means <NUM> extracts the value of the pixel in the change map corresponding to each pixel of the area determined as "with reliability" in the reliability map in association with the peripheral area of the pixel of the image at each time. The data-set generation means <NUM> may extract the value of the peripheral area of the corresponding pixel as the value of the change map.

In the example shown in <FIG>, the data-set generation means <NUM> extracts the value of the area encircled by the broken-line rectangle in the change map corresponding to the area encircled by the broken-line rectangle in the reliability map as the value of the change map. Since the extracted value indicates "with a change", the presence/absence of a change in the data in the first row of the data set shown in <FIG> is represented by a white rectangle.

Note that, when the extracted value indicates "with no change", the presence/absence of a change is represented by a black rectangle. Alternatively, instead of the presence/absence of a change, the value of the change map itself may be included in the data.

The data-set generation means <NUM> further extracts the area encircled by the broken-line rectangle in the image It-<NUM> in association with the area encircled by the broken-line rectangle in the image It. Note that, the data-set generation means <NUM> may extract the center pixel of the rectangle instead of the area encircled by the rectangle.

In addition, the metadata extraction means <NUM> extracts the metadata about the extracted area of the image It-<NUM> and the metadata about the extracted area of the image It. The data-set generation means <NUM> generates each data in the data set shown in <FIG> by combining each extracted image area, each extracted metadata, and the presence/absence of the change. With the above processes, the data set shown in <FIG> is generated.

The change detection means <NUM> in the present exemplary embodiment generates change information (for example, a change map) indicating, for each pixel constituting an image, a plurality of feature values indicating the degree of a change in which a periodic change of a predetermined object is removed from changes between a plurality of images and reliability information (for example, a reliability map) indicating, for each pixel, reliability of each of the plurality of feature values.

Then, the data-set generation means <NUM> in the present example extracts, from the plurality of images, an area including a pixel corresponding to a feature value whose reliability indicated by the generated reliability information is equal to or greater than a predetermined value and extracts, from the generated change information, a feature value equal to or greater than the predetermined value. The data-set generation means <NUM> further generates learning data including each extracted area, the extracted feature value equal or greater than the predetermined value, and data indicating a photographing condition of each of the plurality of images associated with each other.

Hereinafter, the operation of generating a change map and a reliability map by the image processing device <NUM> according to the present example will be described with reference to <FIG> is a flowchart showing the operation of a change map and reliability map generation process by the image processing device <NUM> according to the first example.

First, the earth observation means <NUM> inputs an image photographed at an arbitrary time and the metadata of the image photographed at the arbitrary time to the change detection means <NUM> (step S101).

Then, the satellite image DB <NUM> inputs an image photographed at a reference time and the metadata of the image photographed at the reference time to the change detection means <NUM> (step S102).

Then, the model-parameter computation means <NUM> computes a model parameter at the arbitrary time on the basis of the metadata of the image photographed at the arbitrary time. The model-parameter computation means <NUM> further computes a model parameter at the reference time on the basis of the metadata of the image photographed at the reference time (step S103). The model-parameter computation means <NUM> inputs the computed model parameters to the feature-value computation means <NUM>.

Then, the feature-value computation means <NUM> computes a feature value of a change with no unnecessary changes using the image photographed at the reference time, the image photographed at the arbitrary time, and the model parameters computed in step S103 (step S104). The feature-value computation means <NUM> inputs the computed feature value to the change-pixel detection means <NUM> and the reliability computation means <NUM>.

Then, the change-pixel detection means <NUM> generates a change map representing the presence/absence of a change for each pixel using the computed feature value of the change with no unnecessary changes (step S105).

Then, the reliability computation means <NUM> generates a reliability map representing the reliability of the change map generated in step S105 for each pixel using the computed feature value of the change with no unnecessary changes (step S106). After generating the reliability map, the image processing device <NUM> terminates the change map and reliability map generation process.

Next, the operation of generating a data set by the image processing device <NUM> according to the present example will be described with reference to <FIG> is a flowchart showing the operation of a data set generation process by the image processing device <NUM> according to the first example.

First, the earth observation means <NUM> inputs the image photographed at the arbitrary time to the data-set generation means <NUM>. The earth observation means <NUM> further inputs the metadata of the image photographed at the arbitrary time to the metadata extraction means <NUM> (step S111).

Then, the satellite image DB <NUM> inputs the image photographed at the reference time to the data-set generation means <NUM>. The satellite image DB <NUM> further inputs the metadata of the image photographed at the reference time to the metadata extraction means <NUM> (step S112).

Then, the change detection means <NUM> inputs the generated change map and reliability map to the data-set generation means <NUM> (step S113).

Then, the data-set generation means <NUM> extracts an area corresponding to the periphery of each reliable pixel in the reliability map from each of the image photographed at the reference time, the image photographed at the arbitrary time, and the change map (step S114). The data-set generation means <NUM> inputs each extracted area to the metadata extraction means <NUM>.

Then, the metadata extraction means <NUM> extracts metadata about each area extracted in step S114 from the metadata of the image photographed at the reference time and the metadata of the image photographed at the arbitrary time (step S115). The metadata extraction means <NUM> inputs each extracted metadata to the data-set generation means <NUM>.

Then, the data-set generation means <NUM> generates a data set constituted by data in which each extracted image area, each extracted metadata, and the presence/absence of the change corresponding to the value of the extracted area of the change map are associated with each other (step S116). After generating the data set, the image processing device <NUM> terminates the data set generation process.

The image processing device <NUM> according to the present example includes the change detection means <NUM> that detects a change from images photographed at two different times and the metadata of each of the images and generates a change map and a reliability map indicating the degree of reliability for each pixel.

The image processing device <NUM> further includes the data-set generation means <NUM> that extracts an area corresponding to the periphery of a reliable pixel in the reliability map from each of the images photographed at the two different times and the change map and combines them with the metadata to generate a data set.

The change detection means <NUM> includes the feature-value computation means <NUM> that computes a feature value of a change with no unnecessary changes by limiting the range of the spectrum that changes in accordance with the conditions causing unnecessary changes using model parameters computed from the metadata about the solar zenith angle, the date and time, and the like. The unnecessary changes include changes due to sunshine conditions, changes of atmospheric conditions, and seasonal changes of forests.

The change detection means <NUM> further includes the change-pixel detection means <NUM> that detects a change pixel on the basis of the computed feature value of the change and classifies the detected change pixel, and the reliability computation means <NUM> that computes the reliability of the detection for each pixel.

Thus, the image processing device <NUM> according to the present example is capable of generating a data set required for learning a process of detecting a change only of a detection target without detecting unnecessary changes.

Next, a learning device according to a second exemplary embodiment of the present invention will be described with reference to the drawings. <FIG> is a block diagram showing a configuration example of the learning device according to the second exemplary embodiment of the present invention.

A learning device <NUM> according to the present exemplary embodiment causes a device to learn a process of detecting only a change other than unnecessary changes using a data set constituted by a large number of data including a set of image areas photographed at the same point at two different times, the presence/absence of a change in the image areas, and metadata about each image area.

That is, a change detector that has learned the process of detecting only a change other than unnecessary changes does not detect the unnecessary changes. The unnecessary changes include changes due to sunshine conditions, changes of atmospheric conditions, and seasonal changes of forests.

As shown in <FIG>, the learning device <NUM> includes a model-parameter computation means <NUM> and a machine learning means <NUM>. The learning device <NUM> receives a data set input from the image processing device <NUM> according to the first exemplary embodiment.

In addition, the learning device <NUM> is communicably connected to a change detector <NUM> as shown in <FIG>. The change detector <NUM> having completed the learning detects only a change other than unnecessary changes from the images photographed at the same point at two different times.

The model-parameter computation means <NUM> has a function of computing a model parameter at an arbitrary time on the basis of the metadata of the image photographed at the arbitrary time in the data set and computing a model parameter at a reference time on the basis of the metadata of the image photographed at the reference time in the data set. The function of the model-parameter computation means <NUM> is similar to the function of the model-parameter computation means <NUM> in the first example.

The machine learning means <NUM> has a function of causing a device to learn a process of detecting only a change other than unnecessary changes using a set of image areas photographed at the same point at two different times, the presence/absence of a change in the image areas, and the model parameter about each image area.

Hereinafter, an example in which the learning device <NUM> causes the change detector <NUM> to learn a change detection process will be described with reference to the drawings. <FIG> is an explanatory diagram showing an example in which the learning device <NUM> causes the change detector <NUM> to learn a process of detecting only a change other than unnecessary changes.

The data set shown in <FIG> is the same as the data set shown in <FIG>. The model-parameter computation means <NUM> having received the input data set computes a model parameter on the basis of the metadata of each image. After the computation, the model-parameter computation means <NUM> inputs a data set including the model parameters instead of the metadata to the machine learning means <NUM>.

The machine learning means <NUM> having received the input data set including the model parameters causes the change detector <NUM> to learn a process of detecting only a change other than unnecessary changes.

In the example shown in <FIG>, the machine learning means <NUM> causes the change detector <NUM> to learn a process of outputting, when each of the model parameter at the time (t-<NUM>), the model parameter at the time t, the image area at the time (t-<NUM>), and the image area at the time t is input to a network constituting the change detector <NUM>, the presence/absence of the corresponding change. The model parameter at the time (t-<NUM>) and the model parameter at the time t in the example shown in <FIG> are the solar zenith angle θt-<NUM> and the solar zenith angle θt, respectively.

In addition, the model parameter at the time (t-<NUM>) and the model parameter at the time t may be vectors representing the state of the direct light component sd and the state of the scattered light component ss of the sunlight spectrum shown in the upper of <FIG>, respectively. When the two vectors shown in the upper of <FIG> are directly input to the machine learning means <NUM>, the machine learning means <NUM> removes a periodic change and causes the change detector <NUM> to learn a process of detecting a change other than the periodic change.

The model parameter at the time (t-<NUM>) and the model parameter at the time t may be the vectors representing the state of the NDVI of the plant shown in the middle of <FIG> or the vectors representing the state of the solar azimuth angle relative to the image at the photographing time shown in the lower of <FIG>.

In addition, the network constituting the change detector <NUM> may be any network as long as it is usable for machine learning such as the CNN disclosed in NPL <NUM>, the SAE disclosed in NPL <NUM>, the DBN disclosed in NPL <NUM>, or the like.

The change detector <NUM> having learned the process of detecting only a change other than unnecessary changes detects a change only of a detection target without detecting unnecessary changes from the images photographed at the same point at two different times.

The machine learning means <NUM> in the present exemplary embodiment causes a detector to learn, using learning data including at least a set of image areas representing a periodic change of a predetermined object among changes between a plurality of images and a set of image areas representing a change other than the periodic change among the changes between the plurality of images, a process of detecting a change other than the periodic change among the changes between the plurality of images.

The model-parameter computation means <NUM> in the present exemplary embodiment computes a parameter representing the periodic change of the predetermined object on the basis of data indicating photographing conditions of the image areas included in the learning data. The machine learning means <NUM> causes the detector to learn using the computed parameters and the learning data.

The advantage of machine learning using the model parameters is that machine learning is facilitated. For example, when machine learning is performed with a data set with no model parameters, the data set is required to contain data related to many pattern changes. However, it is difficult to prepare a data set constituted by various types of data.

When machine learning is performed with model parameters, the learning device <NUM> causes the change detector <NUM> to refer to data about similar changes on the basis of the model parameters in order to analogize the pattern of a change although the data set does not include the pattern of the change. That is, it is possible for the user to reduce the types of data included in the data set.

Hereinafter, the operation of the learning device <NUM> according to the present exemplary embodiment causing the change detector <NUM> to learn the change detection process will be described with reference to <FIG> is a flowchart showing the operation of the learning process by the learning device <NUM> according to the second exemplary embodiment.

First, the image processing device <NUM> inputs the generated data set to the learning device <NUM> (step S201).

Then, the model-parameter computation means <NUM> computes a model parameter at the arbitrary time on the basis of the metadata of the image photographed at the arbitrary time. The model-parameter computation means <NUM> further computes a model parameter at the reference time on the basis of the metadata of the image photographed at the reference time (step S202). The model-parameter computation means <NUM> inputs a data set including the computed model parameters to the machine learning means <NUM>.

Then, the machine learning means <NUM> causes the change detector <NUM> to learn a process of detecting a change only of the detection target without detecting unnecessary changes using the input data set (step S203).

Specifically, the machine learning means <NUM> causes the change detector <NUM> to learn a process of detecting a change only of the detection target without detecting unnecessary changes, such as a change of the position of a shadow, a change of the state of clouds, a seasonal change of plants, and the like, using the data set. After the learning, the learning device <NUM> terminates the learning process.

The learning device <NUM> according to the present exemplary embodiment includes the machine learning means <NUM> that causes a device to learn a process of detecting a change only of a detection target without detecting unnecessary changes using data including a set of image areas photographed at the same point at two different times, the presence/absence of a change in the image areas, and model parameters at the observation time of each image area. The unnecessary changes include changes due to sunshine conditions, changes of atmospheric conditions, and seasonal changes of forests.

Thus, the learning device <NUM> according to the present exemplary embodiment is capable of causing the change detector <NUM> to learn a process of detecting a change only of the detection target without detecting unnecessary changes. The change detector <NUM> having learned is capable of detecting a change only of the detection target among changes between a plurality of images with different photographing times.

Note that, the image processing device <NUM> according to the first example and the learning device <NUM> according to the second exemplary embodiment may be used independently or may be used in the same system.

Hereinafter, a specific example of a hardware configuration of the image processing device <NUM> according to the first example and a specific example of a hardware configuration of the learning device <NUM> according to the second exemplary embodiment will be described.

<FIG> is an explanatory diagram showing a hardware configuration example of the image processing device <NUM> according to the first example. The image processing device <NUM> shown in <FIG> includes a central processing unit (CPU) <NUM>, a main storage unit <NUM>, a communication unit <NUM>, and an auxiliary storage unit <NUM>. The image processing device <NUM> may further include an input unit <NUM> for the user to operate and an output unit <NUM> for presenting a processing result or the progress of the processing content to the user.

<FIG> is an explanatory diagram showing a hardware configuration example of the learning device <NUM> according to the present invention. The learning device <NUM> shown in <FIG> includes a CPU <NUM>, a main storage unit <NUM>, a communication unit <NUM>, and an auxiliary storage unit <NUM>. The learning device <NUM> may further include an input unit <NUM> for the user to operate and an output unit <NUM> for presenting a processing result or the progress of the processing content to the user.

Each of the main storage unit <NUM> and the main storage unit <NUM> is used as a work region of data and a temporary save region of data. Each of the main storage unit <NUM> and the main storage unit <NUM> is, for example, a random access memory (RAM).

Each of the communication unit <NUM> and the communication unit <NUM> has a function of inputting and outputting data to and from peripheral devices via a wired network or a wireless network (information communication network).

Each of the auxiliary storage unit <NUM> and the auxiliary storage unit <NUM> is a non-transitory tangible storage medium. The non-transitory tangible storage medium is, for example, a magnetic disk, a magneto-optical disk, a compact disk read only memory (CD-ROM), a digital versatile disk read only memory (DVD-ROM), or a semiconductor memory.

Each of the input unit <NUM> and the input unit <NUM> has a function of inputting data and processing instructions. Each of the input unit <NUM> and the input unit <NUM> is an input device, such as a keyboard or a mouse.

Each of the output unit <NUM> and the output unit <NUM> has a function of outputting data. Each of the output unit <NUM> and the output unit <NUM> is, for example, a display device, such as a liquid crystal display device, or a printing device, such as a printer.

In addition, as shown in <FIG>, the constituent elements of the image processing device <NUM> are connected to a system bus <NUM>. In addition, as shown in <FIG>, the constituent elements of the learning device <NUM> are connected to a system bus <NUM>.

The auxiliary storage unit <NUM> stores, for example, a program for implementing the earth observation means <NUM>, the change detection means <NUM>, the metadata extraction means <NUM>, and the data-set generation means <NUM> shown in <FIG>. The main storage unit <NUM> is used, for example, as a storage region of the satellite image DB <NUM>.

Note that, the image processing device <NUM> may be implemented by hardware. For example, the image processing device <NUM> may have a circuit including a hardware component such as a large scale integration (LSI) incorporating a program for implementing the functions as shown in <FIG>.

The image processing device <NUM> may be implemented by software by executing, by the CPU <NUM> shown in <FIG>, the program which provides the functions of constituent elements shown in <FIG>.

In the case of being implemented by software, the CPU <NUM> loads the program stored in the auxiliary storage unit <NUM> in the main storage unit <NUM> and executes the program to control the operation of the image processing device <NUM>, whereby the functions are implemented by software.

The auxiliary storage unit <NUM> stores, for example, a program for implementing the model-parameter computation means <NUM> and the machine learning means <NUM> shown in <FIG>.

Note that, the learning device <NUM> may be implemented by hardware. For example, the learning device <NUM> may have a circuit including a hardware component such as an LSI incorporating a program for implementing the functions as shown in <FIG>.

The learning device <NUM> may be implemented by software by executing, by the CPU <NUM> shown in <FIG>, the program which provides the functions of constituent elements shown in <FIG>.

In the case of being implemented by software, the CPU <NUM> loads the program stored in the auxiliary storage unit <NUM> in the main storage unit <NUM> and executes the program to control the operation of the learning device <NUM>, whereby the functions are implemented by software.

In addition, a part of or all of the constituent elements may be implemented by a general purpose circuitry, a dedicated circuitry, a processor, or the like, or a combination thereof. These may be constituted by a single chip, or by a plurality of chips connected via a bus. A part of or all of the constituent elements may be implemented by a combination of the above circuitry or the like and a program.

In the case in which a part of or all of the constituent elements are implemented by a plurality of information processing devices, circuitries, or the like, the information processing devices, circuitries, or the like may be arranged in a concentrated manner, or dispersedly. For example, the information processing devices, circuitries, or the like may be implemented as a form in which each is connected via a communication network, such as a client-and-server system or a cloud computing system.

Next, an outline of the present invention will be described. <FIG> is a block diagram showing an outline of the learning device according to the present invention. A learning device <NUM> according to the present invention includes a learning means <NUM> (for example, the machine learning means <NUM>) that, by using learning data including at least a set of image areas representing a periodic change of a predetermined object among changes between a plurality of images and a set of image areas representing a change other than the periodic change among the changes between the plurality of images, causes a detector to learn a process for detecting a change other than the periodic change among the changes between the plurality of images.

When a learning device having such a configuration is used, a change only of a detection target is detected among changes between a plurality of images with different photographing times.

The learning device <NUM> further includes a computation means (for example, the model-parameter computation means <NUM>) that computes a parameter representing the periodic change of the predetermined object on the basis of data indicating photographing conditions of the image areas included in the learning data, and the learning means <NUM> may cause the detector to learn using the computed parameter and the learning data.

When a learning device having such a configuration is used, a periodic change of a predetermined object is expressed more concretely.

Alternatively, the parameter may be a solar zenith angle. Alternatively, the parameter may be a direct light component of the sunlight spectrum and a scattered light component of the sunlight spectrum.

When a learning device having such a configuration is used, a change of the length of a shadow is excluded from a detection target among changes between a plurality of images.

Alternatively, the parameter may be a vegetation index.

When a learning device having such a configuration is used, a seasonal change of plants is excluded from a detection target among changes between a plurality of images.

Alternatively, the parameter may be a solar azimuth angle.

When a learning device having such a configuration is used, a change of the direction in which a shadow is formed is excluded from a detection target among changes between a plurality of images.

<FIG> is a block diagram showing an outline of the image processing device according to the first example. An image processing device <NUM> according to the present example includes a first generation means <NUM> (for example, the change detection means <NUM>) that generates change information indicating, for each pixel constituting an image, a plurality of feature values indicating the degree of a change in which a periodic change of a predetermined object is removed from changes between a plurality of images, and reliability information indicating, for each pixel, reliability of each of the plurality of feature values, an extraction means <NUM> (for example, the data-set generation means <NUM>) that extracts, from the plurality of images, an area including a pixel corresponding to a feature value whose reliability indicated by the generated reliability information is equal to or greater than a predetermined value and extracts, from the generated change information, a feature value equal to or greater than the predetermined value, and a second generation means <NUM> (for example, the data-set generation means <NUM>) that generates learning data including each extracted area, the extracted feature value equal to or greater than the predetermined value, and data indicating a photographing condition of each of the plurality of images associated with each other.

When an image processing device having such a configuration is used, a change only of a detection target is detected among changes between a plurality of images with different photographing times.

<FIG> is a block diagram showing another outline of the image processing device according to the first example. An image processing device <NUM> according to the present example includes a parameter computation means <NUM> (for example, the model-parameter computation means <NUM>) that computes, on the basis of data indicating a photographing condition of each of a plurality of images, a parameter representing a periodic change of a predetermined object displayed in the plurality of images, a feature-value computation means <NUM> (for example, the feature-value computation means <NUM>) that computes, using the computed parameter and the plurality of images, a feature value indicating the degree of a change in which the periodic change is removed from changes between the plurality of images, and a reliability computation means <NUM> (for example, the reliability computation means <NUM>) that computes reliability of the computed feature value.

Claim 1:
A learning device comprising:
a learning means (<NUM>) configured to, by using learning data including at least a set of image areas representing a periodic change of shadow or plants among changes between a plurality of satellite images photographed at the same point at different times and a set of image areas representing a change other than the periodic change among the changes between the plurality of satellite images, cause a detector (<NUM>) to learn a process for detecting a change other than the periodic change among the changes between the plurality of satellite images; and
characterised by:
a computation means (<NUM>) configured to compute a parameter representing the periodic change of shadow or plants on the basis of data indicating photographing conditions of the image areas included in the learning data, wherein
the learning means (<NUM>) is configured to cause the detector (<NUM>) to learn using the computed parameter and the learning data.