Image processing method, image processing apparatus, imaging system, and storage medium, for content concentration estimation

An image processing method includes acquiring an image obtained by imaging of an object, ambient light data during the imaging, and reflection characteristic data and transmission characteristic data which depend on a concentration of a material contained in the object, and separating a reflected light component and a transmitting light component in the image using the ambient light data, the reflection characteristic data, and the transmission characteristic data.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing method that estimates a concentration of a material included in an object from a spectral image.

Description of the Related Art

There has recently proposed a method for estimating a leaf color index using the remote sensing technology for efficient and labor-saving agricultural work. For example. Patent Document 1 discloses a method of estimating a SPAD (Soil & Plant Analyzer Development) value from a spectral measurement result of light reflected by a plant.

The method disclosed in Japanese Patent Laid-Open No. (“JP”) 2002-presumes that the light that has reached a light receiver is only the light reflected by the plant. However, the leaf of the plant is a semitransparent object and thus transmitting light transmitting through the leaf of the plant also reaches the light receiver as well as the reflected light reflected by the leaf of the plant. In addition, a mixture ratio of the reflected light and the transmitting light changes according to the weather (sunny, cloudy, etc.) and the position of the sun (altitude and azimuth). Therefore, the method disclosed in JP 2002-168771 has difficulty in highly accurately estimate the concentration of the material (leaf color, that is, the SPAD value), because the mixture ratio of the reflected light and the transmitting light changes as the weather or the sun position changes.

SUMMARY OF THE INVENTION

The present invention provides an image processing method, an image processing apparatus, an imaging system, and a program, each of which can highly accurately estimate a concentration of a material contained in an object from a spectral image.

An image processing method according to one aspect of the present invention includes acquiring an image obtained by imaging of an object, ambient light data during the imaging, and reflection characteristic data and transmission characteristic data which depend on a concentration of a material contained in the object, and separating a reflected light component and a transmitting light component in the image using the ambient light data, the reflection characteristic data, and the transmission characteristic data.

An image processing apparatus according to another aspect of the present invention includes at least one processor or circuit configured to execute a plurality of tasks including an acquiring task of acquiring an image obtained by imaging of an object, ambient light data during the imaging, and reflection characteristic data and transmission characteristic data which depend on a concentration of a material contained in the object, and a separating task of separating a reflected light component and a transmitting light component in the image using the ambient light data, the reflection characteristic data, and the transmission characteristic data.

An imaging system according to another aspect of the present invention includes an image capturer configured to capture an object, a detector configured to detect ambient light data when the object is captured by the image capturer, and the image processing apparatus.

A non-transitory computer-readable storage medium storing a program according to another aspect of the present invention causes a computer to execute the image processing method.

DESCRIPTION OF THE EMBODIMENTS

Referring now to the accompanying drawings, a detailed description will be given of embodiments according to the present invention. Corresponding elements in respective figures will be designated by the same reference numerals, and a duplicate description thereof will be omitted.

First Embodiment

Referring now toFIG.2, a description will be given of a camera captured model according to a first embodiment of the present invention.FIG.2is an explanatory diagram of a camera captured model according to this embodiment.FIG.2illustrates an object140to be captured by a camera (image pickup apparatus)200. The object140is illuminated by ambient light incident on the object140from each position on a hemispherical space in which the object140is placed. InFIG.2, the ambient light is divided into ambient light120that illuminates the back surface of the object140and ambient light130that illuminates the front surface of the object140.

The object140is a semitransparent object, the ambient light120is diffused and absorbed inside the object140, and part of transmitting light (transmitting light121) reaches the camera200. On the other hand, the ambient light130is diffused and absorbed inside the object140, and part of the reflected light (reflected light131) reaches the camera200. In this embodiment, the so-called diffuse reflected light reflected by such a process will be simply referred to as reflected light, and is distinguished from regular reflection light reflected on the surface of the object140.

Therefore, the image of the object140captured by the camera200is formed by a mixture of transmitting light (transmitting light component)121and reflected light (reflected light component)131at a specific ratio. The mixture ratio of the transmitting light121and the reflected light131changes as the ambient lights120and130fluctuate. In particular, in imaging under outdoor ambient light, the mixture ratio of the transmitting light121and the reflected light131changes depending on the weather (sunny, cloudy, etc.) and the sun position (altitude and azimuth).

When the image is captured under the outdoor ambient light in this way, the mixture ratio of the transmitting light121and the reflected light131is unknown and changes depending on the illumination environment during imaging. Therefore, it is difficult to quantitatively estimate the concentration of the material (chlorophyll or the like) contained in the object from the image captured in such a state. This embodiment can quantitatively estimate the concentration of the material contained in the object by separating the transmitting light121and the reflected light131(separation processing) from the spectral image captured under such ambient light. Hereinafter, the separation processing will be described in detail.

First, this embodiment formulates the camera captured model illustrated inFIG.2, for example, as in the expression (1).
In=IR,n+IT,n=kRIR0,nRn(c)+kTIT0,nTn(c)  (1)

In the expression (1), Inrepresents a luminance value of an image (spectral image) captured by the camera200, and a subscript n represents a wavelength number of the spectral image. For example, when the camera200is an RGB camera, n={1, 2, 3}, and I1, I2, and I3each indicate RGB luminance values. IR, n, and IT, nrepresent luminance values when the reflected light131and the transmitting light121are acquired independently. IR0, nand IT0, nrepresent the illuminances of the ambient lights130and120that illuminate the object140, respectively. Rn(c) and Tn(c) represent spectral reflection characteristics (reflection characteristic data) and spectral transmission characteristics (transmission characteristic data), respectively, depending on the concentration c of the material contained in the object140. In this embodiment, each of the spectral reflection characteristic Rn(c) and the spectral transmission characteristic Tn(c) has previously been stored as known library data in a storage device such as a memory. kRrepresents a ratio of the ambient light130reflected by the object140and reaching the camera200, and kTrepresents a ratio of the ambient light120passing through the object140and reaching the camera200.

The illuminance information IR0, n, and the illuminance information IT0, nof the ambient lights130and120during imaging are known and acquired, for example, by the ambient light information acquirer (detector)110illustrated inFIG.2. In this embodiment, the ambient light information acquirer110includes an ambient light sensor (first ambient light sensor)111and an ambient light sensor (second ambient light sensor)112installed in two different directions. The ambient light sensor111acquires the illuminance information IT0, nof the ambient light120. The ambient light sensor112acquires the illuminance information IR0, nof the ambient light130.

This embodiment acquires the ratio kRof the reflected light (reflected light component), the ratio kTof the transmitting light (transmitting light component), and the concentration c of the material contained in the object140by performing the optimization (optimization calculation) represented by the following expression (2) using the camera captured model formulated in this way.

In this expression (2), “∥ ∥ 2” represents the L2 norm. The separation processing according to this embodiment means the execution of the optimization calculation of the expression (2), but the present invention is not limited to this example. With kR, kT, and c obtained by the optimization, the reflected light (reflected light component) and the transmitting light (transmitting light component) are separated and expressed as illustrated in the following expressions (3) and (4), respectively.
IR,n=kRIR0,nRn(c)  (3)
IT,n=kTIT0,nTn(c)  (4)

Therefore, this embodiment can separate the transmitting light121and the reflected light131from the spectral image in which the transmitting light121and the reflected light131are mixed at an unknown mixture ratio. Further, this embodiment can quantitatively estimate the concentration of the material contained in the object140.

This embodiment relates to an image processing system (imaging system) that estimates the concentration of the object (paddy rice leaf)140from the image (spectral image) acquired by the camera (RGB camera)200. In this embodiment, the concentration c in the expression (2) corresponds to the SPAD value.

Referring now toFIG.1, a description will be given of the configuration of the image processing system100according to this embodiment.FIG.1is a block diagram of the image processing system100. The image processing system100includes an image capturer101, an image processor (image processing apparatus)102, a controller103, a memory104, a communicator105, a display unit106, and an ambient light information acquirer110. The image capturer101includes an imaging optical system101aand an image sensor101b. The image sensor101bphotoelectrically converts an optical image (object image) formed via the imaging optical system101aand outputs an image (image data) to the image processor102. The image processor102has an acquiring means (task)102aand a separating means (task)102b.

The image processing system100can be provided inside the camera200. Alternatively, some functions such as the image processor102of the image processing system100may be implemented in a computer (user PC) away from the camera200or on cloud computing. In this case, the camera200has only part of the image processing system100including the image capturer101.

FIG.2illustrates the object140that is imaged under outdoor ambient light using the image processing system100. As illustrated inFIG.2, the ambient light information acquirer110of the image processing system100includes an ambient light sensor111configured to acquire the illuminance incident on the back surface of the object140, and an ambient light sensor112configured to acquire the illuminance incident on the front surface of the object140.

Referring now toFIG.3, a description will be given of an image processing method according to this embodiment.FIG.3is a flowchart of the image processing method according to this embodiment. Each step inFIG.3is mainly executed by the acquiring means102aor the separating means102bin the image processor102.

First, in the step S201, the image capturer101in the image processing system100images the object140by the signal from the controller103and acquires an RGB image (spectral image). Then, the acquiring means102ain the image processor102acquires the image captured by the image capturer101. At the same time as the step S201, in the step S202, the ambient light information acquirer110(ambient light sensors111and112) acquires (detects), based on the signal from the controller103, the ambient light information (ambient light data) IR0, nand IT0, n. In this embodiment, the ambient light information is information on the tint. Then, the acquiring means102aacquires the ambient light information IR0, nand IT0, ndetected by the ambient light information acquirer110.

The ambient light sensors111and112is made by disposing a diffuser on a sensor having the same spectral sensitivity characteristic as that of the image capturer101, and acquire ambient light information having the same spectral wavelength as that of the image capturer101. The ambient light sensors111and112may include a spectroradiometer, and acquire the ambient light information IR0, nand IT0, nusing the following expressions (5) and (6) with an acquired spectral irradiance E(λ), a spectral transmittance characteristic L(λ) of the imaging optical system, and a spectral sensitivity characteristic Sn(λ) of the image sensor.
IT0,n=∫λn,1λn,2ET(λ,c)L(λ)Sn(λ)dλ(5)
IR0,n=∫λn,1λn,2ER(λ,c)L(Δ)Sn(λ)dλ(6)

In the expressions (5) and (6), ET(λ) is the illuminance of the ambient light120, ER(λ) is the illuminance of the ambient light130, and λn, 1and λn, 2are the shortest wavelength and the longest wavelength, respectively, in the wavelength band in which the image sensor101bhaving the spectral sensitivity characteristic Sn(λ) has sensitivity.

Next, in the step S210, the separating means102bin the image processor102performs separation processing based on the expression (2). Referring now toFIG.4, a description will be given of the separation processing according to this embodiment.FIG.4is a flowchart of the separation processing according to this embodiment. Each step inFIG.4is mainly executed by the separating means102bin the image processor102.

First, in the step S211the separating means102bperforms an optimization calculation based on the expression (2). Next, in the step S212, the separating means102bcalculates the reflected light component and transmitting light component of the expressions (3) and (4). In the steps S211and S212, the separating means102butilizes the reflection characteristic data and the transmission characteristic data of the object that have been previously stored.

Referring not toFIGS.5A and5B, a description will be given of the reflection characteristic data and the transmission characteristic data in this embodiment.FIGS.5A and5Bare explanatory diagrams of reflection characteristic data and transmission characteristic data.FIG.5Ais data showing SPAD value dependencies on the RGB reflectance and the RGB transmittance that have been acquired in advance. A dot plotted point represents the RGB reflectance, a triangularly plotted point represents the RGB transmittance, and a filled color corresponds to the SPAD value. This embodiment separates, as illustrated inFIG.5A, the reflected light and the transmitting light utilizing the fact that the spectral reflection characteristic and the spectral transmission characteristic have different characteristics from each other.

FIG.5Bplots the results of fitting the RGB reflectance and RGB transmittance ofFIG.5Ainto the following expressions (7) and (8) by the least squares method using the SPAD value c as a parameter.
Rn(c)=Σi=0Nan,i·ci(7)
Tn(c)=Σi=0Nbn,i·ci(8)

In expressions (7) and (8), an, iand bn, iare constants determined by the least squares method. InFIG.5B, similar toFIG.5A, a dot plotted point represents the RGB reflectance, and a triangularly plotted point represents the RGB transmittance.

This embodiment has stored information on the expressions (7) and (8) as the reflection characteristic data and the transmission characteristic data of the object, respectively. The reflection characteristic data and the transmission characteristic data are not limited to the above data. For example, as illustrated inFIGS.6A and6B, the reflectance characteristic Rfl(c, λ) and the transmittance characteristic Trs(c, λ) of the object may be used.FIGS.6A and6Bis an explanatory diagram of other reflection characteristic data and transmission characteristic data according to this embodiment.FIG.6Aillustrates a spectral reflectance characteristic of the paddy rice leaf, where the abscissa axis represents a wavelength and the ordinate axis represents a reflectance. The color of each line corresponds to the SPAD value shown on the color bar. Similarly,FIG.6Billustrates the transmittance characteristic of paddy rice, where the abscissa axis represents a wavelength and the ordinate axis illustrates a transmittance.

When the reflection characteristic data and the transmission characteristic data as illustrated inFIGS.6A and6Bare used, Rn(c) and Tn(c) are preferably calculated using the following expressions (9) and (10) with the spectral transmittance characteristic L(λ) of the imaging optical system101aand the spectral sensitivity characteristic Sn(λ) of the image sensor101b.
Rn(c)=Σλn,1λn,2Rfl(c,λ)L(λ)Sn(λ)dλ(9)
Tn(c)=∫λn,1λn,2Trs(c,λ)L(λ)Sn(λ)dλ(10)

In expressions (9) and (10), λn, 1and λn, 2are the shortest wavelength and the longest wavelength, respectively, in the wavelength band in which the image sensor101bhaving the spectral sensitivity characteristic Sn(λ) has sensitivity.

In the step S211the separating means102bperforms the optimization calculation of the expression (2) using the spectral image, the ambient light information, and the reflection characteristic data and the transmission characteristic data of the object140stored in the memory104. The optimization calculation can use a known optimization method, such as a gradient method. As illustrated in the expressions (7) and (8), when Rn(c) and Tn(c) are differentiable functions, the expression (2) is also a differentiable function and thus a faster optimization calculation method such as the Newton method and the trust region method can be used.

Referring now toFIGS.7A to7E, a description will be given of results of the separation processing by the optimization calculation.FIGS.7A to7Eshow the results of the separation processing according to this embodiment and the result of performing the optimization calculation using a trust region method in the step S211inFIG.4.FIG.7Ais an image made by converting an RGB image of paddy rice captured by an RGB camera into a grayscale image.FIGS.7B to7Dare diagrams showing the results of optimizing calculations for the ratio kRof the reflected light, the ratio kTof the transmitting light, and the concentration c of the material for each pixel of the paddy rise leaf.FIG.7Eis a diagram showing a value f of the optimization evaluation function of the expression (2) on a logarithmic scale.

In the step S212inFIG.4, the separating means102bcalculates the reflected light component and the transmitting light component based on the expressions (3), (4), (7), and (8) from the ratio kRof the reflected light, the ratio kTof the transmitting light, and the material concentration c obtained in the step S211. As described above, this embodiment can separate the reflected light component and the transmitting light component from the spectral image of the object. In the processing of calculating the expression (2), the material concentration c of the object140can be estimated.

Second Embodiment

Next follows a description of a second embodiment according to the present invention. This embodiment estimates a SPAD value of a rice leaf from the spectral image acquired by the RGB camera in the same manner as in the first embodiment.

Referring now toFIGS.8and9, a description will be given of a configuration of an image processing system300and the camera captured model according to this embodiment.FIG.8is a block diagram of the image processing system300.FIG.9is an explanatory diagram of the camera captured model, and illustrates the object140that is captured under outdoor ambient light using the image processing system300. The image processing system300according to this embodiment is different from the image processing system100according to the first embodiment having an ambient light information acquirer110in that the image processing system300includes an ambient light information acquirer (detector)310. As illustrated inFIGS.8and9, the ambient light information acquirer310exclusively includes a single ambient light sensor113. Since the other configuration of the image processing system300is the same as that of the image processing system100, a description thereof will be omitted.

Next follows a description of the reason for using only one ambient light sensor113in this embodiment.FIG.10is an explanatory diagram of the ambient light information acquiring model in this embodiment, and illustrates the arrangement of the ambient light sensor111arranged westward and the ambient light sensor112arranged eastward. The ambient light sensors111and112have a configuration in which a diffuser plate is attached to an RGB color sensor.

FIGS.11A and11Bare explanatory diagrams of a time change of the ambient light information, and plots the ambient light information acquired by the ambient light sensors111and112. InFIGS.11A and11B, WBR, b, WBR, r, WBT, b, and WBT, rrare white balance correction coefficients calculated using the following expressions (11) to (14) with the illuminance information IR0, nacquired by the ambient light sensor112and the illuminance information IT0, nacquired by the ambient light sensor111. n={1, 2, 3}, indicating that the values are acquired by the color sensors of R, G, and B in this order.
WBR,b=IR0,2/IR0,3(11)
WBR,r=IR0,2/R0,1(12)
WBT,b=IT0,2/IT0,3(13)
WBT,r=IT0,2/IT0,1(14)

FIG.11Ais a diagram that plots the time change of the white balance correction coefficient when it is cloudy. A black dot represents WBR, b, a white dot represents WBT, b, a black square represents WBR, r, and a white square represents WBT, r. As illustrated inFIG.11A, when it is cloudy, WBR, band WBT, bare equal to each other and WBR, rand WBT, rare equal to each other regardless of the arrangement orientation of the ambient light sensor.FIG.11Bis a diagram that plots the time change of the white balance correction coefficient when it is sunny in the same manner as inFIG.11A. When it is sunny. WBR, band WBT, bare equal to each other and WBR, rand WBT, rare equal to each other at only around midday when the sun crosses the meridian.

Accordingly, this embodiment captures a spectral image when the white balance correction coefficient does not depend on the arrangement orientation of the ambient light sensor (such as within 2 hours before and after the sun crosses the meridian). Thereby, even the single ambient light sensor113can execute the separation processing of the reflected light and the transmitting light.

At such a time (such as within 2 hours before and after the sun crosses the meridian), the white balance correction coefficient does not depend on the orientation of the ambient light sensor. Therefore, the ambient light sensor113according to this embodiment is installed upwardly, for example, as illustrated inFIG.9, and can acquire the ambient light information. Where IR0, nis the ambient light information acquired by the ambient light sensor113and IT0, c=m·IR0, c(m is a proportional constant), the expression (2) for the optimization calculation can be transformed as in the expression (15).

In the expression (15), k′T=m·kT, and Rn(c) and Tn(c) use the data of the expressions (7) and (8). The method of acquiring the ambient light information IR0, nis not limited to the above method, and as illustrated inFIG.12, a standard reflective plate114is imaged by the camera200, and the ambient light information IR0, nmay be acquired from the pixel values of the image of the captured standard reflective plate.FIG.12is an explanatory diagram of a method of acquiring ambient light information using the standard reflective plate114.

Referring now toFIG.13, a description will be given of the image processing method according to this embodiment.FIG.13is a flowchart of the image processing method (SPAD value estimating method) according to this embodiment. Each step inFIG.13is mainly executed by the acquiring means102aor the separating means102bin the image processor102.

First, in the step S401, the image capturer101in the image processing system300images the object140in response to the signal from the controller103and acquires an RGB image (spectral image). Then, the acquiring means102ain the image processor102acquires the image captured by the image capturer101. At the same time as the step S401, in the step S402, the ambient light information acquirer310(ambient light sensor113) acquires (detects) the ambient light information (ambient light data) IR0, nwhen the image is captured in response to the signal from the controller103. Then, the acquiring means102aacquires the ambient light information IR0, ndetected by the ambient light information acquirer310.

Next, in the step S403, the image processor102extracts the captured area (object area) of the paddy rice as the object140. As a method for extracting the paddy rice area, for example, an image may be generated by converting an RGB image into an HSV color space, and pixels within a range of hue angles that can be taken by the paddy rice leaves may be extracted as the paddy rice area.

Next, in the steps S405and S406, the separating means102bin the image processor102performs the separation processing. First, in the step S404, the separating means102bperforms an optimization calculation based on the expression (15) for each pixel of the paddy rice area extracted in the step S403. Next, in the step S405, the separating means102bcalculates the reflected light component I′R, nwhose ambient light component is corrected, using the following expression (16) with the ratio kRof the reflected light and the concentration c calculated in the step S404.
I′R,n=IR,n/IR0,n=kRRn(c)  (16)

Next, in the step S406, the image processor102calculates NGRDI (Normalized Green Red Difference Index) as an index (growth index) that correlates with the SPAD value. NGRDI is calculated based on the following expression (17) using the reflected light component calculated in the step S405.
NGRDI=(I′R,2−I′R,1)/(I′R,2+I′R,1)  (17)

Finally, in the step S407, the image processor102converts NGRDI into a SPAD value using the following expression (18), which is a correlation expression between NGRDI and the SPAD value.
SPAD value=Σi=0NdiNGRDIi(18)

In the expression (18), diis a constant representing a correlation between NGRDI and the SPAD value.

In the separation processing according to this embodiment, the material concentration c (corresponding to the SPAD value) is calculated by the optimization calculation of the expression (15), but the calculated material concentration c and the ratio kRof the reflected light contain errors. Therefore, this embodiment performs processes of the steps S405to S407in order to estimate the material concentration with more redundancy.

The method according to this embodiment can improve the estimation accuracy of the SPAD value by using to estimate the SPAD value the optimization calculation result of only pixels having values f of the optimization evaluation function of the expression (15) are equal to or less than a threshold fth.FIG.14is an explanatory diagram of a difference in estimation accuracy of the SPAD value (material concentration) with respect to the threshold fth. InFIG.14, the abscissa axis represents the threshold fth, and the ordinate axis represents the root mean square error RMSE between the average value and the correct value of the SPAD value estimation results of the pixels equal to or smaller than the threshold fth. As illustrated inFIG.14, the estimation accuracy of the SPAD value can be improved by properly setting the threshold fth.

FIG.15is an explanatory diagram of the SPAD value estimation result, and illustrates the result of estimating the daily change of the SPAD value from the RGB image captured by a fixed-point camera. InFIG.15, the abscissa axis represents the date and the ordinate axis represents the SPAD value. InFIG.15, a squarely plotted point represents an estimation result when the separation processing according to this embodiment is not performed in the steps S404and S405, and a dot plotted point represents an estimation result when the separation processing is performed. An asterisk plotted point represents a correct value, and an average value of the results measured by the SPAD meter is adopted as the correct value for 10 strains of paddy rice in the image estimation area. An error bar represents the standard deviation. Therefore, as illustrated inFIG.15, the separation processing according to this embodiment can quantitatively estimate the material concentration.

Other Embodiments

Thus, in each embodiment, the image processing apparatus (image processor102) has the acquiring means102aand the separating means102b. The acquiring means acquires the image of the object (spectral image), the ambient light data (information on the tint) when the object is imaged, and the reflection characteristic data (Rn(c)) and transmission characteristic data (Tn(c)) that depend on the concentration (SPAD value) of the material (chlorophyll, etc.) contained in the object. The separating means separates the reflected light component (IR, n) and the transmitting light component (IT, n) from the image using the image, the ambient light data, the reflection characteristic data, and the transmission characteristic data. Thereby, the image processing apparatus according to each embodiment can separate the reflected light component and the transmitting light component from the spectral image obtained by imaging the semitransparent object. Therefore, each embodiment can provide an image processing method, an image processing apparatus, an imaging system, and a program, each of which can highly accurately estimate the concentration of a material contained in an object from a spectral image.

The present invention can provide an image processing method, an image processing apparatus, an imaging system, and a program, each of which can highly accurately estimate a concentration of a material contained in an object from a spectral image.

In particular, in each embodiment, a paddy rice leaf were taken as an example as an object, but it can also be applied to another object. In each embodiment, the RGB spectral image is captured as an example, but each embodiment is also applicable to a multiband image and a hyperspectral image having four or more spectral wavelengths. In each embodiment, the image capturer and the ambient light information acquirer are separated from each other, but the image capturer and the ambient light information acquirer may be integrated with each other. The evaluation function for the optimization is not limited to the expressions (2) and (15) and, for example, the L1 norm may be used instead of the L2 norm. Each embodiment illustratively executes the optimization calculation in the image processing apparatus, but the optimization calculation with heavy processing may be executed on the cloud computing.