Biological Information Detection Apparatus and Biological Information Detection Method

A biological information detection apparatus includes: a camera; a frame image analysis unit that detects a region including pixels having a predetermined skin color, as a skin color region, from a frame image taken using the camera, and detects a signal corresponding to a light wavelength from an image signal of each pixel included in the skin color region, as skin color wavelength data; a skin color wavelength difference detection unit that calculates an average value of differences of the skin color wavelength data from predetermined reference wavelength data for the pixels included in the skin color region, and acquires the average value as average wavelength difference data; a pulse wave signal detection unit that detects a signal obtained by smoothing the average wavelength difference data detected in time series, as a pulse wave signal.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a biological information detection apparatus and a biological information detection method that detect the dynamic state of a living body in a noncontact manner in real time.

2. Description of the Related Art

In recent years, attention has been directed toward the technique of detecting the dynamic state of a living body in a noncontact manner in real time using a microwave or a camera. For example, there is a technique of detecting a heart rate and so forth of a subject from a temporal change in a face image of the subject taken using a camera. With miniaturization of a camera module, this technique has been applied to portable terminals such as smart phones and has been rapidly widespread. Further, this technique has been evolved into a technique of measuring blood pressure of a subject in real time by use of a smart phone or the like.

For example, Patent document 1 discloses the technique of spectrally analyzing RGB time-series signals in a region of interest in a subject's (living body's) image to identify the pulse wave signal originating from the blood vessel in the region of interest. Patent document 2 discloses the technique of detecting a pulse wave signal in each of two sites of a subject from images of the two sites, and finds the pulse wave velocity from the pulse wave signals in the two sites to estimate the blood pressure of the subject. A known example of the method of estimating the blood pressure from the pulse wave velocity is a method using the Moens-Korteweg equation (Refer to non-patent document 1).

CITATION LIST

Patent Documents

According to the inventions described in Patent document 1 and Patent document 2, the pulse wave signal is basically acquired by spectrally analyzing the RGB time-series signals of the pixels in the region of interest in the taken images. The images used for that purpose are taken by detecting RGB light reflected from, for example, a subject's face when the face is illuminated with illuminating light. For this reason, the spectral analysis of the RGB time-series signals means spectral analysis of the time series change in the intensity of three colors RGB in reflected light.

Accordingly, when light is steadily applied onto the subject's face as the region of interest, the pulse wave signal can be stably detected. However, when the intensity of illuminating light or natural light varies irregularly, or the shadow on the subject's face vary due to a motion of the subject, the reflected light from the region of interest also largely varies. For this reason, in some cases, the stable pulse wave signals cannot be acquired from the RGB time-series signals of the reflected light.

In this manner, the conventional techniques of detecting biological information such as the heart rate and blood pressure of the subject from the subject's image are susceptible to an environment such as illumination and natural light, and therefore have a problem that the biological information cannot be stably detected.

SUMMARY OF THE INVENTION

In consideration of the above-mentioned problem of the conventional technique, an object of the present invention is to provide a biological information detection apparatus and a biological information detection method that can suppress the influence of an environment such as illuminating light and natural light, and stably detect biological information of a subject.

To achieve the object of the present invention, the biological information detection apparatus of the present invention includes a camera that continuously takes images of a subject at a predetermined time interval; a frame image analysis unit that detects a region including pixels having a predetermined skin color, as a skin color region, from a frame image taken using the camera, and detects a signal corresponding to a light wavelength from an image signal of each pixel included in the skin color region, as skin color wavelength data; a skin color wavelength difference detection unit that calculates an average value of differences of the skin color wavelength data detected by the frame image analysis unit from predetermined reference wavelength data or the skin color wavelength data detected in the frame image preceding the current frame image for the pixels included in the skin color region, and detects the average value as average wavelength difference data; and a pulse wave signal detection unit that detects a signal obtained by smoothing the average wavelength difference data detected in time series, as a pulse wave signal.

The present invention provides a biological information detection apparatus and a biological information detection method that can suppress the influence of an environment such as illuminating light and natural light, and stably detect biological information of a subject.

DETAILED DESCRIPTION OF THE EMBODIMENTS

An embodiment of the present invention will be described below in detail with reference to accompanying figures. Common constituents are given the same reference numerals and description thereof is omitted.

First Embodiment

FIG. 1is a block diagram illustrating an example of a biological information detection apparatus10in a first embodiment. As illustrated inFIG. 1, the biological information detection apparatus10includes a camera100, an image acquisition unit201, a frame image analysis unit220, a reference skin color setting unit205, a skin color wavelength difference detection unit240, a pulse wave signal detection unit260, and a data display unit300.

Here, the biological information detection apparatus10detects a pulse wave signal of a blood flow flowing in a subject's blood vessel from a change of the subject's skin color over time, which is contained in a subject's image taken using the camera100, and acquires or estimates heart rate, blood pressure, and so forth. That is, since it is required to detect the change of the subject's skin color over time (time series change), a digital image camera capable of taking a motion image of, for example, about 30 frames per second is used as the camera100. A subject described herein is a person (human), but may be any animal having a part with less body hair (ex. face) such as a monkey and dog.

Functions of the units constituting the biological information detection apparatus10will be described below. InFIG. 1, the image acquisition unit201receives an image signal101outputted from the camera100for each frame, and outputs an. RGB signal202for each of the pixels constituting the frame.

The frame image analysis unit220receives the RGB signal202of each pixel for each frame, which is outputted from the image acquisition unit201, and outputs a skin color level signal203and a skin color wavelength data signal204for each pixel. Here, the skin color level signal203is a signal indicating that a pixel has a skin color in a predetermined range, and a skin color wavelength data signal204is a signal indicating a value of wavelength of the skin color. With reference toFIG. 4, a condition for determining whether the pixel has the skin color in the predetermined range will be described below. A region including pixels having the skin color in the predetermined range is referred to as a skin color region.

The reference skin color setting unit205sets a value of a reference skin color wavelength data signal206used in the skin color wavelength difference detection unit240. However, a value of the reference skin color wavelength data signal206is set for convenience, and is not limited to any specific value. The value of the reference skin color wavelength data signal206may be, for example, “0”.

The skin color wavelength difference detection unit240receives the skin color level signal203, the skin color wavelength data signal204, and the reference skin color wavelength data signal206of each pixel for each frame. Next, for each of sequentially-taken frames, the skin color wavelength difference detection unit240finds differences between the skin color level signal203of pixels in the skin color region and the reference skin color wavelength data signal206, and then finds an average value of the differences over the pixels in the skin color region. Then, the found average value is outputted as a time-series skin color wavelength difference data signal207.

The pulse wave signal detection unit260uses the time-series skin color wavelength difference data signal207outputted from the skin color wavelength difference detection unit240to generate a pulse wave signal102. Here, the pulse wave signal102corresponds to a blood flow rate, blood pressure or the like in the blood vessel, which changes according to subject's heartbeat. That is, in this embodiment, the pulse wave signal102of the subject can be detected and further the heart rate of the subject can be detected from the pulse wave signal102.

The data display unit300includes a display device such as an LCD (Liquid Crystal Display), and displays the pulse wave signal102and data such as heart rate, which are outputted from the pulse wave signal detection unit260, on the display device.

The functions of the constituents of the biological information detection apparatus10except for the camera100and the data display unit300cart be achieved by a hardware circuit using, for example, a dedicated integrated circuit (FPGA: Field Programmable Logic Array). Alternatively, the functions can be achieved by a computer provided with a processor, a storage device (semiconductor memory, hard disc device, or the like), and an input/output device (keyboard, mouse, display device or the like) However, in this case, the functions of the constituents of the biological information detection apparatus10can be achieved by allowing the processor to execute a predetermined program stored in the storage device.

Subsequently, detailed configuration of the frame image analysis unit220, the skin color wavelength difference detection unit240, and the pulse wave signal detection unit260, which constitute an image processing unit200, will be described.

FIG. 2is a detailed block diagram illustrating an example of the frame image analysis unit220. As illustrated inFIG. 2, the frame image analysis unit220includes an image data storage221, a spatial filter223, an HSV convertor226, a skin color region detector229, and a face detector230.

The image data storage221receives and holds the RGB signal202outputted from the image acquisition unit201(SeeFIG. 1), and outputs a delay REG signal222having a line delay by taps of a convolution kernel of the spatial filter223. The spatial filter223receives the delay RBG signal222, smooths, for example, the. delay RBG signal222of a pixel of interest and surrounding pixels by weighted average calculation or the like, and outputs the smoothed signal as a smoothed RGB signal224.

FIG. 3is a view illustrating an example of the spatial filter223used in the frame image analysis unit220. For example, the spatial filter223in this example applies a convolution kernel (determinant) having3taps in length and width, that is, 3×3 pixels to smoothing processing of each pixel. In this case, 3×3 pixels about the pixel of interest are subjected to convolution using the convolution kernel, and the acquired value becomes the smoothed RGB signal224of the pixel of interest. Components of the determinant of the convolution kernel are, for example, weighted average factors, and may be set as appropriate using average value distribution or Gaussian distribution such that the sum becomes 1.0.

The HSV convertor226(SeeFIG. 2) receives unpack signals225of R (red), G (green), and B (blue) which are unpacked from the smoothed RGB signal224, and converts the unpacked signals225into HSV color space signals including a hue signal204(H), a saturation signal227(S) and a value signal228(V). Although differently named, the skin color wavelength data signal204outputted from the frame image analysis unit220is the same as the hue signal204(H) outputted from the HSV convertor226.

FIG. 4is a view illustrating an example of an HSV color space90and a skin color space900. As illustrated inFIG. 4, the HSV color space90is often represented by cylindrical coordinates. In the cylindrical coordinates, a vertical axis represents value V (Value) indicating color brightness. A radial axis represents saturation S (Saturation) indicating colorfulness. A rotational angle represents hue H (Hue) changing from red to yellow, to green, and finally, purple.

Considering that the hue H is independent from brightness and colorfulness, and the taken image is a detection signal of light, the hue H can be regarded as wavelength of light emitted from each pixel. Thus, in this embodiment, the hue H acquired from each pixel is assumed as a wavelength data signal of the light emitted from each pixel. Similarly, the value V of each pixel can be regarded as the intensity of the light emitted from each pixel.

The skin color space900illustrated inFIG. 4defines a color of human's skin in the HSV color space90. That is, in this embodiment, it is determined whether or not each of values of the hue signal204(H), the saturation signal227(S), and the value signal228(V) of each pixel outputted from the HSV convertor226is included in the skin color space900. When the values are included in the skin color space900, the pixel is regarded as a portion of the human's skin.

Thus, the skin color region detector229receives the hue signal204(H), the saturation signal227(S), and the value signal228(V) from the HSV convertor226, and determines whether or not each of the values of the signals is included in the skin color space900. As a result of the determination, when the value is included in the skin color space900, “1” is outputted as the skin color level signal203, and when the value is not included in the skin color space900, “0” is outputted as the skin color level signal203.

The human's skin color greatly varies depending on individuals, human race, or how to illuminate. Thus, in this embodiment, the user can set the skin color space900.

FIG. 5is a view illustrating an example of a screen displayed for setting the skin color space900. When setting the skin color space900in response to a user's request, the biological information detection apparatus10displays a screen as illustrated inFIG. 5on a predetermined display device. Slide bars401indicating the entire ranges of the hue H, the saturation S, and the value V, and two cursors402that slide along each of the slide bars401are displayed on the screen. The user can freely set the range of the skin color space900by appropriately sliding the cursors402by use of an input device such as mouse (not illustrated).

For example,FIG. 5illustrates the slide bar401for the hue H in the range of 0 degree to 360 degrees. In this slide bar, 0 degrees (360 degrees) indicate red, 120 degrees indicate green, 240 degrees indicate blue, and the hue H in the skin color space900is defined as a region between a color1and a color2. Similarly, for the saturation S, 0% indicates colorless, 100% indicates colorful, and the saturation S in the skin color space900is defined as a region between a saturation1and a saturation2. For the value V, 0% indicates dark color, 100% indicates bright color, the value V in the skin color space900is defined as a region between a value1and a value2.

In the example illustrated inFIG. 5, the ranges of all of the hue H, the saturation S, and the value V in the skin color space900are limited. However, the range of at least the hue H may be limited. In particular, the range of the saturation S may not be limited.

In this manner, the skin color region detector229(SeeFIG. 2) can properly determine whether or not the pixel corresponds to the human's skin portion according to the skin color of the person and the state of illumination. In this specification, a region that is determined as the human's skin portion by the skin color region detector229, and formed of pixels having the skin color level signal203of “1” is referred to as the skin color region.

As described above, the reference skin color setting unit205sets the value of the reference skin color wavelength data signal206, which is used by the skin color wavelength difference detection unit240. The screen illustrated inFIG. 5can be used for this setting. The screen inFIG. 5is used for setting the skin color space900, but the value of the reference skin color wavelength data signal206may be set using the color1and the color2that defines the range of the hue H. In this case, an intermediate value between the color1and the color2may be used as the value of the reference skin color wavelength data signal206.

Referring toFIG. 2, the frame image analysis unit220includes the face detector230. The face detector230cut a facial portion from the frame image to be processed by the frame image analysis unit220. Examples of the method of cutting the facial portion from the frame image include publicly-known Viola-Jones method. In this embodiment, the face detector230cuts the facial portion, and outputs a face-detected region signal231=“1” for each of the pixels of the frame image when the pixel is included in the cut facial portion, and outputs the face-detected region signal231=“0” for each of the pixels of the frame image when the pixel is not included in the cut facial portion.

For the pixel having the face-detected region signal231=“0”, the skin color region detector229outputs “0” as the skin color level signal203without determining whether or not the HSV converted signal of the pixel is included in the skin color space900(SeeFIG. 4). That is, the skin color region detector229detects the skin color region only in the face region cut from the overall frame image by the face detector230.

In this embodiment, the frame image analysis unit220has the face detector230and however, does not need to have the face detector230. In this case, the skin color region detector229also detects the skin color included in the background of the subject, as the skin color region. However, since the skin color region is not the skin color region that changes according to the subject's heartbeat, the skin color region becomes noise for detecting the pulse wave signal102by the biological information detection apparatus10.

Accordingly, the embodiment in which the frame image analysis unit220has the face detector230can detect the pulse wave signal102more accurately than the embodiment in which the frame image analysis unit220does not have the face detector230.

FIG. 6is a detailed block diagram illustrating an example of the skin color wavelength difference detection unit240. As illustrated inFIG. 6, the skin color wavelength difference detection unit240includes a wavelength difference calculator241, a skin color area calculator243, a wavelength difference integrator244, and an average wavelength difference calculator247.

The wavelength difference calculator241receives the skin color level signal203, the skin color wavelength data signal204, and the reference skin color wavelength data signal206of each pixel, and outputs a wavelength difference data signal242set as follows according to the value “1” or “0” of the skin color level signal203That is, when the skin color level signal203is “1”, a value acquired by subtracting the reference skin color wavelength data signal206from the skin color wavelength data signal204is set as the value of the wavelength difference data signal242. When the skin color level signal203is “0”, “0” is set as the value of the wavelength difference data signal242. That is, when the target pixel is the pixel included in the skin color region, a difference between the skin color wavelength data signal204and the reference skin color wavelength data signal206is set as the value of the wavelength difference data signal242, and when the target pixel is not the pixel included in the skin color region, “0” is set as the value of the wavelength difference data signal242.

The skin color area calculator243receives the skin color level signal203indicating that the target pixel is included in the skin color region, counts the number of pixels in the skin color region (skin color level signal203is “1”) in the frame image to be processed, and outputs the count value as a skin color area signal245.

The wavelength difference integrator244receives the wavelength difference data signal242, integrates values of the wavelength difference data signal242for all pixels of the frame image, and outputs the integrated value as an integrated wavelength difference data signal246.

The average wavelength difference calculator247receives the skin color area signal245and the integrated wavelength difference data signal246, and outputs a value obtained by dividing the value of the integrated wavelength difference data signal246by the value of the skin color area signal245, as the skin color wavelength difference data signal207. The skin color wavelength difference data signal207means an average value of the wavelength difference data signal242for all pixels included in the skin color region of the frame image, that is, a change of the average value of the hue H in the skin color region of the subject from a reference value.

The value of the reference skin color wavelength data signal206, which is inputted to the skin color wavelength difference detection unit240, may be “0”. In this case, the skin color wavelength difference data signal207outputted from the average wavelength difference calculator247is acquired by taking an average of the skin color wavelength data signals204over pixels in the skin color region by the number (area) of the skin color region.

FIG. 7is a detailed block diagram illustrating an example of the pulse wave signal detection unit260. As illustrated inFIG. 7, the pulse wave signal detection unit260includes a difference data storage261, a smoothing filter263, a smoothed data storage265, an inclination detector267, a sign data storage269, and extreme value detector271. The pulse wave signal detection unit260generates the pulse wave signal102from the skin color wavelength difference data signal207which is outputted from the skin color wavelength difference detection unit240for every frame, in other words, outputted from the skin color wavelength difference detection unit240over time.

The difference data storage261receives and temporarily stores the skin color wavelength difference data signal207, and outputs a delay skin color wavelength difference data signal262that is the skin color wavelength difference data signal207for some frames preceding the concerned frame. The smoothing filter263receives and smooths the skin color wavelength difference data signal207and the delay skin color wavelength difference data signal262for some frames, that is, outputs a smoothed wavelength difference data signal264obtained by smoothing the skin color wavelength difference data signal207for some frames.

The smoothed wavelength difference data signal264is a signal obtained by smoothing the change (skin color wavelength difference data signal207) of the hue H in the skin color region of the subject in terms of time. The time series change of the hue H in the skin color region of the subject can be regarded as corresponding to a change of the blood flow rate in the blood vessel. Thus, the smoothed wavelength difference data signal264is outputted to the outside as the pulse wave signal102indicating the pulse wave of the blood flow. However, in this embodiment, when the value of the pulse wave signal102is a maximum value or a minimum value, a pulse wave extreme value signal103indicating the maximum value or the minimum value is added to the pulse wave signal102.

Thus, the smoothed data storage265receives the smoothed wavelength difference data signal264, stores values for plural frames, and outputs a smoothed delay wavelength difference data signal266. The smoothed delay wavelength difference data signal266is equivalent to the smoothed wavelength difference data signal264acquired in frames preceding the frame under processing.

The inclination detector267finds a time series change (that is, inclination) of the smoothed wavelength difference data signal264from the smoothed delay wavelength difference data signal266(that is, the smoothed wavelength difference data signal264acquired in frames preceding the concerned frame). Then, a sign of the inclination is outputted as a sign data signal268.

Specifically, the inclination detector267may find the inclination of the smoothed wavelength difference data signal264for two continuous frames, or may find the inclination of the smoothed wavelength difference data signal264obtained by smoothing on average among multiple continuous frames. In the latter case, the inclination detector267may calculate the inclination from an average of the smoothed wavelength difference data signal264for multiple continuous frames, and an average of the smoothed wavelength difference data signal264for multiple previous continuous frames.

The sign data storage269receives the sign data signal268, stores the values of the sign data signal268for multiple frames, and outputs a delay sign data signal270. The delay sign data signal270is equivalent to the sign data signal268acquired in frames preceding the frame under processing.

The extreme value detector271receives the sign data signal268and the delay sign data signal270to find a frame having the inclination sign changed from a positive value to a negative value, or a frame having the inclination sign changed from a negative value to a positive value. This means that the smoothed wavelength difference data signal264at the time when the found frame is obtained changes from an increase to a decrease or from a decrease to an increase, that is, reaches a maximum value or a minimum value.

Thus, the extreme value detector271receives the sign data signal268and the delay sign data signal270, and in the frame having the inclination sign changed from a positive value to a negative value, outputs, for example, “1” as the pulse wave extreme value signal103. In the frame having the inclination sign changed from a negative value to a positive value, the extreme value detector271outputs, for example, “−1” as the pulse wave extreme value signal103. In the frame having the inclination sign kept unchanged, the extreme value detector271outputs, for example, “0” as the pulse wave extreme value signal103.

As described above, in this embodiment, the smoothing filter263smooths the skin color wavelength difference data signal207in terms of time, preventing wrong detection of pulse wave due to a minute change of the skin color wavelength difference data signal207, which is caused by noise and so forth. In this embodiment, the inclination detector267detects a change (inclination) of the smoothed wavelength difference data signal264for adjacent frames, and based on the change (inclination), the extreme value detector271detects a maximum value or a minimum value of difference data. The maximum value or minimum value thus detected is used to count, for example, heart rate.

In the first embodiment described above, the pulse wave signal102is generated based on a change of average hue (H) in pixels determined as skin color among the pixels of the taken face image, that is a change of average wavelength of the skin color. In this case, the influences of the value (V) and the saturation (S) on the pulse wave signal102are eliminated. For this reason, the influences of natural light and shadows are excluded to provide the technique of detecting the pulse wave signal102, which is insusceptible to the environment.

In the first embodiment described above, the camera100is a visible light color camera, and generates an image signal containing three RGB wavelength components. However, this is merely an example, and the camera100may be any camera that can take light reflected from an object (for example, human's face), and output an image signal containing multiple wavelength components For example, at least one of RGB may be included in an infrared or ultraviolet range. To generate such image signal, multiple cameras100may be used.

The camera100may output an image signal containing two wavelength components. For example, when the image signal outputted from the camera100includes only the R signal and the G signal, the generated color space is only the region having the hue (H) in the range of R to G in the HSV color space90illustrated inFIG. 4. However, as long as the skin color space900is included in the region, the above-described processing can be applied.

In the first embodiment, the RGB signal is converted into the signal of the HSV color space90. However, the RGB signal may be converted into a signal of another color space including hue and brightness, such as an HSL (Hue, Saturation, Lightness) color space. In any case, an environment-resistant detection method can be provided by detecting a time series change of light wavelength based on the hue signal of the skin color region. In the case of the HSL color space, lightness (L) is acquired as brightness or intensity of light.

<Modification Example #1 of First Embodiment>

FIG. 8is a block diagram illustrating an example of a biological information detection apparatus10in a modification example #1 of the first embodiment. As illustrated inFIG. 8, the biological information detection apparatus10aincludes the camera100, the image acquisition unit201, the frame image analysis unit220, a skin color wavelength data storage unit205a,the skin color wavelength difference detection unit240, the pulse wave signal detection unit260, and the data display unit300. The configuration of the biological information detection apparatus10ais different from the configuration of the biological information detection apparatus10in the first embodiment (SeeFIG. 1) in that the reference skin color setting unit205in the biological information detection apparatus10is replaced with the skin color wavelength data storage unit205a.

Here, the functions and detailed configuration of the image acquisition unit201and the frame image analysis unit220are the same as those in the first embodiment (SeeFIG. 2and so on) and thus, description thereof is omitted. In this example, the skin color wavelength data storage unit205atemporarily stores the skin color wavelength data signal204of each pixel outputted from the frame image analysis unit220for one or more frames (for example, 3 frames). The skin color wavelength data signal204for the frame preceding the current frame by one or more frames is outputted as a delay skin color wavelength data signal206a.

The functions and detailed configuration of the skin color wavelength difference detection unit240and the pulse wave signal detection unit260are the same as those in the first embodiment (SeeFIG. 6,FIG. 7and so on) and thus, description thereof is omitted. However, this modification example is different from the first embodiment in the signal inputted to the skin color wavelength difference detection unit240.

In the first embodiment described above, the skin color wavelength difference detection unit240(SeeFIG. 6) receives the skin color level signal203, the skin color wavelength data signal204, and the reference skin color wavelength data signal206of each pixel. For pixels in the skin color region, which are identified by the skin color level signal203for each frame image, the skin color wavelength difference detection unit240acquires an average value of the differences between the skin color wavelength data signal204and the reference skin color wavelength data signal206, as the skin color wavelength difference data signal207.

In contrast, in this modification example, the skin color wavelength difference detection unit240receives the skin color level signal203, skin color wavelength data signal204, and the delay skin color wavelength data signal206aof each pixel. For pixels in the skin color region, which are identified by the skin color level signal203for each frame image, the skin color wavelength difference detection unit240acquires an average value of the differences between the skin color wavelength data signal204and the delay skin color wavelength data signal206a,as the skin color wavelength difference data signal207. In this modification example, a reference numeral “206” inFIG. 6denotes the delay skin color wavelength data signal206a.

Here, the skin color wavelength difference data signal207acquired in this modification example can be regarded as a time series change of the average value of the skin color wavelength data signal204of the pixels in the skin color region. In contrast, the skin color wavelength difference data signal207acquired in the first embodiment is a difference from a reference value (the reference skin color wavelength data signal206), as well as an average value of the skin color wavelength data signal204of pixels in the skin color region. Accordingly, the skin color wavelength difference data signal207acquired in this modification example is equivalent to the time-differentiated skin color wavelength difference data signal207in the first embodiment.

The pulse wave signal detection unit260(SeeFIG. 7) outputs the signal (smoothed wavelength difference data signal264) obtained by smoothing the skin color wavelength difference data signal207outputted from the skin color wavelength difference detection unit240by use of the smoothing filter263, as the pulse wave signal102. As described above, the pulse wave signal102in the first embodiment indicates a change of the subject's skin color (hue) that changes according to the blood flow rate that increases/decreases with heartbeat. The pulse wave signal102is a signal represented by a periodic function having a substantially constant cycle, and heart rate or the like as one of biological information of the subject can be easily acquired from the pulse wave signal102.

In this modification example, as in the first embodiment, the pulse wave signal102is obtained by smoothing the skin color wavelength difference data signal207by use of the smoothing filter263. Accordingly, the pulse wave signal102in this modification example is equivalent to a signal acquired by time-differentiating the pulse wave signal102in the first embodiment, and is expressed by a periodic function as in the first embodiment. Thus, also in this modification example, the heart rate or the like as one of biological information of the subject can be easily acquired from the pulse wave signal102as in the first embodiment.

As described above, since the pulse wave signal102acquired in this modification example is acquired based on the skin color wavelength data signal204that represents the hue (H) of each pixel in the skin color region, the influences of the value (V) and the saturation (S) on the pulse wave signal102are eliminated. For this reason, also in this modification example, the in of natural light and shadows are excluded to provide the technique of detecting the pulse wave signal102, which is insusceptible to the environment.

<Modification Example #2 of First Embodiment>

Next, a biological information detection apparatus10bin a modification example #2 of the first embodiment will be described. The entire configuration of the biological information detection apparatus10bin this modification example is the same as the configuration of the biological information detection apparatus10in the first embodiment inFIG. 1(SeeFIG. 1) and thus, illustration thereof is omitted. However, as described with reference toFIG. 9andFIG. 10, detailed configuration of a frame image analysis unit220band a skin color wavelength difference detection unit240bin this modification example is different from the configuration in the first embodiment.

As described below in detail, the biological information detection apparatus10bin this modification example is characterized by suppression of lowering of the detection accuracy and wrong detection for the pulse wave signal102due to rapid variation in natural light.

FIG. 9is a detailed block diagram illustrating an example of the frame image analysis unit220bin the modification example #2 of the first embodiment. As illustrated inFIG. 9, the frame image analysis unit220bin this modification example includes the image data storage221, the spatial filter223, the HSV convertor226, the skin color region detector229, the face detector230, and a signal switch234. The frame image analysis unit220bis basically configured by adding the signal switch234to the frame image analysis unit220in the first embodiment illustrated inFIG. 2. The skin color level signal203is outputted from the signal switch234rather than the skin color region detector229.

The functions of the image data storage221, the spatial filter223, the HSV convertor226, and the face detector230in this modification example are the same as those in the first embodiment. The function of the skin color region detector229is substantially the same as the function in the first embodiment except that an output signal of the skin color region detector229is not the skin color level signal203(SeeFIG. 2), but a skin color detection signal233. However, the skin color level signal203in the first embodiment is substantially the same as the skin color detection signal233in this modification example.

That is, as in the first embodiment, the skin color region detector229in this modification example determines whether or not each of the values of the hue signal204(H), the saturation signal227(S), and the value signal228(V), which are outputted from the HSV convertor226is included in the skin color space900. As a result of this determination, when each value is included in the skin color space900, “1” is outputted as the skin color detection signal233, and when each value is not included in the skin color space900, “0” is outputted as the skin color detection signal233.

The signal switch234receives the value signal228(V) from the HSV convertor226, and the skin color detection signal233from the skin color region detector229. Then, when the value of the skin color detection signal233is “1”, the signal switch234outputs the value signal228(V) from the HSV convertor226as the skin color level signal203b.When the value of the skin color detection signal233is “0”, the signal switch234outputs “0” as the skin color level signal203b.

That is, in this modification example, the value of the skin color level signal203becomes “0” for pixels outside the skin color region, and becomes the value of the value (V) of the pixel for pixels within the skin color region. The skin color level signal203and the skin color wavelength data signal204are outputted from the frame image analysis unit220b.

As in the first embodiment, the face detector230cuts a facial portion from the frame image. When the pixel to be processed is included in the cut facial portion, the face detector230outputs the face-detected region signal231=“1”, and when the pixel to be processed is not included in the cut facial portion, the face detector230outputs the face-detected region signal231=“0”. Then, the skin color region detector229detects the skin color region only in the facial portion of the frame image, which is cut by the face detector230.

FIG. 10is a detailed block diagram illustrating an example of the skin color wavelength difference detection unit240bin the modification example #2 in the first embodiment. As illustrated inFIG. 10, the skin color wavelength difference detection unit240bin this modification example includes the wavelength difference calculator241, a skin color area calculator243b,an area data storage250, the wavelength difference integrator244, an integrated data storage256, and an average wavelength difference calculator247b.

Here, the functions of the wavelength difference calculator241and the wavelength difference integrator244are substantially the same as those in the first embodiment. Accordingly, the wavelength difference calculator241receives the skin color level signal203b,the skin color wavelength data signal204, and the reference skin color wavelength data signal206of each pixel, and outputs the wavelength difference data signal242set as follows according to the value of the skin color level signal203b.That is, when the value of the skin color level signal203is “1”, a value acquired by subtracting the reference skin color wavelength data signal206from the skin color wavelength data signal204is set as the value of the wavelength difference data signal242. When the value of the skin color level signal203is “0”, “0” is set as the value of the wavelength difference data signal242.

The wavelength difference integrator244receives the wavelength difference data signal242, integrates values of the wavelength difference data signal242for all pixels in the concerned frame, and outputs the integrated value as the integrated wavelength difference data signal246.

In contrast, functions of the skin color area calculator243band the average wavelength difference calculator247bare slightly different from the functions of the skin color area calculator243and the average wavelength difference calculator247in the first embodiment.

The skin color area calculator243receives the skin color level signal203representing the value level of the skin color region, counts the number of pixels in the skin color region, that is, the region including no skin color level signal203of “0”, for each frame, and outputs the count value as the skin color area signal245. Further, the skin color area calculator243outputs the inputted skin color level signal203as a value level signal249. The area data storage250receives and stores the skin color area signal245and the value level signal249, and outputs a delay skin color area signal252and a delay value level signal251.

The integrated data storage256temporarily stores values of the skin color wavelength difference data signal207, which are outputted from the average wavelength difference calculator247b,for multiple frames, and outputs a delay integrated skin color wavelength data signal257that is the skin color wavelength difference data signal207for a preceding frame by multiple frames.

The average wavelength difference calculator247breceives the skin color area signal245and the integrated wavelength difference data signal246, and outputs a value obtained by dividing the value of the integrated wavelength difference data signal246by the value of the skin color area signal245, as the skin color wavelength difference data signal207. The function of the average wavelength difference calculator247bis substantially the same as the function of the average wavelength difference calculator247in the first embodiment. However, the average wavelength difference calculator247bin this modification example has following additional functions.

An interframe value level difference signal253inputted to the average wavelength difference calculator247bis a difference between the value level signal249for a concerned frame and the value level signal249(that is, the delay value level signal251read from the area data storage250) for the frame preceding (for example, immediately preceding) the concerned frame. Accordingly, as the interframe value level difference signal253is larger, a change of the value of skin color between frames is larger.

Similarly, an interframe skin color area difference signal254inputted to the average wavelength difference calculator247bis a difference between the skin color area signal245for a concerned frame and the skin color area signal245(that is, the delay skin color area signal252read from the area data storage250) for a frame preceding (for example, immediately preceding) the concerned frame. Accordingly, as the interframe skin color area difference signal254is larger, a change of the area of skin color is larger.

Here, it is assumed that natural light applied to the subject to be processed rapidly changes. In such case, it is considered that the interframe value level difference signal253changes larger than the interframe skin color area difference signal254. In addition, it is considered that the interframe skin color area difference signal254rapidly becomes large.

Thus, in this modification example, the average wavelength difference calculator247breceives the interframe value level difference signal253, the interframe skin color area difference signal254, a value level difference threshold258, and a skin color area difference threshold259in addition to the skin color area signal245and the integrated wavelength difference data signal246. Here, the value level difference threshold258and the skin color area difference threshold259each are a predetermined constant value.

When the interframe value level difference signal253is larger than the value level difference threshold258, the average wavelength difference calculator247h may output the delay integrated skin color wavelength data signal257that is the skin color wavelength difference data signal207for the previous frame (for example, immediately preceding frame), as the skin color wavelength difference data signal207. Alternatively, an average value of the skin color wavelength difference data signal and the delay integrated skin color wavelength data signal257, which are calculated for the concerned frame, may be outputted as the skin color wavelength difference data signal207.

Similarly, when the interframe skin color area difference signal254is larger than the skin color area difference threshold259, the average wavelength difference calculator247bmay output the delay integrated skin color wavelength data signal257that is the skin color wavelength difference data signal for a previous (for example, immediately preceding) frame, as the skin color wavelength difference data signal207. Alternatively, an average value of the skin color wavelength difference data signal and the delay integrated skin color wavelength data signal257, which are calculated for the concerned frame may be outputted as the skin color wavelength difference data signal207.

In this modification example, when the value and the area in the skin color region rapidly changes due to a rapid change of natural light, a rapid change of the skin color wavelength difference data signal207can be suppressed to suppress a rapid change of the pulse wave signal102. Therefore, in this modification example, even in the case of a rapid change of natural light, lowering the detection accuracy and wrong detection about biological information such as heart rate can be suppressed.

Second Embodiment

FIG. 11is a block diagram illustrating an example of a biological information detection apparatus20in accordance with a second embodiment. As illustrated inFIG. 11, the biological information detection apparatus20includes the camera100, the image acquisition unit201, the frame image analysis unit220, a region division unit235, multiple local pulse wave detection units400, a pulse wave velocity calculation unit302, a blood pressure estimation unit320, and the data display unit300. The biological information detection apparatus20estimates a blood pressure value of the subject from a time series change of skin color of the subject in an image taken using the camera100, and displays the estimated blood pressure value on a display device such as LCD via the data display unit300.

The functions of the constituents of the biological information detection apparatus20except for the camera100and the data display unit300can he achieved by a hardware circuit using, for example, a dedicated integrated circuit (FPGA or the like). Alternatively, the functions can he achieved by a computer provided with a processor, a storage device (semiconductor memory, hard disc device, or the like), and an input/output device (keyboard, mouse, display device or the like). However, in this case, the functions of the constituents of the biological information detection apparatus20can be achieved by allowing the processor to execute a predetermined program stored in the storage device.

The functions of the constituents of the biological information detection apparatus20will be described in detail. However, the same constituents as the constituents included in the biological information detection apparatus10in accordance with the first embodiment are given the same reference numerals and description thereof is omitted.

As in the first embodiment, the camera100needs to detect the pulse wave signal based on the time series change of the skin color of the subject, and to estimate blood pressure and therefore, may be a digital video camera capable of taking moving images of about 30 frames per second. Functions and detailed configuration of the image acquisition unit201and the frame image analysis unit220are the same as those in the first embodiment (SeeFIG. 2and so on), and description thereof is omitted.

Accordingly, also in this embodiment, the frame image analysis unit220outputs the skin color level signal203and the skin color wavelength data signal204of each pixel included in the frame image of the target to be processed. Here, the skin color level signal203indicates that the image signal of the concerned, pixel is the signal included in the predetermined skin color space900(SeeFIG. 4), that is, in the skin color region. When the pixel is the signal in the skin color region, the skin color wavelength data signal204is data corresponding to light wavelength of color expressed by the pixel. However, in this embodiment, the hue signal (H) of the concerned pixel is used as the skin color wavelength data signal204.

The region division unit235divides a frame image to be processed into multiple sub-regions501each including, for example, 10×10 pixels (SeeFIG. 14). The region division unit235determines which sub-regions501the skin color level signal203and the skin color wavelength data signal204of each pixel, which are inputted from the frame image analysis unit220, belong to. The region division unit235outputs, as a sub-regional skin color level signal203kand a sub-regional skin color wavelength data signal204kfor each pixel, signals in which a sub-region number for identifying the sub-region501to which the pixel belongs is added to the skin color level signal203and the skin color wavelength data signal204of the pixel.

The sub-regional skin color level signal203kand the sub-regional skin color wavelength data signal204kwith the sub-region numbers, which are outputted from the region division unit235, are classified by the sub-region numbers, and inputted to the local pulse wave detection units400assigned for the sub-region numbers. Accordingly, in this embodiment, the same number of local pulse wave detection units400as the number of the sub-regions501obtained by the region division unit235are prepared.

FIG. 12is a detailed block diagram illustrating an example of the local pulse wave detection unit400in accordance with the second embodiment. As illustrated inFIG. 12, the local pulse wave detection unit400includes the reference skin color setting unit205, the skin color wavelength difference detection unit240, and the pulse wave signal detection unit260. The local pulse wave detection unit400receives the sub-regional skin color level signal203kand the sub-regional skin color wavelength data signal204kof the pixel included in the concerned sub-region, and outputs a sub-regional pulse wave signal102k.Here, the functions and detailed configuration of the reference skin color setting unit205, skin color wavelength difference detection unit240, and the pulse wave signal detection unit260are the same as those in the first embodiment described with reference toFIG. 6andFIG. 7, detailed description thereof is omitted.

However, this embodiment is different from the first embodiment in that the skin color wavelength difference detection unit240of each local pulse wave detection unit400receives only the sub-regional skin color level signal203kand the sub-regional skin color wavelength data signal204kof the pixels in its responsible sub-region501. In summary, the sub-regional pulse wave signal102kto be outputted from the local pulse wave detection unit400is generated for each of the sub-regions501, by using the sub-regional skin color level signal203kand the sub-regional skin color wavelength data signal204kfrom the pixels in the concerned sub-region501. That is, in this embodiment, the sub-regional pulse wave signal102kis not acquired for each frame or facial region, but is acquired for each sub-region501with 10×10 pixels, for example, which is a local part of the frame or region.

FIG. 13is a detailed block diagram illustrating an example of a local pulse wave detection unit400ain a modification example of the second embodiment. As illustrated inFIG. 13, the local pulse wave detection unit400aincludes a skin color wavelength data storage unit205a,the skin color wavelength difference detection unit240, and the pulse wave signal detection unit260. The local pulse wave detection unit400areceives the sub-regional skin color level signal203kand the sub-regional skin color wavelength data signal204k,and outputs the sub-regional pulse wave signal102k.

The local pulse wave detection units400ais configured by replacing the reference skin color setting unit205in the local pulse wave detection units400illustrated inFIG. 12with the skin color wavelength data storage unit205a.That is, the configuration is the same as that in the modification example #1 of the first embodiment. Accordingly, the sub-regional pulse wave signal102koutputted from the local pulse wave detection unit400acorresponds to the signal acquired by time-differentiating the sub-regional pulse wave signal102koutputted from the local pulse wave detection unit400inFIG. 12.

Thus, in the second embodiment, the sub-regional pulse wave signal102kmay be outputted from the local pulse wave detection unit400illustrated inFIG. 12, or may be outputted from the local pulse wave detection unit400illustrated inFIG. 13. Although the second embodiment will be described, below, following description is also applied to the modification example illustrated inFIG. 13.

The pulse wave velocity calculation unit302(SeeFIG. 11) calculates pulse wave velocity based on the sub-regional pulse wave signals102koutputted from the local pulse wave detection units400for the respective sub-regions, and outputs a pulse wave velocity signal303.

FIG. 14is a view illustrating an example of average pulse wave signals102aeach obtained from multiple sub-regions501in a frame image500located at the same vertical position, and the basic concept of calculating the pulse wave velocity. InFIG. 14, the frame image500is represented as a rectangle drawn by a thick solid line. Here, the sub-regions501are multiple regions into which the frame image500is divided and which are drawn by broken lines. An image of a person is displayed in the frame image500, and a skin color region502(shaded portion) is present in the facial portion of the person.

InFIG. 14, the skin color region502refers to a region with pixels having the sub-regional skin color level signal203kof “1”. The sub-regional pulse wave signal102kis generated using the skin color wavelength difference data signal207based on the area (the number of pixels) of the skin color region502included in the sub-regions501and the sub-regional skin color wavelength data signal204kof the pixels in the skin color region502. Accordingly, the sub-regional pulse wave signal102kcannot be acquired from the sub-regions501including no skin color region502. Further, when the area of the skin color region502included in one sub-region501is small, the sub-regional pulse wave signal102kcannot be acquired with high accuracy. Thus, the sub-regional pulse wave signal102kcannot be generated from the sub-regions501when the area ratio of the skin color region502to the sub-regions501is equal to or smaller than 50%, for example (failure of generation).

Further, as illustrated inFIG. 14, the blood flow in the human's face substantially flows from the lower side to the upper side (in the direction of thick arrow). Accordingly, the sub-regional pulse wave signals102khaving waveforms approximately in phase can be acquired from multiple sub-regions501that are located at the same vertical position and aligned in the lateral direction (for example, the hatched sub-regions501inFIG. 14) among the sub-regions501including the skin color region502. On the other hand, a phase difference occurs in the waveforms of multiple sub-regional pulse wave signals102kacquired from the sub-regions501located at different vertical positions in the sub-regions501including the skin color region502. The phase difference is a phase difference between the sub-regional pulse wave signals102k,more specifically, the pulse waves of the blood flow propagating in the blood vessel along with heartbeats.

The average pulse wave signals102aacquired by averaging the sub-regional pulse wave signals102kfrom the sub-regions501located at the same vertical position are drawn on the outer right side of the frame image500inFIG. 14. A time when the average pulse wave signal102areaches an extreme value (a time designated by a frame number or the like) is referred to as an average pulse wave extreme value signal.

Here, the pulse wave velocity (V) can be calculated using a phase difference time Δt between the two average pulse wave signals102aat the sub-regions501located at different vertical positions, and a vertical distance ΔL. That is, the pulse wave velocity (V) is calculated according to an equation: V=ΔL/Δt. The phase difference time Δt between the two average pulse wave signals102acan be readily found as a time difference between average pulse wave extreme value signals103aof the two average pulse wave signals102a.

The average pulse wave signal102ais preferably an average of all the sub-regional pulse wave signals102kacquired from the sub-regions501located at the corresponding vertical position, but may be the sub-regional pulse wave signal102kacquired from one of the sub-regions501located at the corresponding vertical position. However, generally, the use of the average of measurement values can achieve higher accuracy.

FIG. 15is a view illustrating an example of the average pulse wave signals102aeach being an average of the sub-regional pulse wave signals102kobtained from multiple sub-regions501in a facial region510located at the same vertical position, and the basic concept of calculating the pulse wave velocity. InFIG. 15, the facial region510detected by the face detector230is displayed as a rectangle drawn by a thick solid line, and the facial region510are divided into multiple sub-regions501by broken lines. Further, the skin color region502(shaded portion) is present in the facial region510.

FIG. 15is different fromFIG. 16in that the sub-regions501for finding the sub-regional pulse wave signals102kare not set in the entire frame image500, but set in the facial region510detected by the face detector230. Except this,FIG. 15is the same asFIG. 14and description thereof is omitted.

FIG. 16is a view for describing a method of calculating the pulse wave velocity in the case where some of the laterally-aligned sub-regions501located at the same vertical position are pulse wave signal missing sub-regions505. Here, the pulse wave signal missing sub-region505refers to a sub-region501from which the sub-regional pulse wave signals102kcannot be acquired, and inFIG. 16, is represented as a hollow sub-region501. The hatched sub-regions501inFIG. 16represent the sub-regions501from which the sub-regional pulse wave signals102kare acquired.

As described above, to calculate pulse wave velocity, first, an average of the sub-regional pulse wave signals102k,which are acquired from multiple sub-regions501located at the same vertical position and different lateral positions, that is, the average pulse wave signal102ais calculated. InFIG. 16, the laterally-aligned sub-regions501include pulse wave signal missing sub-regions505in some part, but also include sub-regions501from which the sub-regional pulse wave signals102kare acquired. In such case, the average pulse wave signal102acan be acquired by averaging the sub-regional pulse wave signals102kof the sub-regions501from which the sub-regional pulse wave signals102kare acquired.

To put it more specifically using the example inFIG. 16, the laterally-aligned sub-regions501located at the second vertical position from the top include six sub-regions501, four of which are pulse wave signal missing sub-regions505, and two of which are sub-regions where the sub-regional pulse wave signals102kare acquired. In such case, the average pulse wave signal102aat the vertical position can be acquired by averaging the sub-regional pulse wave signals102kfrom the two sub-regions501.

When the average pulse wave signal102ais acquired at each vertical position in this manner, the average pulse wave extreme value signal can be acquired from each of the average pulse wave signals102a.Then, an average value Ave (Δt) can be found as an average of the phase difference time Δt between the average pulse wave extreme value signals at adjacent vertical positions. Here, the pulse wave velocity (V) can be found according to an equation: V=ΔL/Ave (Δt).

FIG. 17is a view for describing a method of calculating the pulse wave velocity in the case where all the laterally-aligned sub-regions501are pulse wave signal missing sub-regions505. In the example illustrated inFIG. 17, at the second and third vertical positions from the top, all the laterally-aligned sub-regions501are the pulse wave signal missing sub-regions505. Thus, at these vertical positions, the average pulse wave signals102acannot be acquired. However, at the first, fourth, and fifth vertical positions, average pulse wave signals102aare acquired.

In such a case, a phase difference time Δt1per vertical distance corresponding to one sub-region is found from the average pulse wave signals102aat the first and fourth vertical positions from the top, and a phase difference time Δt2is found from the average pulse wave signals102aat the fourth and fifth vertical positions from the top. Given that an average value of the phase difference times Δt1and Δt2is expressed as Ave (Δt1, Δt2), the pulse wave velocity (V) can be found according to an equation: V=ΔL/Ave (Δt1, Δt2).

As described above, even when all the laterally-aligned sub-regions501are the pulse wave signal missing sub-regions505at any vertical position, as long as the average pulse wave signals102aare acquired at vertical positions above and below the vertical position, the phase difference time Δt per vertical distance corresponding to one sub-region can be found using the average pulse wave signals102a.Accordingly, the pulse wave velocity (V) can be found.

FIG. 18is a detailed block diagram illustrating an example of the blood pressure estimation unit320in accordance with the second embodiment. As illustrated inFIG. 18, the blood pressure estimation unit320includes a pulse wave velocity storage321, a smoothing filter322, a blood pressure conversion table326and a blood pressure corrector325.

Here, the pulse wave velocity storage321stores values of the pulse wave velocity signal303inputted over the multiple frames, and outputs a delay pulse wave velocity signal327. The smoothing filter322averages the pulse wave velocity signals303and the delay pulse wave velocity signals327inputted over the multiple frames, and outputs a smoothed pulse wave velocity signal323.

The blood pressure conversion table326receives the smoothed pulse wave velocity signal323, searches the table, and outputs a blood pressure conversion signal328on which blood pressure is based. According to the Moens-Horteweg equation, a blood pressure value (P) in the diastolic phase is proportional to the square of the pulse wave velocity (PWV). That is, an equation: P=c×PWV2is satisfied. However, a proportionality constant c depends on various kinds of biological information (age, sex, blood vessel radius, blood density, and so forth) of the subject. Thus, the blood pressure conversion table326receives the value of the smoothed pulse wave velocity signal323as the pulse wave velocity (PWV), and outputs the blood pressure value for predetermined typical biological information as the blood pressure conversion signal328.

The blood pressure corrector325receives the smoothed pulse wave velocity signal323, the blood pressure conversion signal328, and a blood pressure correction parameter324, corrects the blood pressure conversion signal328, and outputs an estimated blood pressure value304. Here, the blood pressure correction parameter324is a numerical value necessary for determining the proportionality constant c, such as age, sex, blood vessel radius, and blood density. That is, the blood pressure corrector325corrects the blood pressure value for typical biological information acquired from the blood pressure conversion table326according to biological information of the subject.

In this embodiment, the blood pressure estimation unit320estimates the blood pressure value of the subject by using the pulse wave velocity signal303, the blood pressure conversion table326, and the blood pressure correction parameter324and however, the estimated blood pressure value304of the subject may be calculated according to a mathematical model such as the Moens-Korteweg equation.

In the second embodiment, the multiple sub-regional pulse wave signals102kacquired from the multiple sub-regions501including the skin color region502is generated based on the sub-regional skin color wavelength data signal204kcorresponding to the hue (H) acquired from pixels in the skin color region502. In this case, the influences of the value (V) and the saturation (S) on the sub-regional pulse wave signals102kare eliminated. In summary, in the second embodiment, the estimated blood pressure value304is calculated using the multiple sub-regional pulse wave signals102k,with the influences of the value (V) and the saturation (S) being eliminated. Accordingly, in the second embodiment, the estimated blood pressure value304, with the influence of natural light, that is, the influences of the value (V) and the saturation (S) being eliminated, can be acquired.

The present invention is not limited to the above-mentioned embodiments and modification examples, and includes other various modification examples. For example, the above-mentioned embodiments and modification examples describe the present invention in detail to facilitate understanding of the present invention, and do not necessarily include all the described constituents. In addition, a unit of the configuration of any embodiment or modification example may be replaced with the configuration of another embodiment or modification example. Alternatively, the configuration of any embodiment or modification example may be combined with the configuration of another embodiment or modification example o Further, part of the configuration of each of the embodiments and modification examples may be altered by addition, deletion, or replacement of a configuration in another embodiment or modification example.