Patent ID: 12189031

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, an embodiment of a distance image acquisition apparatus and a distance image acquisition method according to the invention will be described referring to the accompanying drawings.

[Distance Image Acquisition Apparatus]

FIG.1is a block diagram showing a functional configuration example of a distance image acquisition apparatus10according to the invention.

The distance image acquisition apparatus10shown inFIG.1is a distance image acquisition apparatus of a pulse light detection system, and primarily includes an imaging lens12, a distance image sensor14, an analog-to-digital (AD) converter16, an interface circuit18, a central processing unit (CPU)20, a pulse light emission unit22, an exposure control unit24, and a drive mode selection unit26.

The pulse light emission unit22comprises a near infrared light emitting diode (near infrared LED), and emits pulse light having a given pulse width in synchronization with a light emission timing signal from the exposure control unit24. Pulse light emitted from the near infrared LED of the pulse light emission unit22is near infrared light.

The imaging lens12images reflected light (including reflected light of pulse light irradiated from the pulse light emission unit22and reflected from a subject) from the subject on the distance image sensor14.

The distance image sensor14is constituted of a complementary metal-oxide semiconductor (CMOS) driver which has a vertical driver, a horizontal driver, and the like, and a CMOS type image sensor which is driven by a timing generator. The distance image sensor14is not limited to a CMOS type, and may be an XY address type or charge coupled device (CCD) type image sensor.

The distance image sensor14has a plurality of light receiving elements (photodiodes) in a two-dimensional manner, and is provided with, on an incidence surface of a plurality of light receiving elements, a band-pass filter which transmits only a wavelength bandwidth of near infrared pulse light emitted from the pulse light emission unit22or a visible light cut filter which eliminates visible light. With this, a plurality of light receiving elements of the distance image sensor14function as pixels having sensitivity to near infrared light.

The distance image sensor14is configured such that an exposure period (exposure time and exposure timing) is controlled according to a shutter control signal applied from the exposure control unit24, and electric charge corresponding to the amount of near infrared light entering in the exposure period is accumulated in each light receiving element of the distance image sensor14. Then, a pixel signal (an analog signal corresponding to electric charge accumulated in each pixel) according to the amount of incidence of near infrared light from the subject is read from the distance image sensor14.

Though details will be described below, the exposure control unit24performs one exposure control among first exposure control, in which pulse light is emitted from the pulse light emission unit22, and the difference in the exposure (light reception time of pulse light) between the corresponding light receiving elements of the distance image sensor14according to at least a distance of the subject, second exposure control, in which pulse light is emitted from the pulse light emission unit22, a phase of exposure start with respect to pulse light is different from a phase of exposure start with respect to pulse light of the first exposure control, and all of the light receiving elements of the distance image sensor14are entirely exposed to pulse light reflected from the subject, and a third exposure control, in which light is not emitted from the pulse light emission unit22and exposure to reflected light from the subject is performed, sequentially or selectively.

The analog signal read from the distance image sensor14after the exposure control in the exposure control unit24is converted to a digital signal by the AD converter16, and is loaded on the CPU20by way of the interface circuit18functioning as an image input controller. A CMOS type image sensor may include an AD converter, and in this case, the AD converter16may be omitted.

Though details will be described below, the CPU20has a function as a device control unit which integrally controls the respective units, such as the exposure control unit24, of the distance image acquisition apparatus10, and a function as a drive mode setting unit20A, a distance image generation unit20B, a brightness detection unit20C, and a determination unit20D.

<Basic Principle of Pulse Light Detection System>

Next, the basic principle of the pulse light detection system applied to the invention will be described.

Pulse light irradiated from the pulse light emission unit22of the distance image acquisition apparatus10enters a subject within a distance measurement region, and pulse light reflected from the subject is imaged (received) by the distance image sensor14through the imaging lens12.

As shown inFIG.2A, in a case where there are a far subject T1and a near subject T2within the distance measurement region, pulse light indicated by an arrow is returned earlier from the near subject T2than from the far subject T1, and is received by the distance image sensor14.

FIG.2Bis a diagram showing the relationship of pulse light reflected from the far subject T1and the near subject T2and received by the distance image sensor14and the exposure period, and in particular, shows a case where the far subject T1and the near subject T2have the same reflectance.

In this case, as shown inFIG.2A, in a case where exposure control in which exposure starts in synchronization with the light emission timing of the pulse light and exposure ends when an appropriate time elapses after reflected light is received simultaneously from the far subject T1and the near subject T2, the exposure becomes larger for the near subject T2than for the far subject T1.

Therefore, in a case where the subjects have the same reflectance, it is possible to measure a relative distance according to the magnitude of the exposure.

A near subject T2shown inFIG.3Ahas reflectance lower than that of a far subject T1, and the amount of pulse light reflected and returned from the near subject T2is smaller than the amount of pulse light reflected and returned from the far subject T1. InFIG.3A, the thickness of an arrow represents the amount of pulse light.

In this case, as shown inFIG.3B, even if exposure control is performed in the same manner as inFIG.2B, it is not possible to measure the relative distance of the subject from the exposure obtained by the exposure control.

Therefore, in a case where the reflectance of the subject is not constant, it is necessary to perform multiple times of exposure control in order to eliminate the influence of the reflectance.

In a case where there is near infrared ambient light other than near infrared pulse light (for example, in an imaging environment, such as the outdoors in the daytime), since distance measurement accuracy is deteriorated due to the influence of ambient light, it is preferable to eliminate the influence of ambient light. In this case, the difference in the exposure between exposure control in which pulse light is irradiated and exposure control in which pulse light is not irradiated is taken, whereby it is possible to eliminate the influence of ambient light.

[Drive Mode]

Hereinafter, a drive mode in a pulse light detection system of a first embodiment will be described.

<First Drive Mode>

FIGS.4A to4Dare diagrams showing exposure control and calculation processing of a first drive mode in the pulse light detection system of the first embodiment.

The first drive mode is a mode which is set in a case of an imaging environment, in which there is the influence of ambient light and the reflectance of the subject.

In the first drive mode, as shown inFIGS.4A,4B, and4C, three times of exposure control of the first exposure control, the second exposure control, and the third exposure control are performed.

The first exposure control shown inFIG.4Ais exposure control in which pulse light is emitted from the pulse light emission unit22, and the exposure period is controlled such that the difference in the exposure between the corresponding light receiving elements of the distance image sensor14is generated according to at least the distance of the subject. In the first exposure control, exposure starts when a given time (a time until pulse light is returned from the farthest subject capable of distance measurement) elapses after pulse light is emitted from the pulse light emission unit22, and exposure ends when a time (predetermined exposure time) until pulse light reflected from at least the farthest subject is entirely returned elapses.

According to the above-described first exposure control, in a case where the reflectance of the subject is constant, in comparison of the far subject T1and the near subject T2, the exposure becomes larger for the far subject T1than for the near subject T2.

The second exposure control shown inFIG.4Bis exposure control in which pulse light is emitted from the pulse light emission unit22and a phase of exposure start with respect to pulse light is different from a phase of exposure start with respect to pulse light of the first exposure control, and is exposure control for eliminating change in the exposure in the distance image sensor14due to the difference in the reflectance of the subject. In this example, the second exposure control is exposure control in which all of the light receiving elements of the distance image sensor14are entirely exposed to pulse light reflected from the subject. Specifically, exposure starts in synchronization with the light emission timing of pulse light emitted from the pulse light emission unit22, and exposure ends when a given time (a predetermined exposure time until pulse light is entirely returned from at least the farthest subject capable of distance measurement) elapses. While the “predetermined exposure time” in the first exposure control is the same time as the “predetermined exposure time” in the second exposure control, as described above, the phase of exposure start with respect to pulse light is different.

According to the above-described second exposure control, the exposure is different according to the reflectance of the subject regardless of whether the subject is far or near, and it is possible to acquire information corresponding to the reflectance of the subject with the exposure obtained by the first exposure control.

The third exposure control shown inFIG.4Cis exposure control in which pulse light is not emitted from the pulse light emission unit22and exposure to reflected light from the subject is performed. In comparison with the second exposure control, the third exposure control is different from the second exposure control in that, while pulse light is emitted (irradiated) and the exposure control is performed in the second exposure control, exposure control is performed without emitting pulse light (in a non-irradiation manner) in the third exposure control.

According to the above-described third exposure control, it is possible to acquire the exposure of only ambient light.

As shown inFIG.4D, in a case where output data corresponding to the exposures acquired from the distance image sensor14by the first exposure control, the second exposure control, and the third exposure control are defined as first data L1, second data L2, the third data L3, respectively, information (hereinafter, referred to as “distance information”) D corresponding to the relative distance of the subject can be calculated by the following expression.
D=(L1−L3)÷(L2−L3)  [1]

That is, according to Expression [1], first subtraction data (L1−L3) is calculated by subtracting the third data L3from the first data L1, and second subtraction data (L2−L3) is calculated by subtracting the third data L3from the second data L2. The first subtraction data is data including the distance and reflectance of the subject with the influence of ambient light eliminated, and the second subtraction data is data indicating the reflectance of the subject with the influence of ambient light eliminated. Then, division data is calculated by dividing the first subtraction data by the second subtraction data. The division data becomes data (distance information D) corresponding to the distance (relative distance) of the subject with the influence of ambient light and the influence of the reflectance of the subject eliminated.

Since the relative distance calculated for each light receiving element of the distance image sensor14in the above-described manner is based on a polar coordinate system, the relative distance is converted to an orthogonal coordinate system, whereby it is possible to generate a distance image corresponding to the distance (relative distance) of the subject within the distance measurement region.

The distance information D corresponding to the relative distance of the subject calculated in this manner has the influence of ambient light and the influence of the reflectance of the subject eliminated.

While the “distance image” refers to an image having the distance from the imaging unit to the subject instead of a color or the value of shade of a normal two-dimensional image, an image having the relative distance (distance information D) of the subject calculated based on Expression [1] described above is also an aspect of a “distance image” in the invention.

According to Expression [1] described above, while the distance information D corresponding to the relative distance of the subject with the influence of ambient light and the influence of the reflectance of the subject eliminated is calculated, the invention is not limited thereto, and an absolute distance of the subject with the influence of ambient light and the influence of the reflectance of the subject eliminated can be calculated based on the first data L1, the second data L2, and the third data L3acquired by the first drive mode using a known method, and obtaining the absolute distance of the subject is an aspect of generating a “distance image” in the invention.

<Second Drive Mode>

FIGS.5A to5Care diagrams showing exposure control and calculation processing of a second drive mode in the pulse light detection system of the first embodiment.

The second drive mode is a mode which is set in a case of an imaging environment, in which there is no influence of ambient light and there is the influence of the reflectance of the subject.

In the second drive mode, as shown inFIGS.5A and5B, two times of exposure control of the first exposure control and the second exposure control are performed. The first exposure control and the second exposure control shown inFIGS.5A and5Bare the same as the first exposure control and the second exposure control in the first drive mode shown inFIGS.4A and4B, and thus, detailed description thereof will not be repeated.

As shown inFIG.5C, in a case where output data corresponding to the exposures acquired from the distance image sensor14by the first exposure control and the second exposure control are defined as the first data L1and the second data L2, respectively, the distance information D corresponding to the relative distance of the subject can be calculated by the following expression.
D=L1÷L2[2]

That is, according to Expression [2], division data is calculated by dividing the first data L1by the second data L2. The division data becomes data (distance information D) corresponding to the relative distance with the influence of the reflectance of the subject eliminated. The absolute distance of the subject may be obtained based on the first data L1corresponding to the distance and reflectance of the subject with no influence of ambient light and the second data L2corresponding to the reflectance of the subject with no influence of ambient light.

<Third Drive Mode>

FIGS.6A to6Care diagrams of exposure control and calculation processing of a third drive mode in the pulse light detection system of the first embodiment.

The third drive mode is a mode which is set in a case of an imaging environment, in which the reflectance of the subject is constant (there is no influence of reflectance) and there is the influence of ambient light. The expression “reflectance is constant” refers to, for example, a case where reflectance between subjects (as shown inFIG.2A, the far subject T1and the near subject T2) within a distance measurement region to be imaged is constant, a case where reflectance of a distance measurement target (including a tracking target in a case of imaging while tracking a specific subject) is constant temporally, or the like.

In the third drive mode, as shown inFIGS.6A and6B, two times of exposure control of the first exposure control and the third exposure control are performed. The first exposure control and the third exposure control shown inFIGS.6A and6Bare the same as the first exposure control and the third exposure control in the first drive mode shown inFIGS.4A and4C, and thus, detailed description thereof will not be repeated.

As shown inFIG.6C, in a case where output data corresponding to the exposures acquired from the distance image sensor14by the first exposure control and the third exposure control are defined as the first data L1and the third data L3, respectively, the distance information D corresponding to the relative distance of the subject can be calculated by the following expression.
D=L1−L3[3]

That is, according to Expression [3], subtraction data is calculated by subtracting the third data L3from the first data L1. The subtraction data becomes data (distance information D) corresponding to the relative distance with the influence of ambient light eliminated. As long as information is information of the reflectance of the subject having constant reflectance, the absolute distance of the subject may be obtained based on the first data L1and the third data L3.

<Fourth Drive Mode>

FIGS.7A and7Bare diagrams showing an embodiment of exposure control and calculation processing of a fourth drive mode in the pulse light detection system of the first embodiment.

The fourth drive mode is a mode which is set in an imaging environment, in which there is no influence of ambient light and the reflectance of the subject is constant (there is no influence of reflectance).

In the fourth drive mode, as shown inFIG.7A, only single exposure control of the first exposure control is performed. The first exposure control shown inFIG.7Ais the same as the first exposure control in the first drive mode shown inFIG.4A, and thus, detailed description will not be repeated.

As shown inFIG.7B, in a case where output data corresponding to the exposure acquired from the distance image sensor14by the first exposure control is defined as the first data L1, the first data L1becomes the distance information D (=L1) corresponding to the relative distance of the subject.

The distance information D corresponding to the relative distance of the subject obtained in this manner has no influence of ambient light and the reflectance of the subject as will be apparent from an imaging condition, to which the fourth drive mode is applied. As long as information is information of the reflectance of the subject having constant reflectance, the absolute distance of the subject can be obtained.

Next, a drive mode in a pulse light detection system of a second embodiment will be described.

As described below in detail, the pulse light detection system of the second embodiment is different from the pulse light detection system of the first embodiment in terms of exposure control and calculation processing in the first drive mode and the second drive mode.

<First Drive Mode>

FIGS.8A to8Dare diagrams showing exposure control and calculation processing of a first drive mode in the pulse light detection system of the second embodiment.

The first drive mode is a mode which is set in a case of an imaging environment, in which there is the influence of ambient light and the reflectance of the subject.

In the first drive mode, as shown inFIGS.8A,8B, and8C, the first exposure control, the second exposure control, and the third exposure control are performed.

The first exposure control shown inFIG.8Ais exposure control in which pulse light is emitted from the pulse light emission unit22, and the exposure period is controlled such that the difference in the exposure between the corresponding light receiving elements of the distance image sensor14is generated according to at least the distance of the subject, and corresponds to the first exposure control of the first drive mode in the pulse light detection system of the first embodiment shown inFIG.4A.

The second exposure control shown inFIG.8Bis exposure control in which pulse light is emitted from the pulse light emission unit22, and the phase of exposure start with respect to pulse light is different from the phase of exposure start with respect to pulse light of the first exposure control. In this example, the second exposure control is exposure control in which exposure starts in synchronization with the light emission timing of pulse light emitted from the pulse light emission unit22, and exposure is performed for the same time as the exposure time by the first exposure control. In other words, the second exposure control is exposure control in which the exposure period is set not to overlap the exposure period by the first exposure control shown inFIG.8A, the exposure time is continuous to the exposure time by the first exposure control, and in a case of totaling an output obtained from the distance image sensor14by the first exposure control and an output obtained from the distance image sensor14by the second exposure control, an output with entirely exposed to pulse light reflected from the subject is obtained.

Specifically, the second exposure control is exposure control in which exposure starts in synchronization with the light emission timing of pulse light emitted from the pulse light emission unit22, and exposure ends in synchronization with the lapse of a time corresponding to a half width of the light emission time of pulse light.

It is preferable that a plurality of light receiving elements of the distance image sensor14arranged in a two-dimensional manner are divided into a first light receiving element group for first exposure control and a second light receiving element group for second exposure control, exposure control for the first light receiving element group is performed by the first exposure control, and exposure control for the second light receiving element group is performed by the second exposure control. It is preferable that the first light receiving element group and the second light receiving element group are arranged in a checkered flag pattern.

The third exposure control shown inFIG.8Cis exposure control in which exposure to reflected light from the subject is performed without emitting pulse light from the pulse light emission unit22. In this example, exposure control is performed for an exposure time (the exposure time by the first exposure control, or an exposure time twice longer than the exposure time by the second exposure control) obtained by totaling the exposure time by the first exposure control and the exposure time by the second exposure control.

As shown inFIG.8D, in a case where output data corresponding to the exposures acquired from the distance image sensor14by the first exposure control, the second exposure control, and the third exposure control are defined as the first data L1, the second data L2, and the third data L3, respectively, the distance information D corresponding to the relative distance of the subject is calculated by the following expression.
D=(L1−L3/2)÷(L1+L2−L3)  [4]

That is, according to Expression [4], first subtraction data (L1−L3/2) is calculated by subtracting half (since the exposure time is twice) of the third data L3from the first data L1, and second subtraction data (L1+L2−L3) is calculated by adding the first data L1and the second data L2and subtracting the third data L3from the addition data. The first subtraction data is data including the distance and reflectance of the subject with the influence of ambient light eliminated, and the second subtraction data is data indicating the reflectance of the subject with the influence of ambient light eliminated. Then, division data is calculated by dividing the first subtraction data by the second subtraction data. The division data becomes data (distance information D) corresponding to the relative distance with the influence of ambient light and the influence of the reflectance of the subject eliminated.

The exposure time by the third exposure control shown inFIG.8Ccan be set to a half exposure time (the same exposure time as the exposure time by the first exposure control or the exposure time by the second exposure control), and in this case, the following expression is used instead of Expression [4] described above.
D=(L1−L3)÷(L1+L2−2L3)  [5]
<Second Drive Mode>

FIGS.9A to9Care diagrams showing exposure control and calculation processing of a second drive mode in the pulse light detection system of the second embodiment.

The second drive mode is a mode which is set in a case of an imaging environment, in which there is no influence of ambient light and there is the influence of the reflectance of the subject.

In the second drive mode, as shown inFIGS.9A and9B, the first exposure control and the second exposure control are performed. The first exposure control and the second exposure control shown inFIGS.9A and9Bare the same as the first exposure control and the second exposure control in the first drive mode shown inFIGS.8A and8B, and thus, detailed description thereof will not be repeated.

As shown inFIG.9C, in a case where output data corresponding to the exposures acquired from the distance image sensor14by the first exposure control and the second exposure control are defined as the first data L1and the second data L2, respectively, the distance information D corresponding to the relative distance of the subject is calculated by the following expression.
D=L1÷(L1+L2)  [6]

That is, according to Expression [6], division data is calculated by dividing the first data L1with no influence of ambient light by addition data (L1+L2) (data indicating the reflectance of the subject with no influence of ambient light) obtained by adding the first data L1and the second data L2. The division data becomes data (distance information D) corresponding to the relative distance with the influence of the reflectance of the subject eliminated.

Returning toFIG.1, the drive mode setting unit20A in the CPU20is a part which sets one drive mode among the first drive mode to the fourth drive mode described above, and sets a drive mode automatically according to an imaging environment or sets a drive mode selected by a user's manual operation on the drive mode selection unit26.

The brightness detection unit20C is a part which detects brightness of ambient light, and outputs an integrated average value of a sensor output from the distance image sensor14at the time of performing exposure control (exposure control corresponding to the third exposure control shown inFIG.4C) without emitting pulse light from the pulse light emission unit22as information indicating the brightness of ambient light to the drive mode setting unit20A. For example, in a case of an imaging environment, such as the nighttime, the brightness of ambient light detected by the brightness detection unit20C becomes equal to or less than a threshold corresponding to measurement accuracy. In a case of an imaging environment, such as the indoors with only an artificial light source, since near infrared light is not included in the artificial light source, the brightness of ambient light detected by the brightness detection unit20C becomes equal to or less than the threshold set corresponding to the measurement accuracy. In the first exposure control and the second exposure control, in a case where exposure to ambient light other than pulse light is performed, the distance measurement accuracy is degraded. The threshold is a value which is set corresponding to required measurement accuracy, and needs to be set to a threshold closer to 0 when the measurement accuracy is higher.

The determination unit20D is a part which determines whether or not reflectance of a subject for distance measurement is constant, and for example, in a case where the distance image acquisition apparatus10is used for the purpose of tracking a specific subject (a part of a machine tool, a face of a person, or the like) and measuring a distance of the subject, determines that the reflectance of the subject is constant. A determination result determined by the determination unit20D is output to the drive mode setting unit20A.

In a case where an automatic mode is selected by the drive mode selection unit26, and a drive mode is set automatically, the drive mode setting unit20A sets an optimum drive mode among the first drive mode to the fourth drive mode based on information indicating the brightness of ambient light input from the brightness detection unit20C and determination information input from the determination unit20D regarding whether or not the reflectance of the subject is constant.

That is, the drive mode setting unit20A determines that the brightness of ambient light is greater than the threshold (there is the influence of ambient light) based on information indicating the brightness of ambient light input from the brightness detection unit20C and the determination information input from the determination unit20D, and in a case where it is determined that the reflectance of the subject is not constant, sets the first drive mode shown inFIGS.4A to4DorFIGS.8A to8D.

Similarly, the drive mode setting unit20A determines that the brightness of ambient light is equal to or less than the threshold (there is no influence of ambient light), in a case where it is determined that the reflectance of the subject is constant, sets the second drive mode shown inFIGS.5A to5CorFIGS.9A to9C, in a case where it is determined that the brightness of ambient light is greater than the threshold and it is determined that the reflectance of the subject is constant, sets the third drive mode shown inFIGS.6A to6C, and in a case where it is determined that the brightness of ambient light is equal to or less than the threshold and it is determined that the reflectance of the subject is constant, sets the fourth drive mode shown inFIGS.7A and7B.

In a case where a manual mode is selected by the drive mode selection unit26and one drive mode among the first drive mode to the fourth drive mode is selected by a manual operation, the drive mode setting unit20A sets the manually selected drive mode.

In a case where one drive mode among the first drive mode to the fourth drive mode is set by the drive mode setting unit20A, information indicating the set drive mode is applied to the exposure control unit24.

As described referring toFIGS.4A to4DtoFIGS.9A to9C, the exposure control unit24performs light emission control of pulse light of the pulse light emission unit22and exposure control (exposure control including the first exposure control among the first exposure control, the second exposure control, and the third exposure control) of the distance image sensor14according to the input drive mode.

The distance image generation unit20B acquires the sensor output (the first data L1, the second data L2, or the third data L3corresponding to the exposure of the distance image sensor14by the first exposure control, the second exposure control, and the third exposure control) of the distance image sensor14according to the exposure control by the exposure control unit24through the interface circuit18. Then, in a case where the first drive mode is set, the distance (relative distance) of the subject for each light receiving element of the distance image sensor14is calculated based on the first data L1, the second data L2, and the third data L3corresponding to the exposures by the first exposure control, the second exposure control, and the third exposure control by Expression [1] or Expression [4] described above, and the distance of the subject corresponding to all of the light receiving elements is calculated, thereby generating a distance image corresponding to the relative distance of the subject within the distance measurement region.

Similarly, in a case where the second drive mode is set, the distance image generation unit20B calculates the distance of the subject for each light receiving element of the distance image sensor14based on the first data L1and the second data L2corresponding to the exposures by the first exposure control and the second exposure control by Expression [2] or Expression [6] described above, and calculates the distance of the subject corresponding to all of the light receiving elements, thereby generating a distance image corresponding to the relative distance of the subject within the distance measurement region. In a case where the third drive mode is set, the distance image generation unit20B calculates the distance of the subject for each light receiving element of the distance image sensor14based on the first data L1and the third data L3corresponding to the exposures by the first exposure control and the third exposure control by Expression [3] described above, and calculates the distance of the subject corresponding to all of the light receiving elements, thereby generating a distance image corresponding to the relative distance of the subject within the distance measurement region. In a case where the fourth drive mode is set, the distance image generation unit20B calculates the distance of the subject for each light receiving element of the distance image sensor14based on the first data L1corresponding to the exposure by the first exposure control, and calculates the distance of the subject corresponding to all of the light receiving elements, thereby generating a distance image corresponding to the relative distance of the subject within the distance measurement region.

[Distance Image Acquisition Method]

An embodiment of an imaging processing method according to the invention provides a first drive mode to a fourth drive mode, sets one drive mode among the four drive modes, and performs exposure control and calculation processing in the set drive mode.

Drive mode setting is performed based on determination shown in a flowchart ofFIG.10(drive mode setting step).

That is, it is determined whether or not the reflectance of the subject is constant (Step S10), in a case where the reflectance of the subject is not constant (in a case of “No”), the process transits to Step S12, and in a case where the reflectance of the subject is constant (in a case of “Yes”), the process transits to Step S14.

In Steps S12and S14, it is determined whether or not the brightness of ambient light is equal to or less than the threshold (a threshold for determining whether or not the brightness of ambient light affects the measurement accuracy and set corresponding to the measurement accuracy), and in a case where it is determined in Steps S12and S14that the brightness of ambient light is equal to or less than the threshold (in a case of “Yes”), the second drive mode and the fourth drive mode are set, respectively.

In a case where it is determined in Steps S12and S14that the brightness of ambient light is greater than the threshold (in a case of “No”), the first drive mode and the third drive mode are set, respectively.

With this, the first drive mode is set in a case of an imaging environment in which there is the influence of ambient light and the reflectance of the subject, the second drive mode is set in a case of an imaging environment in which there is no influence of ambient light and there is the influence of the reflectance of the subject, the third drive mode is set in a case of an imaging environment in which the reflectance of the subject is constant (there is no influence of reflectance) and there is the influence of ambient light, and the fourth drive mode is set in a case of an imaging environment in which there is no influence of ambient light and the reflectance of the subject is constant (there is no influence of reflectance).

The above-described drive mode setting may be performed automatically by the drive mode setting unit20A shown inFIG.1based on information indicating the brightness of ambient light input from the brightness detection unit20C and the determination information input from the determination unit20D indicating whether or not the reflectance of the subject is constant, or may be performed based on a user's drive mode selection operation on the drive mode selection unit26.

Next, the distance image acquisition method in each drive mode of the first drive mode to the fourth drive mode will be described. Hereinafter, although a case of a drive mode in the pulse light detection system of the first embodiment will be described, a drive mode in the pulse light detection system of the second embodiment can be performed similarly.

FIG.11is a flowchart showing the distance image acquisition method in a case where the first drive mode is set.

InFIG.11, first, pulse light is emitted from the pulse light emission unit22(Step S100), and the first exposure control shown inFIG.4Ais performed (Step S110). The first data L1corresponding to the exposure of each light receiving element of the distance image sensor14by the first exposure control is acquired from the distance image sensor14.

Subsequently, second pulse light is emitted from the pulse light emission unit22(Step S120), and the second exposure control shown inFIG.4Bis performed (Step S130). The second data L2corresponding to the exposure of each light receiving element of the distance image sensor14by the second exposure control is acquired from the distance image sensor14.

Next, the third exposure control shown inFIG.4Cis performed without emitting pulse light from the pulse light emission unit22(Steps S140, S150). The third data L3corresponding to the exposure of each light receiving element of the distance image sensor14by the third exposure control is acquired from the distance image sensor14. Step S100to Step S150correspond to an exposure control step in the first drive mode.

Next, the calculation shown in Expression [1] described above is executed based on the first data L1, the second data L2, and the third data L3corresponding to the exposures by the first exposure control, the second exposure control, and the third exposure control acquired in Steps S110, S130, and S150, and the distance information D corresponding to the relative distance of the subject is calculated for each light receiving element of the distance image sensor14(Step S160).

The distance image is generated based on the distance information D for each light receiving element of the distance image sensor14calculated in Step S160(Step S170). Steps S160and S170correspond to a distance image generation step.

The processing of Step S100to Step S170may be repeatedly executed at given intervals, and accordingly, it is possible to obtain a distance image in the format of video continuous at given intervals.

FIG.12is a flowchart showing the distance image acquisition method in a case where the second drive mode is set. The steps common to the first drive mode shown inFIG.11are represented by the same step numbers, and detailed description thereof will not be repeated.

An exposure control step in a case where the second drive mode is set corresponds to Step S100to Step S130in the first drive mode, and the processing of Step S140and Step S150is omitted.

Next, the calculation shown in Expression [2] described above is executed based on the first data L1and the second data L2corresponding to the exposures by the first exposure control and the second exposure control acquired in Steps S110and S130, and the distance information D corresponding to the relative distance of the subject is calculated for each light receiving element of the distance image sensor14(Step S200).

The distance image is generated based on the distance information D for each light receiving element of the distance image sensor14calculated in Step S200(Step S210).

Since the second drive mode which is set in a case of an imaging environment, in which there is no influence of ambient light and there is the influence of the reflectance of the subject, has a smaller number of times of imaging than in the first drive mode, and performs less calculation for calculating the distance information D, little shot noise and high distance measurement accuracy are achieved.

FIG.13is a flowchart showing the distance image acquisition method in a case where the third drive mode is set. The steps common to the first drive mode shown inFIG.11are represented by the same step numbers, and detailed description will not be repeated.

An exposure control step in a case where the third drive mode is set corresponds to Steps S100, S110, S140, and S150in the first drive mode, and the processing of Step S120and Step S130is omitted.

Next, the calculation shown in Expression [3] described above is executed based on the first data L1and the third data L3corresponding to the exposures by the first exposure control and the third exposure control acquired in Steps S110and S150, and the distance information D corresponding to the relative distance of the subject is calculated for each light receiving element of the distance image sensor14(Step S300).

The distance image is generated based on the distance information D for each light receiving element of the distance image sensor14calculated in Step S300(Step S310).

Since the third drive mode which is set in an imaging environment, in which there is no influence of the reflectance of the subject and there is the influence of ambient light, has a smaller number of times of imaging than in the first drive mode and performs less calculation for calculating the distance information D, little shot noise and high distance measurement accuracy are achieved.

FIG.14is a flowchart showing the distance image acquisition method in a case where the fourth drive mode is set. The steps common to the first drive mode shown inFIG.11are represented by the same step numbers, and detailed description thereof will not be repeated.

An exposure control step in a case where the fourth drive mode is set corresponds to Steps S100and S110in the first drive mode, and the processing of Step S120to Step S150is omitted.

Next, the distance image is generated based on the first data L1(that is, the distance information D for each light receiving element of the distance image sensor14) corresponding to the exposure by the first exposure control acquired in Step S110(Step S400).

Since the fourth drive mode which is set in a case of an imaging environment, in which there is no influence of the reflectance of the subject and there is no influence of ambient light does not require the acquisition of the exposures for use in eliminating the influence of the reflectance of the subject and the influence of ambient light and does not require calculation among a plurality of exposures, among the first drive mode to the fourth drive mode, the least shot noise and the highest distance measurement accuracy are achieved.

An aspect to which the invention is applicable is not limited to a distance image acquisition apparatus having a single function of acquiring a distance image, and a general digital camera or a video camera which can acquire a color image may have the distance image acquisition function. The invention is also applicable to mobile apparatuses, which have, in addition to the distance image acquisition function, other functions (a call handling function, a communication function, and other computer functions), in addition to cameras having a principal function of distance image acquisition. As other aspects to which the invention is applicable, for example, mobile phones, smartphones, personal digital assistants (PDAs), portable game machines, and the like having a camera function are exemplified. Hereinafter, an example of a smartphone to which the invention is applicable will be described.

<Configuration of Smartphone>

FIG.15shows the appearance of a smartphone500which is an embodiment of a distance image acquisition apparatus.

The smartphone500shown inFIG.15has a flat plate-shaped housing502, and comprises a display input unit520in which a display panel521as a display unit and an operation panel522as an input unit are integrated on one surface of the housing502. The housing502comprises a speaker531, a microphone532, an operating unit540, a camera unit541, and an LED light emission unit542. The configuration of the housing502is not limited thereto, and for example, a configuration in which the display unit and the input unit are independent from each other may be employed, or a configuration having a folding structure or a slide mechanism may be employed.

FIG.16is a block diagram of the smartphone500shown inFIG.15. As shown inFIG.16, the smartphone500comprises, as principal components, a wireless communication unit510, a display input unit520, a call handling unit530, an operating unit540, a camera unit541, a storage unit550, an external input/output unit560, a global positioning system (GPS) reception unit570, a motion sensor unit580, a power supply unit590, and a main control unit501. The smartphone500has, as a principal function, a wireless communication function of performing mobile wireless communication through a base station device and a mobile communication network.

The wireless communication unit510performs wireless communication with the base station device in the mobile communication network according to an instruction of the main control unit501. With the use of the wireless communication, transmission and reception of various kinds of file data, such as music data and image data, and electronic mail data, or reception of Web data, streaming data, or the like is performed.

The display input unit520is a so-called touch panel which displays images (still images and moving images), text information, or the like to visually transfer information to the user and detects a user's operation on the displayed information under the control of the main control unit501, and comprises the display panel521and the operation panel522. In a case where a generated three-dimensional image is viewed, it is preferable that the display panel521is a three-dimensional display panel.

The display panel521uses a liquid crystal display (LCD), an organic electro-luminescence display (OELD), or the like as a display device.

The operation panel522is a device which is placed such that an image displayed on a display surface of the display panel521is visible, and detects one or a plurality of coordinates operated by a user's finger or a stylus. In a case where the device is operated with the user's finger or the stylus, a detection signal generated due to the operation is output to the main control unit501. Next, the main control unit501detects an operation position (coordinates) on the display panel521based on the received detection signal.

As shown inFIG.15, although the display panel521and the operation panel522of the smartphone500are integrated to constitute the display input unit520, the operation panel522is arranged so as to completely cover the display panel521. In a case where this arrangement is employed, the operation panel522may have a function of detecting a user's operation even in a region outside the display panel521. In other words, the operation panel522may have a detection region (hereinafter, referred to as a display region) for a superimposed portion overlapping the display panel521and a detection region (hereinafter, referred to as a non-display region) for an outer edge portion not overlapping the display panel521other than the display region.

Although the size of the display region may completely coincide with the size of the display panel521, both of the size of the display region and the size of the display panel521are not necessarily made to coincide with each other. The operation panel522may have two sensitive regions including an outer edge portion and an inner portion other than the outer edge portion. The width of the outer edge portion is appropriately designed according to the size of the housing502or the like. As a position detection system which is employed in the operation panel522, a matrix switching system, a resistive film system, a surface acoustic wave system, an infrared system, an electromagnetic induction system, an electrostatic capacitance system, and the like are exemplified, and any system may be employed.

The call handling unit530comprises the speaker531and the microphone532, converts speech of the user input through the microphone532to speech data processable in the main control unit501and outputs speech data to the main control unit501, or decodes speech data received by the wireless communication unit510or the external input/output unit560and outputs speech from the speaker531. As shown inFIG.15, for example, the speaker531and the microphone532can be mounted on the same surface as the surface on which the display input unit520is provided.

The operating unit540is a hardware key using a key switch or the like, and receives an instruction from the user. For example, the operating unit540is a push button type switch which is mounted on a lower surface below the display unit of the housing502of the smartphone500, and is turned on in a case of being pressed with a finger or the like and is brought into an off state by restoration force of the panel or the like in a case where the finger is released.

The storage unit550stores a control program or control data of the main control unit501, address data associated with the name, telephone number, and the like of a communication partner, data of transmitted and received electronic mail, Web data downloaded by Web browsing, and downloaded content data, and temporarily stores streaming data or the like. The storage unit550is constituted of an internal storage unit551embedded in the smartphone and an external storage unit552having a slot for a detachable external memory. Each of the internal storage unit551and the external storage unit552constituting the storage unit550is realized using a memory (for example, a Micro SD (Registered Trademark) memory, such as a flash memory type, a hard disk type, a multimedia card micro type, or a card type, or a storage medium, such as a random access memory (RAM) or a read only memory (ROM).

The external input/output unit560plays a role of an interface with all external devices connected to the smartphone500, and is provided for direct or indirect connection to other external devices through communication or the like (for example, a universal serial bus or the like) or network (for example, the Internet, a wireless local area network (LAN), Bluetooth (Registered Trademark), radio frequency identification (RFID), infrared data association (IrDA), Ultra Wideband (UWB) (Registered Trademark), ZigBee (Registered Trademark), or the like).

The external devices connected to the smartphone500are, for example, a wired/wireless headset, a wired/wireless external charger, a wired/wireless data port, a memory card, a subscriber identity module (SIM) card, or a user identity module (UIM) card connected through a card socket, an external audio-video device connected through an audio-video input/output (I/O) terminal, an external audio-video device connected in a wireless manner, a smartphone connected in a wired or wireless manner, a personal computer connected in a wired/wireless manner, a PDA connected in a wired/wireless manner, an earphone, and the like. The external input/output unit can transfer data transmitted from the external devices to the respective components in the smartphone500or can transmit data in the smartphone500to the external devices.

The GPS reception unit570receives GPS signals transmitted from GPS satellites ST1to STn according to an instruction of the main control unit501, executes positioning calculation processing based on a plurality of received GPS signals, and detects the position of the smartphone500having latitude, longitude, and altitude. When position information can be acquired from the wireless communication unit510or the external input/output unit560(for example, a wireless LAN), the GPS reception unit570can detect the position using the position information.

The motion sensor unit580comprises, for example, a three-axis acceleration sensor or the like, and detects physical motion of the smartphone500according to an instruction of the main control unit501. The moving direction or acceleration of the smartphone500is detected by detecting physical motion of the smartphone500. The detection result is output to the main control unit501.

The power supply unit590supplies electric power stored in a battery (not shown) to the respective units of the smartphone500according to an instruction of the main control unit501.

The main control unit501is provided with a microprocessor, operates according to the control program or control data stored in the storage unit550, and integrally controls the respective units of the smartphone500. The main control unit501has a mobile communication control function of controlling respective units of a communication system in order to perform speech communication or data communication through the wireless communication unit510, and an application processing function.

The application processing function is realized by the main control unit501operating according to application software stored in the storage unit550. The application processing function is, for example, an infrared communication function of controlling the external input/output unit560to perform data communication with a device facing the smartphone500, an electronic mail function of transmitting and receiving electronic mail, a Web browsing function of browsing Web pages, or the like.

The main control unit501has an image processing function of displaying video on the display input unit520, or the like based on image data (still image or moving image data), such as received data or downloaded streaming data. The image processing function refers to a function of the main control unit501decoding image data, performing image processing on the decoding result, and displaying an image on the display input unit520.

The main control unit501executes display control on the display panel521and operation detection control for detecting a user's operation through the operating unit540and the operation panel522.

With the execution of the display control, the main control unit501displays an icon for activating application software or a software key, such as a scroll bar, or displays a window for creating electronic mail. The scroll bar refers to a software key for receiving an instruction to move a display portion of an image which is too large to fit into the display area of the display panel521.

With the execution of the operation detection control, the main control unit501detects a user's operation through the operating unit540, receives an operation on the icon or an input of text in an entry column of the window through the operation panel522, or receives a scroll request of a display image through the scroll bar.

In addition, with the execution of the operation detection control, the main control unit501has a touch panel control function of determining whether or not an operation position on the operation panel522is the superimposed portion (display region) overlapping the display panel521or the outer edge portion (non-display region) not overlapping the display panel521other than the display region, and controlling the sensitive region of the operation panel522or the display position of the software key.

The main control unit501may detect a gesture operation on the operation panel522and may execute a function set in advance according to the detected gesture operation. The gesture operation is not a conventional simple touch operation, but means an operation to render a track with a finger or the like, an operation to simultaneously designate a plurality of positions, or an operation to render a track for at least one of a plurality of positions by combining the above-described operations.

The camera unit541is an imaging device which performs electronic imaging using an imaging element, such as a complementary metal oxide semiconductor (CMOS) or a charge-coupled device (CCD). The above-described distance image acquisition apparatus10can be applied to the camera unit541.

In this case, in the distance image acquisition apparatus10, it is preferable that light receiving elements for color imaging (light receiving elements of RGB transmitting light in wavelength bandwidths of red (R), green (G), and blue (B)) and a light receiving element transmitting near infrared light for a distance image are mixed in one imaging element. That is, as an imaging element of the camera unit541, an imaging element in which an R pixel, a G pixel, and a B pixel provided with color filters of RGB and a pixel (a pixel having sensitivity only to infrared light) provided with a visible light cut filter are mixed is preferably used.

The LED light emission unit542has a white LED and a near infrared LED, turns on the white LED in a case where the amount of light of the subject is insufficient in an imaging mode of a color image, and emits pulse light from the near infrared LED according to the drive mode in an imaging mode of a distance image. In a case of the smartphone500having an infrared communication function, the near infrared LED may be used as a light source of infrared communication.

Under the control of the main control unit501, the camera unit541can convert image data obtained by imaging to compressed image data in a format of, for example, Joint Photographic coding Experts Group (JPEG) or the like and can record compressed image data in the storage unit550or can output compressed image data through the external input/output unit560or the wireless communication unit510, and similarly, can record distance image data indicating the distance image in the storage unit550or can output the distance image through the external input/output unit560or the wireless communication unit510. In the smartphone500shown inFIG.15, although the camera unit541is mounted on the same surface as the display input unit520, the mounting position of the camera unit541is not limited thereto, and the camera unit541may be mounted on a rear surface of the display input unit520, or a plurality of camera units541may be mounted. In a case where a plurality of camera units541are mounted, the camera unit541for imaging may be switched to perform imaging alone, or a plurality of camera units541may be used simultaneously to perform imaging.

The camera unit541can be used for various functions of the smartphone500. For example, an image acquired by the camera unit541can be displayed on the display panel521, or an image in the camera unit541can be used as one operation input of the operation panel522. In a case where the GPS reception unit570detects the position, the position may be detected with reference to an image from the camera unit541. In addition, the optical axis direction of the camera unit541of the smartphone500may be determined or a current use environment may be determined with reference to an image from the camera unit541without using the three-axis acceleration sensor or using the three-axis acceleration sensor. Of course, an image from the camera unit541may be used within application software.

[Others]

In this embodiment, although the first drive mode to the fourth drive mode are provided, one drive mode is set from the four drive modes, and the exposure control and the calculation processing corresponding to the set drive mode are performed, the invention is not limited thereto, and the invention may have a configuration in which a plurality of two or more drive modes among the first drive mode to the fourth drive mode (four drive modes) are provided, one drive mode is set from a plurality of drive modes, and exposure control and calculation processing corresponding to the set drive mode are performed.

In this embodiment, although pulse light emitted from the pulse light emission unit is applied as near infrared light, the invention is not limited thereto, and for example, light in a wavelength bandwidth other than near infrared light or white light in a wide band, such as white, may be applied.

EXPLANATION OF REFERENCES

10: distance image acquisition apparatus,12: imaging lens,14: distance image sensor,20: central processing unit (CPU),20A: drive mode setting unit,20B: distance image generation unit,20C: brightness detection unit,20D: determination unit,22: pulse light emission unit,24: exposure control unit,26: drive mode selection unit,500: smartphone,541: camera unit,542: LED light emission unit