Patent ID: 12241731

The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views.

DETAILED DESCRIPTION

In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.

Referring now to the drawings, embodiments of the present disclosure are described below. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

Embodiments will be described hereinafter with reference to the drawings.

First Embodiment

General Arrangement of Displacement Measurement Device100

FIG.1is a diagram illustrating the general arrangement of a displacement measurement device100according to a first embodiment. The displacement measurement device100illustrated inFIG.1is a device configured to irradiate a measurement object10(e.g., a hand of a person) with coherent light, detect a luminance-change coordinate point based on reflected light from the measurement object10, and measure the amount of micro-displacement of the measurement object10based on event data related to the detected luminance-change coordinate point.

The term “luminance-change coordinate point” refers to a pixel where a certain amount or more of luminance change has occurred on an image sensor. The term “event data” refers to data related to a pixel whose luminance has changed a certain amount or more and including the time (T) of the luminance change, the location (X, Y) of the luminance change, and the polarity (P) of the luminance change.

The amount of micro-displacement of the measurement object10, which is measured by the displacement measurement device100, is output to, for example, an apparatus external to the displacement measurement device100and is used by the external apparatus for display to a user or control of an external apparatus to be controlled.

As illustrated inFIG.1, the displacement measurement device100includes an irradiation unit110(also referred to as a “light projection unit”), an interference pattern forming unit120, a luminance-change coordinate point detection unit130, and an information processing unit150.

The irradiation unit110irradiates the measurement object10with coherent light. The irradiation unit110is preferably a laser light source having high coherence to form an interference pattern of light reflected from the measurement object10on a light-receiving surface of the luminance-change coordinate point detection unit130. Examples of the irradiation unit110include a laser diode (LD), a vertical-cavity surface-emitting laser (VCSEL), a small gas laser, and a solid state laser.

The interference pattern forming unit120forms an interference pattern from light reflected from the measurement object10(i.e., from coherent light reflected from the measurement object10). In this embodiment, the interference pattern forming unit120is disposed on an optical path of light reflected from the measurement object10between the measurement object10and the luminance-change coordinate point detection unit130. The interference pattern forming unit120has a function of adjusting the characteristics of an interference pattern received on the light-receiving surface of the luminance-change coordinate point detection unit130so that the amount of displacement of the measurement object10can be appropriately acquired. In an example, the interference pattern forming unit120includes a so-called wavefront control element such as a lens, an aperture, a phase shifter, or a spatial light modulator (SLM).

An example of interference patterns formed by the interference pattern forming unit120is a speckle image. The speckle image refers to random interference patterns caused by the roughness of the surface of the measurement object10. The speckle image reflects the characteristics of light as a wave motion, and has a luminance distribution that changes sensitively to the movement of the measurement object10. In other words, the speckle image is obtained by converting the sensitivity into a scale such that micro-displacement of the measurement object10is captured by the light-receiving surface of the luminance-change coordinate point detection unit130.

The luminance-change coordinate point detection unit130receives, on the light-receiving surface, the interference pattern formed by the interference pattern forming unit120, and detects a luminance-change coordinate point where a certain amount or more of luminance change has occurred, based on the received interference pattern. The luminance-change coordinate point detection unit130outputs event data related to the detected luminance-change coordinate point. An example configuration of the luminance-change coordinate point detection unit130will be described below with reference toFIGS.2and3.

The information processing unit150includes a displacement estimation unit151and an estimated displacement value output unit152.

The displacement estimation unit151calculates an estimated value of the amount of displacement of the measurement object10in a real space, based on the event data output from the luminance-change coordinate point detection unit130.

The estimated displacement value output unit152outputs the estimated value of the amount of displacement of the measurement object10, which is calculated by the displacement estimation unit151.

In the displacement measurement device100according to the first embodiment, the displacement estimation unit151is configured to calculate an estimated value of the amount of displacement of the measurement object10in a real space by using two numerical-sequence processing systems. The two numerical-sequence processing systems are provided in parallel in terms of hardware for two numerical sequences of two elements (the X coordinate and the Y coordinate) representing the locations of luminance-change coordinate points.

In other words, the displacement estimation unit151is configured to perform the two calculations in parallel. One of the calculations is performed by a first numerical-sequence processing system P1, and the other calculation is performed by a second numerical-sequence processing system P2. The first numerical-sequence processing system P1calculates the amount of displacement of the measurement object10along the X axis based on the numerical sequence of X-coordinate values. The second numerical-sequence processing system P2calculates the amount of displacement of the measurement object10along the Y axis based on the numerical sequence of Y-coordinate values. As used herein, the term “in parallel” means that operations can be performed independently of each other, and is used to include a case where time periods during which the operations are performed partially overlap.

Accordingly, the displacement measurement device100according to the first embodiment can detect the amount of displacement of the measurement object10at a high speed with reduced computational load.

Each of the functions of the information processing unit150can be implemented by one or more processing circuits or circuitry. As used herein, the term “processing circuit or circuitry” is used to include a processor programmed to implement each function by software, such as a processor implemented by an electronic circuit, and devices designed to implement the functions described above, such as an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), and existing circuit modules.

Example Configuration of Luminance-Change Coordinate Point Detection Unit130

FIG.2is a diagram illustrating an example configuration of the luminance-change coordinate point detection unit130included in the displacement measurement device100according to the first embodiment.

In the example illustrated inFIG.2, the luminance-change coordinate point detection unit130includes an event-based vision camera131.

The event-based vision camera131is equipped with an event-based vision sensor. The event-based vision sensor is configured to receive an interference pattern, instantaneously (i.e., in a very short time and at a very high speed) detect a luminance-change coordinate point where a certain amount or more of luminance change has occurred in a two-dimensional array of pixels, and output event data including the time (T) of the luminance change, the location (X, Y) of the luminance change, and the polarity (P) of the luminance change. Accordingly, the event-based vision camera131can directly generate event data.

In the displacement measurement device100, the event-based vision camera131of the luminance-change coordinate point detection unit130is used to quickly acquire a speckle image that sensitively changes and reliably capture micro-displacement of the measurement object10.

Other Example Configuration of Luminance-Change Coordinate Point Detection Unit130

FIG.3is a diagram illustrating another example configuration of the luminance-change coordinate point detection unit130included in the displacement measurement device100according to the first embodiment.

In the example illustrated inFIG.3, the luminance-change coordinate point detection unit130includes a frame camera132, a frame-to-frame luminance difference computation unit133, and a luminance-change coordinate point extraction unit134. The frame-to-frame luminance difference computation unit133and a luminance-change coordinate point extraction unit134are implemented by circuitry, such as a CPU.

The frame camera132captures a normal frame image in which the interference pattern appears, and outputs the frame image. The frame-to-frame luminance difference computation unit133calculates a luminance difference between corresponding pixels in two continuous frame images output from the frame camera132. The luminance-change coordinate point extraction unit134extracts, as a luminance-change coordinate point, a pixel where a luminance difference greater than or equal to a certain value is calculated by the frame-to-frame luminance difference computation unit133. The luminance-change coordinate point extraction unit134outputs event data related to the extracted luminance-change coordinate point. The event data includes the time (T) of the luminance change, the location (X, Y) of the luminance change, and the polarity (P) of the luminance change.

Example of Event Data output by Luminance-Change Coordinate Point Detection Unit130

FIGS.4A to4Care illustrations of an example of event data output by the luminance-change coordinate point detection unit130included in the displacement measurement device100according to the first embodiment.

FIGS.4A and4Billustrate examples of a speckle image formed on a light-receiving surface of the event-based vision sensor included in the event-based vision camera131.FIG.4Aillustrates a speckle image400A at time t.FIG.4Billustrates a speckle image400B at time t+Δt.FIG.4Cillustrates event data410, which is an example of the event data output from the event-based vision camera131. The event data410is generated based on the speckle image400A illustrated inFIG.4Aand the speckle image400B illustrated inFIG.4B.

The event-based vision sensor included in the event-based vision camera131is a photodetector that outputs event data. In response to detection of a pixel (i.e., a luminance-change coordinate point) where a luminance change exceeding a predetermined threshold has occurred among pixels arranged in a two-dimensional array (i.e., in response to the occurrence of an event), the event-based vision sensor outputs, as event data, data including the time (T) of the luminance change, the location (X, Y) of the luminance change, and the polarity (P) of the luminance change. The polarity (P) may have a binary value, namely, “1” for increase or “0” for decrease.

In an example, the speckle image400A at the time t illustrated inFIG.4Ais horizontally translated, and, after Δt, the speckle image400B at the time t+Δt illustrated inFIG.4Bis produced. In this case, as indicated by the event data410illustrated inFIG.4C, with the polarities of luminance changes on the event-based vision sensor at the time t+Δt, the speckles are spatially distributed with decrease components410A and increase components410B. InFIG.4C, the decrease components410A are represented as lightly shaded regions, and the increase components410B are represented as heavily shaded regions. In the decrease components410A, the luminance values are decreased by a value greater than or equal to a certain value. In the increase components410B, the luminance values are increased by a value greater than or equal to a certain value.

The event-based vision sensor outputs, for all the pixels in the increase components410B, a time-series data group of signals including the times (T) of signal detection, the pixel locations (X, Y), and the polarity (“1”: increase). The event-based vision sensor also outputs, for all the pixels in the decrease components410A, a time-series data group of signals including the times (T) of signal detection, the pixel locations (X, Y), and the polarity (“0”: decrease). The event-based vision sensor does not output data for the pixels in the other regions (non-shaded regions inFIG.4C) that do not correspond to either the region of the decrease components410A or the region of the increase components410B. Accordingly, the event data410output from the event-based vision sensor has a much smaller amount of data than frame image data.

In this manner, the event-based vision sensor has no limitation in terms of frame rate and can output, as event data, shift information between speckle images at a higher speed than an image sensor that outputs frame image data.

For example, the event-based vision sensor has a sampling time of about 1 μs to 200 μs for all event data in the sensor surface. The sampling time of the event-based vision sensor is much faster than the frame rate of an ordinary video camera or the like. Accordingly, the event-based vision sensor included in the luminance-change coordinate point detection unit130can quickly and reliably detect a shift between speckle images, which changes sensitively to the displacement of the measurement object10.

Principle of Displacement Estimation Using Displacement Estimation Unit151

FIGS.5A to5Care illustrations of the principle of displacement estimation using the displacement estimation unit151included in the displacement measurement device100according to the first embodiment.

FIGS.5A and5Billustrate examples of a frame image based on the event data output from the event-based vision camera131.FIG.5Aillustrates a frame image500A at time t.FIG.5Billustrates a frame image500B at time t+Δts. In the frame images500A and500B, only increase components in which the luminance values are increased by a value greater than or equal to a certain value are illustrated.FIG.5Cillustrates an example of a cross-correlation function calculated based on the frame images500A and500B.

First, in the principle of displacement estimation, to obtain the frame image500A illustrated inFIG.5Afrom the time-series event data output from the event-based vision camera131, an integration time for calculating unit frames is set, a two-dimensional matrix representing an image is prepared, and the number of times a row and column number corresponding to a pixel location appears in the integration time is counted.

Subsequently, in the principle of displacement estimation, as illustrated inFIG.5B, the time of the frame image500A illustrated inFIG.5A, for example, a time distant from the integration start time (t) by Δts, is set as the start point of integration, and the number of times the row and column number appears in the integration time is also counted.

In the frame image500B illustrated inFIG.5B, dotted-line portions represent patterns in the frame image500A illustrated inFIG.5A. The patterns are translated to lightly shaded locations in the frame image500B illustrated inFIG.5B.

Subsequently, in the principle of displacement estimation, a cross-correlation function (seeFIG.5C) between the frame image500A and the frame image500B is calculated.

As illustrated inFIG.5C, when a variable of the correlation function, that is, an amount of displacement, is plotted on the ΔX axis and the ΔY axis, a peak value of the cross-correlation function is obtained at a distance and location (an amount of pixel displacement in the image) corresponding to the amount of translation of the speckle pattern. For example, in the principle of displacement estimation, any one of the frame images500A and500B is shifted in the X-axis direction and the Y-axis direction to obtain an overlapping region between the frame images500A and500B, and such overlapping regions are integrated to calculate the cross-correlation function illustrated inFIG.5C.

In the principle of displacement estimation, in an example, the following method (the Wiener-Khinchin theorem) may be used to calculate the cross-correlation function: The frame images500A and500B are subjected to a Fourier transform, and the complex conjugate of one of the frame images500A and500B is multiplied by the complex conjugate of the other of the frame images500A and500B before an inverse Fourier transform is performed.

Principle of Displacement Estimation Method Using Displacement Estimation Unit151

FIGS.6A to6Care illustrations of the principle of a method for estimating the amount of displacement of a measurement object by using the displacement estimation unit151according to an embodiment.

FIG.6Aillustrates an optical system included in the displacement measurement device100according to an embodiment.FIG.6Billustrates an example of event data groups output from the event-based vision camera131included in the displacement measurement device100according to an embodiment.FIG.6Cillustrates an image of event data groups at two different times.

As illustrated inFIG.6A, the displacement measurement device100irradiates a rough surface10A, which is the measurement object10, with coherent light from the irradiation unit110and receives light reflected from the rough surface10A on the light-receiving surface of the luminance-change coordinate point detection unit130to measure a speckle image formed on the light-receiving surface of the event-based vision camera131.

As illustrated inFIG.6A, if the rough surface10A is displaced, the speckle image on the light-receiving surface of the luminance-change coordinate point detection unit130also translates in the same direction. The ratio of the amount of translation of the speckle image on the light-receiving surface of the luminance-change coordinate point detection unit130to the amount of displacement of the rough surface10A is constant under the same conditions. Accordingly, the displacement measurement device100can estimate the amount of displacement of the measurement object10based on the translation of the speckle image on the light-receiving surface of the luminance-change coordinate point detection unit130.

InFIG.6B, an event data group600is an event data group including 100 items of event data that are continuous in time series. The event data group600is acquired at times before the displacement of the rough surface10A. InFIG.6B, an event data group601is an event data group including 100 items of event data that are continuous in time series. The event data group601is acquired at times after the displacement of the rough surface10A. As illustrated inFIG.6B, each item of event data in the event data groups600and601includes the time of a luminance change, coordinates of the luminance change, and the polarity of the luminance change.

Accordingly, the 100 items of event data included in the event data group600illustrated inFIG.6Bare summed up to obtain a plurality of speckle images610illustrated inFIG.6C.

Further, the 100 items of event data included in the event data group601illustrated inFIG.6Bare summed up to obtain a plurality of speckle images610′ illustrated inFIG.6C.

The plurality of speckle images610are translated to the plurality of speckle images610′ in response to the displacement of the rough surface10A. Accordingly, the difference between the plurality of speckle images610′ and the plurality of speckle images610can be used to estimate the amount of displacement of the rough surface10A.

Since the time between the event data groups600and601is very short, it can be assumed that the rough surface10A and the plurality of speckle images610move with constant velocity. Accordingly, the patterns of the plurality of speckle images610and the patterns of the plurality of speckle images610′ are substantially the same.

Example of Displacement Calculation Method Using Displacement Estimation Unit151

FIGS.7A to7Care illustrations of an example method for calculating the amount of displacement of the measurement object10by using the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

A method for calculating the amount of displacement of the measurement object10will be described with reference toFIGS.7A to7C, typically using five luminance-change coordinate points included in an event data group.

InFIGS.7A to7C, luminance-change coordinate points700are detected at times before the displacement of the rough surface10A. InFIGS.7A to7C, luminance-change coordinate points700′ are detected at times after the displacement of the rough surface10A. The luminance-change coordinate points700′ are obtained by translating the luminance-change coordinate points700.

First, as illustrated inFIG.7A, the displacement estimation unit151focuses on the first luminance-change coordinate point700in the time series and calculates a difference value between the coordinate value of the first luminance-change coordinate point700and the coordinate value of each of the five luminance-change coordinate points700′.

Subsequently, as illustrated inFIG.7B, the displacement estimation unit151focuses on the second luminance-change coordinate point700in the time series and calculates a difference value between the coordinate value of the second luminance-change coordinate point700and the coordinate value of each of the five luminance-change coordinate points700′.

Likewise, the displacement estimation unit151focuses on the third luminance-change coordinate point700in the time series and calculates a difference value between the coordinate value of the third luminance-change coordinate point700and the coordinate value of each of the five luminance-change coordinate points700′.

Likewise, the displacement estimation unit151focuses on the fourth luminance-change coordinate point700in the time series and calculates a difference value between the coordinate value of the fourth luminance-change coordinate point700and the coordinate value of each of the five luminance-change coordinate points700′.

Finally, as illustrated inFIG.7C, the displacement estimation unit151focuses on the fifth luminance-change coordinate point700in the time series and calculates a difference value between the coordinate value of the fifth luminance-change coordinate point700and the coordinate value of each of the five luminance-change coordinate points700′.

In other words, the displacement estimation unit151calculates, for all of the luminance-change coordinate points700included in the event data group before the displacement of the rough surface10A, difference values in coordinate value from each of the luminance-change coordinate points700′ included in the event data group after the displacement of the rough surface10A.

Accordingly, for example, in a case where the event data group before the displacement of the rough surface10A includes 100 items of event data and the event data group after the displacement of the rough surface10A includes 100 items of event data, the displacement estimation unit151calculates 100×100=10000 difference values.

Focus is placed on one luminance-change coordinate point700included in the event data group before the displacement. A plurality of difference values calculated for the luminance-change coordinate point700include one difference value (Δx, Δy) in the same direction and at the same distance as the direction and distance of translation of the entire speckle. This also applies to all of the luminance-change coordinate points700.

As a result, the difference values calculated by the displacement estimation unit151include a number of difference values (Δx, Δy) in the same direction and at the same distance as the direction and distance of translation of the entire speckle, the number of difference values (Δx, Δy) being equal to the number of items of event data included in the event data group before the displacement.

Accordingly, the frequency distribution of the plurality of difference values calculated by the displacement estimation unit151is converted into a histogram having peaks at difference values (Δx, Δy) in the same direction and at the same distance as the direction and distance of translation of the entire speckle.

Frequency Distribution of Difference Values Calculated by Displacement Estimation Unit151

FIG.8is a histogram representing a frequency distribution of difference values calculated by the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

The histogram illustrated inFIG.8represents a frequency distribution of difference values calculated by the displacement estimation unit151using the method described with reference toFIGS.7A to7C. InFIG.8, the horizontal axis represents the difference value between coordinate values (X coordinates or Y coordinates), and the vertical axis represents the frequency.

As illustrated inFIG.8, the histogram representing the frequency distribution of difference values has a peak at a certain difference value. The difference value at the peak is equal to the actual amount of translation (actual amount of movement) of the speckle image.

Accordingly, the displacement estimation unit151can identify the mode of the difference values as the actual amount of translation of the speckle image. Based on the mode, the displacement estimation unit151can estimate the amount of displacement of the rough surface10A of the measurement object10.

As described above, the displacement estimation unit151can directly compute differences between numerical sequences to calculate the amount of translation of a speckle image without generating two speckle images by the integration of two event data groups. Thus, the displacement estimation unit151can calculate the amount of displacement of the measurement object10at a high speed with reduced computational load.

Functional Configuration of Displacement Estimation Unit151

FIG.9is a diagram illustrating an example functional configuration of the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

As illustrated inFIG.9, the displacement estimation unit151includes an event data group generation unit171and an element-specific numerical sequence generation unit172.

The event data group generation unit171acquires a predetermined number of items of event data output from the luminance-change coordinate point detection unit130and generates an event data group including the predetermined number of items of event data.

The element-specific numerical sequence generation unit172generates a numerical sequence for each element in the event data group generated by the event data group generation unit171. Specifically, the element-specific numerical sequence generation unit172generates a first numerical sequence including a set of X coordinates (an example of a “first element”) and a second numerical sequence including a set of Y coordinates (an example of a “second element”).

The displacement estimation unit151includes the first numerical-sequence processing system P1and the second numerical-sequence processing system P2.

The first numerical-sequence processing system P1performs computation of the first numerical sequence including the set of X coordinates (an example of a “first element”) representing the locations of the luminance-change coordinate points extracted from the event data to calculate the amount of displacement of the measurement object10along the X coordinates.

The second numerical-sequence processing system P2performs computation of the second numerical sequence including the set of Y coordinates (an example of a “second element”) representing the locations of the luminance-change coordinate points extracted from the event data to calculate the amount of displacement of the measurement object10along the Y coordinates.

The first numerical-sequence processing system P1and the second numerical-sequence processing system P2each include a numerical-sequence correction unit173, a computation-combination selection unit174, and a displacement derivation unit175.

The numerical-sequence correction unit173performs predetermined correction on the first numerical sequence or the second numerical sequence. Examples of the predetermined correction include a process of extracting a luminance-change coordinate point of event data having either positive or negative polarity, and a sorting process.

The computation-combination selection unit174selects combinations of luminance-change coordinate points to be subjected to computation for the first numerical sequence or the second numerical sequence that is corrected by the numerical-sequence correction unit173. Specifically, the computation-combination selection unit174selects combinations of luminance-change coordinate points included in one event data group and luminance-change coordinate points included in the other event data group (e.g., round-robin combinations or combinations of corresponding luminance-change coordinate points in the descending orders).

The displacement derivation unit175calculates difference values between coordinate values for each of the combinations of luminance-change coordinate points selected by the computation-combination selection unit174. Then, the displacement derivation unit175identifies the mode of the calculated difference values as the actual amount of translation of the speckle image on the X axis or the Y axis. Based on the mode, the displacement derivation unit175estimates the amount of displacement of the measurement object10on the X axis or the Y axis. Further, the displacement derivation unit175outputs the estimated amount of displacement of the measurement object10on the X axis or the Y axis to the estimated displacement value output unit152.

In an example, the displacement estimation unit151does not include the numerical-sequence correction unit173or the computation-combination selection unit174. In other words, in an example, the displacement estimation unit151does not correct numerical sequences or select combinations of luminance-change coordinate points to be subjected to computation. In this example, the displacement estimation unit151may automatically select “round-robin combinations”.

Example of Operation Procedure Using Displacement Estimation Unit151

FIG.10is a flowchart illustrating an example operation procedure performed by the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

First, the event data group generation unit171acquires the event data output from the luminance-change coordinate point detection unit130(step S101). Subsequently, the event data group generation unit171stores the event data acquired in step S101in a memory included in the displacement measurement device100(step S102).

The event data group generation unit171repeatedly performs the processing of steps S101and S102to store a predetermined number of (for example,100) items of event data in the memory. Subsequently, the event data group generation unit171generates an event data group from the predetermined number of items of event data (step S103).

Further, the event data group generation unit171performs the processing of steps S101to S103twice to generate event data groups obtained at two different times.

Subsequently, the element-specific numerical sequence generation unit172generates, for each of the two generated event data groups, a first numerical sequence including a set of X coordinates and a second numerical sequence including a set of Y coordinates (step S104).

Subsequently, in the first numerical-sequence processing system P1, the numerical-sequence correction unit173performs predetermined correction (e.g., classification based on polarity, and sorting) on the first numerical sequence (step S105). Further, the computation-combination selection unit174selects combinations of luminance-change coordinate points to be subjected to computation for the first numerical sequence corrected in step S105(step S106). Subsequently, the displacement derivation unit175calculates difference values between coordinate values for all the combinations of luminance-change coordinate points selected in step S106, and identifies the mode of the calculated difference values as the actual amount of translation of the speckle image on the X axis. Based on the mode, the displacement derivation unit175estimates the amount of displacement of the measurement object10on the X axis (step S107).

In parallel with the processing of steps S105to S107, in the second numerical-sequence processing system P2, the numerical-sequence correction unit173performs predetermined correction (e.g., classification based on polarity, and sorting) on the second numerical sequence (step S108). Further, the computation-combination selection unit174selects combinations of luminance-change coordinate points to be subjected to computation for the second numerical sequence corrected in step S108(step S109). Subsequently, the displacement derivation unit175calculates difference values between coordinate values for all the combinations of luminance-change coordinate points selected in step S109, and identifies the mode of the calculated difference values as the actual amount of translation of the speckle image on the Y axis. Based on the mode, the displacement derivation unit175estimates the amount of displacement of the measurement object10on the Y axis (step S110).

Further, the displacement derivation unit175outputs the amount of displacement of the measurement object10on the X axis, which is estimated in step S107, the amount of displacement of the measurement object10on the Y axis, which is estimated in step S110, to the estimated displacement value output unit152(step S111).

Thereafter, the displacement estimation unit151ends the series of operations illustrated inFIG.10. The displacement estimation unit151repeatedly performs the series of operations illustrated inFIG.10to continuously estimate the amount of displacement of the measurement object10at a high speed.

Example of Amount of Calculation with Displacement Estimation Unit151

FIG.11is a graph illustrating an example of the amount of calculation with the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

The graph illustrated inFIG.11depicts the amount of calculation to be performed with the displacement measurement device100when the event-based vision camera131including 320×240 picture elements outputs 10000 items of event data per second.

In the graph illustrated inFIG.11, the horizontal axis represents an event data group per second, which indicates the velocity resolution of a displacement. The larger the value of the event data group per second is, the faster the displacement of the measurement object10can be followed. When an event data group includes n items of event data, the value of the event data group per second is given by dividing n by the number of items of event data output per second.

In the graph illustrated inFIG.11, legends indicate the round-robin method, which is represented by a solid line with triangles, and the image correlation method, which is represented by a broken line with circles. The round-robin method is used for the displacement measurement device100according to an embodiment. By contrast, the image correlation method is used for an existing displacement measurement device.

The round-robin method is a method for calculating difference values for all combinations of all luminance-change coordinate points included in one event data group and all luminance-change coordinate points included in the other event data group and calculating the mode of the calculated difference values as the amount of displacement of the measurement object10.

The image correlation method is a method for determining the amount of displacement of the measurement object10from a correlation between two images generated from two event data groups.

The round-robin method involves calculation of difference values for all combinations. The amount of calculation for determining difference values by round robin is given by O(n2), and the amount of calculation for determining frequency distributions is given by O(n2). The total amount of calculation is given by O(n2). In the round-robin method, the amount of calculation for determining the amount of displacement of the measurement object10can be expressed by Equation (1) below.
O(n2)=O(n2)+O(n2)  (1)

In the round-robin method, accordingly, the amount of calculation per operation is in proportion to the number n of items of event data included in the event data group. The event data group per second is in inverse proportion to the number n of items of event data included in the event data group. Therefore, the amount of calculation for the event data group per second is in an inverse proportional relationship.

In the image correlation method, two images are subjected to a Fourier transform to generate a composite image, and the composite image is subjected to an inverse Fourier transform. The amount of calculation for the Fourier transform and the inverse Fourier transform is represented by N log N, where N is the number of pixels. The amount of calculation for generating the composite image is N. In the image correlation method, the amount of calculation for determining the amount of displacement of the measurement object10can be expressed by Equation (2) below.
O(NlogN)=O(NlogN)+O(N)  (2)

In the image correlation method, accordingly, the amount of calculation per operation does not depend on the number n of items of event data included in the event data group. Thus, the amount of calculation for the event data group per second is substantially in proportion.

The graph illustrated inFIG.11indicates that, in a situation in which the event data group per second exceeds about 275 (situation in which the measurement object10is displaced at a high speed), the round-robin method, which is used for the displacement measurement device100according to an embodiment, can estimate the amount of displacement of the measurement object10with a smaller amount of calculation than the image correlation method, which is used for the existing displacement measurement device.

Other Example of Displacement Calculation Method Using Displacement Estimation Unit151

FIGS.12A and12Bare illustrations of another example method for calculating the amount of displacement of the measurement object10by using the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

Referring toFIGS.12A and12B, a method for calculating the amount of displacement of the measurement object10will be described typically using five luminance-change coordinate points included in an event data group.

InFIGS.12A and12B, luminance-change coordinate points700are detected at times before the displacement of the rough surface10A. InFIGS.12A and12B, luminance-change coordinate points700′ are detected at times after the displacement of the rough surface10A. The luminance-change coordinate points700′ are obtained by translating the luminance-change coordinate points700.

The numbers assigned to the luminance-change coordinate points700and700′ represent the orders of the coordinate values on the Y axis that are sorted in descending order. InFIG.12B, the luminance-change coordinate points700and700′ illustrated inFIG.12Aare arranged on the same line (i.e., along the Y axis) in descending order of the coordinate values on the Y axis.

In the sort method, the displacement estimation unit151calculates a difference value between the coordinate value of each of the five luminance-change coordinate points700and the coordinate value of the luminance-change coordinate point700′ having the same number.

For example, the displacement estimation unit151calculates a difference value between the coordinate value of the luminance-change coordinate point700assigned “1” and the coordinate value of the luminance-change coordinate point700′ assigned “1”.

For example, the displacement estimation unit151calculates a difference value between the coordinate value of the luminance-change coordinate point700assigned “2” and the coordinate value of the luminance-change coordinate point700′ assigned “2”.

Likewise, the displacement estimation unit151calculates a difference value between the coordinate value of the luminance-change coordinate point700assigned any other number and the coordinate value of the luminance-change coordinate point700′ assigned the same number.

As a result, the displacement estimation unit151can calculate a difference value between each of the plurality of luminance-change coordinate points700and the luminance-change coordinate point700′ assigned the same number. In other words, the displacement estimation unit151can calculate a difference value equal to the total amount of translation. Accordingly, the displacement estimation unit151can calculate difference values with a smaller amount of computation than that in the round-robin method illustrated inFIGS.7A to7C.

In the sort method, in an example, the displacement estimation unit151may calculate difference values between each of the five luminance-change coordinate points700and the luminance-change coordinate point700′ assigned the same number and a plurality of luminance-change coordinate points700′ assigned nearby numbers.

For example, the displacement estimation unit151may calculate a difference value between the luminance-change coordinate point700assigned “2” and the luminance-change coordinate point700′ assigned “2”, a difference value between the luminance-change coordinate point700assigned “2” and the luminance-change coordinate points700′ assigned “1”, and a difference value between the luminance-change coordinate point700assigned “2” and the luminance-change coordinate points700′ assigned “3”.

Accordingly, even if the numbers are not aligned between the luminance-change coordinate points700and the luminance-change coordinate points700′ due to noise, overlapping of event data, or any other factor, the displacement estimation unit151can calculate a difference value between each of the luminance-change coordinate points700and a corresponding one of the luminance-change coordinate points700′.

Example of Amount of Calculation with Displacement Estimation Unit151

FIG.13is a graph illustrating another example of the amount of calculation with the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

The graph illustrated inFIG.13depicts the amount of calculation to be performed with the displacement measurement device100when the event-based vision camera131including 320×240 picture elements outputs 10000 items of event data per second.

In the graph illustrated inFIG.13, legends indicate the round-robin method, which is represented by a solid line with triangles, the image correlation method, which is represented by a broken line with circles, and the sort method, which is represented by a dash-dot line with rectangles. The sort method is another method used for the displacement measurement device100according to an embodiment.

In the sort method, in an example, for each source luminance-change coordinate point (corresponding to each of the luminance-change coordinate points700illustrated inFIGS.12A and12B), the range of the target luminance-change coordinate points (corresponding to corresponding ones of the luminance-change coordinate points700′ illustrated inFIGS.12A and12B) is set to “the same number ±3”. In other words, the number of target luminance-change coordinate points (the number of nearby areas) is set to “7”.

The sort method involves a calculation of n log(n) to sort a sequence of numbers in an event data group. A calculation of 2n log(n) is used to calculate the calculation on two sequences of numbers.

In this example, the sort method involves a calculation of 7n, where the number of items of event data included in the event data group is represented by n and the number of target luminance-change coordinate points (the number of nearby areas) is represented by “7”.

Furthermore, the sort method involves a calculation of O(n log n) to sort a sequence of numbers in an event data group, a calculation of O(n) for coordinate difference calculation, and a calculation of O(n) for frequency distribution calculation. Thus, the sort method involves a calculation of O(n log n) in total.

In the sort method, the amount of calculation for determining the amount of displacement of the measurement object10can be expressed by Equation (3) below.
O(nlogn)=O(nlogn)+O(n)  (3)

In the sort method, accordingly, the amount of calculation per operation is in proportion to n log(n). The event data group per second is in inverse proportion to the number n of items of event data included in the event data group. Accordingly, the amount of calculation for the event data group per second is in proportion to log(n). When n is large, the amount of calculation is substantially constant.

The graph illustrated inFIG.13indicates that the sort method can estimate the amount of displacement of the measurement object10with a smaller amount of calculation than the round-robin method.

Other Example Functional Configuration of Displacement Estimation Unit151

FIG.14is a diagram illustrating another example functional configuration of the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

In the displacement measurement device100illustrated inFIG.14, the event-based vision camera131of the luminance-change coordinate point detection unit130includes a light-receiving element131A having an exposure area131B. The exposure area131B is scanned in the X-axis direction over time. Accordingly, the X-coordinate values of the event data output from the event-based vision camera131are periodically sorted in advance.

With this configuration, the displacement estimation unit151sorts only the second numerical sequence on the Y-axis, and the numerical-sequence correction unit173of the first numerical-sequence processing system P1may be omitted. In the displacement measurement device100illustrated inFIG.14, therefore, the amount of computation performed by the displacement estimation unit151can further be reduced.

Still Other Example Functional Configuration of Displacement Estimation Unit151

FIG.15is a diagram illustrating still another example functional configuration of the displacement estimation unit151included in the displacement measurement device100according to an embodiment.

The displacement measurement device100illustrated inFIG.15is different from the displacement measurement device100illustrated inFIG.9in that the first numerical-sequence processing system P1and the second numerical-sequence processing system P2each include a frequency-distribution array generation unit176in place of the numerical-sequence correction unit173. The frequency-distribution array generation unit176generates two frequency-distribution arrays from two event data groups obtained at different times.

The displacement measurement device100illustrated inFIG.15is further different from the displacement measurement device100illustrated inFIG.9in that the displacement derivation unit175in each of the first numerical-sequence processing system P1and the second numerical-sequence processing system P2uses a convolution operation method instead of the round-robin method to calculate the amount of displacement of the measurement object10.

The displacement measurement device100illustrated inFIG.15is further different from the displacement measurement device100illustrated inFIG.9in that the first numerical-sequence processing system P1and the second numerical-sequence processing system P2each do not include the computation-combination selection unit174.

The method for calculating the amount of displacement of the measurement object10by using the convolution operation method, which is performed by the displacement derivation units175included in the displacement measurement device100illustrated inFIG.15, will be described, with focus on the X-axis coordinate elements of the event data groups.

In the following description, the number of sensors of an event camera in the X-axis direction is denoted by M, a chronologically earlier event data group is denoted by A, and a chronologically later event data group is denoted by B.

The event data group A includes N items of event data, and the X coordinates of the N items of event data are represented by Ax0, Ax1, Ax2, . . . , and AxN−1.

The event data group B includes N items of event data, and the X coordinates of the N items of event data are represented by Bx0, Bx1, Bx2, . . . , and BxN−1.

The coordinates Axi and Bxi (i=0, 1, . . . N−1) are integers in the range (0 to M−1) of the X coordinates.

Arrays focusing on the numbers of items of event data having the same coordinate among the coordinates Ax and Bx, which are respectively the coordinate information of the event data groups A and B, are denoted by s and t, respectively. In the array s, the number of items of event data having Ax expressed by M−1−i within the event data group A is defined as s[i]. In the array t, the number of items of event data having Bx expressed by i within the event data group B is defined as t[i]. From the arrays s and t, an array d obtained by Equation (4) below can be defined.
d[k]=Σi+j=ks[i]·t[j](k=0,1, . . . ,2M−2)  (4)

The value d[k] is equal to the number of sets of integers (i, j) (0≤i and j≤N−1) that satisfy Bxj−Ax_i=k−(M−1).

Transforming the right side of Equation (4) yields Equation (5) below, which can be expressed in the form of convolution.

d[k]=∑i+j=ks[i]·t[j]=∑i=0M-1s[i]·[k-i]⁢(k=0,1,…,2⁢M-2)(5)

Accordingly, if the arrays s, t, and d are subjected to a discrete Fourier transform to obtain arrays S, T, and D, respectively, the array D can be calculated by the following equation.
D[k]=S[k]·T[k](k=0,1, . . . ,2M−2)
The array D (first array) is further subjected to an inverse discrete Fourier transform to obtain the array d (second array). Since the array d indicates the frequency of a difference coordinate, the mode of the array d is extracted to estimate the amount of translation of the speckle.

In the convolution operation method, the frequency of a difference between numerical sequences is determined by using a discrete Fourier transform. It is thus preferable that the first numerical-sequence processing system P1and the second numerical-sequence processing system P2each include an FPGA, a GPU, or the like that can perform advanced computational processing.

Hardware Configuration of Information Processing Unit150

FIG.16is a diagram illustrating an example hardware configuration of the information processing unit150included in the displacement measurement device100according to an embodiment.FIG.16illustrates an example hardware configuration of the information processing unit150in a case where the information processing unit150is implemented by a personal computer.

As illustrated inFIG.16, the information processing unit150is implemented by a computer and includes central processing units (CPUs)201A and201B, a read-only memory (ROM)202, a random access memory (RAM)203, a hard disk (HD)204, a hard disk drive (HDD) controller205, a display206, an external device connection interface (I/F)208, a network I/F209, a data bus210, a keyboard211, a pointing device212, a digital versatile disc rewritable (DVD-RW) drive214, and a media I/F216.

The CPUs201A and201B control the overall operation of the information processing unit150. The ROM202stores a program such as an initial program loader (IPL) to boot the CPUs201A and201B. The RAM203is used as a work area for the CPUs201A and201B. The HD204stores various data such as a program. The HDD controller205controls reading or writing of various data from or to the HD204under the control of the CPUs201A and201B.

The display206displays various kinds of information such as a cursor, a menu, a window, text, or an image. The external device connection I/F208is an interface for connecting to various external devices. Examples of the external devices include, but are not limited to, a universal serial bus (USB) memory and a printer. The network I/F209is an interface for performing data communication using a communication network. The data bus210is an address bus, a data bus, or the like for electrically connecting various components illustrated inFIG.16, such as the CPUs201A and201B, to each other.

The keyboard211is an example of an input unit provided with a plurality of keys to allow the user to enter characters, numerical values, or various instructions. The pointing device212is an example of an input unit that allows the user to select or execute a specific instruction, select a target for processing, or move a cursor being displayed. The DVD-RW drive214controls reading or writing of various data from or to a DVD-RW213, which is an example of a removable recording medium. The removable recording medium is not limited to a DVD-RW and may be a digital versatile disc recordable (DVD-R), for example. The media I/F216controls reading or writing (storing) of data from or to a recording medium215such as a flash memory.

As described above, the information processing unit150includes two CPUs (i.e., the CPUs201A and201B). For example, the CPU201A executes processing of the first numerical-sequence processing system P1. For example, the CPU201B executes processing of the second numerical-sequence processing system P2. The CPUs201A and201B can execute processing operations in parallel with each other. Accordingly, the information processing unit150can execute, in parallel, computation of the first numerical sequence by using the CPU201A (i.e., the first numerical-sequence processing system P1) and computation of the second numerical sequence by using the CPU201B (i.e., the second numerical-sequence processing system P2).

The first numerical-sequence processing system P1and the second numerical-sequence processing system P2may be any combination of hardware components that can execute processing operations in parallel with each other. In an example, the first numerical-sequence processing system P1and the second numerical-sequence processing system P2are not limited to a combination of two CPUs, and a combination of two computation circuits may be used. In another example, a combination of two computers or any other suitable combination of hardware components may be used.

First Example Implementation

FIG.17is a schematic view of a non-contact input apparatus1100, which is a first example implementation of the displacement measurement device100according to an embodiment.FIG.18is a diagram illustrating the cross section of the non-contact input apparatus1100, which is the first example implementation of the displacement measurement device100according to the embodiment.

As illustrated inFIGS.17and18, the non-contact input apparatus1100includes a housing1101, an image display unit1102, an image forming plate1103, an optical window1104, a non-contact input identification unit1105, and a displacement measurement device100. In an example, the non-contact input apparatus1100may include the displacement measurement device100according to an embodiment. InFIGS.17and18, the illustration of the interference pattern forming unit120included in the displacement measurement device100is omitted.

In the non-contact input apparatus1100, the irradiation unit110included in the displacement measurement device100emits coherent light, which emerges as a light sheet, upward and forward from the housing1101. In other words, the irradiation unit110emits coherent light near a virtual image formed by the image display unit1102and the image forming plate1103. In response to the measurement object10(e.g., the finger of the operator) moving transversely to the light sheet for an operation on the virtual image without contact of the measurement object10, reflected light of the light sheet from the measurement object10is incident on the luminance-change coordinate point detection unit130included in the displacement measurement device100in the housing1101through the optical window1104as an interference pattern.

Accordingly, the information processing unit150included in the displacement measurement device100can detect the amount of micro-displacement of the measurement object10and output information indicating the detected amount of micro-displacement of the measurement object10to the non-contact input identification unit1105.

The non-contact input identification unit1105can accurately detect the non-contact operation made by the measurement object10(e.g., a push of the finger, handwriting, or a swipe of the finger), based on the information output from the displacement measurement device100, which indicates the amount of micro-displacement of the measurement object10. Further, the non-contact input identification unit1105can output the detection result to an operation target apparatus or feed back the detection result to the operator. The non-contact input identification unit1105may be implemented by, for example, a PC, which may include a processor, memory, and interface.

In the non-contact input apparatus1100, in an example, the image forming plate1103may be used to form a virtual image from video information or an image displayed on the image display unit1102and display the virtual image above and in front of the housing1101to improve the operability. As illustrated inFIG.18, the image forming plate1103is a member having characteristics of transmission and deflection of light beams. In an example, the image forming plate1103is implemented by a layered reflection structure.

In the non-contact input apparatus1100including the displacement measurement device100according to an embodiment, the displacement measurement device100can quickly and reliably capture a small non-contact movement of the measurement object10(the finger of the operator). In other words, the displacement measurement device100can accurately detect a non-contact operation of the measurement object10(the finger of the operator).

Second Example Implementation

FIG.19is a schematic view of a tremor measurement apparatus1200, which is a second example implementation of the displacement measurement device100according to an embodiment.FIG.20is a diagram illustrating the cross section of the tremor measurement apparatus1200, which is the second example implementation of the displacement measurement device100according to the embodiment.

As illustrated inFIGS.19and20, the tremor measurement apparatus1200includes a housing1201, a cylindrical lens1202, a turning mirror1203, an optical window1204, a support1205, a display device1206, and a displacement measurement device100. The tremor measurement apparatus1200is an example of a “biological micromotion measurement apparatus”. In an example, the tremor measurement apparatus1200may include the displacement measurement device100according to an embodiment. InFIGS.19and20, the illustration of the interference pattern forming unit120included in the displacement measurement device100is omitted. The cylindrical lens1202, the turning mirror1203, and the optical window1204are collectively referred to as an optical system.

The tremor measurement apparatus1200illustrated inFIGS.19and20is an apparatus configured to detect slight movements (e.g., tremors) of a living organism, which is the measurement object10. Tremor is an involuntary movement caused by alternating contraction and relaxation of muscles. Trembling of the hand is a typical example of tremors. Tremors may be caused by conditions such as stress, anxiety, fatigue, hyperthyroidism, and alcoholic withdrawal. Rest tremor is one of the main symptoms in Parkinson's disease.

In the related art, the tremor is measured by measurement of the myoelectric potential or with an acceleration sensor. The tremor measurement apparatus1200illustrated inFIGS.19and20including the displacement measurement device100can capture micro-vibrations of the measurement object10on the micrometer level. As a result, the tremor can be measured with high accuracy in a non-contact environment.

As illustrated inFIG.19, the tremor measurement apparatus1200measures the tremor with the angle between the elbow and the forearm being kept at 45 degrees relative to the horizontal support1205. In the tremor measurement apparatus1200, the irradiation unit110emits coherent light to the back of the hand, and reflected light of the coherent light from the back of the hand is incident on the luminance-change coordinate point detection unit130as an interference pattern.

Accordingly, the information processing unit150included in the displacement measurement device100can detect the amount of micro-displacement of the measurement object10. In other words, the information processing unit150can measure the tremor of the measurement object10with high accuracy. The tremor data measured by the displacement measurement device100is subjected to frequency analysis or the like and can be used to help understand the condition of the person (i.e., the operator) or used as medical data.

While some embodiments of the present disclosure have been described in detail, the present disclosure is not limited to these embodiments and may be modified or changed in various ways without departing from the spirit of the present disclosure as defined in the appended claims.

The functions of the “displacement measurement device” may be physically implemented by one device or physically implemented by a plurality of devices. A plurality of devices that implements the “displacement measurement device” may be referred to as a “displacement measurement system”.

In some embodiments of the present disclosure, the “displacement measurement device” may be applied to apparatuses other than a “biological micromotion measurement apparatus” and a “non-contact input apparatus”. In embodiments of the present disclosure, the “displacement measurement device” may be applied to a game console, an input/output apparatus, and so on. In embodiments of the present disclosure, the “displacement measurement device” may be applied not only to an apparatus that uses the detection of micro-displacement but also to an apparatus for removing small movement errors.

Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.

The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), conventional circuitry and/or combinations thereof which are configured or programmed to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein or otherwise known which is programmed or configured to carry out the recited functionality. When the hardware is a processor which may be considered a type of circuitry, the circuitry, means, or units are a combination of hardware and software, the software being used to configure the hardware and/or processor.

In one example, a displacement measurement system includes: an irradiation unit configured to irradiate a measurement object with coherent light; a luminance-change coordinate point detection unit configured to detect a luminance-change coordinate point where a luminance change has occurred, based on light reflected from the measurement object, and output data related to the luminance-change coordinate point; and circuitry configured to calculate an amount of displacement of the measurement object, based on the data related to the luminance-change coordinate point, by performing computations of a first numerical sequence and a second numerical sequence. The first numerical sequence including a set of first elements each representing a location of the luminance-change coordinate point extracted from the data. The second numerical sequence including a set of second elements each representing a location of the luminance-change coordinate point extracted from the data.