Patent Publication Number: US-2022222845-A1

Title: Position detection system, image processing device, position detection method, and position detection program

Description:
TECHNICAL FIELD 
     The present invention relates to a position detection system, an image processing device, a position detection method, and a position detection program. 
     BACKGROUND ART 
     An event-driven vision sensor in which a pixel having detected a change in the intensity of incident light generates a signal in a time-asynchronous manner has been known. The event-driven vision sensor is advantageous in that the event-driven vision sensor can operate at a low power and a high speed as compared with a frame-type vision sensor that scans all pixels at each predetermined cycle, specifically, an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). A technique relating to such an event-driven vision sensor is described in, for example, PTL 1 and PTL 2. 
     CITATION LIST 
     Patent Literature 
     PTL 1 Japanese Translation of PCT for Patent No. 2014-535098 PTL 2 Japanese Patent Laid-open No. 2018-85725 
     SUMMARY 
     Technical Problem 
     However, regarding the event-driven vision sensor, the advantages as described above have been known, but it is difficult to say that utilization methods in combination with other devices have yet been sufficiently proposed. 
     Accordingly, an object of the present invention is to provide a position detection system, an image processing device, a position detection method, and a position detection program that can obtain advantageous effects by using an event-driven vision sensor in combination with a microlens array. 
     Solution to Problem 
     According to an aspect of the present invention, provided is a position detection system including an event-driven vision sensor that includes a sensor array including sensors each generating an event signal when a change in intensity of incident light is detected, a microlens array that includes at least first and second microlenses for guiding the light to first and second regions of the sensor array, respectively, and a terminal device that has a reception unit for receiving the event signal from the vision sensor, and a position detection unit for detecting a three-dimensional position of the change in the intensity of light, from the event signal generated by each sensor in the first region and the second region. 
     According to another aspect of the present invention, provided is an image processing device including a reception unit that receives an event signal from an event-driven vision sensor including a sensor array including sensors each generating the event signal when a change in intensity of incident light is detected via a microlens array, and a position detection unit that detects a three-dimensional position of the change in the intensity of light, from the event signals. The microlens array includes at least first and second microlenses for guiding the light to first and second regions of the sensor array, respectively. The position detection unit detects the three-dimensional position, from the event signal generated by each sensor in the first region and the second region. 
     According to still another aspect of the present invention, provided is a position detection method including a step of receiving an event signal from an event-driven vision sensor including a sensor array including sensors each generating the event signal when a change in intensity of incident light is detected via a microlens array, and a step of detecting a three-dimensional position of the change in the intensity of light, from the event signals. The microlens array includes at least first and second microlenses for guiding the light to first and second regions of the sensor array, respectively. In the step of detecting the three-dimensional position, the three-dimensional position is detected from the event signal generated by each sensor in the first region and the second region. 
     According to still another aspect of the present invention, provided is a position detection program that causes a computer to realize a function of receiving an event signal from an event-driven vision sensor including a sensor array including sensors each generating the event signal when a change in intensity of incident light is detected via a microlens array, and a function of detecting a three-dimensional position of the change in the intensity of light, from the event signals. The microlens array includes at least first and second microlenses for guiding the light to first and second regions of the sensor array, respectively. The function of detecting the three-dimensional position detects the three-dimensional position, from the event signal generated by each sensor in the first region and the second region. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  is a block diagram for depicting a schematic configuration of a system according to an embodiment of the present invention. 
         FIG. 2  is a diagram for depicting a photographic optical system of the system depicted in  FIG. 1 . 
         FIG. 3  is a flowchart for depicting an example of a position detection method according to the embodiment of the present invention. 
         FIG. 4  is a flowchart for depicting another example of the position detection method according to the embodiment of the present invention. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     Hereinafter, several embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that, in the specification and the drawings, constitutional elements having substantially the same functional configurations will be denoted by the same reference signs, and duplicated description thereof will be omitted. 
       FIG. 1  is a block diagram for depicting a schematic configuration of a position detection system according to an embodiment of the present invention.  FIG. 2  is a diagram for depicting a photographic optical system of the system depicted in  FIG. 1 . As depicted in  FIG. 1 , a position detection system  10  includes a photographic lens  101 , a microlens array  102 , an event driven sensor (EDS)  103  that is an event-driven vision sensor, and an image processing device  200 . The photographic lens  101  is a main lens for imaging an object to be targeted for position detection and includes a general imaging lens. As depicted in  FIG. 2 , the microlens array  102  is arranged on an imaging surface of the photographic lens  101  and is obtained by two-dimensionally arraying a plurality of microlenses. Each microlens has, for example, a circular planar shape, and includes, for example, a solid lens, a liquid crystal lens, a diffraction lens, or the like. It should be noted that microlenses L 1 , L 2 , . . . , Ln are one-dimensionally arrayed in the microlens array  12  depicted in  FIG. 1  and  FIG. 2 , but actually the microlenses are two-dimensionally arrayed. 
     As depicted in  FIG. 2 , among light incident on the photographic lens  101 , the light having passed through a first microlens L 1  included in the microlens array  102  is guided to a first region A 1  on a sensor array of the EDS  103 , and the light having passed through a second microlens L 2  is guided to a second region A 2  on the sensor array of the EDS  103 . As similar to the above, the light having passed through the n-th microlens Ln is guided to an n-th region An on the sensor array of the EDS  103 . In the microlens array  102  and the EDS  103 , n pieces of regions A 1 , A 2 , . . . , An are arranged so as not to overlap with one another. In the EDS  103 , m pieces of sensors are arranged in the respective regions A 1 , A 2 , . . . , An on the sensor array. It should be noted that the number m of sensors corresponding to the respective regions may be constant or may be partially different. 
     The EDS  103  is an example of a vision sensor that generates an event signal when the sensor detects a change in the intensity of light. The EDS  103  includes sensors, not depicted, configuring the sensor array and a processing circuit, not depicted, connected to the sensors. Each sensor includes a light receiving element and generates an event signal when detecting a change in the intensity of incident light, more specifically, a change in luminance (for example, occurring at an edge portion of an object). Since the sensor that has not detected the change in the intensity of incident light does not generate the event signal, the event signal is generated in the EDS  103  in a time-asynchronous manner. The event signal output via the processing circuit includes sensor identification information (for example, the position of a pixel), the polarity of a change in luminance (rising or falling), a time stamp, and the like. 
     Here, the EDS  103  generates the event signal for each sensor. Therefore, if the sensor identification information (for example, the position of a pixel) is associated with each of the regions A 1 , A 2 , . . . , An on the sensor array in advance, it is possible to specify which of the microlenses L 1 , L 2 , . . . , Ln in the microlens array  102  the generated event signal corresponds to. In addition, if the sensor identification information (for example, the position of a pixel) is associated with a position in each of the regions A 1 , A 2 , . . . , An of the sensors in advance, it is possible to specify from which viewpoint the generated event signal has been observed. 
     The image processing device  200  includes functions of a reception unit  201 , a position detection unit  202 , and a movement estimation unit  203  that are implemented by a computer having, for example, a communication interface, a processor, and a memory and realized by the processor operating according to a program stored in the memory or received via the communication interface. Hereinafter, the function of each unit will be further described. 
     The reception unit  201  receives the event signals from the EDS  103  in a wired or wireless manner. As described above, the event signals are generated by the sensors of the EDS  103 . The sensors of the EDS  103  include sensors arranged in regions where each light is guided by at least two microlenses included in the microlens array  102 , and the reception unit  201  receives each event signal generated by the sensor in each region. 
     The position detection unit  202  detects the three-dimensional position of an event, that is, a change in the intensity of light (hereinafter, also referred to as a three-dimensional position where the event occurred or an event position), from the event signals received by the reception unit  201 . More specifically, the position detection unit  202  detects the three-dimensional position where the event occurred by reconstructing the event signals from the sensors in the regions A 1 , A 2 , . . . , An of the EDS  103  on the basis of the known positional relation between the photographic lens  101  and the microlens array  102 . In other words, the position detection unit  202  treats the event signals from the sensors in the regions A 1 , A 2 , . . . , An of the EDS  103  as information indicating a two-dimensional event position observed from m pieces of viewpoints, and detects the three-dimensional position where the event occurred by utilizing parallaxes among these viewpoints. 
     Here, since the three-dimensional position can be detected from the event signals observed from two or more viewpoints mutually having parallaxes, the position detection unit  202  can detect the three-dimensional position where the event occurred if there are event signals generated by the respective sensors arranged in the regions (first and second regions) where each light is guided by at least two microlenses (first and second microlenses). Further, detection accuracy can be further improved by utilizing the event signals observed from three or more viewpoints mutually having parallaxes. Therefore, the position detection unit  202  may detect the three-dimensional position where the event occurred, on the basis of the event signals generated by the respective sensors arranged in the regions where each light is guided by three or more microlenses. 
     The movement estimation unit  203  continuously acquires the detection result of the event position by the position detection unit  202  and estimates the movement of the event position on the basis of time-series detection results. For example, the movement estimation unit  203  may express the movement of the event position in a predetermined period by using a vector, on the basis of time-series changes in the three-dimensional position where the event occurred. Further, the movement estimation unit  203  may estimate the movement of the event position in the future from time-series changes in the past event position. For example, the movement of the event position in the future thus estimated can be utilized for, for example, the future position prediction of an object being present at the event position. 
       FIG. 3  is a flowchart for depicting an example of a position detection method according to the embodiment of the present invention. In the illustrated example, the reception unit  201  of the image processing device  200  receives event signals generated by the sensors of the EDS  103  (Step S 101 ). The event signals received at this time include an event signal necessary for specifying a two-dimensional event position observed from at least two viewpoints. Next, the position detection unit  202  detects the three-dimensional position where the event occurred, on the basis of the event signals received in Step S 101  (Step S 102 ). As described above, the three-dimensional position of the event can be detected by obtaining information indicating the two-dimensional event position observed from at least two viewpoints. 
       FIG. 4  is a flowchart for depicting another example of the position detection method according to the embodiment of the present invention. In the illustrated example, as similar to the example of  FIG. 3 , the reception unit  201  of the image processing device  200  receives event signals (Step S 201 ), and the position detection unit  202  detects the three-dimensional position where the event occurred, on the basis of the received event signals (Step S 202 ). Further, the movement estimation unit  203  determines whether or not the number N of times of detection of the three-dimensional position is two or more (Step S 203 ), and estimates the movement of the event position in a case where the number N is two or more (Step S 204 ). As described above, the movement of the event position is estimated from changes in the three-dimensional position of the event detected twice or more. 
     In the embodiment of the present invention as described above, since the EDS  103  detects a change in the intensity of incident light via the microlens array  102 , the event position can be accurately detected at a high frame rate on the basis of the information of the occurrence position of the event observed from a plurality of viewpoints. Further, high prediction accuracy can be realized in, for example, the future position prediction of an object being present at the event position by estimating the event position on the basis of the time-series detection results of the event position. It is conceivable that the estimation result by the movement estimation unit  203  is used for, for example, avoidance motion control for an approaching object, control of a moving speed and a traveling direction according to an obstacle, and the like other than the above. 
     In addition, in the embodiment of the present invention, the movement of the event position can be estimated at a high frame rate and with high prediction accuracy by a combination of the microlens array  102  and the EDS  103 . Since the event position can be specified by using the EDS  103  without image processing and since the latency of processing for an approaching object or the like is low and it is only necessary to arrange a single EDS  103  corresponding to the microlens array  102 , the same processing as a case in which, for example, a plurality of cameras for obtaining a plurality of viewpoint images is arranged can be realized with space-saving. Further, since it can be determined on the basis of the event signals generated by the EDS  103  that a change in the intensity of light occurred, no additional configuration or control such as infrared (IR) irradiation or an auxiliary light source is required. 
     It should be noted that the position detection system  10  described in the above example may be implemented in a single device, or may be dispersedly implemented in a plurality of devices. For example, the entire position detection system  10  may be implemented in a robot, a drone, or a terminal device including a wearable terminal or the like, or the image processing device  200  may be dispersedly implemented in server devices. 
     In addition, in the position detection system  10  described in the above example, the example in which the microlens array  102  is arranged on the imaging surface of the photographic lens  101  has been depicted, but other arrangements may be employed. For example, the microlens array  102  may be arranged away from the imaging surface of the photographic lens  101 , and a change in the intensity of light may be detected by the EDS  103  via the microlens array  102  on the basis of a virtual image on the imaging surface of the photographic lens  101 . 
     In addition, regarding the microlens array  102  of the position detection system  10  described in the above example, if each microlens is associated with a region on the sensor array of the EDS  103 , each microlens configuring the microlens array  102  may have a uniform focal distance, or microlenses having different focal distances may be regularly or irregularly arranged to configure the microlens array  102 . 
     Although several embodiments of the present invention have been described above in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person ordinarily skilled in the art to which the present invention belongs can perceive various change examples and modification examples within the scope of the technical idea described in the claims, and it can be understood that these obviously belong to the technical range of the present invention. 
     REFERENCE SIGNS LIST 
     
         
           10 : Position detection system 
           101 : Photographic lens 
           102 : Microlens array 
           103 : EDS 
           200 : Image processing device 
           201 : Reception unit 
           202 : Position detection unit 
           203 : Movement estimation unit