Abstract:
A system for three-dimensional image capture while moving. The system includes: an image capture device including at least two digital image sensors placed to capture a stereoscopic image; a processor for obtaining disparity information, associated with the digital images obtained by the image capture device and movement speed of elements in the digital images; a transmitter for sending the digital images to enable control of movement based on the digital images and the associated disparity information; and a controller for controlling the image capture device.

Description:
TECHNICAL BACKGROUND 
       [0001]    The invention relates to a stereoscopic image capture system. Such a system commonly comprises a device described as a 3D camera, or stereoscopic camera. 
         [0002]    The invention particularly considers a stereoscopic image capture system that provides depth information, or equivalent information. The cameras of these systems are described as depth sensors, or depth cameras. These sensors make it possible to obtain images captured from two points of view, which, after processing, make it possible to obtain depth information, or any equivalent information, generally described as disparity information. It is obtained point by point (pixel by pixel) in the image, and referred to as a depth map. 
         [0003]    The invention in particular relates to a stereoscopic image capture system for capturing images while moving, at low or high speed, within an environment made up of stationary or static objects that are opposite the device for capturing images of the relative movements. 
         [0004]    Depth cameras are known based on the measurement of a time of flight of a light wave emitted by illumination means such as LEDs, and reflected the encountered objects. The wave may be an infrared wave. The time of flight, measured pixel by pixel, makes it possible to determine the distance of the reflective surface from the transmitting device. Time of flight cameras have the flaw of being sensitive to disruptions, such as infrared waves emitted by the environment, in particular the sun, or interference, in particular coming from other time of flight cameras, for example encountered during the movement of the vehicle with the camera on board and that passes other vehicles with similar cameras on board. 
         [0005]    LIDAR (light detection and ranging) systems, or laser radars, are also known reading (scanning) the environment with a generally coherent light, emitted by scanning using a heavy mechanical system that must be very precise. The system is cumbersome, and requires regular maintenance, due to the mechanical scanner system. It is more expensive for these two reasons. It is also blinded in case of rain and fog by the reflection caused by the cloudiness. 
         [0006]    Also known is document US 20070296846, which discloses a stereoscopic camera comprising two digital image sensors placed at a distance from one another by a chassis. The positioning of the sensors is obtained by a precise mechanical engagement between the parts. Also known is a stereoscopic camera called Bumblebee comprising two or three digital image sensors, communicating using a FireWire interface (IEEE-1394) and a GPIO (General-Purpose Input/Output) port, having a fixed image capture frequency and a fixed resolution. A first computer program is run on a microcomputer connected to the camera by the two connections, and makes it possible to control the camera, while a second computer program run on the microcomputer makes it possible to perform stereoscopic correlations to generate an image of the disparities. But since the resolution and the image capture frequency are fixed, it is not possible to use this camera dynamically and adaptively in an environment in which some objects are observed in rapid relative movement, and others in slow relative movement. 
         [0007]    Also known is a stereoscopic camera called Duo3D, in which the image capture frequency is configurable by a programming interface command, related in predefined modes to resolutions in both dimensions of the image that are modified by binning. However, this camera is configured for the computerized perception of objects at a short distance, and is not suitable for observing an environment during movement. 
         [0008]    There is therefore a need for a stereoscopic image capture system providing adaptive image capture in the face of relative movements of objects around the image capture device. 
       SUMMARY OF THE INVENTION 
       [0009]    To resolve this problem, a system is proposed for capturing three-dimensional images during a movement, comprising: 
         [0010]    an image capture device including at least two digital image sensors synchronized with one another and arranged to perform a stereoscopic image capture, 
         [0011]    processing means to obtain disparity information associated with the digital images obtained by the image capture device as well as a movement speed of elements in said images, 
         [0012]    a means for transmitting digital images to make it possible to control the movement based on said digital images and the associated disparity information, and 
         [0013]    a control means to command the image capture device by optimizing, to facilitate said movement control, the choice of the dimensions of the field of view of the sensors and the frequency of the image capture, taking into account a maximum throughput tolerated by the transmission means and said movement speed. 
         [0014]    Owing to this system, it is possible to provide a device for controlling the movement of images captured with a frequency and a field of view suitable for the movement, in light of the relative speeds of the objects around the image capture device, with respect to the image capture device, which is then onboard the moving object needing to be controlled. 
         [0015]    Advantageously, the image capture system according to the invention may comprise at least one of the following technical features: 
         [0016]    said speed is assessed by a contrast gradient calculation done on the digital images, 
         [0017]    the means for transmitting digital images comprises a USB 3.0 connection, a Giga Ethernet connection or a Thunderbolt connection, 
         [0018]    the processing means comprises a computer program to be run on a microcomputer, 
         [0019]    the processing means comprises a chipset on a printed circuit substrate, 
         [0020]    the dimensions of the field of view used by the sensors and the image capture frequency can also be set by a user, 
         [0021]    the two sensors are master and slave, respectively, for synchronization purposes, 
         [0022]    the image capture device comprises a digital image processing controller placed on a printed circuit substrate shared by the two sensors, 
         [0023]    the means for transmitting digital images comprises a means for transmitting said images between the image capture device and at least one of the processing means, 
         [0024]    the means for transmitting digital images comprises a means for transmitting said images and the disparity information to a means for controlling the movement, 
         [0025]    if elements are observed that are moving quickly, the capture frequency is increased, while the field of view is restricted, and if elements are observed moving slowly, said frequency is decreased, while the field of view is extended. 
         [0026]    The invention also relates to an object provided with movement means and comprising an image capture system according to the invention, said object comprising means for sending a human operator or an electronic module the disparity information associated with the digital images produced by said system for three-dimensional image capture, in order to control the movement of said object. 
         [0027]    The object may additionally comprise means for moving in the context of autonomous navigation by using the disparity information associated with the digital images obtained by said system for three-dimensional image capture. 
     
    
     
       BRIEF DESCRIPTION OF DRAWING FIGURES 
         [0028]    The invention will be better understood, and other aims, features, details and advantages thereof will appear more clearly, in the following explanatory description done in reference to the appended drawings, provided solely as an example illustrating one embodiment of the invention, and in which: 
           [0029]      FIG. 1  is a general view in space of one embodiment of an image capture device according to the invention, 
           [0030]      FIG. 2  is a diagram showing the operation of the elements of one embodiment of an image capture device according to the invention, 
           [0031]      FIG. 3  is one aspect of the operation of an image capture device according to the invention, 
           [0032]      FIG. 4  shows an outside view of one embodiment of an image capture device according to the invention, 
           [0033]      FIGS. 5 and 6  show example embodiments of the image capture device according to the invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0034]      FIG. 1  provides a schematic view of an electronic board of an image capture device  10  according to one embodiment of the invention. The board  10  is formed by a single printed circuit substrate supporting electronic components. The image capture device board  10  comprises a first digital image sensor  100  and a second digital image sensor  110 , which can be CCD (charge-coupled device) sensors or sensors using another technology, such as the CMOS technology, and receive light through an optical lens focusing system (not shown). 
         [0035]    The selected sensors  100  and  110  have an active surface that can be controlled by a command outside the sensor, and which can be used for different viewing angle dimensions, or different resolutions. They may also be controlled to capture images at different frequencies. 
         [0036]    The sensors  100  and  110  are both positioned on a printed circuit substrate  120 . The substrate  120  can be an elongated rectangle, the large dimension of which is placed along a direction X, and the two sensors are then placed substantially at both ends of the rectangle in the direction X. The two sensors  100  and  110  are positioned with their sensitive surfaces oriented opposite the substrate  120 , turned substantially in a direction Z perpendicular to the direction X, and normal to the plane of the printed circuit substrate  120 . 
         [0037]    On the printed circuit substrate  120  is a controller  130 , in the form of an integrated circuit electrically connected to the two sensors  100  and  110 . The controller  130  receives commands from third-party components and provides operating parameters to the sensors  100  and  110 . It furthermore verifies that these commands have indeed been received and understood by the sensors  100  and  110 . 
         [0038]    Additionally, the printed circuit substrate  120  bears a connector  140 , making it possible to connect the image capture device  10  to a third-party device. 
         [0039]      FIG. 2  functionally shows the image capture device board  10 , and its interaction with an outside processing and control unit  20 . The first sensor  100  and the second sensor  110  are again shown, as well as the controller  130  and the connector  140 . The controller  130  runs several programs constituting several modules  131 ,  132 ,  133  at  134  that will be outlined later. The controller  130 , the sensors  100  and  110  and the connector  140  communicate via tracks of conductive material (copper) of the board  10 . 
         [0040]    Each of the first and second sensors  100  and  110  captures digital images at an identical frequency for both sensors, and set by the controller  130 , which executes a program forming an image sensor processing module  133 . This module  133  sends control commands of the first sensor C 1  and control commands of the second center C 2  to both sensors, in particular to define the capture frequency of the images. 
         [0041]    The first sensor  100  regularly sends a synchronization message M 1  to the second sensor  110 , to allow the latter to align the capture moments of its images with the capture moments of the images by the first sensor  100 . This involves a master-slave mode synchronization, the first sensor  100  acting as master, imposing the capture moments, and the second sensor  110  acting as slave, applying the instruction received to implement the image capture. 
         [0042]    The sensors  100  and  110  send the images they capture to the controller  130 , via messages I 1  and I 2  comprising the color and brightness information of the pixels. The captured and transmitted images are rectangular, the rectangles of the two sensors being identical (same width, same height, same orientation). 
         [0043]    A first program constitutes a receiving and resynchronization module  131 , responsible for receiving the digital images captured by the sensors  100  and  110  and re-synchronizing them such that the following programs implemented by the controller  130  are able to recognize the pairs of images sent by the first and second sensors  100  and  110 . 
         [0044]    A second program constitutes a merging module  132  responsible for merging the two images taken at the same time by the two sensors, and transmitted by the module  131 . The two images are merged into a single digital image, as will be described in connection with  FIG. 3 . The single merged image is sent to the following module. 
         [0045]    The following module, already described, constitutes an image sensor processing (ISP) module  133 . In some embodiments, it may perform parallel calculations to specify the color and light of each pixel. Once its processing is complete, it sends the image to the following module. 
         [0046]    The next program defines a fourth module, which is a conversion module  134 , responsible for converting the data making up the digital images, and potentially other information, into a format compatible with the transmission via a wired connector  150  connected to the connector  140 , and conversely, extracting the information received by the wired connector  150  and the connector  140  to send it to the image sensor processing module  133 . 
         [0047]    The wired module  150 , which for example is flexible, connects the connector  140  to a connector  201  of the outside processing and control unit  20 . The wired connector  150  can perform an electrical or optical transmission. 
         [0048]    The outside processing and control unit  20  comprises a connector  201 , and a processor  200  that runs a processing and control program  202 . It also comprises a conversion module  204  for the reception and sending of data by the connector  201 , which may be run by the processor  200  or a dedicated controller. 
         [0049]    In one embodiment, the transmission by the wired connector  150  is done according to standard USB 3.0. The conversion module  134  and the wired connector  150 , as well as the connectors  140  and  201 , are configured to implement standard USB 3.0. Other standards can be used, in particular the Gigethernet and Thunderbolt standards. 
         [0050]    The processor  200  implements the processing and control program  202 , which uses the digital images transmitted by the image sensor processing module  133 . In particular, the disparity or depth information is calculated for each pair of synchronized images. 
         [0051]    Furthermore, a contrast gradient calculation makes it possible to assess the presence of motion blurring in the images. 
         [0052]    The processing and control program  202  is also able to send commands to the controller  130  of the image capture device  10 , for example via the wired connector  150 , or by another wired or wireless means. 
         [0053]    Depending on the result of the contrast gradient calculation, the processing and control program  202  is able to command automatically, or with agreement from a user to whom the command is suggested by the program, a change in the choice of the dimensions of the field of view of the sensors  100  and  110  and the image capture frequency by the sensors. This choice is framed by the relationship 
         [0000]      width×height×16×frequency max throughput
 
         [0054]    wherein the width and height are expressed in pixels and the maximum throughput in bits/s. The maximum throughput is that of the connection between the image capture device  10  and the outside processing and control unit  20 , which is for example the maximum throughput of a USB 3.0 connection. 
         [0055]    If motion blurring is detected, it is then chosen to decrease the field of view and increase the image capture frequency, to improve the vision of the objects in relative movement around the image capture device  10 . 
         [0056]    If little or no motion blurring is observed, it is chosen to increase the field of view and decrease the image capture frequency, to improve the perception of the environment, including on the sides. 
         [0057]    The outside processing and control unit  20  includes a second connector  203 , as well as a second conversion module  205  for the reception and transmission of data by the connector  203 , which can be done by the processor  200  or a dedicated controller. 
         [0058]    The outside processing and control unit  20  can assume the form of a multifunctional computer or a unit dedicated to controlling the image capture device  10 . In the second case, it may for example be built in the form of a single printed circuit board bearing the connectors  201  and  203 , as well as a processor  200 , which can then run both the program  202  and the modules  204  and  205 . 
         [0059]    The module  205  allows the outside processing and control unit  20  to communicate with a third-party device, via a second wired connection means  250 . The second wired connector  250  can perform an electrical or optical transmission. 
         [0060]    This is for example a USB 3.0 cable, or a Giga Ethernet or Thunderbolt connection. 
         [0061]    The processing and control unit  20  sends the stereoscopic digital images and the disparity or depth information associated therewith via the wired connector  250 . Thus, a three-dimensional depiction of the image captured by the image capture device  10  is provided, either to a human user, or to a software program capable of exploiting it. 
         [0062]      FIG. 3  shows the merging process implemented by the merging module  132 . 
         [0063]    Two images  1000  and  1001  have been captured by the sensors  100  and  110  ( FIG. 1 ). Their content is referenced G and D in the figure, to indicate that these are images captured by the left and right sensors, respectively, of the image capture device  10 . The two images are each a rectangular set of pixels with the same heights and the same lengths. They are subject to merging by the merging module  132 , which creates a single image  2000  made up of a rectangular set of pixels, of the same height as the two images  1000  and  1001  and twice the length relative to the length of the images  1000  and  1001 . The left side of the right image D is placed next to the right side of the left image G. 
         [0064]      FIG. 4  shows a three-dimensional view of one embodiment of the image capture device. It may for example assume the form of a box  10 ′ generally in the shape of a rectangular rhomb or the like, in which the electronic components of the device are contained, and which has two openings to the light on one face, through which the first and second sensors  100  and  110  expose their respective sensitive surfaces to capture digital images. The wired connector  250  allowing the extraction of the digital data is not visible. 
         [0065]      FIG. 5  shows a use of an image acquisition device box  10 ′ in a flying object  5000  of the drone type capable of moving without human intervention, or remotely controlled by a human. The flying object  5000  carries an image acquisition device box  10 ′, as well as an outside processing and control unit  20 , connected by a wired connection means  150 . It also comprises an autonomous navigation module  5100 , interacting with the outside processing and control unit  20 , via a wired connection means  250 . The autonomous navigation unit  5100  uses the stereoscopic images provided by the image capture device and the associated disparity or depth information, pixel by pixel, to make navigation decisions. The navigation decisions are of better quality if the dimensions of the field of view of the sensors and the image capture frequency have been chosen carefully and are readjusted regularly, reactively based on the environment and objects observed in the images and their relative speed with respect to the flying object  5000 . 
         [0066]    Likewise, in  FIG. 6 , the same principle is presented, this time with a rolling object  6000  of the autonomous car type capable of moving without human intervention in a complex environment, for example a road or urban environment. The vehicle can also be a construction vehicle or a wagon moving in a warehouse, for example. 
         [0067]    For the objects  5000  and  6000 , the navigation can be autonomous or computer-assisted navigation, the decision-making in this case always being done by a human, but the latter having additional information obtained owing to the image capture device and the associated system. 
         [0068]    In one alternative, the box  10 ′ of the image capture device can incorporate the control unit  20  inside it, and be on board a vehicle such that the user simply connects the box  10 ′, which is then the only one, to the autonomous navigation module  5100  or  6100  using the wired connection means  250 . The wired connection means  150  is then inside the box  10 ′. 
         [0069]    Within the single box  10 ′, in one alternative, the processor  200  can also be mounted on the board  10 , and communicate via a connection using conductive metal tracks (copper), with the module  133  of the controller  130 , in place of the communication via the wired connection means  150 , the connectors  140  and  201  and the conversion module  134 . 
         [0070]    The maximum throughput value used to command a change in the choice of dimensions of the field of view of the sensors  100  and  110  and the image capture frequency by the sensors is in this case that of the wired connection means  250 . 
         [0071]    In another alternative, the control unit  20  and the autonomous navigation unit  5100  or  6100  are embodied by a same piece of computer equipment, having a processor that runs both the autonomous navigation program and the processing and control program  202 . The connection  250 , the connector  203  and the conversion module  205  are then absent. 
         [0072]    In general, the maximum throughput value used to command a change in the choice of the dimensions of the field of view of the sensors  100  and  110  and the image capture frequency by the sensors is that of the most limiting connection for the transmission of the digital images to obtain disparity information and to send digital images to the navigation module. 
         [0073]    The invention is not limited to the described embodiment, but extends to all alternatives within the scope of the claims.