Patent Publication Number: US-9896205-B1

Title: Unmanned aerial vehicle with parallax disparity detection offset from horizontal

Description:
FIELD OF THE INVENTION 
     The invention relates to unmanned aerial vehicles with parallax disparity detection offset from horizontal. 
     BACKGROUND OF THE INVENTION 
     It is known that unmanned aerial vehicles, or UAVs, may be equipped with optical elements that guide light to image sensors, and that images of an object captured by the image sensors may be used to determine parallax disparity of the object. In such UAVs, the optical elements are arranged so that they are level/horizontal, like human eyes, when the UAVs operate leveled with respect to ground, e.g., not producing horizontal thrust. 
     SUMMARY 
     One aspect of the invention relates to unmanned aerial vehicles with parallax disparity detection offset from horizontal. The unmanned aerial vehicles with parallax disparity detection offset from horizontal is provided by arranging optical elements so that they are separated by both a horizontal distance and a vertical distance when the UAVs operate leveled with respect to ground, e.g. not producing horizontal thrust. 
     A UAV with parallax disparity detection offset from horizontal may include one or more of a housing, a motor, a first optical element, a first image sensor, a second optical element, a second sensor, a processor, and/or other components. The motor may be carried by the housing and may be configured to drive a rotor. The rotor may provide thrust to move the UAV in any direction. The first optical element may be configured to guide light to the first image sensor, and the second optical element may be configured to guide light to the second image sensor. The first optical element and the second optical element may receive light from an object. The first image sensor may be configured to generate a first output signal conveying first visual information regarding the object, and the second image sensor may be configured to generate a second output signal conveying second visual information regarding the object. The visual information may include, by way of non-limiting example, one or more of an image, a video, and/or other visual information. The first optical element, the second optical element, the first image sensor and the second image sensor may be attached to the housing. 
     The first image sensor may include, by way of non-limiting example, one or more of charge-coupled device sensor, active pixel sensor, complementary metal-oxide semiconductor sensor, N-type metal-oxide-semiconductor sensor, and/or other image sensor. The second image sensor may include, by way of non-limiting example, one or more of charge-coupled device sensor, active pixel sensor, complementary metal-oxide semiconductor sensor, N-type metal-oxide-semiconductor sensor, and/or other image sensor. 
     The first optical element may include, by way of non-limiting example, one or more of standard lens, macro lens, zoom lens, special-purpose lens, telephoto lens, prime lens, achromatic lens, apochromatic lens, process lens, wide-angle lens, ultra-wide-angle lens, fisheye lens, infrared lens, ultraviolet lens, perspective control lens, other lens, and/or other optical element. The second optical element may include, by way of non-limiting example, one or more of standard lens, macro lens, zoom lens, special-purpose lens, telephoto lens, prime lens, achromatic lens, apochromatic lens, process lens, wide-angle lens, ultra-wide-angle lens, fisheye lens, infrared lens, ultraviolet lens, perspective control lens, other lens, and/or other optical element. 
     The first optical element and the second optical element may be arranged to be separated by both a horizontal distance and a vertical distance when the UAV operates leveled with respect to ground, e.g. no horizontal thrust is being generated by the rotor. In some implementations, the UAV may be climbing vertically without horizontal movement when the UAV operates leveled with respect to ground. In some implementations, the UAV may be descending vertically without horizontal movement when the UAV operates leveled with respect to ground. In some implementations, the UAV may not be tilting sideways when the UAV operates leveled with respect to ground. 
     The processor may be attached to the housing and may be configured to provide flight control for the UAV. Flight control may include stabilization control, navigation control, altitude control, propulsion control, engine control, and/or other functions needed and/or used during operation of a UAV. The processor may be configured to receive the first output signal and the second output signal, and may be configured to compare the first visual information with the second visual information to determine parallax disparity of the object. In some implementations, the processor may be configured to determine distance between the object and the UAV based on the parallax disparity. 
     These and other objects, features, and characteristics of the present disclosure, as well as the methods of operation and functions of the related components of structure and the combination of parts and economies of manufacture, will become more apparent upon consideration of the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for the purpose of illustration and description only and are not intended as a definition of the any limits. As used in the specification and in the claims, the singular form of “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1A  schematically illustrates an unmanned aerial vehicle in accordance with one or more implementations. 
         FIG. 1B  schematically illustrates an unmanned aerial vehicle in accordance with one or more implementations. 
         FIG. 2  illustrates two viewpoints of a block. 
         FIG. 3A  illustrates parallax of a box due to perspective shift to the left. 
         FIG. 3B  illustrates parallax of a box due to perspective shift to the right. 
     
    
    
     DETAILED DESCRIPTION 
     An unmanned aerial vehicle may be referred to as UAV. The term “unmanned” refers to the capability of the aerial vehicle to operate without requiring a human operator during a flight. In other words, at least some portion of the flight control may be provided remotely and/or by an autopilot. In some implementations, a UAV may carry passengers, cargo, sensors, and/or other physical objects. In some implementations, a UAV may operate autonomously. Alternatively, and/or simultaneously, in some implementations, at least some functionality of a UAV may be controlled and/or modified through remote control, e.g. by a person, for at least some portion of a flight. For example, a human may control and/or assist remotely in a particular maneuver, such as a take-off or landing. 
     A UAV may be a fixed wing aircraft, a helicopter, a multi-rotor aircraft (e.g. a quadcopter), a rotary wing aircraft, and/or another type of aircraft. In some implementations, a UAV may combine features of multiple types of aircraft. A UAV may include one or more components configured to provide thrust. By way of non-limiting example, the one or more components providing thrust may include one or more wings, airfoils, motors, propellers, rotors, rotor discs, and/or other components. 
       FIGS. 1A and 1B  schematically illustrate an unmanned aerial vehicle  10  (also referred to as UAV  10 ), in particular a quadcopter. The quadcopter is an exemplary and non-limiting implementation of UAV  10 . As illustrated in  FIGS. 1A and 1B , UAV  10  may include four motors  12  and four rotors  13 . The number of motors and rotors of UAV  10  is not intended to be limited by any depiction. In some implementations, UAV  10  may include one, two, three, four, five, six, and/or more than six motors and/or rotors. UAV  10  may include one or more of a housing  11 , a motor  12 , a rotor  13 , a first optical element  14 , a second optical element  15 , a first image sensor  16 , a second image sensor  17 , a processor  18 , an electronic storage  19 , and/or other components. 
     Housing  11  may be configured to attach to, support, hold, and/or carry components of UAV  10 . The combination of housing  11  and components attached to, supported, held, and/or carried by housing  11  may be referred to as an unmanned aerial vehicle. 
     Rotor  13  may be driven by motor  12 . In some implementations, rotor  13  may include a rotor blade, a hub, and a mast. The rotor blade may be connected to the hub, the hub may be connected to the mast, and the mast may be connected to motor  12 . In some implementations, rotor  13  may include a rotor blade and a hub. The rotor blade may be connected to the hub, and the hub may be connected to motor  12 . 
     Rotor  13  may provide thrust to move UAV  10  along any direction. In a three-dimensional Cartesian coordinate system, rotor  13  may provide thrust to move UAV  10  along the positive X-axis, the negative X-axis, the positive Y-axis, the negative Y-axis, the positive Z-axis, the negative Z-axis, and any combination thereof. Rotor  13  may provide thrust to rotate UAV  10  along pitch axis, roll axis, yaw axis, and any combination thereof. Rotor  13  may provide thrust to rotate and move UAV  10  at the same time. 
     First optical element  14  may be configured to guide light to first image sensor  16 . Second optical element  15  may be configured to guide light to second image sensor  17 . First optical element  14  may include, by way of non-limiting example, one or more of standard lens, macro lens, zoom lens, special-purpose lens, telephoto lens, prime lens, achromatic lens, apochromatic lens, process lens, wide-angle lens, ultra-wide-angle lens, fisheye lens, infrared lens, ultraviolet lens, perspective control lens, other lens, and/or other optical element. Second optical element  15  may include, by way of non-limiting example, one or more of standard lens, macro lens, zoom lens, special-purpose lens, telephoto lens, prime lens, achromatic lens, apochromatic lens, process lens, wide-angle lens, ultra-wide-angle lens, fisheye lens, infrared lens, ultraviolet lens, perspective control lens, other lens, and/or other optical element. 
     First optical element  14  may guide light received from an object to first image sensor  16  directly, or indirectly through use of one or more light manipulating components. Second optical element  15  may guide light received from an object to second image sensor  17  directly, or indirectly through use of one or more light manipulating components. By way of non-limiting example, a light manipulating components may include one or more of a mirror, a prism, lenses, and/or other light manipulating components. Although first optical element  14  and second optical element  15  are depicted in  FIG. 1  by individual features, one or both of first optical element  14  and/or second optical element  15  may include multiple actual components or pieces. 
     First image sensor  16  may be configured to generate a first output signal conveying first visual information present in the light guided thereto by first optical element  14 . While the object is within a field of view of first optical element  14 , first visual information includes the object. Second image sensor  17  may be configured to generate a second output signal conveying second visual information present in the light guided thereto by second optical element  15 . While the object is within a field of view of second optical element  15 , second visual information includes the object. First image sensor  16  may include, by way of non-limiting example, one or more of charge-coupled device sensor, active pixel sensor, complementary metal-oxide semiconductor sensor, N-type metal-oxide-semiconductor sensor, and/or other image sensor. Second image sensor  17  may include, by way of non-limiting example, one or more of charge-coupled device sensor, active pixel sensor, complementary metal-oxide semiconductor sensor, N-type metal-oxide-semiconductor sensor, and/or other image sensor. 
     The first visual information may include, by way of non-limiting example, one or more of an image, a video, and/or other visual information. The second visual information may include, by way of non-limiting example, one or more of an image, a video, and/or other visual information. One or more of the first visual information and/or the second visual information may be marked, timestamped, annotated, stored, and/or otherwise processed. 
     In some implementations, one or more of first optical element  14 , second optical element  15 , first image sensor  16  and/or second image sensor  17  may be attached directly to housing  11 . By way of non-limiting example, one or more of first optical element  14 , second optical element  15 , first image sensor  16  and/or second image sensor  17  may be in physical contact with housing  11  and may be directly attached to housing  11 , directly supported by housing  11 , directly held by housing  11 , and/or directly carried by housing  11 . 
     In some implementations, one or more of first optical element  14 , second optical element  15 , first image sensor  16  and/or second image sensor  17  may be attached indirectly to housing  11 . By way of non-limiting example, one or more of first optical element  14 , second optical element  15 , first image sensor  16  and/or second image sensor  17  may not be in physical contact with housing  11  and may be indirectly attached to housing  11 , indirectly supported by housing  11 , indirectly held by housing  11 , and/or indirectly carried by housing  11 . For example, one or more of first optical element  14 , second optical element  15 , first image sensor  16  and/or second image sensor  17  may be located in a container, and the container may be directly attached to housing  11 . 
     First optical element  14  and second optical element  15  may be arranged to be separated by both a horizontal distance and a vertical distance when housing  11  is being suspended by rotation of rotor  13 , and while UAV  10  operates leveled with respect to ground. In some implementations, UAV  10  may be climbing vertically without horizontal movement when UAV  10  operates leveled with respect to ground. In some implementations, UAV  10  may be descending vertically without horizontal movement when UAV  10  operates leveled with respect to ground. In some implementations, UAV  10  may not be tilting sideways when UAV  10  operates leveled with respect to ground. This arrangement of first optical element  14  and second optical element  15  may allow for parallax disparity detection offset from horizontal. 
     Parallax refers to the seeming change in position of an object because of a change in the observer&#39;s viewpoint. Parallax disparity is the change in position of an object between two viewpoints. Parallax disparity is inversely proportional to the distance from the viewpoints to the object. Detecting parallax disparity may be less effective when edges of an object has the same orientation as the optical elements. For example, if the optical elements are horizontally arranged, it may be difficult to determine parallax disparity of horizontal edges of an object. Parallax disparity detection offset from horizontal may allow for detection of more meaningful parallax disparity on horizontal edges. 
     A simple case of parallax is illustrated in  FIGS. 2, 3A, and 3B .  FIG. 2  illustrates two different viewpoints for block  27 : viewpoint A  21 , and viewpoint B  22 . From viewpoint A  21 , an observer looking straight ahead would see along straight-ahead line A  25 . From viewpoint A  21 , the observer sees block  27  along viewline A  23 . From viewpoint B  22 , an observer looking straight ahead would see along straight-ahead line B  26 . From viewpoint B  22 , the observer sees block  27  along viewline B  24 . 
       FIG. 3A  illustrates parallax of box  27  due to perspective shift to the left, as seen from viewpoint A  21 . From viewpoint A  21 , the observer sees block  27  to be located at a distance x 1  to the right relative to straight-ahead line A  25 .  FIG. 3B  illustrates parallax of box  27  due to perspective shift to the right, as seen from viewpoint B  22 . From viewpoint B  22 , the observer sees block  27  to be located at a distance x 2  to the left relative to straight-ahead line B  26 . Parallax disparity of box  27  between viewpoint A  21  and viewpoint B  22  is x 1 +x 2 . Thus, distance from viewpoint A  21  and viewpoint B  22  to block  27 , indicated as d in  FIG. 2 , is inversely proportional to x 1 +x 2 . 
     If the focal length of first optical element  14  and first image sensor  16 , and the focal length of second optical element  15  and second image sensor  17  are identical, then distance D from first optical element  14  and second optical element  15  to an object is determined by the following equation: 
               D   =       b   ⁢           ⁢   f         x   1     +     x   2           ,         
where f=focal length, and
 
     b=distance between first optical element  14  and second optical element  15   
     Processor  18  may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, a central processing unit, a graphics processing unit, a microcontroller, an analog circuit designed to process information, and/or other mechanisms for electronically processing information. In some implementations, processor  18  may include a plurality of processing units. In some implementations, processor  18  may be coupled with one or more of RAM, ROM, input/output ports, and/or other peripherals. By way of non-limiting example, a microcontroller may be one or more of 8051, PIC, AVR, and ARM microcontroller. 
     Processor  18  may be coupled, directly or indirectly, to one or more flight control components. By way of non-limiting example, a flight control component may include one or more of an actuator, a motor, a rotor, an accelerometer, a rate of rotation sensor (e.g., a gyroscope), an inertial measurement unit, a compass, a magnetometer, a pressure sensor, a barometer, a global positioning system device, a distance sensor, an image sensor, an optical element, an electronic storage, and/or other flight control components. 
     Processor  18  may be configured by a computer-readable instruction to provide information-processing capability. Information-processing capability includes, but is not limited to, receiving the first output signal generated by first image sensor  16 , receiving the second output signal generated by second image sensor  17 , comparing the first visual information with the second visual information to determine parallax disparity of the object, and/or determining distance between the object and UAV  10  based on the parallax disparity. Comparing the first visual information with the second visual information to determine parallax disparity of the object may include one or more of distortion removal, image rectification, disparity map generation, and/or height map generation. 
     Different approaches may be utilized to triangulate a pixel corresponding between first visual information and second visual information. For example, different approaches may be utilized based on what is known about first optical element  14 , second optical element  15 , first image sensor  16 , and/or second image sensor  17 . 
     In some implementations, a calibrated projection model, a calibrated distortion model, and relative positions and orientations of first optical element  14 , second optical element  15 , first image sensor  16 , and second image sensor  17  may be known. Using these knowns, rectification and undistortion mapping may be applied to first visual information and second visual information. The first visual information and the second visual information may then be processed by one or more depth estimation algorithm, such as block matching, semi-global block matching, efficient large-scale stereo matching, and/or other depth estimation algorithms, and the depth of pixels may be determined. Alternative to applying rectification and/or undistortion mapping, a pixel location may be triangulated by using the locations of the pixel in the first visual information and the second visual information, and the calibration models of first optical element  14 , second optical element  15 , first image sensor  16 , and second image sensor  17 . 
     In some implementations, a calibrated projection model and a calibrated distortion model of first optical element  14 , second optical element  15 , first image sensor  16 , and second image sensor  17  may be known. Relative positions and orientations of first optical element  14 , second optical element  15 , first image sensor  16 , and second image sensor  17  may be estimated by using a set of point correspondences between the first visual information and the second visual information. In parallel to the above estimation, the positions of pixels relative to the positions of first optical element  14 , second optical element  15 , first image sensor  16 , and second image sensor  17  may be estimated. Such estimations may include simultaneous localization and mapping (SLAM). 
     In some implementations, no calibration regarding first optical element  14 , second optical element  15 , first image sensor  16 , and second image sensor  17  may be known. Positions, orientations, and intrinsic parameters of first optical element  14 , second optical element  15 , first image sensor  16 , and second image sensor  17  may be estimated using software frameworks for three-dimensional photogrammetric reconstruction. 
     In some implementations, one or more of the above approaches may be utilized to triangulate a pixel corresponding between first visual information and second visual information. Other approaches are contemplated. 
     In some implementations, the computer-readable instruction may be stored in memory of processor  18 . In some implementations, the computer-readable instruction may be stored in electronic storage  19 . In some implementations, the computer-readable instruction may be received through remote communication, including, but not limited to, radio communication, Bluetooth communication, Wi-Fi communication, cellular communication, infrared communication, or other remote communication. In some implementations, processor  18  may use computer-readable instruction from one or more of memory of processor  18 , electronic storage  19 , and/or remote communication. 
     In some implementations, processor  18  may include a flight control instruction in its memory to provide flight control. In some implementations, a flight control instruction may be stored in electronic storage  19 . In some implementations, a flight control instruction may be received through remote communication, including, but not limited to, radio communication, Bluetooth communication, Wi-Fi communication, cellular communication, infrared communication, and/or other remote communication. By way of non-limiting example, a flight control instruction include one or more of moving UAV  10  in any direction, rotating UAV  10  in any direction, flying UAV  10  in a stable manner, tracking people or objects, avoiding collisions, and/or other functions needed and/or used during operation of unmanned aerial vehicles. By way of non-limiting example, flight control may include one or more of stabilization control, navigation control, altitude control, attitude control, position control, propulsion control, engine control, and/or other control needed and/or used during operation of unmanned aerial vehicles. 
     In some implementations, a flight control instruction may be stored in electronic storage  19 . In some implementations, a flight control instruction may be received through remote communication, including, but not limited to, radio communication, Bluetooth communication, Wi-Fi communication, cellular communication, infrared communication, or other remote communication. In some implementations, processor  18  may use flight control instruction from one or more of memory of processor  18 , electronic storage  19 , and/or remote communication. 
     Electronic storage  19  may include electronic storage media that electronically stores information. The electronic storage media of electronic storage  19  may be provided integrally (i.e., substantially non-removable) with UAV  10  and/or removable storage that is connectable to UAV  10  via, for example, a port (e.g., a USB port, a Firewire port, etc.) or a drive (e.g., a disk drive, etc.). Electronic storage  19  may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EPROM, EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. Electronic storage  19  may store software algorithms, information determined by processor  18 , information received remotely, and/or other information that enables UAV  10  to function properly. For example, electronic storage  19  may store captured visual information (as discussed elsewhere herein), and/or other information. Electronic storage  19  may be a separate component within UAV  10 , or electronic storage  19  may be provided integrally with one or more other components of UAV  10  (e.g., processor  18 ). 
     Although the invention has been described in detail for the purpose of illustration based on what is currently considered to be the most practical and preferred implementations, it is to be understood that such detail is solely for that purpose and that the invention is not limited to the disclosed implementations, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present invention contemplates that, to the extent possible, one or more features of any implementation can be combined with one or more features of any other implementation.