Lidar Sensor with Dynamic Projection Patterns

Systems and methods for sensing objects are provided. An optical apparatus can include a transmitter configured to project on a surface of an object, a first optical pattern having a first set of characteristics and a second optical pattern having a second set of characteristics. The optical apparatus can include a receiver configured to receive first and second reflected optical patterns representing a reflection of the first optical pattern and the second optical pattern from the surface of the object, and generate first and second electrical signals representing the first and the second reflected optical patterns. The optical apparatus can include one or more processors configured to receive the first electrical signals and the second electrical signals, and determine one or more characteristics of the object, including range information of the object.

FIELD

The present disclosure relates generally to sensor apparatuses. In particular, the present disclosure relates to a Lidar with multiple projection patterns.

BACKGROUND

A light detection and ranging (Lidar) sensor is a device, module, machine, subsystem, or system with a purpose to detect range information (e.g., how far an object is from the lidar) of objects in its environment and send the information to other electronics. Lidar can be used in many applications, including automotive, robotics, consumer electronics (e.g., mobile, wearable, or portable devices), and many other suitable applications.

SUMMARY

One example aspect of the present disclosure is directed to an optical apparatus including a transmitter configured to project on a surface of an object, a first optical pattern having a first set of characteristics; and project on the surface of the object, a second optical pattern having a second set of characteristics that are different from the first set of characteristics. The optical apparatus further includes a receiver configured to receive a first reflected optical pattern representing a reflection of the first optical pattern from the surface of the object; generate first electrical signals representing the first reflected optical pattern; receive a second reflected optical pattern representing a reflection of the second optical pattern from the surface of the object; and generate second electrical signals representing the second reflected optical pattern. The optical apparatus further includes one or more processors configured to receive the first electrical signals and the second electrical signals; and determine, based on the first electrical signals and the second electrical signals, one or more characteristics of the object, where the one or more characteristics include range information of the object.

Another example aspect of the present disclosure is directed to a method for operating an optical apparatus including projecting, by a transmitter, a first optical pattern having a first set of characteristics onto a surface of an object; projecting, by the transmitter, a second optical pattern having a second set of characteristics that are different from the first set of characteristics onto the surface of the object; receiving, by a receiver, a first reflected optical pattern representing a reflection of the first optical pattern from the surface of the object; generating, by the receiver, first electrical signals representing the first reflected optical pattern; receiving, by the receiver, a second reflected optical pattern representing a reflection of the second optical pattern from the surface of the object; generating, by the receiver, second electrical signals representing the second reflected optical pattern; and determining, by one or more processors and based on the first electrical signals and the second electrical signals, one or more characteristics of the object, where the one or more characteristics include range information of the object.

Another example aspect of the present disclosure is directed to a light detection and ranging (LIDAR) device including a transmitter configured to project on a surface of an object, a first optical pattern having a first dot density; and project on the surface of the object, a second optical pattern having a second dot density that is different from the first dot density. The LIDAR device further includes a germanium-based receiver formed on a silicon substrate, the germanium-based receiver configured to receive a first reflected optical pattern representing a reflection of the first optical pattern from the surface of the object; generate first electrical signals representing the first reflected optical pattern; receive a second reflected optical pattern representing a reflection of the second optical pattern from the surface of the object; and generate second electrical signals representing the second reflected optical pattern. The LIDAR device further includes silicon-based control circuitry configured to control the transmitter or the germanium-based receiver. The LIDAR device further includes one or more processors configured to receive the first electrical signals and the second electrical signals; and determine, based on the first electrical signals and the second electrical signals, one or more characteristics of the object, where the one or more characteristics include range information of the object.

Other example aspects of the present disclosure are directed to systems, methods, apparatuses, sensors, computing devices, tangible, non-transitory computer-readable media, and memory devices.

DETAILED DESCRIPTION

A light detection and ranging (Lidar) sensor is a device, module, machine, subsystem, or system with a purpose to detect range information (e.g., how far an object is from the lidar) of objects in its environment and send the information to other electronics. Lidar can be used in many applications, including automotive, robotics, consumer electronics (e.g., mobile, wearable, or portable devices), and many other suitable applications.

In some implementations, a lidar may flood (or flash, to be used interchangeably) a targeted scene (e.g., a portion of a sidewalk) with an optical pattern (e.g., a pattern of dots) to simultaneously get multiple detection points of the targeted scene. To increase the distance of 3D seeable range, the flood laser may transmit a dot pattern to concentrate the part of flood laser power while keeping a wide field of view (FOV). For example, a flood laser may have a peak power of 1 W with a spot size (e.g., area of the illumination) of A. If the area A is concentrated to 1/10 while maintaining the flood laser's peak power, the intensity (e.g., W/m2) of the light will increase by 10 times, as the dot density of the light has increased by 10 times. Such increase in dot density could improve the sensitivity at the receiver side (as there are more photons reflecting from a same area), but the resolution of the result (e.g., point cloud) could decrease (as photons will be concentrated to a smaller area, and the distance between two dots will increase).

To solve the low-resolution issue, some users use the algorithm to stitch the low-resolution information leveraging the SLAM (Simultaneous Localization and Mapping) technique. However, the SLAM algorithm assumes that most objects are static, and it may not be accurately applied on moving objects.

This disclosure describes utilizing multiple projector patterns generated either from a single transmitter with a tunable diffuser, or from multiple transmitters, to extend the seeable range and to compensate low resolution. For example, by combining the benefits of a first pattern with high power density/low flood area and a second pattern with low power density/high flood area, a lidar system can extend its detectable range while keeping high resolution for detecting closer-by objects. Moreover, by leveraging the benefit of a wide bandwidth photodetector such as a germanium-on-silicon (GeSi) photodetector, a lidar system may use a first transmitter having a first wavelength (e.g., 940 nm) to keep an overall low power consumption, and also use a second transmitter having a second wavelength (e.g., 1380 nm, which has a much lower absorption coefficient for the human eyes) capable of emitting a higher power to extend seeable range while keeping eye safety.

FIG. 1Ashows an example of a system100that includes a lidar system110and a target object130. The lidar system110includes one or more transmitters112, one or more receivers114, control circuitry116, a scanner118, and one or more processors120.

Each of the one or more transmitters112can include one or more laser sources for emitting optical signals with a specific wavelength or multiple wavelengths (e.g., visible, near infrared (NIR, e.g., wavelength range from 780 nm to 1400 nm, or any similar wavelength range as defined by a particular application), short-wave infrared (SWIR, e.g., wavelength range from 1400 nm to 3000 nm, or any similar wavelength range as defined by a particular application), etc.). The one or more transmitters112are configured to project on a surface of the target object130, a first optical pattern having a first set of characteristics. For example, the one or more transmitters112may emit an optical signal122atowards the target object130. The optical signal122amay have an optical pattern such as a dot pattern as described in reference toFIG. 3A.

The one or more transmitters112are configured to project on the surface of the target object130, a second optical pattern having a second set of characteristics that are different from the first set of characteristics. For example, the one or more transmitters112may emit an optical signal122btowards the target object130. The optical signal122bmay have another optical pattern such as a dot pattern as described in reference toFIG. 3B, where the density of the dot is different.

The first set of characteristics and the second set of characteristics may include a dot density of the dot pattern. Referring toFIG. 3A, a surface (e.g., a surface on the target object130) is illuminated with a first dot pattern302, where a diameter of each dot is designated as d1. Referring toFIG. 3B, the surface is illuminated with a second dot pattern304, where a diameter of each dot is designated as d2that is larger than d1. Each dot in a dot pattern has a dot density representing the number of photons hitting the dot within a given time (e.g., Joule per second per unit area). Assuming that the output laser power of the transmitter112is the same for generating both the first dot pattern302and the second dot pattern304, the first dot pattern density would be higher than the second dot pattern density. A dot pattern having a higher dot density (e.g., the first dot pattern302) generally provides a higher detection range for a lidar system, as there is a better chance that a photon will be reflected back to the receiver for a given spot on an object surface. By contrast, a dot pattern having a lower dot density (e.g., the second dot pattern304) generally provides a higher resolution, as the dot covers more areas on the object surface.

Referring toFIG. 2, a dot pattern may be generated by a combination of one or more lasers202, passive optics204, and a pattern generator206. As an example, after the laser202emits an optical signal, the passive optics204may collimate the optical signal and guide the collimated optical signal to the diffuser204. In some implementations, the pattern generator206may be a diffuser. The diffuser may be implemented using liquid crystal or phase delay, such that a dot pattern may be formed at the illumination plane (e.g., surface of an object) at the same time. In some other implementations, the pattern generator206may be a scanner-based system (e.g., MEMS-based or rotational mirror-based), where a dot pattern may be formed at the illumination plane over a period of time (e.g., within one image frame).

In some implementations, the pattern generator206may be dynamically controlled to form different patterns (e.g., a dot pattern with different dot densities) at different time intervals based on one or more control signals. For example, the pattern generator206may be controlled to form a dot pattern having a higher dot density (e.g., the first dot pattern) during a first time interval (e.g., 0 to 10 msec), and may then be controlled to form a dot pattern having a lower dot density (e.g., the second dot pattern) during a second time interval (e.g., 10 msec to 20 msec).

Referring back toFIG. 1A, each of the one or more receivers114can include one or more photodetectors (e.g., photodiodes, time-of-flight (ToF) sensors, avalanche photodetectors (APD), single-photon avalanche diode (SPAD), etc.) for receiving optical signals with a specific wavelength or multiple wavelengths (e.g., visible, near infrared (NIR), short-wave infrared (SWIR), etc.). The photodetector(s) may be discrete (e.g., a single photodiode) or an integrated array (e.g., a 1-D or 2-D array). The one or more receivers114are configured to receive a first reflected optical pattern representing a reflection of the first optical pattern from the surface of the target object130, and to generate first electrical signals representing the first reflected optical pattern. For example, a photodetector array of the one or more receivers114may receive a reflected optical pattern124athat has been reflected from the target object130. In response to receiving the reflected optical pattern124a,the photodetector array may generate first electrical signals (e.g., currents) representing the reflected optical pattern124a.

The one or more receivers114are configured to receive a second reflected optical pattern representing a reflection of the second optical pattern from the surface of the target object130, and to generate second electrical signals representing the second reflected optical pattern. For example, a photodetector array of the one or more receivers114may receive a reflected optical pattern124bthat has been reflected from the target object130. In response to receiving the reflected optical pattern124b,the photodetector array may generate second electrical signals (e.g., currents) representing the reflected optical pattern124b.

The control circuit116is configured to control the transmitter(s)112and the receiver(s)114. For example, the control circuit116may control a power level of the transmitter(s)112, or may issue control signals to modulate the optical signals of the transmitter(s)112. As another example, the control circuit116may issue control signals to control a timing of readouts at the receiver(s)114. In some implementations, the control circuit116may be formed monolithically with the transmitter(s)112and/or the receiver(s)114. In some other implementations, the control circuit116may be formed separately (e.g., using a CMOS fabrication process) and then coupled (e.g., wire-bond, wafer bonding, etc.) with the transmitter(s)112and/or the receiver(s)114.

Referring toFIG. 6, an example photodetector600is formed on a substrate of a first material (e.g., silicon). The photodetector600may be a single photodiode (e.g., linear photodiode, APD, SPAD, etc.), or a pixel of a germanium-on-silicon-based pixel array configured to receive the first reflected optical pattern and the second reflected optical pattern. The photodetector600includes an absorption region604of a different material (e.g., germanium) for receiving an optical signal to generate electrical signals (e.g., electrons or holes). In some implementations, the photodetector600may be bonded to a different substrate606(e.g., silicon wafer), where control circuit (e.g., control circuit116) has been formed on the substrate606. The photo-generated electrical signals from the absorption region604may be read by the control circuit either through the absorption region604or through the substrate602, depending on the photodetector design and operation.

Referring back toFIG. 1A, the scanner118is configured to scan optical signals transmitted by the transmitter(s)112over time to obtain a representation of a three-dimensional environment. In some implementations, the scanner118may include a MEMS mirror (or MEMS mirror array) that's integrated with the transmitter(s)112. In some other implementations, the scanner118may include a discrete optical mirror or prism. The scanner118may be controlled together with the pattern generator206. In some other implementations, the lidar system110may not include a scanner. For example, the lidar system110may be integrated inside a consumer electronics device, and therefore a scanning function would not be needed. As another example, the lidar system110may be arranged on a vehicle to detect the environment along a fixed orientation.

The one or more processors120may include hardware circuitry (e.g., FPGA, PCB, CPU, etc.) and/or computer storage medium (e.g., memories) that may store instructions for performing computational tasks. The one or more processors120are configured to receive the first electrical signals and the second electrical signals, and determine, based on the first electrical signals and the second electrical signals, one or more characteristics of the target object130, where the one or more characteristics include range information of the object.

As one example, the processor(s)120may determine first range information of the target object130based on the first electrical signals that are generated by the first optical pattern having a higher dot density. The processor(s)120may then determine second range information of the object based on the second electrical signals that are generated by the second optical pattern having a lower dot density but larger dot size. The processor(s)120may then adjust the second range information based on the first range information. In one example scenario, the higher concentration dot would get better signals with less noise, and the lower concentration of dot would get higher spatial resolution but worse signals. The processor(s)120may use the first range information having a lower noise to compensate the noise level of the second range information having high spatial resolution, such that a high spatial resolution 3D image with a lower noise may be example generated. In another example scenario, the higher concentration dot may result in over-exposure (or saturation due to high optical intensity) at the receiver(s)114, and the processor(s)120may use the lower concentration of dot to correct or compensate the first range information, such that a high spatial resolution 3D image with a higher dynamic range may be generated by the processor(s)120.

As another example, after obtaining the first range information and the second range information, the processor(s)120may then select, based on one or more selection criteria, one of the first range information or the second range information to determine the characteristics of the target object130. The selection criteria may include a sensitivity of the receiver, a saturation level of the receiver, and/or a dark current of the receiver. In one example scenario, if the target object130is beyond the detectable range of the lower concentration dot pattern, the lower concentration dot may result in a high noise level at the receiver(s)114. In response to determining that the noise level associated with the lower concentration dot pattern exceeds a threshold, the processor(s)120may use only the higher concentration dot pattern to determine the depth information of the target object.

FIG. 1Billustrate an example of another system101that is similar as the system100as described in reference toFIG. 1A. Here, the transmitter(s)112of the lidar system110may include one or more first lasers configured to transmit optical signals for the first optical pattern, and one or more second lasers configured to transmit optical signals for the second optical pattern. In some implementations, the first laser(s) and the second laser(s) may transmit optical signals having the same wavelength. In some other implementations, the first laser(s) may transmit optical signals having a first wavelength (e.g., 940 nm), while the second laser(s) may transmit optical signals having a second wavelength (e.g., 1310 nm). The receiver(s)114may be divided into multiple regions for receiving optical signals having different wavelengths. For example, the receiver(s)114may include a first germanium-on-silicon pixel array optically coupled to a first filter (e.g., a bandpass filter designed to pass the first wavelength) in order to receive a reflected optical pattern having the first wavelength. The receiver(s)114may further include a second germanium-on-silicon pixel array optically coupled to a second filter (e.g., a bandpass filter designed to pass the second wavelength) in order to receive a reflected optical pattern having the second wavelength. The processor(s)120may process the electrical signals collected from the first and second germanium-on-silicon pixel arrays to determine a characteristic of the target object130as described in reference toFIG. 1A. Operating two (or more) wavelengths at the same time can be beneficial in certain weather conditions, where one wavelength may have a lower water absorption coefficient than the other wavelength, and therefore would enhance the operability of certain applications (e.g., autonomous driving). As another example, operating two (or more) wavelengths at the same time can enable other applications such as material classification.

FIG. 4illustrates an example process for operating a lidar system. The lidar system can be, for example, the lidar system110as described in reference toFIGS. 1A and 1B. The lidar system projects, by a transmitter, a first optical pattern having a first set of characteristics onto a surface of an object (402). For example, the transmitter(s)112may emit an optical signal122atowards the target object130. The optical signal122amay have an optical pattern such as a dot pattern as described in reference toFIG. 3A.

The lidar system projects, by the transmitter, a second optical pattern having a second set of characteristics that are different from the first set of characteristics onto the surface of the object (404). For example, the one or more transmitters112may emit an optical signal122btowards the target object130. The optical signal122bmay have another optical pattern such as a dot pattern as described in reference toFIG. 3B, where the density of the dot is different.

The lidar system receives, by a receiver, a first reflected optical pattern representing a reflection of the first optical pattern from the surface of the object, and generates, by the receiver, first electrical signals representing the first reflected optical pattern (406). For example, a photodetector array of the one or more receivers114may receive a reflected optical pattern124athat has been reflected from the target object130. In response to receiving the reflected optical pattern124a,the photodetector array may generate first electrical signals (e.g., currents) representing the reflected optical pattern124a.

The lidar system receives, by the receiver, a second reflected optical pattern representing a reflection of the second optical pattern from the surface of the object, and generates, by the receiver, second electrical signals representing the second reflected optical pattern. For example, a photodetector array of the one or more receivers114may receive a reflected optical pattern124bthat has been reflected from the target object130. In response to receiving the reflected optical pattern124b,the photodetector array may generate second electrical signals (e.g., currents) representing the reflected optical pattern124b.

The lidar system determines, by one or more processors and based on the first electrical signals and the second electrical signals, one or more characteristics of the object, wherein the one or more characteristics include range information of the object (408). For example, the processor(s)120may determine first range information of the target object130based on the first electrical signals that are generated by the first optical pattern having a higher dot density. The processor(s)120may then determine second range information of the object based on the second electrical signals that are generated by the second optical pattern having a lower dot density but larger dot size. The processor(s)120may then adjust the second range information based on the first range information.

FIGS. 5A-5Cillustrate examples for operating a lidar system, such as the lidar system110as described in reference toFIGS. 1A and 1B.FIG. 5Ashows an example where the transmitter112may be a single source, where the diffuser204may be controlled to dynamically switch diffuse patterns to generate two dot patterns that alternate in time. In this example, one photodetector (or one photodetector array) of the receiver114receives two dot patterns that alternate in time.

FIG. 5Bshows an example where the transmitter(s)112may include multiple (e.g., two) sources, where multiple dot patterns are emitted by the transmitter(s) at alternate times. The two sources here can operate at the same wavelength or different wavelengths. Here, the diffuser204may be static or be controlled dynamically to generate multiple dot patterns for each source that alternate in time. In this example, one photodetector (or one photodetector array) of the receiver114receives two dot patterns that alternate in time.

FIG. 5Cshows an example where the transmitter(s)112may include multiple (e.g., two) sources, where multiple dot patterns are emitted by the transmitter(s) at the same time. The two sources here can operate at the same wavelength or different wavelengths. Here, the diffuser204may be static or be controlled dynamically to generate multiple dot patterns for each source that alternate in time. In this example, multiple photodetectors (or multiple regions of a photodetector array or multiple photodetector arrays) of the receiver114are arranged to receive corresponding optical signals as described in reference toFIG. 1B.

Various implementations may have been discussed using two-dimensional cross-sections for easy description and illustration purpose. Nevertheless, the three-dimensional variations and derivations should also be included within the scope of the disclosure as long as there are corresponding two-dimensional cross-sections in the three-dimensional structures.

Thus, particular embodiments have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results.