Patent Description:
A fundamental challenge of any autonomy-related technology relates to collecting and interpreting information about a vehicle's surrounding environment, along with planning and executing commands to appropriately control vehicle motion to safely navigate the vehicle through its current environment. Therefore, continuing efforts are being made to improve each of these aspects, and by doing so, autonomous vehicles increasingly are able to reliably operate in increasingly complex environments and accommodate both expected and unexpected interactions within an environment. <CIT> relates to a system configured to implement an application that applies artificial intelligence and/or machine-learning techniques to predict an optimal course of action for an autonomous vehicle system to undertake based on suboptimal autonomous vehicle performance and/or changes in detected sensor data. <CIT> may determine a subset of trajectories based on a number of decisions and interactions when resolving an anomaly due to an event or condition. The application may use aggregated sensor data from multiple autonomous vehicles to assist in identifying events or conditions that might affect travel. An optimal subset of trajectories in <CIT> may be formed based on recommendations responsive to semantic changes. <CIT> relates to an object fusion module that normalizes sensor output from vision, RADAR, and LIDAR systems into a common format. Then, the system of <CIT> fuses the object-level sensor data across all systems by associating all objects detected and predicting tracks for all objects. The system of <CIT> uses the data from all sensors combined to develop a single set of knowledge about the objects around the self-driving car, instead of each sensor operating separately. <CIT> relates to a monopulse optical receiving system that is comprised of a plurality of optical sensors for receiving an incident light pulse at angles of incidence within a field of view, and providing in response thereto an output voltage signal whose magnitude is a function of the incident angle, at least two of the sensors being relatively disposed so that a portion of the field of view of one of them overlaps with a portion of the field of view of another one of them to provide an overlapping field of view within which the two sensors provide complementary transmittance gradients. <NPL> discuses a distributed architecture for adaptive sensor fusion introduced for 3D imagery data that makes use of a technique computed with a Bregman-Iteration deconvolution algorithm. The architecture discussed in this paper is a cascaded neural network, which consists of two levels of neural networks. <CIT> relates to a FM LIDAR system including a frequency modulated LIDAR system that incorporates a laser source that is optically coupled to a whispering gallery mode optical resonator. Light from the laser that is coupled into the whispering gallery mode optical resonator is coupled back out as a returning counterpropagating wave having a frequency characteristic of a whispering gallery mode of the optical resonator. This returning wave of <CIT> is used to reduce the linewidth of the source laser by optical injection.

According to an aspect of the present invention, there is provided a method as set out in claim <NUM>. Other embodiments are described in the dependent claims.

In some implementations, phase coherent LIDAR data includes a range-Doppler image.

In some implementations, phase coherent LIDAR data includes an intermediate frequency waveform generated based on mixing of a local optical oscillator with time delayed reflections during the corresponding sensing events.

In addition, some implementations include an autonomous vehicle with one or more processors that are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the methods described herein. Some implementations additionally or alternatively include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the methods described herein.

For example, all combinations of claimed subject matter appearing at the end of this invention are contemplated as being part of the subject matter disclosed herein.

In various levels of autonomous control of a vehicle (e.g., any of SAE Levels <NUM>-<NUM> described above), it is necessary to determine various parameters of the so-called ego vehicle (i.e., the vehicle being autonomously controlled, hereinafter also referred to as the "vehicle" for brevity) - and to adapt autonomous control of the ego vehicle based on the various determined parameters of the ego vehicle itself. In various levels of autonomous control of the ego vehicle, it is also desirable and/or necessary to determine various parameters of additional object(s) that are in the environment of the ego vehicle and that are in addition to the ego vehicle. For example, it can be necessary to determine the pose (position and optionally orientation) of various object(s) in the environment and/or the classification(s) of those object(s). Moreover, for at least some objects it can be desirable to determine movement parameter(s) for those objects such as velocity, acceleration, a candidate trajectory (which can be based on velocity and/or acceleration), etc. For instance, at each of multiple iterations, a trajectory of the ego vehicle can be determined based at least in part on various parameters of object(s) in the ego vehicle's environment, such as pose(s), classification(s), and/or movement parameter(s) (if any) of the object(s). The trajectory can be determined so as to prevent the ego vehicle from contacting and/or being too proximal to any of the object(s).

Implementations disclosed herein are generally directed to determining classification(s) for object(s) in an environment of a vehicle, and controlling the vehicle based on the determined classification(s). For example, autonomous steering, acceleration, and/or deceleration of the vehicle can be adapted based on the determined classification(s) of the additional object(s). For instance, steering and/or acceleration/deceleration parameters determined for a vehicle based on a first environmental object that is determined to be a pedestrian can be different from those determined had the object instead been determined to be an additional vehicle.

In some implementations, a determined pose of an object and a determined classification of the object can be utilized in controlling the vehicle. As one example, if a determined pose of an object is not within (or within a threshold distance of) a current intended path of a vehicle, and the object is determined to have a classification that indicates the object is immobile (e.g., a tree classification), that object can optionally be ignored in determining whether control of the vehicle needs to be adapted. On the other hand, if a determined pose of an object is within (or within a threshold distance of) a current intended path of the vehicle and has a classification that indicates the object is mobile (e.g., vehicle, pedestrian, cyclist, or animal classification), that object can be utilized in determining whether control of the vehicle needs to be adapted.

Further, in some of those implementations, movement parameter(s) for an object (which can optionally be determined based on its classification) can be determined and utilized in controlling the vehicle. For example, velocity and/or acceleration of an object can be utilized to generate a candidate trajectory for the object, and the vehicle controlled based on the candidate trajectory such that the vehicle adapts its own trajectory to avoid being too proximal to the candidate trajectory of the object. How proximal is too proximal can optionally be dependent on, for example, the classification of the object. For example, any value less than <NUM> meter for an object with a bicyclist classification can be too close when the trajectories are substantially parallel, whereas only values less than <NUM> meters for an object with a vehicle classification can be too close when the trajectories are substantially parallel.

In many implementations disclosed herein, poses and classifications of environmental objects are determined based on data from a phase coherent Light Detection and Ranging (LIDAR) component of the vehicle. Moreover, in some of those implementations, velocities and optionally accelerations of the environmental objects are also determined based on the same data from the phase coherent LIDAR component.

Data from a phase coherent LIDAR component of a vehicle is referred to herein as "phase coherent LIDAR data". As described herein, phase coherent LIDAR data can indicate, for each of a plurality of points in an environment, at least a range for the point and a velocity for the point. The range for a point indicates distance to the point from receiver(s) of the phase coherent LIDAR component. The velocity for the point indicates a magnitude (and optionally a direction) of the velocity for the point. The indicated range and velocity for a point is based on a corresponding sensing event of the phase coherent LIDAR component. Accordingly, through sensing events of a sensing cycle, phase coherent LIDAR data indicating instantaneous ranges and instantaneous velocities for a large quantity of points in the environment can be determined. Implementations disclosed herein can determine, based on the instantaneous ranges and instantaneous velocities indicated by at least some of the FMCW LIDAR data, a pose and classification for an object, as well as optionally an instantaneous velocity for the object. In those implementations where an instantaneous velocity for an object is determined based on phase coherent LIDAR data of a sensing cycle, the instantaneous velocity indicates velocity of the object during the sensing cycle, without requiring reference to phase coherent LIDAR data from future sensing cycle(s). This enables quick resolution of an instantaneous velocity of the object, and resultantly enables quick determination of whether adaptation of control of the vehicle is needed in view of the instantaneous velocity (and quick adaptation when it is determined adaptation is needed).

Generally, LIDAR is a technique that uses reflections of electromagnetic waves in the optical spectra to determine ranges of points in an environment. For example, some LIDAR implementations determine the ranges of the points based on measuring the round-trip delay of light pulses to the points. A LIDAR component can include a light source (e.g., a laser) for generating light pulses in the optical spectra, optics (e.g., mirrors) for directing the light source (e.g., to scan the azimuth and elevation), a sensor for measuring reflected pulses, and local processor(s) and/or other electronic components for processing the reflected pulses. The reflected pulses can be processed to determine ranges to points of target(s) in the environment based on time delay between transmitted and reflected pulses (and using the speed of light). In some implementations, LIDAR components generate output, based on the processing, that is a three-dimensional (3D) point cloud with intensity values (based on the intensity of the reflected pulses).

Generally, phase coherent LIDAR is a technique that uses reflections of electromagnetic waves in the optical spectra to determine data that indicates ranges and velocities of points in an environment. A phase coherent LIDAR component includes at least one tunable laser (or other light source) that generates a waveform, and includes at least one receiver, such as a coherent optical receiver. The phase coherent LIDAR component can also include one or more components for directing the tunable laser, such as optics that can be utilized to direct (in azimuth and/or elevation) the waveform from the tunable laser to a plurality of points in an environment. Further, the phase coherent LIDAR component can include local processor(s) and/or other electronic components for processing reflections of the waveform. The tunable laser of a phase coherent LIDAR component is tunable in that it can be controlled so that the waveform it generates is encoded by modifying one or more properties of the waveform across the waveform. The tunable laser can be controlled to generate one or more of various types of encoded waveforms such as chirped, stepped, and/or pulsed waveforms.

A frequency-modulated continuous wave (FMCW) LIDAR component is one particular type of phase coherent LIDAR component. An FMCW LIDAR component at least selectively (e.g., always when in use) controls its tunable laser so that it generates a waveform that is both: encoded through frequency modulation, and continuous. For example, the waveform can be a continuous stepped waveform that is encoded through frequency modulation. For instance, the tunable laser can be controlled to generate a stepped waveform that is encoded through modification of the frequency of the steps. Data from an FMCW LIDAR component of a vehicle is referred to herein as "FMCW LIDAR data". Accordingly, "phase coherent LIDAR data" can be "FMCW LIDAR data" when it is generated by an FMCW LIDAR component based on a waveform that is both encoded through frequency modulation and is continuous.

During a sensing cycle of a phase coherent LIDAR component, the transmitted encoded waveform is sequentially directed to, and sequentially reflects off of, each of a plurality of points in the environment. Reflected portions of the encoded waveform are each detected, in a corresponding sensing event of the sensing cycle, by the at least one receiver of the phase coherent LIDAR component. For example, the reflected portion of a sensing event can be detected by the at least one coherent optical receiver using a coherent-detection scheme in which the reflected portion is mixed with a local optical signal that corresponds to the transmitted encoded waveform. The coherent-detection scheme can be, for example, homodyne mixing, heterodyne mixing, self-heterodyne mixing, or other coherent-detection scheme. The resulting signal generated from mixing the reflected portion with the local optical signal indicates both: (<NUM>) range to the corresponding point, and (<NUM>) velocity of the corresponding point. For example, the resulting signal can be processed by applying a two-dimensional Fourier transform to the signal, to generate a 2D spectrum with a frequency coordinate in the direct dimension and a frequency coordinate in the indirect dimension. The frequency coordinate in the direct dimension can be utilized to determine the range and the frequency coordinate in the indirect dimension can be utilized to determine the velocity. As one particular example, a homodyne detection scheme can be used in which the local optical signal is a split portion of the transmitted encoded waveform and is used as an optical local oscillator signal for the optical receiver. Mixing the local oscillator with a time delayed reflected portion during the sensing event yields a time varying intermediate frequency (IF) waveform, which is directly related to the range of the corresponding point. Further, when the point is moving, a Doppler frequency shift will be superimposed to the IF waveform, which can be used to determine radial velocity of the point. Through detections at multiple sensing events of a sensing cycle, phase coherent LIDAR data is generated for each of a plurality of points in the environment of the vehicle - and indicates range, velocity, and optionally intensity, for each of the points in the environment for the sensing cycle.

As mentioned above, phase coherent LIDAR data generated by a phase coherent LIDAR component is utilized in determining classification(s), pose(s), and/or instantaneous velocity/velocities of object(s) in various implementations disclosed herein. In those implementations, the phase coherent LIDAR data of a sensing cycle indicates, directly or indirectly, a corresponding range and corresponding velocity for each of a plurality of points in an environment of a vehicle to which the phase coherent LIDAR component is coupled. In some of those implementations, the phase coherent LIDAR data includes a three-dimensional (3D) point cloud, with each point in the 3D point cloud defining a corresponding range (via its 3D coordinate in the 3D point cloud), corresponding velocity, and optionally a corresponding intensity. In some other implementations, the phase coherent LIDAR data includes a range-Doppler image, such as a range-Doppler image where each pixel defines a corresponding velocity (and optionally intensity) for a corresponding range and cross-range. In other words, each pixel in the range-Doppler image corresponds to a corresponding range and cross-range (which corresponds to a point in the environment), and defines a velocity for that range and cross-range. In yet other implementations, the phase coherent LIDAR data includes a time varying intermediate frequency (IF) waveform generated by optical receiver(s) of the FMCW LIDAR component based on mixing the local oscillator with the time delayed reflection. The time varying IF waveform can be utilized directly to determine classification(s), pose(s), and/or instantaneous velocity/velocities - and/or can be processed using a Fourier transform and/or other techniques, and the processed data utilized to determine the classification(s), pose(s), and/or instantaneous velocity/velocities. For example, each slice of the time varying IF waveform can be processed to determine range and velocity for a corresponding point in the environment, and the ranges and velocities for multiple points utilized in determining classification(s), pose(s), and/or instantaneous velocity/velocities. Accordingly, one or more of the preceding and/or other types of phase coherent LIDAR data can be generated by the phase coherent LIDAR component and utilized in determining feature(s) of an object according to implementations disclosed herein. The type(s) of phase coherent LIDAR data generated and utilized in determining feature(s) can be dependent on, for example, the particular phase coherent LIDAR component being utilized. Also, it is noted that in some implementations the velocities indicated in received phase coherent LIDAR data can have the ego vehicle's velocity subtracted out (e.g., based on a velocity of the ego vehicle determined based on sensor(s) of the ego vehicle, or based on inferring the velocity of the ego vehicle based on velocities of static objects in the phase coherent LIDAR data). In some other implementations, the ego vehicle's velocity may not be subtracted out of the received phase coherent LIDAR data. In those implementations, the received phase coherent LIDAR data may optionally be processed to subtract out the ego vehicle's velocity.

In some implementations, the phase coherent LIDAR data is from a phase coherent LIDAR monopulse component that includes at least a first and second receiver, such as a first and second coherent optical receiver. In those implementations, each sensing event includes a first receiver sensing event by the first receiver and a second receiver sensing event by the second receiver. For example, the first receiver sensing event can include coherent mixing of a local oscillator with a first detection of a reflected portion of the waveform and the second receiver sensing event can include coherent mixing of the local oscillator with a second detection of a reflected portion of the waveform. The first receiver sensing event and the second receiver sensing event are denoted herein as separate events as they are by separate receivers. However, it is noted that the sensing events can each be of the same reflection and can occur concurrently, or substantially concurrently (e.g., within one millisecond of one another, within one microsecond of one another, or within one nanosecond of one another).

The first and second receivers are positionally offset relative to one another, thereby causing the first and second detections to vary (e.g., temporally and/or characteristically (e.g., in phase)). This enables at least one extra channel of data to be determined for a sensing event based on comparison of the first receiver sensing event and the second receiver sensing event. The extra channel can be utilized to increase the angular accuracy for point(s) determined based on the sensing event. For example, the extra channel can be based on amplitude and/or phase differences between the first and second receiver sensing events (e.g., differences in the time varying IF waveform generated by each of the first and second receivers in mixing with the local oscillator signal). For instance, the amplitude and/or phase differences can be compared to determine an angle (a particular angle or range of angles), for each of one or more point(s) determined based on the sensing event, where each angle is resolved so that the point(s) are smaller than the beam width of the reflection that resulted in the sensing event. Accordingly, the angle (e.g., azimuth and elevation relative to the receiver(s)) for a corresponding range can be determined to a sub-beam width level, thereby providing greater accuracy of the determined angle. Moreover, the point can be localized to a sub-beam width level as the position of the point is defined by the angle and the range.

It is noted that, in some implementations, multiple range/velocity pairs can be determined based on a single sensing event (e.g., based on differences in range and/or velocity among the multiple range/velocity pairs indicated by a range-Doppler image and/or IF waveform). In those implementations, a corresponding angle can be determined for each of the multiple range/velocity pairs, and can each be resolved to a sub-beam width level. In other words, a single sensing event of a sensing cycle can be utilized to determine a corresponding range and corresponding velocity for each of two or more points. In some versions of those implementations, beam width(s) of the electromagnetic output that is output by the laser of a phase coherent LIDAR component can be purposefully fattened/widened to enable covering of more space by the beam width, as compared to non-fattened beam widths. For example, a beam width can be fixed so that, at a given distance from the phrase coherent LIDAR component (e.g., <NUM> meters, <NUM> meters, <NUM> meters, <NUM> meters, or other distance), there are no coverage gaps between the beams of a sensing cycle. Increased angular accuracy techniques described herein can be utilized to determine a corresponding range and corresponding velocity for each of two or more points that are within the fattened beam width of each beam of the sensing cycle. This enables determination of ranges and velocities with a degree of accuracy that, absent increased angular velocity techniques described herein, would otherwise require a greater quantity of beams and sensing events in a sensing cycle. Accordingly, increased angular velocity techniques described herein enable resolution of ranges and velocities with a given degree of accuracy in a sensing cycle, while reducing the quantity of sensing events in the sensing cycle needed to achieve the degree of accuracy. This can decrease the quantity of sensing events that occur in each sensing cycle, thereby conserving hardware resources (e.g., laser resources and/or power source resources). This can additionally or alternatively enable resolution of a given quantity of ranges and velocities for an area in a reduced amount of time, thereby providing excess time for resolving ranges and velocities for additional area(s) (e.g., enabling a larger total area to be covered in each sensing cycle without increasing the duration of the sensing cycle) and/or enabling quick re-resolving of updated ranges and velocities for the area (e.g., enabling sensing cycles to be shortened, without reduction of angular accuracy).

Accordingly, with a phase coherent LIDAR monopulse component, super-pixel accuracy can be achieved while reducing the signal-to-noise ratio (SNR) increasing the noise) through splitting of the reflection across two or more receivers. This can be beneficial in determining ranges and velocities for point(s) of various objects that are far away (from the phase coherent LIDAR monopulse component) and/or small. For example, it can increase the angular accuracy (more accurate positioning, in an angular sense, for a given point) for which range and velocity can be determined for a given return / sensing event - thereby increasing the angular accuracy and enabling determination (or at least more accurate determination) of classification(s), a pose, and/or movement parameter(s) for an object that is far away and/or small, whereas classification(s), pose, and/or movement parameter(s) would otherwise not be determinable (or would at least be less accurate) for the object that is far away and/or small.

In various implementations, with a phase coherent LIDAR monopulse component, super-resolution can additionally and/or alternatively be achieved while reducing the signal-to-noise ratio (SNR) (i.e. increasing the noise) through splitting of the reflection across two or more receivers. The two or more receivers are positionally offset relative to one another, thereby causing respective detections of the receivers to vary (e.g., temporally and/or characteristically (e.g., in phase)). This enables non-redundant LIDAR data from the sensing event at each receiver to be combined to generate LIDAR data (e.g., a range-Doppler image and/or IF waveform) that is of higher resolution (e.g., higher resolution range and/or velocity) than the LIDAR data from any one of the sensing events at a respective receiver. In other words, each sensing event can include a receiver sensing event at each receiver (e.g., a first receiver sensing event at a first receiver, a second receiver sensing event at a second receiver, etc.) and produce respective LIDAR data for the respective receiver sensing event. The non-redundant LIDAR data for the receiver sensing events can be combined to construct super-resolution LIDAR data for the sensing event. The non-redundant LIDAR data can be based on, for example, amplitude and/or phase differences between the respective receiver sensing events (e.g., differences in the time varying IF waveform generated by each of the first and second receivers in mixing with the local oscillator signal). Various techniques can be utilized to generate super-resolution LIDAR data from multiple receiver sensing events, such as frequency domain techniques (e.g., based on shift and aliasing properties of the Continuous and Discrete Fourier Transforms) and/or spatial domain techniques (e.g., non-iterative and/or iterative techniques). Spatial domain techniques can be utilized to apply one or more flexible estimators in constructing super-resolution LIDAR data from lower resolution LIDAR data from receiver sensing events. Such flexible estimators include, for example, Maximum likelihood, Maximum a Posteriori, Projection Onto Convex Sets, Bayesian treatments, etc. Additional description of some phase coherent LIDAR techniques and some phase coherent LIDAR components are provided herein, including monopulse components and techniques.

As mentioned above, during a sensing cycle of a phase coherent LIDAR component, a transmitted encoded waveform is sequentially directed to, and sequentially reflects off of, each of a plurality of points in the environment - and reflected portions of the encoded waveform are each detected, in a corresponding sensing event of the sensing cycle, by the at least one receiver of the phase coherent LIDAR component. During a sensing cycle, the waveform is directed to a plurality of points in an area of an environment, and corresponding reflections detected, without the waveform being redirected to those points in the sensing cycle. Accordingly, the range and velocity for a point that is indicated by phase coherent LIDAR data of a sensing cycle can be instantaneous in that is based on single sensing event without reference to a prior or subsequent sensing event. In some implementations, each of multiple (e.g., all) sensing cycles can have the same duration, the same field-of-view, and/or the same pattern of waveform distribution (through directing of the waveform during the sensing cycle). For example, each of multiple sensing cycles that are each a sweep can have the same duration, the same field-of-view, and the same pattern of waveform distribution. However, in many other implementations the duration, field-of-view, and/or waveform distribution pattern can vary amongst one or more sensing cycles. For example, a first sensing cycle can be of a first duration, have a first field-of view, and a first waveform distribution pattern; and a second sensing cycle can be of a second duration that is shorter than the first, have a second field-of-view that is a subset of the first field-of-view, and have a second waveform distribution pattern that is denser than the first.

Turning now to the Figures, <FIG> illustrates an example autonomous vehicle <NUM> within which the various techniques disclosed herein may be implemented. Vehicle <NUM>, for example, may include a powertrain <NUM> including a prime mover <NUM> powered by an energy source <NUM> and capable of providing power to a drivetrain <NUM>, as well as a control system <NUM> including a direction control <NUM>, a powertrain control <NUM>, and brake control <NUM>. Vehicle <NUM> may be implemented as any number of different types of vehicles, including vehicles capable of transporting people and/or cargo, and capable of traveling by land, by sea, by air, underground, undersea and/or in space, and it will be appreciated that the aforementioned components <NUM>-<NUM> can vary widely based upon the type of vehicle within which these components are utilized.

The implementations discussed hereinafter, for example, will focus on a wheeled land vehicle such as a car, van, truck, bus, etc. In such implementations, the prime mover <NUM> may include one or more electric motors and/or an internal combustion engine (among others). The energy source <NUM> may include, for example, a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, solar panels or other renewable energy source, and/or a fuel cell system. Drivetrain <NUM> may include wheels and/or tires along with a transmission and/or any other mechanical drive components suitable for converting the output of prime mover <NUM> into vehicular motion, as well as one or more brakes configured to controllably stop or slow the vehicle <NUM> and direction or steering components suitable for controlling the trajectory of the vehicle <NUM> (e.g., a rack and pinion steering linkage enabling one or more wheels of vehicle <NUM> to pivot about a generally vertical axis to vary an angle of the rotational planes of the wheels relative to the longitudinal axis of the vehicle). In some implementations, combinations of powertrains and energy sources may be used (e.g., in the case of electric/gas hybrid vehicles), and in some instances multiple electric motors (e.g., dedicated to individual wheels or axles) may be used as a prime mover.

Direction control <NUM> may include one or more actuators and/or sensors for controlling and receiving feedback from the direction or steering components to enable the vehicle <NUM> to follow a desired trajectory. Powertrain control <NUM> may be configured to control the output of powertrain <NUM>, e.g., to control the output power of prime mover <NUM>, to control a gear of a transmission in drivetrain <NUM>, etc., thereby controlling a speed and/or direction of the vehicle <NUM>. Brake control <NUM> may be configured to control one or more brakes that slow or stop vehicle <NUM>, e.g., disk or drum brakes coupled to the wheels of the vehicle.

Other vehicle types, including but not limited to airplanes, space vehicles, helicopters, drones, military vehicles, all-terrain or tracked vehicles, ships, submarines, construction equipment, etc., will necessarily utilize different powertrains, drivetrains, energy sources, direction controls, powertrain controls and brake controls, as will be appreciated by those of ordinary skill having the benefit of the instant disclosure. Moreover, in some implementations some of the components may be combined, e.g., where directional control of a vehicle is primarily handled by varying an output of one or more prime movers. Therefore, implementations disclosed herein not limited to the particular application of the herein-described techniques in an autonomous wheeled land vehicle.

In the illustrated implementation, full or semi-autonomous control over vehicle <NUM> is implemented in a vehicle control system <NUM>, which may include one or more processors <NUM> and one or more memories <NUM>, with each processor <NUM> configured to execute program code instructions <NUM> stored in a memory <NUM>. The processor(s) <NUM> can include, for example, graphics processing unit(s) (GPU(s)) and/or central processing unit(s) (CPU(s)).

Sensors <NUM> may include various sensors suitable for collecting information from a vehicle's surrounding environment for use in controlling the operation of the vehicle. For example, sensors <NUM> can include a frequency-modulated continuous wave (FMCW) Light Detection and Ranging (LIDAR) component <NUM> of the vehicle (FMCW). As described herein, the FMCW LIDAR component <NUM> can generate FMCW LIDAR data that can be utilized in determining a pose, classification, and velocity for each of one or more objects in an environment of the vehicle <NUM>. The FMCW LIDAR data generated by FMCW LIDAR component <NUM> indicates, for each of a plurality of points in the environment of the vehicle, a corresponding range and a corresponding velocity based on a corresponding sensing event of the FMCW LIDAR component. Although an FMCW LIDAR component is illustrated in <FIG> (and other Figures) and described herein in association with many figures, in many implementations a phase coherent LIDAR component that is not an FMCW LIDAR component can instead be utilized. Further, in those implementations phase coherent LIDAR data, that is not FMCW LIDAR data, can be utilized in lieu of FMCW LIDAR data.

Sensors <NUM> can also optionally include a Radio Detection and Ranging (RADAR) component <NUM> and/or a LIDAR component <NUM> (non-phase coherent, that generates data that can be utilized to determine ranges, but not velocities). Further, sensors can also optionally include a satellite navigation (SATNAV) component <NUM>, e.g., compatible with any of various satellite navigation systems such as GPS, GLONASS, Galileo, Compass, etc. The SATNAV component <NUM> can be used to determine the location of the vehicle on the Earth using satellite signals. Sensors <NUM> can also optionally include a camera <NUM>, an inertial measurement unit (IMU) <NUM> and/or one or more wheel encoders <NUM>. The camera <NUM> can be a monographic or stereographic camera and can record still and/or video imagers. The IMU <NUM> can include multiple gyroscopes and accelerometers capable of detecting linear and rotational motion of the vehicle <NUM> in three directions. Wheel encoders <NUM> may be used to monitor the rotation of one or more wheels of vehicle <NUM>.

The outputs of sensors <NUM> may be provided to a set of control subsystems <NUM>, including, a localization subsystem <NUM>, a planning subsystem <NUM>, a perception subsystem <NUM>, and a control subsystem <NUM>. Localization subsystem <NUM> is principally responsible for precisely determining the location and orientation (also sometimes referred to as "pose") of vehicle <NUM> within its surrounding environment, and generally within some frame of reference. Perception subsystem <NUM> is principally responsible for detecting, tracking and/or identifying elements within the environment surrounding vehicle <NUM>. For example, and as described herein, perception subsystem <NUM> can, at each of a plurality of iterations, determine a pose, classification, and velocity for each of one or more objects in the environment surrounding vehicle <NUM>. Further, for example, the perception subsystem <NUM> can track various objects over multiple iterations. For instance, the perception subsystem <NUM> can track an additional vehicle over multiple iterations as the additional vehicle moves relative to the ego vehicle <NUM>. The perception subsystem <NUM> utilize detection, classification, and/or velocity techniques described herein in tracking of an object over multiple iterations. Planning subsystem <NUM> is principally responsible for planning a trajectory for vehicle <NUM> over some timeframe given a desired destination as well as the static and moving objects within the environment. For example, and as described herein, planning subsystem <NUM> can plan a trajectory for vehicle <NUM> based at least in part on a pose, classification, and/or velocity for each of one or more objects in an environment of the vehicle <NUM> in the environment surrounding vehicle <NUM>. In some implementations, planning subsystem <NUM> plans the trajectory for the vehicle <NUM> by generating, and considering, candidate trajectories for each of one or more additional mobile objects in the environment. Planning subsystem <NUM> can determine a candidate trajectory for an object at an iteration based on a pose, classification, and/or velocity for the iteration - and/or the tracking of the object over multiple iterations. Control subsystem <NUM> is principally responsible for generating suitable control signals for controlling the various controls in control system <NUM> in order to implement the planned trajectory of the vehicle <NUM>.

It will be appreciated that the collection of components illustrated in <FIG> for vehicle control system <NUM> is merely exemplary in nature. Individual sensors may be omitted in some implementations. Additionally or alternatively, in some implementations multiple sensors of the types illustrated in <FIG> may be used for redundancy and/or to cover different regions around a vehicle, and other types of sensors may be used. Likewise, different types and/or combinations of control subsystems may be used in other implementations. Further, while subsystems <NUM>-<NUM> are illustrated as being separate from processors <NUM> and memory <NUM>, it will be appreciated that in some implementations, some or all of the functionality of a subsystem <NUM>-<NUM> may be implemented with program code instructions <NUM> resident in one or more memories <NUM> and executed by one or more processors <NUM>, and that these subsystems <NUM>-<NUM> may in some instances be implemented using the same processors and/or memory. Subsystems in some implementations may be implemented at least in part using various dedicated circuit logic, various processors, various field-programmable gate arrays ("FPGA"), various application-specific integrated circuits ("ASIC"), various real time controllers, and the like, and as noted above, multiple subsystems may utilize circuitry, processors, sensors and/or other components. Further, the various components in vehicle control system <NUM> may be networked in various manners.

In some implementations, vehicle <NUM> may also include a secondary vehicle control system (not illustrated), which may be used as a redundant or backup control system for vehicle <NUM>. In some implementations, the secondary vehicle control system may be capable of fully operating autonomous vehicle <NUM> in the event of an adverse event in vehicle control system <NUM>, while in other implementations, the secondary vehicle control system may only have limited functionality, e.g., to perform a controlled stop of vehicle <NUM> in response to an adverse event detected in primary vehicle control system <NUM>. In still other implementations, the secondary vehicle control system may be omitted.

In general, an innumerable number of different architectures, including various combinations of software, hardware, circuit logic, sensors, networks, etc. may be used to implement the various components illustrated in <FIG>. Each processor may be implemented, for example, as a microprocessor and each memory may represent the random access memory (RAM) devices comprising a main storage, as well as any supplemental levels of memory, e.g., cache memories, non-volatile or backup memories (e.g., programmable or flash memories), read-only memories, etc. In addition, each memory may be considered to include memory storage physically located elsewhere in vehicle <NUM>, e.g., any cache memory in a processor, as well as any storage capacity used as a virtual memory, e.g., as stored on a mass storage device or on another computer or controller. One or more processors illustrated in <FIG>, or entirely separate processors, may be used to implement additional functionality in vehicle <NUM> outside of the purposes of autonomous control, e.g., to control entertainment systems, to operate doors, lights, convenience features, etc..

In addition, for additional storage, vehicle <NUM> may also include one or more mass storage devices, e.g., a removable disk drive, a hard disk drive, a direct access storage device (DASD), an optical drive (e.g., a CD drive, a DVD drive, etc.), a solid state storage drive (SSD), network attached storage, a storage area network, and/or a tape drive, among others. Furthermore, vehicle <NUM> may include a user interface <NUM> to enable vehicle <NUM> to receive a number of inputs from and generate outputs for a user or operator, e.g., one or more displays, touchscreens, voice and/or gesture interfaces, buttons and other tactile controls, etc. Otherwise, user input may be received via another computer or electronic device, e.g., via an app on a mobile device or via a web interface.

Moreover, vehicle <NUM> may include one or more network interfaces, e.g., network interface <NUM>, suitable for communicating with one or more networks (e.g., a LAN, a WAN, a wireless network, and/or the Internet, among others) to permit the communication of information with other computers and electronic devices, including, for example, a central service, such as a cloud service, from which vehicle <NUM> receives environmental and other data for use in autonomous control thereof.

Each processor illustrated in <FIG>, as well as various additional controllers and subsystems disclosed herein, generally operates under the control of an operating system and executes or otherwise relies upon various computer software applications, components, programs, objects, modules, data structures, etc., as will be described in greater detail below. Moreover, various applications, components, programs, objects, modules, etc. may also execute on one or more processors in another computer coupled to vehicle <NUM> via network, e.g., in a distributed, cloud-based, or client-server computing environment, whereby the processing required to implement the functions of a computer program may be allocated to multiple computers and/or services over a network.

In general, the routines executed to implement the various implementations described herein, whether implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions, or even a subset thereof, will be referred to herein as "program code. " Program code typically comprises one or more instructions that are resident at various times in various memory and storage devices, and that, when read and executed by one or more processors, perform the steps necessary to execute steps or elements embodying the various aspects of the invention. Moreover, while implementations have and hereinafter will be described in the context of fully functioning computers and systems, it will be appreciated that the various implementations described herein are capable of being distributed as a program product in a variety of forms, and that implementations can be implemented regardless of the particular type of computer readable media used to actually carry out the distribution. Examples of computer readable media include tangible, non-transitory media such as volatile and non-volatile memory devices, floppy and other removable disks, solid state drives, hard disk drives, magnetic tape, and optical disks (e.g., CD-ROMs, DVDs, etc.), among others.

In addition, various program code described hereinafter may be identified based upon the application within which it is implemented in a specific implementation. However, it should be appreciated that any particular program nomenclature that follows is used merely for convenience, and thus the invention should not be limited to use solely in any specific application identified and/or implied by such nomenclature. Furthermore, given the typically endless number of manners in which computer programs may be organized into routines, procedures, methods, modules, objects, and the like, as well as the various manners in which program functionality may be allocated among various software layers that are resident within a typical computer (e.g., operating systems, libraries, API's, applications, applets, etc.), it should be appreciated that the invention is not limited to the specific organization and allocation of program functionality described herein.

Those skilled in the art, having the benefit of the present invention, will recognize that the exemplary environment illustrated in <FIG> is not intended to limit implementations disclosed herein. Indeed, those skilled in the art will recognize that other alternative hardware and/or software environments may be used without departing from the scope of appended claims.

Turning now to <FIG>, two implementations of FMCW LIDAR component <NUM> of <FIG> are schematically illustrated. <FIG> illustrates an FMCW LIDAR component 132A that includes a single receiver 132A3, whereas <FIG> illustrates an FMCW LIDAR monopulse component 132B that includes two receivers 132B3A and 132B3B.

In <FIG>, FMCW LIDAR component 132A includes a tunable laser 132A1 whose optical output is controlled by signal generator 132A2. Signal generator 132A2 drives the laser 132A1 such that the laser 132A1 generates electromagnetic output that is in the optical spectra, and that is encoded through modification of the frequency of the electromagnetic output. The signal generator 132A2 generates a signal to cause the waveform to be encoded. Various types of frequency encoded waveforms can be generated via the signal generator 132A2 and the laser 132A1.

<FIG> illustrates a transmitted beam 192A from the laser 132A1 that is encoded as described in the preceding paragraph, and that is directed toward a point on a target object <NUM>. The transmitted beam 192A is reflected off the point on the target object <NUM> and the reflection 193A is received at the receiver 132A3 during a sensing event. The transmitted beam 192A includes a slice of the encoded waveform and the reflection 193A also includes the slice of the encoded waveform. The slice of the encoded waveform is detected by the receiver 132A3 during the sensing event.

The receiver 132A3 can be a coherent receiver and the reflection 193A can be detected by the receiver 132A3 using a coherent-detection scheme in which the reflection 193A is mixed with a local optical signal, such as a local optical signal that is a split portion of the transmitted beam 192A. It is noted that the detection of the reflection 193A by the receiver 132A3 will be time delayed with respect to detection of the local optical signal (e.g., the split portion of the transmitted beam 192A). The coherent-detection scheme can be implemented by the processor 132A4 and can be, for example, homodyne mixing, heterodyne mixing, self-heterodyne mixing, or other coherent-detection scheme. The resulting signal generated from mixing the reflected portion with the local optical signal indicates both: (<NUM>) range to the point of the target object <NUM>, and (<NUM>) velocity of the point of the target object <NUM>. In some implementations, the processor 132A4 provides the resulting signal as FMCW LIDAR data for the point from the FMCW LIDAR component 132A. In some other implementations, the processor 132A4 performs further processing to generate FMCW LIDAR data for the point that directly indicates the range and the velocity of the point. For example, the range and the velocity of the point can be directly indicated in FMCW LIDAR data that is a 3D point cloud that indicates range and velocity values, or a range-Doppler image. Through detections over multiple sensing events, where a direction of the optical output from the laser 132A1 is varied (e.g., in Azimuth and/or Elevation), FMCW LIDAR data is generated by the FMCW LIDAR component for each of a plurality of points in the environment - and indicates range, velocity, and optionally intensity, for each of the points in the environment.

Although the description of <FIG> above describes an FMCW LIDAR component, it is noted that a phase coherent LIDAR component that is not FMCW can have the same and/or similar components. However, with the non-FMCW LIDAR component, the signal generator 132A2 can drive the laser 132A1 to generate a waveform that is non-continuous and/or that is encoded using a technique other than frequency modulation.

In <FIG>, FMCW LIDAR monopulse component 132B includes a tunable laser 132B1 whose optical output is controlled by signal generator 132B2. Signal generator 132B2 drives the laser 132B1 such that the laser 132B1 generates electromagnetic output that is in the optical spectra, and that is encoded through modification of the frequency of the electromagnetic output. The signal generator 132B2 generates a signal to cause the waveform to be encoded. Various types of encoded waveforms can be generated via the signal generator 132B2 and the laser 132B1.

<FIG> illustrates a transmitted beam 192B from the laser 132B1 that is encoded as described in the preceding paragraph, and that is directed toward a point on a target object <NUM>. The transmitted beam 192B is reflected off the point on the target object <NUM> and, during a sensing event, a first reflection 193B1 is received at the first receiver 132B3A and a second reflection 193B2 is received at the second receiver 132B3B. The transmitted beam 192B includes a slice of the encoded waveform and the reflections 193B1 and 193B2 also include the slice of the encoded waveform. The first and second receivers 132B3A and 132B3B can be coherent receivers and the sensing event can include a first receiver sensing event and a second receiver sensing event. The first receiver sensing event can include coherent mixing of a local oscillator (e.g., a split portion of the transmitted beam 192B) with a first detection of the first reflection 193B1 and the second receiver sensing event can include coherent mixing of the local oscillator with a second detection of the second reflection 193B2. The first and second receiver 132B3A and 132B3B are positionally offset relative to one another, thereby causing the first and second detections to vary. This enables the processor 132B4 to determine at least one extra channel of data for the sensing event based on comparison of the first receiver sensing event and the second receiver sensing event. For example, the processor 132B4 can determine the extra channel based on amplitude and/or phase differences between the first and second receiver sensing events (e.g., differences in the time varying IF generated by each of the first and second receivers in mixing with the local oscillator signal). For instance, the amplitude and/or phase differences can be compared to determine an angle (a particular angle or range of angles), for each of one or more point(s) determined based on the sensing event, where each angle is resolved so that the point(s) are smaller than the beam width of the reflection that resulted in the sensing event. Accordingly, the angle (e.g., azimuth and elevation relative to the receiver(s)) for a corresponding range can be determined to a sub-beam width level. Moreover, the point can be localized to a sub-beam width level as the position of the point is defined by the angle and the range. Accordingly, increased angular accuracy can be achieved while reducing the signal-to-noise ratio (SNR) (i.e., increasing the noise) through splitting of the reflection across two receivers. This can be beneficial in determining ranges and velocities for multiple points of an additional vehicle that is far away (from the FMCW LIDAR monopulse component) and/or small. Additionally or alternatively, and as described herein, the processor 132B4 can determine LIDAR data with super-resolution using non-redundant data from the first and second detections. For example, the processors 132B4 can construct super-resolution LIDAR data from lower-resolution LIDAR data from the first and second detections. Accordingly, super-resolution range and velocity can be determined, while reducing the SNR through the splitting of the reflection across two receivers.

In some implementations, the processor 132B4 provides the signal, generated from the first and second receiver sensing events, as FMCW LIDAR monopulse data for the points from the FMCW LIDAR monopulse component 132B. In some other implementations, the processor 132B4 performs further processing to generate FMCW LIDAR monopulse data for the points that directly indicates the range and the velocity of the points. For example, the range and the velocity of the points can be directly indicated in FMCW LIDAR monopulse data that is a 3D point cloud or a range-Doppler image. Through detections over multiple sensing events where a direction of the optical output from the laser 132B1 is varied (e.g., in Azimuth and/or Elevation), FMCW LIDAR monopulse data is generated by the FMCW LIDAR monopulse component 132B for each of a plurality of points in the environment- and indicates range, velocity, and optionally intensity, for each of the points in the environment.

Although the description of <FIG> above describes an FMCW LIDAR monopulse component, it is noted that a phase coherent LIDAR monopulse component that is not FMCW can have the same and/or similar components. However, with the non-FMCW LIDAR component, the signal generator 132B2 can drive the laser 132B1 to generate a waveform that is non-continuous and/or that is encoded using a technique other than frequency modulation.

Turning now to <FIG>, a block diagram is provided that illustrates implementations of determining classification(s) for object(s) in an environment of a vehicle, and controlling the vehicle based on the classification(s). <FIG> illustrates subsystems <NUM>, <NUM>, <NUM>, and <NUM> of vehicle control system <NUM>, and examples of how they may interact in determining classification(s) for object(s) in an environment of the vehicle, determining pose(s) for the object(s), determining movement parameter(s) (if any) for the object(s), and/or adapting autonomous control based on the determinations.

Localization subsystem <NUM> is generally responsible for localizing a vehicle within its environment. Localization subsystem <NUM>, for example, may determine a pose (generally a position, and in some instances orientation) of the autonomous vehicle <NUM> within its surrounding environment. The localization subsystem <NUM> can determine the pose of the autonomous vehicle <NUM> utilizing FMCW LIDAR data <NUM> from FMCW LIDAR component <NUM> and/or data from other sensor(s) <NUM>, such as camera <NUM>, RADAR component <NUM>, LIDAR component <NUM>, and/or other sensors <NUM> of <FIG> (e.g., IMU <NUM>, encoders <NUM>). The pose determined by the localization subsystem <NUM> can be, for example, relative to a map utilized by the vehicle control system <NUM>. The pose can be updated by the localization subsystem <NUM> as the vehicle <NUM> moves throughout its environment, and the pose provided to (or otherwise made accessible to) one or more of the other subsystems <NUM>, <NUM>, and/or <NUM> for use by the subsystem(s).

Perception subsystem <NUM> is principally responsible for perceiving objects within the environment of the vehicle, and can utilize FMCW LIDAR data <NUM> and optionally data from other sensor(s) <NUM> (e.g., camera <NUM>, RADAR component <NUM>) to detect those objects, classify those objects, determine velocities for those objects, and track those objects.

Perception subsystem <NUM> can include an object detection and classification module 154A that utilizes FMCW LIDAR data <NUM> and/or data from other sensor(s) <NUM> in detecting various objects (e.g., determining the pose of the objects) and classifying the detected objects. For example, the object detection and classification module 154A can process FMCW LIDAR data <NUM> and/or data from other sensor(s) <NUM> in detecting and classifying additional vehicles, pedestrians, animals, trees, signs, bicyclists, and/or other object(s) in the environment of the vehicle <NUM>.

In various implementations, the object detection and classification module 154A can utilize one or more classification models <NUM> in detecting and/or classifying various objects. For example, the classification model(s) can include one or more trained machine learning model(s) such as Bayesian models, random forest models, Markov models, and/or neural network models. For instance, the trained machine learning model(s) can include a neural network model that is trained for utilization in processing FMCW LIDAR data (and/or other data) to generate output that indicates, for each of one or more objects captured by the data, a corresponding pose of the object and corresponding classification(s) of the object. Some non-limiting examples of such neural network models are described in more detail herein (e.g., with reference to <FIG>, <FIG>). In some implementations, in detecting and/or classifying various objects, the object detection and classification module 154A can additionally or alternatively process FMCW LIDAR data <NUM> and/or data from other sensor(s) <NUM> without utilization of machine learning model(s). For example, nearest neighbor and/or other clustering techniques can be utilized to determine portions of data that likely have a corresponding classification.

Regardless of the technique(s) utilized, object detection and classification module 154A can provide an indication of pose(s) and class(es) for object(s) <NUM> to a velocity module 154B of the perception subsystem <NUM>. The provided pose for each object can be, for example, relative to a map utilized by the vehicle control system <NUM> and/or relative to the FMCW LIDAR data <NUM>. The velocity module 154B uses the FMCW LIDAR data <NUM> to determine an instantaneous velocity for each of one or more of the objects <NUM>. In some implementations, the velocity module 154B determines an instantaneous velocity for only those object(s) having certain classification(s) (as indicated by <NUM>) and/or excludes certain classification(s) of objects from instantaneous velocity determination. For example, the velocity module 154B can exclude object(s) having certain classification(s) indicative of non-moving objects, such as a tree classification. Also, for example, the velocity module 154B can exclude object(s) having certain classification(s) indicative of certain relatively slow moving objects, such as a pedestrian classification. As yet another example, the velocity module 154B can determine an instantaneous velocity for object(s) having vehicle, bicyclist, and/or other classification(s) indicative of objects that can be moving relatively fast.

In determining an instantaneous velocity for an object, the velocity module 154B can use the provided pose for that object (provided in <NUM>) to determine a subgroup of FMCW LIDAR data <NUM> that corresponds to the pose (and resultantly, the object). Further, the velocity module 154B can determine the instantaneous velocity as a function of the velocities (including directions and magnitudes) of the FMCW LIDAR data of the subgroup. For example, the velocity module 154B can determine the instantaneous velocity based on a mean, median, and/or other statistical measure of the velocities. As another example, the velocity module 154B can additionally or alternatively distribute velocities of the FMCW LIDAR data of the subgroup into bins (where each bin corresponds to a range of velocities), and determine the instantaneous velocity based on the bin(s) with the largest quantity of occurrences of velocities that conform to the bin(s) (e.g., a histogram analysis). Additional and/or alternative techniques can be utilized.

In some implementations, the velocity module 154B can select a particular instantaneous velocity determination technique for an object, from a plurality of candidate techniques, based on the classification for the object. For example, for an object classification of pedestrian, the instantaneous velocity can be determined based on a mean of the FMCW LIDAR data of the subgroup that corresponds to the pedestrian, whereas an average can instead be utilized for an object classification of a vehicle. As another example, for an object classification of vehicle, the instantaneous velocity can be determined using a histogram analysis, whereas a histogram analysis is not used for an object classification of a pedestrian. In some implementations, the velocity module 154B can additionally or alternatively select a particular instantaneous velocity determination technique for an object, from a plurality of candidate techniques, based on the pose of the object and/or the quantity of FMCW LIDAR data of the subgroup that corresponds to the object. For example, when the pose for an object indicates the object is relatively far away from the vehicle, the velocity module 154B can utilize a technique that is more computationally efficient (but potentially less robust and/or less accurate) than an alternative technique that is instead utilized when the pose indicates the object is relatively close. As another example, when the quantity of FMCW LIDAR data of the subgroup that corresponds to the object is relatively small (e.g., due to the object being far away), the velocity module 154B can utilize a first technique (e.g., average), whereas when the quantity of FMCW LIDAR data of the subgroup is relatively large, the velocity module 154B can utilize a second technique (e.g., histogram analysis).

The perception subsystem <NUM> provides the instantaneous velocity for each of one or more of the objects <NUM>, and the pose(s) and class(es) for the object(s) <NUM> to the planning subsystem <NUM> (or otherwise makes it available to the planning subsystem <NUM>). The planning subsystem <NUM> is principally responsible for planning out a trajectory of the vehicle over some time frame (e.g., several seconds), and may receive input from both localization subsystem <NUM> and perception subsystem <NUM>. The planning subsystem <NUM> can adapt the trajectory of the vehicle based on the provided velocities, poses, and/or classes for the objects. For example, if the data <NUM> and <NUM> indicate, for a given object that is an additional vehicle: a classification of vehicle, a pose for the additional vehicle, and an instantaneous velocity for the additional vehicle - the planning subsystem <NUM> can utilize the pose and the instantaneous velocity for the vehicle in determining one or more candidate trajectories for the additional vehicle. The planning subsystem <NUM> can utilize the instantaneous velocity alone in determining the one or more candidate trajectories for the additional vehicle, or can also utilize one or more previously determined velocities for the additional vehicle. For example, the planning subsystem <NUM> can utilize the instantaneous velocity, as well as one or more immediately preceding determined instantaneous velocities for the additional vehicle (each determined based on FMCW LIDAR data from a prior sensing cycle). Further, the planning subsystem <NUM> can adapt the intended trajectory of the ego vehicle <NUM> so that the ego vehicle <NUM> does not contact and/or come within a threshold distance of the additional vehicle if the additional vehicle traverses one of the candidate trajectories.

As another example, if the data <NUM> and <NUM> further indicate, for a given object that is a pedestrian, a classification of pedestrian, and a pose for the pedestrian (but no velocity for the pedestrian), the planning subsystem <NUM> can utilize the pose and the classification of pedestrian to determine a candidate trajectory area around the pedestrian, where the candidate trajectory area is an area around the pose for the pedestrian that is determined based on a maximum distance a pedestrian could travel before the next object detection event, and/or determined based on a default maximum velocity for an object having a pedestrian classification. Further, the planning subsystem can determine a trajectory of the ego vehicle that avoids the candidate trajectory area around the pose for the pedestrian. As yet another example, if the data <NUM> and <NUM> further indicate, for a given object that is a tree, a classification of tree, and a pose for the tree (but no velocity for the tree), the planning subsystem <NUM> can utilize the pose and the classification of tree to determine a trajectory of the ego vehicle that avoids the pose of the tree. Additional description of these and other techniques for controlling a vehicle based on determined classifications, poses, and velocities of objects are provided herein. For example, additional description of some techniques is provided below in association with description of <FIG> and <FIG>.

The trajectory and/or other data generated by the planning subsystem <NUM> is then passed on to control subsystem <NUM> to convert the desired trajectory into trajectory commands <NUM> suitable for controlling the various vehicle controls <NUM>, <NUM>, and/or <NUM> in control system <NUM>, with localization data also provided to control subsystem <NUM> to enable the control subsystem <NUM> to issue appropriate commands to implement the desired trajectory as the location of the vehicle changes over the time frame.

<FIG> is a block diagram illustrating one example of how pose(s) and class(es) for object(s) <NUM> of <FIG> can be determined utilizing a classification model 170A. Classification model 170A is one example of classification model(s) <NUM> of <FIG>. In <FIG>, the object detection and classification module 154A applies FMCW LIDAR data <NUM> as input to classification model 170A, processes the FMCW LIDAR data <NUM> using the classification model 170A, and generates output 393A based on the processing. In some implementations, the FMCW LIDAR data <NUM> is applied in the same form as it is received from the FMCW LIDAR component <NUM>. In some other implementations, pre-processing of the FMCW LIDAR data <NUM> is performed prior to applying it as input to the classification model 170A. For example, the FMCW LIDAR data <NUM> can be pre-processed by decreasing its resolution to have it conform to input dimensions of the classification model 170A, performing principal component analysis (PCA), padding the data to have it conform to input dimensions of the classification model 170A, and/or otherwise generating an altered version of the FMCW LIDAR data <NUM> before applying it as input. It is understood that the pre-processing performed, if any, will be dependent on the classification model 170A (e.g., its input dimensions and the input data based on which it was trained). Also, in some implementations the FMCW LIDAR data <NUM> is from only a most recent sensing cycle of the FMCW LIDAR component <NUM>. In some other implementations, the FMCW LIDAR data <NUM> can include FMCW LIDAR data from a most recent sensing cycle of the FMCW LIDAR component <NUM>, as well FMCW LIDAR data from one or more prior sensing cycles of the FMCW LIDAR component <NUM> (e.g., from immediately preceding sensing cycles).

The output 393A generated based on the processing using the classification model 170A includes, for each of a plurality of spatial regions<NUM>-N, a corresponding probability for each of classes<NUM>-N. Each of the spatial regions<NUM>-N of the output 393A corresponds to a portion of the FMCW LIDAR data <NUM> and, as a result, corresponds to an area of the environment. For example, the FMCW LIDAR data <NUM> can include a 3D point cloud defining a corresponding range (via its 3D coordinate in the 3D point cloud), corresponding velocity, and optionally a corresponding intensity. In such an example, each of the spatial regions<NUM>-N can correspond to one or more of the 3D points (e.g., a single point or a collection of neighboring points). For instance, each of the spatial regions<NUM>-N can correspond to one of the 3D points of the FMCW LIDAR data <NUM>. As an alternative instance, each of the spatial regions <NUM>-N can correspond to a collection of neighboring 3D points of the FMCW LIDAR data <NUM> (e.g., each spatial region can correspond to a collection of eight of the 3D points, thereby enabling the quantity of spatial regions to be one eighth of the quantity of 3D points). As another example, where the FMCW LIDAR data <NUM> includes a range-Doppler image, each of the spatial regions<NUM>-N can correspond to one or more pixels (e.g., a single pixel or a collection of neighboring pixels) of the range-Doppler image.

Each of the probabilities of the output 393A for each of the spatial regions<NUM>-N corresponds to one of <NUM>-N classes. For example, the class<NUM> probability can be a probability of a vehicle class, the class<NUM> probability can be a probability of a pedestrian class, the class<NUM> probability can be a probability of a bicyclist class, etc. It is noted that while N is utilized to reference the total quantity of spatial regions<NUM>-N, and to reference the total quantity of classes<NUM>-N in output 393A, the total quantity of spatial regions<NUM>-N and the total quantity of classes<NUM>-N can vary from one another in various implementations. As one non-limiting example, over a thousand spatial regions can be provided, whereas only fifty or less classes can be provided.

The object detection and classification module 154A can utilize the output 393A to determine pose(s) and class(es) for object(s) <NUM>. For example, the object detection and classification module 154A can determine that no objects are present in those spatial regions<NUM>-N whose class<NUM>-N probabilities each fail to satisfy a threshold low probability (e.g., <NUM>, <NUM>, or other probability). Also, for example, the object detection and classification module 154A can determine those spatial regions<NUM>-N having one or more of the class<NUM>-N probabilities that satisfy a threshold high probability likely contain an object. Further, the object detection and classification module 154A can determine clusters of neighboring (directly neighboring or within N (e.g., <NUM>, <NUM>, or <NUM>) spatial regions) spatial regions that have probabilities satisfying a threshold for the same class(es) and determine they correspond to the same object of those class(es). For example, if spatial regions<NUM>-<NUM> are neighboring and all have a probability for class<NUM> that satisfies a threshold, the object detection and classification module 154A can determine spatial regions<NUM>-<NUM> correspond to a single object of class<NUM>. Yet further, the object detection and classification module 154A can determine the pose of that single object based on those spatial regions<NUM>-<NUM>. For example, the module 154A can determine the object has a pose corresponding to the FMCW LIDAR data <NUM> that corresponds to spatial regions<NUM>-<NUM>. The determined pose can be with reference to the FMCW LIDAR data <NUM>, or with reference to another frame (e.g., a world frame or other reference frame).

<FIG> is a block diagram illustrating another example of how pose(s) and class(es) for object(s) <NUM> of <FIG> can be determined utilizing a collection of classification models 170B1-170BN. Classification models 170B1-170BN are one example of classification model(s) <NUM> of <FIG>. In <FIG>, classification model 170A is trained to generate output 393A that provides, for each of a plurality of spatial regions<NUM>-N, a corresponding probability for each of classes<NUM>-N. In contrast, in <FIG>, each of the classification models 170B1-170BN is trained to generate corresponding output that provides, for each of a plurality of spatial regions<NUM>-N, a corresponding probability for only one corresponding class.

In <FIG>, the object detection and classification module 154A applies FMCW LIDAR data <NUM> as input to each of classification models 170B1-170BN, processes the FMCW LIDAR data <NUM> using models 170B1-170BN, and generates outputs 393B1-393BN based on the processing. In some implementations, the FMCW LIDAR data <NUM> is applied in the same form as it is received from the FMCW LIDAR component <NUM>. In some other implementations, pre-processing of the FMCW LIDAR data <NUM> is performed prior to applying it as input.

The output 393B1 generated based on the processing using the classification model 170B1 includes, for each of a plurality of spatial regions<NUM>-N, a corresponding probability for class<NUM>. The output 393BN generated based on the processing using the classification model 170BN includes, for each of a plurality of spatial regions<NUM>-N, a corresponding probability for classN. The unillustrated outputs generated based on the processing using the classification models 170B2-170B(N-<NUM>) each includes, for each of the plurality of spatial regions<NUM>-N, a corresponding probability for a corresponding single class. Each of the spatial regions<NUM>-N of the outputs 393B1-393BN corresponds to a portion of the FMCW LIDAR data <NUM> and, as a result, corresponds to an area of the environment.

The object detection and classification module 154A can utilize the outputs 393B1-393BN to determine pose(s) and class(es) for object(s) <NUM>. For example, the object detection and classification module 154A can determine that no objects are present in those spatial regions<NUM>-N whose class<NUM>-N probabilities indicated by outputs 393BA-393BN each fail to satisfy a threshold low probability (e.g., <NUM>, <NUM>, or other probability). Also, for example, the object detection and classification module 154A can determine those spatial regions<NUM>-N having one or more of the class<NUM>-N probabilities indicated by outputs 393BA-393BN that satisfy a threshold high probability likely contain an object of those class(es). Further, the object detection and classification module 154A can determine clusters of neighboring (directly neighboring or within N (e.g., <NUM>, <NUM>, or <NUM>) spatial regions) spatial regions that have probabilities satisfying a threshold for the same class(es) and determine they correspond to the same object of that class. Yet further, the object detection and classification module 154A can determine the pose of that single object based on those spatial regions.

Turning now to <FIG>, an example is illustrated of a training engine <NUM> training the classification model 170A of <FIG> utilizing a training instance 467A of training instances database <NUM>. The training engine <NUM> can train the classification model 170A prior to its usage in <FIG>. The training instance 467A includes a training instance input 468A that includes FMCW LIDAR data (e.g., FMCW LIDAR data from a sensing cycle, or from a plurality of sequential sensing cycles). The training instance 467A also includes a training instance output 469A, that defines, for each of a plurality of spatial regions, whether an object of one or more classes is present in the spatial region. Training instance output 469A is illustrated more particularly in <FIG>. Training instance output 469A defines a plurality of spatial regions<NUM>-N, and, for each of the spatial regions<NUM>-N, a corresponding probability for each of classes<NUM>-N. For example, if only one object having only class<NUM> is present in spatial regions<NUM>-<NUM>, the class<NUM> probability for each of spatial regions<NUM>-<NUM> can be "<NUM>", the probabilities for each of classes<NUM>-N can be "<NUM>" for spatial regions<NUM>-<NUM>, and the probabilities for each of classes<NUM>-N can be "<NUM>" for spatial regions<NUM>-N. The training instance output 469A can, in some implementations, be based on manual annotations of FMCW LIDAR data that indicate classes of objects present in the FMCW LIDAR data.

Turning again to <FIG>, in training the classification model 170A, the training engine <NUM> applies the training instance input 468A to the classification model 170A, and processes the training instance input 468A, utilizing the classification model 170A and based on current parameters of the classification model 170A, to generate an output having a dimension that conforms to the dimension of training instance output 469A. The training engine <NUM> then compares the generated output to the training instance output 469A, and updates one or more parameters of the classification model 170A based on the comparison. For example, the training engine <NUM> can generate an error based on differences between the generated output and the training instance output 469A, and backpropagate a loss (that is based on the error) over classification model 170A to update the classification model 170A. Although only a single training instance 467A is illustrated in <FIG>, classification model 170A will be trained based on a large quantity of training instances of training instances database <NUM>. Those training instances can collectively include training instance inputs with diverse FMCW LIDAR data and diverse training instance outputs. Moreover, although non-batch training is described with respect to <FIG>, batch training may additionally or alternatively be utilized (e.g., where losses are based on errors determined based on a batch of training instances).

<FIG> also illustrates, within dashed lines, example layers that may be included in classification model 170A. In particular, an input layer 170A1, convolution layer(s) 170A2, other layer(s) 170A3, and an output layer 170A4 are illustrated. As described herein, the dimensions/shape of the input layer 170A1 can be dependent on the shape of the FMCW LIDAR data to be applied. Also, the dimensions/shape of the output layer 170A4 can be dependent on the size of the spatial regions, how many class probabilities are to be predicted, etc. In some implementations, multiple convolution layer(s) 170A2 are provided, and max pooling and/or other layers may optionally be interposed between one or more of the convolution layer(s) 170A2. The other layer(s) 170A3 can include such max pooling layers and/or affine layers such as a softmax layer(s) and/or fully connected layer(s) that precede the output layer 170A4. Although <FIG> illustrates example layers that may be included in classification model 170A, additional and/or alternative layers can be included in other implementations. For example, other implementations may not include any convolution layer(s) and/or not include any max pooling layers. For instance, they can instead include alternative layer(s) between the input layer 170A1 and the output layer 170A4.

Turning now to <FIG>, an example environment of a vehicle is illustrated. The environment includes multiple environmental objects <NUM>-<NUM>, and also illustrates examples of some FMCW LIDAR data points that can correspond to the environmental objects. The example FMCW LIDAR data points are illustrated as plus signs (+), minus signs (-), and circles (o). The FMCW LIDAR data points can be, for example, from an instance of (e.g., a sensing cycle of) FMCW LIDAR data that is a 3D point cloud, where each of the points of the 3D point cloud indicates a corresponding range (through its position in the 3D point cloud), and includes a corresponding rate/velocity, and optionally a corresponding intensity value. The plus signs (+) indicate FMCW LIDAR data points that have a positive corresponding velocity value, the minus signs (-) indicate FMCW LIDAR data points that have a negative corresponding velocity value, and the circles (o) indicate FMCW LIDAR data points that have a zero (or near-zero) corresponding velocity value. It is noted that although positive velocity values are all indicated by plus signs (+), and negative velocity values are all indicated by minus signs (-), the magnitudes of those velocity values can vary (e.g., velocity values for the vehicle <NUM> can be greater than those for the pedestrian <NUM>). Also, it is understood that additional FMCW LIDAR data points will be provided with the FMCW LIDAR data, including additional points that correspond to the objects <NUM>-<NUM> - as well as additional points that correspond to additional environmental objects. However, only some of the FMCW LIDAR data points are illustrated in <FIG> for simplicity. Moreover, it is understood that the objects <NUM>-<NUM> will not be rendered in the FMCW LIDAR data as they appear in <FIG> (although they may be visually ascertainable via corresponding FMCW LIDAR data points). However, the objects <NUM>-<NUM> are illustrated in <FIG> for reference purposes. Additionally, it is understood that while <FIG> is described with respect to FMCW LIDAR data points, additional or alternative types of FMCW LIDAR data can be utilized in determining poses, classifications, and movement parameters, such as those types of FMCW LIDAR data also described herein.

In <FIG>, bounding shapes <NUM>-<NUM> are each depicted around a corresponding one of the objects <NUM>-<NUM>. The bounding shapes <NUM>-<NUM> each indicate a subgroup of FMCW LIDAR data that may correspond to a corresponding one of the objects <NUM>-<NUM> and, as a result, each indicate a pose of a corresponding one of the objects <NUM>-<NUM>. Each pose can be a multidimensional pose, such as a three-dimensional (3D) pose that indicates position, or a six-dimensional (6D) pose that indicates position as well as orientation. As described herein, the object detection and classification module 154A can determine the bounding shapes <NUM>-<NUM> based on processing of the FMCW LIDAR data utilizing a classification model <NUM>, and can further determine, utilizing the classification model <NUM>, a corresponding classification for each of the objects <NUM>-<NUM> encompassed by the bounding shapes <NUM>-<NUM>. Although particular bounding shapes are illustrated, it is understood that other bounding shapes can additionally or alternatively be determined and utilized.

In <FIG>, bounding shape <NUM> encompasses FMCW LIDAR data points that correspond to an additional vehicle <NUM>. Each of the FMCW LIDAR data points within the bounding shape <NUM> corresponds to the additional vehicle <NUM> and is indicated with a plus sign (+), which means it has a positive corresponding velocity value (relative to the FMCW LIDAR component <NUM> of the vehicle <NUM>). The object detection and classification module 154A can determine the bounding shape <NUM> based on processing the FMCW LIDAR data over a classification model <NUM>, to generate output that indicates neighboring spatial regions that each have a classification probability for vehicle that satisfies a threshold. The neighboring spatial regions can conform to the bounding shape <NUM>, and they indicate the pose for the additional vehicle <NUM>. Further, the velocity module 154B can determine an instantaneous velocity for the additional vehicle <NUM> based on the FMCW LIDAR data points encompassed by the bounding shape <NUM>. For example, the velocity module 154B can determine the instantaneous velocity based on a mean of the velocities indicated by the FMCW LIDAR data encompassed by the bounding shape <NUM>. As described, in some implementations the velocity module 154B can determine the instantaneous velocity utilizing a technique that is selected based on the object detection and classification module 154A determining the classification of vehicle.

In <FIG>, bounding shape <NUM> encompasses FMCW LIDAR data points that correspond to a tree <NUM>. Each of the FMCW LIDAR data points within the bounding shape <NUM> corresponds to the tree <NUM>. As illustrated, some of the FMCW LIDAR data points that correspond to the branches and/or foliage of the tree <NUM> are indicated with a plus sign (+), while others are indicated with a minus sign (-). This can be due to the branches/foliage moving in different directions (e.g., due to wind) and having different directions of velocities. Some of the FMCW LIDAR data points that correspond to the trunk of the tree <NUM> are indicated with a circle (<NUM>), which means they have a zero (or near-zero) velocity. The object detection and classification module 154A can determine the bounding shape <NUM> based on processing the FMCW LIDAR data over the classification model <NUM>, to generate output that indicates neighboring spatial regions that each have a classification probability for "tree" that satisfies a threshold. The neighboring spatial regions can conform to the bounding shape <NUM>, and they indicate the pose for the tree. In some implementations, the velocity module 154B can skip any determination of an instantaneous velocity for the tree <NUM> based on its classification as determined by the object detection and classification module 154A.

In <FIG>, bounding shape <NUM> encompasses FMCW LIDAR data points that correspond to a pedestrian <NUM>. Each of the FMCW LIDAR data points within the bounding shape <NUM> corresponds to the pedestrian <NUM>. Some of those are indicated with a plus sign (+), (which means they have a positive corresponding velocity value), while others are indicated with a minus sign (-) (which means they have a negative corresponding velocity value), while yet others are indicated with a circle (o) (which means they have a zero or near-zero corresponding velocity value). These differing velocity directions on the pedestrian <NUM> can be due to the pedestrian walking during the sensing cycle. The object detection and classification module 154A can determine the bounding shape <NUM> based on processing the FMCW LIDAR data over a classification model <NUM>, to generate output that indicates neighboring spatial regions that each have a classification probability for "pedestrian" that satisfies a threshold. The neighboring spatial regions can conform to the bounding shape <NUM>, and they indicate the pose for the pedestrian <NUM>. Further, the velocity module 154B can optionally determine an instantaneous velocity for the pedestrian <NUM> based on the FMCW LIDAR data points encompassed by the bounding shape <NUM>. For example, the velocity module 154B can determine the instantaneous velocity based on a median of the velocities indicated by the FMCW LIDAR data encompassed by the bounding shape <NUM>. As described, in some implementations the velocity module 154B can determine the instantaneous velocity utilizing a technique that is selected based on the object detection and classification module 154A determining the classification of "pedestrian".

In <FIG>, bounding shape <NUM> encompasses FMCW LIDAR data points that correspond to a bicyclist <NUM>. Each of the FMCW LIDAR data points within the bounding shape <NUM> corresponds to the bicyclist <NUM>. Some of those are indicated with a plus sign (+), (which means they have a positive corresponding velocity value), while others are indicated with a minus sign (-). The differing velocity directions on the wheels of the bicycle of the bicyclist <NUM> can be due to the wheels rotation of the wheels. The differing velocity directions between the body of the bicyclist <NUM> and one of the legs of the bicyclist <NUM> can be due to the one leg moving in an opposite direction due to pedaling of the bicyclist during the sensing cycle. The object detection and classification module 154A can determine the bounding shape <NUM> based on processing the FMCW LIDAR data over a classification model <NUM>, to generate output that indicates neighboring spatial regions that each have a classification probability for "bicyclist" that satisfies a threshold. The neighboring spatial regions can conform to the bounding shape <NUM>, and they indicate the pose for the bicyclist <NUM>. Further, the velocity module 154B can optionally determine an instantaneous velocity for the bicyclist <NUM> based on the FMCW LIDAR data points encompassed by the bounding shape <NUM>. For example, the velocity module 154B can determine the instantaneous velocity based on histogram analysis of the velocities indicated by the FMCW LIDAR data encompassed by the bounding shape <NUM>. As described, in some implementations the velocity module 154B can determine the instantaneous velocity utilizing a technique that is selected based on the object detection and classification module 154A determining the classification of "bicyclist".

In <FIG>, another example environment of a vehicle is illustrated. The environment includes an additional vehicle <NUM>, and also illustrates three FMCW LIDAR data points 682A, 682B, and 682C that correspond to the additional vehicle <NUM>. The FMCW LIDAR data points can be, for example, from an instance of (e.g., a sensing cycle of) FMCW LIDAR data that is a 3D point cloud, where each of the points of the 3D point cloud indicates a corresponding range (through its position in the 3D point cloud), and includes a corresponding rate/velocity, and optionally a corresponding intensity value. It is understood that additional FMCW LIDAR data points will be provided with the FMCW LIDAR data, such as additional points that correspond to additional environmental objects. However, only some of the FMCW LIDAR data points are illustrated in <FIG> for simplicity. Moreover, it is understood that additional vehicle <NUM> will not be rendered in the FMCW LIDAR data as it appears in <FIG>, but is illustrated in <FIG> for reference purposes. Additionally, it is understood that while <FIG> is described with respect to FMCW LIDAR data points, additional or alternative types of FMCW LIDAR data can be utilized in determining poses, classifications, and movement parameters, such as those types of FMCW LIDAR data also described herein.

In <FIG>, the additional vehicle <NUM> is far away from the FMCW LIDAR component. The FMCW LIDAR data points 682A, 682B, and 682C in <FIG> can be FMCW LIDAR monopulse data points generated by an FMCW LIDAR monopulse component. The example of <FIG> is one example of where FMCW LIDAR monopulse data points can be particularly beneficial. Namely, it illustrates a situation where the increased angular accuracy of the FMCW LIDAR monopulse component enables three FMCW LIDAR data points 682A, 682B, and 682C to be determined, whereas fewer (e.g., two, one, or none) may have been determinable with a FMCW LIDAR non-monopulse component. While <FIG> is one example, it is noted that FMCW LIDAR data points of <FIG> can also be FMCW LIDAR monopulse data points and can also be beneficial since, for example, it increases the angular accuracy of FMCW LIDAR data for the additional objects that can be utilized in determining poses, classifications, and movement parameters according to implementations disclosed herein. Such increased angular accuracy of FMCW LIDAR data can result in more accurate pose, classification, and/or movement parameter determinations in some implementations.

In <FIG>, bounding shape <NUM> is depicted around the additional vehicle <NUM>, and indicates a subgroup of FMCW LIDAR data that corresponds to the additional vehicle <NUM> and, as a result, indicates a pose of the additional vehicle <NUM>. The object detection and classification module 154A can determine the bounding shape <NUM> based on processing of the FMCW LIDAR data utilizing a classification model <NUM>, and can further determine, utilizing the classification model <NUM>, a classification of vehicle for the object encompassed by bounding shape <NUM>. Further, the velocity module 154B can determine an instantaneous velocity for the additional vehicle <NUM> based on the FMCW LIDAR data points 682A, 682B, and 682C. For example, the velocity module 154B can determine the instantaneous velocity based on a largest magnitude of the velocities indicated by the FMCW LIDAR points 682A, 682B, and 682C. In some implementations the velocity module 154B can determine the instantaneous velocity utilizing the largest magnitude based on the relatively low quantity of FMCW LIDAR data points corresponding to the additional vehicle <NUM>.

Poses, classifications, and/or velocities of environmental objects determined according to techniques described herein can be utilized in control of an ego vehicle. Some examples of such adaptation is described with respect to <FIG> and <FIG>, which each illustrate an example of how the vehicle <NUM> can be autonomously controlled based on such considerations.

<FIG> illustrates an example of how a candidate trajectory (if any) can be determined for each of a plurality of objects <NUM>, <NUM>, and <NUM> based on classifications of those objects and/or instantaneous velocities for those objects, and how autonomous control of the vehicle <NUM> can be adapted based on the candidate trajectories.

In <FIG>, a vehicle <NUM>, a tree <NUM>, and a bicyclist <NUM> are illustrated from a birds-eye view. The object detection and classification module 154A can determine corresponding classifications for the vehicle <NUM>, tree <NUM>, and bicyclist <NUM> utilizing techniques described herein. The object detection and classification module 154A can further determine corresponding poses for the vehicle <NUM>, tree <NUM>, and bicyclist <NUM> utilizing techniques described herein. Further, the velocity module 154B can determine an instantaneous velocity for the vehicle <NUM> and an instantaneous velocity for the bicyclist <NUM> utilizing techniques described herein.

The planning subsystem <NUM> can utilize the pose and the instantaneous velocity of the vehicle <NUM> to determine a candidate trajectory 791A of the additional vehicle <NUM>. Further, the planning subsystem <NUM> can utilize the pose and the instantaneous velocity of the bicyclist <NUM> to determine a candidate trajectory 793A of the bicyclist <NUM>. It is noted that no candidate trajectory is determined for the tree <NUM>, as it is of a classification that indicates a static/non-mobile object. Further, it is noted that in various implementations the velocity module 154B may, based on a determined classification of the tree <NUM>, not determine any velocity for the tree <NUM>.

The planning subsystem <NUM> can utilize at least the candidate trajectories 791A and 793A in determining a trajectory 101A for the ego vehicle <NUM>. The trajectory 101A can be determined and implemented over a period of time and represents a straight-line continuation of a current trajectory of the ego vehicle <NUM> since, as can be appreciated from viewing <FIG>, the candidate trajectories 791A and 793A will not cause the objects <NUM> or <NUM> to be too proximal to ego vehicle <NUM> to warrant adapting of the trajectory. Further, tree <NUM> will not alter the trajectory as it is not in the path of the ego vehicle <NUM> and its classification indicates that it is non-mobile.

<FIG> illustrates another example of how a candidate trajectory (if any) can be determined for each of a plurality of objects <NUM>, <NUM>, and <NUM> based on classifications of those objects and/or instantaneous velocities for those objects, and how autonomous control of the vehicle <NUM> can be adapted based on the candidate trajectories.

In <FIG>, a vehicle <NUM>, a tree <NUM>, and a pedestrian <NUM> are illustrated from a birds-eye view. The object detection and classification module 154A can determine corresponding classifications for the vehicle <NUM>, tree <NUM>, and pedestrian <NUM> utilizing techniques described herein. The object detection and classification module 154A can further determine corresponding poses for the vehicle <NUM>, tree <NUM>, and pedestrian <NUM> utilizing techniques described herein. Further, the velocity module 154B can determine an instantaneous velocity for the vehicle <NUM> utilizing techniques described herein. Optionally, the velocity module 154B does not determine an instantaneous velocity for the tree <NUM> or for the pedestrian <NUM>.

The planning subsystem <NUM> can utilize the pose and the instantaneous velocity of the vehicle <NUM> to determine a candidate trajectory 791A of the additional vehicle <NUM>. The trajectory 791A is the same as that illustrated in <FIG>. Further, the planning subsystem <NUM> can utilize the pose of the pedestrian <NUM>, and the classification of the pedestrian <NUM> as a pedestrian to determine a candidate trajectory 794B of the pedestrian <NUM>. The candidate trajectory 794B extends in an area around the pedestrian and can be based on, for example, a default maximum velocity for a pedestrian classification and/or a default maximum distance a pedestrian could traverse prior to a next sensing cycle (in which an updated pose of the pedestrian <NUM> could be determined). It is noted that no candidate trajectory is determined for the tree <NUM>, as it is of a classification that indicates a static/non-mobile object.

The planning subsystem <NUM> can utilize at least the candidate trajectories 791A and 794A in determining a trajectory 101B for the ego vehicle <NUM>. The trajectory 101B can be determined and implemented over a period of time, and represents a deceleration of the current trajectory of the ego vehicle <NUM> since, as can be appreciated from viewing <FIG>, the candidate trajectory 794A of pedestrian <NUM> may cause the pedestrian to be too proximal to ego vehicle <NUM>, unless the ego vehicle is decelerated. Further, deceleration of the ego vehicle <NUM> can be chosen in lieu of steering away from the pedestrian <NUM> in view of the candidate trajectory 791A (e.g., steering away may cause the vehicle to be too proximal to the candidate trajectory 791A of the additional vehicle <NUM>).

<FIG> and <FIG> each illustrates control of the ego vehicle <NUM> at a corresponding given instance based on poses, classifications, and/or movement parameters of environmental objects at the given instance. However, it is understood that control of the ego vehicle <NUM> will continue at each of a plurality of additional instances based on poses, classifications, and/or movement parameters of environmental objects at each of those additional instances, thereby enabling control of the ego vehicle <NUM> to be continuously adapted.

Turning now to <FIG>, a flowchart is provided illustrating an example method <NUM> of determining pose(s) and classification(s) for environmental object(s) based on FMCW LIDAR data, and controlling an autonomous vehicle based on at least some of the pose(s) and classification(s). For convenience, the operations of the flow chart are described with reference to a system that performs the operations. This system may include various components, such as one or more processors of the autonomous vehicle <NUM> of <FIG>. Moreover, while operations of method <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

At block <NUM>, the system receives FMCW LIDAR data from an FMCW LIDAR component of a vehicle. In some implementations, the received FMCW LIDAR data at an iteration of block <NUM> can be a most recent unprocessed (unprocessed using method <NUM>) instance of data generated by the FMCW LIDAR component. The instance of data can be generated by the FMCW LIDAR component during a sensing cycle and can be received by the system, for example, as part of a stream, as data packet(s), or via accessing shared memory. In some implementations, the received FMCW LIDAR data at an iteration of block <NUM> includes only the most recent unprocessed instance of data generated by the FMCW LIDAR component. In some other implementations, the received FMCW LIDAR data at an iteration of block <NUM> also includes one or more prior instances of data generated by the FMCW LIDAR component, such as the most recent N instances of data generated by the FMCW LIDAR component.

At block <NUM>, the system generates output based on processing the FMCW LIDAR data, received at block <NUM>, using at least one classification model. For example, the at least one classification model can include a trained neural network model and the generated output can include, for each of a plurality of spatial regions<NUM>-N, a corresponding probability for each of one or more classes. In such an example, each of the spatial regions<NUM>-N of the output can correspond to a portion of the FMCW LIDAR data and, as a result, corresponds to an area of the environment. Each of the probabilities of the output for each of the spatial regions corresponds to a classification. For example, the output generated using a classification model can indicate, for each of the spatial regions<NUM>-N, a first probability of a vehicle class, a second probability of a pedestrian class, a third probability of a bicyclist class, etc. As another example, the output generated using a classification model can indicate, for each of the spatial regions<NUM>-N, a corresponding probability for only a single class, such as a vehicle class. In those or other examples, multiple classification models can be utilized, each being configured to predict probabilities, for spatial regions, for a subset of classes (e.g., only a single class).

At block <NUM>, the system determines, based on the output and for each of one or more objects, a pose of the object and a classification for the object. For example, the system can determine, based on output of spatial regions<NUM>-N and corresponding class probabilities, that no objects are present in those spatial regions<NUM>-N whose class probabilities each fail to satisfy a threshold low probability. Also, for example, the system can determine that those spatial regions<NUM>-N having a corresponding class probability that satisfies a threshold high probability likely contain an object of that class. Further, the system can determine clusters of neighboring (directly neighboring or within N (e.g., <NUM>, <NUM>, or <NUM>) spatial regions) spatial regions that have probabilities satisfying a threshold for the same class and determine they correspond to a single object of that class. Yet further, the system can determine the pose of that single object based on those spatial regions. The determined pose can be with reference to the FMCW LIDAR data of block <NUM>, or with reference to another frame.

At block <NUM>, the system controls the vehicle based on the pose(s) and classification(s) for object(s) determined at block <NUM>. For example, the system can generate a trajectory for the vehicle based on one or more of the poses and/or classifications determined at block <NUM>. In some implementations, block <NUM> includes sub-block 808A. At sub-block 808A, the system determines, for one or more of the objects, a candidate trajectory for the object. In some implementations, the system may not determine a candidate trajectory for object(s) of certain class(es), such as objects having a classification that corresponds to an immobile object and/or that corresponds to an object that is mobile but has a relatively slow maximum velocity.

At sub-block 808A, the system can optionally determine a candidate trajectory for an object based on an instantaneous velocity for the object. In such a situation, sub-block 808A can include sub-block 808B, in which the system determines, for one or more of the objects, an instantaneous velocity based on FMCW LIDAR data that corresponds to the object. For example, the system can determine an instantaneous velocity for an object based on the velocities indicated by the FMCW LIDAR data (received at block <NUM>) that correspond to the object. For instance, the pose of the object (determined at block <NUM>) can be utilized to select a subgroup of FMCW LIDAR data that corresponds to the object, and the instantaneous velocity determined based on the velocities indicated by the FMCW LIDAR data of the subgroup. In some implementations, the system can select a particular technique for determining an instantaneous velocity of an object, from a plurality of candidate techniques, based on the determined classification for the object, the determined pose for the object, and/or based on the quantity of FMCW LIDAR data that is available for the object. In some implementations, at sub-block 808A, the system can optionally determine a candidate trajectory for an object based on its classification, and without reference to any instantaneous velocity for the object. For example, for an object of a pedestrian classification a candidate trajectory can be determined based on a default maximum instantaneous velocity for a pedestrian.

In implementations where block <NUM> includes sub-block 808A (and optionally sub-block 808B), the system can generate the trajectory for the vehicle based on one or more of the candidate trajectories generated at sub-block 808A (which are based on poses and/or classifications determined at block <NUM>).

Claim 1:
A method, comprising:
receiving, from a phase coherent Light Detection and Ranging, LIDAR, monopulse component (132B) of a vehicle (<NUM>; <NUM>), wherein the phase coherent LIDAR monopulse component is a frequency-modulated continuous wave Light Detection and Ranging, FMCW LIDAR,
monopulse component (132B) of the vehicle, a group of FMCW LIDAR data points (682A, 682B, 682C) collectively capturing a plurality of points in an area of an environment of the vehicle, each of the FMCW LIDAR data points of the group directly indicating a corresponding range and a corresponding velocity for a corresponding one of the points in the environment, and each being generated based on a corresponding sensing event of the phase coherent LIDAR monopulse component,
wherein the corresponding sensing events of the phase coherent LIDAR monopulse component each comprise a first receiver sensing event at a first receiver (132B3A) of the phase coherent LIDAR monopulse component and a second receiver sensing event at a second receiver (132B3B) of the phase coherent LIDAR monopulse component;
determining that a subgroup, of the FMCW LIDAR data points of the group, corresponds to an object (<NUM>; <NUM>-<NUM>; <NUM>-<NUM>) of a particular classification, the determining based on both:
the corresponding ranges for multiple of the FMCW LIDAR data points of the subgroup, and
the corresponding velocities for multiple of the FMCW LIDAR data points of the subgroup; and
adapting autonomous control of the vehicle based on determining that the subgroup corresponds to the object of the particular classification.