Patent Description:
The present invention is defined by a method according to claim <NUM>, a system according to claim <NUM>, a computer readable storage medium according to claim <NUM>, and a host vehicle according to claim <NUM>.

'One or more' includes a function being performed by one element, a function being performed by more than one element, e.g., in a distributed fashion, several functions being performed by one element, several functions being performed by several elements, or any combination of the above.

It will also be understood that, although the terms first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

The terminology used in the description of the various described embodiments herein is for describing embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used herein refers to and encompasses all possible combinations of one or more of the associated listed items. It will be further understood that the terms "includes," "including," "comprises," and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

As used herein, the term "if" is, optionally, construed to mean "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" is, optionally, construed to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]," depending on the context.

<FIG> illustrates a non-limiting example of a radar-data collection system <NUM>, hereafter often referred to as the system <NUM>. As will be described in more detail below, the system <NUM> is useful to create a library of radar-profiles that are radar-returns annotated with various information such as, but not limited to, the identity of the object (e.g. another-vehicle or a stop-sign) detected by a radar. Those in the automated-vehicle object-detection arts will recognize that the identity of an object is often more readily determine based on an image from a camera rather than a radar-return from radar. The library of radar-profiles will be useful for continued operation of a host-vehicle <NUM> if a camera <NUM> on the host-vehicle <NUM> is damaged or otherwise inoperable where the radar <NUM> is relied upon to determine the identity of the object. It is contemplated that the teachings herein are also applicable to annotating cloud-points from a lidar so that a library of lidar-maps could be generated and subsequently used for more effective control of the host-vehicle <NUM> if the camera were to fail, with or without the presence of the radar <NUM> in the system <NUM>.

The host-vehicle <NUM> may be characterized as an automated vehicle, and may be referred to by some as an automated-mobility-on-demand (AMOD) type of vehicle. As used herein, the term automated vehicle may apply to instances when the host-vehicle <NUM> is being operated in an automated-mode, i.e. a fully autonomous mode, where a human-operator (not shown) of the host-vehicle <NUM> may do little more than designate a destination to operate the host-vehicle <NUM>. However, full automation is not a requirement. It is contemplated that the teachings presented herein are useful when the host-vehicle <NUM> is operated in a manual-mode where the degree or level of automation may be little more than providing an audible or visual warning to the human-operator who is generally in control of the steering, accelerator, and brakes of the host-vehicle <NUM>. For example, the system <NUM> may merely assist the human-operator as needed to change lanes and/or avoid interference with and/or a collision with, for example, an object <NUM> such as an other-vehicle <NUM>.

As suggested above, the system <NUM> includes a camera <NUM> and a radar <NUM> that is generally configured or designed for mounting on the host-vehicle <NUM>. To be suitable for mounting on the host-vehicle <NUM>, the camera <NUM> and the radar <NUM> are designed to be reliably operable when subjected to environmental conditions experienced by vehicles such as rain, snow, dirt, chemical contamination, temperature extremes, and the like. Those in the automotive sensing arts will instantly recognize what features are desirable for the camera <NUM> and the radar <NUM> to be suitable for use on the host-vehicle <NUM>.

The radar <NUM> is configured to indicate (i.e. output a signal indicative of or contributing to) a radar-profile <NUM> of an instance of the object <NUM> detected by the radar <NUM>. As used herein, the radar-profile <NUM> may include, but is not limited to, copies of radar-returns (i.e. a copy of a signal output by the radar <NUM>), a mapping (a post processed signal output by the radar <NUM>) of distances and/or directions from the radar <NUM> to various points on the object <NUM>, and/or mapping of range-rate of one or more points on the object <NUM>, i.e. how fast the object <NUM> or points of the object <NUM> are moving relative to (e.g. towards or away from) the radar <NUM>. The distance and range-rate information may be stored as time-domain or frequency-domain data, as will be recognized by those in radar signal processing arts. The radar <NUM> may be a two-dimensional (2D) type of radar that indicates a distance and a direction (e.g. an azimuth-angle) to an instance of a radar return, or may be a three-dimensional (3D) type of radar that also indicates an elevation-angle.

As will be explained in more detail later, the radar-profile <NUM> may include information that may not be readily obtainable directly from signals output by the radar <NUM>. For example, an image <NUM> from the camera <NUM> may be readily analyzed using known processes to determine an identity <NUM> (e.g. small car, trash-bin, large truck, train, stop-sign, yield-sign, railroad-crossing-sign) of the object <NUM>. Those in the radar signal processing arts will recognized that various similar sized objects may be difficult to distinguish, i.e. determine the identity <NUM> of, using only radar-returns from the radar <NUM>. However, the inventors have discovered that once that information (e.g. the identity <NUM>) is known and associated with an instance of the radar-profile <NUM> by way of annotation of the radar-profile <NUM>, then each instance of the radar-profile <NUM> may be referred to at some future date to, for example, determine the identity <NUM> of the object using only radar-returns from the radar <NUM>.

The camera <NUM> may be a monoscopic or stereoscopic or stereoscopic type of camera that is configured to render the image <NUM> of the object <NUM>. The image <NUM> may be a single snap-shot of the object <NUM>, or may be a video composed of a plurality of snapshots. The camera <NUM> may be sensitive to visible-light and/or infrared-light. The camera <NUM> may be co-located with the radar <NUM> as part of an object-detector as suggested in <FIG>, or the camera <NUM> may be spaced apart from the radar <NUM>. For example, the camera may be mounted on the roof of the host-vehicle <NUM>, and the radar <NUM> may be mounted at the front of the host-vehicle <NUM>, e.g. near the headlights or bumper of the host-vehicle <NUM>. The object-detector may be all or part of a perception-sensor used for autonomous operation of the host-vehicle <NUM>.

The system <NUM> includes a controller-circuit <NUM> in communication with the radar <NUM> via a first-input 28A, and in communication with the camera <NUM> via a second-input 28B. The communication may be by way of, but not limited to, wires, fiber-optic, or wireless-communications, as will be recognized by those in the art. The controller-circuit <NUM>, hereafter sometimes referred to as the controller <NUM>, may include one or more instances of a processor <NUM> such as one or more instances of a microprocessor or other control circuitry such as analog and/or digital control circuitry including an application specific integrated circuit (ASIC) for processing data as should be evident to those in the art. While the system <NUM> described herein is generally described in terms of having a single instance of the controller <NUM>, it is recognized that the functions of the controller <NUM> may be shared or distributed among several instances of controllers that are each configured for some specific task. Hereafter, any reference to the controller <NUM> being configured for something is to also be interpreted as suggesting that the processor <NUM> may also be configured for the same thing. It is also recognized that there may be multiple instances of processors in any instance of the controller <NUM>. The controller <NUM> may include memory <NUM>, i.e. non-transitory computer-readable storage-medium, including non-volatile memory, such as electrically erasable programmable read-only memory (EEPROM) for storing one or more routines, thresholds, and captured data. The memory <NUM> may be part of the processor <NUM>, or part of the controller <NUM>, or separate from the controller <NUM> such as remote memory stored in the cloud. The one or more routines may be executed by the controller <NUM> or the processor <NUM> to perform steps for determining the radar-profile <NUM> based on signals received by the controller <NUM> from the camera <NUM> and the radar <NUM>. The controller <NUM> is configured (e.g. programmed) to determine the identity <NUM> (e.g. car, truck, stop-sign) of the object <NUM> in accordance with the image <NUM>, and annotate (i.e. document, characterize, or label) the radar-profile <NUM> in accordance with the identity <NUM>.

<FIG> illustrate non-limiting examples of radar-profile 22A and radar-profile 22B. In this example, the radar-profiles are a 'bird's-eye' or overhead perspective mapping transposition of radar-return 36A and radar-return 36B detected by the radar <NUM>, and include image 24A and image 24B that are associated with radar-return 36A and radar-return 36B, respectively, each of which in this example may be characterized as a reflection-intensity-map <NUM>. The radar-profiles also include annotations <NUM> that indicate the distance to each instance of the other-vehicle, which may be used to determine a size of the other-vehicle in each radar-profile using known geometric algorithms. The images may be analyzed using known image comparison techniques to determine the make/model of each of the other-vehicles shown. Once the radar-profiles are annotated, the radar-profiles may be stored (for future reference) in the memory <NUM> on-board the host-vehicle <NUM> and/or stored remotely in a data-base of annotated radar-profiles, which may be a shared data-base accessible by other vehicles. Accordingly, the system <NUM> may include a transceiver (e.g. Wi-Fi, cellular-network, etc.) used to upload/download the annotated radar-profiles, and later access a library of the annotated radar-profiles.

Continuing to refer to <FIG>, <FIG>, the controller-circuit <NUM> may be configured to determine an orientation-angle <NUM> of the object <NUM> in accordance with, or based on, the radar-profile <NUM>. For example, if the radar-profile includes a distance-map as suggested in <FIG>, and a first-corner or first-edge 44A of the object <NUM> is further away than a second-corner or a second-edge 44B, then that is an indication that the object <NUM> is not oriented normal to the host-vehicle <NUM>. Known geometry techniques can be used to determine the orientation-angle <NUM>. It is further contemplated that a nominal-distance to object <NUM>, a radar-angle (angle between the first-edge 44A and the second-edge 44B), and the orientation-angle <NUM> can used to determine an object-length, e.g. the length (or width) of the object <NUM>. By annotating the radar-profile <NUM> with the orientation-angle <NUM>, a future instance of a radar-return can be more reliably compared to the annotated radar-profile stored in the data-base.

Alternatively, or additionally, the controller-circuit <NUM> may be configured to determine the orientation-angle <NUM> of the object <NUM> in accordance with the image <NUM>. For example, the orientation angle <NUM> of another-vehicle can be determined from the image <NUM> by comparing the height in the image <NUM> of the bottom edge of the front and rear tires of the vehicle in the image. As above, the orientation-angle <NUM> can be used to further annotate the radar-profile <NUM>.

As noted above and shown in <FIG>, the controller-circuit <NUM> may be configured to determine a reflection-intensity-map <NUM> in accordance with the radar-profile. The reflection-intensity-map <NUM> indicates the signal-strength of the reflected radar-signal at various locations in the file of view of the radar <NUM>. <FIG> show two-dimensional reflection-intensity maps, and three-dimensional reflection-intensity-maps are contemplated, although they would be difficult to illustrate here as those in the art would recognize. The reflection-intensity-map of a vehicle suggests that the object has some depth as radar signal may pass through and/or under (reflected by the ground) a vehicle and be reflected by a surface other than what is directly viewable. By contrast, the reflection-intensity-map of a stop-sign would not show any depth, assuming there was not some other object close behind the stop-sign.

It should now be appreciated that the reflection-intensity-map <NUM> of a previously stored instance of the radar-profile <NUM> can be compared to a reflection-intensity-map of a recently received radar-return, and if the two reflection-intensity maps correspond well, then the identity <NUM> associated with that previously stored instance of the radar-profile <NUM> can be presumed to be the identity <NUM> of the recently received radar-return. That a recently received radar-return corresponds to a previously stored instance of the radar-profile <NUM> may be determined with a statistical comparison of reflection-intensity-maps and/or other known radar-target comparison algorithms The reflection-intensity-map <NUM> may be store in terms of time-domain or frequency-domain, which may depend on the type of radar being used.

The controller-circuit <NUM> may be configured to determine a distance <NUM> and/or a direction <NUM> and/or an elevation <NUM> to the object <NUM> in accordance with the radar-profile, and annotate the radar-profile <NUM> in accordance with the distance <NUM> and/or the direction <NUM> and/or the elevation <NUM>. These relatively simple annotations (the values of the orientation-angle <NUM> the distance <NUM> and/or the direction <NUM> and/or the elevation <NUM>) may be used as search markers or tabs to facilitate searching the data-base for a corresponding radar-profile. Other relatively simple characteristics or values such as, but not limited to, object-depth, object-length, object-height, mobile vs. stationary can be used to further annotate the radar-profile <NUM> and further facilitate searching for a corresponding, for example, reflection-intensity-map.

The system <NUM> may also include a sensor <NUM> mounted on the host-vehicle <NUM>. The sensor <NUM> is configured to indicate a pose-angle <NUM> that maybe any one or combination of, but not limited to, pitch-angle, roll-angle, yaw-angle. The sensor <NUM> may also indicate, for example, global-position, speed, heading, acceleration/deceleration of the host-vehicle <NUM>. The controller-circuit <NUM> is in communication with the sensor <NUM> via a third-input <NUM>. The communication may be by way of, but not limited to, wires, fiber-optic, or wireless-communications, as will be recognized by those in the art. If the pose-angle <NUM> is something other than level, data collected from the camera <NUM>, the radar <NUM>, and/or the lidar may need to be compensated so that the frame of reference of all radar-profiles is consistent. Accordingly, the controller-circuit <NUM> is configured to annotate the radar-profile <NUM> in accordance with the pose-angle <NUM>.

The controller-circuit <NUM> (or the processor <NUM>) may also be configured to operate the host-vehicle <NUM> in accordance with the radar-profile <NUM> in response to a determination that a recently received radar-return or radar-map corresponds to the radar-profile <NUM>. That is, as suggested above and by way of example, the system <NUM> or the controller <NUM> or the processor <NUM> compares a recent radar-return to a previously stored radar-profiles to determine the identity of an object <NUM> when the camera is not available, and given that knowledge of the identity of the object <NUM>, operates the host-vehicle <NUM> accordingly. Operation of the host-vehicle <NUM> may include autonomous (i.e. driverless) operation of the vehicle-controls which includes operating the steering, brakes, and accelerator of the host-vehicle <NUM>.

<FIG> illustrates a non-limiting example of a method <NUM> of operating a radar-data collection system <NUM>. As will be explained in more detail below, the method <NUM> helps to overcome the problem of how to continue operating the host-vehicle <NUM> if the camera <NUM> has been obstructed (e.g. covered with mud, ice, or bugs), damaged, or is otherwise not functional.

Step <NUM>, RECEIVE RADAR-PROFILE, may include receiving a radar-return by the radar <NUM>, and generating a radar-profile <NUM> of an object <NUM> detected by the radar <NUM>.

Step <NUM>, RECEIVE IMAGE, may include receiving an image <NUM> of the object <NUM> rendered by the camera <NUM>. If the camera <NUM> and the radar <NUM> have different perspectives of a similar field-of-view because the camera <NUM> and the radar <NUM> are spaced apart, receiving the image may also include applying a known transposition algorithm to the image <NUM> to correct for the differences of perspective.

Step <NUM>, DETERMINE IDENTITY, may include determining an identity of the object <NUM> by way of image-processing the image <NUM>, which may include comparing the image <NUM> to a library of identified-images stored in the data-base. The comparison of the image to the library of identified-images may be by way of, but not limited to, known neural-network algorithms.

Step <NUM>, DETERMINE ORIENTATION-ANGLE, is an optional step that may include determining an orientation-angle <NUM> of the object <NUM> in accordance with the radar-profile <NUM> OR determining an orientation-angle of the object in accordance with the image <NUM>. As used herein, the orientation-angle <NUM> is an indication of the angle of the object relative to a bore-site of the camera <NUM> and/or the radar <NUM>, where an orientation-angle <NUM> of zero (<NUM>°) corresponds to a major plane of the object being normal (i.e. perpendicular) to the bore-site of the camera <NUM> and/or the radar <NUM>.

Step <NUM>, DETERMINE REFLECTION-INTENSITY-MAP, is an optional step that may include determining a reflection-intensity-map <NUM> in accordance with (i.e. based on) the radar-profile <NUM> which is determined from the instant radar-return.

Step <NUM>, RECEIVE POSE-ANGLE, is an optional step that may include receiving a pose-angle <NUM> of a host-vehicle <NUM> from a sensor <NUM> such as an accelerometer or gravity-direction-detector.

Step <NUM>, DETERMINE DISTANCE / DIRECTION / ELEVATION, is an optional step that may include determining a distance <NUM> and/or a direction <NUM> and or an elevation <NUM> to the object <NUM> in accordance with the radar-profile <NUM>.

Step <NUM>, ANNOTATE RADAR-PROFILE, may include annotating the radar-profile with any one or combination of, but not limited to, the identity, the orientation-angle, the reflection-intensity-map, the pose-angle, the distance, the direction, and the elevation. As used herein, to annotate the radar-profile means to record the information with processed and/or unprocessed radar-returns. Once annotated, the instant radar-profile <NUM> may be stored in the data-base as part of a library of annotated radar-profiles that can be accessed/referenced at some later time/date as needed. The annotations <NUM> may be used as search parameters to accelerate the searching of the annotated radar-profiles stored in the data-base. The acceleration of the search is the result of being able to search/compare relatively simple values rather than only being able to make direct comparisons of radar-data.

Step <NUM>, CAMERA OPERATIONAL?, may include sending an image-request to the camera <NUM> and verifying that an image has been provided and optionally that the image as changed from a previous image. That is, verifying that the camera <NUM> is not outputting an image or stuck outputting the same image even though the host-vehicle <NUM> has moved enough that the image should have changed. If the camera <NUM> is deemed to not be operational (NO), the method <NUM> proceeds to step <NUM>. If the camera is functioning (YES), then the image <NUM> may be used to identify the object <NUM> for operating the host-vehicle <NUM> in step <NUM>.

Step <NUM>, IDENTIFY OBJECT BY RADAR-PROFILE, may include ignoring the image <NUM> from the camera <NUM> and determining whatever annotation information can be determine from the radar-signal, and then accessing the data-base to search the annotated radar-profiles and determine the identity <NUM> of the object based on the radar-profile <NUM> rather than the image <NUM>.

Step <NUM>, OPERATE HOST-VEHICLE, may include the controller <NUM> (or the processor <NUM>) operating the vehicle-controls to control the movement of the host-vehicle <NUM> to, for example, transport a passenger of the host-vehicle <NUM> to a destination. That is, operating the host-vehicle <NUM> in accordance with the radar-profile <NUM> in response to a determination that a radar-map (i.e. the instant radar-return received from the radar <NUM>) corresponds to the radar-profile <NUM>, which may have been previously stored in the data-base.

Described herein is a first device <NUM> that includes one or more processors <NUM>; memory <NUM>; and one or more programs <NUM>-<NUM> stored in memory <NUM>. The one or more programs <NUM>-<NUM> including instructions for performing all or part of the method <NUM>. Also, described herein is a non-transitory computer-readable storage-medium <NUM> that includes one or more programs <NUM>-<NUM> for execution by one or more processors <NUM> of a first device <NUM>, the one or more programs <NUM>-<NUM> including instructions which, when executed by the one or more processors <NUM>, cause the first device to perform all or part of the method <NUM>.

Accordingly, a radar-data collection system (the system <NUM>), a controller <NUM> for the system <NUM>, and a method <NUM> of operating the system <NUM> are provided. Radar-profiles <NUM> are generated by annotating data/signals from a radar <NUM> and a camera <NUM>, and then stored for later use if/when the camera <NUM> is not functional. By annotating the radar-profiles, the searching of a library of radar-profiles is accelerated as the annotations can be used as search parameters rather than having to perform a time-consuming one-to-one comparison of recorded radar-returns. That is, the annotations can be used to sort or catalog the radar-profiles that make up the library, thereby enabling an accelerated search.

Claim 1:
A method (<NUM>) comprising:
receiving, with a controller circuit, radar data and/or lidar data representing detections of an object proximate to a host vehicle;
receiving, with the controller circuit, image data representing a captured image (<NUM>) of the object (<NUM>);
determining, with the controller circuit, an identity (<NUM>) of the object (<NUM>) based on the image data;
characterized in that the method further comprises:
determining, with the controller circuit based on the radar data and the image data, a radar profile including a two-dimensional representation of the radar data relative to a host vehicle coordinate system and an associated image of the object, and/or determining, with the controller circuit based on the lidar data and the image data, a lidar-map representation of cloud-points in the lidar data relative the host vehicle coordinate system and the associated image of the object;
annotating the radar-profile (<NUM>) and/or the lidar-map in accordance with the identity (<NUM>) and an orientation of the object; and
storing with the controller circuit the radar-profile and/or the lidar-map in a memory (<NUM>) comprised in the controller-circuit, on-board the host-vehicle, and/or uploading with the controller circuit the radar-profile and/or the lidar-map to a remote data-base of annotated radar-profiles and/or annotated lidar-maps for future reference.