Prediction of pedestrian road crossing with vehicle sensor

A system of controlling operation of a vehicle includes one or more sensors operatively connected to the vehicle. The sensors are configured to obtain respective data of a scene and include a radar unit. A command unit is adapted to receive the respective data and includes a processor and tangible, non-transitory memory on which instructions are recorded. The command unit is configured to determine an orientation angle of a pedestrian in the scene, a Doppler frequency of the pedestrian and a distance of the pedestrian from a border of a road, based in part on the respective data. The orientation angle is based on a heading of the pedestrian relative to a direction of a road. The command unit is configured to designate a status of the pedestrian as either crossing or not crossing based on the distance, the orientation angle and the Doppler frequency of the pedestrian.

INTRODUCTION

The present disclosure relates to prediction of a pedestrian crossing a road using one or more sensors on a vehicle. Mobile platforms, such as motor vehicles, encounter other moving and non-moving objects as they journey through space and time. For example, a vehicle may encounter pedestrians, both moving and standing still. Predicting the behavior of pedestrians is not a trivial matter. In terms of model-based methods, it is difficult to construct explicit mathematical models to describe the probability of a pedestrian crossing a road. For data-driven methods, it is challenging to gather large datasets that capture the randomness of pedestrian crossing behavior and interdependence on other factors.

SUMMARY

Disclosed herein is a system of controlling operation of a vehicle in real-time. The system includes one or more sensors operatively connected to the vehicle. The sensors are configured to obtain respective data of a scene and include a radar unit. A command unit is adapted to receive the respective data and includes a processor and tangible, non-transitory memory on which instructions are recorded. The command unit is configured to determine an orientation angle of a pedestrian in the scene, a Doppler frequency of the pedestrian and a distance of the pedestrian from a border of a road, based in part on the respective data. The orientation angle is based on a heading of the pedestrian relative to a direction of a road. The command unit is configured to designate a status of the pedestrian as either crossing or not crossing based on the distance, the orientation angle and the Doppler frequency of the pedestrian.

The command unit may be adapted to change a vehicle trajectory and/or decelerate the vehicle based in part on the status of the pedestrian. The status is designated as crossing when: (1) the distance is less than a threshold distance; and (2) the orientation angle is less than a threshold angle; and (3) the Doppler frequency of the pedestrian is above a threshold frequency. The status is designated as not crossing otherwise.

In some embodiments, the orientation angle is between the heading of the pedestrian and a normal vector perpendicular to the direction of the road. The threshold frequency may be about 160 Hz and the threshold angle may be about 30 degrees. The command unit is adapted to segment the scene into a road portion and a non-road portion, with the direction and the border of the road being estimated based in part on the segmentation of the scene. The command unit is adapted to detect a location of the pedestrian in three-dimensions and generate a bounding box around the pedestrian in the scene. The orientation and the distance of the pedestrian may be estimated from a center of the bounding box.

In some embodiments, the sensors include a camera. The respective data from the camera may be used to segment the scene. A geographical map may be accessible to the command unit, with the command unit being adapted to employ information from the geographical map to segment the scene. The sensors may include a lidar unit. The respective data from the lidar unit may be employed to generate the bounding box around the pedestrian.

The command unit may be adapted to obtain a vehicle Doppler frequency based in part on the respective data from the radar unit and obtain an extracted frequency value at a center point of a bounding box surrounding the pedestrian in the scene. The command unit may be adapted to obtain the Doppler frequency of the pedestrian by subtracting the vehicle Doppler frequency from the extracted frequency value. The vehicle Doppler frequency is based in part on a carrier frequency of a signal from the radar unit, a speed of light and a speed of the vehicle.

Disclosed is a method of controlling operation of a vehicle in real-time, the vehicle having one or more sensors and a command unit with a processor and tangible, non-transitory memory. The method includes obtaining respective data of a scene, via the one or more sensors, and including a radar unit in the one or more sensors. An orientation angle of a pedestrian in the scene is determined based in part on the respective data. The orientation angle is based on a heading of the pedestrian relative to a direction of a road. The method includes determining a Doppler frequency of the pedestrian and a distance of the pedestrian from a border of the road based in part on the respective data and designating a status of the pedestrian in real-time as either crossing or not crossing based on the distance, the orientation angle and the Doppler frequency of the pedestrian.

DETAILED DESCRIPTION

Referring to the drawings, wherein like reference numbers refer to like components,FIG.1schematically illustrates a system10of controlling operation of a vehicle12. The vehicle12may be a mobile platform such as, but not limited to, a passenger car, sport utility car, light truck, heavy duty truck, ATV, minivan, bus, transit vehicle, bicycle, robot, farm implement (e.g., tractor), sports-related equipment (e.g., golf cart), boat, airplane and train. The vehicle12may take many different forms and include multiple and/or alternate components and facilities. It is to be understood that the vehicle12may take many different forms and have additional components.

Referring toFIG.1, one or more sensors S are operatively connected to the vehicle12. The sensors S may include a radar unit14, a lidar unit16and a camera18. The sensors S are configured to obtain respective data of a scene20, an example of which is shown inFIG.1. The scene20includes pedestrian22on a sidewalk24that is adjacent to or in proximity to a road26. The example scene20includes a building28and a tree30.

The system10includes a command unit50adapted to receive the respective data. The command unit50has a controller C, at least one processor P and at least one memory M (or non-transitory, tangible computer readable storage medium) on which instructions are recorded for executing method100, shown in and described with respect toFIG.2.

The issue being resolved is predicting whether the pedestrian22will cross the road26in the short-term future, e.g., within 1 or 2 seconds. As described below, the command unit50is configured to determine an orientation angle46of the pedestrian22in the scene20, a Doppler frequency of the pedestrian22and a distance D of the pedestrian22from an edge or border48of the road26, based in part on the respective data. The system10is adapted (via execution of method100) to designate a status of the pedestrian22in the scene20as either crossing or not crossing the road26based on the distance, the orientation angle and the Doppler frequency of the pedestrian. Other example scenes220,320having pedestrians222,322are described below relative toFIGS.3-4, respectively.

Referring toFIG.1, the orientation angle46is based on a heading42of the pedestrian22relative to a direction44of the road26. The orientation angle may be defined in various ways. Here, the orientation angle46is between the heading42of the pedestrian22and a normal vector45perpendicular to the direction44, which is parallel to the direction of the road26. The road26may be defined as a stretch of ground on which a vehicle may travel. The road26may be smoothed, semi-paved or paved. The road26may be an unpaved dirt road. The road26includes urban roads (e.g., interstate, collectors), rural roads, the driveway of a home or building or a parking lot. The road26may or may not have lines painted on it. The status of the pedestrian is beneficial for determining vehicle route and speed for an autonomous vehicle.

Referring toFIG.1, the sensors S may be positioned such that their respective field of view substantially overlaps. The radar unit14includes antennas for transmitting electromagnetic waves in at least one of a radio and a microwave domain. The electromagnetic waves reflect off the various objects in the scene20and return to the radar unit14, providing information about their location and speed. The radar unit14may employ phase-shifters to shift the phase of the electromagnetic waves to produce a phased-array beam. The radar unit14provides Doppler measurement, which is an instantaneous radial velocity measurement with relatively low latency and relatively high accuracy. In one example, the accuracy is about 0.1 meters per second.

The lidar unit16uses a laser source to target various objects in the scene20and measures the time for the reflected light to return. Lidar may employ visible, ultraviolet and near infrared electromagnetic radiation. The lidar unit16may include a microscopic array of individual antennas where the timing (phase) of each antenna is controlled to steer a cohesive signal in a specific direction. Other types of lidar or radar systems available to those skilled in the art may be employed.

The camera18provides image data of various targets in the scene20at various times. The image data may include a single visual image or a sequence of visual images. The camera18may include components available to those skilled in the art, such as for example, a lens, beam splitter, mirror, photomultiplier, dispersion devices, apertures, and modulators.

In one embodiment, the command unit50is embedded in the vehicle12. In another embodiment, the command unit50is stored in an “off-board” or remotely located cloud computing service52, shown inFIG.1. The cloud computing service52may include one or more remote servers hosted on the Internet to store, manage, and process data. The cloud computing service52may be at least partially managed by personnel at various locations, such as at a “back office.” The memory M can store command unit-executable instruction sets, and the processor P can execute the command unit-executable instruction sets stored in the memory M.

Communication between the various components of the vehicle12and the cloud unit52may occur through a wireless network54. The wireless network54may be a short-range network or a long-range network. The wireless network54may be a communication BUS, which may be in the form of a serial Controller Area Network (CAN-BUS). The wireless network54may be a serial communication bus in the form of a local area network which may include, but is not limited to, a Controller Area Network (CAN), a Controller Area Network with Flexible Data Rate (CAN-FD), Ethernet, Bluetooth, WIFI and other forms of data. The wireless network52may be a Wireless Local Area Network (LAN) which links multiple devices using a wireless distribution method, a Wireless Metropolitan Area Network (MAN) which connects several wireless LANs or a Wireless Wide Area Network (WAN) which covers large areas such as neighboring towns and cities. Other types of network technologies or communication protocols available to those skilled in the art may be employed.

Referring now toFIG.2, a flowchart of the method100is shown. Method100may be embodied as computer-readable code or instructions stored on and partially executable by the command unit50ofFIG.1. Method100need not be applied in the specific order recited herein. Furthermore, it is to be understood that some steps may be eliminated.

Per block110ofFIG.2, the command unit50is programmed to detect the location and heading42of the pedestrian22in three-dimensions. The respective data from the camera18and facial recognition technology may be employed to detect the location of the pedestrian22. Alternatively, respective data from the lidar unit16may be used to capture the position and the heading of the pedestrian22. The lidar unit16generally provides higher resolution than the radar unit14. A three-dimensional object detection neural network may be used to detect the location of the pedestrian22.

Per block120ofFIG.2, the method100includes generating a bounding box40around the pedestrian22in the scene20based on the respective data from the sensors S. The bounding box40covers the height, width and length of the pedestrian22. In one example, the bounding box40is defined as the minimum or smallest bounding or enclosing box for a set of points defining the pedestrian22in three dimensions. In other words, the bounding box is the box with the smallest volume within which each of the points defining the pedestrian22lie. The command unit50may access a deep learning module or object detection neural network to generate the bounding box40.

Per block130ofFIG.2, the command unit50is programmed to segment the scene20into a road portion (shown shaded inFIGS.1and3-4) and a non-road portion. The segmentation may be done by dividing the pixels or voxels in the image of the scene20into road pixels and non-road pixels. In one embodiment, the segmentation is accomplished using the images/respective data from the camera18. In another embodiment, the segmentation is accomplished using a map obtained from a reference source. For example, the map may be divided into a plurality of grid, with each grid being defined as either a road grid or a non-road grid. The segmentation of the image or map may be done using a neural network. The bounding box40is overlaid or displayed relative to the road portion of the scene20. Also per block130, the command unit50is programmed to calculate the distance D between the pedestrian22and a boundary or edge or border48of the road26, shown inFIG.1. The center of the bounding box40(using the height, width and length) may be taken as the position of the pedestrian22.

Per block140ofFIG.2, the method100includes obtaining an extracted frequency value (F*) for the pedestrian22. The extracted frequency value (F*) may be obtained by taking the Doppler with the maximum intensity value from a data-cube at the range and angle of the center of the bounding box40surrounding the pedestrian22. The location of the extracted frequency value (F*) may be changed from the center-point of the bounding box40. The radar unit provides the range and the center point angle (θ). The initiation of steps or walking by a pedestrian22may be detected with low latency by the radar unit14. In other words, the Doppler frequency in the case of the pedestrian22starting to walk is significantly larger than in the case of the pedestrian22standing.

Per block150ofFIG.2, the command unit50is programmed to obtain the Doppler frequency of the vehicle12that is hosting the radar unit14, (sometimes referred to as the host vehicle). The vehicle Doppler frequency (FV) may be calculated as: FV={2*(FC/c)*[cos(θ)*Vx+sin(θ)*Vy]}. Here, FCis the carrier frequency of the radar unit14, c is the speed of light, θ represents the orientation of the radar unit14, Vx is the velocity of the vehicle12in the x-direction, Vy is the velocity of the vehicle12in the y-direction (where x and y are the Cartesian coordinates with respect to the center point of the radar unit14in the vehicle12). Also per block150, the command unit50is programmed to obtain the Doppler frequency of the pedestrian22by subtracting the vehicle Doppler frequency from the extracted frequency value, as follows: FP=(F*−FV). Here, FPis the Doppler frequency of the pedestrian22, FVis the vehicle Doppler frequency and F* is the extracted frequency value.

Per block160ofFIG.2, the command unit50is programmed to designate a status of the pedestrian22in real-time as either crossing or not crossing based on the distance D, the orientation angle46and the Doppler frequency of the pedestrian22. In a non-limiting example, the threshold distance is set to be about 30 cm. The threshold Doppler value may be set to 160 Hz and the threshold angle may be set to be about 30 degrees.

Referring toFIG.1, the pedestrian22is positioned within threshold distance of the road26and the pedestrian Doppler frequency is greater than the threshold frequency Doppler. The pedestrian22is heading toward the road26as the orientation angle46between the normal vector45(perpendicular to the road direction44) and the heading42is relatively small (approximately 10 degrees here, which is less than the threshold angle). Since each of the conditions is met, the command unit50will designate the status of the pedestrian22as crossing.

FIG.3is a schematic perspective view of another example scene220with a pedestrian222standing on a sidewalk224that is in proximity to a road226(shown shaded). The example scene220includes a building228and vegetation230. Also shown inFIG.3is a bounding box240around the pedestrian222and the heading242of the pedestrian222. Here, the pedestrian Doppler frequency is greater than the threshold frequency Doppler as the pedestrian222is walking and the pedestrian222(at distance D) is positioned within threshold distance of the road226. However, the pedestrian222is not heading toward the road226as the orientation angle between the normal vector245(perpendicular to the road direction244) and the heading242is relatively large (approximately 90 degrees here, which is greater than the threshold angle). Thus, the command unit50will designate the status of the pedestrian222as not crossing.

FIG.4is a schematic perspective view of yet another example scene320with a pedestrian322walking on a sidewalk324that is in proximity to a road326(shown shaded). The example scene320includes a building328and vegetation330. Also shown inFIG.4is a bounding box340around the pedestrian322, the heading342of the pedestrian322and a road direction344. Referring toFIG.4, the pedestrian322is positioned (at distance D) within threshold distance of the road326and the pedestrian Doppler frequency is greater than the threshold frequency Doppler. However, the pedestrian222is not heading toward the road326as the orientation angle between the normal vector345(perpendicular to the road direction344) and the heading342is relatively large (approximately 180 degrees here, which is greater than the threshold angle). Thus, the command unit50will designate the status of the pedestrian322as not crossing.

Proceeding to block170, the command unit50is configured to control operation of the vehicle12based on the predicted pedestrian crossing status. The command unit50may be adapted to change a trajectory of the vehicle12and/or decelerate the vehicle12based in part on the status of the pedestrian. For example, the vehicle12may include an automatic braking module60that is selectively activated to reduce motion or decelerate the vehicle12when the status is designated as crossing. The vehicle12may include a lane change assist module62that is selectively activated to assist the vehicle12in changing lanes when the status is designated as crossing.

In summary, the system10predicts whether a pedestrian22will be crossing the road26by detecting speed of the pedestrian with low latency from Doppler measurements of the radar unit14. The system obtains 3D location and heading of a pedestrian22with indication of proximity to the road26. Road crossing is predicted when the pedestrian22is relatively close to the road26with heading towards the road26and an instantaneous speed is detected from the radar unit14with low latency. The results provide guidance for the vehicle12to reduce its speed, change its direction or other measures.

The command unit50includes a computer-readable medium (also referred to as a processor-readable medium), including a non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random-access memory (DRAM), which may constitute a main memory. Such instructions may be transmitted by one or more transmission media, including coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to a processor of a computer. Some forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, other magnetic media, a CD-ROM, DVD, other optical media, punch cards, paper tape, other physical media, a RAM, a PROM, an EPROM, a FLASH-EEPROM, other memory chips or cartridges, or other media from which a computer can read.

Look-up tables, databases, data repositories or other data stores described herein may include various kinds of mechanisms for storing, accessing, and retrieving various kinds of data, including a hierarchical database, a set of files in a file system, an application database in a proprietary format, a relational database management system (RDBMS), etc. Each such data store may be included within a computing device employing a computer operating system such as one of those mentioned above and may be accessed via a network in one or more of a variety of manners. A file system may be accessible from a computer operating system and may include files stored in various formats. An RDBMS may employ the Structured Query Language (SQL) in addition to a language for creating, storing, editing, and executing stored procedures, such as the PL/SQL language mentioned above.