System and method for using V2X and sensor data

A method and system for traffic control includes receiving at a processing unit sensor data of a site on a road network and receiving at the processing unit a V2X communication. Locations of road users are calculated from the sensor data and the V2X communication enabling the detection of connected and non-connected road users. Once connected and non-connected road users are detected at a site, this information can be used to control traffic.

FIELD

The present invention relates to communication between road users and between road users and infrastructure.

BACKGROUND

In an urban setting there are many blind spots for human road users. Autonomous vehicles can't solve these blind spots as their sensors are of limited field of view in a way similar to the human eye.

Connected road users (such as connected vehicles, bicycles, pedestrians, etc.) represent one of the technologies aimed at solving blind spots and other cases by transmitting information from one road user to the other regarding dangers, location of other road users, etc.

Possible communications between road users include: vehicles to vehicles (V2V) communication and vehicles to pedestrians (V2P) communication. Road users may also communicate with the road infrastructure in vehicle to infrastructure (V2I) communication and pedestrian to infrastructure (P2I) communication. These communication modes are generally termed vehicle to everything (V2X).

Currently, the competing standards used for V2X are DSRC (Dedicated Short Range Communication) and C-V2X/5G cellular based protocols. These two standards deal with the physical level of wireless communication of V2X, namely, the challenges related to low-latency, high reliability and high speed moving objects. Both standards support the same functional layer (transport layer) on which applications can be created.

At the core of V2X communication is a message set that is broadcasted by every connected road user at 10 hz. In the US standard (SAE J2375) the message set is called Basic Safety Message (BSM) or Personal Safety Message for pedestrians (PSM) and in the European standard (ITS-G5) the message set is called Cooperative Awareness Message (CAM). These message sets are mostly the same, functionality-wise.

A message set typically includes information such as: location (latitude and longitude) estimation and the accuracy of the location estimation, bearing in degrees in relation to the north, speed, acceleration, past trajectory and predicted future trajectory.

The information in the message set enables connected road users to use the road more safely and efficiently, reducing traffic congestion, accidents and air pollution.

However, one of the core issues with V2X communication is the need for mass adoption of this technology to make it viable. At the very least, two road users (e.g., two vehicles) must be connected for them to be able to communicate and in order for this technology to provide value. Until mass adoption of V2X communication capabilities, the value of having connectivity is practically none.

Adoption of technology is usually non-linear and can't be properly estimated, especially at the micro level (e.g. estimating how many of the total number of vehicles on a specific street are connected vehicles). The same applies to V2X technology adoption. Until 100% of the road users have V2X communication capabilities, systems using V2X information for decision making, may need to estimate the adoption rate of V2X technology in order to deduce a total amount of road users at a site, based on the amount of connected road users at that site.

In some cases, road infrastructure can communicate with road users. For example, traffic signal preemption (also called traffic signal prioritization) enables to manipulate traffic signals in the path of an emergency vehicle, halting conflicting traffic and allowing the emergency vehicle right-of-way, to help reduce response times and enhance traffic safety. Signal preemption can also be used to allow public transportation priority access through intersections, or by railroad systems at crossings, to prevent collisions.

Traffic signal preemption can be employed by V2I preemption which is based on the transmission of a preemption message (e.g. in SAE J2375—Signal Request Message—SRM) from a connected vehicle to the infrastructure, e.g., a traffic signal controller. Currently, a list of authorized vehicles (e.g., emergency vehicles and public transportation) is used to allow preemption only to listed vehicles.

A few major drawbacks of the current V2I preemption approach include the following:Hacking or malfunction can cause malicious use of the SRM, thereby enabling preemption for non-authorized vehicles;Every authorized vehicle needs to have a V2X subsystem installed, which increases the cost of the vehicle and delays the adoption of preemption in intersections;Non-connected authorized vehicles aren't taken into account using this approach, which means that conflicting demands may not be handled properly. For example, a connected bus crossing the intersection from the north may get priority while at the same time a non-connected police car coming from the west may be delayed due to the priority given to the bus. The fact that the police car is not connected, and thus cannot communicate with the infrastructure, causes priority to be assigned incorrectly.

Even if V2X technology were widely adopted, there would still be scenarios not covered by V2X communication, such as non-connected road users (e.g. a small children) running into the street, V2X communication module malfunctioning, obstacles (such as a pothole) that are not connected, and so on.

For the reasons listed above, current use of V2X technology is inadequate to provide safety and other potential benefits of road users' connectivity.

SUMMARY

Embodiments of the invention provide full coverage of a site on a road network, enabling to detect and identify both connected and non-connected road users at the site, and enabling to emulate a situation where all road users are connected, even road users that are not using V2X communication. Thus, embodiments of the invention provide safety features and other applications enabled by V2X technology to all road users, even in the (extreme) case of having only one connected road user at the site.

Embodiments of the invention employ V2X communication to detect and identify connected road users in the vicinity of or approaching a site and sensors to detect all road users in vicinity of the site.

In one embodiment a traffic control system includes a sensor to detect a road user, the sensor mounted at a site on a road network; a V2X communication module; and a processing unit to receive inputs from the sensor and from the V2X communication module, the inputs including at least a location of a road user. The system may then detect and identify, based on the inputs, connected and non-connected road users.

DETAILED DESCRIPTION

Embodiments of the invention provide locations of all road users at a predetermined site on the road network, enabling efficient traffic control at the site. Embodiments of the invention include detecting total road users and their locations, at a site, based on input from a sensor mounted in vicinity of the site and detecting connected road users and their locations, based on V2X communication. Non-connected road users are then detected by matching each connected road user to one of the total road users. All road users that are not matched are determined to be non-connected road users.

Location of a road user typically refers to coordinates which can be coordinates in the real world (i.e., location in a geographic coordinate system) or pixel coordinates within an image (e.g., a raster image or a point cloud image).

A virtual map, which includes information such as the locations of road users at any given time, can be created and used, for example, to calculate estimated time of arrival (ETA) of different users to different locations. Such a virtual map can be used to efficiently control traffic and in a myriad of safety applications, for example:Warning of collision with objects that aren't in the road user's field of view;Optimizing distance to the next car—by knowing the speed, acceleration and distance to nearby vehicles a connected and autonomous vehicle (CAV) can adapt its own speed and acceleration to keep a safe distance from the nearby vehicles, thereby improving safety and allowing smoother traffic flow;Assisting a CAV with complex maneuvers in urban settings such as left turn movement in signalized intersections in the US (assisted left turn).

The term “road network” refers to the routes and structures used by road users for transportation. For example, roads, highways, junctions, paths, etc., may all be part of the road network.

Infrastructure of a road network includes accessories related to the road network and assisting the road users, such as traffic lights, lighting posts, traffic and other road signs, dynamic message signs (DMS), dynamic lane indicators, etc.

The term V2X used in this description refers generally to communication between all elements on a road network, for example, to communications between road users, between infrastructure and users, between infrastructures, etc.

Although the term “network” in this description and the examples herein all refer to roads, it should be appreciated that the invention relates to any network on which users travel, such as rivers, oceans, air, rails, etc.

In an exemplary system100, which is schematically illustrated inFIG.1A, a processing unit104is in communication with one or more sensors102that can detect a road user and one or more V2X communication modules103that can receive and transmit communication from and to connected road users. The processing unit104receives input (also referred to herein as sensor data) from the sensor102and from the V2X communication module103and can detect and optionally identify, based on the inputs, connected and non-connected road users.

Typically, the inputs from the sensor102and the V2X communication module103include at least the locations of the road users detected by the sensor102and the locations of the connected road users transmitting to the V2X communication module103.

Sensor102may be, for example, optic based, radar based, sonic based or may use other suitable technologies to detect road users. Sensor102may include one or a combination of a camera, radar, lidar and/or other suitable sensors to detect a road user. Sensor102obtains data such as image or other data representing the road user and processing unit104may calculate from the data a location of the road user.

In the exemplary embodiments described herein, the sensor102includes a camera however, other sensors may be used. In one embodiment the sensor102includes a camera containing a CCD or CMOS or another appropriate chip. The camera may be a 2D or 3D camera. Processor104may apply image processing algorithms, such as shape and/or color detection algorithms and/or machine learning models such as convoluted neural networks (CNN) and/or support vector machine (SVM) to detect and possibly classify each road user and may use image processing and tracking algorithms to track each road user to calculate parameters such as location, bearing, speed, acceleration and past and future trajectory of each user.

The V2X communication module103can use suitable communication methods such as DSRC and/or C-V2X/5G to communicate with connected road users. For example, the V2X communication module103may include a DSRC or C-V2X/5G modem to receive data from connected road users using DSRC/C-V2X or fleet telematics (via cellular communication).

The information received from each connected road user typically includes the user's location (in geographic coordinate system), speed, acceleration, bearing, past and predicted future trajectory, similarly to the parameters calculated from the data received from the sensor102. A class (e.g., private car, bus, pedestrian, etc.) and/or identification (e.g., V2X digital certificate, license plate number, etc.) of a road user may also be received via the V2X communication module103.

Processing unit104can generate a signal based on these parameters and send the signal to connected road users and/or road infrastructure, via the V2X communication module103, as further described below.

Processing unit104may include, for example, one or more processors and may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), a microprocessor, a controller, a chip, a microchip, an integrated circuit (IC), or any other suitable multi-purpose or specific processor or controller. Processing unit104may include or may be in communication with a memory unit109. Memory unit109may store at least part of the data received from sensor(s)102and/or the V2X communication module(s)103.

Memory unit109may include, for example, a random access memory (RAM), a dynamic RAM (DRAM), a flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short term memory unit, a long term memory unit, or other suitable memory units or storage units.

In some embodiments the memory unit109stores executable instructions that, when executed by processing unit104, facilitate performance of operations of processing unit104, as described herein.

Components of the system100may be in wired or wireless communication and may include suitable ports and/or network hubs and/or appropriate cabling.

Additionally, system100may include or may be attached to a user interface device having a display, such as a monitor or screen, for displaying e.g., images, virtual maps, instructions and/or notifications (e.g., via text or other content displayed on the monitor). The user interface device may also be designed to receive input from an external user. For example, the user interface device may include a monitor and keyboard and/or mouse and/or touch screen, to enable an external user to interact with the system.

A storage device, connected locally or remotely, e.g., in the cloud, may be used with system100. The storage device may be a server including for example, volatile and/or non-volatile storage media, such as a hard disk drive (HDD) or solid-state drive (SSD). In some embodiments the storage device may include software to receive and manage data input from sensor(s)102and/or V2X communication module(s)103.

As schematically illustrated inFIG.1B, processing unit104receives input from sensor102(step120) and detects total road users from the sensor input (step122). Typically, the group of total users includes all users within the field of view (FOV) of sensor102.

In one embodiment, processor104uses information from the sensor input (namely, the user parameters calculated from the sensor input) to create and maintain a list or other record including identifiers (e.g., a value or other character representing the identity or other parameters of the road user) of the total road users detected in step122. This list includes identifiers of both connected and non-connected users. Typically, a list of total users relates to users at a certain site, which is defined by the FOV of the sensor102.

The processing unit104detects connected road users from input received from the V2X communication module103(step124). Processor104may create and maintain another list or other record based on input from the V2X communication module103. This list includes only connected users transmitting to the V2X communication module103.

Processor104compares total road users to connected users to detect non-connected road users (step126). For example, by comparing or matching the list of total road users to the list of connected road-users, the non-connected users out of the total users can be determined. A road user is considered to be a non-connected road user if there is no connected road user that can be matched to him.

In some embodiments a device may be controlled (e.g., by a signal generated by processing unit104) based on the locations of connected and non-connected road users (step128).

For example, processing unit104may create a message for each non-connected road user detected in step126. The message (e.g. BSM and/or CAM and/or PSM, in current standards) typically contains the calculated user parameters (e.g., location, speed, acceleration, bearing, classification, past and predicted trajectory, etc.) and can be broadcasted via the V2X communication module103modem at the required frequency (e.g. 10 hz for vehicles, 2 hz for pedestrians), to all connected road users in the vicinity of the site.

In another example, which will be further described below, a traffic controller can be controlled by processing unit104in accordance with locations of road users.

Thus, embodiments of the invention enable controlling devices to provide safer and smoother traffic based on locations of both connected and non-connected road users.

Some embodiments described herein enable controlling devices to provide safer and smoother traffic based on other/additional parameters of connected and non-connected road users, such as bearing, speed, acceleration and trajectories of each user.

As described above, processing unit104may create and maintain a list of all road users in vicinity of and approaching a site on the road network, based on inputs from the sensor102and V2X communication module103. The list of total road users is matched to a list of connected road-users and at least the locations of the connected and non-connected road users are determined.

Processing unit104may create a virtual map using the determined locations of the connected and non-connected road users. A virtual map may be created (e.g., calculated) periodically (e.g., at a predetermined frequency). In some embodiments the virtual map may be a dynamic virtual map that is updated periodically.

In one embodiment, by applying object detection and classification algorithms (e.g., using a CNN deep neural network such as YOLO object detection, SSD deep learning or Faster-RCNN) on data input from sensor102, processor104detects and classifies road users and calculates a bounding shape (e.g., a 3D box) for each detected road user, possibly per classification. Accordingly, vehicles may have a different bounding shape than pedestrians, vehicles may have a different bounding shape than trains, etc.

Each bound road user is assigned a tracking ID and is tracked, e.g., by using an object tracking algorithm (such as a Siamese-CNN+RNN, MedianFlow, KLT, etc.).

The pose of each road user, identified by a tracking ID, may be calculated, e.g., based on the direction of each face of the 3D bounding box representing the user.

Parameters of each road user can be calculated based on locations of the user over time. For example, speed can be calculated either directly from radar data and/or by measuring a difference in user locations in images (pixel coordinates) over time.

Acceleration can be calculated by measuring difference in speed over time.

Bearing can be calculated based on the pose of the user and/or based on the angle between two (or more) locations of the same user in two or more different images obtained at different times.

Past trajectory of the user (which can be defined as a list of <location, time> pairs) can be calculated based on locations over time.

Future or predicted trajectory can be calculated using a prediction model (such as a recurrent neural network (RNN)) trained on information including the classification of the road user, past trajectory, speed, acceleration and bearing. The future trajectory can be defined as a list of <location, time> pairs, where time is in the future.

Processor104may then calculate a transformation function (e.g. perspective transformation matrix) that maps the pixel coordinates to a geographic coordinate system. In some cases, processing unit104may calibrate using the location of different known locations in the image, in pixel coordinates, and in the geographic coordinate system (e.g., latitude and longitude). Using a distance measurement function (such as the haversine formula) the distance in meters from two points in the geographic coordinate system can be calculated.

Processing unit104can use the transformation function, e.g., as described above, to create a virtual map from the user parameters calculated from inputs from the sensor102, such as, location and/or pose in pixels/point cloud space, classification, speed, acceleration, bearing and past and predicted trajectory. The map may also include information relating to parameters (such as location, pose, classification, speed, acceleration, bearing and past and predicted trajectory) of connected users, who are not within the FOV of sensor102. This information will typically be received from the V2X communication module103, whereas information relating to parameters of a connected user who is within the FOV of sensor102, will include information from both sensor102and V2X communication module103.

A virtual map200is schematically illustrated inFIG.2. In one embodiment, the virtual map200depicts all the road users215and216with their IDs (ID1and ID2) in a geographic coordinate system. Using, for example, the distance measurement function as described above, and the calculated user parameters, the virtual map200may further be augmented by information215′ and216′ for each road user215and216regarding, for example, the user's location (e.g., in latitude and longitude), speed (e.g., in meters per second), acceleration (e.g., in meters per second squared), bearing (e.g., in angles where 0 is the north), past and predicted trajectory (a list of <location, time> pairs where time might be in the future for the predicted trajectory). Additional parameters, such as class and/or identity (including for example, license plate number, color, shape, etc.), may also be added to the virtual map200. Additional parameters or information that may be added to the virtual map200may include the status of the road user, e.g., connected, non-connected and connected and matched to sensor input.

The virtual map200may include graphic representations of the road network211and of the road network infrastructure212at locations representing their real-world locations. A graphic representation of the road users215and216may be superimposed on the map at appropriate locations.

In some embodiments processing unit104can calculate an estimated time of arrival (ETA) of a specific road user at a real-world location, based on the virtual map200and control a device according to the ETA, as further described below.

As schematically illustrated inFIG.3A, all road users at a site on a road network are detected in step32, based on input from a sensor, such as sensor102. The input from the sensor may include, for example, image data and/or point cloud data.

In step34connected road users are detected based on V2X transmissions, for example, by input from V2X communication module103.

In step36at least one non-connected road user is detected by matching each connected road user (detected in step34) to one of all of the road users (detected in step32), whereby road users that are not matched are determined to be non-connected road users. In one embodiment, matching to determine non-connected users can be done by subtracting a list of connected users from a list of all users.

In step32input from a sensor is analyzed to detect all users. The input from the sensor may include image data and detecting all the road users may include applying object detection algorithms on the image data. In some embodiments the input from the sensor may include data from a radar sensor or lidar sensor (e.g., point cloud data) and detecting all the road users may include using clustering algorithms (such as DBSCAN) or neural networks such CNNs, on the data.

In some embodiments a fine-grained classifier (such as a CNN) can be trained on images of different road users such as vehicles, trains, bicycles, pedestrians, etc. The trained classifier may be used by processing unit104to provide reliable fine-grained classification and identification of road users from image data.

In some embodiments, the matching of each connected road user to one of the road users detected in step32, includes determining that at least one parameter of both road users shows similarity above a threshold.

As schematically illustrated inFIG.3B, a parameter of a road user that was detected from sensor data is determined (step302). If the similarity of the determined parameter to that same parameter of a connected road user, is above a threshold (step304), a match is found (step306). If the similarity is below the threshold, no match is found (step308).

In some embodiments more than one parameter must match above a threshold to confirm a match between two road users.

In one embodiment an object matching algorithm (such as template matching, feature matching, neural network with mapping to a latent vector space and cosine distance loss, etc.) is used to compare parameters of the road users (e.g., location, speed, acceleration, classification and trajectories).

In some cases, parameters of a certain road user determined based on input from the V2X communication module can be compared to the parameters (of that same road user) determined based on input from the sensor. For example, calculating parameters of users from sensor input may include the use of object detection and/or tracking algorithms whereas calculating parameters of connected users which are received from the V2X communication module, include the use of global positioning system (GPS) or inertial measurement unit (IMU) based devices. Comparison between parameters determined by these different techniques enables to determine inherent errors in the input from the V2X communication module and/or errors in calculations based on input from the sensor. In some embodiments the threshold can be set based on the determined inherent errors. For example, the threshold can be set to be above the probability of error (as determined by the determined inherent errors). In other embodiments the threshold is a predetermined threshold.

In a situation where not all road users are connected, connection of road users can be emulated using embodiments of the invention. For example, as schematically illustrated inFIG.3C, all road users at a site on a road network are detected in step312, based on input from a sensor, such as sensor102.

In step314connected road users are detected based on V2X transmissions, for example, by input from V2X communication module103.

In step316at least one non-connected road user is detected by matching all users to connected users, e.g., as described above.

In step318a parameter of a non-connected road user (such as classification, location, bearing, speed, acceleration and past and/or future trajectories) is determined, e.g., as described above.

A message set including the determined parameter is created (step320) and the message set is sent out, e.g., via a V2X communication module, to connected road users and/or to road network infrastructure (step322), enabling the non-connected road user to become “visible” and connected to other users and/or to the infrastructure.

In one embodiment, an example of which is schematically illustrated inFIG.4, a plurality of sensors402are in communication with a control unit406. Control unit406may include a CPU or any other suitable processor and communication capabilities, such as wireless communication capabilities (e.g., Wifi, LoRa, Cellular, etc.) and/or wired communication (e.g., ethernet, fiber, etc.). Additionally, control unit406has V2X communication capabilities.

Control unit406can communicate directly with a road network infrastructure (such as dynamic message signs, dynamic lane indicators, etc.) or via a road network infrastructure controller unit407, which is typically a dedicated computer for controlling infrastructure. For example, each traffic light is connected to a traffic light controller that controls the sequencing and duration of the traffic lights.

The control unit406may also be in communication with one or more V2X communication modules403, which may be located at the same locations of sensors402and/or at appropriate locations to receive and transmit information from and to connected users and/or to the control unit406.

The control unit406can send a signal to the road network infrastructure controller unit407based on the detection of connected and non-connected road users.

In some embodiments the control unit406can communicate with connected road users.

In some embodiments each sensor402and possibly a V2X communication module403and possibly a processing unit, are contained in a single housing401. The housing typically provides stability for sensor402such that it is not moved while obtaining images or other data.

The housing401may be made of durable, practical and safe for use materials, such as plastic and/or metal. In some embodiments the housing401may include one or more pivoting element such as hinges, rotatable joints or ball joints and rotatable arm, allowing for various movements of the housing. For example, a housing can be mounted at a site on a road network to enable several FOVs to the sensor402which is encased within the housing401, by rotating and/or tilting the housing.

In one embodiment, which is schematically illustrated inFIG.5, a network of sensors is deployed at a site on a road network. Each sensor502from the network can be mounted at a different location at the site.

Suitable sites for mounting sensors502include, for example:Intersections, typically signalized intersections, which are a critical part of modern road network and are a decision point and a source of conflict which leads to accidents, especially fatal accidents.Roundabouts, which are an alternative to a signalized intersection which can dramatically reduce fatal accidents but require a considerable amount of landmass.Highway off/on ramps, which are a source of conflict. Ramp metering can also be a decision point which can affect traffic flow.

On a long highway, for example, sensors502can be located in any place suitable to provide a FOV that will cover the highway.

In some embodiments, sensors502are mounted at a location where electricity can be provided and where visibility of the road network is enabled. In other embodiments sensors502and/or other components of the network of sensors, may be mobile and self-powered, e.g., by using solar panels or batteries.

FIG.5shows a typical 4-way intersection500. In this embodiment a sensor502can be installed on each way of the intersection, e.g., on a traffic light mast and/or lighting pole512or any other suitable location that enables full sensor coverage of the center of the intersection and as much coverage (e.g. 200 meters) from the stop line513on each way.

Full sensor coverage means that the sensors502are able to obtain enough data and at a quality to enable detection and classification of road users, e.g., vehicle515.

In this embodiment a control unit506, which may be similar to control unit406described above, is in communication with the sensors network and with a traffic light controller507. The control unit506may also be in communication with V2X communication modules503, which may be located at the same locations of sensors502and/or at appropriate locations to receive and transmit information from and to connected users and/or to the control unit506.

In some embodiments a sensor502and possibly a V2X communication module503can be part of a single unit and several such units in communication with each other and/or in communication with the control unit506, can be located on poles512at the intersection500to provide broader coverage of the intersection.

Control unit506may provide real-time instructions to the traffic light controller507based on inputs from sensors502and V2X communication modules503. This embodiment, which includes using input from one or more sensor502and V2X communication modules503enables relating to all road users at a site (e.g., in vicinity of intersection500) even if they are not connected, providing more accurate and complete control of traffic in order to improve the traffic flow and reduce accidents at the site.

As discussed above, processing unit104can calculate an estimated time of arrival (ETA) of a specific road user at a location, e.g., using a virtual map, and may control a device according to the ETA.

In one embodiment, road network infrastructure can be controlled based on a calculated ETA. For example, authorized road users, such as emergency vehicles (e.g., police cars, fire trucks, ambulances) and public transportation (e.g., buses, trains and ride sharing) can be given priority in signalized intersections to minimize their delay and improve their safety and service level.

In one embodiment, which is schematically illustrated inFIG.6, a road user615is detected in vicinity of intersection600, for example, based on image analysis and/or based on V2X transmissions, as described above. Parameters such as bearing, speed and acceleration of road user615at a first location611can be used to calculate the time it would take the road user615to arrive at a second location613. Typically, the calculations are done using a virtual map, as described above. An ETA of the road user615at the location613on the road network can be generated based on the calculated time.

In one embodiment an ETA historical model (such as RNN) that predicts the ETA of each road user at predetermined locations can be created by taking into account parameters such as speed, acceleration, bearing, classification and past and predicted trajectory.

On top of the historical model, a real-time interaction model (for example a CNN+RNN) based on past and predicted trajectories of all road users from the virtual map takes into account the other road users to further improve the accuracy of the ETA metric.

A control unit607can control a road network infrastructure612, such as a traffic light, based on the generated ETA.

In one example, which is schematically illustrated inFIG.7A, a control unit706controls a road network infrastructure controller (e.g., traffic light controller707) based on prevailing road network rules. The road network rule may include, for example, preemption rules based on municipal or other policies.

In this example, the control unit706receives from virtual map700an indication of a road user, e.g., a road user615, approaching a site, e.g., location613in intersection600. In addition, the ETA of the road user (4 seconds) is provided from virtual map700. Control unit706can identify and classify the road user, e.g., based on inputs from a sensor and/or V2X communication module.

In one embodiment, the road user is an authorized vehicle, namely, a vehicle type authorized to get preemption as defined by city policies. Authorized vehicles may include, for example, emergency vehicles (such as police cars, ambulances, fire trucks) and public transportation vehicles (such as buses and trains). In this case a fine-grained classifier (such as a CNN) can be trained on images of emergency vehicles, public transportation vehicles and other relevant vehicles for preemption. The classifier can be used to provide a reliable fine-grained classification of “authorized vehicles” from data obtained from a sensor, such as image data.

Using the information from virtual map700and fine-grained classification, the control unit706may build an ETA historical model (such as RNN) that predicts the ETA of each “authorized vehicle” to a predetermined location, e.g., to the stop line of an intersection, by taking into account the fine-grained classification of the road user (e.g. bus vs ambulance) and other parameters such as speed, acceleration, bearing and past and predicted trajectories.

The ETA can be added to the information included in virtual map700for each authorized vehicle.

On top of the historical model, a real-time interaction model (for example a CNN+RNN) based on past and predicted trajectories of all road users, takes into account the other road users (e.g. the vehicles in front of the authorized vehicle) to further improve the accuracy of the ETA metric in the virtual map700.

The control unit706can access information from the city's policy722, which determines preemption rules, e.g., which kind of road user has priority over others and when. For example, a bus might be prioritized over a light-rail in the afternoon. The control unit706uses information from virtual map700and from the city's policy722to decide which road user should get priority and therefore which phase of the traffic light controller needs to be served. In one embodiment, the control unit706creates a record (e.g., a list or table or other way of maintaining data) of authorized vehicles sorted by priority and ETA and computes for each authorized vehicle if it can be served without interrupting a higher priority vehicle. For example, consider a case of a light-rail approaching an intersection from the north with an ETA of 10 seconds and a bus approaching from the west with an ETA of 4 seconds. The bus needs 2 seconds to pass the intersection. A city policy of prioritizing light-rail over buses will give priority to the bus even though the light-rail has higher priority, due to the fact that both demands can be served without causing extra delay.

In a different case, where the ETAs for both the bus and the light-rail are similar, then priority will be given to the light-rail in order to minimize delay to the light-rail, as it has higher priority in the city's policy.

The control unit706then controls the traffic light controller707using a preemption signal (e.g. ABC NEMA TS-1, C1 Caltrans, SDLC, NTCIP, etc.) or through a regular call (e.g. using loop emulation, NTCIP call, etc.) in case the traffic light controller707is running in fully-actuated mode.

In some embodiments control unit706may have access to a record of authorized road users721and may compare the identity of the road user with the record of authorized road users721, to determine if the road user is an authorized user.

In one embodiment, which is schematically illustrated inFIG.7B, an authorized user (or other class or identity of a road user) is identified and at least one parameter of the authorized user is calculated (step732). In one embodiment the authorized user is identified based on input from a sensor (e.g., based on image data and/or radar and/or lidar data).

An ETA is calculated for the authorized user, based on the calculated parameter (step734). For example, speed, bearing and acceleration of the identified authorized user can be used to calculate the ETA of the user.

Based on prevailing road network rules736and based on the calculated ETA, road infrastructure can be controlled (step738), for example, to prioritize the authorized road user.

In one embodiment, which is schematically illustrated inFIG.7C, a processing unit receives a preemption message (such as an SRM) from a road user (step742). Typically, the preemption massage is sent from a connected road user.

The road user is identified (step744), for example, based on input from a sensor, and the identified road user is compared to a record, e.g., list of authorized road users (745). If the road user is identified on the list (step746), a road network infrastructure is controlled based on the identification of the user (step748). If the road user is not identified on the list (step744) a signal is generated identifying a malicious road user (step750).

As schematically illustrated inFIG.7D, a signal identifying a malicious road user may cause the road user to be added to a list of suspected malicious users.

In one embodiment, a control unit706receives input from a sensor (e.g., camera) and a V2X communication module. The control unit706is also in communication with a traffic light controller707and has access to several records; a “blacklist”772listing malicious/malfunctioning road users, a “greylist”773listing suspected malicious users, and a “whitelist”774listing confirmed authorized road users (typically a list maintained by the city and/or vehicle manufacturers).

In one embodiment, control unit706receives a preemption request from a connected road user. For example, the connected road user may send an SRM message with its calculated ETA to a location (e.g., location613at intersection600) and indicate it wants priority in a specific part of the road network at a predetermined time. Control unit706may provide preemption for the connected user if the user is listed in the “whitelist”774.

Once it is determined that the connected road user should be within the sensor FOV (e.g., based on the parameters transmitted by the connected user and/or based on its calculated ETA), the connected road user is matched to road users in locations within the sensor FOV, using a virtual map. If no match is found for a predetermined amount of time (e.g., 5 seconds), an identifier of the connected road user (e.g. his V2X digital certificate, license plate number, etc.) is added to the “greylist”773. Information is then sent (e.g. using an email, SMS, NTCIP or any suitable API) by the control unit706to the city's traffic management center (TMC)781and to the Original Equipment Manufacturer—OEM782(e.g. the vehicle manufacturer or operator) regarding a possible malfunction or hacking attempt by the connected road user. The connected road user's identifier may then be moved (at the TMC781and/or OEM782discretion) to the “blacklist”772or to the “whitelist”774.

In another embodiment, the connected road user may be identified or classified (e.g., by using a classifier on image data received from the sensor) as being in the same class (e.g. a bus) that was used in the preemption message. In this case the control unit706can cross-validate the information transmitted from the connected road user by V2X with the sensor information (e.g., by matching the connected road user to a user on a virtual map) and based on a positive match the control unit706can proceed safely with preemption.

If the connected road user is not classified in the same class that was used to request the preemption (e.g. the connected road user is classified as a private car vs a bus) then the connected road user is considered to be malicious and its identifier is added to the “greylist”773for further inspection by the TMC781and/or OEM782, and the preemption is canceled by dropping the call/preemption signal to the traffic light controller707.

In one embodiment, V2X information can be used in order to estimate a number of road users that are outside the sensor FOV in order to predict a future number of users within the FOV, and provide a better decision regarding the traffic signal timing. For example, if it is estimated that 10 vehicles are arriving at a signalized intersection, the green light time may be extended even though there no vehicles currently detected by the sensor.

In one embodiment, which is schematically illustrated inFIG.8, a method is provided for estimating a number of road users at a location on a road network. In this embodiment a processing unit estimates a number of road users arriving at a predetermined location at a certain time, based on inputs from a sensor and a V2X communication module.

In step802a total number of road users at a certain time at a certain site, is calculated, for example, based on input from a sensor mounted on the road network in vicinity of the site.

In step804a number of connected road users at the certain time in vicinity of the site, is calculated, based on input from a V2X communication module.

For example, every connected road user reports its ID, location, speed, bearing, past and predicted trajectory, e.g., using BSM/CAM/PSM messages, via V2X communication, to a processing unit. When a connected road user enters an area of a FOV of a sensor, a match is searched between the connected road user and the road user detected by the sensor. The amount of connected road users and all road users is calculated, as described above.

An adoption rate of V2X technology (defined as percentage of connected road users out of all road users) can be calculated in step806by comparing the total number of road users and the number of connected road users. For example, using the locations of connected road users and total road users, as detected by the sensor, and identifying which of the total road users is a connected road user, an exact measure of the V2X adoption rate can be obtained.

In step808a model to predict a future number of road users at the site, is created using the adoption rate. The model may be built by running an SVM or RNN using the adaption rate calculated in step806, over time. The model may predict the total amount of road users outside the sensor's FOV, based on time of day, observed amount of connected road users, observed amount of all road users, their class (e.g. bus, truck, etc.) and past trajectory of connected road users outside of the sensor FOV.

The prediction model may be refined over time by measuring the prediction error based on the difference between the predicted amount of road users and the actual number of road users as detected by the sensor.

In step810a road network infrastructure may be controlled based on the model.

Thus, according to embodiments of the invention, a processing unit is configured to estimate the number of road users based on an adoption rate (rate of use) of V2X technology.

In some embodiments, behavior parameters of a specific road user can be detected based on a virtual map. For example, a dangerous behavior of a road user and/or a dangerous event can be detected, based on the virtual map.

In one embodiment, an example of which is schematically illustrated inFIG.9A, a method for traffic control includes calculating (e.g., by processing unit104) locations of road users based on sensor data and V2X communication (step92) and creating a virtual map which includes the locations of the road users (step94). A behavior parameter of any specific road user can then be detected from the virtual map (step96). Behavior parameters may include characterizations of the road user's behavior. For example, behavior parameters may include driving directions, acceleration patterns, etc., whereas erratic driving direction, erratic acceleration patters, etc., may indicate dangerous behavior, as further exemplified below.

In step98a signal is generated (e.g., by processing unit104) to control a device based on the detected behavior parameter. For example, the signal may include a V2X communication to other road users to warn them of dangerous behavior of a specific road user. Alternatively, or in addition, the signal may be used to control a road network infrastructure. For example, a traffic light may be controlled to change phases based on detected behavior parameters. Road network infrastructure (e.g., dynamic signs) may be controlled to produce a warning based on the detected behavior parameters.

In some embodiments, an ETA to a real-world location (e.g., a stop line at an intersection) is calculated for the specific road user and for the other road users, based on the virtual map, and a signal is generated based on the ETA.

In some embodiments a probability of a dangerous event (e.g. collision) can be calculated based on the virtual map and based on detected behavior parameters. The signal to control devices (such as a road network infrastructure and/or a V2X communication module to warn other road users) may be generated taking into account the calculated probability.

In some embodiments, input regarding ambient conditions (e.g. weather, lighting) at the real-world locations of the road users and/or at the locations at which they are estimated to arrive, can be received at the processing unit and the probability of a dangerous event can be calculated based on the ambient conditions.

Possibly elements, such as classification of road users (e.g., a heavy-duty truck vs a private vehicle), the weather conditions, time of day, etc., may be weighted and used to determine the probability of a dangerous event.

A more detailed explanation is provided in the description below, exemplifying dangerous behaviors.

In one embodiment, which is schematically illustrated inFIG.9B, there is provided a method for detecting and alerting against red-light-runners.

A control unit906is in charge of deciding which phase (e.g., color of light, direction indicated by light, etc.) of traffic light912is served and for how long. In the example illustrated inFIG.9B, the status of the phase is green. The control unit906maintains the status of all phases, for example, by maintaining a counter to determine how many seconds are left until the phase becomes red.

A virtual map of all road users (including connected road users) is calculated periodically, e.g., at least every 0.1 second (i.e. at 10 hz).

In some embodiments, an image-based weather classifier (such as a CNN) is fed with an image from the sensor902and performs a classification on the weather condition (e.g. light rain, fog, flare from the sun, etc.) of the specific real-world location where the sensor902is installed (e.g., intersection900).

For the phase that is currently on (green) and for every road user (e.g., vehicle915) approaching the traffic light912during the green phase, a probability of crossing the intersection at a red light is calculated based on remaining phase time, location, speed and acceleration of the road user and weather conditions. The probability may be a weighted combination of several elements such as parameters and/or classification of a road user, weather conditions, distance to stop line913and others.

In some embodiments, machine learning algorithms are used to identify dangerous behavior of a road user. For example, a braking prediction model (such as an RNN) may be created based on the time of day, class of road user (pedestrian, private vehicle, truck, etc.), weather conditions (rain, visibility, etc.), speed, acceleration, bearing, past trajectory, distance to stop line913and phase status (e.g., green, yellow). A machine learning model may be trained on data from the specific real-world location (e.g., intersection900) and on a general dataset (e.g., on a database including data from a plurality of intersections).

In another example, a physical model is created (e.g. using classic mechanics). The physical model estimates the braking time (and distance) based on the class of the road user (e.g. a heavy-duty truck vs a private vehicle) which determines the typical deceleration, speed, acceleration, bearing and distance to stop line.

A probability of a road user crossing the intersection900during a red light is based on the relation between breaking time (e.g., as calculated using the models described above) and remaining green time of the phase, calculated using a log function. When the probability of crossing a red light rises above a predefined threshold (e.g. 80% probability of running the red light) a red-light-running (RLR) alert message (e.g. Intersection Collision Avoidance message in SAE J2735) is sent by the control unit906to the connected road users in vicinity of the control unit906. The message is typically sent together with the latest status (i.e. the location, acceleration, bearing, speed, etc.) of the road user and the probability of running the red light.

InFIG.9Ca method for alerting a possible collision, is schematically described.

A virtual map of all road users (including connected road users), e.g., vehicles915and916, is calculated periodically, e.g., at least every 0.1 second (i.e. at 10 hz).

Weather conditions at the location of intersection900are determined, e.g., as described above.

For each road user, e.g., vehicle915, a time to collision (TTC) with every other road user, e.g., vehicle916in his vicinity, is calculated based on the data from the virtual map (location, speed, acceleration of every road user) and the weather conditions. The TTC may be weighted. The weighted TTC may be a combination of parameters, such as:TTC at brake—a standard metric in the traffic engineering world which is calculated based on the distance (calculated using the distance function described above) between two road users, their speed, bearing and an estimation of the breaking time (which may be calculated as described above)Breaking probability, calculated as described above.

When the TTC of a road user rises above a predefined threshold (e.g. 1 second) a collision warning message (e.g. Intersection Collision Avoidance message in SAE J2735) is sent by the control unit906to the connected road users (e.g., vehicle916) in vicinity of the control unit906. The message is typically sent together with the latest status (i.e. the location, acceleration, bearing, speed, etc.) of the road user, e.g., vehicle915.

InFIG.9Da method for alerting road users regarding a dangerous road user is schematically described.

A virtual map of all road users (including connected road users) is calculated periodically, e.g., at least every 0.1 second (i.e. at 10 hz).

Weather conditions at the location of intersection900are determined, e.g., as described above.

A classifier (such as an RNN) for dangerous behavior is trained on a dataset of many road users exhibiting dangerous behavior (e.g. erratic driving directions and/or accelerating patterns, such as, driving out of control, driving in zig zag, not staying in the lanes, pedestrians jumping into the street etc.) and taking into consideration: past trajectory, speed, acceleration, bearing and the class of the road user and location of the other road users. The weather condition classification is also taken into consideration.

The result of the classifier is whether any of the road users are acting normally or exhibiting dangerous behavior and the behavior classification (e.g. out of control) and the confidence in that classification.

When the classifier, using real-time data, classifies that a road user, e.g., vehicle915, is exhibiting a dangerous behavior and the confidence (probability) is above a predefined threshold (e.g. 80%) a warning message is sent by the control unit906together with the latest status of the road user, classified behavior (e.g. out of control) and confidence, to the connected road users, e.g., vehicle916, in the vicinity of the control unit906.

Embodiments of the invention enable using information including locations of connected and non-connected road users in a myriad of solutions to existing and future challenges and opportunities.

Embodiments of the invention bring substantial benefits in terms of safety and comfort, and may also contribute to improved and more granular traffic management, provide a better way to prevent or reduce congestion, and enable fuel savings and reduction of air pollution.