Traffic light state assessment

Systems and method are provided for controlling a vehicle. In one embodiment, a method includes: receiving semantic map data, via a processor, wherein the semantic map data includes traffic light location data, calculating route data using the semantic map data, via a processor; viewing, via a sensing device, a traffic light and assessing a state of the viewed traffic light, via a processor, based on the traffic light location data, and controlling driving of an autonomous vehicle based at least on the route data and the state of the traffic light, via a processor.

INTRODUCTION

The present disclosure generally relates to autonomous vehicles, and more particularly relates to systems and methods for autonomous driving.

An autonomous vehicle is a vehicle that is capable of sensing its environment and navigating with little or no user input. An autonomous vehicle senses its environment using sensing devices such as radar, lidar, image sensors, and the like. The autonomous vehicle system further uses information from global positioning systems (GPS) technology, navigation systems, vehicle-to-vehicle communication, vehicle-to-infrastructure technology, and/or drive-by-wire systems to navigate the vehicle.

Vehicle automation has been categorized into numerical levels ranging from Zero, corresponding to no automation with full human control, to Five, corresponding to full automation with no human control. Various automated driver-assistance systems, such as cruise control, adaptive cruise control, and parking assistance systems correspond to lower automation levels, while true “driverless” vehicles correspond to higher automation levels.

Autonomous vehicles may sense and assess a state of traffic lights at intersections and take appropriate vehicle control action depending upon the traffic light state. However, it can be computationally intensive to look for each traffic light at each intersection and infer which path of travel the traffic light represents amongst a plurality of possibilities. Further, it is important to accurately identify a relevant traffic light.

Accordingly, it is desirable to provide systems and methods that view and assess the state of a relevant traffic light at each intersection along a route with enhanced computing efficiency and/or accuracy. Furthermore, other desirable features and characteristics of the present disclosure will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background.

SUMMARY

Systems and methods are provided for controlling a vehicle.

In one embodiment, an autonomous driving system, includes a guidance system configured to receive semantic map data and to calculate route data using the semantic map data, via a processor. The semantic map data includes traffic light location data. A computer vision system is configured to view and assess a state of a traffic light based on the traffic light location data, via a processor. A vehicle control system is configured to control driving of an autonomous vehicle based at least on the route data and the state of the traffic light, via a processor.

In embodiments, the route data includes a labelled intersection lane and the traffic light location data is associated with the labelled intersection lane.

In embodiments, the semantic map maps an intersection using a plurality of labelled intersection lanes defining paths for traversing the intersection. Each labelled intersection lane includes a traffic light label defining traffic light location data. Traffic light labels of at least some of the labelled intersection lanes identify different traffic lights from each other.

In embodiments, the semantic map includes a labelled lane entering an intersection and at least first and second labelled lanes exiting the intersection. A first labelled intersection lane defines a path connecting the labelled lane entering the intersection and the first labelled lane exiting the intersection and a second labelled intersection lane defines a path connecting the labeled lane entering the intersection and the second labelled lane exiting the intersection. Different traffic light labels are associated with the first and second labelled intersection lanes, respectively, identifying different traffic lights and defining different traffic light location data.

In embodiments, the semantic map includes labelled lanes entering and exiting an intersection, labelled intersection lanes defining paths connecting labelled lanes entering the intersection and labelled lanes exiting the intersection, labelled anchor points where lanes enter the intersection and labelled traffic lights, wherein each labelled anchor point is linked to a labelled traffic light, wherein each labelled intersection lane is associated with a labelled traffic light.

In embodiments, the semantic map includes traffic light identifiers, traffic light type data, intersection lane identifiers, identifiers for lanes entering the intersection and/or identifiers for lanes exiting the intersection.

In embodiments, the route data includes lane identifiers identifying lanes to be used along the route and intersection lane identifiers identifying intersection lanes to be used along the route.

In embodiments, the system includes a data storage device storing the semantic map in the autonomous vehicle.

In embodiments, the traffic light location data defines a location of the traffic light in three spatial dimensions including height. In embodiments, the traffic light location data includes orientation of the traffic light. In embodiments, the traffic light location data includes six degrees of freedom position information. In embodiments, the traffic light location data includes estimates for x, y and z coordinates as well as roll, pitch and yaw.

In embodiments, the computer vision system is configured to control a field of view of a sensing device based on the traffic light location data and/or select a portion of imaging data obtained by at least one sensing device that contains the traffic light based on the traffic light location data, whereby the computer vision system is configured to assess the state of the traffic light by focusing, e.g. focusing image processing, on the traffic light in the selected portion of imaging data.

In embodiments, the computer vision system is configured to receive visual data of the traffic light from a sensing device and is configured to assess a state of the traffic light including a stop and go state. The vehicle control system is configured to control the autonomous vehicle to commence going, to continue going or to stop before the traffic light depending upon the state of the traffic light.

In embodiments, the system includes a positioning system configured to determine position data representing a current position of the autonomous vehicle.

In embodiments, the computer vision system is configured to visualize an upcoming traffic light based on the position data which identifies a position of the vehicle relative to the semantic map, the route data which identifies the upcoming traffic light and the traffic light location data associated with an intersection lane being followed according to the route data. In embodiments, the computer vision system commences visualizing and assessing the state of the traffic light based on geometric (e.g. line of sight) calculations, the positioning data and the traffic light location data determining when the traffic light is first able to visualized or based on a data label in the semantic map identifying where the traffic light is first able to be visualized or based on being within a predetermined distance of the traffic light as determinable from the positioning data and the traffic light location data.

In embodiments, the computer vision system is configured to view the traffic light and/or concentrate image processing on the traffic light based on the current position of the vehicle and the traffic light location data.

In another embodiment, a computer implemented method of autonomous driving includes receiving semantic map data, via a processor. The semantic map data includes traffic light location data. The method includes calculating route data using the semantic map data, via a processor. The method includes viewing, via at least one sensing device, a traffic light and assessing a state of the viewed traffic light, via at least one processor, based on the traffic light location data. The method includes controlling driving of an autonomous vehicle based at least on the route data and the state of the traffic light, via a processor.

In embodiments, the semantic map includes a labelled lane entering an intersection and at least first and second labelled lanes exiting the intersection. A first labelled intersection lane defines a path connecting the labelled lane entering the intersection and the first labelled lane exiting the intersection and a second labelled intersection lane defines a path connecting the labeled lane entering the intersection and the second labelled lane exiting the intersection. Different traffic light labels are associated with the first and second labelled intersection lanes, respectively, which identify different traffic lights and define different traffic light location data.

In embodiments, the traffic light location data defines a location of the traffic light in three spatial dimensions including height. In embodiments, the traffic light location data includes six degrees of freedom position information. In embodiments, the traffic light location data includes estimates for x, y and z coordinates as well as roll, pitch and yaw.

In embodiments, the method includes controlling a field of view of a sensing device based on the traffic light location data and/or selecting a portion of imaging data obtained by at least one sensing device that contains the traffic light, whereby the computer vision system is configured to assess the state of the traffic light by focusing on the traffic light in the selected portion of imaging data.

In embodiments, the method includes tracking a location of the autonomous vehicle in the semantic map based on current position data for the autonomous vehicle, extracting traffic light location data from an upcoming labelled intersection lane in the semantic map that is to be followed according to the route data, and configuring the sensing device to view the traffic light based on the extracted traffic light location data.

In one embodiment, an autonomous vehicle includes a data storage device storing a semantic map, a guidance system configured to receive semantic map data from the semantic map and configured to calculate route data using the semantic map data, via a processor. The semantic map data includes traffic light location data. A computer vision system is configured to view and assess a state of a traffic light based on the traffic light location data, via a processor. A vehicle control system is configured to control driving of the autonomous vehicle based at least on the route data and the state of the traffic light, via a processor.

In embodiments, the semantic map includes a labelled lane entering an intersection and at least first and second labelled lanes exiting the intersection. A first labelled intersection lane defines a path connecting the labelled lane entering the intersection and the first labelled lane exiting the intersection and a second labelled intersection lane defines a path connecting the labeled lane entering the intersection and the second labelled lane exiting the intersection. Different traffic light labels are associated with the first and second labelled intersection lanes, respectively, identify different traffic lights and define different traffic light location data.

In embodiments, a positioning system is configured to determine current position data for the autonomous vehicle and configured to track a location of the autonomous vehicle in the semantic map based on the current position data for the autonomous vehicle. The computer vision system is configured to extract traffic light location data from an upcoming labelled intersection lane in the semantic map that is to be followed according to the route data. The computer vision system is further for configuring the sensing device to view the traffic light and/or focusing image processing on the traffic light based on the extracted traffic light location data.

DETAILED DESCRIPTION

With reference toFIG. 1, a system shown generally at100is associated with a vehicle10in accordance with various embodiments. In general, the system100uses traffic light data stored in a semantic map and intelligently controls the vehicle10based thereon.

As depicted inFIG. 1, the vehicle10generally includes a chassis12, a body14, front wheels16, and rear wheels18. The body14is arranged on the chassis12and substantially encloses components of the vehicle10. The body14and the chassis12may jointly form a frame. The wheels16-18are each rotationally coupled to the chassis12near a respective corner of the body14.

In various embodiments, the vehicle10is an autonomous vehicle and the system100is incorporated into the autonomous vehicle10(hereinafter referred to as the autonomous vehicle10). The autonomous vehicle10is, for example, a vehicle that is automatically controlled to carry passengers from one location to another. The vehicle10is depicted in the illustrated embodiment as a passenger car, but it should be appreciated that any other vehicle including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), marine vessels, aircraft, etc., can also be used. In an exemplary embodiment, the autonomous vehicle10is a so-called Level Four or Level Five automation system. A Level Four system indicates “high automation”, referring to the driving mode-specific performance by an automated driving system of all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request to intervene. A Level Five system indicates “full automation”, referring to the full-time performance by an automated driving system of all aspects of the dynamic driving task under all roadway and environmental conditions that can be managed by a human driver.

As shown, the autonomous vehicle10generally includes a propulsion system20, a transmission system22, a steering system24, a brake system26, a sensor system28, an actuator system30, at least one data storage device32, at least one controller34, and a communication system36. The propulsion system20may, in various embodiments, include an internal combustion engine, an electric machine such as a traction motor, and/or a fuel cell propulsion system. The transmission system22is configured to transmit power from the propulsion system20to the vehicle wheels16-18according to selectable speed ratios. According to various embodiments, the transmission system22may include a step-ratio automatic transmission, a continuously-variable transmission, or other appropriate transmission. The brake system26is configured to provide braking torque to the vehicle wheels16-18. The brake system26may, in various embodiments, include friction brakes, brake by wire, a regenerative braking system such as an electric machine, and/or other appropriate braking systems. The steering system24influences a position of the of the vehicle wheels16-18. While depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system24may not include a steering wheel.

The sensor system28includes one or more sensing devices40a-40nthat sense observable conditions of the exterior environment and/or the interior environment of the autonomous vehicle10. The sensing devices40a-40ncan include, but are not limited to, radars, lidars, global positioning systems, optical cameras40a, thermal cameras, ultrasonic sensors, inertial measurement units, and/or other sensors. The actuator system30includes one or more actuator devices42a-42nthat control one or more vehicle features such as, but not limited to, the propulsion system20, the transmission system22, the steering system24, and the brake system26. In various embodiments, the vehicle features can further include interior and/or exterior vehicle features such as, but are not limited to, doors, a trunk, and cabin features such as air, music, lighting, etc. (not numbered).

The data storage device32stores data for use in automatically controlling the autonomous vehicle10. In various embodiments, the data storage device32stores defined maps100of the navigable environment. In various embodiments, the defined maps100may be predefined by and obtained from a remote system (described in further detail with regard toFIG. 2). For example, the defined maps100may be assembled by the remote system and communicated to the autonomous vehicle10(wirelessly and/or in a wired manner) and stored in the data storage device32. As can be appreciated, the data storage device32may be part of the controller34, separate from the controller34, or part of the controller34and part of a separate system.

In embodiments, the map101is a semantic map101created from a set of labeled (e.g. human labelled) LiDAR maps. That is, mapping vehicles first obtain point cloud or LiDAR maps and the semantic map101is derived from the LiDAR maps. In one example, the semantic map101is encoded with detailed information like driveable areas, lane types, route possibilities through intersections (intersection lanes), traffic light location data, traffic light type data and pedestrian walkways, and is maintained in a Postgres database on the vehicle10. The semantic map101includes a two-dimensional map that is comprised of labels, where the labels have been made based on obtained LiDAR data, with information such as drivable areas, possible routes through intersections, and traffic light data (including traffic light location and type data) added to the two-dimensional map through semantic layers. Labeling is, in some embodiments, performed through a mix of automated processes and human manual annotation.

In various embodiments, one or more instructions of the controller34are embodied in the system100and, when executed by the processor44, retrieve traffic light location data from the semantic map (or maps)100, directs image processing of images obtained by at least one visual sensing device40abased on the traffic light location data, assesses a state of the traffic light and controls the vehicle10according to the state of the traffic light.

With reference now toFIG. 2, in various embodiments, the autonomous vehicle10described with regard toFIG. 1may be suitable for use in the context of a taxi or shuttle system in a certain geographical area (e.g., a city, a school or business campus, a shopping center, an amusement park, an event center, or the like) or may simply be managed by a remote system. For example, the autonomous vehicle10may be associated with an autonomous vehicle based remote transportation system.FIG. 2illustrates an exemplary embodiment of an operating environment shown generally at50that includes an autonomous vehicle based remote transportation system52that is associated with one or more autonomous vehicles10a-10nas described with regard toFIG. 1. In various embodiments, the operating environment50further includes one or more user devices54that communicate with the autonomous vehicle10and/or the remote transportation system52via a communication network56.

The communication network56supports communication as needed between devices, systems, and components supported by the operating environment50(e.g., via tangible communication links and/or wireless communication links). For example, the communication network56can include a wireless carrier system60such as a cellular telephone system that includes a plurality of cell towers (not shown), one or more mobile switching centers (MSCs) (not shown), as well as any other networking components required to connect the wireless carrier system60with a land communications system. Each cell tower includes sending and receiving antennas and a base station, with the base stations from different cell towers being connected to the MSC either directly or via intermediary equipment such as a base station controller. The wireless carrier system60can implement any suitable communications technology, including for example, digital technologies such as CDMA (e.g., CDMA2000), LTE (e.g., 4G LTE or 5G LTE), GSM/GPRS, or other current or emerging wireless technologies. Other cell tower/base station/MSC arrangements are possible and could be used with the wireless carrier system60. For example, the base station and cell tower could be co-located at the same site or they could be remotely located from one another, each base station could be responsible for a single cell tower or a single base station could service various cell towers, or various base stations could be coupled to a single MSC, to name but a few of the possible arrangements.

Apart from including the wireless carrier system60, a second wireless carrier system in the form of a satellite communication system64can be included to provide uni-directional or bi-directional communication with the autonomous vehicles10a-10n. This can be done using one or more communication satellites (not shown) and an uplink transmitting station (not shown). Uni-directional communication can include, for example, satellite radio services, wherein programming content (news, music, etc.) is received by the transmitting station, packaged for upload, and then sent to the satellite, which broadcasts the programming to subscribers. Bi-directional communication can include, for example, satellite telephony services using the satellite to relay telephone communications between the vehicle10and the station. The satellite telephony can be utilized either in addition to or in lieu of the wireless carrier system60.

A land communication system62may further be included that is a conventional land-based telecommunications network connected to one or more landline telephones and connects the wireless carrier system60to the remote transportation system52. For example, the land communication system62may include a public switched telephone network (PSTN) such as that used to provide hardwired telephony, packet-switched data communications, and the Internet infrastructure. One or more segments of the land communication system62can be implemented through the use of a standard wired network, a fiber or other optical network, a cable network, power lines, other wireless networks such as wireless local area networks (WLANs), or networks providing broadband wireless access (BWA), or any combination thereof. Furthermore, the remote transportation system52need not be connected via the land communication system62, but can include wireless telephony equipment so that it can communicate directly with a wireless network, such as the wireless carrier system60.

Although only one user device54is shown inFIG. 2, embodiments of the operating environment50can support any number of user devices54, including multiple user devices54owned, operated, or otherwise used by one person. Each user device54supported by the operating environment50may be implemented using any suitable hardware platform. In this regard, the user device54can be realized in any common form factor including, but not limited to: a desktop computer; a mobile computer (e.g., a tablet computer, a laptop computer, or a netbook computer); a smartphone; a video game device; a digital media player; a piece of home entertainment equipment; a digital camera or video camera; a wearable computing device (e.g., smart watch, smart glasses, smart clothing); or the like. Each user device54supported by the operating environment50is realized as a computer-implemented or computer-based device having the hardware, software, firmware, and/or processing logic needed to carry out the various techniques and methodologies described herein. For example, the user device54includes a microprocessor in the form of a programmable device that includes one or more instructions stored in an internal memory structure and applied to receive binary input to create binary output. In some embodiments, the user device54includes a GPS module capable of receiving GPS satellite signals and generating GPS coordinates based on those signals. In other embodiments, the user device54includes cellular communications functionality such that the device carries out voice and/or data communications over the communication network56using one or more cellular communications protocols, as are discussed herein. In various embodiments, the user device54includes a visual display, such as a touch-screen graphical display, or other display.

The remote transportation system52includes one or more backend server systems, which may be cloud-based, network-based, or resident at the particular campus or geographical location serviced by the remote transportation system52. The remote transportation system52can be manned by a live advisor, or an automated advisor, or a combination of both. The remote transportation system52can communicate with the user devices54and the autonomous vehicles10a-10nto schedule rides, dispatch autonomous vehicles10a-10n, and the like. In various embodiments, the remote transportation system52stores account information such as subscriber authentication information, vehicle identifiers, profile records, behavioral patterns, and other pertinent subscriber information.

In accordance with a typical use case workflow, a registered user of the remote transportation system52can create a ride request via the user device54. The ride request will typically indicate the passenger's desired pickup location (or current GPS location), the desired destination location (which may identify a predefined vehicle stop and/or a user-specified passenger destination), and a pickup time. The remote transportation system52receives the ride request, processes the request, and dispatches a selected one of the autonomous vehicles10a-10n(when and if one is available) to pick up the passenger at the designated pickup location and at the appropriate time. The remote transportation system52can also generate and send a suitably configured confirmation message or notification to the user device54, to let the passenger know that a vehicle is on the way.

As can be appreciated, the subject matter disclosed herein provides certain enhanced features and functionality to what may be considered as a standard or baseline autonomous vehicle10and/or an autonomous vehicle based remote transportation system52. To this end, an autonomous vehicle and autonomous vehicle based remote transportation system can be modified, enhanced, or otherwise supplemented to provide the additional features described in more detail below.

In accordance with various embodiments, the controller34implements an autonomous driving system (ADS)70as shown inFIG. 3. That is, suitable software and/or hardware components of the controller34(e.g., the processor44and the computer-readable storage device46) are utilized to provide an autonomous driving system70that is used in conjunction with vehicle10.

In various embodiments, the instructions of the autonomous driving system70may be organized by function, module, or system. For example, as shown inFIG. 3, the autonomous driving system70can include a computer vision system74, a positioning system76, a guidance system78, and a vehicle control system80. As can be appreciated, in various embodiments, the instructions may be organized into any number of systems (e.g., combined, further partitioned, etc.) as the disclosure is not limited to the present examples.

In various embodiments, the computer vision system74synthesizes and processes sensor data and predicts the presence, location, classification, and/or path of objects and features of the environment of the vehicle10. In various embodiments, the computer vision system74can incorporate information from multiple sensors, including but not limited to cameras, lidars, radars, and/or any number of other types of sensors.

The positioning system76processes sensor data along with other data to determine a position (e.g., a local position relative to a map, an exact position relative to lane of a road, vehicle heading, velocity, etc.) of the vehicle10relative to the environment. The guidance system78processes sensor data along with other data to determine a path for the vehicle10to follow. The vehicle control system80generates control signals for controlling the vehicle10according to the determined path.

In various embodiments, the controller34implements machine learning techniques to assist the functionality of the controller34, such as feature detection/classification, obstruction mitigation, route traversal, mapping, sensor integration, ground-truth determination, and the like.

As mentioned briefly above, the system100ofFIG. 1is included within the ADS70, for example, as will be described in the following. Generally, the autonomous driving system is configured to retrieve traffic light location data from the semantic map101, the computer vision system74is configured to direct image processing of images obtained by the at least one sensing device40abased on the traffic light location data, to assess a state of an associated traffic light, and the vehicle control system80is configured to control the vehicle10based on the state of the traffic light.

For example, as shown in more detail with regard toFIG. 4and with continued reference toFIG. 3, the autonomous driving system70is configured to retrieve semantic map data112from semantic map101stored in storage device32of the autonomous vehicle10. Other embodiments are envisaged in which the map data112is retrieved from a remote storage device via a wireless network.

Referring toFIG. 5 (a), there is illustrated a semantic map101including a two-dimensional base map122and at least one semantic layer120spatially aligned with the base map122. That is, the semantic features in the semantic layer120are geographically aligned with coordinates of the base map122. The at least one semantic layer can be in the form of an overlay. The at least one semantic layer120includes labels for various road features as described heretofore. Of particular relevance to the present disclosure are labels associated with an intersection including traffic lights. In the illustrated embodiment, the at least one semantic layer includes traffic light labels1100-1107, labels for normal lanes entering the intersection1200-1203, labels for normal lanes exiting the intersection1210-1213, and intersection lanes1300defining all possible (allowed per road and driving rules) driving paths connecting the lanes entering the intersection1200-1203and the lanes exiting the intersection1210-1213. Further, included in the at least one semantic layer120are anchor points6000-6003-6003provided at a defined entrance to the intersection where a normal lane1200-1203entering the intersection transitions to intersection lanes1300. The at least one semantic layer includes connector labels1220connecting each anchor point6000-6003to relevant traffic lights. In the exemplary embodiment, the at least one semantic layer120includes traffic light data124(e.g. traffic light identifier and/or location data124) associated with each intersection lane1300. Traffic light location data is either taken directly from the traffic light data or is obtained indirectly through the traffic light identifier. In embodiments, the traffic light location data includes x, y and z coordinates as well as pitch, yaw and roll data.

In embodiments, the labels of the at least one semantic layer include at least one of lines, identifiers, location data, etc. In the illustrated embodiment, the labelled normal lanes entering and exiting the intersection1200-1203,1210-1213and the labelled intersection lanes1300are labelled to include lane identifiers uniquely identifying the lane in the semantic map101, directional information (e.g. a direction that a vehicle is allowed to travel when following the lane), location information and lines defining a path of travel for a vehicle following the lane. Some labelled intersection lanes1300are such that more than one (e.g. two or three) intersection lanes extend from a single lane entering the intersection1200-1203to connect to respective lanes exiting the intersection1210-1213defining, respectively, paths that turn left, turn right and/or go straight. In embodiments, the anchor points6000-6003-6003are each connected to one or more traffic lights1100-1107. The anchor points6000-6003are only connected to traffic lights that are relevant to traffic flow from that anchor point6000and the associated lane entering the intersection1200-1203. In the illustrated embodiment, the labelled connections1220between anchor point6000and traffic lights1105-1106are made through connecting lines such as straight lines. In one embodiment, the traffic lights1100-1107are labelled with traffic light identifiers uniquely identifying each traffic light at the intersection, are labelled with two-dimensional location information relative to the base map122and are labelled with height from road surface information. In a further embodiment, orientation information (e.g. pitch, yaw and roll) is included with the traffic light labels1100-1107in addition to three-dimensional location data. Further, in some embodiments, the traffic lights are labelled with traffic light type data. Different traffic light types are embodied in a type parameter associated with each traffic light. For example, regular Red/Yellow/Green traffic lights are of a different type to Red/Yellow/Green/Green Arrow Left traffic lights.

In embodiments, each intersection lane1300is associated, through association labels, with at least one traffic light1101-1107such that different intersection lanes1300will be associated with different traffic light data124depending upon which traffic lights are relevant for travel along that intersection lane. In one example, the association between traffic light1101-1107label and intersection lane1300label is through embedding a traffic light identifier or other association label with the intersection lane1300label. Alternatively, traffic light location data124(in three dimensions) is embedded in the label for the intersection lane1300.

In various embodiments, the at least one semantic layer120is built by a process as follows. Traffic lights are placed in precise x, y and z space by a mix of manual and automated processes and traffic light labels1100-1107are constructed at that location in the map that include three-dimensional traffic light location data, traffic light identifier data and optional traffic light orientation data. Each traffic light label1100-1107is linked to labelled intersection entry or anchor point6000for all lanes1200-1203that would use the traffic light(s) for guidance. The intersection in the semantic map101includes line strings representing all possible paths of travel that emanate from an anchor point6000, thereby providing intersection lanes1300. Traffic light associations in the form of traffic light data124are automatically populated into these individual paths of travel or intersection lanes1300, based on the anchor points6000-6003and turn type (e.g. a light with red/yellow/green left arrows is only linked to intersection paths of travel that are assigned a “left” turn type).

Returning toFIGS. 3 and 4, whilst continuing to refer toFIG. 5A, use of the semantic map101will be further described. Computer vision system74is configured to retrieve or otherwise receive semantic map data112from the semantic map101as the autonomous vehicle10approaches an intersection. Semantic map data112includes traffic light location data124as part of labels for traffic lights1100-1107and/or as part of labels for intersection lanes1300, as described above. The traffic light location data124included in the semantic map data112allows the computer vision system74to direct image processing to a selected part of a field of view of at least one sensing device40a. In an embodiment, the at least one sensing device40aincludes one or more optical cameras. In various embodiments, computer vision system74is configured to generate camera (or other sensing device) control commands108based on traffic light location data124for controlling at least one of size of field of view, direction of field of view (e.g. panning movements including yaw, tilt and/or roll) and camera zoom. In this way, traffic lights are efficiently identified by the computer vision system74based on the traffic light location data124included in the semantic map101and are visualized by the sensing device40ato allow traffic light assessment by the computer vision system74. In other embodiments, the computer vision system74does not direct the at least one sensing device and corresponding control commands108are not generated. Instead, the computer vision system74focuses image processing on a selected part of captured images based on the traffic light location data124.

In embodiments, the computer vision system74retrieves traffic light type data as part of labels for traffic lights1100-1107. The computer vision system74is configured to perform image processing techniques to compare the type of traffic light identified by traffic light type data with the visual data captured by the at least one sensing device40ato ensure a match, thereby allowing false positives to be reduced. When the visual data does not match the traffic light type, the computer vision system74is configured to continue a neighborhood search for the true traffic light of relevance.

In various embodiments, the at least one sensing device40avisualizes the traffic light based on the control commands108and generates sensed data106, generally in the form of images or video data of at least the traffic light and possibly also neighboring areas. Computer vision system74receives the sensed data106and includes a traffic light assessment module104that processes the sensed data106to assess a state of the traffic light (e.g. red for stop, red and amber to commence going, green for go, amber for commence stopping). The traffic light assessment module104uses image processing techniques and traffic light assessment algorithms to assess the state of the traffic light, in various embodiments. The traffic light assessment module104outputs traffic light state data107, representing any of the possible traffic light states, to the vehicle control system80, as will be described further below. In an alternative or additional embodiment to that shown inFIG. 4, the computer vision system74is configured to focus on a particular traffic light not necessarily by controlling a field of view of at least one sensing device40a, but by localizing a portion of images obtained by the at least one sensing device40athat contains the traffic light. The localized portion of the images is determinable by transforming the real-world coordinates of the traffic light obtained from traffic light location data124in the semantic map data112to image space. In various embodiments, the transformation of real world coordinates to image space makes use of calibration data of the position of the camera40acapturing images of the traffic light relative to the vehicle10, in addition to roll, pitch and yaw of the vehicle10. The roll, pitch and yaw of the vehicle10information is available from the positioning system76. The calibration data is obtainable from the at least one storage device32. The traffic light assessment module104is able to assess just the traffic light contained in the localized portion to determine the traffic light state data107.

In an exemplary embodiment, the computer vision system74is configured to receive positioning data118from positioning system76. Positioning system76is able to determine position of the vehicle10in six degrees of freedom based on GPS data and from lidar-based scan matching techniques. At a general level, visual odometry localizes the vehicle10relative to a three-dimensional map stored in storage device by comparing features derived from a captured three-dimensional point cloud (e.g. a LiDAR point cloud) with corresponding features in the three-dimensional map. In this way, the vehicle10is localized in the three-dimensional map. The positioning system76is configured to translate the localized position in the three-dimensional map to a localized position in the two-dimensional semantic map101as part of positioning data118. From this, relevant traffic light location data124(and other traffic light data such as traffic light type) is obtainable from semantic map data112. Based on the vehicle's current localized position in three dimensions, the three-dimensional position of the traffic light obtained from the traffic light location data124and extrinsic camera calibration data, location of the traffic light in image space is determined by the computer vision system74.

In embodiments, the computer vision system74is configured to select a region of interest of images obtained by the at least one sensing device40acorresponding to the traffic light location in image space determined based on traffic light location data124as described above. Based on geometric considerations, the computer vision system74will select a larger region of interest as the vehicle nears the traffic light. The region of interest will be used in subsequent processing by the traffic light assessment module104.

In various embodiments, the traffic light assessment module104is configured to scale the region of interest to a fixed scale and to run the scaled image data through a neural network configured to identify the state of the traffic light. The traffic light assessment module104is, in some embodiments, configured to check that the traffic light type, as known from traffic light type data included in the semantic map data112, matches the traffic light included in the region of interest. This matching process may be performed using the neural network. In the event of a false positive, the computer vision system74is configured to search in the neighborhood of the region of interest or to base the traffic light assessment on subsequent or previous frames of image data that do not return a false positive.

Referring toFIG. 5B, an example view from a front camera of the autonomous vehicle10included in the at least one sensing device40ais shown. The view is representative of images taken by the front camera of an autonomous vehicle10located at anchor point6000ofFIG. 5A. Two traffic lights labelled as1105,1106in the semantic map101ofFIG. 5Aare visualized by the front camera. In this example, both traffic lights are relevant to going straight ahead and will be associated with the corresponding label for an intersection lane1300following a straight-ahead path in the semantic map. The computer vision system74is configured to extract traffic light location data124included in the semantic map101and associated with the straight-ahead intersection lane and to configure a field of view of the front camera and/or to focus on the traffic light in the images obtained by the camera based on the traffic light location data124, as has been described above. The traffic light location data124in the semantic map101includes three-dimensional location data. Based on a current position of the autonomous vehicle10, which is known from position data118as described below, the traffic light location data124and geometric calculations, the computer vision system74is configured to determine, relative to the vehicle10, the location of the traffic light. In embodiments, the computer vision system74is configured to determine a line of sight of the traffic light based on the location of the traffic light relative to the vehicle and/or is configured to determine a location of the traffic light in image space. By using traffic light location data124to determine a position in image space of the traffic light, traffic lights can be identified and assessed with enhanced accuracy. In particular, systems and methods described herein allow false positive predictions of traffic lights to be filtered out. For example, a tail light could, in theory, be confused with a tail light, but for the fact that the tail light will not correspond with the location of the traffic light according to the traffic light location data124, thereby ruling out or avoiding such a false determination. Further, since it is known from traffic light data124that there should be a traffic light controlling a local intersection, the system can ensure that the vehicle10is controlled in a conservative manner to assume that there is a traffic light, even if one cannot yet be visualized (e.g. because of an obstruction), and proceed through the intersection accordingly. Further, computer processes are made efficient for locating each traffic light at each intersection and for inferring which paths of travel the traffic light represents.

Taking the example ofFIG. 5Bin greater detail, the autonomous driving system70is configured to track the location of the vehicle10along a lane segment1200entering an intersection. The autonomous driving system70is configured to follow the lane segment1200according to the route data114. The autonomous driving system70is configured to retrieve the anchor point6000from the semantic map data101that accords with the route defined by the route data114. The anchor point6000is associated with two traffic lights1105,1106controlling the traffic through the intersection for that lane segment1200. Based on the traffic lights1105,1106associated with the relevant anchor point6000, traffic light location data124for each traffic light1105,1106can be extracted from the semantic map101. The computer vision system74is configured to capture images of the upcoming intersection and a processor thereof is configured to determine local boxes (or other boundaries) that are a small portion of the captured images localized in image space based on the three dimensional traffic light location data124and real space to image space transformation processing. The traffic light assessment module104focuses on small portions of the captured images corresponding to the locations of the traffic lights1105,1106, thereby reducing chance of false positives and enhancing processing efficiency. The vehicle control system80controls the vehicle appropriately according to the traffic light signal.

Based on semantic map data112, positioning data118for the autonomous vehicle10relative to the semantic map101and route data114defining a route to be followed by the autonomous vehicle10, the computer vision system74is able to determine upcoming traffic lights and to commence searching for the relevant traffic light at an appropriate location. That is, route data114defines an upcoming route including normal lanes to follow and intersection lanes. The intersection lanes are associated with traffic light data124in labels in the semantic map data112, as has been described above with respect toFIG. 5A. The computer vision system74is configured, in various embodiments, to initiate visualizing a traffic light according to processes described above when positioning data118, route data114and semantic map data112indicates an approaching, e.g. within a predetermined distance from a current location of the autonomous vehicle10, traffic light. In embodiments, the route data114includes an identifier of an upcoming intersection lane. From the semantic map data112, traffic light data124is associated with the intersection lane, which either includes or is associated with traffic light location data124. The positioning data118allows tracking, e.g. through the positioning system76, of the autonomous vehicle10along the route defined by the route data114and allows calculation of a distance from the traffic light to be calculated. When a proximity to the traffic light is sufficiently close (e.g. as determined by a threshold), the computer vision system74is configured to visualize or aim to visualize the traffic light, or to begin image processing to find and select the traffic light, associated with the intersection lane and begin assessing the state of the traffic light. In additional or alternative embodiments, the semantic map101includes labels in the at least one semantic layer, e.g. associated with the intersection lanes, describing a first line of sight location for the traffic light so that the computer vision system74commences visualizing the relevant traffic light at the earliest possible location along the route.

Positioning data118is obtained through the positioning system76and route data114is obtained through the guidance system78, as will be further described below.

In various embodiments, the positioning system76is configured to determine a location of the autonomous vehicle10based on sensor inputs. In embodiments, the positioning system76is configured to receive global positioning data from a global positioning receiver and sensor imaging from at least one sensor device40a-40n(e.g. a lidar sensor device) to localize a position of the autonomous vehicle relative to the semantic map101. The positioning system76is configured to receive three-dimensional map data and match features obtained through sensor imaging to features in the three-dimensional map data to accurately locate the autonomous vehicle relative to the three-dimensional map. Since the three-dimensional map and the semantic101are calibrated to one another, the positing system76is able to generate positioning data118representing the location of the vehicle10in the semantic map101.

In various embodiments, guidance system78includes a router110configured to use semantic map data112and run a routing algorithm to plot a route from a start location to an end location. The start location may be a current location of the autonomous vehicle10. In embodiments, the end location is a destination entered by an occupant of the autonomous vehicle10or a destination received from a remote source through a wireless network. In embodiments, the route data114comprises a string of lane identifiers including normal lanes and intersection lanes. By following the route defined by the route data114along the semantic map101, labelled data associated with that route is extractable from the semantic map101such as traffic light location data124associated with labelled intersection lanes1300.

In embodiments, the vehicle control system80utilizes the route data114and semantic map data112and algorithmically determines upon actuation commands116to follow the route defined by the route data114. Actuator system30is responsive to the actuation commands116to control vehicle movement as instructed by vehicle control system80. Vehicle control system80receives traffic light state data107and determines appropriate vehicle action based thereon. For example, in the event of a green light state defined by the traffic light state data107, vehicle control system80is configured to start going or to continue going through the intersection as prescribed by the route. In the event of a red light state defined by the traffic light state data107, vehicle control system80is configured to stop the vehicle in advance of the traffic light. The vehicle control system80is configured to output control commands116describing the action to be performed by the autonomous vehicle10in response to the traffic light state.

Referring now toFIG. 6, and with continued reference toFIGS. 1-5, a flowchart illustrates a computer control method400that can be performed by the system100ofFIG. 1in accordance with the present disclosure. As can be appreciated in light of the disclosure, the order of operation within the method is not limited to the sequential execution as illustrated inFIG. 6, but may be performed in one or more varying orders as applicable and in accordance with the present disclosure. In various embodiments, the method400can be scheduled to run based on one or more predetermined events, and/or can run continuously during operation of the autonomous vehicle10.

In step402, the autonomous driving system70receives semantic map data112from the semantic map101stored in the at least one storage device32. The semantic map data112includes labelled traffic light data124including traffic light location data124as part of at least one semantic layer120of labelled road features. The semantic map data may also include traffic light type data as described heretofore. The semantic map data112is used by various components of the autonomous driving system70.

In step404, route data114is calculated by a router110of guidance system78based on input start and destination locations, the semantic map data112, particularly the base map122thereof. The router110implements a routing algorithm to calculate an optimal route between the start and destination locations. The route data114includes identification of normal lanes and intersection lanes that are to be followed by the autonomous vehicle10.

In step406, the autonomous driving system70, e.g. the positioning system76or the computer vision system74, extracts traffic light location data124associated with at least one intersection lane to identify which traffic light(s) is to be viewed and where that traffic light(s) is to be found in real space. Such traffic light location data124is included in the semantic map101and associated with each intersection lane1200-1203. The route data114identifies the intersection lane1200-1203being travelled, thereby allowing the associate traffic light location data124to be extracted. In embodiments, positioning system76localizes the autonomous vehicle10relative to the semantic map101and provides corresponding positioning data118. The computer vision system74determines an upcoming intersection lane from the route data114, the current position defined by the positioning data118and the semantic map101and extracts traffic light location data124associated with the labelled intersection lane in the semantic map101.

In step408, the computer vision system74visualizes the traffic light(s) specified by the traffic light location data124in the semantic map data112and assesses the state thereof. The computer vision system74produces traffic light state data107indicating the assessed state of the traffic light. In some embodiments, the computer vision system74controls a field of view and/or zoom or other parameter of at least one sensing device40a(e.g. optical camera) to capture the traffic light based on the traffic light location data124. To do so, the computer vision system74transforms three-dimensional coordinates of the traffic light location data124from the semantic map101into camera control commands108to visualize the traffic light. In order to determine where the optical camera40ais first able to view the traffic light, geometrical calculations are used, in some embodiments, that determine a line of sight from the at least one sensing device40ato the traffic light location defined by the traffic light location data124. Alternatively, the location at which each traffic light is first able to be viewed (assuming normal visibility conditions) is stored as a label in the semantic map101, optionally associated with each labelled intersection lane1300. In another possibility, the location at which each traffic light is first able to be viewed is determined when the autonomous vehicle is within a predetermined distance of the traffic light, which is calculable from the positioning data118representing a current position of the autonomous vehicle10and the traffic light location data107. In additional or alternative embodiments, the computer vision system74processes a selected portion of captured images that contain the traffic light identified by the traffic light location data124. The portion of the captured images is, in some embodiments, determined by transforming the traffic light location data124into image space to allow the computer vision system74to focus on just a part of the images that contain the traffic light. Using vehicle pose information, the relative location of the traffic light1101-1103, as described in the traffic light location data124, and the vehicle10is determinable. From that relative location and camera calibration information, the location of the traffic light1101-1103in image space is determined.

In step410, the vehicle control system80controls driving of the autonomous vehicle10to follow the route defined by the route data114and to take appropriate action at traffic lights based on the traffic light state data. The vehicle control system80produces actuator commands116for execution by the actuator system30.