Patent ID: 12195038

DETAILED DESCRIPTION

The various implementations discussed hereinafter are generally directed in part to a dual mode map that includes sparse map data for some portions of an environment that lacks some of the data maintained in dense map data for other portions of the environment. Prior to a discussion of these implementations, however, an example hardware and software environment within which the various techniques disclosed herein may be implemented will be discussed.

Hardware and Software Environment

Turning to the Drawings, wherein like numbers denote like parts throughout the several views,FIG.1illustrates an example autonomous vehicle100within which the various techniques disclosed herein may be implemented. Vehicle100, for example, is shown driving on a road101, and vehicle100may include a powertrain102including a prime mover104powered by an energy source106and capable of providing power to a drivetrain108, as well as a control system110including a direction control112, a powertrain control114and brake control116. Vehicle100may be implemented as any number of different types of vehicles, including vehicles capable of transporting people and/or cargo, and capable of traveling by land, by sea, by air, underground, undersea and/or in space, and it will be appreciated that the aforementioned components102-116can vary widely based upon the type of vehicle within which these components are utilized.

The implementations discussed hereinafter, for example, will focus on a wheeled land vehicle such as a car, van, truck, bus, etc. In such implementations, the prime mover104may include one or more electric motors and/or an internal combustion engine (among others), while energy source106may include a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, solar panels or other renewable energy source, a fuel cell system, etc., and drivetrain108may include wheels and/or tires along with a transmission and/or any other mechanical drive components suitable for converting the output of prime mover104into vehicular motion, as well as one or more brakes configured to controllably stop or slow the vehicle and direction or steering components suitable for controlling the trajectory of the vehicle (e.g., a rack and pinion steering linkage enabling one or more wheels of vehicle100to pivot about a generally vertical axis to vary an angle of the rotational planes of the wheels relative to the longitudinal axis of the vehicle). In some implementations, combinations of powertrains and energy sources may be used, e.g., in the case of electric/gas hybrid vehicles, and in some instances multiple electric motors (e.g., dedicated to individual wheels or axles) may be used as a prime mover. In the case of a hydrogen fuel cell implementation, the prime mover may include one or more electric motors and the energy source may include a fuel cell system powered by hydrogen fuel.

Direction control112may include one or more actuators and/or sensors for controlling and receiving feedback from the direction or steering components to enable the vehicle to follow a desired trajectory. Powertrain control114may be configured to control the output of powertrain102, e.g., to control the output power of prime mover104, to control a gear of a transmission in drivetrain108, etc., thereby controlling a speed and/or direction of the vehicle. Brake control116may be configured to control one or more brakes that slow or stop vehicle100, e.g., disk or drum brakes coupled to the wheels of the vehicle.

Other vehicle types, including but not limited to off-road vehicles, all-terrain or tracked vehicles, construction equipment, etc., will necessarily utilize different powertrains, drivetrains, energy sources, direction controls, powertrain controls and brake controls, as will be appreciated by those of ordinary skill having the benefit of the instant disclosure. Moreover, in some implementations some of the components may be combined, e.g., where directional control of a vehicle is primarily handled by varying an output of one or more prime movers. Therefore, the invention is not limited to the particular application of the herein-described techniques in an autonomous wheeled land vehicle.

In the illustrated implementation, autonomous control over vehicle100(which may include various degrees of autonomy as well as selectively autonomous functionality) is primarily implemented in a primary vehicle control system120, which may include one or more processors122and one or more memories124, with each processor122configured to execute program code instructions126stored in a memory124.

A primary sensor system130may include various sensors suitable for collecting information from a vehicle's surrounding environment for use in controlling the operation of the vehicle. For example, a satellite navigation (SATNAV) sensor132, e.g., compatible with any of various satellite navigation systems such as GPS, GLONASS, Galileo, Compass, etc., may be used to determine the location of the vehicle on the Earth using satellite signals. Radio Detection And Ranging (RADAR) and Light Detection and Ranging (LIDAR) sensors134,136, as well as one or more digital cameras138(which may include various types of image capture devices capable of capturing still and/or video imagery), may be used to sense stationary and moving objects within the immediate vicinity of a vehicle. An inertial measurement unit (IMU)140may include multiple gyroscopes and accelerometers capable of detection linear and rotational motion of a vehicle in three directions, while one or more wheel encoders142may be used to monitor the rotation of one or more wheels of vehicle100.

The outputs of sensors132-142may be provided to a set of primary control subsystems150, including, a localization subsystem152, a planning subsystem154, a perception subsystem156, and a control subsystem158. Localization subsystem152is principally responsible for precisely determining the location and orientation (also sometimes referred to as “pose”, which in some instances may also include one or more velocities and/or accelerations) of vehicle100within its surrounding environment, and generally within some frame of reference. Planning subsystem154is principally responsible for planning a trajectory or path of motion for vehicle100over some timeframe given a desired destination as well as the static and moving objects within the environment, while perception subsystem156is principally responsible for detecting, tracking and/or identifying elements within the environment surrounding vehicle100. Control subsystem158is principally responsible for generating suitable control signals for controlling the various controls in control system110in order to implement the planned trajectory or path of the vehicle. Any or all of localization subsystem152, planning subsystem154, perception subsystem156, and control subsystem158may have associated data that is generated and/or utilized in connection with the operation thereof, and that which may be communicated to a teleassist system in some implementations.

In addition, an atlas or map subsystem160may be provided in the illustrated implementations to describe the elements within an environment and the relationships therebetween. Atlas subsystem160may be accessed by each of the localization, planning and perception subsystems152-156to obtain various information about the environment for use in performing their respective functions. Atlas subsystem160may be used to provide map data to the autonomous vehicle control system, which may be used for various purposes in an autonomous vehicle, including for localization, planning, and perception, among other purposes. Map data may be used, for example, to lay out or place elements within a particular geographical area, including, for example, elements that represent real world objects such as roadways, boundaries (e.g., barriers, lane dividers, medians, etc.), buildings, traffic devices (e.g., traffic or road signs, lights, etc.), as well as elements that are more logical or virtual in nature, e.g., elements that represent valid pathways a vehicle may take within an environment, “virtual” boundaries such as lane markings, or elements that represent logical collections or sets of other elements. Map data may also include data that characterizes or otherwise describes elements in an environment (e.g., data describing the geometry, dimensions, shape, etc. of objects), or data that describes the type, function, operation, purpose, etc., of elements in an environment (e.g., speed limits, lane restrictions, traffic device operations or logic, etc.). In some implementations, atlas subsystem160may provide map data in a format in which the positions of at least some of the elements in a geographical area are defined principally based upon relative positioning between elements rather than any absolute positioning within a global coordinate system. It will be appreciated, however, that other atlas or map systems suitable for maintaining map data for use by autonomous vehicles may be used in other implementations, including systems based upon absolute positioning. Furthermore, it will be appreciated that at least some of the map data that is generated and/or utilized by atlas subsystem160may be communicated to a teleassist system in some implementations.

It will be appreciated that the collection of components illustrated inFIG.1for primary vehicle control system120is merely exemplary in nature. Individual sensors may be omitted in some implementations, multiple sensors of the types illustrated inFIG.1may be used for redundancy and/or to cover different regions around a vehicle, and other types of sensors may be used. Likewise, different types and/or combinations of control subsystems may be used in other implementations. Further, while subsystems152-160are illustrated as being separate from processors122and memory124, it will be appreciated that in some implementations, some or all of the functionality of a subsystem152-160may be implemented with program code instructions126resident in one or more memories124and executed by one or more processors122, and that these subsystems152-160may in some instances be implemented using the same processors and/or memory. Subsystems in some implementations may be implemented at least in part using various dedicated circuit logic, various processors, various field-programmable gate arrays (“FPGA”), various application-specific integrated circuits (“ASIC”), various real time controllers, and the like, and as noted above, multiple subsystems may utilize common circuitry, processors, sensors and/or other components. Further, the various components in primary vehicle control system120may be networked in various manners.

In some implementations, vehicle100may also include a secondary vehicle control system170, which may be used as a redundant or backup control system for vehicle100. In some implementations, secondary vehicle control system170may be capable of fully operating autonomous vehicle100in the event of an adverse event in primary vehicle control system120, while in other implementations, secondary vehicle control system170may only have limited functionality, e.g., to perform a controlled stop of vehicle100in response to an adverse event detected in primary vehicle control system120. In still other implementations, secondary vehicle control system170may be omitted.

In general, an innumerable number of different architectures, including various combinations of software, hardware, circuit logic, sensors, networks, etc. may be used to implement the various components illustrated inFIG.1. Each processor may be implemented, for example, as a microprocessor and each memory may represent the random access memory (RAM) devices comprising a main storage, as well as any supplemental levels of memory, e.g., cache memories, non-volatile or backup memories (e.g., programmable or flash memories), read-only memories, etc. In addition, each memory may be considered to include memory storage physically located elsewhere in vehicle100, e.g., any cache memory in a processor, as well as any storage capacity used as a virtual memory, e.g., as stored on a mass storage device or on another computer or controller. One or more processors illustrated inFIG.1, or entirely separate processors, may be used to implement additional functionality in vehicle100outside of the purposes of autonomous control, e.g., to control entertainment systems, to operate doors, lights, convenience features, etc.

In addition, for additional storage, vehicle100may also include one or more mass storage devices, e.g., a floppy or other removable disk drive, a hard disk drive, a direct access storage device (DASD), an optical drive (e.g., a CD drive, a DVD drive, etc.), a solid state storage drive (SSD), network attached storage, a storage area network, and/or a tape drive, among others. Furthermore, vehicle100may include a user interface172to enable vehicle100to receive a number of inputs from and generate outputs for a user or operator, e.g., one or more displays, touchscreens, voice and/or gesture interfaces, buttons and other tactile controls, etc. Otherwise, user input may be received via another computer or electronic device, e.g., via an app on a mobile device or via a web interface, e.g., from a remote operator.

Moreover, vehicle100may include one or more network interfaces, e.g., network interface174, suitable for communicating with one or more networks176(e.g., a LAN, a WAN, a wireless network, and/or the Internet, among others) to permit the communication of information with other vehicles, computers and/or electronic devices, including, for example, a central service, such as a cloud service, from which vehicle100receives environmental and other data for use in autonomous control thereof. In the illustrated implementations, for example, vehicle100may be in communication with various cloud-based remote vehicle services178including, at least for the purposes of implementing various functions described herein, an atlas or map service or system180, a teleassist service or system182, and a live map service or system184. Atlas or map service or system180may be used, for example, to maintain a global repository describing one or more geographical regions of the world, as well as to deploy portions of the global repository to one or more autonomous vehicles, to update the global repository based upon information received from one or more autonomous vehicles, and to otherwise manage the global repository. Teleassist service or system182may be used, for example, to provide teleassist support to vehicle100, e.g., through communication with a teleassist subsystem186resident in primary vehicle control system120, as will be discussed in greater detail below. Live map service or system184may be used to propagate various observations collected by one or more autonomous vehicles to effectively supplement the global repository maintained by atlas or map service or system180. The terms “service” and “system” are generally used interchangeably herein, and generally refer to any computer functionality capable of receiving data from, and providing data to, an autonomous vehicle. In many instances, these services or systems may be considered to be remote services or systems insofar as they are generally external to an autonomous vehicle and in communication therewith.

Each processor illustrated inFIG.1, as well as various additional controllers and subsystems disclosed herein, generally operates under the control of an operating system and executes or otherwise relies upon various computer software applications, components, programs, objects, modules, data structures, etc., as will be described in greater detail below. Moreover, various applications, components, programs, objects, modules, etc. may also execute on one or more processors in another computer coupled to vehicle100via network, e.g., in a distributed, cloud-based, or client-server computing environment, whereby the processing required to implement the functions of a computer program may be allocated to multiple computers and/or services over a network. Further, in some implementations data recorded or collected by a vehicle may be manually retrieved and uploaded to another computer or service for analysis.

In general, the routines executed to implement the various implementations described herein, whether implemented as part of an operating system or a specific application, component, program, object, module, machine learning model, or sequence of instructions, or even a subset thereof, will be referred to herein as “program code.” Program code typically comprises one or more instructions that are resident at various times in various memory and storage devices, and that, when read and executed by one or more processors, perform the steps necessary to execute steps or elements embodying the various aspects of the invention. Moreover, while the invention has and hereinafter will be described in the context of fully functioning computers and systems, it will be appreciated that the various implementations described herein are capable of being distributed as a program product in a variety of forms, and that the invention applies equally regardless of the particular type of computer readable media used to actually carry out the distribution. Examples of computer readable media include tangible, non-transitory media such as volatile and non-volatile memory devices, floppy and other removable disks, solid state drives, hard disk drives, magnetic tape, and optical disks (e.g., CD-ROMs, DVDs, etc.), among others.

In addition, various program code described hereinafter may be identified based upon the application within which it is implemented in a specific implementation. However, it should be appreciated that any particular program nomenclature that follows is used merely for convenience, and thus the invention should not be limited to use solely in any specific application identified and/or implied by such nomenclature. Furthermore, given the typically endless number of manners in which computer programs may be organized into routines, procedures, methods, modules, objects, and the like, as well as the various manners in which program functionality may be allocated among various software layers that are resident within a typical computer (e.g., operating systems, libraries, API's, applications, applets, etc.), it should be appreciated that the invention is not limited to the specific organization and allocation of program functionality described herein.

Those skilled in the art will recognize that the exemplary environment illustrated inFIG.1is not intended to limit the present invention. Indeed, those skilled in the art will recognize that other alternative hardware and/or software environments may be used without departing from the scope of the invention.

Unified Boundary Machine Learning Model For Autonomous Vehicles

In operation, autonomous vehicles may encounter various types of actual and logical boundaries on a roadway that constrain where the autonomous vehicle may travel and/or delineate acceptable lanes within which travel is permitted. Some boundaries may be physical in nature, including, for example, the physical edges of a roadway and/or various physical barriers along a roadway, including, for example, concrete barriers, guardrails, etc. that actually constrain where an autonomous vehicle may travel. In addition, some boundaries may not physically constrain a vehicle but may nonetheless logically delineate the acceptable lanes within which travel is permitted, e.g., painted or taped lines on the road surface the define the extent of each lane of travel, as well as delineate shoulders, exits, entrances, etc. Such logical boundaries may also present differing semantic constraints, e.g., dashed lines that separate adjacent same-direction lanes or that allow for passing, double solid lines that restrict lane changes or passing, etc.

In construction areas, these existing boundaries may also be supplemented by additional boundaries such as repainted or taped lines for temporary traffic diversions (which in some instances may overlap existing lines), spaced apart construction elements such as barrels or traffic cones that appear as distinct objects to the perception system, and construction elements such as jersey barriers that can be difficult to classify due to difficulties in determining object boundaries. Determining the correct lane of travel, as well as an appropriate trajectory that stays within that lane of travel, can therefore be challenging in a number of different scenarios, so a continuing need exists in the art for a manner of detecting and managing the various types of boundaries that an autonomous vehicle may encounter during operation.

One potential approach for addressing the various types of boundaries that an autonomous vehicle might encounter is to use multiple components incorporating engineered/heuristic algorithms and/or trained machine learning models to detect different types of boundaries. Boundaries defined by spaced apart construction elements such as traffic cones or barrels, for example, may be determined using a component that is downstream of a perception component, and in some instances, downstream of a tracking component, and that generates “virtual” boundaries in response to detection by a perception component of multiple construction elements that are found to be within predetermined arrangements relative to one another. Lane boundaries defined by painted or taped lines on the surface of the road may be detected in connection with localization in order to improve the determined global pose of a vehicle by aligning the vehicle within a lane. Physical barriers may be sensed by a perception system and used in connection with object avoidance.

It has been found, however, that detecting various types of boundaries in different components and for different purposes can raise various issues, particularly in construction areas where physical barriers, temporarily lane markers, traffic cones, barrels, etc. are present, and where vehicles may be diverted into other lanes or paths that differ from the lanes defined in map data. For example, determinations of virtual boundaries associated with spaced apart construction elements may benefit from the use of map data to determine how construction elements are aligned relative to the roadway, and further could benefit from the detection of temporary lane markers in such an alignment. When implemented in separate components, however, dependencies may be created, requiring multiple passes and increased system latency.

In the illustrated implementations, however, a unified boundary machine learning model may be trained and used to detect multiple types of boundaries in response to perception data collected from one or more perception sensors, e.g., one or more cameras/image sensors and/or LIDAR sensors. Moreover, in some instances, a unified boundary machine learning model may be incorporated into a multi-head machine learning model that also supports mainline perception, providing reduced system-level computation and memory usage, as well as joint optimization of both functions as well as optimization of contextual interactions between different boundary types.

FIG.2, for example, illustrates an example system200incorporating an autonomous vehicle202in communication with a set of online services204, and supporting a perception-based unified boundary machine learning model205consistent with some implementations. System200also supports both live map and teleassist functionality; however, in other implementations, one or both of live map functionality and teleassist functionality may be omitted.

Within autonomous vehicle202, a motion planner component206receives data, e.g., an augmented road region layout, from a map fusion component208. Map fusion component208may receive a digital map, e.g., a road region layout, from a road region generator component210. Component210may generate a road region layout by accessing an on-board map212to retrieve offline map data suitable for generating a digital map of the environment surrounding autonomous vehicle202during operation. Map fusion component208may also receive observation data from a perception component214, which collects observations using one or more sensors in the manner discussed above, and which in the illustrated implementation utilizes unified boundary machine learning model205. A perception component in some implementations may also provide tracks for various dynamic objects directly to motion planner component206. Where teleassist support is provided, map fusion component208may also receive teleassist data, e.g., teleassist operator input, from a teleassist component216that is in communication with a remote teleassist system218. Furthermore, in some implementations, map fusion component208may also receive additional observation data from a live map system220, such that the data collected from components210,214,216and220may be fused into the augmented road region layout used by motion planner component206when generating a trajectory or path of motion for the autonomous vehicle.

The observation data provided by live map system220is stored in a live map database or data store222, and includes observation data collected from one or more other autonomous vehicles224. Live map system220is generally in bi-directional communication with both autonomous vehicle202and other autonomous vehicle(s)224to both collect observation data for autonomous vehicles202,224and to propagate observation data collected by autonomous vehicles202,224operating in a particular portion or region of the environment to other autonomous vehicles202,224operating in the same portion or region of the environment. In addition, teleassist system218may also be in bi-directional communication with live map database222via live map system220to enable a teleassist operator to store teleassist data, e.g., location-based teleassist triggers, to be propagated to autonomous vehicles via live map system202, as well as to retrieve observation data from live map database222in connection with conducting a teleassist session with autonomous vehicle202. Teleassist system218may also be in bi-directional communication with autonomous vehicle202in some implementations (e.g., via teleassist component216) to allow an autonomous vehicle to provide situational awareness data to a teleassist operator and to allow the teleassist operator to provide suggesting actions to the autonomous vehicle. In some implementations, no direct link may be present between teleassist system218and live map system220, such that communication between these components may be handled through map fusion component208and teleassist component216.

Live map system220and live map database222are representative of an online map system that propagates observation data within an autonomous vehicle fleet, as opposed to an offline map service226, e.g., an atlas system or service, that provides offline map data to the autonomous vehicle fleet. An offline map service226, for example, may include an offline map database228that maintains a global repository of offline map data. A map release component230may be used to generate versioned updates of the offline map database, and a map deploy component232may be used to deploy or propagate the database updates to an autonomous vehicle fleet. It will be appreciated that a global repository of offline map data may be substantial in size, so in some implementations only a portion of the offline map data corresponding to the particular area (e.g., a city, a state, a county, etc. within which autonomous vehicle202operates) may be deployed to the autonomous vehicle and maintained in its on-board map212. In addition, based upon movement of the autonomous vehicle into adjacent areas, additional portions of the offline map data may be communicated to the autonomous vehicle by service226such that the on-board map212includes sufficient offline map data to operate the autonomous vehicle at its current location.

FIG.3next illustrates one example manner of implementing trained unified boundary machine learning model205ofFIG.2.FIG.3, in particular, illustrates an example autonomous vehicle control system240including a perception component242that receives as input perception data, e.g., as captured by one or more cameras or image sensors244(e.g., a camera with a forward-facing field of view) and LIDAR data, e.g., as captured by one or more LIDAR sensors246. Each of image sensor244and LIDAR sensor246may be positioned on an autonomous vehicle to sense the roadway upon which the autonomous vehicle is disposed.

Perception component242includes a trained unified boundary machine learning model, in particular a trained multi-head machine learning model248, that is configured to, in part, generate a plurality of perceived boundaries for the roadway. Model248in particular integrates detection of perceived boundaries associated with a plurality of semantic boundary types for the roadway. A semantic boundary type, within the context of the disclosure, may be considered to refer to a particular classification of boundary within a schema of detectable boundary types. Boundary types in some implementations may be classified based upon whether they are physical or virtual (i.e., whether or not they represent a continuous physical barrier constraining autonomous vehicle movement), whether they are actual or logical (i.e., whether they represent actual physical objects in the environment that an autonomous vehicle should avoid or they represent a logical construct that, for example, for legal or regulatory reasons, they are required to obey), and/or based upon other classifications. For example, a physical barrier such as a guardrail, jersey barrier or permanent concrete barrier may be associated with a physical barrier semantic boundary type, and a painted or taped line on the surface of the roadway that forms the boundary of a lane may be associated with a painted lane semantic boundary type. Likewise, a road edge, e.g., at the boundary between the road and a curb, median, a gravel shoulder, or other non-drivable surface that delineates the edge of the drivable area of a roadway may be associated with a road edge semantic boundary type.

Still other boundary type classifications may be based, for example, upon whether they are permanent or temporary/construction in nature, and in some instances, boundary types may be classified with finer granularity, e.g., to distinguish between guardrails and concrete barriers, between different types of road edges, between different types of painted lane boundaries (e.g., yellow vs. white, single dashed, single solid, double solid, solid and dashed), etc.

In addition, in some implementations, a unified boundary machine learning model such as trained multi-head machine learning model248may also be trained to detect pathways of the autonomous vehicle (the “ego” vehicle) and/or the pathways of other vehicles in the vicinity of the autonomous vehicle. These pathways may also be used in the control of an autonomous vehicle, as the path that a vehicle takes may be indicative of the locations of the drivable lanes on the roadway. In some implementations, pathways may be defined in a similar manner to boundaries, e.g., as collections of linear or curvilinear segments, but with the segments representing a centerline of a path taken by the ego or another vehicle.

In some implementations, boundaries and pathways may collectively be referred to as lane elements, as each may be capable of defining some aspect of a lane in a roadway. A lane, in this regard, may be considered to be a region of a roadway within which a vehicle may potentially be able to operate. A lane element, in turn, may be used to define some aspect of the lane, e.g., a virtual or physical boundary, a centerline, etc. A boundary-type lane element, for example, may define a boundary associated with the lane, while a pathway-type lane element may define a pathway through the lane (e.g., a centerline of the lane). It will be appreciated that a boundary may be represented by a single boundary-type lane element in some implementations, while in other implementations, a boundary may be represented by a collection of boundary-type lane elements defining portions or segments of the boundary. Likewise, a pathway may be represented by a single pathway-type lane element in some implementations, while in other implementations, a pathway may be represented by a collection of pathway-type lane elements defining portions or segments of the pathway.

It should be appreciated, however, that an individual lane element in some implementations may not necessarily be associated with a complete, drivable lane on a roadway, and a lane need not be defined fully by lane elements associated with left and right boundaries and a centerline. Particularly in construction areas, various construction-related boundaries may exist that effectively supersede other detected boundaries such that those other detected boundaries could not be used to define a complete, drivable lane on the roadway due to the presence of other detected lane elements. In addition, in some construction scenarios, vehicles on the roadway may follow different paths than those represented by the detected boundaries, so the pathways defined by those vehicles may, in some instances, define lanes without perceptible boundaries, or lanes that appear to cross or overlap with other lanes defined by the boundaries on the roadway.

At least one of the semantic boundary types supported by a unified boundary machine learning model such as trained multi-head machine learning model248includes a virtual construction semantic boundary type that is associated with a plurality of spaced apart construction elements such as traffic cones or barrels that, while not physical forming a continuous barrier due to the gaps between the elements, nonetheless define a virtual boundary that should be obeyed as a constraint on autonomous vehicle movement. Such construction elements may be separately detectable by the perception system due to the spacing, so a trained unified boundary machine learning model in some implementations may detect and link together these multiple construction elements into a virtual boundary based upon the presence of such construction elements having a spacing and arrangement that defines a suitable boundary along the roadway to constrain vehicle movement. In addition, in some implementations, even though the spaced construction elements are linked together to define a virtual boundary, the mainline perception system may still detect the individual spaced construction elements and output those elements to a tracking component to generate tracks for the construction elements such that the autonomous vehicle is controlled using both the virtual boundary defined for the spaced construction elements as well as the tracks for such construction elements.

In some implementations, for example, trained multi-head machine learning model248may be implemented as a deep neural network (DNN) including an input layer250, one or more intermediate layers252, and an output layer254including one or more mainline perception heads256and one or more unified boundary heads258. In some implementations, for example, one or more intermediate layers252may include one or more convolutional layers. The dimensions/shape of input layer250may be dependent on the shape of the perception data to be applied, while the dimensions/shape of each output head256,258may be dependent on various factors such as how many class probabilities are to be predicted, among others. In some implementations, multiple convolution layers may be provided, and max pooling and/or other layers such as affine layers, softmax layers and/or fully connected layers may optionally be interposed between one or more of the convolution layers and/or between a convolution layer and the output layer. Other implementations may not include any convolution layer and/or not include any max pooling layers, and in still other implementations, other machine learning models may be used, e.g., Bayesian models, random forest models, Markov models, etc.

FIG.4illustrates a system for training model248, e.g., using a training engine260that utilizes training instances262retrieved from a training instances database264. The input266of each training instance, for example, may include perception data such as LIDAR and/or camera/image data, and the output268of each training instance may include object and/or boundary classifications. In some implementations, for example, the training instance output268may define, for each of a plurality of spatial regions, whether an object of one or more classes is present in the spatial region and/or whether a boundary of one or more classes is present in the spatial region. In training model248, training engine260may apply the training instance input266to model248and process the training instance input266, utilizing model248and based on current parameters of model248, to generate an output having a dimension that conforms to the dimension of training instance output268. Training engine260may then compare the generated output to the training instance output268, and update one or more parameters of model248based on the comparison. For example, training engine260may generate an error based on differences between the generated output and the training instance output268, and backpropagate a loss (that is based on the error) over model248to update model248. Although only a single training instance262is illustrated inFIG.4, model248will generally be trained based on a large quantity of training instances of training instances database264. Those training instances can collectively include training instance inputs with diverse perception data and diverse training instance outputs. Moreover, although non-batch training is described with respect toFIG.4, batch training may additionally or alternatively be utilized (e.g., where losses are based on errors determined based on a batch of training instances).

As model248is a multi-head model that incorporates at least one mainline perception head256and at least one unified boundary head258, different subsets of training instances may be used, thereby co-training the different output heads256,258and jointly optimizing the model for both mainline perception and unified boundary perception functionality. At least one subset of the training instances may include input perception data associated with one or more objects sensed within the environment and output classification data that classifies one or more objects to be classified by the least one mainline perception head. At least one subset of the training instances includes input perception data associated with one or more perceived boundaries associated with a virtual construction semantic boundary type for one or more roadways within the environment and output classification data that classifies the one or more perceived boundaries associated with the virtual construction semantic boundary type. In addition, at least one subset of the training instances includes input perception data associated with one or more perceived boundaries associated with an additional semantic boundary type for one or more roadways within the environment and output classification data that classifies the one or more perceived boundaries associated with the additional semantic boundary type. Furthermore, it will be appreciated that at least one training instance may be overlapping in nature, and may include perception data associated with multiple objects and/or multiple boundaries associated with multiple semantic boundary types, thereby further jointly optimizing the model for both mainline perception and unified boundary functionality.

FIG.5illustrates an example roadway270to further describe various semantic boundary types that may be supported in a unified boundary machine learning model in some implementations. An autonomous vehicle272, for example, is illustrated as traveling on a four-lane divided roadway270including a pair of same-direction lanes274,276separated by shoulders278,280from a permanent concrete median barrier282and a non-road (e.g., grass) area284. A road edge286separates shoulder280from non-road area284, and in the example illustration, a construction area is defined by a plurality of spaced apart construction elements288(here, traffic cones) that effectively close lane276and divert all traffic into lane274, followed by a plurality of jersey barriers290that separate lanes274and276.

Within this example roadway, a number of different perceived boundaries, each having a one of a plurality of different semantic boundary types, may be generated and used in the control of an autonomous vehicle.

For example, one semantic boundary type that may be supported in some implementations is a physical barrier semantic boundary type, and two perceived boundaries with such a semantic boundary type are represented by permanent concrete median barrier282and boundary292that is defined by jersey barriers290. Another semantic boundary type that may be supported in some implementations is a road edge semantic boundary type, and a perceived boundary with such a semantic boundary type is represented by road edge286defined between shoulder280and non-road area284.

Yet another semantic boundary type that may be supported in some implementations is a painted lane semantic boundary type, and three perceived boundaries with such a semantic boundary type are represented by solid painted lines294A and294B and dashed painted line294C. Another semantic boundary type that may be supported in some implementations is a virtual construction semantic boundary type, and a boundary with such a semantic boundary type is represented by boundary296defined by traffic cones288.

In addition, as further illustrated inFIG.5, some implementations may also support detection of pathways within a trained unified boundary machine learning model, and a number of perceived pathways are illustrated, e.g., at298A and298B. Perceived pathway298A, for example, may represent an ego path of autonomous vehicle272, while perceived pathway298B may represent a path of another vehicle sensed by the perception system.

Now turning toFIG.6, an example operational sequence300for controlling an autonomous vehicle (e.g., autonomous vehicle202ofFIG.2) using a trained unified boundary model is illustrated. In block302, perception data associated with a roadway upon which an autonomous vehicle is disposed is received from one or more perception sensors (e.g., in a perception component), and in block304, one or more perceived boundaries are generated for the roadway using a trained unified boundary machine learning model as described above. In block306an augmented digital map is then generated using a stored digital map and the generated perceived boundaries (e.g., in a map fusion component). In addition, in this implementation, one or more active lanes in the roadway are determined using the augmented digital map in block308, and a trajectory for the autonomous vehicle is determined using the active lane(s) in block310, both of which may be performed, for example, in a motion planner component. The one or more active lanes may be identified as an ego lane (i.e., a lane in which the autonomous vehicle is or should be operating) or an adjacent lane.

In addition, in some implementations, the motion planner component may also optionally utilize one or more object tracks in order to determine the trajectory for the autonomous vehicle. For example, in block312, one or more objects may be generated using mainline perception, e.g., in a trained mainline perception machine learning model, which in some implementations may be integrated with the trained unified boundary machine learning model into a multi-head machine learning model such as model248ofFIG.3. Then, in block314a tracking component may generate tracks for the generated objects such that the determination of the trajectory in block310also uses the generated object tracks. The autonomous vehicle is then controlled in block316using the determined trajectory.

As noted above, in operational sequence300, one or more active lanes may be determined or synthesized by the map fusion component in block308. In the illustrated implementation, the determination of the active lanes may be based in part on an interpretation of all of the perceived boundaries generated by the trained unified boundary machine learning model in the map fusion component. Thus, the map fusion component may be configured to assess each of the boundaries, in some instances in connection with the digital map, to interpret each boundary and determine therefrom one or more active lanes within which the autonomous vehicle is permitted to drive. In the roadway ofFIG.5, for example, an interpretation of the perceived boundaries in connection with the digital map would result in some implementations in a determination that lane274is active and lane276is inactive, as well as that the drivable area available to the autonomous vehicle is, on one side, bounded by painted line294B, and on the other side, bounded first by painted line294A, then by virtual construction boundary296defined by construction elements288, and then by boundary292defined by jersey barriers290. In addition, the motion planner component could also interpret pathways298A,298B, etc. to select an appropriate trajectory that directs the autonomous vehicle within the active lane.

In other implementations, however, determination of an active lane and/or a drivable area for an autonomous vehicle may be based in part on the interpretation of boundaries upstream of the motion planner component. In some implementations, for example, a unified boundary machine learning model may be trained to also interpret perceived boundaries. As such, rather than generating an augmented digital map using the generated perceived boundaries in a map fusion component, and determining one or more active lanes in the roadway using the augmented digital map in a motion planner component (as is the case with blocks306and308), a trained unified boundary machine learning model may also interpret the perceived boundaries to generate one or more active lanes, such that active lanes (and in some instances, the perceived boundaries as well) may be used to augment a digital map (e.g., in a map fusion component).

Now turning toFIG.7, this figure illustrates an example autonomous vehicle350including an autonomous vehicle control system352that incorporates a unified boundary machine learning model consistent with some implementations. Autonomous vehicle control system352also interfaces with both a live map system and a teleassist system in connection with autonomously operating autonomous vehicle350. It will be appreciated that in other implementations, an autonomous vehicle control system may interface only with a teleassist system or only with a live map system, or may interface with neither, so the invention is not limited to the specific implementation discussed herein. In addition,FIG.7illustrates only the components of autonomous vehicle control system352related to motion and route planning, so other components that are unrelated to such functions are omitted from the figure for simplification purposes.

One or more image sensors or cameras354and one or more LIDAR sensors356operate as the perception sensors used in connection with motion planning. Camera(s)354and LIDAR sensor(s)358output camera data to a perception component358, within which resides an object classifier component360that may be used as a mainline perception component to identify physical objects in the environment surrounding the autonomous vehicle. While component360may detect a wide variety of physical objects, from the perspective of the functionality described herein, two particular types of objects, construction elements and signs, are separately illustrated as being output by object classifier component360. Other types of objects, e.g., other vehicles, pedestrians, etc., may also be detected by component360. It will be appreciated, however, that the types of objects that may be detected are innumerable, and that a perception component may be configured to detect many other types of objects in other implementations, so the output of construction elements and signs as illustrated inFIG.7is merely for the purposes of explanation. In some implementations, object classifier component360may be implemented using one or more trained machine learning models and/or using one or more engineered/heuristic algorithms.

Perception component358also includes a unified boundary machine learning model362configured to detect various types of boundaries capable of being perceived by autonomous vehicle352. In some implementations, unified boundary machine learning model362may be distinct from object classifier component360and/or perception component358; however, as represented at364, in some implementations unified boundary machine learning model362may be integrated with object classifier component360, e.g., with both components representing one or more output heads of a multi-head machine learning model such as described above in connection withFIGS.3-4.

Perception component358may also include a sign classifier component366that receives the objects detected as being signs by object classifier component360and determines the logical significance of such signs, e.g., the type of sign (speed limit, warning, road closure, construction notification, etc.), whether the sign is cautionary or mandatory, a location and/or lane that is affected by the sign (e.g., right lane closed 2 miles ahead), etc. In other implementations, the functionality of components360,366may be combined, or additional purpose-specific classifier components may be used. In some implementations, sign classifier component366may be implemented using one or more trained machine language models and/or using one or more engineered/heuristic algorithms.

The objects detected by components360,366are provided to a tracking component368that maintains tracks for each of the detected objects (as well as other stationary and/or moving objects detected in the environment) over time. It will be appreciated that since the autonomous vehicle is generally in motion, tracks may still be used to represent stationary objects since their relative location to the autonomous vehicle changes over time due to the movement of the autonomous vehicle itself. In some implementations, for example, a track may represent the4D bounds of an object over time, and may be derived from a set of observations (“detections” and “classifications”). In such implementations, a track may define the real-world object in time and space regardless of whether that real world object is moving, and may effectively operate as an interface between perception and motion planning.

Camera(s)354and/or LIDAR sensor(s)356may also provide output to other components in autonomous vehicle control system352. For example, the output may also be provided to an online localization component370that determines the current location and orientation (e.g., a pose) of the autonomous vehicle. A global pose component372may receive the output of LIDAR sensor(s)356to determine a global pose for the autonomous vehicle, which is output to a pose filter component374and fed back as an additional input to global pose component372. The global pose may also be output to a lane alignment component376that receives as input the output of LIDAR sensor(s)356and an output of trained unified boundary machine learning model362, and provides another input to pose filter374, although in some implementations the global pose may not be an input to lane alignment component376. Lane alignment may be an input used for determining a global pose in some implementations. Unified boundary machine learning model362in particular may provide lane alignment component376with one or more perceived painted lane boundaries, i.e., painted or taped lines delineating the lanes on a roadway, to serve in localizing the autonomous vehicle within a lane.

A map fusion component378receives as input the tracks output by tracking component364and the perceived boundaries from unified boundary machine learning model364, which together may be considered to represent at least a portion of the perception observation data collected by the sensors of the autonomous vehicle. In addition, a road region generator component3800retrieves map data from on-board map382and generates a baseline road region layout (RRL) representing a digital map of the area road region surrounding the autonomous vehicle. Map fusion component378fuses these inputs together to generate an augmented road region layout (ARRL) that is provided to a motion planner component384that generates a trajectory or path of motion for the autonomous vehicle at least partially upon the augmented road region layout.

Motion planner component384may also receive as input at least a portion of the tracks output by tracking component368and a desired route from a route planner component386, which is generally generated from on-board map382and provides high level guidance as to a desired route to reach a desired destination.

Moreover, a teleassist component388may be operatively coupled to map fusion component378and motion planner component384to provide an interface with a remote teleassist system (not shown inFIG.7) Teleassist component388may output, for example, observation data to map fusion component378to incorporate into an augmented road region. One non-limiting example is a speed limit command, e.g., to supply a maximum speed an autonomous vehicle may travel at in certain lanes or roadways, or within a construction zone. Teleassist component388may also output, for example, suggested actions to motion planner component384, e.g., to change lanes, to stop or pull over to the side of the road, etc. Teleassist component388may also receive an augmented road region layout from map fusion component378(or any of the other data collected thereby) to assist a remote teleassist operator in providing assistance to the autonomous vehicle during a teleassist session. Further, in some implementations, various triggers (e.g., to trigger a teleassist session in response to detection of speed limit or construction signs), as well as various map issues (e.g., based upon discrepancies determined between digital map data and perceived boundaries) may also be provided by map fusion component378to teleassist component388. Online services390, including a remote teleassist system and a live map system similar to that described above in connection withFIG.2, may be interfaced with map fusion component378and teleassist component388, e.g., to provide observation data and/or location-based teleassist triggers to map fusion component378and to exchange information with a teleassist operator during a teleassist session.

It will be appreciated that the architecture of autonomous vehicle control system352is merely exemplary in nature, and other architectures may be used in other implementations. For example, some or all of the components in autonomous vehicle control system352may be implemented using programmed logic (e.g., implementing engineered/heuristic algorithms) and/or trained machine learning models, and that implementation of such components would be well within the abilities of those of ordinary skill having the benefit of the instant disclosure.

Now turning toFIG.8, in some implementations, a trained unified boundary machine learning model may additionally support one or both of learned tracking and learned attributes. With learned tracking, boundaries (and in some instances, pathways) may be tracked over multiple intervals to associate the boundaries detected in each interval with boundaries detected in other intervals, thereby effectively persisting boundaries across intervals and improving consistency from interval to interval. With learned attributes, one or more attributes may be detected for various boundaries to provide additional semantic content about a boundary or scene.

One type of attribute that may be learned in some implementations is a sidedness attribute, which identifies which side of a detected boundary allows for vehicle travel, and may be utilized downstream, e.g., to reduce uncertainty during motion planning. In some implementations, for example, sidedness may allow an autonomous vehicle to identify when it has entered a non-drivable region and needs to return to a drivable region, or when it is driving adjacent to a non-drivable region and needs to avoid entering that non-drivable region. As one non-limiting example, an autonomous vehicle may be operating on a four-lane divided highway that has been reduced to the left lane in one direction due to temporary construction occurring in the right lane and delimited by barrels. If, however, the autonomous vehicle is in stop and go traffic, and sitting behind a semi-trailer, and if a large break exists between barrels, it may not be readily apparent to the perception system of the autonomous vehicle that it is still within the construction area. Nonetheless, if a sidedness attribute has been set for the boundary associated with the center dashed line that indicates that driving is only permitted to the left of the boundary, the autonomous vehicle may be discouraged from changing into the right lane even though the perception system does not presently detect any construction elements in the roadway.

Another type of attribute that may be learned in some implementations is an active attribute, which generally indicates whether there is an active condition in the vicinity of the boundary that should trigger some operational change in the autonomous vehicle, e.g., to change lanes and/or reduce speed. For example, one type of an active attribute that may be used is an active construction attribute that indicates whether construction is actively occurring proximate the boundary. It will be appreciated that, particularly for large construction projects, active work may only be ongoing in relatively small areas of the overall construction area, e.g., only where construction workers or operational construction equipment are present. Thus, it may be desirable in some implementations to incorporate into a trained unified boundary machine learning model detection of active construction, and to associate with any affected boundaries an active construction attribute and thereby cause the autonomous vehicle to reduce speed and/or change lanes, or perform some other predetermined action.

Likewise, an active attribute in some implementations may be an active hazard attribute associated with the presence of other potential hazards detected on the roadway, e.g., stopped or disabled vehicles, parked police, construction or emergency vehicles, tow trucks, etc. Thus, it may be desirable in some implementations to incorporate into a trained unified boundary machine learning model detection of potential hazards or other conditions along the roadway, and to associate with any affected boundaries an active attribute and thereby cause the autonomous vehicle to reduce speed and/or change lanes, or perform some other suitable action.

Attributes may be associated with boundaries in some implementations, or with individual segments of boundaries, and in some implementations, attributes may be associated with entire scenes.

One example manner of implementing both learned tracking and learned attributes is illustrated by trained unified boundary machine learning model400ofFIG.8. Model400in some implementations may be a trained multi-head machine learning model similar to trained multi-head machine learning model248ofFIG.3, but including additional tracking functionality as well as additional training for detecting attributes such as sidedness and/or active attributes. Model400, for example, may include a plurality of boundary type heads402trained to detect various types of boundaries, and in some instances, one or more pathway heads404trained to detect various types of pathways (e.g., ego pathways and/or other vehicle pathways). Model400receives perception sensor data406, e.g., image data, e.g., as captured by one or more cameras or image sensors (e.g., a camera with a forward-facing field of view) and LIDAR data, e.g., as captured by one or more LIDAR sensors. Additional heads, e.g., mainline object detecting heads, may also be used, but are omitted fromFIG.8for simplicity.

In each boundary type head402, a feature detection stage408distills various learned features, including, in some instances, one or more attributes, associated with a particular boundary type M among the various boundary types supported by the model. Feature vectors are output both to a memory410that maintains one or more (i.e., 1. . N) prior frames or intervals, as well as to a learned tracking stage412, which is trained to track boundaries over multiple frames or segments, and which outputs various detected boundaries, optionally tagged with one or more attributes, as illustrated at414.

Similarly, if pathways are also supported, pathway head404includes a feature detection stage416that distills various learned features, including, in some instances, one or more attributes, associated with one or more types of pathways. Feature vectors are output both to a memory418that maintains one or more (i.e., 1. . N) prior frames or intervals, as well as to a learned tracking stage420, which is trained to track pathways over multiple frames or segments, and which outputs various detected pathways, optionally tagged with one or more attributes, as illustrated at422. In some implementations, the boundary heads and pathway heads may share a backbone network.

It will be appreciated that other types of machine learning architectures or algorithms that support memory or feedback, including, for example, various Recurrent Neural Network (RNN) architectures, Long Short-Term Memory (LSTM) architectures, etc., may be used in various implementations. Implementation and training of a machine learning model to incorporate the aforementioned functionality would be well within the abilities of those of ordinary skill having the benefit of the instant disclosure.

FIG.9illustrates an example roadway to further describe how attributes may be assigned to various perceived boundaries by a unified boundary machine learning model in some implementations. An autonomous vehicle430, for example, is illustrated as traveling on a four-lane divided roadway432including a pair of same-direction lanes434,436separated by shoulders438,440from a permanent concrete median barrier442and a non-road (e.g., grass) area444. A road edge446separates shoulder440from non-road area444, and in the upper half of the example illustration, a construction area448is defined by a plurality of spaced apart construction elements450(here, jersey barriers) that effectively close lane436and divert all traffic into lane434. In the lower half of the example illustration, a non-construction area452is defined, where no construction is currently underway and both lanes434,436are available for travel.

Within this example roadway, a number of different perceived boundaries, each having a one of a plurality of different semantic boundary types, may be generated and used in the control of an autonomous vehicle. For example, perceived boundaries associated with a physical barrier semantic boundary type are represented by permanent concrete median barrier442and boundary454that is defined by jersey barriers450, and a perceived boundary associated with a road edge semantic boundary type is represented by road edge446defined between shoulder440and non-road area444. Perceived boundaries associated with a painted lane semantic boundary type are represented by solid painted lines456and458and dashed painted line460.

As noted above, in some implementations, one or more attributes may be associated with boundaries or portions of boundaries. For example, arrows illustrated next to median barrier442, road edge446and boundaries454,456, and458represent learned sidedness, i.e., the side of the boundary upon which travel is permitted. As noted above, some boundaries do not have a sidedness attribute, e.g., boundary460associated with a dashed line, which allows for travel on each side of the boundary.

In addition, various active attributes may be associated with boundaries, portions of boundaries (e.g., boundary segments) or even regions or scenes. As one example, and as represented by the ‘A’ label on boundary454, at least a portion of boundary454may be associated with an active attribute representative of an active construction area (e.g., as a result of the presence of construction workers462). Likewise, as represented by the ‘A’ label on boundary458, at least a portion of boundary454may be associated with an active attribute representative of a vehicle464(e.g., a disabled vehicle, a police vehicle, an emergency vehicle, a tow truck, etc.) being stopped in shoulder440. In both instances, the use of an active attribute associated with the boundary may be used to trigger an operational change in the autonomous vehicle, e.g., to change lanes and/or reduce speed.

Furthermore, as represented by the ‘I’ labels on other segments of boundaries454and458, attributes may not be assigned to entire boundaries, e.g., only certain boundary segments or areas, such that, for example, where construction is only ongoing in one area but not another, or where no vehicles are present in the shoulder, the actions undertaken in response to active attributes may not be performed.

Dynamic Augmentation Of Autonomous Vehicle Map Data Using Perceived Lane Elements

Now turning toFIG.10, it may also be desirable in some implementations to use the lane elements perceived by a perception system, e.g., using a trained unified boundary machine learning model as discussed above, to identify potential discrepancies between the boundaries, lanes and/or pathways defined in stored map data and what is currently being perceived by the perception system of that autonomous vehicle. Such identified discrepancies may, in turn, be addressed by dynamically augmenting the map data such that downstream consumers of the map data, e.g., a motion controller component, may utilize a pathway generated for a perceived lane in the roadway to operate the autonomous vehicle along the roadway and may utilize one or more perceived pathways for correct actor intent prediction.

FIG.10, in particular, illustrates an example operational sequence500for controlling an autonomous vehicle (e.g., autonomous vehicle202ofFIG.2) based upon stored map data, e.g., as maintained in an on-board map502, as well as perception data, e.g., as output by a perception component504. On-board map502, for example, may be implemented similar to on-board map212as discussed above, while perception component504may be implemented similar to perception component214, and in some implementations may include a trained unified boundary machine learning model implemented in a similar fashion to the various implementations discussed above. In other implementations, however, other perception components, including perception components utilizing other types of trained machine learning models or otherwise, may be used to generate the perception data utilized herein.

The stored map data and perception data are compared in block506to attempt to identify a discrepancy between a mapped lane, i.e., a drivable lane on the roadway defined in the on-board map, and a perceived lane determined from the perception component. As an example, if construction has been initiated along a divided highway and one or more lanes on the roadway have been shifted some distance from their original locations (e.g., by depositing new painted lines on the roadway, or by placing cones, barrels, jersey barriers, etc. along the roadway), the drivable lane(s) of the roadway as detected by the perception component will not align with the mapped lane(s) of the roadway, so it will be understood that the original mapped lanes will not be suitable for use in navigating the autonomous vehicle.

Thus, block508determines whether a discrepancy has been identified, and if so, passes control to block510to determine if a predetermined detection duration has been met. For example, in some implementations, it may be desirable to require a discrepancy to be detected over multiple consecutive detection intervals prior to augmenting the map data, and thereby filter out any mis-identified discrepancies from individual detection intervals. Alternatively, block510may be omitted in some implementations.

Assuming first that a discrepancy has been detected over a sufficient duration, control passes from block510to block512to augment the map data (e.g., to generate an augmented road region) to address the identified discrepancy, e.g., by augmenting the map data to identify a pathway that may be used to navigate the autonomous vehicle along the roadway, such as within a perceived lane determined from the perception data. In addition, in some implementations, the augmentation may also include other operations such as disabling, hiding, deleting, or otherwise deactivating one or more mapped lanes represented in the map data to cause the motion planner to effectively ignore the mapped lane when determining the trajectory of the autonomous vehicle. Augmentation in some implementations may also involve adding new/additional lane content, including lane content for individual or multiple lanes, as well as lanes from painted boundaries or construction inferences, as well as updating/deleting any lane content, or any combination thereof.

Next, in block514one or more active lanes in the roadway are determined using the augmented map data, and a trajectory for the autonomous vehicle is determined using the active lane(s) in block516, both of which may be performed, for example, in a motion planner component. In other implementations, block514may be performed in a map fusion component. In addition, the motion planner component may also optionally utilize one or more object tracks in order to determine the trajectory for the autonomous vehicle. For example, in block518, one or more objects may be generated using mainline perception similar to block312discussed above, and in block520, a tracking component may generate tracks for the generated objects such that the determination of the trajectory in block516also uses the generated object tracks. The autonomous vehicle is then controlled in block522using the determined trajectory. Returning to blocks508and510, if no discrepancy is identified, or if any discrepancy has not been detected over enough detection intervals, block512is bypassed, so that the determination of the active lanes and the trajectory in blocks514and516, and the control of the autonomous vehicle in block518, is based on non-augmented map data.

The stored map data provided by on-board map502, and the perception data output by perception component504, may include various types of data associated with the lanes, boundaries and/or pathways associated with the roadway, e.g., various stored and/or perceived lane elements (including, in some implementations, boundary-type lane elements and/or pathway-type lane elements). In some implementations, some lane elements may also directly define portions of a lane, separate from boundaries and/or pathways. In the illustrated implementation, for example, the stored map data may include a road region layout that includes map data such as various boundaries for the roadway, one or more lanes for the roadway, as well as additional data such as road edges, representing the outer physical extents of the roadway, signs and other static and/or logical objects, and other map data that may be utilized by a motion planner component to generate a trajectory for the autonomous vehicle. The perception data output by perception component504may include various lane elements (e.g., boundaries and/or pathways) detected along the roadway, including semantic boundary types assigned to each of the various boundaries, as well as object tracks associated with various objects in the environment. In other implementations, however, the stored map data and/or the perception data may represent a roadway in other manners, including using various combinations of lane elements, boundaries, lanes and/or pathways, so the invention is not limited to the specific representations of a roadway described herein.

As noted above, the stored map data and the perception data may be used to identify a discrepancy between a mapped lane and a perceived lane in block506ofFIG.10, which is referred to herein as a discrepancy identification operation. One example manner of implementing such an operation is illustrated by operational sequence530ofFIG.11, which begins in block532by determining mapped and perceived lane elements (e.g., boundaries, lanes and/or pathways) from the map and perception data. In some implementations, the map and/or perception data may be output in a format that is suitable for discrepancy identification, while in other implementations, one or both of the map and perception data may be processed to obtain suitable data for performing discrepancy identification. For example, in some implementations, the perception component may output perceived boundaries, but may not output perceived lanes associated with such boundaries, so identification of a discrepancy between a mapped lane and a perceived lane may be based upon analysis of individual mapped and perceived boundaries, rather than a direct comparison of a mapped lane and a perceived lane, and one or both of the map data and the perception data may need to be processed to generate suitable data for discrepancy identification (e.g., to generate a lane from a pair of boundaries, to generate boundaries and/or a centerline from a lane, etc.). Thus, it should be appreciated that while the implementations discussed herein describe the identification of a discrepancy between a mapped lane and a perceived lane, such an identification is not limited to a direct comparison of data structures representing the extents of mapped and perceived lanes, but may also include other comparisons based on data representing other aspects of a roadway (e.g., the boundaries and/or pathways/centerlines that may be used to define a lane in some implementations) and other metadata to inform probabilistic, or confidence based reasoning, or to facilitate a learned model. For example, the perception system in some implementations may provide confidence bounds on the spatial locations and attributes of detected boundaries, occlusion masks, consistency scores (temporal integration), etc.

Next, in block534, geometric entities may be generated to represent the mapped and perceived lane elements (e.g., boundaries, lanes and/or pathways). In one example implementation, in particular, mapped and perceived boundaries may be represented by one or more linear and/or curvilinear geometric entities such as polylines or cubic splines, and discrepancy identification may be based on a comparison of such line representations. In such an implementation, polylines or cubic splines may be generated from mapped lanes in the event that the map data does not include specific boundary data structures. It will be appreciated, however, that other geometric entities, including other straight and/or curved line representations, as well as various polygonal or other two-dimensional shape representations, may be generated in other implementations. In some cases, perceived elements and map elements may be compared directly to each other without transforming them to other representations, and block534may be omitted. In addition, in some implementations, geometric entities may be generated to represent pathways, such that discrepancy identification may be based on a comparison of the geometric entities representing pathways.

Next, in block536, an attempt is made to match the geometric entities generated from the map data with the geometric entities generated from the perception data in order to identify any discrepancies. Discrepancies may be identified, for example, based upon identifying a lane element (e.g., a boundary, lane, or pathway) present in the map data but missing from the perception data, or vice versa, or by identifying that the position of a lane element in the map data that corresponds to a lane element in the perception data is substantially different from the position of that in the perception data. In some implementations, discrepancies may also be identified at least in part based upon boundary attributes or styles (e.g., solid vs. dashed painted lines, yellow vs. white painted lines, etc.). It will be appreciated that thresholds may be used to determine when two geometric entities are close enough to be considered to be matched, as well as to determine when the positions of those geometric entities is different enough to be considered a discrepancy. In addition, weighting may be used in some implementations, e.g., to more heavily weight closer perceived lane elements (which are often more reliable) than more distant perceived lane elements. In still other implementations, a machine learning model may be trained to associate geometric entities with one another to identify discrepancies. It will be appreciated that more than one discrepancy, for more than one lane, may also be detected concurrently in some implementations (e.g., where multiple lanes are shifted in a construction zone).

As a result, one or more discrepancies may be identified in operational sequence530, and in some implementations, such identification may trigger addressing the discrepancy, in particular by performing a map data augmentation operation represented by block512ofFIG.10. One example manner of implementing such an operation is illustrated by operational sequence540ofFIG.12, which begins in block542by disabling all of the mapped lanes in the map data. A mapped lane may be disabled in some implementations by deleting the mapped lanes from the augmented map data, while in other implementations, the mapped lanes may be disabled in other manners, e.g., by marking them as disabled in the augmented map data, or assigning them to a list or other data structure that specifies that they should be skipped when traversing a tree or graph representing a road region. In some implementations, for example, mapped lanes may be defined by pathway elements, and a mapped lane may be disabled by omitting the associated pathway element from the augmented map data.

In addition, in some implementations, it may be desirable to activate a dedicated “perceived” mode for the autonomous vehicle, whereby the autonomous vehicle is controlled to be operated based upon perceived boundaries, lanes and/or pathways. In other implementations, mapped lanes may not be disabled. In still other implementations, an autonomous vehicle may always be operated based on fused lanes, using the HD map as a prior.

Next, in block544, a pathway is generated for a perceived ego lane, i.e., the perceived lane within which the autonomous vehicle is currently operating. The pathway may be represented in a number of manners, e.g., by determining left and right boundaries or guides for the pathway and/or determining a centerline for the pathway, in some instances based upon the positions of the perceived lane elements defining the perceived lane. A pathway may also be defined by a plurality of path segments (e.g., path segments defining a centerline of the path) that are linked to one another in some implementations. In addition, in some implementations, pathways corresponding to one or more additional lanes, e.g., one or more adjacent lanes for the roadway, may also be added, along with relationships between lanes (e.g., to identify when lane changes are permissible or not), thereby enabling a motion planner component to consider the desirability of changing lanes even when operating based upon perceived lane elements. However, in other implementations, only an ego lane and pathway may be generated to simplify augmentation and navigation.

Next, in block546, it may also be desirable in some implementations to update a speed limit and/or other lane attributes for the ego lane and pathway and/or any adjacent lanes or pathways, e.g., based upon the last used speed limit from the map data. In some instances, speed limits that were associated with disabled map lanes may be associated with a perceived lane, or with the edge of the roadway. In addition, other attributes, such as the addition of stop signs or other road signs that apply to one or more lanes.

In some implementations, augmented map data may retain the same general format as the map data output by the on-board map, and an Application Programming Interface (API) used by downstream consumers of map data may be generic to both augmented and un-augmented map data. By doing so, consumers of the map data (e.g., a motion planner component) may be able to use the augmented map data in a similar manner to non-augmented map data, e.g., to query for lanes or pathways in the same manner and/or call the same functions, regardless of whether the map data is augmented or un-augmented.

FIGS.13-15next illustrate an example roadway to further describe how map data may be augmented to enable an autonomous vehicle to operate using perceived lane elements in response to identified discrepancies.FIG.13in particular illustrates a stored map data representation of a roadway550, which defines a pair of same-direction lanes552,554bordered by a pair of shoulders556,558bordering a permanent concrete median barrier560and a non-road (e.g., grass) area562. A road edge564separates shoulder558from non-road area562, while a pair of solid painted lines566and568are represented as boundaries between lane552and shoulder556, and between lane554and shoulder558, respectively. A dashed white painted line570is represented as a boundary between lanes552and554. In some implementations, each lane552,554may represent a pathway for an autonomous vehicle, and may be represented, for example, using a centerline572,574, left boundaries or guides576,578, and/or right boundaries or guides580,582.

With reference toFIG.14, assume that when an autonomous vehicle (not shown inFIG.14) is present on the roadway, it detects a number of perceived boundaries, e.g., a new painted line type boundary584that has been applied to the roadway as well as a virtual construction type boundary586defined by a plurality of detected cones or barrels588and a virtual construction type boundary590defined by a plurality of detected cones or barrels592. Also detectable are the original solid white painted lines566,568and dashed white painted line570. When processing the perception data in the manner described above in connection withFIGS.10-12, a discrepancy may be determined between the mapped lanes552,554and a perceived lane594that is effectively defined by perceived boundaries584,586, e.g., as a result of attempting to match perceived boundaries584,586with the mapped boundaries566,568, and570. Based upon the identified discrepancy, the map data may be augmented, e.g., using a centerline596and/or left and right boundaries or guides598,600to define a pathway along the roadway. In addition, as represented by the omission of lanes552,554fromFIG.14, the mapped lanes may be disabled in the augmented map data, such that a motion planner component consuming the augmented map data will generate a trajectory based upon the single ego lane or pathway now represented in the augmented map data.

Notable in this implementation is that a single ego lane or pathway is generated (e.g., based upon where the autonomous vehicle is currently operating), so even if other potential lanes or pathways are available, no additional lanes are added to the augmented map data. As illustrated inFIG.15, however, in other implementations, it may also be desirable to augment the map data to include a pathway corresponding to one or more adjacent lanes, e.g., for lane602a pathway defined by a centerline604and/or left and right boundaries or guides606,608. In some implementations, it may not be desirable to define a pathway or lane that overlaps shoulder558; however, in other implementations, it may be permissible to do so. Furthermore, the support for adjacent perceived lanes may also allow for lane changes to be planned by a motion planner component in appropriate circumstances.

Dual Mode Map

While augmenting map data with perceived lane elements may address a number of discrepancies, there may be other circumstances that require additional efforts to allow for continued operation of an autonomous vehicle in some environments. In particular, there may be circumstances, e.g., as a result of recently-established construction areas, where the existing map data is insufficient for continued fully autonomous operation of an autonomous vehicle in such areas. Some types of construction, for example, may result in changes to the underlying road surface, and require vehicles to operate in areas that are outside of what is considered to be the drivable surface within the environment. For example, some types of construction may require vehicles to be diverted onto the shoulder of a roadway, or even onto new pavement that has been installed adjacent to the existing roadway.

In addition, some types of construction may require vehicles to be diverted onto a road surface that is normally used for opposite direction traffic. As an example, in some construction areas, where there is a divided highway with two lanes in each direction, some construction projects will close all of the lanes in one direction and place barriers between the two lanes in the other direction, and require vehicles traveling in the direction where both lanes are closed to be routed across the divide between the two sets of lanes (e.g., defined by a grass median or a concrete barrier) to travel in one of the lanes normally assigned to the opposite direction (i.e., for oncoming traffic). In some instances, additional pavement may also be installed across the divide to provide a drivable lane between the two sets of lanes.

As yet another example, changes to an environment may be so substantial that localization is inhibited, causing what may be referred to as a global pose fault. In many instances, localization relies in part on localization data (e.g., surface elements) defining the perceivable surfaces in the environment and maintained as stored map data for the environment. Perception sensor data, e.g., LIDAR data, may then be captured during operation of a vehicle, and the surfaces represented in the perception sensor data may be compared with the stored localization data to determine a global pose of the vehicle (generally position and orientation) that most closely aligns the perception sensor data with the stored localization data. Where changes to an environment, however, are so substantial that no suitable alignment can be found between the perception sensor data and the stored localization data, it may not be possible to determine a current pose for the autonomous vehicle, thereby causing a global pose fault to occur.

A situation requiring an autonomous vehicle to drive off of what is considered the drivable road surface, as well as a situation that causes an autonomous vehicle to be unable to determine its global pose, are both situations that generally have been considered to be situations where full autonomous operation may not be desired. As a result, in many instances an autonomous vehicle may be required to take remedial actions in response to such situations, such as contacting a teleassist system and/or pulling over to the side of the road and stopping.

In some implementations, however, a dual mode map may be used to allow an autonomous vehicle to operate in different modes in different locations. In particular, and with reference toFIG.16, it may be desirable in some implementations to utilize a dual mode map610that includes, for different portions of an environment, dense map data612and sparse map data614. The dense map data612may include, for example, ground surface data616, localization data618, and semantic data620, while sparse map data614may include ground surface data622and localization data624. Sparse map data614, however, may generally omit any semantic data, or at least a portion of the semantic data that may be included in dense map data612. Furthermore, ground surface data622and/or localization data624of sparse map data614in some implementations may be omitted, or if included, may be different from the ground surface data616and/or localization data618of dense map data612in some respects. For example, in some implementations, ground surface estimates may be generated onboard an autonomous vehicle based upon sensor data, while in some implementations, localization data may be generated onboard an autonomous vehicle, e.g., using GNSS(GPS) and/or IMU data, to establish enough of a pose to enable navigation using a sparse map without the use of dense localization/LIDAR priors. Thus, in some implementations, the ground surface data622and/or localization data624of spare map data614may be considered to be a more limited representation of ground surface and/or localization data than provided by ground surface data616and/or localization data618of dense map data612.

Dense map data612, in some implementations, may be considered to be the “full” map data for a particular portion of an environment, representing the stored map data that has been developed via a map generation process and providing a high resolution (high quality) representation of the various static objects in the portion of the environment, e.g., including real word objects or elements such as roads, curbs, buildings, trees, signs, etc., as well as logical elements such as lanes, boundaries, etc. Dense map data may be generated, for example, based upon logs of sensor data collected from multiple vehicles traveling through an environment, and may be developed using a map generation pipeline that incorporates various automated and/or manual steps, including quality assurance and verification. Dense map data may be updated from time to time via a map update process, and in some implementations, may be associated with a release identifier that identifies a version of the dense map data when the dense map data is deployed to a fleet of autonomous vehicles. It will be appreciated that due to the need to verify the accuracy of dense map data before deployment, the timeline for updating dense map data may be on the order of several days.

The portion of an environment represented by dense map data may vary in different implementations. For example, dense map data may be arranged into road regions, tiles, or practically any other suitable organization, and while dense map data may represent differently-sized portions in some implementations, in other implementations, the portions of the environment (e.g., road regions, tiles, etc.) may be similarly sized and arranged in a regular grid.

As noted above, dense map data612in some implementations may include ground surface data616. In this regard, ground surface data may be considered to describe the ground surface of the roadway, and may include at least data that describes the outer physical boundary or boundaries of a roadway. The ground surface data may be two or three dimensional in nature, and in some implementations, may encompass all potentially drivable areas of a roadway, while in other implementations, may exclude some potentially drivable (but ordinarily restricted) areas of a roadway, e.g., shoulders, medians, etc. Boundaries may be represented by combinations of linear, curved, curvilinear, or other suitable boundary representations as discussed above.

Dense map data612may also include localization data618, which generally includes any data suitable for assisting an autonomous vehicle in determining a pose (location and/or orientation) of the vehicle within the environment. In some implementations, for example, localization data may include collections of surfels (surface elements) that represent the surfaces of real world static objects in the environment, such that perception data collected by the perception system (e.g., point clouds generated from LIDAR and/or RADAR data) may be aligned with the surfels when determining the pose of the vehicle. Other data suitable for determining localization may also be used in other implementations, as will be appreciated by those of ordinary skill having the benefit of the instant disclosure.

Dense map data612also includes various types of semantic data620, which generally describe real world static objects or elements in the environment such as roads, curbs, buildings, trees, signs, traffic signals, etc., as well as logical elements such as lanes, boundaries, etc., thereby providing an autonomous vehicle control system with a semantic understanding of the environment. It will be appreciated that it is often the semantic data associated with an environment that becomes invalid in response to changes in the environment, such as road construction, as previously-understood boundaries, lanes, pathways, etc. may no longer be valid after road construction has been initiated in a particular area. Semantic data in some implementations, for example, may include one or more mapped boundaries, one or more mapped lanes, one or more speed limits, one or more mapped signs and/or one or more traffic signals.

Sparse map data614, in many implementations, is intended to be a temporary representation of a portion of an environment that allows an autonomous vehicle to operate, potentially (but not necessarily) in a degraded state, within the portion of the environment, despite the absence of a full representation of that portion of the environment as might be provided by dense map data generated for that portion of the environment. In this regard, sparse map data may also be considered to be a “lightweight” map of the portion of the environment, to distinguish from a full map of the portion of the environment provided by dense map data. In some implementations, sparse map data may omit any semantic data, while in other implementations, sparse map data may include some, but not all, of the semantic data provided in dense map data. Likewise, ground surface and localization data in some implementations may be comparable to that provided by dense map data, while in other implementations, the ground surface and/or localization data provided in sparse map data may be limited relative to that provided by corresponding dense map data.

While sparse map data may not be temporary in some implementations, in the implementations illustrated herein, the sparse map data for a particular portion of an environment may be temporary in nature, and generated in response to a determination that dense map data for the same portion of the environment is not sufficient (e.g., due to a change in roadway boundaries, a global pose fault, etc.) to support continued autonomous operation in the portion of the environment. Moreover, the sparse map data may be generated using a workflow that is outside of, and accelerated relative to, the standard map update process used to maintain map data for a fleet of autonomous vehicles.

In some implementations, for example, once an issue with a portion of the environment is detected, e.g., by one of the autonomous vehicles in a fleet, a rapid remapping process may be used to generate sparse map data for the portion of the environment, which may then be deployed to one or more other autonomous vehicles in the fleet to effectively replace the dense map data for the portion of the environment with the deployed sparse map data such that when the other vehicles are located in the same portion of the environment, the autonomous vehicles may utilize the sparse map data to operate within the portion of the environment. In addition, in a parallel process, or alternatively, a process that continues once the sparse map data has been generated, dense map data for the portion of the environment may later be generated and then deployed to one or more autonomous vehicles in the fleet to replace the sparse map data with the newly deployed dense map data, thereby enabling the newly deployed dense map data to be used in the portion of the environment.

In some implementations, for example, generation of sparse map data in response to detection of a triggering event (e.g., detected construction, detected modified roadway boundary, global pose fault, etc.) may be performed using a workflow that is on the order of several minutes or hours, and generally less than 24 hours, and may utilize over-the-air deployment to autonomous vehicles in a fleet. The generation of the sparse map data, for example, may rely in part on multiple logs collected from one or more vehicles that are operated within the portion of the environment after the triggering event (potentially with the assistance of a human operator or teleassist). Conversely, generation of dense map data may be performed using a workflow that is on the order of several days from the triggering event (e.g., 3 or more days), and may rely on additional logs collected over that timeframe, along with generation of semantic map data, using a map update workflow that is similar to a normal map update process used to maintain a current map for an environment. Furthermore, deployment of the dense map data may take the form of a formal map update, where a new release or version of a map is deployed. Thus, in some implementations, the sparse map data may only be used for a temporary time period, until such time as a formal map update may be performed to provide full scope dense map data after the triggering event.

As noted above, in some implementations, an autonomous vehicle may operate in a dedicated mode when operating within a portion of the environment for which sparse, rather than dense, map data is available. In some implementations, for example, an autonomous vehicle may be operated in a perceived boundary control mode, which operates in a similar manner to that described above in connection withFIGS.10-15, where dynamic augmentation of map data may be used to generate a pathway for a perceived lane in the roadway in response to detection of one or more perceived lane elements by a perception system of the autonomous vehicle. It will be appreciated, in particular, that sparse map data in some implementations may lack any mapped lanes, such that in the perceived boundary control mode, one or more perceived lanes are instead generated while the autonomous vehicle operates using the sparse map data.

FIG.17illustrates an example operational sequence630for controlling an autonomous vehicle using the dual mode map ofFIG.16, e.g., as may be performed by an autonomous vehicle control system. In this implementation, the autonomous vehicle as a default operates in a mapped lane mode, and selectively transitions to a perceived boundary control mode when a determination is made that the autonomous vehicle is in a portion of the environment for which sparse, rather than dense, map data is available. In other implementations, however, the perceived boundary control mode may be the default mode of operation. Sequence630begins in block632, where map data corresponding to the vehicle location is obtained from the stored map. Next, in block634, a determination is made as to whether the vehicle is in a lightweight map area, i.e., whether the map data corresponding to the vehicle's location is sparse map data rather than dense map data.

If not (i.e., the autonomous vehicle is operating in an area supported by dense map data), control passes to block640, where one or more mapped lanes are obtained from the dense map data. Then, in block642, one or more active lanes in the roadway are determined, and in block644, a trajectory for the autonomous vehicle is determined using the active lane(s), both of which may be performed, for example, in a motion planner component. In other implementations, block642may be performed in a map fusion component. In addition, the motion planner component may also optionally utilize one or more object tracks in order to determine the trajectory for the autonomous vehicle, similar to that described above in connection withFIG.10. In some implementations, observations of other autonomous vehicles encountering the portion of the environment and provided via the live map system may also be used to generate active lanes and/or a trajectory. The autonomous vehicle is then controlled in block646using the determined trajectory, effectively following an ego pathway corresponding to the active lane selected in block642and the trajectory determined in block644.

Returning to block634, if the autonomous vehicle is instead operating in an area supported by sparse map data, control instead passes to block648to enable the perceived boundary control mode, and then to block650to disable lane alignment localization, e.g., to disable lane alignment component376ofFIG.7, such that localization of the autonomous vehicle does not rely on any alignment between mapped and perceived lanes. In addition, in some implementations, detection of entry into any areas supported by sparse map data may also be used to reserve teleassist resources in the event that teleassist assistance may be needed.

In the perceived boundary control mode, the sparse map data may be augmented with a pathway generated in response to perceived lane elements, so control passes to block652to generate one or more perceived lanes from perceived boundaries detected by the perception system of the autonomous vehicle and augment the sparse map data to generate augmented sparse map data. Control then passes to blocks642,644, and646to determine one or more active lanes in the roadway using the perceived lanes, generate a trajectory for the autonomous vehicle using the active lane(s) (and optionally also using one or more object tracks), and control the autonomous vehicle using the determined trajectory.

FIGS.18A-18Cillustrate an example operational sequence660for generating, deploying and using a dual mode map consistent with some implementations. As illustrated inFIG.18A, operational sequence660may be initiated within an autonomous vehicle662, referred to herein as a first contact autonomous vehicle, which initially in block664detects a map anomaly when operating within a portion of an environment. For example, as illustrated in block666, the map anomaly in some instances may be a global pose fault, where the autonomous vehicle is unable to determine its current pose, while in other instances, and as illustrated in block668, the map anomaly may be detection of a modified roadway boundary, such as additional pavement that extends beyond mapped roadway boundary, or pavement that directs vehicles into lanes that are mapped for travel in an opposite direction (also referred to herein as oncoming lanes). It will be appreciated that other map anomalies that might warrant the generation of sparse map data may also be detected in other implementations, so the specific examples given herein are not exclusive.

In response to detection of the map anomaly, control passes to block670to generate a notification of the map anomaly, and then to block672to handle the map anomaly. The notification of the map anomaly, for example, may be communicated to an online or offline map service674(FIG.18B), or may be communicated to other remote services available to the autonomous vehicle, e.g., a teleassist service. Handling of the map anomaly in block672may include, for example, initiating a teleassist session to enable a teleassist operator to assist with guidance of the autonomous vehicle through the area of the map anomaly, or alternatively, directing the autonomous vehicle to the side of the roadway (e.g., in the shoulder) to await recovery. In other implementations, e.g., where a human driver is present in the autonomous vehicle, handling of the map anomaly may include transferring control to the human driver to navigate through the area of the map anomaly.

Now turning toFIG.18B, the map service674may receive the notification in block676and initiate in response thereto a lightweight map generation workflow to generate and deploy sparse map data for the portion of the environment associated with the map anomaly. The portion of the environment associated with the map anomaly may include, for example, one or more road regions, tiles, or other suitable organizations of a geographical area. The workflow, for example, may include, in block678, collecting logs from one or more vehicles that have traversed through the portion of the environment associated with the map anomaly. In some instances, for example, log data from the first contact autonomous vehicle662may be used along with log data from one or more other vehicles that are operated within the portion of the environment after detection of the map anomaly. In some implementations, the logs may be collected as vehicles (autonomous and/or non-autonomous) encounter the portion of the environment in their normal operation, while in other implementations, one or more vehicles may be intentionally directed into the portion of the environment in response to detection of the map anomaly to accelerate the log collection process. In addition, in some implementations, the reception of the notification in block676may also result in deployment of live map data via a live map service (e.g., live map service220ofFIG.2) to notify other autonomous vehicles of the map anomaly, and in some instances, cause such other autonomous vehicles to automatically initiate teleassist sessions when entering the portion of the environment, to cede control to a human driver, avoid the portion of the environment, or other proactive measures that may be desirable prior to suitable sparse map data being deployed.

Returning toFIG.18B, after a suitable amount of log data is collected, control passes to block680to generate sparse map data for the portion of the environment associated with the anomaly. For example, a tile or road region may be generated and initially populated with ground surface data, e.g., one or more outer roadway boundaries representing the current boundaries of the roadway as determined from the logs.

Block682then determines whether the map anomaly is associated with a global pose fault, and if so, passes control to block684to generate localization data for the portion of the environment from the logs (e.g., using LIDAR point cloud data or surfels stored in the logs) and incorporate the localization data into the sparse map data. Control then passes to block686to deploy the sparse map data as a lightweight map update (e.g., through a live map service, or via a separate service).

Returning to block682, if the map anomaly is not associated with a global pose fault, control instead passes to block688to reuse localization data from the corresponding dense map data for the portion of the environment associated with the map anomaly, and then to block686to deploy the sparse map data. In particular, where a map anomaly is associated with a modified outer roadway boundary, rather than a global pose fault, the localization data associated with the dense map data may still be suitable for use, so rather than recreate the localization data from the logs, it may be desirable in some implementations to reuse the existing localization data from the dense map data. In other implementations, however, it may be desirable to regenerate localization data from the logs irrespective of the type of map anomaly. In addition, in some implementations, a combination of reused and new localization data may be incorporated into sparse map data, e.g., to regenerate localization data only in areas where sufficient existing localization data does not exist.

It will be appreciated that the generation of the sparse map data may be performed in numerous different manners, and may employ various techniques utilized in connection with generating map data for autonomous vehicles. Such techniques may include manual map data generation, autonomic map data generation, or a combination of manual and automatic map data generation. In some implementations, for example, automatic tools may be used to generate sparse map data, and then manual quality assurance steps may be undertaken to verify the accuracy of the automatically-generated map data, and correct the map data as needed.

Returning toFIG.18A, autonomous vehicle662may receive the lightweight map update in block690and effectively replace the dense map data for the portion of the environment affected by the map anomaly with the sparse map data provided in the lightweight map. Therefore, in block692, the dense map data for the portion of the environment may be invalidated, deactivated or disabled, and in block694the sparse map data provided in the lightweight map update may be stored and activated, such that whenever the autonomous vehicle is located in the portion of the environment, the sparse map data will be used in the manner discussed above in connection withFIG.17. It will be appreciated that various manners of deactivating the dense map data and activating the sparse map data may be used in different implementations, including by selectively overwriting portions of the dense map data, overwriting the dense map data with the sparse map data, or in other manners that will be appreciated by those of ordinary skill having the benefit of the instant disclosure.

Returning again toFIG.18B, after the sparse map data has been generated and deployed as a lightweight map update, map service674may proceed with generating corresponding dense map data in block696. While in some implementations, the dense map data may be generated in a completely separate pipeline or process from the sparse map data, in the implementation ofFIG.18B, the sparse map data is used as a starting point for the dense map data, and the dense map data is generated in part by supplementing the sparse map data with semantic data. It will be appreciated that the generation of the semantic data may be a more resource and time-intensive process than generating the sparse map data, and as such, may take several days to complete in some implementations. However, by virtue of the deployment of the sparse map data prior to generation of the dense map data, a fleet of autonomous vehicles may still be able to operate within the portion of the environment associated with the map anomaly prior to completing the dense map data. It will also be appreciated that the dense map data generation process may rely on additional logs, and may supplement the sparse map data to refine or improve the ground surface and/or localization data that is ultimately deployed as dense map data in some implementations.

Once the dense map data has been generated, refined and validated, the dense map data may be deployed in block698, e.g., as part of a map release, where a new version of the map is deployed to the fleet of autonomous vehicles. With reference toFIG.18A, the deployment of the dense map data may be processed in autonomous vehicle662by receiving the map release in block700, invalidating, disabling or deleting the sparse map data in block702, and storing the newly generated dense map data in block704, such that the autonomous vehicle will thereafter utilize the dense map data when operating in the portion of the environment associated with the map anomaly.

In addition, with reference toFIG.18C, it will be appreciated that the lightweight map update that deploys sparse map data, and the full map release that deploys dense map data, may generally be deployed to multiple autonomous vehicles, including some or all of the autonomous vehicles in a fleet, so that the autonomous vehicles receiving the deployments are also able to navigate through the portion of the environment using the generated sparse and/or dense map data. Thus,FIG.18Cillustrates another autonomous vehicle706that, in response to a lightweight map update, receives the update in block708, invalidates, disables, or deletes the corresponding dense map data in block710, and stores or activates the sparse map data provided in the lightweight map update in block712, in a similar manner to that described above in connection with blocks690-694ofFIG.18A. Likewise, in response to a full map release, receives the release in block714, invalidates, disables, or deletes the sparse map data in block716, and stores or activates the dense map data provided in the full map release in block718, in a similar manner to that described above in connection with blocks700-704ofFIG.18A.

FIGS.19-22next illustrate an example roadway to further describe how sparse map data may be generated for a dual mode map to enable an autonomous vehicle to operate in a portion of an environment subject to a map anomaly, e.g., initiation of a construction project that routes traffic in a roadway into an oncoming lane normally used for oncoming traffic, as well as onto a new segment of roadway added along the shoulder of the roadway.FIG.19in particular illustrates a stored map data representation of a roadway750, which defines, for a north-bound portion of the roadway, a pair of same-direction lanes752,754(with a north-bound direction of travel represented by the upward arrows) bordered by a pair of shoulders756,758bordering a permanent concrete median barrier760and a non-road (e.g., grass) area762. A road edge764separates shoulder758from non-road area762, while a pair of solid painted lines766and768are represented as boundaries between lane752and shoulder756, and between lane754and shoulder758, respectively. A dashed white painted line770is represented as a boundary between lanes752and754. In some implementations, each lane752,754may represent a pathway for an autonomous vehicle, and may be represented, for example, using a centerline772,774, or by left/right boundaries or guides (not shown inFIG.19). It should be noted that in this implementation, shoulders756,758are considered to be normally restricted to traffic, and thus no mapped lanes are represented in the stored map data representation in the regions covered by shoulders756,758.

Similarly, for a south-bound portion of roadway750(only a portion of which is shown), a pair of same-direction lanes776,778(with a south-bound direction of travel represented by the downward arrows) are illustrated, along with a shoulder780bordering concrete median barrier760. A solid painted line782forms a boundary between lane776and shoulder780, and a dashed white painted line784is represented as a boundary between lanes776and778. In some implementations, each lane776,778may represent a pathway for an autonomous vehicle, and may be represented, for example, using a centerline786, as illustrated for lane776, or by left/right boundaries or guides (not shown inFIG.19). Similar to shoulder756, shoulder780is considered to be normally restricted to traffic, and thus no mapped lane is represented in the stored map data representation in the region covered by shoulder780.

It will be appreciated that the stored map data representation of roadway750may be considered to be dense map data providing a “full” map of the roadway, and at least a portion of elements752-786may be considered to represent semantic data associated with the roadway. In addition, the stored map data representation may also include ground surface data that models the surface, as well as one or more road boundaries, e.g., road boundaries788,790,792, that define the outer physical boundaries of the roadway. The stored map data may also include localization data, e.g., surfels associated with the various physical objects (e.g., barrier760) along the roadway.

FIG.20next illustrates roadway750, but after a detected change in the ground surface data, e.g., by an autonomous vehicle794operating along the roadway. InFIG.20, construction has been initiated, resulting in the left lane (lane752) of the north-bound portion of the roadway being routed across a break in barrier760, within which new road surface796has been paved, and into the south-bound portion of the roadway (specifically south-bound lane776), and with the right lane (lane754) of the north-bound portion of the roadway being routed into shoulder758as well as onto a newly-paved road surface798. Cones800lead to jersey barriers802,804,806, which define two new construction-defined lanes808,810.

It will be appreciated that as a result of first contact by autonomous vehicle794with the roadway presented inFIG.20, a map anomaly error may be detected, e.g., as a result of detecting changes in any or all of road edge boundary788(which has been expanded beyond shoulder758and road edge764) and road edge boundaries790,792(which have been broken up by the removal of some of barrier760and the addition of pavement796). Alternatively, a map anomaly error may be detected as a result of a global pose fault, which may occur due to the removal of the portion of barrier760and the addition of jersey barriers802,804and806.

Regardless of the cause of the map anomaly error, a lightweight map update may be generated and deployed as described above in connection withFIGS.18A-18C, resulting in the generation of sparse map data812as illustrated inFIG.21. In sparse map data812, new ground surface data, e.g., road edge boundaries814,816,818, may be included, as well as localization data, e.g., surfels corresponding to barrier820and jersey barriers822,824,826. Semantic data, e.g., boundaries, lanes, pathways, etc., on the other hand, may be omitted from sparse map data812.

Then, as shown inFIG.22, when an autonomous vehicle828encounters the construction area on roadway750, the sparse map data812may be augmented with one or more pathways corresponding to one or more perceived lanes detected by the perception system of the autonomous vehicle, e.g., based on perception of boundaries defined by cones800, barrier820, and jersey barriers822,824,826. For example, two pathways may be detected inFIG.22, corresponding to left and right construction lanes, and defined by centerlines830,832and/or left/right boundaries or guides834/836,838/840.

It will be appreciated that, while certain features may be discussed herein in connection with certain implementations and/or in connection with certain figures, unless expressly stated to the contrary, such features generally may be incorporated into any of the implementations discussed and illustrated herein. Moreover, features that are disclosed as being combined in some implementations may generally be implemented separately in other implementations, and features that are disclosed as being implemented separately in some implementations may be combined in other implementations, so the fact that a particular feature is discussed in the context of one implementation but not another should not be construed as an admission that those two implementations are mutually exclusive of one another. Other variations will be apparent to those of ordinary skill. Therefore, the invention lies in the claims hereinafter appended.