Systems and methods for map verification

Systems and methods are provided for verifying the accuracy of a map. Information regarding objects, such as roadway or roadway-related elements may be encoded into a map to be used for navigation and/or control of a vehicle, such as an autonomous vehicle. In order to verify that the map accurately reflects the roadway or roadway-related elements making up a section of roadway, the vehicle may be driven/ridden along the same section of roadway. A camera feed can be captured of this subsequent traversal of the section of roadway to check if the map includes the requisite roadway or roadway-related elements. The known roadway or roadway-related elements can be occluded from view in the camera feed. Accordingly, if any roadway or roadway-related elements do appear in the camera feed, they can be more easily detected by a verifier, and the processor verification is simplified.

TECHNICAL FIELD

The present disclosure relates generally to autonomous driving, and in particular, some implementations may relate to verifying accuracy of an encoded map to be used during autonomous vehicle operation.

DESCRIPTION OF RELATED ART

Advanced driver-assistance systems (ADAS) can refer to electronic systems that assist a vehicle operator while driving, parking, or otherwise maneuvering a vehicle. ADAS can increase vehicle and road safety by minimizing human error, and introducing some level of automated vehicle/vehicle feature control. Autonomous driving systems may go further than ADAS by leaving responsibility of maneuvering and controlling a vehicle to the autonomous driving systems. For example, an autonomous driving system may comprise some package or combination of sensors to perceive a vehicle's surroundings, and advanced control systems that interpret the sensory information to identify appropriate navigation paths, obstacles, road signage, etc.

BRIEF SUMMARY OF THE DISCLOSURE

In accordance with one embodiment, a computer-implemented method, comprises presenting a traveling scene regarding a section of roadway being traversed by a vehicle, and determining known roadway objects in the traveling scene based on a previously generated map. The computer-implemented method may further comprise occluding the known roadway objects in the traveling scene upon subsequent traversal of the section of roadway. Moreover, the computer-implemented method may comprise receiving input regarding visible roadway objects in the traveling scene during the subsequent traversal of the section of roadway to verify accuracy of the previously generated map.

In some embodiments, the presentation of the traveling scene is performed in real-time during a map verification process.

In some embodiments, the presentation of the traveling scene is performed during an off-line map verification process.

In some embodiments, the determination of the known roadway objects comprises comparing a current location of the vehicle to a commensurate location represented by the previously generated map. In some embodiments, the determination of the known roadway objects further comprises identifying roadway objects encoded in the previously generated map at the current location of the vehicle. In some embodiments, the presentation of the traveling scene comprises presenting a camera feed of the traveling scene captured by a camera of the vehicle. In some embodiments, the occluding of the known roadway objects comprises annotating the camera feed with occluding visual elements overlaid on at least one of a location and a position of each of the known roadway objects as each of the known roadway objects appear in the camera feed.

In some embodiments, the computer-implemented method may further comprise prompting a verifier viewing the traveling scene to indicate appearance of one or more roadway objects therein to elicit the received input.

In some embodiments, a vehicle may comprise a processor, and a memory unit operatively connected to the processor and including computer code, that when executed, causes the processor to: present a traveling scene regarding a section of roadway being traversed by a vehicle; determine known roadway objects in the traveling scene based on a previously generated map; occlude the known roadway objects in the traveling scene upon subsequent traversal of the section of roadway; and receive input regarding visible roadway objects in the traveling scene during the subsequent traversal of the section of roadway to verify accuracy of the previously generated map.

In some embodiments, the computer code, that when executed, may cause the processor to present the traveling scene in real-time during a map verification process.

In some embodiments, the computer code, that when executed, may cause the processor to present the traveling scene during an off-line map verification process.

In some embodiments, the computer code, that when executed causes the processor to determine the known roadway objects further comprises computer code, that when executed, further causes the processor to compare a current location of the vehicle to a commensurate location represented by the previously generated map.

In some embodiments, the computer code, that when executed, causes the processor to determine the known roadway objects further comprises computer code, that when executed, further causes the processor to identify roadway objects encoded in the previously generated map at the current location of the vehicle.

In some embodiments, the computer code, that when executed, causes the processor to present the traveling scene comprises computer code, that when executed, further causes the processor to present a camera feed of the traveling scene captured by a camera of the vehicle.

In some embodiments, the computer code, that when executed, causes the processor to occlude the known roadway objects, comprises computer code, that when executed, further causes the processor to comprises annotating the camera feed with occluding visual elements overlaid on at least one of a location and a position of each of the known roadway objects as each of the known roadway objects appear in the camera feed.

In some embodiments, the memory unit includes further computer code, that when executed, further causes the processor to prompt a verifier viewing the traveling scene to indicate appearance of one or more roadway objects therein to elicit the received input.

DETAILED DESCRIPTION

As alluded to above, ADAS and autonomous driving control systems can be used in vehicles that at least, in part, controls or manages vehicle operation to provide varying levels of automated control or assistance. For ease of reference, the term “autonomous control” will be used herein to refer to such systems. In some vehicles, an override mechanism, such as an override switch, may be used to turn off or disengage a vehicle's autonomous control system.

Autonomous control systems typically generate control signals/instructions for operating a vehicle based on high definition (HD) maps that denote the exact (or as exact as possible) position/location of roadway or road-related elements/objects. These roadway or road-related elements/objects can include, but are not limited to the position of lane lines, three-dimensional (3D) positions of traffic control devices, e.g., traffic signals or lights, traffic signs, painted roadway instructions, etc. Regardless of how HD maps may be generated (some methods are described in greater detail below), the resolution or accuracy of such HD maps is paramount. Any errors in such HD maps typically go unnoticed until, e.g., a verifier such as human passenger or automated detection verification component in an autonomous vehicle detects that some aspect of an HD map is inaccurate. For example, the verifier may notice that the HD map is missing a traffic signal at a particular intersection. However, error detection can be difficult depending on the particular environment. For example, object density in a particular area may be high, where a large number of objects such as pedestrians, buildings, multiple traffic control devices, etc. can make it difficult to verify that all traffic control devices in that area are properly reflected in the HD map.

Accordingly, various embodiments are directed to systems and methods of improved map verification. In some embodiments, a camera feed captured by an autonomous vehicle traversing a route(s) specified in an HD map can be altered in a way that occludes roadway elements or objects whose presence and/or location/position is obtained from the HD map. Thus, when a verifier is viewing the camera feed captured by the autonomous vehicle (either in real-time or at a later time), the verifier can more easily detect when the HD map (or more particularly the information encoded in or as the HD map) is incorrect. For example, if the HD map represents an intersection as having a single traffic signal, when in actuality, the intersection has two traffic signals, the altered camera feed may present a view where the single traffic signal (encoded in the HD map) is blocked out, e.g., with a red box or other visual element, or removed from view. The verifier will therefore detect the existence of the second traffic signal as it will be visible in the altered camera feed, and more easily so because the first traffic signal will not be visible. A user interface or other input mechanism can be provided so that the verifier can enter a note, set a flag, or enter some indication that a roadway element (in this case, a traffic signal) is visible and should be included in the HD map. The user interface can be interactive, and may prompt the verifier with questions asking if the verifier sees any roadway elements, to which the verifier may respond via the user interface/other input mechanism.

It should be noted that the systems and methods described herein can be adapted for use in alternative scenarios, where, e.g., an HD map incorrectly denotes/represents the existence of some road element(s) or object(s). For example, in open areas/spaces, a verifier may notice a blocked out element against, e.g., some section of roadway or some portion of a scene that should not include a road element or object, such as against a natural feature neighboring a road.

The systems and methods disclosed herein may be implemented with or by any of a number of different vehicles and vehicle types. For example, the systems and methods disclosed herein may be used with automobiles, trucks, motorcycles, recreational vehicles, farm vehicles, and other like on-or off-road vehicles. In addition, the principles disclosed herein may also extend to other vehicle types as well. An example hybrid electric vehicle is illustrated and described below as one example.

FIG.1illustrates an example hybrid electric vehicle (HEV)100in which various embodiments for driver disengagement of autonomous vehicle/driving controls may be implemented. It should be understood that various embodiments disclosed herein may be applicable to/used in various vehicles (internal combustion engine (ICE) vehicles, fully electric vehicles (EVs), etc.) that are fully or partially autonomously controlled/operated, not only HEVs.

HEV100can include drive force unit105and wheels170. Drive force unit105may include an engine110, motor generators (MGs)191and192, a battery195, an inverter197, a brake pedal130, a brake pedal sensor140, a transmission120, a memory160, an electronic control unit (ECU)150, a shifter180, a speed sensor182, and an accelerometer184.

Engine110primarily drives the wheels170. Engine110can be an ICE that combusts fuel, such as gasoline, ethanol, diesel, biofuel, or other types of fuels which are suitable for combustion. The torque output by engine110is received by the transmission120. MGs191and192can also output torque to the transmission120. Engine110and MGs191and192may be coupled through a planetary gear (not shown inFIG.1B). The transmission120delivers an applied torque to the wheels170. The torque output by engine110does not directly translate into the applied torque to the wheels170.

MGs191and192can serve as motors which output torque in a drive mode, and can serve as generators to recharge the battery195in a regeneration mode. The electric power delivered from or to MGs191and192passes through inverter197to battery195. Brake pedal sensor140can detect pressure applied to brake pedal130, which may further affect the applied torque to wheels170. Speed sensor182is connected to an output shaft of transmission120to detect a speed input which is converted into a vehicle speed by ECU150. Accelerometer184is connected to the body of HEV100to detect the actual deceleration of HEV100, which corresponds to a deceleration torque.

Transmission120is a transmission suitable for an HEV. For example, transmission120can be an electronically controlled continuously variable transmission (ECVT), which is coupled to engine110as well as to MGs191and192. Transmission120can deliver torque output from a combination of engine110and MGs191and192. The ECU150controls the transmission120, utilizing data stored in memory160to determine the applied torque delivered to the wheels170. For example, ECU150may determine that at a certain vehicle speed, engine110should provide a fraction of the applied torque to the wheels while MG191provides most of the applied torque. ECU150and transmission120can control an engine speed (NE) of engine110independently of the vehicle speed (V).

ECU150may include circuitry to control the above aspects of vehicle operation. ECU150may include, for example, a microcomputer that includes a one or more processing units (e.g., microprocessors), memory storage (e.g., RAM, ROM, etc.), and I/O devices. ECU150may execute instructions stored in memory to control one or more electrical systems or subsystems in the vehicle. ECU150can include a plurality of electronic control units such as, for example, an electronic engine control module, a powertrain control module, a transmission control module, a suspension control module, a body control module, and so on. As a further example, electronic control units can be included to control systems and functions such as doors and door locking, lighting, human-machine interfaces, cruise control, telematics, braking systems (e.g., anti-lock braking system (ABS) or electronic stability control (ESC)), battery management systems, and so on. These various control units can be implemented using two or more separate electronic control units, or using a single electronic control unit.

MGs191and192each may be a permanent magnet type synchronous motor including for example, a rotor with a permanent magnet embedded therein. MGs191and192may each be driven by an inverter controlled by a control signal from ECU150so as to convert direct current (DC) power from battery195to alternating current (AC) power, and supply the AC power to MGs191,192. MG192may be driven by electric power generated by motor generator MG191. It should be understood that in embodiments where MG191and MG192are DC motors, no inverter is required. The inverter, in conjunction with a converter assembly may also accept power from one or more of MGs191,192(e.g., during engine charging), convert this power from AC back to DC, and use this power to charge battery195(hence the name, motor generator). ECU150may control the inverter, adjust driving current supplied to MG192, and adjust the current received from MG191during regenerative coasting and braking.

Battery195may be implemented as one or more batteries or other power storage devices including, for example, lead-acid batteries, lithium ion, and nickel batteries, capacitive storage devices, and so on. Battery195may also be charged by one or more of MGs191,192, such as, for example, by regenerative braking or by coasting during which one or more of MGs191,192operates as generator. Alternatively (or additionally, battery195can be charged by MG191, for example, when HEV100is in idle (not moving/not in drive). Further still, battery195may be charged by a battery charger (not shown) that receives energy from engine110. The battery charger may be switched or otherwise controlled to engage/disengage it with battery195. For example, an alternator or generator may be coupled directly or indirectly to a drive shaft of engine110to generate an electrical current as a result of the operation of engine110. Still other embodiments contemplate the use of one or more additional motor generators to power the rear wheels of a vehicle (e.g., in vehicles equipped with 4-Wheel Drive), or using two rear motor generators, each powering a rear wheel.

Battery195may also be used to power other electrical or electronic systems in the vehicle. Battery195can include, for example, one or more batteries, capacitive storage units, or other storage reservoirs suitable for storing electrical energy that can be used to power MG191and/or MG192. When battery195is implemented using one or more batteries, the batteries can include, for example, nickel metal hydride batteries, lithium ion batteries, lead acid batteries, nickel cadmium batteries, lithium ion polymer batteries, and other types of batteries.

FIG.2illustrates an example autonomous control system200that may be used to autonomously control a vehicle, e.g., HEV100. Autonomous control system200may be installed in HEV100, and executes autonomous control of HEV100. As described herein, autonomous control can refer to control that executes driving/assistive driving operations such as acceleration, deceleration, and/or steering of a vehicle, generally movement of the vehicle, without depending or relying on driving operations/directions by a driver or operator of the vehicle.

As an example, autonomous control may include a lane keeping assist control where a steering wheel (not shown) is steered automatically (namely, without depending on a steering operation by the driver) such that HEV100does not depart from a running lane. That is, the steering wheel is automatically operated/controlled such that HEV100runs along the running lane, even when the driver does not perform any steering operation.

As another example, autonomous control may include navigation control, where when there is no preceding vehicle in front of the HEV100, constant speed (cruise) control is effectuated to make HEV100run at a predetermined constant speed. When there is a preceding vehicle in front of HEV100, follow-up control is effectuated to adjust HEV100's speed according to a distance between HEV100and the preceding vehicle.

It should be understood that manual control or manual driving can refer to a vehicle operating status wherein a vehicle's operation is based mainly on driver-controlled operations/maneuvers. In an ADAS context, driving operation support control can be performed during manual driving. For example, a driver may be actively performing any of a steering operation, an acceleration operation, and a brake operation of the vehicle, while autonomous control apparatus200performs some subset of one or more of those operations, e.g., in an assistive, complementary, or corrective manner. As another example, driving operation support control adds or subtracts an operation amount to or from the operation amount of the manual driving (steering, acceleration, or deceleration) that is performed by the driver.

In the example shown inFIG.2, autonomous control system200is provided with an external sensor201, a GPS (Global Positioning System) reception unit202, an internal sensor203, a map database204, a navigation system205, actuators206, an HMI (Human Machine Interface)207, a monitor device208, a shift lever209, auxiliary devices210. Autonomous control system200may communicate with ECU150, or in some embodiments (may be implemented with its own ECU).

In the example shown inFIG.2, external sensor201is a detector that detects external circumstances such as surrounding information of HEV100. The external sensor201may include at least one of a camera, a radar, and a Laser Imaging Detection and Ranging (LIDAR) unit.

The camera unit may be an imaging device that images the external circumstances surrounding the vehicle. For example, the camera is provided on a back side of a front windshield of the vehicle. The camera may be a monocular camera or a stereo camera. The camera outputs, to the ECU150, image information on the external circumstances surrounding the vehicle. The camera is not limited to a visible light wavelength camera but can be an infrared camera.

The radar unit uses radio waves to detect obstacles outside of the vehicle by transmitting radio waves to the surroundings of the vehicle, and receiving reflected radio waves from an obstacle to detect the obstacle, distance to the obstacle or a relative positional direction of the obstacle. The radar unit outputs detected obstacle information to the ECU150.

The LIDAR unit may operate similar to the manner in which the radar unit operates except that light is used in place of radio waves. The LIDAR unit outputs detected obstacle information to the ECU150.

In the example shown inFIG.2, GPS reception unit202receives signals from three or more GPS satellites to obtain position information indicating a position of HEV100. For example, the position information can include latitude information and longitude information. The GPS reception unit202outputs the measured position information of the vehicle to the ECU150.

In the example shown inFIG.2, the internal sensor203is a detector for detecting information regarding, e.g., a running status of HEV100, operational/operating conditions, e.g., amount of steering wheel actuation, rotation, angle, amount of acceleration, accelerator pedal depression, brake operation by the driver of HEV100. The internal sensor203includes at least one of a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. Moreover, internal sensor203may include at least one of a steering sensor, an accelerator pedal sensor, and a brake pedal sensor.

A vehicle speed sensor is a detector that detects a speed of the HEV100. In some embodiments, HEV100's speed may be measured directly or through calculations/inference depending on the operating conditions/status of one or more other components of HEV100. For example, a wheel speed sensor can be used as the vehicle speed sensor to detect a rotational speed of the wheel, which can be outputted to ECU150.

The acceleration sensor can be a detector that detects an acceleration of the vehicle. For example, the acceleration sensor may include a longitudinal acceleration sensor for detecting a longitudinal acceleration of HEV100, and a lateral acceleration sensor for detecting a lateral acceleration of HEV100. The acceleration sensor outputs, to the ECU150, acceleration information.

The yaw rate sensor can be a detector that detects a yaw rate (rotation angular velocity) around a vertical axis passing through the center of gravity of HEV100. For example, a gyroscopic sensor is used as the yaw rate sensor. The yaw rate sensor outputs, to the ECU150, yaw rate information including the yaw rate of HEV100.

The steering sensor may be a detector that detects an amount of a steering operation/actuation with respect to a steering wheel30by the driver of HEV100. The steering operation amount detected by the steering sensor may be a steering angle of the steering wheel or a steering torque applied to the steering wheel, for example. The steering sensor outputs, to the ECU150, information including the steering angle of the steering wheel or the steering torque applied to the steering wheel of HEV100.

The accelerator pedal sensor may be a detector that detects a stroke amount of an accelerator pedal, for example, a pedal position of the accelerator pedal with respect to a reference position. The reference position may be a fixed position or a variable position depending on a determined parameter. The accelerator pedal sensor is provided to a shaft portion of the accelerator pedal AP of the vehicle, for example. The accelerator pedal sensor outputs, to the ECU150, operation information reflecting the stroke amount of the accelerator pedal.

The brake pedal sensor may be a detector that detects a stroke amount of a brake pedal, for example, a pedal position of the brake pedal with respect to a reference position. Like the accelerator position, a brake pedal reference position may be a fixed position or a variable position depending on a determined parameter. The brake pedal sensor may detect an operation force of the brake pedal (e.g. force on the brake pedal, oil pressure of a master cylinder, and so on). The brake pedal sensor outputs, to the ECU150, operation information reflecting the stroke amount or the operation force of the brake pedal.

A map database204may be a database including map information. The map database204is implemented, for example, in a disk drive or other memory installed in HEV100. The map information may include road position information, road shape information, intersection position information, and fork position information, for example. The road shape information may include information regarding a road type such as a curve and a straight line, and a curvature angle of the curve information on road widths (information on lane widths), information on road slopes, information on the cant angle of a road, and information on the maximum vehicle speed on a road. The map information may also include position information on intersections and branching points, position information on stop lines, position information on pedestrian crossings, and position information on traffic lights or any other traffic signs/elements or other roadway/external environmental elements. In addition, the map information may include position information on white lines (the position information may be stored, not as a line, but as a sequence of points), information on white-line types, information on the positions and the shapes of non-moving obstacles such as curbs, telephone poles, poles, guardrails, walls, and buildings, and information on the positions and the shapes of road surface markings such as symbols and marks painted on a road surface. When autonomous control system200uses a Simultaneous Localization and Mapping (SLAM) technology or position information of blocking structural objects such as buildings and walls, the map information may further include an output signal from external sensor201. In some embodiments, map database204may be a remote data base or repository with which HEV100communicates.

As alluded to above, HD maps may be used for autonomous control of vehicle, such as HEV100. Thus, map database204may comprise HD map information that can exhibit high precision, e.g., at the centimeter level, since autonomous control of HEV100should be precise and without error. The aforementioned external sensors201, for example, may be used to detect/sense roadway objects for inclusion in an HD map of a particular portion of roadway. In some embodiments, an HD map can be manually generated (or at least partially so). A human user may review log data and captured video from a camera feed of HEV100while HEV100traverses that portion of roadway. Using a map editor, the human user may insert or annotate the HD map with all detected roadway elements that belong in the HD map, such as traffic control devices, persistent obstacles, such as island partitions, buildings, but not necessarily transient objects, e.g., pedestrians or other vehicles that may have been detected and reflected in the log and captured in the camera feed. In some embodiments, HD maps may be automatically generated. An example HD map generation system is described in US Patent Application 2020/0096359 (assigned to the present application), and is incorporated herein by reference in its entirety.

In some embodiments, an HD map may comprise various layers, and can include some or more layers including, e.g., a real-time layer, a map priors layer, a semantic map layer, a geometric map layer, and a base map layer. The base map layer may comprise a standard definition map. The geometric map layer may comprise raw sensor data collected by sensors, e.g., external sensor201, GPS reception unit202, and/or internal sensor203. The geometric map layer may output a dense 3D point cloud, which can be post-processed to produce derived map objects that are stored in the geometric map.

The semantic map can build upon the geometric map layer through the addition of semantic objects, e.g., 2D or 3D objects, such as lane boundaries, intersections, parking spots, stop signs, traffic lights, etc. that are used for driving safely, especially in the autonomous driving context. These 2D/2D objects can contain rich information such as traffic speeds, lane change restrictions etc. On the other hand, the map priors layer may comprise dynamic information and human behavior data, e.g., an order according to which a traffic signal may change colors, average speeds of vehicles at parking spots, etc. Autonomy algorithms commonly consume these priors in models as inputs or features and combined with other real-time information. The real-time layer can be dynamically updated, and may include information, such as real-time traffic information.

A logic database217may be a database that stores a plurality of position detection processing logics (position detection logics) used for detecting the position of HEV100. Position detection processing can refer to processing for detecting vehicle position based on the detection result of external sensor201(images captured by the camera, results detected by the radar sensor/LIDAR sensor, etc.). The position detection processing is stored in association with traveling scenes or camera feeds of/captured by HEV100. In addition, logic database217may store camera-captured images each associated with the map information (latitude, longitude) stored in map database204.

Navigation system205may be a component or series of interoperating components that guides a user, e.g., driver or passenger, of HEV100to a destination on a map designated by the user of HEV100. For example, navigation system205may calculate a route followed or to be followed by HEV100, based on the position information of HEV100measured by GPS reception unit202and map information of map database204. The route may indicate a running lane of a section(s) of roadway in which HEV100traverses, for example. Navigation system205calculates a target route from the current position of HEV100to the destination, and notifies the driver of the target route through a display, e.g., a display of a head unit, HMI207(described below), and/or via audio through a speaker(s) for example. The navigation system205outputs, to the ECU150, information of the target route for HEV100. In some embodiments, navigation system205may use information stored in a remote database, like map database204, and/or some information processing center with which HEV100can communicate. A part of the processing executed by the navigation system205may be executed remotely as well.

Actuators206may be devices that execute running controls of HEV100. The actuators206may include, for example, a throttle actuator, a brake actuator, and a steering actuator. For example, the throttle actuator controls, in accordance with a control signal output from the ECU150, an amount by which to open the throttle of HEV100to control a driving force (the engine) of HEV100. In another example, actuators206may include one or more of MGs191and192, where a control signal is supplied from the ECU150to MGs191and/or192to output motive force/energy. The brake actuator controls, in accordance with a control signal output from the ECU150, the amount of braking force to be applied to each wheel of the vehicle, for example, by a hydraulic brake system. The steering actuator controls, in accordance with a control signal output from the ECU150, driving an assist motor of an electric power steering system that controls steering torque.

HMI207may be an interface used for communicating information between a passenger(s) (including the operator) of HEV100and autonomous control system200. For example, the HMI207may include a display panel for displaying image information for the passenger(s), a speaker for outputting audio information, and operation buttons or a touch panel used by the occupant for performing an input operation. HMI207may also or alternatively transmit the information to the passenger(s) through a mobile information terminal connected wirelessly and receive the input operation by the passenger(s) through the mobile information terminal.

Monitor device208monitors a status of the driver/operator. The monitor device208can check a manual driving preparation state of the driver. More specifically, the monitor device208can check, for example, whether or not the driver is ready to start manual operation of HEV100. Moreover, the monitor device208can check, for example, whether or not the driver has some intention of switching HEV100to a manual mode of operation.

For example, the monitor device208may be a camera that can take an image of the driver, where the image can be used for estimating the degree to which the driver's eyes are open, the direction of the driver's gaze, whether or not the driver is holding the steering wheel, etc. Monitor device208may also be a pressure sensor for detecting the amount of pressure the driver's hand(s) are applying to the steering wheel. As another example, the monitor device208can be a camera that takes an image of a hand of the driver.

A shift lever209can be positioned at a shift position, e.g., “A (AUTOMATIC),” “D (DRIVE),” etc. The shift position “A” indicates, for example, an automatic engage mode where autonomous control is engaged automatically. The shift position “D” indicates a triggered engage mode where autonomous control is engaged in response to a driver-initiated request to operate HEV100in an autonomous driving mode.

Auxiliary devices210may include devices that can be operated by the driver of the vehicle, but are not necessarily drive-related, such as actuators206. For example, auxiliary devices210may include a direction indicator, a headlight, a windshield wiper and the like.

ECU150may execute autonomous control of the vehicle, and may include an acquisition unit211, a recognition unit212, a navigation plan generation unit213, a calculation unit214, a presentation unit216, and a control unit215.

Acquisition unit211may obtain the following operation amounts or levels of actuation based on the information obtained by the internal sensor203: steering operation, acceleration operation, and brake operation by the driver during an autonomous control mode; and the level of steering operation, acceleration operation, and brake operation by the driver of the vehicle during a manual control mode.

Recognition unit212may recognize or assess the environment surrounding or neighboring HEV100based on the information obtained by the external sensor201, the GPS reception unit202, and/or the map database204. For example, the recognition unit212includes an obstacle recognition unit (not shown), a road width recognition unit (not shown), and a facility recognition unit (not shown). The obstacle recognition unit recognizes, based on the information obtained by the external sensor201, obstacles surrounding the vehicle. For example, the obstacles recognized by the obstacle recognition unit include moving objects such as pedestrians, other vehicles, motorcycles, and bicycles and stationary objects such as a road lane boundary (white line, yellow line), a curb, a guard rail, poles, a median strip, buildings and trees. The obstacle recognition unit obtains information regarding a distance between the obstacle and the vehicle, a position of the obstacle, a direction, a relative velocity, a relative acceleration of the obstacle with respect to the vehicle, and a category and attribution of the obstacle. The category of the obstacle includes a pedestrian, another vehicle, a moving object, and a stationary object. The attribution of the obstacle can refer to a property of the obstacle such as hardness and a shape of the obstacle.

The road width recognition unit recognizes, based on the information obtained by the external sensor201, the GPS reception unit202, and/or the map database204, a road width of a road in which the vehicle is running.

The facility recognition unit recognizes, based on the map information obtained from the map database204and/or the vehicle position information obtained by the GPS reception unit202, whether or not HEV100is operating/being driven through an intersection, in a parking structure, etc. The facility recognition unit may recognize, based on the map information and the vehicle position information, whether or not the vehicle is running in a school zone, near a childcare facility, near a school, or near a park, etc.

Navigation plan generation unit213may generate a navigation plan for HEV100based on the target route calculated by the navigation system205, the information on obstacles surrounding HEV100recognized by recognition unit212, and/or the map information obtained from map database204. The navigation plan may be reflected in one or more operating conditions/controls to effectuate the target route. For example, the navigation plan can include a target speed, a target acceleration, a target deceleration, a target direction, and/or a target steering angle with which HEV100should be operated at any point(s) along the target route so that the target route can be achieved to reach a desired destination. It should be understood that navigation plan generation unit213generates the navigation plan such that HEV100operates along the target route while satisfying one or more criteria and/or constraints, including, for example, safety constraints, legal compliance rules, operating (fuel/energy) efficiency, and the like. Moreover, based on the existence of obstacles surrounding HEV100, the navigation plan generation unit213generates the navigation plan for the vehicle so as to avoid contact with such obstacles, comport with roadway instructions (speed limit, turn-only lanes, traffic control devices, etc.).

Calculation unit214may calculate a threshold used for determining whether or not to switch from autonomous control to manual driving or vice versa. The determination can be performed based on the operating levels associated with the manner in which the driver is operating HEV100during autonomous control which is obtained by the acquisition unit211. For example, the driver of HEV100may suddenly grasp the steering wheel (which can be sensed by internal sensor203) and stomp on the brake pedal (which can be sensed by monitor device208). The pressure on the steering wheel and the level of actuation of the brake pedal may be excessive enough (exceed a threshold) suggesting that the driver intends to override the autonomous control system200.

Control unit215can autonomously control HEV100based on the navigation plan generated by navigation plan generation unit213. The control unit215outputs, to the actuators206, control signals according to the navigation plan. That is, the control unit215controls actuators206based on the navigation plan, and thereby autonomous control of HEV100is executed/achieved. Moreover, certain levels of operation, e.g., steering wheel actuation, by the driver can be detected by the acquisition unit211. When such level(s) equal or exceed the threshold calculated by the calculation unit214in a period during which autonomous control is being used to operate HEV100, control unit215executes a switching from autonomous control to manual control.

Presentation unit216displays, on a display of the HMI207, various information that may be related to autonomous control, manual control, etc., e.g., indications regarding whether HEV100is operating in autonomous mode or manual mode. As will be described in greater detail below, map verification can be facilitated through presenting and/or annotating a camera feed via presentation unit216and HMI207.

In the context of map verification, navigation system205may calculate a route to be followed by HEV100, the route following or being commensurate with a portion(s) of roadway(s) reflected in an HD map stored in map database204. In this way, the presence and/or position/location of roadway or roadway-related elements or objects denoted or otherwise encoded into the HD map can be verified.

For example, the HD map may be a currently generated HD map reflecting the various characteristics of a portion(s) of roadway as understood to exist through information/data collected by one or more of external sensor201, GPS reception unit202, internal sensor203, and/or information already maintained in map database204. Accordingly, the HD map, in its semantic map layer, may include the location/positioning of roadway or roadway-related objects or elements, such as traffic control devices, signs, and the like.

To verify the accuracy of the HD map, HEV100may traverse the aforementioned route calculated by navigation system205(or otherwise specified to be traversed). While traversing the route, HEV100may use a camera (e.g., external sensor201) to capture a traveling scene. That traveling scene or camera feed can be altered and presented to a verifier that may be riding in HEV100. The altering of the traveling scene or camera feed may comprise occluding known roadway or roadway-related objects or elements encoded in the HD map or some subset thereof. For example, map verification may comprise verifying only certain roadway or roadway-related objects, such as traffic control devices.

For example an overlay unit216amay overlay some visual element, e.g., a shape or other visual element, such as a red or opaquely-colored box, that occludes traffic control devices. In this way, the traffic control devices are not visible to the verifier while the verified is viewing the traveling scene or camera feed on the display of HMI207. Because the verifier cannot see such traffic control devices, the verifier is not distracted by already known/assumed to be known aspects of the route being traversed. That is, the verifier need only focus on roadway or roadway-related objects that do appear, i.e., are not occluded from view. Accordingly, verifying the accuracy of the HD map can be as simple as inputting or flagging an instance when the verifier sees or senses a traffic control device in the traveling scene/camera feed being presented by presentation unit216.

In some embodiments, a black box could be used (although one of ordinary skill in the art would understand that other alternatives could be used, e.g., an overlay image/shape derived from the road element/object being blocked out). For example, in the case of an open sky scene, it could be a sky pattern. In the case of an overlay that is automatically derived from what is being blacked out (i.e. in the open sky case, areas neighboring that area to be removed can, e.g., be determined to be solid blue in color, and then a commensurate shape and/or color could be used to invisibly erase the traffic road element(s)/object(s).

In some embodiments, tagging could, in a speed optimized form, start in the form of a clicker to actuate if a user sees any non-blacked-out road element, e.g., traffic control device, at all. This can be effectuated in real-time. Subsequently, using the gross estimate of error times in the first step, a second step could involve verifiers/labelers working frame-by-frame at the flagged times, and actually marking/otherwise identifying approximate positions within the frame. If this process is performed for two or more cameras at the same timestep, a 3D position becomes available from the triangulation of the drawn-in positions, which can then be handed off to, e.g., a conventional map authoring process to add to the HD map (which then could be fed back in to this described process from the beginning to refine errors. In some embodiments, an inspectors can be asked for partial traffic control device visibility instead of fully visible devices.

Overlay unit216amay leverage the information known about the traffic control devices present in the HD map's semantic map layer (described above). Accordingly, overlay unit216amay generate occluding visual elements based on the characteristics of the traffic control devices reflected in the semantic map layer. As noted above, the semantic map layer may comprise 2D/3D semantic objects that in this example, represent one or more traffic control devices, and the occluding visual elements can be generated to mask the physical characteristics (footprint, outline, shape) of the traffic control devices.

As the camera (external sensor201) is capturing the traveling scene, overlay unit216amay add the occluding visual elements to the camera feed reflecting the traveling scene. Various known mechanisms or future mechanisms of adding such occluding visual elements can be leveraged. In one embodiment, and as described above, logic database217may detect the position of HEV100as it is traversing the calculated route. Thus, a real-time comparison can be made based on the current location of HEV100regarding the HD map being verified and information recognized or assessed by recognition unit212. That is, recognition unit212may provide information regarding recognized traffic control devices that can be compared to the semantic map layer information of the HD map regarding traffic control devices. When a match is found between the information from recognition unit212and the semantic map layer of the HD map, overlay unit216amay be triggered by presentation unit216to insert an appropriate occluding visual element to obscure a traffic control device that is found in both the HD map, and recognized by the recognition unit212.

In another embodiment, recognition unit212need not be involved during the map verification process. For example, overlay unit216amay simply be triggered to insert occluding visual elements to cover traffic control devices from view based solely on the information from the semantic map layer of the HD map from map database204. That is, logic database217can again be used to determine a current location of HEV100, and used to “synchronize” the traveling scene (or location HEV100is currently traveling) to locations on the HD map, and ultimately to the traffic control devices encoded as being present in the HD map. Thus, presentation unit216can present the traveling scene or camera feed on the display of HMI207with the occluding visual elements. Again, the verifier may tag or otherwise input an indication that a traffic control device is missing from or mis-located on the HD map via HMI207.

In yet another embodiment, the map verification process may be performed at a later time, e.g., non-real-time. That is, HEV100may traverse the calculated route, and the camera (external sensor201) may be used to capture the traveling scene. The above-described methods of adding the occluding visual elements can be used while generating the camera feed. The altered camera feed may then be viewed off-line, at a later time, by the verifier. HMI207or a user interface of a map editor may be used by the verifier to input the aforementioned tags, flags, or other indications of traffic control devices missing from the HD map. Alternatively, the camera feed may be captured, and the alteration to the camera feed may also occur off-line after the camera feed is captured. For example, the camera feed may be input in a map editor which has been adapted to have the same/similar functionality as the units/components used to effectuate map verification in HEV100. That is, the map editor may be adapted to compare the semantic map layer of an HD map to information from HEV100's recognition unit212or the camera feed (directly).

FIG.3Aillustrates an example scenario where a vehicle, such as HEV100, is traversing a section of roadway. An example view300from the vehicle includes approaching traffic signals304,306, and308along a section of roadway310. In this example scenario, recognition unit212may obtain information, e.g., from external sensor201, that traffic signals304,306, and308are present at a particular location/position. This location/position may be reflected in the semantic map layer of an HD map stored in map database204.

FIG.3Billustrates an example map verification scenario where map verification is being performed in real-time by a verifier in HEV100as it again travels the same section of roadway310. As noted above, navigation system205may calculate a route to be followed by HEV100. Here, HEV100is once again approaching the location where traffic signals304,306, and308are present (and were previously recognized and incorporated as part of an HD map. Because traffic signals304,306, and308are already part of the HD map, overlay unit216amay overlay traffic signals (presented by presentation unit216via a display of HMI207) in the camera feed/traveling scene being taken by a camera of HEV100(e.g., external sensor201). As noted above, the camera feed, during map verification, is altered to occlude known roadway or roadway-related elements, in this example, traffic signals304,306, and308. Thus, the verifier instead of seeing traffic signals304,306, and308in the view300, the verifier only sees, in this example, occluding visual elements304a,306a, and308ain the form of colored boxes. In this way, and upon traversing this section of roadway310, the verifier need not be distracted unnecessarily. It should be understood that the manner in which a roadway or roadway-related object is occluded can vary. In this example scenario, the traffic signals themselves are occluded from view with occluding visual elements304a,306a, and308a, but in other embodiments, the occluding visual elements can occlude the entirety of a roadway or roadway-related object, e.g., the traffic signal post/pole, in addition to the traffic signal bulb enclosures. Again, because HD maps are so precise, overlay unit216acan be configured to occlude or roadway or roadway-related objects in any manner desired.

As noted above, HMI207may prompt the verifier with questions regarding whether or not the verifier sees any roadway or roadway-related elements, in this case, traffic control devices. Here, the verifier can respond in the negative, or the verifier need not respond at all until a traffic control device appears (that has not been occluded, and thus suggests that the particular traffic control device was previously missed, went un-sensed, etc. In some embodiments, the location/position of a roadway or roadway-related object may not be precisely reflected in the HD map. Thus, overlay unit216amay insert or annotate/append the camera feed with occluding visual elements that are shifted or off-the-mark. Again, the verifier, via HMI207, may have the opportunity to input information or flag this error in the HD map.

FIGS.3C and3Dillustrate another example map verification scenario in accordance with some embodiments. As illustrated inFIG.3C, a vehicle, such as HEV100, is traversing a section of roadway330. Logic database217provides location awareness regarding HEV100. An example view320from HEV100includes approaching traffic signals322and324in an urban environment. In this example scenario, recognition unit212may obtain information, e.g., from external sensor201, that traffic signals322and324are present at a particular location/position. This location/position may be reflected in the semantic map layer of an HD map stored in map database204.

FIG.3Dillustrates an example map verification scenario where map verification is being performed in real-time by a verifier in HEV100as it again travels the same section of roadway330. As noted above, navigation system205may calculate a route to be followed by HEV100. Here, HEV100is once again approaching the location where traffic signals322and324are present. However, only traffic signal322was previously recognized and incorporated as part of an HD map. Because traffic signal322is already part of the HD map, overlay unit216amay overlay traffic signals (presented by presentation unit216via a display of HMI207) in the camera feed/traveling scene being taken by a camera of HEV100(e.g., external sensor201). As noted above, the camera feed, during map verification, is altered to occlude known roadway or roadway-related elements, in this example, traffic signal322. Unlike the previous example described above regardingFIGS.3A and3B, in this example, scenario, an error regarding traffic signal324occurred such that traffic signal has not been occluded from view. Because the verifier sees traffic signal324in the travel scene/camera feed, the verifier can indicate as such via HMI207. In this way, the HD map can be corrected to include traffic signal324in its semantic map layer. It can be appreciated that in this example scenario, with the density of buildings and other roadway or roadway-related objects, map verification can be made more difficult without the occlusion of known roadway or roadway-related objects.

FIG.4is a flow chart illustrating example operations that may be performed to verify the accuracy of a map or other similar collection of data regarding objects encoded or included in the map/collection of data. At operation400, a traveling scene regarding a section of roadway being traversed by a vehicle may be presented. As discussed above, map verification may involve an autonomous vehicle repeatedly traversing sections of roadway based on previously generated maps to determine if the previously generated maps accurately reflect the sections of roadways in terms of roadway object inclusion. It should be understood that this presentation can occur in real-time as the vehicle is re-traversing the sections of roadway, or can occur off-line at a later time. For example, the vehicle may capture camera feeds of the sections of roadway to be later reviewed by a verification entity.

At operation402, known roadway objects in the traveling scene are determined based on a previously generated map. As discussed above, maps, such as HD maps can be created manually or automatically, and involve encoding information regarding roadway objects to be represented therein. However, errors can occur, and certain roadway objects can be mis-located/mis-represented in the map in some way, or can be completely missing.

Determination of known roadway objects is performed so that, at operation404, the known roadway objects in the traveling scene can be occluded upon the subsequent traversal of the section of roadway. That is, in order to ease the burden on a verification entity, such as human verifier viewing the traveling scene/camera feed of the traveling scene, the known roadway objects can be hidden or occluded from view. For example, occluding visual elements, such as boxes, outlines, or other shapes/mechanisms can be overlaid or otherwise used to cover/hide the known roadway objects.

At operation406, input regarding visible roadway objects in the traveling scene are received to verify the accuracy of the previously generated map. As noted above, during operation404, known roadway objects are occluded. Thus, any roadway objects that do happen to appear in the traveling scene/camera feed are assumed to be missing from the previously generated map. The verifier may input some indication of the existence of such missing or mis-located/mis-positioned roadway objects so that the map can be corrected.

It should be noted that although various embodiments described herein involve the use of maps, such as HD maps, the disclosed manner of verification can be applied in other contexts. For example, lists of objects or elements can be verified

Referring now toFIG.5, computing component500may represent, for example, computing or processing capabilities found within a self-adjusting display, desktop, laptop, notebook, and tablet computers. They may be found in hand-held computing devices (tablets, PDA's, smart phones, cell phones, palmtops, etc.). They may be found in workstations or other devices with displays, servers, or any other type of special-purpose or general-purpose computing devices as may be desirable or appropriate for a given application or environment. Computing component500might also represent computing capabilities embedded within or otherwise available to a given device. For example, a computing component might be found in other electronic devices such as, for example, portable computing devices, and other electronic devices that might include some form of processing capability.

Computing component500might include, for example, one or more processors, controllers, control components, or other processing devices. This can include a processor504. Processor504might be implemented using a general-purpose or special-purpose processing engine such as, for example, a microprocessor, controller, or other control logic. Processor504may be connected to a bus502. However, any communication medium can be used to facilitate interaction with other components of computing component500or to communicate externally.

Computing component500might also include one or more memory components, simply referred to herein as main memory508. For example, random access memory (RAM) or other dynamic memory, might be used for storing information and instructions to be executed by processor504. Main memory508might also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor504. Computing component500might likewise include a read only memory (“ROM”) or other static storage device coupled to bus502for storing static information and instructions for processor504.

The computing component500might also include one or more various forms of information storage mechanism510, which might include, for example, a media drive512and a storage unit interface520. The media drive512might include a drive or other mechanism to support fixed or removable storage media514. For example, a hard disk drive, a solid-state drive, a magnetic tape drive, an optical drive, a compact disc (CD) or digital video disc (DVD) drive (R or RW), or other removable or fixed media drive might be provided. Storage media514might include, for example, a hard disk, an integrated circuit assembly, magnetic tape, cartridge, optical disk, a CD or DVD. Storage media514may be any other fixed or removable medium that is read by, written to or accessed by media drive512. As these examples illustrate, the storage media514can include a computer usable storage medium having stored therein computer software or data.

In alternative embodiments, information storage mechanism510might include other similar instrumentalities for allowing computer programs or other instructions or data to be loaded into computing component500. Such instrumentalities might include, for example, a fixed or removable storage unit522and an interface520. Examples of such storage units522and interfaces520can include a program cartridge and cartridge interface, a removable memory (for example, a flash memory or other removable memory component) and memory slot. Other examples may include a PCMCIA slot and card, and other fixed or removable storage units522and interfaces520that allow software and data to be transferred from storage unit522to computing component500.

Computing component500might also include a communications interface524. Communications interface524might be used to allow software and data to be transferred between computing component500and external devices. Examples of communications interface524might include a modem or softmodem, a network interface (such as Ethernet, network interface card, IEEE 802.XX or other interface). Other examples include a communications port (such as for example, a USB port, IR port, RS232 port Bluetooth® interface, or other port), or other communications interface. Software/data transferred via communications interface524may be carried on signals, which can be electronic, electromagnetic (which includes optical) or other signals capable of being exchanged by a given communications interface524. These signals might be provided to communications interface524via a channel528. Channel528might carry signals and might be implemented using a wired or wireless communication medium. Some examples of a channel might include a phone line, a cellular link, an RF link, an optical link, a network interface, a local or wide area network, and other wired or wireless communications channels.