Patent Publication Number: US-2023152467-A1

Title: Lidar based detection of road surface features

Description:
RELATED APPLICATION 
     This application is a continuation of U.S. patent application Ser. No. 16/690,898, filed on Nov. 21, 2019, and entitled “LIDAR BASED DETECTION OF ROAD SURFACE FEATURES”, the entirety of which is incorporated herein by reference. 
    
    
     BACKGROUND 
     An autonomous vehicle (AV) is a motorized vehicle that can operate without human conduction. An exemplary AV includes a plurality of sensor systems, such as, but not limited to, a lidar sensor system, a camera sensor system, and a radar sensor system, amongst others, wherein the AV operates based upon sensor signals output by the sensor systems. For example, a radar system can identify a range from the AV to another vehicle in the driving environment. In another example, an object recognition system may be configured to receive lidar signals output by a lidar sensor system and to identify positions of objects captured in the images. 
     An AV can be deployed as part of an on-demand transportation service. Whereas a quality or feel of the ride of a vehicle may be unimportant in connection with transporting cargo relative to other concerns (e.g., time of travel, speed of travel, fuel consumption, etc.), the apparent quality of the ride may be among the most important aspects of AV performance when transporting passengers. 
     Conventionally, AVs have been configured to avoid certain maneuvers that are likely to cause discomfort to passengers. By way of example, AVs have been configured to accelerate or decelerate slower than a threshold rate to avoid sudden stops and starts that may be uncomfortable or disconcerting to passengers. However, AVs have conventionally been unable to address problems of ride quality that arise from various features of a road surface itself (such as, but not limited to, potholes, speed bumps, broken pavement, utility access covers, etc.). Conventionally, AVs have been unable to detect such road surface features that have a negative impact on the apparent smoothness or comfort of a ride for a passenger. 
     SUMMARY 
     The following is a brief summary of subject matter that is described in greater detail herein. This summary is not intended to be limiting as to the scope of the claims. 
     Described herein are various technologies pertaining to controlling operation of an AV. With more specificity, described herein are various technologies pertaining to an AV that is configured to determine that a surface feature that may affect performance of the AV exists in a road surface in a traveling path of the AV. Still further, technologies are described herein that pertain to an AV that is further configured to execute an avoidance or mitigation maneuver to avoid the feature or mitigate its effect on the performance of the AV. 
     In a non-limiting example, an AV includes a lidar sensor system that comprises at least one lidar sensor. The lidar sensor system outputs lidar data based on sensor signals received from the at least one lidar sensor, where the lidar data is indicative of positions of objects in the driving environment of the AV at one or more times. In an exemplary embodiment, the lidar data comprises a three-dimensional lidar point cloud, wherein each point in the point cloud indicates a three-dimensional position of an object or surface of an object in the driving environment of the AV. In exemplary embodiments, one or more lidar sensors included in the lidar sensor system can be aimed toward a road surface that is in a travel path of the AV. Thus, the lidar sensor system can output data indicative of positions of the road surface and any road surface features on or in a portion of the road surface in the travel path of the AV (e.g., potholes, uneven pavement, speed bumps, utility access covers, and the like). 
     The AV can include a road surface analysis component that is configured to receive lidar data that is indicative of positions of a plurality of points on the road surface in the travel path of the AV from the lidar sensor system. Responsive to receipt of the lidar data, the road surface analysis component determines whether a road surface feature that potentially affects ride quality is present on or in the road surface. In an exemplary embodiment, the road surface analysis component determines whether the road surface feature is present based upon output of a neural network. In such embodiments, the road surface analysis component provides position values of points in a lidar point cloud that are representative of a road surface (e.g., as opposed to other objects in a driving environment of the AV, such as pedestrians, cars, buildings, etc.) to the neural network. Responsive to receipt of the position values, the neural network can output data indicative of a probability that a road surface feature is present on or in the road surface. In various embodiments, the neural network can be trained based upon human-labeled or machine-labeled lidar data, wherein the labeled lidar data is labeled as being representative of a road surface feature or not representative of a road surface feature. 
     In further embodiments, the road surface analysis component determines whether the road surface feature is present based upon an existing height map of the road surface in the travel path of the AV. By way of example, the AV can include a data store having loaded therein a map of roads in an operational region of the AV for use by the AV in connection with navigating through the operational region. The map of roads can include a height map of road surfaces in the operational region. The road surface analysis component can extract a height map of a portion of the road surface based upon lidar data received from the lidar sensor system. The road surface analysis component can then identify that a road surface feature is present on the roadway based upon the extracted height map indicating a different height than the height map stored in the data store. 
     In another exemplary embodiment, the road surface analysis component determines whether the road surface feature is present based upon comparison of a signature of lidar data representative of a road surface to a library of signatures that are labeled as or known to be representative of road surface features that are potentially disruptive to ride quality of an AV. In such embodiment, the road surface analysis component can extract a signature of a road surface feature from lidar data received from the lidar sensor system. The extracted signature can be any of various data pertaining to the received lidar data that can be indicative of the presence of a road surface feature. In a non-limiting example, the road surface analysis component can extract a height map of a portion of the road surface in the travel path of the AV from the points included in the lidar data. Subsequently, the road surface analysis component can compare the height map to a library of height maps that are representative of road surfaces having one or more road surface features present thereon or therein. Based upon the comparing, the road surface analysis component can determine whether the extracted height map is indicative of the presence of a road surface feature in the travel path of the AV. 
     Responsive to the road surface analysis component determining that a road surface feature that can affect ride quality of an AV is present in the travel path of the vehicle, the AV can be controlled to avoid the road surface feature, slow down prior to reaching the road surface feature, or perform some other maneuver that mitigates the effect of the road surface feature on the ride quality. 
     Upon determining that a road surface feature that may be detrimental to ride quality of an AV is present on a roadway, the AV can further update map data that is stored on the AV such that the map data is indicative of the presence of the road surface feature. The AV can further be configured to communicate with a remote server that maintains map data and that is in communication with a plurality of additional AVs. The AV can output data indicative of the presence of the road surface feature to the remote server, and the remote server can communicate with the additional AVs to cause the additional AVs to update their own respective map data with an indication that the road surface feature is present on the roadway. Thereafter, operation of any or all of the AVs can be controlled based upon the map data indicating the presence of the road surface feature (e.g., by causing the AVs to avoid the road surface feature). 
     The above summary presents a simplified summary in order to provide a basic understanding of some aspects of the systems and/or methods discussed herein. This summary is not an extensive overview of the systems and/or methods discussed herein. It is not intended to identify key/critical elements or to delineate the scope of such systems and/or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    illustrates an exemplary AV. 
         FIG.  2 A  is a top-down view of an exemplary driving environment of an AV. 
         FIG.  2 B  is a head-on view of the exemplary driving environment depicted in  FIG.  2 A . 
         FIG.  3    is a functional block diagram of an exemplary perception system of an AV that includes a road surface analysis component. 
         FIG.  4    is a functional block diagram of an exemplary system for control of a fleet of AVs based upon map data that includes road surface feature data. 
         FIG.  5    is a functional block diagram of an exemplary system that is configured to train a neural network component included in the road surface analysis component depicted in  FIG.  3   . 
         FIG.  6    is a flow diagram illustrating an exemplary methodology for controlling operation of a mechanical system of an AV based upon a determined probability that a road surface feature is present on a road surface. 
         FIG.  7    is a flow diagram illustrating an exemplary methodology for training a neural network to be included in a road surface analysis component of an AV. 
         FIG.  8    is an exemplary computing system. 
     
    
    
     DETAILED DESCRIPTION 
     Various technologies pertaining to an AV that is configured to determine whether one or more road surface features are present in or on a roadway are described herein, wherein like reference numerals are used to refer to like elements throughout. With more particularity, technologies described herein facilitate detecting the presence of road surface features that can negatively affect a rider&#39;s perceived ride quality, such as a pothole, a speedbump, or a temporary obstruction (e.g., a fallen limb, a cardboard box, motor accident debris, etc.). In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident, however, that such aspect(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by multiple components. Similarly, for instance, a component may be configured to perform functionality that is described as being carried out by multiple components. 
     Moreover, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from the context, the phrase “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, the phrase “X employs A or B” is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from the context to be directed to a singular form. 
     Further, as used herein, the terms “component” and “system” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor. The computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices. Further, as used herein, the term “exemplary” is intended to mean serving as an illustration or example of something and is not intended to indicate a preference. 
     As described herein, one aspect of the present technology is the gathering and use of data available from various sources to improve quality and experience. The present disclosure contemplates that in some instances, this gathered data may include personal information. The present disclosure contemplates that the entities involved with such personal information respect and value privacy policies and practices. 
     As used herein, the term “road surface feature” is intended to refer to depressions, protuberances, debris, or other features that are present on or in a surface of a roadway, and that affect the orientation of a vehicle as the vehicle travels over them. By way of various examples, the term “road surface feature” is intended to encompass holes, divots, gashes, and other depressions, speed bumps, rumble strips, manhole covers, and other protuberances, debris, and the like. It is to be appreciated that some road surface features may be present in a surface of a roadway, such as speed bumps and pot holes. It is to be appreciated that other road surface features may be present on a surface of a roadway, such as various forms of debris. 
     With reference now to  FIG.  1   , an exemplary AV  100  is illustrated. The AV  100  can navigate about roadways without human conduction based upon sensor signals output by sensor systems of the AV  100 . The AV  100  includes a lidar sensor system  102 , an inertial measurement unit (IMU)  104 , and a plurality of additional sensor systems  106 - 108  (a third sensor system  106  through an Nth sensor system  108 ). The sensor systems  106 - 108  may be of different types. For example, the third sensor system  102  may be a radar sensor system and the Nth sensor system  104  may be a camera (image) system. Other exemplary sensor systems include GPS sensor systems, sonar sensor systems, infrared sensor systems, and the like. The various sensor systems  102 - 108  are arranged about the AV  100 . 
     The AV  100  further includes several mechanical systems that are used to effectuate appropriate motion of the AV  100 . For instance, the mechanical systems can include but are not limited to, a vehicle propulsion system  110 , a braking system  112 , and a steering system  114 . The vehicle propulsion system  110  may be an electric engine, an internal combustion engine, or a combination thereof. The braking system  112  can include an engine brake, brake pads, actuators, a regenerative braking system, and/or any other suitable componentry that is configured to assist in decelerating the AV  100 . The steering system  114  includes suitable componentry that is configured to control the direction of movement of the AV  100 . 
     The AV  100  additionally comprises a computing system  116  that is in communication with the sensor systems  102 - 108  and is further in communication with the vehicle propulsion system  110 , the braking system  112 , and the steering system  114 . The computing system  116  includes a processor  118  and memory  120  that includes computer-executable instructions that are executed by the processor  118 . In an example, the processor  118  can be or include a graphics processing unit (GPU), a plurality of GPUs, a central processing unit (CPU), a plurality of CPUs, an application-specific integrated circuit (ASIC), a microcontroller, a programmable logic controller (PLC), a field programmable gate array (FPGA), or the like. 
     The memory  120  comprises a perception system  122  that is configured to identify objects (in proximity to the AV  100 ) captured in sensor signals output by the sensor systems  102 - 108 . The perception system  122  includes a road surface analysis component  124 . Briefly, the road surface analysis component  124  is configured to determine whether a road surface feature that can affect ride quality of the AV  100  is present on or in a roadway that is in a travel path of the AV  100 . The road surface analysis component  124  can determine whether the road surface feature is present based upon sensor signals output by the sensor systems  102 - 108 . As will be described in greater detail below, the perception system  122  may further include various additional components that are configured to output data indicative of positions and/or predefined types of objects in a driving environment of the AV  100 . In various embodiments, the road surface analysis component  124  can determine whether a road surface feature is present on a roadway based upon data output by such additional components of the perception system  122 , in addition to the sensor signals output by the sensor systems  102 - 108 . 
     The memory  120  additionally includes a control system  126  that is configured to receive output of the perception system  122 , and is further configured to control at least one of the mechanical systems of the AV  100  (the vehicle propulsion system  110 , the brake system  108 , and/or the steering system  114 ) based upon the output of the perception system  122 . In an exemplary embodiment, the road surface analysis component  124 , responsive to determining that a road surface feature is present on or in the roadway, outputs an indication of the road surface feature to the control system  126 . By way of example, the road surface analysis component  124  can output an identified location of a road surface feature to the control system  126 . The control system  126  can then control operation of at least one of the mechanical systems of the AV  100  based upon the indication from the road surface analysis component  124 . In a non-limiting example, the control system  126  can cause the AV  100  to initiate a mitigation maneuver responsive to receiving the indication that the road surface is present on or in the roadway. The mitigation maneuver can include deceleration of the AV  100 , or can include evasive maneuvers such as changing a lane in which the AV  100  is traveling, causing the AV  100  to make a turn, etc. 
     The computing system  116  further includes a datastore  128 . The datastore  128  stores map data  130  that is representative of an operational region of the AV  100  (e.g., a city, town, or other geographic area in which the AV  100  operates). The map data  130  can include, for example, locations of roadways in the operational region, a heightmap of roadways in the operational region, and locations of navigational points of interest in the operational region (e.g., addresses, restaurants, tourist attractions, etc.). The map data  130  can further include avoidance area data. The avoidance area data can indicate one or more avoidance areas through which the AV  100  is prohibited from being routed. 
     As described above, a roadway on which an AV is traveling can have one or more surface features or imperfections. These road surface features can cause a passenger of the AV to perceive that the ride of the AV is rough when the AV travels over the road surface features. By way of example, and with reference now to  FIGS.  2 A and  2 B , two views of an exemplary driving environment  200  of an AV  202  are illustrated. The driving environment  200  includes a roadway  204  on which the AV  202  travels, wherein the roadway  204  can include one or more road surface features that deviate from an average height of the roadway  204 . The roadway  204  includes, for example, a manhole cover  206  and a pothole  208 . When the AV  202  travels over either the manhole cover  206  or the pothole  208 , the manhole cover  206 /pothole  208  subjects the AV  202  to forces that cause a passenger of the AV  202  to perceive jostling or bumpiness in the ride of the AV  202 . 
     The road surface analysis component  124  is configured to determine whether a road surface feature is present on or in a roadway on which the AV  100  is traveling based upon lidar data output by the lidar sensor system  102 . It is to be understood that while the road surface analysis component  124  is described herein as determining whether a road surface feature is present in a travel path of the AV  100 , the road surface analysis component  124  can be configured to determine the presence or absence of road surface features in or on substantially any portion of a road surface that is represented by lidar data output by the lidar sensor system  102 . In an exemplary embodiment, the lidar data comprises a three-dimensional point cloud. The point cloud includes a plurality of points that are each assigned a three-dimensional position in the driving environment (e.g., a coordinate triple). The positions of points in the point cloud are indicative of locations of objects, or surfaces of objects, in the driving environment of the AV  100 . Each of the points can further be assigned an intensity value that is indicative of an intensity of the lidar return received at the lidar sensor system  102  from each of the points. 
     Exemplary operation of the AV  100  in connection with generating lidar data is now set forth. With reference now to  FIG.  2 A , the AV  202  is depicted in the driving environment  200 , wherein the driving environment includes a second vehicle  210 , and two pedestrians  212 ,  214  standing to the side of the roadway  204 . The AV  202  emits a laser beam  216  into the driving environment  200  by way of a lidar sensor system  218 . The laser beam  216  can be scanned 360° around the AV  202 , or within an arc of less than 360° about the AV  202 , and can impinge upon the various objects (e.g., the second vehicle  210  and the pedestrians  212 ,  214 ) in the environment  200 . The lidar sensor system  218  receives reflections of the beam  216 , and based upon such reflections outputs data indicative of positions of a plurality of points of intersection (e.g.,  220 ) of the beam  216  with the objects in the driving environment  200 . By way of example, the lidar sensor system  218  outputs data indicating a three-dimensional position of each of the points  220  (e.g., relative to the AV  202 , relative to a point of reference in the driving environment  200  or outside of the driving environment  200 , etc.). It is to be understood that when the laser beam  216  intersects a surface of the roadway  204 , the lidar sensor system  218  outputs data indicative of positions of the surface of the roadway  204 . 
     It is to be understood that while a single laser beam  216  and a plurality of points of intersection  220  of the beam  216  with objects are depicted in  FIG.  2 A , the lidar sensor system  218  can emit a plurality of laser beams into a driving environment of the AV  202 . For instance, and referring now to  FIG.  2 B , a head-on view of the driving environment  200  from the perspective of a passenger of the AV  202  is depicted. In exemplary embodiments, the lidar sensor system  218  emits a plurality of laser beams into the driving environment of the vehicle  202 , wherein each of the beams has a different angular alignment with respect to a horizontal reference plane of the vehicle  202 . Further, each of a plurality of sensors in the lidar sensor system  218  may be aligned with a different respective lidar beam emitted by the lidar sensor system  218 , such that the lidar sensor system  218  outputs data indicative of a plurality of points of intersection for each of the plurality of beams over a period of time. As depicted in  FIG.  2 B , the plurality of laser beams scan across the objects in the driving environment  200  and intersect the objects along scan lines (e.g.,  222 ). Each of the points  220  where a laser beam of the lidar sensor system  218  intersects an object or the roadway  204  in the driving environment lies along one of the scan lines  222 . Hence, each of the plurality of lidar sensors in the lidar sensor system  218  outputs data indicative of points of intersection of a respective laser beam with various objects in the driving environment  200  along a respective scan line. A lidar point cloud output by the lidar sensor system  218  can therefore be considered a rastered image of objects and surfaces in the driving environment, including a surface of the roadway  204 , comprising as many scan lines as lasers are included in the lidar sensor system  218 . 
     At least a portion of the points  220  are representative of a surface of the roadway  204 . A subset of this portion of the points  220  is representative of the road surface features  206 ,  208 . As will be described in greater detail below, an AV can determine that the road surface features  206 ,  208  are present in the roadway  204  based upon the portion of the lidar points  220  that are representative of the roadway  204  (as opposed to the other objects  210 ,  212 ,  214 ). In exemplary embodiments, the lidar sensor system  102  of the AV  100  can be configured such that a lidar point cloud generated by the lidar sensor system  102  has a high density of points that are representative of a roadway in a travel path of the AV  100 . By way of example, the lidar sensor system  102  can be configured such that a density of points on a road surface within 5 meters of a front of the AV  100  is at least 1000 points per square meter, 1250 points per square meter, or 1450 points per square meter. 
     It is to be appreciated that over the period of a scan of a driving environment of the AV  100  by the lidar sensor system  102 , the lidar sensor system  102  can output data indicative of positions of a larger number of points than are depicted in  FIGS.  2 A and  2 B  (e.g., tens of thousands of points, hundreds of thousands of points, millions of points, etc.). Furthermore, it is to be appreciated that the lidar sensor system  102  can include a greater number of lidar beams and sensors than depicted in either of  FIG.  2 A or  2 B . By way of example, the lidar sensor system  102  can include at least 64 beams and corresponding sensors, at least 128 beams and corresponding sensors, or at least 256 beams and corresponding sensors. 
     Referring again to  FIG.  1   , the AV  100  can be configured such that the perception system  122  selectively outputs lidar data to the road surface analysis component  124 . By way of example, the perception system  122  outputs to the road surface analysis component  124  lidar data that is determined to be representative of a surface of a roadway on which the AV  100  travels. By way of example, the perception system  122  can be configured to identify a subset of points in a lidar point cloud generated by the lidar sensor system  102  that are likely to be representative of the surface of the roadway on which the AV  100  travels. The perception system  122  outputs the subset of points in the point cloud that are representative of the roadway surface to the road surface analysis component  124 . The road surface analysis component  124 , based on the subset of points, determines whether a road surface feature is present on or in the roadway. For example, and referring once again to  FIG.  2 B , there is a subset of the points  220  that is representative of the roadway  204 . The perception system  122  of the AV  100  can determine which of the points  220  is representative of the roadway  204 . The road surface analysis component  124  can then determine that the road surface features  206 ,  208  are present in the roadway  204  based upon the subset of the points  220  that are representative of the roadway  204 . 
     Referring now to  FIG.  3   , an exemplary embodiment of the perception system  122  is shown, wherein the perception system  122  comprises the road surface analysis component  124  and further comprises a lidar segmentation system  302  and a plurality of M additional object recognizer components  304 - 306 , wherein M is a non-zero integer. The lidar segmentation system  302  is configured to perform a segmentation of a lidar point cloud (e.g., as received by the perception system  122  from the lidar sensor system  102 ). Stated differently, the lidar segmentation system  302  is configured to determine which points in the lidar point cloud are representative of which objects in the driving environment of the AV  100 . The lidar segmentation system  302  can further be configured to output an indication of a type of object for each of the objects in the driving environment of the AV  100 . These predefined types of objects can include, but are not limited to, road surface, pedestrian, bike, car, truck, bus, and static, where the type “static” can represent objects that are usually not in motion such as telephone poles, construction equipment, buildings, etc. 
     Each of the additional object recognizer components  304 - 306  independently generates output based upon at least one sensor signal. In non-limiting examples, the second object recognizer component  204  outputs data indicative of types of objects based upon images output by a camera, the Mth object recognizer component outputs data indicative of ranges to objects based upon radar signals, etc. In exemplary embodiments, the perception system  122  can be configured to determine which points in the lidar point cloud are representative of a surface of a roadway based upon data output by the lidar segmentation system  302  and the object recognizer components  304 - 306 . For example, the perception system  122  can identify which lidar points in a lidar point cloud are representative of which of various objects, such as the second vehicle  210  and the pedestrians  212 ,  214 , in the driving environment  200  based upon data output by the lidar segmentation system  302 . The perception system  122  can then identify that the object  210  is a vehicle and that the objects  212 ,  214  are pedestrians based upon output of one or more of the other object recognizer components  304 - 306 . 
     In some embodiments, the perception system  122  can be configured to identify which points in a lidar point cloud are representative of a road surface based upon points in a lidar point cloud not being identified by the lidar segmentation system  302  or the object recognizer components  304 - 306  as any other type of object. In another non-limiting example, the perception system  122  can identify which points in a lidar point cloud are representative of a road surface based upon the map data  130  stored in the data store  128 . For instance, the map data  130  can indicate positions of known roadways in the driving environment of the AV  100 . The lidar segmentation system  302  can identify which points in a lidar point cloud are representative of a road surface based upon the positions of the roadways indicated in the map data  130 . 
     The road surface analysis component  124  receives lidar point cloud data that is representative of a road surface in the driving environment of the AV  100  and determines whether a road surface feature is present on the roadway based upon the lidar point cloud data. 
     In the exemplary perception system  122  of  FIG.  3   , the road surface analysis component  124  comprises a neural network component  308 . The neural network component  308  is configured to receive an input feature vector that is representative of a road surface in a travel path of the AV  100 , and to output data that is indicative of a probability that a road surface feature is present in or on the road surface. The neural network component  308  is or includes a DNN that includes an input layer, one or more hidden layers, and an output layer. Each of the layers comprises one or more nodes, and nodes of each layer are connected to nodes in one or more other layers by way of edges. The edges and the nodes of the DNN can have weights assigned respectively thereto. Thus, each edge can be assigned a different weight (e.g., during a training procedure described in greater detail below), and each node that is assigned a weight can be assigned its own weight. The DNN outputs the data indicative of the probability that the road surface feature is present based upon propagating the vector of input features through the layers of the DNN to the output layer. In an exemplary embodiment, the DNN can be a convolutional neural network (CNN). 
     The input feature vector input to the neural network component  308  can be generated by the road surface analysis component  124  based upon the lidar points received by the road surface analysis component  124  from the perception system  122 . The input features included in the input feature vector can include any of various features pertaining to the lidar points. By way of example, and not limitation, the input features can include x, y, and z coordinate values of a lidar point in the lidar points and an intensity of a lidar return associated with the lidar point. In further non-limiting examples, the input features can include maximum height of the lidar points, minimum height of the lidar points, variance of heights of the lidar points, maximum lidar return intensity of the lidar points, density of the lidar points on the road surface, or substantially any other feature indicative of presence or absence of road surface features. 
     The neural network component  308  outputs data indicative of a probability that a road surface feature is present on or in a road surface. With greater specificity, the output data is indicative of the probability that a road surface feature is present on or in the particular region of a road surface that is represented by the lidar points to which the input feature vector pertains. In an exemplary embodiment, the output layer of the neural network component  308  can be a softmax layer such that the outputs of the output layer can be interpreted directly as probabilities. In such embodiments, the neural network component  308  can be configured to output a plurality of probabilities, wherein each of the probabilities is representative of a probability that a particular type of road surface feature is present in or on the roadway. By way of an example, and not limitation, the neural network component  308  can be configured to output a first probability that a first type of road surface feature is present, a second probability that a second type of road surface feature is present, a third probability that a third type of road surface feature is present, and a probability that no road surface feature is present. Continuing the example, the first probability can be a probability that a pot hole is present in the road surface, the second probability can be a probability that a speed bump is present on the road surface, and the third probability can be a probability that debris is present on the road surface. 
     While operations of the road surface analysis component  124  have been described above in terms of a neural network component  308 , it is to be understood that the road surface analysis component  124  can be configured to determine whether a road surface feature is present on a roadway by other means. By way of example, the road surface analysis component  124  can determine whether a road surface feature is present based upon comparing lidar data representative of a road surface to a plurality of lidar signatures of known road surface features. 
     For instance, signature data  132  can further be included in the data store  128 , wherein the signature data  132  comprises a plurality of lidar signatures of known road surface features. In an exemplary embodiment, each of the signatures comprises a plurality of points from lidar point clouds, wherein the points have been previously determined to be representative of road surface features. The signature data  132  can include one or more signatures of each of various types of road surface features desirably identified by the road surface analysis component  124 . In a non-limiting example, the signature data  132  can include one or more lidar signatures of manhole covers, one or more lidar signatures of potholes, one or more lidar signatures of speedbumps, one or more lidar signatures of motor vehicle accident debris, etc. 
     The road surface analysis component  124  can generate a similarity score between a signature in the signature data  132  and lidar data that is representative of a road surface in a travel path of the AV. The road surface analysis component  124  can determine that a road surface feature is present on a roadway in a travel path of the AV  100  based upon a value of the similarity score relative to a threshold value. For example, in some embodiments the road surface analysis component  124  determines that a road surface feature is present on the roadway based upon the similarity score exceeding a threshold value. In other embodiments, the road surface analysis component  124  determines that a road surface feature is present on the roadway based upon the similarity score falling below a threshold value. The road surface analysis component  124  can generate the similarity score by any of various algorithms. In a non-limiting example, the road surface analysis component  124  computes the similarity score as a sum of distances between pairs of points wherein for each pair one point is drawn from the lidar data and another drawn from a signature in the signature data. In this example, a lower similarity score is indicative of a greater similarity between the lidar data and the signature from the signature data  132 . Accordingly, the similarity score can be considered to be indicative of a probability that a road surface feature is present on a roadway in the travel path of the AV  100 . 
     In another exemplary embodiment, the road surface analysis component  124  can determine whether a road surface feature is present based upon comparing lidar data representative of a road surface to a height map included in the map data  130 . The road surface analysis component  124  can generate a height map of the road surface for a region in a travel path of the AV  100  based upon lidar data received from the lidar sensor system  102 . The road surface analysis component  124  can align the height map of the road surface generated from the lidar data with the height map included in the map data  130 . The road surface analysis component  124  can compare the generated height map with the height map included in the map data  130  to determine whether the generated lidar height map conflicts with the pre-defined height map (included in the map data  130 ). The road surface analysis component  124  can determine that a conflict exists between the lidar height map and the pre-defined height map when the heights of a location on the road surface indicated by each of the height maps differ by greater than a threshold amount. In an exemplary embodiment, the road surface analysis component  124  determines that a road surface feature is present on the roadway responsive to determining that the generated height map and the pre-defined height map conflict at a number of locations that is greater than a threshold number. A number of conflicts between the generated height map and the pre-defined height map can be considered to be indicative of a probability that a road surface feature is present on a roadway in the travel path of the AV  100 . 
     Referring now once again to  FIG.  1   , responsive to determining that a road surface feature is present in a roadway in a travel path of the AV  100 , or determining a probability that a road surface feature is present in the roadway, the road surface analysis component  124  outputs, to the control system  126 , an indication that the road surface feature is present or the probability that the road surface feature is present. The control system  126  controls operation of the mechanical systems of the AV  100  responsive to receiving the indication. In exemplary embodiments, responsive to receiving an indication that a road surface feature is present in the roadway, the control system  126  can control the AV to avoid the road surface feature. In other embodiments, responsive to receiving an indication that a road surface feature is present in the roadway, the control system  126  can control the AV to decelerate prior to traveling over the road surface feature. 
     In further exemplary embodiments, when the road surface analysis component  124  outputs an indication of a probability that a road surface feature is present in the roadway, the control system  126  can perform different actions depending on a magnitude of the probability. By way of example, and not limitation, the control system  126  can compare the probability to a first threshold and a second threshold. Responsive to the probability being below both the first threshold and the second threshold, the control system  126  can take no special action. Stated differently, when the probability is below both thresholds, the control system  126  controls the AV  100  normally. When the probability is above both thresholds, the control system  126  can control the AV  100  to avoid the road surface feature. 
     When the probability is below the second threshold but above the first threshold, the control system  126  can control the AV  100  to decelerate prior to reaching the road surface feature while not causing the AV  100  to avoid the road surface feature. Subsequently, as the AV  100  passes over the road surface feature, the IMU  104  outputs acceleration data that is indicative of an acceleration of the AV  100 . The acceleration data indicates whether the AV  100  is accelerated vertically or laterally by travel over the road surface feature. For instance, when the AV  100  travels over a speed bump or a pot hole, the AV  100  is accelerated vertically as the vehicle travels over the feature. The acceleration may be perceptible by a passenger of the AV  100 , which can reduce the perceived ride quality of the AV  100  to the passenger. When the acceleration data output by the IMU  104  indicates that the AV  100  is subject to an acceleration (e.g., not caused by the mechanical systems  110 - 114  of the AV  100 ) at the location of a potential road surface feature, the perception system  122  can update the map data  130  to indicate the presence of a road surface feature at the location. Therefore, the IMU  104  provides a confirmation of the presence of a road surface feature when the AV  100  travels over the road surface feature. The AV  100  can later be controlled to avoid the road surface feature based upon the road surface feature being indicated in the map data  130 . By way of example, the AV  100  can be routed to avoid a route that passes over a roadway that has the road surface feature thereon. In another example, the AV  100  can be routed over the roadway that includes the road surface feature, but the AV  100  can be controlled to avoid the location of the road surface feature that is indicated on the map (e.g., by changing lanes). 
     In further embodiments wherein the neural network component  308  outputs a probability of each of a plurality of types of road surface features being present on a roadway, the control system  126  can be configured to control the AV  100  based upon the different probabilities. In a non-limiting example, the road surface analysis component  124  can output a first probability that a speed bump is present at a location on the roadway and a second probability that a pothole is present at the location on the roadway. Responsive to determining that the first probability is greater than the second probability, the control system  126  can control the AV  100  to decelerate prior to reaching the location without causing the AV  100  to avoid the location. Responsive to determining that the second probability is greater than the first probability, the control system  126  can control the AV  100  to avoid the location. Thus, the AV  100  can be configured to slow down for certain road surface features and can be configured to avoid other road surface features. 
     The AV  100  can further be configured to communicate the presence of the road surface feature to a server computing device that controls various operations of a fleet of AVs, responsive to confirming that the road surface feature is present. With reference now to  FIG.  4   , an exemplary AV fleet system  400  is shown, wherein the fleet system  400  includes the AV  100 , a plurality of additional AVs  402 - 406 , and a server computing device  408  that is in communication with each of the AVs  100  and  402 - 406  by way of a network  410 . The server computing device  408  includes a data store  412  that has map data  414  loaded thereon. The map data  414  can be master map data that is used by the server computing device  408  in connection with coordinating or controlling operation of the various AVs  100 ,  402 - 406  in the fleet system  400 . Each of the AVs  402 - 406  can include a respective data store  416 - 420  on which is stored a copy of the master map data  414 . In an exemplary embodiment, the perception system  122  of the AV  100  confirms the presence of a road surface feature at a particular location based upon acceleration data output by the IMU  104 . Subsequently, the AV  100  can transmit map update data to the server computing device  408 , wherein the map update data causes the server computing device  408  to update the map data  414  stored at the data store  412  with an indication that the road surface feature is present at the particular location. The server computing device  408  can then transmit the map update data to each of the AVs  402 - 406 , whereupon the AVs  402 - 406  update their respectively stored map data  414  to include the indication that the road surface feature is present at the particular location. In various embodiments, the AV  100  can be configured to update its own map data  130  responsive to determining that the road surface feature is present at the particular location. In other embodiments, the system  400  can be configured such that the AV  100  updates its map data  130  to match the master map data  414  only in response to receiving map update data from the server computing device  408 . This can facilitate consistency of map data maintained by the various AVs  100 ,  402 - 406  included in the fleet system  400 . 
     In various embodiments, map data that includes an indication of a road surface feature can further include an indication of a time at which it was determined that the road surface feature was present. By way of example, and referring to  FIG.  1   , when the perception system  122  updates the map data  130  to include the indication that the road surface feature is present, the perception system  122  can include a time at which the acceleration data that was used to determine that the road surface feature is present was received from the IMU  104 . Thereafter, the control system  126  can control the AV  100  to avoid or not avoid the road surface feature depending on an amount of time that has elapsed from the time at which the road surface feature was determined to be present. By way of example, and not limitation, the control system  126  can control the AV  100  to avoid a road surface feature that is indicated in the map data  130  as having been determined to be present an hour previously. In another non-limiting example, the control system  126  can control the AV  100  to pass over a road surface feature that is indicated in the map data  130  as having been determined to be present more than one week ago. Accordingly, the AV  100  can be configured to avoid recently identified road surface features (e.g., identified less than one month ago, less than two weeks ago, or less than one week ago), which are likely to still be present. The AV  100  can be configured not to avoid road surface features identified long ago (e.g., more than one week ago, more than two weeks ago, or more than one month ago). 
     The AV  100  can sometimes be routed over a previously-identified road surface feature (e.g., previously identified by the AV  100  or another AV that shares map data with the AV  100  such as the AVs  402 - 406  of  FIG.  4   ). When the AV  100  is routed over a previously-identified road surface feature, the perception system  122  of the AV  100  confirms, based upon output of the IMU  104 , whether the previously-identified road surface feature is still present. For example, the perception system  122  can update the map data  130  to indicate that the road surface feature is still present when the acceleration data output by the IMU  104  indicates that the AV  100  was subject to an acceleration due to the road surface feature. The perception system  122  can update the map data  130  to indicate that the road surface feature is not still present when the acceleration data output by the IMU  104  indicates that the AV  100  was not subject to an acceleration by the road surface feature. The AV  100  can therefore keep road surface feature data in the map data  130  up-to-date by allowing for the revisiting of road surfaces indicated in the map data  130  as including road surface features, when enough time has passed that the road surface features may no longer exist. When a fleet of AVs configured in a similar fashion to the AV  100  is operated in an operational region, the fleet of AVs can maintain current road surface feature data throughout the operational region without requiring special dispatches of AVs to perform mapping operations. 
     It is to be appreciated that some road surface features are likely to be present longer than others. For instance, motor vehicle accident debris is likely to be present on a roadway for less time than a pot hole, and a pot hole is likely to be present on a roadway for less time than a speed bump. A speed bump may itself be treated as being effectively permanent. Accordingly, the AV  100  can be configured such that the amount of elapsed time required before the AV  100  is permitted to be routed over a road surface feature is dependent on a type of the road surface feature. 
     By way of example, the perception component  122  can indicate a high probability that a first road surface feature is a pot hole and a high probability that a second road surface feature is motor vehicle accident debris (e.g., based upon output of the neural network component  308 ). In the example, the perception component  122  updates the map data  130  with indications that the first road surface feature and the second road surface feature are present. The indication that the first road surface feature is present can include an indication that the first road surface feature is likely to be a pot hole, and the indication that the second road surface feature is present can include an indication that the second road surface feature is likely to be motor vehicle accident debris. Motor vehicle accident debris is likely to be cleared from a roadway after less time than it takes for a pot hole to be repaired. The control system  126  can be configured to permit routing of the AV  100  over the first road surface feature after a first amount of time has passed, and can be configured to permit routing of the AV  100  over the second road surface feature after a second amount of time has passed, wherein the second amount of time is less than the first amount of time. The AV  100  can therefore be configured to avoid for a longer time road surface features that are likely to be present for a longer time. 
     As noted above, some road surface features may be treated as being effectively permanent. For instance, speed bumps and railroad crossings are generally present on roadways by design and are likely to be present for many years. In some instances, it may be undesirable to route the AV  100  around such road surface features where doing so causes a route taken by the AV  100  to be longer in distance or in time than if the AV  100  were routed over the road surface features. In other cases, the apparent effect on ride quality of may be sufficiently great and the alternative routing cost (e.g., in increased time, increased distance, or increased fare cost to a passenger) sufficiently low that Accordingly, when the road surface analysis system  124  indicates a high probability (e.g., greater than 50%, greater than 75%, or greater than 90%) that a road surface feature is of a type that is likely to be present for long periods of time (e.g., speed bumps, railroad crossings, rumble strips, etc.), the map data  130  can be updated to indicate that the road surface feature is a durable road surface feature. The AV  100  can be configured such that routing of the AV  100  around or over a durable road surface feature is not based upon an elapsed time since the road surface feature was identified. Accordingly, routing of the AV  100  around or over a durable road surface feature can be based primarily or solely on apparent effect on ride quality of the road surface feature (e.g., as indicated by a human-determined subjective measure or as indicated by a magnitude of the acceleration caused by the road surface feature as measured by the IMU  104 ) and the alternative routing cost associated with avoiding the road surface feature. 
     The AV  100  can further be configured such that, when the perception system  122  confirms that a road surface feature is present based upon acceleration data output by the IMU  104 , the neural network component  308  is updated based upon the road surface feature. In an exemplary embodiment, the neural network component  308  can update a weight assigned to at least one of an edge or a node of the DNN of the neural network component  308  responsive to the perception system  122  confirming that the road surface is present based upon the acceleration data. With more specificity, the neural network component  308  can update weights of nodes and edges in the DNN based upon a reinforcement learning algorithm that takes into consideration values of the features of the input feature vector provided as input to the DNN, a probability that a road surface feature is present output by the DNN, and the actual presence or absence of the road surface feature as indicated by the acceleration data output by the IMU  104 . The neural network component  308  can update weights of nodes and edges in the DNN in real-time during operation of the AV  100  in its driving environment. The road surface analysis component  124  therefore refines its model for determining whether road surface features are present as the AV  100  is operating, rather than requiring the AV  100  to be taken offline to retrain the neural network component  308 . 
     With reference now to  FIG.  5   , an exemplary computing system  500  that is configured to train the neural network component  308  is illustrated. By way of example, the computing system  500  can be used to train the neural network component  308  prior to initial deployment of the AV  100 . In another example, the computing system  500  can be used to retrain the neural network  308  during a period in which the AV  100  is taken out of operation for regularly scheduled maintenance. The computing system  500  includes a processor  502  and memory  504 , wherein the memory  504  includes instructions that are executed by the processor  502 . The computing system  500  additionally includes a data store  506  that comprises labeled training data  508 . The labeled training data  508  can include, for example, input feature vectors that are representative of road surfaces that include road surface features and other input feature vectors that are representative of road surfaces that do not include road surface features. The input feature vectors are labeled as being either representative of road surface features or not representative of road surface features. The labeled training data  508  can be or include input feature vectors generated by a road surface analysis component of an AV and labeled by the road surface analysis component based upon output of an IMU, as described above with respect to the road surface analysis component  124  of the AV  100 . For instance, responsive to acceleration data generated by the IMU  104  indicating that a road surface feature is present on a portion of a roadway, the road surface analysis component  124  can label an input feature vector that is representative of the portion of the roadway as being representative of a road surface feature. The road surface analysis component  124  can store the labeled input feature vector in the data store  128  for subsequent use in training a neural network to identify road surface features. 
     The memory  504  includes a neural network training component  510  that trains the neural network component  308 . The neural network training component  510  receives the labeled training data  508 . The neural network training component  510  trains the neural network component  308  such that the probability of a road surface feature being present that is output by the neural network component  308  is higher for input feature vectors that are representative of road surface features and lower for input feature vectors that are not representative of road surface features. The neural network training component  510  can train the neural network component  308  based upon supervised learning algorithms, reinforcement learning algorithms, or other suitable machine learning algorithms. The neural network component  308  can be validated based upon a test training data set  512 , which is also labeled, to ensure that the neural network component  308  produces outputs with acceptable characteristics (e.g., such the neural network component  308  indicates a high probability that a road surface feature is present when the neural network is provided with an input feature vector that is actually representative of a road surface feature that is desirably detected). 
       FIGS.  6  and  7    illustrate exemplary methodologies relating to controlling an AV based upon determining whether road surface feature is present on or in a roadway in a travel path of the AV. While the methodologies are shown and described as being a series of acts that are performed in a sequence, it is to be understood and appreciated that the methodologies are not limited by the order of the sequence. For example, some acts can occur in a different order than what is described herein. In addition, an act can occur concurrently with another act. Further, in some instances, not all acts may be required to implement a methodology described herein. 
     Moreover, the acts described herein may be computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media. The computer-executable instructions can include a routine, a sub-routine, programs, a thread of execution, and/or the like. Still further, results of acts of the methodologies can be stored in a computer-readable medium, displayed on a display device, and/or the like. 
     Referring now to  FIG.  6   , an exemplary methodology  600  for controlling operation of an AV based upon the presence of a road surface feature in or on a roadway in a travel path of the AV is illustrated. The methodology  600  starts at  602 , and at  604  lidar data is received from a lidar sensor that is mounted on an AV. In exemplary embodiments, the lidar data comprises a lidar point cloud, wherein each of the points in the point cloud indicates a location of an object or surface in the driving environment. At least a portion of the points in the lidar point cloud are representative of a road surface in the travel path of the AV. At  606 , a probability that a road surface feature is present on a road surface in a travel path of the AV is determined. In exemplary embodiments, the probability that the road surface feature is present can be determined based upon output of a DNN, as described in greater detail above with respect to the neural network component  308 . In other embodiments, the probability that the road surface feature is present can be determined based upon a similarity score between the lidar data and a lidar signature of a known road surface feature. In still other embodiments, the probability that the road surface feature is present can be determined based upon a degree of conflict between a pre-defined height map of the road surface in the travel path of the AV, and a height map generated based upon the lidar data received at  604 . 
     At  608  a determination is made whether the probability exceeds a threshold probability. If the probability does not exceed the threshold probability, then the methodology  600  returns to  604  and additional lidar data is received from the lidar sensor mounted on the AV. If the probability does exceed the threshold probability, the methodology  600  proceeds to  610  whereupon operation of the AV is controlled to cause the AV to avoid the road surface feature or decelerate prior to reaching the road surface feature. The AV can be controlled to avoid the road surface feature or to decelerate prior to reaching the road surface feature by controlling one or more of a propulsion system of the AV, a braking system of the AV, or a steering system of the AV. The methodology  600  then completes at  612 . It is to be appreciated that the methodology  600  can be repeatedly performed by an AV as the AV travels through its driving environment. 
     Turning now to  FIG.  7   , an exemplary methodology  700  that facilitates training a DNN is illustrated, wherein the DNN is configured for use in a road surface analysis component of an AV to determine whether a road surface feature is present on or in a road surface that is in a travel path of the AV. The methodology  700  starts at  702 , and at  704  labeled training data is received, wherein the training data comprises labeled input feature vectors that are derived from lidar data that is representative of road surfaces. At  706 , a DNN is trained based upon the training data. The DNN is configured such that responsive to receiving a vector of input features that are derived from and representative of lidar data that is itself representative of a road surface, the DNN outputs a probability that a road surface feature is present on or in the road surface. The DNN is trained at  706  such that the DNN outputs a higher probability that a road surface is present on or in the road surface when the DNN is provided a vector of input features that is actually representative of a road surface having a road surface feature thereon/therein. Likewise, the DNN is trained at  706  such that the DNN outputs a lower probability that the road surface is present when the DNN is provided an input feature vector that is representative of a road surface that does not have a road surface feature thereon/therein. The methodology  700  completes at  708 . 
     Referring now to  FIG.  8   , a high-level illustration of an exemplary computing device  800  that can be used in accordance with the systems and methodologies disclosed herein is illustrated. For instance, the computing device  800  may be or include the computing system  116  or the computing system  500 . The computing device  800  includes at least one processor  802  that executes instructions that are stored in a memory  804 . The instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more modules, components, or systems discussed above or instructions for implementing one or more of the methods described above. The processor  802  may be a GPU, a plurality of GPUs, a CPU, a plurality of CPUs, a multi-core processor, etc. The processor  802  may access the memory  804  by way of a system bus  806 . In addition to storing executable instructions, the memory  804  may also store point cloud data, map data, road surface feature data, training data, etc. 
     The computing device  800  additionally includes a data store  808  that is accessible by the processor  802  by way of the system bus  806 . The data store  808  may include executable instructions, sensor data, map data, road surface feature data, training data, etc. The computing device  800  also includes an input interface  810  that allows external devices to communicate with the computing device  800 . For instance, the input interface  810  may be used to receive instructions from an external computing device, etc. The computing device  800  also includes an output interface  812  that interfaces the computing device  800  with one or more external devices. For example, the computing device  800  may transmit control signals to the vehicle propulsion system  110 , the braking system  112 , and/or the steering system  114  by way of the output interface  812 . 
     Additionally, while illustrated as a single system, it is to be understood that the computing device  800  may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device  800 . 
     Various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer-readable storage media. A computer-readable storage media can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproduce data magnetically and discs usually reproduce data optically with lasers. Further, a propagated signal is not included within the scope of computer-readable storage media. Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of communication medium. Combinations of the above should also be included within the scope of computer-readable media. 
     Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc. 
     What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable modification and alteration of the above devices or methodologies for purposes of describing the aforementioned aspects, but one of ordinary skill in the art can recognize that many further modifications and permutations of various aspects are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.