System and method for evaluating the perception system of an autonomous vehicle

A method and apparatus are provided for optimizing one or more object detection parameters used by an autonomous vehicle to detect objects in images. The autonomous vehicle may capture the images using one or more sensors. The autonomous vehicle may then determine object labels and their corresponding object label parameters for the detected objects. The captured images and the object label parameters may be communicated to an object identification server. The object identification server may request that one or more reviewers identify objects in the captured images. The object identification server may then compare the identification of objects by reviewers with the identification of objects by the autonomous vehicle. Depending on the results of the comparison, the object identification server may recommend or perform the optimization of one or more of the object detection parameters.

BACKGROUND

Autonomous vehicles use various computing systems to aid in the transport of passengers from one location to another. Some autonomous vehicles may require some initial input or continuous input from an operator, such as a pilot, driver, or passenger. Other systems, such as autopilot systems, may be used only when the system has been engaged, which permits the operator to switch from a manual mode (where the operator exercises a high degree of control over the movement of the vehicle) to an autonomous mode (where the vehicle essentially drives itself) to modes that lie somewhere in between.

Such vehicles are equipped with various types of sensors in order to detect objects in the surroundings. For example, autonomous vehicles may include lasers, sonar, radar, cameras, and other devices that scan and record data from the vehicle's surroundings. These devices in combination (and in some cases alone) may be used determine the location of the object in three-dimensional space.

In determining whether there is an object near the autonomous vehicle, the computing systems may perform numerous calculations using a number of parameters. Adjustments to these parameters may affect the performance of the computing systems. For example, the adjustments may decrease the likelihood that the computing systems determine the presence of a given object or increase the likelihood that the computing systems do not detect the presence of an object, such as a car, traffic light, or pedestrian.

BRIEF SUMMARY

An apparatus for optimizing object detection performed by an autonomous vehicle is disclosed. In one embodiment, the apparatus includes a memory operative to store a first plurality of images captured by an autonomous vehicle and a second plurality of images, corresponding to the first plurality of images, in which an object label has been applied an object depicted in an image of the second plurality of images. The apparatus may also include a processor in communication with the memory, where the processor operative to receive the first plurality of images from the autonomous vehicle and display a first image from the first plurality of images, wherein the first image comprises an object. The processor may also be operative to receive the object label for the object displayed in the first image from the first plurality of images to obtain a first image from the second plurality of images, compare the received object label with an object label applied by the autonomous vehicle to the object in the first image in the first plurality of images, and determine whether the received object label corresponds to the object label applied by the autonomous vehicle.

In another embodiment of the apparatus, the first plurality of images comprise a first plurality of images captured by a first sensor of a first sensor type and a second plurality of images captured by a second sensor of a second sensor type.

In a further embodiment of the apparatus, wherein the first sensor comprises a camera and the second sensor comprises a laser.

In yet another embodiment of the apparatus, the first plurality of images captured by the first sensor are images captured from a forward perspective of the autonomous vehicle.

In yet a further embodiment of the apparatus, the second plurality of images captured by the second sensor are images captured from a panoramic perspective of the autonomous vehicle.

In another embodiment of the apparatus, the at least one object label comprises a plurality of parameters that define the object label, and the plurality of parameters depend on an image sensor type used to capture the first image from the first plurality of images captured by the autonomous vehicle.

In a further embodiment of the apparatus, the processor is further operative to determine whether the received object label corresponds to the object label applied by the autonomous vehicle by determining whether the object label applied by the autonomous vehicle overlaps any portion of the received object label.

In yet another embodiment of the apparatus, the processor is further operative to determine whether the received object label corresponds to the object label applied by the autonomous vehicle by determining an object identification ratio derived from the received object label and the object label applied by the autonomous vehicle.

In yet a further embodiment of the apparatus, the processor is operative to determine whether the received object label corresponds to the object label applied by the autonomous vehicle based on a first area represented by the intersection of an area of the received object label with an area of the object label applied by the autonomous vehicle, and a second area represented by the union of the area of the received object label with the area of the object label applied by the autonomous vehicle.

In another embodiment of the apparatus, the object label applied by the autonomous vehicle is based on a plurality of object detection parameters, and the processor is further operative to optimize the plurality of object detection parameters when the indication of the correspondence between the received object label and the object label applied by the autonomous vehicle does not exceed a predetermined correspondence threshold.

A method for optimizing object detection performed by an autonomous vehicle is also disclosed. In one embodiment, the method includes storing, in a memory, a first plurality of images captured by an autonomous vehicle and displaying, with a processor in communication with a memory, a first image from the first plurality of images, wherein the first image comprises an object. The method may also include receiving an object label for the object displayed in the first image from the first plurality of images to obtain a first image for a second plurality of images, and comparing the received object label with an object label applied by the autonomous vehicle to the object in the first image in the first plurality of images, and determining whether the received object label corresponds to the object label applied by the autonomous vehicle.

In another embodiment of the method, the first plurality of images comprise a first plurality of images captured by a first sensor of a first sensor type and a second plurality of images captured by a second sensor of a second sensor type.

In a further embodiment of the method, the first sensor comprises a camera and the second sensor comprises a laser.

In yet another embodiment of the method, the first plurality of images captured by the first sensor are images captured from a forward perspective of the autonomous vehicle.

In yet a further embodiment of the method, the second plurality of images captured by the second sensor are images captured from a panoramic perspective of the autonomous vehicle.

In another embodiment of the method, the at least one object label comprises a plurality of parameters that define the object label, and the plurality of parameters depend on an image sensor type used to capture the first image from the first plurality of images captured by the autonomous vehicle.

In a further embodiment of the method, determining whether the received object label corresponds to the object label applied by the autonomous vehicle comprises determining whether the object label applied by the autonomous vehicle overlaps any portion of the received object label.

In yet another embodiment of the method, determining whether the received object label corresponds to the object label applied by the autonomous vehicle comprises determining an object identification ratio derived from the received object label and the object label applied by the autonomous vehicle.

In yet a further embodiment of the method, determining whether the received object label corresponds to the object label applied by the autonomous vehicle is based on a first area represented by the intersection of an area of the received object label with an area of the object label applied by the autonomous vehicle, and a second area represented by the union of the area of the received object label with the area of the object label applied by the autonomous vehicle.

In another embodiment of the method, the object label applied by the autonomous vehicle is based on a plurality of object detection parameters, and the method further comprises optimizing the plurality of object detection parameters when the indication of the correspondence between the received object label and the object label applied by the autonomous vehicle does not exceed a predetermined correspondence threshold.

A further apparatus for optimizing object detection performed by an autonomous vehicle is also disclosed. In one embodiment, the apparatus includes a memory operative to store a plurality of images captured by an autonomous vehicle using object detection parameters, a first plurality of object label parameters determined by the autonomous vehicle, and a second plurality of object label parameters applied by a reviewer having reviewed the plurality of images captured by the autonomous vehicle. The apparatus may also include a processor in communication with the memory, the processor operative to determine whether to optimize the plurality of object detection parameters based on a comparison of the first plurality of object label parameters with the second plurality of object label parameters, and perform an operation on the plurality of object detection parameters based on the comparison of the first plurality of object label parameters with the second plurality of object label parameters.

The operation performed on the plurality of object detection parameters may include identifying a plurality of object detection values, wherein each object detection value corresponds to at least one object detection parameter in the plurality of object detection parameters. For each possible combination of the plurality of object detection values, the operation may include performing an object detection routine on the plurality of images captured by the autonomous vehicle using the plurality of object detection values. The operation may also include selecting the combination of plurality of object detection values that resulted in an optimal object detection routine.

DETAILED DESCRIPTION

This disclosure provides for an apparatus and method directed to optimizing one or more object detection parameters used by a computing system on an autonomous vehicle. In particular, this disclosure provides for an apparatus and method of optimizing the one or more object detection parameters by comparing the identification of objects by the computing system in the autonomous vehicle with the identification of objects by one or more reviewers. The reviewers may review the raw images captured by the autonomous vehicle and the reviewers may manually label the objects depicted in the raw images. By “raw” image, it is meant that the image may not have been marked upon or modified by the autonomous vehicle. In other words, a “raw image” may be an image as captured by a sensor without markings that would alter the view depicted therein. As discussed with reference toFIG. 10below, a reviewer may use the object identification server to create electronic object labels on the raw images captured by the autonomous vehicle. Captured images having been electronically marked with object labels may not be considered raw images.

The manual object labels may then be compared with object labels applied by the computing system of the autonomous vehicle to determine whether the one or more object detection parameters should be optimized. In this manner, the disclosed apparatus and method increases the likelihood that the computing system of the autonomous vehicle recognizes an object depicted in one or more raw images.

FIG. 1illustrates an apparatus102for optimizing the one or more object detection parameters. In one embodiment, the apparatus may include an autonomous vehicle104configured to communicate with an object identification server132. The autonomous vehicle104may be configured to operate autonomously, e.g., drive without the assistance of a human driver. Moreover, the autonomous vehicle104may be configured to detect various objects and determine the types of detected objects while the autonomous vehicle104is operating autonomously.

While certain aspects of the disclosure are particularly useful in connection with specific types of vehicles, the autonomous vehicle104may be any type of vehicle including, but not limited to, cars, trucks, motorcycles, busses, boats, airplanes, helicopters, lawnmowers, recreational vehicles, amusement park vehicles, farm equipment, construction equipment, trams, golf carts, trains, and trolleys.

The autonomous vehicle104may be equipped with various types of sensors106for detecting objects near and/or around with the autonomous vehicle104. For example, the autonomous vehicle104may be equipped with one or more cameras112for capturing images of objects in front of and/or behind the autonomous vehicle104. As another example, the autonomous vehicle104may be equipped with one or more lasers114for detecting objects near and/or around the autonomous vehicle104. Moreover, the autonomous vehicle104may be equipped with one or more radars116for detecting objects near and/or around the autonomous vehicle104.

WhileFIG. 1illustrates that the autonomous vehicle104may be equipped with one or more cameras112, one or more lasers114, and one or more radars116, the autonomous vehicle104may be equipped with alternative arrangements of sensors. For example, the autonomous vehicle104may be equipped with sonar technology, infrared technology, accelerometers, gyroscopes, magnometers, or any other type of sensor for detecting objects near and/or around the autonomous vehicle104.

The autonomous vehicle104may also include a memory108and a processor110operative to capture raw images using the sensors106. While shown as a single block, the memory108and the processor110may be distributed across many different types of computer-readable media and/or processors. The memory108may include random access memory (“RAM”), read-only memory (“ROM”), hard disks, floppy disks, CD-ROMs, flash memory or other types of computer memory.

AlthoughFIG. 1functionally illustrates the processor110, the memory108, and other elements of the autonomous vehicle104as being within the same block, it will be understood by those of ordinary skill in the art that the processor110, the memory108, and the sensors106may actually comprise multiple processors, computers, or memories that may or may not be stored within the same physical housing.

The memory108may be operative to store one or more images118-122captured by one or more of the sensors106. The captured raw images may include raw camera images118captured using the one or more cameras112, laser point cloud images120captured using the one or more lasers114, or radar intensity images122captured using one or more radars. Depending on the type of sensors used by the autonomous vehicle104, the memory108may store other types of images as well.

The images118-122may be formatted in any computer-readable format. For example, the images118-122data may be stored as bitmaps comprised of grids of pixels that are stored in accordance with formats that are compressed or uncompressed, lossless (e.g., BMP) or lossy (e.g., JPEG), and bitmap or vector-based (e.g., SVG), as well as computer instructions for drawing graphics.

The raw camera images116may include one, two, or three-dimensional images having a predetermined number of megapixels. The raw camera images116may further be in color, black and white, or in any other format. The one or more cameras112may be operative to capture the one or more raw camera image(s)118at predetermined time intervals, such as every one millisecond, every second, every minute, or at any other interval of time. Other measurements of capturing images may also be possible, such as 30 frames per second (“fps”) 60 fps, or any other measurement.

The laser point cloud images120may include one or more images comprised of laser points representing a predetermined view angle near and/or around the autonomous vehicle104. For example, the laser point cloud images120may include one or more laser point cloud images representing a 360° view around the autonomous vehicle104. The laser point cloud images120may include a predetermined number of laser points, such as 50,000 laser points, 80,000 laser points, 100,00 laser points, or any other number of laser points. As with the raw camera images118, the autonomous vehicle104may be configured to capture the one or more laser point cloud images120at predetermined time intervals, such as 10 fps, 30 fps, every millisecond, every second, or at any other interval of time.

The radar intensity images122may include one or more images captured using a radar technology. As with the laser point cloud images120or the raw camera images116, the radar intensity images122may be captured at predetermined time intervals.

Although the sensors106may be configured to capture images at predetermined time intervals, the predetermined time intervals may vary from sensor to sensor. Thus, the one or more camera(s)112may be configured to capture one or more raw images118at a time interval different than the one or more laser(s)114, which may also capture one or more laser point cloud images120at a time interval different than the radar(s)116. Hence, it is possible that the autonomous vehicle104is capturing an image, whether using the camera(s)112, the laser(s)114, or the radar(s)116at any given time.

The autonomous vehicle104may also include a processor110operative to control the sensors106to capture the one or more images118-122. The processor110may be any conventional processor, such as commercially available central processing units (“CPUs”). As one example, the processor110may be implemented with a microprocessor, a microcontroller, a digital signal processor (“DSP”), an application specific integrated circuit (ASIC), discrete analog or digital circuitry, or a combination of other types of circuits or logic.

The memory108may also be operative to store an object detector130. The object detector130may be any configuration of software and/or hardware configured to detect an object in an image118-122captured by one or more of the sensors106. As an image is captured by one or more of the sensors106, the image may be communicated to the object detector130, which may analyze the image to determine whether there is an object present in the image. The object in the captured image may be any type of object, such as a vehicle, pedestrian, a road sign, a traffic light, a traffic cone, or any other type of object.

To determine whether an object is present in the image undergoing processing, the object detector130may leverage one or more image parameters124-128. The image parameters124-128may instruct the object detector130when an arrangement of pixels, laser points, intensity maps, etc., should be considered an object. The image parameters124-128may also instruct the object detector130as how to classify the object.

Each of the sensor types may be associated with a corresponding set of image parameters. Thus, the one or more camera(s)124may be associated with camera parameters124, the one or more laser(s)114may be associated with laser parameters126, and the one or more radar(s)116may be associated with radar parameters128. Examples of camera parameters124may include the minimal brightness of a pedestrian, the minimum pixel size of a car object, the minimum width of a car object, and other such parameters. Examples of laser parameters126may include the height of a pedestrian, the length of a car object, an obstacle detection threshold, and other such parameters. Examples of radar parameters128may include minimum distance to an object, a delay threshold for detecting an object, the height of a pedestrian, and other such parameters.

As discussed with reference toFIGS. 12-18, when the object detector130detects an object in an image, the object detector130may define an object label for the detected object. The object label may be defined by a bounding box encompassing the object. In alternative embodiments, the object label may be defined by a bounding oval or other bounding shape.

The object label may have one or more object label parameters that define the shape of the object label. Moreover, the object label parameters may vary depending on the sensor type of the sensor that captured the image. Assuming that the shape of the object label is a bounding box, and that the sensor that captured the image is a one or more of the cameras124, the object label parameters may include a height parameter that defines the height of the bounding box (in pixels), a width parameter that defines the width of the bounding box (in pixels), a first pixel coordinate that defines the latitudinal placement of the bounding box (e.g., an X-coordinate), and a second pixel coordinate that defines the longitudinal placement of the bounding box (e.g., a Y-coordinate). Where the sensor that captured the image is one or more of the lasers126, the object label parameters may also include a third pixel coordinate that defines the physical height of the object or a particular laser point depicted in the captured image (e.g., a Z-coordinate). This third pixel coordinate should not be confused with the height parameter of the object label, because this third pixel coordinate may indicate the elevation of the detected object or of a given laser point (e.g., 3 meters above sea level, 2 meters above sea level, etc.) This third pixel coordinate may further indicate the height of the detected object or laser point relative to the autonomous vehicle104.

In addition, the object label applied by the object detector130may be associated with an image frame number that identifies the image in which the detected object may be located. As a moving object may be located in a number of images, such as a moving vehicle captured by one or more of the cameras124, the moving object may appear in different locations in different images. Hence, the moving object may have a number of different object labels associated with it, and each of the object labels may be associated with a corresponding image number to identify the location of the moving object across multiple images.

The autonomous vehicle104may also be in communication with an object identification server132. The object identification server132may be operative to verify the objects detected by the autonomous vehicle104using the object detector130. Moreover, the object identification server132may facilitate the optimization of one or more of the parameters124-128used by the object detector130to detect objects in the captured images118-122. In one embodiment, the autonomous vehicle104may communicate the object labels, and their corresponding object label parameters, to the object identification server132for verifying that the object labels were correctly, or substantially correctly, applied to objects appearing in one or more of the captured images118-122. The implementation of the object identification server132is discussed with reference toFIG. 12.

The object identification server132may also be in communication with one or more client devices134-138via a network142. The networks140-142may be implemented as any combination of networks. Moreover, the networks140-142may be the same network. The networks140-142may also be various types of networks. As examples, the networks140-142may be a Wide Area Network (“WAN”), such as the Internet; a Local Area Network (“LAN”); a Personal Area Network (“PAN”), or a combination of WANs, LANs, and PANs. Moreover, the networks122-128may involve the use of one or more wired protocols, such as the Simple Object Access Protocol (“SOAP”); wireless protocols, such as 802.11a/b/g/n, Bluetooth, or WiMAX; transport protocols, such as TCP or UDP; an Internet layer protocol, such as IP; application-level protocols, such as HTTP, a combination of any of the aforementioned protocols, or any other type of protocol.

The client devices134-138may be operated by a reviewer that may review one or more of the object labels applied by the object detector130. The client devices134-138in communication with the object identification server132may be any type of client device. As examples, and without limitation, the client devices134-138may include one or more desktop computers and one or more mobile devices. Examples of a mobile device include a laptop, a Personal Digital Assistant (“PDA”), a tablet computer, or other such mobile device. Accordingly, a review may communicate and interact with the object identification server132regardless of whether the client devices134-138are desktop computers, mobile devices (e.g., laptops, smartphones, PDAs, etc.), or any other such client device.

The one or more reviewers may also review one or more of the captured images118-122and may manually apply object labels to objects appearing in the one or more captured images118-122. As discussed below with reference toFIG. 12, the object identification server132may compare the manually applied object labels with the object labels applied by the object detector130of the autonomous vehicle104to optimize one or more of the object detection parameters124-128.

In addition, while the object identification server132is shown separately from the client devices134-138, a reviewer may use the object identification server132without a client device. In other words, the object identification server132may be a desktop computer usable by the reviewer without an intermediary client device.

FIG. 2illustrates one example of the autonomous vehicle104and the placement of the one more sensors106. The autonomous vehicle104may include lasers210and211, for example, mounted on the front and top of the autonomous vehicle104, respectively. The laser210may have a range of approximately 150 meters, a thirty degree vertical field of view, and approximately a thirty degree horizontal field of view. The laser211may have a range of approximately 50-80 meters, a thirty degree vertical field of view, and a 360 degree horizontal field of view. The lasers210-211may provide the autonomous vehicle104with range and intensity information that the processor110may use to identify the location and distance of various objects. In one aspect, the lasers210-211may measure the distance between the vehicle and the object surfaces facing the vehicle by spinning on their axes and changing their pitch.

The autonomous vehicle104may also include various radar detection units, such as those used for adaptive cruise control systems. The radar detection units may be located on the front and back of the car as well as on either side of the front bumper. As shown in the example ofFIG. 2, the autonomous vehicle104includes radar detection units220-223located on the side (only one side being shown), front and rear of the vehicle. Each of these radar detection units220-223may have a range of approximately 200 meters for an approximately 18 degree field of view as well as a range of approximately 60 meters for an approximately 56 degree field of view.

In another example, a variety of cameras may be mounted on the autonomous vehicle104. The cameras may be mounted at predetermined distances so that the parallax from the images of two or more cameras may be used to compute the distance to various objects. As shown inFIG. 2, the autonomous vehicle104may include two cameras230-231mounted under a windshield340near the rear view mirror (not shown).

The camera230may include a range of approximately 200 meters and an approximately 30 degree horizontal field of view, while the camera231may include a range of approximately 100 meters and an approximately 60 degree horizontal field of view.

Each sensor may be associated with a particular sensor field in which the sensor may be used to detect objects.FIG. 3Ais a top-down view of the approximate sensor fields of the various sensors.FIG. 3Bdepicts the approximate sensor fields310and311for the lasers210and211, respectively based on the fields of view for these sensors. For example, the sensor field310includes an approximately 30 degree horizontal field of view for approximately 150 meters, and the sensor field311includes a 360 degree horizontal field of view for approximately 80 meters.

FIG. 4Cdepicts the approximate sensor fields320A-323B and for radar detection units220-223, respectively, based on the fields of view for these sensors. For example, the radar detection unit220includes sensor fields320A and320B. The sensor field320A includes an approximately 18 degree horizontal field of view for approximately 200 meters, and the sensor field320B includes an approximately 56 degree horizontal field of view for approximately 80 meters. Similarly, the radar detection units221-223include the sensor fields321A-323A and321B-323B. The sensor fields321A-323A include an approximately 18 degree horizontal field of view for approximately 200 meters, and the sensor fields321B-323B include an approximately 56 degree horizontal field of view for approximately 80 meters. The sensor fields321A and322A extend passed the edge ofFIGS. 3A and 3C.

FIG. 3Ddepicts the approximate sensor fields330-331of cameras230-231, respectively, based on the fields of view for these sensors. For example, the sensor field330of the camera230includes a field of view of approximately 30 degrees for approximately 200 meters, and sensor field331of the camera231includes a field of view of approximately 60 degrees for approximately 100 meters.

In general, an autonomous vehicle104may include sonar devices, stereo cameras, a localization camera, a laser, and a radar detection unit each with different fields of view. The sonar may have a horizontal field of view of approximately 60 degrees for a maximum distance of approximately 6 meters. The stereo cameras may have an overlapping region with a horizontal field of view of approximately 50 degrees, a vertical field of view of approximately 10 degrees, and a maximum distance of approximately 30 meters. The localization camera may have a horizontal field of view of approximately 75 degrees, a vertical field of view of approximately 90 degrees and a maximum distance of approximately 10 meters. The laser may have a horizontal field of view of approximately 360 degrees, a vertical field of view of approximately 30 degrees, and a maximum distance of 100 meters. The radar may have a horizontal field of view of 60 degrees for the near beam, 30 degrees for the far beam, and a maximum distance of 200 meters. Hence, the autonomous vehicle104may be configured with any arrangement of sensors, and each of these sensors may capture one or more raw images for use by the object detector130to detect the various objects near and around the autonomous vehicle104.

FIGS. 4-9are examples of various images that may be captured by one or more sensors106mounted on the autonomous vehicle104.FIG. 4is a first example of a raw camera image402captured by one or more of the cameras112.FIG. 5is a first example of a laser point cloud image502of the view shown in the first raw camera image402.FIG. 6is a second example of a raw camera image602captured by one or more of the cameras112. Similarly,FIG. 7is an example of a laser point cloud image702of the view shown in the raw camera image602.FIG. 8is yet another example of a raw camera image802, andFIG. 9is an example of a laser point cloud image902of the view shown in the raw camera image802.

As shown in the examples ofFIGS. 5,7, and9, a laser point cloud image may substantially or approximately correspond to a raw camera image captured by a camera. Moreover,FIGS. 5,7, and9demonstrate that the autonomous vehicle104may be configured to capture more than one type of laser point cloud image. The autonomous vehicle104may be similarly configured to capture other types of perspectives using other types of sensors as well (e.g., a panoramic image from a camera).

As the autonomous vehicle104is capturing the one or more images118-122, the object detector130may be analyzing the images to determine whether there are objects present in the captured images118-122. As mentioned previously, the object detector130may leverage one or more object detection parameters124-128in determining whether an object is present in the image. To verify or improve the accuracy of detecting objects by the object detector130, the autonomous vehicle104may also communicate one or more of the captured images118-122to the object identification server132. Communicating the captured images118-122to the object identification server132may occur at any time, such as while the autonomous vehicle104is capturing the one or images124-128, after the autonomous vehicle104has captured the one or more images124-128, or at any other time.

FIG. 10illustrates one example of the object identification server132according to aspects of the disclosure. The object identification server132may include a memory1002and a processor1004. The memory1002may include random access memory (“RAM”), read-only memory (“ROM”), hard disks, floppy disks, CD-ROMs, flash memory or other types of computer memory. In addition, the memory1002may be distributed across many different types of computer-readable media.

The processor1004may be a microprocessor, a microcontroller, a DSP, an ASIC, discrete analog or digital circuitry, or a combination of other types of circuits or logic. In addition, the processor1004may be distributed across many different types of processors.

Interfaces between and within the object identification server132may be implemented using one or more interfaces, such as Web Services, SOAP, or Enterprise Service Bus interfaces. Other examples of interfaces include message passing, such as publish/subscribe messaging, shared memory, and remote procedure calls.

The memory1002may be operative to store one or more databases. For example, the memory1002may store a raw sensor image database1006, an autonomous vehicle object identification database1008, and a reviewer object identification database1010. One or more of the databases1006-1010may be implemented in any combination of components. For instance, although the databases1006-1010are not limited to any single implementation, one or more of the databases1006-1010may be stored in computer registers, as relational databases, flat files, or any other type of database.

Although shown as a single block, the object identification server132may be implemented in a single system or partitioned across multiple systems. In addition, one or more of the components of the object detection server132may be implemented in a combination of software and hardware. In addition, any one of the components of the object identification server132may be implemented in a computer programming language, such as C#, C++, JAVA or any other computer programming language. Similarly, any one of these components may be implemented in a computer scripting language, such as JavaScript, PHP, ASP, or any other computer scripting language. Furthermore, any one of these components may be implemented using a combination of computer programming languages and computer scripting languages.

The raw sensor image database1006may store one or more of the images communicated by the autonomous vehicle104to the object identification server132. Accordingly, the raw sensor image database1006may include images1012captured by one or more of the cameras112, images1014captured by one or more of the lasers114, and images1016captured by one or more of the radars116. The images1012-1016may be formatted in any computer-readable format. For example, the images1012-1016data may be stored as bitmaps comprised of grids of pixels that are stored in accordance with formats that are compressed or uncompressed, lossless (e.g., BMP) or lossy (e.g., JPEG), and bitmap or vector-based (e.g., SVG), as well as computer instructions for drawing graphics. Moreover, the images1012-1016stored in the raw sensor image database1006may correspond to, or be copies of, the images118-122stored in the memory108of the autonomous vehicle104.

The autonomous vehicle object identification database1008may include the object label parameters determined by the object detector130for the objects appearing in the one or more images1012-1016. Thus, in one embodiment, for each object label applied to each object detected by the object detector130, the object identification server132may store the set of parameters that define each of the object labels. As a set of object label parameters define an object label, the autonomous vehicle object identification database1008may be considered to effectively store object labels.

In addition, the autonomous vehicle object identification database1008may store object labels for each type of image. Thus, the autonomous vehicle object identification database1008may store object labels1018for the camera images1012, object labels1020for the laser point cloud images1014, and object labels1022for the radar images1016.

The memory1002may also store a reviewer object identification application1024executable by the processor1004. A reviewer may use the reviewer object identification application1024to identify objects (e.g., apply object labels to objects) appearing in the images1012-1016stored in the raw sensor image database1006. A reviewer may include a human reviewer or a computerized reviewer operative to communicate with the reviewer object identification application1024. While the reviewer object identification application1024is shown as residing in the memory1010of the object identification server132, the object identification application1024may alternatively reside in the memory of a client device in communication with the object identification server132.

To apply object labels to object, the reviewer object identification application1024may display each image to the reviewer. The reviewer may then draw an object label, such as a bounding box or other shape, around an object that the autonomous vehicle104should recognize or detect. The reviewer may also provide identification information for the identified object, such as an object name (e.g., “vehicle,” “bicycle,” “pedestrian,” etc.). Alternatively, the object name may be selectable by the reviewer, such as being selectable as part of a drop-down menu or other graphical menu. The reviewer object identification application1024may then store the object label parameters that define the object label in the reviewer object identification database1010. As discussed previously with regard to the object detector130, the object label parameters may include a width parameter, a height parameter, an X-parameter, a Y-parameter, an image number parameter, and, where the image undergoing review is a laser point cloud image, a Z-parameter.

In addition, the reviewer object identification application1024may employ interpolation techniques to reduce the strain on the reviewer of identifying objects. For example, the reviewer may identify (e.g., by electronically drawing a bounding box around an object using a mouse or other input device) an object appearing in a first image, and the reviewer may identify the object appearing in a last image. The reviewer object identification application1024may then interpolate the object label parameters for the object appearing in images between the first image and the last image. Thus, in instances where an object, such as a moving vehicle traveling alongside the autonomous vehicle104, appears in hundreds or thousands of images, the reviewer object identification application1024may reduce the time and effort required by the reviewer to identify the object in each image.

The reviewer object identification database1010may store the object label parameters determined by the reviewer object identification application1024. The object label parameters may include object label parameters1026for raw camera images, object label parameters1028for laser point cloud images, and object label parameters1030for radar intensity images. The reviewer object identification database1010may store object label parameters for other types of images and/or sensors as well, such as object label parameters for sonar images, infrared images, or any other type of image.

FIGS. 11-16are examples of object labels electronically applied to the images captured by the sensors of the autonomous vehicle104. The object labels shown inFIGS. 11-16have a rectangular shape, but any other shape is also possible. The object labels shown inFIGS. 11-16are examples of object labels that may be applied by the object detector130of the autonomous vehicle104or may have been applied by a reviewer using the reviewer object identification application1024.

Moreover, the object detector130may, as opposed to electronically drawing the object labels on the images, store the object label parameters that define the object label. In contrast, for expediency, a reviewer may draw an object label around an object using an input device in conjunction with the reviewer object identification application1024, and the reviewer object identification application1024may determine the object label parameters based on the dimensions of the drawn object label and other aspects of the given image, such as the X-coordinate pixel and the Y-coordinate pixel derived from the given image's resolution.

FIG. 11shows the raw camera image402ofFIG. 4with three rectangular object labels1102-1106applied to three different objects.FIG. 12shows the laser point cloud image502ofFIG. 5with two rectangular object labels1202-1204applied to two different objects.FIG. 13shows the raw camera image602ofFIG. 6with one object label1302applied to a single object.FIG. 14shows the laser point cloud image702ofFIG. 7with one object label1402applied to a single object.FIG. 15shows the raw camera image802with three rectangular object labels1502-1506applied to three different objects.FIG. 16shows the laser point cloud image802ofFIG. 8with thirteen object labels1602-1626applied to thirteen different objects.

With the object label parameters1026-1030from the reviewers and the object label parameters1018-1022from the autonomous vehicle104, the object identification server132may then proceed to optimizing the object detection parameters124-128used by the object detector130of the autonomous vehicle104. To this end, the object identification server132may include an object identification analyzer1032executable by the processor1004for performing the optimization.

In optimizing the object detection parameters124-128, the object identification analyzer1032may first determine whether an optimization operation should be performed. To make this determination, the object identification analyzer1032may compare the object labels applied by the autonomous vehicle104with the object labels applied by the one or more reviewers. Comparing the object labels of the autonomous vehicle104with the object labels applied by the one or more reviewers may include comparing the object label parameters1018-1022received from the autonomous vehicle104with the corresponding type of object label parameters1026-1030derived from the object labels applied by the one or more reviewers.

In one embodiment, the object identification analyzer1032may compare, for each image, the number of object labels applied by a reviewer with the number of object labels applied by the autonomous vehicle104. In this embodiment, the object identification analyzer1032may determine whether the autonomous vehicle104detected the same number of objects as identified by a reviewer. A predetermined “missed object” threshold may be established for the object identification analyzer1032that establishes the number of permissible objects, as an absolute number or percentage, that the autonomous vehicle104is allowed to not detect. In this embodiment, should the autonomous vehicle104not detect a given percentage or number of objects (e.g., the autonomous vehicle104did not detect 3% of the objects identified by a reviewer), the object identification analyzer1032may display to the reviewer the percentage of objects not detected by the autonomous vehicle104. The object identification analyzer1032may then recommend optimization of the object detection parameters124-128. The object identification analyzer1032may also display the results of this analysis to the reviewer. Alternatively, the object identification analyzer1032may automatically proceed to the optimization of the object detection parameters124-128.

In another embodiment of comparing the autonomous vehicle object label parameters1018-1022with the reviewer object label parameters1026-1030, the object identification analyzer1032may determine whether any of the object labels applied by the autonomous vehicle overlap with any of the object labels applied by the reviewer. In this embodiment, the object identification analyzer1032may determine, not only whether the autonomous vehicle104detected the same number of objects as a reviewer, but whether the autonomous vehicle104detected the same objects. This comparison may also indicate how accurately the autonomous vehicle104detected an object.

To determine the accuracy of the object labels applied by the autonomous vehicle104, the object identification analyzer1032may determine an object label ratio defined as:

where “intersection(object labelautonomous vehicle, object labelreviewer)” is the area of the intersection of the object label applied by the autonomous vehicle104and the object label applied by the reviewer, and “union(object labelautonomousvehicle, object labelreviewer)” is the area of the union of the object label applied by the autonomous vehicle104with the object label applied by the reviewer. Where there is a direct correlation (e.g., a direct overlap) between the object label applied by the autonomous vehicle104and the object label applied by the reviewer, the object label ratio may have a value of 0.5. With an imperfect correlation, the object label ratio may have a value less than 0.5. Each object label in each image may be associated with an object label ratio. The object identification analyzer1032may also assign an object label ratio to objects that the autonomous vehicle104did not detect or an object label ratio to objects that the autonomous vehicle104incorrectly detected (e.g., the autonomous vehicle104detected an object that a reviewer did not identify).

Using the object label ratios, where each object label in each image for a given set of images has an object label ratio, the object identification analyzer1032may determine the mean object label ratio value. Thus, the set of raw camera images1012may have a mean object label ratio value, the set of raw laser point cloud images1014may have a mean object label ratio value, and the set of raw radar intensity images1016may have a mean object label ratio value. Similar to the predetermined “missed object” threshold, the object identification analyzer1032may be configured with an object label ratio threshold (e.g., 0.35%) for each image type, and the mean object label ratio for a given set of images may be compared with the object label ratio threshold.

Since the different types of sensors may detect objects differently, each image type may be associated with a different value for the object label ratio threshold. For example, the set of raw camera images1012may be associated with an object label ratio threshold of 0.4%, the set of raw laser point cloud images1014may be associated with an object label ratio threshold of 0.35%, and the set of raw radar intensity images1016may be associated with an object label ratio of 0.37%. Of course, the set of raw images1012-1016may also be associated with the same value for the object label ratio threshold.

In this embodiment, where the mean object label ratio for a given set of images does not meet (or exceeds) the object label ratio threshold associated with the set of images, the object identification analyzer1032may display to the reviewer a level of inaccuracy in detecting objects by the autonomous vehicle104. The object identification analyzer1032may then recommend optimization of the object detection parameters124-128. The object identification analyzer1032may also display the results of this analysis to the reviewer. Alternatively, the object identification analyzer1032may automatically proceed to the optimization of the object detection parameters124-128.

In yet a third embodiment, the object identification analyzer1032may compare a computed speed of an object labeled by the autonomous vehicle104with a computed speed of an object labeled by a reviewer. The object identification analyzer1032may compute the speed of an object by determining the distance an object travels in a series of one or more images (e.g., since the object identification analyzer1032may be configured with or derive the rate at which the images were captured). The object identification analyzer1032may then determine the differences in speed between objects detected by the autonomous vehicle104and the corresponding objects identified by the reviewer. The object identification analyzer1032may then recommend optimization of the object detection parameters124-128based on the number and value of the determined speed differences. The object identification analyzer1032may also display the results of this analysis to the reviewer. Alternatively, the object identification analyzer1032may automatically proceed to the optimization of the object detection parameters124-128.

The object identification server132may use one or more optimization techniques to optimize the various object detection parameters124-128. In one embodiment, the object identification server132performs the detection of objects with the instructions used by the object detector130using the possible combinations of values of the object detection parameters124-128. For example, suppose that the object detection parameters124for raw camera images include ten parameters, and each parameter may have one of ten values. In this example, the object identification server132may perform the object detection analysis 1010times, and for each performance of the object detection analysis, the object identification server132may store a separate set of object labels (e.g., object label parameters). Thus, in this example, the object detection analysis may result in 1010different sets of object label parameters. The object identification server132may perform this object detection analysis for each sensor type, for each sensor, or for combinations thereof.

Having performed the object detection analysis with the possible combination of values of the object detection parameters, the object identification server132may then invoke the object identification analyzer1032for each set of object label parameters. Thus, using the example above, the object identification analyzer1032may perform 1010analyses with one or more of the comparison embodiments previously discussed (e.g., the “missed object” analysis, the object label ratio analysis, and/or the object speed difference analysis).

The object identification server132may then select or display the combination of values for the object detection parameters that resulted in the most favorable outcome for each analysis. For example, for the “missed object” analysis, the object identification server132may display the set of values for the object detection parameters that resulted in the least number of objects that were not detected or incorrectly detected. As another example, for the object label ratio analysis, the object identification server132may display the set of values for the object detection parameters that resulted in the mean object label ratio closest to (or farthest from) the mean object label ratio threshold. Although the number of analyses may exponentially increase with each object detection parameter, the commercial availability of high-performance processors has made this optimization technique a practical reality. The autonomous vehicle104may then be configured with the optimized values for the selected set of object detection parameters.

FIG. 17illustrates one example of logic flow1702for optimizing one or more sets of object detection parameters124-128of the autonomous vehicle104. As previously discussed, the autonomous vehicle104may capture one or more images118-122using one or more sensors112-116. The autonomous vehicle104may then detect objects in the one or more captured images118-122. In detecting these objects, the autonomous vehicle104may determine object label parameters for each of the detected objects in each of the images. The object label parameters124-128and the captured images118-122may then be communicated to the object identification server132(Block1704).

The object identification server132may then display each of the captured images to a reviewer for identifying objects. In response, the object identification server132may receive object labels applied to the objects identified by the reviewers (Block1706). As with the object label parameters determined by the autonomous vehicle104, the object labels applied by the one or more reviewers may be stored as object label parameters. In certain cases, the object identification server132may apply the object labels to objects (e.g., during interpolation when a reviewer has identified an object in a first image and the object in a last image).

The object identification server132may then determine whether to perform optimization on one or more sets of the object detection parameters124-128of the autonomous vehicle104(Block1708). The object identification132may make this determination using one or more comparison analyses previously discussed (e.g., the “missed object” analysis, the mean object label ratio analysis, and/or the object speed difference analysis).

Depending on the results of the one or more analyses, the object identification server132may recommend optimization of one or more sets of object detection parameters124-128. Alternatively, the object identification server132may automatically perform the optimization. Where the object identifications server132recommends the optimization of one or more sets of object detection parameters124-128, and receives instructions to perform the optimization, the object identification server132may then perform the optimization of the one or more object detection parameters as previously discussed (Block1710). The results of the optimization may then be displayed or incorporated into the autonomous vehicle104(Block1712).

In this manner, the object identification server132facilitates the optimization of various object detection parameters used by the autonomous vehicle104. To increase the accuracy of the objects detected by the autonomous vehicle104, the object identification server132may leverage input provided by reviewers. The input may be the identification of objects in the raw images captured by the autonomous vehicle104. Since the sensors on the autonomous vehicle104may be different sensor types, the object identification server132may leverage different comparison schemes in comparing the object labels of the reviewers with the object labels of the autonomous vehicle. The results of the comparison inform the object identification server132whether to recommend optimizing the object detection parameters. Moreover, leveraging various comparison schemes increases the likelihood that the recommendation by the object identification server132is a correct and valid recommendation. When performed, the optimization of the object detection parameters by the object identification server increases the likelihood that the autonomous vehicle will more accurately detect an object in a captured image. Increases in the accuracy of detecting objects yields such benefits as a safer driving experience, increased response times, increased predicted behavior responses, and other such benefits.

Although aspects of this disclosure have been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of this disclosure as defined by the appended claims. Furthermore, while certain operations and functions are shown in a specific order, they may be performed in a different order unless it is expressly stated otherwise.