Automatically identifying faulty signatures for autonomous driving applications

A method for automatically identifying faulty signatures for autonomous driving applications, the method includes receiving, by a processing circuit, a signature; matching the signature to a group of first signatures that are untagged and are randomly obtained; identifying, based on the matching, first top matching signatures; matching the signature to a group of second signatures that are untagged and are correctly or erroneously indicative of a detection of reference elements; identifying, based on the matching of the signature to the second group of second signatures, second top matching signatures; determining an overlap between the first top matching signatures and the second top matching signatures; and determining whether the signature is faulty or faultless based on the overlap.

The present disclosure relates to the field of autonomous driving, and more particularly, to a method, and a non-transitory computer-readable storage medium for automatically identifying faulty signatures for autonomous driving applications.

BACKGROUND

A perception system is a key building block of all modern advanced driving assistance system (ADAS) and autonomous vehicle (AV) solutions. The system is responsible for detection, tracking and measurement of driving related entities, such as road users, lanes, traffic signs and traffic lights. The output of the perception system is a 3D environmental model, which is used as a basis for every decision making and path planning of the automated vehicle.

All the modern perception systems are based on state-of-the-art deep-learning technology.

The deep learning models can be trained using supervised or unsupervised training. Both methods have limitations that may result in errors.

Therefore, there is a growing need to automatically detect error resulting from a detection process applied by a deep learning model in the context of autonomous driving applications.

SUMMARY

The present disclosure provides a method and a non-transitory computer-readable storage medium for identifying faulty signatures.

In a first aspect of the present disclosure, a method for automatically identifying faulty signatures for autonomous driving applications, the method includes: receiving, by a processing circuit, a signature; matching the signature to a group of first signatures that are untagged and are randomly obtained; identifying, based on the matching, first top matching signatures; matching the signature to a group of second signatures that are untagged and are correctly or erroneously indicative of a detection of reference elements; identifying, based on the matching of the signature to the second group of second signatures, second top matching signatures; determining an overlap between the first top matching signatures and the second top matching signatures; and determining whether the signature is faulty or faultless based on the overlap.

In another aspect of the present disclosure, a non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations for automatically identifying faulty signatures for autonomous driving applications, including receiving, by a processing circuit, a signature; matching the signature to a group of first signatures that are untagged and are randomly obtained; identifying, based on the matching, first top matching signatures; matching the signature to a group of second signatures that are untagged and are correctly or erroneously indicative of a detection of reference elements; identifying, based on the matching of the signature to the second group of second signatures, second top matching signatures; determining an overlap between the first top matching signatures and the second top matching signatures; and determining whether the signature is faulty or faultless based on the overlap.

DESCRIPTION OF EXAMPLE EMBODIMENTS

There is provided a method, a non-transitory computer readable medium and a system for automatically identifying faulty signatures for autonomous driving applications.

Examples of autonomous driving applications includes ADAS applications, autonomous driving applications, and the like.

The different figures illustrates examples of units and/or software and/or information items and/or steps and/or components. These examples are provided for brevity of explanation. At least one of the units and/or software and/or information items and/or steps and/or components is optional or mandatory.

There is provided a computer implemented method and a non-transitory computer readable medium that uses signatures identified as faulty or not faulty—where the identification process is cost effective, does not require manual tagging and is reliable even when the signatures are not associated with a define class. The identification of the signatures as faulty or as not—is reliable.

FIGS.1A,1B and2illustrate examples of a vehicle100, a network132and remote computerized systems134.

InFIG.1Athe vehicle100is illustrated as including sensing system110, a communication system130, one or more memory and/or storage units120, control unit125′, network132in communication with remote computerized systems134.

The one or more memory and/or storage units120is illustrated as storing information191, metadata192, software193and operating system194. The information191, metadata192, software193and operating system194are required for executing one or more methods illustrated in the specification—such as method200.

Processor126ofFIG.1Ais illustrated as including a plurality of processing units126(1)-126(J), J is an integer that exceeds one.

InFIGS.1B and2the control unit125′ is replaced by different components such as advanced driver assistance system (ADAS) control unit123, autonomous driving control unit122, vehicle computer121, and controller125. It is noted that only some of these components may be included in the vehicles.

FIGS.1B and2also provide examples of one or more types of information191and metadata192and/or software193stored in the one or more memory and/or storage units120.

Communication system130, one or more memory and/or storage units120, and processing system124may form a computerized system. The computerized system may include one or more other systems and/or units such as sensing system110(at least the image signal processor114), the ADAS control unit123, the autonomous driving control unit122, the vehicle computer121, and the controller125.

The sensing system110includes optics111, sensing element group112, a readout circuit113, and an image signal processor114. Optics111are followed by a sensing element group such as line of sensing elements or an array of sensing elements that form the sensing element group112. The sensing element group112is followed by a readout circuit113that reads detection signals generated by the sensing element group112. An image signal processor114is configured to perform an initial processing of the detection signals—for example by improving the quality of the detection information, performing noise reduction, and the like. The sensing system110is configured to output one or more sensed information units (SIUs).

The communication system130is configured to enable communication between the one or more memory and/or storage units120and/or the sensing system110and/or any one of the additional units and/or the network132(that is in communication with the remote computerized systems).

The controller125is configured to control the operation of the sensing system110, and/or the one or more memory and/or storage units120and/or the one or more additional units (except the controller).

The ADAS control unit123is configured to control ADAS operations.

The autonomous driving control unit122is configured to control autonomous driving of the autonomous vehicle.

The vehicle computer121is configured to control the operation of the vehicle-especially control the engine, the transmission, and any other vehicle system or component.

The processing system124may include processor126and one or more other processors and is configured to execute any method illustrated in the specification.

The one or more memory and/or storage units120are configured to store firmware and/or software, one or more operating systems, data and metadata required to the execution of any of the methods mentioned in this application.

FIG.1Billustrates the one or more memory and/or storage units120as storing:First group of signatures171.Second group of signatures172.First top matching signatures181. This may include the most matching signatures—for example the top 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 20, 25, 30 or any other number of the most matching signatures. According to an embodiment, a match is determined based on a distance between the (evaluated) signature and signatures of the first group of signatures. Any distance may be calculated. According to an embodiment, a signature includes indexes for data retrieval and the match includes an exact match.Second top matching signatures182. This may include the most matching signatures—for example the top 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 20, 25, 30 or any other number of the most matching signatures.Signature generation software161is configured to generate a signature. The signature may be generated based on the SIU but may be generated based on other information. A detector may be provided when the processing system124executes the signature generation software. Examples of a signature and signature generators that are examples of detectors are illustrated in U.S. patent application Ser. No. 17/309,064 publication serial number 2022/0041184 which is incorporated herein by reference.Signature matching software162is configured to (a) match the signature to signatures of the first group of signatures171to provide the first top matching signatures181, and to (b) match signature to signatures of the second group of signatures172to provide the second top matching signatures182.Matching based signature status determination software163is configured to determine whether the signature is faulty or non-faulty (i.e., faultless) based on an overlap between the first top matching signatures181and the second top matching signatures182. The overlap may be signatures that are included in both the first and second top matching signatures.Parameter adjusting software164is configured to adjust any parameter related to the determination of whether the signature is OK or faulty.One or more signatures is generated by a detector165. The one or more signatures include signatures that may be evaluated to determine whether they are OK or faulty.Operating system166.Historic detector signatures accuracy information167is indicative of an accuracy (reflected by having OK or faulty signatures) of the detector. This history information may provide an indication of whether the detector increases its accuracy, maintains its accuracy, or decreases its accuracy over time. The parameter adjustment software may use such information.Additional software168that may be used to perform any other functionality of the vehicle and/or of any of the other units illustrated inFIG.1B.Validated signatures of the detector175—includes signatures of the detector that were tested to be OK or faulty.A white list176of signatures generated by the detector that are presumed to be OK (non-faulty).A black list177of signatures generated by the detector that are presumed to be faulty.A training dataset178used for training a machine learning process. It may be updated by adding signatures that were found to be OK—and these signatures may be flagged as OK. It may also be updated by adding signatures as being faulty—which are also flagged as faulty.A testing dataset179used for testing a machine learning process. It may be updated by adding signatures that were found to be OK—and these signatures may be flagged as OK. It may also be updated by adding signatures as being faulty—which are also flagged as faulty.

The vehicle computer121may be in communication with an engine control module, a transmission control module, a powertrain control module, and the like

The memory and/or storage units120was shown as storing software. Any reference to software should be applied mutatis mutandis to code and/or firmware and/or instructions and/or commands, and the like.

Any reference to one unit or item should be applied mutatis mutandis to multiple units or items. For example—any reference to processor should be applied mutatis mutandis to multiple processors, any reference to communication system130should be applied mutatis mutandis to multiple communication systems.

According to an embodiment, the one or more memory and/or storage units120includes one or more memory unit, each memory unit may include one or more memory banks.

According to an embodiment, the one or more memory and/or storage units120includes a volatile memory and/or a non-volatile memory. The one or more memory and/or storage units120may be a random access memory (RAM) and/or a read only memory (ROM).

According to an embodiment, the non-volatile memory unit is a mass storage device, which can provide non-volatile storage of computer code, computer readable instructions, data structures, program modules, and other data for the processor or any other unit of vehicle. For example and not meant to be limiting, a mass storage device can be a hard disk, a removable magnetic disk, a removable optical disk, magnetic cassettes or other magnetic storage devices, flash memory cards, CD-ROM, digital versatile disks (DVD) or other optical storage, random access memories (RAM), read only memories (ROM), electrically erasable programmable read-only memory (EEPROM), and the like.

Any content may be stored in any part or any type of the memory unit.

According to an embodiment, the at least one memory unit stores at least one database—such as any database known in the art—such as DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, MySQL, PostgreSQL, and the like.

Various units and/or components are in communication with each other using any communication elements and/or protocols. Communication elements other than communication system130may be provided.

FIGS.1A,1B and2illustrate communication system130as being in communication with various processors and/or units and network132.

The communication system130may include a bus. The represents one or more of several possible types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures can comprise an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, an Accelerated Graphics Port (AGP) bus, and a Peripheral Component Interconnects (PCI), a PCI-Express bus, a Personal Computer Memory Card Industry Association (PCMCIA), Universal Serial Bus (USB) and the like. The bus, and all buses specified in this description can also be implemented over a wired or wireless network connection and each of the subsystems.

Network132that is located outside the vehicle and is used for communication between the vehicle and at least one remote computing system. By way of example, a remote computing system can be a personal computer, a laptop computer, portable computer, a server, a router, a network computer, a peer device or other common network node, and so on. Logical connections between the processor and either one of remote computing systems can be made via a local area network (LAN) and a general wide area network (WAN). Such network connections can be through a network adapter (may belong to communication system130) which can be implemented in both wired and wireless environments. Such networking environments are conventional and commonplace in offices, enterprise-wide computer networks, intranets, and a larger network such as the internet.

It should be noted that at least a part of the content illustrated as being stored in one or more memory/storage units120may be stored outside the vehicle. It should also be noted that the processor may evaluate signatures generated by a plurality of detectors.

FIG.2illustrates an example of a vehicle100, network132, remote computerized systems134, and also illustrates an external memory/storage unit136.

FIG.2differs fromFIG.1Bby illustrating an external memory/storage unit136that stores the first group of signatures171, the second group of signatures172, the training dataset178, the testing dataset179, the white list176and the black list177. In contrary toFIG.1B—the training dataset178, the testing dataset179, the white list176and the black list177are not stored in the one or more memory/storage units120.

FIG.2also differs fromFIG.1Bby illustrating the one or more memory/storage units120as storing validated signatures from a plurality (K) of detectors—from the validated signatures of the first detector175(1) till the validated signatures of the K'th detector175(K).

FIG.2further differs fromFIG.1Bby illustrating the one or more memory/storage units120as storing one or more signatures from the plurality (K) of detectors—from the signatures of the first detector165(1) till the signatures of the K'th detector165(K).

FIG.3illustrates an example of method200that is computer implemented and is for automatically identifying faulty signatures for autonomous driving applications.

According to an embodiment, method200includes step210of receiving, by a processing circuit, a signature associated with an identification of an element, the element is at least one of an object or a road scenario. Accordingly—the signature may be associated with an identification or an object. Alternatively—the signature may be associated with an identification of a scenario.

A scenario may be, for example, at least one of (a) a location of the vehicle, (b) one or more weather conditions, (c) one or more contextual parameters, (d) a road condition, (e) a traffic parameter.

Various examples of a road condition may include the roughness of the road, the maintenance level of the road, presence of potholes or other related road obstacles, whether the road is slippery, covered with snow or other particles.

Various examples of a traffic parameter and the one or more contextual parameters may include time (hour, day, period or year, certain hours at certain days, and the like), a traffic load, a distribution of vehicles on the road, the behavior of one or more vehicles (aggressive, calm, predictable, unpredictable, and the like), the presence of pedestrians near the road, the presence of pedestrians near the vehicle, the presence of pedestrians away from the vehicle, the behavior of the pedestrians (aggressive, calm, predictable, unpredictable, and the like), risk associated with driving within a vicinity of the vehicle, complexity associated with driving within of the vehicle, the presence (near the vehicle) of at least one out of a kindergarten, a school, a gathering of people, and the like. A contextual parameter may be related to the context of the sensed information-context may be depending on or relating to the circumstances that form the setting for an event, statement, or idea.

Examples of situations and of a situation based processing are illustrated in U.S. patent application Ser. No. 16/035,732 which is incorporated herein by reference.

According to an embodiment, step210includes accessing a memory unit or a buffer that stores signatures generated by the detector.

According to an embodiment, step210is followed by steps220and240.

According to an embodiment, step220includes matching the signature to a group of first signatures that are untagged and may be obtained in any manner—for example may be randomly obtained—or in any manner that is made without knowing the content represented by the signatures and/or any manner that is made regardless of the content (which may be unknown) represented by the signatures.

According to an embodiment, step220is followed by step230of identifying, based on the matching of step220, first top matching signatures.

According to an embodiment, step240includes matching the signature to a group of second signatures that are untagged and are correctly or erroneously indicative of a detection of reference elements. For example—a second signature was determined (correctly or erroneously) by the detector to identify a certain reference element. A second signature that was correctly indicative of a detection of a reference element may be a true positive signature or a true negative signature. A second signature was incorrectly indicative of a detection of a reference element may be a false positive signature or a false negative signature.

According to an embodiment, step240is followed by step250of identifying, based on the matching of step240second top matching signatures.

According to an embodiment, steps230and250are followed by step260of determining an overlap between the first top matching signatures to the second top matching signatures.

According to an embodiment, an overlapping signature may appear in both the first and second top matching signatures.

According to an embodiment, an overlapping signature may appear in the first top matching signatures and a close enough signature (within a defined distance) appears in the second top matching signatures.

According to an embodiment, step260is followed by step270of determining whether the signature is faulty or not-faulty based on the overlap.

According to an embodiment step271includes comparing the overlap to a threshold. The threshold value may be dynamically updated—for example may range between 1 to 99 percent or between any subrange that is a part of the range of 1 to 99 percent (for example 5, 15, 20, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95 percent and the like) or any value that equals N1/N2 whereas N1 and N2 are positive numbers whereas N1 is smaller than N2.

According to an embodiment, step272includes determining that the signature is faulty when the overlap is below the threshold.

According to an embodiment, step273includes determining that the signature is non-faulty when the overlap is above the threshold.

According to an embodiment, when the overlap equals the threshold—the classification of the signature as faulty or not is determined by a defined rule.

According to an embodiment, step270is followed by step280of responding to the outcome of step270.

According to an embodiment, the responding may include at least one of:Generating a signature status indication-indicating whether the signature was faulty or not faulty.Storing the signature status indicator in a memory unit and/or storage unit.Transmitting over a communication link or channel the signature status indication.Tagging the signature as faulty or non-faulty.Sensing feedback to a signature generator that generate the signature received in step210.Triggering or requesting or instructing a removal of the signature from a white list-when found faulty.Triggering or requesting or instructing an insertion of the signature to a white list—when found non-faulty.Triggering or requesting or instructing an addition of the signature to a black list—when found faulty.Triggering or requesting or instructing a removal of the signature from a black list-when found faulty.Triggering or requesting or instructing an execution of step290.Triggering an inclusion of the signature in a training dataset used to train a machine learning process. The signature may be added with its status (faulty or unfaulty) or may be added based on its status.Triggering a deletion of the signature from a training dataset used to train a machine learning process. The signature may be removed based on its status (faulty or non-faulty).Triggering an inclusion of the signature in a testing dataset used to test a machine learning process. The signature may be added with its status (faulty or unfaulty) or may be added based on its status.Triggering a deletion of the signature from a testing dataset used to test a machine learning process. The signature may be removed based on its status (faulty or non-faulty).Triggering or requesting or instructing an initiation of provision of another detector to be allocated to generating the signature.Triggering or requesting or instructing an evaluation of the detector that generated the signature.Triggering or requesting or instructing an evaluation of a sensing unit that generated a sensed information unit that was processed to provide the signature.Triggering or requesting or instructing a re-configuration of a sensing unit that generated a sensed information unit that was processed to provide the signature.Triggering or requesting or instructing an determination of a scenario that was supposed to be represented, at least in part, by the signature.Triggering or requesting or instructing to evaluate a compatibility of the sensing unit to sense the scenario.Triggering or requesting or instructing to evaluate a compatibility of the signature generator to generate signatures related to the scenario.Triggering or requesting or instructing an initiation of provision of another detector to be allocated to generate signatures related to the scenario.Monitoring outcomes of multiple iterations of steps210-270, and sending feedback, based on the monitoring, to a signature generator that generated the signature.Tagging the signature, based on the outcome of step270.

According to an embodiment, method200includes step290of automatically adjusting one or more parameters of method200.

According to an embodiment, step290is automatically adjusted per defined period and/or based on events and/or outcomes of the execution of steps210-280.

For example—step290may be triggered when finding a change in the accuracy of the signatures generated by the detector.

If, for example, the percent of faulty signatures generated by the detector decreases over time—the adjustment can be made under the assumption that the detector is more reliable. Step290may include increasing the threshold, reducing the frequency of execution of method200, calculating the updated false positive and the updated true negative (of the signatures evaluated by method200) and updating the tradeoff between the false positive detections and the true positive detections.

A change in the accuracy that mandates a triggering of step290may be defined in various manners—for example having a rule that defines the minimal number of outcome of step270that merits the triggering. Yet for another example—the change may be above a minimal value—for example a change of 0.001 may not merit the triggering of step290. A hysteresis may be applied in order to reduce too frequent changes of any of the parameters.

According to an embodiment, step292includes determining a value of the threshold based on a dynamically set tradeoff between false positive detections and true positive detections.

According to an embodiment, step294includes determining a value of the threshold based on a dynamically determined accuracy metric of a signature generator that generated the signature.

According to an embodiment, step296includes dynamically determining a number of the first signatures of the first top matching signatures.

According to an embodiment, step298includes dynamically adjusting a signature generator that generated the signature based on an outcome of step280.

Any reference in the specification to a method should be applied mutatis mutandis to a device or system capable of executing the method and/or to a non-transitory computer readable medium that stores instructions for executing the method.

Any reference in the specification to a system or device should be applied mutatis mutandis to a method that may be executed by the system, and/or may be applied mutatis mutandis to non-transitory computer readable medium that stores instructions executable by the system.

Any reference in the specification to a non-transitory computer readable medium should be applied mutatis mutandis to a device or system capable of executing instructions stored in the non-transitory computer readable medium and/or may be applied mutatis mutandis to a method for executing the instructions.

Any combination of any module or unit listed in any of the figures, any part of the specification and/or any claims may be provided.

Any one of the perception unit, narrow AI agents, driving decision unit may be implemented in hardware and/or code, instructions and/or commands stored in a non-transitory computer readable medium, may be included in a vehicle, outside a vehicle, in a mobile device, in a server, and the like.

The vehicle may be any type of vehicle such as a ground transportation vehicle, an airborne vehicle, or a water vessel.

The specification and/or drawings may refer to sensed information unit (SIU). The SIU may be an image, a media unit and the like. Any reference to a media unit may be applied mutatis mutandis to any type of natural signal such as but not limited to signal generated by nature, signal representing human behavior, signal representing operations related to the stock market, a medical signal, financial series, geodetic signals, geophysical, chemical, molecular, textual and numerical signals, time series, and the like. The sensed information may be of any kind and may be sensed by any type of sensors-such as a visual light camera, an audio sensor, a sensor that may sense infrared, radar imagery, ultrasound, electro-optics, radiography, LIDAR (light detection and ranging), etc. The sensing may include generating samples (for example, pixel, audio signals) that represent the signal that was transmitted, or otherwise reach the sensor. An SIU may be any arrangement of sensed information—may be of any size and/or format—for example an image, one or more image, an audio packet, a chunk of sensed information, and the like.

Any reference to a SIU should be applied, mutatis mutandis to a processed SIU. A processed SIU may be generated by processing an SIU, processing a previously processed SIU, and the like. The processing may include any operation—such as—filtering, noise reduction, SIU manipulation, padding, and the like.

Any reference to a cluster should be applied mutatis mutandis to a cluster structure. A concept structure may include one or more clusters. Each cluster may include signatures and related metadata.

Any reference to obtaining a content may include receiving the content, generating the content, participating in a processing of the content, processing only a part of the content and/or receiving only another part of the content. Examples of content include one or more signatures, an SIU and the like.

The obtaining of the content include object detection or may be executed without performing object detection.

The specification and/or drawings may refer to a processor. The processor may be a processing circuitry. The processing circuitry may be implemented as a central processing unit (CPU), and/or one or more other integrated circuits such as application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), full-custom integrated circuits, etc., or a combination of such integrated circuits.

Any combination of any steps of any method illustrated in the specification and/or drawings may be provided.

Any combination of any subject matter of any of claims may be provided.

Any combinations of systems, units, components, processors, sensors, illustrated in the specification and/or drawings may be provided.

Any reference to an object may be applicable to a pattern. Accordingly—any reference to object detection is applicable mutatis mutandis to a pattern detection.

A situation may be a singular location/combination of properties at a point in time. A scenario is a series of events that follow logically within a causal frame of reference. Any reference to a scenario should be applied mutatis mutandis to a situation.

The sensed information unit may be sensed by one or more sensors of one or more types. The one or more sensors may belong to the same device or system—or may belong to different devices of systems.

An erroneous signature is a signatures that once used may introduce an error related to object detection. According to an embodiment, the erroneous signature is an ambiguous signature that when used for the object detection, results in inconsistent detection of objects. According to an embodiment, the erroneous signature when used for the object detection, results in at least one of (i) a false negative detection or (ii) a false positive detection. Any reference to an ambiguous signature should be applied mutatis mutandis to any other erroneous signature. Any reference to a false positive signature should be applied mutatis mutandis to any other erroneous signature. Any reference to a false negative signature should be applied mutatis mutandis to any other erroneous signature.

Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative and that alternative embodiments may merge logic blocks or circuit elements or impose an alternate decomposition of functionality upon various logic blocks or circuit elements. Thus, it is to be understood that the systems depicted herein are merely exemplary, and that in fact many other system s may be implemented which achieve the same functionality.

It will be appreciated by people skilled in the art that the embodiments of the disclosure are not limited by what has been particularly shown and described hereinabove. Rather the scope of the embodiments of the disclosure is defined by the appended claims and equivalents thereof.

Further embodiments are listed below.

Embodiment 1. A method for automatically identifying faulty signatures for autonomous driving applications including: receiving, by a processing circuit, a signature; matching the signature to a group of first signatures that are untagged and are randomly obtained; identifying, based on the matching, first top matching signatures; matching the signature to a group of second signatures that are untagged and are correctly or erroneously indicative of a detection of reference elements; identifying, based on the matching of the signature to the second group of second signatures, second top matching signatures; determining an overlap between the first top matching signatures and the second top matching signatures; and determining whether the signature is faulty or faultless based on the overlap.

Embodiment 2. The method according to Embodiment 1, further including comparing the overlap to a threshold.

Embodiment 3. The method according to any of Embodiments 1-2, further including determining that the signature is faulty upon the overlap being below the threshold.

Embodiment 4. The method according to any of Embodiments 1-3, further including determining that the signature is faultless upon the overlap being above the threshold.

Embodiment 5. The method according to any of Embodiments 1-4, further including automatically determining a value of the threshold based on a dynamically set tradeoff between false positive detections and true positive detections.

Embodiment 6. The method according to any of Embodiments 1-5, further including automatically determining a value of the threshold based on a dynamically determined accuracy metric of a signature generator that generated the signature.

Embodiment 7. The method according to any of Embodiments 1-6, further including dynamically determining a number of the first signatures of the first top matching signatures.

Embodiment 8. The method according to any of Embodiments 1-7, further including dynamically adjusting a signature generator that generated the signature based on an outcome of the determining of whether the signature is faulty or faultless.

Embodiment 9. The method according to any of Embodiments 1-8, further including monitoring outcomes of multiple iterations of the receiving of the signature, the matching of the signature to the group of first signatures, identifying the first top matching signatures, the matching of the signature to the group of second signatures, identifying the second top matching signatures, determining the overlap, and the determining of whether the signature is faulty or faultless, and sending feedback, based on the monitoring, to a signature generator that generated the signature.

Embodiment 10. The method according to any of Embodiments 1-9, further including tagging the signature, based on an outcome of the determining of whether the signature is faulty of faultless.

Embodiment 11. A non-transitory computer readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations for automatically identifying faulty signatures for autonomous driving applications, including: receiving, by a processing circuit, a signature; matching the signature to a group of first signatures that are untagged and are randomly obtained; identifying, based on the matching, first top matching signatures; matching the signature to a group of second signatures that are untagged and are correctly or erroneously indicative of a detection of reference elements; identifying, based on the matching of the signature to the second group of second signatures, second top matching signatures; determining an overlap between the first top matching signatures and the second top matching signatures; and determining whether the signature is faulty or faultless based on the overlap.

Embodiment 12. The non-transitory computer readable medium according to Embodiment 11, further including comparing the overlap to a threshold.

Embodiment 13. The non-transitory computer readable medium according to any of Embodiments 11-12, further including determining that the signature is faulty upon the overlap being below the threshold.

Embodiment 14. The non-transitory computer readable medium according to any of Embodiments 11-13, further including determining that the signature is faultless when the overlap is above the threshold.

Embodiment 15. The non-transitory computer readable medium according to any of Embodiments 11-14, further including automatically determining a value of the threshold based on a dynamically set tradeoff between false positive detections and true positive detections.

Embodiment 16. The non-transitory computer readable medium according to any of Embodiments 11-15, further including automatically determining a value of the threshold based on a dynamically determined accuracy metric of a signature generator that generated the signature.

Embodiment 17. The non-transitory computer readable medium according to any of Embodiments 11-16, further including dynamically determining a number of the first signatures of the first top matching signatures.

Embodiment 18. The non-transitory computer readable medium according to any of Embodiments 11-17, further including dynamically adjusting a signature generator that generated the signature based on an outcome of the determining of whether the signature is faulty or faultless.

Embodiment 19. The non-transitory computer readable medium according to any of Embodiments 11-18, further comprising monitoring outcomes of multiple iterations of the receiving of the signature, the matching of the signature to the group of first signatures, identifying the first top matching signatures, the matching of the signature to the group of second signatures, identifying the second top matching signatures, determining the overlap, and the determining of whether the signature is faulty or faultless, and sending feedback, based on the monitoring, to a signature generator that generated the signature.

Embodiment 20. The non-transitory computer readable medium according to any of Embodiments 11-19, further comprising tagging the signature, based on an outcome of the determining of whether the signature is faulty of faultless.