Patent Description:
An increasing number of modern vehicles have advanced driver-assistance systems, ADAS, to increase vehicle safety and more generally road safety. ADAS - which for instance may be represented by adaptive cruise control, ACC, collision avoidance system, forward collision warning, etc. - are electronic systems that may aid a vehicle driver while driving. To function as intended, ADAS may rely on inputs from multiple data sources, such as e.g. automotive imaging, LIDAR, radar, image processing, computer vision, and/or in-car networking.

Moreover, in a not too distant future, autonomous or automated driving systems, ADSs, will to greater extent find their way into modern vehicles. An ADS is a complex combination of various components that can be defined as systems where perception, decision making, and operation of the vehicle are performed by electronics and machinery instead of a human driver, and as introduction of automation into road traffic. This includes handling of the vehicle, destination, as well as awareness of surroundings. While the automated system has control over the vehicle, it allows the human operator to leave all responsibilities to the system. An ADS commonly combines a variety of sensors to perceive the vehicle's surroundings, such as e.g. radar, LIDAR, sonar, camera, navigation system e.g. GPS, odometer and/or inertial measurement units, upon which advanced control systems may interpret sensory information to identify appropriate navigation paths, as well as obstacles and/or relevant signage.

Eventually, essentially fully autonomous vehicles capable of sensing their environment and moving safely along e.g. public roads with little or no human input, may see the light of day.

ADAS and/or autonomous driving features as described above are, however, bound to be required to function with a high integrity to provide sufficiently low risk for the vehicle user(s) as well as other traffic participants. Ensuring that the risk is sufficiently low may require intractable amounts of data for statistical proofs, and would - according to an example - take e.g. approximately a hundred vehicles to drive continuously for five centuries, to acquire. There are several approaches to minimize the risk of the ADAS and/or autonomous driving features before they are launched onto public roads. However, on top of this, it is generally believed that the ADAS and/or autonomous driving features should be monitored once in the field, in order to ensure that they adhere to required safety levels. How such field monitoring should be done and further be accomplished with only the production sensor setup - e.g. the sensors of a launched road traffic vehicle - is however unclear. Further, as the complete sensor setups of ADAS and/or autonomous driving features may generate huge amounts of data, it may be challenging - if not impossible - to send all of that data to a cloud for offline processing and analysis.

A paper from the <NPL>, discloses Assessment of positioning accuracy of vehicle trajectories for different road applications.

It is therefore an object of embodiments herein to provide an approach for in an improved and/or alternative manner monitor performance an requirement fulfilment of performance requirements of a vehicle ADAS or autonomous driving feature.

The object above may be achieved by the subject-matter disclosed herein. Embodiments are set forth in the appended claims, in the following description and in the drawings.

The disclosed subject-matter relates to a method in accordance with claim <NUM>.

The disclosed subject-matter further relates to a field monitor system in accordance with claim <NUM>.

Furthermore, the disclosed subject-matter relates to a vehicle supporting an ADAS or autonomous driving feature, and comprises a field monitor system as described herein.

Moreover, the disclosed subject-matter relates to a computer program product comprising a computer program containing computer program code means arranged to cause a computer or a processor to execute the steps of a the field monitor system described herein, stored on a computer-readable medium or a carrier wave.

The disclosed subject-matter further relates to a non-volatile computer readable storage medium having stored thereon said computer program product.

Thereby, there is introduced an approach according to which one or more ADAS and/or autonomous driving features may be monitored - e.g. in the field - to evaluate whether they are failing and/or are in risk of failing, and take action when such is the case. That is, since there is stored, during a time period ranging from a first time point to a second time point, a range of perception data on which the ADAS or autonomous driving feature is pertinent derived from a perception system adapted to estimate surroundings of the vehicle, there is saved - e.g. in an optional data buffer - obtained data indicative of vehicle surroundings perceptions, such as e.g. detected objects or objects' states and/or vehicle localization, and/or statistical and physical model predictions of future states, derived continuously and/or intermittently from a first time point to a second time point. Thus, for any time period, perception data such as e.g. sensor readings and run-time estimates are saved to reproduce the majority of the temporal dependencies in the ADAS or autonomous driving feature. Moreover, that is, since there is determined based on post-processing a portion of the range of perception data ranging back from the second time point to an intermediate time point between the first and second time points, ground truth intermediate data at the intermediate time point, which ground truth intermediate data reflects estimates of vehicle surroundings at the intermediate time point conditional on the portion of the range of perception data, data comprised in the portion of the range of perception data is post-processed - e.g. run through a backwards filter - that aligns the e.g. objects' current and predicted future states with what happened in the future - i.e. from the intermediate time point to the second time point - as seen from the intermediate time point. By that, ground truth intermediate data - which may be referred to as production ground truth - of the intermediate time point is achieved. Accordingly, the ground truth intermediate data provides improved estimates of the e.g. states of the objects surrounding the vehicle - or the ADAS or autonomous driving feature - at the intermediate time point, conditional on the data portion. Furthermore, that is, since there is determined based on analysis of the ground truth intermediate data in view of intermediate perception data out of the range of perception data corresponding to the intermediate time point, an accuracy deviation measure indicating deviation between the ground truth intermediate data and the intermediate perception data, there is evaluated a discrepancy between the ground truth intermediate data and the intermediate perception data corresponding to the intermediate time point, such as e.g. discrepancies associated with sensor readings and run-time estimates of e.g. localization and/or objects' states surrounding the vehicle or the ADAS or autonomous driving feature, and/or statistical and physical model predictions of future states. Accordingly, with the increased knowledge of vehicle surroundings as time passes from the intermediate time point to the second time point by analyzing data in reverse temporal direction, it may be evaluated how well the ADAS or autonomous driving feature, the surrounding detecting sensors thereof and/or perception system model predictions of future vehicle surroundings performed at said intermediate time point. Moreover, since there is communicated acknowledgement data when the accuracy deviation measure exceeds a performance threshold, the outcome of the evaluation, i.e. the accuracy deviation measure, is considered in view of at least a first threshold, and should said threshold be violated, then an acknowledgment indicative thereof is transmitted from the field monitor system. Accordingly, the field monitor system brings attention to when intermediate perception data deviates from the ground truth intermediate data to an extent greater than the level(s) defined by the performance threshold. Thus, there may be achieved an evaluation in view of failure-risk, failure and/or failure rate - and/or intervention - of the ADAS or autonomous driving feature, the surrounding detecting sensors thereof and/or perception system model predictions of future vehicle surroundings.

For that reason, an approach is provided for in an improved and/or alternative manner monitor performance and/or requirement fulfilment of performance requirements of a vehicle ADAS or autonomous driving feature.

The technical features and corresponding advantages of the above mentioned method will be discussed in further detail in the following.

Non-limiting embodiments of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the disclosure are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Dashed lines of some boxes in the figures indicate that these units or actions are optional and not mandatory.

In the following, according to embodiments herein which relate to performance monitoring and evaluation of an ADAS or autonomous driving feature of a vehicle, there will be disclosed an approach according to which one or more ADAS and/or autonomous driving features may be monitored - e.g. in the field - to evaluate whether they are failing and/or are in risk of failing, and take action when such is the case.

Referring now to the figures and <FIG> in particular, there is depicted a schematic view of an exemplifying field monitor system <NUM> of a vehicle <NUM> according to embodiments of the disclosure. The field monitor system <NUM> is adapted for performance monitoring of an ADAS or autonomous driving feature <NUM> of the vehicle <NUM>. The exemplifying vehicle <NUM> is adapted to support one or more ADAS or autonomous driving features <NUM>, and here, it comprises the field monitor system <NUM>.

The exemplifying vehicle <NUM> may refer to any arbitrary manned or unmanned vehicle, for instance an engine-propelled or electrically-powered vehicle such as a car, truck, lorry, van, bus and/or tractor. Moreover, the term "vehicle" may refer to "launched vehicle", "road-traffic vehicle" and/or "production vehicle", and further to "autonomous and/or at least partly autonomous vehicle", "driverless and/or at least partly driverless vehicle", and/or "self-driving and/or at least partly self-driving vehicle". The exemplifying ADAS or autonomous driving feature may refer to any arbitrary ADAS, ADS or autonomous driving feature, e.g. known in the art and/or yet to be developed.

The phrase "field monitor system" may refer to "production vehicle monitor system", "failure monitor system", "performance monitoring system", "performance validation system, "performance evaluation system" and/or "perception assessment system", whereas "of" a vehicle may refer to "comprised in" a vehicle and/or "on-board" a vehicle. "For" performance monitoring, on the other hand, may refer to "adapted for" performance monitoring, whereas the phrase "for performance monitoring and evaluation" of an ADAS or autonomous driving feature may refer to "for failure and/or failure-risk monitoring and evaluation", "for performance assessment", "for performance validation", "for perception evaluation", "for performance evaluation" and/or merely "for performance monitoring" of an ADAS or autonomous driving feature. According to an example, the phrase "for performance monitoring and evaluation" of an ADAS or autonomous driving feature may refer to "for performance monitoring and evaluation, and potential further evaluation and/or intervention," of an ADAS or autonomous driving feature. The phrase "performance monitoring and evaluation of an ADAS or autonomous driving feature of said vehicle" may refer to "performance monitoring and evaluation associated with an ADAS or autonomous driving feature of said vehicle", and/or "performance monitoring and evaluation of perception estimates on which an ADAS or autonomous feature of said vehicle is pertinent". Moreover, "performance monitoring and evaluation of an ADAS or autonomous driving feature" may refer to "performance monitoring and evaluation of an ADAS or autonomous driving feature and/or a subcomponent or constituent thereof". The phrase "an ADAS or autonomous driving feature of said vehicle", on the other hand, may refer to "an ADAS or automated driving system, ADS, of said vehicle", whereas driving "feature" may refer to driving "system" and/or driving "function". "An" ADAS or automated driving feature may further refer to "at least a first" ADAS or automated driving feature.

The vehicle <NUM> may comprise, be provided with and/or have on-board a perception system <NUM> adapted to estimate surroundings of the vehicle <NUM>, and subsequently adapted to make model predictions of future states of the surroundings. The perception system <NUM> may refer to any commonly known system and/or functionality, e.g. comprised in one or more electronic control modules and/or nodes of the vehicle, adapted and/or configured to interpret sensory information - relevant for driving of the vehicle <NUM> - to identify e.g. obstacles, vehicle lanes, relevant signage, appropriate navigation paths etc. The exemplifying perception system <NUM> may thus be adapted to rely on and obtain inputs from multiple data sources, such as automotive imaging, image processing, computer vision, and/or in-car networking, etc., in combination with sensory information. Such exemplifying sensory information may for instance be derived from one or more optional surrounding detecting sensors <NUM> comprised in and/or provided on-board the vehicle <NUM>. The surrounding detecting sensors <NUM> may be represented by any arbitrary sensors adapted to sense and/or perceive the vehicle's <NUM> surroundings and/or whereabouts, and may e.g. refer to one or a combination of one or more of radar, LIDAR, sonar, camera, navigation system e.g. GPS, odometer and/or inertial measurement units.

As further depicted in <FIG>, the field monitor system <NUM> is - by means of a perception data storing unit <NUM> (shown in <FIG>) - adapted and/or configured for storing, during a time period T ranging from a first time point t<NUM> to a second time point t<NUM>, a range of perception data <NUM> on which the ADAS or autonomous driving feature <NUM> is pertinent, derived from a perception system <NUM> adapted to estimate surroundings of the vehicle <NUM>. Thereby, there is saved - e.g. in an exemplifying optional data buffer <NUM> - obtained data indicative of vehicle surroundings perceptions, such as e.g. detected objects or objects' states and/or vehicle localization, and/or statistical and physical model predictions of future states, derived continuously and/or intermittently from a first time point t<NUM> to a second time point t<NUM>. Thus, for any time period T, perception data such as e.g. sensor readings and run-time estimates are saved to reproduce the majority of the temporal dependencies in the ADAS or autonomous driving feature <NUM>.

The range of perception data <NUM> may be derived from the perception system <NUM> in any arbitrary manner - e.g. known - such as digitally, electronically, wirelessly and/or by wire. Moreover, the range of perception data <NUM> may refer to any range of data of relevance for the ADAS or autonomous driving feature <NUM>, which data further may be indicative of perceptions of surroundings of the vehicle <NUM>, e.g. comprising - and/or being derived from - sensory data from the one or more surroundings detecting sensors <NUM>. Perception data at the first time point t<NUM> may - or may not - differ from perception data at the second time point t<NUM>. Storing of the range of perception data <NUM> may be a continuous ongoing process, and further, perception data of said range <NUM> may be stored continuously and/or intermittently. The time period T - and correspondingly the length of the optional buffer <NUM> - may be of any arbitrary size deemed feasible, e.g. in consideration of data capacity restraints and/or characteristics of the ADAS or autonomous feature <NUM>, and may for instance range from under a second up to several minutes, or more preferred, from a few seconds up to less than a minute.

The term "storing" a range of perception data may refer to "storing in one or more memories", "storing on-board said vehicle", "storing in one or more memories on-board said vehicle", and/or "storing digitally and/or electronically" a range of perception data, and further to "collecting" and/or "obtaining" a range of perception data. The term "range" of perception data, on the other hand, may refer to "amount", "series", "continuous and/or intermittent flow" and/or "collection" of perception data, whereas "perception data" may refer to "continuously and/or intermittently collected perception data". Furthermore, the term "perception" data may refer to "surroundings assessment" data, "sensory" data and/or "temporal dependencies" data, whereas perception "data" may refer to perception "information" and/or "estimates". The phrase "perception data on which said ADAS or autonomous driving feature is pertinent" may refer to "perception data relevant for said ADAS or autonomous driving feature" and/or "perception data by which said ADAS or autonomous driving feature is affected". The term "derived" from a perception system, on the other hand, may refer to "obtained" from a perception system and/or "based on output data" from a perception system, whereas perception system adapted to "estimate surroundings of said vehicle" may refer to perception system adapted and/or configured to "estimate the surroundings of said vehicle", "estimate at least a portion of surroundings of said vehicle", "determine surroundings of said vehicle", "interpret sensory information relevant for driving of said vehicle", and/or "estimate surroundings of said vehicle and make model predictions of future states of the surroundings of said vehicle". "Perception" system may refer to "vehicle surroundings assessment and/or evaluation" system, whereas perception "system" may refer to perception "function and/or functionality". The phrase storing "during" a time period may refer to storing "for" a time period, whereas time period "ranging" from a first time point to a second time point may refer to time period "extending" and/or "taking place" from a first time point to a second time point. "Time period" may refer to "predeterminable time period" and/or "predetermined time period". "Time point", on the other hand, may refer to "point in time", whereas "from a first time point to a second time point" may refer to "from a first time point to a subsequent second time point".

As exemplified in <FIG>, the field monitor system <NUM> is - by means of a ground truth determining unit <NUM> (shown in <FIG>) - adapted and/or configured for determining based on post-processing a portion <NUM> of the range of perception data <NUM> ranging back from the second time point t<NUM> to an intermediate time point tintermediate between the first and second time points t<NUM>, t<NUM>, ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ at the intermediate time point tintermediate, which ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ reflects estimates of vehicle surroundings at the intermediate time point tintermediate conditional on the portion <NUM> of the range of perception data <NUM>. Thereby, data comprised in the portion <NUM> of the range of perception data <NUM> is post-processed - e.g. run through a backwards filter - that aligns the e.g. objects current and predicted future states with what happened in the future - i.e. from the intermediate time point tintermediate to the second time point t<NUM> - as seen from the intermediate time point tintermediate. By that, ground truth intermediate data 6i, 6i', 6i", 6i"', 6i"" - which may be referred to as production ground truth <NUM>, <NUM>', <NUM>", <NUM>‴, 6ʺʺ - of the intermediate time point tintermediate is achieved. Accordingly, the ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ provides improved estimates of the e.g. states of the objects surrounding the vehicle - or the ADAS or autonomous driving feature - at the intermediate time point tintermediate, conditional on the data portion <NUM>.

"Determining" ground truth intermediate data may refer to "calculating" and/or "deriving" ground truth intermediate data, and according to an example further to "determining with support from one or more algorithms" ground truth intermediate data. Determining "based on" post-processing, on the other hand, may refer to determining "by" post-processing, whereas based on "post-processing" may refer to based on "backward-processing" and/or based on "running through one or more backwards filters". The phrase "a portion of said range" of perception data may refer to "at least a portion of said range" of perception data and/or "a selection of said range" of perception data, whereas perception data "ranging back from" may refer to perception data "ranging from". "Ground truth intermediate" data may refer to "processed and/or post-processed intermediate" data, whereas ground truth intermediate "data" may refer to ground truth intermediate "information", "perceptions" and/or "estimates". Ground truth intermediate data "at" the intermediate time point, on the other hand, may refer to ground truth intermediate data "corresponding to" and/or "reflected at" the intermediate time point, whereas ground truth intermediate data "reflecting" estimates may refer to ground truth intermediate data "indicative of" estimates. The intermediate time point tintermediate may - or may not - be equal to or essentially equal to the first time point t<NUM>. "Estimates" of vehicle surroundings may refer to "calculations" of vehicle surroundings, whereas estimates of "vehicle surrounding" may refer to estimates of "sensory data". The phrase estimates of vehicle surroundings "at" the intermediate time point, on the other hand, may refer to estimates of vehicle surroundings "corresponding to" and/or "reflected at" the intermediate time point, whereas "conditional on" the selection of the range of perception data may refer to "based on data comprised in" the portion of the range of perception data and/or "based on post-processing of data comprised in" the portion of the range of perception data.

The field monitor system <NUM> is - by means of a deviation determining unit <NUM> (shown in <FIG>) - adapted and/or configured for determining based on analysis of the ground truth intermediate data 6i, i, 6i", 6i"', 6iʺʺ in view of intermediate perception data 4i, 4i', 4i", 4i"', 4iʺʺ out of the range of perception data <NUM> corresponding to the intermediate time point tintermediate, an accuracy deviation measure indicating deviation between the ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ and the intermediate perception data 4i, 4i', 4i", 4i"', 4iʺʺ. Thereby, there is evaluated a discrepancy between the ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺand the intermediate perception data 4i, 4i', 4i", 4i"', 4iʺʺ corresponding to the intermediate time point tintermediate, such as e.g. discrepancies associated with sensor readings and run-time estimates of e.g. localization and/or objects' states surrounding the vehicle <NUM> or the ADAS or autonomous driving feature <NUM>, and/or statistical and physical model predictions of future states. Accordingly, with the increased knowledge of vehicle surroundings 6i, 6i', 6i", 6i"', 6iʺʺ as time passes from the intermediate time point tintermediate to the second time point t<NUM> by analyzing data in reverse temporal direction, it may be evaluated how well the ADAS or autonomous driving feature <NUM>, the surrounding detecting sensors <NUM> thereof and/or perception system <NUM> model predictions of future vehicle surroundings performed at said intermediate time point tintermediate.

The accuracy deviation may for instance indicate a failure and/or a risk of failure, and may be monitored e.g. in view of quality aspects and/or safety critical aspects. According to an example, in view of safety and/or a safety requirement e.g. on perception performance, the focus may for instance be on near-misses and/or failures to adhere to e.g. a required safety level, whereas in in view of quality, the focus may be for instance on actual failures. It may hence be noted that failures may relate not only to e.g. hardware failures, but additionally or alternatively to failures to meet e.g. a requirement on performance or performance requirement related to safety.

"Determining" an accuracy deviation measure may refer to "calculating" an accuracy deviation measure, whereas determining "based on analysis of said ground truth intermediate data in view of" intermediate data may refer to determining "based on a difference of said ground truth intermediate data as compared to" intermediate data. "Accuracy deviation measure", on the other hand, may refer to merely "accuracy deviation", and further to "ground truth deviation measure", whereas "indicating deviation" may refer to "indicative of deviation" and/or "reflecting deviation". The term "deviation" may refer to "discrepancy". The phrase "intermediate perception data out of said range of perception data corresponding to said intermediate time point", on the other hand, may refer to "intermediate perception data of said intermediate time point".

The increased knowledge of vehicle surroundings as time passes is exemplified in <FIG>. In <FIG>, which illustrates a schematic view of exemplifying ground truth data <NUM> in view of perception data from the range of perception data <NUM> according to embodiments of the disclosure, there is illustrated first time point perception data <NUM> of the first time point t<NUM>, intermediate perception data 4i of the intermediate time point tintermediate, and second time point perception data <NUM> of the second time point t<NUM>, of an object <NUM> - here another vehicle - which is derived from the perception system <NUM>, here with support from the optional surrounding detecting sensors <NUM>. In the illustrated example, the estimate 4i of the surrounding object <NUM> at the intermediate time point tintermediate, i.e. the intermediate perception data 4i, deviates from the determined ground truth intermediate data 6i, in that the object <NUM> according to the intermediate perception data 4i is placed further to the left than according to the ground truth intermediate data 6i.

It is to be understood that there may exist additional perception data and/or ground truth data corresponding to other time points. Further exemplified is for instance how the estimate <NUM> of the surrounding object <NUM> at the first time point t<NUM> - here to greater extent - may deviate from ground truth data <NUM> pertinent the first time point t<NUM>, and similarly, how the estimate <NUM> of the surrounding object <NUM> at the second time point t<NUM> - here to lesser extent - may deviate from ground truth data <NUM> pertinent the second time point t<NUM>.

In <FIG>, which illustrates a schematic view of exemplifying ground truth data <NUM>' in view of perception data from the range of perception data <NUM> according to further embodiments of the disclosure, there is depicted intermediate perception data 4i' of the intermediate time point tintermediate and second time point perception data <NUM>' of the second time point t<NUM> - here represented by free space estimations - which is derived from the perception system <NUM>, here with support from the optional surrounding detecting sensors <NUM>. In the illustrated example, the estimate 4i' of free space at the intermediate time point tintermediate, i.e. the intermediate perception data 4i', deviates from the determined ground truth intermediate data 6i', in that by the second time point t<NUM>, an object <NUM>' - here vehicle - was detected, which was falsely not detected at the intermediate time point tintermediate. This phenomenon may be referred to as a false negative.

In <FIG>, which illustrates a schematic view of exemplifying ground truth data <NUM>" in view of perception data from the range of perception data <NUM> according to yet further embodiments of the disclosure, there is illustrated intermediate perception data 4i" of the intermediate time point tintermediate and second time point perception data <NUM>" of the second time point t<NUM> - here represented by free space estimations - which is derived from the perception system <NUM>, here with support from the optional surrounding detecting sensors <NUM>. In the illustrated example, the estimate 4i" of free space at the intermediate time point tintermediate, i.e. the intermediate perception data 4i", deviates from the determined ground truth intermediate data 6i", in that by the second time point t<NUM>, it could be concluded that there was no presence of an object <NUM>" - here vehicle - which was falsely detected at the intermediate time tintermediate. This phenomenon may be referred to as a false positive.

In <FIG>, which illustrates a schematic view of exemplifying ground truth data <NUM>‴ in view of perception data from the range of perception data <NUM> according to still further embodiments of the disclosure, there is illustrated intermediate perception data 4i‴ of the intermediate time point tintermediate and second time point perception data <NUM>‴ of the second time point t<NUM> - here represented by localization estimations and/or lane curve estimations - which is derived from the perception system <NUM>, optionally with support from the surrounding detecting sensors <NUM>. In the illustrated example, the estimate 4i‴ of the localization of the vehicle <NUM> and/or a lane curve at the intermediate time point tintermediate, i.e. the intermediate perception data 4i"', deviates from the determined ground truth intermediate data 6i"', in that the vehicle <NUM> according to the intermediate perception data 4i‴ is positioned further away from the lane curve than according to the ground truth intermediate data 6i‴. This phenomenon may be referred to as a faulty localization.

According to an example, the perception data may potentially be related to and/or be represented by model predictions of future surroundings that the perception system <NUM> does based on its current understanding of the surroundings. Thus, in <FIG>, which illustrates a schematic view of exemplifying ground truth data 6ʺʺ in view of perception data from the range of perception data <NUM> according to even further embodiments of the disclosure, there is illustrated intermediate perception data 4iʺʺ of the intermediate time point tintermediate and second time point perception data 42ʺʺ of the second time point t<NUM>, derived from the perception system <NUM>, optionally with support from the surrounding detecting sensors <NUM>. Here, the intermediate perception data 4iʺʺ is represented by at least a first prediction model predicting an area 4iʺʺ where an object 7ʺʺ based on its type and current velocity captured at the intermediate time point tintermediate, is predicted to reside at the second time point t<NUM>, whereas the second time point perception data 42ʺʺ is represented by at least a second prediction model - potentially differing from the at least first model - comprising said object 7ʺʺ captured at the second time point t<NUM>. In the illustrated example, the prediction model estimate 4iʺʺ at the intermediate time point tintermediate of the area where the object 7ʺʺ is predicted to reside at the second time point t<NUM>, i.e. the intermediate perception data 4i"", deviates from the determined ground truth intermediate data 6iʺʺ, in that the object 7ʺʺ according to the ground truth intermediate data 6iʺʺ not is within the area 4i"". Here, it may be concluded that there is a discrepancy in some of the measurements and/or predictions models, and hence reason to suspect that the at least first prediction model at least to some extent may have been incorrect.

The field monitor system <NUM> is - by means of an acknowledgement communicating unit <NUM> (shown in <FIG>) - adapted and/or configured for communicating acknowledgement data <NUM> when the accuracy deviation measure exceeds a performance threshold. Thereby, the outcome of the evaluation, i.e. the accuracy deviation measure, is considered in view of at least a first threshold, and should said threshold be violated, then an acknowledgment <NUM> indicative thereof is transmitted from the field monitor system <NUM>. Accordingly, the field monitor system <NUM> brings attention to when intermediate perception data 4i, 4i', 4i", 4i"', 4iʺʺ deviates from the ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ to an extent greater than the level(s) defined by the performance threshold. Thus, there may be achieved an evaluation in view of failure-risk, failure and/or failure rate - and/or intervention - of the ADAS or autonomous driving feature <NUM>, the surrounding detecting sensors <NUM> thereof and/or perception system <NUM> model predictions of future vehicle surroundings.

The performance threshold may be set to any arbitrary limit and/or deviation maximum deemed relevant, e.g. in view of characteristics of - and/or requirements related to - the ADAS or autonomous driving feature <NUM>, the surrounding detecting sensors <NUM> thereof and/or perception system <NUM> model predictions of future vehicle surroundings. The performance threshold may further be variable.

"Communicating" acknowledgement data may refer to "communicating wirelessly and/or by wire" acknowledgement data and/or "communicating in due time and/or when deemed feasible and/or safe" acknowledgement data, whereas "acknowledgement data" may refer to "one or more acknowledgement signals" and/or "an acknowledgement message", and further to "acknowledgement data indicative of, reflecting and/or comprising at least said accuracy deviation measure". The phrase "when said accuracy deviation measure exceeds", on the other hand, may refer to "should said accuracy deviation measure exceed" and/or "if said accuracy deviation measure exceed", whereas "performance" threshold may refer to "adherence" threshold. Moreover, "performance threshold" may refer to "predeterminable and/or predetermined performance threshold", and further to "first performance threshold".

The field monitor system <NUM> may - by means of the acknowledgement communicating unit <NUM> (shown in <FIG>) - optionally be adapted and/or configured for transmitting the acknowledgement data <NUM> wirelessly to a remote entity <NUM>, which acknowledgement data <NUM> comprises the range of perception data <NUM>. Thereby, data related to the intermediate perception data 4i, 4i', 4i", 4i"', 4iʺʺ and/or events at the intermediate time point t<NUM> may be transmitted to the entity <NUM> for offline processing and/or analysis. The timing of transmittal may be arbitrarily selected as deemed feasible, such as e.g. promptly or at a point in time deemed suitable. The latter case may e.g. refer to awaiting a high bandwidth connection to the remote entity <NUM>, such as e.g. a Wifi connection.

The entity <NUM> may refer to any off-board data storage entity - e.g. known - adapted for and/or configured for off-board and/or offline processing and/or analysis, such as e.g. a cloud and/or automotive cloud, cloud network adapted for cloud-based storage, back-end system, and/or one or more servers. "Remote" entity may refer to "off-board" entity and/or "offline" entity, whereas "comprising said range of perception data" may refer to "comprising essentially said range of perception data".

Additionally or alternatively, optionally, the field monitor system <NUM> may - by means of the acknowledgement communicating unit <NUM> (shown in <FIG>) - be adapted and/or configured for transmitting the acknowledgement data <NUM> to an on-board feature control system <NUM> adapted to control the ADAS or autonomous driving feature <NUM>, which acknowledgement data <NUM> comprises an indication to disable the ADAS or autonomous driving feature <NUM>. Thereby, the ADAS or autonomous driving feature <NUM> - and/or one or more surrounding detecting sensors <NUM> associated therewith - or the functionality thereof, may be revoked to, at least temporarily, inhibit further activation(s).

The feature control system <NUM> may refer to any commonly known system and/or functionality, e.g. comprised in one or more electronic control modules and/or nodes of the vehicle, adapted and/or configured to at least in part control the the ADAS or autonomous driving feature <NUM>. "Feature control system" may refer to "feature disablement system" and/or "ADAS or ADS control system", whereas feature control system adapted to "control" may refer to feature control system adapted to "at least partly control". "Comprising an indication" to disable, on the other hand, may refer to "prompting" to disable and/or "comprising instructions prompting" to disable, whereas "indication to disable" may refer to "indication to in due time disable" and/or "indication to disable when deemed feasible and/or safe". Moreover, "disable" the ADAS or autonomous driving feature may refer to "inhibit" and/or "restrict functionality of" the ADAS or autonomous driving feature.

The field monitor system <NUM> may - by means of the acknowledgement communicating unit <NUM> (shown in <FIG>) - optionally be adapted and/or configured for communicating, when the accuracy deviation measure exceeds a further performance threshold, further acknowledgement data <NUM>, which performance threshold deviates from the further performance threshold. Thereby, different acknowledgment data <NUM>, <NUM> may be communicated depending on which threshold is violated. Accordingly, exceeding the performance threshold - upon which the acknowledgement data <NUM> is communicated - may initiate a first action, whereas exceeding the further performance threshold - upon which the further acknowledgement data <NUM> is communicated - may initiate a differing second action. That is, here, the outcome of the evaluation - i.e. the accuracy deviation measure - is considered in view of an additional second threshold, and should said second threshold be violated, then a further acknowledgment indicative thereof is transmitted from the field monitor system. Accordingly, the field monitor system may additionally bring attention to when intermediate perception data 4i, 4i', 4i", 4i"', 4iʺʺ deviates from the ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ to an extent greater than the level(s) defined by the further performance threshold. Thus, there may be achieved a further evaluation in view of failure-risk, failure and/or failure rate - and/or intervention - of the ADAS or autonomous driving feature <NUM>, the surrounding detecting sensors <NUM> thereof and/or perception system <NUM> model predictions of future vehicle surroundings.

The further performance threshold may be set to any arbitrary limit and/or deviation maximum deemed relevant, e.g. in view of characteristics of - and/or requirements related to - the ADAS or autonomous driving feature <NUM>, the surrounding detecting sensors <NUM> thereof and/or perception system <NUM> model predictions of future vehicle surroundings. Moreover, the further performance threshold may be variable.

"Communicating" further acknowledgement data may refer to "communicating wirelessly and/or by wire" further acknowledgement data and/or "communicating in due time and/or when deemed feasible and/or safe" further acknowledgement data, whereas "further acknowledgement data" may refer to "one or more further acknowledgement signals" and/or "a further acknowledgement message". Moreover, "further acknowledgement data" may refer to "further acknowledgement data indicative of, reflecting and/or comprising at least said accuracy deviation measure". The phrase "when said accuracy deviation measure exceeds a further performance threshold", on the other hand, may refer to "should said accuracy deviation measure exceed a further performance threshold" and/or "if said accuracy deviation measure exceed", whereas "further performance" threshold may refer to "further adherence" threshold. Moreover, "further performance threshold" may refer to "predeterminable and/or predetermined further performance threshold", and further to "second performance threshold".

The field monitor system <NUM> may - by means of the acknowledgement communicating unit <NUM> (shown in <FIG>) - optionally be adapted and/or configured for transmitting the further acknowledgement data <NUM> wirelessly to the remote entity <NUM>, which further acknowledgement data <NUM> comprises the range of perception data <NUM>. Thereby, data related to the intermediate perception data 4i, 4i', 4i", 4i"', 4iʺʺ and/or events at the intermediate time point t<NUM> may be transmitted to the entity <NUM> for offline processing and/or analysis, when the further performance threshold is violated. The timing of transmittal may be arbitrarily selected as deemed feasible, such as e.g. promptly or at a point in time deemed suitable. The latter case may e.g. refer to awaiting a high bandwidth connection to the remote entity <NUM>, such as e.g. a Wifi connection.

Alternatively, optionally, the field monitor system <NUM> may - by means of the acknowledgement communicating unit <NUM> - optionally be adapted and/or configured for transmitting the further acknowledgement data <NUM> to the feature control system <NUM>, which further acknowledgement data <NUM> comprises an indication to disable the ADAS or autonomous driving feature <NUM>. Thereby, the ADAS or autonomous driving feature <NUM> - and/or one or more surrounding detecting sensors <NUM> associated therewith - may be revoked to, at least temporarily, inhibit further activation(s), when the further performance threshold is violated.

Further, as illustrated in <FIG>, the range of perception data <NUM> comprises perception data of a surrounding object <NUM>, <NUM>', <NUM>", wherein the accuracy deviation measure reflects a measure in view of the object <NUM>, <NUM>', <NUM>". Thereby, production ground truth <NUM>, <NUM>', <NUM>" may be utilized for performance and/or failure-risk or failure assessment of one or more of the one or more surrounding detecting sensors <NUM> associated with the ADAS or autonomous driving feature <NUM>, used to detect said object <NUM>, <NUM>', <NUM>". That is, if for instance the vehicle <NUM> is approaching an object <NUM>, <NUM>', <NUM>", the sensor estimates may be limited due to the range of the sensors <NUM>. As the vehicle comes nearer the object <NUM>, <NUM>', <NUM>", however, the sensors <NUM> may operate in more favorable ranges and give a higher confidence in the output. The production ground truth <NUM>, <NUM>', <NUM>"may accordingly be used to assess the accuracy of early outputs 4i, 4i', 4i" of the sensors <NUM> when the object <NUM>, <NUM>', <NUM>" may have been at a further distance from the vehicle <NUM>. In a similar manner, should the object <NUM>, <NUM>', <NUM>" for a period of time remain at an essentially constant distance, the increased number of measurements may, when post-processed, increase the confidence in the output <NUM>, <NUM>', <NUM>".

The sensor assessment described above may for instance be done on classification of the object <NUM>, <NUM>', <NUM>". Additionally or alternatively, sensor assessment may be done on state estimating such as e.g. velocity and/or position, and/or object attributes such as e.g. size and/or colour. The sensor assessment may then yield a result, such as an accuracy deviation measure, indicating whether the sensor(s) <NUM> performs as promised or not. In the latter case, which may reflect a situation where the difference is relatively small but the performance threshold nonetheless is violated, the acknowledgement data <NUM> comprising the range of perception data <NUM> may for instance be transmitted to the remote entity <NUM> for offline processing and/or analysis. Additionally, should also the optional further threshold be violated, which may reflect a situation where the difference is relatively large e.g. indicating malfunctioning sensor(s) <NUM>, the further acknowledgement data <NUM> - comprising an indication to disable the ADAS or autonomous driving feature, or one or more of the sensor(s) <NUM> associated therewith - may for instance be transmitted to the on-board feature control system <NUM>.

According to an example, the phrase "said accuracy deviation measure reflecting a measure in view of said object" may refer to "said accuracy deviation measure reflecting a measure in view of said object, whereby performance and/or failure or failure-risk of at least one of said one or more surrounding detecting sensors is evaluated".

Moreover, optionally, the accuracy deviation measure may reflect a measure in view of estimated safety assessment on which the ADAS or autonomous driving feature <NUM> is pertinent. That is, there is a number of different safety or threat measures that may be used as a proxy to assess safety on an ADAS or autonomous driving feature <NUM>. One example of such a safety or threat measure is the Break Threat Number, BTN, defined as the required braking to avoid collision divided by the available braking capacity. A BTN equal to one means collision, and the closer to zero, the further from a critical situation the ADAS or autonomous driving feature <NUM>. Additional measures may be time to collision, being measured as the current states of the ADAS or autonomous driving feature <NUM>, and a target vehicle, and assuming constant velocities of the two. It is known to use extreme value theory to extrapolate data from a field test to validate the potential crash rate. If such a method is used pre-deployment of an ADAS or autonomous driving feature <NUM>, the introduced field monitor system <NUM> may assess whether this holds true also when the ADAS or autonomous driving feature <NUM> is on public roads and/or in customer hands.

As further shown in <FIG>, which is a schematic block diagram illustrating an exemplifying field monitor system <NUM> according to embodiments of the disclosure, the field monitor system <NUM> comprises a perception data storing unit <NUM>, a ground truth determining unit <NUM>, a deviation determining unit <NUM> and an acknowledgement communicating unit <NUM>, all of which already have been described in greater detail above. Furthermore, the embodiments herein for performance monitoring and evaluation of an ADAS or autonomous driving feature <NUM> of the vehicle <NUM>, may be implemented through one or more processors, such as a processor <NUM>, here denoted CPU, together with computer program code for performing the functions and actions of the embodiments herein. Said program code may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the field monitor system <NUM>. One such carrier may be in the form of a CD ROM disc and/or a hard drive, it is however feasible with other data carriers. The computer program code may furthermore be provided as pure program code on a server and downloaded to the field monitor system <NUM>. The field monitor system <NUM> may further comprise a memory <NUM> comprising one or more memory units. The memory <NUM> may be arranged to be used to store e.g. information, and further to store data, configurations, scheduling, and applications, to perform the methods herein when being executed in the field monitor system <NUM>. For instance, the computer program code may be implemented in the firmware, stored in FLASH memory <NUM>, of an embedded processor <NUM>, and/or downloaded wirelessly e.g. from an off-board server. Furthermore, the perception data storing unit <NUM>, the ground truth determining unit <NUM>, the deviation determining unit <NUM>, the acknowledgement communicating unit <NUM>, the optional processor <NUM> and/or the optional memory <NUM>, may at least partly be comprised in one or more nodes <NUM> e.g. ECUs of the vehicle <NUM>. Those skilled in the art will also appreciate that said units <NUM>, <NUM>, <NUM>, <NUM> described above may refer to a combination of analog and digital circuits, and/or one or more processors configured with software and/or firmware, e.g. stored in a memory such as the memory <NUM>, that when executed by the one or more processors such as the processor <NUM> perform as described herein, such as in conjunction with <FIG>. One or more of these processors, as well as the other digital hardware, may be included in a single ASIC (Application-Specific Integrated Circuitry), or several processors and various digital hardware may be distributed among several separate components, whether individually packaged or assembled into a SoC (System-on-a-Chip).

Further shown in <FIG> is the ADAS or autonomous driving feature <NUM>, the perception system <NUM>, the optional one or more surrounding detecting sensors <NUM>, the optional feature control system <NUM>, the acknowledgement data <NUM>, the optional further acknowledgement data <NUM>, and the optional remote entity <NUM>, all of which have been discussed in greater detail above.

<FIG> is a flowchart depicting an exemplifying method performed by a field monitor system <NUM> according to embodiments of the disclosure. Said method is for performance monitoring and evaluation of an ADAS or autonomous driving feature <NUM> of a vehicle <NUM>. The exemplifying method, which may be continuously repeated, comprises one or more of the following actions discussed with support from <FIG>. Moreover, the actions may be taken in any suitable order and/or one or more actions may be performed simultaneously and/or in alternate order where applicable. For instance, Action <NUM> and optional Action <NUM> may be performed in alternate order and/or simultaneously.

In Action <NUM>, the field monitor system <NUM> stores during a time period T ranging from a first time point t<NUM> to a second time point t<NUM> - e.g. with support from the perception data storing unit <NUM> - a range of perception data <NUM> on which the ADAS or autonomous driving feature <NUM> is pertinent, derived from a perception system <NUM> adapted to estimate surroundings of the vehicle <NUM>.

The range of perception data <NUM> comprises perception data of a surrounding object <NUM>, <NUM>', <NUM>", wherein the accuracy deviation measure reflects a measure in view of the object <NUM>, <NUM>', <NUM>". Further optionally, the accuracy deviation measure may reflect a measure in view of estimated safety assessment on which the ADAS or autonomous driving feature <NUM> is pertinent.

In Action <NUM>, the field monitor system <NUM> determines - e.g. with support from the ground truth determining unit <NUM> - based on post-processing a portion <NUM> of the range of perception data <NUM> ranging back from the second time point t<NUM> to an intermediate time point tintermediate between the first and second time points t<NUM>, t<NUM>, ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ at the intermediate time point tintermediate, which ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ reflects estimates of vehicle surroundings at the intermediate time point tintermediate conditional on the portion <NUM> of the range of perception data <NUM>.

In Action <NUM>, the field monitor system <NUM> determines - e.g. with support from the deviation determining unit <NUM> - based on analysis of the ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ in view of intermediate perception data 4i, 4i', 4i", 4i"', 4iʺʺ out of the range of perception data <NUM> corresponding to the intermediate time point tintermediate, an accuracy deviation measure indicating deviation between the ground truth intermediate data 6i, 6i', 6i", 6i"', 6iʺʺ and the intermediate perception data 4i, 4i', 4i", 4i"', 4i"".

In Action <NUM>, the field monitor system <NUM> communicates - e.g. with support from the acknowledgement communicating unit <NUM> - acknowledgement data <NUM> when the accuracy deviation measure exceeds a performance threshold.

Optionally, Action <NUM> of communicating acknowledgement data <NUM> may comprise transmitting the acknowledgement data <NUM> wirelessly to a remote entity <NUM>, which acknowledgement data <NUM> comprises the range of perception data <NUM>.

Additionally or alternatively, optionally, Action <NUM> of communicating acknowledgement data <NUM> may comprise transmitting the acknowledgement data <NUM> to an on-board feature control system <NUM> adapted to control the ADAS or autonomous driving feature <NUM>, which acknowledgement data <NUM> comprises an indication to disable the ADAS or autonomous driving feature <NUM>.

In optional Action <NUM>, the field monitor system <NUM> may, when the accuracy deviation measure exceeds a further performance threshold, communicate - e.g. with support from the acknowledgement communicating unit <NUM> - further acknowledgement data <NUM>, wherein the performance threshold deviates from the further performance threshold.

Optionally, Action <NUM> of communicating further acknowledgement data <NUM> may comprise transmitting the further acknowledgement data <NUM> wirelessly to the remote entity <NUM>, which further acknowledgement data comprises the range of perception data <NUM>.

Additionally or alternatively, Action <NUM> of communicating further acknowledgement data <NUM> may comprise transmitting the further acknowledgement data <NUM> to the feature control system <NUM>, which further acknowledgement data <NUM> comprises an indication to disable the ADAS or autonomous driving feature <NUM>.

Claim 1:
A method performed by a field monitor system (<NUM>) of a vehicle (<NUM>) for performance monitoring and evaluation of an advanced driver-assistance system, ADAS, or autonomous driving feature (<NUM>) of said vehicle (<NUM>), said method comprising:
storing (<NUM>), during a time period (T) ranging from a first time point (t<NUM>) to a second time point (t<NUM>), a range of perception data (<NUM>) on which said ADAS or autonomous driving feature (<NUM>) is pertinent derived from a perception system (<NUM>) adapted to estimate surroundings of said vehicle (<NUM>);
determining (<NUM>) based on post-processing comprising backward-processing a portion (<NUM>) of said range of perception data (<NUM>) ranging back from said second time point (t<NUM>) to an intermediate time point (tintermediate) between said first and second time points (t<NUM>, t<NUM>), ground truth intermediate data (6i, 6i', 6i", 6i"', 6iʺʺ) at said intermediate time point (tintermediate), said ground truth intermediate data (6i, 6i', 6i", 6i"', 6i"") reflecting estimates of vehicle surroundings at said intermediate time point (tintermediate) conditional on i.e. based on post-processing of data comprised in said portion (<NUM>) of the range of perception data (<NUM>);
determining (<NUM>) based on analysis of said ground truth intermediate data (6i, 6i', 6i", 6i"', 6iʺʺ) in view of intermediate perception data (4i, 4i', 4i", 4i"', 4i"") out of said range of perception data (<NUM>) corresponding to said intermediate time point (tintermediate), an accuracy deviation measure indicating deviation between said ground truth intermediate data (6i, 6i', 6i", 6i"', 6i"") and said intermediate perception data (4i, 4i', 4i", 4i"', 4iʺʺ), wherein said range of perception data (<NUM>) comprises perception data of a surrounding object (<NUM>, <NUM>', <NUM>"), said accuracy deviation measure reflecting a measure in view of said object (<NUM>, <NUM>', <NUM>") which evaluates performance and/or failure or failure risk of the at least one of said one or more surrounding detecting sensors sensing the perception data; and
communicating (<NUM>) acknowledgement data (<NUM>) indicative of, reflecting and/or comprising at least said accuracy deviation measure, when said accuracy deviation measure exceeds a performance threshold.