Patent Description:
Autonomous vehicles lack one important actor in the determination of vehicle maintenance need: the driver. The driver usually detects the need of a re-calibration or a maintenance requirement by, e.g., hearing unusual sounds generated by the vehicle or feeling the behaviour of the vehicle in general. Autonomous vehicles typically comprise a vast number of internal sensors, where diagnostics of various driving systems are based on feedback from those sensors. Unfortunately, when signs of maintenance need are detected, such as a mechanical failure, it is often too late to send the vehicle for maintenance. Too late can mean that a faulty component has reached critical failure and may be beyond repair, or even worse, the vehicle may have become undrivable. To detect a flat tire, e.g., several kilometres of driving may be required before it is detected by the internal sensors and corresponding control system. Other issues, like a malfunctioning spring leaf brake for example, are very difficult to detect at all with internal sensors.

<CIT> discloses a test system for an autonomously controllable motor vehicle. A processing device is designed to assess an operability of the motor vehicle based on a comparison of scanned behavior with predetermined behavior.

<CIT> discloses a road test method for an autonomous driving vehicle. The method comprises analyzing information of test parameters corresponding to autonomous driving scenarios to determine a first test result of the vehicle. This analysis is based on evaluation baselines corresponding to autonomous driving scenarios in a process during which a vehicle is travelling along a test route.

<CIT> discloses systems directed to vehicle self-diagnostics, where the vehicle comprises sensors perceiving objects and obstacles in an environment etc. Data from such sensors can be leveraged to determine a behavior associated with the vehicle, which is compared with an expected behavior. Based on any deviation, a fault can be determined, which in turn can diagnosed. Based on diagnosing the fault, the vehicle can determine instructions for redressing the fault.

<CIT> discloses a control system for a transportation vehicle.

<CIT> discloses a method and apparatus for vehicle control.

<CIT> discloses systems and methods for automatic maintenance of autonomous vehicles. However, there is a need for improved maintenance detection for autonomous vehicles.

It is an object of the present disclosure to provide improved maintenance need detection for autonomous vehicles. This object is at least in part obtained by a system for detecting need of preventive maintenance of a vehicle operating as part of an autonomous transport system. The system comprises a control unit and a remote sensor arranged physically separated from the autonomous vehicles. The control unit is arranged to obtain, via the remote sensor, a current driving behaviour of the autonomous vehicle in a test area; obtain a baseline driving behaviour of the autonomous vehicle in the test area; determine a deviation of the current driving behaviour from the baseline driving behaviour; and detect a maintenance need of the autonomous vehicle based on the determined deviation, wherein baseline driving behaviour is obtained from one or more previous observations the autonomous vehicle (<NUM>) in the test area (<NUM>).

This technology avoids breakdown of vehicles by monitoring driving behaviour and comparing that with a baseline. Performing preventive maintenance well ahead of critical failures is cost-effective and increases the efficiency of the autonomous transport system. Machine learning algorithms can be used over time to record previously unknown behaviours from any type of vehicle.

Autonomous transport systems, such as in a mining operation, typically present an environment where each vehicle behaves in a similar way each cycle. This enables an establishment of a well-defined baseline behaviour. Any deviation from that baseline can be a cause for concern and may indicate a maintenance need. The baseline behaviour can be established without assistance from the vehicles, which is an advantage. Forward-feed loops also creates over time a span of expected deviations which could be translated in maintenance cycle optimization, avoiding unnecessary changes of components for instance: Maintenance on a need basis methodology.

Arranging the remote sensor separate from the autonomous vehicle enables the capture of data that may be difficult to obtain from internal sensors. Also, since onboard sensors are located on the ego vehicle which vibrates and presents a dynamic behaviour, some types of failures may be difficult to detect. The system may use a plurality of remote sensors, but a single sensor will suffice for monitoring a fleet of vehicles, which is cost-effective. Furthermore, using a remote sensor rather than internal sensors means that no modifications of the vehicles in the autonomous transport system are necessary, neither by adding additional sensors nor adapting existing sensors.

The system can operate independently from other control systems related to the vehicle. The obtaining of the current driving behaviour, the obtaining of the baseline driving behaviour, the determination of the deviation, and the detection of maintenance can, according to aspects, all be performed within the disclosed system. This facilitates data management since, e.g., there is no need for wireless connections loaded with large amounts of data from, e.g., movies captured by a camera constituting the sensor.

It may be difficult to detect a drifting accuracy of internal sensors inherent in autonomous vehicles when relying on the data output of those internal sensors. The disclosed system, however, can detect such drift. Usually, when internal sensors are drifting, a small degradation of the driving behaviour is noticed over time. This degradation can be observed by the disclosed system as a larger and larger deviation from a baseline driving behaviour.

According to aspects, the control unit is further arranged to determine a fault based on the determined deviation. Determining a fault can speed up the maintenance procedure of the vehicle, which improves the productivity of the autonomous transport system. The fault can comprise, i. , any of tire wear, mechanical defects, suspension system failure, brake failure, steering system failure and sensor position disturbance.

According to aspects, the remote sensor comprises any of a camera, an IR camera, a lidar, a sonar, a microphone, and a radar. Such sensors can obtain various parameters representative of the driving behaviour and are cost-effective. Data from different sensors can be cross-checked which provides accurate information. Also, since there is no need of real-time processing, more complex algorithms can be deployed.

According to aspects, the current driving behaviour and baseline driving behaviour comprise one or more locations or travelled path of the autonomous vehicle in the test area. The travelled path is a good indicator of the driving behaviour; any deviation from a baseline path can indicate a maintenance need way ahead of critical failures. The current driving behaviour and baseline driving behaviour may alternatively, or in combination of, comprise any of velocity, acceleration, yaw, and yaw rate of the autonomous vehicle, which are also parameters representative of the driving behaviour.

According to aspects, maintenance is recommended if the deviation is a above a predetermined threshold deviation. This provides a low-complexity system for detecting a maintenance need.

According to aspects, maintenance is recommended based on a computer-implemented classification model arranged to determine a maintenance need based on the determined deviation. This provides an accurate way of detecting a maintenance need.

According to aspects, the computer-implemented classification model is based on any of a look up table and an analytical function. This can enable a quick setup of the classification model. Alternatively, or in combination of, the computer-implemented classification model is based on any of a neural network, a random forest structure, a support vector machine model, a logistic regression algorithm, a Bayes algorithm, a decision tree algorithm, and a K-nearest neighbours' algorithm. Such machine learning techniques provide adaptability and versatility for different faults or causes for maintenance needs.

As mentioned, the baseline driving behaviour is obtained from one or more previous observations the autonomous vehicle in the test area. This way, it is possible to establish a unique baseline for each vehicle in the cargo transport system. This provides a cost-effective yet accurate way of automatically detecting maintenance needs.

According to aspects, the baseline behaviour is obtained from one or more previous observations of one or more autonomous vehicles in the test area. This way, observations from a plurality of vehicles in the fleet in the autonomous transport system may be used, which may provide a more representative baseline.

According to aspects, the baseline driving behaviour is obtained from a planned driving behaviour of the autonomous vehicle. The planned driving behaviour may be what the autonomous driving system intends, e.g., an intended path. Using such data for the baseline can eliminate or reduce the need of forming the baseline based on test drives in the test area. Furthermore, the system may return a diagnostic based on the deviation to the vehicle or other system part of the autonomous driving.

According to aspects, the control unit is arranged to prompt the autonomous vehicle to perform a test case manoeuvre in the test area. This way, specific manoeuvres that are known to indicate a maintenance need when deviating from the baseline may be observed and analysed. Furthermore, the test case manoeuvres may indicate different types of faults that can be detected by the disclosed system.

There is also disclosed herein methods, computer programs, computer readable media, computer program products, control units, and vehicles associated with the above discussed advantages.

<FIG> shows a heavy-duty vehicle <NUM>. This particular example comprises a tractor unit towing a trailer unit, which in turn tows an additional trailer unit using a dolly unit. The vehicle combination <NUM> may of course also comprise additional vehicle units. The techniques disclosed herein are applicable to rigid trucks, and also to passenger cars, although the main benefit of the proposed technique is obtained when used with heavy-duty vehicle for autonomous transport system.

As mentioned, there is a need for improved systems that can detect maintenance needs of autonomous vehicles. In particular, internal sensors of autonomous vehicles typically detects faults and maintenance needs later than desired. Detecting a drifting accuracy of such internal sensor is also challenging. As of today, any diagnostics in autonomous vehicles are based on the detection of a failure, not on the prevention of it. Moreover, an autonomous vehicle typically believes its path/trajectory is followed as it should, even when an external system would not agree.

Therefore, there is herein disclosed a system <NUM> for detecting a maintenance need of an autonomous vehicle <NUM> operating as part of an autonomous transport system. The system comprises a control unit <NUM> and a remote sensor <NUM> arranged separate from the autonomous vehicle <NUM>. The control unit is arranged to obtain, via the remote sensor <NUM>, a current driving behaviour of the autonomous vehicle <NUM> in a test area <NUM>; obtain a baseline driving behaviour of the autonomous vehicle <NUM> in the test area <NUM>; determine a deviation of the current driving behaviour from the baseline driving behaviour; and detect a maintenance need of the autonomous vehicle <NUM> based on the determined deviation.

The disclosed system is suitable for detecting a maintenance needs of an autonomous vehicle. This includes fully autonomous vehicles and partially autonomous vehicles. In general, however, the system <NUM> and corresponding method disclosed herein can detect a maintenance need of vehicles where it is possible to compare a current driving behaviour to a baseline driving behaviour. To detect a maintenance need herein means to detect an anomaly in the driving behaviour resulting from a faulty component, system, subsystem etc. before critical failure.

The remote sensor <NUM> can be located at specific places where the driving behaviour is representative to parts of or the whole operation route. If anything in the vehicle does not look good, i.e., if the current driving behaviour deviates too much from the baseline driving behaviour according to some pre-determined metric, the vehicle <NUM> can be removed from operation and be brought to the maintenance workshop autonomously.

Arranging the sensor <NUM> separate from the autonomous vehicle <NUM> enables the capture of data that may be difficult to obtain from internal sensors. Arranged separate can, e.g., mean that the sensor is arranged stationary with respect to a road, such as a camera aimed at a section of the road. The system <NUM> may use a plurality of remote sensors <NUM>, but a single sensor will suffice, which is cost-effective. Furthermore, using a remote sensor rather than internal sensors means that no modifications of the vehicles are necessary, neither by adding additional sensors nor by adapting existing sensors.

The system <NUM> can operate independently from other control systems related to the vehicle. The obtaining of the current driving behaviour, the obtaining of the baseline driving behaviour, the determination of the deviation, and the detection of maintenance can, according to aspects, all be performed within the system <NUM>. This facilitates data management since, e.g., there is no need for wireless connections loaded with large amounts of data from, e.g., movies captured by a camera constituting the sensor <NUM>.

The control unit <NUM> can be connected to vehicle <NUM> and other relating systems for autonomous driving. For example, the control unit can be connected to a cloud system with traffic planner functionality, which coordinates which vehicles to send to the workshop and optionally also which selects what kind of service is needed.

In general, the system <NUM> can take all kinds of variations possible in the driving behaviour over time. Based on that, it is possible to predict when the vehicle will should to go through maintenance and/or calibration. This way, it is possible to do maintenance before a complete breakdown.

<FIG> is a plot showing a deviation of driving behaviour versus time. A first line <NUM> shows a deviation increasing over time. A breakdown occurs at time instance <NUM>. This breakdown could have been avoided if maintenance had been done well ahead of the time instance <NUM>. A second line <NUM> also shows a deviation increasing over time. Here, on the other hand, the vehicle is taken out of the cargo transport system for maintenance after the deviation has reached a threshold value <NUM>, and the breakdown therefore is avoided.

As mentioned, it may be difficult to detect a drifting accuracy of internal sensors using the internal sensors of a vehicle. The disclosed system <NUM>, however, can detect such drift. Usually, when internal sensors are drifting, a small degradation of the driving behaviour is noticed over time. This degradation can be observed by the system <NUM> as a larger and larger deviation from a baseline driving behaviour.

In the example of <FIG>, the heavy-duty vehicle <NUM> is driving along a lane on road <NUM> and has just passed through a <NUM>-degree corner. A single sensor <NUM> is arranged at the corner and is arranged to capture the current driving behaviour of the vehicle in the test area <NUM>, which, in this case, covers the corner. In this example, the travelled paths <NUM>, <NUM>, and <NUM> of three points on the back of the vehicle are observed. More generally, the current driving behaviour and baseline driving behaviour may comprise one or more locations or travelled path of the autonomous vehicle <NUM> in the test area <NUM>. The driving behaviour may also or alternatively comprise other data. In particular, the current driving behaviour and baseline driving behaviour may comprise any of velocity, acceleration, yaw, and yaw rate of the autonomous vehicle <NUM>. In general, however, the driving behaviour may comprise any data representative of the operation of the vehicle.

The baseline driving behaviour is obtained from one or more previous observations the autonomous vehicle <NUM> in the test area <NUM>. In the example embodiment of <FIG>, the three travelled paths across the test area may at a first instance be captured and saved as the baseline driving behaviour. Any succeeding passes across the test area may thereafter be compared to the stored baseline behaviour. The baseline behaviour may also be obtained from a plurality of observations of the vehicles in the test area, such as the average of ten controlled test runs.

The baseline driving behaviour may be unique for each single vehicle. In combination of, the base line driving behaviour may be representative for a fleet of the same model of vehicles, or even same type of vehicles. Therefore, the baseline behaviour may be obtained from one or more previous observations of one or more autonomous vehicles <NUM> in the test area <NUM>. Thus, the base line behaviour can be established from a different vehicle than the one the current driving behaviour is obtained from.

According to aspects, the system <NUM> has access to an intended behaviour of the vehicle <NUM>, e.g., the path the autonomous driving system is prompting the vehicle to follow. Therefore, the baseline driving behaviour may be obtained from a planned driving behaviour of the autonomous vehicle <NUM>. In this case, the system <NUM> may return a diagnostic to the vehicle or other system part of the autonomous driving. The system may be integrated as part of the vehicle perception system network via, e.g., V2I (V2X) communication. Additionally, the system may be used to create a distributed perception network for the autonomous cargo transport system.

The baseline driving behaviour may be obtained from a plurality of different ways, such as the ways mentioned above or more generally any way establishing an expected behaviour of the vehicle.

The remote sensor <NUM> may comprise any of a camera, an IR camera, a lidar, a sonar, a microphone, and a radar. The cameras, lidar, sonar, and radar may be used to track one or more points of the vehicle <NUM>. A microphone may be used to detect sounds from the engine, gear boxes, and actuators etc. More generally, the remote sensor can be any type of sensor that can capture data representative of the driving behaviour of the vehicle. As mentioned, the system <NUM> may comprise a plurality of sensors <NUM>, which may be any combination of different types of sensors. The one or more sensors may be comprised in a single unit or be distributed. Similarly, the control unit <NUM> may be integrated with one or more sensors <NUM> or be in a separate unit. The control unit may also be distributed; it can also be cloud based.

The remote sensor <NUM> obtains driving behaviour of the autonomous vehicle <NUM> in a test area <NUM>. This can, e.g., mean a field of view for a camera, radar, lidar etc., and/or an area where the sensor can capture data with a predetermined fidelity, such as resolution, bandwidth, dynamic range etc. As an example, the test area can include a five-by-five meter squared cross section of a road.

According to aspects, maintenance may be recommended by the control unit <NUM> if the deviation is a above a predetermined threshold deviation. For example, if any of the tracked paths in <FIG> of the current observation deviates more than <NUM> from a previous pass in the test area constituting the baseline, maintenance is recommended. Different behavioural parameters, such as the travelled path of a point or the sound of the engine, may be associated with respective threshold deviations. It is also possible that a threshold deviation is obtained from a weighted averaging of deviations of different behavioural parameters.

To identify if maintenance is needed, a computer-implemented classification model can be utilized with advantage. In other words, maintenance may be recommended by the control unit <NUM> based on a computer-implemented classification model arranged to determine a maintenance need based on the determined deviation. In particular, such model may be a machine learning model. Machine learning generally relates to techniques where a model with a pre-determined structure is modified to provide a desired function by means of some form of training. Any machine learning model may be trained using one or more test runs in the test area and/or using simulations. For example, the computer-implemented classification model may be based on any of a neural network, a random forest structure, a support vector machine model, a logistic regression algorithm, a Bayes algorithm, a decision tree algorithm, and a K-nearest neighbours' algorithm. These models are known in general and will therefore not be discussed in more detail herein.

The classification model may also be based on other algorithms/models. In particular, the computer-implemented classification model may be based on any of a look up table and an analytical function.

According to aspects, the control unit <NUM> may further be arranged to determine a fault based on the determined deviation. Certain types of faults may result in a predictable deviation in the current driving behaviour compared to the baseline. A fault can be detected based on, e.g., the magnitude of the deviation of a behaviour parameter of the driving behaviour, such as how much a tracked point of the vehicle deviates from a baseline path. Faults can also be identified using a computer-implemented classification model. The fault may comprise any of tire wear, mechanical defects, suspension system failure, and brake failure. The fault can also be other type defects, wear, or such.

The system <NUM> can further be used to send test cases to the vehicles and monitor the feedback. Such tests can be used as a confirmation of system integrity. In other words, the control unit <NUM> may be arranged to prompt the autonomous vehicle <NUM> to perform a test case manoeuvre in the test area <NUM>. The baseline driving behaviour may be captured during a previous test. Alternatively, or in combination of, the baseline can be established by simulations. A simulation platform may run in the control unit <NUM> in the system <NUM> and can simulate various test cases.

There is also disclosed herein a method for detecting a maintenance need of an autonomous vehicle <NUM>, as is shown in <FIG>. The method comprises.

<FIG> schematically illustrates, in terms of a number of functional units, the components of the control unit <NUM> according to an embodiment of the discussions herein. Processing circuitry <NUM> is provided using any combination of one or more of a suitable central processing unit CPU, multiprocessor, microcontroller, digital signal processor DSP, etc., capable of executing software instructions stored in a computer program product, e.g., in the form of a storage medium <NUM>. The processing circuitry <NUM> may further be provided as at least one application specific integrated circuit ASIC, or field programmable gate array FPGA.

The processing circuitry <NUM> controls the general operation of the control unit <NUM> e.g., by sending data and control signals to the interface <NUM> and the storage medium <NUM>, by receiving data and reports from the interface <NUM>, and by retrieving data and instructions from the storage medium <NUM>.

<FIG> schematically illustrates a computer program product <NUM>, comprising a set of operations <NUM> executable by the control unit <NUM>. The set of operations <NUM> may be loaded into the storage medium <NUM> in the control unit <NUM>. The set of operations may correspond to the methods discussed above in connection to <FIG>.

Claim 1:
A system (<NUM>) for detecting a maintenance need of an autonomous vehicle (<NUM>) operating as part of a cargo transport system, the system comprising a control unit (<NUM>) and a sensor (<NUM>), where the control unit is arranged to
obtain, via the sensor (<NUM>), a current driving behaviour of the autonomous vehicle (<NUM>) in a test area (<NUM>),
obtain a baseline driving behaviour of the autonomous vehicle (<NUM>) in the test area (<NUM>), determine a deviation of the current driving behaviour from the baseline driving behaviour, and
detect a maintenance need of the autonomous vehicle (<NUM>) based on the determined deviation,
characterized in that said sensor (<NUM>) is a remote sensor arranged separate from the autonomous vehicle and in that said baseline driving behaviour is obtained from one or more previous observations of the autonomous vehicle (<NUM>) in the test area (<NUM>).