Patent Description:
A robotic work tool is an autonomous robot apparatus that is used to perform certain tasks, for example cutting lawn grass, demolish an area or cleaning a floor. In order to perform these tasks, the robotic work tool generally uses various sensors in order to perceive its environment and the area within which the robotic work tool is intended to operate. During recent years, the robotic work tools have been developed to use cameras to capture images and video streams of their surroundings. The images or video streams may then be processed in order to detect and classify objects and surfaces within the surroundings of the robotic work tools. A robotic lawn mower according to the state of the art is disclosed by <CIT>.

However, even if the use of cameras to detect and classify objects and surfaces within the surroundings of the robotic work tools generally has improved the operation of the robotic work tools, the inventors have realized that interfering objects, such as dirt, may cover these cameras and disturb the view of the cameras. This may affect the ability to correctly detect and classify objects and surfaces within the surroundings of the robotic work tools. The inventors have realized that there is a need for detecting any such disturbing and interfering objects to prevent that the view of the cameras are disturbed.

Robotic work tools may operate in harsh environments that include a lot of dirt, mud, moisture and other disturbing or interfering object. These interfering objects may accumulate on the camera units of the robotic work tools and gradually eliminate their ability to capture clear images. This may degrade the object detection accuracy of the robotic work tools and lead to unwanted system behaviour. Thus, as mentioned in the background section, there is a need for determining when the object detection accuracy of a robotic work tool is degraded. If the object detection accuracy is degraded, that poses safety risks to the robot work tool's environment and the robot work tool itself. Therefore, a system and a method for detecting when there is an interfering object on a camera unit of a robotic work tool is needed.

In view of the above, it is therefore a general object of the aspects and embodiments described throughout this disclosure to provide a robotic work tool system that detects any disturbing and interfering objects on the camera unit of the robotic work tool, which may degrade the operation of the robotic work tool.

This general object has been addressed by the appended independent claims. Advantageous embodiments are defined in the appended dependent claims.

According to a first aspect, there is provided a robotic work tool system for determining whether there is an interfering object on a camera unit of a robotic work tool.

According to the invention, the robotic work tool is a robotic lawn mower configured to perform a task, such as a work task, autonomously.

The robotic work tool system comprises at least one controller. The at least one controller is configured to obtain a plurality of image frames reflecting the view of the camera unit of the robotic work tool. Each of the obtained image frames is associated with a different position and/or angle compared to the other obtained plurality of image frames. The at least one controller is further configured to determine, based on the obtained image frames, pixel intensity gradients within the view of the camera unit; and to compare each of the determined pixel intensity gradients against at least one threshold. The at least one controller is further configured to determine, based on the comparison, whether there is an interfering object on the camera unit of the robotic work tool. Thereafter, the at least one controller is configured to control a subsequent action of the robotic work tool system based on the determination whether there is an interfering object on the camera unit of the robotic work tool.

In some embodiments, the at least one controller is further configured to determine, based on the obtained image frames and the determination whether there is an interfering object on the camera unit, a proportion of an image frame not occluded by any interfering objects.

In some embodiments, the at least one controller is further configured to determine, based on the determination whether there is an interfering object on the camera unit, an intensity of all pixels within the obtained image frames comprising an interfering object; and to determine, based on the determination whether there is an interfering object on the camera unit, an intensity of all pixels within the obtained image frames not comprising an interfering object. The at least one controller is further configured to compare the determined intensity for pixels comprising an interfering object with the determined intensity for pixels not comprising an interfering object; and to determine, based on the comparison, a Transparency Ratio (TR) for any determined interfering object on the camera unit of the robotic work tool.

In some embodiments, the at least one controller is further configured to determine, based on the determination whether there is an interfering object on the camera unit, a blurriness of all pixels within the obtained image frames comprising an interfering object; and to determine, based on the determination whether there is an interfering object on the camera unit, a blurriness of all pixels within the obtained image frames not comprising an interfering object. The at least one controller is further configured to compare the determined blurriness for pixels comprising an interfering object with the determined blurriness for pixels not comprising an interfering object; and to determine, based on the comparison, a Blurriness Ratio (BR) for any determined interfering object on the camera unit of the robotic work tool.

In some embodiments, the at least one controller is further configured to determine, based on the determined proportion of the image frame not occluded by any interfering objects, the determined TR and the determined BR, an object detection reliability of the robotic work tool system.

In some embodiments, the at least one controller is configured to determine the object detection reliability of the robotic work tool system by determining a weight for each of the determined proportion of the image frame not occluded by any interfering objects, the determined TR and the determined BR. Thereafter each of the weighted determined proportion of the image frame not occluded by any interfering objects, the weighted determined TR and the weighted determined BR is added to a sum in order to determine the object detection reliability of the robotic work tool system.

In some embodiments, when the determined object detection reliability of the robotic work tool system is below an object detection reliability threshold, the at least one controller is configured to control a subsequent action of the robotic work tool system to avoid degraded operation of the robotic work tool due to the low object detection reliability.

In some embodiments, the at least one controller is further configured to receive an indication that any interfering object on the camera unit of the robotic work tool has been removed; and reset, based on the indication, the object detection reliability of the robotic work tool system.

At least one controller is configured to control the subsequent action of the robotic work tool system by transmitting a message to an output device. The message may define reliable vs un-reliable image pixels within the obtained image frames.

In some embodiments, when it is determined that there is an interfering object on the camera unit of the robotic work tool, the at least one controller is configured to control the subsequent action of the robotic work tool system by initiating a cleaning operation of the robotic work tool to remove any interfering object on the camera unit of the robotic work tool.

In some embodiments, when it is determined that there is an interfering object on the camera unit of the robotic work tool, the at least one controller is configured to control the subsequent action of the robotic work tool system by controlling a travel operation of the robotic work tool.

In some embodiments, the plurality of image frames is obtained from the camera unit while at least one of the camera unit and the robotic work tool is moving.

In some embodiments, the plurality of image frames reflecting the view of the camera unit of the robotic work tool is obtained with fixed predefined intervals.

The at least one controller is configured to determine pixel intensity gradients within the view of the camera unit by determining a two-dimensional (2D) pixel intensity gradient image for each of the plurality of obtained image frame; and by determining a pixel-wise norm of each 2D pixel intensity gradient image. Thereafter the determined pixel-wise norm of each 2D pixel intensity gradient image is averaged into a single image frame of average norms of pixel intensity gradients.

In some embodiments, the at least one controller is further configured to determine, based on the determined pixel intensity gradients, contours of an interfering object on the camera unit.

In some embodiments, said at least one threshold comprises a lower threshold and a higher threshold and the at least one controller is configured to determine whether there is an interfering object on the camera unit of the robotic work tool based on the comparison by determining that there is an interfering object on the camera unit of the robotic work tool if the determined pixel intensity gradient is below the lower threshold. Furthermore, it is determined there is non-disturbing interfering object on the camera unit of the robotic work tool if the determined pixel intensity gradient is above the lower threshold and below the higher threshold; and it is determined that there is no interfering object on the camera unit of the robotic work tool if the determined pixel intensity gradient is above the higher threshold.

In some embodiments, the position and/or angle associated with each of the obtained image frames is obtained from at least one position sensor of the robotic work tool.

According to the invention, the robotic work tool system comprises a camera unit.

The robotic work tool comprises a robotic apparatus configured to perform a work task autonomously.

According to a second aspect, there is provided a method implemented by the robotic work tool system according to the first aspect.

In one exemplary implementation, the method is performed by at least one controller, for determining whether there is an interfering object on a camera unit of a robotic work tool. The method comprises obtaining a plurality of image frames reflecting the view of the camera unit of the robotic work tool. Each of the obtained image frames is associated with a different position and/or angle compared to the other obtained plurality of image frames. The method further comprises determining, based on the obtained image frames, pixel intensity gradients within the view of the camera unit and comparing each of the determined pixel intensity gradients against at least one threshold. Thereafter, the method comprises determining, based on the comparison, whether there is an interfering object on the camera unit of the robotic work tool; and controlling a subsequent action of the robotic work tool system based on the determination whether there is an interfering object on the camera unit of the robotic work tool.

In some embodiments, the method further comprises determining, based on the obtained image frames and the determination whether there is an interfering object on the camera unit, a proportion of an image frame not occluded by any interfering objects.

In some embodiments, the method further comprises determining, based on the determination whether there is an interfering object on the camera unit, an intensity of all pixels within the obtained image frames comprising an interfering object. The method further comprises determining, based on the determination whether there is an interfering object on the camera unit, an intensity of all pixels within the obtained image frames not comprising an interfering object; and comparing the determined intensity for pixels comprising an interfering object with the determined intensity for pixels not comprising an interfering object. The method further comprises determining, based on the comparison, a TR for any determined interfering object on the camera unit of the robotic work tool.

In some embodiments, the method further comprises determining, based on the determination whether there is an interfering object on the camera unit, a blurriness of all pixels within the obtained image frames comprising an interfering object; and determining, based on the determination whether there is an interfering object on the camera unit, a blurriness of all pixels within the obtained image frames not comprising an interfering object. The method further comprises comparing the determined blurriness for pixels comprising an interfering object with the determined blurriness for pixels not comprising an interfering object; and determining, based on the comparison, a BR for any determined interfering object on the camera unit of the robotic work tool.

In some embodiments, the method further comprises determining, based on the determined proportion of the image frame not occluded by any interfering objects, the determined TR and the determined BR, an object detection reliability of the robotic work tool system. The object detection reliability may be determined by determining a weight for each of the determined proportion of the image frame not occluded by any interfering objects, the determined TR and the determined BR; and adding each of the weighted determined proportion of the image frame not occluded by any interfering objects, the weighted determined TR and the weighted determined BR to a sum in order to determine the object detection reliability of the robotic work tool system.

In some embodiments, when the determined object detection reliability of the robotic work tool system is below an object detection reliability threshold, the method further comprises controlling a subsequent action of the robotic work tool system to avoid degraded operation of the robotic work tool due to the low object detection reliability.

In some embodiments, the method further comprises receiving an indication that any interfering object on the camera unit of the robotic work tool has been removed; and resetting, based on the indication, the object detection reliability of the robotic work tool system.

In some embodiments, the step of controlling the subsequent action of the robotic work tool system comprises transmitting a message to an output device. The message may define reliable vs un-reliable image pixels within the obtained image frames.

In some embodiments, when it is determined that there is an interfering object on the camera unit of the robotic work tool, the step of controlling the subsequent action of the robotic work tool system comprises initiating a cleaning operation of the robotic work tool to remove any interfering object on the camera unit of the robotic work tool.

In some embodiments, when it is determined that there is an interfering object on the camera unit of the robotic work tool, the step of controlling the subsequent action of the robotic work tool system comprises controlling a travel operation of the robotic work tool.

In some embodiments, the step of determining pixel intensity gradients within the view of the camera unit comprises determining a 2D pixel intensity gradient image for each of the plurality of obtained image frame and determining a pixel-wise norm of each 2D pixel intensity gradient image. It further comprises averaging the determined pixel-wise norm of each 2D pixel intensity gradient image into a single image frame of average norms of pixel intensity gradients.

In some embodiments, the method further comprises determining, based on the determined pixel intensity gradients, contours of an interfering object on the camera unit.

In some embodiments, said at least one threshold comprises a lower threshold and a higher threshold and the step of determining whether there is an interfering object on the camera unit of the robotic work tool based on the comparison comprises determining that there is an interfering object on the camera unit of the robotic work tool if the determined pixel intensity gradient is below the lower threshold; and determining that there is non-disturbing interfering object on the camera unit of the robotic work tool if the determined pixel intensity gradient is above the lower threshold and below the higher threshold. The method step further comprises determining that there is no interfering object on the camera unit of the robotic work tool if the determined pixel intensity gradient is above the higher threshold.

In some embodiments, the robotic work tool system comprises the robotic work tool comprising the camera unit.

In some embodiments, the robotic work tool comprises a robot apparatus configured to perform a work task autonomously.

Some of the above embodiments eliminate or at least reduce the problems discussed above. By determining the pixel intensity gradients and their size within the view of the camera unit of the robotic work tool, it may be determined whether there is any interfering object on the camera of the robotic work tool. Based on this determination, it may be possible to control at least one subsequent action of the robotic work tool such that any safety risks to the robot work tool and its environment may be eliminated, or at least reduced. Thus, a robotic work tool system and method are provided that improve the operation of the robotic work tool.

These and other aspects, features and advantages will be apparent and elucidated from the following description of various embodiments, reference being made to the accompanying drawings, in which:.

The disclosed embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the robotic work tool system are shown. This robotic work tool system may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the robotic work tool system to those skilled in the art.

In one of its aspects, the disclosure presented herein concerns a robotic work tool system for determining whether there is an interfering object on a camera unit of a robotic work tool. The interfering object may be, for example, dirt, mud, a substance, fluid, snow, water or moist. The robotic work tool system for determining whether there is an interfering object on a camera unit may determine if there is an interfering object on a lens of the camera unit. Alternatively, the robotic work tool system may determine if there is an interfering object on a transparent material protecting the camera unit, i.e. in front of the camera.

<FIG> illustrates a schematic view of a robotic work tool system <NUM>. As will be appreciated, the schematic view is not to scale. The present disclosure is now going to be described with reference to <FIG>. The robotic work tool system <NUM> comprises at least one controller <NUM>, <NUM>. As may be appreciated, the robotic work tool system <NUM> may comprise a plurality of controllers <NUM>, <NUM> communicatively coupled to each other. By combining a plurality of controllers <NUM>, <NUM>, even higher processing power may be achieved.

The robotic work tool system <NUM> will mainly be described in general terms of a robotic work tool system <NUM> for determining whether there is an interfering object on a camera unit of a robotic work tool, such as the robotic work tool <NUM> illustrated in <FIG>. It should be understood that the robotic work tool system <NUM> described herein may be implemented together with any type of autonomous machine that may perform a desired activity. The robotic work tool <NUM> may comprise a robotic apparatus configured to perform a task, such as a work task, autonomously. Examples of such types of robotic apparatuses include, without limitation, lawn mowers, construction robotic work tools, cleaning robotic work tools, automatic moving cameras and/or drones, polishing work tools, repair work tools, surface-processing work tools (for indoors and/or outdoors), demolition work tools and/or agricultural, park and green space maintenance robots. Regardless of which type of robotic apparatus that is used, the robotic work tool <NUM> according to the present disclosure comprises a camera unit <NUM> shown in <FIG>.

As illustrated in <FIG>, the at least one controller <NUM> may be located within the robotic work tool <NUM>. Alternatively, or additionally, the at least one controller <NUM> may be communicatively coupled to the robotic work tool <NUM> by a wireless communication interface. The wireless communication interface may also be used to communicate with other devices, such as servers, personal computers or smartphones, charging stations, remote controls, other robotic work tools or any remote device, which comprises a wireless communication interface and a controller. Examples of such wireless communication are Bluetooth®, Global System Mobile (GSM), Long Term Evolution (LTE) and <NUM> or New Radio (<NUM> NR), to name a few. In some embodiments, the robotic work tool system <NUM> may comprise the robotic work tool <NUM> comprising the camera unit <NUM>.

In one embodiment, the at least one controller <NUM>, <NUM> is embodied as software, e.g. remotely in a cloud-based solution. In another embodiment, the at least one controller <NUM>, <NUM> may be embodied as a hardware controller. The at least one controller <NUM>, <NUM> may be implemented using any suitable, publicly available processor, computing means, virtual computer, cloud computer or Programmable Logic Circuit (PLC). The at least one controller <NUM>, <NUM> may be implemented using instructions that enable hardware functionality, for example, by using executable computer program instructions in a general-purpose or special-purpose processor that may be stored on a computer readable storage medium (disk, memory etc.) to be executed by such a processor. The controller <NUM>, <NUM> may be configured to read instructions from a memory <NUM>, <NUM> and execute these instructions to determine whether there is an interfering object on a camera unit <NUM> of the robotic work tool <NUM>. The memory may be implemented using any commonly known technology for computer-readable memories such as ROM, RAM, SRAM, DRAM, FLASH, DDR, SDRAM or some other memory technology.

As illustrated in <FIG>, the robotic work tool <NUM> comprises at least one camera unit <NUM>. The camera unit <NUM> is configured to obtain image frames reflecting the view of the camera unit <NUM>, i.e. the surroundings of the robotic work tool <NUM>. As previously described, these image frames may be processed in order to detect and classify objects and surfaces within the surroundings of the robotic work tool <NUM>. The camera unit <NUM> may be realized in a number of different ways as long as it is configured to capture video streams, still pictures and/or other data reflecting the scenery surrounding the robotic work tool <NUM>.

A first embodiment according to the first aspect will now be described. The at least one controller <NUM>, <NUM> is configured to obtain a plurality of image frames reflecting the view of the camera unit <NUM> of the robotic work tool <NUM>. <FIG> illustrates an example of such an image frame. Each of the obtained image frames is associated with a different position and/or angle compared to the other obtained plurality of image frames. The plurality of image frames may be obtained from the camera unit <NUM> while at least one of the camera unit <NUM> and the robotic work tool <NUM> is moving. The plurality of image frames reflecting the view of the camera unit <NUM> of the robotic work tool <NUM> may be obtained with fixed predefined regular, or irregular, intervals. The intervals may be, for example, time based and/or angular or distance based. Thus, in order to determine whether any interfering object is on the camera unit <NUM>, a plurality of image frames from different positions and/or angles is used, which will reduce the probability for false positive detection of interfering objects. The position and/or angle associated with each of the obtained image frames may be obtained from at least one position sensor <NUM> of the robotic work tool <NUM>. Examples of such position sensor <NUM> include, without limitation, wheel odometry, RTK/GPS, GNSS-navigation and positioning, SLAM, IMU data and INS data.

The at least one controller <NUM>, <NUM> is further configured to determine, based on the obtained image frames, pixel intensity gradients within the view of the camera unit <NUM>. By obtaining image frames from a plurality of different positions and/or angles, the scenery of the camera unit <NUM> changes. However, the interfering objects will appear as stationary points within the camera view and for the interfering objects on the camera unit <NUM>, no motion, or change, will appear. The interfering object will be located at the same spot on the camera unit <NUM> regardless of the position and/or angle of the robotic work tool <NUM> and/or the camera unit <NUM> of the robotic work tool <NUM>. Thus, interfering objects may generally have lower pixel intensity gradient than the scenery, or at least they will have pixel intensity gradients that deviate from expected pixel intensity gradients.

The pixel intensity gradients within the view of the camera unit <NUM> may be determined by the at least one controller <NUM>, <NUM> by determining a two-dimensional (2D) pixel intensity gradient image for each of the plurality of obtained image frame. This may be performed, for example, by using vertical and horizontal Sobel filtering. <FIG> shows an example of an image frame after vertical and horizontal, i.e. 2D, Sobel filtering. Thereafter a pixel-wise norm of each 2D pixel intensity gradient image may be determined and the determined pixel-wise norm of each 2D pixel intensity gradient image may be averaged into a single image frame of average norms of pixel intensity gradients. <FIG> shows an example of such averaged norm image. As previously described, in this single image frame, pixels that were occluded by an interfering object will generally have a very low average norm, while the other pixels will have a higher average norm. The average norm may thereafter, in some embodiments, be equalized using K-means clustering. Such equalization may allow clustering of the average norm values to adapt to external lighting conditions. <FIG> shows an example of an equalized average norm image frame using K-mean clustering.

The at least one controller <NUM>, <NUM> is further configured to compare each of the determined pixel intensity gradients against at least one threshold and to determine, based on the comparison, whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>. Thus, by comparing the pixel intensity gradients against at least one threshold it is possible to determine whether the determined pixel intensity gradients deviate from expected pixel intensity gradients for the scenery and to determine whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>. Accordingly, each pixel may be classified as an interfering pixel, e.g. dirt, or as scene, i.e. the surrounding of the robotic work tool <NUM>.

In some embodiments, the at least one threshold may comprise one predefined threshold, and if it is determined that a pixel intensity gradient is below this threshold, it may be determined that there is an interfering object on the camera unit <NUM>, i.e. that there are stationary feature points on the camera unit <NUM>. However, in preferred embodiments, the at least one threshold may comprise a lower threshold and a higher threshold. In these embodiments, the at least one controller130, <NUM> may be configured to determine whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM> based on the comparison by determining that there is an interfering object on the camera unit <NUM> if the determined pixel intensity gradient is below the lower threshold. It may further be determined that there is a non-disturbing interfering object on the camera unit <NUM> if the determined pixel intensity gradient is above the lower threshold, but below the higher threshold. Thus, if the pixel intensity gradient is between the lower and the higher threshold, there may be an object, or something, on the camera unit <NUM>, but this object is a non-disturbing object. Finally, it may be determined that there is no interfering object on the camera unit <NUM> of the robotic work tool <NUM> if the determined pixel intensity gradient is above the higher threshold. In some further embodiments, the at least one threshold may further comprise a droplet threshold, which is a highest threshold, i.e. a threshold even higher than the previously described higher threshold. For example, a water droplet may distort the camera view into an omnidirectional sub-lens area, where motion flow may be higher than in the rest of the camera field of view. Thus, if the determined pixel intensity gradient is above this droplet threshold, it may be determined that there is an interfering object such as a water droplet on the camera unit <NUM>.

<FIG> illustrates an example image of an equalized average norm image, where segmentation and classification have been applied to the obtained image frames. As seen in <FIG>, this image additionally shows two false positive areas, i.e. areas that incorrectly have been determined, or classified, to comprise interfering objects. However, these false interfering objects may be removed based on their average intensity. Their average intensity in the original image frame may be higher than any expected intensity. Thus, this may be higher than a false positive threshold.

After that it has been determined whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>, the at least one controller <NUM>, <NUM> is configured to control a subsequent action of the robotic work tool system <NUM> based on the determination. Accordingly, different actions may be taken by the robotic work tool system <NUM> depending on whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM> or not.

Accordingly, the present disclosure provides a robotic work tool system <NUM> that may detect interfering objects that cover and disturb the view of a camera unit <NUM> of a robotic work tool <NUM>. As these interfering objects may affect the ability to correctly detect and classify objects and surfaces within the surroundings of the robotic work tool <NUM>, the present disclosure makes it possible to prevent that the object detection accuracy of the robotic work tool <NUM> is degraded. If the object detection accuracy is degraded, safety risks may be posed to the robot work tool's environment and the robot work tool <NUM> itself. However, with the present disclosure these risks are eliminated, or at least reduced. Additionally, the present disclosure provides a way to avoid object detection degradation in an efficient way, which consumes a limited amount of processing powers. This may be especially suitable when using the at least one processor <NUM> within the robotic work tool <NUM> to perform the interfering object detection. The robotic work tool's processor capacity is limited compared to, e.g., a vehicle. Thus, the present disclosure further detects interfering objects on a camera unit using reduced processing power.

As previously described, the at least one controller <NUM>, <NUM> is configured to control a subsequent action of the robotic work tool system <NUM> based on the determination whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>. In some embodiments, the at least one controller <NUM>, <NUM> may be configured to control the subsequent action of the robotic work tool system <NUM> by transmitting a message to an output device <NUM>, <NUM>. The output device <NUM> may be located in the robotic work tool <NUM>, or the output device <NUM> may be located in a device <NUM> that is separate from the robotic work tool <NUM>. The transmitted message may comprise information about whether there is determined to be any interfering objects on the camera unit <NUM> of the robotic work tool <NUM> or not. The transmitted message may define, for example, reliable vs un-reliable image pixels within the obtained image frames. Alternatively, or additionally, the transmitted message may comprise an alarm warning a user of the robotic work tool system <NUM> that there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>.

Alternatively, or additionally, when it is determined that there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>, the at least one controller <NUM>, <NUM> may be configured to control the subsequent action of the robotic work tool system <NUM> by initiating a cleaning operation of the robotic work tool <NUM>. The cleaning operation may be initiated to remove any interfering object on the camera unit <NUM> of the robotic work tool <NUM>. The cleaning operation may, e.g., comprise an automatic cleaning operation of rinsing off the camera unit <NUM> and/or wiping the camera unit <NUM>. In some embodiments, the at least one controller <NUM>, <NUM> may further be configured to, after that a cleaning operation has been performed, perform a new determination whether there are any interfering objects on the camera unit <NUM>. In case it is determined that there still is an interfering object on the camera unit <NUM>, i.e. that an area of the camera unit <NUM> did not get cleaned by the automatic cleaning operation, and/or if the interfering object severely disturbs the operation of the robotic work tool <NUM>, then a message and/or alarm may be sent to an operation central, operator, or nearby user. This may be performed in order to call for human intervention, i.e. cleaning of the camera unit <NUM>.

Alternatively, or additionally, when it is determined that there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>, the at least one controller <NUM>, <NUM> may be configured to control the subsequent action of the robotic work tool system <NUM> by controlling a travel operation of the robotic work tool <NUM>. For example, the robotic work tool <NUM> may be controlled to stop moving, or the robotic work tool <NUM> may be controlled to travel to a cleaning station where it may be cleaned.

In some embodiments, the at least one controller <NUM>, <NUM> may be further configured to perform a basic feature extraction based on the determined pixel intensity gradients. For example, the at least one controller <NUM>, <NUM> may be configured to determine, based on the determined pixel intensity gradients, contours of an interfering object on the camera unit <NUM>. The at least one controller <NUM>, <NUM> may further be configured to determine a bounding box and an extended bounding box, where the two boxes share a same centre and aspect ratio, but where the extended box is larger by a certain percent in each dimension. <FIG> shows an example of an image frame where dirt contours have been applied. <FIG> also shows the concepts of bounding box and extended bounding box.

Furthermore, the at least one controller <NUM>, <NUM> may be configured to determine, based on the obtained image frames and the determination whether there is an interfering object on the camera unit <NUM>, a proportion of an image frame not occluded by any interfering objects. The proportion of an image frame not occluded by any interfering objects may also be referred to as a Clear Area Ratio (CAR) metric. The CAR metric is thus the proportion of the camera view that is not occluded by an interfering object, such as e.g. dirt. A CAR that equals zero means that there is no interfering object on the camera unit <NUM>, while a CAR that equals one means that the entire view of the camera unit <NUM> is covered by interfering objects.

Other features that may be extracted based on the determination whether there is an interfering object on the camera unit <NUM> are a Transparency Ratio (TR) and a Blurriness ratio (BR). These are now going to be described more in detail.

The TR may be determined by the at least one controller <NUM>, <NUM> by determining, an intensity of all pixels within the obtained image frames comprising an interfering object, and by determining, an intensity of all pixels within the obtained image frames not comprising an interfering object. Thereafter, the at least one controller <NUM>, <NUM> may be configured to compare the determined intensity for pixels comprising an interfering object with the determined intensity for pixels not comprising an interfering object, and, based on the comparison, a TR for any determined interfering object on the camera unit <NUM> of the robotic work tool <NUM> may be determined. The TR metric values range between <NUM> to <NUM>, where <NUM> means that the interfering object are fully opaque, while <NUM> means that the interfering object are fully transparent. Thus, the TR is a metric of the transparency of the interfering object on the camera unit <NUM> of the robotic work tool <NUM>.

The BR may be determined by determining, based on the determination whether there is an interfering object on the camera unit <NUM>, a blurriness of all pixels within the obtained image frames comprising an interfering object and by determining, based on the determination whether there is an interfering object on the camera unit <NUM>, a blurriness of all pixels within the obtained image frames not comprising an interfering object. Thereafter, the at least one controller <NUM>, <NUM> may be configured to compare the determined blurriness for pixels comprising an interfering object with the determined blurriness for pixels not comprising an interfering object, and, based on the comparison, a BR for any determined interfering object on the camera unit <NUM> of the robotic work tool <NUM> may be determined. The blurriness may be determined as the variance of a Laplacian filtered region of interest, e.g. the interfering objects. The BR metric values range between <NUM> to <NUM>, where <NUM> means that the interfering object are fully blurred, while <NUM> means that the sharpness of the objects in front of the interfering objects on the camera unit <NUM> is not being degraded by the interfering object.

Based on a determined proportion of the image frame not occluded by any interfering objects, the determined TR and the determined BR, the at least one controller <NUM>, <NUM> of the robotic work tool system <NUM> may be configured to determine an object detection reliability of the robotic work tool system <NUM>. The at least one controller <NUM>, <NUM> may thus be configured to estimate the system's capability to detect objects correctly in the presence of interfering objects on the camera, i.e. an Object Detection Resilience Metric (ODRM). The ODRM metric values range between <NUM> and <NUM>. When ODRM is <NUM>, it means that there is no object detection capability at all. When ODRM is <NUM>, it means that the camera unit <NUM> is clean and that there should be no degradation to the object detection of the robotic work tool system <NUM>. The smaller the ODRM is, the higher the risk of object detection degradation becomes. The ODRM may be used by system architects to create multiple robot functionality levels. For example, when the determined object detection reliability of the robotic work tool system <NUM> is below an object detection reliability threshold, the at least one controller <NUM>, <NUM> may be configured to control a subsequent action of the robotic work tool system <NUM> to avoid degraded operation of the robotic work tool (<NUM>) due to the low object detection reliability. If the ODRM is lower than the object detection reliability threshold, only larger objects may be detected with high confidence, while many smaller objects may be missed. Thus, at such occasions, the at least one controller <NUM>, <NUM> may be configured to control the robotic work tool <NUM> to, for example, stop moving, transmitting an alarm or initiating a cleaning operation. Thus, the at least one controller <NUM>, <NUM> may be configured to control the subsequent action of the robotic work tool system <NUM> to be any of the previously described actions.

In case a cleaning operation of the robotic work tool <NUM> has been performed, the at least one controller <NUM>, <NUM> may further be configured to receive an indication that any interfering object on the camera unit <NUM> of the robotic work tool <NUM> has been removed. Then, the at least one controller <NUM>, <NUM> may be configured to reset, based on the indication, the object detection reliability of the robotic work tool system <NUM>. In other embodiments, the at least one controller <NUM>, <NUM> may be configured to reset the object detection reliability of the robotic work tool system <NUM> when a certain time has elapsed or after a certain movement of the robotic work tool <NUM> and/or the camera unit <NUM> has been performed.

The at least one controller <NUM>, <NUM> may be configured to determine the object detection reliability of the robotic work tool system <NUM> by determining a weight for each of the determined proportion of the image frame not occluded by any interfering objects, the determined TR and the determined BR. Each of the weighted determined proportion of the image frame not occluded by any interfering objects, the weighted determined TR and the weighted determined BR may be added to a sum in order to determine the object detection reliability of the robotic work tool system <NUM>. Thus, another way of expressing the object detection reliability of the robotic work tool system <NUM> may be by the following formula; <MAT> where <NUM> ≤ α ≤ <NUM>, <NUM> ≤ β ≤ <NUM>, <NUM> ≤ γ < <NUM> and α + β + γ = <NUM>.

The weights α, β and γ may be determined, for example, by using a dataset of ground-truth images with known objects. These have been acquired using a known interfering object. Thus, multiple datasets of image frames of objects that were captured using a camera unit <NUM> with interfering objects with various occlusion patterns on them. Each dataset may comprise of multiple image frames of different objects and scenes that were captured with one occlusion pattern. The objects in those images may be manually annotated using bounding boxes and/or semantic segmentation. This annotation may be considered as the ground-truth. An object detection model in inference mode may be used in order to detect the objects in all datasets. For each dataset, a standard object detection quality metric Q, such as Mean Average Precision (mAP) may be determined. A loss function may be defined by Loss=Q^<NUM>-ORDM^<NUM>. The weights α, β and γ may thereafter be selected such that they minimize the error between the predictor ODRM and the standard object detection quality metric Q. The error minimization may be performed by using a standard regression algorithm, e.g. Multiple Linear Regression (MLR).

The robotic work tool system <NUM> presented herein provides a way of detecting, segmenting and analysing interfering objects, such as e.g. dirt, spots and smudges, on a camera unit <NUM> of a robotic work tool <NUM>. This allows the robotic work tool system <NUM> to control subsequent actions based on this and for example, reduce functionality and movement, up to a full stop, of the robotic work tool <NUM> if necessary. This allows any safety risks to the robot work tool <NUM> and its environment to be eliminated, or at least reduced. Thus, a robotic work tool system <NUM> is provided that improves the operation of a robotic work tool <NUM>.

According to a second aspect, there is provided a method implemented in the robotic work tool system <NUM> according to the first aspect. The method will be described with reference to <FIG>.

In one embodiment, the method <NUM> may be performed by a robotic work tool system <NUM> for determining whether there is an interfering object on a camera unit <NUM> of a robotic work tool <NUM>. As illustrated in <FIG>, the method <NUM> starts with step <NUM> of obtaining a plurality of image frames reflecting the view of the camera unit <NUM> of the robotic work tool <NUM>. Each of the obtained image frames is associated with a different position and/or angle compared to the other obtained plurality of image frames. The method <NUM> further comprises step <NUM> of determining, based on the obtained image frames, pixel intensity gradients within the view of the camera unit <NUM> of the robotic work tool <NUM> and step <NUM> of comparing each of the determined pixel intensity gradients against at least one threshold. The method <NUM> further comprises step <NUM> of determining, based on the comparison, whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>. Thereafter, the method <NUM> comprises step <NUM> of controlling a subsequent action of the robotic work tool system <NUM> based on the determination whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>.

An example of a flow according to the method <NUM> is illustrated in <FIG>. In some embodiments, the plurality of image frames reflecting the view of the camera unit <NUM> of the robotic work tool <NUM> may be obtained with fixed predefined intervals. The image frames may be obtained from a video stream from the robot work tool's camera unit <NUM> with decimated frame rate, see step <NUM> and <NUM> of <FIG>. The frame rate decimation may make it possible to use images from diverse locations and shooting angles. As seen in <FIG>, the method <NUM> may, according to some embodiments, comprise step <NUM> and <NUM> of applying vertical and horizontal Sobel filtering.

In some embodiments, as illustrated in <FIG>, the step <NUM> of determining pixel intensity gradients within the view of the camera unit <NUM> may comprise step <NUM> of determining a 2D pixel intensity gradient image for each of the plurality of obtained image frame and step <NUM> of determining a pixel-wise norm of each 2D pixel intensity gradient image. It may further comprise step <NUM> of averaging the determined pixel-wise norm of each 2D pixel intensity gradient image into a single image frame of average norms of pixel intensity gradients.

The example in <FIG> further comprises step <NUM> of equalizing the average norm image using K-means clustering. Thereafter, the frames may be compared against thresholds, step <NUM>, and a binary image of an interfering object, such as dirt, may be created, step <NUM>.

In some embodiments, the method <NUM> may further comprise the steps of performing a basic feature extraction based on the determined pixel intensity gradients, as illustrated in <FIG>. For example, the method may comprise step <NUM> of determining, based on the determined pixel intensity gradients, contours of an interfering object on the camera unit <NUM>. The contours of the interfering object, e.g. dirt, may be determined based on step <NUM> of receiving binary images of dirt, which may have been determined in accordance with the method <NUM> illustrated in <FIG>. Thereafter the method <NUM> may comprise step <NUM> of removing clusters with high average brightness, i.e. false positive interfering objects as illustrated in <FIG>, and step <NUM> of detecting the edges of the interfering object.

Other steps of performing basic feature extraction based on the determined pixel intensity gradients are also illustrated in <FIG>, for example, steps <NUM> and <NUM> of determining a bounding box and an extended bounding box. The two boxes share a same centre and aspect ratio, but where the extended box is larger by a certain percent in each dimension.

Further steps of the basic feature extraction based on the determined pixel intensity gradients may be step <NUM> of determining, based on the obtained image frames and the determination whether there is an interfering object on the camera unit, a proportion of an image frame not occluded by any interfering objects. As previously described, this metric may also be referred to as the CAR metric. The CAR may be determined, for example, by obtaining binary images of dirt, step <NUM> and the calculating dirt spots areas, step <NUM>.

In some embodiments, the method <NUM> may further comprise step <NUM> of determining a TR, as illustrated in <FIG>. The TR may be determined by step <NUM> of determining, based on the determination whether there is an interfering object on the camera unit <NUM>, an intensity of all pixels within the obtained image frames comprising an interfering object. The method <NUM> may further comprise step <NUM> of determining, based on the determination whether there is an interfering object on the camera unit, an intensity of all pixels within the obtained image frames not comprising an interfering object. Thereafter, the TR may be determined, step <NUM>, by comparing the determined intensity for pixels comprising an interfering object with the determined intensity for pixels not comprising an interfering object and determining, based on the comparison, a TR for any determined interfering object on the camera unit <NUM> of the robotic work tool <NUM>.

In some embodiments, the method <NUM> may further comprise step <NUM> of determining a BR, as illustrated in <FIG>. This may comprise step <NUM> of determining, based on the determination whether there is an interfering object on the camera unit <NUM>, a blurriness of all pixels within the obtained image frames comprising an interfering object. It may further comprise step of <NUM> of determining, based on the determination whether there is an interfering object on the camera unit <NUM>, a blurriness of all pixels within the obtained image frames not comprising an interfering object. Thereafter, the BR may be determined, step <NUM>, by comparing the determined blurriness for pixels comprising an interfering object with the determined blurriness for pixels not comprising an interfering object; and determining, based on the comparison, the BR for any determined interfering object on the camera unit <NUM> of the robotic work tool <NUM>.

In some embodiments, the method <NUM> may further comprise determining, based on the determined proportion of the image frame not occluded by any interfering objects, the determined TR and the determined BR, an object detection reliability of the robotic work tool system <NUM>. The object detection reliability may be determined by determining a weight for each of the determined proportion of the image frame not occluded by any interfering objects, the determined TR and the determined BR. Thereafter, each of the weighted determined proportion of the image frame not occluded by any interfering objects, the weighted determined TR and the weighted determined BR may be added to a sum in order to determine the object detection reliability of the robotic work tool system <NUM>.

In some embodiments, when the determined object detection reliability of the robotic work tool system <NUM> is below an object detection reliability threshold, the method <NUM> may further comprise controlling a subsequent action of the robotic work tool system <NUM> to avoid degraded operation of the robotic work tool <NUM> due to the low object detection reliability.

In some embodiments, the method <NUM> may further comprise receiving an indication that any interfering object on the camera unit <NUM> of the robotic work tool <NUM> has been removed; and resetting, based on the indication, the object detection reliability of the robotic work tool system <NUM>.

In some embodiments, the step <NUM> of controlling the subsequent action of the robotic work tool system <NUM> may comprise transmitting a message to an output device <NUM>, <NUM>. The message may define reliable vs un-reliable image pixels within the obtained image frames.

In some embodiments, when it is determined that there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>, the step of <NUM> controlling the subsequent action of the robotic work tool system <NUM> may comprise initiating a cleaning operation of the robotic work tool <NUM>. The cleaning operation may be initiated to remove any interfering object on the camera unit <NUM> of the robotic work tool <NUM>.

In some embodiments, when it is determined that there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM>, the step <NUM> of controlling the subsequent action of the robotic work tool system <NUM> may comprise controlling a travel operation of the robotic work tool <NUM>.

In some embodiments, the plurality of image frames may be obtained from the camera unit <NUM> while at least one of the camera unit <NUM> and the robotic work tool <NUM> is moving.

In some embodiments, the method <NUM> may further comprise determining, based on the determined pixel intensity gradients, contours of an interfering object on the camera unit <NUM>.

In some embodiments, said at least one threshold may comprise a lower threshold and a higher threshold. Then, the step <NUM> of determining whether there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM> based on the comparison may comprise determining that there is an interfering object on the camera unit <NUM> of the robotic work tool <NUM> if the determined pixel intensity gradient is below the lower threshold. The method step may further comprise determining that there is a non-disturbing interfering object on the camera unit <NUM> of the robotic work tool <NUM> if the determined pixel intensity gradient is above the lower threshold and below the higher threshold. The method step may further comprise determining that there is no interfering object on the camera unit <NUM> of the robotic work tool <NUM> if the determined pixel intensity gradient is above the higher threshold.

In some embodiments, the position and/or angle associated with each of the obtained image frames may be obtained from at least one position sensor <NUM> of the robotic work tool <NUM>.

In some embodiments, the robotic work tool system <NUM> may comprise the robotic work tool <NUM> comprising the camera unit <NUM>.

In some embodiments, the robotic work tool <NUM> may comprise a robot apparatus configured to perform a work task autonomously.

With the proposed method <NUM>, it may be possible to detect interfering objects that cover and disturb the view of a camera unit <NUM> of a robotic work tool <NUM>. As these interfering objects may affect the ability to correctly detect and classify objects and surfaces within the surroundings of the robotic work tool <NUM>, it is possible to prevent that the object detection accuracy of the robotic work tool <NUM> is degraded. If the object detection accuracy is degraded, that poses safety risks to both the robot work tool's environment and to the robot work tool <NUM> itself. With the present disclosure these risks are eliminated, or at least reduced. Additionally, the method <NUM> provides a solution to detect interfering objects on a camera unit <NUM> of the robotic work tool <NUM> in an efficient way, which consumes a limited amount of processing powers. Thus, the present disclosure detects interfering objects on a camera unit <NUM> using reduced processing power.

<FIG> shows a schematic view of a computer-readable medium which is configured to carry instructions <NUM> that when loaded into a controller, such as a processor, execute a method or procedure according to the embodiments disclosed above. The computer-readable medium <NUM> is in this embodiment a data disc <NUM>. In one embodiment, the data disc <NUM> is a magnetic data storage disc. The data disc <NUM> is arranged to be connected to or within and read by a reading device, for loading the instructions into the controller. One such example of a reading device in combination with one (or several) data disc(s) <NUM> is a hard drive. It should be noted that the computer-readable medium can also be other mediums such as compact discs, digital video discs, flash memories or other memory technologies commonly used. In such an embodiment, the data disc <NUM> is one type of a tangible computer-readable medium <NUM>.

The instructions <NUM> may also be downloaded to a computer data reading device, such as the controller <NUM>,<NUM> or other device capable of reading computer coded data on a computer-readable medium, by comprising the instructions <NUM> in a computer-readable signal which is transmitted via a wireless (or wired) interface (for example via the Internet) to the computer data reading device for loading the instructions <NUM> into a controller. In such an embodiment, the computer-readable signal is one type of a non-tangible computer-readable medium <NUM>.

Claim 1:
A robotic work tool system (<NUM>) configured to determine whether there is any disturbing and interfering object, such as dirt, on a camera unit (<NUM>) of a robotic lawn mower (<NUM>) which may degrade the operation of the robotic lawn mower (<NUM>),
wherein the robotic lawn mower (<NUM>) is configured to perform a work task autonomously, and wherein the robotic work tool system (<NUM>) comprises at least one controller (<NUM>,<NUM>) configured to:
- obtain a plurality of image frames reflecting the view of the camera unit (<NUM>) of the robotic lawn mower (<NUM>), wherein each of the obtained image frames is associated with a different position and/or angle compared to the other obtained plurality of image frames;
- determine, based on the obtained image frames, pixel intensity gradients within the view of the camera unit (<NUM>);
- compare each of the determined pixel intensity gradients against at least one threshold; determine, based on the comparison, whether there is an interfering object on the camera unit (<NUM>) of the robotic lawn mower (<NUM>); and
- control a subsequent action of the robotic work tool system (<NUM>) based on the determination whether there is an interfering object on the camera unit (<NUM>) of the robotic lawn mower (<NUM>).