Patent Description:
Machines may be used to perform variety of tasks at a worksite. For example, machines may be used to excavate, move, shape, contour, and/or remove material present at the worksite, such as gravel, concrete, asphalt, soil, and/or other materials. These machines can include a bucket used to collect such materials, and the bucket can include a set of GET, such as teeth, to loosen the material. GET can also include shrouds attached to the bucket between teeth to protect the edge of the bucket. Over time, the GET wear and diminish in size reducing their effectiveness making it more difficult for the bucket to collect worksite material. GET can also break from the bucket. When a GET break goes undetected, the GET can mix with the worksite material and can cause damage to downstream processing equipment such as crushers or pulverizers. Work machines may utilize wear detection systems to identify worn or broken GET before damage to downstream equipment occurs.

An attempt to provide a wear detection system is described in <CIT>. The '<NUM> Publication describes a system and tool for monitoring ground-engaging products for earth working equipment that monitors characteristics of those products such as part identification, presence, condition, usage, and/or performance. While the monitoring tool of the '<NUM> Publication can include a light detection and ranging ("LiDAR") sensor, the disclosed LiDAR sensor are static and do not employ adaptive scanning technology that allows for scanning of objections with differing fields of view or resolution. Moreover, the monitoring tools described in the' <NUM> Publication merely offer coarse object detection with accuracy and precision of no better than approximately two centimeters.

Reliance on low-resolution and non-adaptive scanning LiDAR sensors presents disadvantages for detecting wear in GET. First, resolution of approximately two centimeters does provide effective or precise measurement of GET as measurements on the order of magnitude of millimeters (or lower) is preferred. Moreover, non-adaptive scanning LiDAR requires the LiDAR sensor to maintain a fixed field of view, preventing adjustment of image capture and/or change in resolution of captured image information. As a result, the system described in the '<NUM> Publication can be prone to errors in detecting GET wear or loss and is unable to provide measurements at a desirable level of precision. The systems and methods described herein are directed to addressing one or more of these concerns.

<CIT> discloses a computer processor implemented method and system for monitoring a condition associated with operating heavy equipment is disclosed. The method involves receiving a plurality of images at an interface of an embedded processor disposed on the heavy equipment, the images providing a view of at least an operating implement of the heavy equipment. The method also involves processing each of the plurality of images using a first neural network implemented on the embedded processor, the first neural network having been previously trained to identify regions of interest within the image. Each region of interest has an associated designation as at least one of a critical region suitable for extraction of critical operating condition information required for operation of the heavy equipment, and a non-critical region suitable for extraction of non-critical operating condition information associated with the operation of the heavy equipment. The method further involves causing the embedded processor to initiate further processing of image data associated with critical regions to generate local output operable to alert an operator of the heavy equipment of the associated critical operating condition. The method also involves transmitting image data associated with non-critical regions to a remote processor, the remote processor being operably configured for further processing of the image data and to generate output signals representing results of the further processing. The method further involves receiving the output signals generated by the remote processor at one of the embedded processor or another processor associated with a heavy equipment operations worksite, the output signals being presentable via an electronic user interface based at least in part on the output signals to indicate the results of the further processing.

According to a first aspect, a method for detecting wear or loss of a GET includes receiving first imaging data from one or more sensors associated with a work machine. The first imaging data comprises data related to at least one GET of the work machine. The first imaging data identifying the at least one GET at a first resolution. The method identifies a region of interest within the first imaging data that includes the data related to the at least one GET. The method controls a LiDAR sensor to capture second imaging data corresponding to the identified region of interest. The second imaging data identifies the at least one GET at a second resolution higher than the first resolution. The method generates a three-dimensional point cloud of the at least one GET based on the second imaging data and determines a wear level or loss for the at least one GET based on the three-dimensional point cloud.

According to a further aspect, a GET wear detection system includes one or more sensors associated with a work machine, one or more processors, and non-transitory computer readable media storing executable instructions. At least one of the one or more sensors is a LiDAR sensor. The executable instructions when executed by the processor cause the processor to perform operations including receiving first imaging data from the one or more sensors. The first imaging data comprises data related to at least one GET of the work machine. The first imaging data identifies the GET at a first resolution. The operations also include identifying a region of interest within the first imaging data including the data related to the at least one GET and controlling the LiDAR sensor to capture second imaging data corresponding to the identified region of interest at a second resolution that is higher than the first resolution. The operations also include generating a three-dimensional point cloud of the at least one GET based on the second imaging data and determining a wear level or loss for the at least one GET based on the three-dimensional point cloud.

According to another aspect, a work machine includes a bucket comprising at least one GET, a stereoscopic camera comprising a left image sensor and a right image sensor, a LiDAR sensor, one or more processors, and non-transitory computer readable media storing executable instructions that when executed by the one or more processors cause the one or more processors to perform operations. The operations include receiving a left image of the at least one GET captured by the left image sensor and receiving a right image of the at least one GET captured by the right image sensor. The operations also include generating a dense stereo disparity map based on the left image and the right image and identifying a region of interest based on the dense stereo disparity map. The operations further include controlling the LiDAR sensor to capture imaging data based on the identified region of interest. The imaging data captures the at least one GET at a resolution higher than either the left image or the right image. The operations further include generating a three-dimensional point cloud of the at least one GET based on the imaging data and determining a wear level or loss for the at least one GET based on the three-dimensional point cloud.

In the figures, the left-most digit of a reference number identifies the figure in which the reference number first appears.

The present disclosure is generally directed to systems and methods for detecting wear of components of a work machine in an environment, such as a worksite, using one or more sensors. The one or more sensors can include imaging sensors (that could be part of a stereoscopic camera or "stereo camera") and LiDAR sensors capable of capturing imaging data associated with the components. The imaging data can include, but is not limited to, video, images, or LiDAR imaging data. The imaging data is analyzed by a wear detection computer system associated with the work machine -aspects of which may be disposed on the work machine, within the stereo camera, within the LiDAR sensor, or external to these components-to detect wear of the component. The component can be one or more GET of a bucket of the work machine, as one example. The wear detection computer system analyzes the imaging data in one or more phases. In an example first phase, the wear detection computer system receives lower resolution imaging data and uses it to detect a region of interest that includes the analyzed components (e.g., GET). in an example second phase, the wear detection computer system uses the location of the region of interest to control a LiDAR sensor to capture a higher-resolution LiDAR imaging data of the analyzed components. A three-dimensional point cloud of the higher-resolution imaging data is analyzed to determine a wear level or loss of the analyzed components. Using this technique, the wear detection computer system can direct the LiDAR sensor to capture images of the analyzed components at millimeter level of precision (e.g., the LiDAR sensor receives a data "hit" about every <NUM>-<NUM>).

<FIG> is a block diagram depicting a schematic of an example work machine <NUM> including an example wear detection computer system <NUM>. While <FIG> depicts work machine <NUM> as a hydraulic mining shovel, in other examples, work machine <NUM> can include any machine that moves, sculpts, digs, or removes material such as soil, rock, or minerals. As shown in <FIG>, work machine <NUM> can include a bucket <NUM> attached to arm <NUM>. Bucket <NUM> can include one or more ground engaging tools (GET) <NUM>, such as teeth, that assist work machine <NUM> in loosening material. While the examples provided in this disclosure typically refer to GET <NUM> as teeth, other types of GET are contemplated to be within the scope of the embodiments provided by this disclosure. For example, GET can include lip shrouds, edge guards, adapters, ripper protectors, cutting edges, sidebar protectors, tips, or any other tool associated with a work machine that wear over time due to friction with worksite material.

Work machine <NUM> can also include one or more sensors having respective fields of view such as LiDAR sensor <NUM> having field-of-view <NUM> and stereo camera <NUM> having field-of-view <NUM>. Both field-of-view <NUM> and field-of-view <NUM> are directed to bucket <NUM> and GET <NUM>. As shown in <FIG>, field-of-view <NUM> and field-of-view <NUM> are substantially overlapping.

LiDAR sensor <NUM> can include an adaptive scanning LiDAR sensor, i.e., a LiDAR sensor for which its resolution and field of view can be commanded, controlled, and configured. For example, LiDAR sensor <NUM> can include an AEYE 4Sight M™. In some embodiments, field-of-view <NUM> starts with a baseline of <NUM> degrees by <NUM> degrees (representing a "low" resolution range scan) which can then be adjusted by <NUM> degrees to high definition region of interest spanning <NUM> degrees, but other fields of view and angular resolutions may be present in other embodiments. LiDAR sensor <NUM> can be configured to collect as many as <NUM>,<NUM> points per square degree at a frequency of <NUM>. The precision of LiDAR sensor <NUM> is a function of the angular resolution of field-of-view <NUM> and the distance between LiDAR sensor <NUM> and GET <NUM>. As an example, when GET <NUM> is approximately six meters from LiDAR sensor <NUM> and field-of-view <NUM> is configured as <NUM> degrees by <NUM> degrees, a <NUM>,<NUM> points-per-square degree scan would yield LiDAR hits within an captured rectangle of approximately <NUM> meters by <NUM> meters. By refocusing the field of view, a LiDAR hit can register <NUM> millimeters in the horizontal and vertical directions. While the above describes one example LiDAR sensor <NUM>, different LiDAR sensors capable of adaptive scanning can be used in various embodiments.

Stereo camera <NUM> includes a left image sensor and a right image sensor that are spaced apart as to capture a stereo image of objects within field-of-view <NUM>, such as bucket <NUM> and GET <NUM>. In some embodiments, the left image sensor and the right image sensor capture monochromatic images. Stereo camera <NUM> can also include a color image sensor to capture color images of objects within field-of-view <NUM>. In some embodiments, camera <NUM> outputs digital images or work machine <NUM> may include an analog to digital converter disposed between camera <NUM> and wear detection computer system <NUM> to covert analog images to digital images before they are received by wear detection computer system <NUM>.

The one or more sensors of work machine <NUM>, such as LiDAR sensor <NUM> and camera <NUM>, can include a lens cleaning device to remove debris, fog, or other obstructions from the surface (or screen) of the lenses of the one or more sensors in some embodiments. The lens cleaning device can include, for example, a nozzle for emitting compressed air, washer solvent, or washer antifreeze solvent. The lens cleaning device can also include a moving wiper that is configured to contact and wipe the surface of the lens to push debris or other obstructions away from the lens surface. In some embodiments, the cover of the lenses of the one or more sensors may include an actuator that rotates the lens screen (for cylindrical lens screens) or slides the lens screen (for flat lens screens) so that it contacts one or more wipers to remove debris from the screen.

As work machine <NUM> operates within a worksite, it may move arm <NUM> to position bucket <NUM> to move or dig material within the worksite as part of a dig-dump cycle. As work machine <NUM> positions bucket <NUM> through the dig-dump cycle, bucket <NUM> may move in and out of field-of-view <NUM> and field-of-view <NUM>. LiDAR sensor <NUM> and camera <NUM> may be positioned so that they have an unobstructed view of GET <NUM> during the dig-dump cycle. For example, LiDAR sensor <NUM> and camera <NUM> may be positioned on work machine <NUM> so that bucket <NUM> and GET <NUM> are visible at the moment bucket <NUM> empties material within the dig-dump cycle. As another example, LiDAR sensor <NUM> and camera <NUM> may be positioned so that bucket <NUM> enters its field-of-view when arm <NUM> is fully extended or fully contracted within the dig-dump cycle. As explained below with respect to <FIG>, the position of LiDAR sensor <NUM> and camera <NUM> (and accordingly field-of-view <NUM> and field-of-view <NUM>) may vary depending on the type of work machine <NUM> and specifics related to its worksite.

According to some embodiments, work machine <NUM> includes an operator control panel <NUM>. Operator control panel <NUM> can include a display <NUM> which produces output for an operator of work machine <NUM> so that the operator can receive status or alarms related to wear detection computer system <NUM>. Display <NUM> can include a liquid crystal display (LCD), a light emitting diode display (LED), cathode ray tube (CRT) display, or other type of display known in the art. In some examples, display <NUM> includes audio output such as speakers or ports for headphones or peripheral speakers. Display <NUM> can also include audio input devices such as microphone or ports for peripheral microphones. Display <NUM> includes a touch-sensitive display screen in some embodiments, which also acts as an input device.

In some embodiments, operator control panel <NUM> also includes a keyboard <NUM>. Keyboard <NUM> provides input capability to wear detection computer system <NUM>. Keyboard <NUM> includes a plurality of keys allowing the operator of work machine <NUM> to provide input to wear detection computer system <NUM>. For example, an operator may depress the keys of keyboard <NUM> to select or enter the type of work machine <NUM>, bucket <NUM>, and/or GET <NUM> according to examples of the present disclosure. Keyboard <NUM> can be non-virtual (e.g., containing physically depressible keys) or keyboard <NUM> can be a virtual keyboard shown on a touch-sensitive embodiment of display <NUM>.

As shown in <FIG>, wear detection computer system <NUM> includes a one or more processors <NUM>. Processor(s) <NUM> can include one or more of a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), some combination of CPU, GPU, or FPGA, or any other type of processing unit. Processor(s) <NUM> may have numerous arithmetic logic units (ALUs) that perform arithmetic and logical operations, as well as one or more control units (CUs) that extract instructions and stored content from processor cache memory, and then executes the instructions by calling on the ALUs, as necessary, during program execution. Processor(s) <NUM> may also be responsible for executing drivers and other computer-executable instructions for applications, routines, or processes stored in memory <NUM>, which can be associated with common types of volatile (RAM) and/or nonvolatile (ROM) memory.

Wear detection computer system <NUM> also includes a memory <NUM>. Memory <NUM> can include system memory, which may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. Memory <NUM> can further include non-transitory computer-readable media, such as volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. System memory, removable storage, and non-removable storage are all examples of non-transitory computer-readable media. Examples of non-transitory computer-readable media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium which can be used to store the desired information and which can be accessed by wear detection computer system <NUM>.

Memory <NUM> stores data, including computer-executable instructions, for wear detection computer system <NUM> as described herein. For example, memory <NUM> can store one or more components of wear detection computer system <NUM> such as a physical parameter library <NUM>, an image analyzer <NUM>, a wear analyzer <NUM>, and an alert manager <NUM>. Memory <NUM> can also store additional components, modules, or other code executable by processor(s) <NUM> to enable operation of wear detection computer system <NUM>. For example, memory <NUM> can include code related to input/output functions, software drivers, operating systems, or other components.

According to some embodiments, aspects of wear detection computer system <NUM> may be disposed within camera <NUM>. For example, camera <NUM> may include one or more of processor(s) <NUM> and/or memory <NUM>. Similarly, aspects of wear detection computer system <NUM> may be disposed within LiDAR sensor <NUM>. In addition, or alternatively, aspects of wear detection computer system <NUM> may be disposed on work machine <NUM> and outside of LiDAR sensor <NUM> or camera <NUM>.

Physical parameter library <NUM> includes physical parameter sets related to work machine <NUM>, bucket <NUM>, GET <NUM>, LiDAR sensor <NUM> and/or camera <NUM>. For example, physical parameter library <NUM> can include measurement data related to the size of bucket <NUM>, shape of bucket <NUM>, size of GET <NUM>, shape of GET <NUM>, and the spatial relationship between GET <NUM> and bucket <NUM>, and/or the spatial relationship between LiDAR sensor <NUM> and camera <NUM>, as just some examples. Physical parameter library <NUM> can also include parameters related to the size and shape of GET <NUM> in a new or unworn state and parameters related to the size and shape of GET <NUM> when they have reached maximum wear.

Physical parameter library <NUM> can also include templates or reference images related to the combination of bucket <NUM> and GET <NUM> (e.g., a bucket-tool template). For example, for work machine <NUM>, one of the templates stored in physical parameter library <NUM> can include an image of bucket <NUM> with GET <NUM> as bucket <NUM> is expected to be positioned within field-of-view <NUM> and field-of-view <NUM>. The bucket-tool templates can represent GET <NUM> that are unworn (e.g., unworn or expected edges) or GET <NUM> that have reached maximum wear (e.g., a threshold edge). Physical parameter library <NUM> can also include other information related to the wear of GET <NUM> to assist wear analyzer <NUM> in determining when GET have worn to the point of needing replacement. Wear data related to GET <NUM> can be in the form of actual measurement (e.g., metric or imperial dimensions) or in the form of pixel values.

As another example, physical parameter library <NUM> can include CAD-based models of GET <NUM>. The CAD-based models can be models of reference GET <NUM> developed using computer-aided design programs such as AutoCAD®, Autodesk®, SolidWorks®, or other well-known CAD program. The CAD-based models can be used by wear detection computer system <NUM> as reference points to compare observed GET <NUM> sizes and shapes to a model, standard, or unworn GET of the same type to determine wear or loss of GET <NUM>.

Physical parameter library <NUM> can include multiple physical parameter sets where each physical parameter set corresponds to a work machine, bucket, GET, or a combination of these. During operation, an operator may use operator control panel <NUM> to select a physical parameter set from physical parameter library <NUM> matching bucket <NUM> and GET <NUM>, or work machine <NUM>. For example, if the work machine <NUM> is a hydraulic mining shovel having a model number "6015B," the operator may use operator control panel <NUM> to input the model number "6015B," and wear detection computer system <NUM> may load into memory <NUM> a physical parameter set corresponding to a model 6015B hydraulic mining shovel from physical parameter library <NUM>. In some examples, a list of templates available in physical parameter library <NUM> can be shown on display <NUM> upon a power-up or reset operation of wear detection computer system <NUM>, and an operator may select one of the physical parameter sets from the list for operation depending on the model number of work machine <NUM>, bucket type of bucket <NUM>, or type of GET <NUM>.

In some embodiments, the operator may position bucket <NUM> and GET <NUM> within field-of-view <NUM> of camera <NUM> at the beginning of a work shift and cause wear detection computer system <NUM> to capture an image of bucket <NUM> and GET <NUM> using an input on operator control panel <NUM>. Wear detection computer system <NUM> may then perform an image matching process to match bucket <NUM> and GET <NUM> with a physical parameter set and configure itself for the wear detection and image processing processes disclosed herein based on the matching physical parameter set. In some embodiments, wear detection computer system <NUM> may use LiDAR sensor <NUM> and field-of-view <NUM> for this configuration process instead of camera <NUM> and field-of-view <NUM>.

Image analyzer <NUM> can be configured to analyze imaging data captured by either LiDAR sensor <NUM> or camera <NUM> to identify GET <NUM> within field-of-view <NUM> and field-of-view <NUM> and to measure wear of GET <NUM> based on processing of that imaging data. For example, image analyzer <NUM> can receive stereoscopic images from camera <NUM> in the form of left rectified images (captured by the left image sensor of camera <NUM>) and a right rectified image (captured by the right image sensor of camera <NUM>). Image analyzer <NUM> may perform various computer vision techniques on the left rectified image and the right rectified image to identify or determine a region of interest corresponding to GET <NUM>. As another example, image analyzer <NUM> may receive imaging data captured by LiDAR sensor <NUM> which can be used to identify a region of interest corresponding to GET <NUM>. In the disclosed embodiments, image analyzer <NUM> receives data from LiDAR sensor <NUM> to determine wear or loss of GET <NUM>, as described in more detail below.

Image analyzer <NUM> processes two sets of the imaging data when detecting wear or loss of GET <NUM>. The first set of imaging data is captured to identify a region of interest within field-of-view <NUM> or field-of-view <NUM>. The region of interest corresponds to the relative location of GET <NUM> within field-of-view <NUM> or field-of-view <NUM>. The first set of imaging data-for detecting the region of interest-is a broad and lower resolution imaging data capture intended to locate a general region of interest for GET <NUM> and may be referred to as a "coarse scan. " In some embodiments, the first set of imaging data can be captured using camera <NUM>, and image analyzer <NUM> determines the region of interest using computer vision or machine learning techniques. In other embodiments, the first set of imaging data can be captured using LiDAR sensor <NUM> at a first, lower resolution (e.g., <NUM> degrees by <NUM> degrees) that is relatively wide. In some implementations, image analyzer <NUM> receives the first set of imaging data from both LiDAR sensor <NUM> and camera <NUM>.

When image analyzer <NUM> identifies a region of interest corresponding to GET <NUM>, it then controls LiDAR sensor <NUM> to focus on the specific region of interest to perform a higher-resolution scan, or "fine scan. " For example, image analyzer <NUM> may communicate with the application programming interface (API) of LiDAR sensor <NUM> to command it to change field-of-view <NUM> to become narrower with a focus on the identified region of interest. LiDAR sensor <NUM> then performs another scan of GET <NUM> to collect a second set of imaging data. The second set of imaging data-having been captured by LiDAR sensor <NUM> with a narrower field-of-view <NUM>-will be of higher resolution than the first imaging data captured by either LiDAR sensor <NUM> (when set with a wide field of view) or camera <NUM>.

In one embodiment, after receiving the first imaging data (i.e., the lower-resolution imaging data for determining the region of interest) image analyzer <NUM> creates a dense stereo disparity map based on the left rectified image and the right rectified image. Image analyzer may segment the dense stereo disparity map to identify the region of interest. In addition, image analyzer <NUM> may also create a three-dimensional point cloud based on the dense stereo disparity map and may segment the three-dimensional point cloud to identify the region of interest.

In addition to computer vision techniques, or as an alternative to using computer vision techniques, image analyzer <NUM> can also employ deep learning or machine learning techniques to identify regions of interest within left rectified images and/or right rectified images captured by camera <NUM>. For example, image analyzer <NUM> may use a deep learning GET detection algorithm that employs a neural network that has been trained to identify regions of interest based on a corpus of images where individual GET, groups of GET, or GET and bucket combinations have been labeled. Image analyzer <NUM> may also use a deep learning GET-location algorithm that employs a neural network that has been trained to locate GET within an image. The GET-location algorithm can be trained using corpus of images where individual GET have been labeled. Once the GET-location algorithm identifies individual GET within an image, it outputs the corresponding location for the GET. For example, the GET-location algorithm can output a pixel location or a bounding box output related to the location of the GET.

As noted above, once image analyzer <NUM> identifies the region of interest including GET <NUM>, it commands and controls LiDAR <NUM> to focus field-of-view <NUM> on the region of interest. In some embodiments, image analyzer <NUM> uses spatial relationship data between LiDAR sensor <NUM> and camera <NUM> to command LiDAR sensor <NUM> to alter field-of-view <NUM> on the region of interest. Once LiDAR sensor <NUM> receives commands to change its field-of-view, it may alter the configuration of its MEMS (micro-electromechanical system) mirrors to narrow field-of-view <NUM> to capture higher-resolution imaging data related to GET <NUM>.

From the captured higher-resolution imaging data, image analyzer <NUM> can create a three-dimensional point cloud corresponding to GET <NUM>. Each point in the three-dimensional point cloud corresponds to a LiDAR "hit" or detection point captured by LiDAR sensor <NUM>. In some embodiments, the real-life distance between the points can be as small as <NUM> millimeter. In embodiments with sufficiently high resolution (i.e., where the real-life distance between points is less than approximately <NUM>), image analyzer <NUM> communicates the three-dimensional point cloud data to wear analyzer <NUM> for wear detection analysis. In other embodiments, image analyzer <NUM> may perform additional processing of the three-dimensional point cloud data to further refine it for wear analysis.

For example, in some embodiments, image analyzer <NUM> converts the three-dimensional point cloud to a dense mesh surface. Image analyzer <NUM> may further convert the dense mesh surface to a sparse mesh surface before communicating the GET imaging data to wear analyzer <NUM>. Conversion from a three-dimensional point cloud, to a dense mesh surface, then to a sparse mesh surface may be desirable to reduce computational expenditure when comparing the imaging data captured by LiDAR sensor <NUM> to a CAD-based GET model. Conversion from a three-dimensional point cloud, to a dense mesh surface, then to a sparse mesh surface can also filter out noise that may be present in the imaging data due to oversampling.

In some embodiments, wear analyzer <NUM> fuses the lower-resolution, first-received imaging data from camera <NUM> with the higher-resolution data, second-received imaging data received from LiDAR <NUM> to gain confidence in the observed measurement of GET <NUM>. In such embodiments, image analyzer <NUM> performs additional processing on the left image and right image captured by camera <NUM>. For example, once image analyzer <NUM> identifies the regions of interest it can further process them to create a left-edge digital image corresponding to the left rectified image and a right-edge digital image corresponding to the right rectified image. Image analyzer <NUM> may employ gradient magnitude search-based edge detection, but other edge detection techniques employed within the field of computer vision (e.g., zero-crossing based edge detection techniques) could be employed in other embodiments to create the left-edge digital image and the right-edge digital image.

In some examples, image analyzer <NUM> may refine edge estimates of GET <NUM> and/or identify individual GET <NUM> by using an expected location of GET <NUM> within the captured image. For example, image analyzer <NUM> may know the expected position of GET <NUM> relative to bucket <NUM> based on the physical parameter set stored in physical parameter library <NUM> corresponding to the type of bucket <NUM> and GET <NUM> in use. Using this information, image analyzer <NUM> can go to the expected location in selected image and capture a pixel region proximate to the teeth. The pixel region can then be used to further identify the tooth based on computer vision techniques such as application of a convolution filter, segmentation analysis, edge detection, or pixel strength/darkness analysis within the pixel region. In some embodiments, image analyzer <NUM> may use an individual tooth template to apply to the pixel region to further refine the location of the tooth using computer vision techniques. Image analyzer <NUM> may further refine edges using dynamic programming techniques. Dynamic programming techniques can include smoothing based on the strength of the edge, whether the edge is close to a hole or region of uncertainty in the dense stereo disparity map, or other edge detection optimization techniques. Image analyzer <NUM> can also use the output of the GET-location algorithm to gain confidence in the determining the location of the GET and to further refine edge estimates based on the output of the GET-location algorithm.

Image analyzer <NUM> may also create a sparse stereo disparity that is provided to wear analyzer <NUM> that wear analyzer <NUM> can use along with the higher-resolution imaging data captured by LiDAR sensor <NUM> to determine wear or loss in GET <NUM>. In some embodiments, image analyzer <NUM> creates the sparse stereo disparity between the left-edge digital image (associated with the left rectified image) and the right-edge digital image (associated with the right rectified image), and this disparity is used by wear analyzer <NUM>. Alternatively, the sparse stereo disparity may be calculated from a first region of interest image (associated with the left rectified image) and a second region of interest image (associated with the right rectified image) and image analyzer <NUM> may detect an edge from the sparse stereo disparity image.

Wear analyzer <NUM> can be configured to analyze the sparse stereo disparity generated by image analyzer <NUM> for wear. For example, the physical parameter set associated with bucket <NUM> and GET <NUM> can include expected data related to unworn GET <NUM> or a set of unworn GET <NUM> that has been calibrated based on the expected image capture of camera <NUM>. The expected data can be in the form of pixels, actual measurement, a CAD-based model of GET <NUM> or an edge image related to unworn GET, as just some examples. Once wear analyzer <NUM> receives the sparse stereo disparity, it can fuse and correlate that sparse stereo disparity with the three-dimensional point cloud) of the higher-resolution imaging data captured by LiDAR sensor <NUM> (or, in some embodiments, the dense mesh surface or sparse mesh surface determined based on the three-dimensional point cloud) to determine measurement data related to the GET <NUM>. It may then compare the determined measurement data to expected data corresponding to an unworn version of GET <NUM> to determine wear levels, or loss, for GET <NUM>.

In some embodiments, pixel counts associated with the sparse stereo disparity can be used to measure the wear or loss of GET. Pixel counts can include area (e.g., total pixel for the GET), height of the GET in pixels, width of the GET in pixels, the sum of height and width of the GET, as just some examples. The manner of determining pixel counts can vary depending on the shape and style of the GET. For example, for GET that are much longer than they are wide, height pixel counts may be used, whereas for GET that are much wider than they are long, width pixel counts may be used. Various methods for determining pixel counts may be used without departing from the spirit and scope of the present disclosure.

In some embodiments, wear analyzer <NUM> can calculate a similarity score between the determined measurement data (which can include information derived from the higher-resolution LiDAR scan, the sparse stereo disparity determined based on the lower-resolution imagining data from camera <NUM>, or a combination of both) and the expected data corresponding to unworn GET <NUM>. The similarity score can reflect a measure of how well the determined measurement data of GET <NUM> matches the expected data of the physical parameter set. For example, the similarity score can include use of an intersection of union or Jaccard Index method of detecting similarity. In some embodiments, a dice coefficient or F1 Score method of detecting similarity can be employed to determine the similarity score. The similarity score can also include a value reflecting a percentage of how many pixels of the sparse stereo disparity overlap with the expected edge image. In some embodiments, the similarity score may be scaled or normalized from zero to one hundred.

The similarity score can provide an indication of wear of GET <NUM>. For example, a low score (e.g., a range of <NUM> to <NUM>) may indicate that one of GET <NUM> has broken or is missing indicating tooth loss. A high score (e.g., a range <NUM>-<NUM>) may indicate that a tooth is in good health and needs no replacing. A score in between the low and high scores can provide a wear level for the tooth, with higher scores indicating a longer lead time for tooth replacement than a lower score.

In some embodiments, wear analyzer <NUM> can collect measurement data related to GET <NUM> over time and use the collected measurement data to determine a wear level of GET <NUM> and a wear trend of GET <NUM>. For example, work machine <NUM> can be operating in its worksite over several days for a job. As work machine <NUM> moves material during the job, camera <NUM> provides stereo images bucket <NUM> and GET <NUM> to wear detection computer system <NUM>, and image analyzer <NUM> creates sparse stereo disparities for GET <NUM>. Wear analyzer <NUM> can map measurement data (e.g., pixel counts, metric measurements, imperial measurements) associated with the GET <NUM> at several instances of time over the period of time of the job. As bucket <NUM> and GET <NUM> engage with material at the worksite, it is expected that GET <NUM> will diminish in size due to wear. Accordingly, the measurement data associated with GET <NUM> will likewise decrease over time, and the pixel counts over time will reflect a wear trend. Wear analyzer <NUM> can determine a wear level for GET <NUM> at a particular point in time using the wear trend at the particular point in time. The wear level for GET <NUM> may indicate that GET <NUM> need replacement or it may indicate loss of one or more of GET <NUM>. In some embodiments, measurement data associated with GET <NUM> can be stored in memory <NUM> and applied to multiple jobs and multiple worksites, and the wear trend can be applicable to the lifetime of GET <NUM>. In such embodiments, pixel counts associated with GET <NUM> captured by wear analyzer <NUM> may be reset when bucket <NUM> or GET <NUM> are replaced, and wear analyzer <NUM> can restart collection of pixel counts for GET <NUM> from a zero-time point.

Since wear analyzer <NUM> determines a wear trend based on measurement data for GET <NUM> measured over time, wear analyzer <NUM> can also form predictions of when GET <NUM> may need replacement. For example, if wear analyzer <NUM> determines that measurement data associated with GET <NUM> show that GET <NUM> lose <NUM>% of life per ten work hours (because the measurement data decreases by <NUM>% per ten work hours), and GET <NUM> have been used for eight hundred work hours, wear analyzer <NUM> may determine that GET <NUM> need to be replaced within <NUM> hours.

In some embodiments, wear detection computer system <NUM> can include alert manager <NUM>. Alert manager <NUM> can be in communication with wear analyzer <NUM> and may monitor the wear trend and wear level determined by wear analyzer <NUM>. Alert manager <NUM> can provide messaging alerts to operator control panel <NUM> based on information determined by wear analyzer <NUM>. For example, when the wear level reaches a wear threshold value, alert manager <NUM> may generate an alert that is shown on display <NUM> of operator control panel <NUM>. The threshold value can correspond to values indicating extreme GET wear or, in some cases, complete GET loss. The alert may provide an indication to the operator of work machine <NUM> that one or more GET <NUM> need replacement. The wear threshold value can vary from embodiments and may dependent on the type of GET <NUM> and the material at the worksite with which GET <NUM> engage.

Alert manager <NUM> can also provide an alert that GET <NUM> may need replacement at some point in the future, for example, that GET <NUM> may need to be replaced within two weeks. A replacement alert can include information related to wear trend predictions for GET <NUM>. For example, the replacement alert can include a quantification of the wear trend (e.g., GET <NUM> wear <NUM>% per workday), the amount of time the teeth have been in use, or the expected date or time GET <NUM> will reach the wear threshold based on usage data.

In some embodiments, alert manager <NUM> can monitor the wear trend determined by wear analyzer <NUM> and provide a wear level value to display <NUM> to inform operator of work machine <NUM> of the current wear level. For example, if the wear trend indicates that GET <NUM> are <NUM>% worn down, based on the wear trend, alert manager <NUM> may provide an indication that GET <NUM> have <NUM>% of their life left before they need to be replaced. The display <NUM> can also inform an operator that a tooth has broken, indicating tooth loss (e.g., when one or more of GET <NUM> have less than <NUM>% life).

Wear detection computer system <NUM> allows an operator of work machine <NUM> to be informed when GET <NUM> need replacement, or has broken, due to extensive wear. The processes employed by wear detection computer system <NUM>-which are described in more detail below-provide for accurate and precise measurement of GET wear at a scale of less than <NUM> allowing an operator to halt operation of work machine <NUM> in the event of extreme GET wear or loss. The processes and techniques deployed by wear detection computer system <NUM> can be used with a variety of work machines.

For example, <FIG> is a diagram depicting a schematic side view of an example environment <NUM> in which a wheel loader work machine <NUM> is operating. Wheel loader work machine <NUM> can include a bucket <NUM> and one or more GET <NUM>. As shown in <FIG>, a LiDAR sensor <NUM> and a camera <NUM> are positioned so that GET <NUM> and bucket <NUM> are within a field-of-view <NUM> (of LiDAR sensor <NUM>) and field-of-view <NUM> (of camera <NUM>) during a dump end of the dig-dump cycle. As a result, LiDAR sensor <NUM> and camera <NUM> can be configured in such embodiments to capture imaging data when bucket <NUM> is at rest at the dump end of the dig-dump cycle.

As another example, <FIG> is a diagram depicting a schematic side view of an example environment <NUM> in which a hydraulic mining shovel work machine <NUM> is operating. Hydraulic mining shovel work machine <NUM> can include a bucket <NUM> and one or more GET <NUM>. In contrast to the positions of LiDAR sensor <NUM> and camera <NUM> for wheel loader work machine <NUM>, a LiDAR sensor <NUM> and a camera <NUM> are positioned such that GET <NUM> are within a field-of-view <NUM> (of LiDAR sensor <NUM>) and field-of-view <NUM> (of camera <NUM>) during a dig end of the dig-dump cycle. LiDAR sensor <NUM> and camera <NUM> can be configured in such embodiments to capture imaging data when bucket <NUM> is at rest at the dig end of the dig-dump cycle.

In yet another example, <FIG> is a diagram depicting a schematic side view of example an environment <NUM> in which an electric rope shovel work machine <NUM> is operating. Electric rope shovel work machine <NUM> can include a bucket <NUM>, one or more GET <NUM>, a LiDAR sensor <NUM> and a camera <NUM>. As shown in <FIG>, GET <NUM> may be within a field-of-view <NUM> (of LiDAR sensor <NUM>) and field-of-view <NUM> (of camera <NUM>) at a midpoint in the dig-dump cycle, but when bucket <NUM> is relatively close to LiDAR sensor <NUM> and camera <NUM>. In such embodiments, LiDAR sensor <NUM> and camera 428can be configured to capture imaging data when bucket <NUM> enters a range of positions correlating to field-of-view <NUM> and field of view <NUM>.

<FIG> depicts an image data flow diagram <NUM> depicting an example flow of imaging data for a region of interest detection process using computer vision techniques. Image data flow diagram <NUM> includes images that are received, processed, and generated by image analyzer <NUM> when detecting regions of interest within imaging data captured by camera <NUM> related to GET <NUM>. Image data flow diagram <NUM> includes a left image <NUM> and a right image <NUM> captured by camera <NUM>. Left image <NUM> can be a rectified image captured by the left image sensor of camera <NUM>. Right image <NUM> can be rectified image captured by the right image sensor of camera <NUM>. Both left image <NUM> and right image <NUM> include images of bucket <NUM> and GET <NUM>.

Image analyzer <NUM> may process left image <NUM> and right image <NUM> to create disparity map <NUM>. Disparity map <NUM> can be a dense stereo disparity map showing the disparity between each pixel of left image <NUM> and each pixel of right image <NUM>. Using disparity map <NUM> and a physical parameter set <NUM>, obtained from physical parameter library <NUM> and associated with bucket <NUM>, GET <NUM> and/or work machine <NUM>, image analyzer <NUM> can build a three-dimensional point cloud <NUM>. 3D point cloud <NUM> shows disparity between left image <NUM> and right image <NUM> in three dimensions. Image analyzer <NUM> may then perform a segmentation analysis on three-dimensional point cloud <NUM> to identify a region of interest <NUM> including GET <NUM>. Image analyzer <NUM> may use region of interest <NUM> to command and control LiDAR sensor <NUM> to capture higher-resolution imaging data for GET <NUM>.

<FIG> depicts an image data flow diagram <NUM> depicting an example flow of imaging data for a region of interest detection process using deep learning techniques. Similar to image data flow diagram <NUM> described above, the output of the region of interest detection process will be a region of interest <NUM> corresponding to GET <NUM> that image analyzer <NUM> will then use to command and control LiDAR sensor <NUM> to capture higher-resolution imaging data for GET <NUM>. But unlike image data flow diagram <NUM>, image analyzer <NUM> utilizes deep learning techniques to detect region of interest <NUM>.

Image data flow diagram <NUM> includes image <NUM> captured by camera <NUM>. Image <NUM> could be a rectified image captured by either the left image sensor or the right image sensor of camera <NUM>. Image analyzer <NUM> may apply a deep learning GET detection algorithm to image <NUM>. The deep learning GET detection algorithm may employ a neural network that has been trained with a corpus of image data where GET have been individually identified and labeled and/or groups of GET have been individually identified and labeled. When image analyzer <NUM> applies the deep learning GET detection algorithm to image <NUM>, it may identify a plurality of individual GET bounding boxes <NUM> containing images of individual GET <NUM>. In some embodiments, image analyzer <NUM> may also identify a GET group bounding box <NUM> encompassing individual GET bounding boxes <NUM>. Once image analyzer <NUM> identifies GET group bounding box <NUM> it may extract the pixels within it as region of interest <NUM>.

<FIG> depicts an image data flow diagram <NUM> depicting an example flow of imaging data for a region of interest detection process for imaging data captured by LiDAR sensor <NUM>. The imaging data captured by LiDAR sensor <NUM> according to image data flow diagram <NUM> substantially corresponds to the field of view shown in image <NUM>. As shown, the field of view includes bucket <NUM> and GET <NUM>. LiDAR sensor <NUM> performs a LiDAR data capture that includes a plurality of LiDAR "hits" for when LiDAR sensor <NUM> detects an object surface, e.g., a surface corresponding to either bucket <NUM> or GET <NUM>. The LiDAR hits can be represented as three-dimensional point cloud <NUM>, where each point of three-dimensional point cloud <NUM> corresponds to a LiDAR hit. Image analyzer <NUM> determines region of interest of <NUM> based on three-dimensional point cloud <NUM> by performing a segmentation analysis or other object recognition analysis technique. In some embodiments, image analyzer <NUM> may use physical parameter set <NUM> to identify region of interest <NUM>. For example, image analyzer <NUM> may use a bucket-tooth template, CAD-based model of GET <NUM>, or pattern matching techniques to identify region of interest <NUM> within three-dimensional point cloud <NUM>.

<FIG> depicts an image data flow diagram <NUM> depicting an example flow of imaging data for a wear detection process using higher-resolution data captured by LiDAR sensor <NUM>. Once image analyzer <NUM> identifies region of interest <NUM>, it commands and controls LiDAR sensor <NUM> to perform a high-resolution scan of region of interest <NUM> to capture more precise data related to GET <NUM>. LiDAR sensor <NUM> performs the high-resolution scan be adjusting its MEMS mirrors to narrow field-of-view <NUM> to capture objection recognition data limited to region of interest <NUM> (and, in turn, GET <NUM>). Image analyzer <NUM> receives the high-resolution imaging data <NUM> captured by LiDAR sensor <NUM> and can generate a high-resolution three-dimensional point cloud <NUM> where each point in high-resolution three-dimensional point cloud <NUM> corresponds to a LiDAR hit. In some embodiments, image analyzer <NUM> communicates high-resolution three-dimensional point cloud <NUM> to wear analyzer <NUM> so that wear analyzer <NUM> can determine a GET wear or loss measurement <NUM> for GET <NUM>. In some embodiments, image analyzer <NUM> may first convert high-resolution three-dimensional point cloud <NUM> to a dense mesh surface <NUM> or it may further convert dense mesh surface <NUM> to a sparse mesh surface <NUM> before communication to wear analyzer <NUM>.

In some embodiments, wear analyzer <NUM> utilizes information in addition to the information derived from high-resolution imaging data <NUM> to determine GET wear or loss measurement <NUM>. For example, in some embodiments, wear analyzer <NUM> uses information from physical parameter set <NUM>, such as a CAD-based model of GET in an unworn state, as a baseline for determining GET wear or loss measurement <NUM>. Wear analyzer <NUM> can also use previous GET wear or loss measurement <NUM> for GET <NUM>, either instead of, or in addition to, information from physical parameter set <NUM>. In some embodiments, wear analyzer <NUM> may use a sparse stereo disparity <NUM> for region of interest <NUM> generated from imaging data captured by camera <NUM> and fuse it with information derived from high-resolution imaging data <NUM> to determine GET wear or loss measurement <NUM>.

<FIG> shows a flowchart representing an example wear detection process <NUM> to detect wear of GET <NUM>. In some embodiments, process <NUM> can be performed by image analyzer <NUM> and wear analyzer <NUM>. Process <NUM> generally follows the image data flows of <FIG> and should be interpreted consistent with the description of these figures, and the descriptions of image analyzer <NUM> and wear analyzer <NUM> described above with respect to <FIG>. Although the following discussion describes aspects of process <NUM> being performed by image analyzer <NUM> or wear analyzer <NUM>, other components of wear detection computer system <NUM> may perform one or more blocks of process <NUM> without departing from the spirit and scope of the present disclosure.

Process <NUM> begins at block <NUM> where image analyzer <NUM> receives first imaging data from one or more sensors associated with work machine <NUM>. The one or more sensors can include LiDAR sensor <NUM> and camera <NUM>, for example. The first imaging data received at block <NUM> generally corresponds to fields-of-view of the one or more sensors that are wide enough to capture both bucket <NUM> and GET <NUM> of work machine <NUM>. As described in this disclosure, the first imaging data can be considered a "coarse scan" or "lower-resolution scan" of GET <NUM> that can be used to identify a region of interest including GET <NUM>, at block <NUM>.

The region of interest including GET <NUM> can be determined by performing a standard segmentation analysis or edge detection analysis on imaging data captured by camera <NUM> and comparing the results to patterns of GET or bucket and GET combinations stored in physical parameter library <NUM>. Other computer vision techniques such as gradient analysis may be employed. The region of interest including GET <NUM> can also be determined by performing a depth-based segmentation on imaging data captured by LiDAR sensor <NUM> and likewise comparing the results to patterns of GET or bucket and GET combinations stored in physical parameter library <NUM>. In some embodiments, image analyzer <NUM> may use CAD-based models of GET when determining the region of interest at block <NUM>.

At block <NUM>, the image analyzer <NUM> uses the region of interest to command and control LiDAR sensor <NUM> to narrow or adjust its field of view to focus on the region of interest, and by extension, the GET <NUM>. By narrowing and redirecting its field of view, LiDAR sensor <NUM> is capable of capturing a high-resolution scan of GET <NUM>. After LiDAR sensor <NUM> performs the high-resolution scan of GET <NUM>, image analyzer <NUM> receives second imaging data from LiDAR sensor <NUM> at block <NUM>. The second imaging data, having been captured by LiDAR sensor <NUM> with a narrower and more focused field of view, is of higher resolution than the first imaging data received at block <NUM>.

At block <NUM>, image analyzer <NUM> generates a three-dimensional point cloud based on the second imaging data. The points within the three-dimensional point cloud correspond to a LiDAR hit from the higher-resolution scan of GET <NUM>. In some embodiments, the three-dimensional point cloud is of sufficient resolution to calculate a precise GET measurement (e.g., the hits are within two millimeters) and image analyzer <NUM> communicates the three-dimensional point cloud to wear analyzer <NUM>. In some embodiments, image analyzer <NUM> further processes the three-dimensional point cloud by creating a dense mesh surface or a sparse mesh surface before communication to wear analyzer <NUM>.

At block <NUM>, wear analyzer <NUM> uses the information derived from the second imaging data (e.g., the higher-resolution imaging data) to determine a wear level or loss of GET. The wear level or loss may be quantified in real-world measurements (e.g., millimeters), in terms of pixels, or as a percentage of expected size (based, for example, on the CAD-based model for GET <NUM>). As discussed above, wear analyzer <NUM> may use a CAD-based model of GET <NUM> in an unworn state and compare it to the observed GET <NUM> measurement to determine GET wear level or loss. Wear analyzer <NUM> can also use historical measurement data for GET to determine wear level over time or to determine a wear level trend to make a prediction of when GET <NUM> will need replacement. In some embodiments, wear analyzer <NUM> may be configured to determine loss when wear exceeds a threshold. For example, wear analyzer may determine loss of a GET if its size is more then <NUM>% reduced, or reduced by a fixed measurement amount (e.g., <NUM> in length). Wear analyzer <NUM> may generate an alert when wear of GET meets or exceeds the threshold.

Throughout the above description, certain components of wear detection computer system <NUM> were described to perform certain operations. But, in some embodiments of wear detection computer system <NUM>, other components may perform these operations other than what is described above. In addition, wear detection computer system <NUM> may include additional or fewer components than what is presented above in example embodiments. Those of skill in the art will appreciate that wear detection computer system <NUM> need not be limited to the specific embodiments disclosed above.

The systems and methods of this disclosure can be used in association with operation of work machines at a worksite that are excavating, moving, shaping, contouring, and/or removing material such as soil, rock, minerals, or the like. These work machines can be equipped with a bucket used to scoop, dig, or dump the material while at the worksite. The bucket can be equipped with one or more GET to assist with the loosening of the material during operation. The work machines can also include a system having a processor and memory configured to perform methods of wear detection according to the examples described herein. The systems and methods can detect wear or loss of work machine components such as GET so operators of such work machines can take corrective action before a failure damaging downstream processing equipment can occur.

In some examples, the systems and methods capture imaging data associated with GET from one or more sensors of the work machine that is then processed to determine wear or loss of the GET. The one or more sensors can include LiDAR sensors, image sensors, and/or stereoscopic cameras.

In some examples, the one or more sensors collect imaging data twice within the dig-dump cycle of the work machine. The first collection of imaging data is considered a "coarse scan" or "lower-resolution scan" of the bucket and GET. Based on the coarse scan, the systems and methods can determine a region of interest within the first imaging data. The region of interest corresponds to the group of GET (or, in some cases, an individual GET). Based on the region of interest, the systems and methods command the LiDAR sensor to focus its field of view on the GET to perform the second imaging data collection. The second imaging data collection is considered a "fine scan" or a "higher-resolution scan" of the GET.

The process described in the present disclosure provides high-precision measurements of GET while still providing processing efficiency. For example, using coarse scans of the bucket and GET allow the systems and methods to limit high resolution LiDAR scans (which are computationally expensive) to relevant objects within the field of view of the LiDAR sensor. By using the fine scans, measurement of GET within <NUM> or <NUM> millimeters can be achieved.

Moreover, use of LiDAR for fine scans (as opposed image processing or computer vision techniques) presents other advantages. For example, LiDAR-based imaging data capture can perform better than pure image-based data capture when lighting is poor, there are obstructions to cameras due to fog or inclement weather (e.g., rain or snow), when there is low contrast between the GET and excavated material, or when lighting conditions create shadows. The use of LiDAR for fine scans, therefore, can create more accurate wear detection of GET thereby decreasing the likelihood of catastrophic GET loss or wear that can cause damage to downstream processing machines.

Claim 1:
A computer-implemented method, comprising:
receiving first imaging data from one or more sensors (<NUM>, <NUM>) associated with a work machine (<NUM>), the first imaging data comprising data related to at least one ground engaging tool (GET) (<NUM>) of the work machine, the first imaging data identifying the at least one GET at a first resolution;
identifying a region of interest (<NUM>) within the first imaging data, the region of interest including the data related to the at least one GET;
controlling a light detection and ranging (LiDAR) sensor (<NUM>) to capture second imaging data corresponding to the identified region of interest, the second imaging data identifying the at least one GET at a second resolution higher than the first resolution;
generating a three-dimensional point cloud (<NUM>) of the at least one GET based on the second imaging data; and
determining a wear level or loss for the at least one GET based on the three-dimensional point cloud.