Patent ID: 12252870

DETAILED DESCRIPTION

The present disclosure is generally directed to systems and methods for detecting wear of components of a work machine in an environment, such as a worksite, using computer vision techniques. In some examples, a camera associated with a work machine captures video of a component of the work machine. The video is analyzed by a processor associated with the work machine to detect wear of the component. The component may be one or more ground engaging tools (GET) of a bucket of the work machine. In some examples, the system and method select images from the video captured by the camera for processing using a template images illustrating both the bucket and the GET.

For images selected for processing, the system and method determine the number of pixels for the GET and create a graph or mapping of pixel counts for the GET over time. Pixel counts can include area (e.g., total pixel for the GET), height of the GET in pixels, width of the GET in pixels, the sum of height and width of the GET, as just some examples. The manner of determining pixel counts can vary depending on the shape and style of the GET. For example, for GET that are much longer than they are wide, height pixel counts may be used, whereas for GET that are much wider than they are long, width pixel counts may be used. Various methods for determining pixel counts may be used without departing from the spirit and scope of the present disclosure.

In some examples, the processor can determine wear of the GET—and predict when GET need replacement—based on the rate of change of the GET pixel counts over time. The processor can also determine whether one of the GET has broken (e.g., GET loss) based on when the rate of change of the GET pixel counts over time is high. By using the rate of change of GET pixel counts over time to make such determinations, the system and method can reduce errors in wear detection by minimizing the impact of false positives and can do so with less processing resources than more error prone techniques, such as machine learning or neural networks.

FIG.1is a block diagram depicting a schematic of an example work machine100including an example a wear detection computer system110. WhileFIG.1depicts work machine100as a hydraulic mining shovel, in other examples, work machine100can include any machine that moves, sculpts, digs, or removes material such as soil, rock, or minerals. As shown inFIG.1, work machine100can include a bucket120attached to arm122. Bucket120can include one or more ground engaging tools (GET), such as teeth125, that assist work machine100in loosening material. While the examples provided in this disclosure refer to teeth125as GET, other types of GET are contemplated to be within the scope of the embodiments provided by this disclosure. For example, GET can include lip shrouds, edge guards, adapters, ripper protectors, cutting edges, sidebar protectors, tips, or any other tool associated with a work machine that may wear over time due to friction with worksite material.

Work machine100can also include a camera128. Camera128may have a field-of-view129directed to bucket120and teeth125. Camera128can be a mono or stereo camera.

As work machine100operates within a worksite, it may move arm122to position bucket120to move or dig material within the worksite as part of a dig-dump cycle. As work machine100positions bucket120through the dig-dump cycle, bucket120may move in and out of field-of-view129of camera128. Camera128may be positioned so that it has an unobstructed view of teeth125during the dig-dump cycle. For example, camera128may be positioned on work machine100so that bucket120and teeth125are visible at the moment bucket120empties material within the dig-dump cycle. As another example, camera128may be positioned so that bucket120enters its field-of-view when arm122is fully extended or fully contracted within the dig-dump cycle. As explained below with respect toFIGS.2-4, the position of camera128may vary depending on the type of work machine100and specifics related to its worksite.

According to some embodiments, work machine100can include an operator control panel130. Operator control panel130can include a display133which produces output for an operator of work machine100so that the operator can receive status or alarms related to wear detection computer system110. Display133can include a liquid crystal display (LCD), a light emitting diode display (LED), cathode ray tube (CRT) display, or other type of display known in the art. In some examples, display133can include audio output such as speakers or ports for headphones or peripheral speakers. Display133can also include audio input devices such as microphone or ports for peripheral microphones. Display133can include a touch-sensitive display screen in some embodiments, which can also act as an input device.

In some embodiments, operator control panel130can also include a keyboard137. Keyboard137can provide input capability to wear detection computer system110. Keyboard137can include a plurality of keys allowing the operator of work machine100to provide input to wear detection computer system110. For example, an operator may depress the keys of keyboard137to select image templates associated with work machine100, bucket120, or teeth125according to examples of the present disclosure. Keyboard127can be non-virtual (e.g., containing physically depressible keys) or keyboard127can be a virtual keyboard shown on a touch-sensitive embodiment of display133.

As shown inFIG.1, wear detection computer system110can include a processor140. Processor140can include one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), some combination of CPU, GPU, or FPGA, or any other type of processing unit. Processor140may have numerous arithmetic logic units (ALUs) that perform arithmetic and logical operations, as well as one or more control units (CUs) that extract instructions and stored content from processor cache memory, and then executes the instructions by calling on the ALUs, as necessary, during program execution. Processor140may also be responsible for executing drivers and other computer-executable instructions for applications, routines, or processes stored in memory150, which can be associated with common types of volatile (RAM) and/or nonvolatile (ROM) memory.

In some embodiments, wear detection computer system110can include a memory150. Memory150can include system memory, which may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. Memory150can further include non-transitory computer-readable media, such as volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, or other data. System memory, removable storage, and non-removable storage are all examples of non-transitory computer-readable media. Examples of non-transitory computer-readable media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium which can be used to store the desired information and which can be accessed by wear detection computer system110.

Memory150can store data, including computer-executable instructions, for a wear detection computer system110as described herein. For example, memory150can store one or more components of wear detection computer system110such as a template library160, an image selector165, an image analyzer170, a wear analyzer175, and an alert manager180. Memory150can also store additional components, modules, or other code executable by processor140to enable operation of wear detection computer system110. For example, memory150can include code related to input/output functions, software drivers, operating systems, or other components.

Template library160can include one or more template images used by image selector165to identify and select images from a video feed provided to wear detection computer system110by camera128. For example, template library160can include one or more image templates that can be used as part of a segmentation or convolution filter algorithm performed by image selector165to find images from the video feed that are substantially similar to the one or more image templates. According to some embodiments, the templates stored in template library160include images of a bucket and its teeth (e.g., a bucket-tool template). For example, for work machine100, one of the templates stored in template library160can include an image of bucket120with teeth125as bucket120is expected to be positioned within the field of view of camera128. In some examples, teeth125of the one or more templates are unworn, new, or as they appear before they have engaged with material at a worksite.

Template library160can include multiple bucket-tool templates where each bucket-tool template corresponds to a work machine, bucket, tooth, GET, or a combination of these. During operation, an operator may use operator control panel130to select a bucket-tool template from template library160matching bucket120in teeth125, or work machine100. For example, if the work machine100is a hydraulic mining shovel having a model number “6015B,” the operator may use operator control panel130to input the model number “6015B,” and wear detection computer system110may load into memory150a template corresponding to a model 6015B hydraulic mining shovel from template library160. In some examples, a list of templates available in template library160can be shown on display133upon a power-up or reset operation of wear detection computer system110, and an operator may select one of the templates from the list for operation depending on the model number of work machine100, bucket type of bucket120, or tooth type of teeth125.

According to some embodiments, image selector165can perform one or more operations to choose an image for further wear detection analysis. In some embodiments, image selector165uses a bucket-tool template and computer vision techniques to match video frames of the video feed of camera128to the template. Image selector165can utilize a variety of computer vision techniques, either alone or in combination, for matching the bucket-tool template with video frames of the video feed of camera128. For example, image selector can use a conventional template-based computer vision approach, approaches using eigenspaces, cross-correlation approaches, image segmentation, edge detection techniques, convolution filters, or other techniques known in the art for identifying objections within images based on a template, mask, or kernel.

In some embodiments, image selector165can use other techniques for selecting an image for further wear detection analysis. For example, image selector165can be configured to capture an image based on the position of bucket120during the dig-dump cycle of work machine100. In such an example, image selector165can select an image when bucket120can be expected to be in a fixed position, such as at the beginning or end of the dig-dump cycle or when the bucket is fully retracted or extended. When bucket120reaches the fixed position, image selector may capture an image from the video feed of camera128and select it for future wear detection processing. Image selector165can also be configured to select an image at a point during the dig-dump cycle when bucket120and teeth125are closest to camera128, or when camera128has a clear or unobstructed view of bucket120and teeth125.

Image selector165can also use machine learning or neural network techniques to select images for further wear detection analysis. For example, image selector may include a neural network that has been trained using a corpus of training images showing bucket120and teeth125in an optimal position with respect to the field-of-view of camera128and bucket120and teeth125in nonoptimal positions with respect to the field-of-view of camera128. The corpus of training images can also include images of objects from a work site that can potentially trigger false positives such as bucket120with one or more of teeth125missing, bucket120with material stuck to it, buckets or teeth not typically used with work machine100, as just some examples. Once the neural network has been trained, image selector165can use it to process the video feed of camera128and select images for further wear detection processing.

According to some embodiments, image selector165can use a combination of the above identified techniques for identifying images for further wear detection analysis. For example, image selector165may use bucket's120position within a range of positions corresponding to field-of view129within the dig-dump cycle as a trigger to begin applying the bucket-tool template to the images of the video stream captured by camera128. In such embodiments, image selector165may enter a “standby” mode when bucket120is outside the range of position, and a “begin processing” mode when bucket120enters the range of position. As another example, image selector165may use a range of position of bucket120in combination with a trained neural network to select images for further processing in a similar fashion.

Image analyzer170can be configured to analyze images selected by image selector165to further identify individual teeth125within the selected image. In some examples, image analyzer170selects individual teeth125by using an expected location of teeth125within the captured image. For example, if image selector165is using a bucket-tool template, the expected position of teeth125relative to bucket120will be known based on the relative position of the teeth and bucket in the bucket-tool template. Using this information, image analyzer170can go to the expected location in selected image and capture a pixel region proximate to the teeth. The pixel region can then be used to further identify the tooth based on computer vision techniques such as application of a convolution filter, segmentation analysis, edge detection, or pixel strength/darkness analysis within the pixel region. In some embodiments, image analyzer170may use an individual tooth template to apply to the pixel region to further refine the location of the tooth using computer vision techniques.

Wear analyzer175can be configured to analyze tooth images or pixel regions identified by image analyzer170for wear. In some embodiments, image analyzer170analyzes wear based on the associated bucket-tool template used by image selector165to select an image for processing. For example, the associated bucket-tool template can include an image of unworn tools which can be compared to the tooth images identified by image analyzer170based on the size of the unworn tooth from the bucket-tool template and the size of tooth images. In some embodiments, a similarity score can be calculated for the tooth images and the corresponding unworn tooth in the bucket-tool template. The similarity score can reflect a measure of how well the tooth images match the corresponding unworn tooth in the bucket-tool template. For example, the similarity score can include use of an intersection of union or Jaccard Index method of detecting similarity. In some embodiments, a dice coefficient or F1 Score method of detecting similarity can be employed to determine the similarity score. The similarity score can also include a value reflecting a percentage of how many pixels of the tooth images overlap with their corresponding unworn tooth in the bucket-tool template. In some embodiments, the similarity score may be scaled or normalized from zero to one hundred.

The similarity score can provide an indication of wear of teeth125. For example, a low score (e.g., a range of 0 to 20) may indicate that one of teeth125has broken or is missing indicating tooth loss. A high score (e.g., a range 80-100) may indicate that a tooth is in good health and needs no replacing. A score in between the low and high scores can provide a wear level for the tooth, with higher scores indicating a longer lead time for tooth replacement than a lower score.

In some embodiments, wear analyzer175can count pixels associated with images of teeth125over time and use the pixel counts to determine a wear level of teeth125and a wear trend of teeth125. For example, work machine100can be operating in its worksite over several days for a job. As work machine100moves material during the job, camera128provides a video feed of bucket120and teeth125to wear detection computer system110, and image analyzer170identifies pixel regions having teeth for further analysis. Wear analyzer175can map pixel counts associated with the tooth at several instances of time over the period of time of the job. As bucket120and teeth125engage with material at the worksite, it is expected that teeth125will diminish in size due to wear. Accordingly, the pixel counts associated with teeth125will likewise go down over time, and the pixel counts over time will reflect a wear trend. A wear level for teeth125at a particular point in time can be determined using the wear trend at the particular point in time. The wear level for teeth125may indicate that teeth125need replacement or it may indicate tooth loss for one or more of teeth125. In some embodiments, pixel counts associated with teeth125can be stored in memory150and applied to multiple jobs and multiple worksites, and the wear trend can be applicable to the lifetime of teeth125. In such embodiments, pixel counts associated with teeth125captured by wear analyzer175may be reset when bucket120or teeth125are replaced, and wear analyzer175can restart collection of pixel counts for teeth125from a zero-time point.

Since wear analyzer175determines a wear trend based on pixel counts for teeth125measured over time, wear analyzer175can also form predictions of when teeth125may need replacement. For example, if wear analyzer175determines that pixel counts associated with teeth125show that teeth125lose 1% of life per ten work hours (because the pixel counts decrease by 1% per ten work hours), and teeth125have been used for eight hundred work hours, wear analyzer175may determine that teeth125need to be replaced within 200 hours.

In some embodiments, wear detection computer system110can include alert manager180. Alert manager180can be in communication with wear analyzer175and may monitor the wear trend and wear level determined by wear analyzer175. Alert manager180can provide messaging alerts to operator control panel130based on information determined by wear analyzer175. For example, when the wear level reaches a wear threshold value, alert manager180may generate an alert that is shown on display133of operator control panel130. The threshold value can correspond to values indicating extreme tooth wear or, in some cases, complete tooth loss. The alert may provide an indication to the operator of work machine100that one or more teeth125need replacement. The wear threshold value can vary from embodiments and may dependent on the type of teeth125and the material at the worksite with which teeth125engage.

Alert manager180can also provide an alert that teeth125may need replacement at some point in the future, for example, that teeth125may need to be replaced within two weeks. A replacement alert can include information related to wear trend predictions for teeth125. For example, the replacement alert can include a quantification of the wear trend (e.g., teeth125wear 2% per work day), the amount of time the teeth have been in use, or the expected date or time teeth125will reach the wear threshold based on usage data.

In some embodiments, alert manager180can monitor the wear trend determined by wear analyzer175and provide a wear level value to display133to inform operator of work machine100of the current wear level. For example, if the wear trend indicates that teeth125are 60% worn down, based on the wear trend, alert manager180may provide an indication that teeth125have 40% of their life left before they need to be replaced. The display133can also inform an operator that a tooth has broken, indicating tooth loss (e.g., when one or more of teeth125have less than 20% life).

FIG.2is a diagram depicting a schematic side view of an example environment200in which a wheel loader work machine201is operating. Wheel loader work machine201can include a bucket220and one or more ground engaging teeth225. As shown inFIG.2, a camera228is positioned so that teeth225and bucket220are within a field of view229of camera228during a dump end of the dig-dump cycle. As a result, image selector165(FIG.1) can be configured in such embodiments to capture images when bucket220is at rest at the dump end of the dig-dump cycle.

FIG.3is a diagram depicting a schematic side view of an example environment300in which a hydraulic mining shovel work machine301is operating. Hydraulic mining shovel work machine301can include a bucket320and one or more ground engaging teeth325. In contrast to the position of camera228for wheel loader work machine201, camera328is positioned such that teeth325are within field of view329of camera328during a dig end of the dig-dump cycle. Image selector165(FIG.1) can be configured in such embodiments to capture images when bucket320is at rest at the dig end of the dig-dump cycle.

FIG.4is a diagram depicting a schematic side view of example an environment400in which an electric rope shovel work machine401is operating. Electric rope shovel work machine401can include a bucket420, one or more ground engaging teeth425, and a camera428. As shown inFIG.4, teeth425may be within a field of view429of camera428at a midpoint in the dig-dump cycle, but when bucket420is relatively close to camera428. In such embodiments, image selector165(FIG.0.1) can be configured to capture images when bucket420enters a range of positions correlating to field of view429of camera428.

FIG.5depicts an image selection data flow diagram500showing the flow of data for an example image selection process. In some embodiments, template library160may provide bucket-tool template510to image selector165. As described above, bucket-tool template510may be selected by an operator of work machine100before operation. In some embodiments, bucket-tool template510is preloaded in software or firmware of wear detection computer system110. As work machine100operates at a worksite, camera128can provide a video feed containing a plurality of images530to image selector165. Image selector165can analyze the plurality of images using template510. For example, image selector165can use computer vision template matching techniques, convolution filters, segmentation analysis, edge detection, or other computer vision techniques to match template510with the plurality of images530to identify selected image540for further wear detection analysis.

As shown inFIG.5, template510includes an image of both template bucket520and template teeth525. Template bucket520can represent an image of bucket120in a position and orientation consistent with how bucket120appears within the field of view of camera128. Template teeth525can represent new, unworn, or unused versions of teeth125.

In some embodiments, template510includes information providing approximate template tooth locations527that image analyzer170can use for later identification of individual teeth125within selected image540. Template tooth locations527can include pixel offsets from one position of the combined bucket-tool image in template510. For example, the offset may be the number of pixels down and to the right of the upper leftmost corner of bucket520. In some embodiments, template tooth locations527can be absolute pixel positions for template teeth525as opposed to an offset from one edge or corner of bucket520. Template tooth locations527can be used by image analyzer170to approximate corresponding image tooth locations547within selected image540.

FIG.6depicts a pixel count data flow diagram600. Pixel count data flow diagram600represents an example data flow that may occur within image analyzer170to determine pixel counts associated with teeth125. While example data flow diagram600refers to teeth125, other GET are contemplated in different embodiments. In some embodiments, image analyzer170performs a computer vision segmentation analysis on selected image540to separate captured bucket-tool image610from the background of selected image540. Once bucket-tool image610has been segmented from the background, image analyzer170can identify a plurality of tooth images620associated with captured bucket-tool image610, and by extension, selected image540. Image analyzer170can use several techniques for identifying the plurality of tooth images620. For example, as described above, image analyzer170can use template tooth locations527to determine approximate corresponding image tooth locations547, and then extract a pixel region630of predetermined size that is likely to contain an image of an individual tooth. For example, pixel region630may be fifty pixels high by thirty pixels wide and image analyzer170may extract pixel region630for each of corresponding image tooth locations547.

In some embodiments, image analyzer170may employ additional template matching, segmentation, convolution filter, or other computer vision techniques to segment the plurality of tooth images620from captured bucket-tool image610. For example, image analyzer170may employ a template of one tooth to identify the plurality tooth images620.

According to some embodiments, image analyzer170may determine a pixel count associated with the plurality of tooth images620. In some embodiments, image analyzer170may detect pixels635associated with a tooth from selected image540. For example, image analyzer170may determine that pixels635contains four hundred thirty-six pixels. Pixels635can provide a visual representation or abstraction of the actual size and dimension of one of teeth125of work machine100from which wear analyzer can determine wear. Image analyzer170can communicate the pixel count to wear analyzer175for further analysis.

Image analyzer170can also determine an expected amount (e.g., an expected number, an expected location, an expected grouping, etc.) of pixels associated with each tooth based on the number of pixels representing template teeth525in bucket-tool template510. For example, image analyzer can use computer vision techniques, such as edge detection for example, to detect an expected edge640of an unworn tooth based on edges of template teeth525. The number of pixels within expected edge640may represent an expected pixel count for pixels635. In some embodiments, wear analyzer175can use the expected pixel count and the actual number of pixels635to determine wear level of teeth at a particular point in time.

In some embodiments, wear analyzer175can calculate a similarity score between template teeth525(which represent unworn tools) and the plurality of tooth images620as described above. The similarity score can be calculated based on the difference between expected pixel counts for teeth125and number of actual pixels635. In some embodiments, a convolution filter can be used to create similarity scores based on comparing template510with selected image540, or portions of template510(e.g. template teeth525) with corresponding portions of selected image540. In one example, the similarity score can be normalized to a scale of zero to one-hundred, with values below twenty representing low similarity (e.g., a tooth may be missing or broken), values above eighty representing generally unworn tools, and values between twenty and eighty representing varying degrees of wear. While similarity scores may be scaled on a range of zero to one-hundred, other ranges are contemplated within the spirit and scope of the disclosed embodiments.

FIG.7is a diagram of an example wear trend graph700showing a mapping of tool pixel counts over time, represented as plots705. Wear trend graph700can be created by wear analyzer175according to some embodiments. Wear trend graph700can include pixel count axis710and time axis720. While wear trend graph700shows pixel count axis710as the vertical axis and time axis720as the horizontal axis, these axes can be reversed in some embodiments. Pixel count axis710can be incremented by number of pixels. Time axis720can be incremented by any period of time such as minutes, hours, or days for example. In some embodiments, wear analyzer175provides code to operator control panel that causes display133to render a graphical representation of wear trend graph700.

While example wear trend graph700is shown visually inFIG.7, wear analyzer175need not create code for rendering a visual graph. For such embodiments, the data shown in wear trend graph700can be stored in a data structure or object that wear analyzer175uses to determine wear trends, wear levels, and/or loss of teeth125. For such embodiments, wear trend graph700is a visual aid provided for discussion purposes in the present disclosure.

As shown in the example ofFIG.7, wear trend graph700shows plots705for four types of teeth: tooth1731, tooth2732, tooth3737, and tooth4734. In rendered visual representations of wear trend graph700, wear trend graph700can include tooth key730, which identifies the correlation between plots on wear trend graph700and tooth1731, tooth2732, tooth3737, and tooth4734. While wear trend graph700shows data related to teeth, it may show plots related to other types of GET in some embodiments.

According to some embodiments, wear analyzer175can determine a wear trend based on plots705. For example,FIG.7shows wear trend740for plots705of tooth4734. In the example wear trend graph700, wear trend740is linear, but some teeth may have a non-linear wear trend in certain applications. Wear trend graph700can also include wear level threshold750. When wear trend740intersects with wear level threshold750, the tooth corresponding to wear trend740(e.g., tooth4734) has worn to the point of needing replacement or has broken indicating tooth loss. In some embodiments, alert manager180may generate a visual or audio alert informing operator of work machine100that a tooth needs replacement. In some embodiments, wear trend740can be used to detect a future point in time when a tooth may need replacement based on the intersection between an extrapolated version of wear trend740and wear threshold750.

FIG.8shows a flowchart representing an example image selection process800. In some embodiments, process800can be performed by image selector165and image analyzer170. Although the following discussion describes process800as being performed by image selector165and image analyzer170, other components of wear detection computer system may perform one or more blocks of process800without departing from the spirit and scope of the present disclosure.

Process800begins at block810where the image selector captures a video stream of a bucket of the work machine. The video stream can include a plurality of images of the bucket of the work machine, and the bucket's ground engaging tools (GET) (e.g., teeth). At block820, image selector165segments the plurality of images from the video stream using a bucket-tool template. The bucket-tool template, consistent with the above discussion, includes an image of a bucket with unworn GET. The image selector can perform block820using a segmentation analysis where images of the bucket and tooth from the plurality of images of the video feed are segmented from the background. In addition, the bucket-tool template can act as a mask that is applied to the plurality of images to determine similarity between the plurality of images and the bucket-tool template. In some embodiments, the image selector determines a similarity score between the bucket-tool template and the plurality of images. Based on the similarity, image selector165may select an image for wear detection analysis at block830.

At block840, an image analyzer identifies a plurality of GET in the image selected at block830. In some embodiments, the image analyzer can identify the plurality of GET using locations of tools within the bucket-tool template, as described above with respect toFIG.5. Once the plurality of GET have been identified, the wear level for the tools can be determined. Wear level can be determined based on differences between the pixel counts of unworn tools depicted in the bucket-tool template and the pixel counts associated with the plurality of GET. As another example, block850can be performed by comparing the similarity of tools within the bucket-tool template to the plurality of identified GET. As another example, wear levels can be determined consistent with wear detection analysis process900shown inFIG.9.

FIG.9shows a flowchart representing an example wear detection analysis process900. In some embodiments, process900can be performed by one or more components of wear detection computer system110. Although the following discussion describes process900as being performed by certain components of wear detection computer system110, different components of wear detection computer system may perform one or more blocks of process900without departing from the spirit and scope of the present disclosure.

Process900begins at block910where the image selector receives a video stream of a bucket of the work machine. The video stream can include a plurality of images of the bucket of the work machine, and the bucket's ground engaging tools (GET). At block920, the image selector selects an image from the video stream for further wear detection processing. In some examples, the image selector performs block920using a bucket-tool template as described above with respect to image selection process800. In some embodiments, image selector165selects images based on the position of the work machine's bucket in the dig-dump cycle. For example, the image selector can capture an image of the video stream when the bucket is in an optimal position for the camera to capture an image of the bucket and its GET. In other examples, the image selector can use machine learning techniques or neural networks to identify optimal images for wear detection analysis. One or more of the above techniques may be combined to select images at block920.

At block930, an image analyzer identifies GET within the selected image. The image analyzer can detect GET consistent with the embodiments disclosed above with respect toFIGS.6and8. Once the image analyzer identifies the GET, the image analyzer determines pixel counts for the GET at block940. The pixel counts are provided to a wear analyzer in some example. The wear analyzer maps the pixel counts for the GET to an instance in time at block950. The instance in time can be an absolute date-time value (e.g., Oct. 1, 2020 16:04:32) or it could be a relative time based on the operation of the work machine (e.g., number of seconds, minutes, hours of work machine operation).

At block960, the wear analyzer determines the wear level for the GET based on current pixel-time mappings and previous pixel-time mappings. Wear analyzer175can determine a wear level by creating a line of best fit or curve of best fit for the pixel-time mappings. If the wear level is above a wear threshold value (block970: YES) processing returns to block920. If the wear level under a wear threshold value (block970: NO), alert manager180may generate an alert that one or more of the GET need replacement. In some embodiments, after the alert is generated, processing returns to block920.

Throughout the above description, certain components of wear detection computer system110were described to perform certain operations. But, in some embodiments of wear detection computer system110, other components may perform these operations other than what is described above. In addition, wear detection computer system110may include additional or fewer components than what is presented above in example embodiments. Those of skill in the art will appreciate that wear detection computer system110need not be limited to the specific embodiments disclosed above.

INDUSTRIAL APPLICABILITY

The systems and methods described herein can be used in association with operation of work machines at a worksite that are excavating, moving, shaping, contouring, and/or removing material such as soil, rock, minerals, or the like. These work machines can be equipped with a bucket used to scoop, dig, or dump the material while at the worksite. The bucket can be equipped with a series of ground engaging tools (GET) to assist with the loosening of the material during operation. The work machines can also include a system having a processor and memory configured to perform methods of wear detection according to the examples described herein. The system and methods can detect wear of work machine components such as GET. In some examples, the system and methods can capture, from a camera associated with the work machine, video of the work machine component for wear detection processing. In some examples, the system and methods select images from the video captured by the camera for processing using an image template that is compared to the video images. The template can include a sample image of a bucket with unworn GET associated with the work machine. Use of a bucket-tool template, as opposed to a template of a single unworn tooth, can decrease the resources required for image selection processing. Since a bucket-tool template is larger than a template having only one GET, it will take fewer processing cycles to compare the template to the captured images. Use of a bucket-tool template can also increase accuracy of image selection and reduce false positives. Since clumps of dirt, debris, or rock fragments at the worksite can be similar in size and shape to a single tooth, traditional wear detection systems using a single-tooth template can mistake such materials for GET causing processing errors or false positives. Thus, use of a bucket-tool template can decrease errors in processing over a computer vision system or method using single-tooth templates.

In some examples, the system and methods select images for wear detection by identifying individual GET and comparing the pixel counts for the GET in the selected image to an expected pixel count for the GET based on the bucket-tool template. In some examples, the system and methods can identify individual GET based on the location of the unworn GET in the bucket-tool template—it can leverage the approximate location of the GET relative to the bucket in the selected image based on the known location of GET in the bucket-tool template, and standard computer vision techniques (e.g., a convolutional filter) can be employed to identify the specific GET within the selected image. By narrowing the focus of the standard computer vision techniques to the approximate location of the GET in the selected image, processing time is reduced.

In some examples, the system and methods determine wear detection by capturing images of the GET over a period of time. For example, the system and method can capture multiple images of a GET, at multiple instances of time, over the period of time. The system and methods can determine, using the example embodiments disclosed herein, a pixel count for the GET at the multiple instance of time. The pixel counts can be mapped to the instance of time, and the system and method can determine a wear trend based on the rate of change in the pixel count. The system and method can predict when GET need replacement based on the wear trend. By using pixel counts of GET over time, the system and methods can reduce errors in wear detection by minimizing the impact of false positives as such false positives are outliers to the wear trend and represent statistical noise to the wear trend. In addition, the disclosed system and methods require less processing resources than machine learning or neural networks techniques for determining wear of machine components. Thus, the disclosed system and methods are more efficient than wear detection systems that rely on machine learning or neural network techniques for identifying wear using computer vision techniques as they are more accurate and require less processing resources.

While aspects of the present disclosure have been particularly shown and described with reference to the examples above, it will be understood by those skilled in the art that various additional embodiments may be contemplated by the modification of the disclosed devices, systems, and methods without departing from the spirit and scope of what is disclosed. Such embodiments should be understood to fall within the scope of the present disclosure as determined based upon the claims and any equivalents thereof.