Method and apparatus for locating a wear part in an image of an operating implement

A method and apparatus for locating and/or determining the condition of a wear part in an image of an operating implement associated with heavy equipment is disclosed. The method involves capturing at least one image of the operating implement during operation of the heavy equipment, the image including a plurality of pixels each having an intensity value. The method also involves selecting successive pixel subsets within the plurality of pixels, and processing each pixel subset to determine whether pixel intensity values in the pixel subset meet a matching criterion indicating a likelihood that the pixel subset corresponds to the wear part. The matching criterion is based on processing a labeled set of training images during a training exercise prior to capturing the at least one image of the operating implement.

BACKGROUND

This disclosure relates generally to image processing and more particularly to processing of images for locating a wear part in an image of an operating implement.

2. Description of Related Art

Heavy equipment used in mines and quarries commonly includes an operating implement such as a loader, an excavator or a face shovel for digging, loading, manipulating, or moving material such as ore, dirt, or other waste. In many cases the operating implement has a sacrificial Ground Engaging Tool (GET) which often includes hardened metal teeth and adapters for digging into the material. The teeth and/or adapters may become worn, damaged, or detached during operation. Such teeth and/or adapters are commonly referred to as wear parts, and may also include other parts such as lip shrouds between teeth. These wear parts are subjected to a wearing due to contact with often abrasive material and are considered to be sacrificial components which serve to protect longer lasting parts of the GET.

In a mining or quarry operation, a detached wear part, e.g., a missing tooth or adapter, may damage downstream equipment for processing the ore. An undetected wear part can also cause safety risk since if the tooth enters an ore crusher, for example, the tooth may be propelled at a very high speed due to engagement with the crusher blades thus presenting a potentially lethal safety risk. In some cases the wear part may become stuck in the downstream processing equipment such as the crusher, where recovery causes downtime and represents a safety hazard to workers. The wear part may also pass through the crusher and may cause significant damage to other downstream processing equipment, such as for example longitudinal and/or lateral cutting of a conveyor belt. This may be a particular problem with loader or excavator teeth which are typically longer and narrower than shovel teeth. Additionally, knowing the current size and length of wear part may also be of importance in mining or quarry operations. Identifying the condition of wear parts such as their size (length) helps to predict when those wear parts need to be replaced or relocated to prevent damage to the operating implement and also to prevent operational inefficiencies due to unscheduled maintenance.

Camera based monitoring systems are available for monitoring wear parts on operating implements associated with heavy equipment such as front-end loaders, wheel loaders, bucket loaders, backhoe excavators, electric face shovels, and hydraulic face shovels. Such monitoring systems may use bucket tracking algorithms to monitor the bucket during operation, identify the teeth and other wear parts on the bucket, and provide a warning to the operation if a part of the operating implement becomes detached.

There remains a need for methods and apparatus for locating and/or identifying the condition of wear parts within an image of an operating implement associated with heavy equipment.

SUMMARY OF THE INVENTION

In accordance with one disclosed aspect there is provided a method for locating a wear part in an image of an operating implement associated with heavy equipment. The method involves capturing at least one image of the operating implement during operation of the heavy equipment, the image including a plurality of pixels each having an intensity value. The method also involves selecting successive pixel subsets within the plurality of pixels, and processing each pixel subset to determine whether pixel intensity values in the pixel subset meet a matching criterion indicating a likelihood that the pixel subset corresponds to the wear part. The matching criterion is based on processing a labeled set of training images during a training exercise prior to capturing the at least one image of the operating implement.

Processing each pixel subset may involve at least one of directly processing the pixel intensity values, extracting features associated with pixels in the pixel subset, and/or generating a histogram of oriented gradients for the pixel subset.

Processing each pixel subset may involve processing each pixel subset through a corresponding plurality of input nodes of a neural network, each input node having an assigned weight and being operable to produce a weighted output in response to the received intensity value.

The method may involve receiving the weighted outputs from the input nodes at a plurality of hidden nodes of the neural network, each hidden node having an assigned weight and being operable to produce a weighted output in response to the received weighted output from the input nodes.

The method may involve receiving the weighted outputs from the hidden nodes at one or more output nodes, the one or more output nodes having an assigned weight and being operable to produce a weighted output in response to the weighted outputs received from the hidden nodes.

The plurality of hidden nodes comprise may include hidden nodes in one or more layers, each successive layer of nodes operating on the outputs produced by a preceding layer.

Capturing at least one image may involve capturing a sequence of images of the operating implement during operation, the one or more layers including a memory layer including nodes operable to cause results of the processing of previous images of the operating implement to configure the neural network for processing subsequent images of the operating implement.

Processing the labeled set of training images during the training exercise may involve processing labeled sets of sequential training images.

Determining whether pixel intensity values in the pixel subset meet the matching criterion may involve determining whether the weighted output exceeds a reference threshold.

Receiving the weighted outputs from the input nodes at a plurality of hidden nodes may involve receiving the weighted outputs from the input nodes at a first plurality of hidden nodes, and receiving weighted outputs from the first plurality of hidden nodes at a second plurality of hidden nodes, each of the second plurality of hidden nodes having a weight and being operable to produce a weighted output in response to the received weighted output from the first plurality of hidden nodes.

Processing each pixel subset may involve processing each pixel subset using a convolutional neural network having a plurality of layers including at least one convolution layer configured to produce a convolution of the pixels in each pixel subset, and processing the labeled set of training images may involve processing training images to cause the convolutional neural network to be configured to implement the matching criterion for producing a pixel classification output indicating whether pixels in the pixel subsets correspond to the wear part.

Producing the convolution may involve producing the convolution using a sparse kernel having entries separated by rows and columns of zero values.

Producing the convolution may involve producing the convolution using a sparse kernel having entries separated by a plurality of rows and a plurality of columns of zero values.

The convolutional neural network may include a pooling layer configured to process the convolution to provide a plurality of pooling outputs, each pooling output being based on values associated with a plurality of pixels in the convolution.

The pooling layer may implement one of a max-pooling, an average pooling, and a stochastic pooling process.

The method may involve resampling the image to produce a resampled plurality of pixels and processing using the convolutional neural network may involve processing the resampled plurality of pixels, the convolutional neural network having been configured to implement the matching criterion using a correspondingly resampled plurality of training images.

Resampling the pixel data may involve at least one of up-sampling the image and down-sampling the image to produce the resampled plurality of pixels.

Capturing at least one image may involve capturing a sequence of images of the operating implement during operation and the convolutional neural network may include at least one memory layer operable to cause results of the processing of previous images of the operating implement to configure the convolutional neural network for processing subsequent images of the operating implement for producing a pixel classification output for the sequence of images.

Processing the labeled set of training images during the training exercise may involve processing labeled sets of sequential training images.

The labeled training set of images may include a set of images that have been labeled by a user.

The labeled training set of images may include a set of images that have been labeled by a computer implemented labeling process.

The training images may include images of various examples of the wear part labeled as including the wear part, and other images labeled as not including the wear part.

Selecting successive pixel subsets within the plurality of pixels may further involve processing the plurality of pixels to determine whether the operating implement is present in the image, and if the operating implement is present in the image, restricting the plurality of pixels to pixels within a region of interest that includes the operating implement prior to selecting successive pixel subsets within the plurality of pixels.

Processing the plurality of pixels to determine whether the operating implement is present in the image may involve selecting at least one pixel subset within the plurality of pixels, processing the at least one pixel subset to determine whether pixel intensity values in the at least one pixel subset meet an operating implement matching criterion indicating a likelihood that the operating implement is within the at least one pixel subset, and the operating implement matching criterion may be based on processing a labeled set of training images during a training exercise prior to capturing the at least one image of the operating implement.

Selecting successive pixel subsets within the plurality of pixels may involve one of selecting successive pixel subsets having a fixed predetermined size, and calculating a pixel subset size based on the captured image.

The matching criterion may include a plurality of weights corresponding to pixels within the pixel subset and processing each pixel subset may involve for each pixel in the pixel subset, calculating a product of the pixel intensity and the corresponding weight to determine a weighted output for the pixel, and determining whether the pixel subset meets the matching criterion by determining whether a combination of the weighted outputs for the pixel subset exceed a threshold.

Determining whether the weighted outputs for the pixel subset exceed a threshold may involve combining the determined weighted outputs for the pixel subset, determining whether the combined weighted output exceeds a threshold.

Capturing the at least one image may involve capturing a plurality of images of the operating implement during operation of the heavy equipment and the selecting and processing of pixel subsets within the plurality of pixels may be performed for each image and the method may further involve determining whether pixel intensity values in the pixel subsets meet a matching criterion in successive images of the plurality of images.

Capturing the at least one image may include capturing the at least one image using an image sensor having a wavelength sensitivity in at least one of the visible spectrum and the infrared spectrum.

The method may involve determining a dimensional attribute of the wear part.

The method may involve determining at least one of whether the condition of the wear part is satisfactory based on a pre-determined criteria, and a prediction of a time of failure of the wear part based on a rate of wear of the wear part over time.

In accordance with another disclosed aspect there is provided an apparatus for locating a wear part in an image of an operating implement associated with heavy equipment. The apparatus includes an image sensor for capturing at least one image of the operating implement during operation of the heavy equipment, the image including a plurality of pixels each having an intensity value. The apparatus also includes a processor circuit operably configured to select successive pixel subsets within the plurality of pixels. The processor circuit is also configured to process each pixel subset to determine whether pixel intensity values in the pixel subset meet a matching criterion indicating a likelihood that the pixel subset corresponds to the wear part. The matching criterion is based on processing a labeled set of training images during a training exercise prior to capturing the at least one image of the operating implement.

The image sensor may be an image sensor having a wavelength sensitivity in at least one of the visible spectrum and the infrared spectrum

Other aspects and features will become apparent to those ordinarily skilled in the art upon review of the following description of specific disclosed embodiments in conjunction with the accompanying figures.

DETAILED DESCRIPTION

Referring toFIG. 1, an apparatus for locating a wear part and/or determining the condition of the wear part of an operating implement associated with heavy equipment is shown at100. The apparatus100includes an image sensor102and a processor circuit104. The image sensor102is in communication with the processor circuit104via a communications link106. The apparatus100also includes a display108, in communication with the processor circuit104via a communications link110.

Referring toFIG. 2, in one embodiment the image sensor102is mounted on a wheel loader250. The wheel loader250includes a bucket operating implement252carried on side-arms254. The bucket252has a plurality of wearable teeth256, which are subject to wear or damage during operation. The image sensor102is mounted generally between the side-arms and has an associated field of view258. The teeth256on the bucket252will generally move into and out of the field of view258during operation. The mounted image sensor102is shown in more detail in an insert260. In this embodiment the image sensor102is implemented as a thermal imaging sensor, which is sensitive to infrared wavelength ranges. Thermal imaging is particularly suitable for monitoring the teeth256of the wheel loader250shown inFIG. 2since the image sensor102views the back of the bucket252and there is also less chance of ore getting stuck on the bucket and blocking the view of the teeth.

Referring back toFIG. 1, the image sensor102includes a mounting bracket112that mounts the sensor to the wheel loader250under the bucket operating implement252. In general the bracket112is configured to mount to a specific loader type, in this case a Caterpillar™ wheel loader and may provide shock and vibration isolation for the image sensor102. The image sensor102is protected from falling debris from the bucket252by a protective housing114. In some embodiments a lens cleaning system (not shown) may be enclosed within the protective housing114for delivering high pressure washer fluid and/or a compressed air flow for cleaning the upward facing image sensor102, which is exposed during operation.

Referring toFIG. 3, a similar sensor to the sensor102may be installed on other heavy equipment, such as the backhoe excavator shown at120. The backhoe120includes an excavator bucket122having teeth124. In the embodiment shown two possible locations for the image sensor are shown at126and128, each having a respective field of view130and132. The image sensor126,128is shown in more detail in the insert134and includes a visible spectrum image sensor136and an illumination source138for illuminating the field of view130or132.

Referring toFIG. 4a similar sensor to the sensor102may alternatively be installed on a hydraulic face shovel shown at140, which includes an excavator bucket142having teeth144. An image sensor146is installed on a linkage148that supports the excavator bucket142. The image sensor146is shown in more detail in the insert150and includes a visible spectrum image sensor152and an illumination source154for illuminating the excavator bucket142

Referring toFIG. 5, in another embodiment an image sensor212may be mounted on an electric shovel200. The image sensor212is mounted at the end of a boom202of the shovel200and is oriented to provide images of an operating implement of the shovel, in this case a bucket204. The image sensor212is shown in more detail in the insert214and includes a mounting bracket216, a housing218, and in this embodiment, an illumination source220. In this embodiment the image sensor212has a wavelength sensitivity in the visible spectrum but in other embodiments a thermal sensor may be implemented and the illumination source220may be omitted. The mounting bracket216may be configured to provide vibration and/or shock isolation for the image sensor212and illumination source220. The bucket204includes a plurality of teeth206, which in general for an electric shovel are configured as replaceable wear parts. The image sensor212has a field of view208(indicated by broken lines) that includes the bucket204and teeth206. In general, the field of view208is configured such that the bucket204remains in view while the shovel200is excavating an ore face during mining operations. In the embodiment shown inFIG. 5, the processor104and display108are both located within a cabin210of the shovel200. The display108is located to provide feedback to an operator of the shovel200.

In some embodiments, the apparatus100may also include an illumination source (not shown) for illuminating the field of view during low light operating conditions. In embodiments where the image sensor102or212is sensitive to infrared wavelengths, illumination may not be required due to the of teeth becoming warm during operation and providing good infrared image contrast even in low light conditions.

In other embodiments, the image sensor102may be mounted on other heavy equipment, such as hydraulic shovels, front-end loaders, wheel loaders, bucket loaders, and backhoe excavators.

Processor Circuit

A block diagram of the processor104is shown inFIG. 6. Referring toFIG. 6, the processor circuit104includes a microprocessor300, a memory302, and an input output port (I/O)304, all of which are in communication with the microprocessor300. In one embodiment the processor circuit104may be optimized to perform image processing functions. The microprocessor300may include a graphics processing unit334(GPU) for accelerating image processing tasks carried out by the processor circuit104. The microprocessor300also includes an interface port306(such as a SATA interface port) for connecting a mass storage unit308such as a hard drive or solid state drive. Program codes for directing the microprocessor300to carry out functions related to locating teeth within images of the bucket204or252may be stored in the memory302or the mass storage unit308.

The I/O304may also include a network interface310having a port for connecting to a network such as the internet or other local area network (LAN). Alternatively or additionally the (I/O)304may include a wireless interface314for connecting wirelessly to a wireless access point for accessing a network. The local network and/or wireless network may be implemented on the electric shovel200and may be used as the communications links106and110connecting between the image sensor102, the processor circuit104and the display108. Alternatively, the communications links106and110may be implemented using cables. Program codes may be loaded into the memory302or mass storage unit308using either the network interface310or wireless interface314, for example.

The I/O304also includes a display interface320having a display signal output322for producing display signals for driving the display108. In one embodiment display108may be a touchscreen display and the display interface320may also include a USB port324in communication with a touchscreen interface of the display for receiving input from an operator. The I/O304may also have additional USB ports (not shown) for connecting a keyboard and/or other peripheral interface devices.

The I/O304further includes an input port330for receiving image signals from the image sensor102. In one embodiment the image sensor102may be a digital camera and the image signal port330may be an IEEE 1394 (firewire) port, USB port, or other suitable port for receiving image signals. In other embodiments, the image sensor102may be an analog camera that produces NTSC or PAL video signals, for example, and the image signal port330may be an analog input of a framegrabber332.

In other embodiments (not shown), the processor circuit104may be partly or fully implemented using a hardware logic circuit including discrete logic circuits and/or an application specific integrated circuit (ASIC), for example.

Process for Locating the Wear Part

Referring toFIG. 7, a flowchart depicting blocks of code for directing the processor circuit104to locate a wear part, such as teeth256of the bucket operating implement252or teeth206of the bucket204, in an image of an operating implement is shown generally at400. The blocks generally represent codes that may be read from the memory302or mass storage unit308for directing the microprocessor300to perform various functions. The actual code to implement each block may be written in any suitable programming language, such as C, C++, C#, and/or assembly code, for example.

The process400begins at block402, which directs the microprocessor300to cause the image sensor102to capture an image of the operating implement. An example of a captured image of a portion of the bucket operating implement252shown inFIG. 3is shown at500inFIG. 8. The image500includes a plurality of pixels each having an intensity value representing the bucket252. In the embodiment shown the image500has been captured using a thermal imaging system sensitive to infrared wavelength ranges. Heating of the teeth due to friction arising from engagement of the wear parts with ore face being excavated during operations may provide a thermal image under almost any lighting conditions. In other embodiments the image may be captured using a visible wavelength imaging system. The image500includes background areas502that do not include any objects, areas504and506, which include objects that are not part of the operating implement, and an operating implement252. In the image500, the areas502and504and the bucket252have contrasting pixel intensities, which will in general depend on the level of illumination and other factors. Generally, the number of pixels in the image will be large and the size of each pixel will be small (for example 75 pixels per inch of displayed image).

Block404then directs the microprocessor300to select a pixel subset510within the plurality of pixels. For sake of illustration the pixel subset510is shown including only 60 pixels, but in practice the pixel subset would include well in excess of 60 pixels depending on the size of the wear part to be located within the image500. In general the pixel subset510is sized slightly larger than the wear part such that the subset will include the wear part such as the tooth256along with a portion of the background area502and the area of the operating implement252.

The process then continues at block406, which directs the microprocessor300to process the pixel subset510. In this embodiment, processing of the pixel subset510involves determining at block408, whether pixel intensity values in the pixel subset meet a matching criterion. In one embodiment the processing may be in accordance with actual pixel intensity values. In other embodiments other intensity based information may be extracted, for example by dividing the image into connected cells and compiling a histogram of gradient directions or edge directions within each cell.

If at block406, the pixel subset510meets the matching criterion, block408directs the microprocessor300to block410and the pixel subset is flagged as having a high likelihood of corresponding to the wear part i.e. one of the teeth256. Block410also directs the microprocessor300to save the location of the pixel subset510. The location of the pixel subset510may be saved by saving the pixel row and column numbers within the image500for a reference pixel within the flagged pixel subset. For example, a center pixel of the subset510may be saved as indicating the location of the wear part. Alternatively the row and column of the uppermost left hand corner may be used to reference the location of the pixel subset510. Block410then directs the microprocessor to block412. If at block408the pixel subset510does not meet the matching criterion the microprocessor300is directed to block412.

Block412directs the microprocessor300to determine whether further pixel subsets are still to be processed, in which case the microprocessor is directed to block414and is directed to select a successive pixel subset for processing, which involves moving the pixel subset510over in the direction of the arrow512. In one embodiment successive pixel subsets are horizontally overlapped by between about 70%-85% to provide for reliable wear part detection within the image. In other embodiments a greater or lesser overlap between pixel successive subsets may be implemented in accordance with a desired tradeoff between reliable detection and improved processing time per image. For example, where processing speed is not an issue, successive pixel subsets may be spaced apart by only a single pixel.

Block412then directs the microprocessor300back to block406, and blocks406and408are repeated for each successive pixel subset510. Once the pixel subset510reaches a right hand edge of the image500, the pixel subset may be moved down (i.e. to the location of the pixel subset510a) and may be moved either back to the left edge of the image to continue in the direction512. Alternatively, the pixel subset510may be moved from the right edge toward the left edge of the image500in the direction indicated by the arrow514. In one embodiment successive pixel subsets are vertically overlapped by between about 70%-85% to provide for reliable wear part detection within the image, while in other embodiments a greater or lesser vertical overlap between pixel successive subsets may be implemented. If at block412the microprocessor300determines that no further pixel subsets are to be processed, the process ends at block416. If no pixel subsets are flagged at block410, then the image500is considered not to include the wear part.

Matching Criterion

Referring toFIG. 10, a process450for making the determination at block408of whether the pixel subset510meets the matching criterion is described for the pixel subset510b. The process embodiment is described for a pixel subset510bin the image500that is generally centered over the tooth256. The pixel subset510bis shown in enlarged view inFIG. 9along with a portion of the image500that includes the tooth256. Referring toFIG. 9, the pixels602in the pixel subset510bare numbered inFIG. 9using indices x (604) and y (606) for ease of reference. Each pixel602within the pixel subset510bhas an associated weight Wxy, which for sake of illustration is an integer value between 0 and 256 in the embodiment shown. In this embodiment the weights Wxyare predetermined and saved in the memory302of the processor circuit104shown inFIG. 7.

The process450begins at block452, which directs the microprocessor300to read the pixel intensity Ixyof the first pixel (0,0) in the pixel subset510b. Block454then directs the microprocessor300to read the weight Wxyassociated with the first pixel (0,0) in the pixel subset510bfrom the memory302of the processor circuit104.

The process then continues at block456, which directs the microprocessor300to calculate the product of the pixel intensity Ixyand the weight Wxy. Block458then directs the microprocessor300to accumulate a sum S of the values of Rxy. In this embodiment the products of Ixyand Wxyare thus combined by taking a simple sum over the pixels602in the pixel subset510b. If at block460, the pixel (x,y) was not the last pixel (i.e. pixel (5,9)) in the subset, the microprocessor300is directed to block462where the next pixel is selected (for example pixel (0,1)). Block462then directs the microprocessor300back to block452, and blocks452to460are repeated for pixel (0,1). If at block460, the pixel (0,1) was the last pixel (i.e. pixel (5,9)) in the pixel subset510b, the process450is completed and the process returns to block408inFIG. 7. Block458thus directs the microprocessor300to accumulate a sum of the products ΣRxyof pixel intensity Ixyand the weights Wxyfor each of the pixels602in the pixel subset510b.

In this embodiment, at block408the ΣRxyvalue produced by the process450may be compared to a threshold value, and if the threshold is exceeded then the pixel subset510bis considered to correspond to a tooth and would then be flagged accordingly in the process400. When the pixel subset510bis located over a background area such as areas502,504, or506, the correlation between higher weights Wxyin the pixel subset will generally be poor, resulting in lower values of ΣRxy. However, when the pixel subset510bhas a tooth located within the pixel subset, the higher weights Wxyassigned to certain pixels in the pixel subset when multiplied by higher pixel intensities produce higher values of ΣRxy. The threshold may be empirically selected to provide a desired confidence level for identifying tooth images within the image500. Alternatively, the threshold may be dynamically selected based on properties of the image500.

In practice, if there is a significant degree of overlap between successive pixel subsets510, several overlapping pixel subsets may result in ΣRxyvalues above the threshold and would thus be flagged as including a tooth. In this case, an additional step may be added to the process400to select only one pixel subset out of a plurality of overlapping pixel subsets having the highest ΣRxyvalue to avoid multiple detection of the same tooth within the image500.

Generating Matching Criterion

In one embodiment the matching criterion may be generated using a supervised learning process based on images of the wear part. An embodiment of a supervised learning process is shown inFIG. 11at550. While the process550may be implemented on the processor circuit104shown inFIG. 6, it would generally be convenient to use a desktop computer for performing the learning process. The process550begins at block552, which directs the computer to receive a plurality of images. The images may be conveniently stored in a sub-directory on a hard drive of the computer. The plurality of images may include various examples of the wear part being identified, such as tooth images from various different buckets for shovels and/or other heavy operating equipment including teeth from different locations on a particular bucket. The plurality of images may also include images that do not correspond to the wear part, and preferably images of portions of the shovel or other heavy equipment that could be mistaken for a wear part.

Block554then directs the computer to display the first image. Referring toFIG. 12, a screenshot generated by a supervised learning application run on the desktop computer is shown at650. The screenshot650includes a displayed image652of a bucket of a loader (such as the loader250shown inFIG. 3) having a plurality of teeth. The process550then continues at block556, which directs the computer to receive user input of one or more bounding boxes that identify individual teeth in the image. A plurality of such bounding boxes are shown in the screenshot650, each surrounding a respective tooth. The supervised learning application provides a set of control buttons that allow the user to locate the boxes around each tooth using drag and drop functions. The control buttons also provide access to functions for adjusting each box so that a majority of the area of the box is occupied by the tooth while some space remains below and to the sides of the tooth.

Block558then directs the computer to extract the individual tooth images on the basis of the user input bounding boxes654. The pixels within each bounding box654may be saved as a separate image file and either named or grouped in a directory to indicate that the images have been labeled by the user as teeth images.

Block560then directs the computer to determine whether the last image in the plurality of images has been processed. If images remain to be processed, the process continues at block562, where the computer is directed to select the next image. Blocks554-560are then repeated for each successive image until the supervised learning has been completed and all of the teeth in the plurality of images have been extracted and labeled as tooth images.

The process550then continues at block564, which directs the computer to read each extracted images. Block566then directs the computer to process the image to generate and refine the matching criterion based on the image. Block568directs the computer to determine whether further extracted images remain to be processed, in which case block570directs the computer to select the next image for processing. If at block568all of the extracted images have been processed, the computer is directed to block572and the matching criterion is saved as the matching criterion for use in the process400shown inFIG. 7.

In one embodiment, the supervised learning may further involve providing images that are labeled as not including the wear part. For example, referring back toFIG. 12, portions656of the image652may be selected by the user or may be randomly selected and labeled as non-teeth images. The non-wear part images may be used to generate a matching criterion that is less sensitive to generating false positive wear part identifications within the image.

Neural Network Implementation

In one embodiment the processing of the pixel subset at block406inFIG. 7may be implemented using an artificial neural network. Referring toFIG. 13, an example of a small neural network is shown at800. The neural network800includes an input layer802including inputs x1, x2, x3, . . . xj. The inputs xjmay represent pixel intensity values for pixels within the pixel subset, for example. The neural network800also includes one or more hidden layers each including a plurality of nodes or neurons. In this case the neural network800includes hidden layers804and806. Each neuron in the hidden layer804has a respective activation function ƒ1, ƒ2, ƒ3, . . . ƒn, where the activation functions have the form:
ƒ(ΣWxi+b),  Eqn 1
and where W is a weight assigned to each neuron, and a bias b. Each layer of the neural network may or may not have a bias, which is a neuron having a constant value of “1” and is connected it to each neuron in the layer. The weights W of these bias neurons also need to be determined during a training exercise. If the bias is not used then the value of “b” in Eqn 1 is set to zero.

Similarly the hidden layer806includes neurons having activation functions g1, g2, . . . gn. The activation function for each of the neurons in the layer804produce an output in response to the inputs xjwhich are received by neurons in the hidden layer806. The activation functions for each of the neurons in the layer806similarly produce an output in response to the inputs from neurons in the layer804. In other embodiments the hidden layers804and806may include a larger number of neurons, each having an activation function.

The neural network800also includes an output layer808including a neuron having a activation function h, which may have a similar form to the activation function above and produces an output result hWb(xi).

By selecting appropriate weights W and b for the neurons in the layers804,806, and808, the neural network800can be configured to produce an output result that indicates whether an input pixel subset represents a specific wear part or not. Evaluation of the output result for any particular input pixel subset captured during operation would thus involve evaluating the activation functions ƒ1, ƒ2, ƒ3, . . . ƒn, g1, g2, . . . gn, and h using the stored values of W and b to determine outputs for the layers804-808. The output result hWb(xi) would then indicate whether the input pixel subset has been determined to correspond to the wear part or not. In this case the output result may be a confidence value which can be compared with a threshold to convert the result into a binary “0” or “1” indicating whether the wear part has been located or not.

In the above embodiment the processing at block566inFIG. 11may be implemented by training the neural network800using a plurality of pixel subsets representing the specific wear part such as a tooth. The training may be performed prior to capturing images of the operating implement during operation, and may be saved in the memory as a set of data including the weights W and b. Selection of appropriate weights may involve a supervised learning process. In one embodiment, the process may involve a user selecting a variety of pixel subsets which are labelled as including the wear part and then feeding the pixel subsets through the neural network800. The desired output for each image may be designated as y, where in this case y=1 indicates the pixel subset includes an identified wear part while y=0 would indicate that the pixel subset does not include a wear part. A cost function for optimizing the neural network800may then be written as:

J⁡(W,b,xi,yi)=12⁢hW,b⁡(xi)-yi2,Eqn⁢⁢2
which is a half squared error cost function. For a training set having m pixel subsets, the overall cost function is:

Other terms may be added to the cost function above, such as a regularization term that decreases the magnitude of the weights to prevent over fitting. The cost function J is then minimized using a minimization algorithm such as a batch gradient descent minimization that determines values for W and b that provide a closest match between the output result hWb(xi) and the assigned y value for each of the training pixel subsets.

Various other training approaches may be implemented for predetermining the weights W and b associated with the matching criterion. In some embodiments the matching criterion may be completely predetermined during the training exercise. In other embodiments, the matching criterion may be partially predetermined during a training exercise and modified during operation of the heavy equipment in a recurrent neural network implementation as described later herein.

Alternative Implementation

Referring toFIG. 14, a flowchart depicting blocks of code for directing the processor circuit104to locate the a wear part in an image of an operating implement in accordance with an alternative embodiment is shown generally at700. The process begins at block702, which directs the microprocessor300to capture an image of the operating implement including a plurality of pixels. Block704then directs the microprocessor300to select a pixel subset for processing. The image500shown inFIG. 8is reproduced inFIG. 15. However, referring toFIG. 15in the embodiment shown a pixel subset752is selected to specifically cover a number of the teeth256or all of the teeth associated with the operating implement252rather than just a single tooth.

Block706then directs the microprocessor300to process the pixel subset, and block708directs the microprocessor to determine whether pixel intensity values in the pixel subset meet an operating implement matching criterion indicating a likelihood that the operating implement is within the at least one pixel subset. The determination at block708of whether the pixel subset752meets the matching criterion may be implemented generally as described above for the pixel subset510b, except that the weights Wxyin this embodiment are associated with the operating implement as a whole and not just the teeth256. The operating implement matching criterion in block708may also be determined based on processing a labeled set of operator implement training images during a training exercise similar to that described in connection with the process550ofFIG. 11.

If at block708, the pixel intensity values meet the matching criterion, the process continues at block710, which directs the microprocessor300to flag the pixel subset as corresponding to the operating implement. Block710may also direct the microprocessor300to save the pixel subset in the memory300. Block712then directs the microprocessor300to determine whether the last pixel subset in the image500has been processed. If pixel subsets remain to be processed, block712directs the microprocessor300to block714and the next pixel subset is selected and the microprocessor is directed back to block706to process the next selected pixel subset. The pixel subset752is thus scanned through the image500as generally described above for the pixel subset510. If at block708, the pixel intensity values do not meet the matching criterion, the process continues at block712.

If at block712no pixel subsets remain to be processed, block712directs the microprocessor300to block716. Block716then directs the microprocessor300to determine whether the operating implement was located. If at block710any one of the pixel subsets had been flagged as meeting the operating implement matching criterion then the operating implement is considered to have been located and block716directs the microprocessor back to block404of the process400inFIG. 7. Identification of the wear part such as the teeth256may then proceed on the basis of the flagged pixel subset that includes the located operating implement252. If more than one pixel subset has been flagged as meeting the matching criterion, block716will additionally direct the microprocessor300to select the pixel subset with the highest result (i.e. the highest ΣRxy). If at block716the operating implement was not located, block716directs the microprocessor300to block718where the microprocessor is directed to adjust the pixel subset size. Block718then directs the microprocessor300back to block704and blocks702-716are repeated with the adjusted pixel subset. In general, captured images may have varying scale and/or aspect since the bucket252will move with respect to the image sensor102(FIG. 2) during operation providing differing perspectives for successively captured images. The size of the pixel subset752may this be initially set to a default value and later increased to provide a greater likelihood that the operating implement252is located within the image500.

The process700thus facilities first identifying the bucket within the image using a matching criterion based on images of a variety of buckets, and then identifying the wear part such as the teeth once the bucket has been identified in the image.

Tracking the Wear Part

The processes described above have focused on locating a wear part within a single image. In practice, the image sensor102may be implemented using a video camera that produces 30 frames per second. Even in embodiments where the operating implement moves fairly rapidly, a series of image frames will be captured and at least a portion of these image frames may be processed to locate the wear part. For a fixed location of the image sensor102(for example on the boom202of the electric shovel200), the teeth256will appear in many consecutive frames but will have a varying scale depending on how far away the bucket is from the image sensor102. The teeth will also have a varying aspect due to the angle between the bucket252and the field of view208of the image sensor102.

The processing of the images to locate the teeth256may result in a one of the teeth not being located. While this event may be interpreted as an indication that the tooth has become detached or broken off, the event may also be a result of imperfect processing and matching at blocks408and410ofFIG. 7. Referring toFIG. 16, a process embodiment in which tooth identification is based on a plurality of frames is shown generally at850. The process begins at block852, which directs the microprocessor300to capture a plurality of images of the operating implement. Conventional video cameras may produce 30 frames per second, however the processor circuit104may not have sufficient capacity to process frames at this rate and thus some of the frames may be discarded by the processor circuit or the capture rate of the image sensor102may be reduced.

Block854then directs the microprocessor300to process each image and to locate the wear part in the image500(or multiple wear parts in the case of the teeth plurality of teeth256of the bucket252). The processing may be in accordance with the process400inFIG. 7or may be a combination of process400and the process700shown inFIG. 14. The saved row can column location of each resulting pixel subsets that is flagged as including a tooth thus provides a location of the teeth within the image500.

Block856then directs the microprocessor300to process the tooth locations to extract locations for the teeth over a plurality of images. In one embodiment, a one-dimensional (1-D) vector representing the locations of the flagged pixel subsets is generated for each of the plurality of images. The 1-D may be sized in accordance with a known number of teeth256for a particular bucket252. Several of the 1-D vectors are then combined into a two dimensional (2-D) observation matrix. An example of a set of tooth locations over multiple images is depicted graphically inFIG. 17, where locations of each detected tooth are indicated by the numbers 1-9 shown at904. The variation of tooth location can be seen as being restricted to a few different paths, for example the paths904and906of the 5thand 6thteeth. In one embodiment a principle component analysis is applied to extract sub-components indicating principle variations of tooth locations. Referring toFIG. 18, the highest variations in tooth location is found in two principle components shown at920and922.

The process then continues at block858which directs the microprocessor300to determine whether a tooth is consistently missing in successive images or only sporadically missing in one or more images based on the principle component analysis. In one embodiment, principal components are forming a matrix of P by 2*N, where P is a number of principal components that are considered to capture 95% of variation in tooth location. N is the known number of teeth in an undamaged bucket, each having an x and y center location within the image. Assuming that M teeth have been detected within the image (M<N), a 2*M column is selected from the principal component matrix, which has a total 2*N variables. In other words, the 2*(N−M) columns from the principal component matrix are set aside and a sub-principal component is generated, which has a dimension of P by 2*M. The 1-D detected location center of teeth is then projected (which has a length of 2*M to the sub-principal component of size P by 2*M) to obtain a set of coefficients. The projection is solved by least square estimation, and an error of the estimation is computed. The process of selecting 2*M columns out of 2*N columns is then repeated and the estimation error is computed each time. The 2M columns that result in a minimum error, provides an estimate of the location of the detected teeth. Coefficients that correspond to the minimum error are multiplied by the 2*(N−M) columns which were not detected, and will identify the location of the un-detected or missing teeth.

As a result of the teeth detection and tracking in the process850, teeth locations are estimated in successive images and the microprocessor300is directed to discard false positives and estimate the location(s) of possible missing teeth.

If it is determined at block858that a tooth is missing from successive images then the microprocessor is directed to block860and the tooth or other wear part is flagged as being missing. In one embodiment, a missing tooth is identified when the tooth is missing from 15 or more successive images where images are processed at a rate of about 10 images per second.

Block862then directs the microprocessor300to generate an alert for the operator, for example by displaying an alert message on the display108, sounding a buzzer, illuminating a warning light, or a combination of alerts.

If it is determined at block858that there is no consistently missing tooth in successive images, the microprocessor300is directed back to block852, and further images are captured and the process850is repeated.

In some embodiments, following block854, the neural networks800or930may be used not only to detect the wear part, but also to measure a dimensional attribute such as the length and/or size of the wear part. This process may be generally in accordance with the process400inFIG. 7or may be a combination of process400and the process700shown inFIG. 14. The process850may include additional blocks864-870, which may be implemented in parallel with the blocks856-860. Block864directs the microprocessor300to determine the dimensional attributes of the wear part in pixels. Block866then directs the microprocessor300to convert the size of wear part from pixels into real world dimensions, typically expressed in “cm” or “inches”. In one embodiment a reference of known size within the image of the operating implement may be used to provide the scaling between pixels and real world dimensions. In one embodiment, a width of the operating implement or a distance between the pins of two end teeth may be used as the known references. Other parameters that have effect on the length measurement are the position of the operating implement, distance from the operating implement to the image sensor, and its orientation relative to the imaging sensor. A neural network may be trained to find the direction of the operating implement according to the coordinates of the operating implement and teeth in the image.

The process then continues at block868, which directs the microprocessor300to store the dimensional attribute of the wear part in the memory302or mass storage unit308(shown inFIG. 6). Block870then directs the microprocessor300to compare the measured dimension against previously measured dimensions and to determine whether the wear part condition is satisfactory. If the condition is satisfactory block870directs the microprocessor300back to block852and the process is repeated. If at block870the wear part condition is not satisfactory, the microprocessor300is directed to block862and an alert is generated. For example, if at block870, the length dimension of the wear part has been reduced below a replacement threshold criterion the condition may be determined not satisfactory. Alternatively or additionally, the reduction in length dimension may be tracked over time and a rate of wear used to predict failure of the part or provide information for exchanging parts.

Recurrent Neural Network Implementation

In the embodiment shown inFIG. 16, a series of image frames are captured and processed to locate the wear part. However, each successive image frame evaluated in against the same matching criterion established during the training phase. In embodiments that implement a neural network, the network may include a memory layer including nodes operable to cause results of said processing of previous images of the operating implement to configure the neural network for processing subsequent images of the operating implement. Such neural networks that exhibit temporal behavior are known as recurrent neural networks and in one embodiment may include long short-term memory units (LSTM) that is operable to modify the matching criterion based on a time series of inputs (i.e. the successive images). LSTM units are implemented to memorize some prior values for some length of time, thus altering the configuration of the neural network for processing of successive captured images. The recurrent neural network may be trained using sets of sequential labeled training images, for configuring weights of the LSTM units and other nodes of the network. The recurrent neural network may include several layers of LSTM units added to a neural network implementation.

Combinational Neural Network Implementation

Referring toFIG. 19, a convolutional neural network is depicted schematically at930. The captured image is represented by the rectangle932and includes a plurality of pixels934. In this embodiment the image932may include pixel intensity values within a wavelength range of interest, such as infrared wavelengths that convey thermal image features associated with the operating implement. In other embodiments additional pixel data sets of different wavelength ranges may be included. In neural network terms, each pixel934acts as an input neuron for the neural network930.

The neural network930also includes a convolution layer936having a plurality of neurons938. In the embodiment shown, a pixel940in the input image932is to be classified (i.e. as corresponding to a wear part or not corresponding to a wear part), and the classification is performed on the basis of a patch of pixels942surrounding the pixel940. In the embodiment shown, the patch942is illustrated as an 11×11 pixel patch, however the patch may be sized in accordance with the sizes of features in the captured image. In some embodiments, the patch may be selected sized based on an initial size estimate for the patch942.

In the neural network930each neuron938in the convolution layer936is connected to a subset of the input neurons in the image932by defining a convolution kernel944. The convolution kernel944in this embodiment has a size of 3×3 pixels and a set of 9 weights W (946). The kernel944is centered over successive pixels in the patch942of the image932effectively connecting a corresponding neuron938in the convolution layer936to corresponding subsets of the pixels in the captured image932. For the example of pixel940, the convolution kernel944is passed over the patch942and the weights946are applied to the pixel intensity values to produce the output for a neuron in the convolution layer936that corresponds to the input pixel940. The convolution kernel944similarly connects and produces outputs for other corresponding neurons938in the convolution layer936. In this embodiment the convolution kernel944applies the same weights W to each subset of input pixels and thus will become sensitive to the same features in the input pixels when the weights are subsequently determined during a training of the neural network930.

In one embodiment pixel-wise processing may proceed at a stride of 1 or at a stride greater than 1. In general, the stride may be selected by validating the pixel classification output and selecting a stride based on a tradeoff between processing time and the effectiveness of the location of the wear part in the image932. An advantage of having the same weights946for the convolution kernel944is that successive patches942have a large overlap and convolution results may be saved and re-used for each successive patch, thus significantly reducing the number of computations required. This has the effect of significantly reducing processing time, both in training and subsequently when performing real fragmentation assessments using the trained network930.

In other embodiments, a sparse kernel may be used to perform the convolution. A sparse kernel is constructed by inserting rows and columns of zero values in the convolution kernel944. The sparse kernel may have a single row and column of zero values inserted between each element or multiple rows and columns of zero values inserted between elements. The sparse kernel has an advantage over processing using a stride length of greater than 1, particularly where the processing is performed by the GPU334(shown inFIG. 6) since operations are still performed on successive adjacent pixels in the input pixel data sets. Processing by a GPU is very effective under such conditions, while processing as a stride greater than 1 requires that processing of some input pixels is skipped, which makes much less efficient use of GPU processing capabilities.

The neural network930also includes a pooling layer948, including a plurality of pooling neurons950. The pooling layer948combines outputs of the convolution layer936to condense the information to make the neural network930less sensitive to input shifts and distortions. In one embodiment a max-pooling process is applied that finds a maximum output value within a group of outputs from the convolution layer936and sets the output of a corresponding neuron950in the pooling layer948to the maximum output value. For example, the output952in the pooling layer948may be set to the maximum output of the four output neurons954in the convolution layer936. Alternatively, other pooling processes such as average pooling may be implemented where outputs in the convolution layer936are averaged to produce the output in the pooling layer948. In other embodiments, stochastic pooling may be used, where a random output within a group of outputs in the convolution layer936is selected to produce the output in the pooling layer948.

The neural network930further includes an output layer956that includes a neuron958that produces a probability pwthat the image pixel940in the patch942corresponds to a wear part and a neuron960that produces a probabilitypw, that the pixel does not correspond to a wear part. In one embodiment, each of the neurons958and960may be fully connected to the neurons938in the pooling layer948, which means that the neurons in the output layer956may each have multiple inputs that are connected to each of the neurons938.

The embodiment of the neural network930shown inFIG. 19is only one example of a network that may be configured to produce the pixel classification outputs at the output layer956. In general the network930is initially configured and then trained using training images that have been examined and labeled. For example, regions of images may be labeled by an operator to indicate whether the region includes a wear part or does not include a wear part. The images are then saved along with labeling information as labeled training images. It is desirable to have a sufficient number labeled training images under different lighting and other conditions, differing scale, and differing types of operating implements and wear parts. A portion of the labeled training images may be used for training the network930and a further portion may be set aside for validation of the network to evaluate training effectiveness.

Referring toFIG. 20, a screenshot displayed on the display108of the apparatus100ofFIG. 1is shown generally at980. The screenshot980includes various views, including a view982of the bucket, a rear view984showing an area behind the heavy equipment, and left and right views986and988showing areas to the sides of the heavy equipment. The screenshot980also shows a schematic view990representing the teeth of the operating implement. In one embodiment the neural network930may be implemented and configured to first detect the bucket within a larger patch indicated by the rectangle992inFIG. 20. The neural network930may also be implemented and configured to detect individual teeth within the rectangle992that are located within patches indicated by smaller rectangles994. In this embodiment the neural network930has detected that the tooth within the patch996is missing and has reflected this result at998in the schematic view990, showing the represented tooth as being broken or missing.

Resampling Captured Image

In some embodiment, captured images may be of different scales and/or may include the operating implement and wear parts at different scales. The image932may be resampled to represent the wear part using smaller or larger pixels934. As such the image932may be up-sampled and/or down-sampled to produce additional input pixel values for processing. The labeled training images may be similarly scaled during the training operation to different scales, for example 0.5×, 1×, and 2× thus providing additional training inputs for training the network930. The neural network930may thus produce a scaled output at the output layer956for each scaled input pixel values and corresponding set of training images.

In the embodiments disclosed above, a tooth wear part has been used as an example for purposes of the description. However in other embodiments, other wear parts such as a replaceable lip shroud between teeth may also be identified. The above process may be combined with the process700for identifying the bucket and the process400for identifying the teeth to provide high detection reliability. In other embodiments the various disclosed processes may be varied or combined to provide a desired reliability and/or speed of detection.

The above embodiments provide a method and apparatus for reliably detecting a wear part within an image of an operating implement. Images of examples of a variety of corresponding wear parts are used to determine a matching criterion that accounts for minor variations between the wear parts and for other effects such as lighting conditions. False positive identifications may be also be avoided by including easily mistaken images of other parts of the heavy operating equipment or environment in the determination of the matching criterion.

The above embodiments have the advantage over conventional tooth detection methods and systems in that a calibration process is not mandated. In conventional tooth detection systems, a calibration process involving careful marking of each tooth location and orientation in several operating implement images (for example small, medium, and large views of the bucket within the image) and generating calibration parameters is usually required. The resulting calibration parameters teach the image processing algorithms of the conventional tooth detection system where to search for the teeth and at what orientation ranges the teeth may be encountered. While calibration may still be included in the embodiments described herein, the training exercise can effectively eliminate calibration requirements. In some embodiments, only the number of teeth may be required as a calibration parameter, and with sufficient training the determined neural network parameters will take any calibration issues into account. This may significantly reduce the installation and commissioning time, reduce system maintenance requirements, and enhance robustness of the wear part monitoring system.

While specific embodiments have been described and illustrated, such embodiments should be considered illustrative only and not as limiting the invention as construed in accordance with the accompanying claims.