Patent Publication Number: US-2022220697-A1

Title: Object detection system and method

Description:
FIELD 
     This invention relates generally to a detection system for use with a work machine to alert an operator of the work machine to humans or objects too close to the machine. 
     SUMMARY 
     The invention is directed to a detection system. The system comprises a work machine, one or more cameras, a processor, and a warning system. The cameras are configured to capture images of one or more zones surrounding the work machine. The processor is configured to analyze the images captured by the cameras and determine whether any captured image includes a characteristic of one or more predetermined objects within any one or more of the zones. The warning system is controlled by the processor. The warning system sends a warning signal to an operator of the work machine if the characteristic of the predetermined object is within any one or more of the zones. 
     In another embodiment, the invention is directed to a method for detecting objects near a work machine. The method comprises the steps of capturing images of one or more zones surrounding the work machine using one or more cameras and using a processor to analyze the images captured by any one or more of the cameras and determine whether any captured image includes a characteristic of one or more predetermined objects within any one or more of the zones. The method further comprises the step of automatically activating a warning system controlled by the processor if the processor determines the characteristic of any one or more of the predetermined objects is within any one or more of the zones. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a side view of a work machine with a work tool attached. 
         FIG. 2  is a rear perspective view of the work machine and work tool of  FIG. 1  with a detection system of the present invention shown supported on the work machine. 
         FIG. 3  is a top perspective view of the work tool of  FIG. 1  and one or more zones surrounding the work tool that were identified by an operator of the work machine for analysis by the detection system. 
         FIG. 4  is a front perspective view of  FIG. 3 . 
         FIG. 5  is the perspective view of  FIG. 3  with a human form identified in one of the zones. 
         FIG. 6  is the perspective view of  FIG. 5  with a second human form identified in one of the zones. 
         FIG. 7  is a straight on view of a display on an interface for use with the detection system. 
         FIG. 8  is the view of  FIG. 7  with an alternative display shown. 
         FIG. 9  is a flow chart depicting the relationship between the components of the detection system of the present invention. 
         FIG. 10  is a flow chart depicting the method of operation of the detection system of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     With reference to  FIGS. 1-2 , a detection system  10  of the present invention comprises a work machine  12 , one or more cameras  14 , a processor  16 , and a warning system  18 . The work machine  12  comprises a work tool  20  that is attached to a front end  22  or a back end  24  of the work machine  12 . When the work tool  20  is active, it is important for humans or objects to stay away from the work tool and work machine  12  to avoid injury. The detection system  10  may alert an operator of the work machine  12  of humans or objects that are dangerously close to the machine or work tool  20  during operation. 
     The work machine  12  further comprises an engine  26 , a ground supporting member  28 , and an operator station  30  situated on a frame  32 . The operator station  30  shown comprises a seat  34  and steering wheel  36 . Alternatively, the operator station  30  may comprise a platform and joystick controls. As a further alternative, the work machine  12  may not comprise an operator station  30  and instead may be remotely controlled or under a semi-autonomous control. 
     The ground supporting member  28  shown comprises a set of wheels  38 . Alternatively, the ground supporting member  28  may comprise a set of endless tracks. In operation, an operator, for example, uses the steering wheel  36  to guide the wheels  38  of the work machine  12 . In this way, an attentive operator will avoid objects and people. The system  10  of the present invention assists the operator in detecting unperceived or moving objects. 
     The work tool  20  shown is a trencher  40  that is attached to the back end  24  of the work machine  12 . The trencher  40  comprises a plurality of digging teeth  42  that rotate about a trencher boom  44  to uncover a trench. Other work tools, such as vibratory plows, buckets, skid steers, excavator arms, micro-trenching assemblies, grapple arms, stump grinders, and the like may be utilized with the work machine  12 . 
     With reference now to  FIGS. 1-10 , one or more of the cameras  14  are used to capture images  46  of one or more zones  48  surrounding the work tool  20  and the work machine  12 . The cameras  14  may be supported on a boom  50  attached to and extending over the work machine  12 , as shown in  FIG. 2 . This gives the cameras  14  a view of the entire work tool  20  and an area surrounding the work machine  12 . Preferably, at least two cameras  14  are used and are horizontally spaced on the boom  50  to provide stereo or 3-D vision of one or more of the zones  48 . 
     The cameras  14  may face the front end  22  or back end  24  of the work machine  12  depending on the position of the work tool  20  on the machine. Alternatively, a plurality of cameras  14  may be used to capture images of all sides of the work machine  12  if multiple work tools  20  are attached to the machine at one time. A suitable camera for use with the invention is the e-con Systems Capella model or the Leopard stereo camera module, though many different camera systems may be used. 
     The processor  16  may be supported on the work machine  12  at the operator station  30 , as shown. Alternatively, the processor  16  may be at a location remote from the work machine  12 . The processor  16  is electronically connected to an interface  52  having a display  54 , as shown in  FIGS. 7-9 . The interface  52  may be controlled by the operator using a keyboard and mouse or a touch screen. The images  46  captured by the cameras  14  are sent to the processor  16  and depicted on the display  54 . If more than one work tool  20  is attached to the machine  12 , multiple images  46  may be depicted on the display  54  at one time. 
     Prior to operation of the work machine  12 , the operator will identify one or more zones  48  surrounding the work machine  12  to be viewed by the cameras  14 . The zones  48  are identified by selecting one or more boundaries  56  for each zone  48 . The boundaries  56  may be defined by x, y, and z coordinates selected by the operator on the interface  52 , as shown in  FIG. 7 . The taper of the zones  48  may also be selected by the operator on the interface  52 , if any tapering is necessary to better set the size and shape of the zones. 
     The boundaries  56  and taper selected may form different shapes for each zone  48 . The shape of the zones  48  shown are parallelepipeds, but the orientation, size, and shape of the zones may be tailored to: the clock speed or refresh rate of the detection system  10 , the size of the work machine  12 , the dimensions of the work tool  20 , and the operator&#39;s preference. Alternatively, the zones  48  may be preselected and programmed into the processor  16  without input from the operator. 
     The zones  48  are projected on the display  54  overlaying the images  46  captured by the cameras  14 , as shown in  FIGS. 5-8 . The boundaries  56  of the zones  48  are colored or shaded on the display  54 . Different colors or shades may designate different zones  48 . If the operator manipulates the boundaries  56  for the zones  48  on the interface  52 , the changes are reflected on the display  54 . 
     During operation, the processor  16  analyzes the images  46  captured by the cameras  14  and determines whether any captured image includes a characteristic  58  of one or more predetermined objects  60  moving within any one of the zones  48 . The predetermined object  60  shown in  FIGS. 3 and 5-8  is a human form  62 . Alternatively, the predetermined object  60  may be an animal form or any number of moving objects that the work tool  20  might encounter during operation, such as falling tree limbs or rocks. 
     The processor  16  may be programmed with recognition software  61  capable of recognizing angles of the predetermined object  60  during operation. For example, the software may be programmed to recognize angles of the human form  62 . An open source computer vision library software algorithm is capable of making needed recognitions. However, other similar software may be used. 
     If the processor  16  determines the characteristic  58  of the predetermined object  60  is within one of the zones  48 , the recognition software  61  will surround the object with a box  64  on the display  54  and highlight the recognized characteristic. The processor  16  will also trigger the warning system  18  to send a warning signal to the operator. Programming the processor  16  to recognize predetermined objects  60  reduces the likelihood of false positives interrupting operation. Otherwise, for example, debris from the work tool  20  could trigger a response initiated by the processor  16 . 
     The warning signal may comprise an audible alarm  65  or flashing light  66 , as shown in  FIG. 2 . The goal of the warning signal is to allow the operator time to take necessary precautions to avoid injury to the detected object  60  or anyone nearby. The processor  16  may also be programmed to automatically activate an override system  67  incorporated into the work machine  12  that stops operation of the work machine  12  or the work tool  20  if the characteristic  58  of the object  60  is within one of the zones  48 . If more than one zone  48  has been identified, the response triggered by the processor  16  may vary depending on which zone the characteristic  58  of the object  60  is determined to be within. 
     For example, the operator may identify a first zone  68  that is an area within a predetermined distance surrounding the work tool  20 , and a second zone  70  that is an area within a predetermined distance surrounding the first zone  68 . Each predetermined distance may be identical or different. One predetermined distance, for example, may be about two feet. 
     If the characteristic  58  of the object  60  is determined to be only within the second zone  70 , the processor  16  may trigger the warning system  18  to activate a warning signal. In contrast, if the characteristic  58  of the object  60  is determined to be within the first zone  68 , the processor  16  may trigger the override system  67  which stops operation of the work machine  12  or work tool  20 . 
     The specific response triggered by the processor  16  may vary depending on the operator&#39;s preference. The operator may set response preferences prior to operation using the interface  52 . Alternatively, the response preferences may be pre-selected and programmed into the processor  16  without input from the operator. 
     Optical flow software  71  may be used with the processor  16  to determine whether the predetermined object  60  is moving into or out of the zones  48 . Moving objects are seen by the software as groups of moving pixels. The location of the moving pixels on the images  46  is compared on a frame by frame basis. The frames may be compared for example at a rate of ten frames per second to identify any change in position of the moving object. This clock speed or refresh rate of the frames may be increased or decreased depending on the capabilities of the software used. 
     Groups of pixels in the images  46  that are determined to be moving inconsistently with the machine  12  or the ground surface are identified as moving objects and analyzed by the processor  16  to determine if the object contains a characteristic  58  of the predetermined object  60 . If the moving object is determined to have a characteristic  58  of the predetermined object  60  within one of the zones  48 , the processor  16  will trigger the warning system  18  and/or the override system  67 . Both systems may be triggered if the predetermined object  60  moves into different zones  48 . 
     The processor  16  may be programmed to turn off the warning system  18  or reactivate the work tool  20  or work machine  12  if it determines the object  60  has moved out of the zones  48 . Alternatively, the operator may cancel activation of both the warning system  18  and/or the override system  67  if the operator determines the object  60  detected is not in any danger. 
     Groups of pixels in the images  46  that are determined to be moving at the same rate or direction as the ground surface are identified as stationary objects  72  the work machine  12  is moving past. For example, a bush  74  is shown in  FIG. 3  as a stationary object  72  the machine is moving past. The processor  16  may be programmed to ignore stationary objects  72  when comparing frame to frame images  46 . 
     The boundaries  56  defined for each zone  48  may include a floor  76  that is a desired distance above the ground surface. The operator can program the processor  16  to ignore any moving objects detected below the floor  76 . This helps to avoid false positives from moving elements on the work tool  20  or moving dirt or cuttings that may be identified as moving objects. 
     Similarly, the operator may define an area immediately surrounding the work tool  20  as a black zone  78 . This zone  78  may be blacked out from detection by the processor  16  to minimize false warnings and inadvertent shutdowns. The shape of the black zone  78  may be tailored to the shape and size of the work tool  20  used with the work machine  12 . The size and shape of the black zone  78  may also account for the amount of debris dispersed by the work tool  20  during operation. 
     The level of sensitivity of the detection system  10  may be programmed by the operator on the interface  52 . For example, the system  10  may be programmed such that a percentage of the predetermined object  60  must be detected within one of the zones  48  before a response is triggered by the processor  16 . 
     The processor  16  may be programmed to include a data storage device  80 , such as a memory card, to store images  46  captured of all objects  60  detected in the zones  48  during operation. GPS  82  may also be incorporated into the processor  16  to identify the physical location of the object  60  when detected in the zones  48 . The processor  16  may further be equipped with a diagnostics system  84  to verify that the detection system  10  is operable each time the work machine  12  is started. If any portion of the detection system  10  is identified as being inoperable, the processor  16  may disable operation of the work tool  20  or work machine  12  until the problem is corrected. 
     One of ordinary skill in the art will appreciate that modifications may be made to the invention described herein without departing from the spirit of the present invention.