Control of machines through detection of gestures by optical and muscle sensors

A material handler system including a plurality of components and material handler equipment, and methods for utilizing the same, is disclosed. A first component includes gesture command recognition and enhancement for controlling material handler equipment. A second component includes a safeguarding engine for testing the gesture command inputs and improving reliability of the gesture command recognition. A location tracking feature for tracking a location of the material handler equipment in view of a geo-fence is also referenced by the material handler system.

TECHNICAL FIELD

This disclosure relates to a system and method for controlling machines by detecting a person's gesture commands.

BACKGROUND

Often times, heavy material handling operations (e.g., industrial digging sites using a backhoe) involves the teamwork of at least two people to perform the operations. For example, a first person may be operating material handler equipment (e.g., a backhoe) and a second person may be acting as a spotter for the first person to call out and inform the first person of any conditions that the first person may not be aware of.

For some material handler operations, the operator and/or spotter may be required to undergo specialized training to operate the material handler equipment. The use of both the operator and the spotter increases the manpower requirement, and may thus stress the resources of an enterprise running material handling operations.

SUMMARY

According to an embodiment, a computing device including at least a communication interface and a processor is disclosed. The communication interface may be configured to receive gesture sensor data from a gesture detection device, the gesture sensor data including at least one gesture command, receive image data from an image recording device, the image data including a depiction of the at least one gesture command; and receive location information identifying a location of a material handler equipment included in the material handler system. The processor may be in communication with the communication interface. The processor may further be configured to determine a control command for controlling an operation of the material handler equipment based on the gesture sensor data and the image data, compare the location of the material handler equipment included in the location information with geo-fence parameters assigned to the material handler equipment, and control operation of the material handler equipment according to the control command when the location of the material handler equipment is determined to be within the geo-fence parameters.

DETAILED DESCRIPTION

The discussion below makes reference to a material handler system that includes gesture command recognition for controlling operation of material handler equipment included in the material handler system. The system may include two or more sensors for detecting the person's gesture commands, and circuitry for recognizing the gesture commands and implementing a safeguarding strategy that determines when to enable control over the machines based on predetermined safety conditions. Without the gesture command recognition capability, the material handler system may require both an operator agent situated in the material handler equipment for directly operating the material handler equipment, and also a spotter agent located outside of the material handler equipment for providing spotting communications to the operator. The spotting communications may inform the operator of environmental and/or safety conditions that are not viewable by the operator.

With the gesture command recognition, the material handler system may not require the operator described above. For example, the agent located outside of the material handler equipment (i.e., the spotter) described above may be equipped with gesture command devices that enables the agent to control operation of the material handler equipment without further inputs from an operator inside of the material handler equipment. This way, a single agent located outside of the material handler equipment may wear a gesture command device and control operation of the material handler equipment with gesture commands. This does away with the need for an operator seated within the material handler equipment.

With these enhanced technological capabilities, the gesture command capabilities improve the material handler system by reducing a number of human agents needed to operate the material handler equipment. With the enhanced gesture command capabilities, a single agent equipped with the gesture command device described herein and located outside of the material handler equipment, may control operation of the material handler equipment remotely. The material handler equipment may be heavy machinery such as, for example, a backhoe, excavator, or tractor.

FIG. 1shows an exemplary material handler system100that includes material handler equipment103, a gesture detection device102worn by an agent101, and an image recording device104. A computing device110may be installed on the material handler equipment103, where the computing device110includes a communication interface111, a remote interlock interface112, and a material handler tool113that includes a safeguarding engine114and a material handler controller115. A more detailed description of the computer architecture that comprises the computing device110is provided with reference to the computer system200inFIG. 2. Another agent is not seated within the material handler equipment103. The material handler equipment103shown inFIG. 1and described herein may be a backhoe that includes a bucket10, a loader20, and stabilizer legs30, amongst other backhoe components.

The gesture detection device102may be a wrist band, watch, or other wearables that include muscle sensors for detecting muscle movements and activations that may be interpreted as specific gesture commands by the material handler tool113. For example, the muscle sensors on the gesture detection device102may include electrodes for detecting the movement and/or activation of specific muscles on the body. The electrodes may be medical grade stainless steel electromyography (EMG) sensors for detecting the contraction of muscles on a user wearing the gesture detection device102. The gesture detection device102may further include one or more gyroscopes, one or more accelerometers, and/or one or more magnetometers. The gesture detection device102may be a wearable device for sensing movements of a wearer and/or muscle activation of the wearer. For example, the gesture device102may be a Myo™ wearable gesture control and motion control device in the form of an armband. The gesture detection device102may further include a GPS transceiver so that the GPS120may locate the gesture detection device102.

The muscle detection information describing the muscle movements and/or muscle activations may be transmitted by the gesture detection device102to the computing device110. Once received, the material handler tool113running on the computing device110may analyze the muscle detection information and interpret the muscle movements and/or muscle activations to correspond to a specific body movement. The material handler tool113may further match up the interpreted body movement to a predefined gesture command. Alternatively, portions of the material handler tool113may be running on the gesture detection device102directly such that the muscle detection information may be analyzed by the material handler tool113running on the gesture detection device102. According to such embodiments, the gesture detection device102may then interpret the body movement, match the interpreted body movement to a gesture command, and transmit the gesture command to the computing device110.

Examples of body movements that may be used as gesture commands are provided inFIGS. 6-12.FIG. 6shows an exemplary pull-in gesture command.FIG. 7shows an exemplary push-out gesture command.FIG. 8shows an exemplary closed fist gesture command.FIG. 9shows an exemplary open hand gesture command.FIGS. 6-9 and 12show the gesture detection device102worn on and in skin-contact with an agent's wrist, whileFIGS. 10-11 and 13shows the gesture detection device102worn underneath an agent's clothing around a bicep region (in skin-contact). The gesture detection device102may also be worn in skin-contact at other locations such as an agent's forearm.FIG. 10shows a swing left gesture command.FIG. 11shows a swing right gesture command.FIG. 12shows a hand at rest gesture command.FIG. 13shows an arm at rest gesture command.

For exemplary purposes, a closed fist (e.g., closed fist gesture command shown inFIG. 8) may be a gesture command for closing the bucket on the material handler equipment103, an open hand (e.g., open hand gesture command shown inFIG. 9) may be a gesture command for opening the bucket on the material handler equipment103, hands resting on the agent's101waist may be a gesture command for keeping the material handler equipment103still, swinging an arm from the agent's101waist to the right (e.g., swing right gesture command shown inFIG. 11) in a plane may be a gesture command for swinging the bucket on the material handler equipment103to the right, swinging an arm from the agent's101waist to the left in a plane (e.g., swing left gesture command shown inFIG. 10) may be a gesture command for swinging the bucket on the material handler equipment103to the left, the arm pull-in movement (e.g., pull-in gesture command shown inFIG. 6) may be a gesture command for retracting the bucket on the material handler equipment103back in, and the arm pull-out movement (e.g., push-out gesture command shown inFIG. 7) may be a gesture command for extending the bucket on the material handler equipment103out.

The image recording device104may capture one or more still digital images, or record a sequence of one or more digital images such as digital image data, within a field of view of the image recording device104. The material handler system100may be set-up such that the field of view includes both the agent101providing the gesture commands, and the material handler equipment103. According to some embodiments, the image recording device104may further include distance measuring equipment for measuring a distance between the agent101and the material handler equipment103, such as lidar-based distance measuring equipment, radar-based distance measuring equipment, or sonar-based distance measuring equipment. The distance measurement between the agent101and the material handler equipment103may be referenced to ensure the agent101remains a safe distance away from the material handler equipment103. For example, the computing device110may control operation of the material handler equipment103to cease movement of one or more actuators when the distance between the agent101and the material handler equipment103is determined to be less than a predetermined threshold distance.

The material handler equipment103, the gesture detection device102, and the image recording device104may be located within a geo-fence121as defined and identified by a Global Positioning Satellite (GPS)120. The geo-fence121may be a predetermined area that defines a boundary in which mechanical and electrical (i.e., non-human) components of the material handler system100are designated to operate in. For example, the image recording device104and the material handler equipment103may be positioned within the material handler system100to operate within geo-fence121. Likewise, gesture detection device102that is worn by the human agent101may be positioned to operate outside of the geo-fence121. Information on the agent's101location and the geo-fence121may be referenced by the computing device110to keep the agent101outside of the geo-fence121. For example, the computing device110may control operation of the material handler equipment103to cease movement of one or more actuators when the agent101(i.e., the gesture detection device102) is determined to be within the geo-fence121.

The material handler system100may further include a command control server140, where the computing device110installed on the material handler equipment103may rely on the command control server140to accomplish, at least in part, some of the processes and analyses allocated to the material handler tool113described herein. For example, certain processes that require large processing power capabilities such as image recognition for recognizing the agent's101movements from image data captured by the image recording device104may be processed, at least in part, on the command control server140.

The GPS120may communicate with the computing device110directly through the communication interface111. The GPS120and the command control server140may further communication with the computing device110through the communication interface111, via a network130. The network130may be one or more of the following: a local area network; a wide area network; or any other network (such as a cellular communication network) that enables communication of data between computing communication devices. In one embodiment, network130may support wireless communications. Alternatively, or in addition, the network130may support hard-wired communications, such as a telephone line or cable. The network130may also support the Ethernet IEEE (Institute of Electrical and Electronics Engineers) 802.3x specification. The network130may be the Internet and may support IP (Internet Protocol). The network130may be a LAN or a WAN. The network130may be a hotspot service provider network. The network130may be an intranet. The network130may be a GPRS (General Packet Radio Service) network. The network130may be any appropriate cellular data network or cell-based radio network technology. The network130may be an IEEE 802.11 wireless network or any suitable network or combination of networks. Although one network130is shown inFIG. 1, the network130may be representative of any number of networks (of the same or different types) that may be utilized.

In addition to communicating through the network130, the components within the material handler system100may communicate directly with each other via wireless communication protocols such as Bluetooth, NFC (near field communication), and connection to a common WiFi network. For example, the gesture detection device102may communicate directly with the computing device110through the communication interface111. The image recording device104may further communicate directly with the computing device110through the remote interlock interface112. For example, image data and/or distance measurement information obtained by the image recording device104may be communicated to the computing device110through the remote interlock interface112. A more detailed description of the computing device110is provided with reference toFIG. 2.

One or more of the computing devices in the material handler system100may include one or more components described by the exemplary computer architecture of computer system200inFIG. 2.

Computer system200includes a network interface220that allows communication with other computers via a network226, where network226may be represented by network130inFIG. 1. Network226may be any suitable network and may support any appropriate protocol suitable for communication to computer system200. The computer system200may also include a processor202, a main memory204, a static memory206, an output device210(e.g., a display or speaker), an input device212, and a storage device216, communicating via a bus208.

Processor202represents a central processing unit of any type of architecture, such as a CISC (Complex Instruction Set Computing), RISC (Reduced Instruction Set Computing), VLIW (Very Long Instruction Word), or a hybrid architecture, although any appropriate processor may be used. Processor202executes instructions224stored on one or more of the main memory204, static memory206, or storage device216. Processor202may also include portions of the computer system200that control the operation of the entire computer system200. Processor202may also represent a controller that organizes data and program storage in memory and transfers data and other information between the various parts of the computer system200.

Processor202is configured to receive input data and/or user commands through input device212. Input device212may be a keyboard, mouse or other pointing device, trackball, scroll, button, touchpad, touch screen, keypad, microphone, speech recognition device, video recognition device, image recognition device, accelerometer, gyroscope, global positioning system (GPS) transceiver, or any other appropriate mechanism for the user to input data to computer system200and control operation of computer system200. Input device212as illustrated inFIG. 89may be representative of any number and type of input devices.

Processor202may also communicate with other computer systems via network226to receive instructions224, where processor202may control the storage of such instructions224into any one or more of the main memory204(e.g., random access memory (RAM)), static memory206(e.g., read only memory (ROM)), or the storage device216. Processor202may then read and execute instructions224from any one or more of the main memory204, static memory206, or storage device216. The instructions224may also be stored onto any one or more of the main memory204, static memory206, or storage device216through other sources. The instructions224may correspond to, for example, instructions and/or operational commands for implementing the material handler tool113.

Although computer system200is represented inFIG. 2as a single processor202and a single bus208, the disclosed embodiments applies equally to computer systems that may have multiple processors and to computer systems that may have multiple busses with some or all performing different functions in different ways.

Storage device216represents one or more mechanisms for storing data. For example, storage device216may include a computer readable medium222such as read-only memory (ROM), RAM, non-volatile storage media, optical storage media, flash memory devices, and/or other machine-readable media, or any other appropriate type of storage device. Although only one storage device216is shown, multiple storage devices and multiple types of storage devices may be present. Further, although computer system200is drawn to contain the storage device216, it may be distributed across other computer systems that are in communication with computer system200, such as a computer system in communication with computer system200. For example, when computer system200is representative of the computing device110, storage device216may be distributed across to include memory on the command control server140.

Output device210is configured to present information to the user. For example, output device210may be a display such as a liquid crystal display (LCD), a gas or plasma-based flat-panel display, or a traditional cathode-ray tube (CRT) display or other well-known type of display in the art of computer hardware. Accordingly, output device210may be a device for displaying a user interface. In addition or alternatively, output device210may be a speaker configured to output audible information to the user. In addition or alternatively, output device210may be a haptic output device configured to output haptic feedback to the user. Any combination of output devices may be represented by the output device210.

Network interface220provides the computer system200with connectivity to the network226through any compatible communications protocol. Network interface220sends and/or receives data from the network226via a wireless or wired transceiver214. Transceiver214may be a cellular frequency, radio frequency (RF), infrared (IR) or any of a number of known wireless or wired transmission systems capable of communicating with network226or other computer device having some or all of the features of computer system200. Bus208may represent one or more busses, e.g., USB, PCI, ISA (Industry Standard Architecture), X-Bus, EISA (Extended Industry Standard Architecture), or any other appropriate bus and/or bridge (also called a bus controller). Network interface220as illustrated inFIG. 2may be representative of the communication interface111and/or the remote interlock interface112.

Computer system200may be implemented using suitable hardware, software and/or circuitry, such as a personal computer or other electronic computing device. In addition, computer system200may also be a portable computer, laptop, tablet or notebook computer, PDA, pocket computer, appliance, telephone, server computer device, or mainframe computer.

FIG. 3shows an exemplary flow diagram300of logic describing an exemplary process for initializing and calibrating components of the material handler system100. The process for initializing and calibrating the material handler system100may be implemented by, for example, the material handler tool113running on the computing device110.

The gesture detection device102initially connects to the computing device110(301). For example, the gesture detection device102may transmit a connection request signal to the computing device110, where the connection request signal includes device identification information describing the gesture detection device102, as well as a request to connect the gesture detection device102with the computing device. The connection request signal may further include authentication information for authenticating the gesture detection device102to connect with the computing device110.

Upon receiving the connection request from the gesture detection device102, the computing device110may determine whether to allow connection of the gesture detection device102with the computing device110(302). For example, the computing device110may verify the gesture detection device102is allowed to connect with the computing device110by comparing the device identification information with a list of pre-authorized device identifiers. When the device identification information matches a pre-authorized device identifier, the gesture detection device102may be allowed to connect with the computing device110. When the connection request includes authentication information, the authentication information may further be analyzed such that the gesture detection device102is allowed to connect with the computing device110when the authentication information is authenticated. If the gesture detection device102is not allowed to connect with the computing device110, the material handler tool113reverts the process back (301).

After the gesture detection device102is allowed to connect with the computing device110(302), the material handler tool113proceeds to a process for connecting the image recording device104to the computing device110(303). For example, the image recording device104may transmit a connection request signal to the computing device110, where the connection request signal includes device identification information describing the image recording device104, as well as a request to connect the image recording device104with the computing device. The connection request signal may further include authentication information for authenticating the image recording device104to connect with the computing device110. Connecting the image recording device104may also include initializing (e.g., enabling functionality of) a remote interlocking switch. The remote interlocking switch may be a computerized switch for enabling, and disabling, operation of the actuators on the material handler equipment103based on a detected location of the agent101relative to the geo-fence121, and/or based on a detected location of the material handler equipment103and/or image recording device104within the geo-fence121.

Upon receiving the connection request from the image recording device104, the computing device110may determine whether to allow connection of the image recording device104with the computing device110(303). For example, the computing device110may verify the image recording device104is allowed to connect with the computing device110by comparing the device identification information with a list of pre-authorized device identifiers. When the device identification information matches a pre-authorized device identifier, the image recording device104may be allowed to connect with the computing device110. When the connection request includes authentication information, the authentication information may further be analyzed such that the image recording device104is allowed to connect with the computing device110when the authentication information is authenticated. If the image recording device104is not allowed to connect with the computing device110, the material handler tool113reverts the process back (303).

After the image recording device104is allowed to connect with the computing device110(304), the material handler tool113proceeds to calibrate the gesture detection device (305). The gesture detection device102may be calibrated independently, or by communicating calibration information back and forth with the computing device110. The calibration process may include calibration of the sensors, accelerometers, gyroscopes, and/or magnetometers included on the gesture detection device102. Once calibrated, the gesture detection device102may transmit a calibration signal to the computing device110, where the calibration signal indicates the gesture detection device102has been successfully calibrated. If the gesture detection device102is not successfully calibrated, the gesture detection device102may continue efforts to calibrate by reverting back (305).

After the gesture detection device102is successfully calibrated (306), the material handler tool113proceeds to calibrate actuators on the material handler equipment103(307). The material handler actuators may include the bucket, the loader, and the stabilizer legs that are components when the material handler equipment103is a backhoe. Once calibrated, the material handler equipment103may generate a calibration signal to indicate to the material handler tool113that the actuators have been successfully calibrated. If the material handler actuators are not calibrate successfully, the material handler equipment103may continue efforts to calibrate by reverting back (307).

After the material handler actuators are successfully calibrated (308), the image recording device104may be calibrated (309). The calibration of the image recording device104may include calibrating a field of view, focus, and image characteristics of the image recording device104to ensure a scene including the agent101and/or the material handler equipment103is captured by the image recording device104. Once calibrated, the image recording device104may generate a calibration signal indicating the image recording device104has been successfully calibrated, and transmit the calibration signal to the computing device110. If the image recording device104is not calibrate successfully, the image recording device104may continue efforts to calibrate by reverting back (309).

After the image recording device104is successfully calibrated (310), the material handler tool113may proceed to implement a normal run mode for operating the material handler equipment103(311).

FIG. 4shows an exemplary flow diagram400of logic describing an exemplary process for operating the material handler equipment103under a normal run mode. The process for operating the material handler equipment103under the normal run mode may be implemented by, for example, the material handler tool113running on the computing device110.

Movements made by the agent101may be detected by the gesture detection device102and transmitted to the computing device (401). For example, muscle contractions may be detected by EMG sensors, orientations of the agent's101body parts may be detected by gyroscopes, acceleration movements may be detected by accelerometers, and/or magnetic field strength measurements may be detected by a magnetometers of the gesture detection device102. In particular, gesture sensor data describing the movements and other detected measurements by the gesture detection device102may be transmitted to the computing device110. The gesture sensor data may be received by the computing device110through the communication interface111.

Once received, the material handler tool113may determine a quality of the gesture sensor data. The material handler tool113may determine the quality of the gesture sensor data by evaluating such characteristics of the agent's101movements such as the speed and displacement of the detected movements. The material handler tool113may further evaluate the connection quality of the received gesture sensor data by comparing signal information and latency times. The material handler tool113may further evaluate diagnostic information from the gesture detection device102. By evaluating the information included in the gesture sensor data, the material handler tool113determines a quality factor to assign to the gesture sensor data. The quality factor may then be applied at a later step.

In addition, the image recording device104may record image data capturing movements of the agent101. The movements may include one or more gesture commands. The recorded image data may be transmitted to the computing device110through the remote interlock interface112.

Once received, the material handler tool113may determine a quality of the image data (404). The material handler tool113may determine the quality of the image data by evaluating such things as whether or not a signal including the image data is being received (e.g., there may not be successful receipt of an image data signal in cases of weather issues such as fog, high winds, or rain). The material handler tool113may determine the quality of the image data by evaluating whether the image data includes viewable images that depict the agent101and/or the material handler equipment103clearly (e.g., evaluating image quality). The material handler tool113may determine the quality of the image data by evaluating the speed and displacement of the agent's101movements as recognized from the image data. The material handler tool113may determine the quality of the image data by evaluating an intensity of the image characteristics of the image data (e.g., in low and high light conditions the images in the image data may be attenuated). The material handler tool113may determine the quality of the image data by evaluating a comparative distance of the image data (e.g., should the distance from start to finish of the signal change significantly, this may indicate interference or other condition giving a false reading). By evaluating the information included in the image data, the material handler tool113determines a quality factor to assign to the image data. The quality factor may then be applied at a later step.

After determining the quality factor to assign to the gesture sensor data and the quality factor to assign to the image data, the material handler tool113may compute weighting factors for both the gesture sensor data and the image data based on their respective assigned quality factors (405). The weighting factors will be used to determine the fused output of the two input signals (gesture sensor data and image data) that will be eventually sent to the material handler controller115. The weighting factors will take into account the signal quality as well as the movement type described and/or detected by the gesture sensor data and the image data. For example, should the signal quality for the gesture sensor data be determined to be excellent and the movement type small and fine, then the gesture sensor data weighting factor may be generated to be larger than the image data weighting factor. Should the image data quality be determined to be excellent and the movement be large and gross, then the image data weighting factor may be larger than the gesture sensor data weighting factor. In the case where an input signal is determined not to have been received, or the quality of the input signal is near 0, the other input signal may be granted 100% weighting. The weighting factors may be applied at a later step.

The gesture sensor data may be converted to a scaled single value output for each movement detected by the gesture detection device102(406).

The image data may be converted to a scaled single value output for each direction detected by the image recording device104(408).

The material handler tool113may implement an error checking process on the weighting factors (407). The error checking process may include looking for potential errors in the weighting factors such as identifying weighting factors greater than 100%, making sure each individual weighting factor is not greater than 100%, and making sure the sum of each individual weighting factor adds up to 100% (is not larger than 100%). When the error checking process is not satisfied, the material handler tool113reverts the process back (401and402).

When the error checking process is successfully satisfied, the material handler tool113multiplies each scaled single value output from the gesture sensor data and the image data, by their respective weighting factors (409).

The material handler tool113may then combine the values from both the gesture sensor data and the image data and recognize a movement by the agent101based on the weighted gesture sensor data and the weighted image data (410). By recognizing the movement described by the weighted gesture sensor data and the weighted image data, the material handler tool113may then match the recognized movement to a gesture command. The material handler tool113may further translate the gesture command to a material handler actuator motion described as being controlled by the gesture command. Image data may be analyzed using image data processing techniques such as color differentiation to distinguish the pixels depicting the agent101from pixels depicting background objects. The image data may further be analyzed using image processing (e.g., Hough transform) to detect linear boundaries or curved boundaries of arms and fingers depicted within the image data compared against background pixels. A cloud of pixels may be used to model the position of the arms and/or fingers depicted by the image data, where the positioning of the arms and/or fingers depicted by the image data may be compared against reference gestures including arms and/or fingers for the same agent101or a model agent. In terms of the gesture sensor data, the electrical signals detected by the gesture detection device102from the muscles are matched against stored patterns of muscle activation signals that are known to correspond to known reference positioning of arms and/or fingers, reference body movements, and/or reference gestures by the agent101or a model agent.

The material handler tool113may translate the gesture command into corresponding movement of actuators on the material handler equipment103(411). For example, the gesture command may be translated to correspond to moving the bucket10actuator up by 5 ft. The actual translation for moving the bucket10actuator up by 5 ft may correspond to moving the bucket10actuator from its current GPS coordinate to a new GPS coordinate that is 5 ft. above. It follows that the movement of the actuators of the material handler equipment103translated from the gesture command may be in terms of moving the actuator specified by the gesture command to different GPS coordinates.

FIG. 5shows a continuation of the flow diagram400of logic describing the exemplary process for operating the material handler equipment103under the normal run mode.

The material handler tool113may verify the current GPS coordinates of one or more components that are included in the material handler system100. For example, the GPS coordinates of the gesture detection device102, image recording device104, and/or the material handler equipment103may be obtained and verified by the GPS120(414). GPS coordinates for the agent101(by way of detecting the location of the gesture detection device102) may also be determined (415). A predefined fence region (e.g., geo-fence121) may be defined as an area where the material handler equipment103should operate in, and the agent101should not be inside of (416). So when the material handler equipment103is detected to be outside of the geo-fence121, or when the agent101(by way of detecting the location of the gesture detection device102) is detected to be inside of the geo-fence121, further operation of the material handler equipment103may be ceased. In addition, the geo-fence121may extend to a predetermined range surrounding the material handler equipment103itself in order to track the location of specific components of the material handler equipment103. For example, the geo-fence121may be configured to extend to a predetermined depth into the ground below the material handler equipment103. This way, the location of a the bucket10on the exemplary backhoe embodiment of the material handler equipment103may be tracked such that operation of the material handler equipment103may be ceased when the bucket10is determined to be located at a depth beyond the predetermined depth. By configuring the geo-fence121to track the location of the individual components of the material handler equipment103, the agent101may have further back-up control settings to assist the material handler equipment103does not operating beyond predetermined safe boundaries.

To further aid in determining the location of the different components of the material handler system100, a distance of the agent101from the current operation of the material handler equipment103may be determined (412). The distance may be measured, for example, by the image recording device104or determined based on GPS coordinates of the agent101, the gesture detection device102, and/or the material handler equipment103. If the agent is determined to be too close (e.g., less than a predetermined threshold distance from the material handler equipment103), the material handler tool113may also cease operation of the material handler equipment103. The distance of the agent101from the current operation of the material handler equipment103may be compared with the verified location of the geo-fence121(413). If the agent is determined to be located within the geo-fence121, the material handler tool113may further cease operation of the material handler equipment103(422).

The material handler tool113may determine whether the agent101is located outside of the geo-fence121(418). When the material handler tool113determines the agent101is located outside of the geo-fence121, the material handler tool113may compare the obtained GPS coordinates from earlier and compare them against intended GPS coordinates for the different components of the material handler system100(417), and determine whether the comparison allows for the operational movement of the material handler equipment actuator corresponding to the gesture command (419).

When the comparison of the obtained GPS coordinates and the intended GPS coordinates for the different components of the material handler system100indicates the operational movement is allowed (e.g., agent101is outside of geo-fence121, agent101is more than a predetermined threshold distance away from material handler equipment103, and/or a component (e.g., bucket10) of the material handler equipment103is beyond a predetermined depth or range), the material handler tool113may control the material handler equipment103to move the material handler equipment actuator according to the gesture command (420). When the material handler tool113determines the operational movement of the material handler equipment actuator corresponding to the gesture command should not be allowed (e.g., agent101is inside of geo-fence121and/or agent is less than a predetermined threshold distance away from material handler equipment103), a notification may be issued that identifies the gesture command will not be implemented and describing reasons why (421).

The implementations may be distributed as circuitry among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems. Parameters, databases, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, and may be implemented in many different ways, including as data structures such as linked lists, hash tables, arrays, records, objects, or implicit storage mechanisms. Programs may be parts (e.g., subroutines) of a single program, separate programs, distributed across several memories and processors, or implemented in many different ways, such as in a library, such as a shared library (e.g., a Dynamic Link Library (DLL)). The DLL, for example, may store instructions that perform any of the processing described above or illustrated in the drawings, when executed by the circuitry. Various implementations have been specifically described. However, other implementations are also within the scope of this disclosure.