Patent Description:
Industrial automation systems are managed and operated using automation control and monitoring systems. A wide range of applications exist for automation control and monitoring systems, particularly in industrial automation settings. Such applications may include the powering of a wide range of actuators, such as valves, electric motors, and so forth, and the collection of data via sensors. Typical automation control and monitoring systems may include one or more components, such as: programming terminals, automation controllers, input/output (I/O) modules, communication networks, and/or human-machine interface (HMI) terminals.

Generally, certain safety precautions are taken to ensure that devices in industrial automation systems are operated safely. However, conventional industrial automation systems are limited in controlling its devices using the typical automation control and monitoring systems described above. Although these automation control and monitoring systems may be used to manage the operations of the devices within the industrial automation system, improved systems and methods for safely operating devices within an industrial automation system are desirable.

<CIT> relates to systems and methods for controlling industrial equipment in the performance of various industrial activities based on the detected body movement of a user in an industrial automation environment. The method includes employing a time-of-flight sensor to detect movement of a body part of the user, ascertaining whether or not the movement of the body part conforms to a recognized movement of the body part, interpreting the recognized movement of the body part as a performable action, and thereafter actuating industrial machinery to perform the performable action.

<CIT> relates to a computer-implemented method, system, and program which include a behavior processing system for capturing a three-dimensional movement of a user within a particular environment, wherein the three-dimensional movement is determined by using at least one image capture device aimed at the user. The behavior processing system identifies a three-dimensional object properties stream using the captured movement. The behavior processing system identifies a particular defined behavior of the user representing a particular behavioral signal from the three-dimensional object properties stream by comparing the identified three-dimensional object properties stream with multiple behavior definitions each representing a separate behavioral signal for directing control of at least one machine. In response to identifying the particular defined behavior, a machine control system generates a control signal triggered by the particular behavioral signal for directing control of the at least one machine.

It is the object of the present invention to facilitate an improved industrial automation control.

The object is solved by the subject matter of the independent claims.

The present disclosure is generally directed towards a range camera system disposed in an industrial automation system that may detect positions and/or motions of persons and/or objects within a space encompassing the industrial automation system. After detecting the positions and/or motions, the range camera system may perform various operations in the industrial automation system based on those detected positions and/or motions. That is, the range camera system may detect the positions and/or motions of persons and/or objects (e.g., robots, autonomous guided vehicles, or machines) within the industrial automation system space and use the detected positions and/or motions to control various industrial automation devices in the industrial automation system. In addition to detecting positions and/or motions within the industrial automation system, the range camera system may compute a confidence value for each detected position and/or motion. The confidence value may relate to a degree or percentage in which the remote camera system may be certain in the detected position and/or motion. As such, in certain embodiments, the range camera system may send control signals to various devices in the industrial automation system based on the confidence values that correspond to the respective detected positions and/or motions. Additional details regarding the range camera system will be described below with reference to <FIG>.

By way of introduction, <FIG> is a block diagram of an industrial automation system <NUM> operating with a range camera system <NUM>. The range camera system <NUM> may include one or more range cameras <NUM> and a motion recognition system <NUM>. The range camera <NUM> may include any type of camera or sensor system that may detect positions and/or movements of any element in its viewing window. In certain embodiments, the range camera <NUM> may employ various types of technology to enable it to detect positions and movement. By way of example, the range camera <NUM> may include digital cameras, video cameras, infrared sensors, optical sensors (e.g., video/camera), radio frequency energy detectors, sound sensors, sonar sensors, vibration sensors, magnetic sensors, and the like to detect the positions and/or movements of any element in its viewing window. The sensors used for the range camera <NUM> may be capable of detecting sonar waveforms, heat signatures, lasers, floor pressure signatures, air pressure characteristics, and the like.

The range camera <NUM> may also include a sensor that may employ range imaging technology or techniques to produce two-dimensional or three-dimensional images that may be used to indicate the distance between various points in an image acquired by the sensor and the location of the sensor. As such, the sensor employed with range imaging technology may operate according a number of techniques such as stereo triangulation, sheet of light triangulation, structured light, time-of flight, interferometry, coded aperture, and the like.

The detected positions and/or motions received by the range camera <NUM> may be input into the motion recognition system <NUM>, which may interpret the detected positions and/or motions and determine various automation commands that may correspond to the detected positions and/or motions. After determining the automation command that corresponds to the detected positions and/or motions, the motion recognition system <NUM> may send the automation command to the system controller <NUM>, which may implement the automation command on an industrial automation device <NUM> coupled to the system controller <NUM>. For instance, the motion recognition system <NUM> may receive a detected motion from the range camera <NUM> and may interpret the detected motion as a command to stop one of the devices <NUM> in the industrial automation system <NUM>. As such, the motion recognition system <NUM> may send a signal to the system controller <NUM> to stop the respective industrial automation device <NUM>.

Industrial automation devices <NUM> may include any type of machine or device that may operate within the industrial automation system <NUM>. Examples of industrial automation devices <NUM> may include, but are not limited to, actuators, electric motors, electric presses, and the like. Generally, the industrial automation devices <NUM> may be operated or controlled by the system controller <NUM>. As such, the system controller <NUM> may have a processor component, a memory component, a communication component, and the like such that it may operate the industrial automation devices <NUM> by sending control signals to the industrial automation devices <NUM>. In certain embodiments, in addition to receiving automation commands from the motion recognition system <NUM>, the system controller <NUM> may receive various types of information from various sensors <NUM> in the industrial automation system <NUM>. The sensors <NUM> may be any type of sensor such as a push-button sensor, a gas sensor, a temperature sensor, and the like that may be used to provide information to the system controller <NUM>. As such, the system controller <NUM> may operate the industrial automation devices <NUM> based on the information received by the sensors <NUM>, the automation commands interpreted by the motion recognition system <NUM>, or both.

Keeping the foregoing in mind, in certain embodiments, the range camera system <NUM>, the industrial automation devices <NUM>, and the sensors <NUM> may generally be disposed in a space that may be characterized as part of an industrial automation environment <NUM>. The industrial automation environment <NUM> may include a physical space within the industrial automation system <NUM> in which human operators <NUM>, machines <NUM>, or material <NUM> may perform various actions to operate the industrial automation system <NUM>. As such, the range camera system <NUM> may monitor the positions and/or motions of the humans <NUM>, the machines <NUM>, or the material <NUM> as they move within the industrial automation environment <NUM>.

The humans <NUM> may include any person that may be present in the industrial automation environment <NUM>. Generally, the humans <NUM> present in the industrial automation environment <NUM> may include persons trained and authorized to have access to the industrial automation environment <NUM>. The humans <NUM> may perform various operations such as maintenance and the like for any device in the industrial automation system <NUM>. However, it should be noted that the humans <NUM> may also include persons who are not authorized or are undesirable persons (e.g., intruders) present in the industrial automation environment <NUM>. As such, in one embodiment, the range camera system <NUM> may be used to detect whether the humans <NUM> in the industrial automation environment <NUM> are authorized to be in the industrial automation environment <NUM> and send a notification or automation command to the system controller <NUM> based on whether the detected human(s) <NUM> is authorized or expected to be in the industrial automation environment <NUM>.

The machines <NUM> may include any automated or non-automated machine present in the industrial automation environment <NUM>. As such, the machines <NUM> may include robots or mechanical devices that perform various operations on items being manufactured, processed, or the like. For example, the machine <NUM> may include a mechanical claw that may move articles within the industrial automation system. In certain embodiments, the machine <NUM> may include industrial automation devices <NUM> operating within the view window of the range camera <NUM>.

The material <NUM> may include items such as workpieces being moved by the machine <NUM> or the items being manufactured or processed in the industrial automation system <NUM>. For instance, <FIG> depicts the material <NUM> as a box on a conveyor belt. In this instance, the range camera system <NUM> may track the positions of the material <NUM> as it moves along the conveyor belt. The motion recognition system <NUM> may then send information related to the positions of the material <NUM> to the system controller <NUM>, which may use that information to control various industrial automation devices <NUM>. For instance, if the system controller <NUM> determines that the positions of the material <NUM> on the conveyor belt is changing too slowly, the system controller <NUM> may interpret that information as indicating that the material <NUM> is moving too slowly along the conveyor belt. As such, the system controller <NUM> may send a signal to an industrial automation device <NUM>, such as a motor drive coupled to a motor that controls the movement of the conveyor belt, to increase the speed of the motor, thereby increasing the speed at which the material <NUM> may move along the conveyor belt.

In certain embodiments, the range camera <NUM> may be coupled to various components in the industrial automation system <NUM> such as the industrial automation device <NUM> or the like such that the range camera <NUM> may monitor the movements and/or positions of various elements within the industrial automation system <NUM>. For instance, the range camera <NUM> may be coupled to a DIN rail that may be fixed at various locations in the industrial automation system <NUM>, the industrial automation device <NUM>, or the like.

In any case, by using the range camera system <NUM> to monitor various elements within the industrial automation system <NUM>, the system controller <NUM> may better manage the safe operations of the industrial automation system <NUM>. For instance, information acquired from the range camera system <NUM> may be used to determine the location of the human <NUM> with respect to a potential hazard such as a moving industrial automation device <NUM>, a high-temperature industrial automation device <NUM>, high-voltage equipment, and the like. In certain embodiments, the range camera system <NUM> may process data related to body extremity of the human <NUM> that may be moving and the speed at which it may be moving to predict whether the movement may cause the human <NUM> to come in physical contact with a hazardous zone, a machine <NUM>, or the like.

For instance, the range camera system <NUM> may process data received from the range camera <NUM> indicating the positions and movements of various joints of the human <NUM> to determine a probability of the human <NUM> physically contacting or entering a hazardous zone. As such, the range camera system <NUM> may predict that the human <NUM> may come in contact with the hazardous zone when the probability exceeds a threshold. By being able to predict these types of situations, the range camera system <NUM> may take preemptive measures in avoiding undesirable situations such as triggering an action to prevent the dangerous situation or minimize the adverse effects of the undesired situation. For example, the range camera system <NUM> may send the system controller <NUM> automation commands to stop or alter the operations of the industrial automation devices <NUM> or the machine <NUM> to such that the industrial automation devices <NUM> or the machine <NUM> avoid the human <NUM>. Moreover, the range camera system <NUM> may also be used to restart or reengage the halted industrial automation devices <NUM> or machine <NUM> once the human <NUM> is no longer located within the hazardous zone.

The range camera system <NUM> may also send a notification to the system controller <NUM>, other industrial automation systems <NUM>, or the like indicating that the human <NUM> was located in the hazardous zone or that the respective industrial automation devices <NUM> were stopped due the presence of the human <NUM> in the hazardous zone. In this way, the other industrial automation systems <NUM> may modify their operations if it is determined that their operations will be affected by the data received by the motion recognition system <NUM>.

In a similar fashion, the range camera system <NUM> may be used to monitor various the positions and/or motions of the machine <NUM> rather than, or in conjunction with, monitoring the positions and/or motions of the human <NUM>. As such, the range camera system <NUM> may determine whether the machine <NUM> is operating safely within the industrial automation environment <NUM>.

By way of example, <FIG> depicts an example of the industrial automation environment <NUM> that uses a time-of-flight camera system <NUM> as the range camera <NUM> to perform the various techniques described herein. As shown in <FIG>, the time-of-flight camera system <NUM> may be employed in the industrial automation environment <NUM> or in a controlled environment such as a power control room. The time-of-flight camera system <NUM> may include an illumination unit <NUM> and an image sensor <NUM> that may be used to determine the positions and/or motions of the human <NUM>, the machine <NUM>, or the like present in the industrial automation environment <NUM> or in the controlled environment. In one embodiment, the time-of-flight camera system <NUM> may rotate such that it may have a <NUM>-degree line-of-sight around the industrial automation environment <NUM>.

To determine the positions of the human <NUM> or the machine <NUM> in the industrial automation environment <NUM>, the time-of-flight camera system <NUM> may use the illumination unit <NUM> to emit a light signal <NUM> outward into the industrial automation environment <NUM>. The light signal <NUM> may include an intensity-modulated light in the near-infrared range, a laser, or the like. In any case, once the light signal <NUM> hits an element (i.e., the human <NUM> in <FIG>), a reflected light signal <NUM> may be reflected back to the time-of-flight camera system <NUM> and into the image sensor <NUM>. In one embodiment, the reflected light signal <NUM> may be projected into the image sensor <NUM> via a lens that may be disposed on the time-of-flight camera system <NUM>. After sending the light signal <NUM> and receiving the reflected light signal <NUM>, the time-of-flight camera system <NUM> may send data related to the times at which the light signal <NUM> was sent and the reflected light signal <NUM> was received to the motion recognition system <NUM> to determine a distance between the element and the time-of-flight camera system <NUM>. That is, the motion recognition system <NUM> may correlate data related to the light signal <NUM> and the reflected light signal <NUM> such as the times at which the light signal <NUM> was sent and the reflected light signal <NUM> was received to determine a distance between the element and the time-of-flight camera system <NUM>. By tracking this distance over time, the motion recognition system <NUM> may determine the positions of the element over time, the motion of the element, and the like.

In certain embodiments, the motion recognition system <NUM> may include three-dimensional motion paths that may define the path in which the machine <NUM> may operate within the industrial automation environment <NUM>. The three-dimensional motion paths may be determined based on an expected motion of the machine <NUM> with respect to its function within the industrial automation system <NUM>. As such, the motion recognition system <NUM> may detect whether the machine <NUM> is operating as expected with respect to the three-dimensional motion paths and/or whether the machine <NUM> is experiencing any hazardous movements (e.g., moving to fast, proximate to human <NUM>). If the motion recognition system <NUM> detects that the machine <NUM> is operating unexpectedly or with certain hazardous movements, the motion recognition system <NUM> may send a signal to the system controller <NUM>, which may be used to control the machine <NUM>, to operate the machine <NUM> in a safe state, powered off, or the like.

Moreover, in some embodiments, the motion recognition system <NUM> may interpret the motion and/or the speed of the motion of the machine <NUM> or the human <NUM> to predict whether the machine <NUM> may enter a hazardous area or hazardous situation such as contacting the human <NUM>. As such, the motion recognition system <NUM> may send a signal to the system controller <NUM> or directly to the machine <NUM> to alter the operation of the machine <NUM> such that the machine <NUM> may avoid entering the hazardous area or encountering the hazardous situation. That is, the motion recognition system <NUM> may control the operations of the machine <NUM> or any industrial automation device <NUM> based on the motions and/or the positions of the machine, the motions and/or the positions of the human <NUM>, or the like, thereby operating the industrial automation system <NUM> more safely.

Keeping the foregoing in mind, <FIG> illustrates a detailed block diagram <NUM> of the motion recognition system <NUM>. The motion recognition system <NUM> may include a communication component <NUM>, a processor <NUM>, a memory <NUM>, a storage <NUM>, input/output (I/O) ports <NUM>, and the like. The communication component <NUM> may be a wireless or wired communication component that may facilitate communication between the system controller <NUM>, the industrial automation devices <NUM>, the range camera <NUM>, the machine <NUM>, other industrial automation systems <NUM>, and the like. The processor <NUM> may be any type of computer processor or microprocessor capable of executing computer-executable code. The memory <NUM> and the storage <NUM> may be any suitable articles of manufacture that can serve as media to store processor-executable code, data, or the like. These articles of manufacture may represent computer-readable media (i.e., any suitable form of memory or storage) that may store the processor-executable code used by the processor <NUM> to perform the presently disclosed techniques. Generally, the motion recognition system <NUM> may receive motion and/or position data related to the human <NUM>, the machine <NUM>, and/or the material <NUM> and interpret the data to determine automation commands for the industrial automation devices <NUM>. The memory <NUM> and the storage <NUM> may also be used to store the data, the respective interpretation of the data, and the automation command that corresponds to the data. Although the block diagram <NUM> is depicted with respect to the motion recognition system <NUM>, it should be noted that the system controller <NUM> may also include the same components to perform the various techniques described herein. Additional details describing a method in which the motion recognition system <NUM> may use to interpret the motion and/or location data will be discussed below with reference to <FIG>.

Referring now to <FIG>, a method <NUM> for implementing automation commands within the industrial automation system <NUM> based on detected positions and/or motions of an element within the viewable region of the range camera <NUM>. In one embodiment, the techniques described herein with respect to the method <NUM> may be performed by the motion recognition system <NUM>. As such, at block <NUM>, the motion recognition system <NUM> may receive detected positions and/or motions of an element such as the human <NUM>, the machine <NUM>, or the material <NUM>. That is, the range camera <NUM> may receive data related to the positions and/or motions of the element currently present within its viewing window.

In certain embodiments, the data related to the detected positions and/or motions may include one or more optic images of the element, one or more infrared images of the element, or the like. Once the range camera <NUM> acquires the images of the element, the motion recognition system <NUM> may convert the images into two-dimensional figures that represent the element. For instance, <FIG> illustrates an example of a two-dimensional representation <NUM> of the human <NUM> and <FIG> illustrates an example of a two-dimensional representation <NUM> of the machine <NUM>.

Referring briefly to <FIG>, the two-dimensional representation <NUM> of the human <NUM> may include data points <NUM> at various edges or joints on the body of the human <NUM>. Based on the distance between the range camera <NUM> and the human <NUM>, the size of the two-dimensional representation <NUM>, the relative distance between the two-dimensional representation <NUM> and other fixed objects within the view window of the range camera <NUM>, and the like, the motion recognition system <NUM> may generate metadata <NUM> for each data point <NUM>.

The metadata <NUM> may include certain details regarding the respective data point <NUM>. By way of example, the metadata <NUM> may include an x-coordinate value <NUM>, a y-coordinate value <NUM>, a z-coordinate value <NUM>, a confidence value <NUM>, and the like. The confidence value <NUM> may correspond to a degree or amount in which the motion recognition system <NUM> may be certain of the x-coordinate value <NUM>, the y-coordinate value <NUM>, and the z-coordinate value <NUM>. In some embodiments, the motion recognition system <NUM> may generate the confidence value <NUM> for each coordinate value. The confidence value <NUM> may be determined based on visibility of the data point <NUM>, historical data related to the data point <NUM>, or the like.

In the same manner, the two-dimensional representation <NUM> of the machine <NUM> may include data points <NUM> at various edges or joints on the body of the machine <NUM>. Based on the distance between the range camera <NUM> and the machine <NUM>, the size of the two-dimensional representation <NUM>, the relative distance between the two-dimensional representation <NUM> and other fixed objects within the view window of the range camera <NUM>, and the like, the motion recognition system <NUM> may generate metadata <NUM> for each data point <NUM>.

Like the metadata <NUM> described above with reference to <FIG>, the metadata <NUM> may include certain details regarding each respective data point <NUM> such as an x-coordinate value <NUM>, a y-coordinate value <NUM>, a z-coordinate value <NUM>, a confidence value <NUM>, and the like. Like the confidence value <NUM>, the confidence value <NUM> may correspond to a degree or amount in which the motion recognition system <NUM> may be certain of the x-coordinate value <NUM>, the y-coordinate value <NUM>, and the z-coordinate value <NUM>. In some embodiments, the motion recognition system <NUM> may generate the confidence value <NUM> for each coordinate value and may determine the confidence value <NUM> based on visibility of the data point <NUM>, historical data related to the data point <NUM>, an expected positions of the machine <NUM>, or the like.

Keeping the foregoing in mind, the motion recognition system <NUM> may detect a motion or movement pattern of the element based on how the positions of the data points <NUM> or <NUM> change over time. In one embodiment, the motion recognition system <NUM> may acquire the metadata <NUM> or <NUM> related to each data point <NUM> or <NUM> and store the metadata <NUM> or <NUM> in the memory <NUM> or the storage <NUM>. The processor <NUM> of the motion recognition system <NUM> may then analyze the collection of metadata <NUM> or <NUM> to determine how the element is moving. As such, the motion recognition system <NUM> may recognize the motion or gestures of the element.

Referring back to <FIG>, at block <NUM>, the motion recognition system <NUM> may determine a possible automation command for the detected motions received at block <NUM>. In one embodiment, to determine the possible automation command, the motion recognition system <NUM> may compare the detected positions and/or motions to a library of automation commands related to detected positions and/or motions. <FIG> illustrates an example of a library <NUM> of automation motion commands that may be stored in the storage <NUM> of the motion recognition system <NUM>.

The library <NUM> may include a list of expected motions by the element and a corresponding automation command. The list of expected motions may be programmed into the library using a learning mode or the like with the motion recognition system <NUM>. As such, an operator or technician may provide an input to the motion recognition system <NUM> indicating an automation command for a particular industrial automation device <NUM>. The operator may then position himself within the view window of the range camera <NUM> and perform various movements or gestures that he intends to be associated with the inputted automation command. As such, the motion recognition system <NUM> may record how the data points <NUM> move or a pattern of motion of the data points <NUM> that correspond to the human <NUM>. The motion recognition system <NUM> may then store the pattern of motion of the data points <NUM> (automation commands <NUM>) in the library <NUM> and associated the pattern with the respective input automation command.

For example, the operator may provide an input to the motion recognition system <NUM> to enter into a learning mode and specify that the a particular motion or gesture should be associated with an emergency stop an automation command for a particular automation device <NUM>. After receiving these inputs, the motion recognition system <NUM> may detect the movements that correspond to the data points <NUM> of the operator, which may include, for instance, holding one arm out straight with a palm out and figures up, while the operator is in the view window of the range camera <NUM>. Once the motion recognition system <NUM> detects the motion, the motion recognition system <NUM> may store the movement of the data points <NUM> (automation command <NUM>) with an association to an emergency stop automation command in the library <NUM>. In certain embodiments, the motion recognition system <NUM> may enter and exit the learning mode by receiving some input from the operator that does not include any detected motion or gesture. In this case, the configuration of the learning mode may be secured and may not be compromised by any inadvertent motions or gestures.

In certain embodiments, the processor <NUM> may include a motion recognition engine <NUM> that may identify an automation command <NUM> from the library <NUM> based on the detected positions and/or motions of the element. For example, if the motion recognition engine <NUM> detects that the human <NUM> is waving both of his arms left and right, the motion recognition engine <NUM> may compare the detected motion to the motions or patterns of motion stored in the library <NUM> and determine that the motion corresponds to a stop automation command for a particular industrial automation device <NUM>.

The library <NUM> may include a number of motions and a corresponding automation command <NUM> for each motion. The automation commands may include any command to control the automation devices <NUM>, the machine <NUM>, or any other device in the industrial automation system <NUM>. As such, the automation commands may include, but are not limited to, starting a device, stopping a device, increasing a speed or output of a device, decreasing a speed or output of a device, and the like. Moreover, the library <NUM> may include other commands associated with various motions such as disabling the motion recognition system <NUM>, limiting the control or ability of an operator to engage with the motion recognition system <NUM>, or the like. In certain embodiments, the library <NUM> may include motions with respect to fixed objects in the industrial automation environment <NUM>. For instance, a motion of the human <NUM> moving in the direction of some control input device, such as an emergency stop input, may be associated with an automation command for that control input device (e.g., emergency stop).

While the motion recognition engine <NUM> may be used to determine a possible automation command for the detected motions received at block <NUM>, a position determination engine <NUM> may be used to determine a possible automation command for the detected positions received at block <NUM>. As such, the position determination engine <NUM> may determine the positions of the data points <NUM> and <NUM> and may consult the library <NUM> to identify an automation command <NUM> that corresponds to the detected positions received at block <NUM>. In this manner, the library <NUM> may include rules or scenarios in which certain automation commands may be implemented based on the positions of the data points <NUM> and <NUM>.

Keeping this in mind, <FIG> illustrates a space <NUM> within the industrial automation environment <NUM> that may be designated as hazardous areas or areas in which the human <NUM> should not enter or in which only the machine <NUM> should operate. As such, the library <NUM> may include an automation command <NUM> to stop certain industrial automation devices <NUM> that may potentially cause harm to the human <NUM> when the human <NUM> is detected in the hazardous area.

For instance, in one embodiment, the space <NUM> may be marked by borders <NUM>, which may be implemented using a light curtain, electric markers, colored markers, or the like. As such, the library <NUM> may include an automation command to stop industrial automation devices <NUM> or machines <NUM> when one or more humans <NUM> are present within the space <NUM>. In another example, the library <NUM> may include an automation command to stop the operation of the industrial automation device <NUM> or the machine <NUM> when the human <NUM> or an extremity of the human <NUM> is located within an operating space of the industrial automation device <NUM> or the machine <NUM>. For instance, if the industrial automation device <NUM> corresponds to an industrial press machine that compacts material, the library <NUM> may include an industrial automation command that stops the operation of the industrial press machine when the human <NUM> or an extremity of the human <NUM> is detected within the area of the industrial press machine where the material is expected to be placed.

Referring back to <FIG>, at block <NUM>, the motion recognition system <NUM> may determine the confidence value <NUM> or <NUM> for the detected positions and/or motions received at block <NUM>. As mentioned above, the motion recognition system <NUM> may determine the confidence value <NUM> or <NUM> based on visibility of the data point <NUM> or <NUM>, historical data related to the data point <NUM> or <NUM>, expected positions of the human <NUM>, the machine <NUM>, or the material <NUM>, or the like. In one embodiment, the motion recognition system <NUM> may also determine a confidence value for the possible automation command determined at block <NUM> based on the confidence value <NUM> or <NUM>, the difference between the detected positions and/or motions determined at block <NUM> and the positions and/or motions associated with the possible automation command received at block <NUM>, and the like. In another embodiment, the confidence value may be related to the difference between the detected positions and/or motions received at block <NUM> and an expected pattern of motion for the element.

In certain embodiments, the motion recognition system <NUM> may determine a confidence value for the detected positions and/or motions based on whether the corresponding element is within a particular range. For instance, if the element that corresponds to the detected positions and/or motions is within a close range of the camera <NUM>, the detected positions and/or motions may be associated with higher confidence values as compared to elements that may be further from the camera <NUM>. The confidence values may also be stored in an array that may be organized based on a distance between the detected element and the camera <NUM>. As such, the motion recognition system <NUM> may determine a confidence value for a detected positions and/or motions based on a confidence value as listed in the array.

At block <NUM>, the motion recognition system <NUM> may determine whether the confidence value determined at block <NUM> is greater than some threshold. If the confidence value determined at block <NUM> is greater than the threshold, the motion recognition system <NUM> may proceed to block <NUM> and implement the automation command determined at block <NUM>. In addition to or in lieu of implementing the automation command, the motion recognition system <NUM> may send a notification to an administrator or operator of the industrial automation system <NUM> indicating that the automation command is being implemented or the like.

When implementing the determined automation command, in one embodiment, the motion recognition system <NUM> may associate the determined automation command with a particular industrial automation device <NUM>. That is, upon determining the automation command based on the detected positions and/or motions of the element, the motion recognition system <NUM> may determine which automation device <NUM> or machine <NUM> to implement the determined automation command based on the positions of the data points <NUM> and <NUM> within a proximity of a device in the industrial automation system <NUM>. For example, if the motion recognition system <NUM> interprets motions detected by the range camera <NUM> to correspond to stopping an industrial automation device <NUM>, the motion recognition system <NUM> may then, at block <NUM>, implement the automation command on the industrial automation device <NUM> that is within the closest proximity to the element.

Referring back to block <NUM>, if the confidence value determined at block <NUM> is not greater than the threshold, the motion recognition system <NUM> may proceed to block <NUM> and implement a default automation command. The default automation command may be a safe state command for respective automation device <NUM> or machine <NUM>. In one embodiment, the default automation command may include continuing the operation of the respective industrial automation device <NUM>. In another embodiment, at block <NUM>, the motion recognition system <NUM> may send a notification requesting a manual confirmation of the automation command determined at block <NUM> or a notification indicating that the confidence level is low for the respective automation command.

By determining whether to implement the recognized automation command based on the confidence value, the motion recognition system <NUM> may be safeguarded or protected against interpreting random movements and/or positions of the human <NUM>, the machine <NUM>, or the material <NUM> as possible automation commands. However, in certain embodiments, to better ensure that the appropriate automation command is being recognized by the motion recognition system <NUM> or to better ensure that the industrial automation devices <NUM> are being operated safely, the motion recognition system <NUM> may use secondary inputs to supplement, complement, or even replace motion-based and/or position-based inputs recognized by the motion recognition system <NUM>. For example, <FIG> depicts a method <NUM> for controlling one or more industrial automation devices <NUM> within the industrial automation system <NUM> using the range camera system <NUM> and some secondary input.

The secondary input may include any input or signal received or generated by the system controller <NUM> or the motion recognition system <NUM> that may be used to complement or confirm the automation command. As such, the motion recognition system <NUM> may verify the validity of the interpreted automation command based on whether an expected secondary input has been received by the system controller <NUM> or the motion recognition system <NUM>.

Keeping this in mind and referring to <FIG>, at block <NUM>, the motion recognition system <NUM> may receive detected positions and/or motions of the element, as described above with respect to block <NUM> of <FIG>. In one embodiment, the detected positions and/or motions of the element received by the motion recognition system <NUM> may include metadata or information related to the confidence value <NUM> or <NUM> for the detected positions and/or motions received at block <NUM>.

After receiving the detected positions and/or motions of the element, at block <NUM>, the motion recognition system <NUM> may determine a possible automation command for the detected positions and/or motions received at block <NUM>, as described above with respect to block <NUM> of <FIG>. Although not shown in <FIG>, it should be noted that in certain embodiments, the motion recognition system <NUM> may determine a confidence value for the automation command determined at block <NUM> based on the confidence value <NUM> or <NUM>, the difference between the detected positions and/or motions received at block <NUM> and expected positions and/or motions that correspond to the possible automation command determined at block <NUM>, and the like.

At block <NUM>, the motion recognition system <NUM> may determine whether an expected secondary input or automation system signal has been received. As mentioned above, the secondary input may include an input or signal received by the motion recognition system <NUM> to verify that the automation command determined at block <NUM> corresponds to an automation command to be implemented. As such, the secondary input may include the use of manual or other control inputs such as a push button located within the industrial automation system <NUM>, an input provided to the motion recognition system <NUM> via the input/output ports <NUM>, an input received by the motion recognition system <NUM> via a sensor disposed on the industrial automation device <NUM> or within the industrial automation system <NUM>, or the like.

For example, the secondary input may include a manual switch or button operated by the human <NUM>. The secondary inputs may also include a weight detected on a safety/pressure mat, a signal from a light curtain indicating whether the light curtain has been broken, an indication from an area scanner, or inputs from enable pendants, safety gates, guard interlocks, emergency stop switches, or the like.

Another example of the secondary input may include voice commands or confirmations. As such, at block <NUM>, the machine recognition system <NUM> may expect to receive a voice command from the human <NUM> that may correspond to an expected voice command for the respective determined automation command. For instance, the expected secondary input associated with a detected motion that may correspond to a stop automation command may include a speech by the human <NUM> that recites, "stop.

In one embodiment, the secondary input may be the detected positions of the element with respect to some automation device <NUM>, while the automation command determined at block <NUM> may be based on the detected motions of the element. For example, if the motion recognition system <NUM> determines that the motions of the human <NUM> correspond to an automation command that engages a press machine, the motion recognition system <NUM> may, at block <NUM>, determine whether the detected location of the human <NUM> is a safe distance away from the press machine or outside of an area in which a material may be pressed. As such, the location of the human <NUM> may be used as the expected secondary input at block <NUM>. Moreover, in this manner, the motion recognition system <NUM> may coordinate motions detected by the range camera <NUM> with positions detected by the range camera <NUM> to control the operation of various devices within the industrial automation system <NUM>. As such, the motion recognition system <NUM> may ensure that industrial automation devices <NUM> are operated safely and as per their respective specifications.

Referring back to block <NUM>, if the motion recognition system <NUM> receives the expected secondary input, the motion recognition system <NUM> may proceed to block <NUM> and implement the automation command determined at block <NUM>. In one embodiment, in addition to or in lieu of implementing the automation command, the motion recognition system <NUM> may send a notification message (e.g., e-mail, text message) to an administrator or operator of the industrial automation system <NUM> that indicates that the automation command has been implemented, that the automation command has been requested to be implemented, or the like.

If, however, at block <NUM>, the motion recognition system <NUM> does not receive the expected secondary input, the motion recognition system <NUM> may proceed to block <NUM> and implement a default action. In one embodiment, a default action may be defined or determined for each automation device <NUM>. The default action may defined or determined to correspond to a safe-sate (e.g., de-energized) for the respective industrial automation device <NUM>. In other words, if the motion recognition system <NUM> does not receive the expected secondary input, the motion recognition system <NUM> may determine the safe-state for the respective industrial automation device <NUM> and implement the automation command that corresponds to the safe-state. For example, if the expected secondary input corresponds to a signal indicating that the human <NUM> is standing on a pressure mat and therefore located a safe distance away from the respective industrial automation device <NUM>, the motion recognition system <NUM> may determine the safe-state automation command for the respective automation device <NUM> by assuming that the human <NUM> is not standing on the pressure mat. As such, the motion recognition system <NUM> may determine that the safe-state for the respective automation device <NUM> may include deenergizing the respective automation device <NUM>, thereby ensuring that a person is not hurt or an object is not damaged by the respective automation device <NUM>. In this case, the motion recognition system <NUM> may then send a command to the respective automation device <NUM> to operate in the safe-state mode. In certain embodiments, the default action or the safe-state mode may include performing no action at all. As such, the default action may include continuing the operation of the respective automation device <NUM>.

In addition to or in lieu of implementing the default action, the motion recognition system <NUM> may send a notification to an administrator or operator of the industrial automation system <NUM> that indicates that the default action has been implemented, that the secondary input has not been received, or the like. In one embodiment, the notification may include suggestions or instructions as to what the motion recognition system <NUM> may be seeking as the secondary input.

In certain embodiments, after determining that the expected secondary input has been received, the motion recognition system <NUM> may proceed to block <NUM> of <FIG>. As such, the motion recognition system <NUM> may determine the confidence value <NUM> or <NUM> for the detected positions and/or motions received at block <NUM>. Alternatively or additionally, the motion recognition system <NUM> may determine a confidence value for the possible automation command determined at block <NUM>, as described above with respect to block <NUM>. The motion recognition system <NUM> may then follow the process of the method <NUM> to may better ensure that the automation command determined at block <NUM> may correctly match the detected positions and/or motions of the element received at block <NUM>.

In addition to determining automation commands based on detected positions and/or motions of an element, the motion recognition system <NUM> may also analyze a derivative value of the detected positions and/or motions when determining automation commands. That is, the motion recognition system <NUM> may determine one or more derivative values of the detected positions for an element and determine an automation command based on the derivative values. The derivative values of data corresponding to the detected positions of the element may indicate a velocity in which the element is moving. In certain embodiments, the motion recognition system <NUM> may use the velocity information in determining an automation command for the detected positions. For instance, the velocity data may be used to quantify a speed in which the determined automation command should be implemented.

In the same manner, the derivative values of data corresponding to the detected motions of the element may indicate an acceleration or a change in acceleration that corresponds to how the element may be moving. The acceleration or change in acceleration data may be interpreted by the motion recognition system <NUM> to indicate a sense of urgency or an undesired event with respect to the detected motions of the element. For instance, if the human <NUM> moves with increasing speed towards an emergency stop switch, the motion recognition system <NUM> may interpret the high acceleration or high change in acceleration data that corresponds to the motion of the human <NUM> as indicating that a serious issue may be present in the industrial automation system <NUM>. Moreover, the acceleration or change in acceleration data may also be used to predict a location of the human <NUM> (or other tracked object) at various points in time. Additional details with regard to using derivative-based data in determining automation commands for the industrial automation system <NUM> is provided below with reference to <FIG>.

Referring now to <FIG>, the motion recognition system <NUM> may employ a method <NUM> to implement automation commands and/or notifications based on derivative data related to positions and/or motions detected by the range camera <NUM>. At block <NUM>, the motion recognition system <NUM> may receive detected positions and/or motions of an element, as described above with reference to block <NUM> of <FIG>.

After receiving the detected positions and/or motions of the element, at block <NUM>, the motion recognition system <NUM> may determine derivative values for the detected positions and/or motions received at block <NUM>. In one embodiment, for detected position data, the motion recognition system <NUM> may generate a position function that represents the positions of the data points <NUM> or <NUM> over time. The motion recognition system <NUM> may then determine a derivative value for the generated position function to determine a rate of change in the detected positions of the data points <NUM> or <NUM>, which may correspond to a velocity in which the data points <NUM> or <NUM> move.

In another embodiment, for detected motion data, the motion recognition system <NUM> may generate a motion function that represents the motions of the data points <NUM> or <NUM> over time. The motion recognition system <NUM> may then determine a derivative value for the generated motion function to determine a rate of change in the detected motions of the data points <NUM> or <NUM>, which may correspond to an acceleration of the detected motion for the data points <NUM> or <NUM>.

At block <NUM>, the motion recognition system <NUM> may determine possible automation commands based on the detected positions and/or motions received at block <NUM> and/or the derivative values of the detected positions and/or motions determined at block <NUM>. That is, the motion recognition engine <NUM> may determine an automation command that corresponds to the detected positions and/or motions from the library <NUM>, as described above with reference to block <NUM> of <FIG>, and may refine or revise the determined automation command based on the derivative values of the detected positions and/or motions determined at block <NUM>.

For example, in addition to the data related to positions and/or motions of an element, the library <NUM> may also include expected velocities and accelerations that correspond to the data related to the positions and/or motions of the element. In other words, in addition to associating certain positions and/or motions of an element with a particular automation command, the library <NUM> may also associate the certain positions and/or motions of the element of the particular automation command with expected velocities and/or acceleration values that correspond to how the element moves. In this manner, the motion recognition system <NUM> may detect a sense of urgency in the automation command or may flag any unusual behavior of the element that invoked the automation command.

Keeping this in mind, at block <NUM>, the motion recognition system <NUM> may first retrieve an automation command from the library <NUM> based on the detected positions and/or motions of the element. However, the motion recognition <NUM> may then revise or alter the retrieved automation command based on the derivative values of the detected positions and/or motions. By way of example, after determining that the positions and/or motions of an element correspond to an automation command that increases the speed of a motor, the motion recognition system <NUM> may then consult the library <NUM> to determine whether the derivative values of the detected positions and/or motions of the element are within expected ranges of the derivative values of the detected positions and/or motions for the element.

If the derivative values of the detected positions and/or motions of the element are not within the expected ranges of the derivative values, the motion recognition system <NUM> may alter or modify the automation command retrieved from the library <NUM>. For instance, referring back to the example above, if the automation command corresponds to increasing the speed of a motor, the motion recognition system <NUM> may change the automation command to disable the motor. In this example, the motion recognition system <NUM> may disable the industrial automation device <NUM> (i.e., motor) to ensure that the industrial automation device <NUM> is operated safely and to ensure that the appropriate element's positions and/or motions were detected. That is, by recognizing whether the derivative values relatively matched expected the expected derivative values, the motion recognition system <NUM> may provide additional security in preventing unauthorized personnel from operating devices in the industrial automation system <NUM>.

In certain embodiments, the library <NUM> may generally include data related to expected velocity and acceleration values for motions that correspond to each of the automation commands stored therein. In addition to expected velocity and acceleration values for automation commands, the library <NUM> may also store expected velocity and acceleration values that correspond to the general behavior of an element such as the human <NUM>, the machine <NUM>, or the material <NUM>. As such, the expected velocity and acceleration values may include a range of velocity and acceleration values that correspond to normal behavioral movements of the human <NUM>, the machine <NUM>, or the material <NUM>. In certain embodiments, the motion recognition system <NUM> may determine the expected velocity and acceleration values for any element based on historical data acquired by the range camera <NUM>. That is, the motion recognition system <NUM> may detect the positions and/or motions of various elements over time and identify positions and/or motions of the elements that may correspond to normal behavior. The motion recognition system <NUM> may then determine the derivative values for the positions and/or motions of the elements that correspond to the normal behavior and identify a range of derivative values (e.g., velocity and acceleration data) correspond to the normal behavior.

Keeping this in mind, the motion recognition system <NUM> may, at block <NUM>, determine an automation command based on whether the detected positions and/or motions have derivative values that correspond to the expected derivative values. That is, the motion recognition system <NUM> may interpret whether the detected motions of the element correspond to normal movements or abnormal movements and determine an automation command based on whether the element's movements are normal or abnormal. For instance, if the human <NUM> suddenly falls to the floor or becomes immobile, the motion of the human <NUM> falling to the floor or remaining immobile may have derivative values that are outside a range of expected derivative values for a human. As such, the motion recognition system <NUM> may recognize or detect these types of undesirable situations based on the velocity or acceleration data related to the motion of the human <NUM>. In another example, the motion recognition system <NUM> may determine that the machine <NUM> or the material <NUM> may be moving at a velocity or acceleration that may fall outside of the expected range of derivative values. In this case, the motion recognition system <NUM> may determine that the machine <NUM> or the material <NUM> may be moving abnormally, which may result in damage to the machine <NUM>, the material <NUM>, or the like.

If the motion recognition system <NUM> determines the derivative values that are outside a range of expected derivative values, the motion recognition system <NUM> may then determine one or more automation commands for various industrial automation devices <NUM> that may be designed to keep the human <NUM> safe, the industrial automation system <NUM> functioning, or the like. For instance, the motion recognition system <NUM> may predict the movements and operations of the industrial automation devices <NUM> that may be within a proximity of the human <NUM> or that may risk further injuring the human <NUM> and determine automation commands that may minimize any risk of further injury to the human <NUM>. For example, the determined automation command(s) may include stopping the operation of various industrial automation devices <NUM>, machines <NUM>, or the like.

In certain embodiments, the motion recognition system <NUM> may, at block <NUM>, determine possible automation commands based on the derivative values by predicting whether the human <NUM>, the machine <NUM>, the material <NUM>, or the automation device <NUM> may physically contact each other or enter within each other's operating space. That is, using the derivative values of the detected motions received at block <NUM>, the motion recognition system <NUM> may predict whether two elements are moving in a pattern that may result in an injury or damage to either element. Moreover, by using the detected positions and/or motions of the element and the derivative values of the detected positions and/or motions, the motion recognition system <NUM> may predict when the elements may collide, cause injury, cause damage, or the like. As such, the motion recognition system <NUM> may determine one or more automation commands that may prevent injury or damage to any element within the industrial automation system <NUM>. For instance, if the motion recognition system <NUM> predicts that two elements may collide with each other based on the detected positions and/or motions of the elements and the derivative values of the detected positions and/or motions, the motion recognition system <NUM> may determine automation commands for various industrial automation devices <NUM> and/or machines <NUM> that may cause the industrial automation devices <NUM> and/or machines <NUM> to move in an opposite direction than its current direction to avoid the collision.

In some embodiments, the motion recognition system <NUM> may quantify the determined automation command based on the derivative values of the detected positions and/or motions. In other words, the motion recognition <NUM> may attribute some numerical value to an aspect that relates to the determined automation command. For example, if the motion recognition engine <NUM> retrieves an automation command from the library <NUM> that corresponds to stopping the operation of an industrial automation device <NUM>, the motion recognition system <NUM> may use the derivative value of the detected positions and/or motions to quantify a sense of urgency or importance related to the automation command to stop the industrial automation device <NUM>. That is, if the derivative values indicate that the velocity and/or acceleration are greater than an expected range of velocity and/or acceleration, the motion recognition system <NUM> may determine that the corresponding automation command has a high degree of urgency or importance. As such, at block <NUM>, the motion recognition system <NUM> may determine additional automation commands based on the heightened sense of importance. Referring back to the example above, after retrieving an automation command from the library <NUM> that corresponds to stopping the operation of an industrial automation device <NUM>, the motion recognition system <NUM> may generate additional automation commands to stop other related industrial automation devices <NUM> to protect the entire industrial automation system <NUM> if the retrieved automation command is interpreted to have a high importance.

After determining the possible automation command based on the detected positions and/or motions and/or based on the derivative values, the motion recognition system <NUM> may implement the automation command(s) determined at block <NUM>. In some cases, the motion recognition system <NUM> may send a notification to an administrator or operator of the industrial automation system <NUM> that indicates the determined automation command(s). If the motion recognition system <NUM> detected an abnormal or undesirable event, as described above, the motion recognition system <NUM> may send a notification to the administrator or operator of the industrial automation system <NUM> that provides details related to the abnormal or undesirable event.

Referring back to block <NUM>, in certain embodiments, after determining the possible automation command(s), the motion recognition system <NUM> may proceed to block <NUM> of <FIG>. As such, the motion recognition system <NUM> may determine the confidence value <NUM> or <NUM><NUM> for the detected positions and/or motions received at block <NUM>. Alternatively or additionally, the motion recognition system <NUM> may determine a confidence value for the possible automation command determined at block <NUM>, as described above with respect to block <NUM>. In this manner, the motion recognition system <NUM> may better ensure that the automation command determined at block <NUM> may correctly match the detected positions and/or motions of the element received at block <NUM>.

In another embodiment, after determining the possible automation command(s) at block <NUM>, the motion recognition system <NUM> may proceed to block <NUM> of <FIG>. As such, the motion recognition system <NUM> may determine whether an expected secondary input has been received, as described above with reference to block <NUM> of <FIG>. The motion recognition system <NUM> may then verify the validity of the determined automation command based on whether an expected secondary input has been received by the system controller <NUM> or the motion recognition system <NUM>.

Referring again to block <NUM>, the motion recognition system <NUM> may determine possible automation commands based on historical data that may be stored in a memory. That is, the motion recognition system <NUM> may learn to associate particular patterns of motion with a respective automation command using the historical data. For instance, if the motion recognition system <NUM> determines that a particular pattern of motion is typically associated with a particular automation command according to the historical data, the motion recognition system <NUM> may learn or associate the particular pattern of motion with the particular automation command.

In addition to interpreting positions and/or motions of an element in the industrial automation system as described above in <FIG>, <FIG>, and <FIG>, the motion recognition system <NUM> may also distinguish between humans and objects in the industrial automation environment <NUM> and implement various controls and/or notification actions based on whether the element is human or not (<FIG>). For example, the motion recognition system <NUM> may control how lights and air conditioning within the industrial automation environment <NUM> may function based on whether a human or an object is moving in the industrial automation environment <NUM>.

Keeping this in mind, <FIG> depicts a method <NUM> for implementing various control and/or notification actions related to the industrial automation system <NUM> based on whether a human or an object (i.e., non-human) is present in the industrial automation environment. Referring now to <FIG>, at block <NUM>, the motion recognition system <NUM> may receive detected positions and/or motions of an element present in the industrial automation environment <NUM> from the range camera <NUM>, as described above.

At block <NUM>, the motion recognition system <NUM> may determine whether the element corresponds to a human (e.g., the human <NUM>) or an object (e.g., the machine <NUM> or the material <NUM>) based on the detected positions and/or motions received at block <NUM>. In certain embodiments, the motion recognition system <NUM> may compare the positions of the data points <NUM> or <NUM> with templates or known data point arrangements of humans. If the positions of the data points <NUM> or <NUM> correspond to the template or known data point arrangements of humans, the motion recognition system <NUM> may determine that the element is a human. Otherwise, the motion recognition system <NUM> may classify the element as an object or non-human.

In another embodiment, the motion recognition system <NUM> may compare the motions of the element received at block <NUM> with templates or known motion patterns of humans. If the received motions correspond to the template or known motion patterns of humans, the motion recognition system <NUM> may determine that the element is a human. Otherwise, the motion recognition system <NUM> may classify the element as an object or non-human.

Once the motion recognition system <NUM> determines that the element is either a human or an object, the motion recognition system <NUM> may implement a control action and/or a notification action based on whether the element is determined to be a human or an object. The control action may include operating any device in the industrial automation system <NUM> or controlling various environmental parameters in the industrial automation environment <NUM>. For instance, <FIG> illustrates an example environment control system <NUM> that may use the motion recognition system <NUM> to control the various environmental parameters in the industrial automation environment <NUM>, as described above with reference to the method <NUM>.

Referring now to <FIG>, the environment control system <NUM> may include various environment control features such as a heating, ventilation, and air conditioning (HVAC) unit <NUM>, a light control unit <NUM>, or the like. The HVAC unit <NUM> may control the airflow, maintain the air quality. or regulate the temperature in the industrial automation environment <NUM> by providing ventilation, air filtration, and the like. The light control unit <NUM> may control the operations of lights <NUM> used to illuminate the industrial automation environment. In certain embodiments, the HVAC unit <NUM> and the light control unit <NUM> may be coupled to the system controller <NUM> as shown in <FIG>. As such, the system controller <NUM> may provide signals to the HVAC unit <NUM> and the light control unit <NUM> to control the environment within the industrial automation environment <NUM>. However, it should be noted that in some embodiments, the motion recognition system <NUM> may be directly coupled to the environment control devices, as opposed to via the system controller <NUM>.

Keeping the environment control system <NUM> in mind and referring back to block <NUM>, the motion recognition system <NUM> may control the operations of the HVAC unit <NUM>. the light control unit <NUM>, or the like based on whether the detected element is a human or an object. For instance, the conditions within the industrial automation environment <NUM> should not cater to humans when humans are not present therein. As such, in one embodiment, if the motion recognition system <NUM> determines that the element present in the industrial automation environment <NUM> is not a human, the motion recognition system <NUM> may send one or more signals to the system controller <NUM> to modify the operations of the HVAC unit <NUM>, the light control unit <NUM>, and the like. By way of example, the motion recognition system <NUM> may send signals to the system controller <NUM> to stop operating the HVAC unit <NUM> or turn the lights <NUM> off when the motion recognition system <NUM> determines that no human is present in the industrial automation environment <NUM>. In this manner, the industrial automation devices <NUM>, the machines <NUM>, and the material <NUM> may continue to move throughout the industrial automation environment <NUM> while operating in a human-unsuitable (e.g., uncomfortable) condition or low-light condition. That is, since the industrial automation devices <NUM>, the machines <NUM>, and the material <NUM> may not be affected by operating in a human-suitable or human-unsuitable environment, the motion recognition system <NUM> may operate the environment control devices more efficiently by minimizing the use of the HVAC unit <NUM> or the light control unit <NUM>. Moreover, once the received detected positions and/or motions of the element are determined to be human, the motion recognition system <NUM> may, at block <NUM>, adjust the operations of the environment control devices to make the industrial automation environment <NUM> more suitable or comfortable for humans.

Besides controlling the environment control devices in the industrial automation system, the motion recognition system <NUM> may also control the industrial automation devices <NUM> or the machines <NUM> based on whether the element is human or not. For instance, in certain embodiments, the motion recognition system <NUM> may be arranged such that the range camera <NUM> may monitor an area within the industrial automation environment <NUM>, which may be designated as a hazardous area or an area where humans should not enter. As such, if, at block <NUM>, the motion recognition system <NUM> determines that the element in the area corresponds to a human, the motion recognition system <NUM> may send one or more signals to the system controller <NUM> to alter or stop the operation of the industrial automation devices <NUM>, the machines <NUM>, or the like to protect the human now present in the area. In this manner, the motion recognition system <NUM> may allow a wide variety of objects of various sizes and orientations to pass into a hazardous area, while maintaining a safe environment in the industrial automation environment <NUM> when a human is present.

The notification actions may include sending a notification message (e.g., e-mail, text message) to an administrator or operator of the industrial automation system <NUM> that indicates that the automation command has been implemented, that the control action of block <NUM> has been implemented. In one embodiment, the notification message may also include an indication that a human is present in the industrial automation environment <NUM> when it is not authorized to be present and the like.

In addition to implementing control actions and/or notification actions based on whether the detected positions and/or motions are made by humans or non-humans, the motion recognition system <NUM> may also implement control actions and/or notification actions based distinctions detected between humans present in the industrial automation environment <NUM>. For instance, <FIG> depicts an example of a method <NUM> for implementing automation commands and/or notification actions based on distinctions detected between humans in the industrial automation environment.

Referring now to <FIG>, at block <NUM>, the motion recognition system <NUM> may scan the industrial automation environment <NUM> using the range camera <NUM>. At block <NUM>. the motion recognition system <NUM> may identify humans operating or present in the industrial automation environment <NUM>. The motion recognition system <NUM> may then determine which of the present humans are authorized to implement automation commands using the motion recognition system <NUM>.

In one embodiment, the human <NUM> may perform a particular motion or gesture to designate himself as an authorized person in the industrial automation environment <NUM>. As such, the human <NUM> may use a gesture-based sign in motion or authentication process to designate himself as an authorized person.

In the claimed embodiment, the motion recognition system <NUM> may search for individuals wearing a particular article of clothing such as a vest, a button, a belt, gloves, or the like. The particular article of clothing may designate the individual as the authorized human for implementing automation commands. In some instances, humans carrying or wearing an undesirable object such as a gun or laptop may be designated as unauthorized personnel. In these instances, the motion recognition system <NUM> may send a notification or alarm to indicate the presence of unauthorized personnel in the industrial automation environment <NUM>.

In yet another embodiment, the motion recognition system <NUM> may search for an emblem or symbol disposed on the human <NUM> such as a badge to designate the human <NUM> as the authorized human. In some instance, the motion recognition system <NUM> may work in conjunction with the sensors <NUM> to detect a radio frequency identification (RFID) tag or the like disposed on the human <NUM> to designate the human <NUM> as the authorized human.

The motion recognition system <NUM> may also store profiles of authorized individuals in its memory. For instance, the profiles may include data related to facial features, body parameters, or the like associated with authorized humans. As such, the motion recognition system <NUM> may, at block <NUM>, identify authorized personnel based on whether their facial features, body parameters, or the like match that of the facial features, body parameter, or the like stored in the profiles.

The motion recognition system <NUM> may also recognize an authorized human be receiving a gesture or motion from a known authorized human indicating that another human is also authorized. That is, a known authorized human may transfer his authorization status to another human or designate another human in the industrial automation environment <NUM> as authorized using particular motions, gestures, or the like.

The motion recognition system <NUM> may also identify authorized humans by detecting specific color or clothing attributes on humans. In one embodiment, different colors or clothing attributes disposed on a human may indicate different levels of authority. That is, different levels of authority may be associated with different humans in the industrial automation environment <NUM>. Each different level of authority may enable the respective human to implement a different set of automation commands. Although the different levels of authority has been described as being associated on humans based on detecting specific colors or clothing attributes, it should be noted that the motion recognition system <NUM> may designate different humans as having different levels of authority based on particular motions or gestures performed by a human, particular articles of clothing worn by the human, certain emblems, symbols, or tags being detected on the human, stored profiles of the human, as described above.

After the motion recognition system <NUM> identifies the authorized personnel at block <NUM>, the motion recognition system <NUM> may, at block <NUM>, receive detected positions and/or motions that correspond to the identified authorized personnel from the range camera <NUM>. In certain embodiments, the motion recognition system <NUM> may proceed to block <NUM> of <FIG>. As such, the motion recognition system <NUM> may then proceed to block <NUM> and follow the process of the method <NUM> to implement automation commands using confidence values.

After receiving the detected positions and/or motions of the authorized human at block <NUM>. the motion recognition system <NUM> may proceed to block <NUM> of <FIG> and follow the process of the method <NUM>. As such, the motion recognition system <NUM> may determine whether an expected secondary input has been received, as described above with reference to block <NUM> of <FIG>. As such, the motion recognition system <NUM> may verify the validity of the determined automation command of the authorized human based on whether an expected secondary input has been received by the system controller <NUM> or the motion recognition system <NUM>.

Claim 1:
A method for performing industrial automation control, comprising:
detecting, via a sensor system, image data related to positions and/or motions of one or more humans and one or more objects in an industrial automation system;
converting the image data into two-dimensional representations of the one or more humans and the one or more objects, wherein each of the two-dimensional representations consists of a plurality of data points located at edges of the respective two-dimensional representation;
distinguishing, via a programmed computer system, between the one or more humans and the one or more objects by comparing, for each of the two-dimensional representations, the positions of the respective plurality of data points with a plurality of templates, wherein each of the plurality of templates represents a data point arrangement of a human;
designating one of the one or more humans identified by the distinguishing process as an authorized operator based on an article carried or worn by the one of the one or more humans; and
implementing a control action to control operation of a device of the industrial automation system based upon the detected positions and/or motions of the authorized operator.