Fluid human-machine interface

A method of operating a communication device to interface with a machine system comprises projecting a human-machine interface (HMI) system for the machine system on a surface, wherein the HMI system comprises a plurality of commands associated with the machine system, detecting an input from a user, wherein the input comprises a fluid motion in air performed by the user corresponding to a selection of a command of the plurality of commands associated with the machine system, and transferring the selected command for delivery to the machine system.

TECHNICAL FIELD

The invention is related to the field of human-machine interfaces, and in particular, to a system that projects a human-machine interface for a machine on a surface.

TECHNICAL BACKGROUND

Industrial environments include automobile manufacturing factories, food processing plants, and microprocessor fabrication facilities. The typical industrial environment includes various machines, such as drives, pumps, motors, and robots. These machines continually produce data that indicates the current status of the machines, such as the machine's pressure, temperature, or speed.

The typical industrial environment also includes a Human-Machine Interface (HMI). The HMI receives and processes the status data from the machines to generate various graphical displays. The graphical displays indicate the current and historical status of the machines. For example, an HMI graphical display might indicate status metrics of a drive, the pressure of a pump, the speed of a motor, or the output of a robot. The HMI may also control the machines. For example, the HMI might update drive parameters, turn on a pump, speed-up a motor, or stop a robot.

Existing HMI systems often require a user to operate an HMI from a remote location, far away from the actual machine that the HMI is interfacing with and controlling. Even if the user can operate an HMI in close proximity to the machine being monitored or controlled, the HMI is typically a physical device that must be operated by physical touches provided by the user, often requiring the use of a mouse, keyboard, or similar input devices.

TECHNICAL SUMMARY

A method of operating a communication device to interface with a machine system comprises projecting a human-machine interface (HMI) system for the machine system on a surface, wherein the HMI system comprises a plurality of commands associated with the machine system, detecting an input from a user, wherein the input comprises a fluid motion in air performed by the user corresponding to a selection of a command of the plurality of commands associated with the machine system, and transferring the selected command for delivery to the machine system.

In some examples, the method further comprises receiving a user identifier and a machine system identifier, wherein the user identifier identifies the user and the machine system identifier identifies the machine system, and determining a proximity of the user to the machine system, wherein projecting the HMI system for the machine system comprises projecting the HMI system if the proximity of the user to the machine system satisfies a proximity requirement.

In some examples, projecting the HMI system for the machine system comprises projecting the HMI system for the machine system based on the machine system identifier.

In some examples, projecting the HMI system for the machine system comprises selecting the HMI system for the machine system based on the machine system identifier, determining if the user is authorized to view the HMI system based on the user identifier, and if the user is authorized to view the HMI system, projecting the HMI system for the machine system if the proximity of the user to the machine system satisfies the proximity requirement.

In some examples, detecting the input from the user comprises processing the fluid motion performed by the user in relation to the projected HMI system to determine the selected command associated with the machine system.

In some examples, projecting the HMI system for the machine system on the surface comprises projecting the HMI system for the machine system on an exterior surface of the machine system.

In some examples, projecting the HMI system for the machine system on the surface comprises projecting the HMI system for the machine system on a wall.

A communication device configured to interface with a machine system comprises a human-machine interface (HMI) display device, a processing system, and a communication interface. The HMI display device is configured to project an HMI system for the machine system on a surface, wherein the HMI system comprises a plurality of commands associated with the machine system. The processing system is configured to detect an input from a user, wherein the input comprises a fluid motion in air performed by the user corresponding to a selection of a command of the plurality of commands associated with the machine system. The communication interface is configured to transfer the selected command for delivery to the machine system.

A computer readable medium has program instructions stored thereon. When executed by a communication device, the program instructions are configured to direct the communication device to project a human-machine interface (HMI) system for the machine system on a surface, wherein the HMI system comprises a plurality of commands associated with the machine system, detect an input from a user, wherein the input comprises a fluid motion in air performed by the user corresponding to a selection of a command of the plurality of commands associated with the machine system, and transfer the selected command for delivery to the machine system.

DETAILED DESCRIPTION

FIG. 1is a block diagram that illustrates industrial environment100. Industrial environment100comprises machine systems101-103, communication network110, and communication device120. Machine systems101-103are in communication with communication network110over communication link105. Communication device120is in communication with communication network110over communication link121. Communication device120projects Human-Machine Interface (HMI) system104on a surface. User125operates HMI system104using fluid motions in air or space as interpreted by communication device120. Note that the number of machine systems shown inFIG. 1has been restricted for clarity; there would typically be many more.

Industrial environment100comprises an automobile manufacturing factory, food processing plant, microprocessor fabrication facility, or some other type of industrial enterprise. Machine systems101-103comprise drives, pumps, motors, robots, or some other mechanical apparatus, including their associated control systems. A control system comprises, for example, a programmable logic controller (PLC). Additionally, machine systems101-103comprise other, non-mechanical elements, such as a brew kettle in a brewery, a reserve of coal or other resources, or any other element that may reside in an industrial environment100.

Machine systems101-103continually produce operational data over time. The operational data indicates the current status of machine systems101-103, such as parameters, pressure, temperature, speed, or some other status metrics. The operational data may comprise dynamic charts or trends, real-time video, or some other graphical content. Machine systems101-103are capable of transferring the operational data over communication link105to communication network110and to communication device120via network110and communication link121.

Communication network110comprises multiple network elements such as routers, gateways, telecommunication switches, servers, processing systems, or other communication equipment and systems for providing communication and data services. Communication network110could comprise wireless communication nodes, telephony switches, Internet routers, network gateways, computer systems, communication links, or some other type of communication equipment—including combinations thereof. Communication network110may also comprise optical networks, asynchronous transfer mode (ATM) networks, packet networks, local area networks (LAN), metropolitan area networks (MAN), wide area networks (WAN), or other network topologies, equipment, or systems—including combinations thereof. Communication network110may be configured to communicate over metallic, wireless, or optical links. Communication network110may be configured to use time-division multiplexing (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format—including combinations thereof.

Communication device120comprises hardware and circuitry programmed to function as a telecommunications device. Communication device120may comprise a communication interface, user interface, memory device, software, processing circuitry, or some other communication components. For example, communication device120could comprise a telephone, wireless transceiver, mobile phone, computer, personal digital assistant (PDA), mobile Internet device, network interface card, or some communication apparatus—including combinations thereof. In some examples, communication device120could comprise a wireless communication device comprising Radio Frequency (RF) communication circuitry and an antenna. Communication device120could also comprise a camera and a projector. Communication device120is capable of displaying HMI system104on a surface, typically through the use of a projector. Communication device120is also capable of receiving operational data from machine systems101-103over communication link105. Communication device120is capable of processing the operational data of machine systems101-103to generate various graphical displays indicating the current and historical status of machine systems101-103, and can project the graphical displays as part of HMI system104. In addition, communication device120is also capable of displaying various control functions for HMI system104, such as buttons, levers, wheels, knobs, or other control mechanisms which a user125can manipulate through fluid motions in the air or space. By detecting the input motions of the user125and determining the corresponding control functions, communication device120is configured to transfer the control instructions over communication link121to communication network110and to machine systems101-103over communication link105.

HMI system104comprises a Human-Machine Interface for a machine system101-103projected by communication device120on a surface. For example, communication device120could display HMI system104on a surface through the use of a projector component of communication device120. HMI system104allows a user125to view status metrics related to a machine system101-103, and also provides a mechanism for the user125to control machine systems101-103as discussed above. For example, HMI system104might allow a user125to turn on a pump, speed-up a motor, stop a robot, boil a brew kettle, or perform some other type of machine control. To operate such functions, the user would simply provide a fluid motion in the air or space in relation to the projected image on the surface, and a camera of communication device120would monitor these motions and receive the motions as user input data. The user input is then interpreted by communication device120and the appropriate control instructions are sent to the appropriate machine system101-103. In addition, HMI system104may display various graphical displays as projected by communication device120, which could indicate, for example, the pressure of the pump, the speed of the motor, the output of the robot, the temperature of the brew kettle, or some other status metric.

Communication links105and121use metal, air, space, optical fiber such as glass or plastic, or some other material as the transport media—including combinations thereof. Communication links105and121could use various communication protocols, such as time-division multiplexing (TDM), Internet Protocol (IP), Ethernet, telephony, optical networking, hybrid fiber coax (HFC), communication signaling, wireless protocols, or some other communication format—including combinations thereof. Communication links105and121may be direct links or could include intermediate networks, systems, or devices. For example, communication links105and121could comprise a wireless network, wired network, optical network, local area network, wide area network, or some other communication network—including combinations thereof.

FIG. 2is a flow diagram that illustrates an operation of communication device120in industrial environment100. The steps of the operation are indicated below parenthetically. InFIG. 2, communication device120projects human-machine interface (HMI) system104for machine system101on a surface, wherein the HMI system comprises a plurality of commands associated with machine system101(201). The plurality of commands could comprise instructions to control an operation of machine system101, to request status metrics from machine system101, or some other command recognized by machine system101. AlthoughFIG. 1shows communication device120projecting HMI system104on a surface of machine system101, communication device120could also display HMI system104on any surface suitable for projection, such as a wall or door, or even the palm of a hand of user125. In some examples, communication device120could project HMI system104for a plurality of machine systems101-103.

Communication device101detects an input from a user125, wherein the input comprises a fluid motion in air performed by the user125corresponding to a selection of a command associated with machine system101(202). For example, user125can manipulate the plurality of commands displayed on HMI system104through fluid motions in the air or space. The camera component of communication device120is configured to monitor such fluid motions performed by a user125, and interpreting these motions to determine which control function the user125is requesting. In some examples, communication device120detects the input from user125by detecting the fluid motion performed by the user125in relation to the commands being displayed on the HMI system104. In some examples, the user125could provide multiple inputs by utilizing several fingers and/or both hands to provide the fluid motion input. In this case, the user125could differentiate the various inputs by associating different colors with different fingers, such as by wearing gloves with different color-coded finger pockets. Communication device120could then be configured to detect the different colors to determine the precise points of the input based on the colors.

Once communication device120detects an input from a user125corresponding to a selection of a command associated with machine system101, communication device120transfers the selected command for delivery to machine system101(203). Communication device120can be configured to provide the selected command in a format or language recognized by machine system101, so that machine system101is capable of processing the selected command. In other examples, communication device120transfers the selected command to a server or system within communication network110, and the server translates the selected command into a format that machine system101can parse. The server would then transfer the translated command to machine system101.

Advantageously, a user125of communication device120is able to project HMI system104on any surface, enabling the user125to quickly and efficiently access HMI system104to control and interface with a machine system101. For example, HMI system104displayed by communication device120could include various control functions, such as buttons, levers, wheels, knobs, or other control mechanisms which the user125can manipulate through fluid motions in the air or space. A camera component of communication device120is configured to monitor such fluid motions performed by the user125, and interprets these motions to determine which control function the user125is requesting. Communication device120can then execute the control functions determined by the motions of user125by sending appropriate control instructions for delivery to machine systems101-103over communication network110. In addition, user125can request status metrics from one or more machine systems101-103by performing various fluid motions in the air or space in a similar manner to requesting a control function as discussed above.

FIG. 3is a block diagram that illustrates communication device300. Communication device300provides an example of communication device120, although device120may use alternative configurations. Communication device300comprises communication interface301, processing system302, storage system304, HMI display device306, input device307, and communication system308. Storage system304stores communication device software310, HMI operating software311, and HMI interface software312.

Communication interface301comprises components that communicate over communication links, such as network cards, ports, RF transceivers, processing circuitry and software, or some other communication devices. Communication interface301may be configured to communicate over metallic, wireless, or optical links. Communication interface301may be configured to use time-division multiplexing (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format—including combinations thereof. Communication interface301communicates with machine systems101-103over communication network110. Communication interface301is configured to transfer a selected command for delivery to a machine system.

Processing system302comprises microprocessors or other logic circuitry that retrieves and executes communication device software310, HMI operating software311, and HMI interface software312. Storage system304comprises a disk, integrated circuit, flash drive, optical media, or some other memory device.

HMI display device306comprises a projector or some other graphical display projection mechanism. Input device307comprises a camera lens or some other photographic components capable of digital imaging. Communication system308comprises a bus, local area network, or some other communication apparatus. The above-described components (301,302,304, and306-308) of communication device300may be integrated together or distributed among multiple devices.

Communication device software310comprises operating software which may include an operating system, utilities, drivers, networking software, and applications capable of performing the functions described herein for a communication device. In particular, communication device software310directs processing system302to operate HMI display device306to project an HMI system for a machine system on a surface, wherein the HMI system comprises a plurality of commands associated with the machine system. In addition, communication device software310directs processing system302to operate input device307to detect an input from a user, wherein the input comprises a fluid motion in air performed by the user corresponding to a selection of one of the commands associated with the machine system. Finally, communication device software310directs processing system302to direct communication interface301to transfer the selected command for delivery to the machine system.

HMI software311-312comprises an application program, firmware, or some other form of machine-readable processing instructions. HMI operating software311may include an operating system, utilities, drivers, networking, and applications. HMI interface software311comprises an application. When executed by processing system302, HMI software311-312directs communication device300to operate an HMI system as described herein for HMI system104.

FIG. 4is a block diagram that illustrates industrial environment400. Industrial environment400comprises machine systems401-403and communication device420. Machine systems401-403are in communication with communication device420over communication system405. In this example, communication device420displays Human-Machine Interface (HMI) system404on a surface of machine system401. For example, machine system401may include a cabinet that houses elements of system401, and HMI system404could be displayed on the surface of the cabinet of machine system401. User425operates HMI system404using fluid motions in air or space as interpreted by communication device420. Note that the number of machine systems shown inFIG. 4has been restricted for clarity; there would typically be many more.

Industrial environment400comprises an automobile manufacturing factory, food processing plant, microprocessor fabrication facility, or some other type of industrial enterprise. Machine systems401-403comprise drives, pumps, motors, robots, or some other mechanical apparatus, including their associated control systems. A control system comprises, for example, a programmable logic controller (PLC). Additionally, machine systems401-403comprise other, non-mechanical elements, such as a brew kettle in a brewery, a reserve of coal or other resources, or any other element that may reside in an industrial environment400.

Machine systems401-403continually produce operational data over time. The operational data indicates the current status of machine systems401-403, such as parameters, pressure, temperature, speed, or some other status metrics. The operational data may comprise dynamic charts or trends, real-time video, or some other graphical content. Machine systems401-403are capable of transferring the operational data to communication device420over communication system405. In addition, communication device420transfers control instructions to machine systems401-403over communication system405. The control instructions control an operation of one or more machine systems401-403.

Communication system405could comprise a wireless network, wired network, optical network, local area network, wide area network, or some other communication network —including combinations thereof. Communication system405could use various communication protocols, such as time-division multiplexing (TDM), Internet Protocol (IP), Ethernet, telephony, optical networking, hybrid fiber coax (HFC), communication signaling, wireless protocols, or some other communication format —including combinations thereof. Communication system405may be a direct link or could include intermediate networks, systems, or devices. In some examples, communication system405comprises a personal area network (PAN) such as BLUETOOTH, or a wireless local area network (LAN), such as WI-FI, and communication device420communicates locally with machine systems401-403via communication system405using BLUETOOTH, WI-FI, or similar wireless technologies.

Communication device420comprises hardware and circuitry programmed to function as a telecommunications device. Communication device420may comprise a communication interface, user interface, memory device, software, processing circuitry, or some other communication components. For example, communication device420could comprise a telephone, wireless transceiver, mobile phone, computer, personal digital assistant (PDA), mobile Internet device, network interface card, or some communication apparatus—including combinations thereof. In some examples, communication device420could comprise a wireless communication device comprising Radio Frequency (RF) communication circuitry and an antenna. Communication device420could also comprise a camera and a projector. Communication device420is capable of displaying HMI system404on a surface, typically through the use of a projector. Communication device420is also capable of receiving operational data from machine systems401-403via communication system405. Communication device420is capable of processing the operational data of machine systems401-403to generate various graphical displays indicating the current and historical status of machine systems401-403, and can project the graphical displays as part of HMI system404. In addition, communication device420is also capable of displaying various control functions for HMI system404, such as buttons, levers, wheels, knobs, or other control mechanisms which a user425can manipulate through fluid motions in the air or space. By detecting the input motions of the user425and determining the corresponding control functions, communication device420is configured to transfer the control instructions over communication system405to machine systems401-403.

HMI system404comprises a Human-Machine Interface for a machine system401-403projected by communication device420on a surface. In this example, communication device420projects HMI system404on a surface of machine system401through the use of a projector component of communication device420. HMI system404allows a user425to view status metrics related to a machine system401-403, and also provides a mechanism for the user425to control machine systems401-403as discussed above. For example, HMI system404might allow a user425to turn on a pump, speed-up a motor, stop a robot, boil a brew kettle, or perform some other type of machine control. To operate such functions, the user would simply provide a fluid motion in the air or space in relation to the projected image on the surface, and a camera of communication device420would monitor these motions and receive the motions as user input data. The user input is then interpreted by communication device420and the appropriate control instructions are sent to the corresponding machine system401-403. In addition, HMI system404may display various graphical displays as projected by communication device420, which could indicate, for example, the pressure of the pump, the speed of the motor, the output of the robot, the temperature of the brew kettle, or some other status metric.

FIG. 5is a flow diagram that illustrates an operation of communication device420in industrial environment400. The steps of the operation are indicated below parenthetically. InFIG. 5, communication device420receives a user identifier and a machine system identifier, wherein the user identifier identifies a user425and the machine system identifier identifies a machine system401(501). Communication device420could receive the user identifier and the machine system identifier in a variety of ways.

To obtain the user identifier of the user425, communication device420could accept an input from the user425through various input techniques. In one example, communication device420could comprise a microphone, and the user425could instruct communication device420to accept a spoken user identifier by speaking a voice command along with his or her name, identification number, or other unique identifier, such as by speaking the words “user ID John Smith,” for example. Communication device420could employ voice identification techniques to verify the voice print of the user425to authorize the user425to view and operate the HMI system404.

In another example, a camera component of communication device420could photograph various physical attributes of user425to determine the user's identity. For example, this identity determination could be achieved if user425stands in front of a minor and uses his or her fingers to gesture in a circular motion around his or her face, and communication device420could be programmed to recognize the circular motion provided by the user425as a request to verify the facial identity of the user425. In another example, user425could speak a voice command, such as “authenticate user by facial scan,” which would notify communication device420that a facial identification procedure is requested by user425. In any event, the camera could photograph the facial features of user425and compare this photograph to a database of photographs of authorized persons using facial recognition techniques known to one skilled in the art.

In yet another example, user425could hold the camera component of communication device420in front of his or her eyeball, allowing the camera to perform a retinal scan of user425. Communication device420could also include a laser or other suitable retinal scanner for this purpose. In one example, communication device420could be instructed to perform the retinal scan by a voice command spoken by user425, or some other method could be used to instruct communication device420to perform the operation. For example, communication device420could be programmed to recognize an eyeball image when the user's eyeball is in close view of the camera, and upon recognition of the input into the camera as that of a close-up view of an eyeball, perform a retinal scan authorization of the eyeball. Communication device420could then compare the retinal scan of user425to a plurality of authorized retinal scans stored in a database in order to authorize user425to view and operate HMI system404.

In yet another example, the user425could simply hold a finger front of the camera component of communication device420, allowing communication device420to photograph the fingerprint of user425and compare this fingerprint to a plurality of known fingerprints stored in a database in order to authorize user425to view and operate HMI system404. Alternatively, communication device420could comprise a laser or other suitable fingerprint scanning device which the user425could use to scan his or her fingerprint in order to verify authorization to view and use HMI system404in the same manner as described above. Communication device420could be instructed to perform the fingerprint scan by voice command, by recognition of a close-up image of a finger in view of the camera, by receiving a fingerprint data input by the user425using a fingerprint scanning device of communication device420, or some other instruction method.

In addition to the above, traditional input methods could be utilized to provide a user identifier to communication device420. For example, a text-based username and password or other authorization code could be input by user425into communication device420through the use of a keyboard, key pad, touch screen, or other input device.

To obtain the machine system identifier of a particular machine system401-403, communication device420could accept an input of the machine identifier through a variety of similar input techniques. In one example, communication device420could comprise a microphone, and the user425could instruct communication device420to accept a spoken machine system identifier by speaking a voice command along with an identifier of a machine system401-403, such as by speaking the words “machine ID401” to identify machine system401, for example. In addition or alternatively to verifying the user identifier as discussed above, communication device420could employ voice identification techniques to identify the voice print of the user425speaking the machine system identifier, and determine the speaker's user identifier merely from the voice print of the spoken words “machine ID401.” In this manner, communication device420could determine that user425is authorized to view and operate an HMI system404for the requested machine system401merely by receiving the spoken words “machine ID401” and recognizing the voice print of user425as an authorized user identifier.

In some examples, each machine system401-403could comprise an RF tag, RF Identification (RFID), or some other device that transmits an RF signal to communication device420to indicate the machine identifier. The RFID tag may be passive, semi-passive, or active. In this case, communication device420could comprise a wireless communication interface capable of receiving an RFID tag from a particular machine system401-403. Although not required, communication device420could optionally transmit an RF excitation signal to drive transmission of an RFID tag from a machine system401-403. Additionally, a machine system401-403may be configured to transmit an RFID signal only within a given range. For example, communication device420would need to be proximate (in range) to machine system401in order to receive an RFID tag from machine system401that identifies machine system401. In some examples, this could be used to satisfy a proximity requirement, which is discussed in greater detail below.

In another example, the camera component of communication device420could photograph various attributes of a machine system401-403to determine a machine system identifier of a machine system401-403. For example, a machine system identifier may be physically scribed on a surface of a machine system401-403, and the camera component of communication device420could photograph this identifier off of the surface of the machine system. This may be achieved, for example, by the user425using his or her fingers to gesture around an identification number visible on a surface of a machine system401-403in the air or space, and by analyzing the digital image received by the camera, communication device420could determine from the gesture or from the number within the gesture that a machine identification has been requested. In another example, communication device420could be configured to simply identify a machine system401-403based on shape recognition. For example, using the camera component, communication device420could identify machine system403as a drive based on the shape of machine system403matching the shape of a drive. In this manner, communication device420could receive the machine system identifier of a machine system401-403.

Once communication device420receives the user identifier of a user425and the machine system identifier of a machine system401, communication device determines a proximity of the user425to the machine system401(502). Communication device420determines the proximity because device420will typically only display the appropriate HMI system404on the identified machine system401based on a proximity of the user425to the particular machine system401identified by the machine system identifier. In one example, communication device420determines the proximity of the user425to the machine system101identified by the machine system identifier based on the camera's ability to view the identified machine system101or to view a machine system identifier on a surface of the machine system101. Additionally or alternatively, the proximity of the user425to the particular machine system401-403identified may be determined by location determination technology installed within communication device420, such as Global Positioning System (GPS) components, for example. In other examples, communication device420determines the proximity of the user425to the machine system401based on personal area network (such as BLUETOOTH) or other RF connectivity between communication device420and machine system401.

Referring again toFIG. 5, if the proximity of the user425to the machine system401satisfies a proximity requirement, communication device420projects HMI system404for machine system401on a surface, wherein the HMI system comprises a plurality of commands associated with machine system101(503). Typically, communication device420projects the appropriate HMI system404for machine system401as indicated by the machine system identifier that identifies machine system401. The proximity requirement could be determined by threshold values. For example, the proximity requirement could be satisfied when the user425is within ten feet of machine system401. The plurality of commands could comprise instructions to control an operation of machine system401, to request status metrics from machine system401, or some other command recognized by machine system401. In some examples, communication device420could project HMI system404for a plurality of machine systems401-403. Additionally, as discussed above, the proximity requirement may be ensured by configuring a machine system401-403to transmit an RFID signal only within a given range.

The HMI system404displayed by communication device420could further comprise any application associated with or utilized by a particular machine system401-403. For example, HMI system404could include a diagnostic tool for troubleshooting error states related to a machine system401-403, such as a debugger program for debugging the operating code associated with a machine system401-403. Further, HMI system404could comprise a configuration application for a machine system401-403, to allow the user of communication device420to modify configuration settings of machine systems401-403. In other examples, if machine system403comprises a drive, HMI system404displayed by communication device420could comprise a drive explorer application. Additionally, communication device420could project ladder logic for a PLC as part of HMI system404.

Referring again toFIG. 5, communication device420detects an input from a user425, wherein the input comprises a fluid motion in air performed by the user425corresponding to a selection of a command associated with machine system401(504). For example, user425can manipulate the plurality of commands displayed on HMI system404through fluid motions in the air or space. The camera component of communication device420is configured to monitor such fluid motions performed by a user425, and interpreting these motions to determine which control function the user425is requesting. In some examples, communication device420detects the input from user425by detecting the fluid motion performed by the user425in relation to the commands being displayed on the HMI system404. In some examples, the user425could provide multiple inputs by utilizing several fingers and/or both hands to provide the fluid motion input. In this case, the user425could differentiate the various inputs by associating different colors with different fingers, such as by wearing gloves with different color-coded finger pockets. Communication device420could then be configured to detect the different colors to determine the precise points of the input based on the colors.

Once communication device420detects an input from a user425corresponding to a selection of a command associated with machine system401, communication device420transfers the selected command for delivery to machine system401(505). Communication device420can be configured to provide the selected command in a format or language recognized by machine system401, so that machine system401is capable of processing the selected command. In other examples, communication device420transfers the selected command to a server or system within communication network410, and the server translates the selected command into a format that machine system401can parse. The server would then transfer the translated command to machine system401.

Typically, communication device420displays an HMI system404on a surface of a machine system401-403that is specific to the machine system401-403that the HMI system404can monitor and control. For example, in industrial environment400ofFIG. 4, the HMI system404that is displayed on a surface of machine system401would typically monitor and control only machine system401based on the machine system identifier that identifies system401. This would be achieved by user425providing his or her user identifier and a machine system identifier of machine system401as discussed above, and by user425standing within close proximity to machine system401to satisfy the proximity requirement. In other words, communication device420typically projects the HMI system404for machine system401based on the user identifier, the machine system identifier of the machine system401, and a proximity of the user425to the machine system401.

Although the exemplary operation described above requires satisfaction of the proximity requirement before communication device420projects HMI system404for machine system401, exception conditions would allow an authorized user425to override the proximity requirement, which would be useful in the event of an emergency, for example. Under the exception condition, an authorized user425could view an HMI system404on a surface of a first machine system that controls and monitors a different, second machine system, such as machine system402or403, even if that user is far away from those machine systems402or403. For example, user425could speak an emergency override code to bypass the proximity requirement so that communication device420would display an HMI system404that monitors and controls machine system402or403on a surface of machine system401, for example. In addition, communication device420could also display an HMI system404on any surface, such as a wall or door, or even the palm of a hand of user425. An HMI system displayed on any surface could be configured to interface with a plurality of machine systems401-403.

Advantageously, a user425of communication device420is able to project HMI system404on any surface, enabling the user425to quickly and efficiently access HMI system404to control and interface with a machine system401. By projecting HMI system404, a user425of communication device420can manipulate various control functions of HMI404through fluid motions in air or space. For example, HMI system404could display buttons, levers, wheels, knobs, or other control mechanisms, and a camera component of communication device420can monitor the motions performed by the user425in relation to these control mechanisms, and interpret these motions to determine which control function the user425is requesting. Communication device420can then execute the control functions determined by the motions of user425by sending appropriate commands for delivery to machine systems401-403over communication system405. In addition, user425can request status metrics from one or more machine systems401-403by performing various fluid motions in the air or space in a similar manner to requesting a control function as discussed above.

FIG. 6is a block diagram that illustrates an exemplary display of a human-machine interface (HMI) system600projected by a communication device. As shown inFIG. 6, HMI system600displays a plurality of machine systems401-403, a control menu601, and a status menu602. Control menu601comprises a plurality of commands to control machine systems401-403. Status menu602comprises a plurality of commands to request status metrics from machine systems401-403. Note that menus601and602may display more or fewer commands than those depicted inFIG. 6. Also note that HMI system600may display more or fewer machine systems than the three machine systems401-403depicted inFIG. 6.

Machine systems401-403as displayed on HMI system600are graphical representations of machine systems401-403ofFIG. 4. Machine systems401-403could represent machine systems comprising pumps, motors, robots, vats, resources, or any other element residing in industrial environment400, including their associated control systems.

Control menu601could comprise commands to control any operation associated with machine systems401-403. In this example, commands to start or stop a machine system401-403, or to increase a speed of a machine system401-403are provided on control menu601. Status menu602could comprise commands to request any type of information associated with a machine system401-403. Examples of possible information types include maintenance, performance, or historical information, operating status of a machine or process, geographic location of a machine, or a list of other machines connected to a machine. In this example, status menu602provides a user425with options to request pressure, volume, or temperature metrics from machine systems401-403.

InFIG. 6, a user425has selected machine system402, as indicated by the dashed line surrounding machine system402. For example, the user425could indicate machine system402by gesturing around the image of machine system402projected by communication device420in HMI system600. In addition, the user425has provided an input of a selection of a command from control menu601. Specifically, the user425has selected to increase the speed of machine system402. The user425could select the command to increase the speed of machine system402by providing a fluid motion in relation to the “increase speed” command of control menu602on the projected HMI system600. Communication device420is configured to detect the fluid motions performed by the user425to determine the selected command. Communication device420then transfers a control instruction to machine system402that directs machine system402to increase its speed of operation.

In some examples, user425could provide a machine system identifier by gesturing around a particular machine system, as shown by the dashed line around machine system402inFIG. 6. In this case, communication device420could display an HMI system specific for machine system402as indicated by the user's selection of machine system402on HMI system600. Communication device420could then process the operational data of machine system402to generate various graphical displays indicating the current and historical status of machine system402, and can project the graphical displays as part of the HMI system for machine system402through the use of a projector. Communication device420is also capable of displaying various control functions associated with machine system402on the HMI system, such as buttons, levers, wheels, knobs, or other control mechanisms which a user425can manipulate through fluid motions in the air or space. The camera component of communication device420is capable of monitoring such fluid motions performed by the user425, and can interpret these motions to determine which control function the user425is requesting. Communication device420can then execute the control functions determined by the motions of user425by sending appropriate control instructions over communication system405to machine system402.

FIG. 7is a block diagram that illustrates communication device700. Communication device700provides an example of communication device420, although device420may use alternative configurations. Communication device700comprises machine interface701, processing system702, storage system704, HMI display device706, input device707, and communication system708. Storage system704stores communication device software710, HMI operating software711, and HMI interface software712.

Machine interface701comprises components that communicate over communication links, such as network cards, ports, RF transceivers, processing circuitry and software, or some other communication devices. Machine interface701may be configured to communicate over metallic, wireless, or optical links. Machine interface701may be configured to use time-division multiplexing (TDM), Internet Protocol (IP), Ethernet, optical networking, wireless protocols, communication signaling, or some other communication format—including combinations thereof. Machine interface701communicates with machine systems401-103over communication system405. Machine interface701is configured to transfer a selected command for delivery to a machine system.

Processing system702comprises microprocessors or other logic circuitry that retrieves and executes communication device software710, HMI operating software711, and HMI interface software712. Storage system704comprises a disk, integrated circuit, flash drive, optical media, or some other memory device.

HMI display device706comprises a projector or some other graphical display projection mechanism. Input device707comprises a camera lens or some other photographic components capable of digital imaging. Communication system708comprises a bus, local area network, or some other communication apparatus. The above-described components (701,702,704, and706-708) of communication device700may be integrated together or distributed among multiple devices.

Communication device software710comprises operating software which may include an operating system, utilities, drivers, networking software, and applications capable of performing the functions described herein for a communication device. In particular, communication device software710directs processing system702to operate input device707to receive a user identifier and a machine system identifier, wherein the user identifier identifies the user and the machine system identifier identifies the machine system. Further, communication device software710directs processing system702to determine a proximity of the user to the machine system, and if the proximity of the user to the machine system satisfies a proximity requirement, communication device software710directs processing system702to operate HMI display device706to project an HMI system for a machine system on a surface, wherein the HMI system comprises a plurality of commands associated with the machine system. In addition, communication device software710directs processing system702to operate input device707to detect an input from a user, wherein the input comprises a fluid motion in air performed by the user corresponding to a selection of one of the commands associated with the machine system. Finally, communication device software710directs processing system702to direct machine interface701to transfer the selected command for delivery to the machine system.

HMI software711-712comprises an application program, firmware, or some other form of machine-readable processing instructions. HMI operating software711may include an operating system, utilities, drivers, networking, and applications. HMI interface software711comprises an application. When executed by processing system702, HMI software711-712directs communication device700to operate an HMI system as described herein for HMI system404.

The above description and associated drawings teach the best mode of the invention. The following claims specify the scope of the invention. Some aspects of the best mode may not fall within the scope of the invention as specified by the claims. Also, while the preceding discussion describes embodiments employed specifically in conjunction with the monitoring and analysis of industrial processes, other applications, such as the mathematical modeling or monitoring of any man-made or naturally-existing system, may benefit from use of the concepts discussed above. Further, those skilled in the art will appreciate that the features described above can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific embodiments described above, but only by the following claims and their equivalents.