Augmented reality user interface on mobile device for presentation of information related to industrial process, control and automation system, or other system

A method includes identifying one or more first devices contained in at least one image captured by a mobile device. The method also includes obtaining data associated with at least one of: the one or more first devices and one or more second devices connected to or interacting with the one or more first devices. The method further includes presenting information overlaid over the at least one image by the mobile device to a user, where the presented information includes the data or information based on the data. The presented information varies based on perceived distance or proximity of the mobile device to the one or more first devices.

TECHNICAL FIELD

This disclosure generally relates to augmented reality visual displays. More specifically, this disclosure relates to an augmented reality user interface on a mobile device for the presentation of information related to an industrial process, control and automation system, or other system.

BACKGROUND

Industrial process control and automation systems are often used to automate large and complex industrial processes. Industrial processes are typically implemented using large numbers of devices, such as pumps, valves, compressors, or other industrial equipment used to implement various aspects of the industrial processes. Control and automation systems also routinely include large numbers of devices, such as sensors, actuators, and programmable logic controllers (PLCs), used to monitor and control the industrial equipment.

Numerous devices in these types of systems can generate operational, diagnostic, or other data and transmit the data to other components for analysis, storage, or other uses. For example, at least some of this data could be used to identify issues in control and automation systems or in the underlying industrial processes. Maintenance personnel or other personnel could then be dispatched to repair or replace equipment or take other suitable corrective actions to resolve the issues. Similar operations could occur in other systems that include large numbers of devices, such as building management systems.

SUMMARY

This disclosure provides an augmented reality user interface on a mobile device for the presentation of information related to an industrial process, control and automation system, or other system.

In a first embodiment, a method includes identifying one or more first devices contained in at least one image captured by a mobile device. The method also includes obtaining data associated with at least one of: the one or more first devices and one or more second devices connected to or interacting with the one or more first devices. The method further includes presenting information overlaid over the at least one image by the mobile device to a user, where the presented information includes the data or information based on the data. The presented information varies based on perceived distance or proximity of the mobile device to the one or more first devices.

In a second embodiment, a mobile device includes a display, a camera configured to capture at least one image of one or more first devices, and at least one processor. The at least one processor is configured to identify the one or more first devices contained in the at least one image. The at least one processor is also configured to obtain data associated with at least one of: the one or more first devices and one or more second devices connected to or interacting with the one or more first devices. The at least one processor is further configured to present information overlaid over the at least one image on the display to a user, where the presented information includes the data or information based on the data. The presented information varies based on perceived distance or proximity of the mobile device to the one or more first devices.

In a third embodiment, a non-transitory computer readable medium contains instructions that when executed cause at least one processing device to identify one or more first devices contained in at least one image captured by a mobile device. The medium also contains instructions that when executed cause the at least one processing device to obtain data associated with at least one of: the one or more first devices and one or more second devices connected to or interacting with the one or more first devices. The medium further contains instructions that when executed cause the at least one processing device to present information overlaid over the at least one image by the mobile device to a user, where the presented information includes the data or information based on the data. The presented information varies based on perceived distance or proximity of the mobile device to the one or more first devices.

DETAILED DESCRIPTION

FIG. 1illustrates an example industrial process control and automation system100according to this disclosure. As shown inFIG. 1, the system100includes various components that facilitate production or processing of at least one product or other material. For instance, the system100can be used to facilitate control over components in one or multiple industrial plants. Each plant represents one or more processing facilities (or one or more portions thereof), such as one or more manufacturing facilities for producing at least one product or other material. In general, each plant may implement one or more industrial processes and can individually or collectively be referred to as a process system. A process system generally represents any system or portion thereof configured to process one or more products or other materials in some manner.

InFIG. 1, the system100includes one or more sensors102aand one or more actuators102b. The sensors102aand actuators102brepresent components in a process system that may perform any of a wide variety of functions. For example, the sensors102acould measure a wide variety of characteristics in the process system, such as pressure, temperature, or flow rate. Also, the actuators102bcould alter a wide variety of characteristics in the process system. Each of the sensors102aincludes any suitable structure for measuring one or more characteristics in a process system. Each of the actuators102bincludes any suitable structure for operating on or affecting one or more conditions in a process system.

At least one network104is coupled to the sensors102aand actuators102b. The network104facilitates interaction with the sensors102aand actuators102b. For example, the network104could transport measurement data from the sensors102aand provide control signals to the actuators102b. The network104could represent any suitable network or combination of networks. As particular examples, the network104could represent at least one Ethernet network, electrical signal network (such as a HART network), pneumatic control signal network, or any other or additional type(s) of network(s).

The system100also includes various controllers106. The controllers106can be used in the system100to perform various functions in order to control one or more industrial processes. For example, a first set of controllers106may use measurements from one or more sensors102ato control the operation of one or more actuators102b. A second set of controllers106could be used to optimize the control logic or other operations performed by the first set of controllers. A third set of controllers106could be used to perform additional functions. The controllers106can communicate via one or more networks108and associated switches, firewalls, and other components.

Each controller106includes any suitable structure for controlling one or more aspects of an industrial process. At least some of the controllers106could, for example, represent programmable logic controllers (PLCs), proportional-integral-derivative (PID) controllers, or multivariable controllers such as controllers implementing model predictive control (MPC) or other advanced predictive control (APC). As a particular example, each controller106could represent a computing device running a real-time operating system, a WINDOWS operating system, or other operating system.

Operator access to and interaction with the controllers106and other components of the system100can occur via various operator consoles110. Each operator console110could be used to provide information to an operator and receive information from an operator. For example, each operator console110could provide information identifying a current state of an industrial process to the operator, such as values of various process variables and alarms associated with the industrial process. Each operator console110could also receive information affecting how the industrial process is controlled, such as by receiving setpoints or control modes for process variables controlled by the controllers106or other information that alters or affects how the controllers106control the industrial process. Each operator console110includes any suitable structure for displaying information to and interacting with an operator. For example, each operator console110could represent a computing device running a WINDOWS operating system or other operating system.

Multiple operator consoles110can be grouped together and used in one or more control rooms112. Each control room112could include any number of operator consoles110in any suitable arrangement. In some embodiments, multiple control rooms112can be used to control an industrial plant, such as when each control room112contains operator consoles110used to manage a discrete part of the industrial plant.

The control and automation system100here also includes at least one historian114and one or more servers116. The historian114represents a component that stores various information about the system100. The historian114could, for instance, store information that is generated by the various controllers106during the control of one or more industrial processes. The historian114includes any suitable structure for storing and facilitating retrieval of information. Although shown as a single component here, the historian114could be located elsewhere in the system100, or multiple historians could be distributed in different locations in the system100.

Each server116denotes a computing device that executes applications for users of the operator consoles110or other applications. The applications could be used to support various functions for the operator consoles110, the controllers106, or other components of the system100. Each server116could represent a computing device running a WINDOWS operating system or other operating system. Note that while shown as being local within the control and automation system100, the functionality of the server116could be remote from the control and automation system100. For instance, the functionality of the server116could be implemented in a computing cloud118or a remote server communicatively coupled to the control and automation system100via a gateway120. It is also possible to implement common or different functions using the server116and the computing cloud118or remote server.

As noted above, industrial processes are typically implemented using large numbers of devices, such as pumps, valves, compressors, or other industrial equipment. Similarly, industrial process control and automation systems are typically implemented using large numbers of devices, such as the sensors102a, actuators102b, controllers106, and other components inFIG. 1. Numerous devices can generate operational, diagnostic, or other data and transmit the data to other components for analysis, storage, or other uses. In some embodiments, operational data may generally relate to the industrial process(es) being controlled or monitored (such as temperatures, pressures, flow rates, or other values), while diagnostic data may generally relate to information about the equipment itself. However, other or additional forms or types of data could be used in these or other systems.

Getting the appropriate data to the appropriate personnel in a large or complex system can be a challenging task. For example, maintenance personnel or other personnel may need access to operational, diagnostic, or other data in order to properly diagnose and resolve issues within a control and automation system or the underlying industrial processes. This can be difficult when there are numerous components, possibly in close proximity to one another, in the field.

This disclosure provides techniques that utilize mobile devices122to display detailed information for devices in industrial processes, process control and automation systems, or other systems. This is accomplished by augmenting a camera view of one or more devices with one or more augmented reality visual overlays containing detailed information about those devices or about other devices connected to or interacting with those devices. For example, a user could aim a mobile device122at a controller106and view an augmented reality visual overlay containing information about the controller106or about devices coupled to the controller106(such as sensors102a, actuators102b, or other controllers106). The augmented reality visual overlays could include text, symbols, icons, animations, trends, or any other or additional information.

An augmented reality visual overlay can be overlaid over an image or a stream of images of one or more devices, such as by using the image processing features of the mobile device122. The visual overlay can be positioned over the image(s) based on special-purpose markers or natural features of the devices. For example, the mobile device122could use one or more geometries, colors, or textures of one or more controllers106or other devices to identify positioning information for one or more augmented reality objects. The augmented reality objects can then be inserted into the view from the camera of the mobile device122to give the appearance that the augmented reality objects are located on or adjacent to the device(s) being viewed through the camera. The augmented reality objects can be used to provide various levels of information to a user, such as information that varies based on how close the user is to the one or more devices being viewed through the camera. The user could also select various devices or augmented reality objects to obtain more detailed information about particular aspects of the devices or augmented reality objects.

The information used to generate the one or more augmented reality visual overlays could be obtained in any suitable manner. For example, the mobile devices122could obtain data directly from the devices themselves, such as through wireless communications via BLUETOOTH, WiFi, or infrared connections. The mobile devices122could also or alternatively obtain data indirectly from the devices, such as through a historian114, a local server116, or a cloud-based service provided using the computing cloud118.

Any suitable mobile device122having a camera and a display could be used to provide augmented reality visual overlays. In this example, the mobile devices122include a smart phone, smart glasses, and a tablet device. However, any other or additional types of mobile devices could be used here, such as laptop computers. Also, the functionality for providing the augmented reality visual overlays could be implemented in any suitable manner. For instance, a software “app” or other application could be loaded onto the mobile device122and executed by one or more processors of the mobile device122.

Additional details regarding augmented reality visual overlays for use in providing information related to industrial processes, control and automation systems, or other systems are provided below. Note that these details relate to specific implementations of devices that provide or support the visual overlays, specific implementations of the visual overlays, and specific uses of the visual overlays, although other implementations of the devices and visual overlays could be used and the visual overlays could be used for any suitable purposes.

AlthoughFIG. 1illustrates one example of an industrial process control and automation system100, various changes may be made toFIG. 1. For example, the system100could include any number of sensors, actuators, controllers, networks, operator consoles, control rooms, historians, servers, mobile devices, and other components. Also, the makeup and arrangement of the system100inFIG. 1are for illustration only. Components could be added, omitted, combined, further subdivided, or placed in any other suitable configuration according to particular needs. Further, particular functions have been described as being performed by particular components of the system100. This is for illustration only. In general, control and automation systems are highly configurable and can be configured in any suitable manner according to particular needs. In addition, whileFIG. 1illustrates one example operational environment where augmented reality visual overlays could be used to provide information about devices, this functionality can be used in any other suitable system (and that system need not relate to industrial process control and automation).

FIGS. 2 and 3illustrate example devices supporting augmented reality user interfaces for presenting information related to an industrial process, control and automation system, or other system according to this disclosure. In particular,FIG. 2illustrates an example mobile device200, andFIG. 3illustrates an example server or other computing device300. The mobile device200could be used to present augmented reality visual overlays to users, and the computing device300could be used to support the generation or presentation of augmented reality visual overlays to users (such as by providing operational, diagnostic, or other data to the mobile device200). For ease of explanation, the mobile device200and the computing device300are described as being used in the system100ofFIG. 1, although the mobile device200and the computing device300could be used in any other suitable system (whether or not related to industrial process control and automation).

As shown inFIG. 2, the mobile device200includes an antenna202, a radio frequency (RF) transceiver204, transmit (TX) processing circuitry206, a microphone208, receive (RX) processing circuitry210, and a speaker212. The mobile device200also includes a main processor214, a camera216, one or more physical controls218, a display220, and a memory222.

The RF transceiver204receives, from the antenna202, an incoming RF signal, such as a cellular, WiFi, or BLUETOOTH signal. The RF transceiver204down-converts the incoming RF signal to generate an intermediate frequency (IF) or baseband signal. The IF or baseband signal is sent to the RX processing circuitry210, which generates a processed baseband signal by filtering, decoding, and/or digitizing the baseband or IF signal. The RX processing circuitry210can transmit the processed baseband signal to the speaker212or to the main processor214for further processing.

The TX processing circuitry206receives analog or digital data from the microphone208or other outgoing baseband data from the main processor214. The TX processing circuitry206encodes, multiplexes, and/or digitizes the outgoing baseband data to generate a processed baseband or IF signal. The RF transceiver204receives the outgoing processed baseband or IF signal from the TX processing circuitry206and up-converts the baseband or IF signal to an RF signal that is transmitted via the antenna202.

The main processor214can include one or more processors or other processing devices and execute an operating system, applications, or other logic stored in the memory222in order to control the overall operation of the mobile device200. For example, the main processor214could control the transmission and reception of signals by the RF transceiver204, the RX processing circuitry210, and the TX processing circuitry206in accordance with well-known principles. In some embodiments, the main processor214includes at least one microprocessor or microcontroller, although other types of processing devices could also be used.

The main processor214is also capable of executing other processes and applications resident in the memory222. The main processor214can move data into or out of the memory222as required by an executing application. The main processor214is also coupled to the camera216, which provides data to the main processor214for the generation of digital images or video streams. The images or video streams can be presented to a user via the display220.

The main processor214is also coupled to the physical controls218and the display220. A user of the mobile device200can use the physical controls218to invoke certain functions, such as powering on or powering off the device200or controlling a volume of the device200. The display220may be a liquid crystal display (LCD), light emitting diode (LED) display, or other display capable of rendering text and graphics. If the display220denotes a touchscreen capable of receiving input, fewer or no physical controls218may be needed.

The memory222is coupled to the main processor214. Part of the memory222could include a random access memory (RAM), and another part of the memory222could include a Flash memory or other read-only memory (ROM). Each memory222includes any suitable structure for storing and facilitating retrieval of information.

As shown inFIG. 3, the computing device300includes at least one processor302, at least one storage device304, at least one communications unit306, and at least one input/output (I/O) unit308. Each processor302can execute instructions, such as those that may be loaded into a memory310. Each processor302denotes any suitable processing device, such as one or more microprocessors, microcontrollers, digital signal processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuitry.

The memory310and a persistent storage312are examples of storage devices304, which represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory310may represent a RAM or any other suitable volatile or non-volatile storage device(s). The persistent storage312may contain one or more components or devices supporting longer-term storage of data, such as a ROM, hard drive, Flash memory, or optical disc.

The communications unit306supports communications with other systems or devices. For example, the communications unit306could include at least one network interface card or wireless transceiver facilitating communications over at least one wired or wireless network. The communications unit306may support communications through any suitable physical or wireless communication link(s).

The I/O unit308allows for input and output of data. For example, the I/O unit308may provide a connection for user input through a keyboard, mouse, keypad, touchscreen, or other suitable input device. The I/O unit308may also send output to a display, printer, or other suitable output device.

As described above, the mobile device200is used to present one or more augmented reality visual overlays to a user. For example, the main processor214or other component of the mobile device200could receive one or more images from the camera216and identify at least one desired location for one or more augmented reality objects to be overlaid onto the images. The display220could then present the one or more images to the user along with the one or more augmented reality objects overlaid on the images. The one or more augmented reality objects can be based on or present operational, diagnostic, or other data for one or more devices contained within the images or related to the device(s) contained within the images. The computing device300could optionally be used to provide the data to the mobile device200or perform other functions related to the augmented reality visual overlays presented by the mobile device200. However, the mobile device200could perform all of the augmented reality operations itself without the use of any external computing devices.

AlthoughFIGS. 2 and 3illustrate examples of devices supporting augmented reality user interfaces for presenting information related to an industrial process, control and automation system, or other system, various changes may be made toFIGS. 2 and 3. For example, various components in each figure could be combined, further subdivided, or omitted and additional components could be added according to particular needs. As a particular example, each processor214or302could be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). Also, mobile and computing devices come in a wide variety of configurations, andFIGS. 2 and 3do not limit this disclosure to any particular mobile device or computing device.

FIGS. 4A through 5Eillustrate example augmented reality user interfaces according to this disclosure. For ease of explanation, the augmented reality user interfaces are described as being presented on a display220of a mobile device200when used in the system100ofFIG. 1. Thus, the augmented reality user interfaces here are used to provide information about devices in one or more industrial processes and/or one or more industrial process control and automation systems. However, any other suitable user interfaces could be generated and used in any suitable system (whether or not related to industrial process control and automation).

As shown inFIGS. 4A through 4D, different visual overlays can be presented to a user based (at least in part) on the user's distance from one or more devices and on the user's invocations of one or more commands. For example, inFIG. 4A, a visual overlay402could be presented when a user is farther away from one or more devices. This view could be referred to as a “distant” view, although this does not require a specific minimum distance from the devices. In this example, the visual overlay402includes or is presented over at least one image404(possibly an image stream). One or more devices406are included in or captured by the image(s)404, and each device406could be formed using one or more modules408. The visual overlay402here includes one or more augmented reality objects410displayed in association with the devices406. In this particular example, each augmented reality object410includes the name and status of a device406and an icon or other graphic symbol that identifies the status or other information about the device406. For instance, an icon formed using a green circle and a checkmark could indicate that the associated device406is currently operating without any alarms, warnings, or other adverse notifications. In contrast, an icon formed using a red circle and an “X” mark could indicate that the associated device406is currently operating under an alarm, warning, or other adverse notification.

InFIG. 4B, a visual overlay412could be presented when a user is closer to the devices. This view could be referred to as a “medium” view, although this does not require a specific minimum or maximum distance from the devices. In this example, the visual overlay412includes or is presented over at least one image414(possibly an image stream). At least one of the devices406is included in or captured by the image(s)414, along with its associated module(s)408. The visual overlay412here includes one or more augmented reality objects416each in the form of text identifying the name and status of an associated device406. The visual overlay412also includes augmented reality objects418that present statuses for individual modules408of at least one device406. As inFIG. 4A, each augmented reality object418here can take different forms, such as depending on whether an associated module408is currently operating under any alarms, warnings, or other adverse notifications. In this particular example, the statuses are presented for the middle device406within the image414, although repositioning of the mobile device200could allow the statuses to be presented for more than one device406or for different devices406. In some embodiments, at least some of the devices406could have associated graphical identifiers420, such as barcodes or QR codes.

InFIG. 4C, a visual overlay422could be presented when a user is very close to one or more devices406. This view could be referred to as a “near” view, although this does not require a specific minimum or maximum distance from the industrial equipment. In this example, the visual overlay422includes or is presented over at least one image424(possibly an image stream), and the contents of the visual overlay422could be based on what the user is “dwelling” upon in the displayed image(s)424. For example, a graphical marker426(a circle in this example) could be used to identify different physical or augmented reality objects within the displayed image(s)424. In the visual overlay422shown inFIG. 4C, the user is dwelling on the graphical identifier420that uniquely identifies a device406, which causes the visual overlay422to present augmented reality objects428identifying any modules408in the specific device406that are currently experiencing an issue. A user could select one of the augmented reality objects428to view additional details regarding the specific module408and its issue(s). The visual overlay422also presents a toolkit430, which includes various functions that could be selected by the user. These functions could relate to specific settings or parameters of the device406or any of its modules408, and the user could select one or more of the functions to reconfigure or otherwise alter the device406or its modules408.

InFIG. 4D, a visual overlay432is similar to the visual overlay412. In this example, the visual overlay432includes or is presented over at least one image434(possibly an image stream). At least one of the devices406and its associated module(s)408are included in or captured by the image(s)434. Again, the contents of the visual overlay432could be based on what the user is “dwelling” upon in the displayed image(s)434. For example, the user could dwell on a specific module408in a device406using the graphical marker426, causing the visual overlay432to present an augmented reality object436in the form of a window in which data about the specific module408can be viewed and possibly changed.

Note that while the graphical marker426is not shown in the visual overlays402and412, the graphical marker426could be used in those visual overlays or any other visual overlays at any suitable distance(s) from one or more devices406. Also note that while dwelling is one example mechanism for invoking certain augmented reality functions, other mechanisms could also be used. For example, one or more augmented reality objects428associated with a device406could be presented when the device's augmented reality object410is selected in the visual overlay402ofFIG. 4A, such as when a user touches the augmented reality object410on the display220. Similarly, the augmented reality object436could be presented when a module's augmented reality object418is selected in the visual overlay412ofFIG. 4Bor in the visual overlay432ofFIG. 4D, such as when a user touches the augmented reality object418on the display220. In general, various mechanisms and flow paths could be used to select devices or device modules and view information about those devices or device modules.

The visual overlays shown here allow a progressive disclosure of more and more detailed information to a user. For example, when the user positions the mobile device's camera some distance away from a device, a summary view may be given with a “rolled up” (summarized or aggregated) status of the device. Only a few indications (such as just an identification and a tick/cross indicating overall health) may be shown at this distance. An example of this is shown in the visual overlay402. As the user moves the mobile device122closer to the device, progressively more and more detailed information can be given, such as visual overlays that incorporate labels and current values for process variables. Examples of these are shown in the visual overlays412,422, and432. In the examples shown inFIGS. 4A through 4D, this progressive disclosure is performed for modular devices406that include a rack or row containing modules408, where modules408may contain channels or ports. The progressive disclosure shows a single identification and status for the whole rack or row, then the overall status for each module, and then the individual channel or port status on the specific modules that are in the camera view. Thus, the information presented to the user can vary based on distance or proximity to the devices.

The graphical marker426here denotes a “dwell selection” navigation feature that allows a user to locate an augmented reality pointer (which is an overlay object) on a specific component, module, or device feature or on an augmented reality object associated with the component, module, or device for a short period of time. If the graphical marker426remains relatively stationary during this short period of time, the user can be automatically presented with an additional window or other augmented reality object(s) showing more detailed information about the particular component, module, device, or augmented reality object selected by the user. The additional detail window or other augmented reality object(s) may persist until the next “dwell” navigation is performed by the user, or the additional detail window or other augmented reality object(s) may be closed through separate interactions. Among other things, this removes the need for a user to have to make touch interactions on the mobile device to select items that he or she wants more information about or to “drill down” into more details for specific items of interest. However, touch interactions can still be supported here.

FIGS. 5A through 5Eillustrate examples of the types of information that could be presented in the additional detail window (the augmented reality object436) shown in the visual overlay432inFIG. 4D. The additional detail window can be used to provide various information and various levels of information to a user. In some embodiments, the additional detail window could denote a hypertext markup language (HTML) window or other window that allows a user to navigate within the window and select other or additional information for display. Different content could be viewed in the window via appropriate navigation tools (such as hyperlinks) in the window.

InFIGS. 5A through 5E, the augmented reality object436includes two controls502and504located at the top of the window. The control502allows a user to close the augmented reality object436. The control504allows the user to export the information contained in the augmented reality object436to documentation that is viewable in a full screen window. The remaining contents of the augmented reality object436vary based on what information the user has requested to view.

InFIGS. 5A and 5B, the user has selected a specific module408, such as by dwelling on an image of the module408using the graphical marker426or by touching the augmented reality object418associated with that module408. However the module408is selected, the augmented reality object436is displayed containing information for that selected module408. Note thatFIGS. 5A and 5Bform part of the same display, where a user can use a scrollbar506or other mechanism to scroll through the information contained in the augmented reality object436. Here, the augmented reality object436displays information508identifying the selected module408, a status510of the selected module408, and a list512of I/O channels associated with the selected module408. The augmented reality object436could also display information514describing any errors associated with the selected module408.

In this particular example, the information508includes a device name, rack number, and module number associated with the selected module408, although any other or additional identifying information could be used. Also, in this particular example, the status510includes one of multiple icons (depending on whether the selected module408has any errors) and a number of errors associated with the selected module408(if any). However, any other suitable summary of the selected module's status could be used. Further, in this particular example, the list512of I/O channels includes a channel number, one of multiple icons (depending on whether the channel has any errors), a channel name, a spark line graphically showing a value associated with the channel over time, a current value, and a type of channel. The type of channel could have various values, such as analog input (AI), analog output (AO), digital input (DI), digital output (DO), pulse accumulator input (PI), resistance temperature detector (RTD), or thermocouple (TC). Of course, other or additional information associated with each channel of the selected module408could be displayed. In addition, in this particular example, for each error (if any) with the selected module408, the information514identifies the channel name and number, an icon indicating the channel has an error, the type of error, a spark line, a current channel value, a channel type, and a description of the error and how the error might be resolved. Again, other or additional information associated with each error with the selected module408could be displayed.

InFIG. 5C, the user has selected a specific channel of a specific module408, such as by selecting one of the channels in the list512or by selecting one of the errors identified in the displayed information514. However the channel is selected, the augmented reality object436is displayed containing information for that selected channel. Here, the augmented reality object436includes the information508identifying the module408associated with the selected channel and information520associated with the selected channel. In this particular example, the information520is the same as or similar to at least part of the information514contained in FIGS.5A and5B since both relate to the same error, although this need not be the case. The information520could, for example, contain more detailed descriptions of an error and how to resolve the error. Also, if the selected channel is not currently under an error, the displayed information520would be different.

InFIG. 5D, the user has selected a specific device406(like a PLC), such as by dwelling on the graphical identifier420of the device406using the graphical marker426. The device406selected here is the device associated with the module408previously being examined by the user inFIGS. 5A through 5C. However the device406is selected, the augmented reality object436displays information540identifying the selected device406, a status542of the selected device406, and a list544of various characteristics of the selected device406. The augmented reality object436could also display information546describing any errors associated with the selected device406.

In this particular example, the information540includes a device name and rack number for the selected device406, although any other or additional identifying information could be used. Also, in this particular example, the status542includes one of multiple icons (depending on whether the selected device406has any errors) and a number of errors associated with the selected device406(if any). However, any other suitable summary of the selected device's status could be used. Further, in this particular example, the list544includes various names for the device characteristics, one of multiple icons (depending on whether the characteristic has any errors), and other data associated with the characteristic like a spark line or textual data. Of course, other or additional information associated with each characteristic of the selected device406could be displayed. In addition, in this particular example, for each error (if any) with the selected device406, the information546identifies the characteristic name, an icon indicating the characteristic has an error, the type of error, a spark line, a current characteristic value, and a description of how the error might be resolved. Again, other or additional information associated with each error with the selected device406could be displayed. A dashed line548inFIG. 5Dis used to indicate that only some of the contents of the augmented reality object436might be displayed at any given time, and a user could use the scrollbar506or other mechanism to scroll through the information contained in the augmented reality object436.

InFIG. 5E, the user has selected a specific channel, characteristic, or other variable, such as by physically selecting or dwelling on a spark line, channel, or characteristic previously displayed in the augmented reality object436. However the specific channel, characteristic, or other variable is selected, the augmented reality object436displays information560associated with the specific variable and a trend diagram562for the specific variable. In this particular example, the information560includes a device name, rack number, module number, and variable name for the selected variable, although any other or additional identifying information could be used. Also, in this particular example, the trend diagram562plots the value of a variable over time and may include various markings564and annotations566that provide data to a user. Various controls568can be used to control the time period in which the variable's values are plotted in the trend diagram562. Of course, the trend diagram562could have any other suitable form, and any suitable controls568(or no controls568) could be used with the trend diagram562.

Note that any other or additional contents could be presented to the user in the window created using the augmented reality object436. Among other things, the contents could be based on the device or devices currently being viewed through the mobile device's camera and the user's interactions with the augmented reality object436or other augmented reality objects.

AlthoughFIGS. 4A through 5Eillustrate examples of augmented reality user interfaces, various changes may be made toFIGS. 4A through 5E. For example, the contents, layouts, and arrangements of the visual overlays inFIGS. 4A through 5Eare for illustration only. Augmented reality visual overlays could be used to display a wide range of information.FIGS. 4A through 5Eare meant to illustrate some potential uses of the visual overlays, and this disclosure is not limited to these specific augmented reality visual overlays.

FIGS. 6A through 7Dillustrate example techniques for using a mobile device supporting an augmented reality user interface according to this disclosure. For ease of explanation, the techniques shown inFIGS. 6A through 7Dare described as being supported by the mobile device200when used as a mobile device122in the system100ofFIG. 1. Thus, the techniques here are used to provide information about devices in one or more industrial processes and/or one or more industrial process control and automation systems. However, the techniques described here could be used with any suitable devices and in any suitable systems (whether or not related to industrial process control and automation).

As shown inFIGS. 6A through 6C, a user is attempting to locate a specific piece of equipment or a specific module, port, or channel of a specific piece of equipment. The user can use his or her mobile device122to scan tags or other visual indicators602(such as graphical identifiers420) on various pieces of equipment as shown inFIG. 6A, and details for different pieces of equipment can be presented on the mobile device122as shown inFIGS. 6B and 6C. The visual indicators602could denote barcodes, QR codes, or any other suitable graphical indicators.

To scan each visual indicator602, the user could view the visual indicator602using the camera216of his or her mobile device122, at which point the mobile device122can collect information using one or more images of the visual indicator602and identify the piece of equipment associated with the visual indicator602. The mobile device122can then display to the user information about the identified piece of equipment, such as its name and its status. Other or additional information, such as location information provided by a GPS or WiFi transceiver, a wireless beacon, or other mechanism, could also be used to identify a specific piece of equipment.

Prior to reaching the area with the specific piece of equipment, the mobile device122could provide navigation instructions or other information to help the user reach the general area in which the specific piece of equipment is located. Once the desired piece of equipment is identified, the user could use the mobile device122to identify specific modules of the equipment or perform other functions. This approach can help the user to locate desired equipment more quickly, which can often be difficult. For instance, the user may be attempting to physically locate a specific hardware fault on a single I/O channel of a controller having thousands of I/O channels.

As shown inFIGS. 7A through 7D, a user is attempting to troubleshoot a problem with a specific piece of equipment (a PLC in this example). The user can use his or her mobile device122to identify the specific piece of equipment as shown inFIG. 7Aand to obtain a list of augmented reality indicators identifying possible alarms, warnings, or other indicators for that specific piece of equipment as shown inFIG. 7B. The user could then dwell on each augmented reality indicator in order to view a window containing detailed information about that indicator as shown inFIGS. 7C and 7D. The user could use the detailed information for each alarm, warning, or other indicator to try and resolve the associated issue with the specific piece of equipment.

Among other things, this type of functionality could be used to improve the ease of locating devices. For example, a user could activate an “app” or other application on the user's mobile device122when the user is approaching or within a facility or other environment containing a number of devices. If the user wishes to locate a specific device and knows the general area where the specific device is located, the user could go to that general area. If the user does not know the general area where the specific device is located, the user could provide an identification of the specific device, an identification of an issue to be resolved involving the specific device, or other information to the mobile device122, and the mobile device122could provide navigation instructions to reach the specific device. In any case, once in the general area of the specific device, the user could use one or more augmented reality visual overlays to identify the specific device, such as by scanning tags or other visual indicators to identify the specific device or by using a visual overlay to identify a device suffering from a specific problem.

This type of functionality could also be used to improve the performance of other functions involving devices. For example, during the installation of a specific device, the user can use the app or other application on his or her mobile device122to obtain information about the specific device to assist in the physical installation or configuration of the specific device or its subcomponents. As another example, the user can use the app or other application on his or her mobile device122to obtain information about a specific device or about one or more other devices connected to or interacting with that specific device. The user could drill down as needed to identify any problems being experienced and possible solutions to those problems. If a specific module or other subcomponent of a device is identified as the cause of a problem or otherwise needs to be located, the user can use his or her mobile device122to locate that subcomponent. If the user needs to configure or reconfigure a device or any of its subcomponents, the user could do so using his or her mobile device122.

As can be seen here, this type of functionality can greatly increase the ease of performing various tasks in complex systems having large numbers of devices. While specific examples of these tasks are provided above, these are for illustration only, and any number of additional tasks could be supported or performed using this type of functionality.

Moreover, while this type of functionality is often described as being used with industrial processes and industrial process control and automation systems, this type of functionality could find use in a number of other areas. For example, building management solutions are often used to control the equipment used within office buildings or other large structures. As a particular example, automation controllers in an automation system can use operational, diagnostics, or other data to control equipment such as heating, ventilation, and air conditioning (HVAC) equipment and variable air volume (VAV) control equipment in a building or other structure. Operational data that could be used by the building automation controllers may include temperatures in different rooms or other areas of the structure. The same type of approach described above could be used to allow users' mobile devices122to present augmented reality visual overlays associated with devices in the automation system or the underlying control equipment. The visual overlays could be used to assist in functions such as the installation of, navigation to, or maintenance of these devices. As a particular example, the visual overlays could incorporate the display of labels and current values for variables associated with equipment and other operational information. A user's mobile device122could provide navigation instructions to reach specific devices, allow the scanning of tags or other visual indicators, provide progressive information about devices, or perform any other functions described above.

In addition, note that certain functions are described above as being based on distance, such as when progressively more information is provided to a user as the user moves closer to one or more devices. However, changes in actual physical distance may not be required to invoke those functions. For example, a user might not be able to approach one or more devices easily, such as due to physical obstacles. In these or other cases, a user could use the “zoom” feature on his or her mobile device's camera to move closer virtually to the one or more devices. Depending on the resolution and zoom abilities of the mobile device's camera, the user may be able to scan tags or other visual indicators or give the appearance that the user is physically approaching the devices. The contents of an augmented reality visual overlay can therefore vary based on a “perceived” distance or proximity to one or more devices, where the perceived distance or proximity is based on the one or more devices' size(s) or other features within one or more images. The perceived distance or proximity can be based on the actual physical distance(s) to the one or more devices or the zoomed-in size(s) of the one or more devices.

AlthoughFIGS. 6A through 7Dillustrate examples of techniques for using a mobile device supporting an augmented reality user interface, various changes may be made toFIGS. 6A through 7D. For example, augmented reality user interfaces could be used in a number of ways with devices in a system.FIGS. 6A through 7Dare meant to illustrate potential techniques for using visual overlays to assist users in performing certain functions, and this disclosure is not limited to these particular uses.

FIG. 8illustrates an example method800for presenting information related to an industrial process, control and automation system, or other system using an augmented reality user interface according to this disclosure. For ease of explanation, the method800is described as involving the use of the mobile device200in the system100ofFIG. 1. However, the method800could be used with any suitable device and in any suitable system (whether or not related to industrial process control and automation).

As shown inFIG. 8, at least one image captured by a mobile device is obtained at step802. This could include, for example, the processor214of the mobile device200receiving image data from the camera216and processing the image data to generate an image or a stream of images. At least one first device contained in the one or more images is identified at step804. This could include, for example, the processor214of the mobile device200using its image processing features to detect one or more barcodes, QR codes, or other graphical identifiers420contained in the captured image(s). Any other or additional features could be used to identify the first device(s) in the image(s), such as special-purpose markers or natural features of the device(s).

Data associated with the at least one first device or at least one second device connected to or interacting with the first device(s) is obtained at step806. This could include, for example, the processor214of the mobile device200receiving data directly from the first or second device(s) or receiving the data indirectly from the device(s) via another component, such as a historian114, server116, computing cloud118, or remote server. Information is presented to a user of the mobile device at step808. This could include, for example, the processor214of the mobile device200using its image processing features to insert one or more augmented reality objects over the previously-captured image(s) or newly-captured image(s). The presented information can vary based on perceived distance or proximity of the mobile device to the one or more first devices. The information could include at least some of the data obtained in step806or information that is based on that data.

The augmented reality object or objects that are presented to the user in step808could vary based on a number of factors. For example, as noted above, perceived distance or proximity can be based on actual distance or proximity or based on a device's size or other features within the image(s). In some embodiments, the augmented reality objects that are presented to the user could include names and statuses of multiple pieces of equipment when the mobile device200is farther away from the pieces of equipment. The augmented reality objects that are presented to the user could also include statuses of individual modules in at least one of the pieces of equipment when the mobile device200is closer to the pieces of equipment. The augmented reality objects could further include one or more objects identifying issues affecting at least one of the pieces of equipment or a window containing data associated with at least one of the modules in at least one of the pieces of equipment. As noted above, different types of information can be presented to the user using the augmented reality objects, and that information can vary as needed or desired.

Various operations can also occur depending on what the user chooses to do next. For instance, a user request for additional information related to the overlaid information could be detected at step810. This could include, for example, the processor214of the mobile device200detecting the user dwelling on an augmented reality object that was previously overlaid by the mobile device200. This could also include the user physically touching an augmented reality object that was previously overlaid by the mobile device200. Additional information associated with the user request is obtained at step812and presented in an overlaid manner at step814. This could include, for example, the processor214of the mobile device200requesting additional information related to the augmented reality object that the user dwelled upon, and the additional information could be obtained from any suitable source or sources (including the mobile device200itself).

As another example, a user request to invoke a function related to one or more devices could be detected at step816. This could include, for example, the processor214of the mobile device200detecting that the user has selected a function in a toolkit430displayed to the user. Any other or additional mechanism can also be used to allow a user to invoke one or more functions related to the first and second devices. Information associated with the requested function is received and the function is executed at step818. This could include, for example, the processor214of the mobile device200performing functions to allow the user to send commands to one or more devices, to reconfigure one or more devices, or perform other functions related to one or more devices.

AlthoughFIG. 8illustrates one example of a method800for presenting information related to an industrial process, control and automation system, or other system using an augmented reality user interface, various changes may be made toFIG. 8. For example, while shown as a series of steps, various steps inFIG. 8could overlap, occur in parallel, occur in a different order, or occur any number of times. Also, as described above, there are a number of ways in which an augmented reality user interface could be used with devices of a system, and the examples described above with respect toFIG. 8do not limit this disclosure to those example uses.