360° assistance for QCS scanner with mixed reality and machine learning technology

An apparatus, method, and non-transitory machine-readable medium provide for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology. The apparatus includes an optical sensor, a display, a Chatbot, cloud service, and a processor operably connected to the optical sensor and the display. The processor receives diagnostic information from a server related to a field device in an industrial process control and automation system; identifies an issue of the field device based on the diagnostic information; detects, using the optical sensor, the field device corresponding to the identified issue; guides, using the display, a user to a location and a scanner part of the field device that is related to the issue; provides, using the display, necessary steps or actions to resolve the issue; and connects, using a cloud server, a user to get modules of installation, commissioning, annual maintenance (AMC) and training for a quality control system (QCS) as per the selected persona.

TECHNICAL FIELD

This disclosure relates generally to autonomous operating industrial plants. More specifically, this disclosure relates to systems and methods for 360° assistance for a quality control system (QCS) scanner with mixed reality (MR) and machine learning technology.

BACKGROUND

Installation, upgrade, maintenance of QCS scanner requires skilled domain expertise since it involves handling of a radioactive source, precise work with sequence of procedural steps, and should be error free. The people in the industry with the expertise dealing with the QCS scanner are growing older and reaching retirement. New TAC and service regions are facing difficulties supporting the QCS scanner due to a competency gap and lack of expertise of knowledge. The troubleshooting of the QCS scanner requires domain expertise to identify a potential issue precisely and to fix the potential issue. Also, the time required to rectify the issue may vary depending on the field expertise. The training of the QCS scanner requires time on a physical scanner and physical environment for training purposes.

SUMMARY

This disclosure provides systems and methods for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology.

In a first embodiment, an apparatus provides for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology. The apparatus includes an optical sensor, a display and a processor operably connected to the optical sensor and the display. The processor receives diagnostic information from a server related to a field device in an industrial process control and automation system; identifies an issue of the field device based on the diagnostic information; detects, using the optical sensor, the field device corresponding to the identified issue; guides, using the display, a user to a location and a scanner part of the field device that is related to the issue; and provides, using the display, necessary steps or actions to resolve the issue.

In a second embodiment, a method provides for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology. The method includes receiving diagnostic information from a server related to a field device in an industrial process control and automation system; identifying an issue of the field device based on the diagnostic information; detecting, using the optical sensor, the field device corresponding to the identified issue; guiding, using the display, a user to a location and a scanner part of the field device that is related to the issue; and providing, using the display, necessary steps or actions to resolve the issue.

In a third embodiment, a non-transitory medium provides for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology. The instructions cause one or more processors to receive diagnostic information from a server related to a field device in an industrial process control and automation system; identify an issue of the field device based on the diagnostic information; detect, using the optical sensor, the field device corresponding to the identified issue; guide, using the display, a user to a location and a scanner part of the field device that is related to the issue; and provide, using the display, necessary steps or actions to resolve the issue.

DETAILED DESCRIPTION

FIG.1illustrates an example industrial process control and automation system100according to this disclosure. As shown inFIG.1, the system100includes various components that facilitate production or processing of at least one product or other material. For instance, the system100can be used to facilitate control over components in one or multiple industrial plants. Each plant represents one or more processing facilities (or one or more portions thereof), such as one or more manufacturing facilities for producing at least one product or other material. In general, each plant may implement one or more industrial processes and can individually or collectively be referred to as a process system. A process system generally represents any system or portion thereof configured to process one or more products or other materials in some manner.

InFIG.1, the system100includes one or more sensors102aand one or more actuators102b. The sensors102aand actuators102brepresent components in a process system that may perform any of a wide variety of functions. For example, the sensors102acould measure a wide variety of characteristics in the process system, such as pressure, temperature, flow rate, basis weight, moisture, ash, caliper, etc. Also, the actuators102bcould alter a wide variety of characteristics in the process system. Each of the sensors102aincludes any suitable structure for measuring one or more characteristics in a process system. Each of the actuators102bincludes any suitable structure for operating on or affecting one or more conditions in a process system.

At least one network104is coupled to the sensors102aand actuators102b. The network104facilitates interaction with the sensors102aand actuators102b. For example, the network104could transport measurement data from the sensors102aand provide control signals to the actuators102b. The network104could represent any suitable network or combination of networks. As particular examples, the network104could represent at least one Ethernet network, electrical signal network (such as a HART or FOUNDATION FIELDBUS network), pneumatic control signal network, or any other or additional type(s) of network(s).

The system100also includes various controllers106. The controllers106can be used in the system100to perform various functions in order to control one or more industrial processes. For example, a first set of controllers106may use measurements from one or more sensors102ato control the operation of one or more actuators102b. A second set of controllers106could be used to optimize the control logic or other operations performed by the first set of controllers. A third set of controllers106could be used to perform additional functions.

Controllers106are often arranged hierarchically in a system. For example, different controllers106could be used to control individual actuators, collections of actuators forming machines, collections of machines forming units, collections of units forming plants, and collections of plants forming an enterprise. A particular example of a hierarchical arrangement of controllers106is defined as the “Purdue” model of process control. The controllers106in different hierarchical levels can communicate via one or more networks108and associated switches, firewalls, and other components.

Each controller106includes any suitable structure for controlling one or more aspects of an industrial process. At least some of the controllers106could, for example, represent proportional-integral-derivative (PID) controllers or multivariable controllers, such as Robust Multivariable Predictive Control Technology (RMPCT) controllers or other types of controllers implementing model predictive control or other advanced predictive control. As a particular example, each controller106could represent a computing device running a real-time operating system, a WINDOWS operating system, or other operating system.

Operator access to and interaction with the controllers106and other components of the system100can occur via various operator consoles110. Each operator console110could be used to provide information to an operator and receive information from an operator. For example, each operator console110could provide information identifying a current state of an industrial process to the operator, such as values of various process variables and warnings, alarms, or other states associated with the industrial process. Each operator console110could also receive information affecting how the industrial process is controlled, such as by receiving setpoints or control modes for process variables controlled by the controllers106or other information that alters or affects how the controllers106control the industrial process.

Multiple operator consoles110can be grouped together and used in one or more control rooms112. Each control room112could include any number of operator consoles110in any suitable arrangement. In some embodiments, multiple control rooms112can be used to control an industrial plant, such as when each control room112contains operator consoles110used to manage a discrete part of the industrial plant.

Each operator console110includes any suitable structure for displaying information to and interacting with an operator. For example, each operator console110could include one or more processing devices114, such as one or more processors, microprocessors, microcontrollers, field programmable gate arrays, application specific integrated circuits, discrete logic devices, or other processing or control devices. Each operator console110could also include one or more memories116storing instructions and data used, generated, or collected by the processing device(s)114. Each operator console110could further include one or more network interfaces118that facilitate communication over at least one wired or wireless network, such as one or more Ethernet interfaces or wireless transceivers.

In accordance with this disclosure, a technique is provided for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology. One or more components of the system100(e.g., an operator console110) could be configured to perform one or more operations associated with this technique.

AlthoughFIG.1illustrates one example of an industrial process control and automation system100, various changes may be made toFIG.1. For example, industrial control and automation systems come in a wide variety of configurations. The system100shown inFIG.1is meant to illustrate one example operational environment in which a pressure sensor could be used.

FIG.2illustrates an example device for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology according to this disclosure. In particular,FIG.2illustrates an example computing device200. In some embodiments, the computing device200could denote an operator station, server, a remote server or device, or a mobile device. The computing device200could be used to run applications. For ease of explanation, the computing device200is described as being used in the system100ofFIG.1, although the device could be used in any other suitable system (whether or not related to industrial process control and automation).

As shown inFIG.2, the computing device200includes at least one processor202, at least one storage device204, at least one communications unit206, and at least one input/output (I/O) unit208. Each processor202can execute instructions, such as those that may be loaded into a memory210. Each processor202denotes any suitable processing device, such as one or more microprocessors, microcontrollers, digital signal processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or discrete circuitry.

The memory210and a persistent storage212are examples of storage devices204, which represent any structure(s) configured to store and facilitate retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory210may represent a random access memory or any other suitable volatile or non-volatile storage device(s). The persistent storage212may contain one or more components or devices supporting longer-term storage of data, such as a read-only memory, hard drive, Flash memory, or optical disc.

The communications unit206supports communications with other systems or devices. For example, the communications unit206could include at least one network interface card or wireless transceiver facilitating communications over at least one wired or wireless network. The communications unit206may support communications through any suitable physical or wireless communication link(s).

The I/O unit208allows for input and output of data. For example, the I/O unit208may provide a connection for user input through a keyboard, mouse, keypad, touchscreen, gesture control, image processing, or other suitable input device. The I/O unit208may also send output to a display, printer, or other suitable output device.

FIG.3illustrates an exemplary QCS scanner system300of 360° assistance for a QCS scanner310with mixed reality (MR) and machine learning technology according to this disclosure. The embodiment of the exemplary QCS scanner system300illustrated inFIG.3is for illustration only.FIG.3does not limit the scope of this disclosure to any particular implementation.

The QCS scanner system300provides for a mixed reality (MR) (augmented reality (AR)/virtual reality (VR), machine learning and Chatbot solutions resolving potential issues. Using the MR, the commission of the QCS system300is made safer, easier and more user friendly by augmenting the physical conditions with interactive guidance for installation and upgrading of the QCS scanners.

The QCS system300integrates QCS scanner diagnostic messages and fault information with the HoloLens305. The QCS system300receives the solution from a local/centralized solution center and enabling an interactive Chatbot and machine learning for troubleshooting.

The QCS system creates a virtual training for the QCS scanner using VR and AR, which reduces the overall cost of training and physical hardware. The QCS system provides instruction on safe handling of a radioactive source under a hazardous environment. The QCS system creates a mimic of scanner components, which provides the detail information about wiring details, equipment location identification, checkpoints and more.

The AR solution provides an augmented physical scanner with real time data for troubleshooting. The AR solution augments the step-by-step procedure for installing a QCS scanner. The AR solution can upload real-time scanner status.

The machine learning and Chatbot315provides a solution for easy troubleshooting based on previous data with interactive live chat sessions with machine and expert chanters. The machine learning and Chatbot315can record the issue and their resolving steps for future use.

The VR solution can provide an alternate means for practicing installation and commissioning of QCS scanner without the need for access to an expensive physical component. The VR solution can mimic QCS scanner scenarios like real system training of troubleshooting, and can show live status and tips of the QCS system.

The term “360° assistance” of QCS Scanner refers to an overall support of QCS scanner. Four major modules of QCS scanner with respect to support are Module 1: Troubleshooting of QCS scanner issues during on process; Module 2: Training; Module 3: Installation and commissioning of QCS scanner; and Module 4: Annual maintenance/periodic checks. All mentioned modules of QCS scanner support require different/combination of technologies and different approaches to achieve the standardized, time-bound, predictable and robustness in the process.

FIG.4illustrates an exemplary QCS scanner troubleshooting technique400with augmented reality, Chatbot, and machine learning technology according to the embodiments of the present disclosure. The embodiment of the exemplary QCS scanner troubleshooting technique400illustrated inFIG.4is for illustration only.FIG.4does not limit the scope of this disclosure to any particular implementation.

The QCS troubleshooting technique400includes a HoloLens405, a Chatbot410, an issue identification415, an eDocumentation420, a machine learning server425, and an expert support430. The HoloLens405is a holographic computer made to identify the QCS scanner and its internal parts, virtual wiring layout, connection identification, scanner mechanical parts identification and more based on the scanner version also capable of video streaming the scanner for remote assistant.

The Chatbot410can provide an interactive voice based Chatbot technology that accepts the voice input from user and provides the necessary output to guide the user to perform the necessary actions.

The issue identification415involves integrating the HoloLens with the QCS server and QCS scanner to provide scanner related diagnostics based on the diagnostic information. The HoloLens can guide the user to a location or scanner part where the issue occurred and can provide necessary steps or action to be performed to resolve the issue.

The eDocumentation420provides the HoloLens the ability to identify an object and provide information related to the object, e.g. a wiring diagram, mechanical connections, test points, and more. The eDocumentation also provides receive any document a user requests with the help of the Chatbot, which will reduce searching times, data availability, and improve the user experience. The HoloLens can identify an object related to a field device. The user can provide a command that the HoloLens receives using an audio sensor or from an external device. The HoloLens displays a document corresponding to the document type of the identified object.

The machine-learning server425is a cloud service provided to resolve the issue based on a criticality of the issue. The user can connect to the machine learning server using the Chatbot, can request a solution, can provide the solution based on previous occurrences of similar issues, and can record the steps of procedure followed to resolve the current issues through which the system can provide a more robust and accurate solution in the future.

The expert support430is used if the machine-learning server is not able to resolve the issue and a user needs expert support. The HoloLens can request the machine-learning server to connect to an available expert. Once connected with an expert, the expert can explain the issue with actual visuals of the issue in order to resolve the issue. Once the issue is resolved, the machine-learning server can record the steps performed to resolve the issue.

FIG.5illustrates an exemplary QCS scanner training system500with virtual reality and Chatbot technology according to the embodiments of the present disclosure. The embodiment of the exemplary QCS scanner training500illustrated inFIG.5is for illustration only.FIG.5does not limit the scope of this disclosure to any particular implementation.

The QCS training500includes a HoloLens505with a Chatbot510, cloud based training manuals including cloud-based eDocument & video training515and cloud based virtual training module520, and virtual training525.

The wearable or HoloLens505is a holographic computer that can mimic the QCS scanner in a virtual world that can image the virtual QCS scanner. The HoloLens505can show how the physical scanner looks and can show internal parts that can be virtually imaged to aide in user learning for different components (e.g. sensors, mechanics, hardware, and software configurations) before going an actual scanner goes live.

The Chatbot510is an interactive voice-based Chatbot that accepts a voice input from a user and can provide a necessary output to guide the user to perform necessary actions for resolving an issue.

The cloud-based eDocument & video training515can cover a basic introduction of the QCS scanner and the industrial uses. The cloud-based virtual training modules520can cover insights of the QCS scanner, sensors, mechanics, hardware and software configurations, handling, service, and troubleshooting. Examples of training modules can include an installation and commissioning module530, a QCS application module535, a troubleshooting module540, an AMC & service module545, etc.

The virtual training525is used when the user is wearing the HoloLens and connects to the cloud-based training modules. The HoloLens selects a persona of the training module. Using virtual reality, the Chatbot user can commission, troubleshoot, view plant scenario usage without the use of or access to a physical scanner.

FIG.6illustrates an exemplary installation, commissioning and AMC system600of QCS scanner using augmented reality and Chatbot technology according to the embodiments of the present disclosure. The embodiment of the exemplary system600of QCS scanner illustrated inFIG.6is for illustration only.FIG.6does not limit the scope of this disclosure to any particular implementation.

The system600of the QCS scanner includes installation, commissioning and annual maintenance contract (AMC). The system600includes a HoloLens605with a Chatbot610, a cloud-based installation & AMC module615, installation and commissioning620, expert support625, and AMC activities630.

The HoloLens605is a holographic computer made to identify a site location and provide prerequisite conditions and checks for installing a QCS scanner. The HoloLens605can guide the user during installation and commissioning of the QCS scanner, various sensors, and internal parts using augmented reality. The HoloLens can enable the remote assistant for expert advice.

The Chatbot610is an interactive voice-based chatbot that can accept the voice input from the user and can provide necessary outputs to guide the user to perform the necessary action for resolving an issue.

The cloud-based installation & AMC module615includes different modules that can be accessed based on requirements from the cloud that are largely classified in sub-modules. The sub-modules include an installation module635for supporting various versions of QCS scanners, QCS sensor module640, QCS software installation and configuration module645, AMC activities for QCS scanner module650, etc.

The installation and commissioning620includes the HoloLens connecting to a cloud service. The HoloLens selects a QCS scanner version and sensors available for commissioning. The installation and commissioning620can guide the HoloLens with step by step procedures for commissioning the QCS scanner along with software installation and configuration for a full fledge startup of the QCS scanner.

The expert support625can provide the HoloLens with expert support for installation, commissioning, and AMC if a user cannot figure out a part of the process or an issue with the QCS scanner. The HoloLens can connect with an expert and provide a visual of the QCS scanner while the in conversation with a user of the HoloLens. The expert can control the HoloLens to indicate components to the user. That way, the expert can better explain the step or procedure of resolving an issue.

The AMC activities630are based on a customer record system that can create an AMC checklist. The AMC activities630can control the HoloLens to guide the user to perform the ACM activity that enables an engineer to collect on-the-go reports of the activity and comments. The HoloLens can generate the final report of the AMC activity for the customer and user records.

FIGS.7A and7Billustrate an exemplary method700,701for troubleshooting a QCS scanner issue using augmented reality, Chatbot, and machine learning technology according to embodiments of the present disclosure. For example, the method described inFIGS.7A and7Bmay be performed in conjunction with the computing device200inFIG.2.

In operation705, the computing device200can detect an issue with the QCS scanner. In certain embodiments, the computing device200detecting an issue includes receiving diagnostic information from a server related to a field device, such as a QCS scanner, in an industrial process control and automation system. An issue with the field device includes any malfunction that causes the field device to not function at a suitable operational requirement.

In operation710, the computing device200can connect to a QCS server and receive a list of potential issues. The list of potential issues can include typical issues that have been identified on the particular machine itself or from a common malfunction list of the device type.

In operation715, the computing device200can receive a voice command for selecting a potential issue from the list of potential issues. The list of potential issues can be display on the display or provided as audio outputs to the user. In certain embodiments, the computing device200can identify an issue of the field device based on the diagnostic information.

In operation720, the computing device200can capture the QCS scanner using an optical sensor on the HoloLens. The computing device200can detect the field device corresponding to the identified issue using the optical sensor. Once the field device is captured and detected, the computing device can identify specific components that correspond to the identified issue.

In operation725, the computing device200can display an indication on a display of the HoloLens corresponding to a component related to the potential issue of the QCS scanner. The computing device200can guide, using the display, a user to a location and a scanner part of the field device that is related to the issue.

In operation730, the computing device200can receive a request in a voice command for related documents and procedures to resolve the issue. The computing device200can provide the necessary steps or action to resolve the issue. The necessary steps or actions can be displayed on the display. Specific components related to steps in the procedure can be highlighted or marked on the display with any related documents. The computing device200can display the related documents on the display away from the highlighted or marked components.

In operation735, the computing device200can determine whether an issue is resolved. If the issue is resolved, the computing device200proceeds to operation780. If the issue is not resolved, the computing device200proceeds to operation740.

In operation740, the computing device200can detect that the issue is not resolved. The computing device200can receive operating data from the QCS server and determine that the field device is still not operating efficiently.

In operation745, the computing device200can connect the HoloLens to the cloud-based service on a machine-learning server for additional help with the potential issue. The cloud-based service can be directly related to a specific component.

In operation750, the computing device200can provide detailed information of the issue to the machine-learning server using a Chatbot service. The detailed information can include the information of the field device from the QCS server along with any information captured by the computing device200. The information captured by the computing device can include a live feed or frames captured from the optical sensor, audio captured from the user, frames captured from the display of the process used to fix the issue, etc.

In operation755, the computing device200can receive a relevant solution for the issue based on a previous history from the machine-learning server. The previous history includes issues resolved from the field device previously, as well as other field devices of the same type. The machine-learning server can provide an optimal solution based on all the input data or different alternative options.

In operation760, the computing device200can determine whether the issue is resolved. If the issue is resolved, the computing device200proceeds to operation780. If the issue is not resolved, the computing device200proceeds to operation765.

In operation765, the computer device200can request the machine-learning server to connect to a subject matter expert at a technical assistance center (TAC) center. The subject matter expert can be an individual that is experienced with the specific type of field device or an individual that has dealt with the specific issue.

In operation770, the computing device200can provide the detailed information of the issue and a live feed from the optical sensor of the HoloLens to the subject matter expert. The subject matter expert can be connected to live or sent the relevant information related to the field device.

In operation775, the computing device200can receive an expert report for resolving the issue from the subject matter expert. The expert report can include step-by-step instructions for resolving the issue. In each step, different components that correspond to the respective step can be highlighted or have a marker placed on the display for identification.

In operation780, the computing device200can log the issue type and resolve procedure to the machine-learning server. The issue type and resolve procedure can be related to a specific component or assembly of components, or related to a malfunction of the field device. In operation785, the computing device200determines that the issue has been resolved.

AlthoughFIGS.7A and7Billustrates one example of a method700,701for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology, various changes may be made toFIG.7. For example, various steps shown inFIG.7could overlap, occur in parallel, occur in a different order, or occur any number of times.

FIGS.8A and8Billustrate an exemplary flowchart for installation and commissioning according to the embodiment of the present disclosure. For example, the method described inFIGS.8A and8Bmay be performed in conjunction with the installation, commissioning and AMC system600inFIG.6.

In operation805, the installation, commissioning and AMC system600can begin the installation, commissioning, and AMC system of a QCS scanner.

In operation810, the installation, commissioning and AMC system600, from the wearable device, can connect to the cloud server for getting the persona for installation, commissioning, and AMC.

In operation815, the installation, commissioning and AMC system600can determine whether installation and commissioning persona is selected.

In operation820, the installation, commissioning and AMC system600, using a voice command to the wearable device, can request the cloud server for the required installation module for support version of QCS scanner.

In operation825, the installation, commissioning and AMC system600can get the list supported hardware and software installation for the QCS scanner and sensors.

In operation830, the installation, commissioning and AMC system600, using the voice command to wearable, can request the cloud server for the required module, including a QCS scanner installation module, QCS sensors modules, and QCS software installation and configuration module.

In operation835, the installation, commissioning and AMC system600, from the cloud server, can receive a required sequence of a procedure to be performed for installation and commissioning of QCS scanner in wearable device.

In operation840, the installation, commissioning and AMC system600can perform the installation and commissioning from instructions and can evaluate the result on the wearable device.

In operation845, the installation, commissioning and AMC system600can determine if the user is able to perform the instruction or set of instructions.

In operation850, the installation, commissioning and AMC system600can receive help from subject matter expertise to resolve an issue with live visual and interactive chatting.

In operation855, the installation, commissioning and AMC system600can select the AMC module.

In operation860, the installation, commissioning and AMC system600, using a voice command to the wearable device, can request the cloud server for an AMC record and checklist based on the QCS scanner version.

In operation865, the installation, commissioning and AMC system600can receive the list checklist for a selected QCS scanner for AMC.

In operation870, the installation, commissioning and AMC system600, from the cloud server, can receive required sequence of procedure to be performed for AMC QCS scanner in a wearable device.

In operation875, the installation, commissioning and AMC system600can perform the AMC from the instruction and evaluate the result on the wearable device.

In operation880, the installation, commissioning and AMC system600determines whether the user is able to perform the instructions.

In operation885, the installation, commissioning and AMC system600can receive help from a subject matter expert to resolve issues with live visual and interactive chatting.

AlthoughFIGS.8A and8Billustrates one example of a method800,801for 360° assistance for a QCS scanner with mixed reality (MR) and machine learning technology, various changes may be made toFIG.8. For example, various steps shown inFIG.8could overlap, occur in parallel, occur in a different order, or occur any number of times.