Systems and methods for providing extraction on industrial diagrams and graphics

A method to facilitate extraction of display objects for industrial diagrams are disclosed herein. The method comprises: receiving user input indicating a first display object within an industrial diagram; extracting the first display object to generate a first graphic extraction template; identifying one or more regions within the first graphic extraction template; masking the text information; linking each of the one or more regions with at least a portion of an object name of the first display object; extracting all the display objects, from the industrial diagram, that are of the type of the first display object using the first graphic extraction template to generate a first set of extracted graphic objects; and for each of the first set of extracted graphic objects, matching text information within each of the one or more regions with at least a portion of an object name.

TECHNICAL FIELD

Aspects of the disclosure are related to computing hardware and software technology for providing extraction on industrial diagrams and graphics, and more specifically related to computing hardware and software technology for providing extraction on piping and instrumentation diagrams (P&IDs) and human-machine interface (HMI) static graphics.

TECHNICAL BACKGROUND

Various manufacturing processes and other industrial operations occur in industrial automation environments. Some examples of industrial automation environments include industrial mining operations, automobile manufacturing factories, food processing plants, oil drilling operations, microprocessor fabrication facilities, and other types of industrial enterprises. Industrial automation environments typically involve many complex systems and processes which are often spread out over various disparate locations.

Industrial automation environments utilize various machines during the industrial manufacturing process, such as drives, pumps, motors, compressors, valves, robots, and other mechanical devices. These devices have various moving parts and other components that are driven by instructions received from industrial controller systems. Machine builders, solution providers, and other content creators typically produce the control logic needed to run on these industrial controller systems in order to control the mechanical functions of the devices and carry out their intended functions.

Industrial environments also commonly include a human-machine interface (HMI). An HMI typically receives and processes the status data from the machines to generate various graphical displays, which may indicate the current and historical performance of the machines. In traditional implementations, the HMI may also provide a mechanism for an operator to send control instructions to a control system that controls the machines. For example, an operator might use the HMI to direct the control system to update drive parameters, turn on a pump, speed-up a motor, or stop a robot.

Overview

Engineers usually have to manually extract information based on the graphical representation of equipment or instruments found on a P&ID or an HMI display. This manual extraction is time consuming and tedious, and requires manually updating the extracted data whenever there is a change in the P&ID or the HMI display. Systems and methods for providing extraction of display objects within P&ID or HMI are disclosed herein.

One embodiment relates to a method to facilitate extraction of display objects for industrial diagrams. The method includes: receiving an industrial diagram comprising a plurality of display objects; receiving user input indicating a user selected display object; extracting the user selected display object to generate a graphic extraction template, wherein the graphic extraction template has the same visual appearance of the user selected display object; applying mask on one or more regions within the graphic extraction template containing text information specific to the user selected display object; generating a link between each of the one or more regions and at least a portion of an object name of the user selected display object; extracting all the display objects that have the same type as the user selected display object using the graphic extraction template to generate a set of extracted graphic objects; and identifying an object name for each of the set of extracted graphic objects using the one or more links associated with the graphic extraction template.

Another embodiment relates to a non-transitory computer-readable medium comprising computer-executable instructions that, when executed, are configured to cause a processor to perform operations comprising: receiving an industrial diagram comprising a plurality of display objects; receiving user input indicating a user selected display object; extracting the user selected display object to generate a graphic extraction template, wherein the graphic extraction template has the same visual appearance of the user selected display object; applying mask on one or more regions within the graphic extraction template containing text information specific to the user selected display object; generating a link between each of the one or more regions and at least a portion of an object name of the user selected display object; extracting all the display objects that have the same type as the user selected display object using the graphic extraction template to generate a set of extracted graphic objects; and identifying an object name for each of the set of extracted graphic objects using the one or more links associated with the graphic extraction template.

Another embodiment relates to an apparatus to facilitate extraction of display objects for industrial diagram, the apparatus comprising: one or more computer-readable storage media; a processing system operatively coupled with the one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media that, when executed, are configured to cause the processing system to perform operations comprising: receiving an industrial diagram comprising a plurality of display objects; receiving user input indicating a user selected display object; extracting the user selected display object to generate a graphic extraction template, wherein the graphic extraction template has the same visual appearance of the user selected display object; applying mask on one or more regions within the graphic extraction template containing text information specific to the user selected display object; generating a link between each of the one or more regions and at least a portion of an object name of the user selected display object; extracting all the display objects that have the same type as the user selected display object using the graphic extraction template to generate a set of extracted graphic objects; and identifying an object name for each of the set of extracted graphic objects using the one or more links associated with the graphic extraction template.

DETAILED DESCRIPTION

A piping and instrumentation diagram (P&ID) provides a detailed graphical overview of an industrial automation environment, such as a manufacturing factory, processing plant, fabrication facility, and other types of industrial enterprises. Generally, a P&ID comprises a schematic drawing that illustrates the process flow and industrial equipment along with the instrumentation and control devices. A P&ID typically includes mechanical equipment such as pressure vessels, tanks, pumps, motors, compressors, drives, valves and other industrial assets, process pipelines and flow directions, process control and status monitoring instrumentation, and other objects employed in an industrial automation process.

Engineers often work to develop customized HMI displays to monitor and control specific industrial operations. Generally, when creating customized HMI displays, an engineer may refer to the P&ID to get an overview of the plant and see how the various industrial equipment and other objects are laid out and interconnected to one another. The engineer would then typically create the HMI displays according to the P&ID by manually placing and arranging objects linked to various devices and instruments within the HMI displays. Customized HMI displays can also be generated based on existing HMI displays with different format that is not compatible with the customized HMI displays. In this case, engineers have to refer to the existing HMI displays and manually place and arrange objects within the customized HMI displays. In some cases, the P&ID or existing HMI may include a large number of pages to display the whole diagram. It can take months or years of manual labor to create a customized HMI display. These manual process of creating customized HMI displays can be very time consuming and tedious.

The techniques described herein facilitate the creation of customized HMI displays by automatically extracting graphical objects from a P&ID or an existing HMI file that may be imported into an HMI application for inclusion on an HMI display. In at least one implementation, a software tool is provided for extracting static graphical objects from a P&ID or an HMI. The tool allows an engineer to define each type of object and then creates an extraction template for the defined type of object, extracts all the objects that match the extraction template and save the extracted objects in an HMI-compatible file format. Once the HMI graphic objects are generated, an engineer can import the HMI graphics into an HMI application for inclusion on an HMI display, thereby avoiding having to manually draw the object in the HMI application.

Referring now toFIG.1, an industrial diagram of industrial automation environment100is illustrated in an exemplary embodiment. The industrial diagram100may include any suitable industrial diagrams such as P&ID, HMI diagrams, etc. The industrial automation environment is provided as an example environment that may utilize any of the techniques disclosed herein, but note that the present disclosure could equally apply to any other industrial applications. In this example, industrial automation environment comprises a mineral processing facility. The industrial diagram100may include one or more pages according to some embodiments. Each page displays at least a portion of the industrial diagram100. For example, the industrial diagram100may include sixty pages to display the whole diagram. The industrial diagram100includes various components used in the mineral processing facility, such as valves102a, flow sensors104a, tanks106, pumps108a, etc. The industrial diagram100may also include diagram information section103indicating various information related to the industrial automation environment, such as industrial device operating parameters, project name, process name, date, project manager names, etc. In some embodiments, when the industrial diagram100includes more than one page, the industrial diagram100may include page link objects101(such as page link objects101a-101g). Each page link object indicates where the connected object is linked to in another page. In other words, the page link objects101show the connections in the whole diagram that are cut by pages.

The techniques described below with respect toFIGS.1-13could be executed by a computing system, such as computing system810, and could be combined with operation1400ofFIG.14in some implementations. In this example, some of the industrial objects appearing in the industrial diagram of industrial automation environment100is labeled with a tag component. These tag components provide a reference to help uniquely identify each of the objects. In this example, the valves102aare shown as having tag components102b, the flow sensor104ahas tag components104b, mixing tank106, and the pumps108ahas tag components108b.

In some implementations, the industrial diagram file could comprise an image file, a portable document format (PDF) file, or any other type of graphical file format. By loading the industrial diagram into a graphic extractor tool described herein, a user can select a type of graphical display objects and its corresponding tag components within the extractor tool, which may then be created as an extraction template. The extractor tool uses the extraction template to automatically identify and extract the selected type of graphical objects into an HMI-compatible file format which can then be imported into an HMI application for use in creating an HMI display.

With respect toFIGS.2-13, a graphic extractor tool200for extracting one or more types of objects from the industrial diagram100is shown according to an illustrative embodiment. The graphic extractor tool200is configured to extract any objects within an industrial diagram, such as various symbols representing industrial components, texts displaying on the industrial diagram, page link objects, etc. The extraction is provided by creating an extraction template for each type of objects according to user inputs. Each extraction template identifies the shape of a corresponding type of objects, the texts and/or labels associated with the corresponding type of objects, the orientation of the corresponding type of objects, the name structures of the corresponding type of objects, etc. The graphic extractor tool200may use the extraction template to search through the whole industrial diagram100to identify and extract all the objects that has the corresponding type according to some embodiments. The graphic extractor tool200may use the extraction templates to search through one or more user selected pages within the industrial diagram100to identify and extract all the objects, within the selected pages, that has the corresponding type according to some embodiments.

As shown inFIG.2, after a user opens an industrial diagram file within the extractor tool200, the industrial diagram is displayed in a diagram display window202. The diagram display window allows user to zoom in/out of a displayed diagram. In some embodiments, if the industrial diagram has multiple pages, the user can select a page for display. For example,FIG.2illustrates that the industrial diagram includes fifty-six pages, and the user selects the first page for display. In some example embodiments,FIGS.2-9show how the extractor tool200extracts all the valves that has the same type as the valve102a. The extractor tool200is configured to create an extraction template using the valve102aaccording to user inputs. The type of valves such as the valve102ais associated with tag components labeling each valve and indicating a name and/or other information of the valve (e.g., the tag component102b.) The extractor tool200allows the user to draw an outline surrounding the tag component102band reproduce the tag component102bas a primary graphic. The extractor tool200displays the reproduced tag component102cwithin a primary graphic window204. The extractor tool200may reproduce the tag component102busing any suitable image processing algorithms.

As shown inFIG.3, the graphic extractor tool200further allows the user to select one or more regions containing text information that is located around or within the tag component102b. For example, the text information HS and H03 within the tag component102b, and the text information AA01-TK001 around the tag component102b. The extractor tool200determines the location for each user selected text region and link it to the reproduced tag component102c. In some embodiments, the location for each user selected text region may be locations relative to the reproduced tag component102c. Each selected text region is saved and displayed as a text location in a linked text locations window206. For example, within linked text locations window206, a first text location (text location 0) is created and corresponds to the text information AA01-TK001, a second text location (text location 1) is created and corresponds to the text information H03, and a third text location (text location 2) is created and corresponds to the text information HS.

In addition, the extractor tool200applies a mask on each of the text location so that the reproduced tag component102cdoes not include the text information. After applying the mask on each text location, the reproduced tag component102cremains the outlines of the component alone and all the text information is removed from the reproduced tag component102c. For example, as displayed in the primary graphic window204, the text information HS, H03, and AA01-TK001 are removed (e.g., masked out) from the primary graphic. In this way, when the extractor tool200searches through the industrial diagram, the extractor tool200can find and extract all the valve tag components that have the same graphical structure as the primary extraction component102cregardless of what text information is associated with the tag component (e.g., different names). The outline of the reproduced tag component102cis saved as a primary graphic extraction template302. The primary graphic extraction template302, the text locations, and the masks are saved as part of the extraction template.

As shown inFIG.4, the extractor tool200allows the user to draw an outline surrounding the valve102aand reproduce the valve102aas a secondary graphic. The extractor tool200displays the reproduced valve102dwithin a secondary graphic window208. The extractor tool200may reproduce the valve102ausing any suitable image processing algorithms. The reproduced valve102dis saved a s secondary graphic extraction template. If there are one or more regions that contain text information associated with the secondary graphic extraction template, similarly as the primary graphic extraction template302, the graphic extractor tool200applies masks to the text information so that the secondary graphic extraction template contains only outlines of the object and does not contain any specific text information. The extractor tool200creates the extraction template to include the primary graphic extraction template and the secondary extraction template. In some embodiments, if an object is only associated with one graphical component, the extractor tool200creates the extraction template using only the primary graphic extraction template. For example, if there is a valve that does not have a tag component associated with, the valve will be reproduced as a primary graphic extraction template.

As shown inFIG.5, the extractor tool200further allows the user to link an object name of the extraction template (e.g., the valve102a) to the text information associated with the extraction template. The extractor tool200may receive a list of objects names indicating what objects are and their names within the industrial diagram100. The extractor tool200allows the user to select the object name that matches the text information through a dropdown list or through a search box. For example, the user selects the object name AA01_TK001_H03_HS that matches the text information around and within the tag component102b. The extractor tool200further allows the user to select each element of the object name and a corresponding matching element within the text information. For example, the user can select AA01502bwithin the object name and the AA01502awithin the text information of the tag component102b. The user can further use the link button510to link the two elements. Once the two elements502aand502bare linked, the extractor tool200identifies which text region associated with the object is according to which element of an object name. Similarly, the extractor tool200can link the elements504aand504b, elements506aand506b, elements508aand508b, respectively.

When the extractor tool200extracts an object using the extraction template, the extractor tool200can identify which object name corresponds to the extracted object automatically. In some embodiments, if the text information associated with the extraction template does not match any object name within the list, the user can create a name for the specific extraction template in a similar way as other available names. For example, in some embodiments, if the list of object name does not include the name AA01_TK001_H03_HS, the user may use the format of the available name AA01_TK001_H02_HS and the text information associated with the selected object to create a name for the selected object. In these embodiments, when the extractor tool200extracts another object that is not associated with an available name, the extractor tool200can create a name for the object in a way similarly as the user defined the name for the extraction template automatically.

Once the extraction template is created and linked with an object name, the extractor tool200uses the extraction template to search through one or more user defined pages within the industrial diagram automatically. In some embodiments, the extractor tool200receives user input through the RUN SCAN button512to start scan or search through the one or more user defined pages within the industrial diagram. As shown inFIG.6, the extractor tool200can identify all the objects602a-602fthat matches the primary graphic extraction template302. All the identified objects602a-602fare tag components. However, these tag components are not all for the specific type of valve as the valve102awhich is shown in the secondary extraction graphic template102d. For example, the identified tag component602band602dare associated with pumps108instead of valve102a.

As shown inFIG.7, the extractor tool200can also identify all the objects702a-702dthat matches the secondary extraction graphic template102d. All the identified objects702a-702dare valves that has the same type as the valve102a. In order to match each identified tag component with a corresponding identified valve component, the extractor tool200, for each identified valve component, determines a shortest path between the valve component and all the identified tag components. In this way, the extractor tool200can find each pair of matching tag component and valve component. For example, as shown inFIG.8, in order to find the corresponding tag component602afor the valve702a, the extractor tool200calculates a path between the valve702aand each of the identified tag components602a-602f. The extractor tool200further compares all the calculated paths and determines the shortest path. Once the shortest path between the valve component702aand the tag component602ais determined, the extractor tool200determines the valve702ais associated with the tag component602a. Similarly, for each identified valve, the extractor can determine the corresponding through the shortest path method. In this way, the tag components602band602dthat are not associated with any valve can be filtered out automatically. In some embodiments, the extractor tool200can use any suitable shortest path finding algorithms, such as Dijkstra's algorithm.

When the extractor tool200identified all the pairs of valves and corresponding tag components using the extraction template the extractor tool200can generate an extraction output file to include the extracted objects and all the relevant information associated with the extracted objects. As shown inFIG.9, the extraction output file may include, for each extracted valve, the tag name, description, object type, document name of the industrial diagram, the page on which the object is extracted, the HMI display page name, the relative positions of the extracted object within the page, whether the secondary linked graphic found, the secondary rotation, etc. In some embodiments, the extractor tool200may use this extraction output file to generate an HMI file for the extracted components.

In some example embodiments,FIGS.10-13illustrate how the extractor tool200extracts page link objects. As shown inFIG.10, the extractor tool200allows the user to select an object110ain the diagram information section that is associated with page link objects. In some embodiments, the user can select the object110aby drawing a box around the object110a. The extractor tool200reproduces the selected object110ain the primary graphic window and creates a primary page link extraction template110b. The extractor tool200further requires the user to identify regions that contains text information specific to this select object110a, such as the regions112a,114a, and116a. The extractor then applies masks on the user identified regions112c,114c, and116c. In this way, the extractor tool200can use the primary page link extraction template110bto identify all the objects within the diagram that has the similar outlines and similar general text information, such as “PLANT UNIT CODE”, “DOCUMENT TYPE”, and “COUNTING NO” as the selected object110a. The extractor tool200further allows the user to link a text location to each region112a,114a, and116a. For example, the region112athat displays the masked-out object AA01 is linked to the location 0112b, the region114athat displays the masked-out object PFB06 is linked to the location 1114b, and the region116athat displays the masked-out object70005is linked to the location 2116b. In this way, the extractor tool200can use the linked location and region to identify where to extract the specific text information.

As shown inFIG.11, the extractor tool200allows the user to select a page link object111a. The extractor tool200reproduces the page link object and displays the reproduced page link object111bin the secondary graphic window208. The extractor tool200further requests the user to identify the text regions containing specific text information associated with the page link object111a(e.g., the text regions113aand115a). The extractor tool200removes the text information from the text regions by applying masks on the text region. The extractor tool200then creates a secondary page link extraction template using the reproduces page link object111bwith masked-out text regions113band115b. The extractor tool200also allows the user to link the text regions113aand115ato text locations113cand115crespectively. The extractor tool200can create a page link extraction template using the primary and secondary page link extraction templates.

As shown inFIGS.12and13, the extractor tool200can use the page link extraction template to find all the page link objects. The extractor tool200further identifies the specific text information associated with each extracted page link object and determine which two or more page link objects share the same text information. In this way, the extractor tool can find the related page link objects and create a hyperlink to link the relate page link objects. The extractor tool200can display the hyperlinks within a page links window1200. For example, as shown inFIG.12, the extractor tool200identifies the page link objects1202a,1204a, and1206ain page2of the diagram and highlighted the identified objects1202a,1204a, and1206ain the diagram display window202. The extractor tool200also identifies one or more link objects that are related to each of the page link objects1202a,1204a, and1206a. The extractor tool200creates a hyperlink1202to link the page link object1202awith its related page link object1202bas shown inFIG.13. In some embodiments, the extractor tool200creates hyperlink icons within the page links window1200. These hyperlink icons also indicate the page number where the related page link objects are located. When the user clicks the hyperlink icon, the extractor tool200switches from displaying the current page in the diagram display window202to displaying the linked page in the diagram display window202. For example, as shown inFIG.13, when the user clicks hyperlink icon page23, the extractor tool200displays the page13of the diagram in the diagram display window202and highlighted the related page link object1202bin the diagram display window202. In this way, the user can easily navigate through linked pages within the diagram100.

FIG.14is a flow diagram that illustrates an operation1400of a computing system to facilitate extraction of graphic objects for industrial diagrams in an exemplary embodiment. The operation1400shown inFIG.14may also be referred to as a graphic object extraction process1400herein. The steps of the operation are indicated below parenthetically.

At operation1402, a user input indicating a user selected display object within an industrial diagram is received. The industrial diagram includes a plurality of display objects. In some embodiments, the industrial diagram may include one or more pages to display the whole diagram. Each page may display a portion of the diagram. When the industrial diagram includes multiple pages, these pages can be linked by one or more page link objects. In some embodiments, the user selected display object represents a type of objects that the user wants to extract from the industrial diagram. In some embodiments, the user selected display object is selected by the user drawing a box around the display object. an industrial diagram is received for processing extraction of one or more graphic objects.

At operation1404, the user selected display object is extracted to generate a graphic extraction template. The graphic extraction template is generated by reproducing the user selected display object including the shape (e.g., outlines of the object) of the object, any text information associated with the object, the positions of the object within the industrial diagram, and an orientation of the object. In other words, the graphic extraction template has the same visual appearance as the user selected display object.

At operation1406, mask is applied on one or more regions within the graphic extraction template containing text information specific to the user selected display object. In some embodiments, the one or more regions are selected by the user to apply the mask. When the user sees the text information is specific for the user selected display object instead of generic information for a type of display objects, the user can select the regions that contains this specific text information. These selected regions can be masked out (e.g., removed) from the graphic extraction template. In this way, the visual appearance of the masked-out graphic extraction template does not include any text information specific to the user selected object. The locations of the one or more regions that are applied mask are saved.

At operation1408, a link between each of the one or more regions and at least a portion of an object name of the user selected display object is generated. Each region that is masked out with specific text information is linked to a portion of the file so that the graphic extraction template may include the location information of each masked out region and its corresponding portion within an object name.

At operation1410, all the display objects that have the same type as the user selected display object are extracted using the graphic extraction template to generate a set of extracted graphic objects. The display objects that have the same type as the user selected display object when the display objects have the same visual appearance as the graphic extraction template. Since the graphic extraction template is already masked out of the specific text information, the graphic extraction template can be used to identify any display objects that have the same visual appearance as the template.

At operation1412, an object name is identified for each of the set of graphic objects using the one or more links associated with the graphic extraction template. Each of the set of extracted graphic objects corresponds to an object name. The object name can be retrieved from a list of object names associated with the industrial diagram. The object name can be identified by identifying text information within one or more regions that has the same location as the masked out regions of the graphic extraction template. The text information identified within each region can be compared to its corresponding portion within each object name. If each text information of an extracted graphic object matches its corresponding portion of an object name, the object name is identified as associated with the extracted graphic object. In some embodiments, if the text information of an extracted graphic object does not match any object name, an object name is generated using the text information within each region as its corresponding portion within the object name.

Turning now toFIG.15, a block diagram that illustrates an industrial automation environment800in an exemplary implementation is shown. Industrial automation environment800provides an example of an industrial automation environment that may be utilized to implement the predictive maintenance processes disclosed herein, but other environments could also be used. Industrial automation environment800includes computing system810, machine system820, industrial controller825, database system830, and application integration platform835. Computing system810provides an example of a computing system that may be used to execute the graphic object extraction process1400or variations thereof, although other possible computing systems could use alternative configurations. Machine system820and controller825are in communication over a communication link, controller825and database system830communicate over a communication link, database system830and application integration platform835communicate over a communication link, and application integration platform835and computing system810are in communication over a communication link. Note that there would typically be many more machine systems in most industrial automation environments, but the number of machine systems shown inFIG.8has been restricted for clarity.

Industrial automation environment800comprises an industrial mining operation, automobile manufacturing factory, food processing plant, oil drilling operation, microprocessor fabrication facility, or some other type of industrial enterprise. Machine system820could comprise a sensor, drive, pump, filter, drill, motor, robot, fabrication machinery, mill, printer, or any other industrial automation equipment, including their associated control systems. A control system comprises, for example, industrial controller825, which could include automation controllers, programmable logic controllers (PLCs), programmable automation controllers (PACs), or any other controllers used in automation control. Additionally, machine system820could comprise other industrial equipment, such as a brew kettle in a brewery, a reserve of coal or other resources, or any other element that may reside in an industrial automation environment800.

Machine system820continually produces operational data over time. The operational data indicates the current status of machine system820, such as parameters, pressure, temperature, speed, energy usage, operational equipment effectiveness (OEE), mean time between failure (MTBF), mean time to repair (MTTR), voltage, throughput volumes, times, tank levels, or any other performance status metrics. The operational data may comprise dynamic charts or trends, real-time video, or some other graphical content. Machine system820and/or controller825is capable of transferring the operational data over a communication link to database system830, application integration platform835, and computing system810, typically via a communication network. Database system830could comprise a disk, tape, integrated circuit, server, or some other memory device. Database system830may reside in a single device or may be distributed among multiple memory devices.

Application integration platform835comprises a processing system and a communication transceiver. Application integration platform835may also include other components such as a router, server, data storage system, and power supply. Application integration platform835may reside in a single device or may be distributed across multiple devices. Application integration platform835may be a discrete system or may be integrated within other systems, including other systems within industrial automation environment800. In some examples, application integration platform835could comprise a FactoryTalk® VantagePoint server system provided by Rockwell Automation, Inc.

The communication links over which data is exchanged between machine system820, industrial controller825, database system830, application integration platform835, and communication interface808of computing system810could use metal, air, space, optical fiber such as glass or plastic, or some other material as the transport medium, including combinations thereof. The communication links could comprise multiple network elements such as routers, gateways, telecommunication switches, servers, processing systems, or other communication equipment and systems for providing communication and data services. These communication links could use various communication protocols, such as TDM, IP, Ethernet, telephony, optical networking, packet networks, cellular networks, wireless mesh networks (WMN), local area networks (LAN), metropolitan area networks (MAN), wide area networks (WAN), hybrid fiber coax (HFC), communication signaling, wireless protocols, communication signaling, peer-to-peer networking over Bluetooth, Bluetooth low energy, Wi-Fi Direct, near field communication (NFC), or some other communication format, including combinations thereof. The communication links could be direct links or may include intermediate networks, systems, or devices.

Computing system810may be representative of any computing apparatus, system, or systems on which the graphic object extraction process1400disclosed herein or variations thereof may be suitably implemented. In some examples, computing system810could execute an HMI application for developing HMI displays, such as a FactoryTalk® View Studio application provided by Rockwell Automation, Inc. Computing system810provides an example of a computing system that could be used as a either a server or a client device in some implementations, although such devices could have alternative configurations. Examples of computing system810include mobile computing devices, such as cell phones, tablet computers, laptop computers, notebook computers, and gaming devices, as well as any other type of mobile computing devices and any combination or variation thereof. Examples of computing system810also include desktop computers, server computers, and virtual machines, as well as any other type of computing system, variation, or combination thereof. In some implementations, computing system810could comprise a mobile device capable of operating in a server-like fashion which, among other uses, could be utilized in a wireless mesh network.

Computing system810includes processing system801, storage system803, software805, communication interface808, and user interface809. Processing system801is operatively coupled with storage system803, communication interface808, and user interface809. Processing system801loads and executes software805from storage system803. Software805includes application806and operating system807. Application806may include the graphic object extraction process1400in some examples. When executed by computing system810in general, and processing system801in particular, software805directs computing system810to operate as described herein for the graphic object extraction process1400or variations thereof. In this example, user interface809includes display system811, which itself may be part of a touch screen that also accepts user inputs via touches on its surface. Computing system810may optionally include additional devices, features, or functionality not discussed here for purposes of brevity.

Turning now toFIG.16, a block diagram is shown that illustrates computing system900in an exemplary implementation. Computing system900provides an example of any computing system that may be used to execute the graphic object extraction process1400or variations thereof, although other possible computing systems could use alternative configurations. Computing system900includes processing system901, storage system903, software905, communication interface907, and user interface909. User interface909comprises display system908. Software905includes application906which itself includes the graphic object extraction process1400. The graphic object extraction process1400may optionally be implemented separately from application906, as indicated by the dashed line inFIG.9.

Computing system900may be representative of any computing apparatus, system, or systems on which application906and the graphic object extraction process1400or variations thereof may be suitably implemented. Examples of computing system900include mobile computing devices, such as cell phones, tablet computers, laptop computers, notebook computers, and gaming devices, as well as any other type of mobile computing devices and any combination or variation thereof. Note that the features and functionality of computing system900may apply as well to desktop computers, server computers, and virtual machines, as well as any other type of computing system, variation, or combination thereof.

Computing system900includes processing system901, storage system903, software905, communication interface907, and user interface909. Processing system901is operatively coupled with storage system903, communication interface907, and user interface909. Processing system901loads and executes software905from storage system903. When executed by computing system900in general, and processing system901in particular, software905directs computing system900to operate as described herein for the graphic object extraction process1400or variations thereof. Computing system900may optionally include additional devices, features, or functionality not discussed herein for purposes of brevity.

Referring still toFIG.9, processing system901may comprise a microprocessor and other circuitry that retrieves and executes software905from storage system903. Processing system901may be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing system901include general purpose central processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

Storage system903may comprise any computer-readable storage media capable of storing software905and readable by processing system901. Storage system903may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Storage system903may be implemented as a single storage device but may also be implemented across multiple storage devices or sub-systems co-located or distributed relative to each other. Storage system903may comprise additional elements, such as a controller, capable of communicating with processing system901. Examples of storage media include random-access memory, read-only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and that may be accessed by an instruction execution system, as well as any combination or variation thereof, or any other type of storage media. In no case is the computer-readable storage media a propagated signal.

In operation, in conjunction with user interface909, processing system901may load and execute portions of software905, such as the graphic object extraction process1400, to render a graphical user interface for application906for display by display system908of user interface909. Software905may be implemented in program instructions and among other functions may, when executed by computing system900in general or processing system901in particular, direct computing system900or processing system901to receive a selection of a user-defined area that identifies at least a portion of a piping and instrumentation diagram (P&ID) associated with an industrial automation environment. Software905may further direct computing system900or processing system901to analyze the user-defined area of the P&ID to identify at least one object within the user-defined area of the P&ID. In addition, software905directs computing system900or processing system901to extract the at least one object identified within the user-defined area of the P&ID to generate a static graphic object. Software905may further direct computing system900or processing system901to receive edit instructions that describe at least one modification to a visual appearance of the static graphic object. Software905may direct computing system900or processing system901to apply the edit instructions to the static graphic object to generate an HMI graphic object comprising the at least one modification to the visual appearance of the static graphic object.

Software905may include additional processes, programs, or components, such as operating system software or other application software. Examples of operating systems include Windows®, iOS®, and Android®, as well as any other suitable operating system. Software905may also comprise firmware or some other form of machine-readable processing instructions executable by processing system901.

In general, software905may, when loaded into processing system901and executed, transform computing system900overall from a general-purpose computing system into a special-purpose computing system customized to facilitate extraction of display objects for HMI displays as described herein for each implementation. For example, encoding software905on storage system903may transform the physical structure of storage system903. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to the technology used to implement the storage media of storage system903and whether the computer-storage media are characterized as primary or secondary storage.

In some examples, if the computer-readable storage media are implemented as semiconductor-based memory, software905may transform the physical state of the semiconductor memory when the program is encoded therein. For example, software905may transform the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. A similar transformation may occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate this discussion.

It should be understood that computing system900is generally intended to represent a computing system with which software905is deployed and executed in order to implement application906and/or the graphic object extraction process1400(and variations thereof). However, computing system900may also represent any computing system on which software905may be staged and from where software905may be distributed, transported, downloaded, or otherwise provided to yet another computing system for deployment and execution, or yet additional distribution. For example, computing system900could be configured to deploy software905over the internet to one or more client computing systems for execution thereon, such as in a cloud-based deployment scenario.

Communication interface907may include communication connections and devices that allow for communication between computing system900and other computing systems (not shown) or services, over a communication network911or collection of networks. In some implementations, communication interface907receives dynamic data921over communication network911. Examples of connections and devices that together allow for inter-system communication may include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The aforementioned network, connections, and devices are well known and need not be discussed at length here.

User interface909may include a voice input device, a touch input device for receiving a gesture from a user, a motion input device for detecting non-touch gestures and other motions by a user, and other comparable input devices and associated processing elements capable of receiving user input from a user. Output devices such as display system908, speakers, haptic devices, and other types of output devices may also be included in user interface909. The aforementioned user input devices are well known in the art and need not be discussed at length here. User interface909may also include associated user interface software executable by processing system901in support of the various user input and output devices discussed above. Separately or in conjunction with each other and other hardware and software elements, the user interface software and devices may provide a graphical user interface, a natural user interface, or any other kind of user interface. User interface909may be omitted in some examples.

The above description and associated drawings teach the best mode of the invention. The following claims specify the scope of the invention. Some aspects of the best mode may not fall within the scope of the invention as specified by the claims. Also, while the preceding discussion describes embodiments employed specifically in conjunction with the monitoring and analysis of industrial processes, other applications, such as the mathematical modeling or monitoring of any man-made or naturally-existing system, may benefit from use of the concepts discussed above. Further, those skilled in the art will appreciate that the features described above can be combined in various ways to form multiple variations of the invention. As a result, the invention is not limited to the specific embodiments described above, but only by the following claims and their equivalents.