Patent Description:
Industrial automation systems are managed and operated using automation control and monitoring systems (e.g., industrial control system), particularly in industrial automation environments. Such applications may include controlling a wide range of components, such as valves, electric motors, and so forth, and the collection of data via sensors. Typical industrial control systems may include one or more components, such as programming terminals, automation controllers, input/output (I/O) modules, communication networks, human-machine interface (HMI) terminals, and the like.

Generally, industrial control systems operate in the operational technology (OT) environment are used to control industrial devices accessible via an OT network. Although the industrial control systems may be used to manage the operations of the devices within the OT network, there may be additional processes that can be performed with the devices, data generated by the devices that is unused or could be used for more purposes, and there may be devices that are not known to computing devices in the information technology (IT) environment.

<CIT> relates to a method of arranging a cluster of printers that are able to run a customized container image based on the requirements of the applications in the container and the characteristics of the printers in the network. The method includes receiving a container image of software configured to be executed on printers. The method also includes determining and receiving a response to a query for printer characteristics using the server. Additionally, the method includes determining image-eligible micro-services that are configured to execute on a set of selected image-eligible printers selected based on a comparison of the response information with the container-eligible characteristics. Moreover, the method includes determining that the container image is configured to execute image-eligible micro-services. The method further includes, sending the container image to the image-eligible printers and executing the printer management application on the image-eligible printers. <CIT> relates to a method which includes receiving, via a first computing node, a first pod from a second computing node. The method also includes retrieving a first image file that includes a first set of containers from a registry based on the first pod. The first set of containers may cause a control system to halt operations. The method then involves generating a first package based on the first set of containers and storing the first package in a filesystem, receiving a second pod from the second computing node, and retrieving a second image file having a second set of containers from the registry. The second pod includes the second set of containers causes the control system to update software components. The method also involves generating a second package based on the second set of containers and storing the second package in the filesystem.

It is therefore the object of the present invention to provide enhanced employment of software container in a system with a plurality of devices.

In one embodiment, a non-transitory computer-readable medium includes instructions that, when executed by processing circuitry, cause the processing circuitry to receive, from an edge device of an industrial automation system, data indicative of a type of one or more devices in the industrial automation system, one or more types of data generated by the one or more devices, one or more components included in the one or more devices, firmware of the one or more devices, or any combination thereof. When executed, the instructions also cause the processing circuitry to identify, based on the received data, either a firmware update for the industrial automation system or, from a container repository, a container that is implementable on the edge device. Additionally, when executed, the instructions cause the processing circuitry to cause a container image for the container or firmware update for the firmware to be sent to the edge device.

In another embodiment, a computing device includes processing circuitry and a non-transitory computer-readable medium having instructions that, when executed by the processing circuitry, cause the processing circuitry to receive, from an edge device of an industrial automation system, data indicative of a type of one or more devices in the industrial automation system, one or more types of data generated by the one or more devices, one or more components included in the one or more devices, firmware of the one or more devices, or any combination thereof. When executed, the instructions also cause the processing circuitry to determine, based on the received data, a firmware update for the industrial automation system or a container implementable on the edge device from a container repository. Additionally, when executed, the instructions cause the processing circuitry to cause a container image for the container or firmware update for the firmware to be sent to the edge device.

In yet another embodiment, an industrial automation system includes one or more industrial automation devices configured to perform a first industrial process or a portion thereof. The industrial automation system also includes an edge device that is communicatively coupled the one or more industrial automation devices. The edge device includes processing circuitry that is configured to receive, from a cloud computing system, a request for data indicative of a type of one or more industrial automation devices in the industrial automation system, one or more types of data generated by the one or more industrial automation devices, one or more components included in the one or more industrial automation devices, firmware of the one or more industrial automation devices, or any combination thereof. The processing circuitry is also configured to send the data to the cloud computing system in response to receiving the request. Furthermore, the processing circuitry is configured to receive a container image or a firmware update from the cloud computing system after sending the data to the cloud computing system.

The present application is generally directed to techniques that may be implemented in industrial automation systems that employ (software) containers, which are packages of software that may visualize an operating system and be deployed and run on one or more computing devices. As discussed herein, a container agent may receive data regarding devices in an industrial automation system, components of the devices, software or firmware installed on the devices, networks employed in the industrial automation system, and characteristics of the networks. The container agent may, based on the received data, determine one or more solutions that, when implemented by one or more devices of the industrial automation system, may add functionality to the industrial automation system, enable data to be analyzed in a manner not previously used, or enable the industrial automation system to harness data in a different manner. As such, the container agent may determine particular container images or firmware updates and cause the container images (or data for the container images), the firmware updates, or both to be sent to an edge device of the industrial automation system. For example, as discussed below, the container agent may determine that containers corresponding to preventative maintenance applications, power consumption applications, or manufacturing execution system applications may be implemented in an industrial automation system based on the types of devices in the industrial automation system, the components of the devices in the industrial automation system, types of data generated by the devices of the industrial automation system, firmware on the one or more industrial device, or a combination thereof.

By way of introduction, <FIG> is a perspective view of an example industrial automation system <NUM> controlled by one or more industrial control systems <NUM>. The industrial automation system <NUM> includes stations <NUM> having machine components and/or machines to conduct functions within an automated process, such as silicon wafer manufacturing, as is depicted. The automated process may begin at a station 14A used for loading objects, such as substrates, into the industrial automation system <NUM> via a conveyor section <NUM>. The conveyor section <NUM> may transport the objects to a station 14B to perform a first action, such a printing solder paste to the substrate via stenciling. As objects exit from the station 14B, the conveyor section <NUM> may transport the objects to a station 14C for solder paste inspection (SPI) to inspect printer results, to station 14D, station 14E, and station 14F for surface mount technology (SMT) component placement, to a station <NUM> for convection reflow oven to melt the solder to make electrical couplings, and finally to a station <NUM> for automated optical inspection (AOI) to inspect the object manufactured (e.g., the manufactured printed circuit board). After the objects proceed through the various stations, the objects may be removed from the station <NUM>, for example, for storage in a warehouse or for shipment. Clearly, for other applications, the particular system, machine components, machines, stations, and/or conveyors may be different or specially adapted to the application.

For example, the industrial automation system <NUM> may include machinery to perform various operations in a compressor station, an oil refinery, a batch operation for making food items, chemical processing operations, brewery operations, mining operations, a mechanized assembly line, and so forth. Accordingly, the industrial automation system <NUM> may include a variety of operational components, such as electric motors, valves, actuators, temperature elements, pressure sensors, or a myriad of machinery or devices used for manufacturing, processing, material handling, and other applications. The industrial automation system <NUM> may also include electrical equipment, hydraulic equipment, compressed air equipment, steam equipment, mechanical tools, protective equipment, refrigeration equipment, power lines, hydraulic lines, steam lines, and the like. Some example types of equipment may include mixers, machine conveyors, tanks, skids, specialized original equipment manufacturer machines, and the like. In addition to the equipment described above, the industrial automation system <NUM> may also include motors, protection devices, switchgear, compressors, and the like. Each of these described operational components may correspond to and/or generate a variety of operational technology (OT) data regarding operation, status, sensor data, operational modes, alarm conditions, or the like, that may be desirable to output for analysis with IT data from an IT network, for storage in an IT network, for analysis with expected operation set points (e.g., thresholds), or the like.

In certain embodiments, one or more properties of the industrial automation system <NUM> equipment, such as the stations <NUM>, may be monitored and controlled by the industrial control systems <NUM> for regulating control variables. For example, sensing devices (e.g., sensors <NUM>) may monitor various properties of the industrial automation system <NUM> and may be used by the industrial control systems <NUM> at least in part in adjusting operations of the industrial automation system <NUM> (e.g., as part of a control loop). In some cases, the industrial automation system <NUM> may be associated with devices used by other equipment. For instance, scanners, gauges, valves, flow meters, and the like may be disposed on or within the industrial automation system <NUM>. Here, the industrial control systems <NUM> may receive data from the associated devices and use the data to perform their respective operations more efficiently. For example, a controller of the industrial automation system <NUM> associated with a motor drive may receive data regarding a temperature of a connected motor and may adjust operations of the motor drive based on the data.

The industrial control systems <NUM> may be communicatively coupled to a display/operator interface <NUM> (e.g., a human-machine interface (HMI)) and to devices of the industrial automation system <NUM>. It should be understood that any suitable number of industrial control systems <NUM> may be used in a particular industrial automation system <NUM> embodiment. The industrial control systems <NUM> may facilitate representing components of the industrial automation system <NUM> through programming objects that may be instantiated and executed to provide simulated functionality similar or identical to the actual components, as well as visualization of the components, or both, on the display/operator interface <NUM>. The programming objects may include code and/or instructions stored in the industrial control systems <NUM> and executed by processing circuitry of the industrial control systems <NUM>. The processing circuitry may communicate with memory circuitry to permit the storage of the component visualizations.

As illustrated, a display/operator interface <NUM> depicts representations <NUM> of the components of the industrial automation system <NUM>. The industrial control system <NUM> may use data transmitted by sensors <NUM> to update visualizations of the components via changing one or more statuses, states, and/or indications of current operations of the components. These sensors <NUM> may be any suitable device adapted to provide information regarding process conditions. Indeed, the sensors <NUM> may be used in a process loop (e.g., control loop) that may be monitored and controlled by the industrial control system <NUM>. As such, a process loop may be activated based on process inputs (e.g., an input from the sensor <NUM>) or direct input from a person via the display/operator interface <NUM>. The person operating and/or monitoring the industrial automation system <NUM> may reference the display/operator interface <NUM> to determine various statuses, states, and/or current operations of the industrial automation system <NUM> and/or for a particular component. Furthermore, the person operating and/or monitoring the industrial automation system <NUM> may adjust to various components to start, stop, power-down, power-on, or otherwise adjust an operation of one or more components of the industrial automation system <NUM> through interactions with control panels or various input devices.

The industrial automation system <NUM> may be considered a data-rich environment with several processes and operations that each respectively generate a variety of data. For example, the industrial automation system <NUM> may be associated with material data (e.g., data corresponding to substrate or raw material properties or characteristics), parametric data (e.g., data corresponding to machine and/or station performance, such as during operation of the industrial automation system <NUM>), test results data (e.g., data corresponding to various quality control tests performed on a final or intermediate product of the industrial automation system <NUM>), or the like, that may be organized and sorted as OT data. In addition, sensors <NUM> may gather OT data indicative of one or more operations of the industrial automation system <NUM> or the industrial control system <NUM>. In this way, the OT data may be analog data or digital data indicative of measurements, statuses, alarms, or the like associated with operation of the industrial automation system <NUM> or the industrial control system <NUM>.

The industrial control systems <NUM> described above may operate in an OT space in which OT data is used to monitor and control OT assets, such as the equipment illustrated in the stations <NUM> of the industrial automation system <NUM> or other industrial equipment. The OT space, environment, or network generally includes direct monitoring and control operations that are coordinated by the industrial control system <NUM> and a corresponding OT asset. For example, a programmable logic controller (PLC) may operate in the OT network to control operations of an OT asset (e.g., drive, motor). The industrial control systems <NUM> may be specifically programmed or configured to communicate directly with the respective OT assets.

A container orchestration system <NUM>, on the other hand, may operate in an information technology (IT) environment. That is, the container orchestration system <NUM> may include a cluster of multiple computing devices that coordinates an automatic process of managing or scheduling work of individual containers for applications within the computing devices of the cluster. In other words, the container orchestration system <NUM> may be used to automate various tasks at scale across multiple computing devices. By way of example, the container orchestration system <NUM> may automate tasks such as configuring and scheduling of containers, provisioning and deployments of containers, determining availability of containers, configuring applications in terms of the containers that they run in, scaling of containers to equally balance application workloads across an infrastructure, allocating resources between containers, performing load balancing, traffic routing and service discovery of containers, performing health monitoring of containers, securing the interactions between containers, and the like. In any case, the container orchestration system <NUM> may use configuration files to determine a network protocol to facilitate communication between containers, a storage location to save logs, and the like. The container orchestration system <NUM> may also schedule deployment of containers into clusters and identify a host (e.g., node) that may be best suited for executing the container. After the host is identified, the container orchestration system <NUM> may manage the lifecycle of the container based on predetermined specifications. In some embodiments, the container orchestration system <NUM> may be implemented using one or more cloud-based computing devices. Accordingly, the container orchestration system <NUM> may be partially or wholly implemented using cloud-computing resources.

With the foregoing in mind, it should be noted that containers refer to technology for packaging an application along with its runtime dependencies. That is, containers include applications that are decoupled from an underlying host infrastructure (e.g., operating system). By including the run time dependencies with the container, the container may perform in the same manner regardless of the host in which it is operating. In some embodiments, containers may be stored in a container registry <NUM> as container images <NUM>. The container registry <NUM> may be any suitable data storage or database that may be accessible to the container orchestration system <NUM>. The container image <NUM> may correspond to an executable software package that includes the tools and data employed to execute a respective application. That is, the container image <NUM> may include related code for operating the application, application libraries, system libraries, runtime tools, default values for various settings, and the like.

By way of example, an integrated development environment (IDE) tool may be employed by a user to create a deployment configuration file that specifies a desired state for the collection of nodes of the container orchestration system <NUM>. The deployment configuration file may be stored in the container registry <NUM> along with the respective container images <NUM> associated with the deployment configuration file. The deployment configuration file may include a list of different pods and a number of replicas for each pod that should be operating within the container orchestration system <NUM> at any given time. Each pod may correspond to a logical unit of an application, which may be associated with one or more containers. The container orchestration system <NUM> may coordinate the distribution and execution of the pods listed in the deployment configuration file, such that the desired state is continuously met. In some embodiments, the container orchestration system <NUM> may include a master node that retrieves the deployment configuration files from the container registry <NUM>, schedules the deployment of pods to the connected nodes, and ensures that the desired state specified in the deployment configuration file is met. For instance, if a pod stops operating on one node, the master node may receive a notification from the respective worker node that is no longer executing the pod and deploy the pod to another worker node to ensure that the desired state is present across the cluster of nodes.

As mentioned above, the container orchestration system <NUM> may include a cluster of computing devices, computing systems, or container nodes that may work together to achieve certain specifications or states, as designated in the respective container. In some embodiments, container nodes <NUM> may be integrated within industrial control systems <NUM> as shown in <FIG>. That is, container nodes <NUM> may be implemented by the industrial control systems <NUM>, such that they appear as worker nodes to the master node in the container orchestration system <NUM>. In this way, the master node of the container orchestration system <NUM> may send commands to the container nodes <NUM> that are also configured to perform applications and operations for the respective industrial equipment.

With this in mind, the container nodes <NUM> may be integrated with the industrial control systems <NUM>, such that they serve as passive-indirect participants, passive-direct participants, or active participants of the container orchestration system <NUM>. As passive-indirect participants, the container nodes <NUM> may respond to a subset of all of the commands that may be issued by the container orchestration system <NUM>. In this way, the container nodes <NUM> may support limited container lifecycle features, such as receiving pods, executing the pods, updating a respective file system to included software packages for execution by the industrial control system <NUM>, and reporting the status of the pods to the master node of the container orchestration system <NUM>. The limited features implementable by the container nodes <NUM> that operate in the passive-indirect mode may be limited to commands that the respective industrial control system <NUM> may implement using native commands that map directly to the commands received by the master node of the container orchestration system <NUM>. Moreover, the container node <NUM> operating in the passive-indirect mode of operation may not be capable to push the packages or directly control the operation of the industrial control system <NUM> to execute the package. Instead, the industrial control system <NUM> may periodically check the file system of the container node <NUM> and retrieve the new package at that time for execution.

As passive-direct participants, the container nodes <NUM> may operate as a node that is part of the cluster of nodes for the container orchestration system <NUM>. As such, the container node <NUM> may support the full container lifecycle features. That is, container node <NUM> operating in the passive-direct mode may unpack a container image and push the resultant package to the industrial control system <NUM>, such that the industrial control system <NUM> executes the package in response to receiving it from the container node <NUM>. As such, the container orchestration system <NUM> may have access to a worker node that may directly implement commands received from the master node onto the industrial control system <NUM>.

In the active participant mode, the container node <NUM> may include a computing module or system that hosts an operating system (e.g., Linux) that may continuously operate a container host daemon that may participate in the management of container operations. As such, the active participant container node <NUM> may perform any operations that the master node of the container orchestration system <NUM> may perform. By including a container node <NUM> operating in the OT space, the container orchestration system <NUM> is capable of extending its management operations into the OT space. That is, the container node <NUM> may provision devices in the OT space, serve as a proxy node <NUM> to provide bidirectional coordination between the IT space and the OT space, and the like. For instance, the container node <NUM> operating as the proxy node <NUM> may intercept orchestration commands and cause industrial control system <NUM> to implement appropriate machine control routines based on the commands. The industrial control system <NUM> may confirm the machine state to the proxy node <NUM>, which may then reply to the master node of the container orchestration system <NUM> on behalf of the industrial control system <NUM>.

Additionally, the industrial control system <NUM> may share an OT device tree via the proxy node <NUM>. As such, the proxy node <NUM> may provide the master node with state data, address data, descriptive metadata, versioning data, certificate data, key information, and other relevant parameters concerning the industrial control system <NUM>. Moreover, the proxy node <NUM> may issue requests targeted to other industrial control systems <NUM> to control other OT devices. For instance, the proxy node <NUM> may translate and forward commands to a target OT device using one or more OT communication protocols, may translate and receive replies from the OT devices, and the like. As such, the proxy node <NUM> may perform health checks, provide configuration updates, send firmware patches, execute key refreshes, and other OT operations for other OT devices.

With the foregoing in mind, <FIG> is a block diagram of an example industrial control system <NUM> that may be used with the embodiments described herein. The industrial control system <NUM> may include a communication component <NUM>, a processor <NUM>, a memory <NUM>, a storage <NUM>, input/output (I/O) ports <NUM>, a display <NUM>, and the like. The communication component <NUM> may be a wireless or wired communication component that facilitates communication between the container orchestration system <NUM> and the industrial control system <NUM>, or any other suitable electronic device. The processor <NUM> may be any type of computer processor or microprocessor capable of executing computer- executable code. The processor <NUM> may also include multiple processors that may perform the operations described below.

The memory <NUM> and the storage <NUM> may be any suitable article of manufacture that may serve as media to store processor-executable code, data, or the like. These articles of manufacture may represent computer-readable media (i.e., any suitable form of memory or storage) that may store the processor-executable code used by the processor <NUM> to perform the presently disclosed techniques. The memory <NUM> and the storage <NUM> may represent non-transitory computer-readable media (e.g., any suitable form of memory or storage) that may store the processor-executable code used by the processor <NUM> to perform various techniques described herein. It should be noted that non-transitory merely indicates that the media is tangible and not a signal.

The I/O ports <NUM> may couple to one or more sensors <NUM>, one or more input devices, one or more displays, or the like to facilitate human or machine interaction with the industrial control system <NUM>. For example, based on a notification provided to a user via a display <NUM>, the user may use an input device to instruct the adjustment of an OT device.

The display <NUM>, as discussed above, may operate to depict visualizations associated with software or executable code being processed by the processor <NUM>. In one embodiment, the display <NUM> may be a touch display capable of receiving inputs from a user of the industrial control system <NUM>. The display <NUM> may be any suitable type of display, such as a liquid crystal display (LCD), plasma display, or an organic light emitting diode (OLED) display, for example. Additionally, in one embodiment, the display <NUM> may be provided in conjunction with a touch- sensitive mechanism (e.g., a touch screen) that may function as part of a control interface for the industrial control system <NUM>.

Although <FIG> is depicted with respect to the industrial control system <NUM>, it should be noted that the container orchestration system <NUM>, the container nodes <NUM>, and the proxy node <NUM> may also include the same or similar components to perform, or facilitate performing, the various techniques described herein. Moreover, it should be understood that the components described with respect to <FIG> are exemplary figures and the industrial control system <NUM> and other suitable computing systems may include additional or fewer components as detailed above.

With the foregoing in mind, <FIG> illustrates a block diagram that depicts the relative positions of the container node <NUM> and the proxy node <NUM> with respect to the container orchestration system <NUM>. As mentioned above, the container orchestration system <NUM> may include a collection of nodes that are used to achieve a desired state of one or more containers across multiple nodes. As shown in <FIG>, the container orchestration system <NUM> may include a master node <NUM> that may execute control plane processes for the container orchestration system <NUM>. The control plane processes may include the processes that enable the container orchestration system <NUM> to coordinate operations of the container nodes <NUM> to meet the desired states. As such, the master container node <NUM> may execute an application programming interface (API) for the container orchestration system <NUM>, a scheduler component, core resources controllers, and the like. By way of example, the master container node <NUM> may coordinate all of the interactions between nodes of the cluster that make up the container orchestration system <NUM>. Indeed, the master container node <NUM> may be responsible for deciding the operations that will run on container nodes <NUM> including scheduling workloads (e.g., containerized applications), managing the workloads' lifecycle, scaling, and upgrades, managing network and storage resources for the workloads, and the like. The master container node <NUM> may run an API server to handle requests and status updates received from the container nodes <NUM>.

By way of operation, an integrated development environment (IDE) tool <NUM> may be used by an operator to develop a deployment configuration file <NUM>. As mentioned above, the deployment configuration file <NUM> may include details regarding the containers, the pods, constraints for operating the containers/pods, and other information that describe a desired state of the containers specified in the deployment configuration file <NUM>. In some embodiments, the deployment configuration file <NUM> may be generated in a YAML file, a JSON file, or other suitable file format that is compatible with the container orchestration system <NUM>. After the IDE tool <NUM> generates the deployment configuration file <NUM>, the IDE tool <NUM> may transmit the deployment configuration file <NUM> to the container registry <NUM>, which may store the file along with container images <NUM> representative of the containers stored in the deployment configuration file <NUM>.

In some embodiments, the master container node <NUM> may receive the deployment configuration file <NUM> via the container registry <NUM>, directly from the IDE tool <NUM>, or the like. The master container node <NUM> may use the deployment configuration file <NUM> to determine a location to gather the container images <NUM>, determine communication protocols to use to establish networking between container nodes <NUM>, determine locations for mounting storage volumes, locations to store logs for the containers, and the like.

Based on the desired state provided in the deployment configuration file <NUM>, the master container node <NUM> may deploy containers to the container host nodes <NUM>. That is, the master container node <NUM> may schedule the deployment of a container based on constraints (e.g., CPU or memory availability) provided in the deployment configuration file <NUM>. After the containers are operating on the container nodes <NUM>, the master container node <NUM> may manage the lifecycle of the containers to ensure that the containers specified by the deployment configuration file <NUM> is operating according to the specified constraints and the desired state.

Keeping the foregoing in mind, the industrial control system <NUM> may not use an operating system (OS) that is compatible with the container orchestration system <NUM>. That is, the container orchestration system <NUM> may be configured to operate in the IT space that involves the flow of digital information. In contrast, the industrial control system <NUM> may operate in the OT space that involves managing the operation of physical processes and the machinery used to perform those processes. For example, the OT space may involve communications that are formatted according to OT communication protocols, such as FactoryTalk Live Data, EtherNet/IP. Common Industrial Protocol (CIP), OPC Direct Access (e.g., machine to machine communication protocol for industrial automation developed by the OPC Foundation), or any suitable OT communication protocol (e.g. DNP3, Modbus, Profibus, LonWorks, DALI, BACnet, KNX, EnOcean). Since the industrial control systems <NUM> operate in the OT space, the industrial control systems are not capable of implementing commands received via the container orchestration system <NUM>.

In certain embodiments, the container node <NUM> may be programmed or implemented in the industrial control system <NUM> to serve as a node agent that can register the industrial control system <NUM> with the master container node <NUM>. For example, the industrial control system <NUM> may include a programmable logic controller (PLC) that cannot support an operating system (e.g., Linux) for receiving and/or implementing requested operations issued by the container orchestration system <NUM>. However, the PLC may perform certain operations that may be mapped to certain container events. As such, the container node <NUM> may include software and/or hardware components that may map certain events or commands received from the master container node <NUM> into actions that may be performed by the PLC. After converting the received command into a command interpretable by the PLC, the container node <NUM> may forward the mapped command to the PLC that may implement the mapped command. As such, the container node <NUM> may operate as part of the cluster of nodes that make up the container orchestration system <NUM>, while a control system <NUM> (e.g., PLC) that coordinates the OT operations for an OT device <NUM> in the industrial control system <NUM>. The control system <NUM> may include a controller, such as a programmable logic controller (PLC), a programmable automation controller (PAC), or any other controller that may monitor, control, and operate an industrial automation device or component.

The industrial automation device or component may correspond to an OT device <NUM>. The OT device <NUM> may include any suitable industrial device that operates in the OT space. As such, the OT device <NUM> may be involved in adjusting physical processes being implemented via the industrial system <NUM>. In some embodiments, the OT device <NUM> may include motor control centers, motors, human machine interfaces (HMIs), operator interfaces, contactors, starters, sensors, drives, relays, protection devices, switchgear, compressors, network switches (e.g., Ethernet switches, modular-managed, fixed-managed, service-router, industrial, unmanaged, etc.) and the like. In addition, the OT device <NUM> may also be related to various industrial equipment such as mixers, machine conveyors, tanks, skids, specialized original equipment manufacturer machines, and the like. The OT device <NUM> may also be associated with devices used by the equipment such as scanners, gauges, valves, flow meters, and the like. In one embodiment, every aspect of the OT device <NUM> may be controlled or operated by the control system <NUM>.

As such, the control system <NUM> may perform actions based on commands received from the container node <NUM>. By mapping certain container lifecycle states into appropriate corresponding actions implementable by the control system <NUM>, the container node <NUM> enables program content for the industrial control system <NUM> to be containerized, published to certain registries, and deployed using the master container node <NUM>, thereby bridging the gap between the IT-based container orchestrations system <NUM> and the OT-based industrial control system <NUM>.

In addition, the proxy node <NUM> may also perform certain supervisory operations based on its analysis of the machine state data of the respective control system <NUM>. As a result of its analysis, the proxy node <NUM> may issue commands and/or pods to other nodes that are part of the container orchestration system <NUM>. For example, the proxy node <NUM> may send instructions or pods to other worker container nodes <NUM> that may be part of the container orchestration system <NUM>. The worker container nodes <NUM> may corresponds to other container nodes <NUM> that are communicatively coupled to other control systems <NUM> for controlling other OT devices <NUM>. In this way, the proxy node <NUM> may translate or forward commands directly to other control systems <NUM> via certain OT communication protocols or indirectly via the other worker container nodes <NUM> associated with the other control systems <NUM>. In addition, the proxy node <NUM> may receive replies from the control systems <NUM> via the OT communication protocol and translate the replies, such that the nodes in the container orchestration system <NUM> may interpret the replies. In this way, the container orchestration system <NUM> may effectively perform health checks, send configuration updates, provide firmware patches, execute key refreshes, and provide other services to OT devices <NUM> in a coordinated fashion. That is, the proxy node <NUM> may enable the container orchestration system to coordinate the activities of multiple control systems <NUM> and <NUM> to achieve a collection of desired machine states for the connected OT devices <NUM> and <NUM>.

Bearing the discussion of <FIG> in mind, when utilizing an edge orchestration, there may be networks, devices, or both that are unknown from the perspective of devices or systems that are located outside of a particular system. For example, devices that operate or communication networks within the OT space may be unknown to devices or systems in the IT space, such as a cloud-computing system. Edge devices in the OT space may be communicatively coupled to both a device or system in the OT space as well as other devices in the OT space. For instance, in the context of <FIG>, the industrial control system <NUM> (that includes the proxy node <NUM>) and the computing system that are illustrated as being communicatively coupled to the container orchestration system <NUM> may act as edge devices that can share data between the OT space and the IT space. As discussed above, as more information regarding the OT space (e.g., devices and capabilities) are determined container-based solutions may be provided to expand the functionality of the industrial automation system <NUM> or to enable the industrial automation system <NUM> to have new capabilities.

Keeping this in mind, <FIG> is a block diagram of an industrial automation system <NUM>. As illustrated, the industrial automation system <NUM> includes devices <NUM> (referring collectively to devices 102A-102D), an edge device <NUM>, and a cloud computing system <NUM>. The industrial automation system <NUM> may be implemented using components in the industrial automation system <NUM> of <FIG> and <FIG>. For instance, the devices <NUM> may be industrial automation devices such as the industrial controller system <NUM>, stations <NUM> (referring collectively to stations 14A-<NUM>-H), the conveyer section <NUM>, the sensors <NUM>, the display/operator interface <NUM>, devices that implement the container node <NUM> and proxy node <NUM>, and the OT device <NUM>. The edge device <NUM> may be the computing system that implements the container node <NUM> or the industrial control system <NUM> that implements the proxy node <NUM>. The cloud-computing system may be an IT side device or devices that implement the container orchestration system <NUM> and the container registry <NUM>.

Utilizing components of the industrial automation system <NUM> is one example implementation of the industrial automation system <NUM>. In other embodiments, the devices <NUM> may be industrial automation devices such as, but not limited to, motor control centers, motors, HMIs, operator interfaces, contactors, starters, sensors, drives, relays (e.g., overload relays), protection devices, switchgear, compressors, network switches (e.g., Ethernet switches, modular-managed, fixed-managed, service-router, industrial, unmanaged, etc.), controllers, and the like. Furthermore, while the industrial automation system <NUM> is illustrated as having four devices <NUM>, in other embodiments, the industrial automation system <NUM> may include any suitable number of devices <NUM> that are communicatively coupled to the cloud computing system <NUM> via the edge device <NUM>.

The edge device <NUM> may be a device a computing device or industrial automation device on the OT side that is communicatively coupled to the device <NUM> and communicatively couples the devices <NUM> to the cloud computing system <NUM>, which is located on the IT side. In some embodiments, the edge device <NUM> may be a device such as a controller or a drive. In any case, the edge device <NUM> may include processing circuitry (e.g., one or more processors) that executes machine-readable instructions that may be stored on memory or storage (e.g., a non-transitory computer-readable medium) included in the edge device <NUM>.

The cloud computing system <NUM> may be implemented using one or more computing devices that have processing circuitry that can execute machine-readable instructions that may be stored on memory or storage (e.g., a non-transitory computer-readable medium) included in cloud computing system <NUM> or communicatively coupled to the processing circuitry device of the cloud computing system <NUM>.

The devices <NUM> of the industrial automation system <NUM> may be various types of devices that may have various capabilities, collect (or provide) or utilize (e.g., consume) various types of data, and include various components that may also have particular capabilities or collect or utilize certain types of data. Furthermore, the OT side of the industrial automation system <NUM> may utilize a particular network topology (e.g., a star network or ring) for a network that communicatively coupled the devices of the OT side (e.g., the devices <NUM> and edge device <NUM>). As described below with respect to <FIG> and <FIG>, the cloud computing device <NUM> may implement a container agent <NUM> (e.g., software executed by processing circuitry of the container orchestration system <NUM>) that may send requests <NUM> to the edge device <NUM> for information regarding the devices <NUM> or OT devices of the industrial automation system <NUM>. In response to receiving the request <NUM>, a discovery agent <NUM> (e.g., software executed by processing circuitry of the edge device <NUM>) of the edge device <NUM> may generate and send requests <NUM> to the devices <NUM>, which may send responses <NUM> to the edge device <NUM>. Among other things, the requests <NUM> and responses <NUM> may pertain to a type of a particular device <NUM>, capabilities of the device <NUM>, types of data generated or utilized by the device <NUM>, firmware or software on the device <NUM>, or network information for the OT side of the industrial automation system <NUM>. The discovery agent <NUM> may receive the responses <NUM> and send responses <NUM> to the container agent <NUM> of the cloud computing system <NUM> that are indicative or inclusive of data in the responses <NUM> (e.g., individually or after grouping the responses <NUM>). The container agent <NUM> may process or otherwise analyze the responses <NUM> (or data regarding the devices <NUM> or the OT side of the industrial automation system <NUM>), determine one or more potential solutions (e.g., containers, firmware or updates) that may be implemented (e.g., by the edge device <NUM>) to expand the capabilities or functionality of the devices <NUM> or edge device <NUM>, and send the determined solution(s) (e.g., solution data <NUM>).

With this in mind, <FIG> is a flow diagram of a process <NUM> for providing container-based, firmware, and software solutions for an industrial automation system, such as the industrial automation system <NUM> or the industrial automation system <NUM>. The process <NUM> may be implemented by the container orchestration system <NUM> or the cloud computing system <NUM> by executing computer-readable instructions, for example, for the container agent <NUM>. Before describing the process <NUM>, it should be noted that while the process <NUM> is described below as having operations that are performed in one order, in other embodiments, the process <NUM> may omit some of the operations described below, the operations may be performed in a different order, or both omit some operations and have operations performed in a different order.

At process block <NUM>, the cloud computing system <NUM> (or, more specifically, the container agent <NUM> implemented by the cloud orchestration system <NUM>), may send a request for device discovery to an edge device. For example, with reference to <FIG>, the request for device discovery may be the request <NUM> that is sent to the edge device <NUM>. As noted above, in edge orchestration, there may be unknown networks and devices on those networks (e.g., OT networks). The request for device discovery may be a request for information regarding the devices <NUM>, the edge device <NUM>, and any networks of the industrial automation system <NUM>. For example, the request <NUM> may be for information regarding the devices <NUM> (e.g., identities of the devices <NUM>, what types of devices the devices <NUM> are), capabilities of the devices <NUM>, data that can be generated or consumed by the devices <NUM>, components of the devices <NUM>, and software or firmware of the devices <NUM>.

Upon receiving the request <NUM>, the edge device <NUM> may generate and send requests <NUM> to the devices <NUM> to obtain information about the devices <NUM>. For example, the discovery agent <NUM> may perform discovery services and browse networks (e.g., OT networks) to find devices (e.g., devices <NUM>) that are connected to the same network(s) as the edge device <NUM> and send the requests <NUM> to the devices <NUM>. Each of the devices <NUM> may respond to a received request <NUM> and provide the data requested by the request <NUM> (e.g., in the form of responses <NUM>). The edge device <NUM> may generate the responses <NUM> based on the responses <NUM> received from the devices <NUM>. The responses <NUM> may therefore include the information requested in the request <NUM> regarding the devices <NUM>. The responses <NUM> may also include information regarding the network(s) (e.g., OT networks) utilizing the industrial automation system <NUM>.

As an example, the edge device <NUM> may discover (e.g., based on a received response <NUM> from device 102A) that the device <NUM> is a drive as well as components included in the device 102A (e.g., fans, power modules, filters, etc.). As another example, the response <NUM> received from the device 102B may be indicative of the device 102B being an overload relay capable of providing energy data. As yet another example, the edge device <NUM> may determine networks and characteristics of the networks (e.g., network topologies) of the industrial automation system <NUM>. Accordingly, the response <NUM> may include the networks in the industrial automation system <NUM> as well as the topologies of the networks.

Returning to <FIG> and the discussion of the process <NUM>, at process block <NUM>, the cloud computing system <NUM> may receive the device discovery data from the edge device <NUM>, for example, in the form of the response <NUM>. In other words, cloud computing system <NUM> (and the container orchestration system <NUM>) may receive data regarding devices <NUM> (e.g., identities of the devices <NUM>, what types of devices the devices <NUM> are), capabilities of the devices <NUM>, data that can be generated or consumed by the devices <NUM>, components of the devices <NUM>, software or firmware of the devices <NUM>, and networks (e.g., OT side networks) in the industrial automation system <NUM>.

In embodiments of the process <NUM> in which process block <NUM> is omitted, the edge device <NUM> may generate the device discovery data without being prompted by the cloud computing system <NUM>. For example, at periodic intervals or in response to software or firmware updates, a container being deployed on the edge device <NUM> (e.g., one of the container images <NUM>), determining a new device has been connected to a network to which the edge device <NUM> is also connected, or the edge device <NUM> being deployed, the edge device <NUM> may send the requests <NUM> to the devices <NUM>, receive the responses <NUM> from the devices <NUM>, and generate and send the responses <NUM> to the cloud computing system <NUM>. Accordingly, the edge device <NUM> may provide the response <NUM> without the cloud computing system <NUM> first sending the request <NUM>.

At process block <NUM>, container agent <NUM> of the cloud computing system <NUM> may determine a solution recommendation based on the discovery data received from the edge device <NUM>. In other words, the container agent <NUM> may analyze the data received from the edge device <NUM> (e.g., responses <NUM>) to determine which type of devices the devices <NUM> are, particular components included in the devices <NUM>, types of data that the devices <NUM> may generate or consume, and network topologies of networks in the industrial automation system <NUM>. Based on the received data as well as available containers (e.g., container images <NUM> maintained in the container registry <NUM>), available firmware updates, or available software updates, the container agent <NUM> may determine one or more solutions that may be provided to the edge device <NUM>, for example, to expand the functionality of the edge device <NUM> or another device in the industrial automation system <NUM>. For example, the container agent <NUM> may utilize the received data and determine whether there is a container image <NUM> present in the container registry <NUM> (and not on the edge device <NUM> or a particular one of the devices <NUM>) that could be utilized in conjunction with a particular type of device <NUM>, component of a device <NUM>, type of data, type of network indicated in the received data (e.g., the response <NUM>). Moreover, the container agent <NUM> may determine whether there are available updates based on comparing version on device(s) to version maintained in storage communicatively coupled to cloud computing system <NUM> or based on records maintained by cloud computing system <NUM>. Several examples of solutions that the container agent <NUM> may recommend are provided below.

As one example, the container agent <NUM> may determine what the devices <NUM> are (e.g., types of devices) and the replaceable components included in one or more of the devices <NUM>. As such, the container agent <NUM> may determine that a containerized maintenance solution (e.g., a particular container image <NUM> of the container registry <NUM>) could be provided to the edge device <NUM> to deploy the solution within the industrial automation system <NUM> (e.g., within edge device <NUM> itself). More specifically, the maintenance solution may be a container that, when deployed (e.g., as a container image <NUM> on the edge device <NUM> or elsewhere within the OT side of the industrial automation system <NUM>), receives data regarding the devices <NUM> such as sensor data, a history or the devices <NUM> and components thereof (e.g., maintenance histories, operational histories, runtime operating parameters) and utilizes a (predictive) maintenance model (or updates the model based on the data) to determine and recommend maintenance operations that should be performed to the devices <NUM> or components thereof. For example, from a predictive maintenance standpoint, the maintenance solution may determine, based on the model, when part failures may occur and recommend replacing the part prior to its expected failure.

Another example of a container-based solution is an energy application. For example, in one embodiment, if the responses <NUM> indicate that the devices <NUM> (or a particular one of the devices <NUM>) generates power data (e.g., electrical power consumed). The container image for the energy application may receive the power data to perform analytics regarding the power data, which can be used to make determinations regarding electrical power consumed by the devices <NUM> and the industrial automation system <NUM>.

As yet another example, the container agent <NUM> may check for manufacturing execution system (MES) configurations for MES applications that may run on the edge device <NUM>, one of the devices <NUM>, or another device that may be included in the industrial automation system <NUM>. In industrial automation systems (such as the industrial automation system <NUM> or industrial automation system <NUM>), an enterprise resource planning (ERP) system may be implemented by computing devices. The ERP system may receive data regarding production, such as commands to produce a particular amount of a product or item manufactured by the industrial automation system. The ERP system may generate an order for the items to be manufactured and send the order to an MES application, which may dispatch the devices <NUM> of the industrial automation system <NUM> to perform a process to fulfill the order. In other words, the MES application may cause production of the items to fulfill the order. In the context of providing an MES solution, the container agent <NUM> may determine whether a particular device <NUM> or the edge device <NUM> has information (e.g., a particular type of data) or data structures for a particular MES application or version of the MES application and recommend an MES solution (e.g., MES software or a container image for an MES application) that can be deployed to the edge device <NUM>. Moreover, based on the devices <NUM> in the industrial automation system <NUM> and firmware on the devices <NUM>, the container agent <NUM> may determine what sorts of processes may be implementable in the industrial automation system <NUM> using the MES application. The container agent <NUM> may recommend such processes as part of an MES solution. Accordingly, a new type of or control over performing a process may be possible using the device <NUM> of the industrial automation system <NUM>.

As yet another example of a solution that the container agent <NUM> may determine, the container agent <NUM> may check for firmware or software updates for the edge device <NUM> or the devices <NUM>. For example, the container agent may compare versions of software of firmware on the devices and well as device functionality associated with the versions to provide an information regarding upgrading or updating the firmware or software. For example, a firmware solution generated by the container agent <NUM> may be that if particular software or firmware were to be upgraded or updated, particular functionality would be added to one of the devices <NUM>.

Continuing with the discussion of the process <NUM>, at process block <NUM>, the cloud computing system <NUM> may send a recommendation to the edge device <NUM>. The recommendation may include an indication of the one or more solutions determines at process block <NUM>. In other words, the recommendation may indicate containerized solutions, firmware updates, or software updates that may be provided to the edge device <NUM> to be implemented on the edge device <NUM> or a device <NUM> of the industrial automation system <NUM>.

At process block <NUM>, the cloud computing system <NUM> may receive a response from the edge device <NUM> regarding the recommendation. For example, the response may indicate whether to provide the recommended solution or not. In cases in which the recommended solution includes multiple items (e.g., multiple container images <NUM>, firmware or software updates, or a combination thereof), the response may indicate which of the items should be provided to the edge device <NUM>.

At decision block <NUM>, the cloud computing system <NUM>, or more particularly, the container orchestration system <NUM> or container agent <NUM>, may determine whether the recommended solution should be provided to the edge device <NUM> based on the response received from the edge device <NUM>. For instance, as noted above, the response from the edge device <NUM> may indicate which particular solution (e.g., container-based solution, firmware update, or software update), if any, should be provided. If the recommended solution (or at least one of several recommended solutions) is requested (as determined at decision block <NUM>, at process block <NUM>, the cloud computing system <NUM> or container orchestration system <NUM> may provide the recommended solution(s) to the edge device <NUM>. For example, if the requested solution includes a firmware update, the cloud computing system <NUM> may cause the firmware update to be sent to the edge device <NUM> either by retrieving the firmware update and sending the firmware update to the edge device <NUM> or by causing a computing device communicatively coupled to the cloud computing system <NUM> to send the firmware update to the edge device <NUM>. As another example, if the requested solution includes a container-based solution, the cloud computing system <NUM> may cause a container image <NUM> for the container for the sent to the edge device <NUM> by retrieving the underlying data for the container image <NUM> from the container registry <NUM> and sending the data for the container image <NUM> to the edge device <NUM>.

However, if at decision block <NUM> the cloud computing system <NUM> determines that the recommended solution was not requested, at process block <NUM>, the process <NUM> may end. In other words, the recommended solution is not provided to the edge device <NUM>.

It should be noted that process <NUM> or portions thereof may be performed after a solution has been provided to the edge device <NUM>. For example, the container agent <NUM> may continue to receive data (e.g., in responses <NUM>) and check for additional solutions (e.g., new containers, updates for containers, or updates for firmware or software). Furthermore, the container agent <NUM> may also determine the compatibility of existing solutions (e.g., container images <NUM> deployed on the edge device <NUM>) and firmware on the edge device <NUM> or devices <NUM>. For example, when a solution was provided, a first version of firmware may have been installed on a device <NUM>. However, at a later time, the firmware may have been updated to second version. The container agent <NUM> may determine whether the solution is compatible with the second version of the firmware and, if not, provide a solution recommendation. For instance, in the context of an MES solution, the container agent <NUM> may also determine that firmware has been changed from the first version to the second version and that the second version of the firmware is no longer compatible with a process implemented using the MES solution or the container image <NUM> for the MES solution. In such a case, the container agent <NUM> may determine whether another firmware update exists that would restore compatibility, and, if such an update exists, recommend the update to the edge device <NUM>. The container agent <NUM> may also recommend reverting to the first version of the firmware.

The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature that demonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical.

Claim 1:
A non-transitory computer-readable medium (<NUM>, <NUM>) comprising instructions that, when executed by processing circuitry (<NUM>), cause the processing circuitry to:
receive (<NUM>), from an edge device (<NUM>) of an industrial automation system (<NUM>, <NUM>), data (<NUM>) indicative of a type of one or more devices (<NUM>) in the industrial automation system, one or more types of data generated by the one or more devices, one or more components included in the one or more devices, firmware of the one or more devices, or any combination thereof;
identify (<NUM>), based on the received data, a container from a container repository (<NUM>), wherein the container is implementable on the edge device; and
cause (<NUM>) a container image (<NUM>) for the container to be sent to the edge device.