Patent ID: 12224679

DETAILED DESCRIPTION

In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, interfaces, techniques, etc. in order to provide a thorough understanding. In other instances, detailed descriptions of well-known devices, circuits, and methods are omitted so as not to obscure the description with unnecessary detail. Same reference numerals refer to same or similar elements throughout the description.

Virtualization, both as virtual machines (VMs) and as containers, is the foundation of cloud solutions. VMs and containers differ in several ways, one difference being that containers provide a way to virtualize an operating system (OS) so that multiple workloads can run on a single OS instance, while with VMs, the hardware is being virtualized to run multiple OS instances. Container management systems are often referred to as container orchestrators or just orchestrators and provide cloud elasticity. The orchestrator can instantiate new containers on available hardware resources when needed.

The herein presented embodiments take advantage of the fact that this virtualization technology can be adapted and used to manage controllers and/or services. The term Distributed Control Node (DCN) is used in this application when referring to a controller running on dedicated hardware. The term Virtual Distributed Control Node (VDCN) refers to a virtualized, containerized DCN, basically the virtualized DCN firmware. It is noted that virtualization, containers, and orchestrators are used only as illustrative examples, and that there are other ways to implement the herein presented embodiments.

FIG.1is a schematic illustration of an environment in which embodiments may be implemented. A process control system12is illustrated, comprising a plurality of compute nodes23,24,25,26. In preferred embodiments, the process control system12comprises at least two compute nodes23,24in order to enable handling of hardware failure as well as software failures. The compute nodes23,24,25comprises, among themselves, at least two instances13,14of VDCNs deployed in a respective compute node23,24,25,26of a cluster16of at least two compute nodes23,24,25,26. A first VDCN is a primary VDCN13, which is currently controlling an automated process in, for instance, an industrial automation environment. The process control system12also comprises at least a second VDCN14, which is functioning as a backup VDCN in case the primary VDCN13should fail. The primary VDCN13and the backup VDCN14being deployed in different compute nodes23,24ensures handling also of hardware failures.

A device10is provided, in various embodiments, for ensuring uninterrupted operation of the process control system12. InFIG.1, the device10is exemplified by an orchestrator. Network orchestration refers to actions that e.g., a network controller performs in setting up devices, applications, and services in the network to achieve set objectives. However, the device10may be any of a number of devices, for instance a VDCN, a primary VDCN, a backup VDCN, firmware, a container, an orchestrator, an orchestrator running on a dedicated machine, an orchestrator running on an edge device of the process control system12, control plane, and an edge device of the process control system12. In some embodiments, the device10may be implemented such that the functions are distributed among two or more VDCNs. In the following the device10is exemplified by an orchestrator.

The cluster16is the hardware resources that the orchestrator10is aware of and can use for VDCN deployment. The compute nodes23,24,25,26are the nodes that the cluster16consists of, i.e., the physical devices arranged to execute the controlling of the automated processes. The compute nodes23,24,25,26may, for instance, be embedded devices such as Raspberry Pis, industrial personal computers (PC), Distributed Control Node Hardware, which is purposefit, high-end hardware for industrial use, powerful PCs or servers. The orchestrator10may, for instance, run on a dedicated machine that may, but need not, be a compute node. In an industrial context, an edge device might be suitable for running the orchestrator10, as illustrated inFIG.1. The edge device is typically a computationally competent device (or devices) located at the edge of the control network and acts as a gateway and as data pre-processor towards the cloud. Often the edge device is not just a single device, instead it may comprise of several devices to form an on-premises edge cloud, a cloud that may have an orchestrator.

In many control systems, it is desirable to have one and only one primary VDCN at any given time. If not having only one primary VDCN, an alternative is to vote in the consumers of the data produced in the Distributed Control Node (DCN), which would typically be the I/O or I/O connectivity. However, in a generic system with support for a third party, it would not be possible to require all I/O to support voting, and therefore the use of only one primary is preferred.

In the various embodiments of the present teachings, provides a combination of, on the one hand, failure detection and redeployment of, for instance, containers provided by the orchestrator10and, on the other hand, a DCN network-based state transfer, role selection, and failure detection. The combination results in a process control system12where a primary VDCN13is active and controlling the I/O (not illustrated). If the primary VDCN13fails, the backup VDCN14resumes operation, while at the same time, the orchestrator10redeploys a replacement VDCN15for the failed VDCN in another compute node26with available resources. For example, if a hardware failure occurs in a first compute node23, the backup VDCN14becomes the primary, and the orchestrator10deploys a new VDCN15on e.g., a fourth compute node26. This presented combination results in a pseudo100N redundancy, where N is the number of compute nodes23,24,25,26in the cluster16of compute nodes that can host the redeployed backup.

The present teachings describe, in various embodiments, a kind of orchestration of high-availability VDCNs, realized with a layered architecture: the orchestration layer and the VDCN layer. Each layer has its own supervision mechanism of the VDCNs. The combination of the two yields a high availability redundancy and ensures a bumpless takeover by the backup. The rapid failover handling is handled by the VDCN layer and the redundancy mechanism within that, and the reinstatement of the backup is handled by the orchestration layer.

FIG.2illustrates an exemplary use case, comprising an orchestration layer and a VDCN layer, both of which are described next.

Orchestration Layer

The orchestrator10is configured to have at least two instances13,14of the redundant VDCN running in at least two different compute nodes of the cluster16. The orchestrator10is arranged to supervise that the number of VDCN instances matches a configured number. If there is a missing VDCN (e.g., due to a failure), the orchestrator10is arranged to deploy a new VDCN, provided that there are enough resources in the cluster16. In preferred embodiments, a minimum of three compute nodes are used. The orchestrator10may be provided with various features. For instance, the orchestrator10may be configurable for a selectable number of instances of VDCNs. The number may be set depending, for instance, on the type of automation process at hand. The orchestrator10comprises means to supervise that the number of instances of VDCNs is the expected and may be configured to strive to maintain the configured number of instances of VDCNs. This may be implemented in different ways, for instance querying the VDCN(s) and await the response(s) therefrom, and based on the response(s) take appropriate action The orchestrator10is able to deploy new instances of VDCNs on available cluster resources, when it deems that there is a need for it. This is typically the case when the primary VDCN has failed. The orchestrator is able to gather, e.g., actively poll and/or receive, relevant metrics from the cluster, metrics that it needs to, e.g., determine where a VDCN can be deployed. Examples of such metrics comprise: number of nodes available, CPU usage, CPU quota available, memory usage and pressure (accesses per time unit), disk usage and pressure (accesses per time unit).

VDCN Layer

Three mechanisms are provided for obtaining a failure handling giving a bumpless failover. These mechanisms may be used to achieve the corresponding advantages obtained when using a redundant DCN.

The mechanisms are:Failure detection—to detect that the primary has failed.State transfer: This is to enable the backup VDCN to resume where the primary VDCN left of. The state transfer can be implemented in different ways, e.g., by directly transfer the state from the primary VDCN to the backup VDCN, or by making the primary state available for the backup VDCN, e.g. by sharing the state in, for instance, a cloud environment, or else way making the state available such that the backup VDCN may access it.Role selection—in simultaneous startup or in case of a plurality of backups, the role selection will ensure that only one of the backups becomes the new primary when the former fails. It is noted that the failure may be a failure in the compute node, or in the VDCN.

With the above mechanism(s), a set of VDCNs configured for redundancy can determine amongst themselves (or, in some embodiments, be instructed) which individual VDCN is the primary and detect when that role of being the primary should change and backup should resume this role.

It is noted that two instances of VDCNs are required, but the teachings are not limited to two instances. In case even higher availability is desired, a higher number of instances of VDCNs is possible, and the role selection is a key to that. In some cases, a single VDCN may be used, e.g., when the automated process at hand is such that longer periods of time without active control is tolerable. Then only the state transfer is required in the VDCN layer, which may be accomplished by a network share where a replacing VDCN can access the latest state. However, in the description, the use of two instances of VDCNs is used as an exemplary case since it is the most common use case and used in the following exemplary use case.

The process control system12, e.g. the orchestrator10thereof is, in various embodiments, configured to strive towards two instances of the VDCN13,14on two different compute nodes23,24in the cluster16. The orchestrator10may be configured by an engineering tool. The orchestrator10deploys the VDCN13,14on the two different nodes23,24. The role selection (or assignment) ensures that one of them becomes primary and the other backup. The backup VDCN14is synchronized with the latest primary states using a network, for instance using a network-based state transfer protocol. The VDCNs13,14are running, wherein the primary is controlling the I/O, being fieldbus master, running the control application, and publishing signal/variable values in the network, etc. The backup VDCN14is running in redundant mode. When a failure happens on compute node23that is hosting the primary VDCN13, a failure detection, e.g., in the backup VDCN14, detects that and the role selection assigns it the new role as primary. The orchestrator10detects that only one VDCN exists and deploys a new one on an available compute node, i.e., a compute node having resources for it. The redundancy is restored, without human intervention.

Compared to the state of art, the method provides several advantages, such as: automated re-instatement of backup; higher availability and/or reliability owing to lower mean time to repair; no spare parts need to be fetched; spare capacity may already be connected to the cluster.

Embodiments disclosed herein relate to handling of failures in a process control system12, which is arranged to control an automated process, such as, for instance, drilling process automation, automation of oil rigs, robotic process automation, automated production or assembly lines etc. In order to provide improvements for failure handling in a process control system, a method performed in a process control system12for handling failures in the process control system12, a process control system12, a computer program product comprising code, e.g., in the form of a computer program that when run on means, e.g., processing circuitry or a computer, causes the process control system12to perform the method, a device10, and a method performed by the device10. The method in the process control system12may be performed in a distributed manner.

FIG.3is a flowchart of a method according to embodiments. A method40in a process control system12is provided for handling failures. The process control system12has been described, and comprises a plurality of compute nodes23,24,25and at least two instances13,14,15of virtualized distributed control nodes, VDCNs, deployed in a respective compute node23,24,25of a cluster of at least two compute nodes23,24,25. A first VDCN is a primary VDCN13controlling an automated process and at least a second VDCN is a backup VDCN14.

The method40comprises assigning42the role of primary VDCN13to the backup VDCN14. This step may be performed in the backup VDCN14, as has been described earlier.

The method40further comprises detecting44that the number of instances of the VDCNs13,14,15is less than a set number. This step may be performed in the device10, for instance by the orchestrator10, as has been described and exemplified earlier. The detecting42may be implemented in different ways. In an embodiment, the detecting comprises detecting that there is one VDCN missing, i.e., that a configured number of VDCNs differs from the expected one. The VDCN may be missing due to a failure in at least one of: a first compute node23on which the primary VDCN13is deployed and the primary VDCN13.

The method40comprises deploying46a new instance of a VDCN15in one of the compute nodes23,24,25. This step may also be performed by the orchestrator12of the process control system12. In an embodiment, the step of assigning42is thus performed in the backup VDCN14(the backup VDCN14becomes the primary VDCN), while the steps of detecting44and deploying46are performed by the orchestrator10of the process control system12. In other embodiments, the steps of assigning42and detecting44are performed in the backup VDCN while the deploying is performed by the orchestrator10.

It is noted that the described steps42,44,46may be performed in another order, and that e.g., steps42and44may be performed essentially simultaneously. It is also noted (and exemplified earlier) that the method40may be implemented to be performed in a distributed manner within the process control system12, wherein the steps42,44,46are performed by two or more entities.

In an embodiment, the assigning42is performed in the backup VDCN14and the detecting44is performed in a device10of the process control system12.

In various embodiments, the method40comprises detecting a failure in at least one of: a first compute node23on which the primary VDCN13is deployed and the primary VDCN13. This step may, for instance, be performed by a failure detection function in the backup VDCN14.

In various embodiments, the method40comprises cyclically synchronizing at least the backup VDCN14with the latest primary states. Such synchronizing may be performed for several VDCNs if necessary in view of, for instance, the automated processes at hand, that the process control system12is controlling. A transfer of the state from the primary VDCN to the (new) backup VDCN can be made, which enables the backup VDCN to resume where the failed primary VDCN13left off. The synchronization can be instructed by the orchestrator10and the backup VDCN14may perform it, or the synchronization can be made entirely by the backup VDCN14.

The method40for handling failures in a process control system12may be implemented in one or more devices10,14cooperating. In embodiments wherein two or more devices10,14are cooperating, the steps of the method may be divided in different way as has been described. The device may, for instance, be one or more of: a virtualized distributed control node (VDCN), a primary VDCN, a backup VDCN, firmware, a container, an orchestrator, an orchestrator running on a dedicated machine, an orchestrator running on an edge device of the process control system12, control plane, and an edge device of the process control system12. The process control system12comprises two or more instances13,14,15of VDCN, wherein a first VDCN and a second VDCN are deployed in a respective compute node23,24,25of a cluster of compute nodes23,24,25. The number of required compute nodes is thus at least two. The first VDCN is a primary VDCN13and is controlling an automated process, whereas the second VDCN14is a backup VDCN, to be used in case of a failure. There may be several backup VDCNs, for instance, in case the process control system12is controlling a highly critical automation process or is vulnerable to failures.

A corresponding process control system12is also provided. The process control system12comprises a plurality of compute nodes23,24,25and at least two instances13,14,15of virtualized distributed control nodes, VDCNs, deployed in a respective compute node23,24,25of a cluster of at least two compute nodes23,24,25. A first VDCN is a primary VDCN13controlling an automated process and at least a second VDCN is a backup VDCN14.

The process control system12comprises means for assigning the role of primary VDCN to the backup VDCN14, means for detecting that the number of instances13,14,15of the VDCNs is less than a set number and means for deploying a new instance15of a VDCN in one of the compute nodes23,24,25. Such means may, as has been described earlier, e.g., in relation toFIG.3, comprise a device10and/or a VDCN performing all or some of the steps. The means may comprise computer means programmed to carry out the steps. Such computer means may be executed in a single device, or in a distributed manner among two or more devices.

In an embodiment, the process control system the backup VDCN14comprises a failure detection configured to detect the new role as primary VDCN.

In an embodiment, the process control system12comprises a device10configured to ensure existence in the process control system12of at least two instances of VDCNs on a respective computer node23,24,25.

A device10is also provided to be used in the described process control system12for handling failures. The device10is configured to detect that the number of instances of the VDCNs13,14,15is less than a set number and to deploy a new instance of a VDCN15in one of the compute nodes23,24,25. The device10may be one of: a virtualized distributed control node, VDCN, a primary VDCN, a backup VDCN, a firmware, a container, an orchestrator, an orchestrator running on a dedicated machine, an orchestrator running on an edge device of the process control system12, control plane, and an edge device of the process control system12.

A virtualized distributed control node, VDCN,14is also provided, for use in the described process control system12. The VDCN14, in particular a backup VDCN, is configured to detect that a primary VDCN13controlling an automated process is missing. That is, the number of instances of the VDCNs13,14,15is less than a set number. This detection may, as has been described, be performed in various ways. The backup VDCN14is further configured to assign the role of primary VDCN to the backup VDCN14, i.e., to itself. The backup VDCN14thus becomes the primary VDCN and is controlling the automated process at hand. This can be achieved rapidly and without human intervention. If there are at least three compute nodes, each comprising a deployed VDCN instance, there is no need for an immediate replacement of a compute node and/or VDCN (depending on what type of failure that has occurred), since there are still a primary VDCN controlling the automated process(es) and another backup VDCN ready to assume the role of primary VDCN. In cases where there are two compute nodes, one in which the primary VDCN is deployed and one in which the backup VDCN is deployed, there is still no interruption of the automated processes, and thus no production losses. In such cases, wherein maintenance personnel need to replace the failed unit in order to restore full redundancy, it does not necessarily need to be done immediately.

It is noted that there may be more than one backup VDCN, and that e.g., two backup VDCNs may be supervising that the primary VDCN is functioning properly. In such case one of the backup VDCNs may be assigned the role of primary VDCN and the other may be assigned (or keep) the role of backup VDCN.

In an embodiment, the detecting is performed by a failure detection function of the first VDCN14.

FIG.4schematically illustrates, in terms of a number of functional units, the components of a process control system according to an embodiment. Processing circuitry110is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product (as inFIG.6), e.g., in the form of a storage medium130. The processing circuitry110may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

Particularly, the processing circuitry110is configured to cause the process control system12to perform a set of operations, or actions, as disclosed above. For example, the storage medium130may store the set of operations, and the processing circuitry110may be configured to retrieve the set of operations from the storage medium130to cause the process control system12to perform the set of operations. The set of operations may be provided as a set of executable instructions. The processing circuitry110is thereby arranged to execute methods as herein disclosed.

The storage medium130may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

The device10may further comprise a communications interface120for communications with other entities, functions, nodes, and devices, over the interfaces. As such the communications interface120may comprise one or more transmitters and receivers, comprising analogue and digital components.

The processing circuitry110controls the general operation of the process control system12e.g., by sending data and control signals to the communications interface120and the storage medium130, by receiving data and reports from the communications interface120, and by retrieving data and instructions from the storage medium130. Other components, as well as the related functionality, of the process control system12are omitted in order not to obscure the concepts presented herein.

FIG.5schematically illustrates, in terms of a number of functional modules, components of a process control system12according to an embodiment. The process control system12comprises a number of functional modules; an assign module210configured to perform step42, a detect module220configured to perform step44and a deploy module230configured to perform step46. The process control system12ofFIG.5may further comprise a number of optional functional modules implementing the various method steps described herein. In general terms, each functional module210,220,230may be implemented in hardware or in software. Preferably, one or more or all functional modules210,220,230may be implemented by the processing circuitry110, possibly in cooperation with the communications interface120and the storage medium130. The processing circuitry110may thus be arranged to from the storage medium130fetch instructions as provided by a functional module210,220,230and to execute these instructions, thereby performing any actions of the process control system12as disclosed herein.

FIG.6shows one example of a computer product comprising computer readable storage medium according to an embodiment. On this computer readable means340, a computer program320can be stored, which computer program320can cause the processing circuitry110and thereto operatively coupled entities and devices, such as the communications interface120and the storage medium130, to execute methods according to embodiments described herein. The computer program320and/or computer program product330may thus provide means for performing any actions of the process control system12, and devices10,14as herein disclosed. On this computer readable means330, a computer program320can be stored, which computer program320can cause the processing circuitry110and thereto operatively coupled entities and devices, such as the communications interface120and the storage medium130, to execute methods according to embodiments described herein. The computer program320and/or computer program product330may thus provide means for performing any actions of the process control system12as herein disclosed.

In the example ofFIG.6, the computer program product330is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product330could also be embodied as a memory, such as a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program320is here schematically shown as a track on the depicted optical disk, the computer program320can be stored in any way which is suitable for the computer program product330.

The invention has mainly been described herein with reference to a few embodiments. However, as is appreciated by a person skilled in the art, other embodiments than the particular ones disclosed herein are equally possible within the scope of the invention, as defined by the appended patent claims.