Patent Publication Number: US-11022962-B2

Title: High availability industrial automation system having primary and secondary industrial automation controllers and method of communicating information over the same

Description:
BACKGROUND INFORMATION 
     The subject matter disclosed herein relates to industrial automation systems. More specifically, the subject matter disclosed herein relates to industrial automation systems having backup capabilities. 
     As is known to those skilled in the art, industrial controllers are specialized electronic computer systems used for the control of industrial processes or machinery. An example industrial controller is programmable logic controllers (PLC&#39;s) used in a factory environment. Industrial controllers differ from conventional computers in a number of ways. Physically, they are constructed to be substantially more robust against shock and damage and to better resist external contaminants and extreme environmental conditions. The processors and operating systems of industrial controllers allow for real-time control and execute languages for ready customization of programs to comport with a variety of different controller applications. Industrial controllers may have a user interface for accessing, controlling, and monitoring the industrial controller. 
     In a high availability (HA) industrial control system, it may not be enough for an industrial controller to maintain control after a single fault occurs. The user often needs to change the system operations when the HA state has changed. This may include doing a controlled shutdown or going to some other form of functionality, 
     It is known for certain HA industrial control systems to include a data cross loading system for use with an industrial control system having a primary industrial controller and a secondary (or “backup”) industrial controller. Each industrial controller has redundant hardware components and have data structures associated with these components. Further, known HA industrial control systems have a dedicated high speed data link that connects the primary industrial controller with the secondary industrial controller. The high-speed data link allows for a rapid and “bumpless” transfer of control from the primary industrial controller to the secondary industrial controller when a malfunction occurs. The high-speed data link rapidly cross loads data structures from the primary industrial controller to the secondary industrial controller. U.S. Pat. No. 5,777,874, the content of which is incorporated herein by reference, provides an example of a bumpless system having a dedicated high-speed data link between the primary industrial controller and the secondary industrial controller. 
     However, such bumpless systems have not been fully met without incurring various disadvantages. A fully synchronized HA industrial control system, such as disclosed in U.S. Pat. No. 5,777,874, requires a large amount of data to be transferred between the two controllers. This leads to the requirement for the dedicated high speed link between the devices, additional component costs, additional processing power, and greater board real-estate. 
     BRIEF DESCRIPTION 
     An HA industrial control system with less functionality than known HA industrial control systems is desired. Applications exist where full synchronization and high speed performance are not required by the HA industrial control system. Accordingly, embodiments of the invention provide an HA industrial control system without requiring a dedicated high speed link. The removal of the dedicated high-speed data link and related modules help to lower cost of the improved HA system. Additionally and alternatively, embodiments of the invention provide an HA industrial control system with limited program and data synchronization as options. The lower levels of synchronization allow the user to configure HA features to tradeoff features versus performance. Embodiments of the invention allow the user to configure a system which will provide an acceptable level of continuation of control without program or data synchronization. 
     In one embodiment, the invention provides a method of communicating information from a first industrial automation controller to a second industrial automation controller of a high availability network. The method includes receiving a signal indicative to cross load less than full synchronization between the first industrial automation controller and the second industrial automation controller, limiting synchronization between the first industrial automation controller and the second industrial automation controller based on the received signal, and cross loading information from the first industrial automation controller to the second industrial automation controller. The cross loading of information is less than the maximum amount of communicable information capable of being cross loaded. 
     In another embodiment, the invention provides a high availability industrial automation system having a primary industrial automation controller, a secondary (or backup) industrial automation controller, and a communication network connected to the primary industrial automation controller and the secondary industrial automation controller. The primary industrial automation controller includes a processor and a non-transitory storage medium configured to store a plurality of instructions, a plurality of automation tasks, input/output (I/O) data and internal storage data. The processor is operative to execute the plurality of instructions to receive a signal indicative to cross load less than full synchronization between the primary industrial automation controller and the secondary industrial automation controller, limit synchronization between the primary industrial automation controller and the secondary industrial automation controller based on the received signal, and cross load information from the primary industrial automation controller to the secondary industrial automation controller. The cross loading of information is less than the maximum amount of communicable information capable of being cross loaded. 
     These and other advantages and features of the invention will become apparent to those skilled in the art from the detailed description and the accompanying drawings. It should be understood, however, that the detailed description and accompanying drawings, while indicating preferred embodiments of the present invention, are given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the present invention without departing from the spirit thereof, and the invention includes all such modifications. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Various exemplary embodiments of the subject matter disclosed herein are illustrated in the accompanying drawings, in which: 
         FIG. 1  is a block diagram representing a high availability industrial automation system according to one embodiment of the invention; 
         FIG. 2  is a block diagram representing some aspects of the exemplary industrial automation system of  FIG. 1  in greater detail; 
         FIG. 3  is a block diagram representing a high availability industrial automation system according to another embodiment of the invention; and 
         FIG. 4  is a block diagram representing a processor and memory arrangement for one embodiment of a system controller of an industrial automation controller. 
     
    
    
     DETAILED DESCRIPTION 
     In describing the various embodiments of the invention, which are illustrated in the drawings, specific terminology will be resorted to for the sake of clarity. However, it is not intended that the invention be limited to the specific terms so selected and it is understood that each specific term includes all technical equivalents which operate in a similar manner to accomplish a similar purpose. For example, the word “connected,” “attached,” or terms similar thereto are often used. They are not limited to direct connection but include connection through other elements where such connection is recognized as being equivalent by those skilled in the art. 
     With reference to  FIG. 1 , an industrial automation system  5  includes a first (or primary) industrial automation controller  10  and a second (or secondary) industrial automation controller  15 . As illustrated, the first and second industrial automation controllers  10  and  15  are modular and may be made up of numerous different modules connected together on a rack or rail (represented by dashed line  18 ). Additional modules may be added or existing modules removed and the first and second industrial automation controllers  10  and  15  reconfigured to accommodate the new configuration. In the exemplary industrial automation system  5  shown, both the first and second industrial automation controllers  10  and  15  include a power supply module  20 , a controller module  25 , and a network module  30 . Each industrial automation controller  10  and  15  is further shown with an additional module  35  that may be selected according to the controller requirements. An example additional module is an analog or digital input or output module, which will be referred to herein generally as an I/O module. Another example additional module is a redundancy module for maintaining redundant information between the first industrial automation controller  10  and the second industrial automation controller  15 . Other exemplary additional modules include an additional controller module or an additional network module, 
     An operator interface  40  is shown connected to the industrial automation system  5 . The operator interface  40  can include a processing portion  45 , an input device  50 , and an output device  55 . The input device  50  can include, but not limited to, a keyboard, touchpad, mouse, trackball, or touch screen. The output device  55  can include, but not limited to, a display, speaker, or printer. It is contemplated that each component of the operator interface  40  may be incorporated into a single unit, such as an industrial computer, laptop, or tablet computer. It is further contemplated that multiple operator interfaces can be distributed about the industrial automation system  5 . The operator interface  40  may be used to display operating parameters and/or conditions of the controlled machine or process, receive commands from the operator, or change and/or load a control program or configuration parameters. An interface cable  60  connects the operator interface  40  to the first industrial automation controller  10 . 
     The first and second industrial automation controllers  10  and  15  are connected to other devices by a network  62  according to the application requirements. An interface cable  65  connects the network modules  30  of the controllers  10  and  15 . An interface cable  70  connects the first industrial controller to a first remote rack  85 . An interface cable  75  connects the second industrial controller to a second remote rack  90 . An interface cable  80  connects the first remote rack  85  to the second remote rack  90 . It is contemplated that the network cables  65 - 80  may be custom cables configured to communicate via a proprietary interface or may be a standard industrial cable for a non-proprietary network. Example non-proprietary networks include Ethernet/IP, DeviceNet, or ControlNet. The network cables  65 - 80  arrange the controllers and racks of the network  62  in what is referred to as a ring arrangement. It is contemplated that other network arrangements are possible for coupling the elements of the network  62 , including arrangements that have switches that allow for redundancy communication paths, daisy-chain arrangements, star (simple or multi-layered) arrangements, or more complicated topologies such as redundant local area networks (LANs). The network modules  30  are configured to communicate according to the protocol of the network to which it is connected and may be further configured to translate messages between two different network protocols. 
     The first and second remote racks  85  and  90  are positioned at varying positions about the controlled automation system or process. As illustrated, each remote rack  85  and  90  is modular and may include numerous different modules connected together in a rack or mounted to a rail. Additional modules may be added or existing modules removed (e.g., one of the redundant network modules  30 ) and the remote rack  85  or  90  reconfigured to accommodate the new configuration. 
     As illustrated, each remote rack  85  and  90  includes a pair of network modules  30 . Each network module  30  is connected to one of the network cables  70 - 80 , an input module  110 , and an output module  115 . Each pair of network modules  30  allows for network redundancy at the respective remote rack  85  or  90 . Each of the input modules  110  is configured to receive input signals  120  from controlled devices  125 . Each of the output modules  115  is configured to provide output signals  130  to the controlled devices  125 . An input module and an output module may be combined into a singular module, and collectively the input and output modules may be referred to as I/O modules. Optionally, still other modules  140  may be included in the remote rack  85 . It is understood that the industrial automation system  5 , the industrial automation controllers  10  and  15 , and the remote racks  85  and  90  may take numerous other forms and configurations without deviating from the scope of the invention. 
     Referring next to  FIG. 2 , a portion of the exemplary industrial automation system of  FIG. 1  is illustrated in block diagram form. It is contemplated that each of the nodes in the network may include a processor  145  and a memory  150 . The processors  145  are configured to execute instructions and to access or store operating data and/or configuration parameters stored in the corresponding memory  150 . The processors  145  are suitable processors according to the node requirements. It is contemplated that the processors  145  may include a single processing device or multiple processing devices executing in parallel and may be implemented in separate electronic devices or incorporated on a single electronic device, such as a field programmable gate array (FPGA) or application specific integrated circuit (ASIC). The memory devices  150  are non-transitory storage mediums that may be a single device, multiple devices, or may be incorporated in part or in whole within the FPGA or ASIC. Each of the nodes also includes a clock circuit  155 , and each clock circuit  155  is preferably synchronized with the other clock circuits  155  according to, for example, the IEEE-1588 clock synchronization standard. Each clock circuit  155  generates a time signal configurable to report the present time accurate to either microseconds or nanoseconds. Communication between nodes mounted in the same rack or contained within a single housing occurs via a backplane  160  and a corresponding backplane connector  165 . Nodes communicating via network media  65 - 80  include ports  170  configured to process the corresponding network protocol. The input module  110  includes input terminals  175  configured to receive the input signals  120  ( FIG. 1 ) from the controlled devices  125 ,  135 . The input module  110  also includes any associated logic circuitry  180  and internal connections  185  required to process and transfer the input signals  120  from the input terminals  175  to the processor  145 . Similarly, each output module  115  includes output terminals  190  configured to transmit the output signals  130  ( FIG. 1 ) to the controlled devices  125 , 135  ( FIG. 1 ). The output module  115  also includes any associated logic circuitry  195  and internal connections  200  required to process and transfer the output signals  130  from the processor  145  to the output terminals  190 , 
       FIG. 3  shows another exemplary automation system  205  including a first (or primary) industrial automation controller  210  and a second (or secondary) industrial automation controller  215 . The first and second industrial automation controllers  210  and  215  are not shown in  FIG. 3  as being multi-modular (as compared to the controllers  10  and  15  of  FIG. 1 ), but are unitary industrial automation controllers with power supplies and communications built within the controllers. Further, the first and second remote racks  220  and  225  are shown with the Input/Output (I/O) modules  230  as being unitary modules, unlike the distributed I/O modules  110  and  115  shown in  FIG. 1 . 
       FIG. 4  is a block diagram showing one example arrangement for the processor  145  and memory  150  of a controller module  25 . In  FIG. 4 , the processor  145  includes multiple computer processing units. The first computer processing unit (CPU)  305  is a system CPU that executes application code and messages for communication with other components of the industrial automation system  5 . The second CPU  310  is a backplane CPU that transfers I/O data and other module data on a backplane. The two CPUs  305  and  310  operate independently from each other. The memory is shown as having multiple memories. The memories include a project documentation memory  315 , logic and data memory  320 , and I/O memory  325 . The project documentation memory  315  can include memory for comment descriptions, alarm logs, and data properties. The logic and data memory  315  includes memory for program source code, and program data. The I/O memory can include I/O data, message buffers, and similar data. The three memories can similarly operate independently of each other, i.e., the processors can access information from distinct memories at the same time. While one arrangement for a processor  145  and memory  150  is shown in  FIG. 4 , it is contemplated that other arrangements and number of processors and/or memory can be different. 
     Prior HA industrial automation systems include a data cross loading system having a high-speed data link for allowing rapid and bumpless transfer of control from the primary industrial automation controller to the secondary industrial automation controller when a malfunction occurs. However, such bumpless systems have not been fully met without incurring, various disadvantages as already discussed above. 
     For some applications, full synchronization and high speed performance are not required. The industrial automation system  5 , for example, does not utilize a dedicated high speed data link. The removal of the dedicated high speed link and related modules help to lower the cost of the improved industrial automation system  5 . The industrial automation system  5  also utilizes lower levels of synchronization by allowing the user to configure the industrial automation system  5 . Configuring the industrial automation system  5  allows for trading features with performance. 
     During configuration of the industrial automation system  5 , the user can provide configuration input via the operator interface  40 . For example, the user may identify to the industrial automation controller  10  that it is the primary controller and that the industrial automation controller  15  is the secondary (or backup) controller. The user may also identify other components coupled to the industrial automations system  5  to the primary industrial automation controller  10 . Further configuring and programming of the industrial automation system  5  can be done as is customary to one skilled in the art. 
     During operation of the industrial automation system  5 , the primary industrial automation controller  10  controls the first and second remote racks  85  and  90  and cross loads information to the secondary industrial automation controller  15 . The controlling of the industrial automation system  5  and cross loading of information is accomplished over network  62 . 
     More related to embodiments of the invention is the further configuring of the primary industrial automation controller  10 , via the operator interface  40 , to cross load less than full synchronization information between the primary industrial automation controller  10  and the secondary industrial automation controller  15 . For example, the user can enter a command to the primary industrial automation controller  10  to (a) cross load program synchronization information and input/output (I/O) synchronization information, (b) cross load program synchronization information and not I/O synchronization information, (c) cross load I/O synchronization information and not program synchronization information, and (d) cross load neither I/O synchronization information nor program synchronization information. 
     It is envisioned that other information may be cross loaded or prevented from cross loading between the primary and second industrial automation controllers  10  and  15 . An example of program synchronization information includes industrial automation tasks (e.g., event tasks, periodic tasks, continuous tasks) to be used by the first remote rack  85  and/or the second remote rack  90 . An example of I/O synchronization information includes I/O data acquired from the devices  125  during operation of one or more industrial automation tasks. 
     Through the selection of synchronization options, the improved industrial automation system  5  allows for trading features with performance. With option (a), above, the industrial automation system  5  still allows for full synchronization, but the synchronization may be delayed by data congestion on the network  62 . This may create a “bump” in the industrial automation system  5  when the secondary industrial automation controller  15  takes over from the primary industrial automation controller  10  since the cross loading of information may not be fully synchronized. Presumably the bump will be acceptable otherwise a user may select a different industrial automation system with a dedicated synchronization connection. Also this option is selected for the expectation that the network  62  has little data congestion among the various components to allow for the full synchronization to occur. 
     When data congestion is higher than what is acceptable for option (a), the user may select options (b) or (c), above. Options (b) and (c) still allow for some synchronization among the industrial automation controllers  10  and  15  to improve performance over a non-synchronized system; but the reduced synchronization improves data congestion on the network  62  since less than full synchronization occurs. 
     For implementations when I/O synchronization information is not cross loaded (i.e., option (b)), a bump in the industrial automation system  5  occurs when the secondary industrial automation controller  15  takes over from the primary industrial automation controller  10 . The bump occurs since the default I/O synchronization information is not as current as the I/O synchronization information of the primary industrial automation controller  10 . However, the user has deemed the bump to be acceptable. 
     For implementations when program synchronization information is not cross loaded (i.e., option (c)), the secondary industrial automation controller  15  may include a program, such as a shutdown program, that is acceptable to the user when the secondary industrial automation controller  15  takes over from the primary industrial automation controller  10 . The cross loading of I/O synchronization information is deemed by the user as necessary in option (c) to allow the secondary industrial automation controller to function. 
     Lastly, the user may configure the primary industrial automation controller with option (d), above. It is not envisioned that this feature will be selected often since no cross loading will take place between the industrial automation controllers  10  and  15 . However, it is conceivable that the network  62  may become too congested such that no cross loading may be desired. 
     With using the configurable levels of cross loading, embodiments of the invention can also allow for a similar look and feel to the full implementation of the system described in U.S. Pat. No. 5,777,874, but with reduced functionality. This will allow a manufacturer to provide a family of customizable products to the consumer. 
     According to another aspect of the invention, the control program loaded into each of the primary controller  10  and the secondary controller  15  includes a verification tag associated with the program. The verification tag may be generated by programming software when the control program is created prior to loading the program into the controller. Optionally, the verification tag may be generated by the primary controller when a new control program is loaded into the controller. It is further contemplated that the verification tag may be modified if, for example, a technician modifies the control program, or a portion thereof, within the controller after it has already been downloaded. 
     The verification tag maybe generated by any suitable method. It is contemplated that the verification tag may be, for example, a single number or a data string, such as a checksum, generated as a function of the content of the control program; a user defined name assigned to the control program, where the name must be unique for each program; a value generated based on the time the control program is created; an incremental value that is automatically updated by the programming software each time the control program is changed; or any combination thereof. The verification tag is stored in the memory of the primary controller  10 . 
     As indicated above, it is contemplated that the user may configure whether the program is synchronized between the primary and secondary controllers  10 ,  15  as an option of the HA system. When the synchronization occurs, the primary controller  10  transfers the verification tag associated with the control program along with the control program to the secondary controller  15 . If program synchronization is not selected, a separate verification tag is loaded into each controller when the corresponding control program is loaded into the controller. Similarly, should the control program be modified on either the primary or secondary controller  10 ,  15 , the verification tag changes on the controller to which modifications are made. The verification tag for each controller  10 ,  15  is transmitted to the other controller even if the program synchronization is not selected. Because the verification tag is a single number or string of data, it does not require significant bandwidth for transmission. 
     As previously discussed, the HA system typically needs to continue operation under various fault conditions. One such fault condition is the failure of a controller. If the controller presently controlling operation of the industrial automation system  5  (i.e., either the primary controller  10  or the secondary controller  15 ) enters a fault state such that the controller no longer can continue operating, the HA system may execute a switchover such that the other controller continues operation of the system. Because it is possible that the two control programs are not identical (i.e., program synchronization was not selected) the controller is configured to compare the verification tag fir the control program presently loaded in the controller that is to begin operating to the verification tag of the controller previously operating. If the two verification tags are identical, the switchover is allowed to proceed and the secondary controller continues operation of the HA system. 
     If the two verification tags are not identical, the two control programs in each controller were not identical. An error message is generated and transmitted to a technician to identify the fault condition. Further operation may proceed according to a predefined state. In one state, the new controller may immediately shut down due to the differences in the program. In another state, the new controller may enter a controlled shut down state, to allow the controlled system to continue operating for a period of time. In still another state, the new controller may be allowed to take over operation of the controlled process with the different control program but simply post a message alerting the operator of the condition such that the operator may take action according to the application requirements. 
     Variations and modifications of the foregoing are within the scope of the invention. It also being understood that the invention disclosed and defined herein extends to all alternative combinations of two or more of the individual features mentioned or evident from the text and/or drawings. All of these different combinations constitute various alternative aspects of the invention. The embodiments described herein explain the best modes known for practicing the invention and will enable others skilled in the art to utilize the invention. The invention is capable of other embodiments and of being practiced or carried out in various ways.