Abstract:
An industrial control system hardened against malicious activity monitors highly dynamic control data to develop a dynamic thumbprint that can be evaluated to detect deviations from normal behavior of a type that suggest tampering or other attacks. Evaluation of the dynamic thumbprint may employ a set of ranges defining normal operation and reflecting known patterns of interrelationship between dynamic variables.

Description:
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
       [0001]    -- 
       CROSS REFERENCE TO RELATED APPLICATION 
       [0002]    -- 
       BACKGROUND OF THE INVENTION 
       [0003]    The present invention relates to industrial controllers controlling factory automation and/or industrial processes and in particular to a system providing enhanced security for industrial control systems against malicious acts. 
         [0004]    Industrial control systems have traditionally been protected against tampering or malicious activity by the same safeguards used to protect the physical equipment of the factory or the like, that is limiting physical access to the industrial controller and its associated equipment. 
         [0005]    Modern industrial control systems employing distributed processing as well as network and Internet connections have greater exposure to attack. While such systems may be physically secured, more points of security must be established for distributed systems, and network connections to the Internet can render physical security irrelevant. Recent evidence is that access to industrial control systems through the Internet is being exploited by sophisticated and well-funded foreign nations or organizations. In one example, the United States Industrial Control System Cyber Emergency Response Team (ICS-CERT) has provided a warning related to malware (Black Energy) attacking the human machine interfaces (HMI) of programmable logic controllers used to manage and control industrial equipment. There is anecdotal evidence of successful Internet-based attacks directly on industrial control systems. 
         [0006]    Unlike attacks on standard computer equipment and servers, attacks on industrial control systems can conceivably produce damage to physical property and risk to human life. 
       SUMMARY OF THE INVENTION 
       [0007]    The present invention provides a system for increasing the security of industrial control systems by monitoring possible tampering that may only be evident in dynamically changing patterns of operation of the industrial control system. 
         [0008]    In one embodiment, the invention is directed toward an industrial control system having multiple inter-communicating industrial control devices coordinated according to a control program. Each of multiple control devices may have one or more device network ports for communicating with other elements of the industrial control system and electrical connectors for accepting electrical conductors communicating with industrial equipment to receive or transmit electrical signals from or to that industrial equipment for the control of an industrial process. 
         [0009]    The control devices also provide a control device processor communicating with an electronic memory system holding: operating software describing operation of the control device, a data table holding representations of the electrical signals of the electrical connectors and a diagnostic program providing outputs monitoring the operation of the control device. The data table and the outputs of the diagnostic program together define a dynamic device state. 
         [0010]    The operating software executes to (i) read at least a portion of the dynamic device state to generate a dynamic signature, encrypt the dynamic signature, and transmit the dynamic signature over the network port. 
         [0011]    The control system also provides a security controller having a controller network port for communicating with other elements of the industrial control system, a security controller processor communicating with the controller network port and a controller electronic memory system accessible by the security controller processor and holding a security program. 
         [0012]    The security program executes to receive a dynamic signature from a given control device through the network port, decrypt the dynamic signature, analyze the dynamic signature against rules establishing a multi-value range of acceptable dynamic signature values, and provide an output indicating whether the received dynamic signature is outside the multi-value range of acceptable dynamic signature values. 
         [0013]    It is thus a feature of at least one embodiment of the invention to provide a security-hardened dynamic thumbprint that can be used to detect malicious activity interfering with operation of the control system. 
         [0014]    The portion of the dynamic device state may include data indicating electrical signals of the electrical connectors. 
         [0015]    It is thus a feature of at least one embodiment of the invention to deduce possible tampering with both the control devices and the machinery to which they are attached by using an analysis of I/O signals from the control device. 
         [0016]    The security program may execute to receive a dynamic signature from a multiple of given control devices through the network port and analyze the dynamic signature against integrated rules relating to the combined dynamic signatures and establishing a multi-value range of acceptable dynamic signature values. 
         [0017]    It is thus a feature of at least one embodiment of the invention to provide the ability to detect malicious activity through the global analysis of disparate portions of the control system. 
         [0018]    The dynamic signature may include multiple time varying quantities wherein the rules establish multi-value ranges for each quantity. 
         [0019]    It is thus a feature of at least one embodiment of the invention to detect tampering from dynamic variables which do not conform to a static thumbprint through the use of ranges encompassing multiple values of the dynamic signature. 
         [0020]    The multi-value ranges may vary as a function of other varying quantities. 
         [0021]    It is thus a feature of at least one embodiment of the invention to provide a set of sophisticated rules that may recognize correlations or interrelations among different variables. 
         [0022]    The rules may be applied by a supervised machine learning system trained with dynamic signatures from a properly operating industrial control system. 
         [0023]    It is thus a feature of at least one embodiment of the invention to provide a system that can manage the complexity of multiple dimensions of variables for an arbitrary control system providing insight into possible tampering. 
         [0024]    The properly operating industrial control system may be determined at least in part by historical operation of the industrial control system. 
         [0025]    It is thus a feature of at least one embodiment of the invention to provide a system that continues to learn during operation of the industrial control system. 
         [0026]    The dynamic device state may include at least one of a timestamp, a digital signature, a device identification number, and a changing random code. 
         [0027]    It is thus a feature of at least one embodiment of the invention to provide a method of reducing the risk of tampering with the transmitted dynamic device state during transmission. 
         [0028]    The dynamic signature may include outputs from a diagnostic program monitoring operation of the control device. 
         [0029]    It is thus a feature of at least one embodiment of the invention to monitor operating parameters such as CPU utilization, free memory, stack depth, port traffic over a predetermined interval and change in average port traffic that may indicate malicious activity of the nature of a denial of service attack overloading the control device. 
         [0030]    The rules may be at least in part a function of calendar data indicating schedule changes in the industrial control system. Alternatively or in addition, the dynamic signature may include an operating mode of the control device selected from a run state indicating that the control device is running to execute a control program and a programming state indicating that the control device is being programmed with respect to a control program. 
         [0031]    It is thus a feature of at least one embodiment of the invention to inform the rules with scheduled maintenance or changes dining programming to reduce false positive alarms. 
         [0032]    A first control device may produce a first dynamic signature and a second control device may receive the first dynamic signature and produce a second dynamic signature based on a dynamic device state of the second control device and the first dynamic signature and transmit the second dynamic signature over a control system communication port of the second control device. 
         [0033]    It is thus a feature of at least one embodiment of the invention to permit a distributed processing of dynamic security data that can provide insight into analysis of the data and reduce the amount of data that needs to be transmitted. 
         [0034]    These particular objects and advantages may apply to only some embodiments tailing within the claims and thus do not define the scope of the invention. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0035]      FIG. 1  is a simplified industrial control system showing multiple controllers, distributed control modules, connections to the Internet and supervisory systems suitable for use with the present invention; 
           [0036]      FIG. 2  is a functional diagram of an example control device showing various functional components whose data may be incorporated into a thumbprint revealing the security state of those components; 
           [0037]      FIG. 3  is a functional diagram of a security device and a security template used in managing the security signatures generated by the control devices; 
           [0038]      FIG. 4  is a flowchart depicting the steps of populating the security template of  FIG. 3  from various device files; 
           [0039]      FIG. 5  is a flow chart of a configuration tool executed by the controller of  FIG. 3  or other security device in configuring a security system of the present invention and the operation of a security-processing program; 
           [0040]      FIG. 6  is a flowchart of the steps executed by the security-processing program after configuration in executing a response script; 
           [0041]      FIG. 7  is a logical representation of the significance matrix for analyzing the significance of detected errors; 
           [0042]      FIG. 8  is a logical representation of the notification tree providing different notifications depending on their significance levels and responses from notified individuals; 
           [0043]      FIG. 9  is a figure similar to that of  FIG. 2  showing the development of a dynamic thumbprint; 
           [0044]      FIG. 10  is a logical representation of the hierarchy of the industrial control system of  FIG. 1  showing the passing of context information upward through the hierarchy for the processing of dynamic thumbprint data; 
           [0045]      FIG. 11  is a translation table for translating local variable names into the template variable names; 
           [0046]      FIG. 12  is a process diagram of the training of the supervised machine learning system for analyzing dynamic thumbprints; 
           [0047]      FIG. 13  is a flowchart of an authorization protocol used to prevent unauthorized changes in the control hardware; and 
           [0048]      FIG. 14  is a simplified depiction of a global display of security status. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
     Example Control System 
       [0049]    Referring now to  FIG. 1 , an industrial control system  10  suitable for application of the present invention may provide one or more controllers  12   a ,  12   b , operating to execute a control program for the control of an industrial process  14  as is generally understood in the art. The industrial process, for example, may coordinate a set of machines on an assembly line or the like, or interact with actuators and sensors of plant processing materials to control that process, or conduct other similar control applications. 
         [0050]    The industrial controllers  12  may communicate downstream with one or more control devices  16   a - 16   c  providing a direct interface to the elements of the industrial process  14 . Such control devices  16  may include, by way of non-limiting example, one or more I/O modules  16   a  providing input and output lines  18  to and from the industrial process  14  allowing communication with sensors  20  and actuators  22 . Other example control devices  16  may be a motor controller  16   b  controlling power applied to electric motor  23 , or motor drives  16   c  providing more sophisticated motor control, for example, by synthesizing power waveforms to a motor  23 . 
         [0051]    The industrial controllers  12  may communicate with the control devices  16  by means of an industrial control network  24  such as the Common Industrial Protocol (CIP™), EtherNet/IP™, DeviceNet™, CompoNet™, and ControlNet™ managed by the standards organization, ODVA, of Michigan, USA. Such networks provide for high reliability transmission of data in real time and may provide features ensuring timely delivery, for example, by pre-scheduling communication resources such as network bandwidth, network buffers, and the like. 
         [0052]    The industrial controller  12  may also communicate upstream, through a data network  26  (which may, but need not be an industrial control network) via one or more routers or switches  28 , with a central computer system  30 . This latter supervisory computer system  30  may further communicate via the Internet  32  with remote devices  34  such as computer terminals, mobile wireless devices, and the like. Alternatively, there may be a direct connection between the industrial controller  12  and the Internet  32 . 
         [0053]    As is generally understood in the art, each of the control devices  16 , industrial controllers  12 , switches  28 , computer systems  30  and remote devices  34  may provide one or more electronic processors and associated electronic memory holding programs executable by the processors, some of which are described below. 
         [0054]    Referring now to  FIGS. 1 and 2 , a representative control device  16  provides for I/O conductors  36 , for example, wires communicating with sensors  20 , actuators  22 , motors  23 , or the like. These I/O conductors  36  may be releasably connected to the control device  16  via one or more terminal or connector systems  38 , for example, screw terminals. The connector system  38  in turn may communicate with a connection management circuit  40  which can detect, for example, the presence or absence of a conductor  36  attached to the connector system  38 , for example, by monitoring a current loop or by monitoring an applied voltage or the like, or a broken wire or “stuck at” fault by monitoring and absence of signal state change over a predetermined time period or during application of a test signal. 
         [0055]    Signals from the conductors  36  pass through the connector system  38  and connection management circuit  40  and are acquired and stored in an I/O table  42  being part of onboard computer memory  45  comprised of volatile and nonvolatile memory structures. Signals to be output from the control device  16  may be also stored in the I/O table  42  prior to transmission on the conductors  36 . 
         [0056]    A processor  44  within the control device  16  may execute a control program  46 , for example, held in volatile memory, as mediated by operating system  48 , for example, being firmware held in nonvolatile memory. The control program  46  may process inputs received from conductors  36  as stored in I/O table  42 . These inputs may be transmitted to an industrial controller  12  via a network interface  54  allowing communication on the network  24  for processing by a control program held in the industrial controller  12 . The control program  46  and the operating system  48  may be implemented as either firmware or software or a combination of both. 
         [0057]    Conversely, the control program  46  of the control device  16  may also execute to receive outputs from the industrial controller  12  through the network interface  54  to generate output values written to the I/O table  42  and ultimately output over conductors  36 . The control program  46  may also or alternatively execute some control logic to generate its own outputs from received inputs. 
         [0058]    In one embodiment of the present invention, the control device  16  also holds in memory  45  a security program  58  that provides for generating a “thumbprint” according to a thumbprint table  62  and a defense script  64 , both of which will be discussed further below. 
       Static Signatures 
       [0059]    The control device  16  may employ a variety of data structures that reflect the status of the control device, its configuration, and the authenticity of its programs. 
         [0060]    The control program  46  and the operating system  48  may include information such as a revision number and digital signature  49 , for example, the latter using public-key or similar techniques such as asymmetric encryption and cryptographically secure hash functions, that allow determination that the associated firmware or software is from a trusted or valid source. 
         [0061]    Generally, the revision number need not be a single revision number, but could include an aggregated set of revision numbers representing a set of different revision numbers, for example, from different components of the software or from affiliated software or a chain of sequential revisions. Revision numbers may also be associated with firmware or hardware of the device, as will be discussed below. 
         [0062]    In addition, the entire data set of the control program  46  and the operating system  48  may be hashed or otherwise digested to a reduced size sub-thumbprint as will be described below. This digesting process is strictly distinguishable from compression in that the latter anticipates a de-compression or recovery step, but as used herein this digesting process will generally be referred to both as a digesting and/or a compression with this distinction understood. 
         [0063]    A hardware configuration register  50  (implemented in volatile or nonvolatile memory  45  and/or as physical switch positions) may hold settings for controlling the operation of the control device  16  and may additionally provide manufacturing data about the control device  16  including, for example, a serial number, module function type, manufacturer name, manufacture date, and the like. In addition, the hardware configuration register may provide for a read-only memory including an encrypted certification code embedded by the manufacturer indicating authenticity of the hardware. The hardware configuration registers may further provide a storage location for output data from one or more diagnostic programs implemented by the operating system  48 , for example, those that indicate memory or other faults, instruction execution speed, memory capacity or checksum results. In one embodiment, the diagnostic program outputs CPU utilization, free memory, and stack depth. The diagnostic program may also monitor network communication including port traffic over a predetermined interval and/or change in average port traffic such as may indicate a denial of service type attack. 
         [0064]    A transaction log  52  also held in memory  45  may record certain activities affecting the control device  16 , for example, the act of uploading of new control program  46  and/or operating system  48  or changes in switch settings stored in the hardware registers  50 , and may record these activities and the time at which they occurred in the source of the change, for example, including identity of an authorized individual. 
         [0065]    Referring still to  FIG. 2 , during operation of the control device  16 , under the control of the security program  58 , a digital operating thumbprint  70  may be periodically generated and transmitted to a security device  31 , for example, one of the industrial controllers  12  or the computer system  30 . This operating thumbprint  70  ideally captures portions of the data structure described above that can be used to determine whether they have been tampered with or corrupted in some fashion. For maximum flexibility, the contents of the digital operating thumbprint  70  may vary according to a thumbprint table  62  which provides for various transmission modes  72  each associated with different digital thumbprints  70  having different amounts of information and thus representing different degrees of size reduction of the state data of the control device  16 . As such, these different digital operating thumbprints  70  provide a trade-off between thumbprint detail and computational and transmission burden. 
         [0066]    Generally, the operating thumbprint  70  for each mode  72  of the thumbprint table  62  designates a specific set of thumbprint source data  74 , for example, the control program  46 , the firmware operating system  48 , the configuration register  50 , and environmental data held in various components of the control device  16  including the wire connection states of the connection management circuit  40 , its address and/or location in the factory environment (for example held in communication or memory modules), operating temperature and the like from distributed internal sensors. In one example mode  72 , the entire data set from each of the sources is reduced to a digest, for example, using a cyclic redundancy code or hash compression and these compressed representations are assembled to generate one or more digital operating thumbprint  70 . The compression process may be loss-less or lossy but need not allow reconstruction of the digested data. 
         [0067]    This digital operating thumbprint  70  is then transmitted to the remote security-monitoring device where it is compared with a corresponding stored thumbprint to establish within a reasonable probability according to the digest scheme that the source data  74  of the control device  16  has not been modified or tampered with. 
         [0068]    In different modes  72 , the amount of data size reduction provided in the thumbprint table  62  may be reduced or each of the source data  74  compressed separately so that an analysis of the operating thumbprint  70  may reveal the particular source data  74  that has changed or been corrupted. Thus, for example, each of the control program  46  and operating system  48  may be separately compressed into a sub-thumbprint  78 . Mismatches in the comparison of the sub-thumbprint  78  and its corresponding copy at the security device  31  allow for determination of which of the structures of a control program  46  and operating system  48  have changed as opposed to their being simply a change in one of the two programs. The importance of this will be explained below with respect to determining the significance of any mismatch in the thumbprints. Similarly, the wire-off information of the connection management circuit  40  and hardware registers  50  may be transmitted without compression (e.g., as uncompressed sub-thumbprints  78 ) so any detected change in the sub-thumbprint  78  immediately indicates which wire has been disconnected or which hardware value has changed. 
         [0069]    The operating thumbprint  70  may also include digital signature  82 , allowing the detection of tampering of the operating thumbprint  70  after it has been transmitted from the control device  16 . In this respect the operating thumbprint  70  may include a timestamp  79 , a sequence value or randomly generated value  83  that may be synchronously developed at a receiving security device  31  (for example, by a clock or similar algorithm) so that an operating thumbprint  70  may not be intercepted and replaced to spoof the security device  31  into believing that an operating thumbprint  70  has been sent or different operating thumbprint  70  has been sent. The timestamp  79  and the sequence value or randomly generated value  83  encoded in the operating thumbprint  70  prevents ready substitution of values in an intercepted operating thumbprint  70 . The operating thumbprint  70  may also include a device identification number  71  that allows the particular control device  16  sourcing the thumbprint to be positively determined. 
         [0070]    The operating thumbprint  70  has been described only with respect to control devices  16 , but it will be understood that every element of the control system  10  may develop these thumbprints which may be passed upward to a security device  31 . Thus the controllers  12  may also generate thumbprints when the security device  31  is computer system  30 . The exact content and compression of the thumbprint  70  will be device-specific. 
         [0071]    Referring now to  FIG. 3 , the security device  31  managing the analysis of the operating thumbprints  70  will generally include a network interface card  84  communicating with the network  24  to receive the operating thumbprint  70  on a periodic basis, for example, as pushed from the control devices  16  or in response to a poll from the security device  31 . In one embodiment, the polling from the security device  31  is done on a periodic basis, for example, timed from the last transmitted message from the control device  16 . In this way, the control devices  16  may also assess the health or security of the system if necessary when a polling has not been performed after a predetermined time. The polling may be done by employing authentication certificate using a public-key encryption or the like to prevent spoofing of this polling process. 
         [0072]    Generally, the security device  31  also includes a processor system  86  and a memory  88  holding a security-processing program  90 , as will be described, and a populated security table  92  used for security analysis. 
         [0073]    The populated security table  92  may provide an entry for each control device  16  as indicated by entry field  94 . The populated security table  92  may also provide, for each signature mode  72 , thumbprint data  98  including a stored thumbprint  100  for that signature mode  72 , previous valid thumbprints  108 , and a thumbprint map  110 . A timestamp value  102  may be stored in the security table  92  or an associated data structure to indicate the received time of the latest copy of a valid operating thumbprint  70  from a given control device  16 , and a notification tree  104  may be provided which provides contact information for notifications of security issues as will be discussed below. 
         [0074]    The thumbprint map  110  may generally identify each of the sub-thumbprints  78  by the function  112  of the source data  74  (for example: operating system  48 , control program  46 , hardware registers  50 ) and will give a weight  114  indicating the significance of a possible mismatch between stored thumbprint  100  and received thumbprints  70  or sub-thumbprint  78 . The thumbprint map  110  may also provide a response script  118  indicating possible responses to a detected mismatch between the operating thumbprint  70  and the stored thumbprint  100 . Clearly the number of sub-thumbprints  78  and hence the number of thumbprint maps  110  will vary depending on the particular mode  72 . 
         [0075]    Referring now to  FIG. 4 , the information of the populated security table  92  may be rapidly generated by selecting from a number of standard security templates  120  being generally defined for different generic types of control systems  10 . For example, a given packaging line providing for relatively standard control devices  16  may provide a standardized template  120 . 
         [0076]    Each template  120  may provide for generic programs  121  for each of the components of the industrial control system  10  including a generic control program  123  for one or more controllers  12  and generic device programs  125  and security programs  131  for one or more associated control devices  16 . The generic programs  121  will define generic I/O points that allow for electrical communication to sensors or actuators of an industrial process  14  using generic names. As will be discussed below, these generic I/O points may be modified by the user to link them to actual physical I/O in a configured industrial control system. Afterwards the modified generic control program  123  and modified device programs  125  may be loaded into the associated physical components to provide for a rapidly configured security system. 
         [0077]    The standardized template  120  may also be associated with a security-monitoring program  129  that may be uploaded into the supervisory computer system  30  (shown in  FIG. 1 ) for communicating with the security programs  131  to coordinate the security process. 
         [0078]    Once a standardized template  120  is selected, the generation of a populated template may be performed by a template crafting program  126  executed, for example, on the computer system  30  or a controller  12  during the commissioning of the control system  10  as indicated by process block  130 . Each standardized template  120  will have pre-populated elements  122  based on the assumed underlying process, and will also require additional information for the particular industrial process  14 . For example, some of the pre-populated elements  122  may identify general functional blocks needed for the control system  10  of the type assumed by the template  120 . The user, may then select among specific device files  124  representing a particular control device  16 , for example, a given model number of motor controller that meets a functional block requirement (e.g., generic motor controller) of the standardized template  120  but provides specifics with respect to the particular device. Incorporation of device files  124  into the standardized template  120  is indicated by process block  132 . In some embodiments, the specific device files  124  may provide their own versions or modifications or patches to the generic device programs  125  or security programs  131 . Generally hardware manufacturers may supply the necessary device files  124 . 
         [0079]    Standardized template  120  will also include the elements of the security table  92  as discussed above which may be used by the security-monitoring program  129 . Generic elements of the security table  92  may be supplemented by data manually added or edited by the user within the framework provided, for example, to create the notification tree  104 , indicating people to be notified in the event of the thumbprint mismatch. Some pre-populated elements, for example, weights  114  that are ascribed to a particular control device  16  or sub-thumbprint  78 , may be modified or may assume a default value from the standardized template  120 . These editing changes are indicated by process block  134 . 
         [0080]    Referring to  FIGS. 4 and 11 , as noted above order for the pre-established security templates  120  to provide for rules that work not only with the generic process of the security template  120  but also with an actual control process, the process of populating the template values per process block  134  may employ a template translation table  208  which links standardized template device names  220  for generic control devices to actual device names  219  for the actual control devices  16  of the industrial control system  10 . This linking may be performed at a time of commissioning per process block  136  guided by corresponding functions  217  describing the functions of the generic control devices associated with the standardized device names  220 . The standardized template device names  220  built into predefined rules associated with the security templates  120 , as described below, may then be mapped to the actual device names  219  so that the predefined security roles of the security templates  120  may apply to the devices of the particular application without the need to develop the rules for each different application. 
         [0081]    Also at process block  136 , particular generic functions implemented by various input or output variables maybe identified by particular tag names used in a given control program  46 , for example, so that the security device  31  may interpret the function implemented by a particular conductor  36  should it become disconnected from connector system  38 , so that a generated report to a user can indicate the function that was lost not simply an arbitrary wire number. The standardized security templates  120  allow the benefits of a detailed vulnerability analysis of the given types of control systems, identifying likely failures, the significance of those failures and the response to those failures indicated by mismatch thumbprints  70 , to be leveraged among many installations and many users. When the standardized template  120  is fully populated at process block  137 , it may be uploaded to the security device  31  and the security-processing program  90  activated. 
         [0082]    Referring now to  FIG. 5 , before the industrial control system  10  is put into use, the control system  10  may be configured, as indicated by process block  140 , during which the various components may be interconnected by the network  24  and the necessary control program  46  loaded into control devices  16  and hardware register values  50  and other components initialized in the loaded populated security table  92  installed. 
         [0083]    At process block  142  public keys or similar security keys such as asymmetric encryption may be created and distributed to the components of the industrial control system  10  (e.g., the control, devices  16 , the controllers  12 , etc.) to allow for the attachment of digital signatures in the exchange data described above with respect to the thumbprints  70 . At process block  144 , the populated device templates  120  generated for the security device  31  are loaded with stored thumbprint  100  of each of the components of the system  10 . 
         [0084]    During general operation of the control system  10 , thumbprints  70  are solicited from or pushed by the control device  16  to the security device  31 , as indicated by process block  146 , where they are compared as indicated by decision block  148  with the stored thumbprint  100  for the proper mode  72 . This comparison is according to the particular control device  16  from which the operating thumbprint  70  is received. If the operating thumbprint  70  matches the stored thumbprint  100 , then after a delay indicated by process block  150  this process is repeated so that any potential tampering or failure of the control devices  16  may be identified in near real-time. In the event that there is a mismatch between the received thumbprints  70  and the corresponding stored thumbprint  100  in the populated security table  92 , then the security-processing program  90 , at decision block  148 , proceeds to process block  154  and one of a number of different responses from response script  118  may be implemented. This detection may be in real time or may occur on a regularly or randomly scheduled basis. 
         [0085]    It is contemplated that the stored thumbprints  100  may also be subject to periodic comparison to other stored values, for example their values at an earlier time, as held in a second storage location to detect possible tampering with the stored thumbprint  100 . That is, a tracking of the history of the security thumbprints  100  may be performed and any mismatch detected in this tracking may also invoke a response according to process block  154 . 
         [0086]    Referring now to  FIG. 6 , in the event of a mismatch at decision block  148 , the security-processing program  90  will generally implement the response script  118  that may be stored in the populated security table  92 . This response script  118  may perform a number of different tasks including: generating notification reports per process block  156 , performing additional data collection per process block  158 , assessing a significance of the mismatch at process block  160 , and taking mitigating or defensive actions at process block  162 . Each particular step is optional and whether it will be performed is determined by the particular response script  118 . Each of the process blocks  156 - 162  may be repeated in a loop so that the response and analysis constantly evolves with additional information and possibly other changes in the system. 
         [0087]    The reporting of process block  156  may provide for notifications to individuals or groups in a notification tree  104  of  FIG. 3  per process block  164 . Referring momentarily to  FIG. 8 , in one embodiment, notification tree  104  may provide multiple entries each associated with a significance level  166  of the mismatch. Each significance level is linked to an acknowledgment level  169  and contact information  175 . The acknowledgment level  169  may indicate whether a contact individual has acknowledged receipt of that contact. Generally, the contact information  175  may be a network address, a human machine interface, and e-mail address, a mobile device contact number, or any of a variety of different methods of communicating a problem to individuals or groups of individuals and/or other devices including controllers  12  or factory indicators such as lights and beacons. 
         [0088]    The notifications, when to individuals, may be, for example, via e-mail messages or served as a web page and may provide, for example, a graphical display (shown in  FIG. 14 ) that indicates each of the functional elements  300  of the industrial control system  10  and its status with respect to errors in thumbprints  70 , severity of errors, the timing or sequence of errors, and mitigating actions, for example, by color. This information may also be displayed locally on a human machine interface or the like to provide an immediate snapshot of system security in the vicinity of the controlled equipment. 
         [0089]    For a first mismatch, at a first iteration of the loop of process blocks  156 - 162 , the significance level  166  will be zero because significance has not yet been determined at process block  160 . The context for this low significance level may be limited to individuals in charge of routine maintenance or the like or simply to a log file. For example, minor mismatches in thumbprints or sub-thumbprints may be reported only to technical individuals in charge of maintaining the system and may be indicated to be low priority whereas more significant mismatches may provide reports with urgent designations to fast responders and supervisors. As additional mismatches occur and as the loop is executed multiple times, the significance level  166  may rise and the particular contact information  175  identifying individuals to be contacted will change according to the significance of the mismatch and whether or not one or more parties has responded or acknowledged receipt of the notification. In one response script  118 , if no parties acknowledge receipt of the notification in a given period of time, the significance level  166  will rise so that additional contacts may be added or different people may be notified pending on the severity of the potential problem as will be discussed below. 
         [0090]    The reporting of process block  156  may also provide a system alert update being a globally available system security value that may be read by other security devices  31  to allow coordinated effort. This system alert update, indicated by process block  167  may provide information about the mismatch, including any detailed information of the mismatch components, it significance level  166  and possible additional steps being taken. As will be discussed below, the system alert status from other security devices  31  or generated by other control devices  16  in different response script  118  may also be considered with respect to setting the significance level  166  of a particular mismatch. By understanding multiple disparate mismatches, a more nuanced view of the significance of the local mismatch can be determined. 
         [0091]    At data collection process block  158 , additional data may be collected with respect to the mismatch signature typically driven by the significance level  166  but also driven by the type of mismatch. Most notably a finer-grained operating thumbprint  70  may be obtained (e.g., more sub-thumbprints  78 ), as indicated by process block  168 , based on identification of the coarse operating thumbprint  70 . Thus, for example, if the operating thumbprint  70  has very low granularity indicating only a mismatch in data of a collective group of data structures, the fine-grained data collection of process block  168  will provide for a more partitioned sub-thumbprint  78  so that the location of the particular mismatch may be better identified, for example, to a particular data structure or device. This escalation of the detail provided by the thumbprint allows a trade-off between knowledge about the specific problem and network overhead necessary to communicate the thumbprints to be flexibly set. 
         [0092]    At process block  170  of data collection process block  158 , transaction logs  52  may be collected to prevent loss or damage of those transaction logs  52  and to allow analysis of the transaction logs  52  such as may indicate a source of the error (for example, a given human operator making changes to the system). The transaction logs  52  may also inform possible mitigating steps, as will be discussed below, for example, locking out certain personnel from changing the software of the control devices  16 . At process block  173 , system significance level  166  may be read in order to gain an understanding of all possible control devices  16  experiencing signature mismatches (that have uploaded system alerts at process blocks  167 ) and to adjust the data collection level. 
         [0093]    The invention contemplates that some response scripts will operate in a “stealth” mode in which data is collected and possibly stored for a long period of time on activities that do not justify alarms or other notifications. This stealth mode satisfies the trade-off between avoiding frequent false alarms and notifications, while ensuring that long-term trends and minor deviations are nevertheless fully assessed and treated. Minor changes in system security may be automatically implemented in the stealth mode as well, of types provided by the discussed response scripts, but without necessary notifications. 
         [0094]    The data collected during the stealth mode may be separately analyzed, for example, over a longer time period so that a long-term, lower level of alert may eventually be escalated to a higher level simply because of the long-term nature of the detected anomaly, or because of additional information that can be evaluated from long-term data collection. For example, long-term trends or correlations (e.g., security issues associated with the particular individual&#39;s access to the equipment or in another pattern) can then be aggregated and reported or used to trigger higher level responses. 
         [0095]    The assessment of the significance of the mismatch is determined at process block  160  and allows tailoring of any response to mismatches in the thumbprints  70  to a derived severity. By assigning severity levels to any mismatch, false alarms may be reduced while rapidly escalating response, even for minor mismatches, when the type of mismatches indicates possible tampering or interference with operation of the control system  10 . Generally, the significance level  166  will derive from a number of factors that may be investigated at process block  160 . For example, at process block  174 , the location of the mismatch (for example, to a particular component of the control device  16 ) may be used to obtain a weight  114  described above indicating the abstract significance of the error. Thus for example, a disconnection of a wire conductor  36  providing information from a redundant sensor or to a actuator not critical for operation of the industrial control system  10  may have a low weight whereas substantial errors in the control program  46  or operating system  48  may be given higher weight. 
         [0096]    At process block  176 , the change in the system status (for example, derived from system alert update of process block  167  for multiple control devices  16 ) may be analyzed to see if the particular mismatch is part of a pattern of mismatches throughout the control system  10  and to analyze any trending of those mismatches so that mismatches that are part of a rising number of mismatches are given greater weight. The weight may be affected by the number of mismatches or the number of different structures exhibiting mismatches. Analysis of patterns of mismatches among different separated control devices  16  may be incorporated into the response script to identify particular changes that may individually look benign but together suggest more significance and a higher significance level  166 . 
         [0097]    At process block  178 , mitigation options are assessed to see if particular mismatches may be easily mitigated, for example, using redundant control devices  16  or using backup information that may be put into place by command from the security device  31 . If the mismatch may be mitigated, lower significance level  166  may be assigned. 
         [0098]    At process block  180 , the mismatches in current thumbprints  70  and stored thumbprint  100  are compared against any scheduled changes that have been preregistered with the security device  31 , for example, in a calendar-type application. The significance of mismatches that relate to changes that have been preregistered is generally assigned to a lower significance level  166 . Similarly unscheduled changes that occur while the control device  16  is in a configuration or maintenance mode (as set from the control panel of the control device  16 ) may be registered as less severe than when the same configuration changes are detected during runtime. In this way false positives may be reduced. 
         [0099]    Referring now also to  FIG. 7 , process block  160  of assessing the significance level  166  of a mismatch of current thumbprints  70  and stored thumbprint  100  may be implemented by simply summing the weights  114  of the thumbprint map  110  associated with each mismatch. Alternatively a calculation of significance level  166  may be implemented by a set of rules that provides for more sophisticated Boolean combinations of weights and other factors. Most generally, a significance matrix  182  may be developed to map multiple conditions  184  to particular significance levels  166 . Thus, for example, low significance (e.g., 0) may be mapped to conditions such as mismatched control program  46  that is nevertheless indicated to be authentic or occurring during a scheduled maintenance upgrades or a sub-thumbprint  78  that matches a previous thumbprint  108 . Similarly, a wire loss indicated to be on a low importance function may garner a low significance level  166 . A white list may be established indicating, for example, changes or change combinations that are generally benign, for example, expected patterns of changes in the hardware registers  50  may be mapped to low significance level  166 . Changes that occur during a low alert status of the system may be given a low significance level  166 . A low alert status may result from no or low numbers of mismatches or mismatches having low significance levels  166  at different control devices  16  or that occur on hardware that is redundant and thus can be readily mitigated, or when the occurrence of the mismatch has been acknowledgment by the contact individual with an indication that a high significance is not warranted, or should be overridden. In addition, particular input or output points identified to be important or leading indicators of a critical failure (or indicative of proper operations) may be received as inputs for the purpose of establishing an importance of other errors. 
         [0100]    Conversely mismatches caused by inauthentic control programs  46  or operating systems  48 , that also match no previous thumbprint  108 , that occur during unscheduled times, or that are caused by wire-off signals for critical functions may be given a high significance. Just as a white list may be established, a blacklist of configuration changes that are suspected, or have been predetermined to suggest tampering, may create a high significance level  166 . Changes that are individually benign or low significant but where the changes occur during in an environment of other high significance levels  166  or changes associated with a predetermined pattern of mismatches in other similar control devices  16  may also be promoted to a high significance level  166 . Clearly cases where there is no redundant hardware available and no response from individuals contacted as part of the reporting process block  156  may be given greater significance. 
         [0101]    Referring momentarily to  FIG. 13 , each or any one of the control devices  16  and controllers  12  may implement in firmware or software of the operating system  48  a change supervisor  190  that requires certain steps in order for the industrial controller  12  or control device  16  to be modified. These steps may be implemented on the control device  16  itself or on a proxy device designated as the gateway for such changes. The change supervisor  190  may monitor any request for a change in any of the components subject to the thumbprints  70  (e.g., the control program  46 , the firmware operating system  48 , and the configuration register  50 ) at decision block  192 . When a change is requested, an authorization may be requested of the individual seeking to make the change as indicated by process block  194 . This authorization may be a password or a multifactor authorization, for example, requiring password information and a physical key or the like. Ideally the authorization identifies a specific responsible individual. 
         [0102]    The received authorization may be compared against a list of authorized individuals and/or individual clearances at decision block  196 . If the authorization level is not sufficient as determined by decision block  196 , a report may be generated as indicated by process block  198  and this attempt recorded in the transaction log  52  as indicated by process block  200 . Otherwise the change may be implemented as indicated by process block  202  and again the change recorded in the logging process of process block  200 . 
         [0103]    The assessment of the significance level  166  of the mismatch determined at process block  160  is used to generate the reports at process block  156 , potentially suppressing broad dissemination of reports for minor matters while escalating reports for matters of higher significance level  166  as has been discussed. The significance level  166  of the mismatch may also drive the mitigation actions according to process block  162  as the process blocks  156 - 162  are looped through. 
         [0104]    Referring now to  FIG. 6 , the process block  162  performing a mitigating action in the event of a mismatch between the received thumbprints  70  and a stored thumbprint  100  may modify the change supervisor  190  as one possible mitigating action shown by process block  204 . Specifically, in the event of a mismatch, process block  204  may change or increase security levels needed for particular operations. For example, security levels for changes in the control program  46  or operating system  48  may be increased particularly in a situation where it appears that widescale tampering may be occurring. Particular individuals identified from the transaction logs  52  associated with a mismatch, as collected at process block  170 , may have their authorization revoked. Password values used for authentication may be reset requiring new passwords that may be issued under controlled circumstances. 
         [0105]    The mitigation step of process block  162  may also perform other actions. As indicated by process block  206 , operating modes of the control device  16  (e.g., run state versus programming state) may be locked down to prevent pending program changes from being implemented. 
         [0106]    Some types of mismatches may provoke instructions being sent, from the security device  31  to the control device  16  having a mismatch, where the instructions cause the control device  16  to move to a safe state and remain there. A safe state is a predetermined set of input and output values that are likely to be safe, that is to create no or minimized risk of harm to the equipment or users, and to minimize propagation of failure to other components of the control system  10 . The safety states may be predetermined defined in the standard security templates  120  discussed above. Such safe states may, for example, move equipment and the like into safe positions and may deactivate certain activities. 
         [0107]    Additional processes of the mitigation step of process block  162  may instruct the control device  16  to run the defense script  64  mentioned above which enlists the various sensors  20  and actuators  22  for defensive purpose. In one example, the defense script  64  may cause cameras associated with various control devices  16  to be activated to begin logging possibly suspicious activity in the area. Lighting control by control devices  16  may be turned on to reveal intrusions and the like and access gates intended for user safety, controlled by control devices  16 , may be locked to prevent access to the equipment or devices. 
         [0108]    As indicated by process block  209 , the mitigation step of process block  162  may also instruct the activation of redundant equipment that can serve the function of the compromised control devices  16 . Alternatively or in addition, the mitigation step may instruct the control devices  16  to prevent software updating or to provide local signals to operators in the area of the control device  16 , for example, through human machine interface elements such as panel lights, beacons, audio annunciators, or the like. 
         [0109]    Again each of these mitigation activities of process block  162  may be driven by a set of specifically drafted rules or more generally by the significance levels  166  determined above with respect to process block  160 . 
       Dynamic Data 
       [0110]    Referring now to  FIG. 9 , the above description involves obtaining signatures of data that is largely “static” (that is changing slowly or changing not at all during normal operation) or “quasi-static” (that is changing but having a state characterization that is largely static), for example, a dynamic variable that nevertheless typically stays within a predefined range. It is contemplated that the present invention may be expanded to “dynamic” data, for example, current I/O data from I/O table  42  which changes rapidly with operation of the control device  16 , network data from the network interface  55  including port numbers, packet counts, and the like as well as actual received packets, and processor data from the processor  44 , for example, processor utilization percentage, processor fault flags and the like. Again this data may be linked with a timestamp  79 , a digital signature  80 , a device identification number  71 , and/or a changing random code  83  to provide security in the transmission of a dynamic operating thumbprint  70 ′. 
         [0111]    This dynamic operating thumbprint  70 ′ cannot be easily compared against a static stored thumbprint but may nevertheless be compared against rules that, for example, establish ranges of values within which the operating thumbprint  70 ′ or the underlying data should vary, or correlations between values of the underlying data that can be used to detect a deviation from the normal pattern and excursions of these dynamic values. In this case, the stored thumbprint  100  described above may be replaced by more sophisticated dynamic signatures to otherwise provide the detection of mismatches used as has been described above. Referring now to  FIG. 12 , one method of implementing a dynamic stored thumbprint  100  makes use of a machine learning system  201  or the like. This machine learning system  201  may be trained, as is understood in this art, using a teaching set  205  of normal dynamic operating thumbprints  70 ′ together with an intentional corruption of those normal dynamic thumbprints  70 ′ or intentionally manufactured thumbprints implementing hypothetical tampering scenarios. After the machine learning system  201  is trained using the teaching set  205 , it then receives the actual dynamic thumbprints  70 ′ to produce an output  203  that may be used by decision block  148  of  FIG. 5 . 
         [0112]    The dynamic stored thumbprint  100  comprising either set of rules or a machine learning system may also be used for the analysis of static thumbprints  70 , for example, to analyze minor evolution in the otherwise static operating state that would be expected with an industrial control system (otherwise accommodated as upgrading or the like). 
         [0113]    At times, the rules of the dynamic stored thumbprints  100 ′ may be allowed to evolve within certain ranges so as to eliminate false positives caused by natural evolution of the state of the control system. This evolution may be provided, for example, by using historical data to create new training sets that are used to constantly update the dynamic stored thumbprints  100 ′. In this case, a second level of analysis of the dynamic stored thumbprints  100 ′ may be performed, for example, with a longer time frame, to evaluate that evolution of the dynamic stored thumbprints  100 ′ for possible underlying problems that may be detected to trigger a response script of process block  154  described above. 
         [0114]    The implicit rules of the dynamic stored thumbprints  100 ′ may also be randomly perturbed at the range thresholds to change the precise thresholds at which a response script of process block  154  is invoked. This randomization can help defeat “probing” of the dynamic stored thumbprints  100 ′, for example, on a separate industrial control system  10 , where the probing is used to collect information to defeat other industrial control systems  10 . The randomization may be performed, for example, by randomly selecting among different elements of a teaching set to provide slightly different teaching rules generated by a machine learning system  201 , or by randomly adjusting the thresholds of ranges of rules used to evaluate dynamic stored thumbprint  100 ′ by minor amounts that still ensure that the function of the ranges to test for out of range conditions are still substantially met. 
         [0115]    Referring to  FIG. 10 , the potentially large combinatorial space occupied by many dynamic variables can be managed in the present invention by providing a distributed security device  31  in which a mismatch per decision block  148  (of  FIG. 5 ) is analyzed for downstream devices by the next upstream device, limiting the propagation of the dynamic thumbprints  70 ′. To the extent that these dynamic thumbprints  70  cannot be otherwise compressed, this distribution to local analysis of the dynamic values, for example, range checking or the use of a local supervised machine learning system, may be used to convert the dynamic thumbprints  70 ′ into static or quasi-static thumbprints  70 ′ for conventional analysis at a security device  31  using the methods described above. The ability to accurately detect complex patterns in the data of the dynamic thumbprints  70 ′ can be promoted by transmitting the dynamic thumbprints  70 ′ together with context data, for example, a particular control task or local clock value related to the dynamic thumbprints  70 ′ that allows clustering of dynamic operating thumbprint  70 ′ into limited subsets that can be analyzed separately, for example, subsets related to temporal proximity, or subsets related to particular control tasks. 
         [0116]    Accordingly, a dynamic operating thumbprint  70   a ′ and a dynamic operating thumbprint  70   b ′ generated by control devices  16   a  and  16   b , respectively, associated with a given control task may be linked by a context established by context envelope  211  (C) encapsulating the dynamic thumbprints  70   a ′ and  70 W and transmitted with the thumbprints  70   a ′ and  70   b ′. The context envelope  211  may link the thumbprints  70   a ′ and  70   b ′ as relating to a common control task or similar local clock occurrences. This context envelope may be augmented as additional thumbprints  70   c ′ are passed up to the security device  31  so that eventually a dynamic operating thumbprint  70   d ′ with a context envelope  213  is received, this context envelope  213  collecting dynamic thumbprints  70   a ′ and  70   b ′ together in context envelope  211  (C) and connecting context envelope  211  (C) with operating thumbprint  70   c ′ by context envelope  213  (E). This hierarchy of context envelope  211  and  213  allows specialized rules to be applied to each separate context minimizing the complexity of the analysis process 
         [0117]    A similar approach may be used with static thumbprints  70  where upstream devices  215   b  (e.g. a controller  12 ) may aggregate static state thumbprints  70  from downstream devices  215   a  (e.g.  16 ) with the upstream devices  215   b  generating its own static thumbprints  70  being a digest of the received thumbprints  70  from the downstream devices  215   a . These new static thumbprints  70  are then forwarded further upstream to further upstream devices  215   c  and this process may be repeated. Preliminary matching of the thumbprint  72  to stored thumbprints  100  may occur at intermediary upstream devices  215   b  with only the results of those matches (per decision block  148  of  FIG. 5 ) being sent upstream to devices  215   c  with the provision that in the event of a mismatch or at any time, a higher-level security device  31  may request that the raw received thumbprints  70  may be passed through the intermediary devices  215   b  and  215   c  without digesting for analysis at a higher-level. This latter procedure helps prevent tampering with the security mechanism through attacks at intermediary devices  215   b  and  215   c.    
       Safety Monitoring 
       [0118]    The above description has been provided in a context of monitoring an industrial control system against malicious attacks. It will be appreciated that elements of the above system can also be used to detect irregularities in the operation of an industrial control system that do not necessarily result from malicious intent but that may nevertheless affect the integrity or safety of operation of the industrial control system. 
         [0119]    “Safety” as used herein refers to the operation of specialized industrial control systems (“safety systems”) used in environments where the safety of humans requires proper functioning of the control system. Safety systems may include the electronics associated with emergency-stop buttons, light curtains, and other machine lockouts. Traditionally, safety systems have been implemented by a set of redundant circuits separate from the industrial control system used to control the industrial process with which the safety system is associated. Such safety systems were “hardwired” from switches and relays including specialized “safety relays” which provide comparison of redundant signals and internal checking of fault conditions such as welded or stuck contacts. 
         [0120]    Current safety systems may be implemented using specialized computer hardware and network protocols for example as taught by U.S. Pat. Nos. 6,631,476; 6,701,198; 6,721,900; 6,891,850; and 6,909,923 all hereby incorporated by reference. U.S. Pat. No. 7,027,880, also hereby incorporated by reference and assigned to the assignees of the present invention, describes a safety system that uses a “signature” of the software executed by the safety system that can be compared to a signature of a previously certified version of the same software. This comparison process allows rapid re-certification (or determination of proper certification) of the safety system. The present invention may expand upon this concept by using the security signatures described above as safety signatures that provide a complete indication of changes in the industrial control system beyond merely changes in the operating software to also include changes in configuration data and environmental data which together a define control state of the industrial controller. In addition or alternatively, the aggregation of safety signatures from multiple elements of the control device allows for more comprehensive assurance of the integrity of a safety system comprised of multiple elements. As is also described above, the safety system may provide for diagnostics not normally present with safety systems by zeroing in on the cause of the fault to help correct this fault. This zeroing in is accomplished by obtaining increasingly detailed safety signatures in the manner discussed above. 
         [0121]    A failure of the safety signal from any element to match a corresponding stored signature associated with a safety certified state of the industrial control system may cause the system to send alerts to the appropriate personnel in the manner discussed above and also to move the system to a safe state as is also discussed above. 
         [0122]    Certain terminology is used herein for purposes of reference only, and thus is not intended to be limiting. For example, terms such as “upper”, “lower”, “above”, and “below” refer to directions in the drawings to which reference is made. Terms such as “front”, “back”, “rear”, “bottom” and “side”, describe the orientation of portions of the component within a consistent but arbitrary frame of reference which is made clear by reference to the text and the associated drawings describing the component under discussion. Such terminology may include the words specifically mentioned above, derivatives thereof, and words of similar import. Similarly, the terms “first”. “second” and other such numerical terms referring to structures do not imply a sequence or order unless clearly indicated by the context. 
         [0123]    When introducing elements or features of the present disclosure and the exemplary embodiments, the articles “a”, “an”, “the” and “said” are intended to mean that there are one or more of such elements or features. The terms “comprising”, “including” and “having” are intended to be inclusive and mean that there may be additional elements or features other than those specifically noted. It is further to be understood that the method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order discussed or illustrated, unless specifically identified as an order of performance. It is also to be understood that additional or alternative steps may be employed. 
         [0124]    References to “a controller”, “an industrial controller”, and “a computer”, should be understood to include any general computing device suitable for the recited function including workstations, industrial controllers, personal or desktop computers, servers, cloud computers and the like operating locally or remotely to other elements of the invention. 
         [0125]    References to “a microprocessor” and “a processor” or “the microprocessor” and “the processor.” should be understood to include one or more microprocessors that can communicate in a stand-alone and/or a distributed environment(s), and can thus be configured to communicate via wired or wireless communications with other processors, where such one or more processor can be configured to operate on one or more processor-controlled devices that can be similar or different devices. Furthermore, references to memory, unless otherwise specified, can include one or more processor-readable and accessible memory elements and/or components that can be internal to the processor-controlled device, external to the processor-controlled device, and can be accessed via a wired or wireless network. 
         [0126]    The term “network port” should not be construed as limited to particular types of networks or ports but is intended to broadly cover communications via wired and wireless ports, ports connecting to separate media such as cables and optical fibers as well as backplanes, and a variety of protocols including but not limited to RS-232/422, USB, IEEE1394, and 1756-EN2T protocols. 
         [0127]    It is specifically intended that the present invention not be limited, to the embodiments and illustrations contained herein and the claims should be understood to include modified forms of those embodiments including portions of the embodiments and combinations of elements of different embodiments as come within the scope of the following claims. All of the publications described herein, including patents and non-patent publications, are hereby incorporated herein by reference in their entireties.