Patent Publication Number: US-2021194955-A1

Title: Data exchange and processing synchronization in distributed systems

Description:
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
     This invention was made with U.S. Government support under Contract No. DE-0E000831 awarded by the Department of Energy. The Government has certain rights in this invention. 
    
    
     BACKGROUND 
     The present disclosure relates generally to distributed systems. Centrally computed algorithms are being increasingly substituted with distributed control algorithms whereby agent nodes of a distributed system each perform a portion of an algorithm while exchanging data between agent nodes. Conventional execution of these distributed control algorithms requires synchronous communication between agent nodes. However, distributed systems may use asynchronous communication networks between nodes. Alternatively, a synchronous communication network of a distributed system may experience asynchrony due to system failure or communication delay. There remain unmet needs including executing distributed control algorithms within distributed systems that communicate asynchronously. In view of these and other shortcomings in the art, there is a significant need for the unique apparatuses, methods, systems and techniques disclosed herein. 
     DISCLOSURE OF ILLUSTRATIVE EMBODIMENTS 
     For the purposes of clearly, concisely and exactly describing non-limiting exemplary embodiments of the disclosure, the manner and process of making and using the same, and to enable the practice, making and use of the same, reference will now be made to certain exemplary embodiments, including those illustrated in the figures, and specific language will be used to describe the same. It shall nevertheless be understood that no limitation of the scope of the present disclosure is thereby created, and that the present disclosure includes and protects such alterations, modifications, and further applications of the exemplary embodiments as would occur to one skilled in the art with the benefit of the present disclosure. 
     SUMMARY OF THE DISCLOSURE 
     Exemplary embodiments of the disclosure include unique systems, methods, techniques and apparatuses for synchronous communication in distribution systems. Further embodiments, forms, objects, features, advantages, aspects and benefits of the disclosure shall become apparent from the following description and drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  illustrates an exemplary microgrid system including an asynchronous communication network. 
         FIG. 2  is a flowchart illustrating an exemplary process for data exchange in an asynchronous communication network. 
         FIG. 3  is a sequence diagram illustrating the exemplary process illustrated in  FIG. 2 . 
         FIG. 4  is a flowchart illustrating another exemplary process for data exchange in an asynchronous communication network. 
         FIG. 5  is a sequence diagram illustrating the exemplary process illustrated in  FIG. 4   
         FIG. 6  is a set of graphs illustrating state estimation in the exemplary microgrid of  FIG. 1  based on the exemplary processes illustrated in  FIGS. 2 and 4 . 
         FIG. 7  is a table illustrating state estimation results in the exemplary microgrid of  FIG. 1  based on the exemplary processes illustrated in  FIGS. 2 and 4 . 
     
    
    
     DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS 
     With reference to  FIG. 1 , there is illustrated an exemplary networked microgrid system  100 . It shall be appreciated that system  100  may be implemented in a variety of industrial automation or power system applications including utility grid distribution systems, industrial plant distribution systems, and vehicular distribution systems, to name but a few examples. It shall be appreciated that the topology of system  100  is illustrated for the purpose of explanation and is not intended as a limitation of the present disclosure. Although system  100  is illustrated with a single line diagram, system  100  may be structured to transmit single phase alternating current (AC) power, multiphase AC power, or direct current (DC) power. 
     System  100  includes microgrids  110 ,  120 ,  130 ,  140  and  150 . Each microgrid includes at least one load and at least one power source coupled to one of a plurality of buses. Each microgrid includes a plurality of field devices structured to measure electrical or physical characteristics of the microgrid. For example, microgrid  150  includes a plurality of buses including bus  153 , a plurality of loads including load  155 , an AC power source  157 , and a plurality of field devices  152 . 
     AC power source  157  may include a solar panel array, a wind turbine, a natural gas generator, or any other device or system structured to generate power. Load  155  may be any type of device or system structured to consume power. The plurality of field devices  152  may include voltage sensors, intelligent electronic devices (IEDs), remote terminal units (RTUs), relays, reclosers, current sensors, voltage transformers, and current transformers, to name but a few examples. 
     Each microgrid also includes a microgrid control system, also known as an agent node, coupled to the plurality of field devices by way of a plurality of communication channels. For example, microgrid  150  includes microgrid control system  151 , which is coupled to the plurality of field devices  152  by way of a plurality of communication channels. 
     Each microgrid control system includes an input/output device, a processing device, and a memory device. For example, microgrid control system  121  includes an input/output device  123 , a processing device  125 , and a memory device  127 . Microgrid control system  121  may be a stand-alone device, an embedded system, or a plurality of devices structured to perform the functions described with respect to system  121 . For example, system  121  may be incorporated into a microgrid supervisory control and data acquisition (SCADA) gateway. 
     Input/output device  123  enables microgrid control system  121  to communicate with local field devices or other microgrids. Input/output device  123  may include a network adapter, network credential, interface, or a port (e.g., a USB port, serial port, parallel port, an analog port, a digital port, VGA, DVI, HDMI, FireWire, CAT 5, Ethernet, fiber, or any other type of port or interface), to name but a few examples. Input/output device  123  may include more than one of these adapters, credentials, or ports, such as a first port for receiving data and a second port for transmitting data. 
     Processing device  125  may include one or multiple processors, Arithmetic-Logic Units (ALUs), Central Processing Units (CPUs), Digital Signal Processors (DSPs), or Field-programmable Gate Arrays (FPGAs), to name but a few examples. For forms of processing devices with multiple processing units, distributed, pipelined, or parallel processing may be used. Processing device  125  may be dedicated to performance of only the operations described herein or may be used in one or more additional applications. Processing device  125  may be of a programmable variety that executes algorithms and processes data in accordance with operating logic  808  as defined by programming instructions (such as software or firmware) stored in memory  127 . Alternatively or additionally, operating logic for processing device  125  is at least partially defined by hardwired logic or other hardware. Processing device  125  may comprise one or more components of any type suitable to process the signals received from input/output device  123  or elsewhere, and provide desired output signals. Such components may include digital circuitry, analog circuitry, or a combination of both. 
     Memory device  127 , also known as a computer readable medium, may be of one or more types of memory, such as a solid-state variety, electromagnetic variety, optical variety, or a combination of these forms, to name but a few examples. Furthermore, memory device  127  may be volatile, nonvolatile, transitory, non-transitory, or a combination of these types, and some or all of memory device  127  may be of a portable variety, such as a disk, tape, memory stick, or cartridge, to name but a few examples. In addition, memory device  127  may store data that is manipulated by operating logic of processing device  125 , such as data representative of signals received from and/or sent to input/output device  123  in addition to or in lieu of storing programming instructions defining operating logic, just to name one example. Memory device  127  may be included with processing device  125  and/or coupled to processing device  125 . It shall be appreciated that the other microgrid control systems of system  100  may also include components described herein with respect to microgrid control system  121 . 
     Microgrid control systems  111 ,  121 ,  131 ,  141 , and  151  are structured to communicate by way of asynchronous communication network  160  including communication channels coupled with one or more neighboring microgrid control systems. Network  160  may communicate asynchronously because the microgrid control systems do not synchronize to a common time source, such as a GPS time signal, or because network  160  is a synchronized communication network experiencing communication delays. In the illustrated embodiment, microgrid control system  111  communicates with microgrid control systems  121  and  131 ; microgrid control system  121  communicates only with microgrid control system  111 ; microgrid control system  131  communicates with microgrid control systems  111 ,  141 , and  151 ; microgrid control system  141  communicates only with microgrid control system  131 ; and microgrid control system  151  communicates only with microgrid control system  131 . 
     The communication channels of network  160  between the microgrid control systems may be wired or wireless. Each communication channel of network  160  may use standardized communication protocols, such as IEC 61850, OPC UA, Pub-Sub, XMPP, or DDS, to name but a few examples. In certain embodiments where standardized communication protocols are insufficient to define behavior of microgrids for distributed state estimation, semantics of the exchanged messages may be defined using common knowledge representations. 
     For example, the Foundation for Intelligent Physical Agents (FIPA) standard implements Agent Communication Language (ACL) to describe the semantics of exchanged messages. The content of an ACL message includes the object of the action and parameters passed through the message. Sender and receiver parameters designate the name of the sender and intended recipient agents. ACL messages express communicative acts such as informing, requesting, refusing, subscribing, and propagating, to name but a few examples. IEC 61850 communication protocols may be extended using non-standardized IEC 61850 logical nodes as Abstract Communication Service Interface (ACSI) based on ACL communication acts, and can be mapped to GOOSE messages. 
     Each microgrid control system is structured to receive measured electrical and physical characteristics of the corresponding microgrid from the field devices of the microgrid, operate controllable devices of the microgrid, and protect the devices coupled to the microgrid from faults and other conditions. The microgrid control systems, collectively, are structured to execute at least one distributed control application by recursively determining iterations and sharing the determined iterations with other microgrid control systems using the asynchronous communication network  160 . 
     For the purposes of illustration, a distributed control application for estimating states in system  100  shall now be described. State estimation is a process whereby electrical or physical characteristics of all microgrids in networked microgrid system  100  are estimated using input data related to the electrical or physical characteristics of the networked microgrid system to be estimated. The state estimation may include estimating the bus voltages for all buses in networked microgrid system  100 , as well as estimating current flow through all microgrids in networked microgrid system  100 . 
     First, each microgrid control system determines a first iteration using local data from the field devices of the corresponding microgrid. The first iteration may be determined using recursive least squares, to name but one example. Vector set (1) includes a local state estimation vector for every microgrid of system  100 , each vector including an initial voltage and phase angle x 1   (n) (0) estimate for each bus in the microgrid. At the 0 th  iteration, each microgrid sets the initial estimate vector X MGn (0). 
         x   MG     110     T (0)=[ x   1   (n) (0) x   2   (n) (0) x   3   (n) (0) x   4   (n) (0) x   5   (n) (0) x   6   (n) (0) x   7   (n) (0) x   8   (n) (0) x   9   (n) (0)] 
         x   MG     120     T (0)=[ x   1   (n) (0) x   2   (n) (0) x   3   (n) (0) x   4   (n) (0) x   5   (n) (0) x   6   (n) (0)] 
         x   MG     130     T (0)=[ x   1   (n) (0) x   2   (n) (0) x   3   (n) (0) x   4   (n) (0) x   5   (n) (0) x   6   (n) (0) x   7   (n) (0)] 
         x   MG     140     T (0)=[ x   1   (n) (0) x   2   (n) (0) x   3   (n) (0) x   4   (n) (0) x   5   (n) (0) x   6   (n) (0)] 
         x   MG     150     T (0)=[ x   1   (n) (0) x   2   (n) (0) x   3   (n) (0) x   4   (n) (0) x   5   (n) (0) x   6   (n) (0) x   7   (n) (0) x   8   (n) (0) x   9   (n) (0)]  (1)
 
     Next, each microgrid control system transmits its first iteration to the other microgrid control systems while receiving all the first iterations determined by the other microgrid control systems. Using the set of first iterations, each microgrid control system combines the first iterations into a global estimate vector X MGglobal  using vector set (1) 
         x   MGglobal (0)=[ x   MG     110     T (0) x   MG     120     T (0) x   MG     130     T (0) x   MG     140     T (0) x   MG     150     T (0)] T   (2)
 
     Using the global estimate vector x MGglobal , the microgrid control systems of the networked microgrid system use a diffusion strategy and peer-to-peer iterative communication to perform state estimation. The diffusion algorithm considers a collection of N microgrid control systems estimating the same parameter vector, ω 0 , size of N×1. Diffusion algorithms employ two types of strategies: Combine-then-Adapt (CTA) and Adapt-then-Combine (ATC). ATC can be summarized as follows: At every iteration instant i, ATC strategy performs two steps. The first step is an information exchange step where microgrid control system k receives from its neighbors their moments {R u,l ,r du,l }. Microgrid control system k combines this information and uses it to update its existing estimate ω k,i−1  to an intermediate value x MGglobal     k,i   . All other microgrid control systems in the networked microgrids system are performing a similar step and updating their estimates {ω k,i−1 } into intermediate estimates {x MGglobal     k,i   } by using information from their neighbors. The second step is an aggregation step where microgrid control system k combines the intermediate estimates of its neighbors to obtain its update estimate ω k,i . 
     Each iteration determined by agent microgrid control system k is determined using equation set (3) where X MGglobal     k,i   ∈  denotes the state of agent microgrid control system k at iteration i, ω k,i  is the intermediate variable for agent k at iteration i, μ k  is a nonnegative updating parameter, also known as a step size, of agent microgrid control system k, and ∇ ω J l (ω k,i−1 ) is the stochastic gradient for agent microgrid control system k of the intermediate state ω at iteration i. 
     
       
         
           
             
               
                 
                   
                     ATC 
                     
                       MG 
                        
                       _ 
                        
                       State 
                     
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             
                               x 
                               
                                 MGglobal 
                                 
                                   k 
                                   , 
                                   i 
                                 
                               
                             
                             = 
                             
                               
                                 ω 
                                 
                                   k 
                                   , 
                                   
                                     i 
                                     - 
                                     1 
                                   
                                 
                               
                               - 
                               
                                 
                                   μ 
                                   k 
                                 
                                  
                                 
                                   
                                     ∇ 
                                     ω 
                                   
                                    
                                   
                                     
                                       J 
                                       l 
                                     
                                      
                                     
                                       ( 
                                       
                                         ω 
                                         
                                           k 
                                           , 
                                           
                                             i 
                                             - 
                                             1 
                                           
                                         
                                       
                                       ) 
                                     
                                   
                                 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               ω 
                               
                                 k 
                                 , 
                                 i 
                               
                             
                             = 
                             
                               
                                 Σ 
                                 
                                   l 
                                   ∈ 
                                   
                                     N 
                                     k 
                                   
                                 
                               
                                
                               
                                 a 
                                 kl 
                               
                                
                               
                                 x 
                                 
                                   MGglobal 
                                   
                                     k 
                                     , 
                                     
                                       i 
                                       - 
                                       1 
                                     
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     In certain embodiments, μ k  is 0.2. In other embodiments, μ k  is within a range from 0 to 1. The stochastic gradient is the difference of co from one iteration to the other. Each microgrid control system keeps track of its gradient and updates the combination of all of its neighbors accordingly. 
     Coefficient a kl  of trust matrix A is a trust factor of a that each microgrid control system places on the other. The coefficients do not need to be symmetric, i.e., a kl ≠a lk , but each coefficient must be greater than or equal to zero, the sum of trust factors assigned by one microgrid control system to neighbors must equal one, and the trust factor assigned to a microgrid control system which is not a neighbor must be zero. 
         a   kl ≥0, Σ k=1   n   a   kl= 1 and  a   kl =0, if  l⊂N   k   (4)
 
     Trust factors may be assigned such that a trust factor between an agent microgrid control system and its neighbor is equal to the inverse of the maximum of the number of connected microgrid control systems to either the agent microgrid control system or the neighbor microgrid control system, and a trust factor between a microgrid control system and the microgrid control system itself is equal to one minus the sum of the trust factors between the microgrid control system and its neighbors. The equation set (5) illustrates the described assignment, where n k  is the number of connected microgrid control systems of agent microgrid control system k including microgrid control system k itself, n l  is the number of connected microgrid control systems of neighbor microgrid control system l including microgrid control system l itself, and N l  is the neighboring nodes of neighbor l. 
     
       
         
           
             
               
                 
                   
                     a 
                     kl 
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             
                               1 
                               
                                 max 
                                  
                                 
                                   ( 
                                   
                                     
                                       n 
                                       k 
                                     
                                     , 
                                     
                                       n 
                                       l 
                                     
                                   
                                   ) 
                                 
                               
                             
                             , 
                             
                               k 
                               ∈ 
                               
                                 
                                   N 
                                   l 
                                 
                                  
                                 \ 
                                  
                                 
                                   { 
                                   l 
                                   } 
                                 
                               
                             
                           
                         
                       
                       
                         
                           
                             
                               1 
                               - 
                               
                                 
                                   Σ 
                                   
                                     k 
                                     ∈ 
                                     
                                       
                                         N 
                                         l 
                                       
                                        
                                       \ 
                                        
                                       
                                         { 
                                         l 
                                         } 
                                       
                                     
                                   
                                 
                                  
                                 
                                   a 
                                   kl 
                                 
                               
                             
                             , 
                             
                               k 
                               = 
                               l 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
           
         
       
     
     In other words, the trust factor from k to l depends on the maximum number of neighbors either microgrid control system has, and the self-weight brings the total of each row (and column) in A to one. The trust matrix may be recalculated in response to a change in system topology, but otherwise remains static. 
     Using equation set (5), each microgrid control system calculates a trust matrix. The following matrices show the topology matrix of system  100  being converted into a trust matrix using equation set (5). 
     
       
         
           
             
               
                 
                   
                      
                     
                       
                         
                           1 
                         
                         
                           1 
                         
                         
                           1 
                         
                         
                           0 
                         
                         
                           0 
                         
                       
                       
                         
                           1 
                         
                         
                           1 
                         
                         
                           0 
                         
                         
                           0 
                         
                         
                           0 
                         
                       
                       
                         
                           1 
                         
                         
                           0 
                         
                         
                           1 
                         
                         
                           1 
                         
                         
                           1 
                         
                       
                       
                         
                           0 
                         
                         
                           0 
                         
                         
                           1 
                         
                         
                           1 
                         
                         
                           0 
                         
                       
                       
                         
                           0 
                         
                         
                           0 
                         
                         
                           1 
                         
                         
                           0 
                         
                         
                           1 
                         
                       
                     
                      
                   
                   → 
                   
                      
                     
                       
                         
                           0.4167 
                         
                         
                           0.333 
                         
                         
                           0.2500 
                         
                         
                           0.0000 
                         
                         
                           0.0000 
                         
                       
                       
                         
                           0.3333 
                         
                         
                           0.6667 
                         
                         
                           0.0000 
                         
                         
                           0.0000 
                         
                         
                           0.0000 
                         
                       
                       
                         
                           0.2500 
                         
                         
                           0.0000 
                         
                         
                           0.2500 
                         
                         
                           0.2500 
                         
                         
                           0.2500 
                         
                       
                       
                         
                           0.0000 
                         
                         
                           0.0000 
                         
                         
                           0.2500 
                         
                         
                           0.7500 
                         
                         
                           0.0000 
                         
                       
                       
                         
                           0.0000 
                         
                         
                           0.0000 
                         
                         
                           0.2500 
                         
                         
                           0.0000 
                         
                         
                           0.7500 
                         
                       
                     
                      
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
     
     After determining each new iteration, the microgrid control systems transmit their new iteration to the other microgrid control systems and receive the new iterations from each of the other microgrid control systems. The microgrid control system compares its own new iteration with the new iterations determined by the other microgrid control systems to determine whether the most recent set of iterations has converged, that is the standard deviation of the iterations is below a threshold. For example, the threshold may be, σ&lt;0.001 or σ&lt;0.1, to name but a few examples. The microgrid control systems continue to determine new iterations of the state estimation until the final iteration converges. The final iteration, which is the completed state estimation, may be used for market simulations, stability analysis, and demand response, to name but a few examples. 
     With reference to  FIG. 2 , there is illustrated a process  200  for exemplary data exchange between agent nodes during the execution of a distributed application using an asynchronous communication network. For example, where the distributed application is a distributed state estimation application using a recursive algorithm such as weighted least squares, iterations determined by process  200  may be iterations of a state estimation that will be recursively determined until the iterations determined by each agent node converge. 
     It shall be appreciated that while process  200  is directed to steps taken by one agent node of a distribution system, each of the agent nodes in the distribution system executes process  200  in order to execute the distribution application. It shall be further appreciated that a number of variations and modifications to process  200  are contemplated, including, for example, the omission of one or more aspects of process  200 , the addition of further conditionals and operations, and/or the reorganization or separation of operations and conditionals into separate processes. 
     Process  200  begins at operation  201  where the agent node determines a first iteration using data not received from other agent nodes. The first iteration may be determined using measurements received from measuring devices in direct communication with the agent node. The iteration is one iteration of a first set of iterations that are generated by the agent nodes of the distribution system. 
     Process  200  proceeds to operation  203  where the agent node determines a local vector clock by either generating or updating the local vector clock. The local vector clock includes a plurality of elements, wherein one element corresponds to the number of iterations determined by the agent node and the other elements correspond to the number of received iterations determined by each of the other agent nodes of the distribution system. For example, the local vector clock for the agent node may be represented by vector V k  where element V k [k] is the number of iterations that have been determined by the agent node and element V k [l] is the number of iterations received by the agent node and determined by another agent node of the distribution system, agent node l. Every time the agent node determines a new iteration, V k [k] is incremented by one. Every time the agent node receives an iteration from agent node l, V k [/] is incremented by one. 
     Process  200  proceeds to operation  205  where the agent node transmits the first iteration and the local vector clock to one or more neighboring agent nodes of the distribution system. Neighboring agent nodes are agent nodes of the distribution system in direct communication with the agent node. 
     Process  200  proceeds to operation  207  where the agent node receives an iteration of the first set of iterations determined by agent node l and a remote vector clock determined by agent node l. In certain embodiments, the agent node may receive more than one iteration of the first set of iterations from agent node l and a remote vector clock reflecting the multiple iterations being received by the agent node. 
     Process  200  proceeds to operation  209 , where the agent node updates the local vector clock based on the received remote vector clock. The agent node may update the local vector clock using the following equation: 
         V   k [ l ]=max( V   k [ l ], V   l [ l ])  (7)
 
     Process  200  proceeds to conditional  211  where the agent node determines whether the agent node has received all iterations of the first set of iterations from the other agent nodes based on the updated vector clock. For example, the agent node may determine all iterations of the first set of iterations have been received by determining each element of V k  is at least one. If the agent node has received all iterations of the first set of iterations, process  200  proceeds to conditional  215 . If the agent node has not received all iterations of the first set of iterations, process  200  proceeds to operation  213 . 
     During operation  213 , the agent node transmits the iteration or iterations received in operation  207  and the local vector clock updated in operation  209  to one or more neighboring agent nodes. Process  200  then proceeds to operation  207 . Process  200  continues in a loop of operation  207 , operation  209 , conditional  211 , and operation  213  until the agent node receives all the iterations of the first set of iterations. 
     At conditional  215 , the agent node determines whether the most recently determined iteration by the agent node is a final iteration. For example, an agent node may need to determine if the most recently completed set of iterations has not converged. If the agent node needs to determine a new iteration, process  200  proceeds to operation  217 . If the agent node does not need to determine a new iteration, process  200  proceeds to operation  219 . 
     During operation  217 , the agent node determines a new iteration based on the first set of iterations in response to determining the most recently determined iteration by the agent node is not a final iteration. The agent node also updates the local vector in response to determining the new iteration. The new iteration is an iteration of a second set of iterations determined by the agent nodes of the distribution system. In certain embodiments, the agent node determines the new iteration before at least one other agent node has received all iterations of the first set of iterations. 
     Process  200  proceeds to operation  218  where the agent node transmits the new iteration and updated local vector clock from operation  217  to one or more neighboring agent nodes. Process  200  then returns to operation  207 . Process  200  continues to receive iterations from other agent nodes, updating the vector clock, and determining new iterations based on the updated vector clock until the agent node determines the most recently determined new iteration is the final iteration. 
     During operation  219 , the agent node transmits the final iteration to local control applications in order to operate the distribution system based on the final iteration. For example, where the final iteration is a microgrid state estimation, the agent node may change controllable device set points based on the final iteration. 
     With reference to  FIG. 3 , there is a sequence diagram  300  illustrating the exemplary data exchange process  200  in  FIG. 2  executed by system  100  in  FIG. 1 . As shown in diagram  300 , each agent node determines three iterations. While the first iterations for each agent node are determined simultaneously, the second and third iterations are determined by the agent nodes at different times. Using asynchronous communication network  160 , each agent node receives a complete set of iterations at different times, determines the new iteration using the complete set of iterations, and then transmits the new iteration with an updated vector clock to the other agent nodes. 
     With reference to  FIG. 4 , there is illustrated a process  400  for exemplary data exchange between agent nodes during the execution of a distributed application using an asynchronous communication network. For example, where the distributed application is a distributed state estimation application using a recursive algorithm such as weighted least squares, iterations determined by process  400  may be iterations of a state estimation that will be recursively determined until the iterations determined by each agent node converge. 
     It shall be appreciated that while process  400  is directed to steps taken by one agent node of a distribution system, each of the agent nodes in the distribution system executes process  400  in order to execute the distribution application. It shall be further appreciated that a number of variations and modifications to process  400  are contemplated, including, for example, the omission of one or more aspects of process  400 , the addition of further conditionals and operations, and/or the reorganization or separation of operations and conditionals into separate processes. 
     Unlike process  200 , process  400  does not wait for an agent node to receive all iterations of a set of iterations before determining a new iteration. Instead, as described in more detail below, each agent node determines a new iteration using the iterations received during a ticker period preceding the time for determining the new iteration. 
     Process  400  begins at operation  401  where the agent node determines a trust matrix A including a plurality of trust factors, each trust factor corresponding to a weight applied to iterations of the plurality of iterations determined by one of the plurality of agent nodes. The weights may be determined using equation set (5), to name but one example. 
     Process  400  proceeds to operation  403  where the agent node determines a first iteration. The agent node may determine the first iteration using local data or measurements that are not transmitted through another agent node. For example, the agent node may determine the first iteration using the following equation set, where ψ k,i ∈  denotes the state of agent microgrid control system k at iteration i, ω k,i  is the intermediate variable for agent k at iteration i, μ k  is a nonnegative step size of agent microgrid control system k, a lk  is the trust factor for iterations determined by agent node l and received by agent node k, and ∇ ω J l (ω k,i−1 ) is the stochastic gradient for agent microgrid control system k of the intermediate state ω at iteration i: 
       ψ k,i =ω k,i−1 −μ k Σ l∈N     k   [∇ ω   J   l (ω k,i−1 )]*
 
       ω k,i =Σ l∈N     k     a   lk ψ l,i   (8)
 
     Process  400  proceeds to operation  405  where the agent node receives one or more iterations from one or more of the other agent nodes during a first ticker period. The first ticker period begins after the first iteration and ends before the beginning of operation  407 . 
     Process  400  proceeds to operation  407  where the agent node updates the trust matrix based on the iterations received during operation  405 . To update the trust matrix, the agent node decreases any weight applied to iterations received from an agent node that did not transmit an iteration to the agent node during the ticker period. Updating the trust matrix may also include increasing the weight applied to iterations determined by the agent node in an amount equal to the amount any weights were decreased. For example, the trust matrix may be updated using the following equation set: 
     
       
         
           
             
               
                 
                   
                     a 
                     lk 
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             
                               
                                 
                                   
                                     if 
                                      
                                     
                                         
                                     
                                      
                                     
                                       ( 
                                       
                                         
                                           ψ 
                                           
                                             l 
                                             , 
                                             i 
                                           
                                         
                                         ≠ 
                                         
                                           ψ 
                                           
                                             l 
                                             , 
                                             
                                               i 
                                               - 
                                               1 
                                             
                                           
                                         
                                       
                                       ) 
                                     
                                   
                                   &amp; 
                                 
                                  
                                 
                                     
                                 
                                  
                                 
                                   ( 
                                   
                                     
                                       ψ 
                                       
                                         l 
                                         , 
                                         i 
                                       
                                     
                                     ≠ 
                                     0 
                                   
                                   ) 
                                 
                               
                             
                             
                               
                                 
                                   a 
                                   lk 
                                 
                                 &gt; 
                                 0 
                               
                             
                           
                           
                             
                               otherwise 
                             
                             
                               
                                 
                                   a 
                                   lk 
                                 
                                 = 
                                 0 
                               
                             
                           
                         
                          
                         
                           
 
                         
                          
                         
                           a 
                           kk 
                         
                       
                       = 
                       
                         1 
                         - 
                         
                           
                             Σ 
                             
                               l 
                               ∈ 
                               
                                 
                                   N 
                                   k 
                                 
                                  
                                 \ 
                                  
                                 
                                   { 
                                   k 
                                   } 
                                 
                               
                             
                           
                            
                           
                             a 
                             lk 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
           
         
       
     
     Process  400  proceeds to operation  408  where the agent node updates the step size based on the iterations received during the ticker period in operation  403 . To update the step size, the agent node decreases any step size applied to iterations received from an agent node that did not transmit an iteration to the agent node during the ticker period. For example, the step size may be updated using the following equation set: 
     
       
         
           
             
               
                 
                   
                     a 
                     lk 
                   
                   = 
                   
                     { 
                     
                       
                         
                           
                             
                               
                                 if 
                                  
                                 
                                     
                                 
                                  
                                 
                                   ( 
                                   
                                     
                                       ψ 
                                       
                                         l 
                                         , 
                                         i 
                                       
                                     
                                     ≠ 
                                     
                                       ψ 
                                       
                                         l 
                                         , 
                                         
                                           i 
                                           - 
                                           1 
                                         
                                       
                                     
                                   
                                   ) 
                                 
                               
                               &amp; 
                             
                              
                             
                                 
                             
                              
                             
                               ( 
                               
                                 
                                   ψ 
                                   
                                     l 
                                     , 
                                     i 
                                   
                                 
                                 ≠ 
                                 0 
                               
                               ) 
                             
                           
                         
                         
                           
                             
                               μ 
                               k 
                             
                             &gt; 
                             0 
                           
                         
                       
                       
                         
                           otherwise 
                         
                         
                           
                             
                               μ 
                               k 
                             
                             = 
                             0 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
     Process  400  proceeds to operation  409  where the agent node determines a second iteration after the ticker period based on the iterations received during operation  405  and the updated trust matrix from operation  407 . In certain embodiments, the agent node determines a second iteration based in part on the step size updated in operation  408 . For example, the agent node may determine the second iteration using equation set (8). 
     Process  400  proceeds to conditional  411  where the agent node determines if the most recently determined iteration is the final iteration. For example, the agent node compares its own new iteration with other iterations received from other agent nodes to determine whether they have converged, that is the standard deviation of the iterations is below a threshold. For example, the threshold may be, σ&lt;0.001 or σ&lt;0.1, to name but a few examples. In certain embodiments, the agent node may determine the most recent iteration is the final iteration by determining whether the agent node has completed a fixed number of iterations. 
     If the most recently determined iteration is not the final iteration, process  400  proceeds to operation  405  and process  400  repeatedly receives new iterations during a new ticker period, updates the trust matrix based on the new iterations, and determines a new iteration using the iterations received during the new ticker period and the updated trust matrix until the agent node determines the most recent iteration is the final iteration. 
     During operation  413 , the agent node transmits the final iteration to local control applications in order to operate the distribution system based on the final iteration. For example, where the final iteration is a microgrid state estimation, the agent node may change controllable device set points based on the final iteration. 
     With reference to  FIG. 5 , there is a sequence diagram  500  illustrating exemplary data exchange process  400  in  FIG. 4  executed by system  100  in  FIG. 1 . Diagram  500  illustrates a series of iterations performed by each agent node separated by a plurality of ticker periods. For example, agent node  121  determines iterations  501  and  503  separated in time by ticker period  505 . As illustrated in diagram  500 , the plurality of ticker periods for each agent node may be an equal length in time. In other embodiments, the plurality of ticker periods for an agent node may include ticker periods of varying lengths in time. As illustrated in diagram  500 , the ticker periods for agent nodes  121 ,  131 , and  141  are different lengths in time compared to the ticker periods for agent node  111 . 
     With reference to  FIG. 6 , there is a plurality of graphs  600  including graph  610  and graph  620 . Graph  610  illustrates the state estimation of bus  153  in system  100  using the exemplary data exchange process  200  in  FIG. 2 . Graph  610  includes a plurality of estimation lines  611 , each line representing the values of each iteration determined by one agent node. By the 15 th  iteration determined by each agent node, the state estimation has converged. 
     Graph  620  illustrates the state estimation of bus  153  in system  100  using the exemplary data exchange process  400  in  FIG. 4 . Graph  620  includes a plurality of estimation lines  621 , each line representing the values of each iteration determined by one agent node. By the 50 th  iteration determined by each agent node, the state estimation has converged. It is important to note that while process  400  requires more iterations to converge, the number of agent nodes and the ticker period lengths may be structured such that process  400  takes less time to converge than process  200 . As the number of agent nodes increases, process  200  takes more time to converge. Therefore, it is advantageous to use process  200  for smaller sets of agent nodes and process  400  for larger sets of agent nodes. 
     With reference to  FIG. 7 , there is a table illustrating microgrid state estimation results using node intercommunication based on exemplary process  200  and exemplary process  400 . Each row of the table includes estimation and actual values of the voltage magnitude and phase angle of a bus in a microgrid. Despite performing state estimation using an asynchronous communication network, the states estimation results of process  200  and process  400  correspond to the actual values being estimated by the state estimation. 
     Further written description of a number of exemplary embodiments shall now be provided. One embodiment is a method comprising: determining, with a plurality of agent nodes structured to communicate asynchronously in a distributed system, a first set of iterations including an iteration determined by each of the plurality of agent nodes; determining, with a first agent node of the plurality of agent nodes, a local vector clock; receiving, with the first agent node, a first iteration of the first set of iterations and a remote vector clock determined based on the first iteration; updating, with the first agent node, the local vector clock based on the received remote vector clock; and determining, with the first agent node, a first iteration of a second set of iterations based on the first set of iterations after determining all iterations of the first set of iterations have been received based on the local vector clock. 
     In certain forms of the foregoing method, the local vector clock includes a plurality of elements, wherein a first element of the local vector clock corresponds to a number of iterations determined with the first agent node, and wherein each of the remaining elements of the local vector clock corresponds to a number of received iterations determined by one of the other agent nodes. In certain forms, the method further comprises transmitting, with the first agent node, the first iteration of the first set of iterations and the updated local vector clock in response to determining all iterations of the first set of iterations determined by the other agents nodes have not been received. In certain forms, determining, with the first agent node, the first iteration of the second set of iterations occurs in response to the first agent node determining the first iteration is not a final iteration, and wherein the method comprises updating the local vector clock based on the first iteration of the second set of iterations, and transmitting, with the first agent node, the first iteration of the second set of iterations and the local vector clock updated based on the first iteration of the second set of iterations. In certain forms, the first iteration of the second set of iterations is determined by the first agent node before at least one agent node of the plurality of agent nodes has received all iterations of the first set of iterations. In certain forms, the plurality of agent nodes is structured to communicate without being synchronized to a common time source. In certain forms, the method further comprises repeatedly receiving iterations, updating the local vector clock, and determining a new iteration based on the local vector clock until the first agent node determines the most recently determined iteration is a final iteration. In certain forms, the plurality of agent nodes are microgrid controllers, wherein the final iteration is a state estimation, and wherein the method comprises operating, with the first agent node, a controllable device of a microgrid based on the final iteration. In certain forms, the plurality of agent nodes are industrial automation controllers, wherein the final iteration is a state estimation, and wherein the method comprises operating, with the first agent node, a controllable device of an industrial automation system based on the final iteration. 
     Another exemplary embodiment is a distributed system comprising: a plurality of agent nodes including a first agent node; and a communication network structured to allow the plurality of agent nodes to communicate asynchronously; wherein the plurality of agent nodes each include a set of instructions stored on a memory device which, when executed by a processing device of the corresponding agent node, is effective to determine a first set of iterations including an iteration determined by each of the plurality of agent nodes, wherein the first agent node is structured to execute the set of instructions stored on the memory device of the first agent node with the processing device of the first agent node effective to: determine a local vector clock; receive a first iteration of the first set of iterations and a remote vector clock determined based on the first iteration; update the local vector clock based on the received remote vector clock; and determine a first iteration of a second set of iterations based on the first set of iterations after determining all iterations of the first set of iterations have been received based on the local vector clock. 
     Another exemplary embodiment is a method comprising: operating a distributed system including a first agent node and a plurality of agent nodes structured to communicate asynchronously; determining, with each agent node of the plurality of agent nodes, a plurality of iterations separated in time by a plurality of ticker periods; determining, with the first agent node, a trust matrix including a plurality of trust factors, each trust factor corresponding to a weight applied to iterations of the plurality of iterations determined by one of the plurality of agent nodes; determining, with the first agent node, a first iteration; receiving, with the first agent node, a portion of the plurality of iterations determined by a portion of the plurality of agent nodes during a first ticker period beginning after the first iteration; updating, with the first agent node, the trust matrix based on the portion of the plurality of iterations; and determining, with the first agent node, a second iteration after the first ticker period based on the portion of the plurality of iterations and the updated trust matrix. 
     In certain forms of the foregoing method, the method further comprises determining, with the first agent node, a second plurality of iterations including the first iteration and the second iteration, the plurality of iterations being separated in time by a second plurality of ticker periods, wherein the second plurality of ticker periods are each equal in length of time. In certain forms, a portion of the ticker periods of the first plurality of ticker periods are a length in time different than the length of time of the ticker periods of the second plurality of ticker periods. In certain forms, updating the trust matrix is based on the portion of the plurality of iterations received during the first ticker period includes decreasing any weight applied to iterations received from a remainder of the plurality of agents nodes that did not transmit an iteration to the first agent node during the ticker period. In certain forms, updating the trust matrix based on the portion of the plurality of iterations received during the first ticker period includes increasing the weight applied to iterations determined by the first agent node in an amount equal to the amount any weights were decreased. In certain forms, the method further comprises determining a step size based on the portion of the plurality of iterations received during the first ticker period, wherein determining the second iteration after the first ticker period is based in part on the determined step size. In certain forms, the first iteration, the second iteration, and the plurality of iterations are determined by the plurality of agent nodes based on recursive least squares. In certain forms, the first iteration, the second iteration, and the plurality of iterations are state estimations, and wherein the method further comprises determining, with the first agent node, a series of iterations including the first iteration and the second iteration until the first agent node determines a final iteration. In certain forms, the plurality of agent nodes are microgrid controllers, wherein the final iteration is a final state estimation, and wherein the method comprises operating, with the first agent node, a controllable device of a microgrid based on the final iteration. 
     A further exemplary embodiment is a distributed system comprising: a first agent node; a plurality of agent nodes; and a communication network structured to allow the first agent node and the plurality of agent nodes to communicate asynchronously; wherein the plurality of agent nodes each include a first set of instructions stored on a first memory device which, when executed by a first processing device of the corresponding agent node, is effective to determine a plurality of iterations separated in time by a plurality of ticker periods, wherein the first agent node is structured to execute a second set of instructions stored on a second memory device of the first agent node with a second processing device of the first agent node effective to: determine a trust matrix including a plurality of trust factors, each trust factor corresponding to a weight applied to iterations of the plurality of iterations determined by one of the plurality of agent nodes, determine a first iteration, receive a portion of the plurality of iterations determined by a portion of the plurality of agent nodes during a first ticker period beginning after the first iteration, update the trust matrix based on the portion of the plurality of iterations, and determine a second iteration after the first ticker period based on the portion of the plurality of iterations and the updated trust matrix. 
     While the present disclosure has been illustrated and described in detail in the drawings and foregoing description, the same is to be considered as illustrative and not restrictive in character, it being understood that only certain exemplary embodiments have been shown and described, and that all changes and modifications that come within the spirit of the present disclosure are desired to be protected. It should be understood that while the use of words such as “preferable,” “preferably,” “preferred” or “more preferred” utilized in the description above indicate that the feature so described may be more desirable, it nonetheless may not be necessary, and embodiments lacking the same may be contemplated as within the scope of the present disclosure, the scope being defined by the claims that follow. In reading the claims, it is intended that when words such as “a,” “an,” “at least one,” or “at least one portion” are used there is no intention to limit the claim to only one item unless specifically stated to the contrary in the claim. The term “of” may connote an association with, or a connection to, another item, as well as a belonging to, or a connection with, the other item as informed by the context in which it is used. The terms “coupled to,” “coupled with” and the like include indirect connection and coupling, and further include but do not require a direct coupling or connection unless expressly indicated to the contrary. When the language “at least a portion” and/or “a portion” is used, the item can include a portion and/or the entire item unless specifically stated to the contrary.