Consensus loss in distributed control systems

A device may correspond to a physical access controller in a distributed physical access control system. A method, performed by the device in a distributed system, may include detecting that another device in the distributed system has become unavailable; determining that a loss of consensus has occurred in the distributed system based on detecting that the other device has become unavailable; generating a list of available devices in the distributed system; and sending an alarm message to an administrative device, wherein the alarm message indicates the loss of consensus and wherein the alarm message includes the list of available devices.

FIELD

This disclosure generally relates to access control systems, and more specifically, to consensus loss in distributed access control systems.

BACKGROUND INFORMATION

A distributed system may include components that communicate and coordinate their actions to achieve a set of tasks. For example, computers in a distributed system may communicate over a network in order to coordinate to solve an algorithm. In order to successfully coordinate simultaneous operations on independent computers, such as data and resource management or synchronization of processing steps, a distributed algorithm may need to be implemented. Various distributed algorithms have been developed to coordinate components of a distributed system.

SUMMARY

According to one aspect, a method, performed by a device in a distributed system, may include detecting, by the device, that another device in the distributed system has become unavailable; determining, by the device, that a loss of consensus has occurred in the distributed system based on detecting that the other device has become unavailable; generating, by the device, a list of available devices in the distributed system; and sending, by the device, an alarm message to an administrative device, wherein the alarm message indicates the loss of consensus and wherein the alarm message includes the list of available devices.

Additionally, the method may include receiving, from the administrative device, an instruction to create a consensus-based distributed system that includes at least some of the available devices included in the list; and creating the consensus-based distributed system in response to receiving the instruction.

Additionally, creating the consensus-based distributed system may include selecting a number of devices required to generate a quorum, wherein the number corresponds to a smallest number of devices from the available devices needed to achieve a majority.

Additionally, the method may include assigning a voting weight to a device of the at least some of the available devices, wherein the voting weight gives more weight to a vote received from the device than to a vote received from another device of the at least some of the available devices when the device is voting for a quorum in the created the consensus-based distributed system.

Additionally, the voting weight may be based on at least one of a location of the device; a membership of the device in a particular subset of the plurality of physical access control devices; a device type associated with the device; a threat level assigned to the device; or a measure of reliability associated with the device.

Additionally, the voting weight may be based on at least one of one or more logged events associated with the device; a frequency of activation of a peripheral device associated with the device; or a frequency of administrative login associated with the device.

Additionally, detecting that another device in the distributed system has become unavailable may include at least one of detecting that a connection to the other device has been lost; receiving an indication of device failure from the other device; or detecting that the other device is associated with a security breach.

Additionally, sending the alarm message to the administrative device may include at least one of sending a Web Services message to the administrative device; sending an email message to the administrative device; sending a Short Message Service message to the administrative device; or sending a real-time quality of service message to the administrative device.

Additionally, the method may include maintaining the list of available devices in the distributed system; and updating the list of available devices at particular intervals.

Additionally, the distributed system may include a distributed dataset, and the method may further include generating an emergency distributed dataset, in response to determining that a loss of consensus has occurred in the distributed system, wherein the emergency distributed dataset is based on the distributed dataset; and creating an emergency consensus-based distributed system that includes the available devices included in the list, in response to determining that a loss of consensus has occurred in the distributed system, wherein the emergency consensus-based distributed system uses the emergency distributed dataset.

Additionally, the method may further include receiving, from the administrative device, an instruction to create a consensus-based distributed system that includes the available devices included in the list; and incorporating data from the emergency distributed dataset into the distributed dataset, in response to receiving the instruction to create the consensus-based distributed system.

Additionally, the device may correspond to an embedded system.

Additionally, the distributed system may include a distributed physical access control system, and wherein the device and the other device correspond to physical access control units.

According to another aspect, a device of a distributed system may include logic configured to detect that another device in the distributed system has become unavailable; determine that a loss of consensus has occurred in the distributed system based on detecting that the other device has become unavailable; generate a list of available devices in the distributed system; and send an alarm message to an administrative device, wherein the alarm message indicates the loss of consensus and wherein the alarm message includes the list of available devices.

Additionally, the logic may be further configured to receive, from the administrative device, an instruction to create a consensus-based distributed system that includes at least some of the available devices included in the list; and create the consensus-based distributed system in response to receiving the instruction.

Additionally, when the logic is creating the consensus-based distributed system, the logic may be further configured to select a number of devices required to generate a quorum, wherein the number corresponds to a smallest number of devices from the available devices needed to achieve a majority.

Additionally, when the logic is creating the consensus-based distributed system, the logic may be further configured to assign a voting weight to a device of the at least some of the available devices, wherein the voting weight gives more weight to a vote received from the device than to a vote received from another device of the at least some of the available devices when the device is voting for a quorum in the created the consensus-based distributed system.

Additionally, the voting weight may be based on at least one of a location of the device; a membership of the device in a particular subset of the plurality of physical access control devices; a device type associated with the device; a threat level assigned to the device; a measure of reliability associated with the device; one or more logged events associated with the device; a frequency of activation of a peripheral device associated with the device; or a frequency of administrative login associated with the device.

Additionally, when the logic is detecting that another device in the distributed system has become unavailable, the logic may be configured to detect that a connection to the other device has been lost; receive an indication of device failure from the other device; or detect that the other device is associated with a security breach.

Additionally, when sending the alarm message to the administrative device, the logic may be further configured to send a Web Services message to the administrative device; send an email message to the administrative device; send a Short Message Service message to the administrative device; or send a real-time quality of service message to the administrative device.

Additionally, the distributed system may include a distributed dataset, and the logic may be further configured to generate an emergency distributed dataset, in response to determining that a loss of consensus has occurred in the distributed system, wherein the emergency distributed dataset is based on the distributed dataset; create an emergency consensus-based distributed system that includes the available devices included in the list, in response to determining that a loss of consensus has occurred in the distributed system, wherein the emergency consensus-based distributed system uses the emergency distributed dataset; receive, from the administrative device, an instruction to create a consensus-based distributed system that includes the available devices included in the list; and incorporate data from the emergency distributed dataset into the distributed dataset, in response to receiving the instruction to create the consensus-based distributed system.

Additionally, the distributed system may include a distributed physical access control system, and wherein the device and the other device correspond to physical access control units.

According to yet another aspect, a distributed system may include a plurality of physical access control devices, wherein a particular one of the plurality of physical access control devices is configured to detect that another device in the distributed system has become unavailable; determine that a loss of consensus has occurred in the distributed system based on detecting that the other device has become unavailable; generate a list of available devices in the distributed system; send an alarm message to an administrative device, wherein the alarm message indicates the loss of consensus and wherein the alarm message includes the list of available devices; receive, from the administrative device, an instruction to create a consensus-based distributed system that includes at least some of the available devices included in the list; and create the consensus-based distributed system in response to receiving the instruction.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Implementations described herein relate to consensus loss in distributed control systems. In one embodiment, a distributed control system may include a distributed physical access control system. A physical access control system may include one or more control or system units, each controlling physical access to an area of a facility. For example, control or system unit may obtain credentials from a user and may unlock a door lock if the user's credentials are verified. In other implementations, a distributed control system may include a distributed building management system, a distributed monitoring system, a distributed security system, and/or another type of a distributed control system.

A distributed control system may perform distributed computing operations using a distributed algorithm. For example, a distributed control system may manage a distributed data base. A distributed algorithm may include a consensus algorithm. A consensus algorithm may require that a consensus be reached by the nodes in a distributed system before an operation is performed. For example, to update a distributed database employing a consensus algorithm, the nodes in the distributed system reach a consensus to perform the update. When consensus is reached, the nodes (e.g., all the nodes) may then agree to a proposed operation (e.g., a change to a distributed database). A consensus may be reached if a quorum is obtained for a proposed operation. A quorum may be reached if a majority of the nodes votes for the proposed operation. In this embodiment, requiring a majority of nodes to vote for a change may ensure that, if two conflicting proposed changes are being considered, at least one node receives both of the proposed changes and selects to vote for one of the proposed changes.

A consensus loss in a distributed system may occur if a quorum cannot be reached. For example, if a particular number of nodes fails or is unreachable, a quorum for a proposed operation may not be obtained and the distributed system may not be able to continue to operate. A consensus loss may be automatically handled by changing the number of nodes in the system or by bringing reserve nodes into the system. However, in distributed control systems, such solutions may not be desirable. For example, changing the number of nodes in the system or bringing in reserve nodes may change the functionality of the system. As another example, reserve nodes may not be available. As yet another example, if communication between two subsets of nodes in a distributed control system is interrupted and a new consensus-based system is automatically created using the available nodes in each subset, a split-brain problem may occur. In a split-brain problem, the two subsets each form their own distributed system, resulting in two versions of a distributed dataset being maintained.

Implementations described herein may include a device in a distributed system configured to detect that another device in the distributed system has become unavailable and to determine that a loss of consensus has occurred in the distributed system based on detecting that the other device has become unavailable. The device may be configured to generate a list of available devices in the distributed system and to send an alarm message to an administrative device. The alarm message may indicate that a loss of consensus has occurred and may include the list of available devices. An administrator may determine whether to create a new distributed system based on the generated list of available devices. Generation of a new distributed system using the available devices may be desirable in some situations and not desirable in other situations. The device may be configured to receive, from the administrative device, an instruction to create a consensus-based distributed system that includes at least some of the available devices included in the list and to create the consensus-based distributed system in response to receiving the instruction. Generating a list of available devices and requiring approval from an administrator to form a new consensus-based system using the available devices may prevent a split-brain problem in a consensus-based distributed system.

FIG. 1is a block diagram of an exemplary environment100in which the systems and/or methods described below may be implemented. As shown inFIG. 1, environment100may include a control system110(e.g., a distributed physical access control system), a network120, and an administration device130.

Distributed control system (DCS)110may include a distributed computing system that includes system units115-A to115-N (referred to collectively as “system units115” and individually as “system unit115”). System unit115may be implemented as an embedded system. In some implementations, system unit115may include a physical access control device. For example, system unit115may include an access controller that controls access to a secured area, such as a room or a group of rooms. System unit115may receive credentials (e.g., access card credentials) via a reader device and to determine whether the credentials are authentic and are associated with authority to access the secure area. If so, the access controller may issue a command to open a lock on a door or perform other operations associated with granting access to the secure area. In other implementations, system unit115may include a different type of security device, such as a monitoring device, a device that controls the operation of a machine, etc.

DCS110may include one or more distributed datasets. A distributed dataset includes a dataset associated with multiple devices, wherein the In one embodiment, multiple devices may communicate and coordinate with each other to make changes to the dataset. In one embodiment, a local copy of the distributed dataset is maintained by each device that is associated with the distributed dataset and if a change is agreed upon by the devices, the change is replicated to the local copies of the distributed datasets. In another embodiment, not all the devices store a local copy of the distributed dataset, for example.

In some embodiments, consensus is reached in order for a change to be made in the distributed dataset (e.g., a consensus-based distributed database). In other embodiments, a change may be made to the distributed dataset without consensus. A distributed dataset may be associated with all system units115or may be associated with a subset of system units115. A system unit115may propose a change to a consensus-based distributed dataset. If the change is accepted by a quorum of system units115associated with the distributed dataset, a consensus may be reached and the change may be propagated to each local copy of the distributed dataset in each associated system unit115. Thus, a consensus with respect to a change in the distributed dataset may be reached if a quorum of the associated system units115votes for the change. A quorum may correspond to the smallest majority of the associated system units115. Thus, if a distributed dataset is associated with N system units115, a quorum may be reached if N/2+1 associated system units115votes for the change if N is an even number, or if (N−1)/2+1 associated system units115votes for the change if N is an odd number. Requiring a smallest majority to reach a quorum may ensure that when considering two conflicting proposals, at least one system unit115receives both proposals and selects one of the proposals for consensus.

A consensus-based distributed dataset may ensure that any system unit115associated with the distributed dataset includes the information (e.g., all the information in one embodiment) managed by the distributed dataset. For example, a distributed dataset may include access rules and the access rules may be available to any system unit115associated with the distributed dataset. Thus, as a result of the one or more distributed datasets, in one embodiment, DCS110may correspond to a decentralized system with no central controlling device, such as a server device. In other embodiments, DCS110may include both a decentralized system and a central controlling device (such as a server device). Changes to DCS110may be configured at any system unit115and if the change is associated with a distributed dataset, the change may be propagated to other system units115associated with the distributed dataset. Furthermore, DCS110may exhibit robustness with respect to device failure, as a single point of failure may be avoided. For example, if a particular system unit115fails, the other system units115may continue to operate without loss of data (or with the minimization of loss of data). Moreover, DCS110may be changed dynamically. For example, applications may be added at any time and new datasets may be stored in system units115as needed.

DCS110may also include datasets that are not distributed. As an example, a first system unit115may include a local dataset that is not included in any other system unit115. As another example, a first system unit115may include a local dataset that is replicated to a second system unit115in a non-distributed manner, such as by being mirrored. As yet another example, a first system unit115may include a first version of a local dataset and a second system unit115may include a second version of the local dataset, wherein the first system unit115maintains the first version of the local dataset and the second system unit115maintains the second version of the local dataset. In yet another example, a first system unit115may derive a first local dataset from a distributed dataset and a second system unit115may derive a second local dataset from the distributed dataset, wherein the first local dataset is different from the second local dataset.

Network120may enable system units115to communicate with each other and/or may enable administration device130to communicate with particular system units115. Network120may include one or more circuit-switched networks and/or packet-switched networks. For example, network120may include a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a Public Switched Telephone Network (PSTN), an ad hoc network, an intranet, the Internet, a fiber optic-based network, a wireless network, and/or a combination of these or other types of networks.

Administration device130may enable an administrator to connect to a particular system unit115in order to configure DCS110, change a configuration of DCS110, receive information from DCS110, and/or otherwise administer DCS110. Administration device130may include any device configured for communicating with a system unit115. For example, administration device130may include a portable communication device (e.g., a mobile phone, a smart phone, a phablet device, a global positioning system (GPS) device, and/or another type of wireless device); a personal computer or workstation; a server device; a laptop, tablet, or another type of portable computer; and/or any type of device with communication capability.

AlthoughFIG. 1shows exemplary components of environment100, in other implementations, environment100may include fewer components, different components, differently arranged components, or additional components than depicted inFIG. 1. Additionally or alternatively, any one device in environment100(or any group of devices) may perform functions described as performed by one or more other devices in environment100. For example, in some implementations, system units115may include an input and/or output device (e.g., keyboard/keypad and display, touchscreen, etc.) and administration device130may not be needed.

FIG. 2is a block diagram illustrating exemplary components of a system unit115. As shown inFIG. 2, system unit115may include a controller210and one or more peripheral devices230. Controller210may control the operation of system unit115, may communicate with other system units115, may communicate with administration device130, and/or may control peripheral devices230. Controller210may include a bus212, a processor214, a memory216, a network interface218, a peripheral interface220, and a housing222.

Bus212may include a path that permits communication among the components of controller210. Processor214may include any type of single-core processor, multi-core processor, microprocessor, latch-based processor, and/or processing logic (or families of processors, microprocessors, and/or processing logics) that interprets and executes instructions. In other embodiments, processor214may include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and/or another type of integrated circuit or processing logic.

Memory216may include any type of dynamic storage device that may store information and/or instructions, for execution by processor214, and/or any type of non-volatile storage device that may store information for use by processor214. For example, memory216may include a random access memory (RAM) or another type of dynamic storage device, a read-only memory (ROM) device or another type of static storage device, a content addressable memory (CAM), a magnetic and/or optical recording memory device and its corresponding drive (e.g., a hard disk drive, optical drive, etc.), and/or a removable form of memory, such as a flash memory.

Network interface218may include a transceiver (e.g., a transmitter and/or a receiver) that enables controller210to communicate (e.g., transmit and/or receive data) with other devices and/or systems via wired communication links (e.g., conductive wire, twisted pair cable, coaxial cable, transmission line, fiber optic cable, and/or waveguide, etc.), wireless communication links (e.g., radiofrequency (RF), infrared, and/or visual optics, etc.), or a combination of wireless and wired communication links. Network interface218may include a transmitter that converts baseband signals to RF signals and/or a receiver that converts RF signals to baseband signals. Network interface218may be coupled to an antenna for transmitting and receiving RF signals.

Peripheral interface220may be configured to communicate with one or more peripheral devices230. For example, peripheral interface220may include one or more logical components that include input and/or output ports, input and/or output systems, and/or other input and output components that facilitate the transmission of data to peripheral devices230. As an example, peripheral interface220may communicate with peripheral devices using a Serial Peripheral Interface Bus protocol, such as a Wiegand protocol, an RS-485 protocol, and/or another type of protocol. As another example, peripheral interface220may use a different type of protocol. In one embodiment, network interface218may also act as a peripheral interface for coupling peripherals230to controller210.

Housing222may enclose the components of controller210and may protect the components of controller210from the environment. In one embodiment, housing222may include one or more of peripheral devices230. In another embodiment, housing222may include administration device130. Housing222may define the boundaries of one system unit115and/or controller210from other system units115and/or controllers210in a multi system unit115/controller115system.

As described below, controller210may perform certain operations relating to managing consensus loss in a distributed system. Controller210may perform these operations as a result of hardwired circuitry of an ASIC. Controller210may also (or alternatively) perform these operations in response to processor214executing software instructions contained in a computer-readable medium, such as memory216. A computer-readable medium may include a non-transitory memory device. A memory device may be implemented within a single physical memory device or spread across multiple physical memory devices. The software instructions may be read into memory216from another computer-readable medium or from another device. The software instructions contained in memory216may cause processor214to perform processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

Peripheral devices230may include one or more devices that provide information to controller210, that are controlled by controller210, and/or that otherwise communicate with controller210. For example, peripheral devices230may include a reader device240, a lock device250, a sensor260, and/or an actuator270. While a single reader device240, a single lock device250, a single sensor260, and a single actuator270are shown inFIG. 2for illustrative purposes, in practice, peripheral devices230may include multiple reader devices240, multiple lock devices250, multiple sensors260, and/or multiple actuators270. In some implementations, peripheral devices230may not include one or more of the devices shown inFIG. 2. Additionally or alternatively, peripheral devices230may include any other type of security device not shown inFIG. 2.

Reader device240may include a device that reads credentials from a user and provides the credentials to controller210. For example, reader device240may include a keypad configured to receive an alphanumeric personal identification number (PIN) from a user; a card reader to configure a card that stores a card code on a magnetic strip or another type of storage device, such as an RFID tag; a fingerprint reader configured to read a user's fingerprint; an iris reader configured to read a user's iris; a microphone and a voice signature identifier configured to record a user's voice signature; an near field communication NFC reader; and/or another type of reader device. Reader device240may include any type security device that can provide credentials, and may include one or more sensor devices, such any sensor device described below with reference to sensor260. For example, reader device240may include a camera used for facial recognition and/or a microphone used for voice recognition.

Lock device250may include a lock controlled by controller210. Lock device250may lock a door (e.g., prevent it from opening or closing), a window, an HVAC vent, and/or another type of access opening to a secure area. For example, lock device250may include an electromagnetic lock; a mechanical lock with a motor controlled by controller210; an electromechanical lock; and/or another type of lock. Furthermore, lock device250may lock/unlock operation of a machine, transport vehicle, elevator, and/or an electrical device.

Sensor260may include a sensor device. As examples, sensor260may include a door sensor to sense whether a door is open or closed; a visible light monitoring camera, an infrared (IR) light monitoring camera, a heat signature monitoring camera, and/or another type of monitoring device; an alarm sensor, such as a motion sensor, a heat sensor, a pressure sensor, and/or another type of alarm sensor; an audio recording device (e.g., microphone); a tamper sensor, such as a position sensor located inside system unit115; and/or a “request to exit” button located within a secured area associated with system unit115; and/or another type of sensor device.

Actuator270may include an actuator device. As an example, actuator270may control an illumination device. As other examples, actuator270may include a burglar alarm activator; a speaker to play messages or generate alarm signals; a display device; a motor to move sensor260(e.g., control the field of view of a camera or other monitoring device); a motor for opening/closing a door, window, HVAC vent, and/or another opening associated with a secure area; a motor to secure lock device250in a locked or unlocked position; a fire extinguishing device; and/or another type of actuator device.

AlthoughFIG. 2shows exemplary components of system unit115, in other implementations, system unit115may include fewer components, different components, additional components, or differently arranged components than depicted inFIG. 2. Additionally or alternatively, any component of system unit115(or any group of components) may perform the task or tasks described as performed by one or more other components of system unit115. For example, in some implementations, peripheral interface220may correspond to a network interface. As another example, in some implementations, peripheral devices230may be connected to controller210via network interface218, rather than via peripheral interface220.

Further, although DCS110may include a physical access distributed control system, other implementations may control systems other than physical access systems. On the other hand, DCS110may include any type of physical access control systems (e.g., in an operational environment), such as a control system for opening and/or closing a door or controlling physical access to a building or facility. DCS110may also include a system to control a fan (e.g., start or stop), to initiate an alarm in a building management system (e.g., failed authentication, successful authentication, etc.), or to control a robot arm in an industrial automation system.

FIGS. 3A and 3Bare block diagrams illustrating exemplary functional components of system unit115. The functional components of system unit115may be implemented, for example, via hard-wired circuitry of one or more ASICs. Additionally or alternatively, the functional components of system unit115may be implemented by processor214executing instructions from memory216.FIG. 3Aillustrates the functional layers of system unit115. As shown inFIG. 3A, system unit115may include an application programming interface (API) layer310, an application layer320, a distribution layer340, and a storage layer350.

API layer310includes an API configured to communicate, e.g., with administration device130. As an example, when an administrator uses administrator device130to log into system unit115, API layer310may communicate with administrator device130to authenticate the administrator. As another example, API layer310may communicate with administrator device130to change a configuration of system unit115. API layer310may receive data from administrator device130and provide the data to distribution layer340and/or to storage layer350. API layer310may also communicate with administrator device130to install an application in application layer320. API layer310may be configured to handle different administrator types. For example, API layer310may include an API to handle a Web Services administrator, a Linux administrator, an Open Network Video Interface Forum (ONVIF) administrator, and/or another type of API.

Application layer320includes one or more applications installed on system unit115.FIG. 3Bshows exemplary applications. As shown inFIG. 3B, application layer320may include an access control logic application322, a door control application324, a reader control application326, an event handling application328, a schedule handling application330, and/or a consensus loss module332.

Access control logic application322may determine whether to grant access based on received credentials and based on stored access rules. Door control application324may control one or more doors and/or associated lock devices250. For example, door control application324may determine whether a door is open or closed and/or locked or unlocked and may operate one or more device to open or close the door and/or to lock or unlock the door. Reader control application326may control one or more reader devices240and may obtain and process credentials received from the one or more reader devices240. Event handling application328may maintain a log of events recorded or generated by system unit115and/or recorded by another system unit115. Event handling application328may ensure that locally recorded or generated events are distributed to other system units115in DCS110in order to maintain a distributed system event log in all (or at least some) system units115. Thus, logged events may be retrieved from any system unit115that is associated with the system event log. Schedule handling application330may manage one or more schedules associated with system unit115. For example, access rules for particular groups of users may change based on particular times of day.

Consensus loss module332may detect consensus loss with respect to a distributed dataset. For example, consensus loss module332may determine which system units115are available and may determine whether the number of available system units115, associated with a particular distributed dataset, is sufficient to reach a quorum with respect to proposed operation on the distributed dataset. If the number of available system units115drops below the number required to reach a quorum, consensus loss module332may detect a consensus loss with respect to the particular distributed dataset. Consensus loss module332may generate a list of available system units115, associated with the particular distributed dataset, may generate a consensus loss alarm that includes the generated list, and may send the generated consensus loss alarm to a designated device. The designated device may correspond to administration device130and/or another device, such as a mobile communication device (e.g., mobile phone) of an administrator. If the administrator selects to generate a new set of system units115for the distributed dataset, the administrator may send an instruction to system unit115to generate the new set of system units115. Consensus loss module332may generate the new set of system units115associated with the distributed dataset.

Other applications (not shown inFIG. 3B) may be included in application layer320. As an example, an alarm application may generate a report and/or an alarm and send the report and/or alarm to administrator device130(and/or to another designated device) and/or one or more other system units115. As another example, a task-specific control application may process events associated with system unit115, such as door opening events, sensor events, actuator events, and/or other types of events.

Distribution layer340may manage one or more distributed datasets associated with system unit115. For example, distribution layer340may maintain secure connections with other system units115over network120. (e.g., a Transport Layer Security (TLS) connection). Furthermore, distribution layer340may use a protocol (e.g., a PAXOS protocol) to establish a consensus with respect to a change in a particular consensus-based distributed dataset. As an example, distribution layer340may send a proposal for a change to other system units115associated with the distributed dataset and may receive a quorum for the change from the other system units115. As another example, distribution layer340may vote for a proposal received from another system unit115. As yet another example, distribution layer340may receive an indication that a consensus has been reached for a change without having voted for the change. When an indication of consensus for a change is received, distribution layer340may make the change in the local copy of the distributed dataset.

Storage layer350stores one or more datasets associated with system unit115. A dataset stored in storage layer350may correspond to a local dataset or may correspond to a distributed dataset. A local dataset may store information associated with (and/or only associated with) the particular system unit115that stores the local dataset. A distributed dataset may store information associated with other system units115associated with the distributed dataset.

Exemplary information that may be included in storage layer350is shown inFIG. 3B. As shown inFIG. 3B, storage layer350may include configuration data352, credentials data354, access rules data356, and consensus database358. Configuration data352may store configuration data associated with a particular system unit115, such as hardware configuration of controller210, peripheral devices230connected to controller210, application installed in application layer320, and/or other types of configuration information. Credentials data354may store credentials associated with system unit115. Access rules356may store access rules associated with system unit115. Consensus database358may store information that may be used by consensus loss module332to detect consensus loss. Exemplary information that may be stored in consensus database358is described below with reference toFIG. 4.

AlthoughFIGS. 3A and 3Bshow exemplary functional components of system unit115, in other implementations, system unit115may include fewer functional components, different functional components, differently arranged functional components, or additional functional components than depicted inFIGS. 3A and 3B. Additionally, any one of the components (or any group of components) of system unit115may perform functions described as performed by one or more other functional components of system unit115.

FIG. 4is a diagram of exemplary information that may be stored in the consensus database358. As shown inFIG. 4, consensus database358may include one or more distributed dataset records400. Each dataset record400may store information relating to a particular dataset with which system unit115is associated. Dataset record400may include a distributed dataset field410, a set of system unit fields420-A to420-N, a corresponding set of availability fields430-A to430-N, a consensus field440, and an alarm field450.

Distributed dataset field410may store information identifying a particular distributed dataset associated with system unit115. Each system unit field420may store information identifying another system unit115associated with the distributed dataset. A corresponding availability field430may store information about whether the other system unit115is available. Consensus field440may store information relating to consensus requirements associated with the particular distributed dataset. For example, consensus field440may store information about how many votes are needed from other system units115to reach a quorum for establishing consensus relating to proposed operation on the particular distributed dataset. Furthermore, consensus field440may store an indication that consensus loss has been detected if consensus loss module332detects consensus loss for the particular distributed dataset. In some implementations, different system units115may assigned different voting weights when voting for a proposed operation on the particular distributed dataset. The voting weight information may be stored in system unit fields420and/or in consensus field440.

Alarm field450may store information relating to generation of a consensus loss alarm in response to a consensus loss being detected. For example, alarm field450may identify a particular communication method to be used in sending the consensus loss alarm and/or may identify a particular destination device to which the consensus loss alarm is to be sent.

AlthoughFIG. 4show exemplary components that may be stored in consensus database358, in other implementations, consensus database358may include fewer components, different components, differently arranged components, or additional components than depicted inFIG. 4.

FIG. 5is a flowchart for processing consensus loss in a distributed control system in one embodiment. In some implementations, the process ofFIG. 5may be performed by controller210. In other implementations, some or all of the process ofFIG. 5may be performed by another device or a group of devices separate from controller210and/or including controller210.

The process ofFIG. 5may include detecting that a system unit in a distributed system has become unavailable (block510). System units115may exchange heartbeat messages at particular intervals. If a heartbeat message is not received from another system unit115within a particular time period, consensus loss module332may determine that the other system unit115has become unavailable. As an example, the other system unit115may have malfunction or lose power. As another example, a connection to the other system unit115may fail as a result of a broken link, network congestion, malfunctioning interface or networking device, and/or another cause. Consensus loss module332may store an indication of the unavailability of the other system unit115in consensus database358.

As yet another example, a failing system unit115may send a message to other system units115indicating device failure, causing the other system units115to identify the failing system unit115as unavailable. As yet another example, a compromised system unit115may detect a security breach. For example, the compromised system unit115may include a tamper switch that becomes activated if controller210of the other system unit115is manipulated. In response to detecting the tampering, the compromised system unit155may send a message to the other system units115indicating a security breach and may deactivate itself. The other system units115may identify the compromised system unit115as unavailable in response to receiving the security breach message. Consensus loss module332may maintain a list of available system units115, associated with distributed dataset, and may update the list of available devices at particular intervals.

A determination may be made that consensus loss has occurred in the distributed system based on detecting that the system unit has become unavailable (block520). For example, consensus loss module332may determine how many system units115, associated with a distributed dataset, are available and may determine whether the number of available system units115is less than the number of system units115required to reach a quorum to establish a consensus about a proposed operation on the distributed dataset. If the number of available system units115is less than the number of system units115required to reach the quorum, consensus loss module332may determine that a consensus loss has occurred for the distributed dataset.

A list of available system units may be generated (block530) and an alarm message, which includes the generated list, may be sent to an administration device (block540). For example, consensus loss module332may access consensus database358and may determine which system units115, associated with the distributed dataset, are available. Consensus loss module332may access alarm field450to determine what communication method to use to send the consensus loss alarm and may determine a particular destination device to which the consensus loss alarm should be sent. The destination device may be administration device130and/or another device, such as an administrator's mobile communication device.

As an example, alarm field450may specify that a Web Services message should be sent to the destination device and consensus loss module332may send a Web Services message to the destination device. As another example, alarm field450may specify that an email message, a Short Message Service, or a real-time quality of service message should be sent to the destination device and consensus loss module332may send an email message, a Short Message Service, or a real-time quality of service message to the destination device.

Instructions may be received to create a consensus-based distributed system that includes at least some of the system units included in the list (block550) and a consensus-based distributed system may be generated based on the received instructions (block560). For example, system unit115may receive instructions from the destination device (e.g., administration device130) to generate a consensus-based distributed system for the distributed dataset. The received instructions may include a list of system units115that are to be included in the new distributed system. As an example, the administrator may select to include all the available devices listed in the consensus loss alarm in the new distributed system. As another example, the administration may select to exclude some of the available devices from the new distributed system.

In some implementations, administration device130may send the instructions to a particular system unit115that is to be included in the new distributed system. In other implementations, administration device130may send the instructions to multiple (and possibly all) system units115that are to be included in the new distributed system. A system unit115that received the instructions may generate a new distributed dataset based on the distributed dataset associated with the consensus loss alarm and may then proceed to associate the other system units115, which are to be included in the new distributed system, with the new distributed dataset.

System unit115may, when creating the new distributed system, select a number of system units115required to reach a quorum based on the smallest number of devices needed to achieve a majority. Thus, for example, if 6 system units115were selected by the administrator for the new distributed system, a quorum of 4 system units115may be set for the new distributed system.

In some implementations, different voting weights may be assigned to different system units115when voting for a quorum for a proposed operation for a distributed dataset. A particular system unit115may be given a higher or lower voting weight than other system units115based on the location of the particular system unit115. For example, a particular location may be considered important and a system unit115at the particular location may thus be given a higher voting weight than other system units115.

A particular system unit115may be given a higher or lower voting weight than other system units115based on a membership in a group of system units115. For example, a distributed dataset may be associated with multiple groups of system units115based on one or more criteria and system units115in a first group may be given a higher voting weight than system units115in a second group. A particular system unit115may be given a higher or lower voting weight than other system units115based on a device type of the particular system unit115. For example, a first system unit115may correspond to a new version with additional functionality and may be given a higher voting weight than other system units115.

A particular system unit115may be given a higher or lower voting weight than other system units115based on a threat level assigned to the particular system unit115. For example, a particular system unit115may be located in a high threat area, such as a public location, and may thus be given a lower voting weight than other system units115. A particular system unit115may be given a higher or lower voting weight than other system units115based on a measure of reliability associated with the particular system unit115. For example, a particular system unit115may have a high failure rate (e.g., due to environmental conditions) and may be given a lower voting weight than other system units115. A particular system unit115may be given a higher or lower voting weight than other system units115based on one or more logged events associated with the particular system unit115. For example, a particular system unit115may include a motion sensor that logs a large number of motion detection events and thus the particular system unit115may be given a higher voting weight than other system units115.

A particular system unit115may be given a higher or lower voting weight than other system units115based on a frequency of activation of a peripheral device associated with the particular system unit115. For example, a particular system unit115may experience high traffic, causing a lock device250to be activated with a higher frequency than lock devices250associated with other system units115, and may be given a higher voting weight than other system units115. A particular system unit115may be given a higher or lower voting weight than other system units115based on a frequency of administrative login associated with the particular system unit115. For example, a particular system unit115may be used by administration device130more frequently than other system units115and may be given a higher voting weight than other system units115.

In some implementations, system units115may need to continue to use a distributed dataset after a consensus loss is detected and before a response is received from administration device130instructing system units115to generate a new distributed dataset. Thus, in some implementations, in response to detecting a consensus loss, system unit115may generate an emergency distributed dataset based on the distributed dataset and based on the list of available system units115. The emergency distributed dataset may be used by the system units115until an administrator responds to the generated consensus loss alarms. After instructions are received from administration device130to create a new distributed system, data from the emergency distributed dataset may be incorporated into the new distributed dataset.

In some implementations, after the new distributed dataset has been created, one or more system units115, that were previously associated with the distributed dataset and that have become unavailable, may again become available. For example, a faulty system unit115may be repaired or a broken connection may be restored. A restored system unit115may request to rejoin the distributed dataset. While the restored system unit115has been unavailable, the restored system unit115may have accumulated data relevant to the distributed dataset. The restored system unit115may request to be added to the new distributed system that was created after consensus loss. A system unit115that is associated with the new distributed dataset may receive a request from the restored system unit115to be added to the new distributed dataset and may add the restored system unit115to the new distributed dataset. The system unit115may subsequently receive a request from the restored system unit115to add device data to the new distributed dataset, may determine whether the device data is consistent with the new distributed dataset; and may add the device data to the new distributed dataset, if the device data is consistent with the distributed dataset.

FIG. 6is a block diagram illustrating exemplary components of administration device130. As shown inFIG. 6, administration device130may include a bus610, a processor620, a memory630, an input device640, an output device650, and a communication interface660.

Bus610may include a path that permits communication among the components of administration device130. Processor620may include any type of single-core processor, multi-core processor, microprocessor, latch-based processor, and/or processing logic (or families of processors, microprocessors, and/or processing logics) that interprets and executes instructions. In other embodiments, processor620may include an ASIC, an FPGA, and/or another type of integrated circuit or processing logic.

Memory630may include any type of dynamic storage device that may store information and/or instructions, for execution by processor620, and/or any type of non-volatile storage device that may store information for use by processor620. For example, memory630may include a RAM or another type of dynamic storage device, a ROM device or another type of static storage device, a CAM, a magnetic and/or optical recording memory device and its corresponding drive (e.g., a hard disk drive, optical drive, etc.), and/or a removable form of memory, such as a flash memory.

Input device640may allow an operator to input information into administration device130. Input device640may include, for example, a keyboard, a mouse, a pen, a microphone, a remote control, an audio capture device, an image and/or video capture device, a touchscreen display, and/or another type of input device. In some implementations, administration device130may be managed remotely and may not include input device640. In other words, administration device130may be “headless” and may not include a keyboard, for example.

Output device650may output information to an operator of administration device130. Output device650may include a display, a printer, a speaker, and/or another type of output device. For example, administration device130may include a display, which may include a liquid-crystal display (LCD) for displaying content to an administrator. The display may correspond to a touchscreen. In some embodiments, administration device130may be managed remotely and may not include output device650. In other words, administration device130may be “headless” and may not include a display, for example.

Communication interface660may include a transceiver that enables administration device130to communicate with other devices and/or systems via wireless communications (e.g., RF, infrared, and/or visual optics, etc.), wired communications (e.g., conductive wire, twisted pair cable, coaxial cable, transmission line, fiber optic cable, and/or waveguide, etc.), or a combination of wireless and wired communications. Communication interface660may include a transmitter that converts baseband signals to RF signals and/or a receiver that converts RF signals to baseband signals. Communication interface660may be coupled to an antenna for transmitting and receiving RF signals.

Communication interface660may include a logical component that includes input and/or output ports, input and/or output systems, and/or other input and output components that facilitate the transmission of data to other devices. For example, communication interface660may include a network interface card (e.g., Ethernet card) for wired communications and/or a wireless network interface (e.g., a WiFi) card for wireless communications. Communication interface660may also include a USB port for communications over a cable, a Bluetooth™ wireless interface, an RFID interface, an NFC wireless interface, and/or any other type of interface that converts data from one form to another form.

As described below, administration device130may perform certain operations relating to management of consensus loss alarms received from system units115. Administration device130may perform these operations in response to processor620executing software instructions contained in a computer-readable medium, such as memory630. Memory630may be implemented within a single physical memory device or spread across multiple physical memory devices. The software instructions may be read into memory630from another computer-readable medium or from another device. The software instructions contained in memory630may cause processor620to perform processes described herein. Alternatively, hardwired circuitry may be used in place of, or in combination with, software instructions to implement processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

AlthoughFIG. 6shows exemplary components of administration device130, in other implementations, administration device130may include fewer components, different components, additional components, or differently arranged components than depicted inFIG. 6. Additionally or alternatively, one or more components of administration device130may perform one or more tasks described as performed by one or more other components of administration device130.

FIG. 7is a flowchart for processing consensus loss alarms received from system units of a distributed control system in one embodiment. In some implementations, the process ofFIG. 7may be performed by administration device130. In other implementations, some or all of the process ofFIG. 7may be performed by another device or a group of devices separate from administration device130and/or including administration device130.

The process ofFIG. 7may include receiving one or more consensus loss alarms from system units of a distributed control system (block710). Administration device130may include a consensus loss alarm application. The consensus loss alarm application may be configured to receive consensus alarms from system units115of DCS110. When consensus loss is detected by system unit115, the consensus alarm may receive one or more consensus loss alarms. In some implementations, the consensus loss alarms may be presented as a sequence of alerts that are provided as they are received. Additionally or alternatively, the consensus loss alarms may be organized and consolidated before being presented to the administrator.

The received alarms may be organized based on distributed datasets associated with the alarms (block720) and a user interface may be generated based on the organized alarms (block730). For example, if a consensus loss is detected for a distributed dataset, the functioning system units115(e.g., all functional system units115) that are associated with the distributed dataset may send a consensus loss alarm to administration device130. Some of the received consensus loss alarms for a particular distributed dataset may include identical lists of available devices. The consensus loss alarm application may organize the received lists of available devices so that duplicate lists need to be presented. An alert may be generated on administration device130and, when an administrator selects to view the generated alert, the generated user interface may be presented to the administrator.

The user interface may be updated if additional alarms are received (block740). As additional consensus loss alarms are received, the consensus loss alarm application may update the generated user interface to include information received from the additional consensus loss alarms received by administration device130from system units115.

FIG. 8is a floor plan diagram illustrating an exemplary physical layout800of system unit115. As shown inFIG. 8, physical layout800may include a wall810, a door820, controller210, reader device240, lock device250, sensor260, and actuator270.

Wall810encloses a secure area830, such as a room in a building. Door820provides access for a user to secure area830. In this embodiment, controller210is installed inside secure area830. In other embodiments, controller210may be installed in an insecure area850. Reader device240is installed outside secure area830and lock device250is installed inside secure area830to wall810and door820. Sensor260, in this example, is a monitoring device mounted outside secure area830in an insecure area outside door820. Actuator270includes a motor used to control the field of view of the monitoring device in this example.

When a user enters credentials into reader device240(e.g., by entering a PIN, scanning an access card, scanning an iris, etc.), controller210may use the credentials to authenticate the identity of the user and may perform a lookup in an access rules table to determine whether to grant access to the user based on the identity of the user and the access rules. If controller210determines that access should be granted, controller210activates lock device250to unlock door820, thus granting access to the user to secure area830.

AlthoughFIG. 8shows exemplary components of physical layout800, in other implementations, physical layout800may include fewer components, different components, additional components, or differently arranged components than depicted inFIG. 8. Additionally or alternatively, any one component (or group of components) in physical layout800may perform a task or tasks described as performed by one or more other components of physical layout800.

FIG. 9is a floor plan diagram illustrating an exemplary physical layout900of DCS110. As shown inFIG. 9, physical layout900may include a building910with rooms920-A through920-F. A local network930, such as an Ethernet network, may interconnect system units115-A through115-F. In this example, system unit115-A controls two doors into room920-A; system unit115-B controls an outside door into room920-B; system unit115-C controls one door from room920-B to room920-C, system unit115-D controls one door from room920-C to room920-D; system unit115-E controls one door from room920-D to room920-E; and system unit115-F controls an outside door into room920-F.

In this example, system units115-A to115-F do not include a central controlling device (e.g., a server) and may include one or more distributed datasets. For example, system units115-A through115-F may maintain a distributed credentials table, a distributed access rules table, and/or a distributed events log. Assume an administrator uses administration device130to log into system unit115-A to add a user and to add credentials associated with a user. Those added credentials may be distributed to the other system units115that control doors to rooms to which the user has access. If system unit115-B fails, for example, data collected by system unit115-B may continue to be available as a result of a distributed events log included in the other system units.

AlthoughFIG. 9shows exemplary components of physical layout900, in other implementations, physical layout900may include fewer components, different components, additional components, or differently arranged components than depicted inFIG. 9. For example, in another embodiment, a central controlling device (e.g., a server) may be used in conjunction with one or more distributed datasets. Additionally or alternatively, one or more components of physical layout900may perform one or more tasks described as performed by one or more other components of physical layout900.

FIGS. 10A-10Care diagrams of an exemplary consensus loss scenario according to one or more implementations described herein.FIG. 1OA illustrates a possible network topology1000of system units115in physical layout900ofFIG. 9. As shown inFIG. 10A, local network930may include an Ethernet switch1010-A and an Ethernet switch1010-B. Ethernet switch1010-A may be connected to controllers of system units115-A,115-B, and115-C. Ethernet switch1010-B may be connected to controllers of system units115-D,115-E, and115-F. Ethernet switches1010-A and1010-B may be connected with an Ethernet cable. Assume that the Ethernet cable is damaged, resulting in broken link1020. Broken link1020may cause system units115-A,115-B, and115-C to be unable to communicate with system units115-D,115-E, and115-F.

FIG. 10Billustrates a table1030that may correspond to part of the information stored in consensus database358. As shown inFIG. 10B, table1030includes information relating to distributed datasets associated with system units115-A to115-F. Table1030may include information relating to a master database distributed dataset, a building counter distributed dataset, and a room C counter distributed dataset. A master database distributed dataset may be associated with system units115-A to115-F. The master database may store, for example, encrypted credentials information, access rules information, system units115configuration information, and/or other type of information. In one embodiment, in this example, all of system units115-A to115-F may be associated with the master database distributed dataset. Furthermore, a quorum of 4 system units may be required to reach a consensus relating to changes to the master database distributed dataset. In other embodiments in other examples, not all of system units115-A to115-F are necessarily associated with the master database distributed dataset.

The building counter distributed dataset may store information relating to a building counter application. The building counter application may keep track of people entering and leaving building910and may maintain information about how many people are currently inside building910. Since only system units115-B and115-F control exits for building910, only system units115-B and115-F are associated with the building counter distributed dataset. Furthermore, a quorum of 2 system units may be required to reach a consensus relating to changes to the building counter distributed dataset.

The room C counter distributed dataset may store information relating to a counter application for room920-C. For example, room920-C may correspond to a cafeteria and a room C counter application may keep track of how many people are currently in the cafeteria. Since only system units115-A,115-C, and115-D control doors to room920-C, only system units115-A,115-C, and115-D are associated with the room C counter distributed dataset. Furthermore, a quorum of 2 system units may be required to reach a consensus relating to changes to the building counter distributed dataset.

Broken link1020may result in the generation of consensus loss alarms.FIG. 10Cillustrates a signal flow1040that includes consensus loss alarms generated by particular ones of system units115-A to115-F in response to broken link1020. System unit115-A can no longer reach system units115-D,115-E, and115-F. Therefore, system unit115-A cannot reach the required quorum of 4 with respect to the master database distributed dataset. Therefore, system unit115-A may generate and send a master database consensus loss alarm1042to administration device130. Master database consensus loss alarm1042may list system units115-A,115-B, and115-C as the available devices. Furthermore, since system unit115-A can still reach system unit115-C, system unit115-A can reach the required quorum of 2 with respect to the room C counter distributed dataset and thus no loss of consensus will be detected for the room C counter distributed dataset.

System unit115-B can no longer reach system units115-D,115-E, and115-F. Therefore, system unit115-B cannot reach the required quorum of 4 with respect to the master database distributed dataset. Therefore, system unit115-B may generate and send a master database consensus loss alarm1044to administration device130. Master database consensus loss alarm1044may list system units115-A,115-B, and115-C as the available devices. Furthermore, system unit115-B cannot reach the required quorum of 2 with respect to the building counter distributed dataset. Therefore, system unit115-B may generate and send a building counter consensus loss alarm1046to administration device130. Building counter consensus loss alarm1046may list system unit115-B as the only available device.

System unit115-C can no longer reach system units115-D,115-E, and115-F. Therefore, system unit115-C cannot reach the required quorum of 4 with respect to the master database distributed dataset. Therefore, system unit115-C may generate and send a master database consensus loss alarm1048to administration device130. Master database consensus loss alarm1048may list system units115-A,115-B, and115-C as the available devices. Furthermore, since system unit115-C can still reach system unit115-A, system unit115-C can reach the required quorum of 2 with respect to the room C counter distributed dataset and thus no loss of consensus will be detected for the room C counter distributed dataset.

System unit115-D can no longer reach system units115-A,115-B, and115-C. Therefore, system unit115-D cannot reach the required quorum of 4 with respect to the master database distributed dataset. Therefore, system unit115-D may generate and send a master database consensus loss alarm1050to administration device130. Master database consensus loss alarm1050may list system units115-D,115-E, and115-F as the available devices. Furthermore, system unit115-D cannot reach the required quorum of 2 with respect to the room C counter distributed dataset. Therefore, system unit115-D may generate and send a room C counter consensus loss alarm1052to administration device130. Room C counter consensus loss alarm1052may list system unit115-D as the only available device.

System unit115-E can no longer reach system units115-A,115-B, and115-C. Therefore, system unit115-E cannot reach the required quorum of 4 with respect to the master database distributed dataset. Therefore, system unit115-E may generate and send a master database consensus loss alarm1054to administration device130. Master database consensus loss alarm1054may list system units115-D,115-E, and115-F as the available devices.

System unit115-F can no longer reach system units115-A,115-B, and115-C. Therefore, system unit115-F cannot reach the required quorum of 4 with respect to the master database distributed dataset. Therefore, system unit115-F may generate and send a master database consensus loss alarm1056to administration device130. Master database consensus loss alarm1056may list system units115-D,115-E, and115-F as the available devices. Furthermore, system unit115-F cannot reach the required quorum of 2 with respect to the building counter distributed dataset. Therefore, system unit115-F may generate and send a building counter consensus loss alarm1058to administration device130. Building counter consensus loss alarm1058may list system unit115-F as the only available device.

FIG. 11is a diagram of an exemplary user interface1100according to one or more implementations described herein. As shown inFIG. 11, user interface1100may be generated by administration device130in response to receiving the consensus loss alarms shown inFIG. 10C. User interface1100may include a master database distributed dataset list1110, a building counter distributed dataset list1120, and a room C counter distributed dataset list1130.

Master database distributed dataset list1110may include two lists. The first list may be generated based on information received from master database consensus loss alarms1042,1044, and1048, and the second list may be generated based on information received from master database consensus loss alarms1050,1054, and1056. In this case, the administrator may select one of the two lists as the new consensus-based system for the master database distributed dataset and de-select the other list in order to avoid a split-brain problem of having two different versions of the master database distributed dataset. For example, if the administrator selects system units115-A,115-B, and115-C, then system units115-D,115-E, and115-F will not be able to make changes to the master database.

Building counter distributed dataset list1120may include also two lists. The first list may be generated based on information received from building counter consensus loss alarm1046and the second list may be generated based on information received from building counter consensus loss alarm1058. In this case, the administrator may select both of the lists, which may result in a split-brain problem. However, the administrator may consider it important to continue to keep track of people entering and leaving the building and may choose to manually reconcile the building counter datasets from system units115-B and115-F.

Room C counter distributed dataset list1130may include a list generated based on information received from room C counter consensus loss alarm1052. The administrator may choose to ignore this alarm as the administrator may recognize, based on the configuration of the system, that system units115-A and115-C are still able to obtain a quorum with respect to the room C counter distributed dataset.

This application incorporates by reference the following applications filed the same day as the present patent application Ser. No. 14/028,218, titled “Event Timeline Generation” application Ser. No. 14/028,118, titled “Control System Configuration Within an Operational Environment” application Ser. No. 14/028,236, titled “Anonymous Decisions in an Access Control System” application Ser. No. 14/028,059, titled “Joining a Distributed Database” application Ser. No. 14/028,198, titled “Distributed Events in an Access Control System” application Ser. No. 14/028,230, titled “Managing Application Data in Distributed Control Systems”; and application Ser. No. 14/028,208, titled “Distribution of User Credentials”.

For example, while series of blocks have been described with respect toFIGS. 5 and 7, the order of the blocks may be modified in other implementations. Further, non-dependent blocks may be performed in parallel.