Abstract:
A system, method, computer program and article of manufacture for updateable fan-out replication with reconfigurable master association in a large, multi-node LDAP environment. A replication ring supports the addition of fan-out nodes as children to each primary node that sits on a replication ring. The fan-out nodes can be cascaded in multiple parent/child relationships and can support full replication or a subset of the parent data. Each child/parent relationship is defined by an agreement. Each fan-out node replicates changes to their immediate children and parent based upon the change details and the configured agreement, distributing the replication load.

Description:
RELATED APPLICATION  
       [0001]     This application claims the benefit of U.S. Provisional Application No. 60/632,922, entitled “UPDATEABLE FAN-OUT REPLICATION WITH RECONFIGURABLE MASTER ASSOCIATION”, filed Dec. 3, 2004, which is hereby incorporated by reference in its entirety. 
     
    
     BACKGROUND AND SUMMARY  
       [0002]     This invention relates to computer systems, and more particularly to replication of data.  
         [0003]     Data replication is the process of maintaining multiple copies of a database object in a distributed database system. Performance improvements can be achieved when data replication is employed, since multiple access locations exist for the access and modification of the replicated data. For example, if multiple copies of a data object are maintained, an application can access the logically “closest” copy of the data object to improve access times and minimize network traffic. In addition, data replication provides greater fault tolerance in the event of a server failure, since the multiple copies of the data object effectively become online backup copies if a failure occurs.  
         [0004]     One type of database application for which data replication is particularly useful is the replication of data for directory information systems. Directory information systems provide a framework for the storage and retrieval of information that is used to identify and locate the details of individuals and organizations, such as telephone numbers, postal addresses, and email addresses.  
         [0005]     One common directory system is a directory based on the Lightweight Directory Access Protocol (“LDAP”). LDAP is an object-oriented directory protocol that was developed at the University of Michigan, originally as a front end to access directory systems organized under the X.500 standard for open electronic directories (which was originally promulgated by the Comite Consultantif International de Telephone et Telegraphe “CCITT” in 1988). Stand alone LDAP server implementations are now commonly available to store and maintain directory information. Further details of the LDAP directory protocol can be located at the LDAP-devoted website maintained by the OpenLDAP Organization at http://www.openldap.org.  
         [0006]     LDAP directory systems are normally organized in a hierarchical structure having entries organized in the form of a tree, which is referred to as a directory information tree (“DIT”). The DIT is often organized to reflect political, geographic, or organizational boundaries. A unique name or ID (which is commonly called a “distinguished name”) identifies each LDAP entry in the DIT. An LDAP entry is a collection of one or more entry attributes. Each entry attribute has a “type” and one or more “values.” Each entry belongs to a particular object class. Entries that are members of the same object class share a common composition of possible entry attribute types.  
         [0007]     Some LDAP replication systems utilize a replication ring. The replication ring is a loop of primary LDAP nodes which replicate the same LDAP object across the nodes.  FIG. 1A  shows replication ring  100 , and primary nodes  102 ,  104 , and  106 . To add a node to the replication ring, a new node takes its place on the ring such as primary node  108 . Each primary node pushes its changes to each of the other primary nodes. Thus, each node pushes each change n−1 times, where n is the number of nodes. In a system with a large number of nodes, the replication process can be burdensome on each node. A solution is needed to add new nodes to a replication ring without creating additional replication burden to the existing nodes.  
         [0008]     Another drawback to some replication systems is that each node must be identical. This required updating to data on each node that may not be required on each node. This is burdensome and inefficient.  
         [0009]     One embodiment for adding new nodes to a replication ring without burdening the existing master nodes includes creation of fan-out nodes, where fan-out nodes are nodes that do not have to be placed on the replication ring.  
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0010]      FIG. 1A  is a representation of a replication ring with master nodes.  
         [0011]      FIG. 1B  is a representation of process  150 , the replicating LDAP directory overview.  
         [0012]      FIG. 2A  is a representation of a replication ring with fan-out nodes.  
         [0013]      FIG. 2B  is a representation of a master association reconfiguration.  
         [0014]      FIG. 3A  is a representation of process  300 , the change handling process for Fan-out replication purpose.  
         [0015]      FIG. 3B  is a representation of process  301 , the operation replay process for keeping parent consistent with child.  
         [0016]      FIG. 4  is a representation of a process to determine if a change should be propagated down to a child.  
         [0017]      FIG. 5  is a representation a system on which highly available updateable LDAP replication occurs.  
     
    
     DETAILED DESCRIPTION OF INVENTION  
       [0018]     The present solution is directed to replication of data in an LDAP multi-node environment. One embodiment involves the creation of fan-out nodes, and redirecting changes at the fan-out node to its primary node.  
         [0019]     In some embodiments a replication ring is a communication ring on which sit one or more nodes to be replicated. Nodes that sit directly on the replication ring are referred to as master nodes, or primary nodes. A primary node can have child nodes referred to as fan-out nodes. A fan-out node is a node in a replication network that does not sit on the replication ring. Each fan-out node can be a child to a primary node or a parent or a child to other fan-out nodes. An example embodiment of this parent/child relationships is shown in  FIG. 2A . Note this simplified illustration shows a small number of levels of nodes, however the embodiments are not limit in number of nodes or levels. Node  202  is one of the primary nodes on replication ring  200 . Fan-out nodes  204 ,  214 , and  216  are in communication with, and children of, primary node  202 . Fan-out nodes  206  and  208  are in communication with, and children of, fan-out node  204 . Fan-out nodes  210  and  212  are in communication with, and children of, fan-out node  208 . Each fan-out node can have an exact copy of its parent node&#39;s data, or in some embodiments, a subset of the parent node&#39;s data.  
         [0020]      FIG. 1B  shows an overview of the directory replication process  150 . In process action  152 , one or more primary nodes are created in a replication ring. One or more fan-out nodes are created as children of the primary nodes or as children of other fan-out nodes in process action  154 . A change is implemented in process action  156 . The change is replicated throughout the network in process action  158 .  
         [0000]     Agreement  
         [0021]     To define the parent to child relationship between nodes in the network, a “relationship agreement” is configured. One parameter of the agreement characterizes the nature of the relationship as uni-directional or bi-directional. A uni-directional relationship is one in which changes only pass from parent to child. A bi-directional relationship is one in which changes pass from child to parent in addition to passing from parent to child. Another parameter of the agreement can define which data the parent and child share. For example, the agreement can specify that a child node possesses only a subset of the parent&#39;s data.  
         [0022]     Other parameters of the agreement can define the types of changes that will pass between parent and child. For example, the agreement can specify to pass changes of a specific change type, changes for certain object classes, or changes for certain attribute types. In some embodiments the relationship is defined in a set of agreements. For example, data passing from parent to child can be defined in one agreement, while data passing from child to parent can be defined in another agreement.  
         [0000]     Master Association Reconfiguration  
         [0023]     Master association reconfiguration is the process by which the fan-out node re-associates itself. Re-association occurs when a master is no longer available (e.g. goes down, loses power). The rules of re-association are as follows.  
         [0024]     A first rule of re-association may provide for the case in which a parent node is unavailable and the parent node is a master in the replication ring. In this case, the immediate child node of the unavailable parent can re-associate to any master in the replication ring.  
         [0025]     A second rule of re-association may provide for the case in which the parent node that is unavailable is not in the replication ring but is a fan-out master. In this case, all the parent node&#39;s immediate children can re-associate to the parent of the unavailable master. In other words, the fan-out tree collapses.  
         [0026]     This process is illustrated in  FIG. 2B . For example, if node  208  lost power, child nodes  210  and  212  would re-associate to node  204 , which is the parent of  208 . If primary node  202  went down, child nodes  204 ,  214 , and  216  would re-associate to primary node  203 .  
         [0000]     Change Log  
         [0027]     Some embodiments utilize a change log. Changes are propagated throughout the replication network by directing a change log to each node. The change log records operational information and origin information. The operational information provides the receiving node the required information so that the receiving node can process the change. For example, the operational information may include effected entry(ies), effected attribute(s), and the change value(s). The origin information identifies the origin and immediate source of the change log so that the receiving node can determine whether the change needs to be propagated to its other relational nodes. For example, the origin information may include some, or all, of the following: 
        Type which indicates if the change log is original or regenerated     Origin Node: Node/Master where the user made the change     Origin Time: Time when the change was made     ID at Origin Node: ID for the change log where the change log originated     Regenerated Node: Node where the change was regenerated (Replication process applied the original change log and the change log was regenerated)     Regenerated Time: Time when the change was regenerated     ID at Regenerate Node: ID where the change was regenerated     Identity: Identity of user/process who performed the operation that resulted in change log creation or change log regeneration        
 
         [0036]     In some embodiments, the life of a change log lasts for one change at one node. Once that change is enacted at that node, the life of that particular change log is over. To continue to propagate the change throughout the network, the change log is regenerated. Each change log, regenerated or not, ceases to exist at the node once it is consumed by all its immediate child nodes. The regenerated change log includes all the same operational information as the original change log to allow the change to be implemented, however, the origin information will differ. The Identity, Regenerated Node, Regenerated Time and ID at the Regenerated Node are different for each regenerated change log, while the Origin Node, Origin Time, and ID at the Origin Node remain the same for each regeneration of the change log for a particular change.  
         [0000]     Change Management  
         [0037]     An LDAP information system can be used to provide a framework for the storage and retrieval of information that is used to identify and locate the details of individuals and organizations, such as telephone numbers, postal addresses, and email addresses. Recall from above that LDAP directory systems are normally organized in a hierarchical structure having entries organized in the form of a tree, which is referred to as a directory information tree (“DIT”), which may be organized to reflect political, geographic, or organizational boundaries. A unique name or ID identifies each LDAP entry, which is a collection of one or more entry attributes.  
         [0038]     A change to the database can come in the form of an attribute change, or an entry addition, deletion or rename. The changes can originate at any node in the system including fan-out nodes. One embodiment of process  300  is shown in  FIG. 3A . In process  300  a change occurs at a node which results in change log creation or regeneration and this change log is received by the child node for its consumption. In another embodiment, when an operation takes place at a node, it is also replayed at its immediate parent, if the agreement allows, therefore realizing the operation at the concerned node and its immediate parent. This replaying process, process  301 , is illustrated in  FIG. 3B .  
         [0039]     Process  300  in  FIG. 3A  is triggered each time a node retrieves a change from its parent/master, independent of the origin of the change. That is, it does not matter if the change at the parent/master originated at that node or was propagated from another node, the change management process is the same. A change is received in process action  302 . That is, an original change log or a regenerated change log is received. The change is then evaluated as per agreement in process action  303 , and if it qualifies, then it is implemented in the local directory in process action  304 . If the change does not qualify, in process action  303 , then the process  300  stops. Both the relationship agreement and the change log are taken into consideration when deciding whether to propagate changes. Process action  303  can also take place before the change is received by the node and for security reasons the parent does not even send it to the child if it does not qualify according to the agreement. Process  300  includes Downward Propagation Process  312 , while Upward Propagation Process  314  is described in Process  301  as illustrated in  FIG. 3B .  
         [0040]     In determining whether to propagate changes down to a node&#39;s children, process action  306  in  FIG. 3A  considers the relationship agreement and the change log. If the change log indicates a change which falls into the realm of changes for which the relationship agreement between the child node and the parent node has been configured, and the change log has not come from the child under consideration, the change will be propagated down to that child node. If the change log indicates a change which does not fall into the realm of changes for which the relationship agreement between the child node and the parent node has been configured, or if the change came from that child node (i.e., the origin node is the child node), the change will not be propagated down to that child node. This determination is made for each child node individually. An example illustration of this process is shown in  FIG. 4 .  
         [0041]      FIG. 4  is a representation of a process  400  to determine if a change should be propagated down to a child. As shown in  FIG. 4 , a change received from a parent may be evaluated in process step  402 . In process step  404 , it may be determined whether the origin node for the change is the child node, i.e., it is determined whether the change came from the child node. If it is determined that the origin node for the change is the child node, the process may stop.  
         [0042]     If it is determined that the origin node is not the child node, it may be determined in process step  406  whether the change is in the agreement between the child and the parent. In other words, it is determined whether the falls into the realm of changes for which the relationship agreement between the child node and the parent node has been configured. If it is determined that the change is not in the agreement, the process may stop.  
         [0043]     If it is determined that the change is in the agreement, the change may be realized at the child in process action  408 . For example, the change may be realized as in process step  308  of  FIG. 3A .  
         [0044]      FIG. 3B  is a representation of process  301 , the operation replay process for keeping parent consistent with child. Process  301  is triggered when an operation is performed at the fan-out node by an end user in process action  315 . The local database is updated as a result of the operation in process action  316  and a change log is created, as it is for any directory operation, in process action  317 . If the agreement for upward propagation requires the operation to be replayed at the parent, as determined by process action  318 , the directory of the parent is modified in process action  319 , and operation  301  may then be replayed at the parent node.  
         [0045]     In determining whether an operation needs to be replayed at its immediate parent node, process action  318  in process  301  considers the relationship agreement and the operation. If the relationship agreement between the child node and the parent node indicates a bi-directional relationship which encompasses the operation to be replayed at the parent, then the same operation will be replayed immediately at the parent in process action  319 . If the relationship agreement between the child node and the parent node does not indicate a bi-directional relationship, or the bi-directional relationship does not encompasses the changes indicated in the operation, the operation will not be replayed to the parent node and the Upward Propagation Process  314  stops.  
         [0046]     Lower level details regarding the replication process can be found in U.S. Pat. No. 6,615,223, which is hereby incorporated by reference as if fully set forth herein.  
       ILLUSTRATIVE EXAMPLE  
       [0047]     The following example illustrates process  300  in  FIG. 3A , and refers to the fan-out node in  FIG. 2A . This description is for illustrative purposes only and is not meant to limit the embodiments. Each relationship can be individually configured is not limited by this example.  
         [0048]     For illustration purposes, suppose fan-out node  208  receives a change for an operation that took place at node  204  ( 302 ). The change log received by  208  will have the origin information as node  204 . The change is then evaluated and if it complies with the agreement ( 303 ) it is implemented at local directory  208  ( 304 ). It is determined that there are child nodes  210  and  212  from  208  ( 306 ) and therefore a change log is regenerated at node  208  ( 308 ). The change is then propagated down to children  210  and  212  by process  300 . The regenerated change log at node  208  reflects origin information of node  204  and regenerated node information of node  208 .  
         [0049]     Each node  210  and  212  receive the change ( 302 ), evaluate if the change needs to be realized at the local directory ( 303 ) and if true, modify their local directory ( 304 ), and determine the regeneration of the change ( 306 ). Since neither node  210  nor  212  have children this downward propagation stops here.  
         [0050]     Consider another operation that is performed at node  208  by end user ( 315 ) which triggers process  301 . Operation is performed at the local node  208  ( 316 ) and a change log is created for this operation ( 317 ). It is determined that the operation at node  208  is to be replayed to its parent node  204  ( 318 ). That is, the relationship agreement for node  208  and  204  specifies a bi-directional relationship that includes the particular type of operation/data to be propagated upwards. The operation is then replayed at the parent node  204  as if node  208  is performing the operation on behalf of end-user ( 319 ) and that completes the upward change propagation.  
         [0051]     If the change were to be propagated up to node  202 , then the change is propagated throughout the replication ring  200  to keep the replication ring nodes identical. In using these embodiments a replication network can be configured with particular relationship agreements which control the propagation of changes throughout a network without requiring that each node implement each change.  
         [0000]     System Architecture Overview  
         [0052]     The execution of the sequences of instructions required to practice the invention may be performed in embodiments by a computer system  1400  as shown in  FIG. 5 . In an embodiment, execution of the sequences of instructions required is performed by a single computer system  1400 . According to other embodiments, two or more computer systems  1400  coupled by a communication link  1415  may perform the sequence of instructions required in coordination with one another. In order to avoid needlessly obscuring the embodiments, a description of only one computer system  1400  will be presented below; however, it should be understood that any number of computer systems  1400  may be employed.  
         [0053]     A computer system  1400  according to an embodiment will now be described with reference to  FIG. 5 , which is a block diagram of the functional components of a computer system  1400 . As used herein, the term computer system  1400  is broadly used to describe any computing device that can store and independently run one or more programs.  
         [0054]     Each computer system  1400  may include a communication interface  1414  coupled to the bus  1406 . The communication interface  1414  provides two-way communication between computer systems  1400 . The communication interface  1414  of a respective computer system  1400  transmits and receives electrical, electromagnetic or optical signals, that include data streams representing various types of signal information, e.g., instructions, messages and data. A communication link  1415  links one computer system  1400  with another computer system  1400 . For example, the communication link  1415  may be a LAN, in which case the communication interface  1414  may be a LAN card, or the communication link  1415  may be a PSTN, in which case the communication interface  1414  may be an integrated services digital network (ISDN) card or a mode, or the communication link  1415  may be the Internet, in which case the communication interface  1414  may be a wireless modem, a digital modem a cable model or an network card.  
         [0055]     A computer system  1400  may transmit and receive messages, data, and instructions, including program, i.e., application, code, through its respective communication link  1415  and communication interface  1414 . Received program code may be executed by the respective processor(s)  1407  as it is received, and/or stored in the storage device  1410 , or other associated non-volatile media, for later execution.  
         [0056]     In an embodiment, the computer system  1400  operates in conjunction with a data storage system  1431 , e.g., a data storage system  1431  that contains a database  1432  that is readily accessible by the computer system  1400 . The computer system  1400  communicates with the data storage system  1431  through a data interface  1433 . A data interface  1433 , which is coupled to the bus  1406 , transmits and receives electrical, electromagnetic or optical signals, that include data streams representing various types of signal information, e.g., instructions, messages and data. In embodiments, the functions of the data interface  1433  may be performed by the communication interface  1414 .  
         [0057]     Computer system  1400  includes a bus  1406  or other communication mechanism for communicating instructions, messages and data, collectively, information, and one or more processors  1407  coupled with the bus  1406  for processing information. Computer system  1400  also includes a main memory  1408 , such as a random access memory (RAM) or other dynamic storage device, coupled to the bus  1406  for storing dynamic data and instructions to be executed by the processor(s)  1407 . The main memory  1408  also may be used for storing temporary data, i.e., variables, or other intermediate information during execution of instructions by the processor(s)  1407 .  
         [0058]     The computer system  1400  may further include a read only memory (ROM)  1409  or other static storage device coupled to the bus  1406  for storing static data and instructions for the processor(s)  1407 . A storage device  1410 , such as a magnetic disk or optical disk, may also be provided and coupled to the bus  1406  for storing data and instructions for the processor(s)  1407 .  
         [0059]     A computer system  1400  may be coupled via the bus  1406  to a display device  1411 , such as, but not limited to, a cathode ray tube (CRT), for displaying information to a user. An input device  1412 , e.g., alphanumeric and other keys, is coupled to the bus  1406  for communicating information and command selections to the processor(s)  1407 .  
         [0060]     According to one embodiment, an individual computer system  1400  performs specific operations by their respective processor(s)  1407  executing one or more sequences of one or more instructions contained in the main memory  1408 . Such instructions may be read into the main memory  1408  from another computer-usable medium, such as the ROM  1409  or the storage device  1410 . Execution of the sequences of instructions contained in the main memory  1408  causes the processor(s)  1407  to perform the processes described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the invention. Thus, embodiments of the invention are not limited to any specific combination of hardware circuitry and/or software. Logic refers to hardware, software or any combination of the two.  
         [0061]     The term “computer-usable medium,” as used herein, refers to any medium that provides information or is usable by the processor(s)  1407 . Such a medium may take many forms, including, but not limited to, non-volatile, volatile and transmission media. Non-volatile media, i.e., media that can retain information in the absence of power, includes the ROM  1409 , CD ROM, magnetic tape, and magnetic discs. Volatile media, i.e., media that can not retain information in the absence of power, includes the main memory  1408 . Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise the bus  1406 . Transmission media can also take the form of carrier waves; i.e., electromagnetic waves that can be modulated, as in frequency, amplitude or phase, to transmit information signals. Additionally, transmission media can take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.  
         [0062]     In the foregoing specification, the embodiments have been described with reference to specific elements. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope. For example, the reader is to understand that the specific ordering and combination of process actions shown in the process flow diagrams described herein is merely illustrative, and the invention can be performed using different or additional process actions, or a different combination or ordering of process actions. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.