Abstract:
A data management system with data stored in multiple disparate formats in synchronized stores, method of synchronizing the data and recovering from synchronization failures and program product therefor. Data changes in one data store are cached in a universal format in an active synchronizer and forwarded to a second store from the universal format cache. Standby synchronizers provide failover handling by identifying synchronization failures and self-selecting a replacement synchronizer to serve as the active synchronizer.

Description:
BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention is related to data management and more particularly, to reliably maintaining synchronized data stored in disparate data formats. 
     2. Background Description 
     Frequently, data is collected and shared amongst a number of applications. Each application may require presenting the data in a specific format, e.g., in a relational database, while another may require the same data as files stored hierarchically as a collection of files in central storage. Consequently, whenever data is being shared amongst a number of applications, it is likely that at least one application requires the data in a different format than the others. Thus, when data is shared by a number of applications, it is likely that the applications require the data in disparate formats. Accordingly, each format must be current and accurately reflect the same data content stored in each other format. 
     For example, a business concern may collect and maintain data for its employees, such as personnel, pay and user data. Each of the various applications, e.g., for presence based communications applications, personnel management applications and business communications applications, may require the data in a specific format that is different from and incompatible with the others. For example, one application may require hierarchically formatted data, e.g., stored in Active Directory. Another may require the data in a relational database, e.g., SQL Server. Status changes for each employee (e.g., an employee is promoted, dies or is terminated), require data updates the in each format in each store location and so, the data stores must be reliably synchronized. 
     Unfortunately, if synchronization is prevented/disrupted (i.e., the synchronization fails), old data may cause errors that may continue until the failure is subsequently discovered. Worse still, subsequent updates may obfuscate the failure and make recovery from such errors more difficult. For example, a terminated employee may be marked in personnel records as such, but continue to have remote access to company resources and sensitive information. Direct deposits may continue to a deceased employee&#39;s checking account long after some records reflect the employee&#39;s demise. Consequently, synchronizing shared data in disparate formats may be of critical importance. 
     Thus, there is a need for automatically synchronizing data stored in disparate formats and, more particularly, for automatically recovering from synchronization failures to shared data content stored in multiple disparate formats. 
     SUMMARY OF THE INVENTION 
     It is a purpose of the invention to reliably provide shared data in disparate formats; 
     It is another purpose of the invention to reliably synchronize contents of files in disparate formats storing shared data; 
     It is yet another purpose of the invention to seamlessly recover from failures in synchronizing contents of files storing shared data in disparate formats; 
     It is yet another purpose of the invention to automatically identify synchronization failures in synchronizing contents of files storing shared data in disparate formats and seamlessly recover from such failures. 
     The present invention relates to a data management system with data stored in multiple disparate formats in synchronized stores, method of synchronizing the data and recovering from synchronization failures and program product therefor. Data changes in one data store are cached in a universal format in an active synchronizer and forwarded to a second store from the universal format cache. Standby synchronizers provide failover handling by identifying synchronization failures and self-selecting a replacement synchronizer to serve as the active synchronizer. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The foregoing and other objects, aspects and advantages will be better understood from the following detailed description of a preferred embodiment of the invention with reference to the drawings, in which: 
         FIG. 1  shows an example of a preferred embodiment data synchronization system with a failover facility for maintaining data synchronization of shared data stored in multiple disparate formats; 
         FIG. 2  shows an example of a dynamically loaded XML configuration file; 
         FIG. 3  shows an example of a source data object, an universal data object and a target data object; 
         FIGS. 4A-B  show general and specific UDC examples; 
         FIG. 5  shows an example of self-selecting the active synchronizer at start up; 
         FIG. 6  shows a flow diagram example of failover handling using a preferred Progress-Connector. 
     
    
    
     DESCRIPTION OF PREFERRED EMBODIMENTS 
     Turning now to the drawings and more particularly,  FIG. 1  shows an example of a preferred embodiment data management system  100  with failover facility for maintaining data synchronization of shared data stored in multiple disparate formats according to the present invention. The preferred data synchronization system  100  includes at least one active synchronizer  102  receiving data updates to shared data stored in a source store  104  in one format, e.g., in hierarchical format. The active synchronizer  102  caches updates in a universal format and forwards cached updates to one or more target store(s)  106  for storage in another format, e.g., in a relational database. In some applications serial synchronization may be desired, e.g., from a first store in a first format to a second store in a second format and, then, from the second store to a third store in a third format. For such serial applications, the active synchronizer  102  treats the second store as the target store  106  in the first synchronization and, then, as the source store  104  in the second synchronization. The active synchronizer  102  simultaneously monitors synchronization progress and logs the progress in a progress store  108 . The active synchronizer  102  hosts and maintains an Universal Data Cache (UDC)  110  that caches updates. An extensible Markup Language (XML) configuration file  112  defines the UDC  110 , an In-Connector  114  to the source store  104 , an Out-Connector  116  for each target store  106  and a Progress-Connector  118  to the progress store  108 . The connectors  114 ,  116 ,  118  may be dynamically created at runtime, for example, using the Reflection transformation in the NET framework from Microsoft Corporation. 
     For example, a preferred embodiment system may be deployable with two selectable modes as selected and defined by the XML configuration file  112 . In a first mode the system may have as the source store  104  an extended schema capability in, for example, HiPath® OpenScape™ Active Directory (AD) Connector services from Siemens Communication Inc. In this first mode, the active synchronizer  102  may synchronize an SQL Server database in target store  106  with updates to the AD source store  104 . In the second unextendable schema mode, the one store acts first as the target store  106  and then, as the source store  104 . So first, the active synchronizer  102  may be synchronizing user data from the AD source store  104  into an Active Directory in Application Mode (ADAM) in target store  106 . Then, the active synchronizer  102  synchronizes all data from ADAM target store  104  to the SQL target store  106 . 
     Although shown with a single synchronizer  102  in this example, preferably, the data synchronization system  100  includes multiple synchronizers deployed on separate servers (not shown) sharing the progress store  108 , with only one synchronizer  102  active at a given time and the other remaining synchronizer(s) in standby. The progress store  108  stores at least sufficient information to determine synchronization status and sufficient information to recover from an identified failure. For example, the progress store  108  may include an indicator (ReplicationInProgress) that identifies the state of the replication process; an active synchronizer ID (ActiveSynchronizer) that identifies which synchronizer currently is running in active mode; and, a timestamp (LastSynchronization) that indicates the time the last successful synchronization occurred. The progress store  108  can be of any type of storage. However, preferably, the progress store is an SQL database or XML file for dynamic modification. Accordingly, a static store such as an Active Directory is least preferred. 
     The active synchronizer  102  periodically synchronizes data between two data stores  104 ,  106 , while all of the synchronizers monitor synchronization progress to intercept and recover from update failures. When the active synchronizer  102  fails to start a scheduled synchronization, the Progress-Connector  118  manages automatically self-selecting one standby synchronizer and designating the selected synchronizer as active, as described in more detail hereinbelow. The progress store  108  is shared by all synchronizers through the Progress-Connector  118 . Since the progress store  108  is otherwise isolated from the active synchronizer  102 , any data store (even the target store  106 ) may serve as the progress store  108 . 
     In particular for the above HiPath® OpenScape™ example, the In-Connector  114  may be a generic Lightweight Directory Access Protocol (LDAP) In-Connector retrieving changes from AD and ADAM. This generic In-Connector may use standardized LDAP DirSync control to retrieve changes from the source store  104 . The Out-Connector  106  may be a LDAP Out-Connector, for each update storing the data changes and cookie into the ADAM, i.e., for an ADAM target store  106 . Since access to a SQL database target store  106  must be through the data access layer, the Out-Connector  116  to a SQL database target store  106  may be a generic SQL Out-Connector implemented in the HiPath® OpenScape™ Data Access Layer, for example. 
       FIG. 2  shows an example of a XML configuration file  112 , e.g., from a computer-readable medium such as stores  104 ,  106  or  108 , with library names and object names defining the connectors  114 ,  116  that are loaded dynamically. The XML configuration file  112  also defines the Progress-Connector  118 , which does not include object mapping. The Progress-Connector  118  may be dynamically created at runtime and connects the active synchronizer  102  to the shared progress store  108  for failover handling. Standby synchronizers also connect to the shared progress store  108  through the Progress-Connector  118 . The active synchronizer  102  parses the XML configuration file  112  to create the UDC  110  and dynamically create the connectors  114 ,  116 ,  118 , which may be modified/updated dynamically at run-time. 
     Advantageously, since the active synchronizer  102  monitors the XML configuration file  112  and caches changes/updates in the UDC, the active synchronizer  102  may add other target stores at runtime without stopping. Since a newly added store does not contain cookies, instead of a delta synchronization, the active synchronizer  102  detects the absence of a valid cookie, which triggers a full store synchronization for the new store. Further, the connectors  114 ,  116  may be updated in the XML configuration file  112  and dynamically loaded. So, the active synchronizer  102  does not require code changes to change object and attributes for the connectors  114 ,  116 . This avoids shutting down the active synchronizer  102  to change the connectors because XML configuration file  112  changes are applied to upgrade the connectors  114 ,  116  at runtime. 
     The Progress-Connector  118  stores progress information dynamically in the progress store  108 . Thus since the progress store  108  and Progress-Connector  118  are shared with all synchronizers, standby synchronizers can monitor each synchronization to identify when the active synchronizer  102  is failing or has failed and self-select a new active synchronizer, when necessary. During the first synchronization, all of the synchronizers participate in self-selecting the synchronizer with the shortest synchronization interval as the active synchronizer  102 . If all synchronizers use the same interval the synchronizer first executing a synchronization becomes active and remains the active synchronizer  102  until an error, i.e., the active synchronizer  102  misses a synchronization. The Progress-Connector  118  sets the ReplicationInProgress flag during each synchronization to prevent the active synchronizer  102  from restarting synchronization and to prevent standby synchronizers from going active. The Progress-Connector  118  also prevents race conditions from two synchronizers simultaneously checking the ReplicationInProgress flag by locking the ReplicationInProgress flag for one of the two. 
       FIG. 3  shows an example with reference to  FIG. 1  of a source data object  120  from source store  104 , represented as an universal data object  122  cached in the UDC  110  and, as a target data object  124  provided to a target store  106 . The XML configuration file  112  defines the form of objects traversing each connector  114 ,  116 ,  118  and attributes of data traversing each. The In-Connector  114  retrieves changes/updates from the source store  104  and attaches an object name and collects and formats change data as source data objects  120 . Each source data object is also associated with a cookie that identifies data changes/updates (e.g., with a timestamp) since the last synchronization interval. Thus, the cookie insures that target store(s) only receive current delta changes on the next synchronization. The UDC  110  modifies and caches each source data object  120  as an universal data object  122  and cookie. The Data Out-Connector  116  converts the universal data object  122  to a target data object  124  that is provided to the target store  106 . Only the In-Connector  114  and the Out-Connectors  116  contain object metadata for mapping universal data objects to another format, i.e., the source object and attribute names into a universal name and then, the universal names to the target object and attribute names. Thus, metadata is different for each connector  114 ,  118  and, especially for each Out-Connecter in a system  100  with multiple target stores. 
     After formatting a source data object  120 , the In-Connector  114  passes the source data object  120  and its associated cookie to the active synchronizer  102 . In the active synchronizer  102 , the UDC  110  contains source and target data format definitions and cached update objects (preferably, all as metadata that include the synchronized data itself), as well as cached corresponding cookies for each object. The synchronizer  102  also monitors the XML configuration file  112  for changes (e.g., added target stores) and dynamically updates metadata in the UDC  110  for any identified changes. Each Out-Connector  116  converts the normalized data to the appropriate format and forwards the formatted data to the corresponding target data store  106 . The UDC  110  seamlessly mates specifically formatted files in data stores  104 ,  106 . So, although an update enters in one format and exits in one or more other formats, as an object passes from the In-Connector  114  through the synchronizer  102  to the Out-Connector  116 , the object maintains the same normalized format. 
       FIGS. 4A-B  show a more detailed general example of UDC  110  and, a specific example with the source data object  122  of  FIG. 3  cached in the UDC  110 . As noted hereinabove, the UDC  110  stores connector metadata  132 ,  134 - 136 , cached data change objects  138 - 140  and associated cookies  142 - 144 . In this example, the Progress-Connector  118  is implemented as an Out-Connector and represented as one of the connector metadata  132 ,  134 - 136 . Thus, connector metadata (except the Progress-Connector  118 ) includes for each In-Connector and Out-Connector, a library and object name  146 , connector settings  148  and object metadata  150 - 152 . Each object metadata includes a source object name  154 , a target object name  156  and attribute metadata  158 . Attribute metadata  158  includes a source attribute name  160 , a target attribute name  162  and an indication whether the attribute metadata  158  is a primary key  164 . Changes/updates in the XML configuration file  112  are reflected in metadata  132 ,  134 - 136 . Further, the relationships between data formatted for the source store and target store are indicated in the example of  FIG. 4B  with the linking arrows  166 ,  168 ,  170 . Thus, as can be seen, the UDC  110  is self-describing such that the data objects  138 - 140  are represented within the synchronizer  102  as normalized data, independent of object and attribute naming requirements for either the source store  104  and/or the target store  106 . Advantageously, In-Connector  114  and Out-Connector(s)  116  may have any suitable configuration for any selected data format. 
       FIG. 5  shows an example of a method of self-selecting and starting the active synchronizer, e.g.,  102  in  FIG. 1 . First in step  180 , the active synchronizer  102  dynamically instantiates the connectors using the library name and object name of the connector from the XML configuration file  112 . In step  182  the active synchronizer  102  creates the UDC  110  from the metadata of the objects and attributes from the XML configuration file. Then in step  184 , the active synchronizer  102  sets up a watcher to monitor the XML file  112  for runtime changes, e.g., modifications to the existing Out-Connectors; adding another Out-Connector; adding objects/attributes to be synchronized; changing synchronizer settings such as the synchronization interval. Then, in step  186  the active synchronizer  102  sets up a synchronization interval timer. In step  188  the active synchronizer  102  begins updating for changes/updates and all of the synchronizers begin monitoring for impending failures. After each synchronization interval  190  monitoring pauses, e.g. every 60 sec. In step  192  each synchronizer executes the failover logic guided by the Progress-Connector  118  to check whether it is the active synchronizer. The active synchronizer also determines whether it should start the synchronization interval. In step  194 , the active synchronizer requests that the Out-Connector(s)  116  get the cookie(s) from the target store(s)  106  that are stored in the UDC  110 . In step  196  changes that have occurred since the last synchronization interval and associated cookies are retrieved/received through the In-Connector  114  and cached in the UDC  110 . In step  198  the cached changes are passed to the Out-Connector(s)  116  and to the target store(s) with the associated cookie. 
       FIG. 6  shows a flow diagram  200  example of failover handling using a Progress-Connector (e.g.,  118  in  FIG. 1 ) according to a preferred embodiment of the present invention. The First synchronization interval  202  begins with the Progress-Connector  118  implementing a defined interface to access the progress store  108 . In step  204  the active and standby synchronizers read progress information retrieved from the progress store  108 . Each synchronizer uses the failover logic to check whether it is required to execute a synchronization interval and stay (or become) the active synchronizer. So, in step  206  each synchronizer checks whether an update is already in progress, i.e., another synchronizer is active. If another synchronizer is active, then in step  208 , the synchronizer ends the synchronization interval. However, if another synchronizer is not active, then in step  210 , the synchronizer checks whether it is designated as the active synchronizer or, the current synchronization is the first synchronization. If another synchronizer is currently active (i.e., it is not active) and a previous synchronization has occurred, then in step  212 , the synchronizer checks whether the active synchronizer has missed or failed to complete an update. If the active synchronizer has not missed or failed to complete an update, then in step  208 , the synchronizer ends the synchronization interval. Otherwise, in step  214  the active synchronizer has failed and the synchronizer replaces the failed synchronizer. The active synchronizer (either identified in step  210  or newly designated in step  214 ) begins the update in step  216  by setting a progress flag in the progress store. In step  218  the target stores are synchronized. Then, in step  220  the active synchronizer is designated active host in the progress store  108  and in step  208  the active synchronizer ends the synchronization interval. 
     Optionally, for a single synchronizer system or where the failover facility is unnecessary, a progress store is unnecessary. The XML configuration file need not list a Progress-Connector in a single synchronizer embodiment. Instead, the synchronizer may use a default Cache Progress-Connector that keeps the progress information in the Progress-Connectors memory cache. Otherwise, however, the single synchronizer functions identically to the above described preferred multiple synchronizer embodiments with XML defined Progress-Connectors. 
     Advantageously, the failover facility of the Progress-Connector and progress store provides a unique solution to significantly ameliorate data synchronization failures. The preferred failover facility insures that a single active synchronizer much more reliably synchronizes multiple target data stores with data stored in disparate formats with updates from a single source. Special storage is not required for the progress store because the Progress-Connector can use any data format. So, the progress store may be part of one target store, e.g., a SQL database. Thus, the preferred data synchronization system has wide application wherever reliable data synchronization is required. 
     Additionally, the self-describing universal format of data cached in the UDC is independent of the needs of In-Connectors and Out-Connectors, as well as the Progress-Connector. So, the synchronizer can synchronize data from any source store in any format to any target store in any other format. Also, because the synchronizers continually monitor the XML configuration file, the data synchronization system is flexible enough that it may be reconfigured in runtime by changing the XML configuration file and without changing the synchronizer. New data stores, even storing data in formats not previously handled, can easily be plugged in by changing the XML configuration file. New connectors can be added in runtime for the new stores and created using off-the-shelf tools. Typical such formats may include but are not limited to, for example, data structures such as an LDAP directory, SQL database, XML or any other defined structure files. Generic Connectors such as a LDAP In-Connector used in HiPath™ OpenScape™ can be used in any synchronizer and are fully configured by the XML configuration file. Optionally, a custom connector may be easily created, for a HiPath™ OpenScape™ SQL Out-Connector. Thus, a preferred embodiment system eliminates the need for a new custom designed synchronization system for each situation. 
     While the invention has been described in terms of preferred embodiments, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims. It is intended that all such variations and modifications fall within the scope of the appended claims. Examples and drawings are, accordingly, to be regarded as illustrative rather than restrictive.