Abstract:
Retention of data is managed and data holds are enforced. At least one event consumer data source contains data that is subject to a retention schedule that defines rules for the disposition of the data. An enterprise retention management (ERM) application accepts at least one hold request that defines at least a portion of the data that must be held from disposition. A disposition request is also accepted by the ERM that comprises an instruction to dispose of at least a portion of the data based on the rules within the retention schedule. The disposition request is filtered to dispose of the portion of data identified in the disposition request minus the held data, which comprises disposition survivor data. An event-driven disposition mechanism disposes of the disposition survivor data as soon as possible after the disposition survivor data is released from hold or unlocked.

Description:
BACKGROUND OF THE INVENTION 
     1. Technical Field 
     The invention relates to systems for the disposition of data. More specifically, the invention relates to a method and apparatus for handling edge-cases of event-driven disposition. 
     2. Description of the Related Art 
     Business organizations retain electronic documents, records and other data in storage for extended periods of time for a number of reasons including easy access, internal policy, and regulation compliance, among other various reasons. For instance, government regulation may require an organization to retain certain securities information for a given duration for SEC compliance. Likewise, some organizations retain electronic records of documents for audit and/or litigation purposes. 
     Some known data storage systems involve storing data with an associated retention mark. The retention mark indicates a time period for the retention of the data and when the time period lapse, the data is typically disposed of automatically. Currently, a system administrator must manually manipulate the retention for a given datum when the retention is triggered by an external event. Organizations managing a large amount of stored data incur time-consuming and costly expenses in performing the updating and data disposition manually. 
     Furthermore, an error in data disposition may result in dire consequences. For instance, in cases where data was not disposed of, too much data has been disposed of, or wrong data has been disposed of, an organization may incur unwanted legal and business consequences. Therefore, there is a need in eliminating human factor from data disposition as much as possible. 
     There is no uniform view on how to manage disposition of data. This needs to be changed to get under control growing storage and legal costs associated with storing unnecessary information. 
     Different types of data is associated with different retention schedules, i.e. rules describing how long the information should be preserved in a certain data source, what is the event that triggers measuring of the disposition period, and what should be done with the information when the disposition period is due. 
     The methods of automatic event driven disposition described in U.S. patent application Ser. No. 12/164,468, filed on Jun. 30, 2008, published as United States Patent Publication No. 2009/0328070 and the methods of handling files on hold described in aU.S. patent application Ser. No. 12/164,524, filed on Jun. 30, 2008, published as United States Patent Publication No. 2009/0326969, now U.S. Pat. No. 7,792,945 address the functionality of an Information Lifecycle Management System that automatically disposes of the documents in multiple data sources that may be a subject for a legal hold. However, they do not provide expected results when it comes to:
         Making sure that documents that have been on hold during the execution of a disposition request are disposed of according to their retention schedules after they have been released from hold. Hereinafter, we use the term “Disposition Survivors” for these documents.   Making sure that disposition of documents including Disposition Survivors is handled properly in scenarios when a retention schedule changes   Making sure past events are handled properly.       

     It would be advantageous to provide a method and apparatus for handling edge-cases of event-driven disposition. 
     SUMMARY OF THE INVENTION 
     An embodiment of the invention provides a method and apparatus for handling edge-cases of event-driven disposition. In an embodiment, a method and apparatus for managing retention of data and for enforcing data holds, comprises at least one event consumer data source containing data that is subject to a retention schedule that defines rules for the disposition of the data. An enterprise retention management (ERM) application is provided to which the at least one event consumer is coupled. Means are provided for accepting at least one hold request in the ERM. The hold request defines at least a portion of the data that must be held from disposition, thus forming held data. Means are also provided for accepting a disposition request in the ERM. The disposition request comprises an instruction to dispose of at least a portion of the data based on the rules within the retention schedule. Means are further provided for filtering the disposition request, thus forming a filtered request comprising an instruction to dispose of a portion of data comprising the portion of data identified in the disposition request minus the held data. The held data comprises disposition survivor data. Means are also provided for executing the filtered request. Finally, an event-driven disposition mechanism is provided for disposing of the disposition survivor data as soon as possible after the disposition survivor data is released from hold or unlocked. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block schematic diagram showing a business event driven disposition method according to the invention; 
         FIG. 2  is a timeline showing a disposition survivor according to the invention; 
         FIG. 3  is a flow diagram showing the performance of a disposition in a disposition survivor-safe consumer according to the invention; 
         FIG. 4  is a flow diagram showing logic for sending disposition reminders to simple disposition survivor-unsafe consumers according to the invention, optional logic is indicated in  FIG. 4  with a dashed line; and 
         FIG. 5  is a flow diagram showing logic for sending disposition reminders to retention-capable disposition survivor-unsafe consumers according to the invention, optional logic is indicated in  FIG. 5  with a dashed line. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       FIG. 1  is a block schematic diagram of a business event driven disposition method according to the invention. One or more event producer data source  101 ,  102  through n are provided. In some embodiments of the invention, the event producer data sources  101 ,  102  through n transmit information about business events. Examples of business events include the termination of an employee, an employee transfer, a change in familial status of an employee, a management decision to discontinue a product or a management decision to discontinue a service. Although these specific examples of business events are disclosed explicitly, it will be readily apparent to those having ordinary skill in the art that any business event that can affect the retention of data is equally envisioned. In some other embodiments of the invention, the data sources  101 ,  102  through n transmit hold instructions. Hold instructions comprise a readable instruction defining a portion of data that must be put on hold from disposition. 
     Business events are uploaded to an Enterprise Retention Management System  105  (hereinafter referred to as an “ERM”). In some embodiments of the invention, the event producer data sources  101 ,  102  through n push the business event to the ERM  105 . In other embodiments, the business event is pulled from one of the event producer data sources  101 ,  102  through n into the ERM  105 . In yet other embodiments, the business event is manually entered into the ERM  105 . In some embodiments of the invention, a graphical user interface  175  is provided. 
     Also coupled to the ERM  105  are one or more event consumer data sources  198 ,  199  through m. A link is mapped between business events originating in the data sources  101 ,  102  through n and the data within the event consumers  198 ,  199  through m. As such, business events are able to affect the retention schedule of the data within the data sources  198 ,  199  through m. 
     The process of linking, mapping and propagating disposition request between event producers and event consumers is more fully explained in copending U.S. patent application Ser. No. 12/164,468, filed on Jun. 30, 2008, published as United States Patent Publication No. 2009/0328070, entitled Event Driven Disposition (hereinafter referred to as the “Event Driven Disposition” disclosure), which is incorporated herein in its entirety by this reference thereto. 
     According to some embodiments of the invention, one or more connectors  150 ,  151  through p and  152 ,  153  through q are established for communication between the event producers  101 ,  102  through n and the ERM, as well as the ERM and the event consumers  198 ,  199  through m. The connectors  150 ,  151  through p and  152 ,  153  through q perform a number of services. Some examples of services include configuration services, polling services, filtering services, parameter resolving services, and hold services. In some embodiments of the invention, the connectors provide reminder services. Although specific examples of services are provided, it will be readily apparent to those having ordinary skill in the art that the connectors perform a wide variety of services relating to retention, storage, holding, disposing, and/or reminder services, among others. 
     Methods and systems for managing electronic discovery and retention-holds are described in copending U.S. patent application Ser. No. 11/963,383, filed on Dec. 21, 2007, published as United States Patent Publication No. 2009-0165026, entitled Method and Apparatus For Electronic Data Discovery, which is incorporated herein in its entirety by this reference thereto. According to this disclosure, e-discovery collections and hold are performed through an Electronic Discovery Management Application (hereinafter referred to as an “EMA”). The EMA uses a connector coupled to a data source, which exposes a configuration service and a number of other services needed for processing collection and hold requests. 
     According to some embodiments of the invention, one or more of the event producer data sources  101 ,  102  through n transmit holds and the connectors  150 ,  151  through p and  152 ,  153  through q are capable of managing collections and holds. Documents that are on hold should not be disposed of in response to a disposition request. Therefore, the invention provides connectors that keep track of documents that are on hold and that make sure these documents are not disposed of. Additionally, there may be a situation when certain documents, which should have been disposed of were preserved because they were put on hold (in place) or locked in some other way. Therefore, the invention provides an event-driven disposition solution that disposes of these documents soon after they have been released from hold, or unlocked some other way. Furthermore, the retention schedule for a portion of data may evolve. For instance, a change in tax code may require a business to keep documents for a longer period of time for compliance. For stored taxed documents already on hold, it is important not to dispose of the documents between the time the hold is withdrawn and the time required by the new regulation. Therefore, the invention manages reminders, such that the data are not prematurely disposed of in light of an evolving retention schedule. 
     The specific setup for managing retention schedules, holds and reminders depends largely on what type of event consumer data source is being considered. In some embodiments, a method of determining what type of data sources are coupled to the ERM is accomplished with a polling step. For example, in the Event Driven Disposition disclosure, a method of determining information about event consumer is accomplished when event consumers are polled by ERM through a configuration service exposed by event consumer connector to return the types of disposition requests they can consume as well as the type of event consumer. 
     Likewise, according to some embodiments of the invention, a step of polling the event consumers is used to determine what the event consumers understand about holds and collections. In some embodiments of the invention, the polling step occurs when the ERM is configured. In some embodiments, the polling step occurs after the initial ERM configuration and before a hold request is propagated. In yet other embodiments, the information about what the event consumers understand about collections and/or hold is manually configured. 
     Disposition Survivors 
     There may be a situation when certain documents, which should have been disposed of were preserved because they resided on hold (in place) or locked in some other way. It would be beneficial for an event-driven disposition solution to be able to dispose of these documents soon after they have been released from hold (or unlocked). 
       FIG. 2  is a timeline showing a disposition survivor according to the invention. In  FIG. 2 , documents A and B initially reside in a data source pursuant to a hold request. Document B is put on hold. A disposition request is made and document A is disposed of, but document B is preserved because it is on hold. A hold release request is made and document B is then released from hold. As such, it is a disposition survivor. 
     This problem may or may not exist in the data source/connector (hereinafter “Consumer”) depending on Consumer capabilities. Based on these capabilities, Consumers can be classified as discussed below. 
     Disposition Survivor-Safe 
     This is a Consumer that can dispose of disposition survivors without external help. 
     An example of this is a file system connector that is responsible for putting files on hold and stores the list of files on hold in its database. Once such a connector receives a disposition request, it tries to delete all the files matching the request criteria unless they are recorded as being on hold in the connector&#39;s database. If a file on hold is identified during the disposition request, the connector persists a current disposition request, or it does not delete the request if it has been persisted, into its own persistence layer (database) and periodically tries to re-execute this request until there are no files matching the request criteria that remain on hold. After that the request is deleted from the database. 
       FIG. 3  is a flow diagram showing the performance of a disposition in a disposition survivor-safe consumer according to the invention. In  FIG. 3 , the process starts ( 300 ) and files are identified for disposition ( 302 ). A determination is made is there are more file for disposition ( 304 ) and, if so, the next file is chosen ( 306 ). A determination is made whether or not the file is on hold ( 308 ); if so, the transaction persists ( 310 ). Otherwise, the file is deleted ( 312 ) and the process continues ( 304 ). If there are no more files ( 304 ), a determination is made whether or not the transaction persists ( 314 ); if so, the system waits ( 318 ). Otherwise, the process ends ( 316 ). If the wait exceeds a maximum disposition transaction lifespan ( 320 ) the process ends ( 322 ). Otherwise, the process continues ( 302 ). 
     A disposition survivor-safe connector can also be implemented on top of a data source that prevents the connector from deleting a document, even if the connector does not know that the document is on hold. For example, an event consumer connector on top of a Documentum™ record management repository tries to delete a document from the repository and fails when the document has been put on hold by a Retention Policy Services module of Documentum™. Once this has happened, the connector persists the current disposition request into its own database and periodically tries to re-execute this request until there are no files matching the request criteria that remain on hold. After that, the request is deleted from the database. This algorithm is a modification of the previous one, except that instead of consulting its own database, the connector consults the underlying data source&#39;s hold module to check which documents are on hold. 
     Disposition Survivor-Unsafe 
     In other scenarios, the Consumer cannot dispose of disposition survivors. We call such Consumer&#39;s disposition survivor-unsafe. 
     Note that in copending U.S. patent application Ser. No. 12/164,468, filed on Jun. 30, 2008, published as United States Patent Publication No. 2009/0328070, we also classified Consumers into retention-capable, i.e. those who can enforce retention schedules for themselves and require only the knowledge of a disposition triggering event; and simple, i.e. those who cannot enforce retention schedules and require immediate disposition requests to be dispatched to them at the time of disposition. These two classifications are orthogonal to each other, although in real life retention-capable Consumers are likely to be disposition survivor-safe, and simple Consumers are likely to be disposition survivor-unsafe. 
     Sending Disposition Reminders 
     Disposition Survivor-Unsafe Consumers 
       FIG. 4  is a flow diagram showing logic for sending disposition reminders to simple disposition survivor-unsafe consumers according to the invention, optional logic is indicated in  FIG. 4  with a dashed line. The process starts ( 400 ). To make disposition survivor-unsafe Consumers capable of disposing of disposition survivors, the logic sends the Consumer a disposition reminder request to re-evaluate a disposition request ( 402 ). Optionally, a determination is made whether or not a connector reported that not all documents were being deleted ( 404 ). If all documents are deleted, the process ends ( 406 ). Otherwise, the process proceeds to wait ( 408 ). Optionally, a determination is made whether or not a maximum disposition transaction lifespan is exceeded ( 410 ). If so, the process ends ( 412 ); otherwise, the process continues ( 402 ). 
     In a response to the reminder, the Consumer re-executes the disposition attempt, and disposes of disposition survivors that were released from hold since the initial disposition request or the previous disposition reminder. 
     The logic should be informed whether a Consumer is disposition survivor-safe or unsafe. This can be achieved through a response to a configuration service request. In addition to the information returned in a configuration service response described in copending U.S. patent application Ser. No. 12/164,468, filed on Jun. 30, 2008, published as United States Patent Publication No. 2009/0328070, and U.S. patent application Ser. No. 12/164,524, filed on Jun. 30, 2008, published as United States Patent Publication No. 2009/03266969, now U.S. Pat. No. 7,792,945, the configuration service response may contain any or all of the following parameters:
         A Consumer disposition survivor-safe (disposition survivor-unsafe) flag is set, indicating whether the Consumer is disposition survivor-safe or unsafe.   A reminder frequency (reminder period) parameter is set, indicating how often the reminder should be sent to the consumer.   A maximum disposition transaction lifespan parameter is set, indicating for how long the ERM should remind the consumer about disposition and/or for how long disposition a survivor-safe Consumer attempts to get rid of disposition survivors. These may be two different parameters.       

     The ERM can overwrite the default values coming from configuration service, as well as to reset default values. 
     Although the existence of a configuration service is highly desirable, the same information can be also received by logic through a user interface and configuration files. 
     For simple Consumers, reminders are sent periodically after the initial disposition request has been issued. 
     For retention-capable Consumers, the ERM starts sending the reminders after the estimated time of first disposition attempt. The ERM sends a scheduled disposition request immediately after it receives an event. Therefore, it does not control the actual disposition process on the Consumer side. 
     Retention-capable consumers can initiate the first disposition attempt. But disposition survivor-unsafe Consumers cannot initiate additional attempts. Therefore, they must be reminded to do that by the ERM. 
     To send a reminder, the ERM calculates the time when the Consumer should have started the initial disposition attempt if it acts according to the retention schedule. Then, the ERM waits for a reminder frequency period and issues a reminder. 
     Such a consumer is very unlikely ever to exist because, if it is aware of a retention schedule and can perform an initial disposition attempt, most likely it is able to perform subsequent attempts. Or it would be easier to implement such connector as a simple connector. 
     A disposition reminder request can have the same format as an immediate disposition request. In particular, it may contain all the information necessary to perform the disposition attempt so the Consumer does not need to remember disposition criteria between attempts. 
     Handling Changing Triggering Event Types in Retention Schedules 
     If the event type changed in a retention schedule, the ERM:
         Stops sending disposition reminders to disposition survivor-unsafe Consumers;   Sends the new retention schedule to retention-capable consumers; and   Notifies the ERM administrator that the event type coming from the producer may not longer match the retention schedule associated with the route.
 
Handling Changing Disposition Periods in Retention Schedules for Simple Consumers Once the Event is Inside the ERM, but has not yet been Propagated to the Consumer
       

     As described in copending U.S. patent application Ser. No. 12/164,468, filed on Jun. 30, 2008, published as United States Patent Publication No. 2009/0328070, simple Consumers are those who cannot enforce retention schedules and who require immediate disposition requests to be dispatched to them at the time of disposition. Copending U.S. patent application Ser. No. 12/164,468, filed on Jun. 30, 2008published as United States Patent Publication No. 2009/0328070, suggests that the ERM calculate the disposition time for such a consumer and send an immediate disposition request to the Consumer at the time when disposition should be executed. 
     There may be a situation when a retention period in a retention schedule associated with the route pointing to a simple event consumer has been changed after the event entered the ERM, but before the disposition request has been sent to the Consumer. 
     In this situation, the ERM should: 
     
         
         
           
             Calculate the new time if the immediate disposition request; 
             If the newly calculated time is in future, do nothing; and 
             Issue an immediate disposition request immediately if the newly calculated time happened in the past. 
           
         
       
    
     To achieve this, the ERM does not try to calculate the disposition dates for simple consumers upfront. Instead, it periodically checks whether the event is ready to be forwarded to the event consumer by comparing the current date time with the date time calculated based on event occurrence date and retention rules. This allows the ERM to change retention rules with ease, without the need to recalculate anything in event tables. 
     Alternatively, the ERM may calculate the disposition time upfront, but recalculate it each time retention schedule has changed for disposition transaction that refer to this retention schedule. 
     Handling Changing Disposition Periods in Retention Schedules after Immediate Disposition Request has been Sent to a Simple Consumer 
     Simple Disposition Survivor-Unsafe Consumer 
     There may be a situation when the underlying retention schedule changed between the initial disposition request call and a disposition reminder request. It may result in the outcome for a disposition survivor-unsafe consumer that a disposition time has come and the ERM already sent a disposition request. As a result, the connector performed one or multiple attempts to delete the documents. But there are still some documents on hold. So the connector is waiting for disposition reminders to perform more attempts. 
     In this case, if the new disposition time is in future, the ERM refrains from sending reminders until the new disposition time comes. If the new disposition time is in the past, the ERM continues sending reminders. 
     Simple Disposition Survivor-Safe Consumer 
     As opposed to the previous scenario, a disposition survivor-safe Consumer has a built-in mechanism that identifies disposition survivors periodically or on event-driven basis, and that tries to dispose of them without any interaction with the ERM. It is essential to suspend this mechanism for a while when the disposition time shifts to the future. To achieve this, in a response to a configuration service request, a disposition survivor-safe Consumer specifies the maximum disposition transaction lifespan, which is the maximum period after which the Consumer stops the attempts to dispose of remaining disposition survivors. If this period is not specified, it can be treated by the ERM as infinity, or the ERM may impose its own maximum disposition transaction lifespan for the Consumer. 
     Once a retention time in the retention schedule is changed, and the new time happens to be in future, the ERM identifies all the disposition transactions where disposition survivor-safe Consumers are involved and for which maximum disposition transaction lifespan is in future and an immediate disposition request has already been sent. For example, if on Mar. 2, 2007 the schedule is changed, transaction  123  started on Jan. 1, 2007, and its maximum disposition transaction lifespan is two years, and an immediate disposition request has been sent on Feb. 1, 2007, such a transaction is selected. 
     Consumers involved in such transactions receive stop disposition message from the ERM, which contains identifiers of disposition transactions that must be stopped. When the new disposition time is due for each of these transactions, the ERM re-issues an immediate disposition request. 
     Once a retention time in the retention schedule is changed, and the new time happens to be in past, the ERM identifies all the disposition transactions affected by this retention schedule. If no immediate disposition request has been issued within this transaction yet, the ERM issues such a request. Otherwise, it does nothing. 
     Handling Changing Disposition Periods in Retention Schedules after Immediate Disposition Request has been Sent to a Retention-Capable Consumer 
     As opposed to simple Consumers, retention-capable Consumers receive scheduled disposition requests immediately after the event is registered by the EMA. As a result, it is highly unlikely (and can be made impossible) that the retention period has been changed after the event is received by ERM but before it has been propagated to Consumer. 
     Retention-Capable Disposition Survivor-Safe Consumer 
     This Consumer can store and understand retention schedules and manage the disposition of disposition survivors. When the retention period changes, the ERM propagates the new retention schedule to such a Consumer as fast as possible if the Consumer involved in any disposition transaction whose start time plus maximum disposition transaction lifespan is in the future; and the consumer is expected to act accordingly. 
     Retention-Capable Disposition Survivor-Unsafe Consumer 
       FIG. 5  is a flow diagram showing logic for sending disposition reminders to retention-capable disposition survivor-unsafe consumers according to the invention, optional logic is indicated in  FIG. 5  with a dashed line. In  FIG. 5 , the process starts ( 500 ) and a scheduled disposition request is sent ( 502 ). The system waits for an estimated time of a first disposition attempt plus a remainder period ( 504 ). The disposition reminder request is then sent ( 506 ). Optionally, a determination is made whether of not the connector reported that all documents are being deleted ( 506 ). If not, the process ends ( 510 ); otherwise, the systems waits ( 512 ). A further optionally determination is ay be made whether or not a maximum disposition transaction lifespan is exceeded ( 514 ). If not, the process continues ( 506 ); otherwise the process ends ( 516 ). 
     When retention period has changed and the Consumer involved in any disposition transaction whose start time plus maximum disposition transaction lifespan is in the future, the ERM:
         Sends a new retention schedule to the consumer and expects the consumer to start its initial disposition attempts according to the new schedule.   Calculates the disposition time according to the schedule; and   If new estimated time of first disposition attempt plus reminder frequency period is still in future, refrains from sending reminders;   Otherwise, it starts sending reminders.
 
Triggering Legacy Events
       

     Routes can be established after the ERM receives events from event producers, presumably because other routes used the same Producers. The ERM can either propagate only the events that entered the ERM after the route has been established, or it can also propagate old events to the newly established route. This behavior can also be dynamically configured on the application level and applies all the routes created afterwards. Or, it can be decided during the route setup by offering the user to propagate old events and even allowing him to choose particular events that must be propagated. 
     Although the invention is described herein with reference to the preferred embodiment, one skilled in the art will readily appreciate that other applications may be substituted for those set forth herein without departing from the spirit and scope of the present invention. Accordingly, the invention should only be limited by the Claims included below.