ACE: a Novel Software Platform to Ensure the Long Term Integrity of Digital Archives

We develop a new methodology to address the integrity of long term archives using rigorous cryptographic techniques. A prototype system called ACE (Auditing Control Environment) was designed and developed based on this methodology. ACE creates a small-size integrity token for each digital object and some cryptographic summary information based on all the objects handled within a dynamic time period. ACE continuously audits the contents of the various objects according to the policy set by the archive, and provides mechanisms for an independent thirdparty auditor to certify the integrity of any object. In fact, our approach will allow an independent auditor to verify the integrity of every version of an archived digital object as well as link the current version to the original form of the object when it was ingested into the archive. We show that ACE is very cost effective and scalable while making no assumptions about the archive architecture. We include in this paper some preliminary results on the validation and performance of ACE on a large image collection. Introduction One of the most challenging problems facing digital archives is how to ensure the authenticity of their holdings over the long term (tens or hundreds of years). Unless the authenticity of an archive can be assured, it may not be possible to use the archive’s holdings to support any significant endeavor. Digital information is in general quite fragile, especially over time. Errors can be introduced because of hardware and media degradation, hardware and software malfunction, operational errors, security breaches, and malicious alterations, to name a few of the obvious ones. Other potential sources of errors, which are particularly relevant for long term archives, include major hardware and software systems changes due to technology evolution, and the possibility of major natural hazards and disasters such as fires, floods, and hurricanes. Two additional factors complicate this problem further. First, an object may be subjected to a number of transformations during its lifetime, including those migrative transformations due to format obsolescence. These transformations may alter the object in unintended ways. Second, most current integrity checking mechanisms are based on some type of cryptographic techniques, most of which are likely to become less immune to potential attacks over time and hence they will need to be replaced by stronger techniques. Therefore any approach to ensure the authenticity of a long term archive has to also be able to address these two problems as well. Several technical approaches have been proposed to address the long term integrity of digital archives, including those that appeared in [1], [3], [4], [6], [7], [8], and [9], but none seems to offer a solid approach that is applicable to the different emerging architectures for digital archives (including centralized and distributed archives) and that is capable to continually monitor and verify the integrity of the data in a cost effective way. In this paper, we introduce a general software environment called ACE (Auditing Control Environment), which is based on a rigorous cryptographic approach and yet quite efficient and can interoperate with any archiving architecture. Using the new framework, we introduce procedures to continually verify the integrity of the archive. Our approach will allow an independent auditor to verify the integrity of every version of an archived digital object as well as link the current version to the original form of the object when it was ingested into the archive. Specifically, ACE is based on creating a small-size integrity token for each digital object upon its deposit into the archive (or upon registration of the object of an existing archive), to be stored either with the object itself or in a registry at the archive as authenticity metadata. Cryptographic summary information that depends on all the objects registered during a dynamic time period is stored and managed separately. The summary information is very compact and is of size independent of the number or sizes of the objects ingested. Regular audits will be continuously conducted, which will make use of the integrity tokens and the summary integrity information to ensure the integrity of both the objects and the integrity information. In our prototype, audits can also be triggered by an archive manager or by a user upon data access. However we are assuming that the auditing services are not allowed to change the content of the archive even if errors are detected. The responsibility for correcting errors is left to the archive administrator after being alerted by the auditing service. Overview of the ACE Approach ACE adopts a two-tier approach. The first tier deals with creating a small size Integrity Token (IT) (Figure 1) for each digital object upon its deposit into the archive (or upon registration of the object of an existing archive), to be stored either with the object itself or in a registry at the archive as authenticity metadata. Cryptographic Summary Information (CSI) depending on all the objects registered during a dynamically adjustable time interval is stored and managed independently of and separately of the archive. The ITs and CSIs are used to continually verify the authenticity of the corresponding digital object. The second tier involves the generation of very compact witness values that cryptographically depend on all the objects ingested during a relatively long time period (such as a week). Figure 1. Integrity Token ACE: First Tier The first tier integrity information types (IT and CSI) are generated in two steps; aggregative registration and hash-linking. The aggregative registration of the objects is typically invoked during ingestion, and composed of aggregation rounds. The interval of each round is determined dynamically based on the number of registration requests and time passed. This dynamic aggregation period allows us to control both the maximum size of ITs and maximum wait-time for registration. During an aggregation round, the hashes of all the objects submitted for registration as well as random hashes as necessary are aggregated using an authentication tree such as the Merkle’s tree [10]. Note that, in practice, the hash of the object is submitted as a part of the registration request (IT Req in Figure 1). The internal node in the authentication tree has the hash value of the concatenated hashes at the children. We insert random hash values into each round to ensure that the tree will always have a certain minimal number of leaves. Figure 2 shows an authentication tree for a round involving eight objects with hash values 7 2 1 0 ,. , , , h h h h . Figure 2. Authentication Tree (IT Reqi contains hi) Note that the value at the root is a hash value that depends in a cryptographic sense on all the objects processed during a round. For each object, we assemble a short list of hashes from the tree, called an aggregation proof, to enable the derivation of the root value from the hash of the object. We time stamp all the objects participating in each round with the same time stamp. The second step consists of linking the hash value generated at each round with the hash values generated at the previous rounds using a structure that depends on the linking scheme used. In our prototype, we use a simple binary linking scheme that computes the hash value of the previous Cryptographic Summary Information (CSI) concatenated with the hash value of the current round as illustrated in Figure 3. This is the same scheme as suggested in [5] and [24]. In this binary linking scheme, the only two data necessary to construct CSI is the previous CSI and the root value of the authentication tree. The former is included in IT (LSI in Figure 1), whereas the latter can be re-computed using the aggregation proof. In the other words, IT has all the information to re-compute the corresponding CSI at any time.