Information Management: The Challenges of Managing Electronic Records
Released: 2010-06-17T00:00:00

Letter

Background

The federal government collects, generates, and uses large amounts of information in electronic form, from enormous geographic databases to individual e-mails. Much of that information can constitute official federal records, and agencies must have ways to manage such records.

Federal Agencies and NARA Have Responsibilities for Federal Records Management

Under the Federal Records Act, each federal agency is required to make and preserve records that (1) document the organization, functions, policies, decisions, procedures, and essential transactions of the agency and (2) provide the information necessary to protect the legal and financial rights of the government and of persons directly affected by the agency’s activities. If these records are not effectively managed, individuals might lose access to benefits for which they are entitled, the government could be exposed to legal liabilities, and historical records of vital interest could be lost forever. In addition, agencies with poorly managed records risk increased costs when attempting to search their records in response to Freedom of Information Act requests or litigation-related discovery actions. Finally, without effective management of the documentation of government actions, the ability of the people to hold the government accountable is jeopardized.

Effective records management is also an important tool for efficient government operation. Without adequate and readily accessible documentation, agencies may not have access to important operational information to make decisions and carry out their missions.

Accordingly, to ensure that they have appropriate recordkeeping systems with which to manage and preserve their records, agencies are required to develop records management programs. These programs are intended, among other things, to provide for accurate and complete documentation of the policies and transactions of each federal agency, to control the quality and quantity of records they produce, and to provide for judicious preservation and disposal of federal records.

Among the activities of a records management program are identifying records and sources of records and providing records management guidance, including agency-specific recordkeeping practices that establish what records need to be created in order to conduct agency business.

Under the Federal Records Act and the regulations issued by NARA, records must be effectively managed throughout their life cycle, which includes records creation or receipt, maintenance and use, and disposition. Agencies create records to meet the business needs and legal responsibilities of federal programs and (to the extent known) the needs of internal and external stakeholders who may make secondary use of the records. To maintain and use the records created, agencies are to establish internal recordkeeping requirements for maintaining records, consistently apply these requirements, and establish systems that allow them to find records that they need. Disposition involves transferring records of permanent, historical value to NARA for archiving and destroying all other records that are no longer needed for agency operations.

One key records management process is scheduling, the means by which NARA and agencies identify federal records and determine time frames for disposition. Creating records schedules involves identifying and inventorying records, appraising their value, determining whether they are temporary or permanent, and determining how long records should be kept before they are destroyed or turned over to NARA for archiving. For example, one general records schedule permits civilian agencies to destroy case files for merit promotions (2 years after the personnel action is completed, or after an audit by the Office of Personnel Management, whichever is sooner). No record may be destroyed or permanently transferred to NARA unless it has been scheduled, so the schedule is of critical importance. Without schedules, agencies would have no clear criteria for when to dispose of records and, to avoid disposing of them unlawfully, would have to maintain them indefinitely.

Scheduling records, electronic or otherwise, requires agencies to invest time and resources to analyze the information that an agency receives, produces, and uses to fulfill its mission. Such an analysis allows an agency to set up processes and structures to associate records with schedules and other information (metadata) to help it find and use records during their useful lives and dispose of those no longer needed.

Records schedules are based on content and are media-neutral; that is, electronic records are classified on the same basis—by content— as physical records. In addition, agencies are to compile inventories of their information systems, after which the agency is required to develop a schedule for the electronic records maintained in those systems.

NARA also has responsibilities related to scheduling records. NARA works with agencies to help schedule records, and it must approve all agency records schedules. It also develops and maintains general records schedules covering records common to several or all agencies. According to NARA, records covered by general records schedules make up about a third of all federal records. For the other two thirds, NARA and the agencies must agree upon agency-specific records schedules.

Under the Federal Records Act, NARA is given general oversight responsibilities for records management as well as general responsibilities for archiving—the preservation in the National Archives of the United States of permanent records documenting the activities of the government. Of the total number of federal records, less than 3 percent are permanent. (However, under the act and other statutes, some of the responsibilities for oversight over federal records management are divided across several agencies. Under the Federal Records Act, NARA shares a number of records management responsibilities and authorities with the General Services Administration (GSA). The Office of Management and Budget (OMB) also has records management oversight responsibilities under the Paperwork Reduction Act and the E- Government Act.)

For records management, NARA is responsible for issuing guidance; working with agencies to implement effective controls over the creation, maintenance, and use of records in the conduct of agency business; providing oversight of agencies’ records management programs; approving the disposition (destruction or preservation) of records; and providing storage facilities for agency records. The act also gives NARA the responsibility for conducting inspections or surveys of agency records and records management programs.

Federal Records Management Has Been Given Low Priority and Has Had Persistent Weaknesses

Historically, despite the requirements of the Federal Records Act, records management has received low priority within the federal government. As early as 1981, in a report entitled Federal Records Management: A History of Neglect, we stated that “persistent records management shortcomings” had been attributed to causes that included “lack of commitment by top management, emphasis on agency missions, and the low priority of records management.” Almost 30 years later, the priority problem has remained remarkably persistent.

For instance, a 2001 study prepared for NARA by SRA International, Inc., on perceptions in the federal government with respect to records management, concluded that recordkeeping and records management in general receive low priority, as evidenced by lack of staff or budget resources, absence of up-to-date policies and procedures, lack of training, and lack of accountability. This assessment also concluded that although agencies were creating and maintaining records appropriately, most electronic records remained unscheduled, and records of historical value were not being identified and provided to NARA for archiving.

In 2002, drawing on the 2001 study, we reported that the low priority given to records management programs was a factor in program weaknesses. We noted that records management is generally considered a “support” activity. Because support functions are typically the most dispensable in agencies, resources for and focus on these functions are often limited.

In 2008, we reported on weaknesses in federal e-mail management at four agencies. The four agencies reviewed generally managed e- mail records through paper-based processes, rather than using electronic recordkeeping. (A transition to electronic recordkeeping was under way at one of the four agencies, and two had long-term plans to use electronic recordkeeping.) We attributed weaknesses in agency e-mail management (such as senior officials not conforming to regulations) to factors including insufficient training and oversight regarding recordkeeping practices (as well as the onerousness of handling large volumes of e-mail)—similar to the effects of low priority described by SRA. Accordingly, we recommended that agencies with weaknesses in oversight, policies, and practices develop and apply oversight practices, such as reviews and monitoring of records management training and practices, that would be adequate to ensure that policies were effective and that staff were adequately trained and were implementing policies appropriately.

Further evidence of the persistence of the priority issue was provided in 2008, when NARA surveyed federal senior managers about their perception of records management. According to the survey, only 64 percent of managers saw records management as a useful tool for mitigating risk.

In April 2010, NARA released a report on its first annual records management self-assessment, which analyzed responses to a survey sent in September 2009 to 245 federal cabinet-level agencies, agency components, and independent agencies. According to NARA, the survey results showed that almost 80 percent of agencies were at moderate to high risk of improper disposition of records. For example, the survey found that not all agencies had appropriate policies in place for handling e-mail, and that only a little over half of the responding agencies had training in place for high-level executives and political appointees on how to manage e-mail; this is consistent with the finding in our 2008 report on e-mail practices regarding insufficient training and oversight regarding recordkeeping practices. NARA rated almost half of the responding agencies (105 of 221) as high risk in the area of e-mail.

NARA’s survey also indicated, among other things, that a large proportion of agencies have not scheduled existing systems that contain electronic records. In December 2005, NARA issued a bulletin requiring agencies to have NARA-approved records schedules for all records in existing electronic information systems by September 30, 2009. 27 percent of agencies responding to NARA’s September 2009 agency self-assessment survey indicated that fewer than half of their electronic systems were scheduled. Such large numbers of unscheduled systems are a problem for agencies because their records cannot legally be disposed of, with the consequences for increased cost and risk mentioned earlier.

NARA concluded that the varying levels of agency compliance with its records management regulations and policies have implications for the government’s effectiveness and efficiency in conducting its business, protecting citizens’ rights, assuring government accountability, and preserving our national history.

Our Previous Reports Have Recommended Strengthening NARA’s Oversight Approach

The Federal Records Act gave NARA responsibility for oversight of agency records management programs by, among other functions, making it responsible for conducting inspections or surveys of agencies’ records and records management programs and practices; conducting records management studies; and reporting the results of these activities to the Congress and OMB.

We have made recommendations to NARA in previous reports that were aimed at improving NARA’s insight into the state of federal records management as a basis for determining where its attention is most needed. In 1999, in reporting on the substantial challenge of managing and preserving electronic records in an era of rapidly changing technology, we noted that NARA did not have governmentwide data on the electronic records management capabilities and programs of all federal agencies. Accordingly, we recommended that NARA conduct a governmentwide survey of these programs and use the information as input to its efforts to reengineer its business processes. However, instead of doing a governmentwide baseline assessment survey as we recommended, NARA planned to obtain information from a limited sample of agencies, stating that it would evaluate the need for such a survey later.

In 2002, we reported that because NARA did not perform systematic inspections of agency records management, it did not have comprehensive information on implementation issues and areas where guidance needed strengthening. We noted that in 2000, NARA had suspended agency evaluations (inspections) because it considered that these reached only a few agencies, were often perceived negatively, and resulted in a list of records management problems that agencies then had to resolve on their own. However, we concluded that the new approach that NARA initiated (targeted assistance) did not provide systematic and comprehensive information for assessing progress over time. (Only agencies requesting assistance were evaluated, and the scope and focus of the assistance were determined not by NARA but by the requesting agency.) Accordingly, we recommended that it develop a strategy for conducting systematic inspections of agency records management programs to (1) periodically assess agency progress in improving records management programs and (2) evaluate the efficacy of NARA’s governmentwide guidance.

In response to our recommendations, NARA devised a strategy for a comprehensive approach to improving agency records management that included inspections and identification of risks and priorities. Subsequently, it also developed an implementation plan that included undertaking agency inspections based on a risk-based model, government studies, or media reports.

In 2008, we reported that under its oversight strategy, NARA had performed or sponsored six records management studies in the previous 5 years, but it had not conducted any inspections since 2000, because it used inspections only to address cases of the highest risk, and no recent cases met its criteria. In addition, NARA’s reporting to the Congress and OMB did not consistently provide evaluations of responses by federal agencies to its recommendations, as required, or details on records management problems or recommended practices that were discovered as a result of inspections, studies, or targeted assistance projects.

Accordingly, we recommended that NARA develop and implement an oversight approach that provides adequate assurance that agencies are following NARA guidance, including both regular assessments of agency records and records management programs and reporting on these assessments. NARA agreed with our recommendations and devised a strategy that included annual self- assessment surveys, inspections, and reporting. It has now begun implementing that strategy, having released the results of its first governmentwide self-assessment survey, as mentioned earlier.

Managing Records in Electronic Form Presents Major Challenges

As we have previously reported, electronic records pose major management challenges: their volume, their complexity, and the increasingly decentralized environment in which they are created. E- mail epitomizes the challenge, as it is not only voluminous and complex, but also ubiquitous. ● Huge volumes of electronic information are being created.

Electronic information is increasingly being created in volumes that pose a significant technical challenge to our ability to organize it and make it accessible. An example of this growth is provided by the difference between the digital records of the George W. Bush administration and that of the Clinton administration: NARA has reported that the Bush administration transferred 77 terabytes of data to the Archives on leaving office, which was about 35 times the amount of data transferred by the Clinton administration. Another example is the Department of Energy’s National Energy Research Scientific Computing Center, which said that, as of January 2009, it had over 3.9 petabytes of data (that is, about 4,000,000,000,000,000 bytes) in over 66 million files and that the volume of data in storage doubles almost every year. ● Electronic records are complex.

Electronic records have evolved from simple text-based files to complex digital objects that may contain embedded images (still and moving), sounds, hyperlinks, or spreadsheets with computational formulas. Some portions of electronic records, such as the content of dynamic Web pages, are created on the fly from databases and exist only during the viewing session. Others, such as e-mail, may contain multiple attachments, and they may be threaded (that is, related e-mail messages are linked into send–reply chains). They may depend heavily on context. For example, to understand the significance of an e-mail, we may need to know not only the identity but the position in the agency of the sender and recipients. (Was it sent by an executive or a low-level employee?) In addition, new technologies, such as blogs, wikis, tweets, and social media, continue to emerge, posing new challenges to records managers. ● Identification and classification of electronic records are difficult in a decentralized computing environment.

The challenge of managing electronic records significantly increases with the decentralization of the computing environment. In the centralized environment of a mainframe computer, it is comparatively simple to identify, assess, and manage electronic records. However, in the decentralized environment of agencies’ office automation systems, every user can create electronic files of generally unstructured data that may be formal records and thus should be managed. Documents can be created on individuals’ desktop computers and stored on local hard drives. E-mail can come from outside the agency. In cases like these, the agency generally depends on the individual to identify the document or the e-mail as a record, and, through placing it in a recordkeeping system, associate it with its appropriate schedule, make it searchable and retrievable, and preserve it until it is due for disposal.

As we reported in 2008, e-mail is especially problematic. E-mail embodies several major challenges to records management: ● It is unstructured data, and it can be about anything, or about several subjects in the same message, making it difficult to classify by content. ● There is a very large volume of it: one study estimates that a typical corporate user sends or receives around 110 messages a day. Further, there may be many copies of the same e-mail, which can increase storage requirements or require a means of determining which copy to keep. Keeping large numbers of messages potentially increases the time, effort, and expense needed to search for information in response to a business need or an outside inquiry, such as a Freedom of Information Act request. ● It is complex: e-mail records may have multiple attachments in a variety of formats, they may include formatting that is important for meaning, and they include information about senders, recipients, and time of sending. Recordkeeping systems must be able to capture all this information and must maintain the association between the e-mail and its attachment(s). ● Its relevance depends on context. It may be part of a message thread that is necessary to understand its content, or it may discuss other documents or issues that are not well identified. An e-mail that says “I agree. Let’s do it” may be about a major decision or about going to lunch next week. ● It may not be obvious who is responsible for identifying an e-mail as a record and at what point. NARA regulations require that both senders and recipients may be responsible for identifying records. However, an e-mail may have multiple recipients and be forwarded to still other recipients.

Agency Commitment Is a Prerequisite for Addressing the Electronic Records Challenge

As NARA has pointed out, the decision to move to electronic recordkeeping is inevitable, but as we and NARA have previously reported, implementing such systems requires that agencies commit the necessary resources for planning and implementation, including establishing a sound records management program as a basis. Further, automation will not, at least at the current state of the technology, solve the “end user problem”—relying on individual users to make sound record decisions. Nor will automation solve the problem of lack of priority, which, as our previous work has shown, is of long standing. However, several developments could lead to increased senior-level attention to records management: NARA’s use of public ratings as a spur to agency management, growing recognition of risks entailed in poor information and records management, the requirements and emphasis of the recent Open Government Directive, and the influence of congressional oversight. Senior management commitment, if followed through with effective implementation, could improve the governmentwide management of electronic and other records.

Electronic Recordkeeping Systems Are Challenging to Implement and Will Not Yet Solve the End User Problem

Moving to electronic recordkeeping is not a simple or easy process. Agencies must balance the potential benefits against the costs of redesigning business processes and investing in technology. Our previous work has shown that such investments, like any information technology investment, require careful planning in the context of the specific agency’s circumstances, in addition to well- managed implementation.

In 2007, a NARA study team examined the experiences of five federal agencies (including itself) with electronic records management applications, with a particular emphasis on how these organizations used these applications to manage e-mail. Among the major conclusions was that although the functionality of the software product itself is important, other factors are also crucial, such as agency culture and the quality of the records management program in place. With regard to e-mail in particular, the survey concluded that for some agencies, the volume of e-mail messages created and received may be too overwhelming to be managed at the desktop by thousands of employees across many sites using a records management application alone. A follow-up study in 2008 added that although a records management application offers compliant electronic recordkeeping, “it can be expensive to acquire, time consuming to prepare for and implement, requires user intervention to file records, and can be costly over the long haul for data migration and system upgrades.” NARA found that in most instances agencies had to work to overcome user resistance to using the system.

This user challenge has led records management experts to believe that end users cannot be relied on to manage e-mail records, or indeed any other types of records. A recent Gartner study concluded that user-driven classification of records, especially e-mail, has failed and will continue to fail; a study by the Association for Information and Image Management (AIIM) stated “it is simply not plausible to expect all creators of records to perform accurate, manual declaration and classification.” According to Gartner, “What enterprises really need (and want) is a mechanism that automatically classifies messages by records management type … without user intervention.” At the time of writing (August 2007), Gartner described such technology as “in its infancy,” but expected it to mature rapidly because of high demand.

This technology, automated records classification (sometimes called “autocategorization”), might help address the user problem. (The Air Force is currently working with autocategorization projects.) However, like other information technology, it requires resources for setup and maintenance to be effective, and it is not simple to implement. Further, according to AIIM, autocategorization might not work for an agency’s particular documents or file plan, and might not be sufficiently accurate or cost effective.

Some proposals have been made to simplify the e-mail problem. Gartner recommends treating e-mail as a separate issue from general records management, perhaps by putting all e-mail in a single category of temporary records with a uniform retention period. Similarly, the Director of Litigation in NARA’s Office of General Counsel has suggested keeping all e-mail created by key senior officials (with some additional designations by agency components) as permanent and treating all the rest as temporary. Both proposals would make managing e-mail simpler, but could increase the risk that significant information will not be preserved.

Recent Developments Could Help Spur More Action on Information and Records Management

Raising the priority of records management has been and continues to be an uphill battle. As we have reported, government needs to prioritize the use of resources, and records management has not been a high priority. Further, records management can also be time- and resource-consuming and technically difficult to implement.

NARA can influence this situation by providing effective guidance and assistance to agencies, as well as through its oversight and reporting activities. With its recently initiated annual self- assessment survey, NARA is responding to our earlier recommendations on oversight by beginning an effort to develop a comprehensive view of the state of federal records management as a basis for determining where its attention is most needed. Reporting the results of the survey, with scores for individual agencies and components, to the Congress, OMB, and the public is one way to help bring the records management issue to the attention of senior agency management.

Another factor that could help raise awareness of the value of records management is the growing recognition of the risks of weak electronic records and information management, as a result of fear of potentially large costs to organizations that have to produce electronically stored information to respond to litigation, as well as well-publicized incidents of lost records, including e-mail. This recognition of risk is coupled with increased awareness of the value of organizations’ information assets; according to AIIM, the field of enterprise content management (which includes records management) has been accepted, driven by the need to control the content chaos that pervades local drives, file shares, email systems, and legacy document stores. As a result, according to an AIIM survey, the highest current priorities for ECM activity are electronic records management and managing e-mails as records.

Further, recent Open Government initiatives, which emphasize the importance of making information available to the public for transparency and accountability, could be an additional impetus to addressing electronic records management. OMB’s Open Government Directive makes a direct link between open government and records management by requiring that each agency’s Open Government Plan include a link to a publicly available Web site that shows how the agency is meeting its existing records management requirements. More generally, the directive urges agencies to use modern technology to disseminate useful information. According to an Administration official, records management plays a crucial role in open government by ensuring accountability through proper documentation of government actions. Increased attention to information and records management could provide another spur encouraging agencies to devote resources to managing their electronic records more effectively.

Finally, the priority that agencies give to addressing weaknesses may be increased by hearings such as this, which show that the Congress recognizes the importance of good records management for the efficient, effective, and accountable operations of government.

In summary, federal records management has been given low priority for many years. However, the explosion of electronic information and records is an increasing risk to agencies, and could even become a drag on agencies’ ability to perform their missions if not brought under control.

Raising visibility, as NARA is doing by publishing the results of its self-assessment survey, can raise the perception among senior agency officials of the importance of records management. Also significant is the push for Open Government, which, by heightening the importance of agencies’ providing information to the public, makes information a more central part of their missions and could help highlight the actual importance to agencies of actively managing their information. Strong indications from the Congress that records management needs more attention could also raise the priority among agency management.

Mr. Chairman, this completes my prepared statement. I would be pleased to respond to any questions you or other Members of the Subcommittee may have at this time.

Contact and Acknowledgments

If you should have questions about this testimony, please contact me at (202) 512-6304 or melvinv@gao.gov. Other major contributors include Barbara Collier, Lee McCracken, J. Michael Resser, and Glenn Spiegel.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.