# Presentation: 259090

## Results of IAC Study of Metricsin Electronic Records Management (ERM)Systems

- Dr. Rick Klobuchar
- Vice President and Chief Technology Officer
- SAIC -Enterprise Solutions Business Unit
- 2829 Guardian Lane
- Virginia Beach, VA  23452
- _[richard.l.klobuchar@saic.com](mailto:richard.l.klobuchar@saic.com)_
- (757) 631-2335

- Dr. Mark Giguere
- Lead IT (Policy & Planning)
- ERM E-Gov co-Program Manager
- Modern Records Programs
- NARA
- _mark.giguere@nara.gov_
- (301) 837-1744

**Notes:**
Good Morning.  My name is Mark Giguere and I am the co-Program Manager of the President’s Electronic Records Management Initiative, for which NARA was the Managing Partner.  I am jointly presenting, along with Dr. Rick Klobuchar of SAIC, the results of a study conducted by the Industry Advisory Council.  Dr. Klobuchar kindly provided the slides that form the basis of this presentation and will field questions at the conclusion of the pecha kucha sessions.  

This study was conducted, at the request of the Office of Management and Budget, in an attempt to define metrics which would be used to evaluate deployments of electronic records management systems.  The IAC was recommended in an attempt to survey and evaluate the ‘best metric practices’ which could be found in the industry.

## Introduction and Principal Conclusions

- How does one measure the impact of an ERM system to the bottom line business or mission of an organization?
- What is the business case for an enterprise ERM system?
- Principal conclusions:
- No silver bullet
- No universal COTS tool or product
- No one metric captures the success of an ERM system and relates unambiguously to the bottom line
- Notwithstanding: Some common categories of metrics in use today
- Some metrics less burdensome to capture than others
- Some metrics just reflect a measure of IT system performance
- Some metrics reflect mission success more directly than others
- Measurement of ERM performance is currently immature
- Most measurements tend to be IT-related rather than related to records management itself
- Valid comparisons of ERM practices across organizations are difficult to make, and probably should not be made

**Notes:**
Because of the substantial costs of any IT capital investment, it is incumbent upon managers to establish metrics that can be used to measure return on investment of the expenditure.  Deployment of electronic records management systems, and the establishment of useful and tenable metrics related to such, present unique challenges.  This is both due to the relatively immature nature of the technology and, more importantly, to the high degree of variability in records management organizations manifest in Federal agencies.

## Bottom Line

- The inescapable conclusion:
- There is no simple, single answer!
- There is no Swiss Army Knife-like tool
- Tradeoffs must be made to arrive at metrics that are:
- Meaningful to measure ERM success (e.g., “good” *vs. “*bad” metrics), and
- Not too burdensome to capture on an enterprise-wide basis
- “What gets measured is what gets done”
- Aggregation of metrics into a single coherent picture of bottom line performance isproblematic

**Notes:**
Defining metrics for electronic records management systems deployments is a complex.  In part, this is due to the fact that records management is facilitative to, and should underlie all business processes – that is to say there may be many intervening ‘degrees of freedom’ between measurement of automated records management processes and performance of the ‘records-managed’ business process.  Further complicating matters is that metrics must not only be easy to gather, but ideally, uniform across a deployment which may ‘touch’ several line of business processes (depending on the scope of the deployment).

## Concerns to Consider

- Metrics for Public Services Relating to ERM
- Spirit of the eGovernment initiative is to provide a Government that *“works better and costs less.”*
- Quantifiable and well-defined ERM metrics relating to capacity, throughput, security (especially data and records integrity), assured service availability, ubiquitous access, lower cost, improved turnaround times, etc. are of interest.
- Also concerned about particular metrics that are unreliable, non-specific, intractable to interpret, or too burdensome or onerous to collect.

**Notes:**
An additional level of complexity for metrics associated with government ERM projects is the desire for metrics that are not only well-defined, reliable, and not overly onerous to collect, but they need to testify to the bottom line consideration that public monies are being well spent and contributing to more efficient government operations.  That is to say, they need to document a ‘bottom line’ impact that is understandable to the average citizen.

## Major Factors to Consider

- Who is the Consumer?
- Nature of the “consumer” is an important factor
- “Who” and/or “what” the metrics are sampling
- “Public at large”
- Specific customers
- Agency/company employees
- Federal agencies,
- Other government agencies
- corporations, or
- Foreign users, etc.
- What is the ERM Business Practice?
- What specific “bottom-line” agency and/or industry business practices the metrics supported.  For example:
- Servicing FOIA requests
- Support for legal discovery
- Historical research
- Genealogy
- Auditing and controls
- Regulatory compliance
- Public information dissemination
- Statistical analysis
- Archival records management
- Grants management
- ERM systems operations and management
- Specific mission support (e.g., medical, environmental, emergency and disaster, defense)

**Notes:**
Relating the effect of electronic records management automation to ‘bottom line’ impact is a function of the perspective of the consumer – in the case of ERM, that consumer is often not a direct consumer of records management, but of a ‘downstream’ business process.  In this respect, the IAC study was most useful because it was able to characterize ERM metrics in the context of those which had been most useful in industry business practice.

## Principals in Defining ERM Metrics

- Not everything that can be measured needs to be measured nor should it be
- Metrics should have a purpose for continuing improvement
- Best to design the capture and management of metrics into a system upfront or provide for an SLM approach
- Important “paper *vs*. electronic” paradigm issues to be understood

**Notes:**
What was derived from the combined industry experiences of casting ERM metrics?  While there is a wide variety of things that can be measured, that, in and of itself, is not sufficient.  Metrics should be cast in a durable form such that they can serve the basis for a measurement of continuing improvement, with forethought given in the design of the system to automated capture of those metrics.

## Broad Categories of ERM Metrics

- Access to ERM Services
- Accuracy
- Capacity
- Efficiency
- Participation
- Productivity
- Search and Retrieval
- System
- User Satisfaction
- Utilization
- Legal *

- * Suggested to the IAC team by Robert Williams  of Cohasset Associates

**Notes:**
Industry ‘best practice’ measures of suitable ERM metrics were found to fall in a broad variety of categories ranging from:  improvements in efficiency of processes (such as the capture of appropriate and accurate records), enhanced search and retrieval of records, savings resulting from process efficiencies in responding to discovery requests.

## “Good” vs. “Bad” Metrics

- Many metrics are potentially ambiguous, intractable, unreliable, or burdensome to capture
- Among the more problematic metrics:
- Record search time
- Record retrieval time
- Number of seats (or licenses)
- Session time, and the
- Raw number of records in the system
- All of the above can be captured
- However, interpretation of each can be quite controversial
- A long session time, for example, could be indicative of great success or utter failure
- Search times can be curiosity-driven as in surfing the Web
- Level of commitment and persistence of user can not be easily measured
- Some people are just better than others at“finding things”
- Training, domain knowledge, and time-of-daycan be important mitigating factors

**Notes:**
The IAC study also uncovered categories of unsuitable metrics, that while easily captured and indeed may show improvement over time, are ambiguous in their relation to the ‘bottom line’.  Others are ambiguous, and may not accurately represent what they are purportedly measuring.   What is important about the IAC’s recognition of categories of ‘bad’ metrics is a heightened appreciation for the necessity of uniform interpretation, within the deployment, of what agreed upon metrics mean.

## Sample Candidate Metrics for ERM Systems

| Measurement Category | Metric | Capture Method | Capture Medium | Capture
Burden | Comments |
| --- | --- | --- | --- | --- | --- |
| Access to Services | Hours of operation | Manual | Periodic audit | Low | Almost certainly greatly improved with automation |
|  | Access Points | Automated | System | Low | Almost certainly greatly improved with automation |
| Accuracy | Percentage of records correctly declared  | Manual | Periodic audit | High | Measure of quality |
|  | Percentage of records correctly classified | Manual | Periodic audit | High | Measure of quality |
| Capacity | Size of holdings, i.e., number of records (possibly by record type) | Automated | System | Low | No indication of quality |
| Efficiency | Ease of performing daily tasks | Manual | Survey | High | Purely subjective but indicative of success and acceptance of ERM |
| Participation | Number of seats | Automated | System | Low | No indication of quality |
|  | Number of people declaring records | Manual | Live Oversight | Medium | Indicative of acceptance of system |
|  | Number of people classifying records | Manual | Live Oversight | Medium | Indicative of acceptance of system |
|  | Number of people retrieving records | Manual | Live Oversight | Medium | Indicative of acceptance of system |

**Notes:**
So what is the actual grist of the IAC study?  It’s most valuable contribution to the study of ERM metrics is a systematic analysis broad measurement categories of metrics, cross-referenced  by the nature, medium, and burden of the capture mechanism.  All of these are impartially defined within the context of best industry practices and provide a valuable catalog of metrics that can be considered within Federal government deployments.

## Sample Candidate Metrics for ERM Systems (cont.)

| Measurement Category | Metric | Capture Method | Capture Medium | Capture
Burden | Comments |
| --- | --- | --- | --- | --- | --- |
| Productivity | Number of requests processed per week | Automated | System | Low for one system; High across enterprise | Difficult to measure enterprise-wide across multiple processes; may only be useful as a sampling metric, e.g., for FOIA requests only |
| Search and Retrieval | System search time | Automated | System | Low | No indication of quality |
|  | System retrieval time | Automated | System | Low | No indication of quality |
|  | Number of successful searches | Automated | System | Low | Difficult to interpret; returned result is not necessarily desired result |
|  | Number of search indexes | Automated | System | Low | Indicator of complexity and therefore ease of use |
|  | Number of classification categories | Automated | System | Low | Indicator of complexity and therefore ease of use |
| System | Throughput, i.e., transactions per hour or per unit of time | Automated | System | Low | Measures IT performance, not success of ERM |
|  | Response time, i.e., time to retrieve a record | Automated | System | Low | Measures IT performance, not success of ERM |
|  | Availability, i.e., system uptime | Automated | System | Low | Measures IT performance, not success of ERM |

**Notes:**
So how might agencies use the results of the IAC study of industry best practice recommendations for ERM metrics?  Careful consideration of this catalog of best industry practices can lead to identification of metrics appropriate to your specific deployment and satisfactory to OMB.

## Sample Candidate Metrics for ERM Systems (cont.)

| Measurement Category | Metric | Capture Method | Capture Medium | Capture
Burden | Comments |
| --- | --- | --- | --- | --- | --- |
| User Satisfaction | User satisfaction rating | Manual | Survey | High | Nearly universal metric for ERM exemplars |
| Utilization | Number of people retrieving records | Automated | System | Low | Indicative of acceptance of system; no indication of success or satisfaction |
|  | Virtual Visitors | Automated | System | Low | Indicative of acceptance of system; no indication of success or satisfaction |
| Legal | Numbers and types of process violations that are caught, missed, and/or are attempted | Semi-Automatic | System | Medium | Measure of accuracy and quality of the ERM processes with potential legal weight, significance, and bearing |
|  | Fraction of the inventory of electronic records within an ERM system that is in the wrong state | Semi-Automatic | System | Medium-High | Indicative of the quality of the processes and services provided within an ERM system |

- _Note_:  Any of these metrics should be used to measure improvement over time relative to a baseline.
- The numbers are not meaningful in and of themselves. Additionally, the Study Group determined
- that there is no universal, “silver bullet” metric.

**Notes:**
Agencies should take care when applying metrics to ERM systems.  Any of the suggested metrics might be ‘doable’ – they might be able to be consistently captured with little burden.  However, one must ask one’s self, do they meaningfully measure the ‘bottomline’ in a manner which demonstrates to improvement in an understandable manner to the relevant constituencies.

## Summary

**Notes:**
In conclusion, the casting of metrics for ERM system deployments must be approached with careful consideration.

NARA would like to thank the IAC for its invaluable assistance and I would like to thank Dr. Klobuchar for sharing this opportunity to present these results with him, as well as for his fielding your questions at the conclusion of the pecha kusha sessions.

Please feel free to contact either of us if you’ve any questions about either the IAC study or the ERM E-Government Initiative.  We hope you have found this useful.