# Presentation: 937270

## Risk-Based Approach

- Scott M Revolinski
- Washington Safety Management Solutions
- Carolyn Apperson-Hansen
- Cleveland Clinic Foundation

- Part 11 Public Meeting

- 6/11/04

**Notes:**
Good afternoon.  I am Carolyn Apperson-Hansen and I would like to present a risk-based approach that Scott Revolinski and I have used to validate electronic record security and integrity with the clinical research system for the Department of Biostatistics at the Cleveland Clinics.  Scott Revolinski brought the risk-based approach for validation of human safety significant computer systems that has been developed by WSMS for use with the Department of Energy’s operation of nuclear reactors and criticality safety analysis.  We believe that our efforts to utilized this approach with the clinical research use of electronic records can be a template for a more generalized risk-based approach for the selection of controls to ensure electronic records security, authenticity, and integrity.

## Major Points

- Use of risk based evaluation of electronic records systems
- Defining which electronic records are Part 11 electronic records

- Part 11 Public Meeting

- 6/11/04

**Notes:**
Two major areas where the risk-based approach can assist Part 11 compliance efforts are:
First, evaluation of electronic record systems to identify potential hazards to electronic record security and integrity.  Identification of these hazards leads to development of risk mitigation controls targeted to ensure security and integrity without unwarranted burden.

Secondly, this evaluation can provide the documentation identifying which electronic records are Part 11 electronic records and provide the justification why certain electronic records do not require Part 11 compliance.

## Major Points

**Use of risk based evaluation of electronic records systems**
- Defining which electronic records are Part 11 electronic records

- Part 11 Public Meeting

- 6/11/04

**Notes:**
I want to turn first to a discussion of risk assessment of electronic records systems.

## Existing Risk Guidance

- Use of COTS software in medical devices

- Part 11 Public Meeting

- 6/11/04

**Notes:**
The FDA guide for the use of commercial off the shelf software in medical devices is referenced in the new Part 11 Scope and Application guide  This COTS software guide provides information for risk-based validation which can be used with electronic records systems beyond COTS software in medical devices.

A significant concept in this (and all other FDA guides for software validation) is that commercial off the shelf software cannot be considered validated except through a process where the User identifies their requirements of the software and verifies (through testing) that the requirements are met.  That is, a vendor cannot sell “validated” software.

This concept is very useful in evaluating electronic record flow through a system in that all sub-processes within the record flow cannot be considered validated until the specific requirements for these sub-processes are identified and verified.

## Existing Risk Guidance

- COTS software in medical devices guide:
- Basic Validation Documentation
- Residual Risk Discussion

- Part 11 Public Meeting

- 6/11/04

**Notes:**
Beyond the obvious use of the guide, two significant items from this guide that we use in our electronic record assessment are the:

Basic Validation Documentation list provided.  This provides a guideline as to the minimum documentation requirements for a validation.  Largely this consists of information about the software, its use and the configuration management process.

and

The Residual Risk Discussion.  As we know that the risk level will not be driven to zero this discussion provides a guideline as to what basis may be used to accept software with unmitigated potential hazards.  While commercial distribution of software does not alone justify ‘software validation’, the wide use of such software can justify acceptance of residual risk.  We use available documented experience with electronic record systems to document the acceptability of any residual risk after all financially and technologically feasible risk mitigation controls are established.

## Risk Assessment Difficulties

- Most electronic records cannot be easily related to human safety categories:
- major, moderate, minor
- Failure probability for software cannot be defined

- Part 11 Public Meeting

- 6/11/04

**Notes:**
Risk is generally defined as the combination of the hazard significance and the hazard probability.  Problems with using the risk-based approach in  the COTS software guide are that:

the guide is written for computer systems used within or in the development of medical devices and assumes a recognizable link to human safety.  The importance of the computer system to human safety is to be categorized as

major - kills people
moderate - significant injury
minor - minor or no injury

However,  generalized use of electronic records in other areas of the medical industry do not have this easily recognized link to human safety.  A more generalized approach to categorize the hazard significance is required.

Secondly, the failure probability for software cannot be defined.  Failure probabilities for other aspects of the electronic record control system can be estimated.  But, overall the risk-based approach presented in this guide needs some adaptation. 

## General Risk Guide (draft) 

*Risk Management as an Integral Part of the Quality Management System*, SG3 / N15R6 (draft)
- Using Failure Mode and Effects Analysis (FEMA) techniques leads to a risk index.  This is used to categorize the significance of various hazards

- Part 11 Public Meeting

- 6/11/04

**Notes:**
Recently, a more general draft guide was presented for risk-based evaluations of medical devices.  This guide, while still targeted to medical devices, contains a perspective for software evaluation beyond that in the COTS software guide.  Here a general use of the Failure Mode and Effects Analysis (or FEMA) techniques is indicated.  The results of this analysis lead to a subjective risk index which can be used to compare the significance of identified hazards. 

In this example, a matrix of the hazard occurrence probability and the hazard severity form a risk index that is sub-group into categories of high, medium and low.  This type of relationship is subjective and somewhat de-coupled from the overall risk of the parent system.  However, if consistency is carefully applied these categories are useful in measuring the number and / or extent of robustness needed to mitigate the hazards.  

Scott and I have used the subjective FEMA  techniques to evaluate the significance of hazards to electronic record security and integrity when there is no direct connection to human safety.  The overall system being evaluated (clinical research in our specific case) has clear significance to human safety, but individual electronic records being used within the system cannot be identified as having major, moderate, or minor significance as utilized in the COTS software guide.

## What’s Missing

- Relationship between electronic records and human safety
- Evaluation of electronic record hazard probabilities and consequences
- Basis for judging robustness of risk mitigation controls

- Part 11 Public Meeting

- 6/11/04

**Notes:**
The new Scope and Application Guide leads us to document our risk-based evaluation.  As mentioned, there are some gaps between the conditions defined in the available guidance and  the information required to complete a risk assessment of an electronic records system.  Examples are:
In the general case, the relationship between most electronic records and the human safety effects of the medical products is difficult to identify.
Hazard probabilities of software failures cannot be identified.  There is no basis for comparing the risk (combination of hazard probability and consequences) of software failures with the risk of other failures in the electronic record system.
Once the hazards are identified, there is no basis for determining when the chosen risk mitigation controls are sufficiently effective.  The intended uses of the risk mitigation controls are to eliminate concerns regarding the use of electronic records without unnecessary cost and infringement upon the use of technological advances.  It is not credible to consider that risk can be eliminated.   What is the basis for accepting the residual risk?

## Systems Approach

- Evaluate association between product quality / human safety and System under consideration
- Examine electronic records use within System
- Identify electronic record security and integrity hazard scenarios
- Establish multiple layers of risk mitigation controls commensurate with hazard probability and consequences

- Part 11 Public Meeting

- 6/11/04

**Notes:**
A systems approach to risk-based validation of electronic record security and integrity requires a basis for identifying hazards to the electronic records and building risk mitigation controls consistent with risk of the overall system to product quality and / or human safety.

An example of how to do this is to first evaluate the overall system and its risks without regard to the use of electronic records.  This risk then bounds the significance of the risk associated with the electronic records within any sub-process of the system.  This defines a ‘normalization factor to bias the FMEA risk indices. 
Then the use or flow of electronic records within the sub-process is examined and 
scenarios are identified to assess the hazards to the security and integrity of the electronic records.
The probability and severity of these hazard occurrences is estimated leading to risk indices.
At least one risk mitigation control should be applied for each scenario.  The number and type (e.g., physical barrier, procedural) is increased for increasing risk index.  However, the controls are to be tempered based upon the system overall risk. 

## Risk Assessment Summary

- In re-evaluating the risk-based approach to Part 11 requirements consider providing:
- an approach to estimate the relationship between electronic records and human safety (e.g., FMEA – Systems Approach)
- types of risk mitigation controls (e.g., layers of procedural and / or physical barriers)
- basis for accepting residual risk

- Part 11 Public Meeting

- 6/11/04

## Major Points

- Use of risk based evaluation of electronic records systems
**Defining which electronic records are Part 11 electronic records**

- Part 11 Public Meeting

- 6/11/04

**Notes:**
With this systems approach to risk-based evaluations in mind, let me now turn to the second topic of defining which electronic records are Part 11 electronic records.

## Existing Application Guidance

- Scope and Application Guide for Part 11

- Part 11 Public Meeting

- 6/11/04

**Notes:**
The new Scope and Application Guide seeks to define a narrower scope for application of Part 11.  The narrower scope is centered on 
Records specifically required by predicate rules when maintained electronically
Electronic records submitted to the FDA
Electronic records shown to be significant to product quality and safety
Electronic records with electronic signatures

This narrow scope greatly facilitates the industry’s application of Part 11, but adds a complexity in that now justification must be shown as to why certain electronic records where not considered Part 11 electronic records.

## Application Difficulties

- Most predicate rules were written before use of computers for records management
- Predicate rule requirements for Part 11 issues aren’t addressed
- Few electronic records have direct links to product quality or human safety yet Part 11 application is anticipated

- Part 11 Public Meeting

- 6/11/04

**Notes:**
Feeding into this complexity are the following difficulties:

Most predicate rules do not clearly identify what records are required.  In a few places the need for a record is specifically spelled out, however in most places the need for such documentation is observed only by inference.  For example, a recent warning letter identified computer software used within a process to be a required record.  I’m not disagreeing with this interpretation as it is very like to be an electronic record significant to product quality, but inclusion of the software into the Part 11 umbrella is seen only by inference.

The Scope and Application Guide directs us to look to the Predicate rules to determine the need for such things as validation and audit trail requirements.  However, very few predicate rules reference these requirements.  It is not my assumption that they are not required if not specifically identified.  Like the above mentioned computer software, these things are likely required to ensure product quality.

Finally, as mentioned before, few electronic records have direct links to product quality and / or human safety.  For example, are training records Part 11 records are not?  They are specifically required in some predicate rules, but not all.

## Risk-Based Systems Approach

- Business risk is the inappropriate use of narrow Part 11 scope
- Systems approach to determine use of electronic records and potential regulatory impacts
- Justification documentation

- Part 11 Public Meeting

- 6/11/04

**Notes:**
The path to compliance we have chosen follow the previous risk-based discussion closely.  However, here the risk is not to the security or integrity of the electronic records, but rather to the inappropriate inclusion or exclusion of electronic records from the Part 11 compliance umbrella.

This becomes a financial decision.  Including too many electronic records within the Part 11 compliance program elevates the cost of the program.  While exclusion of some records may result in some form of regulatory action or inability to use the records for future clinical research (in my specific case).

In some cases, the reference in the predicate rules makes the decision clear.  But more frequently, the decision must be made based upon a subjective evaluation of how the electronic record is used within the overall system.  This decision and the documentation justifying the decision follows a process very similar to the FMEA Systems Approach described earlier.

In the end, the justification for each choice must be documented.

## Application Summary

- In re-evaluating the scope of Part 11 application consider providing:
- some examples of when electronic records could be considered Part 11 records and when they are not (when not specifically identified in the predicate rules)
- basis for deciding what is significant to product quality / human safety

- Part 11 Public Meeting

- 6/11/04

## Contact Information

- Scott Revolinski
- Washington Safety Management Solutions
- 2131 S. Centennial Avenue
- Aiken, SC 29803
- _[s.revolinski@wxsms.com](mailto:s.revolinski@wxsms.com)_
- 803-502-9678

- Carolyn Apperson-Hansen
- Cleveland Clinic Foundation
- 9500 Euclid Avenue
- Cleveland, Ohio 44195
- _[capperso@bio.ri.ccf.org](mailto:capperso@bio.ri.ccf.org)_
- 216-445-3618

- Part 11 Public Meeting

- 6/11/04

**Notes:**
Thank you for this opportunity to speak with you.  If Scott or I can be of any assistance in clarifying these ideas please feel free to contact us.