Triangle PEERS

FDA Electronic Records; Electronic Signatures

Guidance Document Comments

Triangle PEERS is pleased to submit comments in response to the FDA\'s
Guidance for Industry 21 CFR Part 11: Electronic Records; Electronic
Signature - *Validation* issued in August 2001. Triangle PEERS is an
informal association in North Carolina for entities and individuals with
an interest in the implementation of 21 CFR Part 11.

##### [VALIDATION]{.underline} 

***Summary of General Comments***

1.  More guidance that 'drills-down' into applicability for different
    types/categories of software that are industry recognized (e.g.,
    in-house developed, off-the-shelf, office automation tools such as
    MS Word, Excel etc.) would be helpful. Additionally, there is a need
    for more guidance relating to medical and laboratory instruments, as
    they are comprised increasingly of software components. Guidance is
    not requested down to the level of specific programs within each
    generally recognized category (such as Access for databases or
    WordPerfect for word processors). However, guidance is requested for
    the general types of computerized systems and usage (regulatory vs
    non-regulatory) of software and hardware within these categories.

2.  Some terminology is inconsistently used. New terms have been
    introduced so that considerable translation is required. Terminology
    is not always consistent with industry usage. For example, rather
    than using validation testing terminology typical of the
    pharmaceutical industry (IQ/OQ/PQ-User Acceptance), this Guidance
    introduces new terminology. Particularly unclear is where PQ-User
    Acceptance testing is covered.

```{=html}
<!-- -->
```
3.  The organization of the document seems reversed, as it is presented
    in the opposite order to how validation activities actually take
    place. It should be presented in the order that represents the
    chronology of System Development Life-Cycle events as conducted, not
    as audited. For example, in testing, there is no expectation that a
    \'development\' environment is sufficient, or a production-like
    environment is required. To some extent, this cannot be specified by
    the FDA due to the multitude of methodologies. However, clearly
    required development type testing (5.4.2 read Unit Testing,
    \"white-box\" testing) that must be done by programmers is
    chronologically listed after Integration and Business-oriented
    Testing (5.4.1 user-site tests, simulation tests). This can be
    confusing, especially since the protocol-based testing (OQ/PQ) is
    discussed even earlier in the document (section 5.2). Again, these
    are laid out largely in reverse chronological order. Section 5.5
    essentially consists of peer review/structured walkthrough
    procedures, which would usually fall between 5.4.2 and 5.4.1.
    Organizing the tasks in chronological order for those involved in
    implementing computer system validation would be much more helpful.

4.  This new guidance does not supply any additional detail that relates
    back to the 1997 guidance document, General Principles of Software
    Validation. It is unclear why there should be a separate validation
    guideline if 21 CFR Part 11 items are to be included in a CSV
    documentation set. Perhaps the original CSV guidance document should
    be modified. The content is not specific to 21 CFR Part 11 and
    should instead be released as General Guidance rather than being
    tied to Electronic Records; Electronic Signatures (ERES). The
    implication is that if a system falls outside of the ERES scope,
    then computer system validation (CSV) is not a regulatory
    requirement. If this is true, it should be stated explicitly. There
    is no guidance that helps us to understand that 'X' activities help
    to meet 'Y' regulation/requirement.

5.  Specific details of the validation system life cycle steps,
    controls, and documentation were specifically not described in this
    Guidance, because the agency feels that much has already been
    published on good software development and implementation practices,
    as evidenced by the extensive references list in Appendix A.
    Industry is being urged by the FDA to implement compliant system
    solutions, including purchased products. A common method for
    verifying product features and adequate system development, release,
    and control processes is for the potential customer to perform an
    audit of the vendor. There is still confusion on the part of the
    vendors as to what is expected (for good system development
    practices and associated documentation). A common response from the
    vendor is "Where does it say it's required, in Part 11?" It would be
    very helpful for industry to have something more "official" (such as
    Guidance) to provide as supporting information for vendor audit
    observations.

## Validation Guidance Comments by Section

# Section 2. *Scope*

Further clarification is needed as to which systems may require
electronic signature capability, i.e., where signatures are required by
the predicate rule. Several sections, e.g., sections 2, 4 and 5.1,
reference procedures and controls "directed to . . . ensuring signer
non-repudiation." However it should be clear that this is only relevant
for systems which incorporate electronic signatures (required by the
predicate rule), i.e., one may have electronic records without
electronic signatures (but not vice versa).

[Section 2.1 *Applicability*]{.underline}

This section seems to imply that 21 CFR Part 11 is not included in the
scope of 21 CFR Part 11 (e.g., does not act as its own predicate rule).
This would then mean that records such as system developer
qualifications, system documentation, change control records, and access
control records, if maintained electronically, would not be covered by
21 CFR Part 11, unless they are specifically required by other predicate
rules. Is this a correct interpretation? Note that this is also implied
in the definition of "Predicate Rule" in the Glossary Guidance.

# Section 2.2 *Audience*

Did the author intend to show another bullet point beyond those already
listed?

# Section 5.1 *System Requirements Specifications*

Modify text as follows:

"*you should obtain evidence that the computer system implements those
needs correctly and that they are traceable **to system design
requirements and specifications.**"*

-change to --

*"you should obtain evidence that the computer system implements those
needs correctly and that they are traceable **to
requirements/specifications.**"*

Bullets 2 and 3 should be removed at the end of section 5.1 or revised
to include clearer, less obscure examples.

The statement regarding having requirements before one can test is one
of the best points in this guidance. This issue cannot be repeated
enough ways. If anything, there should be specific reference to some of
the background materials on gathering, expressing, and reviewing user
requirements. The task is not an easy one, because framing requirements
clearly and without ambiguity is dauntingly difficult.

The expectations for validation of work over the Internet are fairly
clear, but encryption is not. Since encryption actually transforms the
data, how is this expected to be validated? Is an audit trail required?

Regarding scanning and scalability, this is outside the scope of Part 11
and should be deleted or clarified. Is it necessary to have
specifications on scanning or scalability?

It is not clear what is expected in validating scanned images. What
about validating the software that de-skews and cleans up the images? At
what stage is an audit trail required: after the image is captured
[or]{.underline} after the image is cleaned and stored?

The document parenthetically defines an end user as "a person regulated
by the FDA." The wording should be modified to "a person whose actions
are regulated by FDA." Actions are regulated, not people.

[Section 5.2 *Documentation of Validation Activity*]{.underline}

Delete the first sentence of this section.

# Section 5.2.1 *Validation Plan*

Clarification is needed on expectations for "schedule of validation
actions." "Schedule" implies including the time frame for validation
activities. Is there a benefit to including this in the Validation Plan
since software cannot be released for use until validation has been
completed? Any schedules seem more appropriate in documents other than
the Validation Plan. Also, clarification of what is meant by "who is
responsible for each validation activity" would be helpful. Is this a
general term (user vs. developer) rather than a reference to specific
individuals?

# Section 5.2.2 *Validation Procedures*

This section discusses 'Validation Procedures' in terms of validation
[testing]{.underline} deliverables, when in fact validation activities
occur concurrent with and throughout the Systems Development Life-Cycle
(SDLC). All major SDLC deliverables should be listed, not just the
testing phase deliverables. The point of section 5.2.2, as it is
understood, is that there should be a 'standardized methodology' for
validation, and the guidance should explicitly make this point.

The section omits conclusions in aggregate (i.e., one needs more than
just individual component results). Separation of the Validation
Protocol (5.2.2) from the execution (5.2.3) is clear. It emphasizes
individual result documentation and \'measurable\' outcomes. The
expectation that the report actually reaches some conclusion about the
test results is not evident. There is no discussion regarding any
limitations and exclusions on the system, nor whether the overall
protocol achieved its objectives (e.g., the application was found to
meet functional requirements as tested by the protocol).

What is meant by the phrase "computer system configuration"? Does this
refer to the database environment, forms, server, etc., or is this the
specific PC configuration as well? Procedures would not normally
describe the actual configuration and the expected outcomes for testing,
but would rather focus on tasks. The configuration should be documented
in the system design specification and also in the Validation Plan or
test plan (e.g., the configuration that will be tested).

[Section 5.2.3 *Validation Report*]{.underline}

Revise "Validation Report" to "Validation Report/Summary"

This section should indicate that the validation report has been
reviewed and approved \"prior to the release of the system to the
production environment."

Modify text as follows:

*"Whenever possible, test results should be expressed in quantified
terms rather than stated as "pass/fail."* should be changed to,
*"Whenever possible, test results should be documented in a manner
permitting pass/fail decisions to be reached."* Or perhaps, "*Acceptance
of the systems should be tied to acceptance criteria.*"

Regarding "quantifying" test results, a properly designed test will be
expressed in such a way that the result is binary: either it worked
correctly or did not work correctly. Usually, there is no quantifiable
variable, only a question of whether "X" occurred or not. Pass/Fail vs.
quantified terms erroneously implies that Pass/Fail should not be used
in testing. In addition to clarifying this statement, providing examples
of how Pass/Fail can be effectively used would be helpful.

Clarification and/or examples of "quantified terms" would be helpful.
Does "quantified terms" mean including the data demonstrating the pass
or fail for a test?

Perhaps change "quantified terms" to "objective terms" as expressed in
Section 5.2.5 of the General Principles of Software Validation; Final
Guidance for Industry and FDA Staff.

[\
Section 5.3 *Equipment Installation*]{.underline}

Equipment Installation. Clarify this section to specify whether this
refers to a discussion of Installation Qualification (IQ).

The order of activities performed is not consistent with practice. For
example, Equipment Installation Qualification (5.3) takes place after
the testing instructions, but it is indicated that it should come first.

[Section 5.4 *Dynamic Testing*]{.underline}

There does not appear to be any discussion of security testing. This is
obviously a major concern within the context of 21 CFR Part 11.

# Section 5.4.1 *Key Testing Considerations*

This indicates that some testing is to be performed in the actual
production environment. This may be feasible only for new systems, or
when conducting a parallel test with an already existing environment.
Best practice is to test in a separate environment to avoid affecting
production processes.

5.4.1 -- first bullet

Terminology clarification is needed as this implies that stress testing
is the same as load testing, which is not accurate.

Items appear to overlap due to being derived from several different
sources.

There is clearer language available on stress/volume testing; guidance
is not offered on how to decide how far to take this. Some rules of
thumb would be helpful.

Is "Key Testing Considerations" a discussion of the testing that would
occur during systems development (this is not clear), but is not
expected in validation testing (simulation tests, stress testing)?

5.4.1 - second bullet

Clarification is needed to confirm whether simulation tests are the same
as User Acceptance Testing (UAT). Also, clarify whether tests are run in
a production environment.

5.4.1 -- third bullet

The last sentence should be removed. It suggests parallel testing and it
is impractical at an industry level to maintain a second system that
mirrors production and can be run at the same time.

This could be interpreted to require significant levels of stress
testing. It would be impossible to have hundreds of people on the system
for hours at a time to try to support an extreme definition for stress
testing.

#### 

### Section 5.4.2 *Software testing should include*:

Some of the terms used in the documentation such as Dynamic Testing,
Functional Testing, etc., should be in the Glossary Of Terms.

5.4.2 -- first bullet

Remove the second sentence of the structural testing bullet. It is not
necessary if defined initially and then used consistently.

What is described is not dynamic testing, but rather static testing, and
therefore this discussion belongs in section 5.5. Note, also, that not
all white-box testing is via inspection. Any tests that require
knowledge of the way the code is constructed would be considered
white-box (also called "glass box," since the tester can see inside).
Specifically, some test approaches are contrived so that they exercise
all branches or paths of a given piece of code -- frequently limited to
a specific module, i.e., applied in the context of unit testing.

Remove the term "contemporary," as it does not add contextual meaning.

5.4.2 -- second bullet

Remove the last sentence of the functional testing bullet. This is not
necessary if defined initially and then used consistently.

5.4.2 -- third bullet

This is the first place that "program build testing" is mentioned. More
often, the references talk about unit testing, integration testing, and
system testing as separate activities. We suggest changing the name of
the last bullet from "Program build testing" to "System Testing." In the
software developer's world, "program build testing" has the more
specific meaning of verifying the completeness and readiness of a new
build, not the functionality therein. This seems to be a misnomer and an
inadequate substitute for System Testing -- the more industry-accepted
term for this group of test activities. Or, as used in the guidance, is
"program build testing" meant to be equivalent to "unit testing"?

[Section 5.4.3 *How test results should be expressed.*]{.underline}

Regarding "quantifying" test results, a properly designed test will be
expressed in such a way that the result is binary: either it worked
correctly or did not work correctly. Usually, there is no quantifiable
variable, only a question of whether "X" occurred or not. Correctly
designed testing is by its very nature pass/fail.

See also our comments under Section 5.2.3.

####### Section 5.5 *Static Verification Techniques*

"Static Verification Techniques" would encompass any form of inspection,
walkthrough, or technical review. In particular, link this back to the
comments about user requirements in section 5.1. Many authors in the
references cited by this guidance note that easily half of all software
defects can be traced to defects in requirements. Therefore the most
productive type of review/inspection an organization can conduct would
examine the requirements. Are the requirements clear and unambiguous?
Are they implementable? Are they testable? And of course, are they
complete?

What is the point of the qualified phrase "Where available, knowledge of
these activities and their outcomes can help to focus testing
efforts..."? The point should be expressed in a more straightforward
manner, such as, "When used, these activities and their outcomes can
help to focus testing efforts."

[Section 5.6 *Extent of Validation*]{.underline}

This appears to be essentially a 'Scope' section and should be moved
nearer to the Validation Plan statement earlier in the document (Section
5.2). Section 5.6 is important information when developing the
Validation Plan (5.2.1). This establishes the rationale behind the
documented approach. It would be helpful to make this connection
explicitly by proximity in the document.

[Section 5.7 *Independence of Review*]{.underline}

This section indicates that validation should be performed by persons
other than those responsible for building the system. It seems that the
Software Development Life-Cycle (SDLC) concept would indicate that
validation is, in part, an on-going activity concurrent with the
building as well as the testing of the system. Using the phrase,
"computer system validation should be performed by person other than
..." implies once again that computer system validation is equivalent to
and limited to testing activities instead of the sum-total of activities
conducted throughout the SDLC.

Also, the mention of Quality Assurance (QA) in this section appears to
be the only mention of QA in the entire document. There is no reference
to QA review or approval of any of the Validation Plan, procedures, or
report. This section (Section 5.7) would be an appropriate place to
indicate the need for review of the validation activities/deliverables
by QA (i.e., audit. The emphasis needs to be on the presence of a
quality system.

As in section 5.2.2, the term "validation" is being used to refer to
testing only.

# Section 5.8 *Change Control (Configuration Management)*

This section inaccurately implies that change control is equal to and
limited to configuration management. The heading of this section
misleads the reader to focus on the infrastructure- based change control
(configuration management, production environment change control, etc.).
This is a critical area of change control but is not the only area.
Release Change Management (management of changes in the development
environment covers source code management, CSV deliverable change
management, problem log management, etc.) is also essential.
Parenthetically, Document Change management is tightly linked to the
other two, and this aspect is highlighted in the Part 11 rule but absent
in this Guidance.

The concept of analyzing for possible regressions, then testing those
areas is understood, but more often regression testing is not performed
that way. Rather, an extensive suite of carefully designed and
controlled regression tests is carried out to provide assurance that
nothing was broken in other areas of the code.

What exactly is expected in regression analysis? How should this be
documented?

The expectation of regression analysis needs to be clarified. It could
be an endless task to evaluate the impact of each new or revised
software program on all the already validated software on that system.

[Section 6.1 *Commercial, Off-the-Shelf Software*]{.underline}

The term "customization" (used in the second from the last sentence of
this section) should be defined. We also suggest that the phrase "that
they prepare" be deleted, since parties other than the end user may
prepare customizations.

[Section 6.1.3 *Functional Testing of Software*]{.underline}

What is meant by the phrase, "more extensive functional testing" when in
a prior sentence it is already stated that functional testing should
cover "all" functions of the program?

[Section 6.2.1 *Internet Validation*]{.underline}

The author creates a contradiction in terminology within the first
sentence. "Internet Validation" is not an appropriate term and should be
modified to read "Internet Qualification." As with any infrastructure
component, the Internet fundamentally cannot be validated because it is
not a specified process. It can be qualified for use, can have
configuration change management, and a specific application can be
validated with the Internet qualification as an underlying component. As
such, the title on section 6.2.1 should be changed to Internet
Qualification.

In the second paragraph of this section, the text needs clarification
when discussing Internet-related activities. For example, there is no
use of expected 21 CFR Part 11 terminology such as 'open system' and
'closed system'. Additionally, there is no definition or guidance
regarding how one defines the 'end' of the Internet communications
pipeline. Is it the firewall at each end? Is it the web server?

Another bullet should be added: "Use of secure (encrypted) connections
between host and client".

The expectations for validation of work over the Internet are fairly
clear, but encryption is not. Since encryption actually transforms the
data, how is this expected to be validated? Is an audit trail required?

# Section *International and National Consensus Standards References*

If this document references the IEEE Software Engineering Standards, it
should mention a more up-to-date version rather than the 1994 version:

Vol 1: Customer and Terminology Standards (ISBN 0-7381-1559-2)

Vol 2: Process Standards (ISBN 0-7381-1560-6

Vol 3: Product Standards (ISBN 0-7381-1561-4)

Vol 4: Process Standards (ISBN 0-7381-1562-2)

Triangle PEERS appreciates the opportunity to submit comments to FDA\'s
Guidances on 21 CFR Part 11.

Sincerely,

Triangle PEERS

Research Triangle Park, North Carolina