######### APPENDIX A: EXAMINATION PROCEDURES {#appendix-a-examination-procedures .Appx-Head-1}

**EXAMINATION OBJECTIVES:** The objectives of Development and
Acquisition assessments are to identify weaknesses or risks that could
negatively impact an organization, to identify entities whose condition
or performance requires special supervisory attention, and to
subsequently effect corrective action.

Examiners should not expect organizations to employ formal project
management techniques in all situations. Reviews should be risk focused
and center on ensuring project management standards, controls, and
procedures are present and commensurate with the characteristics and
risks of the projects under review.

Examiners are not required to include lengthy responses to each
Objective or bulleted item. Often, examiners should be able to simply
note an item is adequate or inadequate with yes or no responses.
However, examiners must adequately document material findings.
Documentation must be sufficient to support the assignment of the
Development and Acquisition component rating of the Federal Financial
Institutions Examination Council's Uniform Rating System for Information
Technology.

PROCEDURES

+---------------------------+-----+-----------------------------------+
|                           | **W | **Comment**                       |
|                           | ork |                                   |
|                           | Pa  |                                   |
|                           | per |                                   |
|                           | Re  |                                   |
|                           | fer |                                   |
|                           | enc |                                   |
|                           | e** |                                   |
+===========================+=====+===================================+
| Objective 1: Determine    |     |                                   |
| the scope of the          |     |                                   |
| Development and           |     |                                   |
| Acquisition review.       |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Identify strengths    |     |                                   |
| and weaknesses relating   |     |                                   |
| to development,           |     |                                   |
| acquisition, and          |     |                                   |
| maintenance activities,   |     |                                   |
| through a review of:      |     |                                   |
|                           |     |                                   |
| -   Prior reports of      |     |                                   |
|     examination;          |     |                                   |
|                           |     |                                   |
| -   Internal and external |     |                                   |
|     audits;               |     |                                   |
|                           |     |                                   |
| -   Regulatory, audit,    |     |                                   |
|     and security reports  |     |                                   |
|     from key service      |     |                                   |
|     providers;            |     |                                   |
|                           |     |                                   |
| -   Organizational        |     |                                   |
|     charts;               |     |                                   |
|                           |     |                                   |
| -   Network topology      |     |                                   |
|     maps; and             |     |                                   |
|                           |     |                                   |
| -   Résumés of technology |     |                                   |
|     managers.             |     |                                   |
+---------------------------+-----+-----------------------------------+
| 2\. Review management's   |     |                                   |
| response to report and    |     |                                   |
| audit findings to         |     |                                   |
| determine:                |     |                                   |
|                           |     |                                   |
| -   The adequacy and      |     |                                   |
|     timing of corrective  |     |                                   |
|     actions;              |     |                                   |
|                           |     |                                   |
| -   The resolution of     |     |                                   |
|     root causes rather    |     |                                   |
|     than just specific    |     |                                   |
|     issues; and           |     |                                   |
|                           |     |                                   |
| -   The existence of      |     |                                   |
|     outstanding issues.   |     |                                   |
+---------------------------+-----+-----------------------------------+
| 3\. Review applicable     |     |                                   |
| documentation and         |     |                                   |
| interview technology      |     |                                   |
| managers to identify:     |     |                                   |
|                           |     |                                   |
| -   The type and          |     |                                   |
|     frequency of          |     |                                   |
|     development,          |     |                                   |
|     acquisition, and      |     |                                   |
|     maintenance projects; |     |                                   |
|                           |     |                                   |
| -   The formality and     |     |                                   |
|     characteristics of    |     |                                   |
|     project management    |     |                                   |
|     techniques;           |     |                                   |
|                           |     |                                   |
| -   The material changes  |     |                                   |
|     that impact           |     |                                   |
|     development,          |     |                                   |
|     acquisition, and      |     |                                   |
|     maintenance           |     |                                   |
|     activities, such as:  |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Proposed or enacted   |     |                                   |
|     changes in hardware,  |     |                                   |
|     software, or vendors; |     |                                   |
|                           |     |                                   |
| -   Proposed or enacted   |     |                                   |
|     changes in business   |     |                                   |
|     objectives or         |     |                                   |
|     organizational        |     |                                   |
|     structures; and       |     |                                   |
|                           |     |                                   |
| -   Proposed or enacted   |     |                                   |
|     changes in key        |     |                                   |
|     personnel positions.  |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 2: Assess the   |     |                                   |
| level of oversight and    |     |                                   |
| support provided by the   |     |                                   |
| board and management      |     |                                   |
| relating to development,  |     |                                   |
| acquisition, and          |     |                                   |
| maintenance activities.   |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Assess the level of   |     |                                   |
| oversight and support by  |     |                                   |
| evaluating:               |     |                                   |
|                           |     |                                   |
| -   The alignment of      |     |                                   |
|     business and          |     |                                   |
|     technology            |     |                                   |
|     objectives;           |     |                                   |
|                           |     |                                   |
| -   The frequency and     |     |                                   |
|     quality of            |     |                                   |
|     technology-related    |     |                                   |
|     board reporting;      |     |                                   |
|                           |     |                                   |
| -   The commitment of the |     |                                   |
|     board and senior      |     |                                   |
|     management to promote |     |                                   |
|     new products;         |     |                                   |
|                           |     |                                   |
| -   The level and quality |     |                                   |
|     of board-approved     |     |                                   |
|     project standards and |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   The qualifications of |     |                                   |
|     technology managers;  |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| -   The sufficiency of    |     |                                   |
|     technology budgets.   |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 3: Assess the   |     |                                   |
| organizational structure  |     |                                   |
| in relation to the        |     |                                   |
| appropriateness of        |     |                                   |
| assigned responsibilities |     |                                   |
| concerning technology     |     |                                   |
| systems and initiatives.  |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Evaluate              |     |                                   |
| organizational            |     |                                   |
| responsibilities to       |     |                                   |
| ensure the board and      |     |                                   |
| management:               |     |                                   |
|                           |     |                                   |
| -   Clearly define and    |     |                                   |
|     appropriately assign  |     |                                   |
|     responsibilities;     |     |                                   |
|                           |     |                                   |
| -   Appropriately assign  |     |                                   |
|     security, audit, and  |     |                                   |
|     quality assurance     |     |                                   |
|     personnel to          |     |                                   |
|     technology-related    |     |                                   |
|     projects;             |     |                                   |
|                           |     |                                   |
| -   Establish appropriate |     |                                   |
|     segregation-of-duty   |     |                                   |
|     or compensating       |     |                                   |
|     controls; and         |     |                                   |
|                           |     |                                   |
| -   Establish appropriate |     |                                   |
|     project, technology   |     |                                   |
|     committee, and board  |     |                                   |
|     reporting             |     |                                   |
|     requirements.         |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 4: Assess the   |     |                                   |
| level and characteristics |     |                                   |
| of risks associated with  |     |                                   |
| development, acquisition, |     |                                   |
| and maintenance           |     |                                   |
| activities that could     |     |                                   |
| materially impact the     |     |                                   |
| organization.             |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Assess the risks      |     |                                   |
| identified in other       |     |                                   |
| objectives and evaluate   |     |                                   |
| the adequacy of risk      |     |                                   |
| management programs       |     |                                   |
| regarding:                |     |                                   |
|                           |     |                                   |
| -   Risk identification   |     |                                   |
|     and assessment        |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Risk reporting and    |     |                                   |
|     monitoring            |     |                                   |
|     procedures; and       |     |                                   |
|                           |     |                                   |
| -   Risk acceptance,      |     |                                   |
|     mitigation, and       |     |                                   |
|     transfer strategies.  |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 5: Assess the   |     |                                   |
| adequacy of development   |     |                                   |
| project management        |     |                                   |
| standards, methodologies, |     |                                   |
| and practices.            |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Evaluate the adequacy |     |                                   |
| of development activities |     |                                   |
| by assessing:             |     |                                   |
|                           |     |                                   |
| -   The adequacy of, and  |     |                                   |
|     adherence to,         |     |                                   |
|     development standards |     |                                   |
|     and controls;         |     |                                   |
|                           |     |                                   |
| -   The applicability and |     |                                   |
|     effectiveness of      |     |                                   |
|     project management    |     |                                   |
|     methodologies;        |     |                                   |
|                           |     |                                   |
| -   The experience of     |     |                                   |
|     project managers;     |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     project plans,        |     |                                   |
|     particularly with     |     |                                   |
|     regard to the         |     |                                   |
|     inclusion of clearly  |     |                                   |
|     defined:              |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Phase expectations;   |     |                                   |
|                           |     |                                   |
| -   Phase acceptance      |     |                                   |
|     criteria;             |     |                                   |
|                           |     |                                   |
| -   Security and control  |     |                                   |
|     requirements;         |     |                                   |
|                           |     |                                   |
| -   Testing requirements; |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| -   Documentation         |     |                                   |
|     requirements;         |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The formality and     |     |                                   |
|     effectiveness of      |     |                                   |
|     quality assurance     |     |                                   |
|     programs;             |     |                                   |
|                           |     |                                   |
| -   The effectiveness of  |     |                                   |
|     risk management       |     |                                   |
|     programs;             |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     project request and   |     |                                   |
|     approval procedures;  |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     feasibility studies;  |     |                                   |
|                           |     |                                   |
| -   The adequacy of, and  |     |                                   |
|     adherence to,         |     |                                   |
|     standards and         |     |                                   |
|     procedures relating   |     |                                   |
|     to the:               |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Design phase;         |     |                                   |
|                           |     |                                   |
| -   Development phase;    |     |                                   |
|                           |     |                                   |
| -   Testing phase; and    |     |                                   |
|                           |     |                                   |
| -   Implementation phase; |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The adequacy of       |     |                                   |
|     project change        |     |                                   |
|     controls;             |     |                                   |
|                           |     |                                   |
| -   The appropriate       |     |                                   |
|     inclusion of          |     |                                   |
|     organizational        |     |                                   |
|     personnel throughout  |     |                                   |
|     the project's life    |     |                                   |
|     cycle;                |     |                                   |
|                           |     |                                   |
| -   The effectiveness of  |     |                                   |
|     project communication |     |                                   |
|     and reporting         |     |                                   |
|     procedures; and       |     |                                   |
|                           |     |                                   |
| -   The accuracy,         |     |                                   |
|     effectiveness, and    |     |                                   |
|     control of project    |     |                                   |
|     management tools.     |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 6: Assess the   |     |                                   |
| adequacy of acquisition   |     |                                   |
| project management        |     |                                   |
| standards, methodologies, |     |                                   |
| and practices.            |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Assess the adequacy   |     |                                   |
| of acquisition activities |     |                                   |
| by evaluating:            |     |                                   |
|                           |     |                                   |
| -   The adequacy of, and  |     |                                   |
|     adherence to,         |     |                                   |
|     acquisition standards |     |                                   |
|     and controls;         |     |                                   |
|                           |     |                                   |
| -   The applicability and |     |                                   |
|     effectiveness of      |     |                                   |
|     project management    |     |                                   |
|     methodologies;        |     |                                   |
|                           |     |                                   |
| -   The experience of     |     |                                   |
|     project managers;     |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     project plans,        |     |                                   |
|     particularly with     |     |                                   |
|     regard to the         |     |                                   |
|     inclusion of clearly  |     |                                   |
|     defined:              |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Phase expectations;   |     |                                   |
|                           |     |                                   |
| -   Phase acceptance      |     |                                   |
|     criteria;             |     |                                   |
|                           |     |                                   |
| -   Security and control  |     |                                   |
|     requirements; and     |     |                                   |
|                           |     |                                   |
| -   Testing, training,    |     |                                   |
|     and implementation    |     |                                   |
|     requirements;         |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The formality and     |     |                                   |
|     effectiveness of      |     |                                   |
|     quality assurance     |     |                                   |
|     programs;             |     |                                   |
|                           |     |                                   |
| -   The effectiveness of  |     |                                   |
|     risk management       |     |                                   |
|     programs;             |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     project request and   |     |                                   |
|     approval procedures;  |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     feasibility studies;  |     |                                   |
|                           |     |                                   |
| -   The adequacy of, and  |     |                                   |
|     adherence to,         |     |                                   |
|     standards that        |     |                                   |
|     require               |     |                                   |
|     request-for-proposals |     |                                   |
|     and                   |     |                                   |
|     invitations-to-tender |     |                                   |
|     to include:           |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Well-detailed         |     |                                   |
|     security,             |     |                                   |
|     reliability, and      |     |                                   |
|     functionality         |     |                                   |
|     specifications;       |     |                                   |
|                           |     |                                   |
| -   Well-defined          |     |                                   |
|     performance and       |     |                                   |
|     compatibility         |     |                                   |
|     specifications; and   |     |                                   |
|                           |     |                                   |
| -   Well-defined design   |     |                                   |
|     and development       |     |                                   |
|     documentation         |     |                                   |
|     requirements;         |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The adequacy of, and  |     |                                   |
|     adherence to,         |     |                                   |
|     standards that        |     |                                   |
|     require:              |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Thorough reviews of   |     |                                   |
|     vendors' financial    |     |                                   |
|     condition and         |     |                                   |
|     commitment to         |     |                                   |
|     service; and          |     |                                   |
|                           |     |                                   |
| -   Thorough reviews of   |     |                                   |
|     contracts and         |     |                                   |
|     licensing agreements  |     |                                   |
|     prior to signing;     |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The adequacy of       |     |                                   |
|     contract and          |     |                                   |
|     licensing provisions  |     |                                   |
|     that address:         |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Performance           |     |                                   |
|     assurances;           |     |                                   |
|                           |     |                                   |
| -   Software and data     |     |                                   |
|     security provisions;  |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| -   Source-code           |     |                                   |
|     accessibility/ escrow |     |                                   |
|     assertions;           |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The adequacy of       |     |                                   |
|     project change        |     |                                   |
|     controls;             |     |                                   |
|                           |     |                                   |
| -   The appropriate       |     |                                   |
|     inclusion of          |     |                                   |
|     organizational        |     |                                   |
|     personnel throughout  |     |                                   |
|     the project's life    |     |                                   |
|     cycle;                |     |                                   |
|                           |     |                                   |
| -   The effectiveness of  |     |                                   |
|     project communication |     |                                   |
|     and reporting         |     |                                   |
|     procedures; and       |     |                                   |
|                           |     |                                   |
| -   The accuracy,         |     |                                   |
|     effectiveness, and    |     |                                   |
|     control of project    |     |                                   |
|     management tools.     |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 7: Assess the   |     |                                   |
| adequacy of maintenance   |     |                                   |
| project management        |     |                                   |
| standards, methodologies, |     |                                   |
| and practices             |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Evaluate the          |     |                                   |
| sufficiency of, and       |     |                                   |
| adherence to, maintenance |     |                                   |
| standards and controls    |     |                                   |
| relating to:              |     |                                   |
|                           |     |                                   |
| -   Change request and    |     |                                   |
|     approval procedures;  |     |                                   |
|                           |     |                                   |
| -   Change testing        |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Change implementation |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Change review         |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Change documentation  |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Change notification   |     |                                   |
|     procedures            |     |                                   |
|                           |     |                                   |
| -   Library controls; and |     |                                   |
|                           |     |                                   |
| -   Utility program       |     |                                   |
|     controls.             |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 8: Assess the   |     |                                   |
| effectiveness of          |     |                                   |
| conversion projects.      |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Evaluate the          |     |                                   |
| effectiveness of          |     |                                   |
| conversion projects by:   |     |                                   |
|                           |     |                                   |
| -   Comparing initial     |     |                                   |
|     budgets and projected |     |                                   |
|     time lines against    |     |                                   |
|     actual results;       |     |                                   |
|                           |     |                                   |
| -   Reviewing project     |     |                                   |
|     management and        |     |                                   |
|     technology committee  |     |                                   |
|     reports;              |     |                                   |
|                           |     |                                   |
| -   Reviewing testing     |     |                                   |
|     documentation and     |     |                                   |
|     after-action reports; |     |                                   |
|                           |     |                                   |
| -   Reviewing conversion  |     |                                   |
|     after-action reports; |     |                                   |
|                           |     |                                   |
| -   Interviewing          |     |                                   |
|     technology and user   |     |                                   |
|     personnel; and        |     |                                   |
|                           |     |                                   |
| -   Reviewing suspense    |     |                                   |
|     accounts for          |     |                                   |
|     outstanding items.    |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 9: Assess the   |     |                                   |
| adequacy of quality       |     |                                   |
| assurance programs.       |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Assess the adequacy   |     |                                   |
| of quality assurance      |     |                                   |
| programs by evaluating:   |     |                                   |
|                           |     |                                   |
| -   The board's           |     |                                   |
|     willingness to        |     |                                   |
|     provide appropriate   |     |                                   |
|     resources to quality  |     |                                   |
|     assurance programs;   |     |                                   |
|                           |     |                                   |
| -   The completeness of   |     |                                   |
|     quality assurance     |     |                                   |
|     procedures (Are the   |     |                                   |
|     deliverables of each  |     |                                   |
|     project, and project  |     |                                   |
|     phase, including the  |     |                                   |
|     validation of initial |     |                                   |
|     project assumptions   |     |                                   |
|     and approvals,        |     |                                   |
|     appropriately         |     |                                   |
|     assured?);            |     |                                   |
|                           |     |                                   |
| -   The scalability of    |     |                                   |
|     quality assurance     |     |                                   |
|     procedures (Are the   |     |                                   |
|     procedures            |     |                                   |
|     appropriately         |     |                                   |
|     tailored to match the |     |                                   |
|     characteristics of    |     |                                   |
|     the project?);        |     |                                   |
|                           |     |                                   |
| -   The measurability of  |     |                                   |
|     quality assurance     |     |                                   |
|     standards (Are        |     |                                   |
|     deliverables assessed |     |                                   |
|     against predefined    |     |                                   |
|     standards and         |     |                                   |
|     expectations?);       |     |                                   |
|                           |     |                                   |
| -   The adherence to      |     |                                   |
|     problem-tracking      |     |                                   |
|     standards that        |     |                                   |
|     require:              |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Appropriate problem   |     |                                   |
|     recordation;          |     |                                   |
|                           |     |                                   |
| -   Appropriate problem   |     |                                   |
|     reporting;            |     |                                   |
|                           |     |                                   |
| -   Appropriate problem   |     |                                   |
|     monitoring; and       |     |                                   |
|                           |     |                                   |
| -   Appropriate problem   |     |                                   |
|     correction;           |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The sufficiency of,   |     |                                   |
|     and adherence to,     |     |                                   |
|     testing standards     |     |                                   |
|     that require:         |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The use of            |     |                                   |
|     predefined,           |     |                                   |
|     comprehensive test    |     |                                   |
|     plans;                |     |                                   |
|                           |     |                                   |
| -   The involvement of    |     |                                   |
|     end users;            |     |                                   |
|                           |     |                                   |
| -   The documentation of  |     |                                   |
|     test results;         |     |                                   |
|                           |     |                                   |
| -   The prohibition       |     |                                   |
|     against testing in    |     |                                   |
|     production            |     |                                   |
|     environments; and     |     |                                   |
|                           |     |                                   |
| -   The prohibition       |     |                                   |
|     against testing with  |     |                                   |
|     live data;            |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The sufficiency and   |     |                                   |
|     effectiveness of      |     |                                   |
|     testing programs      |     |                                   |
|     regarding:            |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The accuracy of       |     |                                   |
|     programmed code;      |     |                                   |
|                           |     |                                   |
| -   The inclusion of      |     |                                   |
|     expected              |     |                                   |
|     functionality; and    |     |                                   |
|                           |     |                                   |
| -   The interoperability  |     |                                   |
|     of applications and   |     |                                   |
|     network components;   |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   The independence of   |     |                                   |
|     quality assurance     |     |                                   |
|     personnel.            |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 10: Assess the  |     |                                   |
| adequacy of program       |     |                                   |
| change controls.          |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1.  Evaluate the          |     |                                   |
|     sufficiency of, and   |     |                                   |
|     adherence to:         |     |                                   |
|                           |     |                                   |
| -   Routine and emergency |     |                                   |
|     program-change        |     |                                   |
|     standards that        |     |                                   |
|     require appropriate:  |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Request and approval  |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Testing procedures;   |     |                                   |
|                           |     |                                   |
| -   Implementation        |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Backup and backout    |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Documentation         |     |                                   |
|     procedures; and       |     |                                   |
|                           |     |                                   |
| -   Notification          |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Controls that         |     |                                   |
|     restrict the          |     |                                   |
|     unauthorized movement |     |                                   |
|     of programs or        |     |                                   |
|     program               |     |                                   |
|     modules/objects       |     |                                   |
|     between development,  |     |                                   |
|     testing, and          |     |                                   |
|     production            |     |                                   |
|     environments;         |     |                                   |
|                           |     |                                   |
| -   Controls that         |     |                                   |
|     restrict the          |     |                                   |
|     unauthorized use of   |     |                                   |
|     utility programs,     |     |                                   |
|     such as:              |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Policy prohibitions;  |     |                                   |
|                           |     |                                   |
| -   Monitoring of use;    |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| -   Logical access        |     |                                   |
|     controls;             |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Library controls that |     |                                   |
|     restrict unauthorized |     |                                   |
|     access to programs    |     |                                   |
|     outside an            |     |                                   |
|     individual's assigned |     |                                   |
|     responsibilities such |     |                                   |
|     as:                   |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Logical access        |     |                                   |
|     controls on all       |     |                                   |
|     libraries or objects  |     |                                   |
|     within libraries; and |     |                                   |
|                           |     |                                   |
| -   Automated library     |     |                                   |
|     controls that         |     |                                   |
|     restrict library      |     |                                   |
|     access and produce    |     |                                   |
|     reports that identify |     |                                   |
|     who accessed a        |     |                                   |
|     library, what was     |     |                                   |
|     accessed, and what    |     |                                   |
|     changes were made;    |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| ```{=html}                |     |                                   |
| <!-- -->                  |     |                                   |
| ```                       |     |                                   |
| -   Version controls that |     |                                   |
|     facilitate the        |     |                                   |
|     appropriate retention |     |                                   |
|     of programs, and      |     |                                   |
|     program               |     |                                   |
|     modules/objects,      |     |                                   |
|     revisions, and        |     |                                   |
|     documentation.        |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 11: Assess the  |     |                                   |
| adequacy of               |     |                                   |
| patch-management          |     |                                   |
| standards and controls.   |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Evaluate the          |     |                                   |
| sufficiency of, and       |     |                                   |
| adherence to,             |     |                                   |
| patch-management          |     |                                   |
| standards and controls    |     |                                   |
| that require:             |     |                                   |
|                           |     |                                   |
| -   Detailed hardware and |     |                                   |
|     software inventories; |     |                                   |
|                           |     |                                   |
| -   Patch identification  |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Patch evaluation      |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Patch request and     |     |                                   |
|     approval procedures;  |     |                                   |
|                           |     |                                   |
| -   Patch testing         |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Backup and backout    |     |                                   |
|     procedures;           |     |                                   |
|                           |     |                                   |
| -   Patch implementation  |     |                                   |
|     procedures; and       |     |                                   |
|                           |     |                                   |
| -   Patch documentation.  |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 12: Assess the  |     |                                   |
| quality of application,   |     |                                   |
| system, and project       |     |                                   |
| documentation, and the    |     |                                   |
| adequacy of documentation |     |                                   |
| controls.                 |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Assess the adequacy   |     |                                   |
| of documentation controls |     |                                   |
| by evaluating the         |     |                                   |
| sufficiency of, and       |     |                                   |
| adherence to,             |     |                                   |
| documentation standards   |     |                                   |
| that require:             |     |                                   |
|                           |     |                                   |
| -   The assignment of     |     |                                   |
|                           |     |                                   |
|   documentation-custodian |     |                                   |
|     responsibilities;     |     |                                   |
|                           |     |                                   |
| -   The assignment of     |     |                                   |
|     document authoring    |     |                                   |
|     and approval          |     |                                   |
|     responsibilities;     |     |                                   |
|                           |     |                                   |
| -   The establishment of  |     |                                   |
|     standardized document |     |                                   |
|     formats; and          |     |                                   |
|                           |     |                                   |
| -   The establishment of  |     |                                   |
|     appropriate           |     |                                   |
|     documentation library |     |                                   |
|     and version controls. |     |                                   |
+---------------------------+-----+-----------------------------------+
| 2\. Assess the quality of |     |                                   |
| application documentation |     |                                   |
| by evaluating the         |     |                                   |
| adequacy of internal and  |     |                                   |
| external assessments of:  |     |                                   |
|                           |     |                                   |
| -   Application design    |     |                                   |
|     and coding standards; |     |                                   |
|                           |     |                                   |
| -   Application           |     |                                   |
|     descriptions;         |     |                                   |
|                           |     |                                   |
| -   Application design    |     |                                   |
|     documents;            |     |                                   |
|                           |     |                                   |
| -   Application           |     |                                   |
|     source-code listings  |     |                                   |
|     (or in the case of    |     |                                   |
|     object-oriented       |     |                                   |
|     programming: object   |     |                                   |
|     listings);            |     |                                   |
|                           |     |                                   |
| -   Application routine   |     |                                   |
|     naming conventions    |     |                                   |
|     (or in the case of    |     |                                   |
|     object-oriented       |     |                                   |
|     programming: object   |     |                                   |
|     naming conventions);  |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| -   Application operator  |     |                                   |
|     instructions and user |     |                                   |
|     manuals.              |     |                                   |
+---------------------------+-----+-----------------------------------+
| 3\. Assess the quality of |     |                                   |
| open source-code system   |     |                                   |
| documentation by          |     |                                   |
| evaluating the adequacy   |     |                                   |
| of internal and external  |     |                                   |
| assessments of:           |     |                                   |
|                           |     |                                   |
| -   System design and     |     |                                   |
|     coding standards;     |     |                                   |
|                           |     |                                   |
| -   System descriptions;  |     |                                   |
|                           |     |                                   |
| -   System design         |     |                                   |
|     documents;            |     |                                   |
|                           |     |                                   |
| -   Source-code listings  |     |                                   |
|     (or in the case of    |     |                                   |
|     object-oriented       |     |                                   |
|     programming: object   |     |                                   |
|     listings);            |     |                                   |
|                           |     |                                   |
| -   Source-code routine   |     |                                   |
|     naming conventions    |     |                                   |
|     (or in the case of    |     |                                   |
|     object-oriented       |     |                                   |
|     programming: object   |     |                                   |
|     naming conventions);  |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| -   System operation      |     |                                   |
|     instructions.         |     |                                   |
+---------------------------+-----+-----------------------------------+
| 4\. Assess the quality of |     |                                   |
| project documentation by  |     |                                   |
| evaluating the adequacy   |     |                                   |
| of documentation relating |     |                                   |
| to the:                   |     |                                   |
|                           |     |                                   |
| -   Project request;      |     |                                   |
|                           |     |                                   |
| -   Feasibility study;    |     |                                   |
|                           |     |                                   |
| -   Initiation phase;     |     |                                   |
|                           |     |                                   |
| -   Planning phase;       |     |                                   |
|                           |     |                                   |
| -   Design phase;         |     |                                   |
|                           |     |                                   |
| -   Development phase;    |     |                                   |
|                           |     |                                   |
| -   Testing phase;        |     |                                   |
|                           |     |                                   |
| -   Implementation phase; |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| -   Post-implementation   |     |                                   |
|     reviews.              |     |                                   |
+---------------------------+-----+-----------------------------------+
| Note: If examiners employ |     |                                   |
| sampling techniques, they |     |                                   |
| should include planning   |     |                                   |
| and testing phase         |     |                                   |
| documentation in the      |     |                                   |
| sample.                   |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 13: Assess the  |     |                                   |
| security and integrity of |     |                                   |
| system and application    |     |                                   |
| software.                 |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Evaluate the security |     |                                   |
| and integrity of system   |     |                                   |
| and application software  |     |                                   |
| by reviewing:             |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     quality assurance and |     |                                   |
|     testing programs;     |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     security and          |     |                                   |
|     internal-control      |     |                                   |
|     design standards;     |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     program change        |     |                                   |
|     controls;             |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     involvement by audit  |     |                                   |
|     and security          |     |                                   |
|     personnel in software |     |                                   |
|     development and       |     |                                   |
|     acquisition projects; |     |                                   |
|     and                   |     |                                   |
|                           |     |                                   |
| -   The adequacy of       |     |                                   |
|     internal and external |     |                                   |
|     security and control  |     |                                   |
|     audits.               |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 14: Assess the  |     |                                   |
| ability of information    |     |                                   |
| technology solutions to   |     |                                   |
| meet the needs of the end |     |                                   |
| users.                    |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Interview end users   |     |                                   |
| to determine their        |     |                                   |
| assessment of technology  |     |                                   |
| solutions.                |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 15: Assess the  |     |                                   |
| extent of end-user        |     |                                   |
| involvement in the system |     |                                   |
| development and           |     |                                   |
| acquisition process.      |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Interview end users   |     |                                   |
| and review development    |     |                                   |
| and acquisition project   |     |                                   |
| documentation to          |     |                                   |
| determine the extent of   |     |                                   |
| end-user involvement.     |     |                                   |
+---------------------------+-----+-----------------------------------+
| Conclusions               |     |                                   |
+---------------------------+-----+-----------------------------------+
| Objective 16: Document    |     |                                   |
| and discuss findings and  |     |                                   |
| recommend corrective      |     |                                   |
| actions.                  |     |                                   |
+---------------------------+-----+-----------------------------------+
| 1\. Document findings and |     |                                   |
| recommendations regarding |     |                                   |
| the quality and           |     |                                   |
| effectiveness of the      |     |                                   |
| organization's            |     |                                   |
| Development and           |     |                                   |
| Acquisition standards and |     |                                   |
| procedures.               |     |                                   |
+---------------------------+-----+-----------------------------------+
| 2\. Discuss preliminary   |     |                                   |
| findings with the         |     |                                   |
| examiner-in-charge        |     |                                   |
| regarding:                |     |                                   |
|                           |     |                                   |
| · Violations of laws,     |     |                                   |
| rulings, or regulations;  |     |                                   |
| and                       |     |                                   |
|                           |     |                                   |
| · Issues warranting       |     |                                   |
| inclusion in the report   |     |                                   |
| of examination.           |     |                                   |
+---------------------------+-----+-----------------------------------+
| 3\. Discuss your findings |     |                                   |
| with management and       |     |                                   |
| obtain commitments for    |     |                                   |
| corrective actions and    |     |                                   |
| deadlines for remedying   |     |                                   |
| significant deficiencies. |     |                                   |
+---------------------------+-----+-----------------------------------+
| 4\. Discuss findings with |     |                                   |
| the examiner-in-charge    |     |                                   |
| regarding:                |     |                                   |
|                           |     |                                   |
| · Recommendations         |     |                                   |
| regarding the Development |     |                                   |
| and Acquisition rating;   |     |                                   |
| and                       |     |                                   |
|                           |     |                                   |
| · Recommendations         |     |                                   |
| regarding the impact of   |     |                                   |
| your conclusions on the   |     |                                   |
| composite rating(s).      |     |                                   |
+---------------------------+-----+-----------------------------------+
| 5\. Document your         |     |                                   |
| conclusions in a memo to  |     |                                   |
| the examiner-in-charge    |     |                                   |
| that provides             |     |                                   |
| report-ready comments for |     |                                   |
| all relevant sections of  |     |                                   |
| the report of             |     |                                   |
| examination.              |     |                                   |
+---------------------------+-----+-----------------------------------+
| 6\. Organize your work    |     |                                   |
| papers to ensure clear    |     |                                   |
| support for significant   |     |                                   |
| findings and              |     |                                   |
| recommendations.          |     |                                   |
+---------------------------+-----+-----------------------------------+

  ----------------------- ----------- ----------- -----------------------
  Examiner                            Date        

                                                  

                                                  

  Reviewer's Initials                             

                                                  
  ----------------------- ----------- ----------- -----------------------