![](media/image1.jpeg){width="2.9722222222222223in"
height="2.946527777777778in"}

Human Factors Assessments

in

Investment Analysis:

Definition and Process Summary for Cost, Risk, and Benefit

Ver 1.4

June 9, 2006

Federal Aviation Administration

Human Factors Research and Engineering Division

AAR-100

POC: Glen Hewitt, (202) 267-7163

Human Factors Assessments in Investment Analysis:

Definition and Process Summary for Cost, Risk, and Benefit

## 

**1. Purpose:** This document provides a brief description of a "Human
Factors Assessment" especially those conducted during the Investment
Analysis process.

**2. Definition:** The Human Factors Assessment is a process that is
integrated with other system acquisition processes and provides
essential components to the products of the Investment Analysis (IA).
Three of these human factors components are: a) the human-system
performance contribution to program benefits, b) an assessment of the
human-system performance risks, and c) the estimated costs associated
with mitigating human factors risks and with conducting the engineering
program support. The human factors components related to benefits,
risks, and costs are integrated with other program components in the IA
products and system acquisition documentation.

**3. Background:** During the conduct of the FAA Acquisition Management
System (AMS) Investment Analysis phase, human factors research and
engineering practitioners conduct various activities to provide critical
information to the program activities and documentation. The purpose of
conducting human factors research and engineering activities (as
outlined in Chapter 5 of the FAA Human Factors Job Aid, dated December
2003, is to ensure that

:

-   Human-system performance capabilities and limitations are properly
    reflected in the system requirements (e.g., Attachment 1 of the OMB
    Exhibit 300, Program Requirements)

-   Human-system performance characteristics and their associated cost,
    benefits, and risks assist in deciding among investment alternatives
    (e.g., Attachment 2 of the OMB Exhibit 300, Business Case Analysis)

-   Human-system performance risks and their mitigation are
    appropriately addressed in program development and implementation
    plans and activities (e.g. Attachment 3 of the OMB Exhibit 300,
    Implementation Strategy and Planning)

**4. Process Description:** Investment analyses often follow a broadly
accepted process for the inclusion of the human factors contributions to
IA products. These contributions may support the "comparative"
evaluation of solution alternatives being considered or support the
"detailed" definition of one or more selected alternatives. Investment
Analysis Teams (including benefits, cost, and risk assessment sub-teams)
designate human factors practitioners to support the IA process. In
conjunction with producing the Human Factors Assessment for Investment
Analysis, these human factors practitioners also support IA team
activities including:

-   Investment Analysis Plan

-   Requirements Definition Activities

-   Market Survey

-   Alternative Solution Identification and Analysis

-   Affordability Assessment and Trade Studies

-   Acquisition Program Baseline Development

-   Investment Analysis report, briefing, and recommendations

#### Human Factors Assessments in the Investment Analysis

Figure 1: The relationship of Human Factors Assessment key products to
the major products of the Investment Analysis phase

In addition to other support activities, human factors practitioners
provide input to IA products related to benefits, risks, and costs as
depicted in Figure 1 and outlined below:

**4.1 Benefits:** Efforts related to identifying system/program benefits
provided in quantitative and qualitative terms must be congruous with
human-in-the-loop performance limitations and performance enhancements.
There are various ways by which to conduct the benefits analysis for
human-in-the-loop performance impacts, including the methodology
described in "Framework for Evaluation of Human-System Issues with
ASDE-X and Related Surface Safety Systems" prepared by Raja Parasuraman
(Catholic University of America), John Hansman (Massachusetts Institute
of Technology), Steven Bussolari (Massachusetts Institute of Technology)
dated January 12, 2002, which can be viewed at the FAA Human Factors
website (https://hf.faa.gov/). The activities necessary to identify
human-in-the-loop performance limitations provide the basis for
conducting a human factors risk analysis (see "Risks" below).
Human-in-the-loop performance enhancements are likely to reflect both
cost avoidance opportunities (e.g., lower staffing, lower training time,
lower costs) and operational improvements (e.g., increased safety, more
effective procedures, increased performance and productivity). \[Note:
These performance enhancements may not come free of costs which must
also be captured in the IA cost analysis.\]

**4.2 Risks:** Efforts related to identifying system/program risks must
include consideration of the human factors and human-system performance
risks. Human factors risk analyses should provide information on what is
known and unknown about the human-system performance in meeting minimum
or desired system performance requirements. Human factors considerations
that are relevant to meeting system performance and functional
requirements include:

1)  human performance (e.g., human capabilities and limitations,
    workload, function allocation, hardware and software design,
    decision aids, environmental constraints, and team versus individual
    performance)

2)  training (e.g., length of training, training effectiveness,
    retraining, training devices and facilities, and embedded training)

3)  staffing (e.g., staffing levels, team composition, and
    organizational structure)

4)  personnel selection (e.g., minimum skill levels, special skills,
    anthropometrics, demographics, and experience levels)

5)  safety and health aspects (e.g., hazardous materials or conditions,
    system or equipment design, operational or procedural constraints,
    biomedical influences, protective equipment, and required warnings
    and alarms).

The risk analyses and products provide, for each alternative, the full
range of human factors and human-system interface requirements (e.g.,
cognitive, organizational, physical, functional, environmental)
necessary to achieve an acceptable level of performance for operating,
maintaining, and supporting the system. For each of the identified
risks, the human factors practitioner provides a risk description and
rating, a mitigation strategy, and an estimated cost for resolving or
mitigating the identified risk (see "Costs" below). (Identified risks
may also serve as an indication of the level of the complexity of the
human-system interface that impacts the cost of the human factors
effort.) The mitigation strategies should be defined in enough detail to
outline the essential actions or activities that need to be conducted
during the design and development phase. These mitigation strategies and
activities are later incorporated into the program acquisition strategy
and the integrated program plan (e.g., Attachment 3 of the OMB Exhibit
300, Implementation Strategy and Planning).

There are various ways to identify and categorize the human factors
risks including an approach described in Attachment 1 that entails
rating 24 standard human factors risk areas. Once human factors risks
are identified and mitigation strategies devised, they may be translated
into one or many overall current and predicted residual risks for human
factors using the commonly accepted risk grid below (Figure 2). Using
the grid enables the human factors risk(s) to be easily integrated with
other program risks. Additional guidance is provided in table 1 through
table 3 below to assist in risk identification and in the estimation of
the severity and probability of an adverse event related to human
factors. One method for estimating risk for each of the 24 human factors
issues (function allocation, work load, situational awareness, staffing,
etc.) is to use a set of assessment criteria such as those identified in
Attachment 2. Other methods may be used.

Figure 2: Risk Grid

Table 1 provides a general checklist to assist in the identification of
risks related to human-system effectiveness, suitability, and user
acceptance.

**Table** 1 Human Factors Risk Checklist

+-----------------------------------------------------------------------+
| **Human Factors**                                                     |
+-----------------------------------------------------------------------+
| **Human-in-the-loop Effectiveness**                                   |
|                                                                       |
| -   Inadequate definition of human-in-the-loop operational objectives |
|                                                                       |
| -   Inadequate specification of human-in-the-loop benefits            |
|                                                                       |
| -   Inadequate analysis of human-in-the-loop system capability to     |
|     deliver expected benefits or enhancement                          |
|                                                                       |
| -   Human error mechanisms not fully identified                       |
|                                                                       |
| -   Time required to perform tasks is unknown                         |
|                                                                       |
| -   Automation does not provide the necessary functionality to        |
|     support effective decision-making/problem-solving                 |
|                                                                       |
| **Human-in-the-loop Suitability**                                     |
|                                                                       |
| -   Lack of consistency, compatibility, or congruity with operational |
|     environment or legacy systems.                                    |
|                                                                       |
| -   Human-system design/interface induces new/additional human error  |
|     potential                                                         |
|                                                                       |
| -   Inadequate incorporation of functional requirements to support    |
|     user-system performance goals                                     |
|                                                                       |
| **User Acceptability**                                                |
|                                                                       |
| -   New tasks impose excessive attentional, memory, or workload       |
|     demands                                                           |
|                                                                       |
| -   Requires new teaming and communication links                      |
|                                                                       |
| -   Operations interface is unacceptable to user                      |
|                                                                       |
| ```{=html}                                                            |
| <!-- -->                                                              |
| ```                                                                   |
| -   Maintenance interface is unacceptable to user                     |
+-----------------------------------------------------------------------+

Table 2 provides criteria for estimating the probability of an adverse
event related to human factors.

Table 2: Estimating the Probability of an Adverse Event Related to Human
Factors

+-----------------------+-----------------------+----------------------+
| High Probability of   | **Medium Probability  | **Low Probability of |
| an Adverse Event      | of an Adverse Event** | an Adverse Event**   |
+=======================+=======================+======================+
| If one or more of the | If one or more of the | If all of the        |
| following conditions  | following conditions  | following conditions |
| are present:          | are present:          | are present:         |
|                       |                       |                      |
| 1)  System            | 1)  System            | 1)  System           |
|     requirements or   |     requirements or   |     requirements and |
|     designs lack      |     designs include   |     designs include  |
|     human-system      |     incomplete        |     human-system     |
|     performance       |     human-system      |     performance      |
|     objectives or are |     performance       |     objectives       |
|     derived without   |     objectives or are |     derived from     |
|     comprehensive     |     derived with      |     comprehensive    |
|     human-in-the-loop |     limited           |                      |
|     performance       |     human-in-the-loop |    human-in-the-loop |
|     research,         |     performance       |     performance      |
|     studies, or       |     research,         |     research,        |
|     analyses.         |     studies, or       |     studies, and     |
|                       |     analyses.         |     analyses.        |
| 2)  Human-in-the-loop |                       |                      |
|     performance goals | 2)  Human-in-the-loop | 2)  Analysis         |
|     are unstated or   |     performance goals |     indicates that   |
|     not achievable    |     are partially     |                      |
|     within the        |     stated or         |    human-in-the-loop |
|     proposed          |     partially         |     performance      |
|     operational and   |     achievable within |     goals are        |
|     maintenance       |     the proposed      |     achievable       |
|     concepts or using |     operational and   |     within the       |
|     the proposed      |     maintenance       |     proposed         |
|     design approach.  |     concepts or using |     operational and  |
|                       |     the proposed      |     maintenance      |
| 3)  Human interface   |     design approach.  |     concepts and     |
|     issues and risk   |                       |     using the        |
|     mitigation        | 3)  Human interface   |     proposed design  |
|     strategies are    |     issues and risk   |     approach.        |
|     not adequately    |     mitigation        |                      |
|     supported by      |     strategies are    | 3)  Human interface  |
|     research,         |     partially         |     issues and risk  |
|     funding,          |     supported by      |     mitigation       |
|     technical         |     research,         |     strategies are   |
|     expertise, or     |     funding,          |     adequately       |
|     other resources.  |     technical         |     supported by     |
|                       |     expertise, or     |     research,        |
| 4)  Proposed          |     other resources.  |     funding,         |
|     automation lacks  |                       |     technical        |
|     analyses to       | 4)  Analyses show     |     expertise, and   |
|     ensure full       |     proposed          |     other resources  |
|     functionality or  |     automation        |     needed to        |
|     information to    |     supports partial  |     complete the     |
|     support user      |     functionality and |     design within    |
|     tasks.            |     information       |     program          |
|                       |     needed to support |     constraints.     |
| 5)  User tasks and    |     user tasks.       |                      |
|     skills are not    |                       | 4)  Automation       |
|     well defined or   | 5)  User tasks and    |     provides full    |
|     do not conform to |     skills are        |     functionality to |
|     current skill     |     defined but       |     support user     |
|     levels.           |     changing user     |     decision-making. |
|                       |     roles require     |                      |
| 6)  Human-system task |     reevaluation of   | 5)  User tasks and   |
|     performance times |     skills and        |     skills are well  |
|     are unknown or    |     training.         |     defined or       |
|     not quantified.   |                       |     remain           |
|                       | 6)  Human-system task |     essentially      |
| 7)  Potential for     |     performance times |     unchanged.       |
|     human error has   |     are partially     |                      |
|     not been          |     known or          | 6)  Human-system     |
|     quantitatively    |     partially         |     task performance |
|     analyzed or the   |     quantified.       |     times are known  |
|     impact on         |                       |     and acceptable.  |
|     human-in-the-loop | 7)  Potential for     |                      |
|     system            |     human error has   | 7)  Potential for    |
|     capabilities is   |     been partially    |     human error has  |
|     unknown or        |     analyzed or       |     been             |
|     changing.         |     impact on         |     quantitatively   |
|                       |     human-in-the-loop |     analyzed and     |
| 8)  Physical or       |     system            |     impact on        |
|     cognitive         |     capabilities is   |                      |
|     human-system      |     partially known.  |    human-in-the-loop |
|     integration       |                       |     system           |
|     design elements   | 8)  Physical or       |     capabilities is  |
|     are, individually |     cognitive         |     known.           |
|     or in the         |     human-system      |                      |
|     aggregate,        |     integration       | 8)  Physical or      |
|     unknown or        |     design elements   |     cognitive        |
|     sufficiently      |     are, individually |     human-system     |
|     deficient to      |     or taken          |     integration      |
|     detract from      |     together,         |     design elements  |
|     efficient or      |     partially known.  |     are individually |
|     effective task    |                       |     and taken        |
|     performance.      | 9)  Some elements of  |     together         |
|                       |     the integration   |     sufficiently     |
| 9)  Requirements for  |     of the system or  |     mature to assure |
|     integration of    |     its components    |     efficient or     |
|     the system or its |     into the user     |     effective task   |
|     components into   |     work environment  |     performance.     |
|     the user work     |     are new or        |                      |
|     environment are   |     changing.         | 9)  Integration of   |
|     undetermined or   |                       |     the system or    |
|     changing.         | 10) User groups       |     its components   |
|                       |     partially         |     into the user    |
| 10) User groups do    |     contribute to     |     work environment |
|     not contribute to |     requirements      |     is fully         |
|     requirements      |     development,      |     compatible with  |
|     development,      |     analysis, and     |     the larger       |
|     design, or        |     design.           |     system and       |
|     analysis.         |                       |     operations.      |
|                       |                       |                      |
|                       |                       | 10) User group input |
|                       |                       |     is an integral   |
|                       |                       |     part of          |
|                       |                       |     requirements     |
|                       |                       |     development,     |
|                       |                       |     design, and      |
|                       |                       |     analysis.        |
+-----------------------+-----------------------+----------------------+

Table 3 provides criteria for estimating the severity of an adverse
event relating to human factors.

### Table 3: Estimating the Severity of an Adverse Event Relating to Human Factors

+------------------------+-----------------------+---------------------+
| **Substantial Severity | **Moderate Severity   | **Minor Severity of |
| of Impact**            | of Impact**           | Impact**            |
+------------------------+-----------------------+---------------------+
| If one or more of the  | If one or more of the | If all of the       |
| following conditions   | following conditions  | following           |
| are present:           | are present:          | conditions are      |
|                        |                       | present:            |
| 1)  Size of the        | 1)  Size of the       |                     |
|     workforce affected |     workforce         | 1)  Workload        |
|     by system changes  |     affected by       |     analyses assure |
|     is large and       |     system changes is |     that staffing   |
|     staffing levels    |     small and         |     levels support  |
|     and system         |     staffing levels   |     system          |
|     performance goals  |     and system        |     performance     |
|     are not supported  |     performance goals |     goals.          |
|     by workload        |     are partially     |                     |
|     analyses.          |     supported by      | 2)  Analyses        |
|                        |     workload analyses |     indicate        |
| 2)  Analyses indicate  |     or by current     |     personnel skill |
|     personnel skill    |     staffing.         |     and ability     |
|     and ability        |                       |     requirements    |
|     requirements are   | 2)  Analyses indicate |     are met by      |
|     changing or unmet  |     personnel skill   |     current         |
|     by current         |     and ability       |     workforce.      |
|     workforce.         |     requirements are  |                     |
|                        |     partially met by  | 3)  Early training  |
| 3)  Early training     |     current           |     analyses        |
|     analyses are       |     workforce.        |     influenced      |
|     lacking or fail to |                       |     alternative     |
|     influence          | 3)  Early training    |     analysis and    |
|     selection of       |     analyses          |     design to       |
|     design             |     partially         |     ensure ease in  |
|     alternatives for   |     identify factors  |     performing all  |
|     critical tasks     |     affecting design  |     critical tasks. |
|     such as problem    |     alternatives for  |                     |
|     solving and        |     critical tasks.   | 4)  A               |
|     decision-making.   |                       |     human-centered  |
|                        | 4)  Physical and      |     design approach |
| 4)  Physical and       |     cognitive         |     has been used   |
|     cognitive          |     human-system      |     to design the   |
|     human-system       |     integration       |     physical and    |
|     integration design |     design elements   |     cognitive       |
|     elements and       |     and integration   |     human-system    |
|     integration of the |     of the system and |     integration     |
|     system and its     |     its components    |     elements and    |
|     components into    |     into the user     |     the integration |
|     the user work      |     work environment  |     of the system   |
|     environment have   |     have been         |     and its         |
|     not been fully     |     partially         |     components into |
|     analyzed or do not |     analyzed or       |     the user work   |
|     comply with human  |     partially comply  |     environment.    |
|     factors            |     with human        |                     |
|     engineering best   |     factors           | 5)  System changes  |
|     practices.         |     engineering best  |     affect no       |
|                        |     practices.        |     safety critical |
| 5)  System changes     |                       |     components and  |
|     affect safety      | 5)  System changes    |     analyses have   |
|     critical           |     affect minor      |     proven system   |
|     components and     |     safety components |     safety and      |
|     analyses have not  |     or analyses show  |     workforce       |
|     yet proven system  |     limited impact on |     health are      |
|     safety and         |     system safety and |     assured.        |
|     workforce health   |     workforce health. |                     |
|     are assured.       |                       |                     |
+------------------------+-----------------------+---------------------+

> 4.3 Costs: Efforts related to identifying system/program costs must
> include the estimated costs for mitigating the human factors risks and
> the costs for providing the necessary human-system performance
> enhancements. There are various ways to estimate human factors costs
> including those addressed in "Human Factors Program Cost Estimation -
> Potential Approaches," prepared by Dr, Parimal Kopardekar, March 23,
> 2002, at the FAA Human Factors website (https://hf.faa.gov/). An
> abbreviated method for estimating the human factors costs may be
> employed such as that described in Attachment 3. The estimated human
> factors cost provides an input to the total program cost estimates.
> Costs attributable to the human factors effort may include those
> sources listed at Attachment 4. In order to be integrated into the
> total program costs and the acquisition program baseline (APB), the
> human factors costs are distributed into the categories of the
> appropriate acquisition phase or work breakdown structure (see
> Attachment 5).

**5. Summary:** The steps involved in the Human Factors Assessment (HFA)
for Investment Analysis are summarized in Attachment 6. A template for
preparing the Human Factors Assessment is at Attachment 7. The HFA
provides essential input to IA products. These inputs may be in the form
of "a comparative HFA" by providing relative evaluations for the
different alternatives being considered or a "detailed HFA" for one or
more selected alternatives. In either case, the HFAs consist of
activities that:

-   Integrate human performance considerations into IA products and
    processes

-   Include but are not limited to benefits, risk, and cost assessments

-   Provide benefit, risk, and cost information that are combined with
    other IA and program products

**6. Point of Contact:** For additional information, contact ATO-P Human
Factors Research and Engineering (Glen Hewitt, 202-267-7163).

**Attachment 1**

**Human Factors Risks Areas**

Risk areas that may need to be assessed include those listed and
described generally in the table below. The table provides a convenient
way to indicate the risk rating (e.g. High, Medium, or Low), the impact
on cost (either a dollar value or an impact rating), and the associated
mitigation strategy. Note: A rating for one human factors risk area is
generally considered sufficient to rate the overall area of human
factors at that level of risk. Also, because the accumulation of several
low risks may result in a higher probability of an adverse event, the
cumulative impact of low several risk areas should be assessed. For the
purposes of establishing a consistent approach to accumulated risks,
generally six or more risks in a lower level may be considered
sufficient to raise the overall human factors risk rating to the next
higher level.

Table 1.1 Human Factors Risk Areas

+----------------+---------------------+------+------+----------------+
| Risk Areas     | General Description | Risk | Cost | Mitigation     |
|                |                     |      |      |                |
|                |                     | Ra   | Ra   |                |
|                |                     | ting | ting |                |
+----------------+---------------------+------+------+----------------+
| Allocation of  | Assigning those     |      |      |                |
| Function       | ro                  |      |      |                |
|                | les/functions/tasks |      |      |                |
|                | for which the human |      |      |                |
|                | or equipment        |      |      |                |
|                | performs better     |      |      |                |
|                | while enabling the  |      |      |                |
|                | human to maintain   |      |      |                |
|                | awareness of the    |      |      |                |
|                | operational         |      |      |                |
|                | situation.          |      |      |                |
+----------------+---------------------+------+------+----------------+
| **A            | Accommodating the   |      |      |                |
| nthropometrics | physical attributes |      |      |                |
| and            | of its user         |      |      |                |
| Biomechanics** | population (e.g.,   |      |      |                |
|                | from the 1st        |      |      |                |
|                | through 99th        |      |      |                |
|                | percentile levels). |      |      |                |
+----------------+---------------------+------+------+----------------+
| **CHI          | Employing effective |      |      |                |
| (              | and consistent user |      |      |                |
| Computer-Human | dialogues,          |      |      |                |
| Interaction)** | interfaces, and     |      |      |                |
|                | procedures across   |      |      |                |
|                | system functions.   |      |      |                |
+----------------+---------------------+------+------+----------------+
| **             | Applying system     |      |      |                |
| Communications | design              |      |      |                |
| and Teamwork** | considerations to   |      |      |                |
|                | enhance required    |      |      |                |
|                | user communications |      |      |                |
|                | and teamwork.       |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Culture of   | Addressing the      |      |      |                |
| Personnel**    | organizational and  |      |      |                |
|                | sociological        |      |      |                |
|                | environment into    |      |      |                |
|                | which any change,   |      |      |                |
|                | including new       |      |      |                |
|                | technologies and    |      |      |                |
|                | procedures, will be |      |      |                |
|                | introduced.         |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Displays and | Designing and       |      |      |                |
| Controls**     | arranging displays  |      |      |                |
|                | and controls to be  |      |      |                |
|                | consistent with the |      |      |                |
|                | operator's and      |      |      |                |
|                | maintainer's tasks  |      |      |                |
|                | and actions.        |      |      |                |
+----------------+---------------------+------+------+----------------+
| **D            | Preparing user      |      |      |                |
| ocumentation** | documentation and   |      |      |                |
|                | technical manuals   |      |      |                |
|                | in a suitable       |      |      |                |
|                | format of           |      |      |                |
|                | information         |      |      |                |
|                | presentation, at    |      |      |                |
|                | the appropriate     |      |      |                |
|                | reading level, and  |      |      |                |
|                | with the required   |      |      |                |
|                | degree of technical |      |      |                |
|                | sophistication and  |      |      |                |
|                | clarity.            |      |      |                |
+----------------+---------------------+------+------+----------------+
| *              | Accommodating       |      |      |                |
| *Environment** | environmental       |      |      |                |
|                | factors (including  |      |      |                |
|                | extremes) to which  |      |      |                |
|                | the system will be  |      |      |                |
|                | subjected and       |      |      |                |
|                | understanding the   |      |      |                |
|                | associated effects  |      |      |                |
|                | on human-system     |      |      |                |
|                | performance.        |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Functional   | Applying            |      |      |                |
| Design**       | human-centered      |      |      |                |
|                | design for          |      |      |                |
|                | usability and       |      |      |                |
|                | compatibility with  |      |      |                |
|                | operational and     |      |      |                |
|                | maintenance         |      |      |                |
|                | concepts.           |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Human        | Examining design    |      |      |                |
| Error**        | and contextual      |      |      |                |
|                | conditions          |      |      |                |
|                | (including          |      |      |                |
|                | supervisory and     |      |      |                |
|                | organizational      |      |      |                |
|                | influences) as      |      |      |                |
|                | causal factors      |      |      |                |
|                | contributing to     |      |      |                |
|                | human error, and    |      |      |                |
|                | consideration of    |      |      |                |
|                | objectives for      |      |      |                |
|                | error tolerance,    |      |      |                |
|                | error prevention,   |      |      |                |
|                | and error           |      |      |                |
|                | c                   |      |      |                |
|                | orrection/recovery. |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Information  | Enhancing operator  |      |      |                |
| Presentation** | and maintainer      |      |      |                |
|                | performance through |      |      |                |
|                | the use of          |      |      |                |
|                | effective and       |      |      |                |
|                | consistent labels,  |      |      |                |
|                | symbols, colors,    |      |      |                |
|                | terms, acronyms,    |      |      |                |
|                | abbreviations,      |      |      |                |
|                | formats, and data   |      |      |                |
|                | fields.             |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Information  | Ensuring the        |      |      |                |
| Requirements** | availability and    |      |      |                |
|                | usability of        |      |      |                |
|                | information needed  |      |      |                |
|                | by the operator and |      |      |                |
|                | maintainer for a    |      |      |                |
|                | specific task when  |      |      |                |
|                | it is needed, and   |      |      |                |
|                | in a form that is   |      |      |                |
|                | directly usable.    |      |      |                |
+----------------+---------------------+------+------+----------------+
| **I/O          | Selecting input and |      |      |                |
| Devices**      | output (I/O)        |      |      |                |
|                | methods and devices |      |      |                |
|                | that allow          |      |      |                |
|                | operators or        |      |      |                |
|                | maintainers to      |      |      |                |
|                | perform tasks,      |      |      |                |
|                | especially critical |      |      |                |
|                | tasks, quickly and  |      |      |                |
|                | accurately.         |      |      |                |
+----------------+---------------------+------+------+----------------+
| **KSAs**       | Measuring the       |      |      |                |
|                | knowledge, skills,  |      |      |                |
|                | and abilities       |      |      |                |
|                | (KSAs) required to  |      |      |                |
|                | perform job-related |      |      |                |
|                | tasks, and          |      |      |                |
|                | determining         |      |      |                |
|                | appropriate         |      |      |                |
|                | selection           |      |      |                |
|                | requirements for    |      |      |                |
|                | users.              |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Operational  | Ensuring that the   |      |      |                |
| Suitability**  | system              |      |      |                |
|                | appropriately       |      |      |                |
|                | supports the user   |      |      |                |
|                | in performing       |      |      |                |
|                | intended functions  |      |      |                |
|                | while maintaining   |      |      |                |
|                | interoperability    |      |      |                |
|                | and consistency     |      |      |                |
|                | with other system   |      |      |                |
|                | elements or support |      |      |                |
|                | systems.            |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Procedures** | Designing operation |      |      |                |
|                | and maintenance     |      |      |                |
|                | procedures for      |      |      |                |
|                | simplicity,         |      |      |                |
|                | consistency, and    |      |      |                |
|                | ease of use.        |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Safety and   | Preventing/reducing |      |      |                |
| Health**       | operator and        |      |      |                |
|                | maintainer exposure |      |      |                |
|                | to safety and       |      |      |                |
|                | health hazards.     |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Situational  | Enabling operators  |      |      |                |
| Awareness**    | or maintainers to   |      |      |                |
|                | perceive and        |      |      |                |
|                | understand elements |      |      |                |
|                | of the current      |      |      |                |
|                | situation, and      |      |      |                |
|                | project them to     |      |      |                |
|                | future operational  |      |      |                |
|                | situations.         |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Special      | Minimizing the need |      |      |                |
| Skills and     | for special or      |      |      |                |
| Tools**        | unique operator or  |      |      |                |
|                | maintainer skills,  |      |      |                |
|                | abilities, tools,   |      |      |                |
|                | or characteristics. |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Staffing**   | Accommodating       |      |      |                |
|                | constraints and     |      |      |                |
|                | efficiencies for    |      |      |                |
|                | staffing levels and |      |      |                |
|                | organizational      |      |      |                |
|                | structures.         |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Training**   | Applying methods to |      |      |                |
|                | enhance operator or |      |      |                |
|                | maintainer          |      |      |                |
|                | acquisition of the  |      |      |                |
|                | knowledge and       |      |      |                |
|                | skills needed to    |      |      |                |
|                | interface with the  |      |      |                |
|                | system, and         |      |      |                |
|                | designing that      |      |      |                |
|                | system so that      |      |      |                |
|                | these skills are    |      |      |                |
|                | easily learned and  |      |      |                |
|                | retained.           |      |      |                |
+----------------+---------------------+------+------+----------------+
| **V            | Designing visual    |      |      |                |
| isual/Auditory | and auditory alerts |      |      |                |
| Alerts**       | (including error    |      |      |                |
|                | messages) to invoke |      |      |                |
|                | the necessary       |      |      |                |
|                | operator and        |      |      |                |
|                | maintainer          |      |      |                |
|                | response.           |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Workload**   | Assessing the net   |      |      |                |
|                | demands or impacts  |      |      |                |
|                | upon the physical,  |      |      |                |
|                | cognitive, and      |      |      |                |
|                | decision-making     |      |      |                |
|                | resources of an     |      |      |                |
|                | operator or         |      |      |                |
|                | maintainer using    |      |      |                |
|                | objective and       |      |      |                |
|                | subjective          |      |      |                |
|                | performance         |      |      |                |
|                | measures.           |      |      |                |
+----------------+---------------------+------+------+----------------+
| **Work Space** | Designing adequate  |      |      |                |
|                | work space for      |      |      |                |
|                | personnel and their |      |      |                |
|                | tools or equipment, |      |      |                |
|                | and providing       |      |      |                |
|                | sufficient space    |      |      |                |
|                | for the movements   |      |      |                |
|                | and actions that    |      |      |                |
|                | personnel perform   |      |      |                |
|                | during operational  |      |      |                |
|                | and maintenance     |      |      |                |
|                | tasks under normal, |      |      |                |
|                | adverse, and        |      |      |                |
|                | emergency           |      |      |                |
|                | conditions.         |      |      |                |
+----------------+---------------------+------+------+----------------+

**Attachment 2**

**An Assessment Criteria for Human Factors Risks**

A set of assessment criteria for estimating risk for each of the 24
human factors issues (function allocation, work load, situational
awareness, staffing, etc.) is below.^1^ Others may be used including
those that estimate severity and probability of adverse events.

  --------------- ----------------- ------------------ --------------------
  **CRITERION**   **RISK LEVEL**                       

  **For each      **Low**           **Medium**         **High**
  human factors                                        
  issue:**                                             

  Clarity and     High (good        Medium             Low (not clear or
  completeness of clarity and                          complete)
  program         completeness)                        
  objectives,                                          
  requirements                                         
  and constraints                                      

  Availability of High (the issue   Medium (the issue  Low (the issue has
  human factors   has been          has been partially not been evaluated
  data (human     evaluated)        evaluated)         or it is unknown
  performance,                                         what data is
  capabilities,                                        available)
  user                                                 
  acceptance,                                          
  etc.)                                                

  Level of change Low (very few     Medium (some       High (many changes
  required to     changes and/or    changes and/or     and/or changes with
  resolve the     changes with      changes with       significant impact
  issue (user     negligible impact moderate impact    will be required to
  interface,      will be required  will be required   meet human factors
  procedures,     to meet human     to meet human      requirements, or,
  training,       factors           factors            the level of change
  standards,      requirements)     requirements)      required is unknown)
  policy)                                              

  Complexity of   Low (few          Medium             High (many
  change required complications to                     complications to
  to resolve the  changes)                             changes)
  issue                                                

  Program         High (obvious     Medium             Low (disparate views
  commitment to   dedication to                        or irreconcilable
  resolving the   resolution)                          differences)
  issue                                                

  Clarity of plan High              Medium             Low (no plan or many
  to resolve the  (comprehensive,                      deficiencies in
  issue           clear plan)                          plan)

  Safety risk     Low (safety risk  Medium             High (the safety
  associated with is known)                            risk is high or
  the issue                                            unknown)
  --------------- ----------------- ------------------ --------------------

^1^Based upon writings by Jay Martin

**Attachment 3**

**Abbreviated Human Factors Cost Estimation Method**

**General:** The cost of conducting human factors engineering support
has been estimated to be between 0.5% and 6% of the program's
developmental costs (depending upon many factors). Cost estimating
methods (such as those identified in the concept paper "Human Factors
Program Cost Estimation - Potential Approaches," prepared by Dr, Parimal
Kopardekar, March 23, 2002, at the FAA Human Factors website focus on
detailed elements that affect human factors costs. The concept paper
also enumerates some overarching macroscopic cost drivers. At the early
stages of a program, macroscopic factors that may be used to estimate
the human factors costs as a percentage of the program's developmental
cost include:

1.  *Definition of and Agreement on System Requirements* -- The
    specificity and clarity of the human-system interface requirements,
    operational and maintenance concepts, concepts of use, expected task
    performance levels, and procedural guidelines determine the amount
    of uncertainty and risk in meeting system performance objectives and
    is a key factor affecting cost.

2.  *The complexity of the human-system integration* -- The complexity
    of the integration between the operator/maintainer and the system
    (including the human-system interface such as that reflected in the
    display design) increases the developmental and evaluations costs.

3.  *Organizational culture and nature of relationships among
    management, user, and provider unions, industry, and other
    stakeholders (e.g., interests converge or negotiations are
    necessary)* \-- The climate of an organization plays a role in
    determining how easily the new changes will be implemented. Some
    changes are easier to implement than others due to their perceived
    or actual acceptability. The changes that face resistance become
    costly since part of the cost goes towards ensuring that the
    resistance is managed. Often, a concept or technology offers
    differing benefits (or losses) to different stakeholders even
    though, on the average, they are beneficial to the NAS. Under such
    circumstances, time and cost need to be devoted to gain mutual
    consensus. Such processes increase the cost.

4.  *Pace of program (e.g., aggressive, normal, slow)* -- As the program
    schedule becomes more aggressive, more resources are needed and the
    cost increases in a shorter period.

5.  *Safety and security considerations (e.g., higher security, or
    normal security)* -- As the safety and security requirements for
    equipment, technology, procedures, or decision support tools
    increase, more developmental activities and evaluations need to be
    conducted to assure the safety standards are met. This leads to
    higher cost.

6.  *Collaboration with international, external, or domestic
    organizations for standardization and other reasons* -- Early
    collaboration with international and domestic partners increases the
    likelihood of ensuring that all requirements are taken into
    consideration. However, increased collaboration increases the cost
    and/or schedule of a program due to costs associated with increased
    deliberation and consensus on meeting broader requirements.

**Cost Estimates:** Derived from a survey of human factors
professionals, Table 2.1 (below) provides attributes affecting program
human factors costs, their relative weighting, attribute descriptors,
and cost values to be used as additive elements of the total program
human factors development costs. Especially early in the acquisition,
these may be used (and modified as appropriate) to estimate the human
factors program cost. For example, if a program entails "normal" or
"moderate" levels of all cost factors, the total human factors cost may
be estimated at approximately 3% (from .75+.6+.6+.45+.3+.3) of the
program developmental costs. Or, if the program entails "normal" or
"moderate" levels of all cost factors except a "very high" human-system
integration/complexity," the total human factors costs may be estimated
at approximately 4.2% (from .75+1.2+.6+.45+.3+.3) of the program
developmental costs.

**Risk-Based Cost Estimates:** A "risk" or "confidence" level should be
associated with the estimated human factors cost. Notionally, the cost
should be estimated at the 80 or 90% confidence level to assure limited
risk to the program. This risk-based cost estimate may be derived from
the "most likely" or "best" estimate of the human factors costs, or it
may be directly estimated by identifying the set of activities (and
their costs) required to address 80 or 90% of the risk (or by reducing
the "unmitigated" risks to 20 or 10%). Depending upon the type of
acquisition, the distribution of human factors cost risk may be best
estimated by a lognormal distribution. In the absence of additional
risk-based cost information and for the purpose of relating estimated
cost and risk level, the following relationship is provided:

Figure 2.1: Distribution of estimated human factors costs

Note: This figure shows several estimated human factors cost
relationships, including:

1\. Confidence Levels for Cost Estimates:

\- High level of confidence = 90% of the cost risk is identified by the
cost estimate

\- Low level of confidence = 10% of the cost risk is identified by the
cost estimate

\- Most Likely = Best estimate of the cost

2\. Confidence Level Differences: The difference between the High
confidence levels and Most Likely ("X") is twice the difference between
Most Likely and Low confidence levels ("X/2").

3\. Confidence Level Values: High confidence estimates are 140% of Most
Likely estimates; and Low confidence estimates are 20% below Most Likely
estimates.

> Table 2.1: Program Attributes Affecting Human Factors Costs

<table style="width:100%;">
<colgroup>
<col style="width: 27%" />
<col style="width: 0%" />
<col style="width: 12%" />
<col style="width: 0%" />
<col style="width: 10%" />
<col style="width: 0%" />
<col style="width: 11%" />
<col style="width: 0%" />
<col style="width: 10%" />
<col style="width: 0%" />
<col style="width: 12%" />
<col style="width: 0%" />
<col style="width: 12%" />
<col style="width: 0%" />
</colgroup>
<thead>
<tr class="header">
<th colspan="2"><p><strong>Program Attributes</strong></p>
<p><strong>(RelativeWeight: % of HF Cost)</strong></p></th>
<th colspan="2"></th>
<th colspan="10"><p><strong>Attribute Descriptors and</strong></p>
<p><strong>Cost Values (as a percentage of program developmental
cost)</strong></p></th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td colspan="2"><blockquote>
<p>Definition or Agreement on Human-System Interface Requirements</p>
</blockquote></td>
<td colspan="2">Descriptors</td>
<td colspan="2">Highly Defined and Resolved</td>
<td colspan="2"><p>Moderately</p>
<p>Defined and Resolved</p></td>
<td colspan="2"><p>Somewhat</p>
<p>Defined or Resolved</p></td>
<td colspan="2"><p>Moderately</p>
<p>Undefined or Unresolved</p></td>
<td colspan="2">Highly Undefined or Unresolved</td>
</tr>
<tr class="even">
<td colspan="2"><blockquote>
<p>(25%)</p>
</blockquote></td>
<td colspan="2">Cost %</td>
<td colspan="2">0.13</td>
<td colspan="2">0.37</td>
<td colspan="2">0.75</td>
<td colspan="2">1.0</td>
<td colspan="2">1.5</td>
</tr>
<tr class="odd">
<td colspan="2"><blockquote>
<p>Human-System Integration Complexity</p>
</blockquote></td>
<td colspan="2">Descriptors</td>
<td colspan="2">Very Low</td>
<td colspan="2">Low</td>
<td colspan="2">Moderate</td>
<td colspan="2">High</td>
<td colspan="2">Very High</td>
</tr>
<tr class="even">
<td colspan="2">(20%)</td>
<td colspan="2">Cost %</td>
<td colspan="2">0.1</td>
<td colspan="2">0.3</td>
<td colspan="2">0.6</td>
<td colspan="2">0.8</td>
<td colspan="2">1.2</td>
</tr>
<tr class="odd">
<td colspan="2"><blockquote>
<p>Organizational Culture, Stakeholder Interests</p>
</blockquote></td>
<td colspan="2">Descriptors</td>
<td colspan="2">Very Conducive</td>
<td colspan="2">Conducive</td>
<td colspan="2">Moderate</td>
<td colspan="2">Resistant</td>
<td colspan="2">Very Resistant</td>
</tr>
<tr class="even">
<td colspan="2"><blockquote>
<p>(20%)</p>
</blockquote></td>
<td colspan="2">Cost %</td>
<td colspan="2">0.1</td>
<td colspan="2">0.3</td>
<td colspan="2">0.6</td>
<td colspan="2">0.8</td>
<td colspan="2">1.2</td>
</tr>
<tr class="odd">
<td colspan="2"><blockquote>
<p>Program Pace</p>
</blockquote></td>
<td colspan="2">Descriptors</td>
<td colspan="2">Very Slow</td>
<td colspan="2">Slow</td>
<td colspan="2">Moderately Paced</td>
<td colspan="2">Aggressive</td>
<td colspan="2">Very Aggressive</td>
</tr>
<tr class="even">
<td><blockquote>
<p>(15%)</p>
</blockquote></td>
<td colspan="2">Cost %</td>
<td colspan="2">0.07</td>
<td colspan="2">0.23</td>
<td colspan="2">0.45</td>
<td colspan="2">0.6</td>
<td colspan="2">0.9</td>
<td></td>
</tr>
<tr class="odd">
<td colspan="2"><blockquote>
<p>Safety Considerations</p>
</blockquote></td>
<td colspan="2">Descriptors</td>
<td colspan="2"><p>Very</p>
<p>Low</p></td>
<td colspan="2"><p>Somewhat</p>
<p>Low</p></td>
<td colspan="2">Moderate</td>
<td colspan="2"><p>Somewhat</p>
<p>High</p></td>
<td colspan="2">Very High</td>
</tr>
<tr class="even">
<td><blockquote>
<p>(10%)</p>
</blockquote></td>
<td colspan="2">Cost %</td>
<td colspan="2">0.05</td>
<td colspan="2">0.15</td>
<td colspan="2">0.3</td>
<td colspan="2">0.4</td>
<td colspan="2">0.6</td>
<td></td>
</tr>
<tr class="odd">
<td colspan="2"><blockquote>
<p>External/</p>
<p>International Collaboration Needs</p>
</blockquote></td>
<td colspan="2">Descriptors</td>
<td colspan="2">Very Low</td>
<td colspan="2">Low</td>
<td colspan="2">Moderate</td>
<td colspan="2">High</td>
<td colspan="2">Very High</td>
</tr>
<tr class="even">
<td><blockquote>
<p>(10%)</p>
</blockquote></td>
<td colspan="2">Cost %</td>
<td colspan="2">0.05</td>
<td colspan="2">0.15</td>
<td colspan="2">0.3</td>
<td colspan="2">0.4</td>
<td colspan="2">0.6</td>
<td></td>
</tr>
<tr class="odd">
<td colspan="2"><strong>TOTAL</strong></td>
<td colspan="2"></td>
<td colspan="2"><strong>.5%</strong></td>
<td colspan="2"><strong>1.5%</strong></td>
<td colspan="2"><strong>3%</strong></td>
<td colspan="2"><strong>4%</strong></td>
<td colspan="2"><strong>6%</strong></td>
</tr>
</tbody>
</table>

**Attachment 4**

**Types of Human Factors Costs**

In estimating human factors costs, one should include the various
sources of personnel, material, and incidental costs. These cost sources
include government costs (Federal and contractor support) and
vendor/developmental contractor costs. Costs may be incurred during any
of the phases of the system lifecycle and estimates should include (but
not be limited to) those associated with two general categories of
activities (adapted from Mantei and Teorey, 1988):

> [Development Activities]{.underline} [Usability
> Activities]{.underline}
>
> Feasibility study and analysis Market analysis
>
> User/target audience description Cost/benefit/risk analysis
>
> Requirements definition Product acceptance analysis
>
> Design and analysis Task analysis
>
> Prototype construction User testing and evaluation
>
> System implementation Product survey
>
> Product testing Post-implementation assessment
>
> Design update and maintenance

For the purposes of this document, the following types of costs may be
considered to fall under human factors program management costs when
addressing human performance or human-system interface issues:

1.  Design engineering and analysis cost

2.  Software personnel programming cost

3.  Human factors staff cost

4.  Laboratory/facility cost

5.  Study participant cost

6.  Subject matter expert cost

7.  User needs assessment cost

8.  Concept studies cost

9.  Prototype and usability assessment costs

10. Modeling and fast-time simulation cost

11. Human-in-the-loop simulation cost

12. Experiment/study plan development cost

13. Scenario development cost

14. Scenario shakedown cost

15. Final simulation cost

16. Data collection cost

17. Data analysis cost

18. Final report development cost

19. Coordination, communication, and implementation costs

\[Note: If a study, experiment, test, or other activity involves human
performance assessment, human-system interface design, or human
performance data collection or analysis, costs associated with human
factors tasks should be considered as part of human factors cost. For
example, if human factors personnel participate in maintainability
testing to gather maintainer performance data, the cost of this
participation is allocated to human factors, but not the total cost of
the maintainability testing.\]

**Attachment 5**

**Human Factors Cost Distribution**

Human factors costs may need to be estimated for the various phases of a
program or for various activities within a phase. The two tables below
provide a (notional) relative distribution of human factors costs:

**1. For Acquisition Phases:** Using the FAA Acquisition Lifecycle
Management Policy phases, the human factors costs are parsed across
phases. The relative human factors cost distribution may be adjusted as
appropriate for the particular acquisition, the program's specific human
factors risk, and the mitigation strategies selected. These costs are
then to be integrated with other program cost estimates to support the
Acquisition Program Baseline and program planning.

**Table 4.1 Human Factors Costs by Acquisition Phase/WBS Level 1**

  ------------------------------------------------- ---------------------
  **Program Cost Category**                         **Relative Human
                                                    Factors Cost
                                                    Distribution**

  Mission Analysis                                  5

  Requirements Determination and Investment         20
  Analysis                                          

  Solution Development and Implementation           60

  In-service Management                             10

  Disposition and Service Life Extension            5

  TOTAL                                             100%
  ------------------------------------------------- ---------------------

**2. For Work Breakdown Structure (WBS) Level 2:** Using the FAA Work
Breakdown Structure (especially element 3.0, Solution Development) the
human factors costs are parsed into the program element cost categories
below. The relative human factors cost distribution may be adjusted as
appropriate for the particular acquisition, the program's specific human
factors risk, and the mitigation strategies selected. These costs are
then to be integrated with other program cost estimates to support the
Acquisition Program Baseline and program planning.

**Table 4.2 Human Factors Costs by WBS Solution Development Category**

  ------------------------------------------------- ---------------------
  **Program Cost Category**                         **Relative Human
                                                    Factors Cost
                                                    Distribution**

  3.1 Program Management                            10%

  3.2 System Engineering                            15%

  3.3 HW/SW Design, Development, Procurement, and   50%
  Production                                        

  3.4 Facilities and Physical Infrastructure Design 5%
  and Development                                   

  3.5 Test and Evaluation                           10%

  3.6 Documentation                                 5%

  3.7 Logistics Support                             5%

  TOTAL                                             100%
  ------------------------------------------------- ---------------------

**Attachment 6**

**Summarized Steps for Human Factors Assessments in Investment
Analysis**

**STEP 1: Conduct Benefits Analysis to Determine:**

a)  Human-in-the-loop system performance limitations

b)  Human-in-the-loop system performance enhancements

**STEP 2: Conduct Risk Analysis to Determine the:**

a)  Human factors risks

b)  Probability/severity ratings for human factors risks

c)  Human factors mitigation strategies/activities

**STEP 3: Conduct Cost Analysis to:**

a)  Estimate human factors mitigation costs

b)  Estimate costs associated with human factors benefits/enhancements

c)  Aggregate human factors mitigation and enhancement costs

d)  Calculate risk-based human factors costs at 80 or 90% level

e)  Distribute 80 or 90% risk-based human factors cost among WBS
    elements

**STEP 4: Integrate the Results with Other Investment Analysis and
Program Products**

**Attachment 7**

**Template for Human Factors Assessments in Investment Analysis**

Revise as necessary:

EXECUTIVE SUMMARY

INTRODUCTION

BACKGROUND / SCOPE OF EFFORT

USERS (OPERATORS AND MAINTAINERS) AND THEIR TASKS

HF ASSESSMENT APPROACH

SOURCE INFORMATION CONSIDERED

SYSTEM CRITICAL CHARACTERISTICS AND FEATURES

APPLICABLE HF GUIDANCE / STANDARDS

DIMENSIONS OF HUMAN FACTORS SUPPORT (HUMAN FACTORS ISSUE AREAS)

Estimates of Probability, Severity, and Overall HF Risk for 24 HF
Dimensions

DISCUSSION OF OVERALL HF RISK ESTIMATES

Recommended Risk Mitigation Strategies

HF ASSESSMENT: SUMMARY AND CONCLUSIONS