**Macro Evaluation Model for Research Programs on Human Fatigue**

**SUMMARY REPORT**

**September 2005**

# 

#### TABLE OF CONTENTS

[**executive summary 1**](#executive-summary)

[I. PURPOSE and BACKGROUND 2](#__RefHeading___Toc151365970)

[II. Overview.................. 2](#ii.-overview)

[III. INTRODUCTION TO LOGIC MODELS
3](#iii.-introduction-to-logic-models)

[IV. project products 5](#iv.-project-products)

[APPENDIX I: Macro Fatigue Management Rail Model
7](#appendix-i-macro-fatigue-management-rail-model)

[APPENDIX II: Detailed Fatigue Management Rail Model
8](#appendix-ii-detailed-fatigue-management-rail-model)

[APPENDIX III: Coordination Plan: Fatigue Management Activities of the
Office of Research and Development and the Office of Safety
12](#appendix-iii-coordination-plan-fatigue-management-activities-of-the-office-of-research-and-development-and-the-office-of-safety)

[APPENDIX IV: Cross-Modal Fatigue Management Model
17](#appendix-iv-cross-modal-fatigue-management-model)

[APPENDIX V: User Guide for Cross-Modal Model
23](#appendix-v-user-guide-for-cross-modal-model)

[APPENDIX VI: Glossary for Cross-Modal Model
31](#appendix-vi-glossary-for-cross-modal-model)

#  

# executive summary 

**Purpose and Background**

Two offices, the Office of Research and Development (R&D) and the Office
of Safety (OS) at the Federal Railroad Administration (FRA) of the
United States Department of Transportation (DOT), are both involved in
fatigue management (FM) activities and evaluation. The long-term goal of
the project was to contribute to the development of an integrated
planning and evaluation strategy including a coordination plan for the
fatigue management activities of the two offices. A working group
comprised of staff from the two offices and from the U.S. DOT Volpe
National Transportation Center (Volpe) contributed to planning and
reviewing the various activities and products of the project. Concurrent
with this project, the Project Director was also working on a related
cross-modal FM project for DOT Human Factors Coordinating Committee
(HFCC). Senior managers from the air, marine, motor carrier, and rail
modes engaged in a process similar to the one used here for the FRA.
Thus, selected relevant products from the HFCC project are also
addressed in this report.

**Methodology**

In order to achieve their common and respective goals and carry out the
mission of the FRA, the Office of R&D and OS engaged in a participatory
process to identify: 1) priority *FM issues*; 2) *activities* to address
those issues; 3) intended *outcomes*; and 4) *coordination* processes.
The research team interviewed selected headquarters staff from R&D and
OS, and OS field staff. The results of these interviews were summarized
in standard text reports and in visual graphic displays. These graphics
are in the format of a conceptual framework called *logic models*, which
are widely used for planning and evaluation. Logic models help to
identify causal linkages underlying decision making. They are now being
used by the Office of Management and Budget (OMB) as evidenced in their
new Performance Assessment Rating Tool (PART) for evaluating executive
agencies.

**Overview**

This Summary Report describes some of the major products developed for
this project. In order to increase accessibility and usage, it includes
copies of these products. The Report also describes and contains copies
of selected products from the aforementioned HFCC cross-modal related
project.

[]{#__RefHeading___Toc151365970 .anchor}**I. PURPOSE and BACKGROUND**

This project was designed to contribute to the development of an
integrated planning and evaluation strategy for the fatigue management
activities (FM) of the Office of Research and Development (R&D) and the
Office of Safety (OS) at the Federal Railroad Administration (FRA) of
the United States Department of Transportation (DOT). In order to
achieve this goal, the two offices engaged in a participatory process to
identify: 1) priority FM issues; 2) activities to address those issues;
and 3) intended outcomes. The offices also participated in identifying
how they could enhance their communication and collaboration to
coordinate their respective FM activities and evaluations.

The two offices, R&D and OS, recognized that since they were both
involved in FM activities and evaluation, they could facilitate
achieving their mutual and respective goals by greater integration of
their work. The offices were supported by the essential role played by
U.S. DOT Volpe National Transportation Center (Volpe) in planning and
implementing the project. The project received direction at the various
phases from a small working group of staff members from R&D, OS, and
Volpe. This working group worked collaboratively in designing and
reviewing the products of this project.

Concurrent with this project, the Project Director was also working on a
related cross-modal FM project for DOT Human Factors Coordinating
Committee (HFCC). This allowed for a degree of synergy and mutual
benefit for each project. For example, senior managers from the air,
marine, motor carrier, and rail modes engaged in a participatory process
similar to the one used here for the FRA, to identify crosscutting FM
issues, activities and outcomes. Selected relevant products from the
HFCC project are also addressed in this report.

# II. Overview 

This is a summary document. It contains a brief description of the
documents produced for this project, including the methodology used to
develop each product, as well as some possible and already realized
usages of the various products. The report also describes supplementary
materials developed for the aforementioned HFCC Cross-Modal project. To
facilitate access and usability, the Appendices contain the documents
described in the body of the report, which includes all those produced
for this project and selected ones for the Cross-Modal project.

In order to enhance understanding and usage of the graphics and models
of this project, this report begins with a brief introduction to logic
models. Logic models are the type of conceptual model that is the
foundation for most of the graphic models of this project.

# III. INTRODUCTION TO LOGIC MODELS

Logic models graphically depict information and are widely used in the
evaluation field. They provide conceptual frameworks that can help to
identify causal connections, the *if*...*then*...statements underlying
decision making. For example, *if* we provide fatigue management
education, *then* employees are more likely to use their off-duty time
effectively. Such models have proved so useful that the Office of
Management and Budget (OMB) has included them in their new Performance
Assessment Rating Tool (PART) for evaluating executive agency
performance.

Logic models can be used for planning, evaluation, collaborative
consensus-building, and knowledge development. They are useful tools to
clarify thinking and facilitate concerted action by multiple groups or
persons. The process of developing and using logic models brings people
together to discuss and determine common meanings and priorities.

A logic model provides a framework that organizes and condenses
information and makes information more conceptually and physically
accessible. The basic elements of a logic model are "action" and
"result" (see Figure 1). Common elements also include "need" (see Figure
2). Then each of these main categories gets specified according to the
particular purpose and users of the model (see Figure 3: Variation and
Specification of Elements).

Figure 1: Basic Elements of Logic Models

![](media/image1.wmf){width="6.416666666666667in"
height="3.6006944444444446in"}

Figure 2: Common Elements

![](media/image2.wmf){width="5.166666666666667in"
height="2.998611111111111in"}

# 

# 

Figure 3: Variation and Specification of Elements

# ![](media/image3.wmf){width="5.993055555555555in" height="3.1527777777777777in"} 

#  IV. project products 

1.  **Macro Fatigue Management Rail Model----Appendix I**

The Macro Fatigue Management Rail Model (Macro Rail Model) is a logic
model that summarizes information related to fatigue management
activities. This information was collected in interviews with FRA staff.
Selected headquarters staff at R&D and OS, and OS field staff
participated in semi-structured interviews to identify priority
*issues*, *activities* to address the issues, and intended *outcomes* of
the activities. The interviews were a step towards developing an
integrated FM strategy for Office of R&D and OS. The Macro Rail Model
could be distributed internally to R&D and OS to communicate and
prioritize issues.

2.  **Detailed Fatigue Management Rail Model----Appendix II**

The *Detailed* Fatigue Management Rail Model (Detailed Rail Model)
provides a more specified depiction of the information displayed in the
*Macro* Rail Model. It was developed by the same method as the Macro
Model and could have the same general use. However, its greater detail
allows users to get to a more specific level that may be needed to
carry-out the planning, implementation and evaluation activities that
are more generically described in the Macro Rail Model. Users will find
the model helpful at any stage in the life of activities----initial,
ongoing, and completed.

3.  **Coordination Plan: Fatigue Management Activities of the Office of
    Research and Development and the Office of Safety----Appendix III**

In addition to identifying the FM issues, activities, and results
(summarized in the rail models and report, Appendices I-II),
interviewees offered their opinions on how the two offices could improve
coordination of their FM activities. This information is graphically
displayed in the Coordination Plan: Fatigue Management Activities of the
Office of Research and Development and the Office of Safety (Appendix
III). The interviewees offered broad support for increasing
communication and collaboration between the two offices. In addition,
the working group determined that a graphic template depicting the
points of coordination would be helpful (see Coordination Plan, Figure
1, Appendix IIII). Participants also discussed how the two offices have
some common roles, some distinct primary roles, and supporting roles
(Figure 2, Appendix IIII). Figure 3, Appendix IIII graphically depicts
how coordination of these roles could take place in the example of
planning short-term and long-term projects.

4.  **Cross-Modal Fatigue Management Model----Appendix IV**

The Cross-Modal Fatigue Management Model (Cross-Model Model) was
produced for the aforementioned project funded by the Human Factors
Coordinating Committee (HFCC), *Operator Fatigue Program Evaluation
Tools and Methods* and led by the same project director as for this FRA
project. In the HFCC project, government stakeholders from each of the
four modes----air, marine, motor carrier and rail----participated in a
series of small focus groups to develop the Cross-Modal Model (Appendix
IV). The Cross-Modal Model is similar to the Detailed Rail Model
(Appendix II). The Cross-Modal Model was distributed to rail industry
stakeholders at the North American Rail Alertness Partnership meeting in
2004. Distribution and use of this model served the ultimate purpose of
this project----of developing methods and processes for industry-wide
strategic planning and evaluation. The Cross-Modal Model exhibits a
broader level of generality than the Rail model, but it also it
demonstrates the overlap and similarity of FM issues facing stakeholders
in all modes of transportation. Further, it provides an example of how
logic models vary depending on the creators and purposes.

5.  **User Guide for Cross-Modal Model----Appendix V**

This User Guide (Appendix VII) accompanies the Cross-Modal Model
(Appendix IV) and was also developed for the HFCC project. While its
focus and content targets the Cross-Modal Model, the general process it
describes---how to read and use a logic model---applies to the use of
other logic models such as the Macro and Detailed Rail Models. The
document is included here to enhance the reader's understanding of the
Cross-Modal Model and also to clarify understanding and enhance usage of
the rail models.

6.  **Glossary for Cross-Modal Model----Appendix VI**

The Glossary for Cross-Modal Model (Appendix VI) accompanies the
Cross-Modal Model (Appendix V). While its focus and content targets the
Cross-Modal Model, many of the terms apply to logic models in general.
Therefore, it is included here both to enhance the reader's
understanding of the Cross-Modal Model, and also to facilitate usage of
the Macro and Detailed Rail Models.

# 

# APPENDIX I: Macro Fatigue Management Rail Model 

# ![](media/image4.wmf)  {#section-6 .unnumbered}

# APPENDIX II: Detailed Fatigue Management Rail Model 

#### ![](media/image5.wmf)  {#section-7 .unnumbered}

#### ![](media/image6.wmf)APPENDIX II: Detailed Fatigue Management Rail Model (cont'd)

#### APPENDIX II: Detailed Fatigue Management Rail Model (cont'd)

> ![](media/image7.wmf)

#### APPENDIX II: Detailed Fatigue Management Rail Model (cont'd)

![](media/image8.wmf)

# APPENDIX III: Coordination Plan**: Fatigue Management Activities of the Office of Research and Development and the Office of Safety**

## Purpose

This document is part of a larger project aimed at developing an
integrated strategy and a coordination plan for fatigue research and
activities at the U.S. Department of Transportation's (DOT) Federal
Railroad Administration (FRA) for the Office of Safety (OS) and the
Office of Research and Development (Office of R&D). It is intended for
FRA use and internal distribution only.

## Background 

Staff members from the Office of R&D and the OS recognize that since
both offices are involved in fatigue research and activities, they may
facilitate achieving their mutual and respective goals by enhancing
communication, collaboration, and coordination between the offices.

## Methodology

This document is based upon information gathered in semi-structured
in-person and telephone interviews conducted with selected FRA staff
from the Office of R&D and the OS, both from headquarters and from the
field. Interviewees also identified selected documents, from which
information was also gathered.

## Participant Reported Goals and Roles for the OS and Office of R&D

Participants expressed the goal of creating greater collaboration
between the OS and Office of R&D to build on their respective strengths.
They suggested increasing communication by using a variety of
techniques. Participants also cited the value of maintaining and
supporting a fatigue liaison at FRA headquarters to track fatigue
activities and facilitate communication between the OS and Office of
R&D, and between headquarters and OS field staff. In order to enhance
coordination, interviewees suggested that communication and
collaboration need to occur at the major steps in each other's projects.
They noted it is particularly important to identify issues and plan
together from the outset. Participants also discussed the roles of each
office.

## GRAPHIC DISPLAYS OF INTERACTIONS AND ROLES 

In order to support the achievement of the reported goals of increased
communication, collaboration and coordination, this document presents
three graphic displays: 1) Coordination Plan; 2) Roles; and 3)
Short-Term and Long-Term Projects.

The Coordination Plan (Figure 1) indicates four distinct major project
phases:

1.  Planning

2.  Implementation

3.  Evaluation

4.  Sustainability

We envision in-person meetings between OS and Office of R&D at each of
the phases or milestones. Such meetings will provide an important
opportunity to achieve the expressed goals.[^1] Contact with external
and internal stakeholders also can take place at each phase as needed.
Furthermore, several modes of communication----email, intranet web
postings, written summaries and other meetings----can supplement the
milestone meetings.

The roles for OS and Office of R&D are displayed in Figure 2. This
graphic depiction emphasizes that while each office plays *primary*
roles that differ, they perform *common* roles. In addition,
participants identified *supporting* roles that draw on their respective
strengths and mutual goals and that pertain to their common and primary
roles.

One of the distinctions between OS and the Office of R&D is the emphasis
on short-term turnaround for OS as contrasted to longer-term research
for R&D, as shown in Figure 3. This graphic illustration depicts how
both offices need to be involved in identifying issues, assessing the
short versus long-term issues, and collaborating on both types of
projects.

![](media/image9.wmf)

![](media/image10.wmf)

![](media/image11.wmf)

![](media/image12.wmf)

![](media/image13.wmf)

# APPENDIX IV: Cross-Modal Fatigue Management Model

![](media/image14.wmf)

#### APPENDIX IV: Cross-Modal Fatigue Management Model cont.

#### 

![](media/image15.wmf)

#### APPENDIX IV: Cross-Modal Fatigue Management Model cont.

#### 

![](media/image16.wmf)

#### APPENDIX IV: Cross-Modal Fatigue Management Model cont.

#### 

![](media/image17.wmf)

#### APPENDIX IV: Cross-Modal Fatigue Management Model cont.

#### 

![](media/image18.wmf)

#### APPENDIX IV: Cross-Modal Fatigue Management Model cont.

#### 

![](media/image19.wmf)

# APPENDIX V: User Guide for Cross-Modal Model 

**Overview of User Guide**

Strategic Program and Evaluation Uses

The Cross Modal Fatigue Management Transportation Program and Evaluation
Model (the Model) organizes and visually displays information so it is
more accessible and usable for program planning and evaluation planning.
It supports strategic data driven decision-making. The Model is helpful
in engaging a diverse group of stakeholders and facilitating common
understandings and consensus.

**Government Fatigue Managers and Wider Industry Users**

This User Guide is intended to clarify the use of the accompanying Model
that has been constructed by and for government managers in fatigue.
However, we expect the wider transportation industry to find it useful
both in its current form and when adapted for specific purposes.

How to Use the Model

The first step is to examine the Model and its content, including needs
and examples of programs that currently address these needs, and the
intended outcomes. The second step is to identify next program and
evaluation planning priorities and which aspects of the model should be
further developed to address those priorities.

**Definitions and Descriptions\--Logical Linkages of Needs, Programs,
and Results**

This document and the Glossary define both the structural elements, e.g.
the column headings, and the cell content In this way meaning can be
clarified but the Model can remain concise. The Guide also explains how
the columns, the structural elements are causally and logically linked
to each other.

This model includes three major key structural elements:

-   Needs

-   Programs

-   Results

The causal logic of *if... then...* is evidenced in the left to right
visual format of the model---*if* a need *then* a program that addresses
the need, and *if* such a program, *then* what intended results.

**Usage and Purposes**

Primary Usage

This model is one type of Logic Model. Logic models clarify issues for
planning and evaluating programs. The visual display of information
helps clarify the logical sequence of programs and their intended
results. In this case the Model demonstrates linkages between needs,
programs, and results

It shows what's going on now, identifies clarifies gaps, and makes
information more accessible. Thus, it supports data driven policy,
program, and evaluation decision-making and long-range thinking.

Other Specific Usages

The model is helpful in collaborating across modes and groups of
stakeholders. It communicates and promotes improved sharing of
information. It is well suited for presentations since it condenses a
lot of information into a concise visual format. It will be helpful in
educating and creating a greater awareness of the need for fatigue.
Improved collaborating and sharing of information are helpful in making
policy and funding decisions.

Additional Benefits

Working with the Model, using it, and developing other versions of the
model will build capacity among users for more rigorous approaches to
program and evaluation planning and utilization of results. The content
of this Model focuses on fatigue but the method has broader application
to other content areas.

The Model can also be a first step towards an "intelligent" knowledge
based management system. Such a system would allow the user to click on
entries in the model for more information. This additional information
could provide more specificity, sources for more information, target
populations, measures, and distinctions of short, intermediate,
long-term and unintended outcomes. It can be developed so data can be
entered at local, state, and regional levels. It can also be designed to
vary the control of access for confidentiality and to compute and
present data in various formats.

Limitations and Cautions

In its current matrix format, the Model implies linear relationships.
This linearity makes it difficult to depict the complexity and
interconnectedness of the various components. When depicted in graphic
software more complexity such as feedback loops and cross-overs between
the categories can be depicted.

This version of the Model is limited to examples of programs, components
and outcome. It does not attempt to be comprehensive. Thus, it does not
allow the user to fully assess the current state of practices nor to
identify gaps. It focuses on intended not achieved outcomes.

**Intended Users**

Primary: Government Fatigue Program Managers

This version of the fatigue Model was constructed for government fatigue
program managers, evaluation and Government Performance Results Act
(GPRA) staff, and related senior policy staff. This would include staff
in each of the four modes of transportation---air, marine, rail and
trucking---as well as staff that works across modes at the DOT level.
The categories of the Model and the content were developed with these
primary users in mind. Other government users outside of DOT may also
find the method and content of interest.

Additional: Wider Industry

This version of the Model includes information pertaining to current DOT
projects across the four modes of transportation. We expect that
stakeholders in the wider industry from labor and management will find
both the structure and the content of interest. The structure or the
categories that are included in the Model can be modified for specific
purposes by any of the user groups. For example, a particular company
might want to take the same categories-needs, programs and results and
specify the content for their company. An industry-wide association
might want to fill in the content to see the range of kinds of programs
and outcomes available. Companies and industries readily identify where
there are gaps in current program and evaluation activities and thus,
help develop priorities for new projects and activities.

**How to Use the Model**

Program Planning

Fatigue program managers and senior policy staff who make decisions
regarding fatigue programs can use the model to see what program
managers identify as cross-cutting priorities for the four modes of
transportation. They also will be able to specify examples of programs
that currently address those needs as well intended outcomes of those
programs.

Staff and working groups can then decide what their priority uses of the
model are and further develop it accordingly. For example, they may want
a comprehensive list across the modes of the current programs or
activities addressing each of the six high priority needs. Or perhaps
they might want to select one or more of the need areas and identify
important types of programs that need further funding.

Evaluation Planning

The Model can be used for macro level evaluation planning or more
detailed project specific planning. For macro level planning, the use is
similar to macro program planning described above. The first step could
be a comprehensive identification of existing outcomes for selected
programs. This could be further specified by the measures that have been
used to document the outcomes as well as distinctions between short,
intermediate, and long-term outcomes. This would allow the users to
learn about the range of existing measures and outcomes. Compared with
intended outcomes it could also clarify gaps in measuring intended
outcomes.

At a more micro level for specific project evaluations, the Model can be
expanded to include more detail on: the program components;
identification of target populations for the components; and intended
short, intermediate, and long-term outcomes for each program component
and target population. Targets for specific levels of outcomes or
performance goals for the outcomes can also be developed at the project
level.

Participatory Consensus Building

Individuals and groups can work together to develop and use the model
for macro and project specific program and evaluation planning. A
workshop or facilitated meeting is a helpful method. This allows various
stakeholders to work together, understand each other's perspectives, and
come to some consensus as to how needs, programs, and results will be
defined, categorized, and prioritized.

**Definitions and Descriptions**

Structure- Column Headings and Sub Headings

***Needs**---**Need Areas and Example Indicators*****.** In this version
of the model, the first column is "Needs". The sub column heading
distinguishes between the "Need Areas" and "Example Indicators". The six
"Need Areas" have been defined as priorities by a cross modal group of
senior government fatigue program managers who view these as a
*comprehensive* set of current needs under which all relevant activities
can be categorized. These need areas can also be seen as goal areas
within the fatigue management domain.

The "Example Indicators" refer to information or documentation that
confirms or justifies why the need area has been selected. Each "need
area" has several *example* indicators. Users can presumably provide
more examples as well as more detail for each example as needed.

***Programs**---**Example Programs and Example Components*****.** The
model is used by reading from left to right. Thus, for each "Need Area"
in the left column, the following right column "Example Programs" lists
various types of programs that address the aforementioned needs. In most
cases there are more program examples that could be added to create a
comprehensive list of the kinds of current or past programs or policies
that address each need area listed in column one. Programs refer to not
only specific defined programs and projects but also policies and
activities that are undertaken to achieve remedying the need.

"Example components" pertain to the program examples in the previous
column. Again, we presume that in most cases there are more components
that can be specified and that these are just illustrative examples. If
the model is further specified to depict all existing programs and
policies then the components list would need to be expanded. This is the
place where the details of what the program or policy actually include
needs to be described and listed. For example, two educational awareness
programs might both have brochures but one might also have video or
audio tapes and without including the tape component, one might overlook
this important element of the program.

In expanded versions of the model it will be important for some usages
to include "Target Populations" as a third sub-component of "Programs".
Knowing who the programs and policies target and who they reached is an
integral part of program and evaluation planning and assessment. It is
also in this middle category of "Programs" that users may want to insert
a column with program implementation information.

Indicators of program implementation are often referred to as program
outputs. These outputs allow one to assess the degree to which the
program was implemented as planned. This is essential for interpreting
the results or outcomes. If the intended outcomes are not achieved, it
is important to know if that is because the program is not capable of
producing the outcomes or if it was not implemented as intended. For
example, if no one ever attended educational training sessions or never
used the educational materials, then the lack of positive outcomes could
be because of this poor implementation. The outputs and other indicators
of program implementation need to indicate a reasonable level of program
implementation before it is appropriate to measure outcomes.

***Results**---**Example Intended Intermediate Outcomes*****.** Intended
outcomes parallel the indicators of the needs and are at the
intermediate outcome level. These are not the shortest, most immediate
outcomes for the programs nor are they the ultimate, long-term outcomes.
Generally, these are results that should be seen from three to five
years after the program activities. In other versions of the Model,
short-term and long-term outcomes may also be specified. Specifying
short-term outcomes is important in fine-tuning the program activities
and is helpful for program improvement. Generally if short-term outcomes
are not occurring, some changes in the program implementation or outcome
measurement needs to happen. Specifying long-term outcomes helps to see
the ultimate gains that are projected from current activities.

In this version of the Model, the results are *intended* outcomes. In
another version, it might be helpful to specify *actual* outcomes for
implemented programs. A comparison of intended and actual outcomes is
important to determine if the programs are achieving their intended
purposes. Further, if there is not evidence of achieving the intended
outcomes then it is important to try to understand why. This should lead
to an examination of the program activities and how they were
implemented, how the outcomes were measured, the design of the
evaluation, and perhaps a reassessment of the assumption that the
program can reasonably be expected to lead to the intended outcomes.

***Long-Term Individual Outcomes***. The last page of the model includes
individual outcomes that could result from many of the intended
intermediate changes. We include them because they can help the user
understand another level of the kinds of results or outcomes that can
occur from the fatigue mitigation programs and policies. When using the
model to further specify outcomes, it would be helpful to consider
assessing existing data on these outcomes and developing validated
measures when current measures are deemed lacking.

Structure---Row Headings

The first four "needs" are sequential from top to bottom. The first is
to "assess needs", the second to "define and measure fatigue", the third
is to change "the organization" and the fourth is to "change the
individual". This sequencing is especially important in this linear
matrix format. For example, it depicts the necessity of having
organizational options in place before one can expect individual change.
Without the organization offering educational programs or scheduling
changes, it is not reasonable to expect the individual to change
attitudes or behaviors that affect their fatigue and performance.

Using graphic software to display the model will allow to depict a less
linear and more interconnected nature of some of parts of the model. For
example, example programs may address more than one need area and may
result in or reinforce outcomes from other rows.

The last two "needs"---"tools and technology" and "implementation,
effectiveness & evaluation"---can be applied to the first four need
areas. However managers chose to include these as separate "need areas".
Including these areas focuses on how these are primary processes through
which a variety of specific needs can be addressed.

Cell Content

The contents of the model or specifics of the programs and results are
examples and are not intended to be comprehensive. In the course of
using the Model for various purposes, users can elaborate on the number
of examples or on the kinds and extent of information for those
examples. Specific definitions of terms used in the cells are included
below in the Glossary (see Section 5). Intended Users

REFERENCES FOR LOGIC MODELS

Chinman M., Imm P., and Wandersman A. (2003). Getting to Outcomes 2003:
Promoting Accountability Through Methods and Tools for Planning,
Implementation, and Evaluation. Manual prepared for Centers for Disease
Control and Prevention participatory research grant, University of South
Carolina.

Community Anti-Drug Coalitions of America (2000). *Coalition Building
103: Strategic Planning,* Strategizers Series #39 901, N. Pitt Street,
Suite 300, Alexandria, VA 22314.

Corbeil, R. (1986). Logic on logic models. *Evaluation Newsletter.*
Ottawa: Office of the Comptroller General of Canada, September.

Den Heyer, Molly (2002). The temporal logic model concept. *Canadian
Journal of Program Evaluation*, *17*(2), 27-47.

Funnell, S. (1997). Program logic: An adaptable tool for designing and
evaluating programs. *Evaluation News and Comment*, *6*(1), 5-17.

Jordan, G.B. and Mortensen, J. (1997). Measuring the performance of
research and technology programs: A Balanced scorecard approach,
*Journal of Technology Transfer*, 22(2), Summer.

Jullian, J.A., Jones, A., and Dey, D. (1995). Evaluation and the logic
model: Program planning and evaluation tools, *Evaluation and Program
Planning*, *18*, (4), 333-341.

Labin, S. (2004, November). *Logic models: For Strategic Planning and
Knowledge Development.* Presented at the annual meeting of the American
Evaluation Association, Atlanta, GA.

McLaughlin, John A., and Jordan, Gretchen B., Logic models: A tool for
telling your performance story. *Evaluation and Program Planning*, New
York: Elsevier Science

McDonald, R. and Teather, G. (1997). Science and technology policy
evaluation practices in the government of Canada. *Policy Evaluation in
Innovation and Technology: Towards* *Best Practices*, Proceedings:
Organization for Economic Co-operation and Development.

Mayeske.G. and Lambur, M. (2001). *How to Design Better Programs: A
Staff Centered Stakeholder Approach to Program Logic Modeling*, Program
Design Institute, 2433 Vineyard Lane, Crofton, MD 21114.

Millar, Annie, Simeone, Ronald S., and Carnevale, John T. (2001). Logic
models: A systems tool for performance management. *Evaluation and
Program Planning*, Elsevier Science, 24, 73-81.

Parsons, B. (1999). Making Logic Models More Systemic: An Activity,
American Evaluation Association session, Boulder, CO: InSites.

Rogers, Patricia J., Petroscino, Anthony, Huebner, Tracy A, and Hacsi,
Timothy A. (2000). Program theory evaluation: Practice, and problems.
*New Directions for Evaluation*, Fall, Jossey-Bass, 5-13.

Rush, B. and Ogborne, A. (1991). Program logic models: Expanding their
role and structure for program planning and evaluation. *Canadian
Journal of Program Evaluation* *6*(2).

Torvatn, Hans (1999). Using program theory models in evaluation of
industrial modernization programs: Three case studies. *Evaluation and
Program Planning*, *22*(1), February.

United Way of America (1996). *Measuring Program Outcomes: A Practical
Approach*. Arlington, VA.: United Way of America. Or see
http://www.unitedway.org/outcomes/mpo.htm

University of Wisconsin Cooperative Extension (2003). Evaluation Logic
Model Bibliography, Program Development and Evaluation (web site),
http://www.uwex.edu/ces/pdande/evaluation/evallogicbiblio.html.

U.S. Agency for International Development (1999). *Results Framework*.
Introduction/Context for Planning System,
http://www.info.usaid.gov/pubs/sourcebook/usgov/mfr.html; Material
developed by MSI, http://www.info.usaid.gov/pubs/r4workshop/STRATEG6/.

Unrau, Y. (1993). A program logic model approach to conceptualizing
social service programs. *The Canadian Journal of Program Evaluation*,
*8*(1), 117-134.

W. K. Kellogg Foundation. (2004). *Logic model development guide: Using
logic models to bring together planning, evaluation, and action* (Item
\# 1209). Battle Creek, MI: Author.

Wholey, J. S. (1983). *Evaluation and Effective Public Management*.
Boston: Little, Brown.

# APPENDIX VI: Glossary for Cross-Modal Model 

The Glossary defines many of the terms in the Cross-Modal Model. It
clarifies usage of words that have specialized research and evaluation
meaning and elaborates on the acronyms for better understanding for a
wider audience. It is intended to be used in conjunction with the Model
and allows the Model to display information concisely.

Actigraph

A wristwatch-like recording device that detects wrist movements. These
wrist movements have demonstrated a significant relationship with EEG
levels as indicators of sleep. It can also be used to give bio feedback
to wearers regarding their risk of fatigue.

Benchmarking

Comparing the quality or performance at a specified level to that of a
similar situation. The level can be defined from the situation at hand
or from another similar situation as a point of comparison.[^2]

Best Practices

Procedures and activities that have been found to be effective in
achieving goals and in producing positive outcomes[^3]. This
effectiveness should be confirmed with reliable and valid outcome data.
See *promising practices*.

CEO

Chief Executive Officer or head of a company.

Bio Compatible

Consistent with human physiology. In this context, it includes circadian
rhythms that influence human fatigue and alertness patterns associated
with light, darkness, and time of day.

Cognitive Task Analysis (CTA)

Specification of the kinds and levels of cognitive or brain functioning
that are required for different kinds of tasks or work operations, in
contrast to overall physical strength. For example, handing out tickets
requires low cognitive functioning, and driving a train requires higher
cognitive skill. Generally speaking, the greater the need for higher
order cognitive reasoning, the more important are alertness and a low
risk of fatigue.

Components

Elements or parts of the programs or activities. What the program is in
contrast to where or with whom the activities are carried out.

Context Evaluation

Assesses needs, problems, assets, and opportunities to help define and
judge goals and priorities. The first stage of the CIPP model (Context,
Input, Process, and Product Evaluations[^4].

Crew Resource Management (CRW)

Uses principles of team building, situational awareness, assertiveness
training, and cross utilization, CRM has been found to have significant
benefits for safety, productivity, and morale. CRM's began in the
airline industry as "cockpit resource management" and developed to
include more employees than just the cockpit crew.

Data Surveillance System

An information system that provides systematically collected data that
can be used for monitoring or determining progress of operations or
events. May include specifications on how data is collected, what
measures and data sources are used, and how information is updated.

Error Taxonomy

Defines common fatigue factors in terms of degradation, maintenance, or
enhancement of performance. Establishes principles for classifying. *See
taxonomy.*

Fast Tool-Fatigue Avoidance Scheduling Tool

See SAFTE Tool**.**

Field Measures

Ways to count or verify events or effects outside of the laboratory in
real life situations.

Focus Groups

A method of qualitative inquiry whereby a small group of people meets
with a leader to provide input on their perceptions and experiences.
Potential strengths are the depth of information that can be provided
and the synergy between the group members. Potential limitations are
group pressure inhibiting honest feedback and the inability to reliably
generalize to larger sets of persons if the group does not well
represent the larger set.

FMR-Fatigue Management Reference Guide

Catalogue of major fatigue countermeasure programs. Produced in 2003 by
Battelle for DOT.

ID

To identify using various methods of inquiry, such as phone, Internet,
e-mail, and literature searches.

Implementation

The execution or carrying out of what is planned. Assessing the degree
of program implementation is the core of process evaluation and is
essential in planning and interpreting outcome evaluations. See *process
evaluation.*

Incidence

The number of new cases of a particular category or condition that arise
in a specified area during a specified period of time[^5],[^6] See
*prevalence* for contrast.

Individual

A particular person who is under the individual's control or choice.
Programs and activities are often aimed at affecting individuals'
knowledge, skills, attitudes, and behaviors.

Industry or Industry-Wide

Refers to the set of stakeholders, including private companies, unions,
government, non-profits, and researchers in a particular mode of
transportation. Thus, it is not restricted to private companies.

Intended Outcomes

Those results which the program or activity hopes for or directs its
attention toward. achieving. They are distinguished from actual or
measured outcomes, which are observed or documented, and from unintended
outcomes, which the program did not plan or expect to occur.

Intermediate Outcomes

Results that occur several, three years to five, years after the program
activities. They are distinguished from *outputs* and *short* and
*long-term outcomes.* See also *outcomes.*

Interviews

A method of inquiry that is accomplished with verbal dialogue, in person
or by telephone. It usually includes some common questions but may
include variation in questions across a set of interviews. It may use
open ended or pre determined closed ended responses. See *surveys*

Logic Model

A visual depiction of a program's underlying causal logic. It identifies
conceptual linkages between identified problems, activities, and
outcomes. The model is read from left to right with implied
*if...then*\...statements: *if* a given need, *then* given programs,
*if* such a program, *then* expected or achieved results.[^7] It
explains the program and what it intends to accomplish and is the core
of program planning and evaluation[^8],[^9]

Long-term Outcomes

Typically these are results that occur within six to ten years after the
onset of program activities. See also *outcomes*. Distinguished from
*outputs* and *short* and *intermediate outcomes*.

Needs or Need Areas

Can be addressed by specific activities, policies and programs. Need
areas here are the specific areas identified as priorities. They create
a comprehensive, finite set.

Needs Assessments

A systematic process for gathering information about current conditions.
The findings establish the need for an intervention or program[^10].
They can include surveys, interviews, demographic and data from relevant
databases describing current conditions.

Organizational

Under the control or subject to choice by the group with a defined and
structured purpose. In this case, the group is often a specific company
within an industry. Distinguished from *individual*.

Outcomes

Results or changes for individuals or groups of individuals during or
after participating in program activities or in this case, in fatigue
countermeasures. These changes are usually defined in terms of
knowledge, attitude, skills, or behaviors[^11],[^12] Distinguished from
*outputs* and further specified as *short, intermediate or long-term*.

Outputs

Indicators of program implementation, such as products or services
delivered by a program[^13]. Outputs are usually defined in terms of
observable or easily measurable behaviors or events. Examples of outputs
for an employee educational program: the number of persons attending
sessions; number of persons receiving materials; or number of persons
who reported having reviewed the materials. Usually occurs during
program implementation. Distinguished from *short-term outcome.*

Populate

To further specify or provide more information on identified categories.

Prevalence

The total number of people who fall into a particular category or
condition at a specified time[^14]. For example, in accident data, the
total number of people involved in night time accidents. Distinguished
from *incidence*.

Process Evaluation

An assessment of the implementation of program activities and the
qualities, strengths and weaknesses of the implementation[^15]. A
process evaluation usually focuses on internal dynamics and actual
operations[^16]. See *implementation*.

Programs

A set of activities with clearly stated goals[^17]. For this model,
programs can include policies and a loose collection of activities.
Program examples are illustrative and not comprehensive of all programs
that address the identified need. Neither are the examples comprehensive
of all programs that could produce the same outcomes.

Promising Practices

In contrast to best practices that are confirmed with data, promising
practices are procedures that have been identified by word of mouth or
with less formal or more limited data. See also *best practices.*

SAIC-SAFTE Tool- Sleep, Activity, Fatigue and Task Effectiveness Model
(SAFTE).

Mathematical model of on-duty work schedules analyzed for various
degrees of fatigue risks. It is based on research of circadian rhythms
and has been used for accident investigation and as an exemplar of how
this and similar tools can be validated. It uses Fatigue Avoidance
Scheduling Tool (FAST) software, developed for the U.S. Air Force,
Department of Defense also by Dr. Hursh.

Science-Based

Data or information which is gained by commonly accepted scientific
practices, including statistical sampling, and rigorous study design and
measurement procedures.

Self Report

Information based on participant self-disclosure. Self report can be
affected by individual perception and one's motivation to report
accurately.

Short-term Outcome

A result or effect that occurs immediately or soon after a program or
policy is implemented, usually within one to two years. They are often
defined in terms of a change in knowledge or attitudes. For example for
an employee educational program: increased knowledge of problems
associated with fatigue. See also *output* and *intermediate-outcome*s.

Surveys

A method of inquiry that usually involves asking similar questions in a
structured format and often with structured or pre determined responses.
Generally surveys involve a large number of people and a written format
although they may be verbally administered. See also *interviews*.

Systems Analysis

A particular body of knowledge and method for analyzing complex physical
or social systems. It makes explicit the complex array of factors and
interrelationships that affect a given situation.

Target Populations

The groups or persons intended to be affected by a particular activity,
policy or program. Program outcomes usually articulate change in these
groups or persons.

Taxonomy

A system of categorizing and defining a set of issues or objects. It
defines principles for classifying.

TOT- Trainer of Trainers

Applies to education, training and other knowledge dissemination
practices. TOT leverages training or education by training those who
train others.

Time-Lag

The time interval in research and evaluation between an event and the
measurement or occurrence of effects or results of the event.

Validated

Confirmed to be validated or to be measuring what is intended to be
measured. The word comes from the research term, validity. For example,
a valid measure of date of birth would be birth records. In contrast,
asking populations who are age-sensitive to disclose their age would be
a less valid measure. Difficult to test when no agreed upon standard
exists[^18].

Vertical and Horizontal Loading

Provides information on the breadth and depth of job related activities.
Breadth is the comprehensive understanding of a system. Depth is more
focused detailed work for a portion of the system. In combination can be
used to manage the flow of work.

Work-Load Analysis

Measuring, defining, and categorizing the various aspects of work tasks
including the cognitive and physical effort required to perform the
tasks appropriately.

Work Schedule Analysis

Measuring, defining, and categorizing various aspects of work schedules,
usually in terms of duration, times of day, frequency, and patterns
across a given time period, such as a week or month or several months.
Fatigue management software that performs work schedule analysis, such
as the Ximes or Fast tools, identify risks of fatigue based on patterns
of duty and off-duty time worked.

Ximes-RAS Work Schedule Representation and Analysis Software

Analyzes actual on-duty hours including hours per week and length of
duty. Shows graphic displays of working schedules by visual of a 24 hour
clock and analyzes schedules by their regularity and irregularity.
Categories documented include description of shifts, rests, sleep, and
commuting time.

REFERENCES for GLOSSARY

Chinman M, Imm P, Wandersman A (2004). Getting to Outcomes 2004:
Promoting Accountability Through Methods and Tools for Planning,
Implementation, and Evaluation. Santa Monica, CA: RAND Corporation,
TR-TR101. Available at http://www.rand.org/publications/TR/TR101/.

Gordis MD, L. (1996). *Epidemiology*. Philadelphia, PA: W.B. Saunders
Company.

Rossi, P.H., Freeman, H. E., & Lipsey, M. W., *Evaluation: A systematic
approach* (6^th^ ed.). California: SAGE Publications.

Stufflebeam, Daniel. The CIPP Model for Evaluation. Presented at the
2003 Annual Conference of the Oregon Program Evaluators Network (OPEN),
Portland, Oregon.

United Way of America (1996). Measuring *program outcomes: A practical
approach* (3^rd^ printing).

University of Wisconsin-Extension. *Providing leadership for program
evaluation Everyday logic model.*

Wandersman, A., Imm, P., Chinman, M., & Kaftarian, S. (2003). *Getting
to outcomes: methods and tools for planning, self-evaluation and
accountability*. National Center for the Advancement of Prevention. GTO.
<http://www.stanford.edu/> davidf/GTO_Volume_I.pdf and
<http://www.stanford.edu/> davidf/GTO_Volume_II.pdf

W. K. Kellogg Foundation (2001). *Logic model development guide: Using
logic models to bring together planning, evaluation & action.*

#  

[^1]: These phases have been developed for this project but are informed
    by the planning and evaluation framework *Getting to Outcomes*
    (GTO). Chinman M., Imm P., Wandersman A., (2003). Getting to
    Outcomes 2003: Promoting Accountability Through Methods and Tools
    for Planning, Implementation, and Evaluation. Manual prepared for
    Centers for Disease Control and Prevention's participatory research
    grant, University of South Carolina.

[^2]: Wandersman, pg. 65.

[^3]: Wandersman, pg. v.

[^4]: Stufflebeam, pg. 2.

[^5]: Rossi, Freeman, and Lipsey, pg. 444.

[^6]: Gordis, p. 30.

[^7]: United Way, pg. 38. Wisconsin U Extension. pg. 2-8, 2-11-2-20.

[^8]: Wisconsin U Extension. pg. 2-8 , 2-11-2-20

[^9]: Kellogg

[^10]: Wandersman, pg. viii

[^11]: United Way, pg. 2-3.

[^12]: University of Wisconsin, Program Development and Evaluation.
    Glossary.

[^13]: Rossi, Freeman, Lipsey, pg. 201

[^14]: Rossi, Freeman, Lipsey, pg. 446.

[^15]: Wandersman, pg. viii.

[^16]: Patton, pg. 206.

[^17]: Wandersman, pg. viii.

[^18]: Rossi, Freeman, Lipsey, pg. 249.