Study

Report

2008-05

Program Evaluation for U.S. Army

Lifelong Learning Centers (LLCs):

Extension to Military Operational

Specialty (MOS)-Based LLCs

Anna Cianciolo

Command Performance Research, Inc.

United States Army Research Institute

for the Behavioral and Social Sciences

20080807 181

May 2008

Approved for public release; distribution is unlimited.
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the Department of the Army

Deputy Chief of Staff, G1

Authorized and approved for distribution:

BARBARA A. BLACK, Ph.D. MICHELLE SAMS, PhD.

Research Program Manager Director

Training and Leader Development

Technical review by

Kimberly A. Metcalf, U.S. Army Research Institute

Kathleen A. Quinkert, U.S. Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Study Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: DAPC-ARI-ZXM,

2511 Jefferson Davis highway, Arlington, Virginia 22202-3926.

FINAL DISPOSITION: This Study Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for the Behavioral

and Social Sciences.

NOTE: The findings in this Study Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from... to)

May 2008 Final June 2007 - January 2008

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Program Evaluation Metrics for U.S. Army Lifelong Learning

Centers (LLCs): Extension to Military Operational DASW01-03-D-0015

Specialty(MOS)-Based LLCs

5b. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5c. PROJECT NUMBER

D730

Annq T r.inln I.nrnmnrl Parfnrrnarw- Rpaprr.h Inn I

5d. TASK NUMBER

353

Se. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

Command Performance Research, Inc. Human Resources Research

206 N. Randolph St. Suite 428 Organization

Champaign, IL 61820-3978 66 Canal Center Plaza

Suite 400

Alexandria, VA 22314

9. SPONSORINGIMONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

U.S. Army Research Institute for the Behavioral and Social ARI-FLRU

Sciences

2511 jefferson Davis Hiqhway

Arlington, Virginia 22202-3926 11. MONITOR REPORT NUMBER

Study Report 2008-05

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

Contracting Officer's Representative and Subject Matter POC: Dr. Kimberly Metcalf

14. ABSTRACT (Maximum 200 words):

Lifelong Leaming Centers (LLCs) are the physical instentiation of the Army Training and Doctrine Command's (TRADOC's)

lifelong leaming concept. Previous research by the Army Research Institute for the Behavioral and Social Sciences (ARI)

established a framework for assessing the effectiveness of LLCs. The present investigation sought to examine the generalizability

of the LLC Assessment Framework and to apply the framework to assessing the Fort Gordon LLC. This effort also examined the

requirements for enabling LLC self-assessment. Some modification to the assessment framework was necessary to include

external factors that moderate the relation between outputs and outcomes. The assessment of the Fort Gordon LLC revealed

that, overall, the outputs necessary to achieve educational transformation and Impact were produced by the LLC staff and

affiliated stakeholders. It was determined that people knowledgeable of educational theory and technology and capable of having

extensive face-to-face contact with LLC stakeholders will be necessary for LLC self-assessment. This report provides

recommendations for leveraging the LLC capabilities and for addressing external factors such that future staffing and

development decisions enhance the impact of the lifelong leaming initiative.

15. SUBJECT TERMS

Lifelong learning, technology-assisted instruction, computer-based training, educational program evaluation, learning effectiveness metrics

,BTA19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

~'~ABSTRACT OFPAGES

16. REPORT 17. ABSTRACT 18. THIS PAGE Ellen Kinzer

Unclassified Unclassified Unclassified Unlimited 153 Technical Publications Specialist

703-602-8047

Ii
Study Report 2008-05

Program Evaluation Metrics for

U.S. Army Lifelong Learning Centers (LLCs):

Extension to Military Operational

Specialty (MOS)-Based LLCs

Anna T. Cianciolo

Command Performance Research, Inc.

ARI-Fort Leavenworth Research Unit

Leader Development

Stanley M. Halpin, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences

2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926

May 2008

Army Project Number Personnel and Training

665803D730 Analysis Activities

Approved for public release; distribution is unlimited.

iii
iv
FOREWORD

As the Army's lead laboratory for research, development, and analysis on training and other human

dimensions of operational performance, the U.S. Army Research Institute for the Behavioral and

Social Sciences (ARI) is poised to address emerging topics as requested by Army organizations. One

such emerging topic is lifelong learning, a concept adopted by the U.S. Army Training and Doctrine

Command (TRADOC) to fundamentally change the timing and accessibility of Army schoolhouse

training and education. ARI has supported the development of the lifelong learning concept by

funding an initial investigation into the metrics necessary for assessing the effectiveness and impact

of lifelong learning centers (LLCs), the physical instantiation of the lifelong learning concept.

The present research was conducted at the request of TRADOC's Training Development and

Delivery Directorate to follow up the previous ARI-funded exploration of LLC assessment. The goal

of the present effort was threefold: (1) to ensure that the LLC Assessment Framework initially

developed would be broadly applicable across current and future LLCs that provide military

operational specialty qualification education; (2) to conduct an assessment of the LLC implemented

at Fort Gordon, GA; and (3) to draw up a plan for continued LLC self-assessment.

This final report presents in detail the research conducted and its findings. A revised LLC

Assessment Framework is provided with the changes and their associated justifications

documented. A narrative of the Fort Gordon LLC assessment method and findings is presented,

as are recommendations strengthening the LLC in order to achieve optimal impact. Finally, a

proposed plan for LLC self-assessment is provided along with a rationale for its design. Part of

this work (the Fort Gordon LLC assessment findings) was presented to the Fort Gordon

Directorate of Training in December 2007. The complete body of work will be presented to the

next Council of Colonels to be held to discuss TRADOC's lifelong learning initiative.

MICHELLE SAMS, Ph.D.

Director

v
ACKNOWLEDGEMENTS

The author would like to gratefully acknowledge the investment of Ms. Glenna Dobie, COL

James Markley, and ARI in the present research effort. The extensive research presented in this

report could not have been accomplished without the generous donation of time by a great

number of people for interviews, focus groups, coordination, classroom observation, factchecking, and discussion. The author is especially grateful for the assistance of Mr. Pat Baker,

Mr. Claydis Bradley, LTC Patrick Campbell, Ms. Bennita Freeman, Ms. Sonya Johnson, Mr.

Warren Nickson, Mr. Carlyle Melancon, Mr. John Overton, Mr. Mike Sizemore, students of the

25N 10 course, and students of the Signal Captains' Career Course. Tremendous gratitude to Ms.

Skinner, who provided many hours of valuable research assistance, and to the reviewers of this

report, Drs. Kathleen Quinkert and Kimberly Metcalf, for making it a much better document than

it would have been otherwise.

vi
EXECUTIVE SUMMARY

Research Requirement:

The lifelong learning concept has been advanced as an Army-wide solution to the

problem of meeting the educational demands of a rapidly changing operational environment.

Lifelong learning is defined as "a mixture of traditional schoolhouse resident education with

education presented in other locations at the individual's teachable moment" (TRADOC, 2004).

Lifelong Learning Centers (LLCs) comprise a suite of technologies that enable, among other

things, online posting of schoolhouse curriculum materials, courseware downloads, and

distributed collaboration among users.

Previous research by the U.S. Army Research Institute for the Behavioral and Social

Sciences (ARI) established a process for assessing the effectiveness of LLCs in enhancing

learning and readiness. The final report for that effort presented a comprehensive framework for

conceptualizing how resources invested in LLCs could produce change to instruction, learning,

and organizational effectiveness. The framework was used to conduct a formative assessment of

a pilot LLC established at Fort Leavenworth to deliver leader education, which demonstrated that

the framework was a feasible and useful tool for conducting LLC program evaluation.

The limited scope of the previous ARI effort, however, prevented an explicit test of the

generalizability of the assessment framework. It remains to be determined whether the

framework applies as expected to other LLCs, especially those that deliver military operational

specialty (MOS) qualification instruction. New metrics and measures associated with the range

of training and education strategies provided by LLCs, if required, must be developed. The

present research sought to examine the generalizability of the LLC Assessment Framework and

to apply the framework to conducting a formative assessment of the Fort Gordon LLC. This

effort also investigated the requirements for enabling LLCs to conduct self-assessment such that

the U.S. Army Training and Doctrine Command (TRADOC) could leverage ARI research to

further the development of its lifelong learning initiative.

Procedure:

An in-depth set of interviews and focus groups were conducted with the large variety of

stakeholders involved with the Fort Gordon LLC, a well-established MOS-based LLC and the

prototype for the lifelong learning concept. The training and education strategies enabled by the

Fort Gordon LLC represent the majority of learning strategies supported by other current and

anticipated LLCs. The information gathered was used to determine the applicability of the

assessment framework presented in Cianciolo (2007) and to make modifications to enhance

generalizability where necessary. The revised framework was used to assess the leaming

effectiveness and readiness impact of the Fort Gordon LLC. Six aspects of the Fort Gordon LLC

were examined: MOSQ Instruction, Assignment-Oriented Training, Simulations, Discussion

Forums, Leader Education, and On-Demand Training. Assessment involved interviews, focus

groups, classroom observations, system analysis, and archival data review where access to

resources (e.g., operational units) for collecting data was limited. Requirements and methods for

vii
conducting LLC self-assessment were determined via a combination of interviews, literature

review, and examination of LLC assessment lessons learned.

Findings:

The LLC Assessment Framework proved to be largely generalizable across different

types of LLCs, but some modification was necessary in order to reflect additional determinants

of organizational impact. These additional determinants were factors external to the sphere of

influence of the LLC, chiefly personnel management and institutional training procedures, which

can moderate the relation between outputs and outcomes. Some modification also was made to

metrics and measures in order to include more generalizable and usable methods for capturing

LLC outcomes.

The assessment of the Fort Gordon LLC revealed that, overall, the outputs necessary to

achieve educational transformation and impact (e.g., downloadable computer-based training

products) were produced by the LLC staff and affiliated stakeholders. Where expected outputs

were not observed, closer coordination between the LLC and stakeholders in the proponent

schoolhouse appeared to be the necessary remedy. The link between outputs and outcomes was

not readily observed in the present research effort. Larger organizational factors, such as Army

policy regarding who may train which MOSs, played a role in limiting impact by preventing

uniform access to instructional materials, reducing the time available to conduct training

anytime, anywhere, and constraining the adaptivity of training systems to evolve in response to

changing educational requirements.

Analysis of LLC self-assessment requirements indicated that assessment could not be

readily automated. Rather, the evaluation process requires people knowledgeable of educational

theory and technology and capable of having extensive face-to-face contact with LLC

stakeholders within and outside of the schoolhouse. This requirement may be met by relatively

minor modifications to current staffing levels at the program management level. Moreover,

existing institutional resources may be leveraged to support the assessment process.

Utilization and Dissemination of Findings:

Assessing LLCs informs decision making by shedding light on the educational impact of

financial investment in lifelong learning. The revised framework for LLC assessment reveals the

factors outside of the immediate LLC context that can influence the link between the outputs of

the initiative and the outcomes it achieves. In-depth assessment is necessary to demonstrate how

existing investment and processes may be better leveraged to maximize the benefit of the dollars

spent. Without assessment, decisions to save cost may needlessly reduce the effectiveness of

education, which may in turn affect readiness and morale. The present research provides

recommendations for leveraging the capabilities inherent in LLCs and for addressing external

influences to success such that future staffing and development decisions enhance the impact of

the lifelong learning initiative.

viii
PROGRAM EVALUATION METRICS FOR U.S. ARMY LIFELONG LEARNING

CENTERS: EXTENSION TO MOS-BASED LLCS

CONTENTS

Page

IN T R O D U C T IO N ...................................................................................................................... I

Overview of the Lifelong Learning Initiative ................................................................. 1

Background of the Present Research .......................................................................... 2

Overview of the Present Research ............................................................................ 3

LLC ASSESSMENT FRAMEWORK MODIFICATIONS ................................................... 4

The Original LLC Assessment Framework ............................................................... 5

Overview of the Fort Gordon LLC ............................................................................ 6

Modifications to Logic Model Components ............................................................. 10

Modifications to Metrics .......................................................................................... 13

Modifications to Measures ........................................................................................ 14

ASSESSMENT OF THE FORT GORDON LLC .............................................................. 15

Assessment of MOSQ Instruction ............................................................................ 16

Assessment of Assignment-Oriented Training ........................................................ 24

Assessment of Simulations ..................................................................................... 29

Assessment of Discussion Forums ......................................................................... 35

Assessment of Leader Education ............................................................................ 44

Assessment of On-Demand Training ...................................................................... 52

LLC SELF-ASSESSMENT ................................................................................................. 57

LLC Assessment Lessons Learned .......................................................................... 57

Implications for Using the LLC Assessment Framework ........................................ 61

A Proposed LLC Self-Assessment Plan ................................................................... 62

C O N C L U SIO N S ....................................................................................................................... 66

Limitations of the Present Research ....................................................................... 67

Recommendations for Future Research and Practice .............................................. 68

C losing Thoughts ..................................................................................................... 68

R E FE R E N C E S ......................................................................................................................... 7 1

APPENDIX A - ACRONYMS ............................................................................................ 75

APPENDIX B - REVISED LLC ASSESSMENT FRAMEWORK ................................... 77

LIST OF FIGURES

Figure 1. Logic Model for LLC Impact (from Cianciolo, 2007) ............................................ 5

Figure 2. Revised LLC Logic Model ................................................................................... 10

Figure 3. Schematic of the Proposed LLC Assessment Plan ............................................... 63

ix
LIST OF TABLES

table 1. Components and Associated Elements in the Revised LLC Logic Model ............. 12

Table 2. Estimated Expenses Associated with 25B10 Instruction from Signal Center Pilot

S tu d y ........................................................................................................................... 2 1

Table 3. Content (File) Overlap Among Tobyhanna and Sacramento HTRCs ................... 23

Table 4. Knowledge Management Assignment Completion Rate ........................................ 38

Table 5. Percentage of Non-required Data Fields Filled Out .............................................. 40

Table 6. Type of Initiating Posts for One-Post and Multi-Post Threads ............................. 43

Table 7. Capstone Exercise Performance of Resident- and Blended-SCCC Student Groups.. 51

x
INTRODUCTION

"Like many units, I suspect we received a lot of equipment that no one in the company had ever

seen or used: Polish Mine Boots ... Talon Robots, civilian global positioning systems and laser

range finders, Falcon View Mapping Software ... frequency jammers, and Polaris and John Deer

all-terrain vehicles, to give a few examples." - MSG C. Peterson

The Soldiers in today's Army face unprecedented levels of complexity in their operating

environment. The particular challenges of asymmetric warfare and counterinsurgency are well

recognized and documented (e.g., FM3-24: Counterinsurgency; see also Chiarelli & Michaelis,

2005; Williams, 2003), but as the above quotes from Long Hard Road: NCO Experiences in

Afghanistan and Iraq (2007) illustrate, these challenges only partially represent the body of

knowledge and skills that must be acquired to achieve success. The Army itself is rapidly

changing, adopting new command, control, and communications technologies, new unit

structures and personnel management practices, and new performance objectives. Moreover, the

politically sensitive and highly public nature of actions taken by U.S. Soldiers significantly

broadens the impact of mistakes. Consequently, Soldiers must develop the capability to execute

more difficult tasks, more often, and under greater pressure.

The Army's institutional training and education system has changed in response to the

demands of the modem operating environment. For instance, the Army Distance Learning

Program (now called the Army Distributed Learning Program) was initiated in the 1990s to

facilitate the participation of students outside the schoolhouse in standardized institutional

education. The anticipated benefits of distance learning included an increased graduation rate,

reduced course duration, enhanced personnel readiness status, and cost savings (Leonard,

Winkler, Hove, Ettedgui, Shanley, Sollinger, 2001; Shanley, Leonard, & Winkler, 2001).

A limitation of the distance learning program, however, was that it was yoked to the

institutional timeframe for updating and providing formal education. It enhanced the formal

education process, but was not designed to address the learning needs of Soldiers outside of this

context. Students continued to receive instruction in lock-step with career milestones rather than

conducting training on an as-needed basis in response to rapid changes in the operational

environment. A new approach was required to enable training and education that both met

rigorous standards for content and instructional strategy and enabled the development of just-intime competency throughout a Soldier's career.

Overview of the Lifelong Learning Initiative

The U.S. Training and Doctrine Command's (TRADOC's) lifelong learning concept has

emerged as this approach. Lifelong learning is defined as "a mixture of traditional schoolhouse

resident education with education presented in other locations at the individual's teachable

moment" (TRADOC, 2004). Simply stated, the purpose of the lifelong learning concept is to

fundamentally change the way Soldiers interact with their proponent schoolhouse. Its intent is to

enable anytime, anywhere access to institutionally approved learning content by leveraging

information technology and advanced instructional strategies, including performance assessment

and after action reviews (Wilson & Helms, 2003). Such access not only would benefit Soldiers,

I
but the schoolhouse as well, by linking the operating and generating forces in a rapid cycle of

learning content development and distribution.

The seed for lifelong learning was planted at the U.S. Army Signal Center, located at Fort

Gordon, GA, which sought to meet the training requirements of communications Soldiers whose

equipment was updated far more frequently than formal education could be offered. The lifelong

learning concept was later developed as an Army-wide solution to the problem of meeting the

educational demands of a rapidly changing operational environment. The long-term vision for

the lifelong learning initiative is to enable the delivery of anytime, anywhere instruction by all

TRADOC schoolhouses.

Lifelong Learning Centers (LLCs) are the concrete instantiation of the lifelong learning

concept, the portals through which globally distributed learners reach back to the institution.

They provide web-based access to the instruction provided by the proponent schoolhouse. LLCs

comprise a suite of technologies that enable, among other things, online posting of schoolhouse

curriculum materials, courseware downloads, and distributed collaboration among users. In

addition to the Fort Gordon LLC, several LLCs have been established at other schoolhouses,

including the Command and General Staff College (Fort Leavenworth) and the Maneuver

Support Center (Fort Leonard Wood), among others.

Currently, the technical management of LLCs occurs on site with the supported

proponent. Future plans include consolidating technical management into one Enterprise LLC

located at Fort Eustis, with supporting regional hubs on location with the original three LLCs

(listed above) plus a few additional locations. Such integration is expected to reduce costs and

facilitate management by centralizing technical support, content management, and user

development (i.e., course developers, instructors, discussion facilitators, etc.). In concert with the

evolving processes of technical management and instructional delivery, assessment strategies for

capturing LLC effectiveness and impact also have developed (Cianciolo, 2007).

Background of the Present Research

The present research follows from a previous ARI effort to explore metrics and methods

for assessing LLCs (Cianciolo, 2007). Initial attempts by LLC program managers and staff to

develop metrics for success focused on the activities conducted by LLC personnel (e.g., number

of courses placed online) and other easily quantifiable data (e.g., number of students enrolled).

Although such metrics were important reflections of personnel task execution, they did not

address larger questions of interest to decision makers at the organizational level. Unanswered

questions centered on the "so what?" of implementing LLCs. Did LLCs enhance the efficiency

and effectiveness of instruction? Did they improve readiness? Did they reduce the costs of

education? ARI sought to develop a generalizable framework for assessing LLCs that could be

used to answer these questions. A secondary purpose of that investigation was to conduct a

formative assessment of the pilot LLC established at the Command and General Staff College,

Fort Leavenworth.

As reported in Cianciolo (2007), a combination of literature review, interaction with

stakeholders, and data collection, was used to perform the research. Specifically, a

2
comprehensive literature review on the lifelong learning initiative (e.g., strategic plans,

marketing literature, progress reports, etc.) was conducted, as was a review of the scientific and

professional literature on technology-assisted instruction, program evaluation, and organizational

behavior. Interviews and focus groups were conducted with a variety of stakeholders involved

with the Fort Leavenworth LLC, including program managers and leadership, technical staff,

instructors, course developers, and students. Where possible, input from people involved with the

broader lifelong learning concept, such as representatives of the initiative's executive agent, also

was collected. Following the literature review and interviews, surveys were administered to

provide quantitative, representative data on the implementation and effectiveness of the Fort

Leavenworth LLC. This research produced a comprehensive, generalizable framework for

conceptualizing how resources invested in LLCs can produce change to instruction, learning, and

organizational performance. The formative assessment of the Fort Leavenworth LLC

demonstrated that the framework was a feasible and useful tool for conducting LLC program

evaluation. Using the framework, the influence of the LLC technologies on instructional

efficiency and effectiveness relative to other factors could be identified.

An important limitation of the initial LLC assessment research was that the design of the

assessment framework was based on an unrepresentative sample of LLCs (N = I, Fort

Leavenworth). The small size of this sample prevented an explicit test of the generalizability of

the LLC Assessment Framework. The purpose of the Fort Leavenworth LLC was to support the

delivery of field grade leader education, but other LLCs were established (or have been

conceptualized) to enable junior leader education, military operating specialty qualification

(MOSQ) instruction, and/or military graduate-level education (e.g., judge advocate instruction).

It therefore remains to be determined whether the framework applies as expected to these other

LLCs, particularly those that deliver MOSQ instruction. Leader and advanced military education

vice MOS-qualification instruction have different learner audiences and require different

learning environments, instructional strategies, and performance assessment and feedback

methods. Moreover, the instruction supported by MOS-based LLCs may have a more direct

impact on mission readiness (e.g., through the just-in-time development or retraining of skills

necessary to perform mission essential tasks) and potentially broader implications for cost

savings (e.g., by using simulations as a substitute for expensive hands-on training equipment).

Overview of the Present Research

The present research was sought by TRADOC in order to extend the findings of the

previous ARI research and to leverage ARI support for continued development of the lifelong

learning initiative. Its first objective was to examine the generalizability of the LLC Assessment

Framework and to revise the framework to ensure its broad applicability. To meet this goal, the

sample size of LLCs considered was expanded to two--the Fort Leavenworth LLC and the Fort

Gordon LLC. Although still small, this sample size represented two-thirds of the population of

well-established LLCs and addressed the distinct needs of MOS-based LLCs relative to LLCs

focused on leader education. Selecting the Fort Gordon LLC also enabled detailed analysis of the

broad range of training and education strategies that LLCs could offer, including simulation

downloads and training on demand. The range of learning strategies featured in the Fort Gordon

LLC was expected to represent the majority of strategies applied by current and future LLCs. An

in-depth examination of the Fort Gordon LLC was conducted via interviews, literature review

3
(where applicable), and program analysis. The ways in which the assessment framework had to

be modified to address the diverse needs of LLCs were identified and integrated into a revised

framework.

The second objective of this research was to assess the effectiveness and impact of the

Fort Gordon LLC. Six aspects of the Fort Gordon LLC were examined: MOSQ Instruction,

Assignment-Oriented Training, Simulations, Discussion Forums, Leader Education, and OnDemand Training. Data were collected using a variety of methods, including interviews and

focus groups, classroom observation, archival data analysis, and system analysis. Data collection

alternatives to surveys were emphasized in order to explore the feasibility and utility of measures

that limit demand on LLC users and stakeholders.

The third objective of this research was to explore the requirements of LLC selfassessment. This exploration was to include an investigation of the feasibility of automated data

collection as well as the capabilities and number of required personnel and technologies.

Requirements and methods for conducting LLC self-assessment were determined via a

combination of interviews, literature review, and examination of LLC assessment lessons

learned.

This final report documents the modifications made to the LLC Assessment Framework

and presents the completely revised framework in Appendix B. Next, the assessment of the Fort

Gordon LLC is presented, detailing the method and findings. Finally, recommendations for

enabling LLC self-assessment and for conducting future LLC assessment research are provided.

For readability purposes, this report was written in a modular format such that the text on each

area covered (Assessment Framework Modifications, Fort Gordon LLC Assessment, and LLC

Self-Assessment) may be read as a stand-alone document. The Conclusions section at the end of

this report summarizes the report and makes recommendations for strengthening LLC

assessment research.

LLC ASSESSMENT FRAMEWORK MODIFICATIONS

There are several differences between MOS-based LLCs and LLCs based on leader

education that have implications for how each type of LLC should be assessed. Every attempt to

maximize the generalizability of the LLC Assessment Framework was made in the previous ARI

research by including program-level stakeholders in the design process as well as high-level

representatives of LLCs other than the one at Fort Leavenworth. An in-depth examination of the

Fort Gordon LLC was conducted to put the generalizability of the assessment framework to the

test. The Fort Gordon LLC was selected because it was the longest running LLC and because it

provided a large variety of instruction that could be considered representative of the types of

instruction most LLCs would offer (e.g., MOSQ instruction, simulation downloads, on-demand

training, leader education, etc.). In this section, a brief review of the original LLC Assessment

Framework is provided, followed by an overview of the Fort Gordon LLC and a detailed

discussion of the revisions necessary to the original assessment framework based on the present

research.

4
The Original LLC Assessment Framework

The design of the original LLC Assessment Framework was based on the logic model

approach to conducting program evaluation (McLaughlin & Jordan, 2004). Logic modeling links

resources invested in a program to organizational impact through staff and stakeholder activities,

program outputs, and anticipated outcomes at the individual and small group or community

level. A logic model can be thought of as a "high-payoff target list" for conducting assessment.

That is, the elements of a logic model are those assessment targets that must be captured in order

to conduct an informative, diagnostic program evaluation. A logic model also may be thought of

as a hypothesis or "qualitative causal model" of how a program achieves impact. Assessment

results at the end of the causal chain (i.e., impact) may be explained by findings further up along

the causal chain (e.g., effectiveness of staff activities, productivity). Logic models are commonly

used for assessing programs that do not have simple return-on-investment metrics, such as social

service interventions (e.g., parenting classes, health literacy initiatives), and may readily be

adopted for non-profit educational initiatives.

Resources Technology, personnel, leadership,

Ifacilities, 

needs assessment

Activities Use of resources by personnel to

I create a blended learning environment

- Learner access to and use of the

Blede Lblended learning environment's

Blended Learning content and technology

Characteristics of the blended

Short-term Outcome - learning environment that differentiate

Enhanced Instruction/Learning i from instructor-focused learning

I and that address known shortfalls

Intermediate-term Outcome - Readiness, time to competency and

Enhanced Performance fifdong learning orientation

Long-term Outcome -

Organizational Excellence/Lifelong Learning Culture shift

Figure 1. Logic Model for LLC Impact (from Cianciolo, 2007)

5
Figure 1 above shows the logic model presented in Cianciolo (2007). This logic model

served as the architecture for the original LLC Assessment Framework. Briefly, resources were

the investment of money, labor, facilities, and technology into an LLC. Activities were the use of

resources by personnel to enable anytime, anywhere access to proponent learning content. The

output of an LLC was represented by the access to and use of the portal and its contents by

Soldiers. Short-term, intermediate-term, and long-term outcomes represented the instructional,

individual, and organizational benefits achieved by providing anytime, anywhere access to

proponent learning content. For a detailed breakdown of the logic model components into their

constituent elements the reader is referred to Cianciolo (2007).

Overview of the Fort Gordon LLC

The lifelong learning concept emerged from the conditions faced by signal Soldiers and

leaders working in the contemporary operating environment. Specifically, the increased

importance of communications personnel to mission success-combined with rapid changes to

communication technologies and the side-by-side use of new and legacy systems-created a

demand for more frequent, more rapid training (Farrell, 2001). For highly technical specialties,

access to formal training not only was a function of room in the schoolhouse but also of the

availability of equipment for hands-on training. Training requirements quickly outpaced the U.S.

Army Signal Center's ability to provide in-house instruction, so a new way of doing business

was required to meet the need. The lifelong learning concept provided this new business model

(Wilson & Helms, 2003). The broad range of training and education strategies enabled by the

Fort Gordon LLC represents a majority of the learning strategies adopted by other current and

future LLCs, making the Fort Gordon LLC a useful testbed for evaluating the generalizability of

the LLC Assessment Framework.

The Fort Gordon LLC was established in March, 2002 by the Signal Center Directorate

of Training to implement the lifelong learning concept (Walton, 2003). The University of

Information Technology (UIT), as the LLC was then known, began by providing a collaborative

space and a variety of training products (e.g., equipment simulations) online via a central

resource center or portal. This resource center enabled anytime, anywhere access to proponent

schoolhouse content through individual workstations or the establishment of virtual campuses

located throughout the world.

The early success of the UIT led to the Army-wide adoption of the lifelong learning

concept (TRADOC, 2004), with the UIT serving as the first and model instantiation and

becoming known as the Fort Gordon LLC. Additional pilot LLCs were launched at the

Command and General Staff College at Fort Leavenworth and the Maneuver Support Center at

Fort Leonard Wood, among other locations, as part of a TRADOC program to stand up LLCs at

every proponent schoolhouse.

In late 2006 and in 2007, the mission of the Fort Gordon LLC expanded significantly,

and has been folded into the larger LandWarNet eUniversity initiative. LandWarNet is the Army

component of the Global Information Grid, providing Army-wide on-demand information

collection, processing, storage, and dissemination. It consists of all of the Army's Department of

Defense/Joint communications and computing systems and services, software, data security

6
services, and other associated services. The LandWarNet eUniversity, which is supported by Fort

Gordon LLC staff, equipment, and affiliated personnel (e.g., training content developers),

provides essential LandWarNet network training and knowledge management to non-signal

Soldiers and to civilians who touch the network. Through the LandWarNet eUniversity the Fort

Gordon LLC has an Army-wide impact on information technology education and training.

To manage the scope of the present analysis and to maintain consistency with previous

research (i.e., Cianciolo, 2007), the LandWarNet eUniversity was not examined as a whole.

Rather, focus was maintained on those aspects of LandWarNet eUniversity that reflected the

generic LLC model adopted by other proponents. Each of these aspects, which represent the

outputs of resource investment and staff activities, is described in detail below.

Land WarNet e University - Signal

The LandWarNet eUniversity-Signal (eSignal) portal is the most direct reflection of the

generic LLC model. That is, it features BlackboardTm as a central element for hosting

standardized, approved proponent curriculum materials in a web-based environment that is

accessible to learners (regardless of location) 24 hours a day, 7 days a week. Also consistent with

the generic LLC model, eSignal provides one-stop access to online discussion forums. eSignal

differs significantly from other LLCs in the nature of its instructional content and, by extension,

its expected impact on mission readiness.

Military operational specialty qualification (MOSQ) instruction. In contrast to more

"university-based" LLCs, such as the Fort Leavenworth LLC, which focus on leader education,

eSignal primarily hosts military operational specialty qualification (MOSQ) instruction. As of

the writing of this report, qualification courses for 20 MOS are hosted on Blackboard for learners

in residence at Fort Gordon. eSignal also makes MOSQ instruction available to the Total Army.

The qualification course for one MOS (25B 10: Information Systems Operator - Analyst) is

hosted in Blackboard for students in the Army Reserves, with 167 graduates to date. Learners in

the Reserve Component participate in classroom-based MOSQ instruction at Regional HighTech centers located in Sacramento, CA and Tobyhanna, PA. Additionally, a 25B 10 pilot

blended-learning course for Army National Guard Soldiers was conducted in 2005. Distance

learning was provided to individual students and an abridged classroom component was held at

the Professional Education Center located in Little Rock, AR.

Anytime, anywhere MOSQ instruction is expected to have a more direct impact on

readiness than leader education for multiple reasons. First, MOSQ training addresses specific

skills that constitute effective individual and collective performance in the field. Absence of

these skills affects readiness by reducing the number of Soldiers in a unit who can carry out the

unit's mission essential tasks. The absence of leader skills certainly impairs readiness, but the

link between formal leader education and leader effectiveness in the field is subject to a greater

number of moderating factors [i.e., acquisition and use of experience-based, or tacit knowledge

(Tan & Libby, 1997; Wagner & Sternberg, 1985)],

Second, MOSQ instruction available "anywhere" enables Soldiers to learn independently

of seats in the schoolhouse. Backlogs of students awaiting reserved seats in the schoolhouse

7
reduce readiness by decreasing the percentage of personnel in units who are qualified to perform

their jobs (Cianciolo, 2007; Shanley, Leonard, & Winkler, 2001). Moreover, personnel who are

not available to the unit because they are attending schoolhouse instruction (or because they are

not yet MOS-qualified) reduce the training and mission readiness of the unit (Leonard et al.,

2001). Such absences are a particular problem for the Reserve Component, which consistently

has difficulty reaching optimal training status (Sortor, Lippiatt, Polich, & Crowley, 1994).

Making MOSQ training available anywhere enables Soldiers to learn from home, from local

technical centers, or from deployed locations, thus avoiding long-term absences from the unit.

The resident learning process may also benefit from anytime, anywhere education by

enabling students to take greater responsibility for their own learning, to prepare better for

classroom discussion and exercises, and to process course information in greater depth.

Improvements in learning processes enabled by advanced technologies are expected to enhance

readiness indirectly over the longer term, and have been discussed previously (Cianciolo, 2007),

so they are not further elaborated here.

Simulations. Providing anytime, anywhere access to MOSQ instruction is not the only

means by which eSignal (and other MOS-based LLCs) can have a direct impact on readiness.

Equipment simulations hosted in eSignal support MOS sustainment and refresher training as well

as just-in-time skill development. eSignal hosts several equipment simulations (e.g., AN/TSC85/93, Joint Network Node, FBCB2, etc.), which are available as downloads to registered users

who have common access cards. These simulations are enabled by advanced interactive

multimedia instruction technologies that use a scaffolding approach to building procedural

technical skills (Frank, G., Whiteford, B., Hubal, R., Sonker, P., Perkins, K., Arnold, P., et al.,

2004). The simulations first assist learners in acquiring a skill by demonstrating each step of the

procedural task along with text-based descriptions of the task. Next, learners practice the skill,

receiving corrective feedback from the simulation when they make mistakes. Finally, learners

validate their skills by executing a task in the simulation without feedback. The simulations

produce a GO/NOGO report for learners to self-evaluate their proficiency (Frank et al., 2004).

Technical discussion forums. eSignal also hosts discussion forums to support rapid,

horizontal information and knowledge sharing among communication professionals. The longterm vision for eSignal knowledge management is that it will serve as one-stop access to all

communication-related discussion forums. As of 1 October 2007, eSignal hosts 40 technical

forums (with 841 topics), in which signal Soldiers share information related to technical

troubleshooting of the eSignal website, all aspects of communications equipment (e.g., setup,

use, troubleshooting, maintenance, lessons learned, etc.), and the larger signal profession.

LandWarNet Leaders Forum

Currently supporting the Fort Gordon LLC's knowledge management component is the

Battle Command Knowledge System (BCKS). BCKS hosts and manages the LandWarNet

Leaders Forum, which is used by the Signal Center to provide early exposure to knowledge

management and the use of BCKS during the Basic Officer Leadership Course (BOLC), the

Basic Non-Commissioned Officer's Course (BNCOC), and the Advanced Non-Commissioned

Officer's Course (ANCOC). The intent behind the Leaders Forum is to facilitate the

8
development of signal Soldiers into active participants in horizontal knowledge sharing and

professional community growth throughout their careers. The Leaders Forum also provides easy

access to other BCKS professional forums, including NCO Net, PlatoonLeader and

CompanyCommand.

The Leaders Forum is not directly accessible through the eSignal portal. It is accessed

through BCKS via Army Knowledge Online. Future plans include combining the eSignal and

BCKS forums in order to provide one-stop knowledge management. In addition, external

communications-related forums, to include the signal Warrant Officers' forum and the 53

Listserve (for automation personnel), will be folded into the eSignal site. These plans are

consistent with the recent integration of the Army's Directorate of Information Management

portal for training, education, and knowledge sharing into the larger LandWarNet eUniversity.

On-Demand Training

On-Demand Training, enabled by online courseware delivery, makes MOSQ,

sustainment, refresher, and just-in-time training available to signal units in the field. On-Demand

Training may take the form of (1) mobile training teams (MTTs), which travel from the Signal

Center to present tailored classroom instruction at remote locations by accessing content posted

in the LLC; (2) virtual mobile training teams, which deliver special purpose instruction through

the LLC to deployed units; and (3) unit universities, which enable remote, unit-based classroom

instruction by accessing content posted in the LLC1.

Importantly, On-Demand Training makes it possible to provide training where previously

it was impossible or at least very difficult. The 24/7 availability of On-Demand Training enables

Soldiers to develop critical skills independently of the institutional education cycle. Particularly

in the case of technical skills, the requirement to enhance or refresh a Soldier's capability occurs

several times between scheduled educational milestones. For instance, the refresh rate for

communications equipment is approximately 18 months, but the span between formal training

opportunities in the schoolhouse typically is six years. In this way, On-Demand Training

supports the Army's apprenticeship model of training, in which the majority of learning occurs

on the job. On-Demand Training provides the mentor to which Soldiers are apprenticed with upto-date training content, advanced technology, and sound pedagogy.

As of the writing of this report, one MTT from Fort Gordon has been deployed to Fort

Hood to conduct post-deployment 25U30 MOSQ instruction. One virtual mobile training team

was assembled to assist a unit in Iraq with tearing down, moving, and setting up a new piece of

satellite communications equipment. The short-course (videotaped demonstrations with

instructors and manufacturer operation, equipment, and repair manuals) took the LLC staff two

weeks to prepare with an estimated cost savings of greater than $400,0002. Forty-seven Unit

Other components of the Fort Gordon LLC, including simulation downloads, the technical discussion forums, and

non-resident 25B 10 qualification instruction, may be considered on-demand training due to their independence from

the schoolhouse. The distinction between these outputs of the LLC is made to clarify the discussion and later

analyses of finctionality and effectiveness.

2 Estimated cost savings were presented by the Fort Gordon LLC in a VIP Brief on the Fort Gordon LLC, dated July

2007.

9
Universities have been established, up from seven in November of 2006. Ultimately, Unit

Universities will be made available to all types of unit that have a signal element, which could

number more than 600.

Mobile training teams (virtual or otherwise) are activated at the request and expense of a

unit. To establish a Unit University, a unit commander or training manager (e.g., the battalion S3

or training NCO) contacts the LandWarNet eUniversity staff and requests access to particular

MOS course material. If the requested MOS courseware is not among the 20 MOS course

curricula already available on the LandWarNet eUniversity server, the LandWarNet staff

procures the training by contacting the instructors who deliver the course at the Signal Center.

Placing the courseware onto a website created for the requesting unit launches the Unit

University with the sustainment training serving as the initial focus. Once a Unit University is

established, users also have one-stop access to other features of the LandWarNet eUniversity,

such as equipment simulation downloads.

Modifications to Logic Model Components

Few modifications to the underlying logic model components of the LLC Assessment

Framework were anticipated because the logic model used to create the framework was theorybased and designed to generalize across technology-assisted educational initiatives, both civilian

and military (see Cianciolo, 2007). It was discovered through analysis of the Fort Gordon LLC,

however, that some refinement to the logic model and its constituent elements was necessary to

represent a more complete hypothesis of cause and effect with regard to LLC impact. The main

drivers of change were (1) the realization that the timeframe of impact (short-term, intermediateterm, and long-term) was unrelated to the breadth of impact (individual, unit, organization); and

(2) the discovery of external factors that moderate the link between LLC outputs and the

expected outcomes. The revised LLC logic model is shown below in Figure 2.

Resources

Activities

External I

Factors ft t1pu i ts

Individua Organizational

Outcomes Unit Outcomes

Outcomes

Figure 2. Revised LLC Logic Model

10
Timeframe of Impact

The Fort Gordon LLC's delivery of training outside the context of formal educational

milestones (e.g., training on demand, simulation downloads), makes it possible to achieve

individual outcomes during the short-term that were originally thought to occur during the

intermediate-term (e.g., enhanced readiness). Outcomes thought to occur at the organizational

level, over the long-term, may actually occur at the individual level (e.g., culture shift) or over

the short-term (e.g., cost savings) as opportunities to provide just-in-time training arise. Given

the generally limited scope of program assessments and the constantly changing nature of Army

initiatives such as the Fort Gordon LLC, considering outcomes in terms of the timing of impact

may draw attention to assessment targets that are not observable or particularly meaningful.

A more apt categorization of outcomes discriminates between individual, unit, and

organizational outcomes, rather than short-term, intermediate-term, and long-term outcomes.

" Individual outcomes are those changes in learner behavior and capability accomplished

via technology-assisted instruction, training on demand, and associated enhancements to

instructional efficiency and effectiveness.

" Unit outcomes are those changes in unit effectiveness enabled by anytime, anywhere

access to proponent learning content.

* Organizational outcomes are those changes to organizational functioning at the

classroom, proponent, and Army level accomplished via lifelong learning.

External Factors

As the diversity of an LLC's educational offerings increases, and the breadth of outreach

expands, the opportunity arises for external factors outside the direct sphere of influence of the

LLC to play a role in achieving impact. In these cases, organizational readiness factors, which

are commonly recognized to influence the effectiveness of educational initiatives (e.g., Dean,

Biner, & Coenen, 1996; Salas, Rhodenizer, & Bowers, 2000; Leonard, Winkler, Hove, et al.,

2001), become even more important determinants of success. The requirement for coordination

with external actors is especially strong in a case such as the Fort Gordon LLC, where impact

depends on the actions of others who do not work within the same hierarchy or reward

structures. Examples of external factors influencing LLC effectiveness include (but are not

limited to) (1) the duration of the institutional curriculum development cycle; (2) the availability

of computing facilities located on post to support blended resident instruction; (3) the incentive

program for engaging in required at-home learning; and (4) Army policy regarding who can

teach what qualification courses. The actors who could influence these factors range widely from

unit commanders, to human resource managers, to Army-level plans and operations personnel.

External factors were not represented in the original logic model because they were

considered outside of the sphere of influence of the LLC initiative. However, they do play a

critical role in enabling success and therefore should be recognized in an LLC assessment effort.

Failure to account for external factors would prevent a comprehensive understanding at the

program and Army level of how resources must be leveraged or processes enhanced to maximize

impact. As an analogy, the lifelong learning concept has been considered in this research

11
program to be the main effort, but its success is determined by the actions of its supporting

efforts. If the commander is unaware of how supporting effort activity influences the success of

the main effort, he lacks the critical information he needs to ensure mission success.

Table I below presents the revised logic model elements list. As with the original LLC

logic model, each element is tied to an element category and a logic model component.

Table . Components and Associated Elements in the Revised LLC Logic Model

Money

o Costs directly and indirectly attributable to delivering proponent courses (resident, nonresident, and simulation-supported), including technical staff costs

o Costs directly and indirectly attributable to delivering training on demand

Resources o Costs directly and indirectly attributable to implementing discussion forums

Personnel

o Leadership, technical staff, course developers, courseware producers, and instructors

Fixed Assets

o Technology, equipment, supplies, and facilities used to develop, implement, and maintain

LLCs and to provide proponent learning content

Technical Staff

o Setup, integrate, customize, and manage LLC components

o Migrate course content across LLC components

o Provide training to instructors, curriculum developers, and other users on the LLC

components

o Provide technical support to students, faculty, curriculum developers, and other users

o Provide support for answering Field Army users' operational questions

Curriculum DeveloRers

o Collaboratively generate course content using LLC applications

o Lead the development of standard operating procedures (SOPs) for leveraging the capabilities

of the LLC components

o Mentor late adopters on system functionalities to enhance course development

Instructors

o Deliver curriculum materials by posting them in the LLC

o Customize course content based on student feedback and access to other resources (e.g., Army

Knowledge Management)

Activities o Perform course administrative duties

o Evaluate student progress and report grades to school administrators

o Lead the development of SOPs for leveraging system capabilities

o Mentor late adopters on application functionalities to enhance course instruction

CBT/WBT Courseware Production Team

o Maintain project teams to perform CBT/WBT analysis, design, development, implementation,

maintenance, and validation

o Provide contractual, technical, and educational/quality oversight of contractor-developed

CBT/WBT

o Maintain a database of CBT/WBT technologies, capabilities, and techniques

Leadership

o Provide and communicate vision

o Initiate and oversee user and stakeholder needs assessment

o Market the LLC concept to stakeholders

o Procure resources to maintain/update the LLC, oversee operations, prioritize limited resources

across LLC functions

o Initiate/organize the development of SOPs for leveraging system capabilities

Outputs 24/7 Uniform Access

o Access to the system

o Actual use of the system

12
ComNuter-/Web-based courseware for delivering rvronent courses

o Courseware that is readily available to meet course needs

External Culture

Factors o Unit commander support of/emphasis on informal education

Resources

o Time available for learners to conduct studies in the context of other work

o Fiscal incentives available for learners to conduct studies at home

o Technology and procedures available to track individual learners

o Contracting cycle

o Computing facilities

Policies

o Army-level policy supporting anytime/anywhere learning

o Army-level policy supporting rapid course updates

Outcomes Improved Student Performance

(Individual) o Enhanced higher-order thinking

o Enhanced skill development

o Enhanced reflective capability

o Enhanced learner independence and responsibility

0 Enhanced learning self-efficacy

o Enhanced motivation

Adoption of Lifelony Learning Orientation

o Distal motivation to engage in opportunities to learn

o Enhanced collaboration orientation

o Internalization of anytime, anywhere learning

Enhanced Mission Readiness

o Just-in-time competency

o Enhanced skill retention

o Enhanced (affective) organizational commitment

o Enhanced socialization in organizational goals and values

o Reduction in work-education-Family conflict

Outcomes Enhanced Mission Readiness

(Unit) o Enhanced unit status reporting

o Reduced time to optimal training status

o Enhanced MOSQ training status

o Enhanced collective training

Outcomes Immroved Teaching and Learning Environment

(Organization) o Enhanced relevance of training and educational content

0 Enhanced instructional efficiency

o Instructor as a facilitator of adult learning

" Presence of a learning community

0 Advanced CBT/WBT courseware for distributed/distance learning

Enhanced Educational Cost-Effectiveness

0 Enhanced cost-outreach

0 Enhanced throughput effectiveness

0 Reduced recycle rate

0 Enhanced CBT/WBT courseware development cost-effectiveness

o Reduced the range equipment/supplies requirements

Modifications to Metrics

As with the LLC logic model, relatively little modification to metrics was anticipated.

Where modifications were anticipated, they were expected to involve additions to outcomes in

order to reflect the unique ways in which MOS-based LLCs enhance readiness at the unit level.

In fact, several metrics were added. In general, metrics were added in order to better represent

13
the resources, activities, outputs, and outcomes associated with providing learning opportunities

other than course instruction (i.e., simulation downloads, discussion forums, and training on

demand). Although the original LLC Assessment Framework accounted for the different types of

learner reached by the Fort Leavenworth and Fort Gordon LLCs, the framework did not account

for the diverse range of instructional methods possible. The complete set of metrics for the

revised LLC Assessment Framework is shown in Appendix B.

Modifications to Measures

Some modification to measures was anticipated. Primarily, it was expected that

modifications would include (1) the addition of measurement methods that could serve as

feasible alternatives to surveys; (2) the modification of existing measures to make them more

general across LLCs (i.e., not tied to a specific mode of course delivery or type of curriculum);

(3) the addition of measures to address the broader range of assessment opportunities enabled by

the diverse outputs of different types of LLC; and (4) the addition of measures to address the

creation of new metrics, where applicable. All of these anticipated modifications were made.

First, interviews and focus groups were added as feasible alternatives to surveys. These

measurement methods provide significantly more diagnostic information than do surveys and

permit approximately the same sample representativeness as surveys in certain cases (i.e., when

participants are not located at the schoolhouse), given the relatively low response rate to surveys

by LLC users in the field. The decision to use surveys versus focus groups should be based on

the diversity and accessibility of the population to be sampled, as well as the purpose of the

information (developmental or sampling) to be acquired through the research process.

Second, some measures were modified or added to enable more generic methods of

assessing student performance and more broadly applicable measures of resources. For instance,

financial data may not be available in some cases, so reasonable methods for estimating costs

must be specified. The revised measures support this estimation process. In addition, where

special purpose measures were listed explicitly in the original assessment framework (e.g., the

1009 form used by the Command and General Staff College to assess higher-order thinking),

more generic forms of assessment were supplied. The type of instruction to which measures

could be applied also was specified in a way that would be more user friendly and generalizable

across LLCs. Rather than specifying the mode of instruction (i.e., distributed-collaborative,

resident, etc.), the appropriate LLC output was specified (e.g., leader education, MOSQ

instruction, On-Demand Training, etc.). This specification may help users of the revised

framework to more quickly determine the metrics and measures that apply to their particular

LLC.

Third, archival materials, such as programs of instruction and course crosswalks

(comparing resident to distance learning), were identified as additional measures for several

metrics. Such measures draw the attention of assessors who are relatively new to the Army

training system to the possibility of cost-effective, informative alternatives to novel data

collection. Additional measures not explicitly mentioned in the assessment framework include

previous scientific and institutional studies. As is demonstrated in the Fort Gordon LLC

assessment below, the analysis of previous research provided valuable supporting information.

14
Historical documents cannot be directly linked to any one type of metric 3, however, so they were

not included, but should be explored by future assessors.

Finally, measures were added to address the new metrics. Refer to Appendix B for the

complete, revised LLC Assessment Framework.

ASSESSMENT OF THE FORT GORDON LLC

This section presents the assessment of the Fort Gordon LLC. Six aspects of the Fort

Gordon LLC were assessed: (1) MOSQ Instruction; (2) Assignment-Oriented Training; (3)

Simulations; (4) Discussion Forums; (5) Leader Education; and (6) On-Demand Training. For

each aspect assessed, the discussion is organized as follows:

" The Sample

* Assessment Questions & Method

" Summary of Findings

* Detailed Findings

" Recommendations

The Fort Gordon LLC is wide ranging, with 20 MOSQ courses posted online, 46 Unit

Universities, and well over 100 simulation and courseware downloads. The limited scope and

duration of the present assessment effort necessitated that only a sample of each of the

components of LLC be selected for analysis. This sample was chosen based on seniority (e.g.,

courses that were among the first to be hosted on the LLC were selected) and on input from

project stakeholders at the Signal Center Directorate of Training. In most cases, the sample

involved in data collection was representative (if not the population itself). Where the sample

may not be representative, caveats are provided. The investigation presented below does not

feature controlled scientific experimentation, but rather reflects the attempt to explore causality

using a qualitative (logic) model in a naturalistic organizational setting. Quantitative data were

collected everywhere possible and presented in table format, but both summarized and detailed

findings are provided in narrative format due to the complex and somewhat qualitative nature of

the research.

It should be noted that the Fort Gordon LLC is rapidly changing such that the assessment

results presented here reflect a historical snapshot of LLC functioning. For example, the

curriculum for the Signal Captains' Career Course, assessed as part of this investigation, recently

underwent significant change in order to standardize the content of the Active and Reserve

Component courses. The majority of the Signal Captains' Career Course students interviewed

had begun the course using the previous format, completing it under the new format. A small

number of the Unit Universities initially selected for examination could not be assessed because

the units themselves were deactivated. The Leaders Forum staff acquired direct BCKS assistance

during the course of the research effort, and is in the process of developing near-term plans for

3 The presence of these documents is not systematic, although their most likely focus is organizational impact, such

as cost savings, learning effectiveness, and readiness enhancement. Most commonly, these studies are authored by

government and federally funded research institutions, including the TRADOC Analysis Center, RAND

Corporation, and ARI.

15
significant changes to the organization of forums. In each section below, an attempt has been

made to document the implications of rapid change for interpreting the assessment findings.

Continuous assessment would be necessary to ensure that the need for and impact of change is

tracked.

Assessment of MOSQ Instruction

The Sample - 25B10

To fully study the impact of the Fort Gordon LLC on MOSQ instruction, a course had to

be selected such that the effects of the LLC on delivering and administering instruction could be

examined. As described in Cianciolo (2007), educational technology may transform the

classroom experience both by how it is used to convey curriculum materials during class time

and by how it is used to conduct administrative activities outside the classroom. The lack of

computer lab facilities at Fort Gordon prevents anytime, anywhere access to course materials by

many resident students, so the Fort Gordon LLC currently is not being used to conduct such

administrative activities as assigning or collecting homework, posting announcements, or sharing

course content among resident students. The non-resident student and/or instructor experience

with accessing and using LLC technologies from a distance therefore was the only such

experience that could be examined. Examining a course that provides resident and non-resident

instruction also was necessary to determine whether the LLC enabled standardization and

increased outreach of proponent curriculum materials. Increased outreach is a critical mechanism

by which the cost per student of a course is reduced. At the time the present research was

conducted, the 25B 10 course was the only MOSQ course offered in both resident and nonresident format using the Fort Gordon LLC technologies (i.e., Blackboard). For this reason, the

25B 10 course was selected for analysis and a sample size of one equals the population.

The 25B 10 course provides instruction for "entry-level" (Skill Level 1) information

systems operator-analysts. The course lasts approximately 19 weeks, and is segmented into 18

annexes, covering such topics as information assurance, network essentials, troubleshooting and

repair, routers and switches, basic operations of various commercial off-the-shelf applications

(e.g., Microsoft Outlook, UNIX, and SOLARIS operating systems), and basic operations of

proprietary tactical equipment (i.e., selected components of the Army Battle Command System).

Annexes comprise lectures, demonstrations, simulations, and hands-on training in different

sequences depending on the annex. The 25B 10 non-resident course is offered to students in the

Reserve Component via High-Tech Regional Centers, where students may come together for

lectures and hands-on training. Curriculum materials (e.g., lecture slides) also may be accessed

remotely by non-resident students via Blackboard.

Assessment Questions & Method

The analysis of MOSQ instruction assessed the processes and outcomes associated with

delivering the 25B 10 course to resident and non-resident learners via the Fort Gordon LLC.

Specifically, the following questions were asked and associated metrics used to focus data

collection:

16
o Ouestion: Does delivering 25B 10 using Blackboard enhance course relevance?

o Metric: Estimated frequency of instructor augmentation to standardized course

curriculum relative to pre-LLC situations

o Question: Does using Blackboard to administer the 25B10 course in residence enhance

instructional efficiency?

o Metric: Estimated % reduction in time spent in classroom doing administrative

tasks, including announcements, handouts, and testing using LLC

o Metric: Estimated % increase in time available to assist students having

difficulties

o Ouestion: What are the cost savings associated with administering 25B10 online?

o Metric: Estimated % reduction in cost-per-student with LLC versus prior to LLC

implementation

o Metric: % reduction in travel, housing, and student pay expenses

o Question: Does administering the 25B10 course online enable uniform access to training

content?

o Metric: % curriculum materials found in Blackboard that are common across user

locations

The above metrics were partially captured via a series of interviews and a focus group.

Specifically, the Division Chief of 25B resident instruction was interviewed, as were the former

Fort Gordon LLC Program Manager, the head instructors for the non-resident 25B 10 course

administered at two Reserve Component High-Tech Regional Centers (Tobyhanna, PA and

Sacramento, CA) and at the Army National Guard Professional Education Center located in

Little Rock, AR. A focus group was conducted with three resident 25B10 instructors and the

Deputy Division Chief of 25B resident instruction. The interviews and focus group were

conducted as an altermative to surveys for assessing instructor activities and impressions.

Archival materials also were examined, including the 25B 10 program of instruction, the 25B 10

non-resident curriculum materials, and a cost-savings analysis of a pilot non-resident 25B 10

course conducted at the Army National Guard Professional Education Center.

Summary of Findings

In the case of 25B 10, analysis of activities and outputs indicated that there was great

potential for online course delivery via the Fort Gordon LLC to enhance the outreach,

administration, standardization, currency, and cost-effectiveness of MOSQ instruction. The LLC

technical staff performed the activities required to post curriculum materials online, creating

three 25B10 Blackboard sites that enabled students in both the Reserves and National Guard to

earn MOS-qualification without the obligation to leave their units or families and incur the

related travel, lodging, and duty pay costs associated with resident instruction. Blackboard sites

for the resident 25B 10 course allowed instructors to present course materials independently of

classroom arrangements. Moreover, administering exams via Blackboard allowed instructors to

reduce significantly the amount of time spent administering exams and to use the additional time

for remedial instruction. The LLC help desk provided assistance to course developers and

instructors upon request.

17
Linking the online delivery of 25B 10 materials to fully realized benefits, however, was

not a straightforward matter due to factors outside the sphere of influence of the Fort Gordon

LLC technical staff and the Signal Center Directorate of Training. These factors included (1) the

length of the curriculum review and revision process for programs of instruction; (2) lack of

public computing facilities at Fort Gordon; and (3) Army-level restrictions on who may teach

MOSQ courses.

The length of the curriculum review process made it very difficult for LLCs to facilitate

meaningful change to institutional curriculum content. Although some modification could be

made without being subject to the institutional review process, the nature of such change was

superficial. Substantive updates to address new equipment or equipment versions could not be

approved or resourced without formal review. Moreover, instructor access to Blackboard sites

was restricted such that content edits or augmentation had to be made through course developers.

This restriction ensured standardization, but limited instructor participation in enhancing course

currency. Lack of public computing facilities limited student access to course materials outside

of the classroom, making optimal use of class time difficult to achieve for resident instructors.

Without common access to computers after hours, course administration had to remain a

classroom activity. Army-level restrictions on who may teach MOSQ courses prevented the

broad implementation of non-resident 25B 10 instruction hosted via the LLC and the associated

cost savings.

Some modification to internal processes would have facilitated the use of Blackboard to

enhance instructional efficiency and to enable uniform access to course content. Different

programs of instruction were used in all three of the 25B 10 courses analyzed (resident, nonresident/Tobyhanna, and non-resident/Sacramento), in part due to lags in larger training

processes or schedules but also due in part to lack of coordination among the managers of

resident instruction and the LLC staff. Ensuring standardization was not viewed as a role for the

LLC staff (rather, this role was to post content as requested) or the resident instruction managers

(rather, communication outside the schoolhouse was handled by the Army's Automated Systems

Approach to Training personnel). Similarly, the relative roles of the LLC staff and resident

instruction managers in ensuring that instructors were facile with technology-assisted instruction

and Blackboard features were undefined. More tightly integrated roles, guided by a systems

perspective on producing educational change, were necessary to sustain the momentum of

proactive individuals who leveraged the capability of LLC technologies.

Detailed Findings

Course relevance. Army training and education courses go through a lengthy review

process prior to substantive changes being made to the program of instruction. Substantive

changes are those changes that lead to new requirements for resources, such as hands-on training

equipment, simulation-based courseware, time, and personnel. Substantive changes may include

the introduction/removal of a topic area or the addition/deletion of content to an existing topic

area (such that the required number of hours for instruction changes). The resources necessary to

meet the new requirements are not provided by the Training and Doctrine Command until the

revised program of instruction has been reviewed and approved. The review process takes

approximately three years, a timeframe that lags behind communications equipment

18
modifications. Because substantive changes to a curriculum are subject to the institutional review

process, the LLC may help to address this lag if it is used by instructors and/or course developers

to rapidly make minor revisions to course content or to disseminate new information or

downloads that students could access during time not used for classroom instruction.

The three 25B10 instructors and the Deputy Division Chief of 25B resident instruction,

who participated in the focus group, indicated that approximately 30% of a lesson plan may be

changed without requiring the lengthy institutional review process. These changes include

corrections to typos and other errors in the lecture slides as well as relatively minor revisions of

content. For example, slides could be added to a lecture to reflect a change in equipment

functionality or the presentation strategies within a lecture could be modified such that a more

multimedia approach is used. Focus group participants reported that they were not allowed to

make such changes themselves (i.e., they had read-only access to Blackboard), but were required

to work with course developers who have administrative access to curriculum materials. Access

restrictions were built into the curriculum management process to ensure the quality control and

standardization of course materials. Moreover, public access to computers outside of the

classroom was not widely available for resident students, so the Fort Gordon LLC was not used

to distribute extracurricular materials. For these reasons, focus group participants did not report

engaging in active modification of course content.

To the extent that the sample of instructors interviewed was representative and reporting

was accurate, the Fort Gordon LLC appears to have had relatively little impact on enabling

enhanced relevance of MOSQ training content such that MOSQ instruction optimally meets the

readiness requirements of field units. Online posting of institutional course content can

strengthen the tie between the non-resident curriculum and the needs of the Army as reflected in

the proponent's priorities for instruction. However, the critical changes that must be made to

address the lag between curriculum revision and equipment updates are largely beyond the

sphere of influence of the LLC, requiring modification to the curriculum development and

resourcing process more so than implementation of advanced training delivery technologies.

Such technologies are most effective for standardizing course currency across resident and nonresident learners or providing up-to-date content that is not tied to a particular institutional

course. This finding is consistent with Cianciolo (2007).

It should be noted that the adoption of new communications equipment by field units also

lags behind the release cycles of equipment modifications or replacements. The educational and

readiness costs associated with curriculum lag are unknown because the degree of mismatch

between training content and fielded equipment is undocumented. The Fort Gordon LLC may

better address readiness shortfalls resulting from curriculum lag by providing training on demand

rather than by attempting to change institutional processes.

Instructional efficiency. Instructional efficiency is enhanced by technology when it is

used to reduce the amount of classroom time spent on non-teaching activities and increases instructor time spent on tailoring the learning experience to the needs of a particular class or

particular individuals (e.g., Bourne, 1998). Focus group participants (N = 3 25B 10 instructors

and the Deputy Division Chief of 25B) reported that there was relatively little reduction in

classroom administration time enabled by hosting 25B10 course content online in Blackboard.

19
The reason they provided was that resident 25B 10 students do not have uniform access to

computing facilities outside of the classroom. They stated that announcements, homework

assignments, and other materials posted online would not be accessible to most students, so class

time was used for course administration instead. In addition, they noted that the 25B 10 course

has very few group assignments, which students use face-to-face time to complete. Non-resident

instructors at the High-Tech Regional Centers reported not conducting online course

administration for similar reasons.

Focus group participants did indicate, however, that using Blackboard could vastly

simplify the administration of quizzes and exams, which would in turn allow them to make better

use of class time. They reported that students took all quizzes and most exams in Blackboard

with grading done automatically by the system. Prior to administering assessments in

Blackboard, all tests were paper-based, requiring instructors to determine grades by hand and to

provide delayed feedback to students. Administering assessments in Blackboard could enable

instructors to monitor real-time trends in student test performance such that group-level feedback

could be delivered immediately following evaluation and individual students experiencing

problems could be assisted prior to follow-on instruction. Automatic grading also could enable

instructors to spend less time working at home and more time working with students having

problems with the course content. Reducing the rate at which failing students are recycled

through the 25B 10 course would reduce the housing, duty pay, and other costs associated with

spending extra time at the schoolhouse retaking course content.

Focus group participants estimated that approximately 20% of instructors currently were

able to leverage Blackboard's capability to administer exams as described above. They also

reported that no formal process existed for training 25B 10 instructors to use Blackboard in this

way. One of the two non-resident 25B 10 instructors interviewed (representing 50% of the

population of such instructors) stated that a high frequency of connectivity problems altogether

prevented the use of Blackboard for assessment administration. The other non-resident instructor

interviewed did not report such problems, but also did not indicate that extensive use was made

of Blackboard's assessment function to enhance instruction, perhaps due to staffing shortfalls.

In the case of instructional efficiency, as in the case of course relevance, factors outside

the sphere of influence of the Fort Gordon LLC played a role in limiting its impact on how

teaching and learning are conducted. Individual instructors who proactively leveraged

Blackboard capabilities were able to introduce meaningful change to their classroom conduct,

given the limitations imposed by external factors. Outreach by the LLC technical staff could

accelerate the process by which other instructors gain proficiency with Blackboard's grade book

features, thus helping them to leverage technology for transforming the classroom. The larger

matter of course administration requires a shift in how resident student's access instruction (and

how such access is granted by the authentication software on Army servers) such that anytime,

anywhere access is a meaningful possibility for all students.

Cost savings. Reducing the per-student cost of MOSQ instruction may be accomplished

by (1) reducing the number of students who must travel to the schoolhouse for instruction; (2)

reducing the amount of time students spend in the schoolhouse; and/or (3) increasing the number

of students who participate in instruction (given roughly the same number of instructors,

20
simulations, classrooms, etc.). Providing MOSQ instruction online can reduce per-student costs

along the second of these dimensions, according to a pilot study launched by the Signal Center in

July 2004.

The purpose of the 2004 pilot study was to explore the feasibility of providing 25B 10

instruction to non-resident students in the Army National Guard. Prior to the pilot study,

National Guard students were required to go to Fort Gordon to complete the 19.5-week 25B10

course. There were 17 students in the pilot non-resident course, which was administered as a

blend of at-home instruction and a one-week capstone exercise held at the Professional

Education Center in Little Rock, AR. The pilot study enabled the Signal Center to assess the

reduction in salary (i.e., daily base pay), lodging, and meal expenses that would result from

shortening the amount of time spent in the schoolhouse from 19.5 weeks (137 days) to I week.

According to this pilot study, expenses were estimated as shown in the following table.

Table 2. Estimated Daily Expenses for 25B 10 Instruction from the Signal Center Pilot Study

Expense Categryr Amount

Daily Base Pay $73.33

Lodging $69.00 (Fort Gordon, GA); $71.00 (Camp Robinson, AR)

Meals $39.00

The data in the table indicate that the per-student duty pay, lodging, and meals costs for

sending National Guard Soldiers to Fort Gordon for the full 25B 10 course were $24,842.21 and

were $1,283.31 for sending these same students to Camp Robinson for the one-week capstone

exercise. This represents a 95% reduction in these per-student expenses, and a 93% reduction in

cost-per-student when travel expenses ($550) are factored in. When the other costs associated

with instruction (e.g., instructor salaries, teaching facilities upkeep and maintenance, hands-on

training equipment, etc.) are considered, the % reduction in per-student cost will decrease

somewhat further, but it is unlikely that it will be reduced to triviality.

For example, if the additional per-student expenses associated with providing nonresident training are estimated to be $25,000, the % reduction in cost-per-student decreases to

approximately 50%. Note, however, that a $25,000 per-student cost, in addition to duty pay,

lodging, meals, and travel expenses, would be extremely high. Such a per-student cost might be

incurred if (a) expensive, sole-purpose facilities must be built to hold the non-resident capstone

exercise; (b) the increase in the number of non-resident instructors (and/or hands-on training

equipment) is not matched by a decrease in the number of resident instructors (and/or

equipment); and (c) the courseware and simulations used for at-home instruction are very

expensive and have a short useful life.

Due to Army-level constraints on who is allowed to provide 25B MOSQ instruction, the

25B 10 non-resident course has not been administered to National Guard students since the 2004

pilot. For this reason, additional cost savings associated with qualifying 25B 10 National Guard

Soldiers via non-resident instruction have not been achieved. Analogous per-student cost savings

may not be seen for 25B 10 students in the Reserve Component who take the course at High-Tech

Regional Centers. This is because at High-tech Regional Centers, 25B 10 students spend

approximately 13 weeks of the course in the classroom. It should be noted, however, that

21
approval recently was granted for the National Guard to teach the 25B 10 MOSQ course, with the

necessary resourcing to conduct the course expected sometime in 2008. Reinstatement of this

course presumably will resume the associated cost savings.

Uniform access. Uniform access to curriculum materials includes the equivalence of

instructional content provided to resident and non-resident students as well as the equivalence of

the learning experience (i.e., what is done with instructional content) across both types of learner.

It enhances course relevance for non-resident learners in addition to instructional quality. Nonresident instructors, as well as the former LLC Program Manager and the Division Chief of 25B

resident instruction, indicated that posting 25B 10 materials online significantly enhanced the

quality and standardization of lecture materials for non-resident students. They stated that online

posting of course content made lectures less dependent upon the individual strengths of the

instructor and more tightly tied to the learning priorities identified by the proponent. The present

analysis examined the percent overlap in 25B10 instructional materials made available to nonresident learners and explored the similarities/dissimilarities in various aspects of learning

experience, including technology use and classroom time.

The Fort Gordon LLC has enabled the online posting of 25B 10 curriculum materials,

however the lead 25B 10 instructors at the Tobyhanna and Sacramento High-Tech Regional

Centers reported that their course was not based on the same program of instruction currently in

use at the Signal Center. One of the interviewees reported that updates to the resident 25B 10

program of instruction were not disseminated to the Reserve Component but that such updates

could be acquired if one knew the right person to ask. Both interviewees also reported that the

course content available to them through the LLC was not the most recent version developed, nor

was the recency of course content consistent among them (i.e., they reported that the High-Tech

Regional Center in Tobyhanna was using the 2005 version of the courseware whereas the 2004

version was being used in Sacramento). These claims were borne out by examination of the nonresident 25B 10 sites present on the LLC. Resident instructors indicated that lags in the Army's

Automated Systems Approach to Training process, which posts programs of instruction,

prevented simultaneous distribution of the programs of instruction. They also reported that

different non-resident locations use different programs of instruction depending on the resources

(e.g., hands-on training equipment) available to them.

Examination of the 25B 10 course syllabus available via the LLC revealed that both

Tobyhanna and Sacramento had the same syllabus, dated August 2003. The contact information

present in the syllabus was similarly outdated, with the names of technical support personnel and

key leaders in the Signal Center Directorate of Training no longer valid. The course map

provided in the syllabus featured different annexes than those present in either the Tobyhanna or

Sacramento Blackboard sites. The annexes present in the course websites reflected more recent

annex titles, with greater correspondence (than those in the syllabus) to the annex titles in the

2005 resident 25B10 program of instruction (used for 2006-07 resident instruction). Tobyhanna's

annex titles had 100% overlap with those in the 2005 program of instruction (i.e., the annex

names matched exactly). Sacramento's annex titles had 29% (7/24) overlap with the 2005

program of instruction and 69% (9/13) overlap with the 2004 program of instruction. A more

recent (2006) resident 25B 10 course map provided for the present research reflected partial

differences in annex titles from the 2005 program of instruction. The results of this examination

22
suggest that the resident and non-resident instructors interviewed had accurate situation

awareness regarding the recency and overlap of their course materials.

Table 3. Content (File) Overlap Among Tobyhanna and Sacramento HTRCs

Annex Overlap Explanation

Overview & AIS Security (Both HTRCs) 63% Sacramento had overview documentation and IASO

(5/8) brief that Tobyhanna did not

A Plus (Both HTRCs) 93% Chapter 8 presentation differed across locations;

(54/58) Tobyhanna had an exam that Sacramento did not

Network Essentials + (Tobyhanna) 26% Fourteen files were Chapter documents that

Network Essentials (Sacramento) (17/65) Sacramento had but Tobyhanna did not; Another 14

files were lecture notes that Tobyhanna had, but

Sacramento did not; 14 files were the chapter

presentation slides themselves, which were different

across locations; Sacramento had a student critique

and special purpose exercises that Tobyhanna did not

TCP/IP (Both HTRCs) 18% Sacramento had practical exercises that Tobyhanna

(3/17) did not; Sacramento had content for Lesson 2, but

Tobyhanna did not; Lesson plans and presentation

slides themselves were different

Troubleshooting (Both HTRCs) 100%

(36/36)

Routers & Switches (Tobyhanna) 35% Sacramento had Routers folders with no match in

Routers/Switches (Sacramento) (8/23) Tobyhanna's Routers & Switches annex; Tobyhanna

had Router folders with no match in Sacramento's

Routers annex

Windows XP/2003 Server (Tobyhanna) 49% Sacramento and Tobyhanna had completely nonWindows XP/Outlook (Sacramento) (42/85) overlapping Windows XP content and completely

Windows Server 2003 (Sacramento) non-overlapping Outlook content; Tobyhanna had

Exchange & Outlook content for installing Exchange 2003 that Sacramento

(Tobyhanna) did not

Exchange Server

(Sacramento)

Systems Administration/Network 100% Tobyhanna did not have systems administration or

Management/Security + (21/21) network management content as was suggested by the

(Tobyhanna)Security + (Sacramento) annex tide

UNIX (Both HTRCs) 100%

(23/23)

SOLARIS (Both HTRCs) 100%

(18/18)

MCS (Tobyhanna) 0% Sacramento did not have this annex

FBCB2 (Tobyhanna) 0% Sacramento did not have this annex

TIMS (Tobyhanna) 0% Sacramento did not have this annex

DTOPS/AIS (Tobyhanna) 0% Sacramento did not have this annex

Communications Exercise (Tobyhanna) 0% Sacramento did not have this annex

25B 10 Field Training Exercise 0% Sacramento did not have this annex

(Tobyhanna)

Examination of the 25B 10 course materials in the Tobyhanna and Sacramento High-Tech

Regional Center (HTRC) sites revealed marked differences both in organization and content. In

both HRTC sites, the folder structure occasionally required students to drill down two or three

folders to reach a single document or presentation. Sacramento had numerous duplicate folders

23
and files. Across the two locations, files were occasionally named differently, but upon

examination were discovered to have the same content. Table 3 above details the content

differences among the two HTRCs as well as associated explanations. As shown in the table,

content overlap in the majority of annexes was incomplete. The non-overlap appeared to have

been due largely to files that were present in one location but not in another.

On the basis of this analysis, it appears that the LLC staff has been dutifully posting

25B 10 content online, but that uniform access to MOSQ content has not yet been achieved. To

some extent, the Fort Gordon LLC may not be able to facilitate fully uniform content due to

differences in programs of instruction used. Even if the programs of instruction were the same

for resident and non-resident 25B 10 students, there would exist some inconsistency in the

learning experience across both types of learner. For instance, the head instructor from

Sacramento reported that his HTRC lacked most of the hands-on training equipment necessary to

teach 25B 10 (in particular the annexes on the Army Battle Command System). He appeared to

be unaware that some simulations to support his instruction were available in the eSignal

downloads portal. A process involving the coordination of resident instruction managers and the

LLC staff would potentially circumvent the challenges to presenting uniform course content by

(1) facilitating communication and document sharing between the schoolhouse and lead

instructors in the Reserves; (2) simplifying and standardizing the interface design for posting

course content; and (3) developing a course site interface that provides one-stop access to lecture

slides and related simulations.

Recommendations

The disconnect between LLC outputs and organizational outcomes revealed in the

present analysis should not lead one to conclude that achieving such outcomes for MOSQ is

impossible. Nor should this finding be interpreted that the Fort Gordon LLC has failed to

function according to specification. The roles of the LLC staff and the managers of resident

instruction evolved independently and therefore functioned independently at the time this

research effort was conducted. Closer coordination, guided by shared understanding of how the

LLC can uniquely improve MOSQ instruction, would increase resident instructors' ability to

leverage LLC technologies to transform the classroom and enhance proponent outreach through

uniform content push instead of individualized content pull. Factors outside the sphere of

influence of the LLC (i.e., Army-level policies, availability of public computing facilities, and

institutional curriculum review processes) must be addressed at a higher level, but data collected

at the lower level can illustrate the need and direction for change.

Assessment of Assignment-Oriented Training

The Sample - 25S10

The 25S10 Course, which provides MOSQ instruction for "entry-level" (Skill Level 1)

satellite communications operators-maintainers, was selected as an example for studying the role

that the Fort Gordon LLC plays in enabling the cost-effective delivery of assignment-oriented

training (AOT). Regardless of the curriculum-delivery medium, AOT shortens time to

competency and reduces skill decay by targeting only the critical tasks that students will need to

24
perform in the duty position they assume immediately after graduating. Lengthy Advanced

Individual Training courses (e.g., 40+ weeks) are broken down into coherent technical "tracks"

that target the skills necessary to be qualified to operate unique equipment and/or systems in a

particular echelon unit (Barrett, 2001; Kinney, 2005). For example, taken as a whole, the 25S 10

course provides common core instruction, instruction for tactical satellite operation, and

instruction for strategic satellite operation. All 25S 10 students take the common core and then

take either tactical or strategic operator instruction, depending on the type of unit to which they

expect to be assigned. Presumably because students taking AOT spend significantly less time in

the schoolhouse, course throughput may increase as a result, further reducing the cost per student

to deliver instruction (Wilson & Helms, 2003).

LLCs can support AOT by enabling students to take, as needed, additional coursework

via distance learning if/when their duty position changes (Vann, 2003). The alternate track of the

course would be amenable to remote delivery and self-study because students already have a

foundational understanding of satellite operation on which to build (see, for example, Shanley,

Leonard, & Winkler, 2001). Moreover, allowing students to take the alternate track of the course

via distance learning reduces student time in the schoolhouse, which in turn reduces training cost

and enhances readiness because learners can remain with their unit and families (Leonard,

Winkler, Hove, et al., 2001). The 25S 10 course was selected for analysis because it was among

the first of the MOSQ courses to be made available in the AOT format (at that time, the 25S

MOS was called 31S). It was transitioned to AOT format in 2002.

Assessment Questions & Method

The analysis of AOT impact assessed the outcomes of delivering the 25S10 course in an

AOT format. Specifically, the following questions were asked and associated metrics used to

focus data collection:

o Question: Does delivering 25S10 in AOT format reduce time to competency?

o Metric: % reduction in course duration

o Question: Does delivering 25S10 in AOT format enhance course relevance?

o Metric: Unit commander/Training supervisor perceptions of course relevance

o Metric: Student perceptions of course relevance

o Question: Does delivering 25S10 in AOT format increase student throughput?

o Metric: % increase in enrollment and graduation after LLC implementation

o Question: Does LandWarNet eSignal enhance the delivery of AOT?

o Metric: % intended target audience (i.e., 25S10 follow-on students) registered to

use the system

The primary means used to capture the above metrics were interviews. The Division

Chief for 25S instruction was interviewed, as was the Chief of the Resident Training

Management Branch4 , the LandWarNet eSignal Program Manager, the Signal Center Director of

Training who oversaw initial AOT implementation, and three Signal Center Quality Assurance

4 During the course of this research effort, this person assumed a new position as the Chief of Training Management

at the Signal Regimental NCO Academy. The previous position is referred to in this report because it was in this

capacity that the interviewee served during the timeframe of interest.

25
personnel. The program of instruction for the 25S 10 course also was examined to partially

determine the change in course duration affected by AOT. Unfortunately, data from the Army

Training Requirements and Resources System (ATRRS) database could not be used directly to

answer questions about student throughput. ATRRS-based information about student throughput

was retrieved indirectly through discussion with the Chief of Resident Training Management. In

addition, because the available ATRRS records did not go back far enough to compare course

enrollment/graduation numbers before and after the implementation of AOT, discussion centered

on enrollment/graduation trends over the years of offering 25S 10 in AOT format. Archival

studies conducted by the Quality Assurance Office and by the TRADOC Analysis Center

(Kinney, 2005) were examined for information about the impact of AOT on Soldiers and units in

the field.

Summary of Findings

The results of the 25S 10 assessment tentatively suggest that providing Advanced

Individual Training courses in AOT format can significantly reduce course duration, thus saving

money and potentially reducing time to competency. Strong statements about the enhancement

of operational performance or unit readiness due to reduced time to competency could not be

made, but no negative impact on performance was demonstrated. Training and human resources

management issues appeared to prevent AOT from achieving its full, envisioned impact on

student throughput and online course delivery through the Fort Gordon LLC.

Outreach to operational units who received AOT graduates appeared to be quite

challenging. Despite a smoothly functioning process, surveys administered remotely by

proponent quality assurance personnel received few responses. In contrast, an archival study of

AOT effectiveness (Kinney, 2005) produced a wealth of data, but it lasted two and a half years

and involved six site visits conducted by a team of researchers. These findings suggest that the

success of future assessments of LLC impact on unit outcomes will depend on the coordinated

effort of parties at the proponent and Army level, including the TRADOC Analysis Center.

Detailed Findings

Time to Competency. Using the reduction in course duration as a metric, delivering the

25S 10 course in AOT format significantly reduced time to competency. Prior to AOT, the 25S 10

course lasted approximately 46 weeks 5. According to the 2007 Fiscal Year, 2nd Quarter program

of instruction, students taking 25S 10 in the AOT format only spent 26 weeks in the schoolhouse.

This represents a reduction of 43% time in the schoolhouse. An analogous reduction in cost

associated with housing each student could be expected, although travel costs to and from the

schoolhouse would not be affected (and could possibly increase if re-assigned graduates returned

to the schoolhouse for alternate-track instruction).

Course Relevance. Past assessments of AOT conducted by the Signal Center have

involved capturing commander/supervisor and student impressions of course relevance via

5 Because a pre-AOT version of the 25S10 course was not available, the duration of the 25S10 course prior to

implementation of AOT format is the average of two estimates (42 weeks and 50 weeks) given by two different

interviewees.

26
survey. Surveys were administered by the Quality Assurance Office via the Automatic Survey

Generation (AUTOGEN) tool. The most recent AUTOGEN survey on 25S10 was administered

from April to Nov 2005, and data were reported internally in July 2006. This survey asked 25S 10

graduates (tactical and strategic tracks) and their supervisors to report on their ability to perform

critical technical tasks to standard6 . In the present research, the intent was to work with this

existing survey data such that duplication of effort could be avoided and outreach beyond the

institutional setting could be achieved. Unfortunately, the response rate (for both graduates and

supervisors) to the surveys was too low to draw conclusions. The response rate for supervisors of

tactical 25S10 graduates was 0%. For graduates it was 3%. For the strategic track 25S10

graduates and supervisors, the response rate was 0%.

In the absence of AUTOGEN survey data, a preliminary assessment of AOT conducted

by the TRADOC Analysis Center (Kinney, 2005) offered a partial answer to the question of

AOT impact on unit outcomes. In this study, research team members visited 17 signal battalions

in six locations, conducting surveys and interviews with the graduates of four AOT courses

(25F10, 25P10, 25Q10, and 25Q10) and their supervisors. According to Kinney, 60% (9/15) of

25S 10 graduate supervisors reported that AOT-trained Soldiers were "basically the same" as

traditionally trained Soldiers. Twenty-seven percent (4/15) claimed that AOT-trained Soldiers

were worse, and 13% (2/15) claimed they were better. In any case, supervisors were unanimous

in reporting that they treated AOT-trained and traditionally trained Soldiers the same with regard

to unit training and work assignments. These results tentatively suggest that the readiness impact

of AOT may be seen primarily in greater institutional training efficiency, rather than enhanced

unit proficiency.

Another way to look at course relevance is to determine the percentage of students who

are placed in units for which they have received the "correct" track of the course. As noted in

Shanley, Crowley, Lewis, Masi, Straus, Leuschner, et al. (2005) and Kinney (2005), AOT places

a nontrivial burden on personnel management systems to link individuals' training and duty

assignments. Kinney (2005) detailed several human resources management issues that gave rise

to the malassignment of AOT graduates, including (1) lack of automated databases for tracking

and assigning Soldiers with unconventional identifiers (i.e., additional skill identifiers instead of

MOS designations); (2) confusion over manual data entry responsibilities; and (3) confusion over

how to assign Soldiers with unconventional identifiers. Interviews with both the Division Chief

and the Chief of the Resident Training Management Branch indicated that the accuracy of

student placement was very high (>95%). Kinney (2005) largely supports these observations,

citing an 8% malassignment rate for AOT overall and a 10% malassignment rate for 25S 10.

Student Throughput. As described previously, delivering the 25S10 course in an AOT

format reduced the length of the course by 43%. Reducing the course by this much time

introduced the possibility that the course may be delivered roughly twice as often, thus

approximately doubling the student load each academic year. Increased throughput has obvious

implications for reducing the cost per student (cost of administering the course per student would

be roughly halved) and enhancing readiness (enhanced training and personnel status), which

were expected benefits of AOT detailed by the initial planners of the Fort Gordon LLC (Wilson

& Helms, 2003). Interviews with both the 25S Division Chief and the Chief of the Resident

6 Consistent with TRADOC practice, "standard" was defined in the survey as 80% GO.

27
Training Management Branch indicated, however, that student throughput has not changed since

the implementation of AOT. The Resident Training Management Chief citied a level trend in

ATRRS enrollment/graduation data since the initial years of AOT.

The 25S Division Chief provided some explanation for this level trend. He reported that

the introduction of AOT has led instructors to become more specialized such that their ability to

teach across topics has become more limited. Instead of the same instructor teaching both the

tactical and strategic tracks of the course, different instructors teach each segment because they

are held at the same time (because both tactical and strategic students begin at the same time by

taking the common core). Teaching the course in overlapping tracks therefore would require

more instructors, rather than the same number, and more instructors are not available. 25S

historically has been a MOS with persistent shortages, as reflected in the monthly readiness

reports of the Chief of Staff of the Army (Shanley, Leonard, & Winkler, 2001). Soldiers with the

necessary skills to operate satellite communications equipment therefore are assigned to units

that need them.

Online Delivery ofAOT. The adoption of the Fort Gordon LLC to deliver AOT is

reflected in the percentage of the intended target audience registered to use the system for this

purpose. In the case of the 25S 10 course, the intended target audience would be those students

who are reassigned after graduation and must complete a different track of 25S 10 instruction.

According to the initial plans for AOT, the Gordon LLC technologies would be used to deliver

this follow-on instruction (Wilson & Helms, 2003). Currently however, the LLC is not used to

deliver follow-on 25S 10 instruction, largely due to difficulties with the automated management

of individual training assignments. Students instead return to the schoolhouse to complete their

training.

That said, true cost savings associated with delivering follow-on AOT training via the

LLC are demonstrated when the intended user audience is very large. In the case of 25S 10, this

is not the case. The 25S Division Chief estimated that only a very small percentage (fewer than

5%) of 25S10 graduates require follow-on instruction. The cost savings associated with reducing

course duration therefore reflect the majority of cost savings that could be achieved via AOT, at

least for the 25S 10 course.

Recommendations

Recommendations for enhancing the impact of AOT on unit outcomes must be directed

outside of the scope of the LLC. Factors affecting impact apparently were driven by a lack of

policy on how to interpret and manage unconventional identifiers (i.e., additional skill

identifiers) for tracking training status and making unit assignments. If developed, this policy

could be used to design databases for automatically performing these management functions.

Perhaps more importantly, such policy could serve as a springboard for developing personnel

management practices that align the particular capabilities of individual Soldiers to the specific

competency requirements of units. Such practices would significantly expand the possibilities for

implementing lifelong learning on an Army-wide scale.

28
Assessment of Simulations

The Sample - 25N10 & Joint Network Node (JNN) Simulation

The 25N 10 course provides qualification instruction for the 25N MOS created in

October, 2005. It prepares "entry-level" (Skill Level 1) nodal network systems operatorsmaintainers to perform basic operations with the Joint Network Node (JNN) Network, which the

Army has selected to replace legacy Mobile Subscriber Equipment. At the time of writing this

report, the 25N 10 course did not have a non-resident version and course content was not hosted

in the Fort Gordon LLC for resident students. All instruction was performed in the classroom via

lecture and practical exercise. Since its inception, the 25N10 course has made use of equipment

simulations to facilitate classroom instruction and practical exercise performance. This practice

is consistent with the historical use of training technology (i.e., interactive videodisc) to support

Army communications training in the schoolhouse (Winkler & Polich, 1990).

Specifically, simulated JNN equipment was used to supplement hands-on training by

giving 25N10 students the opportunity to practice basic operations on a desktop computer while

waiting to conduct hands-on training with actual equipment. As described earlier in this report,

equipment simulations such as the JNN courseware use the scaffolding approach to instruction,

walking students through an Acquire "4 Practice ") Validate sequence for each skill to be

learned. The same simulation used in the 25N10 course also has been made available as a standalone download in the Fort Gordon LLC. Learners outside of the schoolhouse setting (e.g.,

deployed signal Soldiers) who have common access cards can retrieve the simulation from

LandWarNet eSignal, and many have. As of September 17, 2007 the JNN simulation had been

downloaded 2,371 times for refresher training or preparation for hands-on, on-the-job learning

with new equipment.

By studying the use of equipment simulations in schoolhouse instruction and by nonresident learners, it is possible to assess their effectiveness as learning tools in a variety of

settings. It is because of JNN simulation availability in multiple learning contexts that it and the

25N 10 course were selected for analysis. Simulations have been proposed to be at least a partial

alternative to hands-on training, thereby reducing time on equipment and equipment-related

costs, while simultaneously maintaining pedagogical standards (e.g., Wilson & Helms, 2003;

Winkler & Polich, 1990). It should be noted that one annex of only one course, using a single

suite of simulations provided by a single contractor, was assessed in the present research. In

addition, the 25N 10 course, having never been administered without simulations, cannot permit

before-after comparisons of some metrics of interest, including time savings, equipment

requirements, and student throughput 7

. A rigorous, scientific study has been conducted to assess

the use of simulations for Army communications training (Winkler & Polich, 1990), which was

used to provide a context for interpreting the findings of the present research. On the basis of this

additional context, some specification of the generalizability of the present findings is provided

below.

' Other types of analyses to capture these metrics were performed as part of this research. Consistent with the

qualitative causal approach, these alternative analyses focused on the processes that would enable such outcomes as

time savings, equipment requirements, and throughput.

29
Assessment Questions & Method

The analysis of simulation implementation and use for lifelong learning focused on the

output and outcomes associated with using equipment simulation to provide schoolhouse and onthe-job instruction on basic Joint Network Node-Network operations. Specifically, the following

questions were asked and associated metrics used to focus data collection:

o Question: Does the use of simulations in 25N10 save time on equipment?

o Metric: % reduction in time on equipment

o Question: Does the use of simulations in 25N10 enhance instructional efficiency?

o Metric: % instructors who report using increased class time to enhance instruction

o Question: Does posting the JNN simulation on the LLC enable uniform access to JNNrelated learning content?

o Metric: Equivalence of learning experience across resident and non-resident

learners

o Question: Does the JNN simulation enable just-in-time competency?

o Metric: % on-the-job simulation users who report that the JNN simulation enabled

just-in-time competency

o Question: Does the use of simulations for 25N10 broaden thinking about where and how

technical training can be accomplished?

o Metric: % students and instructors who believe that simulations can serve as a

substitute for hands-on, classroom training

To capture metric data, a combination of focus groups, classroom observation, and

survey was used. Specifically, two focus groups were conducted, one with 25N10 students (N=

16) and one with 25N 10 instructors (N = 5). The purpose of these focus groups was to gather

impressions of the effectiveness and utility of the JNN simulation for resident 25N 10 instruction

in general. Classroom observations (with different students and instructors than those

interviewed initially) then were conducted during one of the annexes of the 25N 10 program of

instruction. This annex focused on the basic functions of the Promina system. Two lecture --

simulation "- hands-on training sequences (Database Management and Basic Trunks Services)

from this annex were observed. Itshould be noted that this small sample of students and the

particular course annex selected may not be representative, however the findings from the

present analysis is consistent with previous scientific analyses and with adult learning theory so

the sample is likely not an outlier. An online survey of on-the-job users of the JNN simulation

was administered via LandWarNet eSignal. The survey was posted in the Downloads site of

eSignal and was open for approximately 3 weeks, but received no responses.

Summary of Findings

The JNN simulation was actively used in the 25N10 course and frequently downloaded

from the eSignal Downloads site. Instructors used the time saved by learning via desktop

simulation to enhance the training they provided with hands-on equipment. Use of equipment

simulations allowed for greater practice time on equipment procedures, which has positive

theoretical implications for skill retention. Instructors and students alike endorsed simulationsupported instruction and had forward-thinking views about how to use simulations to improve

30
learning. Anecdotal evidence indicated that the stand-alone JNN simulation download could be

used in a manner similar to schoolhouse classroom to enable just-in-time competency

development. However, more research will be necessary to understand how simulation

downloads are used by distributed learners. The impact of the JNN simulation on just-in-time

competency development could not be determined due to lack of survey response data.

The present findings, coupled with previous scientific study (Winkler & Polich, 1990)

suggested that desktop simulations can partially substitute for hands-on training. The general

caution that simulations cannot replace hands-on instruction was supported by the 25N 10

classroom observations. Inconsistencies between the JNN simulation and the actual equipment

caused difficulty for some students as they transferred automatic, but incorrect skills from their

desktop computer to the hands-on training equipment. Characteristics of the actual JNN

equipment omitted from the simulations had to be addressed using hands-on training time.

Instructors observed that a faster contracting cycle would significantly accelerate the delivery of

up-to-date simulations.

Detailed Findings

Time on equipment. Focus group interviews revealed a difference of opinion regarding

the degree to which simulations reduced the time required to achieve proficiency with actual

JNN equipment. 25N 10 students reported that little transferable learning was accomplished with

the simulations, citing fidelity shortfalls and simplistic learning objectives as reasons for limited

utility. For example, students indicated that they could get the correct answers during the

Practice and Validate phases of the simulation without knowledge of the underlying principles of

how the simulated equipment worked. They also stated that the simulations did not allow them to

make mistakes that they could make on the actual equipment, thus preventing them from

learning how to undo their actions in an operational situation.

In contrast, 25N 10 instructors reported that using simulations to practice foundational

skills allowed students to achieve proficiency with actual equipment faster than they would

without such practice. Although they acknowledged limitations of simulations, instructors saw

accelerated learning as a key benefit of their adoption. Instructors also noted, however, that

enhanced proficiency with hands-on equipment did not actually reduce student time on the

equipment, but rather allowed instructors to make better use of the equipment time available.

Classroom observation supported both of these contrasting perspectives. First, the

observations that students made about the simulations were partially validated. The simulations

did have some functional dissimilarity from the actual equipment, which affected skill transfer

for some students. A small minority of students found it frustrating to override incorrect, but

automatic skills developed using the simulations (e.g., use of arrow keys and command line text)

when they transitioned to hands-on training. Unlearning automatic skills increased hands-on

equipment time for these students relative to those who did not have such difficulty. Skill

development using the desktop simulation was subject to decay, and when students made

mistakes on basic procedures, they did not know how to undo them on the actual equipment. In

addition, there were aspects of the actual equipment, such as status lights, which convey

important information about the success of task execution, that were not part of the simulations.

31
Students' questions during the hands-on training that were observed often addressed the meaning

of the status lights and their relation to the task at hand.

Second, classroom observations validated the instructors' report of reduced time to

competency and enhanced use of class time. Although printed materials with detailed

instructions were available to students during hands-on instruction, at least half of the students

did not need to refer to them to complete hands-on exercises. The amount of time spent acquiring

the analogous skills using the desktop simulation was approximately 10-15 minutes. Students

using hands-on equipment moved quickly through the basic skills, and instructors optimized

(rather than shortened) the learning experience by drawing linkages between task performance

and the components of the equipment not reflected in the simulation (e.g., status lights and

installation activities). Given the roughly equivalent skill transfer associated with simulationsupported instruction versus hands-on only training found in Winkler and Polich (1990), one

might assume that the same amount of time spent learning on the desktop simulations would

have been spent learning on the hands-on training equipment. This represents a significant time

savings that could be used to enhance the hands-on training experience, even though time on

equipment was not reduced.

An important implication of coupling desktop simulation with hands-on training was that,

in one sense, time on equipment actually increased. That is, students had the opportunity to

increase the time spent on practicing equipment procedures, regardless of whether the equipment

was real or virtual. The Acquire -) Practice -- Validate sequence used in the desktop simulation

required students to practice the same procedure three times, whereas the hands-on training

would only have supported going through the procedure enough times to demonstrate

proficiency. Moreover, students could practice the procedures on the desktop simulations as

many times as they wished, depending on their self-estimate of proficiency. The additional

training time, which could be as short as 5 minutes (i.e., the observed average time spent on

hands-on training demonstrating proficiency) or as long as students wished, would enhance skill

retention for which initial levels of proficiency is the leading predictor (Arthur, Bennett, Stanush,

& McNelly, 1998). The present findings are consistent with Winkler and Polich (1990).

Instructional Efficiency. Instructor interviews in both focus groups and classroom settings

indicated that, in general, time on equipment was not reduced by simulations, but rather that

simulations enhanced scheduled hands-on training. 25N 10 classroom observation supported this

perspective by revealing that hands-on training was used to increase the basic proficiency

accomplished via desktop simulation practice and to develop skills not addressed by the

equipment simulations. As described in the previous section, instructors enhanced hands-on

training by explaining the meaning of equipment status lights, providing students with

opportunities to make and undo errors, and highlighting links between the classroom theory and

practical procedures.

Equivalence of learning experience. To the extent that technical skill development using

simulations involves analogous instructional strategies across diverse learning contexts (e.g.,

classroom vs. field settings), the learning experience and projected effectiveness of the training

may be considered to be equivalent. Clark (1994) has argued (and others, e.g., Sitzmann,

Kraiger, Stewart, & Wisher, 2006, have demonstrated) that instructional technology is only as

32
effective as the pedagogy it supports. The training and education literature indicates that some

key components of effective pedagogy, especially as it relates to technical skill development and

computer-based instruction include scaffolding, interactivity, and application (Abell, 2003;

Firdyiwek, 1999; Hays, Stout, and Ryan-Jones, 2005). Particularly for those people using

simulations to learn outside of the classroom context, therefore, structured engagement with

content and contact with others, i.e., peers and mentors, during the learning process should be

considered critical to learning outcomes. This is especially true when simulations deviate from

actual equipment function, which is commonly the case because equipment versions change

faster than their associated desktop simulations do.

In the 25N 10 course, simulations were used as learning tools within the context of

lectures about how the system works generally. The lecture slides presented high-level bullet

points and diagrams, and the instructor augmented this content with additional information about

how the equipment functions and its integration into the larger signal environment. The

instructor also engaged the students by asking questions and using humor. Classroom

observations further indicated that 25N 10 students and instructors had high levels of interaction

while using the simulations and transitioning from simulations to hands-on equipment. In

particular, instructors made students aware of the specific ways the simulations behaved

differently from the actual equipment. For example, during the classes observed, there was a

particular function for which the simulation behaved in exactly the opposite way than the actual

equipment. For the present researcher, this difference caused a great deal of confusion while

observing hands-on training, but students had been made aware of it in an earlier lecture (not

observed) and so knew to work around the problem. Instructors also encouraged students to use

the simulations to enhance retention through repeated practice before and after hands-on training.

All of these practices, according to commonly accepted learning theory (e.g., American Distance

Education Consortium, 2003; Chickering & Gamson, 1987; University of Illinois, UrbanaChampaign, 1999), should be expected to maximize the learning benefit of using simulations.

Unfortunately, due to the lack of response to the JNN survey administered to eSignal

users, a direct comparison could not be made between the simulation-supported learning

experience in the classroom and in the field. Some differences in learning experience may be

assumed, however, given that the JNN simulation download was not coupled with 25N course

materials, such as lecture slides. Possibly more importantly, the download was not accompanied

by explanations of the differences between the desktop simulation and various versions of the

actual JNN equipment. Discussions in the eSignal technical forums would be one place to

provide peer mentorship on these topics, but such discussions were not observed in the analyses

of the forums detailed later in this report. Discussions did address difficulties in downloading the

large files associated with the JNN simulation and brought up questions of versioning.

An alternative source of mentorship for on-the-job learners might include more senior

Soldiers who have greater experience with JNN operations. This possibility was presented as

likely in the focus groups with 25N10 students. One unit trainer interviewed as part of this

research effort reported that he used the JNN simulation download to teach 12 subordinates using

a classroom setting. The classroom setting enabled students to learn individually at their own

workstations but in a group setting similar to that observed for 25N 10 instruction. Beyond these

33
indirect inferences and anecdotal evidence, no further conclusion may be made about the

equivalence of the learning experience for schoolhouse and on-the-job learners.

Just-in-time competency. Unfortunately, due to lack of response to the JNN survey, the

ability of desktop simulation to enable just-in-time competency could not be analyzed.

Download data alone cannot be used as a proxy for just-in-time competency because it is

unknown what is done with the simulations after they have been retrieved. The previous example

of the unit trainer who conducted classroom JNN training with the simulation download suggests

that just-in-time competency can be supported by simulations, however. This person actively

sought out the simulation to conduct training while his unit waited to receive the actual

equipment assigned to them. Greater detail on how the downloads are used will go a long way

toward understanding LLC impact on just-in-time competency.

Broader thinking. As described previously, it has been proposed that simulations may be

considered at least a partial substitute for hands-on training with actual equipment (Wilson &

Helms, 2003; Winkler & Polich, 1990). Such a substitution would potentially reduce the costper-student of MOSQ instruction in the schoolhouse (and at other locations) by reducing the

need for costly training equipment. In addition, the education of non-resident learners, such as

those in the Reserves who complete their instruction at High-Tech Regional Centers, could be

made more equivalent to that of students in the schoolhouse by increasing access to equipment

function. The adoption of simulations as an alternative to hands-on training requires broader

thinking about what may be considered a learning environment such that learners (1) seek and

use simulations rather than awaiting classroom instruction; (2) actively use simulations to aid in

initial learning and retention; and (3) encourage the use of simulations for the training of others.

Members of both the 25N 10 student and instructor focus groups unanimously supported

the use of equipment simulations for technical skill development. Moreover, they had forwardthinking ideas about simulation properties that would better address higher-level cognitive skills.

These ideas included embedding short scenarios into the simulation instruction that required

learners to troubleshoot equipment malfunctions and networking simulated systems together

such that students could experience the challenges to achieving the interoperability of joint

network nodes. That said, there also was unanimous agreement among the members of both the

student and instructor focus groups that simulations should not be considered a substitute for

hands-on training with actual equipment. Their concerns stemmed from the conclusion that

simulations could not fully replicate the functionally of actual equipment. These conclusions

were supported by classroom observations, which indicated that dissimilarity partially

challenged the learning process. Winkler and Polich (1990) also found that compared to students

trained solely on hands-on equipment, students trained primarily using alternative technology

(interactive videodisc) had slightly higher error rates in hands-on test conditions.

Achieving the technological sophistication necessary to fully replicate JNN functionality

would be costly, technically difficult, and would require that the contracting cycle (to

develop/modify simulations) keep pace with the equipment update cycle. It is unknown whether

effort and resources invested in developing high-fidelity networked simulations would produce

the cost-benefit tradeoff expected from simulations.

34
Recommendations

More information on the impact of simulation downloads on unit outcomes is required.

LLC staff could support efforts to collect external data on simulation use in the field by

providing relatively simplistic methods for reporting on effectiveness. For instance, a rating

capability and comments area associated with each download in the Downloads page of eSignal

would present a quick and simple way for remote users to provide feedback and information

about simulation functionality. Follow-on discussion led by the eSignal technical forum

facilitators could provide further detail on how simulations are used in the field. This information

would be useful not only for assessing simulation use but also for communicating to the field

Army the training caveats that must be observed in order to use the simulations effectively. In

this way, the proponent could assist in the mentoring process for distributed learners through the

LLC staff. Closer coordination between the LLC staff and the resident instruction managers

would increase the likelihood that other supporting materials, such as lecture slides, were posted

along with simulation downloads. Some efforts to provide this type of supporting information

were begun during the course of this research effort.

Taking a longer-term view, for simulations to have an impact on the bottom line (i.e.,

cost savings associated with reduced time on equipment), as well as retention (Arthur et al.,

1998) they will have to better replicate system functionality through a combination of in-depth

task analysis, effective instructional design, and an accelerated contracting cycle. An in-depth

task analysis would enable simulation developers to better understand both the explicit and

implicit tasks that must be executed to operate equipment. Instructional design then could

exercise more of the higher-level cognitive skills that trainees must use to monitor, troubleshoot,

and integrate the equipment through a combination of scaffolding and scenario-based training.

Simultaneous use (by large groups of students) of networked desktop simulations for such

practice would shorten the number of hours spent on actual equipment, with equipment being

used to validate rather than instruct. Networked simulations also could support the integration of

on-the-job learners into ongoing resident instruction. An accelerated contracting cycle would

enable more rapid modification to existing simulations, thus allowing updates to simulation

functionality to keep pace with analogous equipment updates.

Assessment of Discussion Forums

The Sample - LandWarNet eSignal Technical Forums and LandWarNet Leaders Forum

Both the LandWarNet eSignal technical forums and the LandWarNet Leaders Forum

were assessed as part of this research. Discussion forums represent the significant potential of

lifelong learning centers to foster and enable culture shift toward anytime, anywhere

participation in professional self-development, organizational enculturation, and knowledge

management. Discussion forums also play a key role in fostering the development of

professional learning communities, which help to give a common identity to distributed learners.

Examining the Leaders Forum provided the opportunity to explore the current status of the Fort

Gordon LLC in fostering culture shift toward the use of discussion forums for lifelong learning

and horizontal information sharing. Analyzing the eSignal technical forums enabled an

35
investigation of how the Fort Gordon LLC supports these ongoing efforts by distributed Signal

Soldiers.

It is much easier to build a collaborative meeting space than it is to foster transformative

use of that space (Cianciolo, Heiden, & Prevou, 2006; Wenger, McDermott, & Snyder, 2002).

Yet, user activity is critical to achieving the organizational impact expected to result from

investments in collaborative meeting space. For this reason, assessing the status of the discussion

forums component required going beyond the impressive number of forums already established

in eSignal and investigating how these forums were used and to what extent they fostered

knowledge development and professional growth. The development of social and intellectual

capital in discussion forums is a critical determinant of professional growth (Cianciolo et al.,

2006; Lesser & Storck, 2001) and also is feasibly (and relatively objectively) measured, so it

served as the focus of the present research. Four aspects of social and intellectual capital growth

defined elsewhere (Cianciolo et al., 2006; Dixon, Allen, Burgess, Kilner, & Schweitzer, 2005)

were used to frame assessment: connections, context, content, and conversation.

Assessment Questions & Method

The analysis of discussion forums as contributors to lifelong learning community

development focused the output and outcomes associated with implementation of the technical

forums and Leaders Forum. Specifically, the following questions were asked and the associated

metrics were used to focus data collection:

o Question: Are the discussion forums being actively used?

o Metric: % of registered users actively using the system (adoption rate)

o Question: Are the discussion forums supporting the development of interpersonal

connections?

o Metric: % of posts that contain referrals (to self or others) for additional

information or expertise

o Metric: % users with basic biographical information viewable by others

o Question: Are the discussion forums supporting the development of organizational

context?

o Metric: Presence of "built-in" opportunities to participate in shared experiences

o Question: Are the discussion forums fostering the use and distribution of content?

o Metric: % of knowledge contributions (i.e., downloads) made by target members

(as opposed to facilitators)

o Metric: % of knowledge contributions (i.e., downloads) with contextualizing

descriptions

o Question: Are the discussion forums fostering active conversation?

o Metric: % of discussion contributions made by target members (as opposed to

facilitators)

o Metric: % of initial posts followed by a meaningful response within 24 hours (i.e.,

not just an acknowledgement)

o Metric: % of conversation threads whose posts contain references to one another

o Metric: % of incidents of unprofessional commentary

36
To assess the Fort Gordon LLC discussion forums, sections of the forums themselves

were analyzed directly. For the Leaders Forum, the discussion and knowledge-posting activity of

particular individuals were analyzed. These individuals were selected based on their class

enrollment. Beginning in the Spring of 2007, students in the Basic Officer Leadership Course

(BOLC), the Basic Non-Commissioned Officer Course (BNCOC), and the Advanced NonCommissioned Officer Course (ANCOC) received an assignment to demonstrate certain

knowledge management proficiencies using the Leaders Forum. These proficiencies included

starting a discussion, participating in a discussion, uploading a document to the forum, and

editing a profile containing biographical information. The intent of this assignment was to foster

interest in and ability to use the eSignal technical forums and the BCKS professional forums.

Students who had received the assignment and who had several months to complete it prior to

the conduct of this investigation were selected to analyze the rate of participation in the

assignment and in the Leaders Forum generally. The total number of students selected was 68

(44 BOLC students, 10 BNCOC students, and 14 ANCOC students).

In the case of the eSignal technical forums, conversations in particular topic areas were

selected for analysis on the basis of their presumed relevance to the forum members and the

present research effort. These topic areas included notes from the field (including lessons learned

from Operation Iraqi Freedom and Operation Enduring Freedom), technical support (for

communications systems and for eSignal itself), and specialty areas for 25B, 25N, and 25S.

Topic areas that were left out included TRADOC Lifelong Learning Center (resource

requirements and master plans), general discussions (e.g., high-level conversation on information

assurance, LandWarNet, LINUX, etc.), and specialty areas not including 25B, 25N, and 25S

(e.g., S6, Warrant Officer, 25U, etc.). The number of conversations analyzed was 179,

representing 18% (179/1003) of the number of conversations currently ongoing at the time of

this analysis and 19% of the number of discussion posts (810/4247).

Summary of Findings

Analysis of the well-established eSignal technical forums revealed steady use of the

forums over the past four years, which have been populated by numerous professional, helpful

conversations. The percentage of registered users participating in discussions was below that

described as typical for active forums (1-3% compared to 25-35%), but this level of activity

reflected a foundation on which the forums can be built further. The technical forums began as

mechanism for providing technical support to LLC users, but have evolved to meet more diverse

needs. Discussions were primarily conducted among users of the forums, rather than by the LLC

technical staff, suggesting that forum members felt some sense of ownership over the

collaborative space. Levels of professionalism were very high, especially in an unmoderated

setting. These characteristics suggest that active cultivation of discussion in the forums would be

successful in further developing the signal learning community.

The Leaders Forum represented a pilot effort for exploring how to cultivate discussion

and other activity using BCKS as the online collaborative space. The focus of this pilot was on

establishing the mechanics of knowledge management, such as the use of collaborative software.

During the course of this investigation, lessons were being learned about what sort of content

could stimulate conversation and what situational characteristics acted to motivate Soldiers to

37
use online discussions to communicate with each other. Activity in the forums reflected

acquisition of the "science" of knowledge management, but the development and implementation

of a strategic plan for integrating existing forums into the knowledge management assignment

would enhance the cultivation of communities of lifelong learners.

Detailed Findings

Adoption of Forums. To explore the use of discussion forums, the percentage of BOLC,

BNCOC, and ANCOC students who demonstrated particular proficiencies in Leaders Forum as

outlined in their knowledge management assignment was determined. These proficiencies, as

listed above, were: (1) start a discussion; (2) participate in a discussion; (3) upload a document to

the forum; and (4) edit biographical information. These proficiencies were selected from a larger

list (e.g., including "know the difference between tacit and explicit knowledge" and "download a

relevant piece of knowledge") because they could be linked directly to specific individuals and

could be assessed by analyzing forum activity. The results of the analysis are shown below, and

include only those students who were registered in the forum [93% (13/14) of the ANCOC

students were not registered in the forum and 18% (8/36) of the BOLC students were not

registered].

Table 4. Knowledge Management Assignment Completion Rate

Proficiency BOLC (N= 36) BNCOC (N= 10) ANCOC (N= 1)

Start 53%(19/36) 10%(1/10) 100%(1/1)

discussion

Participate in 58%(21/36) 0%(0/10) 0%(0/1)

discussion

Upload file 0% (0/36) 10% (1/10) 0% (0/1)

Edit bio 86%(31/36) 10%(1/10) 100%(1/1)

As shown in the table above, the adoption rate of the Leaders Forum was moderate.

Although ANCOC student participation was occasionally at 100%, this sample size was only one

student, and should not be considered a representative sample. The moderate participation rate

was apparently driven by the BOLC students, who completed the assignment at a significantly

higher rate than the BNCOC students. Becoming a member of Leaders Forum is one of the

knowledge management proficiencies that was not assessed because it was unclear whether

students were responsible for registering themselves or if they were enrolled as a group by

someone else. To the extent that registration was voluntary, adoption rates were best for the

BNCOC students (100%). Moreover, all of the BNCOC students were cross-registered as

members of NCO Net - another BCKS forum. High percentages of both BNCOC students and

ANCOC students [90% (9/10) and 93% (13/14), respectively] were potentially cross-registered

as members in the eSignal technical forums.8 Three of the 44 BOLC students (7%) were crossregistered in other BCKS forums and 27% (12/44) were potentially cross-registered in the

8 Cross-registration in the eSignal technical forums was determined by searching the technical forum member list for

the names of the BOLC, BNCOC, and ANCOC students. The technical forums member list did not include

biographical data, so where common names (e.g., Ralph Jones) were found, it could not be verified for certain

whether the student name and technical forum member name belonged to the same person.

38
eSignal technical forums. None of the students in any course made content or conversation

contributions in the e-Signal technical forum discussions analyzed.

The relatively high frequency of forum cross-membership among the BNCOC and

ANCOC students suggests that limited activity among Leaders Forum members was not

necessarily due to unwillingness to access knowledge management resources. An alternative

explanation is that the number of passionate, active participants in an online community typically

represents a small proportion (25-35%) of the member population (Wenger et al., 2002). At least

in the case of the BOLC students, member activity well exceeded this range. Examination of the

posts made by BOLC discussion participants, however, indicated that approximately 36%

(28/77) of the contributions were made primarily to complete the assignment requirement, rather

than to engage the professional community and conduct self-development. For example, one of

these posts said "I'm trying this out for my homework." A chain of posts following an initial post

said "interesting," then "really interesting," then "seriously interesting." It appeared that the

"science" of knowledge management was successfully acquired by these students, as expected,

but that the pilot forum was not yet a ready canvas for the "art" of knowledge management.

A second, more likely alternative explanation is that the students had other, more

effective means for sharing information with their peers. First, the students were co-located at

Fort Gordon, so much of their communication could happen face-to-face instead of online.

Second, limited public computing facilities on post reduced the ease with which students could

actively engage in online discussions outside of the classroom when they were not co-located.

Third, participation in Leaders Forum was part of an assignment and not a natural outgrowth of

interest in what the forum had to offer. The pilot Leaders Forum purposefully was established

largely devoid of content in order to serve as a place for demonstrating and cultivating the

collaborative knowledge development processes. However, the initial content in a forum is

critical for engaging members in active participation (see, e.g., Dixon et al., 2005). Moreover, it

appeared that the content needs of the Leaders Forum target audience could be addressed by

visiting other forums, such as NCO Net or PlatoonLeader.mil.

At the time of conducting these analyses, there were 56,154 registered users in the

eSignal technical forums. Of this number, 299 unique members contributed to the conversations

sampled as part of this investigation, representing less than 1% of the membership. Additional,

unique forum members would almost certainly be represented if all of the conversations had

been sampled, but by the same token, other members who participated in the sampled

discussions would not. Assuming that the same ratio of unique contributors to conversation posts

found in the analyzed conversations [37% (299/810)] also characterized the unanalyzed

conversations, then approximately 3% (1,571/56,154) of registered users contributed to ongoing

conversations. This percentage is less than the 25-35% active participation rate cited in Wenger,

et al (2002). In the discussions analyzed, roughly the same number of posts were made in 2004,

2005, 2006, and 2007, which indicated a steady trend of conversation activity in the technical

forums.

Connections. Seventy percent (33/47) of the Leaders Forum members (representing

primarily BOLC students, N = 31) edited their biographical information beyond the basic

required information for the registering in the forum. The required biographical information

39
included rank, branch, duty status, profile (name and email), key words (to be used as search

terms when finding people in the forum), and willingness to share knowledge (true/false). Nonrequired information included 19 additional data entry fields, including spoken languages,

expertise, theater deployments, reason for joining the forum, etc. The middle column of Table 5

below shows the percentage of forum members who filled out each of these additional fields.

Table 5. Percentage of Non-required Data Fields Filled Out

Non-required Percentage Data Entry, Non- Percentage Data Entry,

Data Entry Field Required Biographical Information All Data Entry Fields

(Total N = 47) (Total N = 47)

Middle Initial 74% (N= 35) 74% (N= 35)

Reason For Joining Forum 49% (N = 23) 40% (N = 19)

Mobile Phone 49% (N = 23) 49% (N= 23)

Functional Area/MOS 45% (N= 21) 47% (N = 22)

Education 38% (N= 18) 38% (N= 18)

Grade/Rank or Title 38% (N= 18) 62% (N = 29)

Spoken Languages 36% (N= 17) 36% (N= 17)

Work Phone 23% (N= 11) 23% (N= 11)

Job Experience 21% (N= 10) 21% (N= 10)

Theater Deployments 21% (N= 10) 23% (N= 11)

DSN 15% (N= 7) 15% (N= 7)

ASI 9%(N=4) 9% (N= 4)

Fax 6% (N= 3) 6% (N= 3)

Expertise/Competencies 4% (N = 2) 13% (N = 6)

Additional Background 2% (N= 1) 26% (N= 12)

Title 0% (N= 0) 47% (N= 22)

Homepage 0% (N= 0) 0%(N=0)

Fax 0% (N= 0) 0% (N= 0)

ICQ Number 0% (N= 0) 0% (N= 0)

Asshown in this column, 14 of the 19 non-required biographical information fields were

used. Of these fields, the percentage filled out ranged from 2% to 74%, with the majority (58%)

of percentages being 15% or higher. Notably, the Expertise/Competencies field only had a 4%

fill rate, whereas Middle Initial field had a 74% fill rate. Alone, however, these unanalyzed

percentages did not accurately reflect the degree to which the Leaders Forum pilot facilitated the

growth of social connections.

First, the user audience should be considered when interpreting what the above metrics

mean. For example, the majority of the Leaders Forum members in the present sample (46/47)

were near the beginning of their Army careers and so likely did not have theater deployments or

expertise to report. The relatively detailed background and expertise description provided by the

single ANCOC forum member relative to the limited descriptions provided by the BOLC and

BNCOC students suggests this is a possibility. Second, some of the important information asked

for in the required fields had been entered into the required Keywords field, including entries for

MOS, expertise, and additional background. When these entries were taken into account, the

percentage of forum members providing information increased. Third, the nature of the

40
information provided in the non-required data entry fields should be considered. For example,

30% of the additional information provided in the required Keywords field was unrelated to the

business of the forum (e.g., "Married" and "thorough and open-minded"). Removing these

entries reduced the percentage of forum members providing information. The revised

contribution rates are shown above in Table 6 above in the right-hand column. These revised

percentages reflect a moderate degree of facilitation of social network growth, but it should be

noted that the majority of entries in the biographical information fields were minimal (e.g., oneword entries). Here too, it appears that the students using the Leaders Forum effectively learned

the mechanics of knowledge management, which was a goal of this pilot forum, but that the

development of professional community would require additional effort.

None of the user profiles in the technical forums contained biographical information. The

design of the technical forum user profile did not permit the entry or display of such information.

Another reflection of social network development may be found in the referrals that

discussants make to themselves or others who could provide additional information or expertise.

None of the posts in the Leaders Forum referred conversation participants to others. Thirty-eight

percent of forum members sampled indicated that they were unwilling to share information,

although it should be noted that it is unclear whether this selection was a conscious choice or the

unintentional acceptance of a default setting (e.g., some members who selected "false" to

characterize their willingness to share information provided contact information). In the sampled

discussions in the technical forum, 11% of posts referred others to the poster or some other

person for additional expertise or information.

Context. Both the eSignal technical forums and the Leaders Forum were examined for

the presence of "built-in" opportunities to stimulate discussion. Such opportunities stand out in

contrast to "organic" conversations that arise out of regular activity in the forum. Instead, built-in

opportunities are organized events such as digital stories, book reviews, collective exercises, and

the like, which are released in the forum purposefully to generate a shared experience and get

conversation going (Cianciolo, Cianciolo, Prevou, & Morris, 2007). Built-in opportunities for

discussion may be contributed by facilitators and forum members and help to build professional

community and stimulate the exchange of tacit knowledge. Neither the technical forums nor the

Leaders Forum included these manufactured opportunities. The technical forums did include a

small handful of polls, which may be used to give members a sense of their community

demographics and opinions, but without discussion these do not serve as a way to generate

shared experience.

It should be noted that in some sense the technical forums were not designed to facilitate

the development of professional community, but more to serve as information portals for people

with explicit technical questions. It may be the case that the purpose of these forums is not

served by introducing built-in opportunities for discussion because users have opportunities to

develop professional community in other forums, such as NCO Net, PlatoonLeader.mil, and

CompanyCommand.mil. Although in a pilot stage, the Leaders Forum ultimately is intended to

foster the development of community and knowledge sharing skills, so these engineered

opportunities will be useful for stimulating activity and shared experience.

41
Content. Two metrics were assessed to explore the quality of the content in the forums.

These metrics were selected to determine the relative activity of forum members compared to

facilitators in posting content and the degree to which content posted to the forum was

actionable. Although it is important for facilitators to post content to a forum, it is an important

sign of community health that forum members share the responsibility for disseminating

information and solutions. Without this kind of balance, the forum becomes a knowledge

repository designed in a top-down fashion rather than an emergent body of knowledge created

through professional exchange. Actionable content is content (e.g., a training plan, a

presentation, a standard operating procedure, etc.) that is supported with contextualizing

information. Contextualizing information is critical for potential users to understand how and

when such content may be applied (Dixon et al., 2005) and includes descriptions of what the

content is, when it was created, how it was used, who used it, and so on.

In the Leaders Forum sample studied, one forum member posted content, but it was not

coupled with contextualizing information. In the technical forums no content uploads were made,

although several posts included pasted content from external sources or links to external sources.

More in-depth analysis of content quality was not conducted because relatively few contributions

were made by the samples studied.

Conversation. Four metrics were used to assess conversation activity and quality.

Conversation activity is reflected in the length of time between posts, particularly posts that

initiate a discussion, asking for help or opinions. These posts should be followed up with some

kind of acknowledgement and initial comment within 24 hours in order to foster confidence in

forum users that the forum can meet their needs. Conversation quality is reflected in the degree

to which conversations have a professional tone, reflect the active participation of the community

(as opposed to just facilitators), and feature links to one another that represent exchange among

professionals. Some additional content analysis based on the rubric presented in Cianciolo et al.,

(2006) was performed. The purpose of this analysis was to investigate the utility and

effectiveness of the conversation.

Of the 179 initiating discussion posts analyzed in this investigation, 44% were responded

to within 24 hours, reflecting a relatively low rate of conversation activity (90% or more is

ideal). Less than I% of these posts (N = 6) had an unprofessional tone, reflecting a high level of

professionalism in this unmoderated forum. Unprofessional tone was defined as saying negative

things about another person or group of people. Twelve percent (97/810) of the discussion posts

referred to other discussion posts in a thread. In the discussions analyzed, participants seemed to

make an active effort to summarize or clarify the discussion by referring to what others said.

Seventeen percent (N = 138) of posts were made by facilitators, which indicated that the large

majority of ongoing discussion was conducted by forum members.

To analyze conversation utility and effectiveness, discussion-initiating posts were first

separated according to whether they received a response (no response = 1 post-thread; I or more

responses = multi-post thread). Next, the initiating posts were categorized into types,

corresponding to the post type definitions provided in Cianciolo et al., (2006). These post types

were: Invitation to Build "New" Knowledge, Request for Input, Directed Question, Request for

Knowledge Resources, Request for Expert, Information Post, and Unknown. Two additional post

42
types, Rant and Inferred Question, were added for the present analysis. A Rant was defined as a

post that points out deficiencies of some kind in a process, policy, or procedure but that is not

explicitly seeking feedback or broader opinion. An Inferred Question was defined as a post that

does not explicitly ask a question but that indicates the poster is having a problem of some kind,

perhaps a form or requesting help that is typical of technical support forums (e.g., "I click the

link and it goes to the wrong page."). Table 6 below shows the breakdown of initiating post type

within both types of thread (1-post and multi-post).

As shown in the table, 15% (26/179) of the initiating posts asking questions did not

receive a follow-up post in the forum. The majority (92%) of these posts were inferred questions,

requests for experts, and requests for resources. The majority of initiating posts were followed by

a response. The average number of posts in a multi-post discussion thread was 4, ranging from I

to 55. Directed questions and requests for input were especially effective at generating responses

in the forum. Requests for resources also generated a response more often than not, although

requests for experts did not.

Table 6. Type of Initiating Posts for One-Post and Multi-Post Threads

Post Type # Initiating % of Total # Initiating Multi- % of Total Post_Type1 -Post Threads Post Threads

Directed Question 0 0% 55 36%

Invitation to Build "New" 1 4% 1 1 %

Knowledge

Information Post 0 0% 9 6%

Inferred Question 11 42% 16 11%

Request for Input 0 0% 39 25%

Request for Resources 4 15% 19 12%

Request for Expert 9 35% 5 3%

Rant 1 4% 6 4%

Unknown 0 0% 3 2%

Total 26 100% 153 100%

The utility or effectiveness of a multi-post thread was determined by applying the criteria

outlined in Cianciolo et al. (2006) for each type of initiating post. The criteria capture whether

the multiple posts in a thread adequately addressed the question raised (or information shared) by

the initiating post. For example, the initiating post types of Invitation to Build "New"

Knowledge, Request for Input, Rant, and Information Post were examined for the degree to

which they stimulated participation among the broader population. The initiating post types of

Directed Question, Request for Expert, Request for Knowledge Resources, and Inferred Question

were examined for the degree to which the question posed was actually answered inside the

forum. Using these criteria, 77% (137/179) of the initiating posts were followed by useful and

effective conversation.

43
Recommendations

The technical forums have a solid foundation on which to build further participation and

to stimulate the evolution of the forums' purpose from technical support to a professional

development learning community. Activities that may increase participation in the technical

forums should enhance awareness of ongoing activity in the forum, to include featuring new and

hot conversations on the main eSignal page and encouraging members to use the forum for

professional exchange. LLC staff members who serve as forum facilitators could further fuel

activity by conducting periodic needs analyses and addressing these in the forum with "guest"

discussants.

The Leaders Forum pilot reflects the critical recognition that knowledge management

competencies are an important determinant of lifelong learning. The recommendations presented

here are intended to help the Fort Gordon LLC develop both the "art" and "science" of the

horizontal information sharing process. Two key lessons learned that should be carried away

from the current status of the pilot effort are (1) content plays a key role in stimulating discussion

and community development; and (2) people use the most expedient mode to communicate with

one another. Greater success in stimulating culture shift and adoption of knowledge management

would be achieved by using the knowledge management assignment to guide students'

interactions with well-established forums that connect them with the larger, distributed

professional community.

In order to reduce the likelihood that unproductive conversations are conducted by novice

students in the professional community, the assignment could instruct students to pull content

from the established forums and then distribute and discuss it in the Leaders Forum. Consistent

with the current approach, such facilitation could be accomplished in a classroom setting, and

could be implemented as a one-day seminar when students are exposed to and enrolled in the

appropriate BCKS forums. To stimulate participation, the students could be given class time to

explore the BCKS forums, to use the Leaders Forum to share what they learned, and to have live

conversations about items of particular interest they discovered. Increased levels of instructor

moderation of the Leaders Forum would also encourage discussion as well as ensure that the

culture of knowledge management is understood and adopted by the students.

Assessment of Leader Education

The Sample - Signal Captains' Career Course (SCCC)

The SCCC serves as a representative example for exploring how leader education can be

conducted using LLCs. Leader education is a component of most proponent schoolhouse course

offerings, so it is useful to understand how the technologies that constitute an LLC not only

permit the cost-effective, wide-reaching delivery of course materials but also enable equivalent

learning experiences across diverse learning environments. An initial analysis of LLC impact on

leader education (intermediate-level education for field grade officers) was presented in

Cianciolo (2007), but in that study greater emphasis was placed on the equivalence of learning

content vice equivalence of learning experience or environment. Although the same tasks and

standards may be applied to both classroom and blended modes of education delivery,

44
differences in instructional strategies and learning contexts may moderate educational impact

(Clark, 1994; Firdyiwek, 1999). The potential for cost savings associated with delivering blended

instruction, such as reduced housing and student compensation costs, is widely recognized. In the

present research, it was necessary to explore the application of metrics that capture instructional

strategy and learning experience and their impact on learning performance.

Broadly speaking, the SCCC prepares signal captains to lead a company-sized signal unit

and to serve as the signal officer (S6) on a combined arms staff (see, e.g., North, 2006). The

recently revised SCCC consists of a TRADOC-prescribed common core and seven instructional

modules that cover general knowledge (e.g., general leader skills, personnel management,

logistics, etc.), signal theory, information technology, information management, Department of

Defense communications, network management, and combined arms planning (North, 2006). In

the resident version of the course, these topics are covered via a combination of face-to-face

lecture, hands on training, practical exercise, and some computer-based instruction. In the

blended version of the course, the common core and seven modules are allocated to five phases

of instruction--three of which are individual distance learning (Phases I, II, and IV) and two are

resident phases held at the Signal Center (Phases III and V)--and a single block of selfdevelopment.

Assessment Questions and Method

In the present research, outputs and outcomes associated with equivalence of resident

SCCC and blended (distance learning + resident) SCCC were explored in order to determine the

educational impact of delivering leader education via a combination of resident and distance

learning (DL) formats. Specifically, the following questions were asked:

o Question: Does administering the SCCC online enable uniform access to training

content?

o Metric: Content equivalence across resident and online learners

o Metric: Instructional strategy equivalence across resident and online learners

o Question: Does administering the SCCC online produce uniform learning performance

across resident and online learners?

o Metric: Performance equivalence across resident and online learners

A combination of methods was used to explore these questions, including interviews,

archival data analysis, focus groups, and classroom observation. Interviews were conducted with

two instructors who oversaw SCCC-blended, including administration, content delivery, and

performance evaluation during both distance learning (DL) and resident phases of the course.

One of these instructors also conducted performance evaluation during the capstone exercises of

SCCC-resident. Brief, informal discussions also were held with two SCCC course developers

and a more in-depth discussion was held with an Army Training and Doctrine Command quality

assurance officer who was concurrently evaluating the content equivalence of SCCC-resident

and SCCC-blended. Overall, the purpose of these interviews was to determine the nature of

instruction in the two versions of the SCCC.

45
Archival data analysis consisted of examination of the SCCC-resident and SCCC-blended

programs of instruction and course "crosswalk" diagrams. 9 Also studied was the course map for

SCCC-blended. The purpose of the archival data analyses was to explore the content

equivalence of SCCC-resident and SCCC-blended. An additional purpose was to determine the

different modes of instruction (e.g., hands on training vs. equipment simulation) used to deliver

SCCC content (i.e., instructional strategy equivalence).

Two focus groups (N = 7 and N = 8) were conducted with SCCC-blended students. The

purpose of these focus groups was to explore further instructional strategy equivalence. The

questions asked of focus group participants centered on their experiences during the distance

learning phases of the course. Specifically, the questions asked about (a) interactivity among

students and with the instructor while distributed; (b) usability/access issues associated with

online content delivery; (c) impressions of content, delivery, and learning achieved; (d) tradeoffs

involved in completing coursework at home vs. at a schoolhouse; and (e) participation in the

block of voluntary self-development courseware.

Classroom observations were conducted during the combined arms staff exercise held at

the end of both SCCC-resident and SCCC-blended. In the combined arms staff exercise,

students role-played the members of a Stryker Brigade Combat Team staff and conducted the

military decision-making process to form a plan for addressing stability operations in

Azerbaijan. The mission analysis brief and course of action brief of two groups of SCCCblended students and one group of SCCC-resident students were assessed using the Tactical

Thinking Behaviorally Anchored Rating Scales (Phillips, Ross, & Shadrick, 2006) as a guiding

framework. Specifically, the briefs were assessed for the stage of cognitive development

(novice, advanced beginner, competent, proficient, and expert) reflected in the slides presented

and associated discussion. General observations were written down and then collectively

associated with a stage of development.

It should be noted that the sample of SCCC student groups interviewed and observed may

not be representative and that a larger sample is desirable for drawing strong conclusions. The

findings based on this sample are consistent with previous research, so it is possible to conclude

that this sample is not an outlier.

Summary of Findings

The present analysis indicated that blended learning solutions, partially hosted in the Fort

Gordon LLC, enabled roughly uniform distribution of leader education content across learner

types. Variation in learning experience across each type of learner was identified, however this

dissimilarity did not appear to create meaningful differences in performance during the capstone

exercise observed. Student groups having received both types of instruction demonstrated novice

levels of performance, with slightly more advanced cognitive development demonstrated by the

resident group. The blended group did require coaching, however, in order to match the level of

performance demonstrated by the resident group.

9 Crosswalks compare both versions of the course, showing areas of shared/unique topic coverage and

similarities/differences in recommended hours for completion.

46
Differences in learning experience stemmed primarily from a lack of interactivity among

the student and his or her instructors and peers during the DL phases of blended instruction and

from a lack of incentives and processes in place to reward effective self-instruction strategies for

distance learners. As noted in Leonard, Winkler, Hove, et al. (2001), implementing procedures

for resourcing, supporting, and encouraging at-home instruction is a critical contributor to the

success of DL. Although the performance implications associated with unequal learning

experiences were not seen in the present research, learner dissatisfaction with DL may foster

negative attitudes towards the capability of the Army to train effectively and the Army's

motivations for providing training at home. Negative attitudes such as these may reduce affective

organizational commitment and individual readiness.

Detailed Findings

Equivalence of Course Content. Examination of the program of instruction for both

versions of the SCCC largely indicated course content equivalence. That is, the same course

content was made available to all learners, regardless of location, and involved roughly the same

amount of recommended hours (760 hours for resident, 670.5 hours for blended)' 0 . Differences

in course content actually received arose when (a) there were usability or access problems with

the distance learning technologies; and (b) required components in the resident version of the

course were voluntary in the blended version of the course.

First, interviews with instructors revealed that the TRADOC-prescribed common core

component, administered online for students taking the blended version of the course, had so

many technical problems that many students have been allowed to graduate without having

completed the common core. The TRADOC prescribed common core accounts for 64 hours of

instruction (18 topics), covering branch-general topics such as equal opportunity, law of war, and

information assurance. Resident students, in contrast, take the common core via computer-based

instruction delivered in the schoolhouse, avoiding online delivery problems. For this reason,

resident students complete this component of the course whereas blended students may not.

Students interviewed in focus groups most frequently reported that they had begun the SCCC

prior to its conversion to the five-phase format in 2006, so they did not take the common core

online. Of two interviewees who did report taking the common core, one reported significant

technical difficulty and one did not. Focus group participants also reported technical difficulties

when accessing other online portions of the course, but these problems did not prevent them

from completing the coursework.

Second, some components of the seven instructional modules were taught in a classroom

setting for resident students whereas SCCC-blended students received these same components as

voluntary self-development (totaling about 161 hours of instruction). It may be the case that

distance learning students complete the voluntary self-development, but given the large number

10 Differences in hours between the two versions of SCCC do not reflect the subtraction of a particular annex or

annexes of instruction. Rather, small differences accrue over content areas such that although SCCC-blended has

more recommended hours for some topics, SCCC-resident overall has a greater number of recommended hours. In

general, SCCC-blended topics taught in residence have several fewer (--40 hours per topic) recommended hours than

the same topics taught in SCCC-resident. This difference enables SCCC-blended students to return more quickly to

their civilian jobs and families.

47
of hours necessary to complete the required distance learning coursework in the context of

civilian job and Family demands, it is likely that few actually do. Focus group participants

unanimously reported that they did not complete any of the self-development coursework. Some

of the people interviewed did not know that such coursework was available.

For these two reasons, differences in course content actually received represented

somewhere between 161-225 hours of instruction and, more importantly, 16-34 content topics. It

should be noted that prior to the introduction of online instruction, the non-resident portion of the

blended-SCCC was conducted via mail correspondence. The course was conducted in two

phases instead of five, with the second phase conducted in residence at Fort Gordon over two

weeks. Blended-SCCC students received a great deal less content than resident students, and

what content they received was strictly text-based without interactivity. In focus groups and

interviews, this educational context was reported as less preferable by students and course

developers alike. Lack of equivalence in course content among the two current versions of the

SCCC should be considered in this light.

Equivalence of Instructional Strategy. The application of an effective instructional

strategy that leverages technical capability is critical for achieving the educational impact made

possible via blended learning solutions for leader training (Clark, 1994; Firdywik, 1999). In the

case of blended learning, the conditions must be set for effective self-instruction strategy.

Leonard, Winkler, Hove, et al. (2001) outlined five methods for ensuring the effective

implementation of blended learning, as follows:

1. Carefully selecting segments of instruction amenable to instructorless or distributed

learning

2. Using appropriate and relevant instructional media

3. Ensuring that sufficient changes are made to existing processes and support activities

(e.g., to enable instructor support for online learners)

4. Providing adequate resources to implement blended learning (e.g., technology, technical

support personnel)

5. Providing sufficient incentives for students, commanders, and supporting activities (e.g.,

human resources) to play their proper roles

Although not all of these methods are conventionally associated with instructional

strategy, they do affect the quality of self-instruction through the availability of (a) quality

courseware; (b) instructors and other support to interact with distributed students; and (c) time

set aside specifically for online learning.

Examination of the methods used to convey course content indicated rough equivalence

between the resident and blended SCCC. First, segments of the curriculum that required group

exercises and collaborative learning (e.g., staff exercises, network management) were conducted

as resident phases in blended-SCCC, with individual leader and technical skills (e.g., staff

knowledge, signal theory) reserved for online instruction. Online instruction largely was

delivered via narrated PowerPoint lectures and interactive courseware, including some

equipment simulation, reflecting the appropriate media selections.

48
Despite the application of these methods, significant differences in learning experience

did exist, and were related to both the instructorless nature of the DL segments of the course and

the instructional media used. One noteworthy difference between the two versions of the SCCC

was the degree of student interaction with instructors and with each other while learning the

content covered in the online phases of the blended SCCC (Phases II and IV). Students in both

focus groups unanimously agreed that greater interaction, particularly with instructors, during the

distributed phases of instruction would be desirable. Specifically, they reported being more

engaged by "authentic" or generative assessment activities (e.g., creating a memorandum) in

which they produced an actual leader product and received feedback directly from the instructor.

They did not feel that the assessment activities in the interactive courseware were useful for their

learning or retention. They reported that these assessments were too easy to answer correctly

without complete knowledge or proficiency with the content. Students also reported technical

difficulties with scoring interactive courseware assessments, such that correct answers were

counted as wrong and vice versa.

Students also desired greater interactivity with each other through group assignments on

leader tasks not currently covered by the instruction. They did not wish for synchronous

interaction, citing scheduling challenges associated with conducting widely distributed group

activities. Nor did they wish for longer resident phases. Rather, they expressed a desire for

content that was more targeted to their needs as leaders and managers and for interaction with

others as appropriate to facilitate such learning. The collaborative capabilities of the software

used by the Fort Gordon LLC to deliver curriculum materials (i.e., Blackboard) permits

discussion and community development among distributed learners, although it is not currently

used in this way. Proactively leveraging collaborative capabilities may enhance the social aspects

of distance learning, provide a straightforward way for students to find and help each other, and

may increase the number of SCCC-blended students who actively engage in course materials

through interaction with the instructor.

Another important difference observed between the learning experiences of resident- and

blended-SCCC students was tied to the methods noted by Leonard, Winkler, Hove, et al. (2001)

that relate to providing support and incentives to enable online instruction. Specifically, students

reported not being compensated for the time they spent completing online coursework, nor were

they given set-aside training time by their commanders or employers to focus on education. As a

result, students completed the minimum necessary amount of course content as quickly as

possible when they could work it around other demands. The methods they reported using

included: staying up late, taking vacation or sick days off of work, putting off instruction until

the last minute and cramming, and simultaneously doing coursework and recreational activities

(e.g., watching TV). Focus group interviewees unanimously felt that non-resident instruction was

beneficial but that the way it had been implemented placed a significant burden on the learner.

Although it was not explicitly assessed as part of this research effort, it is likely that such a belief

would not enhance affective organizational commitment, and could possibly have a negative

impact.

Similarly, instructors did not receive assistance or time set aside for actively facilitating

online learning, which can be very time consuming but essential for effective distributed

instruction (Abell, 2000; Hiltz, 1998; Kreijns, Kirschner, Jochems, 2003). The current design of

49
most of the online courseware used in blended-SCCC obviates the instructor except as a course

administrator. Although it is recognized that instructorless education would save money, there is

a potential cost-benefit tradeoff that is worth exploring.

Equivalence of Learning Performance. The revealed differences in learning experience

raise questions about the equivalence of learning performance across resident- and blendedSCCC students. As described previously, both resident- and blended-SCCC capstone staff

exercises were observed in order to investigate learning performance. This exercise was selected

because it was held at the end of the course, thus increasing the likelihood that any observed

differences would be due to prior instruction. In both the resident and blended versions of the

SCCC, the combined arms staff exercise follows training in staff knowledge and requires

students to play staff roles with which they may be unfamiliar. The mission analysis and course

of action development phases were observed, along with their associated briefings to the

instructor.

Some differences in how the capstone exercises were conducted for resident- and

blended-SCCC students should be described. First, blended students were allocated one less day

to complete the exercise. These students made up for the missing day by working longer hours

on the days they conducted the exercise. The result was that resident students worked for five,

approximately 8-hour days and the blended students worked for four, approximately 10-hour

(and sometimes longer) days. Second, the instructor was much more involved in guiding the

exercise and providing mentorship with blended students than with resident students. Third, the

two blended student groups observed (N = 7, N = 8) were slightly smaller than the resident

student group (N= 9) and the resident group had a foreign national student from Azerbaijan, the

country serving as the area of operations for the exercise. Fourth, the groups had differing

numbers of members who had served on a battalion staff (N = 4 and 2 for the blended groups and

N = I for the resident group). Fifth, and finally, the groups differed in how seriously they took

the exercise. The resident group was much more relaxed and unconcerned about their

performance than either of the two blended groups to whom it meant a great deal to perform

well.

Table 7 below shows examples of the behaviors sampled during classroom observations

and the assignment of a level of cognitive development. As shown in the table, the performance

of both groups largely reflected reliance on rules (e.g., doctrine, the operations order), variability

in analytical capability, and focus on key mission factors somewhat in isolation. According to

Phillips, Ross, and Shadrick (2006), these behaviors reflect a "Novice" level of cognitive

development. A Novice level of cognitive development is to be expected from the students at this

stage in their careers given very limited experience with the military decision making process

and combined arms planning. Students at this stage had not yet acquired the knowledge that

enables more analytical and flexible thinking, and were asked to play roles that they will not play

in the operational environment.

One exception to this observation was course of action development as conducted by the

resident students. In this phase of the exercise, the resident students demonstrated some ability to

recognize meaningful elements in the information provided and specify their implications,

particularly for enemy action. They also went beyond established rules to address unknowns

50
about enemy strength. For this reason, their performance in this phase was assigned an

"Advanced Beginner" rating.

Table 7. Capstone Exercise Performance of Resident- and Blended-SCCC Student Groups

Student Exercise Phase Example Behaviors Level of Cognitive

Group Development

- High-level descriptions of geography, population,

and political/economic situation (i.e., regurgitation

of operations order)

- Limited enemy analysis (i.e., goals not explicitly

stated)

- Battlefield effects analysis identified most key

Mission Analysis implications of weather and terrain Novice

(MA) - Facts and assumptions moderately well identified

and articulated - Implied, specified, and essential tasks reflected

some conflation of the mission and essential tasks,

some overlap in specified tasks among warfighting

Resident functions, recognition of constraints satisfactory

- Some assumptions reflected lack of knowledge

- Implications of enemy courses of action well

articulated

- Good analysis of enemy patterns

- Enemy adaptations to friendly strengths seen as

Course of Action "weaknesses" instead of thinking enemy tactics

(soAD - Effective analysis of differing tactics among Advanced Beginner Development (COAD) differing threat groups and most likely and

dangerous COA

- Modified force ratio matrix to address unknowns

- Evaluation criteria with arbitrary weights that

couldn't be justified

- Significant amount of time (1+ hrs) spent reading

doctrinal manuals prior to beginning analysis

- Some conflation of facts and tasks

- Instructor provided guidance on considering all

assets, information requirements, and facts vs.

Mission Analysis assumptions Novice

(MA) - Slide organization deviates from normal procedure,

presentation somewhat confusing

- Commander's Critical Information Requirements

Blended cut and pasted from OPORD

- One group to do significant revisions of MA brief

- Restated mission one level of analysis too high

- Instructor advised students not to exert command

and control too many levels down

Course of Action - Conflation of the mission, purpose, and tasks to be

accomplished with the COAs Novice

Development (COAD) - Levels of analysis at the platoon level (i.e., too

low)

- Good evaluation criteria

The data in Table 7 suggest that there was little practical difference in the performance of

blended- and resident-SCCC students. Students achieved similar levels of performance, although

blended student groups received a great deal more mentoring and guidance from the instructor to

51
achieve the level of performance they did. One of the blended groups had to redo significant

portions of their mission analysis before completing their brief to the instructor.

Recommendations

Solutions for building interpersonal interactivity into online courseware generally are not

simple, nor are they inexpensive. Instructor time is required for collaboration and additional

course developer time (and expertise) is required to build computer-based training that has

interactivity as its key mode for delivering instruction and conducting assessment. Blackboard

currently supports interactivity among instructors and students, but procedures must be put into

place and faculty developed in order to leverage these capabilities. In the case of leader

education, as in the case of MOSQ instruction, the development of instructors, course

developers, and the overall technology-assisted instruction process can only be accomplished

through close coordination and well-defined roles among resident instruction managers and the

LLC staff. In the MANSCEN LLC, courseware development is a critical function of the LLC

staff, a model potentially worth exploring for the Fort Gordon LLC.

Assessment of On-Demand Training

The Sample - Mobile Training Teams and Unit Universities

As stated previously in this report, much of what the Fort Gordon LLC offers to lifelong

learners may be considered training on demand. Simulation downloads, online courseware, and

discussion forums all enable 24/7 access to learning content at the initiative of the learner.

However, in order to clarify the presentation of this report, these aspects of the Fort Gordon LLC

have been considered separately. Mobile training teams and Unit Universities represent outputs

uniquely produced by the Fort Gordon LLC, whereas simulations, blended learning, and

knowledge management have been implemented extensively throughout the Army. For this

reason, mobile training teams and Unit Universities require their own, novel assessment criteria

and focused analysis.

The critical function served by On-Demand Training is the provision of MOS

sustainment training independently of the institutional education cycle. The purpose of OnDemand Training is to support the Army's apprenticeship model of instruction, enabling

pedagogically sound and technologically advanced on-the-job training. It is possible for OnDemand Training to improve the cost-effectiveness of training, but it is more important for

enhancing mission readiness because it serves as an alternative to institutional modes of

instruction. Enhanced mission readiness stems from the accelerated skill development and

increased retention enabled by just-in-time instruction.

Assessment Questions & Method

The present research assessed the output and outcomes associated with the use of OnDemand Training for enhancing proponent outreach. Specifically, the following questions were

asked and associated metrics used to focus data collection:

52
o Question: Is On-Demand Training being adopted?

o Metric: % of target audience registered to use the system

o Ouestion: Does On-Demand Training enable just-in-time learning?

o Metric: % of registered users who state that training received was timely and

could not have been possible any other way

o Question: Is On-Demand Training cost-effective?

o Metric: Reduction in cost to conduct mobile training teams before and after LLC

implementation

These questions were answered through a combination of interviews and review of

archival data. Interviews were conducted with the Fort Gordon LLC Extension Campus

Coordinator, the LLC Program Manager, the Signal Center Chief of Resident Training

Management, the Chief of the Training Development Cell located at the Signal Center NCO

Academy, and ten training supervisors who established or oversaw Unit Universities with the

Fort Gordon LLC. Interviews with training supervisors were conducted in order to capture

concrete information about how the Unit Universities enabled just-in-time training and enhanced

readiness. Archival data reviewed included cost-effectiveness analyses conducted by the Fort

Gordon LLC staff and mobile training team cost data provided by the Signal Center Chief of

Resident Training Management.

Interviews conducted with Unit University training supervisors represent a relatively

modest sample due to several factors. First, three Unit Universities were used to conduct nonresident 25B10 instruction (i.e., they supported the High-Tech Regional Centers at Tobyhanna

and Sacramento and the Professional Education Center in Little Rock), which was discussed

previously in this report. These Unit Universities were not considered part of On-Demand

Training, bringing the sample number down to 44. Second, of the seven original Unit

Universities (up and running by November 2006), two of the associated units had been

deactivated. Third, of the remaining 42 Unit Universities, 18 training supervisors were contacted

and eight did not respond to a request for an interview. Fourth and finally, of the ten training

supervisors contacted, five indicated that their Unit Universities were too recently established to

have much to say about impact. Because the remaining 24 (42-18) Unit Universities had also

been very recently established, further attempts to contact training supervisors were suspended.

Summary of Findings

Unit trainers interviewed unanimously and enthusiastically endorsed the importance of

Unit Universities to enhancing their units' mission readiness. The LLC staff has launched as

many Unit Universities as personnel and hardware limitations would allow, with recent upgrades

to hardware poised to accelerate the process in the immediate future. Most of the unit trainers

interviewed had newly established Unit Universities, but easily cited the ways in which they

thought training on demand would provide unique and cost-effective opportunities for refresher

and new equipment training. Mobile training teams supported by the LLC technologies enabled

at least partial cost reduction to the Signal Center's ongoing efforts to reach out to operational

units in need of pre- and post-deployment training. The delivery of training to a deployed unit

via a virtual mobile training team represented a unique mode of proponent outreach that made

53
cost-effective use of existing LLC personnel and schoolhouse experts through close

coordination.

Kinney (2005) presented some external factors that unit trainers cited as important for

enabling lifelong learning in units. Among these factors were time to train, command emphasis,

and incentives to train during down time. All three of these factors also were cited by

interviewees as currently limiting the impact of Unit Universities. Other limiting factors cited by

unit trainers were lack of awareness of Unit Universities and the difficulty associated with

acquiring course content that is not readily available online. Although the first three factors are

outside the sphere of influence of the LLC and the Signal Center, these additional factors are not.

They could be addressed through closer coordination among content developers and providers

and increased marketing outreach of the lifelong learning concept.

Detailed Findings

Adoption of On-Demand Training. At the time this analysis was conducted, the Fort

Gordon LLC hosted 47 Unit Universities. The Extension Campus Coordinator estimated that 180

Unit Universities would be necessary to reach every unit in the Signal Corps and that more than

600 would be necessary to reach signal and non-signal units who could benefit from reach back

to the Signal Center. Forty-seven Unit Universities represents approximately 30% of the Signal

target audience and 8% of the Army-wide target audience, a modest percentage that has grown

significantly in the past year (1% to 8%). According to the LLC staff interviewed, constraints on

the growth of the Unit University initiative have been due largely to limited server space to host

Unit Universities (which produced concerns about backup activities) and lack of staff time

available to get the word out about On-Demand Training. Both LLC staff members interviewed

reported a surge in interest in Unit Universities following visits by the former Deputy Chief of

Signal to bring word of LandWarNet eSignal to operational units. Recent acquisitions of new

hardware should enable the hosting of more Unit Universities. If the present rate of growth (i.e.,

roughly six times the number of Unit Universities in one year as were present the year previous)

continues, the Army-wide target audience should be reached within approximately 2 years. A

more realistic estimate, given current limitations in personnel and courseware availability would

probably be about 10 years.

It is difficult to estimate an appropriate adoption rate for mobile training teams, given the

spontaneous nature of requests for On-Demand Training and the potentially very broad target

audience. For live mobile training teams, LLC capabilities are used more as a facilitative

capability (i.e., to save printing and reproduction costs) than as an enabling capability (i.e.,

making it possible to send out a mobile training team). The Signal Center Chief of Resident

Training Management stated that it can be most cost-effective to conduct training at the

schoolhouse, so emphasis is placed on maximizing the ability of units to come to Fort Gordon.

For this reason, effective use of the Fort Gordon LLC to deliver blended learning, among other

services, may actually help reduce the frequency of mobile training team deployment.

One way to estimate the adoption rate for virtual mobile training teams is to compare the

frequency of usage compared to live mobile training teams. The Chief of Resident Training

Management reported that since October 2005, the Signal Center sent close to 20 live mobile

54
training teams per year. Since its inception in 2002, Fort Gordon has deployed one virtual mobile

training team. As with Unit Universities, staffing shortfalls and limited marketing activity appear

to have contributed to modest adoption rate. It is not widely known that the Fort Gordon LLC

can provide virtual mobile training teams. Even so, significant, close coordination among LLC

staff and Signal Center instructors and course developers would be necessary to prepare the

appropriate training content.

Just-in-time training. Of the ten Unit University training supervisors interviewed, five

indicated that their Unit University was too newly established to yet have an idea of its impact.

Nine of the 10 interviewees were very enthusiastic about the possibilities for their Unit

University to enable MOS sustainment training and reach back to the schoolhouse." These

interviewees reported that they established a Unit University to enhance mission readiness

through (1) training on new equipment assigned to the unit; (2) training on MOS-specific skills

that were needed for deployment, but that could-not be trained feasibly in the schoolhouse (e.g.,

not enough seat reservations); (3) increased confidence enabled by reach back to Signal Center

resources; (4) reinforcement of existing theory or skills; (5) reduced time away from the unit for

training; and (6) reduced training cost.

It is important to note that the motivation to establish Unit Universities appeared to have

come from a need to enhance readiness more directly than by improving personnel or training

percentages as stated in a unit readiness report. Improving, for example, a personnel percentage

requires that the number of duty-MOS-qualified Soldiers in a unit changes. Such a change

requires institutional education conducted either entirely at the schoolhouse or provided as a

blended learning solution or mobile training team. The Unit Universities of nine of the ten

training supervisors interviewed were established rather as workarounds to the challenges of

going through the formal educational system in a timely fashion. The nature of how readiness is

enhanced by On-Demand Training further highlights the unique characteristics of this initiative

and its ability to provide training that could not be conducted in any other way.

That said, the established Unit Universities have not yet achieved optimum effectiveness

in enhancing just-in-time competency, as reported by the training supervisors interviewed. The

primary barriers to effectiveness that they listed were (1) limited time to conduct on-the-job

training; (2) lack of incentives for conducting additional training outside of daily duties; (3) lack

of command emphasis on using digital training for augmenting work performance; and (4)

challenges getting schoolhouse courseware released online or getting special purpose courseware

created. The majority of these barriers are beyond the direct sphere of influence of the LLC,

although initiating and maintaining the development of social networks among itself, unit

commanders, and schoolhouse education providers will be critical for facilitating the delivery of

training on demand and ensuring it will be used.

Cost-effectiveness. The total cost of sending mobile training teams (MTTs) varies a great

deal, depending on the location where the team is to be sent (costs include travel and possibly

1 One of the ten interviewees was using his Unit University to deliver the non-resident Sergeants Major course. He

was also very enthusiastic about the capabilities of his Unit University, but from the stance of providing institutional

instruction rather than just-in-time training. Fifteen percent (7/47) of Unit Universities are used to support blended

learning for institutional education.

55
hazardous duty pay), the duration of the training (instructor costs), the number of students

(instructor and printing and reproduction requirements), and equipment requirements. The Fort

Gordon LLC can assist with reducing these costs by making course content available online,

thereby reducing the printing and reproduction costs of conventional MTTs, or by providing

training virtually (i.e., virtual MTTs). Both types of MTT have been deployed by the Fort

Gordon LLC.

MTTs with training content (e.g., lecture slides) hosted in an LLC represent relatively a

minor reduction in cost, printing and reproduction only. In this case, instructors still must be sent

to the training location, as (potentially) does the equipment. The feasibility of sending a mobile

training team therefore is a function of the number of students who need the training and the

readiness cost (i.e., time away from unit) associated with sending these students to the

schoolhouse, where equipment and instructors are already located, or deploying these personnel

without the training. One conventional MTT enabled by the Fort Gordon LLC was sent to Fort

Hood, and one to be sent back to the same location was determined to be infeasible given the

relatively small number of students. Factors therefore somewhat beyond the sphere of influence

of the LLC (i.e., availability of equipment at the unit to be trained and instructor costs) ultimately

determined the deployability of a conventional MTT.

Since its launch in 2002, the Fort Gordon LLC has deployed one virtual mobile training

team. As presented in the Fort Gordon LLC VIP brief, a unit in Iraq discovered that it had to

work with a piece of satellite communications equipment with which it was completely

unfamiliar. The unit requested training on this equipment in order to successfully move, operate,

and maintain the equipment. In two weeks' time, the LLC staff, together with instructors at the

Signal Center, developed videotaped demonstrations of how to perform the required functions

and hosted it online through the LLC along with operation manuals, maintenance manuals, and

instructor slides and notes. The estimated cost savings associated with conducting the training

virtually was more than $400,000.

Present examination of archival cost estimates for sending a MTT to Iraq indicated that

the vast majority of the costs incurred by such a team would be personnel salary and travel costs.

Blended learning solutions that have been piloted to conduct 25B 10 MOSQ instruction with the

National Guard demonstrates an option for decreasing the actual costs of deploying virtual

mobile training teams by reducing the duration of the team's deployment. In any case, a cost

approaching $400,000 to send a mobile training team to a hazardous area such as Iraq, raised the

question of how often this occurs and therefore how many opportunities an LLC has to reduce

costs by providing virtual teams instead of live ones. The Fort Gordon LLC has not had the

opportunity to provide another virtual mobile training team (instead of a live mobile training

team) in the six years it has been established. This reality highlights the importance of LLCs and

virtual mobile training teams to making training possible, perhaps more so than saving money to

conduct training.

Recommendations

The impact of On-Demand Training on mission readiness and cost savings is a direct

function of the adoption of the Fort Gordon LLC by units in the field. Increased adoption of Unit

56
Universities and virtual mobile training teams will improve the likelihood that skills can be

acquired or refreshed just in time and enhances the effectiveness (both in terms of pedagogy and

resource requirements) of proponent outreach to lifelong learners. The rate of expansion of Unit

Universities suggests that the adoption of On-Demand Training is accelerating, but also that

there are opportunities for the Fort Gordon LLC and Signal Center to extend their outreach

through increased marketing activities.

In order to fully understand the readiness impact of Unit Universities, some follow-up

research is necessary. This research should be facilitated by the LLC staff through working

documentation of the following information:

o A history for each Unit University (i.e., how/why it was established, how long it took,

how the requester heard about LandWarNet, etc.)

o The training content hosted in each Unit University

o The history of the training content (i.e., whether it was developed specifically for the Unit

University or collected from an existing source, the social networks involved in making

the content available, etc.)

In addition, it would be beneficial if the LLC staff conducted periodic follow-up phone

interviews with the training supervisors in charge of the Unit Universities so that stories of how

the system is working and what the challenge areas are can be collected and acted upon.

Documentation of the external factors that limit the impact of Unit Universities may support

efforts to exact change in other areas that could make training more adaptive and flexible in the

future.

LLC SELF-ASSESSMENT

LLC self-assessment enables TRADOC to leverage the findings of this ARI program of

research in order to develop the lifelong learning initiative well into the future. For this reason,

effort was devoted to exploring the requirements for conducting LLC self-assessment using the

revised assessment framework and analyzing the feasibility of these requirements. The goal was

to develop a plan for conducting LLC self-assessment that could be readily adopted and also

produce meaningful data about learning effectiveness and organizational impact. Assessment

lessons learned throughout the research program were collected to assist in forming this plan and

are summarized below.

LLC Assessment Lessons Learned

Lesson #1 - The Rate of Change in LLC Activities and Outputs is Very High

As described previously, interpreting the results of LLC assessments must take into

account the fact that LLCs change rapidly and often. Among the LLC characteristics that change

are:

o Terminolo v: The use of multiple terms to refer to the same things (or vice versa, to use

the same term to refer to multiple things) is widespread. For example, the Fort Gordon

57
LLC also is known to various users as the University of Information Technology,

LandWarNet e-Signal and LandWarNet eUniversity. Other multi-use terminology

includes referents to modes of delivering instruction (e.g., distributed vs. distance

leaming vs. blended learning and simulation vs. simulator vs. courseware), which often

are used independently of their definitions.

o Courses offered: LLCs are characterized by significant, responsive growth to training and

education requirements that emerge from both proponent schoolhouses and the field. The

result of this adaptivity is the emergence of different types of courses which are best

delivered using different types of instructional strategy, educational technology, and, of

course, different content. The LLCs adapt in response, adding new components,

modifying interface layouts, and bhanging technical capabilities.

o Support infrastructure: Since the Signal Center's University of Information Technology

was established as the first LLC and the initial executive agent for the lifelong learning

initiative, several changes have been made regarding where strategic planning and

decision making are executed. The location and means for providing server support and

technical assistance also have undergone change, with plans for further evolution.

o Use practices: The use of LLCs (and, by extension, interface their layouts) evolves as the

technical staff, instructors, course developers and other stakeholders develop processes

for working together and enhance their ability to leverage the technologies involved.

Other driving forces behind change in use practices involves the integration of additional

stakeholders who enhance LLC functioning, such as faculty developers, quality assurance

professionals, marketing personnel, course administrators, and so on.

o Personnel: LLCs are run by contractor personnel, which change at least on a periodic

basis when a challenger successfully wins a re-compete. Changes in contractors generally

do not involve a complete turnover in personnel, but what change in personnel does occur

results in process and data loss as new hires are brought up to speed.

o Assessment goals: As LLCs evolve, the reason for assessing them changes, as does the

target of assessment. Assessment that was initially conducted to justify the existence of a

particular LLC (or of the broader LLC concept) may later be conducted to evaluate the

impact of introducing specific technologies, faculty development interventions, support

infrastructure modifications, or other initiatives. As LLCs become broader in scope, the

assessment spotlight may shift across components because an assessment of all

components simultaneously would be too labor intensive.

o Technology: The technology used as the backbone of LLCs is always subject to change.

Existing technologies may be substituted by more cost-effective or powerful alternatives.

New technologies may be added to leverage advances in capability or to enable the

achievement of newly identified learning objectives. Individual proponents may augment

the LLC foundational technology with applications of their own in order to meet the

particular needs of their learners.

Lesson #2 - The Assessment Methods Necessary to Capture Meaningful Data Require a "Human

in the Loop"

A range of methods may be used to capture assessment metrics, including system

analysis (e.g., content analysis of discussion forums), surveys, interviews, focus groups, archival

data analysis (e.g., financial data, programs of instruction, course grades, etc.), and classroom

58
observation. Unfortunately, none of these methods are suited for full automation for the

following reasons:

1. Although the LLC Assessment Framework identifies metrics that will provide

information about what data to collect, a person is needed to develop the measures used

to capture metric data. Reuse of such measures as surveys, interview and focus group

protocols, observer checklists, content analysis rubrics, etc., is possible, and

recommended as a means of partial automation, but some modification will be necessary

over time to address the kinds of changes described above.

2. Some data may be collected automatically (e.g., survey data), but most methods require a

person to gather data from other people (e.g., interviews, focus groups, classroom

observations). There are methods for partially automating this process, for example,

providing means for users to submit comments and feedback instead of conducting

interviews or using grades instead of classroom observations. The tradeoff associated

with partial automation is between information volume and ease of administration.

Generally speaking, the easier it is to administer a measure, the less information it

provides. A human is required to judge the cost-benefit tradeoff of selecting among

assessment methods.

3. Metric data come from numerous sources, requiring the participation and support of

people both within and outside the proponent schoolhouse. Automated data collection

likely would require the integration of multiple data sources including not only databases

(automated and human-manipulated) but also paper-based repositories. A person would

be necessary to integrate the information from these numerous sources.

4. Once metric data have been captured, a person is required to analyze and interpret the

data. Survey software can automatically summarize response data, for example, but it

cannot identify the implications of these data for the functioning of the LLC. Similarly,

activity data may be collected automatically by LLC technologies, but these data paint

only a partial picture of effectiveness and impact. Several data inputs must be analyzed

and integrated to fully understand LLC functioning.

5. Interpretation of metric data also needs a human to specify the caveats to generalizing

and acting on the findings. For example, a person must be involved in determining

whether a representative sample was used for classroom observations and, when the

sample is not representative, what limits are placed on where or when the findings apply.

6. Finally, a person is required to communicate the assessment results to others. At some

point in the distant future, dashboard technologies (requiring human data entry and/or

significant database integration) could be designed to support the flexible and broad

reporting of assessment results. The design of such a dashboard, however, requires a

more stable assessment context than is currently the case for LLCs. Assessment metrics

must be extensively validated and widely accepted before serving as the design basis for

an assessment dashboard.

Lesson #3 - Face Time is Critical for Conducting Assessment

Face time is critical for conducting assessment for multiple reasons. First, visits to the

proponent schoolhouse and interviews with key stakeholders are required to fully understand the

complexities of that proponent's LLC. Although on paper LLCs share the same basic

59
technologies, functions, and goals, LLCs differ widely in their histories and particular areas of

emphasis. Physically being present at the schoolhouse facilitates contact with a larger and more

representative sample of people involved in running the LLC and providing instruction.

Stakeholders overlooked by an assessor who has not visited the proponent could potentially

provide useful information or could provide access to additional people of which the assessor

was unaware. When the connections among stakeholders and other people with information arise

during site visits, it is a simple matter to capitalize on such networks by setting up meetings or

dropping by an office. Personal introductions facilitated by stakeholders increase the likelihood

of a productive exchange with people unfamiliar with the assessor.

Second, face-to-face meetings help to build trust on the part of stakeholders and other

providers, which, in an evaluative context, may not come easily. Multiple meetings often are

required to gain access to people and data that can present the most complete picture of LLC

functioning. Periodic, face-to-face updates to stakeholders and providers on assessment findings

go a long way toward ensuring shared understanding of how the program is doing and achieving

mutual satisfaction with how results are presented. Email, phone and, to some degree,

videoteleconference contact among assessors, stakeholders, and other providers reduce the

observability of visual cues of discomfort with the process and therefore prevent opportunities to

amend one's approach. In addition, physical distance between these people makes the assessor

seem more remote, allowing interpretations of the assessor's motives to become more subject to

the concerns and unverified assumptions of the people whose program is being assessed.

Lesson #4- Collecting Readiness Data Requires Extensive Field Work

Aside from theoretical, proxy measures of readiness, such as organizational commitment

(e.g., McGonigle, Casper, Meiman, Cronin, Cronin, & Harris, 2005), there are two ways to

collect readiness data. First, organizational-level readiness data, such as personnel and training

readiness status, may be collected via queries of various Army databases. Directly querying these

databases necessitates the appropriate authority and need to know, which typically is reserved for

investigators from within the Army or those research institutions closely affiliated with it (e.g.,

RAND Corporation, Institute for Defense Analyses). Without this authority and affiliation,

extensive social networks are required to collect the same data indirectly. Such social networks

must involve well-established relationships with Army personnel and/or large contractors who

work actively with the necessary databases and are willing to support the investigation.

Second, the collection of unit-level readiness data, such as time to competency, requires

close contact with unit trainers and commanders in the field who receive graduates and who

participate actively in fostering lifelong learning in their subordinates. As demonstrated in the

present research effort, it is possible to collect such readiness data via phone interviews and

surveys, but a "boots-on-the-ground" perspective would be more effective in collecting plentiful

and detailed information about how engagement in lifelong learning can be translated into the

savings of life, time, and money (particularly in a deployed setting). Such detail would enhance

the qualitative causal model of LLC impact and lead to the development of new metrics and

measures. Unfbrtunately, such detail would be costly and time consuming to gather, in part

because lifelong learning graduates are widely distributed and difficult to track individually.

60
Implications for Using the LLC Assessment Framework

Lesson #1 Implications

The rapid rate of change discovered in Lesson #1 (Rapid Change) indicates that some

modification to the LLC Assessment Framework will be required to conduct LLC selfassessments in the future. The present research suggests that the nature of these modifications

will be the addition or modification of logic model component elements, as well as new or

modified metrics and measures, provided that new use practices or technologies (not anticipated

by the current framework) are assessed. Activities identified in the revised LLC Assessment

Framework likely will have to be adjusted somewhat as the LLC concept itself is expanded to

regional and enterprise level. For instance, actions on the part of regional and enterprise LLC

technical staff to facilitate collaboration (distributed and face-to-face) with course developers,

instructors, administrators, and subject matter experts must be a critical target for future LLC

assessments.

A second implication of Lesson # I is that the cost of automating LLC assessment likely

would significantly outweigh the benefit. Oversight would be required to ensure that automated

assessment measures use the correct terminology, address the appropriate technologies, apply to

the right courses, and so on, which will change often over time. Similarly, administration would

be necessary in order to verify the accuracy and quality of data collected automatically in the

context of frequent, rapid change. The actions that assessment personnel would have to take to

exert the proper control over data flow would amount to functionally the same activities that

would be conducted to assess the LLCs without automation.

Lesson #2 Implications

Lesson #2 (Person in the Loop) indicates that LLC self-assessment is far from becoming

an automated process. A person is required to (1) determine the minor modifications to the LLC

Assessment Framework for particular applications; (2) develop or revise measures (or decide to

use the measures as is); (3) conduct data collection, analysis, and interpretation; and (4)

communicate results to others. To perform these functions, this person (or people) must have a

working knowledge of adult learning theory and practice, instructional design, educational

technology, research design, and elementary statistics.

Automating the assessment process, at least in the early stages of the lifelong learning

initiative, not only would not be feasible given the rapid rate of change, the nascent state of

metrics and measures, and widely distributed sources of assessment data, but also would lack the

diagnostic detail necessary to understand the basis of shortfalls and stimulate growth. For this

reason alone, rapid advancement toward automation at this time should not be encouraged.

Rather, the time and effort required to design automated data collection should be invested in

developing the relationships and research methods necessary to integrate the diverse sources of

information about LLC effectiveness and impact. Automation then could be used to facilitate an

existing process rather than to determine that process.

61
Lesson #3 Implications

The key implication of Lesson #3 (Face Time) is that applying the LLC Assessment

Framework must involve a significant amount of time devoted to being physically present at the

proponent schoolhouse supported by the LLC. Visits to the schoolhouse must involve face-toface contact with key stakeholders and other providers in which interpersonal exchange fosters

trust and mutual understanding. Time spent at the schoolhouse, particularly if the assessor is only

a temporary visitor, should leverage the close proximity of stakeholders, using the opportunity to

ensure that a large, representative sample of people is contacted.

Lesson #4 Implications

It should not be concluded from Lesson #4 (Field Work) that the unit readiness metrics

specified in the LLC Assessment Framework cannot be addressed. Rather, the necessary

authority, need to know, social connections, and outreach process must be in place to make

readiness data collection feasible and cost-effective. The existing connections between field units

and LLC staff who provide On-Demand Training could facilitate data collection, and did, in the

present research. In addition, LLC staff processes could be tailored to develop a formal method

for maintaining such contact. Closer coordination at the program/organizational level between

training and personnel managers could facilitate the data sharing necessary to determine LLC

impact. Research conducted by institutions closely affiliated with the Army should be explicitly

devoted to identifying the necessary means to capture readiness data from the field and the

challenges to applying these methods.

A Proposed LLC Self-Assessment Plan

As alluded to above, a successful LLC self-assessment program should feature the

following characteristics:

o A staff of people with working knowledge of (ideally experts in) adult learning theory

and practice, educational technology, instructional design, research design, and

elementary statistics to design and oversee the assessment process;

o An assessment process that involves significant face time between assessors,

stakeholders, and other providers; and

o Sufficient authority and social networks to orchestrate extensive field data collection.

The program also must be resource-efficient, chiefly by leveraging existing resources,

capitalizing on previous work, re-using assessment instruments, and using automation, where

feasible.

The LLC self-assessment program requires these features in order to (1) maintain

knowledge of the rapid, frequent changes to LLC characteristics; (2) determine whether

modifications to the LLC Assessment Framework are necessary to conduct assessment; (3)

develop, as needed. new metrics and measures; (4) conduct data collections; (5) analyze,

interpret, and disseminate findings; and (6) provide actionable recommendations for future LLC

62
growth. Figure 3 below shows the general layout of a proposed LLC self-assessment plan, which

is discussed in more detail below.

Human

{AResoances

Instructors, Ceomand

Course o mmat

Developers

(Proponent) LLC

Program

Management

(TRADOC)

LLC Staff

Quality s Augmentee

Assurance

(Proponent) Field

I Units

External

Assessors

LLC

Assessment Existing

Framework Resources

Figure 3. Schematic of the Proposed LLC Assessment Plan

External Assessors

The people ultimately responsible for conducting LLC self-assessment should be

disinterested parties, independent of the lifelong learning initiative. They should not be employed

by an organization whose access to funding or other resources is linked to assessment findings.

Assessment personnel also should not be the same people responsible for making decisions about

funding or the allocation of other resources to the LLC. Maximizing the independence of

assessors in this way reduces the likelihood that biases are consciously or (more likely)

unconsciously used to guide the investigative "spotlight" or the interpretation of data.

In the proposed LLC self-assessment plan, external assessors are assessment specialists

independent of the proponent and LLC program management who plan assessment, collect data,

63
analyze findings, and present results. They use the LLC Assessment Framework as a basis for

designing their assessment plan, as well as other resources, as necessary. For instance, the

Advanced Distributed Learning Co-Laboratory (ADL Co-Lab) provides training and other

support materials for educational program evaluation. These materials include a worksheet for

planning training evaluation, sample survey questions to capture student and instructor reactions

to training, and slides/handouts detailing the training evaluation process. Although the sample

survey questions relate to only one aspect of the LLC Assessment Framework, the planning

worksheet and evaluation process slides/handout could be used to assist with assessment more

broadly. All of these documents may be retrieved from the ADL Co-Lab by creating an account

with adlcommunity.net and searching the contents of the Research and Evaluation folder.

Another useful resource for assessors is TRADOC's Pamphlet 350-70-4 (2004) and

Accreditation Standards and Guide (2007). The pamphlet details the assessment process,

including how to plan assessment, collect data, analyze findings, and disseminate results. The

evaluation guide for accreditation presents 31 standards for evaluating proponent instruction as

well as detailed instructions for applying them. Several of these standards overlap with the LLC

Assessment Framework methods. Hays, Stout, and Ryan-Jones (2005) presents assessment

checklists to be used when assessing CBT/WBT. Although none of these documents alone is

fully applicable to assessing LLCs, these resources, combined with the LLC Assessment

Framework and associated reports, should provide sufficient guidance for knowledgeable

individuals to conduct a program evaluation consistent with the Army's expectations.

External assessors may come from within the government or outside of it. One option for

external assessment provided by the government is the ADL Co-Lab. The ADL Co-Lab assists in

evaluating the efficiency and effectiveness of other initiatives that fall under the general lifelong

learning rubric, including the U.S. Joint Forces Command individual augmentee training

program and the Defense Ammunition Center's internship training program, among others.

Another external assessor within the government could be the TRADOC Analysis Center.

Assessors from outside the government may come from the plethora of contractors who

specialize in instructional design and educational program evaluation.

External assessors should be capable of recognizing the larger social context in which

assessment takes place. People with this ability are better able to understand (1) the concerns that

stakeholders and other providers will have about being assessed; (2) the methods for building

trust and exchange; (3) the organizational context that places external constraints on LLC

impact; and (4) the importance of framing results in a way that is true to the findings but

satisfactory to the stakeholders involved. Sensitivity to the larger social context prepares

assessors to apply their assessment-related expertise in a way that will produce the most

meaningful findings, accepted results, and actionable recommendations. Understanding the

larger context would be greatly enhanced by an effective LLC Staff Augmentee, whose role is

described below.

LLC StaffAugmentee

The proposed LLC Staff Augmentee plays the critical role in LLC assessment by

facilitating the connection between external assessors, LLC program management, proponent

64
stakeholders, external data sources (e.g., personnel databases), and external stakeholders (i.e.,

field units). Like the external assessor, this person has expertise in the areas of adult learning

theory, technology-assisted instruction, and educational program evaluation. The LLC Staff

Augmentee is situated with the proponent, but is a government employee of the larger LLC

program. This person therefore would have the expertise and access necessary to serve as a

multi-party liaison who is capable of coordinating closely with the LLC technical staff,

schoolhouse faculty and course developers, subject matter experts, proponent quality assurance

personnel, and field unit trainers. This person would be able to reach back to the program level

of LLC management to facilitate access to external database information and field unit

leadership, which will enable data collection that is external to the proponent schoolhouse. As an

employee at the program level, the LLC Staff Augmentee also would be responsible for ensuring

that the assessment and evolution of a particular LLC feeds the larger process, including the

production of generalizable assessment methods and the development of a shared knowledge

base of LLC program effectiveness.

The knowledge and close coordination of the LLC Staff Augmentee would facilitate data

collection by assisting external assessors in identifying stakeholders (inside and outside of the

schoolhouse), procuring archival data, and achieving access to schoolhouse courses, field unit

trainers, and personnel and training bases. The LLC Staff Augmentee also would provide

external assessors with the current technical status of the LLC, as well as the history of changes,

the larger organizational context, and related documentation. The implementation of assessors'

recommendations would be enabled by the LLC Staff Augmentee's collaboration with proponent

quality assurance personnel responsible for faculty development. The LLC Staff Augmentee

would contribute his or her expertise in technology-assisted instruction to designing and

delivering faculty development at the request of quality assurance.

Quality Assurance (Proponent)

The expertise of proponent quality assurance personnel should be leveraged to assist with

data collection both internal and external to the schoolhouse. These individuals have significant

experience with conducting assessment of Army training initiatives and have ready access to

existing resources that could support the LLC evaluation effort. Ongoing quality assurance

studies should be brought to the attention of external assessors by the LLC Staff Augmentee so

that efforts are not duplicated and time is saved by re-using assessment processes. Moreover, the

LLC Staff Augmentee should identify recurring LLC assessment targets and work with quality

assurance to include these as part of their investigation of proponent effectiveness. Additional

contracted personnel may be required to assume added quality assurance duties, but quality

assurance leaders have experience recruiting, selecting, and training educational assessors and

may be considered more independent than members of an LLC staff who might be considered as

alternatives.

LLC Program Management (TRADOC)

The role of LLC program management in the proposed assessment plan is to ensure that

lifelong learning assessmentand development are tightly coupled through coordinated processes

across the LLCs. These processes should not necessarily be standardized, given the unique needs

65
of each LLC, but they should avoid duplication of effort, collectively paint the "big picture" of

the success of lifelong learning, and ultimately facilitate the connection between the lifelong

learning initiative, the proponent schoolhouses, and the field Army. The LLC program

management would accomplish such synchronized diversity by maintaining reporting authority

over the LLC Staff Augmentees but situating them with the proponents. Augmentees would be

able to develop the critical relationships necessary to enable lifelong learning but also would

have the obligation to coordinate with the augrnentees of other LLC staffs and report to the LLC

program management about the success of the larger lifelong learning effort. In addition, the

organization-level connections of the LLC program management with the Human Resources

Command and the key leadership of field units would facilitate the augmentee reach back for the

collection of data external to the proponent schoolhouse.

Reconsidering the Purpose of the LLC Staff

A critical characteristic of the proposed LLC self-assessment plan is that it involves a

slight, but important modification to the envisioned purpose of the LLC staff. The proposed plan

is predicated on the assumption that the success of lifelong learning is as much a function of

social networks as it is of technical outputs and that the LLC staff could be uniquely situated to

provide both. Currently, the role of the LLC staff is considered to be largely technical, involving

the construction and maintenance of the LLC portals and the provision of technical support. In

contrast, responsibility is somewhat more diffuse and less well articulated for ensuring (1) that

relationships are developed with proponent stakeholders to rapidly produce effective training on

demand and to conduct meaningful assessment; (2) that proponent faculty and course developers

understand how to leverage technology for assisting adult learners; (3) that LLC functionality fits

into the larger context of not only the proponent's objectives, but those of organizational Army;

and (4) that a body of knowledge is developed to track and facilitate the growth of the lifelong

learning concept. Carrying out this responsibility requires the cooperation and coordination of a

diverse set of organizations that may or may not be aware (or supportive) of their role in

enabling lifelong learning. A central source of strong leadership that understands the needs of the

proponent, the LLC program management, and the larger context is needed to maximize impact.

LLC technical staffs as currently constructed do not have the personnel, expertise, or

authority to take on these challenging and complex roles. Yet, as the findings of the LLC

assessment research program suggest to date, these roles are essential to ensuring the success of

lifelong learning by strengthening the link between technical outputs, external factors, and

impact. The addition of the LLC Staff Augmentee in the proposed assessment plan is not simply

a recommended method for conducting LCC assessment, but also represents an important first

step in acknowledging and enacting a more proactive role of LLC program management in

equipping LLC staffs to achieve the larger goals of lifelong learning.

CONCLUSIONS

The present research explored the generalizability of a framework for assessing the

effectiveness and impact of LLCs, the concrete instantiation of the U.S. Army's lifelong learning

concept. The analysis presented in this report involved a substantial increase in the types of

instruction and learning audiences supported by lifelong learning than has been conducted

66
previously, producing a more representative sample on which to base the framework's design. It

was determined that some modification was necessary to the assessment framework to address

the variety of possible functions an LLC could provide, but that the logic modeling approach

remained a valid method for conducting qualitative causal analysis.

A secondary purpose of the present research was to conduct an assessment of the Fort

Gordon LLC. The results of this assessment indicated that although the expected outputs for this

LLC generally were delivered, external factors moderated the link between outputs and

outcomes at all levels (individual, unit, and organizational). That is, in the absence of controlled

experimental study, inferences could be made regarding the determinants of program strengths

and weaknesses by using a qualitative causal model. Shaping the influence of external factors

required closer coordination among the LLC staff and proponent offices as well as closer

coordination among decision makers at the organizational and Army level.

A third purpose of this research effort was to specify an assessment plan that would

enable LLCs to adopt the revised LLC Assessment Framework to conduct self-assessment. The

design of this plan was based on lessons learned in the present investigation as well as previous

ARI LLC assessment research (Cianciolo, 2007). A key feature of this plan is the emphasis

placed on reconsidering the role of LLC staffs such that they serve a critical social as well as

technical function. The self-assessment plan addressed the need to more closely coordinate

efforts by situating program-level assessment personnel at the proponent. Staffing in this way

would facilitate the social connections necessary to conduct assessment internally and externally

and would ensure that assessment findings were actively used to enhance the larger lifelong

learning initiative.

Limitations of the Present Research

The limitations of the present research involve the lack of empirical basis of the Fort

Gordon LLC assessment. First, emphasis on exploring methods to substitute for surveys,

combined with the short duration of this research effort, reduced the representativeness of the

sample used in some cases (e.g., only three SCCC classes were observed, only three 25B10

instructors were interviewed). In these instances, converging evidence was used to support lack

of data, but further study is warranted. Where representative sampling was possible (e.g., as in

discussion forum analysis), it was conducted. Moreover, in several instances, the relatively small

sample actually was the population (e.g., 25B10 was the only MOSQ course delivered both in

the schoolhouse and to non-resident students).

Second, where comparisons across conditions were of interest (e.g., training effectiveness

with and without simulations), lack of authority to manipulate treatment conditions prevented

controlled experimental study. Where possible, archival data and/or quasi-experimental designs

(e.g., as in the SCCC) were used--and the qualitative causal model provided a great deal of

converging evidence--but systematic study would provide a stronger basis for conclusions.

An additional, important limitation of the present research was the lack of external data

that could provide information about the impact of lifelong learning on unit readiness.

AUTOGEN surveys, as well as surveys created as part of this research effort, suffered from

67
insufficient response rates. Phone interviews with unit trainers who established Unit Universities

were helpful, but failed to provide detailed accounts of the impact of lifelong learning. The

enthusiasm of unit trainers for LLCs was great, but direct contact with and observation of these

trainers' units is necessary to understand the concrete reality of how training on demand has

changed the readiness posture of units in the field.

Recommendations for Future Research and Practice

Future research should focus on identifying and applying the means necessary to conduct

external data collection. Visits to line units actively using lifelong learning should be paid in

order to observe directly the impact that anytime, anywhere instruction has on performance. Face

time with these units also would enable the collection of comprehensive interviews and focus

groups to gather detailed information for developing additional outcome metrics. Such

investigation would reveal, for example, the acceptable amount of lag between the request for

and delivery of On-Demand Training. Physical presence with the units also would reveal

additional external factors that influence the adoption of lifelong learning and potential methods

for addressing these factors and expanding LLC marketing outreach. The schoolhouse

environment and, to some extent, the distance learning setting is fairly well understood, but the

operational learning environment remains largely unexplored.

Future research also should explore and identify the additional modes of instruction and

learning audiences that are being or will be addressed by current and envisioned LLCs. Although

a wide sampling of educational opportunities was featured in this report and in Cianciolo (2007),

there is sufficient reason to believe that this sample was not comprehensive. This research could

be conducted by reviewing each LLC's planning documentation and conducting site visits to

each proponent to interview key stakeholders.

Future practice must ensure that the critical, proactive social role of the LLC staff be

acknowledged and actively developed. Initiating self-assessment as envisioned in this report

would represent a significant step in this direction. Future research supporting this effort should

explore the particular social networks and processes necessary to enable effective LLC

functioning at the regional and enterprise level, as well as at the proponent level. Changes to

technical support structures must retain the critical coordinative and collaborative roles served by

a fully functional LLC staff.

Closing Thoughts

In summary, the present investigation attempted to achieve three goals: (1) to examine

the generalizability of the LLC Assessment Framework put forth in previous ARI research; (2) to

apply the revised framework to assessing the Fort Gordon LLC; and (3) to explore the

requirements for enabling LLC's to conduct their own assessment in the future. The findings of

this research were that the assessment framework was generalizable, but required modification in

order to address the broad range of training and education strategies supported by the Fort

Gordon LLC. It is expected that this variety of strategies represents the majority of learning

approaches that will be taken by other current and future LLCs, but some modification to the

framework will be necessary for each assessment that is conducted. The Fort Gordon LLC was

68
found generally to be accomplishing its output goals but that external factors sometimes

moderated the link between outputs and expected outcomes. Recommendations for enhancing

outcomes and for conducting future LLC assessment emphasized the importance of coordination

among the LLC technical staff, proponent stakeholders, the LLC program management, and

other stakeholders at the Army level. LLCs represent the future of adaptive training and

education, and it is hoped that these findings will foster continued growth and the broad impact

envisioned for the lifelong learning initiative.

69
70
REFERENCES

Abell, M. (2000). Soldiers as distance learners: What Army trainers need to know. Proceedings

of the Interservice/Industry Training, Education, & Simulation Conference (I/ITSEC).

Orlando, FL.

Abell, M. (2003). Deepening distributed learning: Motivating Soldiers to learn, grow, achieve.

Proceedings of the Interservice/Industry Training, Education, & Simulation Conference

(/ITSEC). Orlando, FL.

American Distance Education Consortium. (2000). ADEC Guiding principles for

distance teaching and learning. Available at:

http://www.adec.edu/admin/pavers/distanceteaching pDrnciRles.html.

Arthur, W., Jr., Bennett, W., Jr., Stanush, P. L., & McNelly, T. L. (1998). Factors that influence

skill decay and retention: A quantitative review and analysis. Human Performance,

11(1), 57-101.

Barrett, J. (2001). What assignment-oriented training means to the Signal Regiment. Army

Communicator, 26(4).

Boume, J. R. (1998). Net-learning: Strategies for on-campus and off-campus network-enabled

learning. Journal of Asynchronous Learning Networks, 2(2), 70-88.

Chiarelli, P. W., & Michaelis, P. R. (2005). Winning the peace: The requirement for fullspectrum operations. Military Review, July-August, 4-17.

Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate

education. Retrieved June 9, 2006 from the World Wide Web. Available at

http://honolulu.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/7princip.ht

M.

Cianciolo, A. T. (2007). Program Evaluation Metrics for U.S. Army Lifelong Learning Centers

(Technical Report No. 1196). Arlington, VA: U.S. Army Research Institute for the

Behavioral and Social Sciences.

Cianciolo, A. T., Cianciolo, D., Prevou, M. I., & Morris, R. F. (2007). Using digital storytelling

to stimulate discussion in Army professional forums. Proceedings of the

Interservice/Industry Training, Education, & Simulation Conference (I/ITSEC). Orlando,

FL

Cianciolo, A. T., Heiden, C., & Prevou, M. (2006). Assessing Army professional forums: Metrics

for effectiveness and impact (Research Report No. 1862). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences.

71
Clark, R. E. (1994). Assessment of distance learning technology. In E. L. Baker & H. F. O'Neil,

Jr. (Eds.), Technology assessment in education and training (pp. 63-78). Hillsdale, NJ:

Erlbaum.

Dean, R., Biner, P., & Coenen, M. (1996). Distance education effectiveness: A systems approach

to assessing the effectiveness of distance education. Education at a Distance Journal,

10(3), J17-J20.

Dixon, N. M., Allen, N., Burgess, T., Kilner, P., & Schweitzer, S. (2005). CompanyCommand:

Unleashing the power of the Army profession. West Point, NY: Center for the

Advancement of Leader Development and Organizational Learning.

Farrell, P. (2001). Transforming information-technology training: A master plan for change.

Army Communicator, 26(4).

Firdyiwek, Y. (1999). Web-based courseware tools: Where is the pedagogy? Educational

Technology, 39(1), 29-34.

Frank, G., Whiteford, B., Hubal, R., Sonker, P., Perkins, K., Arnold, P., et al. (2004).

Performance assessment for distributed learning using after action review reports

generated by simulations. Proceedings of the Interservice/Industry Training, Simulation,

and Education Conference. Orlando, FL.

Hays, R. T., Stout, R. J., & Ryan-Jones, D. L. (2005). Quality evaluation tool for computer- and

web-delivered instruction (Technical Report No. 2005-002). Orlando, FL: Naval Air

Warfare Center Training Systems Division.

Hiltz, S. R. (1998). Collaborative learning in asynchronous learning networks: Building

learning communities. Invited Address at WEB98, Orlando, Florida. Retrieved from the

World Wide Web on March 18, 2004. Available at:

http://eies.niit.edu/-hiltz/collaborative learning in as3nch.htm.

Kinney, P. A. (2005). Assignment oriented training (A OT) training effectiveness analysis (TEA)

interim report: Phase I results (Technical Report No TRAC-WSMR-TEA-05-024). Fort

Monroe, VA: U. S. Army Training and Doctrine Command (TRADOC).

Kreijns, K., Kirschner, P. A., Jochems, W. (2003). Identifying the pitfalls for social interaction in

computer-supported collaborative learning environments: A review of the research.

Computers in Human Behavior, 19(3), 335-353.

Leonard, H. A., Winkler, J. D., Hove, A., Ettedgui, E., Shanley, M. G., & Sollinger, J. (2001).

Enhancing stability and professional development using distance learning. Santa

Monica, CA: RAND Corporation.

Lesser, E. L., & Storck, J. (2001). Communities of practice and organizational performance. IBM

Systems Journal, 40(4), 831-841.

72
Long Hard Road: NCO Experiences in Afghanistan and Iraq. (2007). Fort Bliss, TX: U. S. Army

Sergeants Major Academy.

McGonigle, T. P., Casper, W. J., Meiman, E. P., Cronin, C. B., Cronin, B. E., & Harris, R. R.

(2005). The relationship between personnel support programs and readiness: A model to

guide future research. Military Psychology, 17(1), 25-39.

McLaughlin, J. A., & Jordan, G. B. (2004). Using logic models. In J. S. Wholey, H. P. Hatry, &

K. E. Newcomer (Eds.), Handbook ofpractical program evaluation (2"d ed, pp. 7-32).

San Francisco: Jossey-Bass.

North, B. (2006). Signal captains career course. Army Communicator, 31(2), 4-6.

Phillips, J. K., Ross, K. G., & Shadrick, S. B. (2006). User's guide for tactical thinking

behaviorally anchored rating scales. (Research Product 2006-05). Fort Knox, KY: U.S.

Army Research Institute for the Behavioral and Social Sciences.

Salas, E., Rhodenizer, L., & Bowers, C. A. (2000). The design and delivery of crew resource

management training: Exploiting available resources. Human Factors, 42(3), 490-511.

Shanley, M. G., Crowley, J. C., Lewis, M. W., Masi, R., Straus, S. G., Leuschner, K. J., et al.

(2005). Transformation and The Army School System. Santa Monica, CA: RAND

Corporation.

Shanley, M. G., Leonard, H. A., & Winkler, J. D. (2001) Army distance learning: Potential for

reducing shortages in Army enlisted occupations. Santa Monica, CA: RAND

Corporation.

Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of

web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59(3),

623-666.

Sortor, R. E., Lippiatt, T. F., Polich, J. M., & Crowley, J. C. (1994). Training readiness in the

Army reserve components. Santa Monica, CA: RAND Corporation.

Tan, H., & Libby, R. (1997). Tacit managerial versus technical knowledge as determinants of

audit expertise in the field. Journal ofAccounting Research, 35(1), 97-113.

U.S. Army Training and Doctrine Command. (2004). The growth and future of lifelong learning.

Retrieved February 18, 2006 from the World Wide Web. Available at

http://www.tradoc.army.mil/pao/web specials/lifeloni learnin/lllgrowth.htm.

U.S. Army Training and Doctrine Command. (2004). Pamphlet 350-70-4: Systems approach to

training: Evaluation. Washington, DC.

73
U.S. Department of the Army. (2006, December). FM 3-24: Counterinsurgency. Washington,

DC.

University of Illinois Faculty Seminar. (1999). Teaching at an Internet distance: The pedagogy

of online teaching and learning. Retrieved May 8, 2006 from the World Wide Web.

Available at: http://www.vpaa.uillinois.edu/tid/report.

Vann, L. S. (2003). Training update: Assignment-oriented training. Army Communicator, 28(3),

24.

Wagner, R. K., & Sternberg, R. J. (1985). Practical intelligence in real-world pursuits: The role

of tacit knowledge. Journal of Personality & Social Psychology, 49(2), 436-458.

Walton, B. (2003) Training update: Lifelong learning. Army Communicator, 2 7(3), 29-30.

Wenger, E., McDermott, R., & Snyder, W. M. (2002). Cultivating communities ofpractice.

Boston, MA: Harvard Business School Press.

Williams, T. J. (2003). Strategic leader readiness and competencies for asymmetric warfare.

Parameters, Summer, 19-35.

Wilson, W. R., & Helms, R. F., II. (2003). A business model for lifelong learning. Proceedings

of the Interservice/Industry Training, Simulation, and Education Conference. Orlando,

FL.

Winkler, J. D., & Polich, J. M. (1990). Effectiveness of interactive videodisc in Army

communications training. Santa Monica, CA: RAND.

74
APPENDIX A - ACRONYMS

ADL Co-Lab Advanced Distributed Learning Co-Laboratory

ANCOC Advanced Non-Commissioned Officer Course

AOT assignment-oriented training

ARI U.S. Army Research Institute for the Behavioral and Social Sciences

ATRRS Army Training Requirements and Resources System

AUTOGEN automatic survey generation

BCKS Battle Command Knowledge System

BNCOC Basic Non-Commissioned Officer Course

BOLC Basic Officer Leadership Course

CBT computer-based training

DL distance learning

HTRC high-tech regional center

JNN joint network node

LLC lifelong learning center

LTC lieutenant colonel

MANSCEN Maneuver Support Center

MOS military operating specialty

MOSQ military operating specialty quilified

MTI" mobile training team

NCO non-commissioned officer

SCCC Signal Captains Career Course

SOP standard operating procedure

TRADOC U.S. Army Training and Doctrine Command

UIT University of Information Technology

WBT web-based training

75
76
x

ANN

< $1 II.

~ Li

U .W
zx

xx

x

4)4+ 0

U- z

' ~ E I 0 0 0-0 .0 4

0 Q0

),40 > 4)E

-0 -e - 0 A Q. 

>

In.. 0 ou 0 in U

V) 04)

70

4. 79 e

u
x

.. 2

=== . = . " + .

A0 0

0 0 00

0" oO

- V

go:o

CLU

0 0 .2

> 0<
zC

4)

U ~ C utu )~.0 0 

0

p~~~~1 4 4 o ~ )~34 'fw)

0-~ 0.0

00

0

J4 0*

0 or 7!

CL> -,- rL -!j r. 

Ca.

o~o

SA

go:~ 4

~4)go:

00

LQ 0~)~
x x

Elli

+ >0

Ii a

a' iII

- J.,

I iin
tu t

=A

o 0. 0

e0 o wi -0

o '0

4..

0- 0

~U 0 00

El E S 9 i

bo on. -0 1b

0 0

~u~o0mm

40 0 "0

0,*

-0 G.A 0_

rd) U

__ _ La _ _ 1!___ _ __L r
ral

r,

++ 

0

0 

o, Mon.

8 0 0.8

C .2

~~Op

Z J -- j;5

4) 0

44S -80 0 . i.
a o

0>00

U I;?!l i !! _ 

Z

iIi

C00

or

Hli .0

.

'

v io .0

Is I I 

Jd 0 rA2
000

00

C44

200

6e0~ ~

I.-

"' s

0 . t)p.

o 'o -.
xx

W 0 o

-0 

0 00

0 2 -r-2i9 1 r

~~.00

2 AA

400.

044. m .7

0 III!
zx

Xx

+ I mn ~V

co Be

lot i

- _ 00
4 x

>S

- ~4'00

00 =0 a 3' 00Z0.

0 9 : I IonI

0 o

U ~is

EE Is~
xx

x x X

at t

rA~ IIG 3

RIO

Ln 0V 0 - Gn

01 c) 'C l0 00

00 oI1 .I 1 o 0a 00 0

ol -S -2 g r r F o

.970 Jo0J

.0d f r

__ _ 7___ 7___S_<_S <

Ho 0 0 -o--

o_ U __ Qn__ _ _ 0_ __ ____t

L Sm

MOE- r
zx

00 00

-0

o a)

< .- 

CLu

.< 2) 02

4)00

Cuum

or *- 010 oowQ

0C U) r. -. 4 T L "g > Cu PA 0

e '044 ) 0

4 4) 0.

2- 00 2
x x

W X x x x

o o "90 o6

I . + I' --

. . cr - - cf

o< ..- o . o,. a< 

-a. -

C.~

- 4!I

9w Or o -0 ooN- A

Jo. o

0) 0 0
w W

+0

00

.00 U 0 0

.2 o

> , -M

c .2 c W-8

o coad

0 0

O. -z 0 50 

0t~

Z:U C3~-

00$
CL1

"i-C

SIl

,0 0- 

° !il,
r,

0)

>6>

0 0 0

CL0 0 4

0 .U a C)

0. o.. 0 Y o Cc, o

o o>

ba 4,'.2 M 0

C) U

0
d I £ i , 

ISIII" I° ii

00

_ = .' !La .0 -° lii I

o 0. Si ii < = AL__I

i '==* £i!jo

=:'i I vc
x X

x

04)

2 o

~-.0

.8 o

cc 0 C ~ O 0 - <~u 0 .2~L

CL _ _ _ __3

~0 y C

0. o-0 o 0-9

0 ~ .

M '0

00
X ~X X

z

rAx

Ila

>1 4)

C/). cnQf0 r

cy 0004 (1

0- - CO 0-S40g

-Z ,000' GO ~ I Z V

CLo

00.

0.6

0.0 0 C

ixQ

L _ _:1_ _ _ _ _ _ _ _ _ * 0

LZ0 __ _ _ _ _ _ _ _ _ _ _
X x X

z

0 !2 C) 'o 1' ., a '

000 : 0 0 ,

10. 0

OD 0

0~ 0

o-0 7 9

wo E

4"". ol Jo

14. 0" § .91 G20 C-, 7.9 ~ e 

a a)
zx

l

5 ~ ~ 7 011~i1

i jjli °

I ill = i i iI'

fil o I

oh . . "jiu

++,,++,..I++ + .+ I ' L i++ ++.

_, +!',+ + I- .

II g + -t "ii +t!!I + i.

+++ *111+ I+ ,, i*II+ El+ a+ + + -° , .1111111 ' +-+:'

.+'+H , I .
2.z0 4 ))

w

4w)

A 0

E

00

'0, -0

0

0 n

0)A 0.

SA .

0. 5

o o

2 o4)UO.

I--
0

0 T

uo 1 -5

o~~

C))
.O4

0r~

~ ..

f) W)

o WO0 -. 0 -

I-0

to0 0~ 0A V~

2A o

>., 0 m .' U

-00c Or.- r. : r:

0) 144 -4 ; --.

mo-2

Mo.

Cu
x x

x x

400

,,,,

o. go

. 011

V 0
z

U rA

-woto u

CL.4

.22

0 r u OAv
zx

00

u U 

00

0- 0~ 

V u

'liii~J~ilh ___ Iu :.

.9
x

0 x

C4)~ - 0~ 4) 044)

Ib - 0

0 o

.0 0 *- r U

rv 'S 750. t2 Z

o -0

V) 0'

U,S 0

Jd -

Si

.~-0

.~'U0

10 IS4)

9%. 4))

4)

0 ' 0m
@0

'4 xo.;-j 

IM

+I

lillli I Il

i>i .

Ad iiiI 1
W 00

00

00> 0 o

09 3V6 o0c, A 0 i o .- L) U 

o "a.0 0 o 'oa.0.

-h -

2uU

U~~ 0 1S

cq2~ (A0 <
x

x ~ x

>+

00 - s 0 0

02-2 0

CL
xx

.0U

CO4) a

00 0,

-vr0

020

000

a 0

w- 0

4W))
, X X X

zx

0 C

b.

0u 0

~ .4)

40.~ S

Cu n

.10

13 ~ * C2 C

- 40.

"a~ 0 .- l4

0 0 0 ?

0

UU

0 q

'.'-X S

p @ 

0_ o -o . , , .,
z

0 0

oo

v 0

cr 0

or + 0

+ 0

0) 0

U)0

Oo 0 00a

n6 g a'. 0

.0 0 ts 0

144
40..

-A 0

4-h
u 

ot

CL~ 00Q. 

uz t

lu:3.00

oJ .4 0 0 0

u 8 ~ u~ > o. IZ u "00 o

> )C

u~ I- 
0

C6 0

rz

uI /

lidI

. - o

*1 g ie " " . i .
z×

0 0

oo 0 ' .L. . IiEli

•0 c a 00 :

bL

o~ 0~1. U

- - - - ----
z

ILC0

7E
o aPi a

'g8 0

All~Qy
X

i x×

02.

'I iII ..

Ioi

11111A3

al

'~JUIi
z~

~o~o a 0 X

=x

*.1 0

;]2 g

00

.0 -9- 0 '

I= .2 o

4) - 90

- - 0

CL

2 .2 00 0 R
xx

.d,

w 00

_ _ _ _ _ _ _ __ _ O- .d0 

WAC

Ir 0. a 0

8 441 g 4 ~ 20 ~ C --o 9 o

Ic k
zx

zx

2.0 .0

0 0 0 0

I-c

aS.

-0 0 0o

0 0 0

00 1. 0

0 0 -. 0

-- ' 0 00=

u Q0 r 4

454)

U

Zn 1.0o 0 

U0'.0

04w)o
zx

iii

040

00

44-
zX

0-

0. 0

0 0 r

0.~-0 0

.00

C'

CL0

0 .0 0 0 co. 0 U

8 6 0 2 0 O0 B

0U 0

o 0

.~0

U U U>
ILth..

L 0
z

z

0 00

W4o 00

V -bo00

.00

Ub
X x

x X

tZ

,..I.

= Jo

Mi

• _ _ _ .0. _ _ _ _ i!II _ _ _ _ _

UA

UOU
0U 

-

a.. , >1:= -e

OA

U El

, *0

]Ah.
z

W W

WWJ

~~ol

.2 40 A W CY

Ulo

'rA ~ ~ ~ go I 

0o
x X

z

zx x

o 0 0 0 0

0) cc 0 0) 0

N~~~ 'Q' - .- 0 0 -0)4'

00

A..

0A - 0~c 
X

X x x

x x

ccot 0

lu c

0 00

0o r_

"'-an

0 00

0 0 ,1

: ~~0 o

00w
Gx

0 0

J2 C CY

s. 0

4) g:

12 -

78 0.

0

LL

UD
x

• I I ; t1I!, = 

4i !ti ii

I j'Ij.o =

tI

tt~ 00

I ~04

II ti :
Xx x

zx

C

y CY ce =0

on w cn-r Ol 0u1

00 w

W.2o .2 01 01 0

40 Lu a 90

O 8 rr cl 21-0 z2 0 .10

co
xx

a

"U1 iiix

xi x

1C11

lii7i'f
00

00

r0

I- , 8 C2 

,, 8, CY

2 2c 2V. :

o CI0 e

CL (44 '~Q0 L;4 oo 0 m .0 0~ rA Cu tg

07. - I~- .~ Cu.

6. a~ 0

0 co ~ cc

to. -6. V) 8 -

0Cu

0Cuo 0u *Cua

- >

Q.,0 0C cs u0 
xx

z x

E -a

U3-

0 0g - ol. o an

____ il ilJilJl .CL*~ 3 C4

.= 7gU

rA~

,0 - boa

0 1

LI__ _ i
= 0

to 00 .

+

|01"' I: X " e o . .-, '

-R o an *= 0-' "

0~ -0 4)E

A 1 = .. A -o v ,. ,,'

)0 0

rA 00

0 2=

0

02

I) 0

CL

-J

rh.

0 'A "

E.' s' .. ;.o -

0 N a-Qbe5

cc0

"000

0- ca

U~ A~
z

z

R. 0- co0

-~ ~ ~~i~ ce~ 0 - 0 L,=

0 84

.8.-0;~~

0L 0

00

5*~~C 0 00~. ~
x

E +

12 -0 0 ,

0 C

-e-Y

C3- ro : 1 .

C0 :u oo0a rA10 0

0S S 0 02 - 0 g 0

10, 10

- )' . U

00

Co~~1 0.8C C C .5 *0 0 0 'A~- ~ .5.

.IR 0 rA -0 'A

4.-

~ 0 UU0

CL
xx

x x

A oo

71

I40-

al 0- .0 0

48. 40C
Or

0 0 0.

00

rirA

0

S2 .. 2 20. .08 4

0,0

=or r- Ai

0 w 0O 0

,. r

000 C3 0 C3 o 0 ______ 0

814,.0, .10 U .e O 8 S00 ou 0 , 

cu

U.~Uc0

~~4 Oo~2

E~'~ 0u
40-

o 0 .

VV

+Q

=0o

R .5
fu4

20.0 00 a 11

o0 CL0

_ oss il o IL

Vu 4A

____ ___
zX

.Sri

93)

000

00

... ta , <42.

iwO

)_. . .

ol 0

lu A

A 3 c
z

rA x X

U,z

Ig0

UU

c O +, -

0'

AHl-'-

r. C H .= W) c

C)-C -O 

L U

rL 0

0 r C.) 0

Or >-~U

an 0

- --

~ -~ ~0 W

~LL0
xx

zx

X

,

Oa + 9o. 0. .9 .0

UU

0

.-. 0

a i -