a

GAO

United States Government Accountability Office

Report to the Ranking Minority Member, 

Committee on Armed Services, House of 

Representatives

July 2004 MILITARY 

EDUCATION

DOD Needs to 

Develop Performance 

Goals and Metrics for 

Advanced Distributed 

Learning in 

Professional Military 

Education

GAO-04-873
www.gao.gov/cgi-bin/getrpt?GAO-04-873.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Neal P. Curtin 

at (757) 552-8100 or curtinn@gao.gov. 

Highlights of GAO-04-873, a report to 

the Ranking Minority Member, 

Committee on Armed Services, 

House of Representatives 

July 2004

MILITARY EDUCATION

DOD Needs to Develop Performance 

Goals and Metrics for Advanced 

Distributed Learning in Professional 

Military Education 

DOD does not have specific performance goals and metrics with which 

to assess ADL effectiveness in PME. Furthermore, although GAO and 

private-sector organization have established frameworks for assessing the 

effectiveness of educational programs by focusing on metrics for learning 

outcomes—that is, the knowledge, skills, and abilities that students attain 

through learning activities—DOD’s oversight focuses instead on educational 

inputs such as facilities, student to faculty ratios, and student body 

composition. Since ADL is still a new and evolving tool, systematic 

evaluative processes have not yet been required. Without clear goals and an 

effective process for evaluating the results of ADL application, DOD cannot 

ensure that its program is achieving an appropriate return on investment and 

other goals. 

The criteria for converting PME courses and curricula to ADL vary by school 

and by military service, are based on subjective choices as to which content 

is suited for online delivery, and are focused solely on nonresident programs. 

The private sector similarly lacks systematic criteria in its use of ADL. 

However, DOD’s implementation of ADL programs for PME compares 

favorably with private-sector institutions. 

Cultural, technological, and resource challenges affect ADL implementation. 

For example, some military policies reflect a lower estimation of the value of 

nonresident PME, and many respondents to a survey of ADL students and 

alumni indicated that its quality and achievement of outcomes did not 

compare favorably, in their view, with those of resident education programs. 

The technological challenges of balancing computer access with network 

security, along with resource challenges of funding and increased burdens 

on limited administrative staff, are additional concerns. 

DOD Nonresident PME Programs Currently Using ADL Applications 

When and how 

developed 

No. of 

students 

 Program 

length 

No. of 

courses 

U.S Army War 

College 1999 (In-House) 654 2 yrs. 

10 + 2 resident 

courses 

Naval War College 2002 (Contractor 

and In-House) 1,799 18-24 mos. 3 

Air Command and 

Staff College 1999 (In-House) 12,069 Up to 18 mos. 6 

Source: GAO. 

As part of its transformation to 

prepare the armed forces to meet 

current and future challenges, the 

Department of Defense (DOD) is 

expanding its use of advanced 

distributed learning (ADL) 

techniques in senior- and 

intermediate-level officer 

professional military education 

(PME) (see table at right). ADL 

instruction does not require an 

instructor’s presence, and it 

facilitates the use of varied learning 

management systems. To date, the 

application of ADL has been 

targeted to nonresident students. 

To determine whether DOD uses a 

systematic process for evaluating 

the results of ADL application, 

GAO was asked to examine DOD’s 

metrics for assessing program 

effectiveness, to compare DOD’s 

criteria for converting courses to 

ADL with those of private-sector 

institutions, and to identify the 

challenges to ADL implementation. 

GAO recommends that the 

Secretary of Defense promote 

(1) the development of specific 

performance effectiveness goals 

for ADL in PME schools and (2) the 

use of ADL technologies to provide 

and establish metrics for learning 

outcomes. DOD partially concurred 

with the first recommendation and 

fully concurred with the second. 

DOD supports the use of specific 

effectiveness goals for PME, but 

believes such goals are not 

appropriate for any specific 

delivery method. 
Page i GAO-04-873 ADL in Military Education

Contents

Letter 1

Results in Brief 2

Background 4

DOD Does Not Have Specific Metrics for Assessing Performance 

Goals or Learning Outcomes 6

ADL Conversion Varied by School and by Service Based 

on Subjective Assessments of Content Suitability 13

Cultural, Technological, and Resource Barriers and Challenges 

Affect ADL Implementation in PME Programs 16

Conclusions 19

Recommendations for Executive Action 20

Agency Comments and Our Evaluation 20

Appendixes

Appendix I: Scope and Methodology 22

Appendix II: Methodology for Our Survey of Nonresident PME Students 

and Graduates 25

The Study Population 25

Developing the Survey 26

The Sample Design and Administration 26

Sampling Error 27

Nonsampling Error and Data Quality 28

Appendix III: Survey Responses 29

Introduction 29

Frame of Reference 29

Appendix IV: ADL Applications and Additional Features 

of Nonresident Programs 41

U.S. Army War College 41

Naval War College 41

Air Command and Staff College 42

Joint Forces Staff College 42

Additional Features of Nonresident PME Programs 43

Appendix V: Comments from the Department of Defense 45

GAO’s Comment 47
Contents

Page ii GAO-04-873 ADL in Military Education

Tables Table 1: Enrollment Statistics for Resident and Nonresident 

Students for Each Senior- and Intermediate-Level PME 

School for the Academic Year 2003-2004 5

Table 2: Disposition of Sample 27

Table 3: Levels-of-Learning Definitions 44

Abbreviations

ACSC Air Command and Staff College

ADL advanced distributed learning

DOD Department of Defense

JFSC Joint Forces Staff College

NWC Naval War College

PME professional military education

USAWC U.S. Army War College

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately.
Page 1 GAO-04-873 ADL in Military Education

United States Government Accountability Office

Washington, D.C. 20548

A

July 30, 2004 Letr

The Honorable Ike Skelton

Ranking Minority Member

Committee on Armed Services

House of Representatives

Dear Mr. Skelton:

As part of its transformation to prepare the armed forces to meet current 

and future challenges, the Department of Defense (DOD) is transforming 

the way it trains to favor more rapid and responsive deployment. A 

significant element of its training transformation strategy is the application 

of advanced distributed learning (ADL), a technique of instruction that 

does not require an instructor’s presence; can use more than one form of 

media; and emphasizes the use of reusable content, networks, and learning 

management systems. DOD has been expanding its use of ADL in its 

program of professional military education (PME). PME provides military 

officers with a wide array of college-level academic courses in both 

resident and nonresident settings; to date, the application of ADL has been 

targeted to nonresident students. As a new tool, ADL is being examined to 

determine whether DOD is applying a systematic performance evaluation 

approach, particularly in light of the increased rate at which 

servicemembers are being deployed worldwide. Without clear goals and an 

effective process for evaluating the results of ADL application, DOD cannot 

ensure that its program is achieving an appropriate return on investment 

and other goals.

We were asked to review DOD’s use of ADL in senior- and intermediatelevel officer PME, and specifically:

1. to examine the metrics DOD uses to assess the effectiveness of ADL 

in PME,

2. to determine what processes and criteria DOD uses to select the 

courses or curricula it converts to ADL and how these criteria compare 

with those of other institutions in meeting ADL objectives in 

nonresident education, and

3. to identify what barriers and challenges exist for implementing ADL 

in PME.
Page 2 GAO-04-873 ADL in Military Education

We also reviewed and assessed the policies and guidance of several DOD 

offices responsible for providing oversight for PME activities. These offices 

included the Under Secretary of Defense for Personnel and Readiness and 

the Joint Staff’s Joint Education Branch. We also studied experience in the 

private education sector and in other parts of the government in measuring 

the effectiveness of education programs. In addition, we surveyed 

437 current students and graduates of senior- and intermediate-level PME 

programs to obtain their perspectives on their PME experience. 

Appendixes I and II describe our scope and methodology in more detail.

We conducted our review from March 2003 through June 2004 in 

accordance with generally accepted government auditing standards.

Results in Brief DOD does not have specific performance goals or metrics with which to 

assess the effectiveness of ADL in PME, and its oversight activities focus 

on educational inputs rather than on learning outcomes. While DOD 

believes ADL has had a positive impact, its views are based on anecdotal 

information; clear goals and an effective process for evaluating results of 

ADL implementation are absent. Although numerous organizations have 

roles in providing oversight of PME activities, with several specifically 

responsible for ensuring that PME meets general standards of 

accreditation, DOD accreditation activities, like those in the private sector, 

focus primarily on educational process inputs—for example, facilities or 

student to faculty ratios. But we and a private-sector organization have 

established guidelines and frameworks for assessing the effectiveness of 

educational programs that stress a focus on measurable outcomes—that is, 

the knowledge, skills, and abilities a student acquires from a course. 

Furthermore, ADL has a unique ability to capture, retain, store, and 

document interactions in an online environment, thus providing the 

opportunity to demonstrate student skill improvements and to customize 

performance metrics. However, we found no evidence to indicate that DOD 

is using this ability. 

The processes for converting PME courses and curricula to ADL vary 

by school and by military service, and they feature a mixture of in-house 

and contractor approaches. PME schools generally focus their ADL 

applications on nonresident education programs, and they tend to 

convert an entire curriculum as a package rather than in a modular, 

course-by-course manner. No systematic criteria inform PME schools’ 

decisions about which courses or curricula to convert to ADL. Instead, 

schools make individual, subjective choices as to which content is best 
Page 3 GAO-04-873 ADL in Military Education

suited for online rather than another delivery method. Notably, we found 

that nonmilitary educational institutions also lack systematic criteria when 

converting courses or curricula to ADL. DOD’s approaches are in fact 

consistent with mainstream practice, and in some cases, compare 

favorably with the best implementations.

Numerous cultural, technological, and resource challenges affect ADL 

implementation in PME programs, some which may affect ADL expansion 

or maintenance. Cultural issues include concerns by PME school officials 

about ADL’s acceptance as an appropriate learning method and the 

appropriate extent of its use for nonresident education. In our survey, 

nonresident students expressed concerns about the quality of their 

courses, regardless of nonresident delivery method, as compared with 

those taken in residence. Technological challenges, particularly those 

concerning the optimal balance between student access (computer 

availability and freedom of information) and network security (protection 

of sensitive information and use of military installation firewalls), remain 

to be addressed. With respect to resources, there are concerns about ADL’s 

ability to compete for limited funding and about the potentially 

burdensome administrative impact on nonresident program staff.

To better assess the effectiveness of ADL in professional military 

education, we recommend that DOD promote (1) the development of 

specific performance effectiveness goals for ADL in PME schools and 

(2) the use of ADL technologies to provide and establish metrics for 

learning outcomes.

In commenting on a draft of this report, DOD partially concurred with our 

first recommendation and fully concurred with the second. DOD supports 

the use of specific effectiveness goals for PME, but believes such goals are 

not appropriate for any specific delivery method. DOD stated that current 

accreditation practices are already promoting the data collection 

capabilities of ADL technologies for assessing multiple delivery methods.
Page 4 GAO-04-873 ADL in Military Education

Background Each military service has separate PME schools for senior- and 

intermediate-level officers. As defined by the Joint Staff’s Officer 

Professional Military Education Policy,1

 the senior-level schools, typically 

for O-5 and O-6 ranked officers, focus on warfighting within the context of 

strategy. The intermediate-level schools, typically for O-4 ranked officers, 

focus on warfighting within the context of operations.2

 (See table 1 for a list 

of PME schools and enrollment totals.)

The senior- and intermediate-level PME schools are not alike in terms of 

program offerings for resident and nonresident students. As indicated in 

table 1, while all senior-level PME schools offer resident programs, only the 

Army War College and the Air War College have analogous nonresident 

programs. Also as indicated in table 1, all intermediate-level PME schools 

offer resident and nonresident programs.

DOD has approximately 39,318 students enrolled in its senior- and 

intermediate-level PME schools. The vast majority of these enrollees 

are nonresident students. Of the total enrolled, approximately 3,788, or 

10 percent, are taking course work as resident students; the rest, or 

90 percent, are nonresident enrollees.

1

 Department of Defense, Chairman of the Joint Chiefs of Staff Instruction 1800.01A, 

Officer Professional Military Education Policy, December 2000. 

2

 There are also senior- and intermediate-level schools sponsored by DOD through its 

National Defense University. These schools are designed to educate officers on joint 

matters. The senior-level schools are the National War College, the Industrial College of 

Armed Forces, and the Joint and Combined Warfighting School–Senior at the Joint 

Forces Staff College. The intermediate-level school is the Joint and Combined Warfighting 

School–Intermediate at the Joint Forces Staff College.
Page 5 GAO-04-873 ADL in Military Education

Table 1: Enrollment Statistics for Resident and Nonresident Students for Each Senior- and Intermediate-Level PME School for 

the Academic Year 2003-2004

Source: DOD.

Note: N/A–School without a nonresident component.

a

According to Army Command and General Staff College officials, the nonresident student total 

fluctuates and could be plus or minus 2,000 students on any given day.

b

Naval War College’s nonresident programs are offered at the intermediate level (equivalent to the 

Naval Command and Staff College) through its College of Distance Education.

c

The Marine Corps College of Continuing Education is the nonresident component of the Marine Corps 

Command and Staff College.

Nonresident PME exists to provide PME to a larger population than can be 

supported in resident institutions. Nonresident PME provides alternative 

learning-style options for officers not selected for residence or unable to 

participate in residence due to operational commitments. The military 

services have had nonresident PME programs for many years. The Naval 

War College (NWC) has had a department for correspondence courses 

since 1914. The U.S. Army War College (USAWC) has provided a 

nonresident course offering since 1968. The Air Force’s nonresident 

programs were created in 1947 for its senior-level PME school and 1948 for 

its intermediate-level PME school.

PME institutions Resident students Nonresident students

Joint Senior-Level Schools

National War College 200 N/A

Industrial College of the Armed Forces 309 N/A

Joint Combined-Level School 

Joint Forces Staff College 229 N/A

Senior-Level Schools

Air War College 265 6,100

Army War College 340 654

College of Naval Warfare–Naval War College 209 N/A

Marine Corps War College 16 N/A

Intermediate-Level Schools

Air Command and Staff College 587 12,069

Army Command and General Staff College 1,183 10,000a

College of Naval Command and Staff–Naval War College 256 1,799 b

Marine Corps Command and Staff College 194 c

Marine Corps College of Continuing Education c 4,908

Total 3,788 35,530
Page 6 GAO-04-873 ADL in Military Education

Paper-based correspondence is the traditional nonresident PME delivery 

mode. Students complete correspondence courses individually with limited 

faculty contact. Course materials and submissions are exchanged between 

students and faculty primarily by mail. PME schools have implemented 

other delivery modes, including seminars conducted at remote sites by 

PME faculty and CD-ROM distribution. Increasingly, PME schools are using 

ADL3

 techniques in their nonresident program offerings.

Several ADL applications are currently in use at senior- and intermediatelevel PME schools, and all of them are focused on nonresident programs. 

They are offered at the U.S. Army War College, the Naval War College, and 

the Air Command and Staff College (ACSC). A planned ADL offering for 

reserve component staff is under development at the Joint Forces Staff 

College. See appendix IV for details on these programs. The addition of an 

ADL application for the Army Command and General Staff College 

nonresident PME course is anticipated for fiscal year 2005.

DOD Does Not 

Have Specific Metrics 

for Assessing 

Performance Goals or 

Learning Outcomes

DOD does not have specific performance goals and metrics to assess the 

effectiveness of ADL in PME. While DOD believes ADL has had a positive 

impact, its views are based on anecdotal information, rather than a 

systematic performance measurement. Thus, DOD cannot determine 

whether ADL is meeting performance goals in comparison to other delivery 

methods. Although numerous organizations are providing oversight of PME 

activities, with several specifically responsible for ensuring that PME 

meets general standards of accreditation, these organizations do not focus 

on student learning outcomes—that is, the knowledge, skills, and abilities a 

student acquires from a course. Instead, DOD accreditation activities, like 

those in the private sector, focus primarily on educational process inputs, 

such as quality of facilities and student faculty ratios. We and a privatesector organization have recently established guidelines and frameworks 

for assessing the effectiveness of educational programs that stress a focus 

on measurable outcomes. ADL is a new and evolving tool for which 

systematic evaluation requirements have not been established. ADL has a 

unique ability to capture, retain, store, and document interactions in an 

online environment, which provides the opportunity to demonstrate 

3

 Advanced distributed learning, as defined by DOD’s April 1999 ADL Strategic Plan and May 

2000 ADL Implementation Plan, expands distance learning by emphasizing computer-based 

instruction; common standards; and use of reusable content, networks, and learning 

management systems in an “anytime, anyplace” environment.
Page 7 GAO-04-873 ADL in Military Education

student skill improvements, and thus to customize performance metrics. 

However, we have found no evidence to indicate that DOD is utilizing this 

ability.

Numerous Organizations 

Have Roles in Providing 

Oversight of PME Activities

Numerous oversight organizations review PME activities, with several 

organizations specifically designed to ensure that PME conforms to general 

standards of accreditation. The preeminent mechanism for oversight is the 

Joint Chiefs of Staff’s Process for Accreditation of Joint Education. The 

process is designed to provide oversight and assessment of PME 

institutions for purposes of strengthening and sustaining Joint Professional 

Military Education.4

 It is a peer-review process involving a self-study 

component and a team assessment. The review sequence includes 

certification, accreditation, and reaffirmation of accreditation status. 

Accreditation can currently be granted for up to 5 years, and all PME 

programs with current ADL applications are Joint Staff-accredited. The 

Joint Staff also sponsors the Military Education Coordinating Council, an 

advisory body composed of high-ranking PME leadership. The purpose of 

the Council is to address key issues of interest for joint military education, 

to promote cooperation and collaboration among member institutions, and 

to coordinate joint education initiatives.

The military services have responsibility for the service PME institutions in 

terms of managing PME content and quality and conducting all levels 

within the guidelines of the military educational framework. Consistent 

with Title 10 of the United States Code, the Secretary of Defense requires 

that each PME institution periodically review and revise curriculum to 

strengthen focus on joint matters and on preparing officers for joint duty 

assignments.

PME is also reviewed by other internal and external organizations. Each 

PME institution has a Board of Visitors/Advisors that provides guidance 

over PME activities. The Board of Visitors/Advisors is composed of military 

and/or civilian academic officials who are nominated by PME schools and 

appointed by service secretaries to provide advice on educational and 

institutional issues. Service PME institutions have other internal and 

4

 Joint Professional Military Education is a Joint Chiefs of Staff-approved body of objectives, 

policies, procedures, and standards supporting the educational requirements for joint 

officer management. Joint Professional Military Education is a portion of PME that supports 

fulfillment of the educational requirements of joint officer management.
Page 8 GAO-04-873 ADL in Military Education

external advisory committees that perform activities such as providing 

advice, communicating feedback from major commands, and conducting 

curriculum review. Service Inspector General offices have conducted 

periodic reports and assessments on PME schools. The military services’ 

education and training commands also provide oversight of PME activities, 

though not day-to-day administration. Additionally, private-sector regional 

accreditation agencies assess senior- and intermediate-level PME 

programs. Their accrediting activities generally guide the Joint Staff’s 

review process.

Performance-Effectiveness 

Metrics for ADL 

Implementation Are Lacking

PME schools have not established, and oversight organizations have not 

reviewed, specific goals or metrics of performance effectiveness for ADL 

implementation. As was stated in our recently issued guide for establishing 

a framework for assessing training and development efforts in the federal 

government, “it is increasingly important for agencies to be able to evaluate 

training and development programs and demonstrate how these efforts 

help develop employees and improve the agencies’ performance.”5

The Sloan Consortium—a private-sector organization that maintains a 

repository of information on distance education—views metrics as 

crucial for assessing program effectiveness. For example, metrics can 

(1) demonstrate that the “learning effectiveness” of nonresident education 

is at least as good as that of its resident counterpart, (2) identify cost 

comparisons that can be used to develop better strategic plans, and 

(3) provide information on student retention and completion rates. As was 

stated in our report on oversight for the military academies, such elements 

embody the principles of effective management, in which achievements are 

tracked in comparison with plans, goals, and objectives, and the 

differences between actual performance and planned results are analyzed.6

PME schools identified advantages of ADL over other means of delivery, 

but the advantages appeared to be anecdotally derived. PME school 

officials stated that ADL has resulted in quality improvements in PME 

delivery, especially when compared with paper-based correspondence. 

5

 U.S. General Accounting Office, Human Capital: A Guide for Assessing Strategic 

Training and Development Efforts in the Federal Government, GAO-04-546G

(Washington, D.C.: March 2004).

6

 U.S. General Accounting Office, Military Education: DOD Needs to Enhance 

Performance Goals and Measures to Improve Oversight of Military Academies, 

GAO-03-1000 (Washington, D.C.: Sept. 10, 2003).
Page 9 GAO-04-873 ADL in Military Education

These advantages include (1) better facilitation of student and faculty 

interaction; (2) increased flexibility in modifying course material; 

(3) reductions in time required to complete programs; (4) better leveraging 

of resources for administrative support; and (5) establishment of learning 

management systems that monitor student progress and produce 

management reports. But there were no indications that evidence for these 

advantages were based on an evaluative effort to compare differences 

between ADL and paper-based correspondence courses. Since PME 

schools have not detailed a comprehensive process for evaluating ADL 

benefits over paper-based correspondence, it cannot be determined 

whether ADL is meeting performance goals based on appropriate returns 

on investment, student retention, student access to courses, or other goals 

that schools use to measure program effectiveness.

Additionally, we did not observe any oversight agency focus on specific 

metrics of ADL effectiveness. According to Joint Staff officials, they 

perform reviews of nonresident programs as part of their accreditation 

activities. However, their reports focus on the nonresident program as a 

whole and not on particular methods of delivery. ADL is a new and evolving 

tool, and a systematic assessment of these applications has not yet been 

required. The three regional accreditation agencies that review PME 

schools with ADL implementations show variances in nonresident program 

evaluation policy.7

 One agency stated that nonresident programs are not 

separately evaluated, although the programs may be included within the 

scope of the institution’s existing accreditation. Another agency stated that 

additional procedures must be performed before nonresident programs are 

included within the scope of the institution’s accreditation. The third 

agency required schools to evaluate its nonresident programs to ensure 

comparability to resident programs. In addition, we have not observed any 

Office of the Secretary of Defense or Board of Visitors/Advisors reviews in 

relation to ADL effectiveness for nonresident PME.

While we did not observe measures of effectiveness specifically geared 

toward ADL applications, PME schools with ADL applications did perform 

program effectiveness assessments for nonresident education by the way 

of student satisfaction assessments as part of the Joint Staff accreditation 

7

 The Middle States Association of College and Schools is the regional accrediting agency for 

the U.S. Army War College. The New England Association of Schools and Colleges is the 

regional accrediting agency for the Naval War College. The Southern Association of Colleges 

and Schools is the regional accrediting agency for the Air Command and Staff College.
Page 10 GAO-04-873 ADL in Military Education

process. These assessments used in-course student surveys, graduate 

surveys, and supervisory surveys to obtain feedback as part of a systematic 

approach to instructional design and to update and improve curriculum 

offerings.

• USAWC performs surveys of students, alumni, and general officers with 

USAWC graduates in their commands. Students are surveyed for each 

course regarding particular aspects of the course and general degrees of 

satisfaction. A survey of alumni is conducted every 2 years. A general 

officer survey, designed to assess general officer impressions of alumni, 

will now be conducted annually instead of every 3 years, as in the past. 

Prior feedback from general officer surveys reported that the 

curriculum should emphasize application of strategic thinking to 

national security issues. USAWC also performs internal course 

evaluations as part of its curriculum assessment process. USAWC 

faculty members are required to undergo training to provide a degree of 

standardization in instruction and evaluation. This standardization, 

especially for evaluation, is more stringent for nonresident education. 

USAWC can conduct trend analyses for student performance and 

student satisfaction to determine statistical significances.

• NWC uses student and alumni surveys to assess the academic program’s 

effectiveness. Depending on the department, student assessments 

include daily sessions critiques, lecture critiques, end-of-course 

critiques, major exercise critiques, and exam critiques. Alumni are sent 

questionnaires 2 years after graduation asking for feedback on their 

educational experience. All academic departments conduct an extensive 

analysis of various student surveys to determine areas of the curriculum 

that are not meeting student needs so that these areas can be improved. 

Surveys are based on standards promulgated by accrediting agencies 

and external organizations to help objectively measure institutional 

excellence. Resident and nonresident student programs are measured 

the same since a single faculty is responsible for both. Peer evaluation of 

faculty members is used to sustain teaching method quality.

• ACSC uses internal and external evaluations at all phases of its 

curriculum development process. It conducts end-of-course surveys 

that focus on delivery and educational support and end-of-year surveys 

for students to provide feedback about whether they believed the school 

(1) prepared them to lead commands, (2) accomplished its mission, 

(3) was institutionally effective, and (4) was beneficial to 

professional development. Surveys are also given to graduates and 
Page 11 GAO-04-873 ADL in Military Education

graduate supervisors to obtain perspectives on whether the 

school (1) accomplished its mission and institutional effectiveness; 

(2) enhanced graduates’ ability to think operationally and critically; 

(3) prepared graduates to assume leadership duties, and (4) made the 

experience valuable in professional development.

Metrics for Learning 

Outcomes Are Lacking

Student learning outcomes, as stated by the Council for Higher Education 

Accreditation—a national association representing accrediting 

organizations—are “properly defined in terms of the knowledge, skills, and 

abilities, that a student has attained at the end (or as a result) of his or her 

engagement in a particular set of higher education experiences.”8

 PME 

schools generally are not assessed for student learning outcomes as a 

means of determining program effectiveness. The Joint Staff’s 

accreditation organization responsible for assessing PME schools has 

primarily focused on inputs to the educational process. As detailed in its 

policy, its educational standard assessment and self-study requirements 

focus on internal aspects such as organizational structure, facilities, 

curricula, student to faculty ratios, student body composition/mix, and 

faculty qualifications. However, as stated in our recently published guide 

for assessing training and development programs, the focus on evaluating 

activities and processes takes away from evaluating training and 

development’s contribution to improved performance, reduced costs, or 

greater capacity to meet new and emerging transformation challenges.9

 The 

Joint Staff has identified the usefulness of student learning outcomes and is 

currently in the process of developing student learning outcomes for PME 

and procedures to include them in the accreditation process.

8

 Council for Higher Education Accreditation, Statement of Mutual Responsibilities 

for Student Learning Outcomes: Accreditation, Institutions, and Programs 

(Washington, D.C.: Sept. 2003).

9

 See GAO-04-546G.
Page 12 GAO-04-873 ADL in Military Education

Our recently published report on distance education states that there is 

increased interest in using outcomes more extensively as a means of 

ensuring quality in all forms of education, including nonresident 

education.10 The Council for Higher Education Accreditation has issued 

guidelines on nonresident education and campus-based programs that call 

for greater attention to student learning outcomes, and the congressionally 

appointed Web-based Education Commission11 has also called for greater 

attention to student outcomes. The Commission said that a primary 

concern related to program accreditation is that “quality assurance has 

too often measured educational inputs (e.g., number of books in the 

library, etc.) rather than student outcomes.”

Private-sector educational institutions are just beginning to emphasize 

the evaluation of learning outcomes as a viable measure of program 

effectiveness. For example, the University of Maryland University College, 

a school with a comparably large distance education program and which 

serves a large number of military personnel, is piloting a project to identify 

and measure learning outcomes in five general areas—writing efficiency 

and oral communication, technology fluency, information literacy, 

quantitative literacy, and scientific literacy. The university will use 

knowledge captured by its distance education database to serve as a basis 

for this determination.

Accrediting agencies and our recent report on training and development 

program assessments are also emphasizing the evaluation of learning 

outcomes as a measure of program effectiveness. Some of the regional 

agencies that accredit programs at the senior- and intermediate-level PME 

schools generally recognize the importance of student learning outcomes 

and have instituted practices that reflect some aspects of a systematic, 

outcome-based approach called for in GAO-04-279.12 However, these 

agencies vary in the extent to which standards and policies address student 

10 U.S. General Accounting Office, Distance Education: Improved Data on Program Costs 

and Guidelines on Quality Assessment Needed to Inform Federal Policy, GAO-04-279

(Washington, D.C.: Feb. 26, 2004).

11 The Congress established the Web-based Education Commission to prepare a report to the 

President and the Congress that contains recommendations for legislation and 

administrative actions, including those pertaining to the appropriate federal role in 

determining the quality of educational software products. Members of the Commission 

included senators, representatives, and leaders from postsecondary institutions.

12 See GAO-04-279.
Page 13 GAO-04-873 ADL in Military Education

learning outcomes for distance education. Our training and development 

assessment guide states that agencies need credible information on how 

training and development programs affect organizational performance, and 

that decision makers will likely want to compare the performance of these 

programs with that of other programs. Furthermore, programs lacking 

outcome metrics will be unable to demonstrate how they contribute to 

results.

We surveyed nonresident PME current students and graduates to obtain 

their perspectives on the achievement of PME learning objectives and 

PME’s impact on their career objectives. (See appendix III for presentation 

of survey results.) Because we only surveyed nonresident students, we 

could not compare the results with those of resident students. However, we 

believe the data can be useful for DOD to consider in its continuing study of 

program effectiveness.

ADL Can Be Used to 

Capture Valuable 

Measurement Data

ADL has a unique ability to capture, retain, store, and document 

interactions in an online environment, which provides the opportunity 

to demonstrate student skill improvements, and thus to customize 

performance metrics. Since work is done on a computer, various data 

points are automatically collected as a student works, including the time 

spent, specific pages of the text visited, use of online help, and 

communication with others. University of Maryland University College 

officials pointed out ADL’s unique ability when compared with other 

delivery methods to retain, capture, store, and document baseline data that 

can be used as the basis for performance metrics. These officials said they 

would use such data in designing performance measures for learning 

outcomes. However, we found no evidence to indicate that DOD is using 

this ability. DOD may be missing an opportunity to enhance its ability to 

measure effectiveness.

ADL Conversion Varied 

by School and by 

Service Based 

on Subjective 

Assessments of 

Content Suitability

The processes for converting PME courses to ADL varied by school and by 

military service, and they feature a mixture of in-house and contractor 

approaches. PME schools generally focus their ADL applications on 

nonresident programs to improve their efficiency and effectiveness. In 

most cases, conversion decisions were made in collaboration with school 

academic boards. PME schools did not identify any systematic criteria as 

the basis for their decisions as to which courses or curricula to convert to 

ADL. They subjectively focused ADL conversion on the suitability of 
Page 14 GAO-04-873 ADL in Military Education

content for Web-based applications. Curriculum conversions were made 

because of a DOD-wide need (1) to improve access to a diverse officer 

corps and (2) to increase the efficiency of educational delivery. Since 

nonmilitary educational institutions also lack systematic criteria for 

converting courses or curricula to ADL for nonresident education, DOD’s 

approaches are in fact consistent with mainstream practice, and in some 

cases, compare favorably with best practices in nonmilitary education.

DOD Processes and Criteria 

for ADL Conversion Varied

The processes for converting PME courses to ADL varied by school and by 

military service, and they feature a mixture of in-house and contractor 

approaches. However, the conversions were focused on the schools’ entire 

nonresident programs. USAWC’s and ACSC’s ADL applications were 

developed and managed by in-house staff and faculty. USAWC and ACSC 

used staff consisting of instructional designers and courseware developers 

interacting with respective faculty to develop courses. NWC’s ADL 

application combined the use of contractor and in-house support. 

Contractor staff created Web-based applications for two of the three 

courses in NWC’s curriculum. NWC officials learned enough from 

contractor efforts to create a Web-based application for the remaining 

course with in-house staff. In all cases, the ADL applications were applied 

to affect the entire nonresident curriculum and, in most cases, were 

preceded by reviews and final decisions from the schools’ academic 

boards.

The PME schools did not identify any systematic criteria that inform their 

selection of course for conversion to ADL. Rather, they made subjective 

decisions as to the appropriate parts of the curriculum that should be 

delivered online based on content suitability. While USAWC’s application is 

delivered mostly through the Internet, print media delivers a portion of the 

course content as well. USAWC’s application also includes two 2-week 

resident components in which students are brought together to achieve 

learning objectives best suited to resident instruction. These objectives 

include verbal communication, interaction in live settings, interpersonal 

skills used in direct relationships, and other skills that are important 

components of the resident experience. USAWC’s application also uses 

asynchronous “threaded discussions,” in which faculty members initiate 

online discussions with students on various academic topics. NWC’s and 

ACSC’s applications, which do not include resident components, blend 

content delivery using print media, CD-ROM, and the Internet. In NWC’s 

application, print media is used for material that is mostly text and requires 

limited interactive capabilities; CD-ROMs are delivered to students for 
Page 15 GAO-04-873 ADL in Military Education

material that is not routinely updated; and students are assigned to cohort 

teams that allow online interactive opportunities and group discussions. 

In ACSC’s application, almost all nonresident materials are provided to 

students on CD-ROMs as well as on the Internet to allow as much flexibility 

as possible to complete nonresident courses.

Generally, PME school officials stated that ADL conversions were made 

because of nonresident PME’s need (1) to respond to multiple learning 

styles in a diverse officer corps, (2) to increase the efficiency of 

educational delivery, and (3) to improve the quality of the educational 

offering. Additionally, NWC saw ADL as a means of potentially affecting 

retention and increasing enrollment. USAWC saw ADL as a means of 

responding to student demands for a more efficient and relevant 

educational experience. ACSC saw ADL as an improved means for 

delivering course material to increasing numbers of deployed officers.

Nonmilitary ADL 

Conversion Decisions Were 

Similar

In nonmilitary applications of ADL, we observed processes and criteria for 

conversion decisions that were similar to DOD’s. Course conversions 

followed analogous processes and criteria decisions, driven by factors such 

as interest and student enrollment that were not systematic. University of 

Maryland University College officials stated that their conversion process 

includes a dedicated staff of instructional designers and subject matter 

experts (usually faculty) who produce course conversion to distance 

learning content within an established framework to ensure maintenance 

of standards. Their criteria for conversion focus on high-student demand 

and high levels of interest and on course work that required less “hands-on” 

training, such as business and information technology courses. Further 

research of private-sector practices supports the observation that the lack 

of systematic criteria is consistent with mainstream practice in ADL 

adoption for nonresident education.

DOD’s approaches for course conversion are thus consistent with 

mainstream practice, and in some cases, compare favorably with the best 

practices in nonmilitary education. For example, NWC’s ADL application 

received the Crystal Award in 2002 from the Association for Educational 

Communications and Technology based on “innovative and creative use of 

the medium, instructional value and relevance, soundness of instructional 

strategy, quality of production, and evidence of successful outcomes.” As of 

June 2004, USAWC’s nonresident education program is fully accredited by 

the Middle States Commission of Higher Education for the awarding of 

Master’s of Strategic Studies Degrees. USAWC is the first military 
Page 16 GAO-04-873 ADL in Military Education

institution to achieve degree-granting authority for its nonresident 

ADL-based program.

Cultural, 

Technological, and 

Resource Barriers and 

Challenges Affect ADL 

Implementation in 

PME Programs

PME schools identified a number of cultural, technological, and resource 

challenges that affect ADL implementation and may affect future 

maintenance or expansion of ADL efforts. Cultural issues such as 

appropriate extent of ADL incorporation, general perceptions about 

nonresident education, and limited ADL research in military education 

affect the degree of ADL implementation. Technological trade-offs and 

nonresident program resourcing also affect continued ADL efforts.

Cultural PME officials question the appropriate extent to which ADL should be used 

in nonresident education and how closely it can, or should, enable 

nonresident education to approximate resident education. It is generally 

recognized that resident programs are better in providing acculturation,13

interactive skills, and simulations that are critical for professional 

development of officers, and that there are challenges in providing such 

aspects in nonresident education. But some observers believe that 

nonresident education should not be compared to resident education and 

that nonresident education represents a vital broadening experience in its 

own right. In addition, there are indications that an ADL approach could 

significantly enrich nonresident content by excluding teaching methods 

that do not work in residence, allowing students the flexibility to focus on 

material that requires further study without class disruption, and serving as 

the basis for applications that can be used to upgrade teaching methods in 

resident programs.

13 Acculturation is defined as a developmental activity that involves the adoption of 

customs, protocols, and doctrine. The acculturation process is designed to prepare officers 

for shared leadership positions while reinforcing total force partnerships.
Page 17 GAO-04-873 ADL in Military Education

ADL implementation could also be affected by certain negative perceptions 

concerning nonresident education that are held by students, and in some 

cases, reflected in policy. Our survey of nonresident PME current students 

and recent graduates indicated that about 49 percent of current students 

and 48 percent of graduates believe they are not as well prepared as are 

their resident student counterparts, regardless of nonresident delivery 

method.14 While not universal between the services, we observed instances 

of military education policies that reinforce the perception that 

nonresident education is not as desirable as resident education. The Air 

Force’s PME Instruction AFI36-2301 states that “ideally, all officers will 

attend PME in residence,” and that limited resources restrict resident 

attendance to the “best qualified.” Furthermore, “completing nonresident 

PME programs will not affect eligibility for resident PME programs.” 

Indeed, we were told of instances where officers who after completing the 

nonresident PME program subsequently enrolled in the corresponding 

senior- or intermediate-level course in residence.

The extent of ADL implementation in nonresident education is affected by 

the role that PME completion plays in promotional consideration. 

Programs maintained to foster promotional consideration—that is, 

“personnel-oriented”—might not be compatible with programs 

emphasizing the learning outcomes brought about by ADL—that is, 

“education-oriented.” Our survey shows that promotional considerations, 

rather than learning outcomes, are the focus for students in nonresident 

education. An estimated 73 percent of current students and 84 percent of 

recent graduates listed as their predominant reason for participating in 

nonresident PME a desire to improve their promotional chances or to meet 

promotional requirements. In addition, of an estimated 22 percent of PME 

graduates who were promoted to a higher rank after PME completion, an 

estimated 88 percent stated that PME contributed to a “great” or “very 

great extent” in their promotions.15 But ADL adoption goals should focus 

more on learning outcomes than on simply “checking the boxes” for 

promotional opportunity enhancement.

PME officials state that there are concerns that ADL advantages could be 

oversold to the point that ADL may be used to supersede resident programs 

14 The percentages reported here are based on a sample of current students and graduates 

and are estimates. All percentage estimates from the survey reported have margins of error 

of plus or minus 10 percentage points or less, unless otherwise noted.

15 The confidence interval for this figure was +9 percent and –17 percent.
Page 18 GAO-04-873 ADL in Military Education

and shift the burden of PME administration. DOD officials, already viewing 

ADL delivery methods as attractive from a cost savings perspective, are 

desirous to expand such programs, even at the expense of resident 

programs. However, PME officials believe that ADL expansion should be 

considered only after completely understanding its impact on military 

operations and recognizing the resident program’s role as the basis for a 

nonresident program. In addition, PME officials noted that ADL could be 

used as a means of shifting the burden from the school to the student and 

the student’s commands without providing appropriate command support, 

compensation, or resources.

While ADL research exists for military training courses, there is only 

limited research on ADL’s impact on military education, especially in terms 

of its impact on learning that requires interactive elements. A DOD official 

stated that it is well-known in studies that distance learning (usually 

teleconferencing) instruction offers no significant differences as compared 

with classroom instruction. However, officials believe that more work 

should be done to collect information that ADL improves learning in 

military education and that these studies should focus on collaborative 

learning environments and the extent of their translation. In addition, 

further efforts should look at commercial education studies 

(undergraduate and graduate education) and its transfer to military 

education.

Technological PME school officials have stated that decisions are needed on trade-offs 

between increased demands for student access (more servers, more 

bandwidth, or reduced firewalls) and the maintenance for network 

security. Such decisions are complicated by what is viewed as a lack of 

standardization in DOD and within individual services on security 

requirements.

In our nonresident survey, approximately 19 percent of current students 

and 16 percent of recent graduates experienced computer/Internet related 

problems affecting their PME experience. Some identified problems 

included servers, bandwidth issues, and security firewalls. PME schools 

considered using “.edu” domains to make courseware more available to 

students because of the difficulty of interacting “.mil” domains with 

systems outside of military organizations. However, such moves would be 

expensive and would conflict with increasing requirements to reduce the 

number of servers and personnel needed to operate these systems. Some 

reported problems involved limited bandwidth issues. While such problems 
Page 19 GAO-04-873 ADL in Military Education

can be overcome, they require time and money to resolve. Firewalls 

maintained for security purposes have caused schools to limit library 

access to nonresident students due to perceived security threats.

Resources and Funding Most ADL efforts at PME schools were fielded independently with limited 

budgets and staffing. USAWC’s and ACSC’s ADL applications were 

developed and supported with in-house staff responsible for managing 

resident and nonresident programs. These applications were fielded 

independently within the services. PME officials stated that PME schools’ 

ability to fund ADL applications is limited due to DOD’s priority to focus 

more on its training and operational activities. An emerging funding issue 

involves the use of copyrighted material in ADL applications. Increasing 

costs in using copyrighted material for course work could result in limiting 

course flexibility and methodologies.

New technologies, such as ADL, create new requirements for faculty 

personnel with higher technical expertise and more equipment and 

structure than traditional programs. Faculty members must be skilled to 

perform in online and classroom settings. PME schools are beginning to 

observe they must offer faculty opportunities to teach courses in multiple 

medias or risk losing qualified faculty to competitors.

Conclusions Although PME schools receive oversight from a number of organizations, 

we observed that neither the schools nor the oversight agencies had 

focused on (1) establishing specific performance effectiveness goals for 

ADL implementation or (2) measuring learning outcomes as a means of 

evaluating program effectiveness. The Joint Staff’s accreditation reports on 

nonresident education do not detail performance goals for any particular 

delivery method. The military services, which have primary responsibility 

for PME oversight, view the accreditation process provided by the Joint 

Staff as the primary means of ensuring the effectiveness of nonresident 

education. DOD is not alone in this problem—the lack of metrics for 

performance effectiveness and learning outcomes is pervasive throughout 

all educational institutions. Our prior efforts indicate that most public and 

private institutions lack a framework with which to assess implementation 

of training and development efforts. However, agencies need clear 

information on how training and development efforts affect organizational 

performance, and decision makers will likely want to compare the 

performance of these programs with that of other programs. Without clear 
Page 20 GAO-04-873 ADL in Military Education

goals and an effective process for evaluating the results of ADL application, 

DOD cannot ensure that ADL is achieving appropriate return on 

investment, student retention, student access, and other goals in 

comparison with prior efforts. Furthermore, programs lacking outcome 

metrics will be unable to demonstrate how they contribute to results. 

Moreover, by not capturing and using student data that are uniquely 

available through ADL techniques, DOD is missing the opportunity to 

develop the basis for effectiveness metrics and knowledge about 

learning outcomes.

Recommendations for 

Executive Action

We recommend that the Secretary of Defense direct the Under Secretary of 

Defense for Personnel and Readiness, in concert with the Joint Staff, 

service headquarters, and the PME schools, to take the following two 

actions:

• promote the development of specific performance effectiveness goals 

for ADL and

• promote the use of ADL technologies to capture data to provide 

knowledge about learning outcomes.

Agency Comments and 

Our Evaluation

DOD partially concurred with our first recommendation and fully 

concurred with the second. DOD supports the use of specific effectiveness 

goals for PME, but believes such goals are not appropriate for any specific 

delivery method. While we acknowledge DOD’s concerns with focusing on 

a specific delivery method, we believe that ADL is unlike other means of 

delivery because of its potential to modernize the educational experience 

and because its use is rapidly expanding in other areas of PME. We believe 

it would be worthwhile for DOD to know specifically how well ADL 

performs, especially in comparison with other delivery methods, in order to 

better understand its appropriate use for PME. DOD concurred with our 

second recommendation and stated that current accreditation practices are 

already promoting the data collection capabilities of ADL technologies for 

assessing multiple delivery methods. DOD’s comments are included in this 

report as appendix V. DOD also provided technical changes, which we 

incorporated as appropriate.
Page 21 GAO-04-873 ADL in Military Education

We are sending copies of this report to congressional members as 

appropriate. We will also send copies to the Secretary of Defense; the 

Secretaries of the Air Force, Army, and Navy; the Commandant of the 

Marine Corps; and the Chairman of the Joint Chiefs of Staff. We will make 

copies available to others on request. In addition, this report will be 

available at no charge on the GAO Web site at http:/www.gao.gov.

If you or your staff have any questions, please call me on (757) 552-8100 or 

Clifton Spruill, Assistant Director, on (202) 512-4531. Major contributors to 

this report were Arnett Sanders, Maewanda Michael-Jackson, Jean Orland, 

David Dornisch, Terry Richardson, and Cheryl Weissman.

Sincerely yours,

Neal P. Curtin

Director, Defense Capabilities and Management
Page 22 GAO-04-873 ADL in Military Education

Appendix I

Scope and Methodology ApApenednxidexiIs

We reviewed Department of Defense’s (DOD) implementation of advanced 

distributed learning (ADL) in senior- and intermediate-level professional 

military education (PME) to determine processes and criteria used for 

converting courses, metrics to assess ADL effectiveness and its fulfillment 

of learning objectives, and barriers and challenges in ADL implementation. 

We collected, reviewed, and analyzed relevant program information and 

conducted interviews with DOD officials responsible for ADL 

implementation in PME programs and oversight responsibilities. We 

initially obtained data from the Principal Deputy Under Secretary of 

Defense for Personnel and Readiness to identify PME programs with ADL 

applications. A review of the data indicated that there were three existing 

programs. We identified, interviewed, and obtained data from officials from 

PME schools with ADL applications. Those schools were

• U.S. Army War College, Carlisle, Pennsylvania;

• Naval War College, Newport, Rhode Island; and

• Air Command and Staff College, Montgomery, Alabama.

We also interviewed and obtained data from officials at the Joint Forces 

Staff College in Norfolk, Virginia, on their pending ADL application.

We also interviewed and obtained data from agencies within DOD 

responsible for oversight of PME activities. Those agencies included

• The Joint Chiefs of Staff’s Operational Plans and Joint Force 

Development Directorate (J-7), Joint Doctrine, Education, and Training 

Division, Joint Education Branch, Arlington, Virginia;

• The Office of the Under Secretary of Defense for Personnel and 

Readiness, Deputy to the Under Secretary for Readiness, Office of 

Readiness and Training Policy and Programs, Arlington, Virginia;

• The Office of the Under Secretary of Defense for Personnel and 

Readiness, Deputy to the Under Secretary for Military Personnel Policy, 

Arlington, Virginia;

• Department of the Army, Office of the Deputy Chief of Staff, Operations 

and Plans, Arlington, Virginia;
Appendix I

Scope and Methodology

Page 23 GAO-04-873 ADL in Military Education

• Department of the Navy, Office of the Chief of Naval Operations, 

Personal Development and Accessions Division, Washington, D.C.;

• Department of the Air Force, Office of the Deputy Chief of Staff for 

Personnel, Learning, and Force Development, Arlington, Virginia;

• U.S. Marine Corps Combat Development Center, Training and Education 

Command, Quantico, Virginia;

• U.S. Army War College Board of Visitors, Carlisle, Pennsylvania;

• Naval War College Board of Advisors, Newport, Rhode Island; and

• Air University Board of Visitors, Montgomery, Alabama.

To determine sufficient metrics to assess ADL effectiveness, we provided 

PME program officials with a detailed list of questions that included those 

relating to effectiveness and learning objectives. We reviewed written 

responses, if provided, and followed up with site visits and correspondence 

with oversight agencies to clarify or obtain additional information if 

necessary. We also obtained and analyzed data from a survey of 

nonresident PME current students and graduates, which included 

questions designed to obtain perceptions on program effectiveness. Details 

of the survey methodology are presented in appendix II.

To determine processes and criteria DOD used for ADL conversion, we 

provided PME program officials with a detailed list of questions that 

included those relating to process and criteria decisions. We reviewed 

written responses, if provided, and followed up with site visits to clarify or 

obtain additional information if necessary. To determine whether criteria 

were consistent with those of other institutions performing distance 

education, we researched prior literature on this topic and conducted a site 

visit to the University of Maryland University College in Adelphi, Maryland. 

The school was identified in our prior reports on distance education as 

having a program with a large distance education population, as well as 

educating a significant number of military officers. We also contacted and 

received data from the Sloan Consortium, an organization designed to 

encourage collaborative sharing of knowledge and effective practices to 

improve online education.

To determine barriers and challenges to ADL implementation, we provided 

PME program officials with a detailed list of questions that included those 
Appendix I

Scope and Methodology

Page 24 GAO-04-873 ADL in Military Education

relating to barriers and challenges. We reviewed written responses, if 

provided, and followed up with site visits and correspondence with DOD 

oversight agencies to clarify or obtain additional information if necessary. 

We also obtained and analyzed data from a survey of nonresident PME 

current students and graduates, which included questions designed to 

obtain perceptions on barriers and challenges in completing PME courses. 

Details of the survey methodology are presented in appendix II.
Page 25 GAO-04-873 ADL in Military Education

Appendix II

Methodology for Our Survey of Nonresident 

PME Students and Graduates ApendxiI

To obtain military officers’ perspectives on nonresident PME in terms of 

impact on careers, achievement of learning objectives, and obstacles and 

challenges, we conducted a statistically representative survey of current 

nonresident senior- and intermediate-level PME students and graduates of 

these schools from April 1999 to March 2003, roughly the period coinciding 

with initial ADL implementation at several senior- and intermediate-level 

schools. We present the survey questions and response results in 

appendix III.

The Study Population The population for the nonresident PME survey consisted of current 

students and graduates who fulfilled the following criteria:

1. Respondents were identified as enrolled in a senior- and 

intermediate-level nonresident PME program of study from April 1999 

to March 2003. We decided on this time period to ensure that our 

respondents would have begun their programs after Web-based PME 

had been clearly established as a mode of instruction or have been in 

PME long enough to have meaningful responses to our questions.

2. Respondents participated in a senior- and intermediate-level 

nonresident PME program of study, as opposed to individual PME 

courses undertaken via continuing education programs.

3. Respondents are currently active in the U.S. military services or 

reserves, excluding U.S. civilians; U.S. Coast Guard members; and 

international members, either military or civilian.

4. Respondents participated (i.e., currently enrolled or graduated) 

in a nonresident PME program in one of the six senior- and 

intermediate-level PME schools: U.S. Army War College, Army 

Command and General Staff College, Air War College, Air Command 

and Staff College, Naval War College, and Marine Corps College of 

Continuing Education.

The survey asked respondents about PME’s impact on furthering career 

objectives, their achievement of learning objectives, and obstacles and 

challenges of the programs. Specific questions concerned students’ 

satisfaction with their overall program, various modes of program delivery, 
Appendix II

Methodology for Our Survey of Nonresident 

PME Students and Graduates

Page 26 GAO-04-873 ADL in Military Education

and with technologies used; students’ time and duty management concerns; 

and reasons for participation in nonresident PME.

Developing the Survey To develop areas of inquiry for the survey, we reviewed our previous work 

related to distance education and PME. We reviewed a series of survey 

questionnaires developed by us and by DOD. We used these sources and 

our own analysis to develop an initial set of questions. We further 

developed and refined the questionnaire by obtaining and incorporating 

written comments regarding the initial questions from administrators and 

other representatives of the senior- and intermediate-level PME schools.

In addition to an internal expert technical review by our Survey 

Coordination Group, we pretested the survey with five individuals whose 

personal characteristics corresponded to our eligibility criteria. We 

identified pretest subjects through our contacts who were current military 

personnel or who knew military personnel and our PME points of contact.

The Sample Design and 

Administration

We conducted the survey between January and April of 2004 on a random 

sample of 437 current students and graduates of nonresident PME 

programs using a self-administered Web-based questionnaire. We drew the 

names of our respondents from an overall population data set we 

constructed that combined separate data sets received from each of the 

senior- and intermediate-level PME schools. For each data set, we 

requested the officer’s name, school attended, month and year of initial 

enrollment, component (defined as either active duty or reservist), and 

mode of PME delivery. We requested e-mail addresses and, if needed, 

phone numbers from potential respondents after they were drawn from the 

population data sets. We stratified our sample by component in order to 

better understand any differences between these components.

We activated the survey Web site and informed our sample respondents 

of the Web site, their logon name, and passwords by e-mail on January 30, 

2004. To maximize the response rate, we sent five subsequent follow-up 

e-mail reminders to nonrespondents in February and March 2004. At the 

time of the third mailing, we also telephoned many of the nonrespondents 

to encourage them to complete the survey. We ended data collection 

activities on April 30, 2004.
Appendix II

Methodology for Our Survey of Nonresident 

PME Students and Graduates

Page 27 GAO-04-873 ADL in Military Education

Of the 437 selectees included in our sample, we received 273 useable 

questionnaires. We defined useable as respondents who completed the 

survey and were not identified as out-of-scope. During the survey, we 

deemed 67 of the 437 to be outside the scope of our survey after 

determining that they did not meet at least one of our eligibility criteria. 

Disregarding these 67 responses, our overall response rate was 

73.8 percent (273/370). Table 2 shows the final disposition of the sample 

(the 437 respondent accounts activated) by strata.

Table 2: Disposition of Sample

Source: GAO.

The estimates we make in this report are the result of weighting the survey 

responses to account for effective sampling rates in each stratum. These 

weights reflect both the initial sampling rate and the response rate for each 

stratum. As with many surveys, our estimation method assumes that 

nonrespondents would have answered like the respondents.

Sampling Error For the estimates we present in this report, we are 95 percent confident 

that the results we would have obtained had we studied the entire 

population are within +/-10 or fewer percentage points of our estimates 

(unless otherwise noted). Because we surveyed a sample of recent 

nonresident PME students, our results are estimates of student and 

graduate characteristics and thus are subject to sampling errors. Our 

confidence in the precision of the results from this sample is expressed in 

95 percent confidence intervals, which are expected to include the actual 

results for 95 percent of the samples of this type. We calculated confidence 

intervals for our results using methods appropriate for a stratified 

probability sample.

Stratum Sample Useable Out of scope

Number of 

nonrespondents

Active Duty 219 136 31 52

Reserve Duty 218 137 36 45
Appendix II

Methodology for Our Survey of Nonresident 

PME Students and Graduates

Page 28 GAO-04-873 ADL in Military Education

Nonsampling Error and 

Data Quality

We conducted in-depth pretesting of the questionnaire to minimize 

measurement error. However, the practical difficulties in conducting 

surveys of this type may introduce other types of errors, commonly known 

as nonsampling errors. For example, measurement errors can be 

introduced if (1) respondents have difficulty interpreting a particular 

question, (2) respondents have access to different amounts of information 

in answering a question, or (3) errors in data processing occur. We took 

extensive steps to minimize such errors in developing the questionnaire, 

collecting the data, and editing and analyzing the information. The 

Web-based data management system we used provides a systematized 

process for processing, transferring, and analyzing data that also protects 

against nonsampling errors. In addition, we performed tests to ensure the 

reliability and usefulness of the data provided by the PME schools. These 

included computer analyses to identify inconsistencies both within and 

across the data sets and other errors in the data sets from which we 

developed our overall sampling frame. We also interviewed agency officials 

knowledgeable about the data. We determined that the data were 

sufficiently reliable for the purposes of this report.
Page 29 GAO-04-873 ADL in Military Education

Appendix III

Survey Responses ApendxiI

Introduction Welcome DoD Distance Learning Student or Graduate.

The U.S. General Accounting Office (GAO) - an agency of the U.S. Congress 

has been requested to review various aspects of nonresident professional 

military education (PME). Part of that effort is to evaluate whether the 

appropriate decisions are being made in regards to nonresident education 

for intermediate and senior-level military officers. As part of this effort, we 

are assessing opinions of nonresident PME graduates and current students 

on (1) the achievement of PME learning objectives, (2) obstacles and 

challenges in completing PME, and (3) PME’s impact on furthering career 

objectives. 

The survey should take approximately 10 minutes to complete. 

Participation in this survey is voluntary but encouraged. Your responses 

will be confidential and the results of the survey will be reported in 

aggregate form only. 

Before choosing an answer, please read the full question and all response 

choices carefully. 

Thank you in advance for your participation.

Frame of Reference Please consider only the following when answering the questions on this 

survey: 

1. Nonresident Professional Military Education programs of study that 

you have participated in, as opposed to individual PME courses 

undertaken via continuing education programs. 

AND

2. Nonresident Professional Military Education programs of study that 

you began in April 1999 or after.

Our survey of nonresident Professional Military Education students was 

divided into two main parts, one with questions appropriate for current 

students and one with similar questions worded slightly differently for 

graduates. There are also questions on demographics to which both current 

students and graduates responded. Survey questions and responses for 
Appendix III

Survey Responses

Page 30 GAO-04-873 ADL in Military Education

graduates are indicated in italics and those for current students are in plain 

text.

The information provided here represents weighted data. For information 

on weighting, see appendix II.

Except where noted by the following, all percentage estimates have 95% 

confidence intervals within +/- 10 percentage points:

a

Confidence interval exceeds +10 percentage points

b

Confidence interval exceeds +25 percentage points and estimate is unreliable

Questions 1 and 31 are intentionally omitted because they contained 

instructions telling respondents which questions to answer.

Survey of Nonresident Professional Military Education Graduates and 

Current Students

Q2. The name of program in which currently enrolled:

Q32. From which school did you graduate?

Q3. In what month and year did you begin your PME program? 

Q33. In what month and year did you begin your PME program (if you graduated from more than one program, answer for the most recent 

one)?

Air Command 

and Staff 

College 

Nonresident 

Program 

(percent)

Air War 

College 

Nonresident 

Program 

(percent)

Army 

Command and 

General Staff 

College 

Nonresident 

Program 

(percent)

Army War 

College 

Distance 

Education 

Program 

(percent)

Marine Corps 

College of 

Continuing 

Education 

(percent)

Naval College 

of Distance 

Education 

(percent)

Current Students 30 25 31 1 10 2

Graduates 40 27 27 2 1 3

1999 

(percent)

2000 

(percent)

2001 

(percent)

2002 

(percent)

2003 

(percent)

Other 

(percent)

Current Students 1 3 14 30 47 5

Graduates 15 26 34 21 5 0
Appendix III

Survey Responses

Page 31 GAO-04-873 ADL in Military Education

Q4. What mode of instruction have you used most often in your nonresident Professional Military program?

Q34. What mode of instruction did you use most often in your nonresident Professional Military program?

Q5. In what month and year do you expect to complete your PME studies program?

Q35. In what month and year did you complete your Professional Military Education program?

Q6. In a typical week, approximately how many hours did you spend in Professional Military Education-related activities, including 

preparation, study, working on-line, and time in class?

Q36. In a typical week, approximately how many hours did you spend in Professional Military Education-related activities, including 

preparation, study, working on-line, and time in class?

Q7. Does the military or your employer afford you time during your work-week for Professional Military Education?

Q37. Did the military or your employer afford you time during your work-week for Professional Military Education?

Q8. How many hours do you work in paid employment in a typical work-week (outside of Professional Military Education-related activities)?

Q38. During the period of time that you were completing your Professional Military Education program, how many hours did you work in 

paid employment in a typical work-week (outside of Professional Military Education-related activities)?

Seminar or 

Classroom 

Instruction 

(percent)

Web-Based 

Correspondence 

(percent)

Paper-Based 

Correspondence 

(percent)

Current Students 24 25 51

Graduates 42 18 40

2003 

(percent)

2004 

(percent)

2005 

(percent)

2006 

(percent)

2007 

(percent)

Other 

(percent)

Current Students 1 68 21 4 0 6

1999 

(percent)

2000 

(percent)

2001 

(percent)

2002 

(percent)

2003 

(percent)

2004 

(percent)

Graduates 1 10 18 31 39 1

Mean

Current Students 5.8

Graduates 8.4

Yes 

(percent)

No 

(percent)

Current Students 23 77

Graduates 42 58

Mean

Current Students 52.0

Graduates 47.5
Appendix III

Survey Responses

Page 32 GAO-04-873 ADL in Military Education

Q9. Listed below are various reasons why someone would participate in a nonresident PME program. What is your greatest reason for 

participating in a nonresident Professional Military Education program? 

Q39. Listed below are various reasons why someone would participate in a nonresident PME program. What was your greatest reason for 

participating in a nonresident Professional Military Education program

Q10. To this point in time, how satisfied or dissatisfied are you with your Professional Military Education program?

Q40. Overall, how satisfied or dissatisfied are you with the Professional Military Education program in which you graduated?

Q11. To what extent, if any, has your Professional Military Education program benefited your military career to this point in time?

Q41. To what extent, if any, has graduation from your Professional Military Education program benefited your military career to this point 

in time?

Current 

Students 

(percent)

Graduates 

(percent)

To develop professionally 20 13

To gain access to better assignments 1 0

To gain knowledge in my field or in fields of interest to me 2 0

To improve my chances of, or meet the requirements for, 

promotion 73 84

To network with other officers 1 0

To obtain college credit 0 0

Other 4 3

Very satisfied 

(percent)

Somewhat 

satisfied 

(percent)

Neither 

satisfied nor 

dissatisfied 

(percent)

Somewhat 

dissatisfied 

(percent)

Very 

dissatisfied 

(percent)

Current Students 16 35 21 21 8

Graduates 20 45 15 15 5

Very great 

extent 

(percent)

Great extent 

(percent)

Moderate 

extent 

(percent)

Little extent 

(percent)

No extent 

(percent)

Current Students 1 21 42 20 16

Graduates 15 21 30 22 12
Appendix III

Survey Responses

Page 33 GAO-04-873 ADL in Military Education

Q12. To what extent, if any, do you believe the knowledge you are acquiring in your Professional Military Education program will improve 

your effectiveness in future assignments?

Q42. To what extent, if any, do you believe the knowledge you acquired in your Professional Military Education program has improved your 

effectiveness in job assignments?

No Parallel Question for Current Students

Q43. Have you been promoted to a higher rank since you completed your Professional Military Education program?

No Parallel Question for Current Students

Q44. To what extent, if any, do you believe that completion of your Professional Military Education program contributed to your promotion?

Q13. To what extent, if any, does your Professional Military Education program enable you to acquire the knowledge you are expected to 

obtain?

Q45. To what extent, if any, did your Professional Military Education program enable you to acquire the knowledge you were expected to 

obtain?

Q14. Was/is any part of your Professional Military Education program taken through seminar/classroom-based instruction?

Q46. Was any part of your Professional Military Education program taken through seminar/classroom-based instruction?

Very great 

extent 

(percent)

Great extent 

(percent)

Moderate 

extent 

(percent)

Little extent 

(percent)

No extent 

(percent)

Current Students 5 22 42 25 6

Graduates 9 16 39 24 12

Yes 

(percent)

No 

(percent)

Graduates 78 22

Very great 

extent 

(percent)

Great extent 

(percent)

Moderate 

extent 

(percent)

Little extent 

(percent)

No extent 

(percent)

Graduates 53a 34a 3a 9a 0

Very great 

extent 

(percent)

Great extent 

(percent)

Moderate 

extent 

(percent)

Little extent 

(percent)

No extent 

(percent)

Current Students 5 25 43 21 6

Graduates 9 28 40 19 4

No 

(percent)

Yes 

(percent)

Current Students 67 33

Graduates 50 50
Appendix III

Survey Responses

Page 34 GAO-04-873 ADL in Military Education

Q15. Overall, how would you rate the quality of this seminar/classroom-based learning?

Q47. Overall, how would you rate the quality of this seminar/classroom-based learning?

Q16. Was/is any part of your Professional Military Education program taken through paper-based correspondence?

Q48. Was any part of your Professional Military Education program taken through paper-based correspondence?

Q17. Overall, how would you rate the quality of this paper-based correspondence learning?

Q49. Overall, how would you rate the quality of this paper-based correspondence learning?

Q18. Was/is any part of your Professional Military Education program taken through the World-Wide Web or Internet?)

Q50. Was any part of your Professional Military Education program taken through the World-Wide Web or Internet?

Q19. Overall, how would you rate the quality of this web/Internet-based learning?

Q51. Overall, how would you rate the quality of this web/Internet-based learning?

Excellent 

(percent)

Very good 

(percent)

Good 

(percent)

Fair 

(percent)

Poor 

(percent)

Current Students 8a 66a 11a 13a 3a

Graduates 18a 35a 31a 14 3

No 

(percent)

Yes 

(percent)

Current Students 36 64

Graduates 40 60

Excellent 

(percent)

Very good 

(percent)

Good 

(percent)

Fair 

(percent)

Poor 

(percent)

Current Students 3 27a 40a 26a 4

Graduates 6 31 33 20 10

No 

(percent)

Yes 

(percent)

Current Students 64 36

Graduates 60 40

Excellent 

(percent)

Very good 

(percent)

Good 

(percent)

Fair 

(percent)

Poor 

(percent)

Current Students 17a 22a 40a 15a 5a

Graduates 8a 39a 41a 10a 2
Appendix III

Survey Responses

Page 35 GAO-04-873 ADL in Military Education

Q20. How easy or difficult has it been for you to use web/Internet-based learning?

Q52. How easy or difficult was it for you to use web/Internet-based learning?

Q21. How easy or difficult have you found interaction with faculty during your web/Internet-based learning?

Q53. How easy or difficult did you find interaction with faculty during your web/Internet -based learning?

Q22. How easy or difficult have you found interaction with other students during your web/Internet-based learning?

Q54. How easy or difficult did you find interaction with other students during your web/Internet-based learning?

Q23. How well does the courseware/course software work on the computer equipment to which you have access for taking web/Internetbased learning?

Q55. How well did the courseware/course software work on the computer equipment to which you had access for taking web/Internetbased learning?

Very easy 

(percent)

Somewhat 

easy 

(percent)

Neither easy 

nor difficult 

(percent)

Somewhat 

difficult 

(percent)

Very difficult 

(percent)

Current Students 34a 21a 32a 8a 5a

Graduates 40a 37a 10a 12a 0

Very easy 

(percent)

Somewhat 

easy 

(percent)

Neither easy 

nor difficult 

(percent)

Somewhat 

difficult 

(percent)

Very difficult 

(percent)

Current Students 26a 11a 35a 11a 16a

Graduates 18a 48a 29a 5a 0

Very easy 

(percent)

Somewhat 

easy 

(percent)

Neither easy 

nor difficult 

(percent)

Somewhat 

difficult 

(percent)

Very difficult 

(percent)

Current Students 16a 31a b 16a 11a

Graduates 15a 41a 26a 15a 4a

Excellent 

(percent)

Very good 

(percent)

Good 

(percent)

Fair 

(percent)

Poor 

(percent)

Current Students 8a 41a 32a 19a 0

Graduates 21a 43a 28a 5 2
Appendix III

Survey Responses

Page 36 GAO-04-873 ADL in Military Education

Q24. How reliable is your network access for taking web/Internet-based learning (e.g. ability to connect; to upload and download 

assignments, etc).

Q56. How reliable was your network access for taking web/Internet-based learning (e.g. ability to connect; to upload and download 

assignments, etc)?

Q25. Compared to resident Professional Military Education students in the school in which you are enrolled, of the following options, do 

you believe you are prepared:

Q57. Compared to resident Professional Military Education program graduates of your school, of the following options, do you believe you 

are prepared:

Q26. Overall, what has been the primary challenge, if any, affecting your Professional Military Education program?

Q58. Overall, what was the primary challenge, if any, affecting your Professional Military Education program?

Q27. Have you experienced any computer/Internet-related problems affecting your Professional Military Education program?

Q59. Did you experience any computer/Internet-related problems affecting your Professional Military Education program?

Very reliable 

(percent)

Somewhat 

reliable 

(percent)

As reliable as 

unreliable 

(percent)

Somewhat 

unreliable 

(percent)

Very 

unreliable 

(percent)

Current Students 41a 40a 14a 3a 3a

Graduates 52a 45a 220

better than 

resident 

students. 

(percent)

as well as 

resident 

students. 

(percent)

worse than 

resident 

students. 

(percent)

Don’t know 

(percent)

Current Students 2 27 49 23

Graduates 3 30 48 19

Current 

Students 

(percent)

Graduates 

(percent)

Computer/Internet-related problems 3 0

Deployment cycle 12 5

Domestic circumstances 12 10

Maintaining focus 16 12

Present job duties 45 53

Not applicable, I am not experiencing & have not experienced any challenges 

to this point in time. 6 17

Other 8 3

No 

(percent)

Yes 

(percent)

Current Students 81 19

Graduates 84 16
Appendix III

Survey Responses

Page 37 GAO-04-873 ADL in Military Education

Q28. What specific computer/Internet-related problems have you incurred?

Q60. What specific computer/Internet-related problems have you incurred?

Q29. At any point during your Professional Military Education program, have you had to defer/disenroll from your studies?

Q61. At any point during your Professional Military Education program, did you have to defer/disenroll from your studies?

Q30. What was the primary reason you had to defer/disenroll from your studies?

Q62. What was the primary reason you had to defer/disenroll from your studies?

Open-ended comments not shown here.

Q63. What is the highest degree or level of school that you have completed? 

Current 

Students Graduates

Yes 

(percent)

Yes 

(percent)

a. Bandwidth/ network speed 63a b

b. Inadequate uploading or downloading 

ability/lack of high-speed internet 

equipment 57a b

c. Inadequate technical support b b

d. Defective/ incompatible equipment 25a b

e. Lack of computer skills 5a b

f. Lack of network availability/access to 

internet 52a b

g. Security/Firewall issues 26a b

h. Other b b

No 

(percent)

Yes 

(percent)

Current Students 66 34

Graduates 86 14

Current 

Students 

(percent)

Graduates 

(percent)

High school or equivalent 0 0

1 or more years of college, no degree 0 0

Associate’s degree 1 0

Bachelor’s degree 42 25

Master’s degree 44 61

Doctoral or professional school degree 10 12

Other 3 2
Appendix III

Survey Responses

Page 38 GAO-04-873 ADL in Military Education

Q64. In what branch of the military do you serve?

Q65. What duty capacity best describes you during the majority of your Professional Military Education program?

Q66. What component best describes you during the majority of your Professional Military Education program?

Q67. Are you a member of the National Guard?

Current 

Students 

(percent)

Graduates 

(percent)

Air Force 54 66

Army 33 30

Marines 11 1

Navy 2 3

Current 

Students 

(percent)

Graduates 

(percent)

Non-Active Duty 43 26

Active Duty 57 74

Current 

Students 

(percent)

Graduates 

(percent)

Active Component 46 67

Reserve Component 54 33

Current 

Students 

(percent)

Graduates 

(percent)

Yes 35a 45a

No 65a 55a
Appendix III

Survey Responses

Page 39 GAO-04-873 ADL in Military Education

Q68. What military occupational category best describes you during the majority of your Professional Military Education program? 

Q69. What was your rank when you began your Professional Military Education program?

Q70. What is your current rank?

Current 

Students 

(percent)

Graduates 

(percent)

Administrative 12 9

Engineering & Maintenance Officers 10 13

General Officers & Executives 4 7

Health Care Officers 18 13

Intelligence Officers 6 1

Scientists & Professionals 9 10

Supply & Procurement & Allied 

Officers 7 8

Tactical Operation Officers 28 31

Non-Occupational 0 1

Not Listed Above/Other 6 7

Current 

Students 

(percent)

Graduates 

(percent)

O-2 1 1

O-3 18 18

O-4 65 59

O-5 13 21

O-6 2 1

Other 2 0

Current 

Students 

(percent)

Graduates 

(percent)

O-2 0 0

O-3 4 3

O-4 70 59

O-5 24 27

O-6 3 7

O-7 0 1

Other 0 3
Appendix III

Survey Responses

Page 40 GAO-04-873 ADL in Military Education

Q71. If you have any other comments related to your PME education, training, assignments, distance learning, or any other matters related 

to this questionnaire, please note them here.

Open-ended comments not shown here.
Page 41 GAO-04-873 ADL in Military Education

Appendix IV

ADL Applications and Additional Features 

of Nonresident Programs ApendxiIV

We observed three current ADL applications at PME senior- and 

intermediate-level schools. These schools have geared their ADL efforts to 

their nonresident programs. The programs vary from service to service in 

terms of enrollment, structure, duration, and credits received for 

graduation. In addition, we observed additional features of nonresident 

programs that affect the nature of their ADL applications.

U.S. Army War College The U.S. Army War College (USAWC), the Army’s senior-level PME school, 

initiated its Web-based nonresident education program in April 1999. The 

program went online in an evolutionary process until the spring of 2002, 

whereby students received both text and online versions. Since the spring 

of 2002, all nonresident students have received their education via a 

combination of ADL technology and appropriate text. Nonresident 

students are board selected to participate in the program. It is a 2-year 

Web-based program that is the only delivery method offered to nonresident 

students. The program has a “blended” component, whereby 2 of its 

12 courses are taken in residence at USAWC. Also, distance courses are 

presented to students as a group or cohort; that is, students enroll at the 

beginning of the nonresident school year and must complete a sequenced 

load of 5 courses during the first year, followed by an additional 5 courses 

during the second year. The resident courses are of 2-week duration and 

are conducted at the end of each academic year. The nonresident program 

is designed to parallel the resident program, and graduates from both 

programs are awarded Master’s Degrees in Strategic Studies.

Naval War College The Naval War College’s (NWC) nonresident education programs are 

concentrated in its College of Distance Education, its only nonresident 

college and one of five colleges under the NWC umbrella. The College of 

Distance Education, an intermediate-level PME school, offers several 

nonresident options. The fleet seminar program has existed in various 

forms at the school since 1974; the Web-enabled correspondence program 

has been operating fully since October 2002; and the CD-ROM-based 

correspondence program, effective in April 2004, was designed to replace 

the phased-out paper-based correspondence course. Nonresident options 

are open to all officers and qualified personnel. The Web-based course can 

be completed in 18-24 months. While there is no formal resident portion to 

this course, students are assigned to cohort teams to facilitate team and 

faculty communication. This nonresident course is closely aligned with the 

resident course, and graduates are allowed to obtain graduate hour credits. 
Appendix IV

ADL Applications and Additional Features 

of Nonresident Programs

Page 42 GAO-04-873 ADL in Military Education

In the case of several seminars of the fleet seminar program, students can 

apply for admission to a program of graduate study leading toward a 

Master’s of Arts Degree in National Security and Strategic Studies.

Air Command and Staff 

College

The Air Command and Staff College (ACSC), the Air Force’s 

intermediate-level PME school, implemented its nonresident program in its 

present form in September 1999. There are two methods for completing the 

nonresident program: by seminar or by correspondence. The ACSC 

nonresident program is open to all officers and qualified personnel. The 

most recent version of the program consists of six courses organized into 

two semesters. The seminar method, which can take up to 11 months to 

complete, is conducted weekly, is typically composed of 3-18 students, and 

is led by assigned seminar leaders in order to facilitate group discussions. 

The correspondence program, a self-study program delivered in a balanced 

manner consisting of paper, CD-ROM, and Web-based delivery, requires up 

to 18 months to complete. Students move interchangeably between both 

programs, but they must achieve a minimum score of 70 percent on each of 

the six examinations and must complete four interactive Web-based 

exercises. The nonresident programs are designed to mirror resident 

programs, and there are multiple versions in use by ACSC nonresident 

students. These programs do not award master’s degrees, but the American 

Council of Education recommends up to 21 semester hours of graduate 

credit for course completion.

Joint Forces Staff 

College

National Defense University’s Joint Forces Staff College (JFSC) is 

piloting an Advanced Joint Professional Military Education pilot course 

for senior- and intermediate-level reserve officers. Initially launched in 

September 2003, it is designed to last 38 weeks. The period consists of 

35 weeks of Web-based education and 3 weeks of resident education with 

1 week occurring after the first 8 weeks of Web-based education, and the 

last 2 weeks at the end of the 38-week period. JFSC, already responsible for 

the resident Joint PME Phase II course used to complete the education 

process for joint specialty officers, was tasked to develop a Joint PME 

course for reserve officers in response to the fiscal year 1999 National 

Defense Authorization Act and the Joint Staff Guidance in May 2000. While 

there is no joint specialty officer requirement for reserve officers, JFSC 

was required to prepare reserve officers for joint duty assignments by 

providing a course similar in content to its resident Joint PME course, and 

to do so by utilizing current distance learning applications.
Appendix IV

ADL Applications and Additional Features 

of Nonresident Programs

Page 43 GAO-04-873 ADL in Military Education

Additional Features 

of Nonresident 

PME Programs

There are additional features of PME programs that affect the nature of 

their ADL applications. Those features include:

• Student Board Selection—Nonresident students are selected to attend 

the PME schools either through an annual board selection process or 

through open admissions. Only USAWC selects its nonresident students; 

the other programs with ADL applications have open-admission 

policies.

• Joint Professional Military Education—A significant portion of the 

PME curriculum involves study of joint service issues along with 

service-specific issues. Officers who successfully complete senior- or 

intermediate-level PME course work are awarded Joint PME Phase I 

credit, which is required for those who wish to serve as joint specialty 

officers. All nonresident programs with ADL applications grant Joint 

PME Phase I credit.

• Service Promotion Impact—PME officials stated that PME program 

completion and other forms of higher education are factors used in 

consideration for promotion and vary among the services. Generally, the 

Air Force requires completion of a corresponding PME level of study 

before a candidate is considered for the next promotion level. The Army, 

while not as strict as the Air Force, places a high value on PME and 

graduate education in promotion decisions. The Navy, placing a higher 

premium on operational experience, currently is less inclined to 

recognize PME as a credential for promotion.

• Learning Objectives Between Programs—PME officials stated that, as 

outlined by Joint Staff policies, learning objectives for nonresident 

courses are required to be the same for resident courses, regardless of 

the method of delivery. PME schools have instituted internal control 

processes to ensure the achievement of learning objectives for all 

programs, irrespective of delivery method. Generally, PME schools 

apply similar evaluation systems and criteria to both resident and 

nonresident programs.

• Levels-of-Learning—PME schools teach to differing achievement levels 

across and within the services, and they have designed their curricula 

accordingly. School officials refer to these achievement levels as 

“levels-of-learning” based on a taxonomy defined in the Joint Staff 

policy. (See table 3 for a detailed definition of levels-of-learning 
Appendix IV

ADL Applications and Additional Features 

of Nonresident Programs

Page 44 GAO-04-873 ADL in Military Education

designations.) For the schools with ADL applications, their desired 

levels of learning for nonresident programs may or may not be 

equivalent to the corresponding resident programs:

• USAWC—Synthesis/Analysis (same as for resident program).

• NWC—Application (resident program calls for synthesis/analysis).

• ACSC—Comprehension (resident program calls for 

synthesis/resident).

• JFSC (Planned)—Application (same as for resident program).

Table 3: Levels-of-Learning Definitions

Source: DOD.

Note: These terms, listed in increasing levels of achievement, are used to define the Joint PME 

learning objectives for PME schools.

Levels of learning Definitions

Knowledge The ability to remember previously learned material. This level involves recall of a wide range of material, from 

specific facts to complete theories, but all that is required is bringing to mind appropriate information. Terminology 

for achievement: defines, describes, identifies, labels, lists, matches, names, outlines, reproduces, selects, and 

states.

Comprehension The ability to grasp the meaning of material. Translating material from one form to another, interpreting material, 

or estimating future trends may show this level. Terminology for achievement: converts, defends, distinguishes, 

estimates, explains, extends, generalizes, gives examples, infers, paraphrases, predicts, rewrites, summarizes, 

translates, and understands.

Value The internalization and consistent display of a behavior. The levels of valuing consist of acceptance of a value, 

preference for a value, and commitment (conviction).

Application The ability to use learned material in new and concrete situations. This level includes application of rules, 

methods, concepts, principles, laws, and theories. Terminology for achievement: changes, computes, 

demonstrates, discovers, manipulates, modifies, operates, predicts, prepares, produces, relates, shows, solves, 

and uses.

Analysis The ability to break down material into its component parts so that its organizational structure may be understood. 

This level includes identification of the parts, analysis of the relationships between parts, and recognition of the 

organizational principles involved. Terminology for achievement: breaks down, diagrams, differentiates, 

discriminates, distinguishes, illustrates, infers, outlines, points out, selects, separates, and subdivides.

Synthesis The ability to put parts together to form a new whole. This level involves production of unique communications, a 

plan of operations, or a set of abstract relations. Terminology for achievement: categorizes, combines, compiles, 

composes, creates, devises, designs, explains, generates, modifies, organizes, plans, rearranges, reconstructs, 

relates, reorganizes, revises, rewrites, summarizes, tells, and writes.

Evaluation The ability to judge the value of material for a given purpose. Judgments are to be based on defined internal 

(organizational) or external (relevance to the purpose) criteria. Criteria are subject to value judgments. 

Terminology for achievement: appraises, criticizes, discriminates, explains, justifies, interprets, and supports.
Page 45 GAO-04-873 ADL in Military Education

Appendix V

Comments from the Department of Defense ApendxiV

Note: A GAO comment 

supplementing those in 

the report text appears 

at the end of this 

appendix.

See comment 1.
Appendix V

Comments from the Department of Defense

Page 46 GAO-04-873 ADL in Military Education

Note: Page numbers in the 

draft report may differ from 

those in this report.
Appendix V

Comments from the Department of Defense

Page 47 GAO-04-873 ADL in Military Education

GAO’s Comment The following is GAO’s comment on the letter from the Department of 

Defense dated July 23, 2004.

1. When we initiated this engagement in February 2003, a key objective 

was to determine (1) the assumptions for DOD’s decision to move 

officer senior- and intermediate-service schools from 1-year residency 

to shorter periods by using ADL and (2) which courses and schools 

would be affected. Immediately after fieldwork commenced, however, 

DOD informed us that it was no longer actively pursuing that approach. 

In April 2003, after consulting with the congressional requester, we 

informed our DOD point of contact regarding our pursuit of the 

engagement’s remaining objectives.

(350327)
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To 

have GAO e-mail you a list of newly posted products every afternoon, go to 

www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional 

Relations

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

Government Accountability Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00