Military Education: DOD Needs to Develop Performance Goals and Metrics for Advanced Distributed Learning in Professional Military Education
Published: 2004-07-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: As part of its transformation to prepare the armed forces to meet current and future challenges, the Department of Defense (DOD) is expanding its use of advanced distributed learning (ADL) techniques in senior- and intermediate-level officer professional military education (PME). ADL instruction does not require an instructor's presence, and it facilitates the use of varied learning management systems. To date, the application of ADL has been targeted to nonresident students. To determine whether DOD uses a systematic process for evaluating the results of ADL application, GAO was asked to examine DOD's metrics for assessing program effectiveness, to compare DOD's criteria for converting courses to ADL with those of private-sector institutions, and to identify the challenges to ADL implementation.
section_title: What GAO Found
paragraphs: DOD does not have specific performance goals and metrics with which to assess ADL effectiveness in PME. Furthermore, although GAO and private-sector organization have established frameworks for assessing the effectiveness of educational programs by focusing on metrics for learning outcomes--that is, the knowledge, skills, and abilities that students attain through learning activities--DOD's oversight focuses instead on educational inputs such as facilities, student to faculty ratios, and student body composition. Since ADL is still a new and evolving tool, systematic evaluative processes have not yet been required. Without clear goals and an effective process for evaluating the results of ADL application, DOD cannot ensure that its program is achieving an appropriate return on investment and other goals. The criteria for converting PME courses and curricula to ADL vary by school and by military service, are based on subjective choices as to which content is suited for online delivery, and are focused solely on nonresident programs. The private sector similarly lacks systematic criteria in its use of ADL. However, DOD's implementation of ADL programs for PME compares favorably with private-sector institutions. Cultural, technological, and resource challenges affect ADL implementation. For example, some military policies reflect a lower estimation of the value of nonresident PME, and many respondents to a survey of ADL students and alumni indicated that its quality and achievement of outcomes did not compare favorably, in their view, with those of resident education programs. The technological challenges of balancing computer access with network security, along with resource challenges of funding and increased burdens on limited administrative staff, are additional concerns.

section_title: Letter
paragraphs: As part of its transformation to prepare the armed forces to meet current  and future challenges, the Department of Defense (DOD) is transforming  the way it trains to favor more rapid and responsive deployment. A  significant element of its training transformation strategy is the application  of advanced distributed learning (ADL), a technique of instruction that  does not require an instructor’s presence; can use more than one form of  media; and emphasizes the use of reusable content, networks, and learning  management systems. DOD has been expanding its use of ADL in its  program of professional military education (PME). PME provides military  officers with a wide array of college-level academic courses in both  resident and nonresident settings; to date, the application of ADL has been  targeted to nonresident students. As a new tool, ADL is being examined to  determine whether DOD is applying a systematic performance evaluation  approach, particularly in light of the increased rate at which  servicemembers are being deployed worldwide. Without clear goals and an  effective process for evaluating the results of ADL application, DOD cannot  ensure that its program is achieving an appropriate return on investment  and other goals.
We were asked to review DOD’s use of ADL in senior- and intermediate- level officer PME, and specifically: 1. to examine the metrics DOD uses to assess the effectiveness of ADL  in PME, 2. to determine what processes and criteria DOD uses to select the  courses or curricula it converts to ADL and how these criteria compare  with those of other institutions in meeting ADL objectives in  nonresident education, and 3. to identify what barriers and challenges exist for implementing ADL  in PME.
We also reviewed and assessed the policies and guidance of several DOD  offices responsible for providing oversight for PME activities. These offices  included the Under Secretary of Defense for Personnel and Readiness and  the Joint Staff’s Joint Education Branch. We also studied experience in the  private education sector and in other parts of the government in measuring  the effectiveness of education programs. In addition, we surveyed  437 current students and graduates of senior- and intermediate-level PME  programs to obtain their perspectives on their PME experience.  Appendixes I and II describe our scope and methodology in more detail.
We conducted our review from March 2003 through June 2004 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: Each military service has separate PME schools for senior- and  intermediate-level officers. As defined by the Joint Staff’s Officer  Professional Military Education Policy, the senior-level schools, typically  for O-5 and O-6 ranked officers, focus on warfighting within the context of  strategy. The intermediate-level schools, typically for O-4 ranked officers,  focus on warfighting within the context of operations. (See table 1 for a list  of PME schools and enrollment totals.)
The senior- and intermediate-level PME schools are not alike in terms of  program offerings for resident and nonresident students. As indicated in  table 1, while all senior-level PME schools offer resident programs, only the  Army War College and the Air War College have analogous nonresident  programs. Also as indicated in table 1, all intermediate-level PME schools  offer resident and nonresident programs.
DOD has approximately 39,318 students enrolled in its senior- and  intermediate-level PME schools. The vast majority of these enrollees  are nonresident students. Of the total enrolled, approximately 3,788, or  10 percent, are taking course work as resident students; the rest, or  90 percent, are nonresident enrollees.
Nonresident PME exists to provide PME to a larger population than can be  supported in resident institutions. Nonresident PME provides alternative  learning-style options for officers not selected for residence or unable to  participate in residence due to operational commitments. The military  services have had nonresident PME programs for many years. The Naval  War College (NWC) has had a department for correspondence courses  since 1914. The U.S. Army War College (USAWC) has provided a  nonresident course offering since 1968. The Air Force’s nonresident  programs were created in 1947 for its senior-level PME school and 1948 for  its intermediate-level PME school.
Paper-based correspondence is the traditional nonresident PME delivery  mode. Students complete correspondence courses individually with limited  faculty contact. Course materials and submissions are exchanged between  students and faculty primarily by mail. PME schools have implemented  other delivery modes, including seminars conducted at remote sites by  PME faculty and CD-ROM distribution. Increasingly, PME schools are using  ADL techniques in their nonresident program offerings.
Several ADL applications are currently in use at senior- and intermediate- level PME schools, and all of them are focused on nonresident programs.  They are offered at the U.S. Army War College, the Naval War College, and  the Air Command and Staff College (ACSC). A planned ADL offering for  reserve component staff is under development at the Joint Forces Staff  College. See appendix IV for details on these programs. The addition of an  ADL application for the Army Command and General Staff College  nonresident PME course is anticipated for fiscal year 2005.
section_title: DOD Does Not Have Specific Metrics for Assessing Performance Goals or Learning Outcomes
paragraphs: DOD does not have specific performance goals and metrics to assess the  effectiveness of ADL in PME. While DOD believes ADL has had a positive  impact, its views are based on anecdotal information, rather than a  systematic performance measurement. Thus, DOD cannot determine  whether ADL is meeting performance goals in comparison to other delivery  methods. Although numerous organizations are providing oversight of PME  activities, with several specifically responsible for ensuring that PME  meets general standards of accreditation, these organizations do not focus  on student learning outcomes—that is, the knowledge, skills, and abilities a  student acquires from a course. Instead, DOD accreditation activities, like  those in the private sector, focus primarily on educational process inputs,  such as quality of facilities and student faculty ratios. We and a private- sector organization have recently established guidelines and frameworks  for assessing the effectiveness of educational programs that stress a focus  on measurable outcomes. ADL is a new and evolving tool for which  systematic evaluation requirements have not been established. ADL has a  unique ability to capture, retain, store, and document interactions in an  online environment, which provides the opportunity to demonstrate  student skill improvements, and thus to customize performance metrics.  However, we have found no evidence to indicate that DOD is utilizing this  ability.
subsections: section_title: Numerous Organizations Have Roles in Providing Oversight of PME Activities
paragraphs: Numerous oversight organizations review PME activities, with several  organizations specifically designed to ensure that PME conforms to general  standards of accreditation. The preeminent mechanism for oversight is the  Joint Chiefs of Staff’s Process for Accreditation of Joint Education. The  process is designed to provide oversight and assessment of PME  institutions for purposes of strengthening and sustaining Joint Professional  Military Education. It is a peer-review process involving a self-study  component and a team assessment. The review sequence includes  certification, accreditation, and reaffirmation of accreditation status.  Accreditation can currently be granted for up to 5 years, and all PME  programs with current ADL applications are Joint Staff-accredited. The  Joint Staff also sponsors the Military Education Coordinating Council, an  advisory body composed of high-ranking PME leadership. The purpose of  the Council is to address key issues of interest for joint military education,  to promote cooperation and collaboration among member institutions, and  to coordinate joint education initiatives.
The military services have responsibility for the service PME institutions in  terms of managing PME content and quality and conducting all levels  within the guidelines of the military educational framework. Consistent  with Title 10 of the United States Code, the Secretary of Defense requires  that each PME institution periodically review and revise curriculum to  strengthen focus on joint matters and on preparing officers for joint duty  assignments.
PME is also reviewed by other internal and external organizations. Each  PME institution has a Board of Visitors/Advisors that provides guidance  over PME activities. The Board of Visitors/Advisors is composed of military  and/or civilian academic officials who are nominated by PME schools and  appointed by service secretaries to provide advice on educational and  institutional issues. Service PME institutions have other internal and  external advisory committees that perform activities such as providing  advice, communicating feedback from major commands, and conducting  curriculum review. Service Inspector General offices have conducted  periodic reports and assessments on PME schools. The military services’  education and training commands also provide oversight of PME activities,  though not day-to-day administration. Additionally, private-sector regional  accreditation agencies assess senior- and intermediate-level PME  programs. Their accrediting activities generally guide the Joint Staff’s  review process.
section_title: Performance-Effectiveness Metrics for ADL Implementation Are Lacking
paragraphs: PME schools have not established, and oversight organizations have not  reviewed, specific goals or metrics of performance effectiveness for ADL  implementation. As was stated in our recently issued guide for establishing  a framework for assessing training and development efforts in the federal  government, “it is increasingly important for agencies to be able to evaluate  training and development programs and demonstrate how these efforts  help develop employees and improve the agencies’ performance.”  The Sloan Consortium—a private-sector organization that maintains a  repository of information on distance education—views metrics as  crucial for assessing program effectiveness. For example, metrics can  (1) demonstrate that the “learning effectiveness” of nonresident education  is at least as good as that of its resident counterpart, (2) identify cost  comparisons that can be used to develop better strategic plans, and  (3) provide information on student retention and completion rates. As was  stated in our report on oversight for the military academies, such elements  embody the principles of effective management, in which achievements are  tracked in comparison with plans, goals, and objectives, and the  differences between actual performance and planned results are analyzed.
PME schools identified advantages of ADL over other means of delivery,  but the advantages appeared to be anecdotally derived. PME school  officials stated that ADL has resulted in quality improvements in PME  delivery, especially when compared with paper-based correspondence.
These advantages include (1) better facilitation of student and faculty  interaction; (2) increased flexibility in modifying course material;  (3) reductions in time required to complete programs; (4) better leveraging  of resources for administrative support; and (5) establishment of learning  management systems that monitor student progress and produce  management reports. But there were no indications that evidence for these  advantages were based on an evaluative effort to compare differences  between ADL and paper-based correspondence courses. Since PME  schools have not detailed a comprehensive process for evaluating ADL  benefits over paper-based correspondence, it cannot be determined  whether ADL is meeting performance goals based on appropriate returns  on investment, student retention, student access to courses, or other goals  that schools use to measure program effectiveness.
Additionally, we did not observe any oversight agency focus on specific  metrics of ADL effectiveness. According to Joint Staff officials, they  perform reviews of nonresident programs as part of their accreditation  activities. However, their reports focus on the nonresident program as a  whole and not on particular methods of delivery. ADL is a new and evolving  tool, and a systematic assessment of these applications has not yet been  required. The three regional accreditation agencies that review PME  schools with ADL implementations show variances in nonresident program  evaluation policy. One agency stated that nonresident programs are not  separately evaluated, although the programs may be included within the  scope of the institution’s existing accreditation. Another agency stated that  additional procedures must be performed before nonresident programs are  included within the scope of the institution’s accreditation. The third  agency required schools to evaluate its nonresident programs to ensure  comparability to resident programs. In addition, we have not observed any  Office of the Secretary of Defense or Board of Visitors/Advisors reviews in  relation to ADL effectiveness for nonresident PME.
While we did not observe measures of effectiveness specifically geared  toward ADL applications, PME schools with ADL applications did perform  program effectiveness assessments for nonresident education by the way  of student satisfaction assessments as part of the Joint Staff accreditation  process. These assessments used in-course student surveys, graduate  surveys, and supervisory surveys to obtain feedback as part of a systematic  approach to instructional design and to update and improve curriculum  offerings.
USAWC performs surveys of students, alumni, and general officers with  USAWC graduates in their commands. Students are surveyed for each  course regarding particular aspects of the course and general degrees of  satisfaction. A survey of alumni is conducted every 2 years. A general  officer survey, designed to assess general officer impressions of alumni,  will now be conducted annually instead of every 3 years, as in the past.  Prior feedback from general officer surveys reported that the  curriculum should emphasize application of strategic thinking to  national security issues. USAWC also performs internal course  evaluations as part of its curriculum assessment process. USAWC  faculty members are required to undergo training to provide a degree of  standardization in instruction and evaluation. This standardization,  especially for evaluation, is more stringent for nonresident education.  USAWC can conduct trend analyses for student performance and  student satisfaction to determine statistical significances.
NWC uses student and alumni surveys to assess the academic program’s  effectiveness. Depending on the department, student assessments  include daily sessions critiques, lecture critiques, end-of-course  critiques, major exercise critiques, and exam critiques. Alumni are sent  questionnaires 2 years after graduation asking for feedback on their  educational experience. All academic departments conduct an extensive  analysis of various student surveys to determine areas of the curriculum  that are not meeting student needs so that these areas can be improved.  Surveys are based on standards promulgated by accrediting agencies  and external organizations to help objectively measure institutional  excellence. Resident and nonresident student programs are measured  the same since a single faculty is responsible for both. Peer evaluation of  faculty members is used to sustain teaching method quality.
ACSC uses internal and external evaluations at all phases of its  curriculum development process. It conducts end-of-course surveys  that focus on delivery and educational support and end-of-year surveys  for students to provide feedback about whether they believed the school  (1) prepared them to lead commands, (2) accomplished its mission,  (3) was institutionally effective, and (4) was beneficial to  professional development. Surveys are also given to graduates and  graduate supervisors to obtain perspectives on whether the  school (1) accomplished its mission and institutional effectiveness;  (2) enhanced graduates’ ability to think operationally and critically;  (3) prepared graduates to assume leadership duties, and (4) made the  experience valuable in professional development.
section_title: Metrics for Learning Outcomes Are Lacking
paragraphs: Student learning outcomes, as stated by the Council for Higher Education  Accreditation—a national association representing accrediting  organizations—are “properly defined in terms of the knowledge, skills, and  abilities, that a student has attained at the end (or as a result) of his or her  engagement in a particular set of higher education experiences.” PME  schools generally are not assessed for student learning outcomes as a  means of determining program effectiveness. The Joint Staff’s  accreditation organization responsible for assessing PME schools has  primarily focused on inputs to the educational process. As detailed in its  policy, its educational standard assessment and self-study requirements  focus on internal aspects such as organizational structure, facilities,  curricula, student to faculty ratios, student body composition/mix, and  faculty qualifications. However, as stated in our recently published guide  for assessing training and development programs, the focus on evaluating  activities and processes takes away from evaluating training and  development’s contribution to improved performance, reduced costs, or  greater capacity to meet new and emerging transformation challenges. The  Joint Staff has identified the usefulness of student learning outcomes and is  currently in the process of developing student learning outcomes for PME  and procedures to include them in the accreditation process.
Our recently published report on distance education states that there is  increased interest in using outcomes more extensively as a means of  ensuring quality in all forms of education, including nonresident  education. The Council for Higher Education Accreditation has issued  guidelines on nonresident education and campus-based programs that call  for greater attention to student learning outcomes, and the congressionally  appointed Web-based Education Commission has also called for greater  attention to student outcomes. The Commission said that a primary  concern related to program accreditation is that “quality assurance has  too often measured educational inputs (e.g., number of books in the  library, etc.) rather than student outcomes.”
Private-sector educational institutions are just beginning to emphasize  the evaluation of learning outcomes as a viable measure of program  effectiveness. For example, the University of Maryland University College,  a school with a comparably large distance education program and which  serves a large number of military personnel, is piloting a project to identify  and measure learning outcomes in five general areas—writing efficiency  and oral communication, technology fluency, information literacy,  quantitative literacy, and scientific literacy. The university will use  knowledge captured by its distance education database to serve as a basis  for this determination.
Accrediting agencies and our recent report on training and development  program assessments are also emphasizing the evaluation of learning  outcomes as a measure of program effectiveness. Some of the regional  agencies that accredit programs at the senior- and intermediate-level PME  schools generally recognize the importance of student learning outcomes  and have instituted practices that reflect some aspects of a systematic,  outcome-based approach called for in GAO-04-279. However, these  agencies vary in the extent to which standards and policies address student  learning outcomes for distance education. Our training and development  assessment guide states that agencies need credible information on how  training and development programs affect organizational performance, and  that decision makers will likely want to compare the performance of these  programs with that of other programs. Furthermore, programs lacking  outcome metrics will be unable to demonstrate how they contribute to  results.
We surveyed nonresident PME current students and graduates to obtain  their perspectives on the achievement of PME learning objectives and  PME’s impact on their career objectives. (See appendix III for presentation  of survey results.) Because we only surveyed nonresident students, we  could not compare the results with those of resident students. However, we  believe the data can be useful for DOD to consider in its continuing study of  program effectiveness.
section_title: ADL Can Be Used to Capture Valuable Measurement Data
paragraphs: ADL has a unique ability to capture, retain, store, and document  interactions in an online environment, which provides the opportunity  to demonstrate student skill improvements, and thus to customize  performance metrics. Since work is done on a computer, various data  points are automatically collected as a student works, including the time  spent, specific pages of the text visited, use of online help, and  communication with others. University of Maryland University College  officials pointed out ADL’s unique ability when compared with other  delivery methods to retain, capture, store, and document baseline data that  can be used as the basis for performance metrics. These officials said they  would use such data in designing performance measures for learning  outcomes. However, we found no evidence to indicate that DOD is using  this ability. DOD may be missing an opportunity to enhance its ability to  measure effectiveness.
section_title: ADL Conversion Varied by School and by Service Based on Subjective Assessments of Content Suitability
paragraphs: The processes for converting PME courses to ADL varied by school and by  military service, and they feature a mixture of in-house and contractor  approaches. PME schools generally focus their ADL applications on  nonresident programs to improve their efficiency and effectiveness. In  most cases, conversion decisions were made in collaboration with school  academic boards. PME schools did not identify any systematic criteria as  the basis for their decisions as to which courses or curricula to convert to  ADL. They subjectively focused ADL conversion on the suitability of  content for Web-based applications. Curriculum conversions were made  because of a DOD-wide need (1) to improve access to a diverse officer  corps and (2) to increase the efficiency of educational delivery. Since  nonmilitary educational institutions also lack systematic criteria for  converting courses or curricula to ADL for nonresident education, DOD’s  approaches are in fact consistent with mainstream practice, and in some  cases, compare favorably with best practices in nonmilitary education.
subsections: section_title: DOD Processes and Criteria for ADL Conversion Varied
paragraphs: The processes for converting PME courses to ADL varied by school and by  military service, and they feature a mixture of in-house and contractor  approaches. However, the conversions were focused on the schools’ entire  nonresident programs. USAWC’s and ACSC’s ADL applications were  developed and managed by in-house staff and faculty. USAWC and ACSC  used staff consisting of instructional designers and courseware developers  interacting with respective faculty to develop courses. NWC’s ADL  application combined the use of contractor and in-house support.  Contractor staff created Web-based applications for two of the three  courses in NWC’s curriculum. NWC officials learned enough from  contractor efforts to create a Web-based application for the remaining  course with in-house staff. In all cases, the ADL applications were applied  to affect the entire nonresident curriculum and, in most cases, were  preceded by reviews and final decisions from the schools’ academic  boards.
The PME schools did not identify any systematic criteria that inform their  selection of course for conversion to ADL. Rather, they made subjective  decisions as to the appropriate parts of the curriculum that should be  delivered online based on content suitability. While USAWC’s application is  delivered mostly through the Internet, print media delivers a portion of the  course content as well. USAWC’s application also includes two 2-week  resident components in which students are brought together to achieve  learning objectives best suited to resident instruction. These objectives  include verbal communication, interaction in live settings, interpersonal  skills used in direct relationships, and other skills that are important  components of the resident experience. USAWC’s application also uses  asynchronous “threaded discussions,” in which faculty members initiate  online discussions with students on various academic topics. NWC’s and  ACSC’s applications, which do not include resident components, blend  content delivery using print media, CD-ROM, and the Internet. In NWC’s  application, print media is used for material that is mostly text and requires  limited interactive capabilities; CD-ROMs are delivered to students for  material that is not routinely updated; and students are assigned to cohort  teams that allow online interactive opportunities and group discussions.  In ACSC’s application, almost all nonresident materials are provided to  students on CD-ROMs as well as on the Internet to allow as much flexibility  as possible to complete nonresident courses.
Generally, PME school officials stated that ADL conversions were made  because of nonresident PME’s need (1) to respond to multiple learning  styles in a diverse officer corps, (2) to increase the efficiency of  educational delivery, and (3) to improve the quality of the educational  offering. Additionally, NWC saw ADL as a means of potentially affecting  retention and increasing enrollment. USAWC saw ADL as a means of  responding to student demands for a more efficient and relevant  educational experience. ACSC saw ADL as an improved means for  delivering course material to increasing numbers of deployed officers.
section_title: Nonmilitary ADL Conversion Decisions Were Similar
paragraphs: In nonmilitary applications of ADL, we observed processes and criteria for  conversion decisions that were similar to DOD’s. Course conversions  followed analogous processes and criteria decisions, driven by factors such  as interest and student enrollment that were not systematic. University of  Maryland University College officials stated that their conversion process  includes a dedicated staff of instructional designers and subject matter  experts (usually faculty) who produce course conversion to distance  learning content within an established framework to ensure maintenance  of standards. Their criteria for conversion focus on high-student demand  and high levels of interest and on course work that required less “hands-on”  training, such as business and information technology courses. Further  research of private-sector practices supports the observation that the lack  of systematic criteria is consistent with mainstream practice in ADL  adoption for nonresident education.
DOD’s approaches for course conversion are thus consistent with  mainstream practice, and in some cases, compare favorably with the best  practices in nonmilitary education. For example, NWC’s ADL application  received the Crystal Award in 2002 from the Association for Educational  Communications and Technology based on “innovative and creative use of  the medium, instructional value and relevance, soundness of instructional  strategy, quality of production, and evidence of successful outcomes.” As of  June 2004, USAWC’s nonresident education program is fully accredited by  the Middle States Commission of Higher Education for the awarding of  Master’s of Strategic Studies Degrees. USAWC is the first military  institution to achieve degree-granting authority for its nonresident  ADL-based program.
section_title: Cultural, Technological, and Resource Barriers and Challenges Affect ADL Implementation in PME Programs
paragraphs: PME schools identified a number of cultural, technological, and resource  challenges that affect ADL implementation and may affect future  maintenance or expansion of ADL efforts. Cultural issues such as  appropriate extent of ADL incorporation, general perceptions about  nonresident education, and limited ADL research in military education  affect the degree of ADL implementation. Technological trade-offs and  nonresident program resourcing also affect continued ADL efforts.
subsections: section_title: Cultural
paragraphs: PME officials question the appropriate extent to which ADL should be used  in nonresident education and how closely it can, or should, enable  nonresident education to approximate resident education. It is generally  recognized that resident programs are better in providing acculturation,  interactive skills, and simulations that are critical for professional  development of officers, and that there are challenges in providing such  aspects in nonresident education. But some observers believe that  nonresident education should not be compared to resident education and  that nonresident education represents a vital broadening experience in its  own right. In addition, there are indications that an ADL approach could  significantly enrich nonresident content by excluding teaching methods  that do not work in residence, allowing students the flexibility to focus on  material that requires further study without class disruption, and serving as  the basis for applications that can be used to upgrade teaching methods in  resident programs.
ADL implementation could also be affected by certain negative perceptions  concerning nonresident education that are held by students, and in some  cases, reflected in policy. Our survey of nonresident PME current students  and recent graduates indicated that about 49 percent of current students  and 48 percent of graduates believe they are not as well prepared as are  their resident student counterparts, regardless of nonresident delivery  method. While not universal between the services, we observed instances  of military education policies that reinforce the perception that  nonresident education is not as desirable as resident education. The Air  Force’s PME Instruction AFI36-2301 states that “ideally, all officers will  attend PME in residence,” and that limited resources restrict resident  attendance to the “best qualified.” Furthermore, “completing nonresident  PME programs will not affect eligibility for resident PME programs.”  Indeed, we were told of instances where officers who after completing the  nonresident PME program subsequently enrolled in the corresponding  senior- or intermediate-level course in residence.
The extent of ADL implementation in nonresident education is affected by  the role that PME completion plays in promotional consideration.  Programs maintained to foster promotional consideration—that is,  “personnel-oriented”—might not be compatible with programs  emphasizing the learning outcomes brought about by ADL—that is,  “education-oriented.” Our survey shows that promotional considerations,  rather than learning outcomes, are the focus for students in nonresident  education. An estimated 73 percent of current students and 84 percent of  recent graduates listed as their predominant reason for participating in  nonresident PME a desire to improve their promotional chances or to meet  promotional requirements. In addition, of an estimated 22 percent of PME  graduates who were promoted to a higher rank after PME completion, an  estimated 88 percent stated that PME contributed to a “great” or “very  great extent” in their promotions. But ADL adoption goals should focus  more on learning outcomes than on simply “checking the boxes” for  promotional opportunity enhancement.
PME officials state that there are concerns that ADL advantages could be  oversold to the point that ADL may be used to supersede resident programs  and shift the burden of PME administration. DOD officials, already viewing  ADL delivery methods as attractive from a cost savings perspective, are  desirous to expand such programs, even at the expense of resident  programs. However, PME officials believe that ADL expansion should be  considered only after completely understanding its impact on military  operations and recognizing the resident program’s role as the basis for a  nonresident program. In addition, PME officials noted that ADL could be  used as a means of shifting the burden from the school to the student and  the student’s commands without providing appropriate command support,  compensation, or resources.
While ADL research exists for military training courses, there is only  limited research on ADL’s impact on military education, especially in terms  of its impact on learning that requires interactive elements. A DOD official  stated that it is well-known in studies that distance learning (usually  teleconferencing) instruction offers no significant differences as compared  with classroom instruction. However, officials believe that more work  should be done to collect information that ADL improves learning in  military education and that these studies should focus on collaborative  learning environments and the extent of their translation. In addition,  further efforts should look at commercial education studies  (undergraduate and graduate education) and its transfer to military  education.
section_title: Technological
paragraphs: PME school officials have stated that decisions are needed on trade-offs  between increased demands for student access (more servers, more  bandwidth, or reduced firewalls) and the maintenance for network  security. Such decisions are complicated by what is viewed as a lack of  standardization in DOD and within individual services on security  requirements.
In our nonresident survey, approximately 19 percent of current students  and 16 percent of recent graduates experienced computer/Internet related  problems affecting their PME experience. Some identified problems  included servers, bandwidth issues, and security firewalls. PME schools  considered using “.edu” domains to make courseware more available to  students because of the difficulty of interacting “.mil” domains with  systems outside of military organizations. However, such moves would be  expensive and would conflict with increasing requirements to reduce the  number of servers and personnel needed to operate these systems. Some  reported problems involved limited bandwidth issues. While such problems  can be overcome, they require time and money to resolve. Firewalls  maintained for security purposes have caused schools to limit library  access to nonresident students due to perceived security threats.
section_title: Resources and Funding
paragraphs: Most ADL efforts at PME schools were fielded independently with limited  budgets and staffing. USAWC’s and ACSC’s ADL applications were  developed and supported with in-house staff responsible for managing  resident and nonresident programs. These applications were fielded  independently within the services. PME officials stated that PME schools’  ability to fund ADL applications is limited due to DOD’s priority to focus  more on its training and operational activities. An emerging funding issue  involves the use of copyrighted material in ADL applications. Increasing  costs in using copyrighted material for course work could result in limiting  course flexibility and methodologies.
New technologies, such as ADL, create new requirements for faculty  personnel with higher technical expertise and more equipment and  structure than traditional programs. Faculty members must be skilled to  perform in online and classroom settings. PME schools are beginning to  observe they must offer faculty opportunities to teach courses in multiple  medias or risk losing qualified faculty to competitors.
section_title: Conclusions
paragraphs: Although PME schools receive oversight from a number of organizations,  we observed that neither the schools nor the oversight agencies had  focused on (1) establishing specific performance effectiveness goals for  ADL implementation or (2) measuring learning outcomes as a means of  evaluating program effectiveness. The Joint Staff’s accreditation reports on  nonresident education do not detail performance goals for any particular  delivery method. The military services, which have primary responsibility  for PME oversight, view the accreditation process provided by the Joint  Staff as the primary means of ensuring the effectiveness of nonresident  education. DOD is not alone in this problem—the lack of metrics for  performance effectiveness and learning outcomes is pervasive throughout  all educational institutions. Our prior efforts indicate that most public and  private institutions lack a framework with which to assess implementation  of training and development efforts. However, agencies need clear  information on how training and development efforts affect organizational  performance, and decision makers will likely want to compare the  performance of these programs with that of other programs. Without clear  goals and an effective process for evaluating the results of ADL application,  DOD cannot ensure that ADL is achieving appropriate return on  investment, student retention, student access, and other goals in  comparison with prior efforts. Furthermore, programs lacking outcome  metrics will be unable to demonstrate how they contribute to results.  Moreover, by not capturing and using student data that are uniquely  available through ADL techniques, DOD is missing the opportunity to  develop the basis for effectiveness metrics and knowledge about  learning outcomes.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense direct the Under Secretary of  Defense for Personnel and Readiness, in concert with the Joint Staff,  service headquarters, and the PME schools, to take the following two  actions:  promote the development of specific performance effectiveness goals   promote the use of ADL technologies to capture data to provide  knowledge about learning outcomes.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD partially concurred with our first recommendation and fully  concurred with the second. DOD supports the use of specific effectiveness  goals for PME, but believes such goals are not appropriate for any specific  delivery method. While we acknowledge DOD’s concerns with focusing on  a specific delivery method, we believe that ADL is unlike other means of  delivery because of its potential to modernize the educational experience  and because its use is rapidly expanding in other areas of PME. We believe  it would be worthwhile for DOD to know specifically how well ADL  performs, especially in comparison with other delivery methods, in order to  better understand its appropriate use for PME. DOD concurred with our  second recommendation and stated that current accreditation practices are  already promoting the data collection capabilities of ADL technologies for  assessing multiple delivery methods. DOD’s comments are included in this  report as appendix V. DOD also provided technical changes, which we  incorporated as appropriate.
We are sending copies of this report to congressional members as  appropriate. We will also send copies to the Secretary of Defense; the  Secretaries of the Air Force, Army, and Navy; the Commandant of the  Marine Corps; and the Chairman of the Joint Chiefs of Staff. We will make  copies available to others on request. In addition, this report will be  available at no charge on the GAO Web site at http:/www.gao.gov.
If you or your staff have any questions, please call me on (757) 552-8100 or  Clifton Spruill, Assistant Director, on (202) 512-4531. Major contributors to  this report were Arnett Sanders, Maewanda Michael-Jackson, Jean Orland,  David Dornisch, Terry Richardson, and Cheryl Weissman.
section_title: Scope and Methodology
paragraphs: We reviewed Department of Defense’s (DOD) implementation of advanced  distributed learning (ADL) in senior- and intermediate-level professional  military education (PME) to determine processes and criteria used for  converting courses, metrics to assess ADL effectiveness and its fulfillment  of learning objectives, and barriers and challenges in ADL implementation.  We collected, reviewed, and analyzed relevant program information and  conducted interviews with DOD officials responsible for ADL  implementation in PME programs and oversight responsibilities. We  initially obtained data from the Principal Deputy Under Secretary of  Defense for Personnel and Readiness to identify PME programs with ADL  applications. A review of the data indicated that there were three existing  programs. We identified, interviewed, and obtained data from officials from  PME schools with ADL applications. Those schools were  U.S. Army War College, Carlisle, Pennsylvania;  Naval War College, Newport, Rhode Island; and  Air Command and Staff College, Montgomery, Alabama.
We also interviewed and obtained data from officials at the Joint Forces  Staff College in Norfolk, Virginia, on their pending ADL application.
We also interviewed and obtained data from agencies within DOD  responsible for oversight of PME activities. Those agencies included  The Joint Chiefs of Staff’s Operational Plans and Joint Force  Development Directorate (J-7), Joint Doctrine, Education, and Training  Division, Joint Education Branch, Arlington, Virginia;  The Office of the Under Secretary of Defense for Personnel and  Readiness, Deputy to the Under Secretary for Readiness, Office of  Readiness and Training Policy and Programs, Arlington, Virginia;  The Office of the Under Secretary of Defense for Personnel and  Readiness, Deputy to the Under Secretary for Military Personnel Policy,  Arlington, Virginia;  Department of the Army, Office of the Deputy Chief of Staff, Operations  and Plans, Arlington, Virginia;  Department of the Navy, Office of the Chief of Naval Operations,  Personal Development and Accessions Division, Washington, D.C.;  Department of the Air Force, Office of the Deputy Chief of Staff for  Personnel, Learning, and Force Development, Arlington, Virginia;  U.S. Marine Corps Combat Development Center, Training and Education  Command, Quantico, Virginia;  U.S. Army War College Board of Visitors, Carlisle, Pennsylvania;  Naval War College Board of Advisors, Newport, Rhode Island; and  Air University Board of Visitors, Montgomery, Alabama.
To determine sufficient metrics to assess ADL effectiveness, we provided  PME program officials with a detailed list of questions that included those  relating to effectiveness and learning objectives. We reviewed written  responses, if provided, and followed up with site visits and correspondence  with oversight agencies to clarify or obtain additional information if  necessary. We also obtained and analyzed data from a survey of  nonresident PME current students and graduates, which included  questions designed to obtain perceptions on program effectiveness. Details  of the survey methodology are presented in appendix II.
To determine processes and criteria DOD used for ADL conversion, we  provided PME program officials with a detailed list of questions that  included those relating to process and criteria decisions. We reviewed  written responses, if provided, and followed up with site visits to clarify or  obtain additional information if necessary. To determine whether criteria  were consistent with those of other institutions performing distance  education, we researched prior literature on this topic and conducted a site  visit to the University of Maryland University College in Adelphi, Maryland.  The school was identified in our prior reports on distance education as  having a program with a large distance education population, as well as  educating a significant number of military officers. We also contacted and  received data from the Sloan Consortium, an organization designed to  encourage collaborative sharing of knowledge and effective practices to  improve online education.
To determine barriers and challenges to ADL implementation, we provided  PME program officials with a detailed list of questions that included those  relating to barriers and challenges. We reviewed written responses, if  provided, and followed up with site visits and correspondence with DOD  oversight agencies to clarify or obtain additional information if necessary.  We also obtained and analyzed data from a survey of nonresident PME  current students and graduates, which included questions designed to  obtain perceptions on barriers and challenges in completing PME courses.  Details of the survey methodology are presented in appendix II.
section_title: Methodology for Our Survey of Nonresident PME Students and Graduates
paragraphs: To obtain military officers’ perspectives on nonresident PME in terms of  impact on careers, achievement of learning objectives, and obstacles and  challenges, we conducted a statistically representative survey of current  nonresident senior- and intermediate-level PME students and graduates of  these schools from April 1999 to March 2003, roughly the period coinciding  with initial ADL implementation at several senior- and intermediate-level  schools. We present the survey questions and response results in  appendix III.
subsections: section_title: The Study Population
paragraphs: The population for the nonresident PME survey consisted of current  students and graduates who fulfilled the following criteria: 1. Respondents were identified as enrolled in a senior- and  intermediate-level nonresident PME program of study from April 1999  to March 2003. We decided on this time period to ensure that our  respondents would have begun their programs after Web-based PME  had been clearly established as a mode of instruction or have been in  PME long enough to have meaningful responses to our questions. 2. Respondents participated in a senior- and intermediate-level  nonresident PME program of study, as opposed to individual PME  courses undertaken via continuing education programs. 3. Respondents are currently active in the U.S. military services or  reserves, excluding U.S. civilians; U.S. Coast Guard members; and  international members, either military or civilian. 4. Respondents participated (i.e., currently enrolled or graduated)  in a nonresident PME program in one of the six senior- and  intermediate-level PME schools: U.S. Army War College, Army  Command and General Staff College, Air War College, Air Command  and Staff College, Naval War College, and Marine Corps College of  Continuing Education.
The survey asked respondents about PME’s impact on furthering career  objectives, their achievement of learning objectives, and obstacles and  challenges of the programs. Specific questions concerned students’  satisfaction with their overall program, various modes of program delivery,  and with technologies used; students’ time and duty management concerns;  and reasons for participation in nonresident PME.
section_title: Developing the Survey
paragraphs: To develop areas of inquiry for the survey, we reviewed our previous work  related to distance education and PME. We reviewed a series of survey  questionnaires developed by us and by DOD. We used these sources and  our own analysis to develop an initial set of questions. We further  developed and refined the questionnaire by obtaining and incorporating  written comments regarding the initial questions from administrators and  other representatives of the senior- and intermediate-level PME schools.
In addition to an internal expert technical review by our Survey  Coordination Group, we pretested the survey with five individuals whose  personal characteristics corresponded to our eligibility criteria. We  identified pretest subjects through our contacts who were current military  personnel or who knew military personnel and our PME points of contact.
section_title: The Sample Design and Administration
paragraphs: We conducted the survey between January and April of 2004 on a random  sample of 437 current students and graduates of nonresident PME  programs using a self-administered Web-based questionnaire. We drew the  names of our respondents from an overall population data set we  constructed that combined separate data sets received from each of the  senior- and intermediate-level PME schools. For each data set, we  requested the officer’s name, school attended, month and year of initial  enrollment, component (defined as either active duty or reservist), and  mode of PME delivery. We requested e-mail addresses and, if needed,  phone numbers from potential respondents after they were drawn from the  population data sets. We stratified our sample by component in order to  better understand any differences between these components.
We activated the survey Web site and informed our sample respondents  of the Web site, their logon name, and passwords by e-mail on January 30,  2004. To maximize the response rate, we sent five subsequent follow-up  e-mail reminders to nonrespondents in February and March 2004. At the  time of the third mailing, we also telephoned many of the nonrespondents  to encourage them to complete the survey. We ended data collection  activities on April 30, 2004.
Of the 437 selectees included in our sample, we received 273 useable  questionnaires. We defined useable as respondents who completed the  survey and were not identified as out-of-scope. During the survey, we  deemed 67 of the 437 to be outside the scope of our survey after  determining that they did not meet at least one of our eligibility criteria.  Disregarding these 67 responses, our overall response rate was  73.8 percent (273/370). Table 2 shows the final disposition of the sample  (the 437 respondent accounts activated) by strata.
The estimates we make in this report are the result of weighting the survey  responses to account for effective sampling rates in each stratum. These  weights reflect both the initial sampling rate and the response rate for each  stratum. As with many surveys, our estimation method assumes that  nonrespondents would have answered like the respondents.
section_title: Sampling Error
paragraphs: For the estimates we present in this report, we are 95 percent confident  that the results we would have obtained had we studied the entire  population are within +/-10 or fewer percentage points of our estimates  (unless otherwise noted). Because we surveyed a sample of recent  nonresident PME students, our results are estimates of student and  graduate characteristics and thus are subject to sampling errors. Our  confidence in the precision of the results from this sample is expressed in  95 percent confidence intervals, which are expected to include the actual  results for 95 percent of the samples of this type. We calculated confidence  intervals for our results using methods appropriate for a stratified  probability sample.
section_title: Nonsampling Error and Data Quality
paragraphs: We conducted in-depth pretesting of the questionnaire to minimize  measurement error. However, the practical difficulties in conducting  surveys of this type may introduce other types of errors, commonly known  as nonsampling errors. For example, measurement errors can be  introduced if (1) respondents have difficulty interpreting a particular  question, (2) respondents have access to different amounts of information  in answering a question, or (3) errors in data processing occur. We took  extensive steps to minimize such errors in developing the questionnaire,  collecting the data, and editing and analyzing the information. The  Web-based data management system we used provides a systematized  process for processing, transferring, and analyzing data that also protects  against nonsampling errors. In addition, we performed tests to ensure the  reliability and usefulness of the data provided by the PME schools. These  included computer analyses to identify inconsistencies both within and  across the data sets and other errors in the data sets from which we  developed our overall sampling frame. We also interviewed agency officials  knowledgeable about the data. We determined that the data were  sufficiently reliable for the purposes of this report.
section_title: Survey Responses
subsections: section_title: Introduction
paragraphs: Welcome DoD Distance Learning Student or Graduate.
The U.S. General Accounting Office (GAO) - an agency of the U.S. Congress  has been requested to review various aspects of nonresident professional  military education (PME). Part of that effort is to evaluate whether the  appropriate decisions are being made in regards to nonresident education  for intermediate and senior-level military officers. As part of this effort, we  are assessing opinions of nonresident PME graduates and current students  on (1) the achievement of PME learning objectives, (2) obstacles and  challenges in completing PME, and (3) PME’s impact on furthering career  objectives.
The survey should take approximately 10 minutes to complete.
Participation in this survey is voluntary but encouraged. Your responses  will be confidential and the results of the survey will be reported in  aggregate form only.
Before choosing an answer, please read the full question and all response  choices carefully.
Thank you in advance for your participation.
section_title: Frame of Reference
paragraphs: Please consider only the following when answering the questions on this  survey:  1. Nonresident Professional Military Education programs of study that  you have participated in, as opposed to individual PME courses  undertaken via continuing education programs.  2. Nonresident Professional Military Education programs of study that  you began in April 1999 or after.
Our survey of nonresident Professional Military Education students was  divided into two main parts, one with questions appropriate for current  students and one with similar questions worded slightly differently for  graduates. There are also questions on demographics to which both current  students and graduates responded. Survey questions and responses for  graduates are indicated in italics and those for current students are in plain  text.
The information provided here represents weighted data. For information  on weighting, see appendix II.
Questions 1 and 31 are intentionally omitted because they contained  instructions telling respondents which questions to answer.
section_title: Survey of Nonresident Professional Military Education Graduates and Current Students
paragraphs: Q2. The name of program in which currently enrolled: Q32. From which school did you graduate?
Air War  College  Nonresident  Program  (percent)
Army  Command and  General Staff  College  Nonresident  Program  (percent)
Army War  College  Distance  Education  Program  (percent)
Naval College  of Distance  Education  (percent) (percent)
Q3. In what month and year did you begin your PME program?
Q33. In what month and year did you begin your PME program (if you graduated from more than one program, answer for the most recent  one)? (percent) (percent) (percent) (percent)
Q4. What mode of instruction have you used most often in your nonresident Professional Military program?
Q34. What mode of instruction did you use most often in your nonresident Professional Military program? (percent) (percent)
Q5. In what month and year do you expect to complete your PME studies program?
Q35. In what month and year did you complete your Professional Military Education program? (percent) (percent) (percent) (percent) (percent) (percent) (percent) (percent) (percent) (percent)
Q6. In a typical week, approximately how many hours did you spend in Professional Military Education-related activities, including  preparation, study, working on-line, and time in class?
Q36. In a typical week, approximately how many hours did you spend in Professional Military Education-related activities, including  preparation, study, working on-line, and time in class?
Q7. Does the military or your employer afford you time during your work-week for Professional Military Education?
Q37. Did the military or your employer afford you time during your work-week for Professional Military Education? (percent)
Q8. How many hours do you work in paid employment in a typical work-week (outside of Professional Military Education-related activities)?
Q38. During the period of time that you were completing your Professional Military Education program, how many hours did you work in  paid employment in a typical work-week (outside of Professional Military Education-related activities)?
Q9. Listed below are various reasons why someone would participate in a nonresident PME program. What is your greatest reason for  participating in a nonresident Professional Military Education program?
Q39. Listed below are various reasons why someone would participate in a nonresident PME program. What was your greatest reason for  participating in a nonresident Professional Military Education program Current  Students  (percent) (percent)
To gain knowledge in my field or in fields of interest to me To improve my chances of, or meet the requirements for,  promotion Q10. To this point in time, how satisfied or dissatisfied are you with your Professional Military Education program?
Q40. Overall, how satisfied or dissatisfied are you with the Professional Military Education program in which you graduated?
Somewhat  satisfied  (percent) (percent) (percent) (percent)
Q11. To what extent, if any, has your Professional Military Education program benefited your military career to this point in time?
Q41. To what extent, if any, has graduation from your Professional Military Education program benefited your military career to this point  in time? (percent) (percent) (percent) (percent) 12 Q12. To what extent, if any, do you believe the knowledge you are acquiring in your Professional Military Education program will improve  your effectiveness in future assignments?
Q42. To what extent, if any, do you believe the knowledge you acquired in your Professional Military Education program has improved your  effectiveness in job assignments? (percent) (percent) (percent) (percent)
Q43. Have you been promoted to a higher rank since you completed your Professional Military Education program? (percent)
Q44. To what extent, if any, do you believe that completion of your Professional Military Education program contributed to your promotion? (percent) (percent) (percent) (percent)
Q13. To what extent, if any, does your Professional Military Education program enable you to acquire the knowledge you are expected to  obtain?
Q45. To what extent, if any, did your Professional Military Education program enable you to acquire the knowledge you were expected to  obtain? (percent) (percent) (percent) (percent)
Q14. Was/is any part of your Professional Military Education program taken through seminar/classroom-based instruction?
Q46. Was any part of your Professional Military Education program taken through seminar/classroom-based instruction? (percent) 50 Q15. Overall, how would you rate the quality of this seminar/classroom-based learning?
Q47. Overall, how would you rate the quality of this seminar/classroom-based learning? (percent) (percent) (percent) (percent)
Q16. Was/is any part of your Professional Military Education program taken through paper-based correspondence?
Q48. Was any part of your Professional Military Education program taken through paper-based correspondence? (percent)
Q17. Overall, how would you rate the quality of this paper-based correspondence learning?
Q49. Overall, how would you rate the quality of this paper-based correspondence learning? (percent) (percent) (percent) (percent)
Q18. Was/is any part of your Professional Military Education program taken through the World-Wide Web or Internet?)
Q50. Was any part of your Professional Military Education program taken through the World-Wide Web or Internet? (percent)
Q19. Overall, how would you rate the quality of this web/Internet-based learning?
Q51. Overall, how would you rate the quality of this web/Internet-based learning? (percent) (percent) (percent)
Q20. How easy or difficult has it been for you to use web/Internet-based learning?
Q52. How easy or difficult was it for you to use web/Internet-based learning? (percent) (percent) (percent) (percent)
Q21. How easy or difficult have you found interaction with faculty during your web/Internet-based learning?
Q53. How easy or difficult did you find interaction with faculty during your web/Internet -based learning? (percent) (percent) (percent) (percent)
Q22. How easy or difficult have you found interaction with other students during your web/Internet-based learning?
Q54. How easy or difficult did you find interaction with other students during your web/Internet-based learning? (percent) (percent) (percent) (percent)
Q23. How well does the courseware/course software work on the computer equipment to which you have access for taking web/Internet- based learning?
Q55. How well did the courseware/course software work on the computer equipment to which you had access for taking web/Internet- based learning? (percent) (percent) (percent) (percent) 2 Q24. How reliable is your network access for taking web/Internet-based learning (e.g. ability to connect; to upload and download  assignments, etc).
Q56. How reliable was your network access for taking web/Internet-based learning (e.g. ability to connect; to upload and download  assignments, etc)? (percent) (percent) (percent) (percent)
Q25. Compared to resident Professional Military Education students in the school in which you are enrolled, of the following options, do  you believe you are prepared: Q57. Compared to resident Professional Military Education program graduates of your school, of the following options, do you believe you  are prepared: as well as  resident  students.  (percent) worse than  resident  students.  (percent) (percent)
Q26. Overall, what has been the primary challenge, if any, affecting your Professional Military Education program?
Q58. Overall, what was the primary challenge, if any, affecting your Professional Military Education program?
Current  Students  (percent) (percent)
Not applicable, I am not experiencing & have not experienced any challenges  to this point in time.
Q27. Have you experienced any computer/Internet-related problems affecting your Professional Military Education program?
Q59. Did you experience any computer/Internet-related problems affecting your Professional Military Education program? (percent) 16 Q28. What specific computer/Internet-related problems have you incurred?
Q60. What specific computer/Internet-related problems have you incurred? (percent) (percent)
Q29. At any point during your Professional Military Education program, have you had to defer/disenroll from your studies?
Q61. At any point during your Professional Military Education program, did you have to defer/disenroll from your studies? (percent)
Q30. What was the primary reason you had to defer/disenroll from your studies?
Q62. What was the primary reason you had to defer/disenroll from your studies?
Q63. What is the highest degree or level of school that you have completed?
Current  Students  (percent)
Q64. In what branch of the military do you serve? (percent)
Q65. What duty capacity best describes you during the majority of your Professional Military Education program? (percent)
Q66. What component best describes you during the majority of your Professional Military Education program? (percent)
Q67. Are you a member of the National Guard? (percent) 55Q68. What military occupational category best describes you during the majority of your Professional Military Education program?
Current  Students  (percent) (percent)
Q69. What was your rank when you began your Professional Military Education program? (percent) (percent) 3 Q71. If you have any other comments related to your PME education, training, assignments, distance learning, or any other matters related  to this questionnaire, please note them here.
We observed three current ADL applications at PME senior- and  intermediate-level schools. These schools have geared their ADL efforts to  their nonresident programs. The programs vary from service to service in  terms of enrollment, structure, duration, and credits received for  graduation. In addition, we observed additional features of nonresident  programs that affect the nature of their ADL applications.
section_title: U.S. Army War College
paragraphs: The U.S. Army War College (USAWC), the Army’s senior-level PME school,  initiated its Web-based nonresident education program in April 1999. The  program went online in an evolutionary process until the spring of 2002,  whereby students received both text and online versions. Since the spring  of 2002, all nonresident students have received their education via a  combination of ADL technology and appropriate text. Nonresident  students are board selected to participate in the program. It is a 2-year  Web-based program that is the only delivery method offered to nonresident  students. The program has a “blended” component, whereby 2 of its  12 courses are taken in residence at USAWC. Also, distance courses are  presented to students as a group or cohort; that is, students enroll at the  beginning of the nonresident school year and must complete a sequenced  load of 5 courses during the first year, followed by an additional 5 courses  during the second year. The resident courses are of 2-week duration and  are conducted at the end of each academic year. The nonresident program  is designed to parallel the resident program, and graduates from both  programs are awarded Master’s Degrees in Strategic Studies.
section_title: Naval War College
paragraphs: The Naval War College’s (NWC) nonresident education programs are  concentrated in its College of Distance Education, its only nonresident  college and one of five colleges under the NWC umbrella. The College of  Distance Education, an intermediate-level PME school, offers several  nonresident options. The fleet seminar program has existed in various  forms at the school since 1974; the Web-enabled correspondence program  has been operating fully since October 2002; and the CD-ROM-based  correspondence program, effective in April 2004, was designed to replace  the phased-out paper-based correspondence course. Nonresident options  are open to all officers and qualified personnel. The Web-based course can  be completed in 18-24 months. While there is no formal resident portion to  this course, students are assigned to cohort teams to facilitate team and  faculty communication. This nonresident course is closely aligned with the  resident course, and graduates are allowed to obtain graduate hour credits.
In the case of several seminars of the fleet seminar program, students can  apply for admission to a program of graduate study leading toward a  Master’s of Arts Degree in National Security and Strategic Studies.
section_title: Air Command and Staff College
paragraphs: The Air Command and Staff College (ACSC), the Air Force’s  intermediate-level PME school, implemented its nonresident program in its  present form in September 1999. There are two methods for completing the  nonresident program: by seminar or by correspondence. The ACSC  nonresident program is open to all officers and qualified personnel. The  most recent version of the program consists of six courses organized into  two semesters. The seminar method, which can take up to 11 months to  complete, is conducted weekly, is typically composed of 3-18 students, and  is led by assigned seminar leaders in order to facilitate group discussions.  The correspondence program, a self-study program delivered in a balanced  manner consisting of paper, CD-ROM, and Web-based delivery, requires up  to 18 months to complete. Students move interchangeably between both  programs, but they must achieve a minimum score of 70 percent on each of  the six examinations and must complete four interactive Web-based  exercises. The nonresident programs are designed to mirror resident  programs, and there are multiple versions in use by ACSC nonresident  students. These programs do not award master’s degrees, but the American  Council of Education recommends up to 21 semester hours of graduate  credit for course completion.
section_title: Joint Forces Staff College
paragraphs: National Defense University’s Joint Forces Staff College (JFSC) is  piloting an Advanced Joint Professional Military Education pilot course  for senior- and intermediate-level reserve officers. Initially launched in  September 2003, it is designed to last 38 weeks. The period consists of  35 weeks of Web-based education and 3 weeks of resident education with  1 week occurring after the first 8 weeks of Web-based education, and the  last 2 weeks at the end of the 38-week period. JFSC, already responsible for  the resident Joint PME Phase II course used to complete the education  process for joint specialty officers, was tasked to develop a Joint PME  course for reserve officers in response to the fiscal year 1999 National  Defense Authorization Act and the Joint Staff Guidance in May 2000. While  there is no joint specialty officer requirement for reserve officers, JFSC  was required to prepare reserve officers for joint duty assignments by  providing a course similar in content to its resident Joint PME course, and  to do so by utilizing current distance learning applications.
section_title: Additional Features of Nonresident PME Programs
paragraphs: There are additional features of PME programs that affect the nature of  their ADL applications. Those features include:  Student Board Selection—Nonresident students are selected to attend  the PME schools either through an annual board selection process or  through open admissions. Only USAWC selects its nonresident students;  the other programs with ADL applications have open-admission  policies.
Joint Professional Military Education—A significant portion of the  PME curriculum involves study of joint service issues along with  service-specific issues. Officers who successfully complete senior- or  intermediate-level PME course work are awarded Joint PME Phase I  credit, which is required for those who wish to serve as joint specialty  officers. All nonresident programs with ADL applications grant Joint  PME Phase I credit.
Service Promotion Impact—PME officials stated that PME program  completion and other forms of higher education are factors used in  consideration for promotion and vary among the services. Generally, the  Air Force requires completion of a corresponding PME level of study  before a candidate is considered for the next promotion level. The Army,  while not as strict as the Air Force, places a high value on PME and  graduate education in promotion decisions. The Navy, placing a higher  premium on operational experience, currently is less inclined to  recognize PME as a credential for promotion.
Learning Objectives Between Programs—PME officials stated that, as  outlined by Joint Staff policies, learning objectives for nonresident  courses are required to be the same for resident courses, regardless of  the method of delivery. PME schools have instituted internal control  processes to ensure the achievement of learning objectives for all  programs, irrespective of delivery method. Generally, PME schools  apply similar evaluation systems and criteria to both resident and  nonresident programs.
Levels-of-Learning—PME schools teach to differing achievement levels  across and within the services, and they have designed their curricula  accordingly. School officials refer to these achievement levels as  “levels-of-learning” based on a taxonomy defined in the Joint Staff  policy. (See table 3 for a detailed definition of levels-of-learning  designations.) For the schools with ADL applications, their desired  levels of learning for nonresident programs may or may not be  equivalent to the corresponding resident programs:  USAWC—Synthesis/Analysis (same as for resident program).
NWC—Application (resident program calls for synthesis/analysis).
ACSC—Comprehension (resident program calls for  synthesis/resident).
JFSC (Planned)—Application (same as for resident program).
section_title: Comments from the Department of Defense
subsections: section_title: GAO’s Comment
paragraphs: The following is GAO’s comment on the letter from the Department of  Defense dated July 23, 2004. 1. When we initiated this engagement in February 2003, a key objective  was to determine (1) the assumptions for DOD’s decision to move  officer senior- and intermediate-service schools from 1-year residency  to shorter periods by using ADL and (2) which courses and schools  would be affected. Immediately after fieldwork commenced, however,  DOD informed us that it was no longer actively pursuing that approach.  In April 2003, after consulting with the congressional requester, we  informed our DOD point of contact regarding our pursuit of the  engagement’s remaining objectives.
section_title: GAO’s Mission
paragraphs: The Government Accountability Office, the audit, evaluation and  investigative arm of Congress, exists to support Congress in meeting its  constitutional responsibilities and to help improve the performance and  accountability of the federal government for the American people. GAO  examines the use of public funds; evaluates federal programs and policies;  and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost  is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts  newly released reports, testimony, and correspondence on its Web site. To  have GAO e-mail you a list of newly posted products every afternoon, go to  www.gao.gov and select “Subscribe to Updates.”
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Congressional Relations
section_title: Public Affairs