Military Education: DOD Needs to Enhance Performance Goals and Measures to Improve Oversight of Military Academies
Published: 2003-09-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Graduates of the service academies operated by the Army, Navy, and Air Force currently make up approximately 18 percent of the officer corps for the nation's armed services. The academies represent the military's most expensive source of new officers. The Department of Defense (DOD) pays the full cost of a student's 4-year education at the academies; and the related cost has increased over the past 4 years. Admission to the academies is highly competitive. The academies use a "whole person" method to make admission decisions. Recent studies by the Air Force raised questions about possible adverse effects of whole person admissions policies on student quality. GAO was asked to review all three service academies and specifically address the extent to which (1) DOD oversees the service academies, (2) applicants are granted waivers of academic standards, and (3) various groups of students differ in admissions scores and academy performance.
section_title: What GAO Found
paragraphs: The Office of the Under Secretary of Defense for Personnel and Readiness (OUSD/P&R), the services, and the academies' boards of visitors conduct considerable oversight of the academies' operations and performance, but they lack a complete oversight framework. A complete oversight framework includes performance goals and measures against which the academies' performance could be better assessed. OUSD/P&R and the services use the number and type of commissioned officers as the primary measure of academy performance. OUSD/P&R requires and receives reports on academy performance from the services. While data submitted in these reports provide perspective on current performance compared with past performance, without stated performance goals and measures, these reports do not offer OUSD/P&R or the services as good an insight into the academies performance as they could. Additionally, though the academy boards of visitors serve as an external oversight mechanism to focus attention on a wide range of issues, they also do not assess the academies' performance against established performance goals and measures. The academies do not grant waivers from academic criteria or have absolute minimum scores for admission. However, under the whole person approach, the academies can admit some applicants whose academic scores are lower than might normally be competitive for admission, but who in their totality (academics, physical aptitude, and leadership) are evaluated by academy officials as being capable of succeeding at the academy. In our review of the academy classes that started in 1998 (class of 2002), we found that despite differences among various groups of students in their admissions scores and similar differences in their performance while at the academies, the differences in performance were not sizable. Some groups, such as females, performed better in some categories than the class as a whole and worse in others. Some groups (minorities, preparatory school graduates, recruited athletes, and students in the lower 30 percent of their class in terms of academic admissions scores) performed at lower levels on average in all categories than the class as a whole.

section_title: Letter
paragraphs: Graduates of the service academies operated by the Army, Navy, and Air  Force make up approximately 18 percent of the officer corps for the  nation’s armed services. The academies represent the most expensive  source of new officers, compared with other sources for officers, such as  Reserve Officer Training Corps (ROTC) programs at colleges and  universities or officer candidate/training schools for individuals who  already have college degrees. The Department of Defense (DOD) pays the  full cost of providing the 4-year programs of academic education, military  training, physical conditioning, and pay for each student. In fiscal year  2002, DOD reported costs per graduate for the U.S. Military Academy, the  U.S. Naval Academy, and the U.S. Air Force Academy were approximately  $349,000, $275,000, and $333,000, respectively. These costs have increased  over the past 4 years. To ensure the best value for the investment in the  academies, effective management principles are critical. Such principles  include a complete oversight framework, with clear roles and  responsibilities, as well as performance goals and measures against which  to objectively assess performance.
With each academy accepting about 1,200 of its more than 10,000  applicants a year, admission to the academies is highly competitive.  Applicants must be selected or obtain a nomination, such as from a  senator, representative, the President, or the Vice President, based on the  categories established by law. Most nominations are reserved for  Congress, which, therefore, has a central role in admitting students to the  academies. In addition to basic age and medical qualifications, the  academies’ admissions process involves an assessment of applicants’  academic achievement (e.g., Scholastic Aptitude Test—SAT—scores and  grade point averages), physical aptitude, and extracurricular activities  (i.e., leadership potential). Academy officials combine these assessments  into a “whole person” admissions score that is used to determine an  applicant’s potential to graduate from an academy and potential fitness as  a commissioned officer. Applicants compete for admission based on these  scores.
Air Force studies have raised questions about possible adverse effects of  whole person admissions policies on student quality. For example, the Air  Force found that its whole person assessments and resulting admissions  scores have led the Air Force Academy to admit an increasing number of  students whose academic qualifications are below academic minimums, as  well as to admit an increasing number of students recruited largely to  participate in varsity intercollegiate athletics.
The House of Representatives report on defense appropriations for fiscal  year 2003 directed that we perform reviews of all three service academies  and their respective preparatory schools. As part of the review of the  service academies, we were also directed to obtain student and faculty  perceptions of various aspects of student life at the academies. Issues  associated with the academy preparatory schools and the results of  surveys on aspects of student life are addressed in separate reports.
As agreed with your offices, this report addresses the following questions,  to what extent  (1) does DOD oversee the academies’ operations and performance?  (2) are applicants granted waivers from academic criteria for admissions?  (3) do various groups of students differ in admissions scores and academy  performance?
In addition to reviewing documents and interviewing officials at all three  academies, the service headquarters, the Office of the Under Secretary of  Defense for Personnel and Readiness (OUSD/P&R), and the academies’  boards of visitors, we reviewed admissions policies and procedures and  observed their use by academy officials in evaluating applications being  considered for the incoming class of 2007. We also obtained and analyzed  admissions and performance data for the student class that graduated in  2002. To compare student admissions qualifications and performance at  the academies, we identified six major groups of students common to all  academies: females, minorities, academy preparatory school graduates,  recruited athletes, prior enlisted personnel, and students whose academic  admission scores fell in the lower 30 percent of the entering class. Data on  student performance included academic grade point average; military  performance average, which is similar to a performance evaluation for  commissioned officers; and class rank. It also included graduation rate.  Other issues, such as recent controversies associated with alleged sexual  assault, did not fall within the scope of this review. Further details on our  scope and methodology are in appendix I.  We conducted our work from  October 2002 through May 2003 in accordance with generally accepted  government auditing standards.
subsections: section_title: Background
paragraphs: The Departments of the Army, the Navy, and the Air Force each have their  own educational institutions (academies) to produce a portion of each  branch’s officer corps:    U.S. Military Academy (West Point, N.Y.), established in 1802;    U.S. Naval Academy (Annapolis, Md.), established in 1845; and    U.S. Air Force Academy (Colorado Springs, Colo.), established in 1954.
The academies are structured to provide a curriculum critical to the  development of successful future officers in academic, military, and  physical areas of achievement. Additionally, the academies emphasize the  moral and ethical development of students through their respective honor  codes and concepts.
There are approximately 4,000 students enrolled at each of the three  service academies at any given time, each comprising four classes. In  December 2002, Congress authorized an annual increase of up to 100  students until the total number reaches 4,400 for each academy. In 2002  the Military Academy graduated 968 students; the Naval Academy 977  students; and the Air Force Academy 894 students. Faculty at the U.S.  Military Academy and the U.S. Air Force Academy are comprised  predominantly of military officers (79 and 75 percent, respectively), while  at the U.S. Naval Academy 59 percent of the faculty are civilians. Table 1  shows the composition of the faculty at the service academies.
DOD reports that the total cost to operate all three academies in fiscal  year 2002 was $990.7 million. Table 2 shows the reported operating costs  and cost per graduate for each academy from fiscal year 1999 through  fiscal year 2002. We did not independently verify these costs.
Prospective students must meet basic eligibility requirements for  appointment to an academy. They must (1) be unmarried, (2) be a U.S.  citizen, (3) be at least 17 years of age and must not have passed their  twenty-third birthday on July 1 of the year they enter an academy, (4) have  no dependents, and (5) be of good moral character.
After determining eligibility, a candidate submits an application to a  preferred academy or academies. Each submitted application is required  to include information such as, but not limited to, the candidate’s (1) SAT  scores (or American College Testing—ACT—examination scores); (2) high  school grade point average (and class rank, if possible); (3) physical  aptitude scores; (4) medical examination results; and (5) extracurricular  activities. The academies admit those candidates that have secured a  nomination and who represent, in the opinion of academy officials, the  best mixture of attributes (academic, physical, and leadership) necessary  to ensure success at the academies and as military officers.
The military academies use a “whole person” method to assess potential  candidates in three major areas: (1) academics, (2) physical aptitude, and  (3) leadership potential. Each academy uses the same basic approach.  Admissions assessments are weighted toward academic scores that  include objective tests and high school performance. Leadership potential  is measured by assessing athletic and non-athletic extracurricular  activities. Subjective assessments of potential candidates in these major  areas also contribute to final admissions “scores.” Such assessments  include interviews with prospective candidates, teacher/coach  evaluations, and analyses of writing samples. Though medical criteria  differ between services, the medical examinations are conducted  according to the same standards, under a joint DOD Medical Examination  Review Board that manages the medical examination process and records  for applicants to all academies.
Each academy is authorized to permit up to 60 foreign students to attend  at any given time on a reimbursable basis by their country of origin. This  number does not count against the authorized student strength of the  academies. The admission of foreign students is covered by separate  policies and procedures. Our review was limited to the policies and  procedures for admitting U.S. citizens to the academies. Figure 1 shows  the basic steps in the admissions process for all U.S. applicants.
Students who are disenrolled from an academy after the start of their third  year may be required to complete a period of active duty enlisted service  of up to 4 years or may be required to reimburse the federal government  for the cost of their education. Those who are disenrolled in their first 2  years do not incur an active service or reimbursement obligation.
subsections: section_title: United States Military Academy Admissions Process
paragraphs: The United States Military Academy’s admissions evaluation considers  academics, leadership, and physical aptitude. Academic considerations  include above-average high school or college academic records as well as  strong performance on SAT/ACT. Additionally, the Military Academy  considers recommendations from English, mathematics, and science  teachers. The leadership potential considers demonstrations of leadership  and initiative in sports, school, community, or church activities and strong  recommendations from faculty and community leadership and is a more  subjective assessment of character. Physical aptitude is based on a scored  standardized test. This test is made up of pull-ups for men or the flexed- arm hang for women, push-ups, standing long jump, basketball throw, and  shuttle run. Figure 2 shows the areas considered and the weights assigned  to each area in the U.S. Military Academy’s whole person admissions  process.
section_title: United States Naval Academy Admissions Process
paragraphs: The United States Naval Academy’s admissions evaluation considers  academics, leadership, physical aptitude, and technical interest. Academic  considerations include above-average high school or college academic  records as well as strong performance on SAT/ACT. Additionally, the  Naval Academy considers recommendations from English and  mathematics teachers. Assessment of leadership potential represents a  subjective evaluation of character in which the academy considers  demonstrations of leadership in terms of extracurricular activities in  sports, school, community, or church and strong recommendations from  faculty and community leadership. Physical aptitude is based on a scored,  standardized test consisting of pull-ups for men or the flexed-arm hang for  women, push-ups, standing long jump, basketball throw, and shuttle run.  Additionally, the Naval Academy considers the technical interest of a  prospective student, which is measured through a questionnaire in the  application packet and used to gauge interest in pursuing a technical  degree. The intent of this requirement is to admit students that are  interested in pursuing technical degrees, specifically nuclear and maritime  engineering. The admissions board can also apply further points to an  applicant’s overall whole person score based on further consideration of  an applicant’s record, including such things as the results of the evaluation  form filled out by the Naval Academy representative who interviewed the  applicant. Figure 3 shows the areas considered and the weights assigned  to each area in the U.S. Naval Academy’s whole person admissions  process.
section_title: United States Air Force Academy Admissions Process
paragraphs: The United States Air Force Academy’s admissions evaluation considers  academics, leadership, and an assessment by the selections panel.  Academic considerations include above-average high school or college  academic records as well as strong performance on SAT/ACT.  Additionally, the Air Force Academy considers recommendations from  English and mathematics teachers. Under leadership potential, the  academy considers extracurricular activities in sports, school, community,  or church and strong recommendations from faculty and community  leadership. Finally, the Air Force Academy Selections Panel makes an  assessment of all potential students. This assessment is composed of a  pass/fail score from the physical aptitude examination and the evaluation  of the academy’s liaison officer evaluation, made after interviewing the  applicant. The physical aptitude examination is made up of pull-ups for  men or the flexed-arm hang for women, push-ups, standing long jump,  basketball throw, and shuttle run. The leadership potential area and the  admissions board include the more subjective assessments of a potential  student. Figure 4 shows the areas considered and the weights assigned to  each area in the U.S. Air Force Academy’s whole person admissions  process.
section_title: Nomination and Appointment of Candidates
paragraphs: The President of the United States alone appoints candidates to the  academies. Before receiving an appointment, all candidates must secure  one or more nominations according to the following categories:    congressional (including a U.S. senator, representative, delegate, or the    service-connected (including, among others, children of disabled  veterans, enlisted personnel in the active or reserve components, and  students from ROTC programs or other designated honor school  graduates); and    other (including the academy superintendents’ nominees and other  nominees to bring the incoming class to full strength).
Figure 5 shows the approximate distribution of categories of academy  nominations, based on the types and numbers of nominees per category  allowed by law.
section_title: Oversight of the Academies
paragraphs: Oversight of the academies is the responsibility of three principal  organizations: OUSD/P&R, the service headquarters, and the board of  visitors of each academy. According to Department of Defense Directive  1322.22 (Service Academies), OUSD/P&R serves as the DOD focal point  for matters affecting the academies and has responsibility to assess  academy operations and establish policy and guidance for uniform  oversight and management of the military academies. The military  departments perform the primary DOD oversight function for their  respective academies. The superintendent of each academy reports  directly to the uniformed head of his respective service (the Chiefs of Staff  for the Army and the Air Force and the Chief of Naval Operations for the  Navy), in accordance with the chain of command for each service. Each  academy also has a board of visitors, mandated by law, that is comprised  of congressional members and presidential appointees. These boards  focus attention and action on a wide range of operational and quality of  life issues at the academies.
As educational institutions, the service academies are also overseen by  several nongovernmental organizations that are outside DOD purview.  Each academy undergoes periodic review by a higher-education  accreditation body associated with its region of the country, usually  involving a full review every 10 years with an interim review every 5 years.  The accreditation bodies review such areas as core curriculum, strategic  planning, self-assessments, diversity of faculty and students, and faculty  credentials. The athletic programs of the academies are also subject to  periodic certification by the National Collegiate Athletic Association. This  body reviews academy athletics in terms of such issues as finances and  impact on the education mission of the academies. We limited our review  of oversight of the academies to DOD organizations and the boards of  visitors.
section_title: DOD Lacks a Complete Framework for Oversight of the Academies
paragraphs: The OUSD/P&R, the services, and the academies’ boards of visitors  conduct many oversight activities, but they lack a complete oversight  framework. A complete oversight framework includes not only clear roles  and responsibilities, but also performance goals and measures against  which to objectively assess performance. Such elements embody the  principles of effective management in which achievements are tracked in  comparison with plans, goals, and objectives and the differences between  actual performance and planned results are analyzed. Without formal  goals and measures, oversight bodies do not have sufficient focus for their  efforts and cannot systematically assess an organization’s strengths and  weaknesses nor identify appropriate remedies that would permit DOD to  achieve the best value for the investment in the academies. In a prior  report, GAO concluded that better external oversight of the academies  was needed to provide useful guidance and suggestions for improvement.  The report recommended that DOD improve oversight of the academies  through such measures as establishing a focal point for monitoring  academy issues in the Office of the Secretary of Defense and establishing  guidance on uniform cost reporting.
OUSD/P&R and the services have established clear roles and  responsibilities for oversight of the academies, with the former serving as  the focal point for issues affecting all academies and the latter having  direct oversight authority over their respective academies. DOD  established guidance in 1994 for the oversight of the academies and for  uniform reporting of costs and resources. OUSD/P&R is directly involved  in those policy issues that affect all academies and require DOD-level  attention and legislative matters. For example, the office was recently the  DOD focal point on the issue of increasing authorized enrollment at the  academies from 4,000 to 4,400. With respect to the academies, the office is  chiefly concerned with monitoring the degree to which the services are  meeting their goals for the accession of new officers. The office also  coordinates major studies that affect the academies, such as a November  1999 report on the career progression of minority and women officers.
The services are responsible for direct oversight of their respective  academies; and the academies are treated similarly to major military  commands. The superintendents of the academies are general/flag officers  who report directly to the uniformed heads of their services (the Chiefs of  Staff for the Army and the Air Force and the Chief of Naval Operations for  the Navy). In addition to overseeing the academies’ budget through the  same approval process as a major command activity, the services oversee  the academies’ operations and performance primarily through the  academies’ goal of meeting service officer accession targets. The  superintendents are responsible for meeting those targets and, in so doing,  are given wide discretion in such areas as modifying their specific  admissions objectives and the process for matching graduates with service  assignments. The service headquarters use a number of mechanisms to  oversee academy performance. For example, each service headquarters  provides officer accession targets to the academies so that the assignment  of graduates and the make up of incoming student classes can be modified  as necessary. In addition to general numbers of officers, each service also  has a number of specialty officer fields that need to be filled, and the  services also monitor the extent to which the academies will be able to  meet those accession goals.
The services also directly oversee the academies by requiring the  superintendents to report on and discuss their operations. For example,  the Air Force uses an annual forum of the most senior Air Force officers to  focus on the Air Force Academy with respect to how it is meeting the  needs of the operational Air Force. The Navy uses similar senior officer  conferences and frequent interaction between the superintendent and  Navy headquarters to conduct oversight. The Army uses the U.S. Military  Academy Forum, comprised of senior Army officers, to address academy  operations issues. The superintendents of the three academies also hold  annual meetings to discuss issues common to all academies. These  mechanisms have resulted in such academy actions as curriculum changes  to increase the number of technical degree majors, increasing language  requirements, and increasing the number of students attending the  academies.
While OUSD/P&R and the services conduct a wide variety of oversight  activity, there are few stated performance goals against which to measure  academy operations and performance. Each of the academies has a  strategic plan that is focused on providing quality military and professional  training and education in order to commission highly capable junior  officers. These plans are approved by the service headquarters but are not  generally used by the services as benchmarks against which to measure  academy performance, and they do not contain specific goals against  which to measure student performance. OUSD/P&R is required to assess  and monitor academy operations based on the information provided in  annual reports it requires from the service secretaries. These reports  provide data on various aspects of performance, such as student  demographics and trends, student quality, admissions and attrition trends,  compensation for students and faculty, leadership and honor systems, and  incidents of indiscipline.
The reports provide OUSD/P&R and the services with information on  current and past performance for academy operations, but apart from  officer accession goals, neither OUSD/P&R nor the services have specific  stated performance goals against which to compare the information  provided in the assessment reports, thus they do not have an explicit basis  for judging the adequacy of their performance. For example, the data  collected by the academies show that graduation rates have increased in  the last 10 years; however, there is no stated goal for a graduation rate  against which to judge whether this rate of increase is adequate. Other  data collected by the academies indicate that the percentage of females  and minorities has fluctuated over the last 3 years, but apart from  admissions targets used by the U.S. Military Academy, there are no stated  goals against which to assess these trends. Additionally, academy officials  regularly analyze data on student body performance to determine the  extent to which admissions standards can be changed to affect student  body performance. However, there are no stated goals for student body  performance, apart from minimum graduation standards, that might help  the academies and other oversight bodies assess overall student  performance.
The oversight efforts of each academy’s board of visitors are similarly  limited by the absence of sufficient performance goals and measures. Each  of the academies has a board of visitors, mandated by law and comprised  of Members of Congress and presidential appointees, that is outside the  DOD chain of command. The boards have a broad legal mandate to inquire  into all aspects of academy operations. The boards meet several times a  year to be briefed on and discuss academy operations and must conduct  an annual visit to their respective academies. During these visits, the  boards are briefed by academy staff on such issues as admissions,  curriculum, recruiting, athletics, morale and welfare, and construction  programs; they also interview students to obtain their perceptions of life at  the academies. The boards also address inquiries to academy staff, which  are usually followed up at subsequent meetings, and they make  suggestions to improve operations or quality of life at the academies. For  example, boards of visitors have recommended increased recruiting of  qualified minority applicants from various congressional districts and  increased surveying of students on quality of life issues.
The boards submit annual reports to the President on the status of and  issues at the academies but do not evaluate academy operations and  performance against established performance goals. The boards of visitors  do not have dedicated staffs to conduct their work, and though board  members may inquire into any aspect of academy operations, the agenda is  set largely by the briefings presented to the boards by academy officials.  Academy officials with whom we spoke were generally satisfied with the  oversight provided by the boards of visitors, though there were concerns  at the Air Force Academy about poor attendance by board members  during annual visits to the academy.
section_title: Whole Person Approach Allows Academies Flexibility to Admit Students with a Range of Qualifications
paragraphs: The academies do not grant waivers from academic criteria but do not  have absolute minimum scores for admission. Under the whole person  approach, the academies can admit some applicants whose academic  scores are lower than might normally be competitive for admission, but  who in their totality (academics, physical aptitude, and leadership  potential) are deemed an acceptable risk and qualified to attend an  academy. This admissions approach is consistent with the intent of the  academies to admit students who also demonstrate leadership and  initiative characteristics, which cannot be quantified by purely objective  scoring methods.
When conducting their admissions processes, the academies do not set  absolute minimum scores for academic ability. Rather, they establish a  range of scores that would be considered competitive, based on past  incoming class performance and academy research on the overall quality  of the applicant pool. Prior to 2002, the Air Force Academy set absolute  minimum academic scores, and a waiver was required to further consider  an applicant who fell below that minimum, no matter how high his or her  scores in the leadership area. However, the Air Force Academy no longer  has absolute minimums and uses the same competitive range approach as  the other academies. Under this approach, if an applicant’s academic score  is lower than the competitive range guidelines, academy officials have  some flexibility to further consider the applicant. Academy officials will  re-examine the applicant’s record for information that might provide  further insight about his or her academic achievement. For example,  officials may contact high school teachers to inquire about the types and  difficulty of the classes the applicant has been taking and his or her  performance in those classes. Academy officials will also weigh the extent  to which the leadership component of the applicant’s whole person score  offset the low component. The applicant is considered a risk and is  evaluated through a deliberative process by academy officials on the basis  of their judgment of whether the applicant is fully qualified and capable of  succeeding at that academy. The subjective nature of this approach is  consistent with the intent of the whole person concept, by which the  academies want to admit students who also demonstrate leadership  characteristics that cannot be quantified by purely objective scoring  methods. Academy officials do not consider these judgments to constitute  a waiver of academic standards, but rather a judicious assessment of the  whole person. The process for assessing those applicants whose academic  scores are lower than might normally be competitive is nonetheless similar  to the former Air Force Academy process for granting waivers.
With over 10,000 applicants for each academy each year and about 1,200  students admitted, the academic standards are high. Academy data show  that the academic quality of the applicants has remained high over the past  4 years, and the competitive ranges for academic scores used by the  academies have remained the same or have increased during this time.  However, it is possible for students to be admitted whose academic scores  were not as competitive as some other applicants who may not have been  admitted. Senators, representatives, and delegates may submit up to 10  nominees for each student vacancy available to him or her per academy.  They may choose to designate one as a principal nominee. If an applicant  receives a principal nomination and is in all other respects qualified, the  academies must admit that applicant, even over an applicant on the same  senator’s, delegate’s, or representative’s nomination list with higher  academic and/or whole person scores. The other nominated names  become alternates for possible admission later in the admissions process.
Though some academies award credit for the extent to which an applicant  surpasses the standards of the physical aptitude examination, there are  minimum standards for the physical test that must be met. None of the  academies uses a system of “waivers,” except for medical conditions. An  applicant can be waived for a medical condition, based on the deliberation  and judgment of DOD medical personnel and the academy superintendent.
For example, an applicant who is disqualified due to a vision condition  may apply for and receive a waiver, based on subsequent surgical vision  correction or determination by the academy superintendent that the  applicant would be able to serve on active duty without the vision  condition being a problem.
section_title: No Significant Differences in Admissions and Academy Performance between Various Groups of Students
paragraphs: In our review of the academy classes that started in 1998 (class of 2002),  we found differences among various groups of students in their  admissions scores and similar differences in their performance while at  the academies, but the differences were not significant in magnitude. In  terms of performance after admission to the academies, differences  between these student groups and the class as a whole were also not  sizable. We reviewed data for the following distinct groups:    minorities,    academy preparatory school graduates,  recruited athletes,  lower 30 percent of class by academic admissions scores.
For the class data we reviewed, minorities, academy preparatory school  graduates, recruited athletes, and prior enlisted students all had lower  average admissions scores than the average for the class as a whole,  though these differences varied. The differences between groups and the  class as a whole were not sizable, generally falling within 5 percent. Those  differences that were statistically significant and outside the 5 percent  range were still generally less than 10 percent of the class as a whole.  Tables 3, 4, and 5, show the average admissions scores for the selected  groups in the class that started in 1998 at the Military, Naval, and Air Force  Academies, respectively. Although each academy uses the same  fundamental whole person approach, they use different scales to calculate  scores. Therefore, the academic and whole person scores cannot be  compared across academies.
Of those students in the lower 30 percent of the class in terms of academic  admissions scores, about 44 percent were recruited athletes, between 25  and 31 percent were minorities, and between 20 and 34 percent were  preparatory school graduates. Table 6 shows the percentage of the  selected groups making up the lower 30 percent of the classes in terms of  their academic admissions scores, by academy.
We also found differences in performance after admission to the  academies between selected groups and the class as a whole. For  example, females at the Naval Academy had a lower graduation rate than  the class as a whole, but they had a higher average academic grade point  average (cumulative GPA) than the class as a whole and higher average  class rank (order of merit). The differences in performance between the  selected groups and the class as a whole were not sizable, generally falling  within 5 percent. Those differences that were statistically significant and  outside the 5 percent range were still generally less than 10 percent of the  class as a whole. Tables 7, 8, and 9 show how the selected groups  performed at the Military, Naval, and Air Force Academies, respectively.  See appendix II for further information on comparisons of performance by  defined student groups.
Some groups—such as minorities, preparatory school graduates, recruited  athletes, and students in the lower 30 percent of their class in terms of  academic admissions scores—performed at lower levels on average in all  categories than the class as a whole, but these differences varied between  academies and by category and were not sizable. For example, one of the  lowest average academic grade point averages for the groups we reviewed  was 2.61 and the average for the class as a whole at that academy was 2.93.  A 2.0 grade point average is required to graduate for academic and military  averages. Similarly, the lowest graduation rate for the class we reviewed  was 65 percent for the students in the lower 30 percent of their class in  terms of academic admissions scores at one academy. The average  graduation rate for the class as a whole was 74 percent.
Our analysis of data for the students who entered the academies in 1998  (class of 2002) indicates that admissions scores are generally good  predictors of performance at the academies. Of the admissions scores, the  academic component of the whole person scores was often the best  predictor of academic performance at the academies, and the whole  person scores in their entirety were often the best predictors of military  performance at the academies. Both academic and whole person  admissions scores were good predictors of class rank. In general, whole  person admissions scores were better predictors of graduation rate than  the academic admissions scores alone.
section_title: Conclusion
paragraphs: Although the service academies receive oversight from a number of  organizations and have established guidance for that oversight that  includes the reporting of a wide range of data on academy operations,  without clear and agreed-upon performance goals, there is no objective  yardstick against which to fully measure academy performance and  operations, apart from the officer accessions goals currently used.  Establishment of such performance goals is consistent with the principles  of effective management and would enhance the quality of oversight  already performed by OUSD/P&R, the services, and the academy boards of  visitors, permitting them to more clearly note those areas in which the  academies excel, highlight areas where improvement is warranted, and  achieve the best value for the nation’s investment in the academies.
section_title: Recommendation for Executive Action
paragraphs: To improve DOD oversight of the operations and performance of the  service academies, we recommend that the Secretary of Defense direct the  OUSD/P&R, in concert with the services, to further enhance performance  goals and measures whereby the information required in annual  assessment reports can be better evaluated. These performance goals  should be developed for each academy and, where appropriate, in  common for all academies. The specific goals should coincide with  performance elements agreed upon by the services and OUSD/P&R and  might include such things as graduation rates, demographic composition  of student classes, assessments of officer performance after graduation,  and other performance information already collected by the academies,  including performance characteristics of various groups of students.
section_title: Agency Comments
paragraphs: In comments on a draft of this report, DOD agreed with our  recommendation to further enhance performance goals and measures for  the service academies whereby the information required in annual  assessment reports can be better evaluated.  DOD further stated that the  Office of the Under Secretary of Defense for Personnel and Readiness  OUSD/P&R will (1) monitor development of improved goals and measures  by the service academies, to include facilitating the development of  common performance goals where appropriate and (2) update DOD   Directive 1322.22, Service Academies, as required.  DOD’s written  comments are included in their entirety in appendix III.
We are sending copies of this report to the appropriate congressional  committees; the Secretaries of Defense, the Army, the Navy, and the Air  Force; and the Director, Office of Management and Budget. We will also  make copies available to others upon request. In addition, the report will  be available at no charge on the GAO Web site at http://www.gao.gov.
Please contact me on (202) 512-5559 if you or your staff have any  questions concerning this report. Key contributors are listed in  appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess the extent to which DOD oversees the service academies’  operations and performance, we interviewed officials at the Office of the  Under Secretary of Defense for Personnel and Readiness; the Army, Navy,  and Air Force headquarters; and the U.S. Military, U.S. Naval, and U.S. Air  Force Academies. We reviewed documents on service and DOD oversight  criteria and structures, reporting mechanisms, academy strategic plans,  academy annual reports on operations and performance, boards of  visitors’ minutes and reports, and superintendents conference reports. We  also attended a U.S. Naval Academy Board of Visitors meeting at the Naval  Academy in December 2002 and a U.S. Military Academy Board of Visitors  meeting in Washington, D.C., in March 2003. Additionally, we reviewed  criteria on the principles of effective management, such as those found in  Standards for Internal Control in the Federal Government.
To assess the extent to which academy applicants are granted waivers  from academic admissions criteria, we interviewed officials from the  Military, Naval, and Air Force Academies and reviewed documents on  admissions policies, standards, and practices. We discussed with academy  officials their execution of the whole person approach, including how they  assess applicants’ records, the weights applied to the various components  of the whole person score (academic, leadership, and physical aptitude),  and the justification for points given to various aspects of an applicant’s  scores. We also reviewed data from each academy on trends in academic  admissions scores. During site visits to each academy, we observed the  evaluation of applicant packages for the incoming class of 2007 by  academy officials, including how the whole person approach was applied  for admissions scores. We also observed meetings of senior officials at  each academy where applicants’ records were evaluated and final  admissions decisions were made.
To assess the extent to which admissions and academy performance  scores differ between various groups of students, we analyzed admissions  scores and academy performance scores for all students who started at  the three academies in 1998 and should have graduated in 2002. This  represented the most recent group of students for which complete data  were available. We requested and received from each academy a database  that included data on both admission scores and information about  students’ performance while attending the academy. We did not  independently assess data reliability, but we obtained assurances about  data completeness, accuracy, and reliability from academy officials  responsible for maintaining data at each academy. We analyzed these data  separately for each academy since each academy calculated admission  scores or performance scores somewhat differently. We identified six  major groups of students common to all academies: females, minorities,  academy preparatory school graduates, recruited athletes, prior enlisted  personnel, and students whose academic admission scores fell in the  lower 30 percent of the entering class (we chose the latter group in order  to capture information on students whose academic admissions scores  may have been lower than might normally be competitive). Information  specifying a student’s membership in each of these groups was provided in  the databases from the academies. To assess differences, we first  compared the mean performance scores for each group to the overall  mean for each performance measure for the entire class. See appendix II  for details on the results of our analysis of the relationships between  admissions and performance scores.
In addition, we assessed the relationship between admissions scores and  performance at the academies by using the whole person admission score  and the academic component of the admissions score. We estimated the  effects of those scores on four measures of performance for students at  the academies: (1) cumulative grade point average (GPA), (2) cumulative  military performance average (MPA), (3) order of merit (class standing),  and (4) graduation rate. We used cumulative GPA upon graduation as an  indicator of academic performance at the academies and military  performance averages upon graduation as an indicator of military  performance at the academies. Order of merit is a measure of class  standing at each academy that combines academic and military grade  performance and is a final rank for each graduating student. At both the  Air Force Academy and the Naval Academy, order of merit is an actual  class rank number. At the Military Academy, however, order of merit  could range between 0 and 4.0 and was given on the same scale as grade  point averages. For each academy, we analyzed the association of both the  academic component scores and whole person admission scores with  each of the performance scores using regression models. Relationships  between the admissions scores and cumulative GPA, cumulative MPA, and  order of merit were estimated using linear regression models. The  relationships between these two admissions scores and the likelihood of  graduating were estimated using logistic regression models. See appendix  II for more details on the results of those analyses.
Issues related to alleged sexual assaults at the academies fell outside the  scope of our objectives. We conducted our work from October 2002  through May 2003 in accordance with generally accepted government  auditing standards.
section_title: Appendix II: Results of Statistical Analysis of Class of 2002 Admissions and Academy Performance Scores
paragraphs: This appendix provides the results of our analyses of both admissions and  performance scores for the class of 2002 at the U.S. Military Academy, the  U.S. Naval Academy, and the U.S. Air Force Academy.
subsections: section_title: Admissions and Performance Scores
paragraphs: We obtained data from all three service academies that included  information on admissions scores (academic and whole person),  performance scores while at the academy (cumulative academic grade  point average, military performance average, and order of merit), attrition  information where applicable, and various demographic characteristics for  all students entering each academy in 1998. Table 10 shows the minimum,  maximum and average admissions and performance scores for students at  each academy. Table 11 shows graduation rates at each academy.
Next, we compared the average admissions scores, performance scores,  and graduation rates of the six student groups to these overall scores and  rates. Tables 12, 13, and 14 show the average admission scores and the  four measures of student performance for the overall sample, and for the  six student groups, for each of the academies. Because we have data for  the population of students in this class and there is no sampling error, the  standard error of these estimates are small and differences that could be  considered small in magnitude may in fact be statistically significant. In  the tables below, differences that are statistically significant (p<.05) and  exceed 5 percent are considered meaningful and noted, though such  differences may not be practically significant when compared with class  performance requirements overall. For example, at the Naval Academy the  overall average academic admissions score is 618, 5 percent of 618 is  about 31. Only those group average academic admissions scores that are  statistically significant and more than 31 points below 618 are noted with  an “a.” Differences that are greater than 10 percent are marked with a “b.”
section_title: Relationships between Admissions and Performance Scores
paragraphs: Regression models were used to assess the relationship between  admission scores and performance at the three academies. We used linear  regression models to examine relationships between admission scores and  GPA, MPA, and order of merit. To examine the relationship between  admission scores and the likelihood of graduating we used a logistic  regression model. Both the academic admission score and the whole  person score were included as independent variables in each model. We  estimated separate regression models for each academy. The results of  these regressions are shown in tables 15 and 16.
The tables show both regression coefficients and standardized  coefficients. In general, regression coefficients are interpreted as the  predicted change in the dependent variable for every unit change in the  independent variables. Here, we have scaled the admissions scores so that  the regression coefficients in the table can be interpreted as the predicted  change in the relevant measure of success for every 100-point increase in  the academic or “whole person” admission score. For example, overall at  the U.S. Air Force Academy, for every 100-point increase in the academic  admission score we expect to see a 0.06 increase in GPA. For every 100- point increase in the “whole person” score, we expect to see a 0.18  increase in GPA. Both relationships are statistically significant, meaning  that both the academic score and the “whole person” score are significant  predictors of cumulative GPA at the academy.
We cannot compare the size of these coefficients across the three  academies, though, because the academic and “whole person” scores are  on different scales. Because the size of the unstandardized regression  coefficients is affected by the scale of the independent variables (the  admissions scores), we use standardized regression coefficients to  compare them. These appear in parentheses in the tables. To estimate  these coefficients, all of the coefficients are standardized by dividing the  regression coefficient by the ratio of the standard deviation of the success  measure to standard deviation of the admission score. The standardized  regression coefficients, therefore, represent the change in the measure of  success for each change of one standard deviation in admission scores.  Using standardized coefficients, one can conclude that the coefficient that  is larger in magnitude has a greater effect on the measures of success.  Using the same U.S. Air Force Academy example, we see that while the  relationships between both academic and “whole person” scores and GPA  are significant, the relationship between academic scores and GPA is  actually a stronger one than the relationship between the “whole person”  score and GPA. Overall, while the academic scores are often a better  predictor of academic performance at the academies (GPA), the “whole  person” scores are often better predictors of military performance (MPA).  The academic admissions scores have no effect on MPA at the Military and  Air Force Academies and the whole person scores, not the academic  admissions scores, predict likelihood of graduating at all three academies.
We also used the R statistic to estimate how much of the variation in each  performance score can be explained by both academic and whole person  admission scores. The admission scores explained about 30 percent of the  variation in GPAs at both the Naval and Air Force Academies and about 40  percent of the variation in GPAs at the Military Academy. The admission  scores explained between a quarter and a third of the variation in order of  merit across the three academies. However, admission scores did not  explain as much of the variation in either military performance scores or  graduation rates. Therefore, while both types of admission scores are  significant predictors of performance at the academy, they only explain  between 7 and 40 percent of the variation in performance at the  academies, and only a very small percentage of the variability in the  likelihood of graduating. Other factors not studied here, such as the  military training and academic environment students experience at the  academies, may contribute to performance more than just students’  admissions scores do.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Gabrielle M. Anderson, Herbert  I. Dunn, Brian G. Hackett, Joseph W. Kirschbaum, Wendy M. Turenne, and  Susan K. Woodward also made key contributions to this report.
section_title: Related GAO Products
paragraphs: Military Education: DOD Needs to Align Academy Preparatory Schools’  Mission Statements with Overall Guidance and Establish Performance  Goals. GAO-03-1017. Washington, D.C.: September 2003.
Military Education: Student and Faculty Perceptions of Student Life at  the Military Academies. GAO-03-1001. Washington, D.C.: September 2003.
DOD Service Academies: Problems Limit Feasibility of Graduates  Directly Entering the Reserves. GAO/NSIAD-97-89. Washington, D.C.:  March 24, 1997.
DOD Service Academies: Comparison of Honor and Conduct  Adjudicatory Processes. GAO/NSIAD-95-49. Washington, D.C.: April 25,  1995.
DOD Service Academies: Academic Review Processes. GAO/NSIAD-95-57.  Washington, D.C.: April 5, 1995.
DOD Service Academies: Update on Extent of Sexual Harassment.  GAO/NSIAD-95-58. Washington, D.C.: March 31, 1995.
Coast Guard: Cost for the Naval Academy Preparatory School and Profile  of Minority Enrollment. GAO/RCED-94-131. Washington, D.C.: April 12,  1994.
Military Academy: Gender and Racial Disparities. GAO/NSIAD-94-95.  Washington, D.C.: March 17, 1994.
DOD Service Academies: Further Efforts Needed to Eradicate Sexual  Harassment. GAO/T-NSIAD-94-111. Washington, D.C.: February 3, 1994.
DOD Service Academies: More Actions Needed to Eliminate Sexual  Harassment. GAO/NSIAD-94-6. Washington, D.C.: January 31, 1994.
Academy Preparatory Schools. GAO/NSIAD-94-56R. Washington, D.C.:  October 5, 1993.
Air Force Academy: Gender and Racial Disparities. GAO/NSIAD-93-244.  Washington, D.C.: September 24, 1993.
Military Education: Information on Service Academies and Schools.  GAO/NSIAD-93-264BR. Washington, D.C.: September 22, 1993.
Naval Academy: Gender and Racial Disparities. GAO/NSIAD-93-54.  Washington, D.C.: April 30, 1993.
DOD Service Academies: More Changes Needed to Eliminate Hazing.  GAO/NSIAD-93-36. Washington, D.C.: November 16, 1992.
DOD Service Academies: Status Report on Reviews of Student  Treatment. GAO/T-NSIAD-92-41. Washington, D.C.: June 2, 1992.
Service Academies: Historical Proportion of New Officers During  Benchmark Periods. GAO/NSIAD-92-90. Washington, D.C.: March 19, 1992.
DOD Service Academies: Academy Preparatory Schools Need a Clearer  Mission and Better Oversight. GAO/NSIAD-92-57. Washington, D.C.:  March 13, 1992.
Naval Academy: Low Grades in Electrical Engineering Courses Surface  Broader Issues. GAO/NSIAD-91-187. Washington, D.C.: July 22, 1991.
DOD Service Academies: Improved Cost and Performance Monitoring  Needed. GAO/NSIAD-91-79. Washington, D.C.: July 16, 1991.
Review of the Cost and Operations of DOD’s Service Academies. GAO/T- NSIAD-90-28. Washington, D.C.: April 4, 1990.