United States General Accounting Office

GAO Report to Congressional Requesters

March 1998 GENDER ISSUES

Analysis of

Methodologies in

Reports to the

Secretaries of Defense

and the Army

GAO/NSIAD-98-125

GAO United States

General Accounting Office

Washington, D.C. 20548

National Security and

International Affairs Division

B-279460 

March 16, 1998

The Honorable Charles S. Robb

Ranking Minority Member

Subcommittee on Readiness

Committee on Armed Services

United States Senate

The Honorable Ike Skelton

Ranking Minority Member

Committee on National Security

House of Representatives

The Honorable Martin T. Meehan

House of Representatives

Over the past year, several groups have examined gender-related issues

affecting initial entry training in the Department of Defense (DOD). The

studies that have gotten the most attention are the Army’s Senior Review

Panel’s report on sexual harassment, the report of the Federal Advisory

Committee on Gender-Integrated Training and Related Issues, and the

report of the Defense Advisory Committee on Women in the Services.

Because the groups produced somewhat different results, our objectives

were to (1) describe how the groups conducted their work, (2) determine

how well the work supported making conclusions and recommendations,

(3) assess the availability of documentation supporting the report, and

(4) determine the extent to which the final report described the study

methodology and disclosed limitations.

Since each of the three studies used focus groups as either their primary

data gathering method or in concert with other means of collecting

information, we used social science literature on focus groups to assess

their use of that methodology. Focus groups use a small group discussion

format, guided by a moderator, to gather information on a given topic.

Focus groups are useful in providing an understanding of the depth and

the variety of opinions, feelings, or beliefs, but are not useful for

quantifying the extent to which a particular view or opinion is held. For

these reasons, focus groups are a qualitative technique best used when the

aim is to explore an issue or, at the other end of the research process, to

assess reaction to specific proposals. As with any methodology, its value is

dependent upon the extent to which it is appropriate to the objectives of

the inquiry, how well the methodology is conducted, and the way in which

Page 1 GAO/NSIAD-98-125 Gender Issues
B-279460 

the results are used. More information on focus group methodology is

found in appendix I.

Background The reports of sexual misconduct at Aberdeen Proving Grounds led the

Secretary of Defense to establish the Federal Advisory Committee on

Gender-Integrated Training and Related Issues and to ask the Defense

Advisory Committee on Women in the Services to meet with trainees and

trainers. These incidents also prompted the Secretary of the Army to

establish the Senior Review Panel on Sexual Harassment.

In November 1996, the Secretary of the Army established the Senior

Review Panel on Sexual Harassment. The panel’s mission was to make

recommendations to improve the human relations environment in which

soldiers live and work, with the specific goal of eradicating sexual

harassment in the Army. The panel consisted of seven members, including

two retired general officers recalled to active duty, two active duty general

officers, a senior noncommissioned officer, and two DOD civilians. The

Senior Review Panel forwarded its report1

 and recommendations to the

Secretary of the Army in July 1997. It included 40 recommendations, of

which 14 dealt with training and related issues.

In June 1997, the Secretary of Defense established the Federal Advisory

Committee on Gender-Integrated Training and Related Issues. The

Secretary of Defense established the Committee as a result of the sexual

misconduct incidents at Aberdeen Proving Grounds. Former Senator

Nancy Kassebaum Baker chaired a panel of 11 that included civilians,

retired officers, and a retired senior noncommissioned officer. The

Secretary directed the Committee to assess the training programs and

policies of the Army, Navy, Air Force, and Marine Corps and make

recommendations to improve initial entry training. The Committee issued

its report2

 to the Secretary of Defense on December 16, 1997. It made 30

recommendations covering the full cycle from recruitment through basic

and advanced training.

The Defense Advisory Committee on Women in the Services (DACOWITS)

has been advising secretaries of Defense since George Marshall

established the Committee in 1951. DACOWITS, which consists of 30 to 40

civilians, makes recommendations to the Secretary on the roles of women

in the Armed Forces and on quality of life issues affecting readiness. As

1

An Army summary of the findings of this report appears in appendix II.

2

A DOD summary of the report’s key recommendations and findings appears in appendix II.

Page 2 GAO/NSIAD-98-125 Gender Issues
B-279460 

part of its mission, DACOWITS members conduct annual visits to selected

Army, Air Force, Navy, Marine Corps, and Coast Guard installations, both

here and overseas. These visits serve two purposes: (1) to provide the

Secretary of Defense with insight into the thoughts and perceptions of

servicemembers in the fleet and the field and (2) to determine what issues

DACOWITS will concentrate on in the future. In November 1996, the former

Secretary of Defense requested that DACOWITS visit training installations to

meet with trainees and trainers in the training environment. In

February 1997, the current Secretary of Defense endorsed the request.

DACOWITS provided a report3 to the Secretary of Defense summarizing these

visits. In its report, DACOWITS recommended continued visits to training

installations, but made no recommendations on military training.

Results in Brief The three studies have different objectives and were conducted somewhat

differently. We compared the methodologies of the three studies with the

principles of focus group methodologies set forth in social science

literature. We found a wide variance among the three studies. The Army’s

Senior Review Panel’s study most closely followed the methodology for

conducting focus groups recommended by the literature. Specifically, we

found:

• The Army’s Senior Review Panel on Sexual Harassment used four methods

to collect data: individual interviews, focus groups, surveys, and

observations. During its 8 months of work, the panel visited 59

installations worldwide, conducted interviews with 808 military and

civilian Army personnel, ran focus groups with over 8,000 soldiers and

civilians, and surveyed 22,952 individuals. The use of multiple methods of

data gathering, the rigor with which the various methods were conducted,

and the publication of the data in the report provides ample support for

making conclusions and recommendations.

• The Federal Advisory Committee on Gender-Integrated Training and

Related Issues used focus groups as its primary method of data gathering.

Although the FAC conducted over 300 focus groups and individual

interviews, their value for making conclusions and recommendations is

limited because the Committee did not (1) systematically collect the same

information from all groups, (2) document the information generated in

each of the interviews and focus groups, or (3) explain how what was

heard in the interviews and focus groups led to their conclusions and

recommendations. In addition, the length of the focus group sessions, the

3

A summary of the findings excerpted from the report’s executive summary can be found in 

appendix II.

Page 3 GAO/NSIAD-98-125 Gender Issues
B-279460 

number of participants, and the number of questions addressed may not

have provided adequate time for full participation of the respondents on

all issues. Given these limitations, the extent to which the Committee’s

work supports its conclusions and recommendations cannot be

determined.

• The Defense Advisory Committee on Women in the Services also used

focus groups of trainees, trainers, and supervisors in the Army, Air Force,

Navy, Marine Corps, and Coast Guard to identify what issues concerned

women and men at training installations. Members of DACOWITS held focus

group discussions at 12 schools at 9 installations in the United States and

prepared a summary report of the results at each installation. The

DACOWITS Chair used these to prepare a report to the Secretary of Defense

that accurately reflected the opinions and perceptions cited in the

individual installation reports. We noted that the DACOWITS focus groups

were (1) larger than recommended in the literature, (2) were sometimes

not long enough to allow meaningful participation, and (3) were not

recorded or documented on a group-by-group basis. The DACOWITS report

summarized the opinion and perception data obtained from focus groups.

It made no conclusions or recommendations on military training based on

that information.

The Army’s Senior

Review Panel on

Sexual Harassment

The Army’s Senior Review Panel on Sexual Harassment formed four

teams, one to review Army policies and three for data collection. Each

field team consisted, on the average, of six military personnel and one

civilian. The Chair, the Vice-Chair, or the Deputy Assistant Secretary of the

Army (a member of the panel) accompanied each field team during their

visits. Other panel members traveled with the teams as often as possible.

Visits lasted 1 to 4 days depending on the numbers of participants in the

various activities. Before the visits, the participants for the individual

interviews, focus groups, and survey were selected and scheduled.

Generally, the visit started with a briefing to present the purpose of the

activity and a description of the team’s data collection efforts. Next, the

team divided into smaller groups to conduct individual interviews,

conduct focus groups, or administer surveys. These activities ran

concurrently and team members rotated to different activities at different

times. Visits ended with a briefing providing commanders the opportunity

to begin corrective actions.

Panel members and the working group collaborated in writing the panel’s

report. Once a near final draft was generated, the panel members met for a

Page 4 GAO/NSIAD-98-125 Gender Issues
B-279460 

final review and agreement on the content. The panel on Sexual

Harassment issued its report to the Secretary of the Army in July 1997.

Methodology The Army’s Senior Review Panel on Sexual Harassment used four methods

to collect data: individual interviews, focus groups, surveys, and

observations. According to the senior social scientist detailed to the panel,

the field teams conducted interviews and focus groups using carefully

developed protocols to obtain human relations environment information.

Members of field teams conducted individual interviews with 808 military

and civilian Army leaders and personnel in Army support groups. Focus

groups consisted of randomly selected, single-gender groups of 8 to 12

people organized by ranks or categories.4

 Participants totaled 7,401

soldiers and 1,007 civilians. Facilitators and note takers of the same

gender as the groups conducted the sessions. All data obtained through

these two activities were entered into a computer for analysis. The

working group, which consisted of more than 40 military and civilian

personnel, developed main themes or categories and placed the

perceptions data under the categories. Data were then analyzed by rank,

by gender, and by question.

The written surveys addressed leadership, cohesion, and sexual

harassment. Field teams administered the surveys to 22,952

servicemembers. Surveys were developed for trainees, trainers, and the

general Army population. The working group analyzed survey data using a

standard statistical analysis software package. Statistically projectable

results appear in the report by question and, in some cases, by gender.

Observations were made during visits to barracks and other facilities and

to engage in informal conversations with military and civilian personnel,

family members, and others.

The seven panel members, supported by the working group, collected data

at 59 Army installations worldwide selected using a stratified random

sampling design. Stratification was based on the type and location of the

installation. The study took 8 months to complete and obtained

information from over 32,000 Army personnel.

4

Rank or category included, junior enlisted soldiers, junior noncommissioned officers, senior

noncommissioned officers, company grade officers, field grade officers, civilian employees, drill

sergeants, instructors, and trainees.

Page 5 GAO/NSIAD-98-125 Gender Issues
B-279460 

Use of Multiple

Methodologies and

Extensive Documentation

Allowed Making

Conclusions and

Recommendations

The panel’s methodology supported making conclusions and

recommendations. Focus groups were used in conjunction with surveys to

not only confirm the survey data but also provide texture and perspective

to the data. The focus groups were of an appropriate size and were all

asked the same questions, in the same order by trained moderators.

However, the number of questions asked of many of the focus groups was

significantly greater than the five or six questions recommended by focus

group literature. For example, the set of questions for trainee focus groups

consisted of 15 questions and the set for trainers consisted of 13 questions.

Focus group discussions were not tape recorded because it was feared

that this would inhibit the participants, but notes were taken by a note

taker and were content-analyzed. The notes from each focus group session

were destroyed, after the responses were entered in the database and

verified for accuracy, to assure that participant confidentiality was

maintained. Destroying the original documentation to assure

confidentiality is considered an appropriate measure by social scientists.

In addition, the completed survey forms were also destroyed to assure

participant confidentiality.

In volume two, the panel provides an extensive discussion of its

methodology. Volume two provides details on how participants were

selected, copies of the focus group questions, the surveys, and the

interview questions. Results of the surveys were included in the report as

well as the most frequently heard responses in the focus groups.

Furthermore, a sufficient amount of data are presented in volume one of

the report, which outlines the panel’s conclusions and recommendations,

to allow the reader to evaluate them.

Lack of Survey Pretest Results

in Controversy

An area of controversy arose because the survey developers did not

pretest the survey questions. We were told by the senior social scientist

attached to the panel, that tight time frames precluded the panel from

carrying out a pretest of the survey form. Normally, a pretest is performed

to identify problem questions, problems with language interpretations,

unclear instructions, or to determine if there are some questions that

respondents will refuse to answer. In this instance, the survey form

contained six questions that some respondents in early administrations

found inflammatory, offensive, and an invasion of privacy and resulted in

some refusals to complete the survey. Subsequently, those questions were

eliminated and a revised form was used. The data on the six questions

were not included in the database, which resulted in an accusation that the

panel had eliminated important data from its analysis. The report

disclosed the problem and its resolution in the methodology section. We

Page 6 GAO/NSIAD-98-125 Gender Issues
B-279460 

believe that the panel acted responsibly in eliminating the offending

questions to avoid having a negative effect on the survey return rate. The

controversy, however, demonstrates the importance of pretesting survey

forms before conducting a survey.

The Federal Advisory

Committee on

Gender-Integrated

Training and Related

Issues

The Federal Advisory Committee on Gender-Integrated Training and

Related Issues saw its role as listening to the views of trainees, trainers,

supervisors, and service officials and providing the Secretary of Defense

with its best judgment about what should be done to improve training.

Small teams of Committee members visited 17 training installations and

operational units to gather opinions. Most Committee members visited

installations from two services. While the Committee Chairman visited

installations for all four services, no Committee member or Committee

staff member visited all of the installations.

Once at an installation, the Committee members followed the same

general schedule: reveille, breakfast with new servicemembers, meetings

with command officials, and interviews and focus groups before lunch.

After lunch with support personnel, the Committee members conducted

additional focus groups and interviews. At the end of each visit, they met

with command officials to discuss their findings. Installation visits

generally lasted 1 day, although visits to basic training sites were 2-day

trips. The visits to the training installations and operational units occurred

in September and October 1997.

The Committee had two public meetings, the first in July 1997 and the

second in October 1997. At the July meeting, service representatives

provided information on the services’ recruiting and training programs. At

the October 1997 meeting, Committee members discussed their

observations and agreed to a partial list of recommendations for the

report. The Committee’s staff drafted the report based on the discussions

they heard during their installation visits and the public meeting, and

memorandums submitted by some of the Committee members in

preparation for the October meeting. Committee members received the

draft report in early December and revisions were made based on their

comments. The Committee Chair discussed the report with Committee

members in a series of one-on-one telephone calls to arrive at the final

recommendations.

Page 7 GAO/NSIAD-98-125 Gender Issues
B-279460 

Methodology The Committee’s primary means of collecting information involved focus

group discussions. The Committee held 199 focus groups, soliciting

opinions from more than 1,000 trainees, 500 trainers, 300 first-term

servicemembers, and 275 supervisors at U.S. training installations and

operational units. Participants were randomly selected under the

supervision of the installations’ inspectors general. Trainees who

participated in the focus groups were within 2 weeks of completing their

training. Participants in the trainer focus groups were trainers for at least 1

year. First term participants were in their initial assignment and had been

on the job between 6 and 18 months. Generally, the Committee met with

equal numbers of females and males, although, because of the limited

number of female trainers and supervisors, this was not always possible.

Focus groups included about 10 to 15 people each and were

gender-segregated. All of the focus groups were moderated by Committee

members, and generally two Committee members or a Committee member

and a Committee staff member attended each session. The Committee

members worked from a set of questions tailored for each service and

each type of focus group. Although the number of questions varied by type

of focus group, the set of questions for all basic training focus groups

consisted of 20 questions, some of which had multiple parts. While some

focus groups were scheduled to last only 30 to 45 minutes, most focus

group sessions lasted nearly an hour.

Committee members also conducted over 100 interviews with service

officials, including commanding officers, inspectors general, company or

squadron commanders, and senior noncommissioned officers. They also

met with representatives of support groups such as chaplains, equal

opportunity officers, medical officers, and legal officers.

Problems With the

Methodology Limit the

Usefulness of the Report

The value of the information included in the Committee’s report for

making conclusions and recommendations is limited because the

Committee did not follow recommended focus group methodology. The

Committee believed that a more flexible approach to the discussions

would enhance the quality of the exchange between the participants and

the Committee members. However, the fact that the same questions were

not asked of each similar focus group, along with the number of questions,

size of the groups, and length of the sessions may have combined to limit

full discussion. In addition, the focus groups’ discussions were not

systematically recorded. As a result, the extent to which the

Page 8 GAO/NSIAD-98-125 Gender Issues
B-279460 

recommendations are supported by the Committee’s work cannot be

assessed.

The Committee staff provided the Committee members with questions for

the focus groups. However, according to the staff director, the Committee

members were told that the questions were guidelines and that they did

not have to be asked as written. Because the Committee members had the

flexibility to ask any question they desired, the responses should not be

compared with each other. Also, the number of questions provided to the

panel members were far more than the five to six focus group literature

recommends. For example, the staff provided 20 questions, some of which

had several parts, for the Committee to ask Army trainees at Fort Leonard

Wood, Missouri. Fifteen trainees participated in the 1-hour focus groups at

this installation. If the entire hour was spent on the questions, there would

have only been 3 minutes spent on each question and only 15 seconds for

each participant to respond. We do not believe that would have been

enough time for a meaningful discussion of a question. Finally, even if all

the questions were asked as they were written, they were not always

asked in the same order each time. Social science literature suggests that

the same questions asked in a different sequence may result in different

responses.

The absence of documentation of the comments made in the individual

focus groups was the most serious methodological shortcoming. While the

Committee members took notes during each focus group, these notes

were not made part of the Committee’s records nor were they summarized

and included in the report. Without documentation, it is impossible to

determine if the Committee’s work supports its recommendations. Also,

the lack of documentation prevented the Committee from analyzing the

data to know what comments they heard or how often similar comments

were made. Knowing how often a particular kind of comment was made

and the subgroup of the person who made it are ways of putting the

comments in perspective and filtering biases.

The report of the Federal Advisory Committee on Gender-Integrated

Training and Related Issues does not include a sufficient discussion of the

Committee’s methodology and work process. For example, the report

states that the Committee conducted discussion groups with randomly

selected servicemembers, but it does not explain the random selection

process. In addition, the use of terminology such as “randomly selected”

implies a level of scientific rigor that was not achieved in this study. The

report does not identify the make-up of the discussion groups, discuss

Page 9 GAO/NSIAD-98-125 Gender Issues
B-279460 

what type of data analysis was done or not done, or mention any

limitations with the data. Limitations that we believe should have been

mentioned are that the report was based on opinions and the results

cannot be generalized to the entire military training population.

Also, the report often presents opinions in a manner that they can be

misinterpreted as facts based on empirical data. For example, the report

says that the Committee members observed that integrated housing is

contributing to a higher rate of disciplinary problems, but, according to the

Chairman, the Committee did not obtain any data to support this

statement. In addition, the report contains many statements that include

words like “most”, “many”, and “majority”. These words lead a reader to

believe that the Committee counted responses to particular questions or

polled the focus group participants. The Chairman said that the Committee

does not have quantitative data.

The report also does not explain the process the Committee used to

formulate its recommendations. Although the Committee held a public

meeting in October 1997 after its installation visits had been completed,

the recommendations on separate barracks for male and female recruits

and on the organization of gender-segregated platoons, divisions, and

flights were not made until after that meeting. Furthermore, those

recommendations were not discussed by the Committee as a whole, but

rather in a series of calls to individual Committee members.

The Defense Advisory

Committee on Women

in the Services

The mission of the DACOWITS effort was to provide the Secretary of Defense

with an overview of broad issues raised by trainees and trainers of both

genders throughout initial entry training. A secondary purpose was to help

determine what issues DACOWITS would concentrate on in the future. The

Chair and the Executive Director of DACOWITS selected seven members (all

were women) to visit training installations. Members were selected based

on their DACOWITS experience and the quantity and quality of their previous

installation reports. Typical visits were conducted by one DACOWITS

member and lasted 2 days. Visits began with a briefing by the commanding

officer about the school and its mission, followed by trainee, trainer, and

supervisor focus groups. At the end of a visit, the DACOWITS member met

with command officials to share the results of the focus groups. Reports,

summarizing the most frequently heard comments from the various focus

groups, were written at the conclusion of each visit. In addition, the seven

members met at DACOWITS’ 1997 fall conference to discuss the results of

their visits. Using the reports and the conference discussion, the 1997

Page 10 GAO/NSIAD-98-125 Gender Issues
B-279460 

DACOWITS Chair wrote the report. The report was released by the Secretary

of Defense in January 1998.

Methodology DACOWITS used focus groups as its primary means of data gathering.

Overall, they solicited the opinions of over 1,200 trainees, trainers, and

supervisors in the Army, the Navy, the Marines, the Air Force, and the

Coast Guard in focus group discussions at 12 gender-integrated training

schools at 9 installations. The schools included enlisted basic,

intermediate, and advance training, and officer advanced training. Most

focus groups were gender-segregated and trainees, trainers, and

supervisors were in separate focus groups as well. DACOWITS requested

trainees with at least 40 percent of training completed. Many trainees had

completed their training and were awaiting graduation.

The groups averaged 20 participants and sessions lasted about 60 minutes,

although some were shorter. Before meeting with the Committee

members, focus group participants viewed an 18-minute video that

explained the mission of DACOWITS and highlighted some of the gender

equality issues that DACOWITS had worked on in the past such as sexual

harassment, discrimination, child care, and the combat exclusion policy.

The video set the stage for the two open-ended questions that all the

participants were asked: (1) “How is it going?” and (2) “If you had five

minutes to speak with the Secretary of Defense, what would you tell him?”

According to the former Chair, DACOWITS uses these questions during all

installation visits. Training installation visits took place between July and

November 1997.

At the conclusion of each visit, a DACOWITS member completed a

standardized installation visit report summarizing the most frequently

heard comments from the focus groups. The comments included in these

reports were entered into a computer and sorted by frequency across the

services as well as by individual service. Issues were included in the report

to the Secretary based on frequency. The individual installation visit

reports support the opinions and perceptions that appear in the report to

the Secretary of Defense.

Problems With the

Methodology Limit the

Usefulness of the Report

Some focus groups may have been too large or may not have had enough

time to allow ample participation by most of the participants. The

literature suggests that focus groups should be no larger than 

12 participants. During the DACOWITS visits to the training schools, some

Page 11 GAO/NSIAD-98-125 Gender Issues
B-279460 

groups were as large as 20 participants. Groups larger than 12 usually do

not allow sufficient opportunity to actively participate in the discussion

and are more difficult to manage. Also, the majority of the sessions were

about an hour long and some ran for only 45 minutes, about half the time

recommended by focus group literature.

DACOWITS used two questions to generate discussion. However, time may

still have been a problem, since the questions were very open-ended and

could be taken in virtually any direction by a participant. This would likely

have the effect of increasing the amount of time needed as each

participant not only answered the discussion questions, but also reacted

and responded to the issues raised by others.

DACOWITS did not document the individual focus groups as recommended

by focus group literature. Instead, DACOWITS members prepared installation

visit reports which summarized the opinions they heard most frequently.

While the installation reports document the work performed and the

issues surfaced during the training installation visits, they do not capture

enough information about the discussions in each focus group to be really

useful. For example, they do not provide enough information on the rank

or gender of the groups that raised the issue which would help put the

comments into perspective.

As we stated earlier, all of the DACOWITS members making installation

training installation visits were women. Some focus group literature

suggests that the gender of the moderator and the gender of the focus

group should be the same, particularly when the issues being discussed

are sensitive or have a direct bearing on the opposite sex. Also, some

focus group literature suggests that men are more likely to tell a woman

moderator what they think will impress or please her rather than what

they think. The use of female moderators for male focus groups, in

conjunction with the women’s advocacy impression that the video is likely

to have conveyed, may have made some males hesitant to raise issues or

perceptions that might be construed as anti-female. Because DACOWITS did

not document each of its focus groups it is impossible to determine if the

use of women moderators with all-male focus groups had an effect on the

responses of the male participants.

The DACOWITS report provides some methodological information for the

reader, but does not provide some key information. First, the report does

not provide any details on how the Committee members documented the

focus groups. Second, the report does not clearly explain the process used

Page 12 GAO/NSIAD-98-125 Gender Issues
B-279460 

by DACOWITS to determine what issues would be included in the report.

Third, while the report provides some detail about the make-up of the

focus groups it does not describe how the focus group participants were

selected. It should be noted however, as recommended by focus group

literature, the report clearly states its two major limitations: (1) the

opinion and perception information included in the report has not been

independently validated or confirmed and ( 2) the Committee did not visit

any gender-segregated training facilities. Also, in accordance with the

limitations of the methodology, the DACOWITS report made no conclusions

or recommendations on military training.

Agency Comments We provided a draft of this report to DOD, the Chairman of the Federal

Advisory Committee on Gender-Integrated Training and Related Issues,

and the former Chair and Military Director of DACOWITS for comment.

We discussed our report with Department of the Army officials, who

concurred with our observations on the Army’s Senior Review Panel on

Sexual Harassment. We also discussed the draft report with the Executive

Director of the Federal Advisory Committee on Gender-Integrated

Training and Related Issues who suggested some clarifications to the

report, which we considered and made as appropriate. In addition, we

discussed the draft with the military director of DACOWITS, who stated that

DACOWITS does not aim to meet the standards of academic research but

instead uses focus groups to collect opinions and identify issues for

further study. Finally, we discussed the draft with the former Chair of

DACOWITS who suggested some technical corrections which we made as

appropriate.

Scope and

Methodology

We reviewed the reports from the Army’s Senior Review Panel on Sexual

Harassment, the Federal Advisory Committee on Gender-Integrated

Training and Related Issues, and DACOWITS. We reviewed literature on the

conduct and use of focus groups, since that was a common methodology

across the three studies. We focused on the methodological information

provided in the reports, including any limitations on the use of the

information. We reviewed supporting documents to determine if the

evidence collected supports making conclusions and recommendations.

We did not evaluate the validity of specific conclusions and

recommendations made by any of the studies.

Page 13 GAO/NSIAD-98-125 Gender Issues
B-279460 

We met with the Chairman and Executive Director of the Federal Advisory

Committee on Gender-Integrated Training and Related Issues, the former

Chair and Military Director of DACOWITS, and with the senior social scientist

of the Army’s Senior Review Panel on Sexual Harassment to thoroughly

explore the approach and methodology used in these efforts.

Our review was requested by the former Ranking Minority Member of the

House National Security Committee and Mr. Meehan. We are addressing

the report to the current Ranking Minority Member of the House National

Security Committee, Mr. Skelton, as a courtesy. We are addressing this

letter to Senator Robb because it is related to other work on gender issues

in the military that we have undertaken at his request.

We conducted our review in February and March 1998 in accordance with

generally accepted government auditing standards.

We are sending copies of this report to the Secretaries of Defense, the

Army, the Air Force, and the Navy; the Chairman of the Joint Chiefs of

Staff; and the Commandant of the Marine Corps. We will make copies

available to any other interested parties.

The major contributors to this report were Carol R. Schuster, William E.

Beusse, Carole F. Coffey, George M. Delgado, and Kathleen M. Joyce. If

you or your staff have any questions concerning this report, please call me

on (202) 512-5140.

Mark E. Gebicke

Director, Military Operations

 and Capabilities Issues

Page 14 GAO/NSIAD-98-125 Gender Issues
Page 15 GAO/NSIAD-98-125 Gender Issues
Contents

Letter 1

Appendix I 

Focus Group

Methodology

18

Appendix II 

Summary of Reports

21

Abbreviations

DACOWITS Defense Advisory Committee on Women in the Services

DOD Department of Defense

Page 16 GAO/NSIAD-98-125 Gender Issues
Page 17 GAO/NSIAD-98-125 Gender Issues
Appendix I 

Focus Group Methodology

Focus groups1 are carefully planned small group discussions involving

people with similar characteristics who are knowledgeable about an issue

but do not know each other well. The views expressed in focus groups are

not necessarily representative of a population and statistical estimates

cannot be derived from the results. Furthermore, focus groups cannot be

used to determine the extent of a problem. Focus groups produce

qualitative data that provide insights into attitudes, perceptions, and

opinions of the participants. They are most often used before, during, or

after quantitative research procedures such as surveys. For example, focus

groups can be used before a survey is undertaken to help a research team

learn about the target audience or determine the appropriateness of the

questionnaire. Focus groups are often used with surveys to confirm

findings and to obtain greater breadth and depth of information. Finally,

focus groups are often used as a follow-up to surveys to help interpret

responses. On occasion, focus groups are used alone when opinions and

perceptions are more important than how many people hold such views.

The size of the focus group is an important, but often overlooked, element

of a successful group discussion. The literature on focus groups suggests

that an appropriate size for a focus group is 6 to 12 people. A focus group

with fewer than six participants sometimes has problems with

productivity because the group has fewer experiences to share. Also, small

groups can be more easily affected by people who know each other, by

uncooperative participants, or by participants who view themselves as

experts on the topic. Groups that have more than 12 people usually do not

allow people sufficient opportunity to actively participate in the

discussion, making the groups difficult to manage.

The composition of the focus group is also important. Participants should

share some similar characteristics but be diverse enough to allow for

differences of opinions. The topic of discussion and the information to be

obtained dictate the types of characteristics shared. However, generally

participants should be similar in age, occupation, education, and social

class. Focus groups with distinct differences among participants such as

trainees and trainers or junior and senior enlisted personnel do not work

well because of limited understanding of other lifestyles and situations.

Furthermore, some participants may be inhibited and defer to those they

believe to be better educated or more experienced or of a higher social

1

This section drew from the following: Richard A. Krueger, Focus Groups: A Practical Guide for

Applied Research (Newbury Park, California: Sage Publications,1988); David L. Morgan, Focus Groups

As Qualitative Research (Newbury Park, California: Sage Publications,1988); and David W. Stewart

and Prem N. Shamdasani, Focus Groups: Theory and Practice (Newbury Park, California: Sage

Publications,1990)

Page 18 GAO/NSIAD-98-125 Gender Issues
Appendix I 

Focus Group Methodology

class. Sometimes, the gender of participants can affect the outcome of a

focus group and some social scientists recommend against mixing genders

because men and women tend to perform for each other. When the

opinions of disparate groups are needed, focus group literature

recommends holding separate groups for each distinct group.

Focus group discussions are conducted informally and guided by trained

moderators who encourage participants to share their thoughts and

experiences. Trained, experienced moderators are critical to the success

of a focus group. An unqualified moderator can easily undermine the

reliability and validity of focus group findings. Successful moderators are

good listeners, who can make people feel relaxed and anxious to talk.

Moderators must control a group without being obvious and be aware of

time. Since literature suggests that focus groups should be scheduled for

90 minutes and run no more than 120 minutes, moderators need to be able

to keep the discussion on track and move the participants from one topic

to the next. Moderators should be aware of the influence that they have on

the type and amount of data obtained. Moderators must be aware of their

own biases that might affect the validity of the data and take care not to

provide cues to participants about desirable responses and answers. If

dealing with sensitive subjects where views could vary according to

factors such as gender or race, it is recommended that the moderator be

similar in gender or race to the participants. Finally, moderators must have

sufficient knowledge of the topic to put comments in perspective and

followup on critical areas of concern.

Questions are the heart of the focus group discussion. The literature on

focus groups suggests five or six questions for a discussion group. The

questions need to be carefully thought-out and phrased to result in the

maximum amount of information in the limited time available. Questions

should not suggest potential answers and yes or no questions should be

avoided. Questions should be asked in the same order in every focus group

and questions should be sequenced from most important to least

important to ensure that the most necessary information is obtained from

the participants if time runs out. The sequence is important because the

questions may interact with one another to form the stimulus that

generates the responses. If the questions are asked in a different order at

each focus group, the stimulus is changed and the response will be

different.

The results of the focus groups’ discussions should be documented on a

session by session basis. Focus group literature agrees that the best way to

Page 19 GAO/NSIAD-98-125 Gender Issues
Appendix I 

Focus Group Methodology

do that is by tape recording supplemented with written notes. However, if

a tape recording is not feasible or inhibiting to the participants, note taking

can be sufficient provided they are complete enough to be analyzed.

A systematic analysis of focus group data is also important. The analysis

can be either qualitative or quantitative, but it must be systematic and

verifiable. It must be systematic in that it follows a documented

step-by-step process and verifiable to permit others to arrive at similar

conclusions using available documents and the raw results. Social

scientists have noted that there is a tendency for novice researchers to see

selectively only those parts of the discussion that confirms their particular

point of view. Often, a researcher will go into the discussion with certain

hunches of how participants might feel. As a result, the researcher tends to

look for evidence to support these hunches and overlook data that present

different points of view. A systematic and verifiable process helps

researchers in filtering out bias and assuring that they present the data as

objectively as possible.

Once data are collected and analyzed, the data should be reported and, if

appropriate, conclusions and recommendations made. A report should

clearly state what the purpose of the study was, what its scope was, how

the data were collected and analyzed, and what, if any, significant

limitations exist on the data or the use of the data. For example, studies

that used focus groups as the primary method of data collection should

clearly state that the data being reported is opinion or perception. If the

opinions have been substantiated by other types of data, this should be

clearly stated in the report. The report should also include the results of

the focus groups, and the results should be clearly stated so that a reader

can come to the same conclusions as the report writers.

Page 20 GAO/NSIAD-98-125 Gender Issues
Appendix II 

Summary of Reports

The Secretary of the

Army’s Senior Review

Panel on Sexual

Harassment

The following is excerpted from Army press reports that accompanied the

report of the Senior Review Panel’s report on Sexual Harassment in the

Army as well as the executive summary of the report:

The panel found:

• Sexual harassment exists throughout the Army, crossing gender, rank, and

racial lines; gender discrimination is more common than sexual

harassment.

• Army leaders are the critical factor in creating, maintaining and enforcing

an environment of respect and dignity in the Army; too many leaders have

failed to gain the trust of their soldiers.

• The Army lacks institutional commitment to the Equal Opportunity

program and soldiers distrust the equal opportunity program.

• Trainees believe the overwhelming majority of drill sergeants and

instructors perform competently and well, but “respect” as an Army core

value is not well institutionalized in the Initial Entry training process.

Recommendations of the panel were broad-based and covered a wide

variety of Army processes including: leader development, equal

opportunity policy and procedures, initial entry training soldierization,

unit and institutional training, command climate, and oversight.

Federal Advisory

Committee on

Gender-Integrated Training

and Related Issues

The following is an excerpt from the press release that accompanied the

report of the Federal Advisory Committee on Gender-Integrated Training

and Related Issues which delivered its report to the Secretary of Defense

in December 1997:

“The panel studied the full training cycle including recruiting, basic training, and advanced

skills training. Its recommendations covered the training cadre, housing of recruits, fitness

programs and follow-on advance training. Among the several recommendations made for

recruiting, the panel proposed better preparing recruits mentally and physical for basic

training. It also recommended ways to improve the training cadre. It recommended that

physical training requirements be toughened and made more uniform throughout the

services. The panel also suggested that emphasis on discipline be carried over from basic

to advance training. The panel recommended that value training be incorporated into all

initial entry training programs and that training get more resources.

“During visits to training installations, the panel concluded that men and women should be

housed in separate barracks and train separately at the operational unit level — the Army

platoon, the Navy division and the Air Force flight. In the Marine Corps men and women

Page 21 GAO/NSIAD-98-125 Gender Issues
Appendix II 

Summary of Reports

live, eat, and train separately. The panel recommended that gender-integrated training

continue for field training, technical training and classroom work.”

Defense Advisory

Committee on Women in

the Services

The following is an extract from the executive summary of the report to

the Secretary of Defense from the Defense Advisory Committee on Women

in the Services:

“The scope of DACOWITS’ training installation visits included all elements of initial entry

training, including basic training, advanced individual training, and officer advanced

training. The majority of issues raised by trainees, trainers, and supervisors of trainers

were similar across all of the Armed Forces.

“The most frequently raised issues by women and men and trainees and trainers alike were

artificial gender relationships imposed at training installations, the persistence of gender

discriminatory behaviors at many locations; the relationship between trainer attitudes and

gender climates; the under valuation of trainers, especially women trainers; the need for

greater gender integration to train field and fleet ready servicemembers, the need to

increase physical training opportunities and standards; the need to improve screening of

new recruits and to harmonize recruiting quality and practices; the under resourcing of

training schools and the need to improve support services for women trainees.”

(703235) Page 22 GAO/NSIAD-98-125 Gender Issues
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100