Gender Issues: Analysis of Methodologies in Reports to the Secretaries of Defense and the Army
Published: 1998-03-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO reviewed three studies on gender-related issues affecting initial entry training in the Department of Defense (DOD), focusing on: (1) how the groups conducted their work; (2) how well the work supported making conclusions and recommendations; (3) the availability of documentation supporting the report; and (4) the extent to which the final report described the study methodology and disclosed limitations.
section_title: What GAO Found
paragraphs: GAO noted that: (1) the Army's Senior Review Panel on Sexual harassment used four methods to collect data: individual interviews, focus groups, surveys, and observations; (2) during its 8 months of work, the panel visited 59 installations worldwide, conducted interviews with 808 military and civilian Army personnel, ran focus groups with over 8,000 soldiers and civilians, and surveyed 22,952 individuals; (3) the use of multiple methods of data gathering, the rigor with which the various methods were conducted, and the publication of the data in the report provides ample support for making conclusions and recommendations; (4) the Federal Advisory Committee (FAC) on Gender-Integrated Training and Related Issues used focus groups as its primary method of data gathering; (5) although FAC conducted over 300 focus groups and individual interviews, their value for making conclusions and recommendations is limited because the Committee did not: (a) systematically collect the same information from all groups; (b) document the information generated in each of the interviews and focus groups; or (c) explain how what was heard in the interviews and focus groups led to their conclusions and recommendations; (6) in addition, the length of the focus group sessions, the number of participants, and the number of questions addressed may not have provided adequate time for full participation of the respondents on all issues; (7) given these limitations, the extent to which the Committee's work supports its conclusions and recommendations cannot be determined; (8) the Defense Advisory Committee on Women in the Services (DACOWITS) also used focus groups of trainees, trainers, and supervisors in the Army, Air Force, Navy, Marine Corps, and Coast Guard to identify what issues concerned women and men at training installations; (9) members of the DACOWITS held focus group discussions at 12 schools at 9 installations in the United States and prepared a summary report of the results at each installation; (10) the DACOWITS Chair used these to prepare a report to the Secretary of Defense that accurately reflected the opinions and perceptions cited in the individual installation reports; (11) the DACOWITS focus groups were: (a) larger than recommended in the literature; (b) were sometimes not long enough to allow meaningful participation; and (c) were not recorded or documented on a group-by-group basis; and (12) the DACOWITS report summarized the opinion and perception data obtained from focus groups; and (13) it made no conclusions or recommendations on military training based on that information.

section_title: Letter
paragraphs: Over the past year, several groups have examined gender-related issues affecting initial entry training in the Department of Defense (DOD). The studies that have gotten the most attention are the Army’s Senior Review Panel’s report on sexual harassment, the report of the Federal Advisory Committee on Gender-Integrated Training and Related Issues, and the report of the Defense Advisory Committee on Women in the Services. Because the groups produced somewhat different results, our objectives were to (1) describe how the groups conducted their work, (2) determine how well the work supported making conclusions and recommendations, (3) assess the availability of documentation supporting the report, and (4) determine the extent to which the final report described the study methodology and disclosed limitations.
Since each of the three studies used focus groups as either their primary data gathering method or in concert with other means of collecting information, we used social science literature on focus groups to assess their use of that methodology. Focus groups use a small group discussion format, guided by a moderator, to gather information on a given topic. Focus groups are useful in providing an understanding of the depth and the variety of opinions, feelings, or beliefs, but are not useful for quantifying the extent to which a particular view or opinion is held. For these reasons, focus groups are a qualitative technique best used when the aim is to explore an issue or, at the other end of the research process, to assess reaction to specific proposals. As with any methodology, its value is dependent upon the extent to which it is appropriate to the objectives of the inquiry, how well the methodology is conducted, and the way in which the results are used. More information on focus group methodology is found in appendix I.
subsections: section_title: Background
paragraphs: The reports of sexual misconduct at Aberdeen Proving Grounds led the Secretary of Defense to establish the Federal Advisory Committee on Gender-Integrated Training and Related Issues and to ask the Defense Advisory Committee on Women in the Services to meet with trainees and trainers. These incidents also prompted the Secretary of the Army to establish the Senior Review Panel on Sexual Harassment.
In November 1996, the Secretary of the Army established the Senior Review Panel on Sexual Harassment. The panel’s mission was to make recommendations to improve the human relations environment in which soldiers live and work, with the specific goal of eradicating sexual harassment in the Army. The panel consisted of seven members, including two retired general officers recalled to active duty, two active duty general officers, a senior noncommissioned officer, and two DOD civilians. The Senior Review Panel forwarded its report and recommendations to the Secretary of the Army in July 1997. It included 40 recommendations, of which 14 dealt with training and related issues.
In June 1997, the Secretary of Defense established the Federal Advisory Committee on Gender-Integrated Training and Related Issues. The Secretary of Defense established the Committee as a result of the sexual misconduct incidents at Aberdeen Proving Grounds. Former Senator Nancy Kassebaum Baker chaired a panel of 11 that included civilians, retired officers, and a retired senior noncommissioned officer. The Secretary directed the Committee to assess the training programs and policies of the Army, Navy, Air Force, and Marine Corps and make recommendations to improve initial entry training. The Committee issued its report to the Secretary of Defense on December 16, 1997. It made 30 recommendations covering the full cycle from recruitment through basic and advanced training.
The Defense Advisory Committee on Women in the Services (DACOWITS) has been advising secretaries of Defense since George Marshall established the Committee in 1951. DACOWITS, which consists of 30 to 40 civilians, makes recommendations to the Secretary on the roles of women in the Armed Forces and on quality of life issues affecting readiness. As part of its mission, DACOWITS members conduct annual visits to selected Army, Air Force, Navy, Marine Corps, and Coast Guard installations, both here and overseas. These visits serve two purposes: (1) to provide the Secretary of Defense with insight into the thoughts and perceptions of servicemembers in the fleet and the field and (2) to determine what issues DACOWITS will concentrate on in the future. In November 1996, the former Secretary of Defense requested that DACOWITS visit training installations to meet with trainees and trainers in the training environment. In February 1997, the current Secretary of Defense endorsed the request. DACOWITS provided a report to the Secretary of Defense summarizing these visits. In its report, DACOWITS recommended continued visits to training installations, but made no recommendations on military training.
section_title: The Army’s Senior Review Panel on Sexual Harassment
paragraphs: The Army’s Senior Review Panel on Sexual Harassment formed four teams, one to review Army policies and three for data collection. Each field team consisted, on the average, of six military personnel and one civilian. The Chair, the Vice-Chair, or the Deputy Assistant Secretary of the Army (a member of the panel) accompanied each field team during their visits. Other panel members traveled with the teams as often as possible.
Visits lasted 1 to 4 days depending on the numbers of participants in the various activities. Before the visits, the participants for the individual interviews, focus groups, and survey were selected and scheduled. Generally, the visit started with a briefing to present the purpose of the activity and a description of the team’s data collection efforts. Next, the team divided into smaller groups to conduct individual interviews, conduct focus groups, or administer surveys. These activities ran concurrently and team members rotated to different activities at different times. Visits ended with a briefing providing commanders the opportunity to begin corrective actions.
Panel members and the working group collaborated in writing the panel’s report. Once a near final draft was generated, the panel members met for a final review and agreement on the content. The panel on Sexual Harassment issued its report to the Secretary of the Army in July 1997.
subsections: section_title: Methodology
paragraphs: The Army’s Senior Review Panel on Sexual Harassment used four methods to collect data: individual interviews, focus groups, surveys, and observations. According to the senior social scientist detailed to the panel, the field teams conducted interviews and focus groups using carefully developed protocols to obtain human relations environment information. Members of field teams conducted individual interviews with 808 military and civilian Army leaders and personnel in Army support groups. Focus groups consisted of randomly selected, single-gender groups of 8 to 12 people organized by ranks or categories. Participants totaled 7,401 soldiers and 1,007 civilians. Facilitators and note takers of the same gender as the groups conducted the sessions. All data obtained through these two activities were entered into a computer for analysis. The working group, which consisted of more than 40 military and civilian personnel, developed main themes or categories and placed the perceptions data under the categories. Data were then analyzed by rank, by gender, and by question.
The written surveys addressed leadership, cohesion, and sexual harassment. Field teams administered the surveys to 22,952 servicemembers. Surveys were developed for trainees, trainers, and the general Army population. The working group analyzed survey data using a standard statistical analysis software package. Statistically projectable results appear in the report by question and, in some cases, by gender. Observations were made during visits to barracks and other facilities and to engage in informal conversations with military and civilian personnel, family members, and others.
The seven panel members, supported by the working group, collected data at 59 Army installations worldwide selected using a stratified random sampling design. Stratification was based on the type and location of the installation. The study took 8 months to complete and obtained information from over 32,000 Army personnel.
section_title: Use of Multiple Methodologies and Extensive Documentation Allowed Making Conclusions and Recommendations
paragraphs: The panel’s methodology supported making conclusions and recommendations. Focus groups were used in conjunction with surveys to not only confirm the survey data but also provide texture and perspective to the data. The focus groups were of an appropriate size and were all asked the same questions, in the same order by trained moderators. However, the number of questions asked of many of the focus groups was significantly greater than the five or six questions recommended by focus group literature. For example, the set of questions for trainee focus groups consisted of 15 questions and the set for trainers consisted of 13 questions. Focus group discussions were not tape recorded because it was feared that this would inhibit the participants, but notes were taken by a note taker and were content-analyzed. The notes from each focus group session were destroyed, after the responses were entered in the database and verified for accuracy, to assure that participant confidentiality was maintained. Destroying the original documentation to assure confidentiality is considered an appropriate measure by social scientists. In addition, the completed survey forms were also destroyed to assure participant confidentiality.
In volume two, the panel provides an extensive discussion of its methodology. Volume two provides details on how participants were selected, copies of the focus group questions, the surveys, and the interview questions. Results of the surveys were included in the report as well as the most frequently heard responses in the focus groups. Furthermore, a sufficient amount of data are presented in volume one of the report, which outlines the panel’s conclusions and recommendations, to allow the reader to evaluate them.
subsections: section_title: Lack of Survey Pretest Results in Controversy
paragraphs: An area of controversy arose because the survey developers did not pretest the survey questions. We were told by the senior social scientist attached to the panel, that tight time frames precluded the panel from carrying out a pretest of the survey form. Normally, a pretest is performed to identify problem questions, problems with language interpretations, unclear instructions, or to determine if there are some questions that respondents will refuse to answer. In this instance, the survey form contained six questions that some respondents in early administrations found inflammatory, offensive, and an invasion of privacy and resulted in some refusals to complete the survey. Subsequently, those questions were eliminated and a revised form was used. The data on the six questions were not included in the database, which resulted in an accusation that the panel had eliminated important data from its analysis. The report disclosed the problem and its resolution in the methodology section. We believe that the panel acted responsibly in eliminating the offending questions to avoid having a negative effect on the survey return rate. The controversy, however, demonstrates the importance of pretesting survey forms before conducting a survey.
section_title: The Federal Advisory Committee on Gender-Integrated Training and Related Issues
paragraphs: The Federal Advisory Committee on Gender-Integrated Training and Related Issues saw its role as listening to the views of trainees, trainers, supervisors, and service officials and providing the Secretary of Defense with its best judgment about what should be done to improve training. Small teams of Committee members visited 17 training installations and operational units to gather opinions. Most Committee members visited installations from two services. While the Committee Chairman visited installations for all four services, no Committee member or Committee staff member visited all of the installations.
Once at an installation, the Committee members followed the same general schedule: reveille, breakfast with new servicemembers, meetings with command officials, and interviews and focus groups before lunch. After lunch with support personnel, the Committee members conducted additional focus groups and interviews. At the end of each visit, they met with command officials to discuss their findings. Installation visits generally lasted 1 day, although visits to basic training sites were 2-day trips. The visits to the training installations and operational units occurred in September and October 1997.
The Committee had two public meetings, the first in July 1997 and the second in October 1997. At the July meeting, service representatives provided information on the services’ recruiting and training programs. At the October 1997 meeting, Committee members discussed their observations and agreed to a partial list of recommendations for the report. The Committee’s staff drafted the report based on the discussions they heard during their installation visits and the public meeting, and memorandums submitted by some of the Committee members in preparation for the October meeting. Committee members received the draft report in early December and revisions were made based on their comments. The Committee Chair discussed the report with Committee members in a series of one-on-one telephone calls to arrive at the final recommendations.
subsections: section_title: Methodology
paragraphs: The Committee’s primary means of collecting information involved focus group discussions. The Committee held 199 focus groups, soliciting opinions from more than 1,000 trainees, 500 trainers, 300 first-term servicemembers, and 275 supervisors at U.S. training installations and operational units. Participants were randomly selected under the supervision of the installations’ inspectors general. Trainees who participated in the focus groups were within 2 weeks of completing their training. Participants in the trainer focus groups were trainers for at least 1 year. First term participants were in their initial assignment and had been on the job between 6 and 18 months. Generally, the Committee met with equal numbers of females and males, although, because of the limited number of female trainers and supervisors, this was not always possible.
Focus groups included about 10 to 15 people each and were gender-segregated. All of the focus groups were moderated by Committee members, and generally two Committee members or a Committee member and a Committee staff member attended each session. The Committee members worked from a set of questions tailored for each service and each type of focus group. Although the number of questions varied by type of focus group, the set of questions for all basic training focus groups consisted of 20 questions, some of which had multiple parts. While some focus groups were scheduled to last only 30 to 45 minutes, most focus group sessions lasted nearly an hour.
Committee members also conducted over 100 interviews with service officials, including commanding officers, inspectors general, company or squadron commanders, and senior noncommissioned officers. They also met with representatives of support groups such as chaplains, equal opportunity officers, medical officers, and legal officers.
section_title: Problems With the Methodology Limit the Usefulness of the Report
paragraphs: The value of the information included in the Committee’s report for making conclusions and recommendations is limited because the Committee did not follow recommended focus group methodology. The Committee believed that a more flexible approach to the discussions would enhance the quality of the exchange between the participants and the Committee members. However, the fact that the same questions were not asked of each similar focus group, along with the number of questions, size of the groups, and length of the sessions may have combined to limit full discussion. In addition, the focus groups’ discussions were not systematically recorded. As a result, the extent to which the recommendations are supported by the Committee’s work cannot be assessed.
The Committee staff provided the Committee members with questions for the focus groups. However, according to the staff director, the Committee members were told that the questions were guidelines and that they did not have to be asked as written. Because the Committee members had the flexibility to ask any question they desired, the responses should not be compared with each other. Also, the number of questions provided to the panel members were far more than the five to six focus group literature recommends. For example, the staff provided 20 questions, some of which had several parts, for the Committee to ask Army trainees at Fort Leonard Wood, Missouri. Fifteen trainees participated in the 1-hour focus groups at this installation. If the entire hour was spent on the questions, there would have only been 3 minutes spent on each question and only 15 seconds for each participant to respond. We do not believe that would have been enough time for a meaningful discussion of a question. Finally, even if all the questions were asked as they were written, they were not always asked in the same order each time. Social science literature suggests that the same questions asked in a different sequence may result in different responses.
The absence of documentation of the comments made in the individual focus groups was the most serious methodological shortcoming. While the Committee members took notes during each focus group, these notes were not made part of the Committee’s records nor were they summarized and included in the report. Without documentation, it is impossible to determine if the Committee’s work supports its recommendations. Also, the lack of documentation prevented the Committee from analyzing the data to know what comments they heard or how often similar comments were made. Knowing how often a particular kind of comment was made and the subgroup of the person who made it are ways of putting the comments in perspective and filtering biases.
The report of the Federal Advisory Committee on Gender-Integrated Training and Related Issues does not include a sufficient discussion of the Committee’s methodology and work process. For example, the report states that the Committee conducted discussion groups with randomly selected servicemembers, but it does not explain the random selection process. In addition, the use of terminology such as “randomly selected” implies a level of scientific rigor that was not achieved in this study. The report does not identify the make-up of the discussion groups, discuss what type of data analysis was done or not done, or mention any limitations with the data. Limitations that we believe should have been mentioned are that the report was based on opinions and the results cannot be generalized to the entire military training population.
Also, the report often presents opinions in a manner that they can be misinterpreted as facts based on empirical data. For example, the report says that the Committee members observed that integrated housing is contributing to a higher rate of disciplinary problems, but, according to the Chairman, the Committee did not obtain any data to support this statement. In addition, the report contains many statements that include words like “most”, “many”, and “majority”. These words lead a reader to believe that the Committee counted responses to particular questions or polled the focus group participants. The Chairman said that the Committee does not have quantitative data.
The report also does not explain the process the Committee used to formulate its recommendations. Although the Committee held a public meeting in October 1997 after its installation visits had been completed, the recommendations on separate barracks for male and female recruits and on the organization of gender-segregated platoons, divisions, and flights were not made until after that meeting. Furthermore, those recommendations were not discussed by the Committee as a whole, but rather in a series of calls to individual Committee members.
section_title: The Defense Advisory Committee on Women in the Services
paragraphs: The mission of the DACOWITS effort was to provide the Secretary of Defense with an overview of broad issues raised by trainees and trainers of both genders throughout initial entry training. A secondary purpose was to help determine what issues DACOWITS would concentrate on in the future. The Chair and the Executive Director of DACOWITS selected seven members (all were women) to visit training installations. Members were selected based on their DACOWITS experience and the quantity and quality of their previous installation reports. Typical visits were conducted by one DACOWITS member and lasted 2 days. Visits began with a briefing by the commanding officer about the school and its mission, followed by trainee, trainer, and supervisor focus groups. At the end of a visit, the DACOWITS member met with command officials to share the results of the focus groups. Reports, summarizing the most frequently heard comments from the various focus groups, were written at the conclusion of each visit. In addition, the seven members met at DACOWITS’ 1997 fall conference to discuss the results of their visits. Using the reports and the conference discussion, the 1997 DACOWITS Chair wrote the report. The report was released by the Secretary of Defense in January 1998.
subsections: section_title: Methodology
paragraphs: DACOWITS used focus groups as its primary means of data gathering. Overall, they solicited the opinions of over 1,200 trainees, trainers, and supervisors in the Army, the Navy, the Marines, the Air Force, and the Coast Guard in focus group discussions at 12 gender-integrated training schools at 9 installations. The schools included enlisted basic, intermediate, and advance training, and officer advanced training. Most focus groups were gender-segregated and trainees, trainers, and supervisors were in separate focus groups as well. DACOWITS requested trainees with at least 40 percent of training completed. Many trainees had completed their training and were awaiting graduation.
The groups averaged 20 participants and sessions lasted about 60 minutes, although some were shorter. Before meeting with the Committee members, focus group participants viewed an 18-minute video that explained the mission of DACOWITS and highlighted some of the gender equality issues that DACOWITS had worked on in the past such as sexual harassment, discrimination, child care, and the combat exclusion policy. The video set the stage for the two open-ended questions that all the participants were asked: (1) “How is it going?” and (2) “If you had five minutes to speak with the Secretary of Defense, what would you tell him?” According to the former Chair, DACOWITS uses these questions during all installation visits. Training installation visits took place between July and November 1997.
At the conclusion of each visit, a DACOWITS member completed a standardized installation visit report summarizing the most frequently heard comments from the focus groups. The comments included in these reports were entered into a computer and sorted by frequency across the services as well as by individual service. Issues were included in the report to the Secretary based on frequency. The individual installation visit reports support the opinions and perceptions that appear in the report to the Secretary of Defense.
section_title: Problems With the Methodology Limit the Usefulness of the Report
paragraphs: Some focus groups may have been too large or may not have had enough time to allow ample participation by most of the participants. The literature suggests that focus groups should be no larger than  12 participants. During the DACOWITS visits to the training schools, some groups were as large as 20 participants. Groups larger than 12 usually do not allow sufficient opportunity to actively participate in the discussion and are more difficult to manage. Also, the majority of the sessions were about an hour long and some ran for only 45 minutes, about half the time recommended by focus group literature.
DACOWITS used two questions to generate discussion. However, time may still have been a problem, since the questions were very open-ended and could be taken in virtually any direction by a participant. This would likely have the effect of increasing the amount of time needed as each participant not only answered the discussion questions, but also reacted and responded to the issues raised by others.
DACOWITS did not document the individual focus groups as recommended by focus group literature. Instead, DACOWITS members prepared installation visit reports which summarized the opinions they heard most frequently. While the installation reports document the work performed and the issues surfaced during the training installation visits, they do not capture enough information about the discussions in each focus group to be really useful. For example, they do not provide enough information on the rank or gender of the groups that raised the issue which would help put the comments into perspective.
As we stated earlier, all of the DACOWITS members making installation training installation visits were women. Some focus group literature suggests that the gender of the moderator and the gender of the focus group should be the same, particularly when the issues being discussed are sensitive or have a direct bearing on the opposite sex. Also, some focus group literature suggests that men are more likely to tell a woman moderator what they think will impress or please her rather than what they think. The use of female moderators for male focus groups, in conjunction with the women’s advocacy impression that the video is likely to have conveyed, may have made some males hesitant to raise issues or perceptions that might be construed as anti-female. Because DACOWITS did not document each of its focus groups it is impossible to determine if the use of women moderators with all-male focus groups had an effect on the responses of the male participants.
The DACOWITS report provides some methodological information for the reader, but does not provide some key information. First, the report does not provide any details on how the Committee members documented the focus groups. Second, the report does not clearly explain the process used by DACOWITS to determine what issues would be included in the report. Third, while the report provides some detail about the make-up of the focus groups it does not describe how the focus group participants were selected. It should be noted however, as recommended by focus group literature, the report clearly states its two major limitations: (1) the opinion and perception information included in the report has not been independently validated or confirmed and ( 2) the Committee did not visit any gender-segregated training facilities. Also, in accordance with the limitations of the methodology, the DACOWITS report made no conclusions or recommendations on military training.
section_title: Agency Comments
paragraphs: We provided a draft of this report to DOD, the Chairman of the Federal Advisory Committee on Gender-Integrated Training and Related Issues, and the former Chair and Military Director of DACOWITS for comment.
We discussed our report with Department of the Army officials, who concurred with our observations on the Army’s Senior Review Panel on Sexual Harassment. We also discussed the draft report with the Executive Director of the Federal Advisory Committee on Gender-Integrated Training and Related Issues who suggested some clarifications to the report, which we considered and made as appropriate. In addition, we discussed the draft with the military director of DACOWITS, who stated that DACOWITS does not aim to meet the standards of academic research but instead uses focus groups to collect opinions and identify issues for further study. Finally, we discussed the draft with the former Chair of DACOWITS who suggested some technical corrections which we made as appropriate.
section_title: Scope and Methodology
paragraphs: We reviewed the reports from the Army’s Senior Review Panel on Sexual Harassment, the Federal Advisory Committee on Gender-Integrated Training and Related Issues, and DACOWITS. We reviewed literature on the conduct and use of focus groups, since that was a common methodology across the three studies. We focused on the methodological information provided in the reports, including any limitations on the use of the information. We reviewed supporting documents to determine if the evidence collected supports making conclusions and recommendations. We did not evaluate the validity of specific conclusions and recommendations made by any of the studies.
We met with the Chairman and Executive Director of the Federal Advisory Committee on Gender-Integrated Training and Related Issues, the former Chair and Military Director of DACOWITS, and with the senior social scientist of the Army’s Senior Review Panel on Sexual Harassment to thoroughly explore the approach and methodology used in these efforts.
Our review was requested by the former Ranking Minority Member of the House National Security Committee and Mr. Meehan. We are addressing the report to the current Ranking Minority Member of the House National Security Committee, Mr. Skelton, as a courtesy. We are addressing this letter to Senator Robb because it is related to other work on gender issues in the military that we have undertaken at his request.
We conducted our review in February and March 1998 in accordance with generally accepted government auditing standards.
We are sending copies of this report to the Secretaries of Defense, the Army, the Air Force, and the Navy; the Chairman of the Joint Chiefs of Staff; and the Commandant of the Marine Corps. We will make copies available to any other interested parties.
The major contributors to this report were Carol R. Schuster, William E. Beusse, Carole F. Coffey, George M. Delgado, and Kathleen M. Joyce. If you or your staff have any questions concerning this report, please call me on (202) 512-5140.
section_title: Focus Group Methodology
paragraphs: Focus groups are carefully planned small group discussions involving people with similar characteristics who are knowledgeable about an issue but do not know each other well. The views expressed in focus groups are not necessarily representative of a population and statistical estimates cannot be derived from the results. Furthermore, focus groups cannot be used to determine the extent of a problem. Focus groups produce qualitative data that provide insights into attitudes, perceptions, and opinions of the participants. They are most often used before, during, or after quantitative research procedures such as surveys. For example, focus groups can be used before a survey is undertaken to help a research team learn about the target audience or determine the appropriateness of the questionnaire. Focus groups are often used with surveys to confirm findings and to obtain greater breadth and depth of information. Finally, focus groups are often used as a follow-up to surveys to help interpret responses. On occasion, focus groups are used alone when opinions and perceptions are more important than how many people hold such views.
The size of the focus group is an important, but often overlooked, element of a successful group discussion. The literature on focus groups suggests that an appropriate size for a focus group is 6 to 12 people. A focus group with fewer than six participants sometimes has problems with productivity because the group has fewer experiences to share. Also, small groups can be more easily affected by people who know each other, by uncooperative participants, or by participants who view themselves as experts on the topic. Groups that have more than 12 people usually do not allow people sufficient opportunity to actively participate in the discussion, making the groups difficult to manage.
The composition of the focus group is also important. Participants should share some similar characteristics but be diverse enough to allow for differences of opinions. The topic of discussion and the information to be obtained dictate the types of characteristics shared. However, generally participants should be similar in age, occupation, education, and social class. Focus groups with distinct differences among participants such as trainees and trainers or junior and senior enlisted personnel do not work well because of limited understanding of other lifestyles and situations. Furthermore, some participants may be inhibited and defer to those they believe to be better educated or more experienced or of a higher social class. Sometimes, the gender of participants can affect the outcome of a focus group and some social scientists recommend against mixing genders because men and women tend to perform for each other. When the opinions of disparate groups are needed, focus group literature recommends holding separate groups for each distinct group.
Focus group discussions are conducted informally and guided by trained moderators who encourage participants to share their thoughts and experiences. Trained, experienced moderators are critical to the success of a focus group. An unqualified moderator can easily undermine the reliability and validity of focus group findings. Successful moderators are good listeners, who can make people feel relaxed and anxious to talk. Moderators must control a group without being obvious and be aware of time. Since literature suggests that focus groups should be scheduled for 90 minutes and run no more than 120 minutes, moderators need to be able to keep the discussion on track and move the participants from one topic to the next. Moderators should be aware of the influence that they have on the type and amount of data obtained. Moderators must be aware of their own biases that might affect the validity of the data and take care not to provide cues to participants about desirable responses and answers. If dealing with sensitive subjects where views could vary according to factors such as gender or race, it is recommended that the moderator be similar in gender or race to the participants. Finally, moderators must have sufficient knowledge of the topic to put comments in perspective and followup on critical areas of concern.
Questions are the heart of the focus group discussion. The literature on focus groups suggests five or six questions for a discussion group. The questions need to be carefully thought-out and phrased to result in the maximum amount of information in the limited time available. Questions should not suggest potential answers and yes or no questions should be avoided. Questions should be asked in the same order in every focus group and questions should be sequenced from most important to least important to ensure that the most necessary information is obtained from the participants if time runs out. The sequence is important because the questions may interact with one another to form the stimulus that generates the responses. If the questions are asked in a different order at each focus group, the stimulus is changed and the response will be different.
The results of the focus groups’ discussions should be documented on a session by session basis. Focus group literature agrees that the best way to do that is by tape recording supplemented with written notes. However, if a tape recording is not feasible or inhibiting to the participants, note taking can be sufficient provided they are complete enough to be analyzed.
A systematic analysis of focus group data is also important. The analysis can be either qualitative or quantitative, but it must be systematic and verifiable. It must be systematic in that it follows a documented step-by-step process and verifiable to permit others to arrive at similar conclusions using available documents and the raw results. Social scientists have noted that there is a tendency for novice researchers to see selectively only those parts of the discussion that confirms their particular point of view. Often, a researcher will go into the discussion with certain hunches of how participants might feel. As a result, the researcher tends to look for evidence to support these hunches and overlook data that present different points of view. A systematic and verifiable process helps researchers in filtering out bias and assuring that they present the data as objectively as possible.
Once data are collected and analyzed, the data should be reported and, if appropriate, conclusions and recommendations made. A report should clearly state what the purpose of the study was, what its scope was, how the data were collected and analyzed, and what, if any, significant limitations exist on the data or the use of the data. For example, studies that used focus groups as the primary method of data collection should clearly state that the data being reported is opinion or perception. If the opinions have been substantiated by other types of data, this should be clearly stated in the report. The report should also include the results of the focus groups, and the results should be clearly stated so that a reader can come to the same conclusions as the report writers.
section_title: Summary of Reports
subsections: section_title: The Secretary of the Army’s Senior Review Panel on Sexual Harassment
paragraphs: The following is excerpted from Army press reports that accompanied the report of the Senior Review Panel’s report on Sexual Harassment in the Army as well as the executive summary of the report:
Sexual harassment exists throughout the Army, crossing gender, rank, and racial lines; gender discrimination is more common than sexual harassment.
Army leaders are the critical factor in creating, maintaining and enforcing an environment of respect and dignity in the Army; too many leaders have failed to gain the trust of their soldiers.
The Army lacks institutional commitment to the Equal Opportunity program and soldiers distrust the equal opportunity program.
Trainees believe the overwhelming majority of drill sergeants and instructors perform competently and well, but “respect” as an Army core value is not well institutionalized in the Initial Entry training process.
Recommendations of the panel were broad-based and covered a wide variety of Army processes including: leader development, equal opportunity policy and procedures, initial entry training soldierization, unit and institutional training, command climate, and oversight.
section_title: Federal Advisory Committee on Gender-Integrated Training and Related Issues
paragraphs: “The panel studied the full training cycle including recruiting, basic training, and advanced skills training. Its recommendations covered the training cadre, housing of recruits, fitness programs and follow-on advance training. Among the several recommendations made for recruiting, the panel proposed better preparing recruits mentally and physical for basic training. It also recommended ways to improve the training cadre. It recommended that physical training requirements be toughened and made more uniform throughout the services. The panel also suggested that emphasis on discipline be carried over from basic to advance training. The panel recommended that value training be incorporated into all initial entry training programs and that training get more resources. “During visits to training installations, the panel concluded that men and women should be housed in separate barracks and train separately at the operational unit level — the Army platoon, the Navy division and the Air Force flight. In the Marine Corps men and women live, eat, and train separately. The panel recommended that gender-integrated training continue for field training, technical training and classroom work.”
section_title: Defense Advisory Committee on Women in the Services
paragraphs: “The scope of DACOWITS’ training installation visits included all elements of initial entry training, including basic training, advanced individual training, and officer advanced training. The majority of issues raised by trainees, trainers, and supervisors of trainers were similar across all of the Armed Forces. “The most frequently raised issues by women and men and trainees and trainers alike were artificial gender relationships imposed at training installations, the persistence of gender discriminatory behaviors at many locations; the relationship between trainer attitudes and gender climates; the under valuation of trainers, especially women trainers; the need for greater gender integration to train field and fleet ready servicemembers, the need to increase physical training opportunities and standards; the need to improve screening of new recruits and to harmonize recruiting quality and practices; the under resourcing of training schools and the need to improve support services for women trainees.”
The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.