Equal Employment Opportunity: DOD's EEO Pilot Program Under Way, but Improvements Needed to DOD's Evaluation Plan
Published: 2006-05-05T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Delays in processing of equal employment opportunity (EEO) complaints have been a long-standing concern. In 2000, as part of the Department of Defense's (DOD) fiscal year 2001 authorization act, Congress authorized DOD to carry out a 3-year pilot program for improving processes to resolve complaints by civilian DOD employees by testing procedures that would reduce EEO complaint processing times and eliminate redundancy, among other things. The act requires two reports from GAO--90 days after the first and last fiscal years of the pilot program's operation. In December 2005 and January 2006, we provided briefings on our initial review of the pilot program. This report (1) describes key features and status of the three programs and (2) assesses DOD's plan for evaluating the effectiveness of the pilot program.
section_title: What GAO Found
paragraphs: In August 2004, the Secretary of Defense authorized 2-year programs in (1) Defense Logistics Agency (DLA), (2) the Defense Commissary Agency (DeCA), and (3) components of the U.S. Air Force (USAF) which became operational in fiscal year 2005. While the legislation stated that the pilot program is exempt from procedural requirements of current Equal Employment Opportunity Commission (EEOC) regulations, to a large extent two of the three programs were designed and are operating within existing EEOC requirements, with a specific emphasis on alternative dispute resolution (ADR) as encouraged in DOD's memo soliciting pilot program proposals. ADR techniques include, but are not limited to, conciliation, facilitation, mediation, or arbitration and usually involve the intervention or facilitation by a neutral third party. After the first year, program officials reported low case activity and stated that they plan to request approval from the Secretary to continue their respective programs for a third year. To carry out the programs, officials used similar strategies--outreach to inform eligible staff about the pilot programs, staff training, and the use of electronic data collection--but implemented them differently. Our assessment of DOD's evaluation plan for the pilot program found both strengths and limitations. A sound evaluation plan contains such features as criteria for determining program performance and measures that are directly linked to program objectives. Such key features increase the likelihood that the evaluation will yield sound results, thereby supporting effective program and policy decisions. Lacking these key features, DOD is limited in its ability to conduct an accurate and reliable assessment of the program's results, and Congress is limited in its ability to determine whether features of the overall program have governmentwide applicability. Officials from DOD's pilot program oversight entities have acknowledged shortcomings and have indicated a willingness to modify the plan.

section_title: Letter
paragraphs: Federal employees or applicants for employment who allege that they have  been discriminated against by a federal agency may file a complaint with  that agency. The Equal Employment Opportunity Commission (EEOC) has  established procedures for federal agencies to process equal employment  opportunity (EEO) complaints, including time frames for taking certain  actions, and use of alternative dispute resolution (ADR) programs. The  federal EEO complaint process consists of two stages—informal, or  precomplaint counseling, and formal, when a complaint is filed with the  agency. Delays in the processing of EEO complaints have been a long- standing concern of EEOC, other federal agencies, and Congress. As an  example of such delays, from fiscal years 1993 to 2003, the time the federal  government took to investigate a formal EEO complaint rose from an  average low of 171 days to an average high of 343 days.
In 2000, as part of DOD’s fiscal year 2001 authorization act, Congress  authorized the Department of Defense (DOD) to carry out a 3-year pilot  program for improving processes for the resolution of EEO complaints by  civilian employees of DOD. The legislation provided that the pilot program  was to include procedures to reduce EEO complaint processing times,  eliminate redundancy, reinforce accountability, and provide for early  resolution. While not prescribing details of how the pilots were to be  designed or operated, the authorizing legislation provided that complaints  processed under the pilot program shall be subject to the procedural  requirements established for the pilot program and exempt from the  procedural requirements or other regulations, directives, or regulatory  restrictions prescribed by EEOC. The programs DOD authorized in August  2004 were in (1) the Defense Commissary Agency (DeCA), (2) Defense  Logistics Agency (DLA), and (3) components of the U.S. Air Force (USAF),  and they became operational in fiscal year 2005.
The 2001 authorization act requires GAO to submit two reports on the pilot  program—90 days after the first and last fiscal years of pilot program  operation. We provided briefings on the results of our initial review of  common features and preliminary case activity for the three programs to  interested congressional committees in December 2005 and January 2006.  This report provides additional information by (1) describing key features  and status of the programs and (2) assessing DOD’s plan for evaluating the  effectiveness of the pilot program.
To describe the three programs and their key features, we reviewed  documents provided by DOD officials and interviewed those officials. In  addition, we reviewed information about the federal regulations governing  the federal EEO complaint process and our reports. To assess DOD’s plan  for evaluating the effectiveness of the pilot program, we reviewed the plan  and DOD’s 6- and 9-month evaluation reports. We also interviewed DOD  officials about the evaluation reports. We consulted social science and  evaluation literature, along with our publicized guidance, to identify key  features of an evaluation plan. On the basis of criteria gathered from these  sources, we identified strengths and limitations of DOD’s plan. As the pilot  program and the evaluation efforts were already under way when we began  our review, we focused on the plan’s strengths and those specific  limitations where it would still be possible for DOD to implement  improvements.
Program officials provided data on case activity for the three programs.  The data are preliminary and because we do not use them to develop  findings, we did not conduct a data reliability assessment. Because we did  not assess the reliability of the data, the data are of undetermined  reliability.
We conducted our review in the Washington, D.C., metropolitan area from  August 2005 through March 2006 in accordance with generally accepted  government auditing standards.
subsections: section_title: Background
paragraphs: As of January 1, 2000, all federal agencies covered by EEOC regulations  were required to establish or make available an ADR program for both the  informal and formal complaint stages of the EEO process.
On March 9, 2000, at a joint hearing held by the Subcommittee on Civil  Service of the House Committee on Government Reform and the  Subcommittee on Military Readiness of the House Armed Services  Committee, the Navy discussed the results of its experiences under its 18- month pilot program for resolving EEO complaints through the use of ADR,  which resulted in resolution on an average of 31 days.
The Floyd D. Spence National Defense Authorization Act, for fiscal year  2001, authorized the Secretary of Defense to carry out at least three pilot  programs—one at a military department and two at DOD agencies. The programs were authorized to operate for 3 years. The act exempts the  programs from EEOC’s procedural requirements or restrictions.
In 2004, DOD authorized the following as pilot programs: (1) DLA, which  provides worldwide logistics support—munitions and supplies—for the  missions of military departments; (2) DeCA, which operates a worldwide  chain of commissaries providing groceries to military personnel, retirees,  and their families at a discount; and (3) 31 bases of the USAF, accounting  for about one-third of USAF bases with federal EEO programs. The pilot  programs were authorized by the Secretary for 2 years with an option for  an additional (third) year.
The legislative objectives for the programs are to:  reinforce local management and chain of command accountability, and  provide the parties involved with early opportunity for resolution.
The legislation also provides that pilot program participants voluntarily  participate in the pilot program, and that participants maintain their right  to appeal final agency decisions to EEOC and file suit in federal district  court as is the case in the federal EEO complaint process.
The Office of the Deputy Undersecretary of Defense for Civilian Personnel  Policy, the Office of the Deputy Undersecretary for Equal Opportunity, and  the Office of Complaint Investigations within the Civilian Personnel  Management Service have ongoing responsibility for oversight, monitoring,  and evaluation of the overall pilot program.
Under EEOC regulations, during the informal, or precomplaint counseling  stage, ADR techniques can be used. Counselors are to advise individuals  that, when the agency agrees to offer ADR in the particular case, they may  choose to participate in either counseling or in ADR. If the matter is not  resolved by counseling or if ADR is unsuccessful, the counselor is required  to inform the employee in writing of his or her right to file a formal  discrimination complaint with the agency. ADR can also be used after an  agency receives a formal complaint.
After a complainant files a formal discrimination complaint, the agency  must decide whether to accept or dismiss the complaint and notify the  complainant. If the agency dismisses the complaint, the complainant can  appeal the dismissal to EEOC. If the agency accepts the complaint, it has  180 days to investigate the accepted complaint and provide the  complainant with a copy of the investigative file. Within 30 days of receipt  of the copy of the investigative file, the complainant must choose between  requesting (1) a hearing and decision from an EEOC administrative judge  (AJ) or (2) a final decision from the agency. When a hearing is not  requested, the agency issues a final decision. A complainant may appeal an  agency’s final decision to EEOC.
In cases where a hearing is requested, the AJ has 180 days to issue a  decision and send the decision to the complainant and the agency. If the AJ  issues a finding of discrimination, he or she is to order appropriate relief.  After the AJ decision is issued, the agency can issue a final order notifying  the complainant whether or not the agency will fully implement the  decision of the AJ, and the employee can file an appeal with EEOC. If the  agency issues an order notifying the complainant that the agency will not  fully implement the decision of the AJ, the agency also must file an appeal  with EEOC at the same time. See appendix I for more details and  associated time frames related to the EEO complaint process.
section_title: The Three Programs Emphasize ADR Techniques, Share Common Implementation Strategies, and Report Low Case Activity
paragraphs: Although features of the three programs vary by agency and focus on  different stages of the complaint process, they all emphasize the use of  ADR techniques available under the current federal EEO process. They  also share common implementation strategies, including outreach to  eligible staff to inform them about the programs, staff training, and  electronic data collection. In its 9-month evaluation, DOD observed that  pilot program activity had been lower than anticipated; DOD did not  provide a baseline for its comparison or elaborate on the reason for this  occurrence. After 12 months, program officials continue to report low case  activity.
subsections: section_title: DOD’s Pilot Program Emphasizes ADR Techniques
paragraphs: In developing the overall EEO pilot program, DOD allowed DLA, DeCA,  and USAF to determine their individual program design. However, in its  memo soliciting pilot program proposals, DOD encouraged potential  participants to work with the Office of Complaint Investigations to develop  the format and content of their proposals, offering the assistance of the  Office’s experienced staff of certified complaint investigators and  mediators with success in using ADR techniques. Two of the programs— DLA and DeCA—emphasize the use of ADR in the informal stage,  consistent with federal EEO regulations. Program officials said that their  programs are attempting to address the legislative objective of providing  early opportunity for resolution by focusing on ADR. The third program, in  selected bases of the USAF, changes the formal stage of the federal EEO  process by combining the investigative and hearing phases after a  complainant has filed a formal complaint. This program also emphasizes  the use of ADR techniques both during the informal stage as well as at the  time a complainant files a formal complaint.
subsections: section_title: DLA’s Pilot Program
paragraphs: DLA’s program, Pilot for Expedited Complaint Processing (PECP), began in  October 2004 at DLA headquarters in Fort Belvoir, Va. DLA considers several types of cases, such as those that challenge government policy,  inappropriate for PECP and screens them out.
The PECP process is similar to the informal stage of the current EEO  process. DLA officials said the PECP process has three steps. The first step  occurs when an employee who believes he or she has been discriminated  against makes initial contact with DLA’s EEO office. An EEO Intake  Specialist collects specific information about the employee’s concerns and  drafts an intake report, which includes a description and basis of the claim.  The EEO Intake Specialist advises the employee orally and in writing about  (1) PECP and how it compares to the federal counseling process and (2)  the employee’s right to opt out of the pilot program at any time before the  filing of a formal complaint. The second step begins when the employee  chooses to participate in PECP. At this time, the EEO Intake Specialist  discusses and offers the employee ADR. The EEO Intake Specialist also  informs the employee that participating in ADR is optional and can be used  at any stage of the complaint process. The EEO Intake Specialist considers  two methods of ADR— mediation or facilitation. Mediation is the primary  method used by PECP. According to DLA, the method of ADR used is  based on the employee’s claim and the EEO Intake Specialist’s assessment  of the method that would more likely encourage communication between  the employee and management and resulting resolution.
Under the third step, ADR takes place. DLA pilot program officials  acknowledged that the pilot program’s ADR features do not differ from  those offered under the current EEO process. DLA has an ADR program  called Reach Equitable Solutions Voluntarily and Easily (RESOLVE), which  is used when mediation is offered. RESOLVE is managed by DLA’s General  Counsel. According to DLA officials, RESOLVE mediators cannot mediate  precomplaints or complaints involving organizations they may service in  another capacity, thus ensuring the neutrality of the mediator.
section_title: DeCA’s Pilot Program
paragraphs: DeCA’s program, Early Resolution Opportunity (ERO), began in February  2005 and covers 23 stores in three zones (DeCA West Zone 16-San Diego,  Calif.; DeCA East Zone 28-Virginia Beach, Va; and DeCA East Zone 6-San  Antonio, Tex.). Using ADR techniques, ERO seeks to provide early  resolution opportunities, because according to DeCA, ineffective  communication between employees and supervisors or managers often  results in perceptions of discrimination. Moreover, DeCA believes that  disputes can be resolved before they enter the informal counseling stage if  a trained EEO facilitator can intervene to negotiate resolution. Cases that  involve alleged violent acts, theft, sexual harassment, termination, or may  be precedent setting, are ineligible for ERO.
ERO is divided into two steps. In the first step, a trained DeCA facilitator  attempts to resolve a claim before the start of the informal stage of the  current process. Employees at stores participating in ERO can call a toll- free number to discuss their concerns with a trained facilitator. For  example, an employee could call about perceived discrimination over  schedule changes, and the facilitator may discuss what had occurred and  rationale for schedule changes (e.g., to cover absences). According to  DeCA officials, some employees “self screen” during the facilitation  process, deciding not to pursue an EEO complaint or to pursue another  avenue, such as the negotiated grievance process.
The second step of ERO, which follows if facilitation is unsuccessful in  resolving the employee’s concerns, involves calling in a third-party  mediator. According to a DeCA official, DeCA uses mediators from DOD’s  Office of Complaint Investigations, because they are trained, experienced  ADR professionals, and have a greater perception of neutrality as they do  not work for DeCA.
If mediation fails, an individual may choose to file a formal complaint.  According to a DeCA official, ERO seeks to reduce the processing time of  the formal stage. To help achieve this goal, DeCA reduces processing times  for two phases of the formal stage of the complaint process: (1) after a  complainant files a formal complaint, DeCA has set a goal in ERO of   14 days to accept, partially accept, or dismiss it; and (2) after the report of  investigation is completed, DeCA sends a notice informing the complainant  that he or she has 7 days to either request a hearing or a final agency  decision, reducing the time from 30 days under EEOC regulations. In  addition, to further reduce processing time for ERO cases, paper  documents are replaced with electronic files. Finally, according to a DeCA  official, officials from DeCA and the Office of Complaint Investigations can  download relevant case documents from a secure shared drive for  complaints filed under both ERO and under the current EEO process.
section_title: USAF’s Pilot Program
paragraphs: USAF’s program, called Compressed Orderly Rapid Equitable (CORE),  focuses on the formal phase of the EEO complaint process. The program  began January 1, 2005, at 29 continental U.S. sites and 2 overseas offices  that we refer to as test bases. Although the 31 test bases account for less  than one-third of all USAF bases with EEO programs, they produce over 80  percent of all USAF EEO complaints. Cases that involve class and mixed- case complaints or cases related to claims already accepted under the  current federal EEO complaint process are not eligible to participate in  CORE.
CORE has a two-step process that begins at the time the complainant files a  formal complaint. Until a complaint is filed, USAF officials attempt early  resolution of allegations of discrimination in the informal stage using the  current federal EEO process. If resolution is not achieved during this  stage, the complainant must choose between CORE and the current federal  EEO process. The first step of CORE involves mediation. If the  complainant declines mediation or mediation is unsuccessful, step two  begins, and a CORE Fact-Finding Conference is conducted. USAF defines  this conference as a “non-adversarial, impartial fact-gathering procedure.”  The conference is conducted by a CORE fact-finder, provided by the Office  of Complaint Investigations. During the conference, the fact-finder hears  testimony from witnesses and receives documentary evidence; also at this  time, a verbatim transcript is taken by a certified court reporter. Following  the conference, the fact-finder completes the record of the complaint and  recommends a decision in the case to the director of the USAF Civilian  Appellate Review Office. The director of the USAF Civilian Appellate  Review Office may accept, reject, or modify the fact-finder’s recommended  decision. The director then prepares a final agency decision for signature  by the director of USAF Review Boards Agency. The director of USAF  Review Boards Agency issues the final agency decision. Any further action  on the complaint, including rights to appeal to EEOC and file a lawsuit, are  governed by current federal EEO complaint procedures.
According to USAF, by combining the investigative and hearing phases of  the current federal EEO complaint process, USAF aims to issue a final  agency decision within 127 days or less of filing the formal complaint; the  current process can take up to 360 days plus another 70 days to provide the  complainant and the agency their allotted time for decision making. USAF  officials also indicated that through the CORE Fact-Finding Conference,  each complainant gets their “day in court,” whereas under the current EEO  process, complainants often wait months to request a hearing and can have  their complaint dismissed by an EEOC AJ without a hearing.
section_title: DOD’s EEO Pilot Programs Share Common Implementation Strategies
paragraphs: The three programs share common implementation strategies but  implement them differently. In our review of the programs and subsequent  discussions with DOD and program officials, DeCA and USAF conducted  some level of outreach to program-eligible employees to inform them about  the programs. For example, DeCA officials went to participating stores and  handed out brochures describing ERO. According to USAF program  officials, outreach on CORE included sending a letter to all participating  bases from the Chief of Staff for Personnel as well as a notice to the unions.  Additionally, CORE was publicized in USAF news service and  governmentwide media.
We also found that agencies varied in how they trained their EEO  employees about the programs. USAF officials used contractors to train  some employees in CORE over a 1-week period; in turn, those employees  trained others. DLA officials had informal in-house employee training.  DeCA sent EEO officials and an attorney from its headquarters trained in  ERO to each of its three zones to train EEO managers as well as managers  and supervisors at its 23 stores.
Finally, all three programs used electronic data collection for tracking and  monitoring, with each program developing its own electronic data  collection method. For example, USAF uses EEO-Net system and software  to collect program data. USAF also uses USAF-specific software, the Case  Management and Tracking System, to manage the EEO process, including  CORE, and an electronic case identifier to mark CORE cases to help in  monitoring those program cases that reach EEOC on appeal. DeCA  currently uses an Access database to track ERO activity, and DLA uses an  Excel spreadsheet to track PECP activity.
Officials from both the programs and DOD’s EEO pilot program oversight  entities have indicated their willingness to share information. As we have  previously reported, by assessing their relative strengths and limitations  through collaboration, agencies can look for opportunities to address  resource needs by leveraging each others’ resources and obtaining  additional benefits that would not be available if they were working  separately. While the focus of our earlier work was on coordination  between agencies from different departments, the findings would also be  applicable to agencies within a department that are engaged in similar  activities.
section_title: DOD’s EEO Pilot Programs Report Low Case Activity
paragraphs: In its 9-month evaluation report, DOD stated that program activity for all  three programs had been lower than anticipated. At the end of the first  year, program officials reported continued low program activity. However,  in its report, DOD did not provide a baseline for its comparison or  elaborate on the reason for this occurrence. Instead, DOD’s evaluation plan  states that data collected during the pilot program are to be measured  against fiscal year 2004 baseline data. Therefore, we are including fiscal  year 2004 data as reported to EEOC for each program for comparison  purposes. Since many cases are still going through the program process, for  comparison, we report only the number of initial contacts or formal  complaints.
According to DeCA officials, from January 1, 2005, through January 31,  2006, 42 employees contacted DeCA’s EEO office; of those, 41 were offered  participation in ERO, and all opted for ERO. Of those who completed ERO,  16 did so with resolution; 9 did so without resolution, and 14 are still in process. Data are not available for DeCA test stores for fiscal year 2004, the  year before ERO was implemented.
According to DLA officials, from January 1, 2005, through January 31, 2006,  15 employees contacted DLA’s EEO office; of those, 13 were offered  participation in PECP, and 12 opted for it. Of those who completed PECP,  10 did so with resolution; 1 declined participation, and 1 withdrew the  precomplaint; 1 opted out of PECP. For fiscal year 2004, the year before  DLA implemented PECP, 26 employees contacted DLA’s headquarters EEO  office.
According to USAF officials, from January 1, 2005, through January 31,  2006, a total of 634 formal complaints were filed USAF-wide. The CORE  process was available to 534 of the complainants. Of those complainants  offered CORE, 104 opted to process their complaint using CORE. Of these  104, 63 have been closed with resolution, and 28 CORE cases are still in  progress. Thirteen complainants opted out of CORE and chose to return to  the current EEO process. For fiscal year 2004, the year before USAF  implemented CORE, 667 formal complaints were filed USAF-wide; of these,  488 were filed at what are now CORE test sites.
DOD’s 9-month report stated that case activity was lower than expected. As  a result of the low case activity, program officials have said they will seek  to extend their respective programs for an additional (third) year.  According to the authorizing memo from DOD implementing the pilot  program, in April 2006 program officials can request to extend the pilot  program for a third year. At the time of this report, DeCA and USAF had  made requests of DOD to extend the operation of their pilot programs for a  third year.
section_title: Although Containing Some Strengths, Limitations in Its Evaluation Plan Will Hinder DOD’s Ability to Assess Pilot Program Results
paragraphs: Our initial assessment of DOD’s evaluation plan for the pilot program found  both strengths and limitations. One strength of the plan was the inclusion  of forms for collecting baseline data (before the programs began) and pilot  program data, which provides a tool for the pilot programs to measure  some aspects of their progress. Although DOD developed an evaluation  plan for the overall pilot program, the plan lacked some key features of a  sound evaluation plan, including measures that are directly linked to the  program objectives, criteria for determining pilot program performance,  and an appropriate data analysis plan for the evaluation design. Without  such features, DOD will be limited in its ability to conduct an accurate and  reliable assessment of the programs’ results. In addition, the lack of  established key evaluation features in DOD’s plan increases the likelihood  of insufficient or unreliable data, further limiting confidence in pilot  program results. Without confidence in pilot program results, DOD will be  limited in its decision making regarding this pilot program, and Congress  will be limited in its decision making about the pilot program’s potential  broader application.
Officials from DOD’s pilot program oversight entities have acknowledged  shortcomings and have indicated a willingness to modify the plan.
subsections: section_title: Strengths of DOD’s Evaluation Plan
paragraphs: Considering the evaluation plan itself and interviews with DOD officials,  we found that DOD’s plans for assessing the pilot programs had some  strengths, including:   Forms in the evaluation plan for collecting baseline data (before  the pilot programs began) and pilot program data. According to  the evaluation plan, baseline data from fiscal year 2004 are recorded on  a template (i.e., a modified version of EEOC Form 462) appropriate to  the part of the complaint process the pilot program focuses on. Data  collected during the programs will be measured against the baseline  data collected in the prior year’s EEOC Form 462. Pilot program and  nonpilot program data are to be collected by an Individual Data Report  form, which is to collect processing-time data, comparative information  on early ADR, and early management involvement in cases at each pilot  program site. Comparing data from the modified EEOC Form 462 to  data from the program as well as to nonprogram cases is expected to  help DOD determine whether processing times and redundancy were  reduced concerning early resolution and streamlining as a result of the  pilot program.
Detailed time frames, roles and responsibilities, and report  planning in the evaluation plan. The evaluation plan includes a  schedule that details tasks, roles and responsibilities, and milestones for  completing set tasks in evaluating the pilot program. This schedule  provides a framework that is organized and easy to follow.
Inclusion of reasonable research design. The evaluation plan  includes a reasonable method for assessing pilot program results.  Because the pilot program legislation mandates voluntary participation  in the program, DOD was restricted from one form of design (i.e.,  randomly assigning employees alleging or filing complaints of  discrimination to participate in the pilot program). As a result, DOD  chose to compare prepilot and postpilot program data as well as pilot  and nonpilot program cases.
In addition, DOD officials said that the plan can be adjusted to the extent  feasible to ensure that the data collected are sufficient for evaluating the  pilot program.
section_title: Without Key Evaluation Plan Features, DOD Will Be Limited in Its Ability to Assess Pilot Programs’ Results
paragraphs: DOD’s plan for evaluating the effectiveness of the pilot program lacks some  key features that are essential to assessing performance. Well-developed  evaluation plans, which include key evaluation features, have a number of  benefits, perhaps most importantly, increasing the likelihood that  evaluations will yield methodologically sound results, thereby supporting  effective program and policy decisions. The lack of established key  evaluation features in DOD’s plan increases the likelihood of insufficient or  unreliable data, limiting confidence in pilot program results. Without  confidence in pilot program results, DOD will be limited in its decision  making regarding this pilot program, and Congress will be limited in its  decision making about the pilot program’s potential broader application.
Some key features of a sound evaluation plan include: well-defined, clear, and measurable objectives;  measures that are directly linked to the program objectives;  criteria for determining pilot program performance;  a way to isolate the effects of the pilot programs;  a data analysis plan for the evaluation design; and  a detailed plan to ensure that data collection, entry, and storage are  reliable and error-free.
DOD’s evaluation plan contains the following limitations:  The objectives in DOD’s evaluation plan are not well defined or clear,  which makes measurement problematic. For example, the evaluation  plan identifies management accountability as an objective without  defining it, who it applies to, and how it will be measured. Without well- defined, clear, and measurable objectives, the appropriate data may not  be collected, thus hindering the assessment of pilot program progress.
DOD’s data collection efforts are not linked to objectives in the  evaluation plan. For example, the evaluation plan contains a variety of  surveys that the individual pilot programs can use to measure customer  satisfaction, but customer satisfaction is not included in the evaluation  plan as an objective of the plan. Directly linking objectives and  measures is a key feature of an evaluation plan. Without such linkage,  data collection efforts may not directly inform stated objectives, and in  turn, may not inform the evaluation effort.
DOD’s evaluation plan does not establish standards for evaluating pilot  program performance. For example, DOD’s plan does not state the  amount or type of change required to indicate that a pilot program has  succeeded in reducing processing time. Without targets or standards for  determining success, it will be difficult to determine if the pilot program  was effective.
DOD’s evaluation plan does not mention controlling for possible  outcomes that are attributable to factors other than the effects of the  programs. A preferred research method is to use random assignment of  program participants to provide greater confidence that results are  attributable to a program. As we mentioned, DOD was restricted from  randomly assigning employees alleging or filing complaints of  discrimination to participate in the pilot program. As a result, other  factors, such as the type of complaint, complainant, or the mediator may  affect pilot program outcomes. Establishing controls for such factors  could help isolate the effects attributable to the pilot programs. When an  evaluation design involves, for example, a comparison between prepilot  and postpilot program conditions, the research design should include  controls to ensure that results will be attributable to the pilot program  and not to other factors.
DOD’s evaluation plan does not explain how the data will be analyzed.
Although the evaluation plan has templates for collecting data, including  pilot program baseline data, individual data reports, and various  surveys, it does not state how the data collected will be analyzed. A data  analysis plan is a key feature of an evaluation plan as it sets out how  data will be analyzed to determine if program objectives have been met.  Without a data analysis plan, it is not clear how the data will be analyzed  to inform the objectives of the evaluation and assess the performance of  the programs.
DOD’s plan does not explain how the integrity of the data collected will  be ensured. A detailed plan to ensure that data collection, entry, and  storage are reliable and error-free is a key feature of an evaluation plan  that gives greater confidence to data quality and reliability and to any  findings made from these data. Without a detailed plan to ensure that  data collection, entry, and storage are reliable and error-free, confidence  in pilot program results will be limited.
section_title: Conclusions
paragraphs: All three programs share a common feature of emphasizing the use of ADR  to meet the legislative mandate to improve the efficiency of the EEO  complaint process. In addition, although authorized to operate outside of  current EEOC regulations, to a large extent, two of the three programs  have been designed by DOD to operate within the requirements of current  regulations. While sharing common strategies in such areas as electronic  data collection, the pilot programs implemented them differently. As the  challenges of the 21st century grow, it will become increasingly important  for DOD to consider how it can maximize performance and results through  the improved collaboration of its organizations. Officials from the  programs and DOD’s EEO pilot program oversight entities have indicated  their willingness to share information and strategies.
To better ensure that it will provide useful results, DOD needs to make  changes to its evaluation plan. Although DOD’s evaluation plan had some  strengths, the plan’s shortcomings may impede DOD’s ability to produce  sound results that can inform both program and policy decisions regarding  the overall pilot program. The lack of key evaluation features, such as clear  and measurable objectives, measures linked to these objectives, and  established criteria for determining pilot program performance may limit  confidence in pilot program results.
section_title: Recommendations for Executive Action
paragraphs: To improve the performance and results of the pilot program, we  recommend that the Secretary of Defense direct the Deputy  Undersecretary of Defense for Civilian Personnel Policy, the Deputy  Undersecretary for Equal Opportunity, and the Civilian Personnel  Management Service to take the following actions: Establish regular intra-agency exchanges of information on outreach  strategies, training, and electronic data collection from which the pilot  programs could achieve potential benefits that would not be available if  working separately.
Develop a sound evaluation plan to accurately and reliably assess the pilot  programs’ results, including such key features as  well-defined, clear, and measurable objectives;  measures that are directly linked to the program objectives;  criteria for determining pilot program performance;  a way to isolate the effects of the pilot programs;  a data analysis plan for the evaluation design; and  a detailed plan to ensure that data collection, entry, and storage are  reliable and error-free.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Secretary of Defense for his review  and comment. The Principal Deputy Undersecretary of Defense provided  written comments, which are included in appendix II.
DOD generally agreed with our recommendations. Regarding the  establishment of regular intra-agency exchanges of information among the  pilot programs to leverage potential benefits, DOD stated that it will hold  quarterly meetings with pilot program managers. Concerning the  development of an evaluation plan that accurately and reliably assesses the  pilot programs’ results, DOD partially concurred with the recommendation  and stated that it would consider and incorporate the recommended key  features into the evaluation plan as appropriate. However, DOD also stated  that the purpose of the plan was to assist pilot program evaluators in their  work by specifying those procedures, tools, and objectives that would be  unique to the pilot programs. In its comments, DOD reasons that because  all pilot program officials agreed on a particular objective, which was  common to both the pilot and traditional EEO complaint procedures, that it  was not necessary to link data collection efforts to that objective or  incorporate either the objective or the data collection effort in the  evaluation plan. Because the plan is the long-term guide for the pilot  program evaluation process and because staff changes occur, it is  important that DOD include all objectives and methods they intend to use  in the plan, allowing the evaluation process to be more transparent and  provide clearer guidance to the pilot program officials on evaluation  procedures.
In its response, DOD also commented on our observation that to a large  extent two of the three pilot programs were designed and are operating  within existing EEOC requirements. DOD noted that this was due in large  part to a presidential memorandum issued when the legislation was signed.  The memorandum, which addressed the implementation of the pilot  program, required that a complaining party be allowed to opt out of the  pilot program at any time. According to DOD, adhering to this requirement  necessitated using a similar design to the current EEO process so that  complaining parties who decided to opt out would not be penalized by  having to start at the very beginning of the current EEO complaint process.  It is not clear to us that ensuring the ability to opt out at any point  necessitates returning the complaining party to the very beginning of the  current EEO process in all cases. Rather, the complaining party would be  returned to the current EEO process at an appropriate point based on what  was achieved through the pilot program process. Overall, we see nothing in  the presidential memorandum that would limit DOD’s legitimate use of the  procedural flexibility granted by Congress through the pilot program  authority.
We will send copies of this report to other interested congressional parties,  the Secretary of Defense, and the Chair of EEOC. We also will make copies  available to others upon request. In addition, the report is available on  GAO’s home page at http://www.gao.gov.
If your staff have questions about this report, please contact me on (202)  512-9490. Key contributors to this report are listed in appendix III.
section_title: EEO Laws and Regulations Applicable to Federal Employees
paragraphs: Title VII of the Civil Rights Act of 1964, as amended, makes it illegal for  employers, including federal agencies, to discriminate against their  employees or job applicants on the basis of race, color, religion, sex, or  national origin. The Equal Pay Act of 1963 protects men and women who  perform substantially equal work in the same establishment from sex- based wage discrimination. The Age Discrimination in Employment Act of  1967, as amended, prohibits employment discrimination against individuals  who are 40 years of age or older. Sections 501 and 505 of the Rehabilitation  Act of 1973, as amended, prohibit discrimination against qualified  individuals with disabilities who work or apply to work in the federal  government. Federal agencies are required to provide reasonable  accommodation to qualified employees or applicants for employment with  disabilities, except when such accommodation would cause an undue  hardship. In addition, a person who files a complaint or participates in an  investigation of an equal employment opportunity (EEO) complaint or who  opposes an employment practice made illegal under any of the  antidiscrimination statutes is protected from retaliation. The Equal  Employment Opportunity Commission (EEOC) is responsible for enforcing  all of these laws.
Federal employees or applicants for employment who believe that they  have been discriminated against by a federal agency may file a complaint  with that agency. The EEOC has established regulations providing for the  processing of federal sector employment discrimination complaints. This  complaint process consists of two stages, informal, or precomplaint  counseling, and formal. Before filing a complaint, the employee must  consult an EEO counselor at the agency in order to try to informally resolve  the matter. The employee must contact an EEO counselor within 45 days of  the matter alleged to be discriminatory or, in the case of a personnel action,  within 45 days of the effective date of the action. Counselors are to advise  individuals that, when the agency agrees to offer alternative dispute  resolution (ADR) in the particular case, they may choose to participate in  either counseling or in ADR.
Counseling is to be completed within 30 days from the date the employee  contacted the EEO office for counseling unless the employee and agency  agree to an extension of up to an additional 60 days. If ADR is chosen, the  parties have 90 days in which to attempt resolution. If the matter is not  resolved within these time frames, the counselor is required to inform the  employee in writing of his or her right to file a formal discrimination  complaint with the agency. The written notice must inform the employee of  the (1) right to file a discrimination complaint within 15 days of receipt of  the notice, (2) appropriate agency official with whom to file a complaint,  and (3) duty to ensure that the agency is informed immediately if the  complainant retains counsel or a representative.
After a complainant files a formal discrimination complaint, the agency  must decide whether to accept or dismiss the complaint and notify the  complainant. If the agency dismisses the complaint, the complainant has 30  days to appeal the dismissal to EEOC. If the agency accepts the complaint,  it has 180 days to investigate the accepted complaint and provide the  complainant with a copy of the investigative file. Within 30 days of receipt  of the copy of the investigative file, the complainant must choose between  requesting (1) a hearing and decision from an EEOC administrative judge  (AJ) or (2) a final decision from the agency. When a hearing is not  requested, the agency must issue a final decision within 60 days. A  complainant may appeal an agency’s final decision to EEOC within 30 days  of receiving the final decision.
In cases where a hearing is requested, the AJ has 180 days to issue a  decision and send the decision to the complainant and the agency. If the  AJ issues a finding of discrimination, he or she is to order appropriate  relief. After the AJ decision is issued, the agency has 40 days to issue a final  order notifying the complainant whether or not the agency will fully  implement the decision of the AJ, and the employee has 30 days to file an  appeal with EEOC of the agency’s final order. If the agency issues an  order notifying the complainant that the agency will not fully implement  the decision of the AJ, the agency also must file an appeal with EEOC at the  same time. Parties have 30 days in which to request reconsideration of an  EEOC decision. Figure I illustrates the EEO complaint process.
If a complaint is one that can be appealed to the Merit Systems Protection  Board (MSPB) such as a removal, reduction in grade or pay, or suspension  for more than 14 days, the complaint is a “mixed-case complaint.” EEOC  regulations provide that an individual may raise claims of discrimination in  a mixed case, either as a mixed-case EEO complaint with the agency or a  direct appeal to MSPB, but not both.
A complainant may file a civil action in federal district court at various  points during and after the administrative process. The filing of a civil  action will terminate the administrative processing of the complaint. A  complainant may file a civil action within 90 days of receiving the agency’s  final decision or order, or EEOC’s final decision. A complainant may also  file a civil action after 180 days from filing a complaint with his or her  agency, or filing an appeal with EEOC, if no final action or decision has  been made.
section_title: Comments from the Department of Defense
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgements
paragraphs: In addition to the individual named above, Belva M. Martin, Assistant  Director; Karin K. Fangman; Cindy Gilbert; Emily Hampton-Manley;  Anthony Patterson; Rebecca Shea; Linda Sidwell (detailee); and Kiki  Theodoropoulos made key contributions to this report.