A Checklist for Evaluating Federal ADR Programs: Long Form1
 

I.	 Pre-design Planning and Preparation 

A. Determine goals and objectives for the evaluation 

1.	 Determine goals and objectives that are sensitive to the needs and 

interests of those requesting the evaluation and the audience. 

2.	 Link goals and objectives of the evaluation closely with the goals and 

objectives of the program being evaluated. 

a.	 What were the original program goals and objectives? 

b.	 Is the program, as implemented, consistent with the original 

design? 

3.	 Use performance indicators to help define or select ADR program 

goals2

. 

a.	 How well is the program working? 

b.	 Should changes be made? 

c.	 Should the program be continued or expanded? 

d.	 How well does ADR work in particular federal contexts? 

B. Identify the audience for the evaluation, and how best to 

meet their needs 

1.	 Identify potential audiences. 

a.	 Is the program reaching its intended audiences? 

b.	 Are all potential audiences available to participate in the 

evaluation? 

2.	 Identify issues that each audience is interested in and wants 

addressed. 

a.	 ADR program officials 

i.	 How is the program working, and how might it be improved? 

ii.	 What is the Impact on case inventory (backlogs)? 

iii.	 What is the effect on long-term relationships among 

disputants? 

iv.	 How well is information about the program being dispersed? 

1

 Based onBased on Evaluating ADR Programs: A Handbook for Federal Agencies, Evaluating ADR Programs: A Handbook for Federal Agencies, Administrative
 

Conference of the United States Dispute Systems Design Working Group, March 1995.
 2 Performance Indicators for ADR Program Evaluation (November 1993), Administrative (November 1993), Administrative
 

Conference's Dispute Systems Design Working Group. 1 
b.	 Other agency officials 

i.	 Budget officers - What is the cost? What are the savings? 

ii.	 Offices of General Counsel - How much time was needed? 

What is the nature of the outcomes? 

iii.	 Inspector General - What are the settlement issues? Is 

long-term compliance promoted? 

iv.	 Other Managers - How effectively was the program 

implemented? 

c.	 Members of Congress 

i.	 How does ADR use affect budgets? 

ii.	 How are related laws being implemented? 

d.	 Public 

i.	 How efficiently is the agency resolving disputes? 

ii.	 Are participants satisfied? 

e.	 Program users 

i.	 How typical was their experience compared to other users? 

C. Determine how evaluation results will be used 

1.	 To measure program effectiveness (impact, outcome, summative 

evaluations): 

a.	 Is the program meeting its goals? 

b.	 Is it having the desired impact? 

c.	 Should the program be continued or expanded? 

2.	 Program design and administration (process or formative evaluations): 

a.	 How is the program operating? 

b.	 How can a continuing program be improved? 

D. Consider issues of timing and expense 

1.	 Determine when a program evaluation should be undertaken. 

a.	 Consider these factors: 

i.	 Whether the program has been in operation long enough to 

ensure that there is enough data to examine 

ii.	 Whether the program has resolved start up problems 

2 
iii.	 Pilot - whether the evaluation will be completed early enough 

to be a factor in the decision to continue or expand the 

program 

iv.	 Whether there are other deadlines relating to future 

decision-making which will affect the usefulness of 

evaluation results 

b.	 An evaluation measuring a program's effectiveness should wait 

until possible impact is measurable. 

c.	 An evaluation of a program design can reasonably be done 

anytime, but it is best if start up problems have been eliminated. 

2.	 Determine how much the evaluation will cost. 

a.	 Consider these factors: 

i.	 Number and complexity of performance indicators 

ii.	 Type of design 

iii.	 Level of statistical significance required of results 

iv.	 Availability of acceptable data for comparison purposes 

v.	 Who is selected to carry out the evaluation 

b.	 Control costs through careful planning and appropriate 

adjustments in the design phase. 

E. Select an evaluator 

1.	 Select an evaluator with certain qualifications. 

a.	 Objectivity 

b.	 Experience in conducting evaluations 

c.	 Technical expertise in data collection and analysis 

d.	 Understanding of the organization or context in which the program 

operates 

e.	 Good interpersonal and management skills 

2.	 From several possibilities, choose an evaluator that best fits your 

needs. 

a.	 An outside evaluator 

i.	 Someone outside the agency 

ii.	 Greatest potential for impartiality 

iii.	 Relatively expensive 

3 
b.	 Someone within the agency, but outside the program 

i.	 An evaluation capacity within the agency 

ii.	 Neutrality at a potentially lower cost 

c.	 Someone involved with the ADR program 

i.	 People involved with program implementation or design 

ii.	 Least expensive, best understanding of program context 

iii.	 Potential perceptions of a lack of impartiality 

d.	 Team representing internal and external groups 

i.	 Avoids disadvantages of other options 

3.	 Create an advisory committee. 

a.	 Informal group of stakeholders 

b.	 Sounding board on design and implementation issues, helps refine 

focus, and reviews and comments on methodology 

4.	 Use a liaison from within program to work with the evaluator to ensure 

access to necessary information. 

II. Designing the Evaluation 

A. Gain a thorough understanding of the ADR program design 

and operation 

1.	 Review documents and interview key program managers and 

stakeholders. 

2.	 Ask questions that will help you to gain an understanding the program. 

a.	 What are the program goals and objectives? 

b.	 How are goals defined and prioritized? 

c.	 Can program goals be measured? 

d.	 Have measurable indicators been established? 

e.	 Are data already being collected to measure those objectives? 

f.	 Who are the program participants? 

g.	 How are cases referred to the program? 

h.	 What functions and responsibilities are given to participants? 

i.	 What constraints are there to program operation? 

j.	 What limits are put on the program or participants? 

k.	 What funding and staffing resources are available and being used? 

l.	 What type of training is provided for staff and participants? 

m. Are there written policies and procedures for program operation? 

4 
 

n.	 What types of program data are available and can they be used? 

o.	 What additional information is needed? 

p.	 What is the estimated time and cost of gathering additional 

information? 

B. Translate evaluation goals and objectives into measurable 

performance indicators3 

C. Determine appropriate design strategy 

1.	 Case studies: 

a.	 Focus on single, cohesive groups or cases in which ADR was 

elected 

b.	 More descriptive 

c.	 Comparisons can be made to similar situations where ADR was not 

used. 

d.	 Inferential and hold no scientific weight 

2.	 Time series designs: 

a.	 Collect information about a particular group over periods of time 

b.	 No separate comparison group. 

3.	 Comparison group designs: 

a.	 Divide individual persons or groups into different groups at the 

beginning of the study 

i. Experimental group - those participating in the program 

ii. Control group - those not participating in the program 

b.	 Useful in determining whether outcomes are the result of ADR or 

an outside cause 

c.	 "True experimental" design 

i.	 random assignment of individuals and cases to experimental 

or control group 

ii.	 best way to ensure outcomes are a result of the program 

iii.	 not always possible 

d.	 "Quasi-experimental" design 

i.	 non-random assignment to groups 

ii.	 use naturally occurring groups - cases disposed of before 

the ADR program began (control) compared with cases 

3

 Refer toRefer to Performance Indicators for ADR Program Evaluation (November 1993), Administrative (November 1993), Administrative 

Conference's Dispute Systems Design Working Group. 5 
 

handled under the ADR program (experimental) 

D. Determine data needs and availability of data 

1.	 Consider how data will be analyzed to ensure proper data collection. 

2.	 Determine what types of data are needed. 

a.	 Program Data: 

i.	 Design information 

ii.	 Program Operation 

b.	 Comparison Data: 

i.	 The agency without ADR (cost and time) 

ii.	 Use a control group 

c.	 Information relating to possible other causes: 

i.	 Changes in agency policy or regulations 

ii.	 Changes in economic environment 

iii.	 Particular skill of participating individuals 

E. Decide how to collect the data4 

1.	 Use documents as a source of data. 

a.	 Identify types of documents and the data to be obtained from them: 

Plans, Procedures, Case files, Accounting records, Budget, 

documents, Contracts, Data bases, Existing studies, reports, 

and secondary analyses of the ADR program 

b.	 Consider whether all records or a sample will be reviewed 

c.	 Develop checklists, coding sheets, or standardized forms to help 

simplify reviews 

2.	 Use people as a source of data. 

a.	 Identify potential sources: 

Agency and program staff, program participants, subject 

matter experts, and stakeholders 

4

 For examples of surveys and questionnaires for use by federal agencies in ADR program For examples of surveys and questionnaires for use by federal agencies in ADR program 

evaluations, refer to the prototype data collection materials prepared by RAND for the evaluations, refer to the prototype data collection materials prepared by RAND for the

Administrative Conference Administrative Conference 6 
b.	 Decide on a method for obtaining information from people 

i.	 Standardized interviews 

ii.	 Informal conversational interviews 

iii.	 Telephone interviews 

iv.	 Focus group interviews 

v.	 Participant surveys 

vi.	 Mail surveys 

3.	 Directly observe the program. 

III.Implementing the Design 

A. Collect the data 

1.	 Select people to survey based on their position or role in the program. 

a.	 Key informants with special program or subject knowledge 

b.	 All affected people 

c.	 Sample - must be representative of the larger group 

i.	 Randomly selected based on a listing 

ii.	 Purposefully selected based on a characteristic or being a 

representative of a subgroup or a certain viewpoint 

2.	 Ensure the collection of good quality data. 

a.	 Reliable data: Make sure that repeated uses of the evaluation tool 

(survey question) yield basically the same data 

b.	 Valid data - Make sure that the evaluation tool generates results 

that provide an accurate representation of the program being 

measure 

c.	 Avoid threats to the validity of data 

i. Instrumentation effect - when the observed effect is due to 

changes in the questionnaire from one use to the next or 

when the questionnaire itself affects responses 

ii Hawthorne effect - when the observed effect is due to factors 

in the experiment itself (awareness of study's importance) 

7 
iii Selection bias - when the observed effect is due to 

differences between the types of individuals in the sample 

and comparison group 

3	 Be aware of certain things while collecting data. 

a	 Confidentiality - often required to protect privacy of individuals and 

increases willingness to cooperate with the evaluation 

b	 Paperwork Reduction Act - agencies "shall not conduct or sponsor 

the collection of information" without the approval of OMB 

c	 Uniformity - all data must be collected in the same way 

d Pre- testing - ensure that questions are clear and will provide the 

information needed for the evaluation analysis 

B0 Analyze and interpret the data 

1	 Choose between simple descriptive methods and more complex 

statistical methods based on: 

a Evaluation objectives 

b Audience expectations 

c Amount of time to complete report 

2	 If necessary, consult with statisticians or research experts. 

3	 Use statistics software packages to ease the process of recording and 

analyzing data, especially when extensive surveying of participants is 

necessary. 

4	 Consider some factors before selecting analytical techniques. 

a Will the audience understand sophisticated analytical techniques? 

b What margin of error will the audience be comfortable with? 

c Is a generalization from a sample to a population desirable? 

d Are the evaluators attempting to show a causal relationship 

between the program and measured results? 

III Writing and Distributing the Report 

A0Determine best method for communicating results 

1	 Briefings and presentations: 

8 
a Allow quick and selective communication of important information 

b Avoid bias in selecting material to present 

c Include some discussion of methodology and cautions about limits 

and appropriate use of data 

d Provide for interaction or feedback from audience to allow 

identification of issues and potential problems. 

2	 Written reports:
 

a Provide more detail on methodology and results
 

b Often required by legislation or executive decision
 

c Ensures that there is one "official" source of information on
 

evaluation results and methodology 

B0 Decide what kind of information needs to be 

communicated 

1 Goals and objectives of the evaluation
 

2 Description of the program and how it works
 

3 Description of evaluator's methodology
 

4 Presentation of evaluation findings
 

5 Discussion of program strengths and weaknesses
 

6 Implications for program administration
 

7 Recommendations
 

C0 Enhance the effectiveness of your presentation 

1	 Involve potential users as early as possible in determining 

presentation format and style. 

2 Tailor presentation method, format, and style to audience needs. 

3 Be clear and accurate. 

4 Be honest and direct. 

5 Simplify and reduce complex data to an understandable form and use 

graphic illustrations where appropriate. 

6 Provide an executive summary or abstract. 

7 Make survey instruments and other data collection tools available. 

8 Note the limitations on the interpretation and use of evaluation data. 

9 Expect the need for follow-up. 

10 Be flexible and responsive. 

D0 Disseminating of evaluation results 

1	 Decide who will be responsible for dissemination of results -this 

should be thought about early on and again when results become 

9 
available, may be decided by contract or agreement, or discussed 

informally by evaluators and decision-makers 

a Evaluator
 

b Program officials
 

c Evaluation requestor
 

d Cooperatively
 

2	 Decide when to release evaluation results - may be defined by 

contract or agreement or discussed by evaluators and decisionmakers 

a Publicity - select slower news days 

b Coordinate release with other agency activities 

c Releasing preliminary data before all data is collected is risky 

3	 Decide how widely results will be disseminated - cost, convenience 

and interest level will determine whether dissemination is wide or 

narrow 

4	 Decide how results will initially be disclosed - Memorandum, press 

release, press conference; determined by those who have the 

authority to make the disclosure 

10