bluuebunny/crossref_metadata_2025 huggingface dataset:
A Low-Cost Follow-Up Resource: Using the World Wide Web to Maximize Client Location Efforts
Authors: Passetti Lora L. , Godley Susan H. , Scott Christy K. , Siekmann Melissa
Published: August 2005

Many evaluations depend on follow-up interviews, and low attrition rates can help strengthen internal and external validity. The process of locating participants can consume significant evaluation resources. The World Wide Web is an inexpensive tracking tool offering cost-effective and time-saving access to several information networks and agencies that may provide information on program participants. The use of the World Wide Web as a tracking tool is illustrated by describing its application during follow-up work in the evaluation of a substance abuse treatment program for pregnant and postpartum women. This article serves as a guide to location-related links and their potential applications with a variety of participant groups.
Toward Clarification of Program Evaluation: A Proposal with Implications for the Possible Certification of Evaluators
Authors: Sawin Enoch I.
Published: August 2005

Recent attention to the topic of the possible certification of evaluators highlights several of the central problems facing the field. Controversies exist regarding the purposes and methods of evaluation. Indeed, there is no generally accepted definition of the term evaluation. Such problems would make it very difficult to formulate criteria for certification, if that step is taken eventually. Some plausible procedures are described for classifying different approaches to evaluation, and for subsequent elaboration in terms of philosophical foundations, theory, definitions, purposes, methodology, and ethical guidelines for each major category or cluster of categories. Such work could act as the foundation for subsequent efforts on certification and, in any case, would be enlightening about the current state and potential future directions of evaluation.
Legal and Ethical Issues in Evaluating Abortion Services
Authors: Ferris L. E.
Published: August 2005

When evaluation studies are conducted in a sensitive area, ethical and legal implications are bound to challenge evaluators. All too often, evaluators must deal with competing responsibilities in evaluating these programs or services. This article focuses on several ethical and legal issues that arose during an evaluation of abortion services. We discuss how we developed decision rules and considered trade-offs in dealing with these ethical and legal issues so that rational and objective decisions could be reached. We place this discussion within the context of balancing the utility and propriety evaluation standards with respect to obtaining true informed consent and protecting the privacy and confidentiality of data when evaluating abortion services. The article concludes with recommendations concerning the evaluation of abortion services.
Measuring Program Outcomes: Using Retrospective Pretest Methodology
Authors: Pratt Clara C. , McGuigan William M. , Katzev Aphra R.
Published: August 2005

This study used longitudinal data from 307 mothers with firstborn infants participating in a home-visitation, child-abuse prevention program. A self-report measure of specific constructs the program hoped to affect showed that the retrospective pretest methodology produced a more legitimate assessment of program outcomes than did the traditional pretest-posttest methodology. Results showed that when response shift bias was present, traditional pretest-posttest comparisons resulted in an underestimation of program effects that could easily be avoided by the retrospective pretest methodology. With demands for documenting program outcomes increasing, retrospective pretest designs are shown to be a simple, convenient, and expeditious method for assessing program effects in responsive interventions. The limits of retrospective pretests, and methods for strengthening their use, are discussed.
The Prose and Cons of Poetic Representation in Evaluation Reporting
Authors: MacNeil Cheryl
Published: August 2005

The evaluation of modern day social programming presents challenges to the evaluation community. Among these are the need for others to understand the complexity of unique social circumstances and the inclusion of a diverse range of evaluation participants. Such challenges may call for alternative methods of representation in evaluation reporting. In this article I make a case for and demonstrate the use of poetic transcription as one form of presenting evaluation findings. Taking language from focus group interviews, I constructed a poem intended to provoke report readers to engage in a process of reflective meaning making about the program being evaluated. I discuss my rationale for using poetic transcription as a form of evaluation reporting, the method by which I constructed the representation, and the reactions of the different stakeholders involved in the evaluation.
Presidential Address: Building Evaluation’s Collective Capacity
Authors: Leviton Laura C.
Published: August 2005

Building collective capacity can improve the quality of evaluators’ work, as well as their quality of life. Strengthening collective capacity will help evaluation to influence the policy and program worlds, which continue to misunderstand, misuse, and sometimes fear evaluation. Building this capacity means strengthening the relationships among evaluators themselves, for mutual support, information, collaboration, and sharing. Some barriers to strengthening these relationships include a tendency to operate in a lonely and independent context; an implicit denigration of skills for dealing with people and organizations; and the absence of a forum for discussing these issues. Evaluators can improve their collective capacity in at least three ways: 1) improved membership services from evaluation associations to reduce the vulnerability of individual evaluators to negative reactions to their work; 2) educating evaluation clients more broadly and proactively about what is required for a high quality and useful evaluation; and 3) mechanisms to support and strengthen the evaluation community. For AEA, the local affiliates are key to these improvements.
Multiple Perspectives on Training Evaluation: Probing Stakeholder Perceptions in a Global Network Development Firm
Authors: Michalski Greg V. , Cousins J. Bradley
Published: August 2005

Although stakeholder-based evaluation has been fairly well developed in the general program evaluation literature, it remains barely recognized in training evaluation practice. This article aims to contribute to our understanding of multiple stakeholder perceptions about training evaluation in an organizational context. Extending prior empirical work that described differences in stakeholder perceptions about valued training outcomes, the current case study examines multiple stakeholder perspectives of the purposes, processes, and consequences of evaluation in a global telecommunications network development firm. Semistructured interviews with 15 individuals in three stakeholder groups were audio recorded, transcribed, and coded for analysis. Diverging from common training evaluation practice, which generally fails to acknowledge and frequently limits stakeholder involvement, the current results provide evidence for a multiplicity of stakeholder views. These views were found to be related to the job role of a stakeholder in the organization. For example, training sponsors (line managers with budgetary discretion) described largely formative evaluation purposes, as well as instrumental and symbolic forms of utilization, to improve training for their employees. Training participants (employees with direct or indirect reporting relationships with the managers interviewed) described mixed (formative-summative) purposes, as well as instrumental and conceptual forms of utilization, to improve specific courses and to make informed course selections. Training providers (internal training specialists who develop and deploy training for the previous two groups as internal organizational clients) described evaluation in terms of mixed purposes, and instrumental and symbolic forms of utilization, mostly to highlight training merit and worth and to sustain and expand training budgets. These differing perspectives are discussed in terms of a three-dimensional schema of collaborative inquiry and evaluation. Implications for training and program evaluation research and practice are also discussed.
Evaluation Checklists: Practical Tools for Guiding and Judging Evaluations
Authors: Stufflebeam Daniel L.
Published: August 2005

This article describes a project designed to provide evaluators, their clients, and other stake-holders with checklists for guiding and assessing formative and summative evaluations. The checklists pertain to program, personnel, and product evaluations, and reflect different conceptualizations of evaluation. They are constructed for use in planning, contracting, conducting, reporting, and judging evaluations. The checklists, along with papers on the logic and methodology of checklists and guidelines for developing checklists, are conveniently available on the Western Michigan University Evaluation Center’s Web site www.wmich.edu/evalctr/checklists/.
Asking Questions About Behavior: Cognition, Communication, and Questionnaire Construction
Authors: Schwarz Norbert , Oyserman Daphna
Published: August 2005

Evaluation researchers frequently obtain self-reports of behaviors, asking program participants to report on process and outcome-relevant behaviors. Unfortunately, reporting on one’s behavior poses a difficult cognitive task, and participants’ reports can be profoundly influenced by question wording, format, and context. We review the steps involved in answering a question about one’s behavior and highlight the underlying cognitive and communicative processes. We alert researchers to what can go wrong and provide theoretically grounded recommendations for pilot testing and questionnaire construction.
Use and Non-use of Evaluation Results: Case Study of Environmental Influences in the Private Sector
Authors: Russ-Eft Darlene , Atwood Regina , Egherman Tori
Published: August 2005

This case study details an evaluation of a sales program within a business services organization, here called XYZ Corporation. Logic models were employed to identify the processes, as well as the short- and long-term outcomes. Twenty-three sales people, 10 implementation specialists, and nine consultants from XYZ were interviewed, along with 40 XYZ clients. In addition, written surveys and archival data were gathered. Although timely evaluation reports were produced and presented, reorganization at XYZ Corporation limited direct instrumental use of the evaluation. The present article focuses on use and non-use of the evaluation results, with particular emphasis on process use, and it discusses factors contributing to use and non-use of this private sector evaluation.
Unequal Power—Changing Landscapes: Negotiations between Evaluation Stakeholders in Latino Communities
Authors: Clayson Zoe Cardoza , Castañeda Xóchitl , Sanchez Emma , Brindis Claire
Published: August 2005

The analyses presented in this article are based on the authors’ experiences over the past 5 years evaluating community initiatives located in several low-income, California Latino communities. The dynamic interactions between major stakeholder groups including funders, community-based organization staff, community members, and evaluators are examined through a context-sensitive lens. A case study of El Centro Familiar provides a vehicle for presenting our roles as an interpreters, translators, mediators, and storytellers. The challenges of blending a critical theory social science approach with community-focused evaluation practice form the basis of the discussion.
Evaluator as Steward of Citizen Deliberation
Authors: MacNeil Cheryl
Published: August 2005

I suggest that evaluators have much to offer in being stewards of citizen deliberation during the conduct of evaluative inquiry, especially in regards to the professional community’s guiding principle of “Responsibility to the general and public welfare.” The deliberative forum is reviewed as one evaluation methodology for bringing the theory of deliberative democratic evaluation into practice. The paper offers a number of reflections from the field based on my experiences of adopting a stewardship role during my evaluation practice. These include: (1) working with the tacit program culture, (2) consciousness of risk, (3) creating dissonance, (4) the role of coaching, and (5) sustaining deliberation in program communities. Examples are offered in thinking about constructing deliberative forums throughout evaluation processes in practical contexts. The paper positions evaluation practice as an important mechanism for contributing toward a civil society and asks evaluators to consider their role in being stewards of citizen deliberation.
Enhancing Practical Evaluation Training Through Long-Term Evaluation Projects
Authors: Trevisan Michael S.
Published: August 2005

There is common understanding and wide agreement for practical experiences in evaluation training programs. The argument for this idea is that practical training will provide students the real world experiences needed for preparation as professional evaluators and that these experiences are not possible in didactic course work. Several strategies for providing practical evaluation experiences can be found in the literature. They are limited by their short-term nature, typically bound by a semester time frame. This paper offers another possibility for providing practical evaluation training; namely, university-supported, long-term funded evaluation projects. These projects are managed through a university center that provides assistance to clients in student assessment and program evaluation. After a description of the center, the benefits and challenges of providing these experiences for students are discussed.
Getting to the Bottom Line: A Method for Synthesizing Findings Within Mixed-method Program Evaluations
Authors: McConney Andrew , Rudd Andy , Ayres Robert
Published: August 2005

Evaluators who are concerned more with pragmatics than with competing epistemologies have brought multi- and mixed-method evaluations into common practice. Program evaluators commonly use multiple methods and mixed data to capture both the breadth and depth of information pertaining to the evaluand, and to strengthen the validity of findings. However, multiple or mixed methods may yield incongruent results, and evaluators may find themselves reporting seemingly conflicting findings to program staff, policy makers, and other stakeholders. Our purpose is to offer a method for synthesizing findings within multi- or mixed-method evaluations to reach defensible evaluation (primarily summative) conclusions. The proposed method uses a set of criteria and analytic techniques to assess the worth of each data source or type and to establish what each says about program effect. Once on a common scale, simple math allows synthesis across data sources or types. The method should prove a useful tool for evaluators.
Postmodern Evaluation—or Not?
Authors: Mabry Linda
Published: August 2005

Postmodernism, once an obscure little cloud, has descended as a fog of disbelief and cynicism. Should evaluators change, worry, laugh, cry? The postmodern challenge to social science, particularly to program evaluation, and its potential contribution are the subject of vignettes from practice and a limited exploration, one that borrows occasionally from playfully absurdist postmodern style.
Parting Thoughts on Foundation Evaluation
Authors: Wisely D. Susan
Published: August 2005

Why has progress in evaluation in private foundations been so slow and intermittent? Four possible answers are presented and discussed. Despite the conditions that have slowed evaluation in foundations, there is reason for hope for the future.
The Use of Role-Play in Teaching Evaluation
Authors: Alkin Marvin C. , Christie Christina A.
Published: August 2005

This paper describes the use of role-play in creating an experiential learning environment for graduate students enrolled in a comparative evaluation theory and/or an evaluation procedures course. Role-play exercises have been designed for each course that increase student involvement in the learning process, which promotes a deeper understanding of and engagement in course content. The specific exercises used, which are described in detail, require students to think beyond the words they read—to learn by doing in a safe environment.
Presidential Address: On Mainstreaming Evaluation
Authors: Sanders James R.
Published: August 2005

The practice of evaluation in organizations continues to be limited by perceptions that evaluation is a marginal activity. Arguments demonstrating the importance of evaluation have been ineffective in moving most organizations toward integrating evaluation into their daily routines. A multifaceted approach to making evaluation a part of organizational culture is proposed. It includes using allies, examples, models, research, process guides, and trainers/developers. This article is based on The Presidential Address given by Dr. James R. Sanders at the 2001 Annual Meeting of the American Evaluation Association (AEA) in St. Louis, MO. The theme for the conference was “Mainstreaming Evaluation.”
Evaluation Capacity in K-12 School Counseling Programs
Authors: Trevisan Michael S.
Published: August 2005

The K-12 school counseling field continues to develop and implement Comprehensive, Developmental, Guidance and Counseling (CDGC) programs to organize services provided by school counselors. Given the program evaluation requirement to guide refinement and renewal efforts, enhancement of CDGC evaluation infrastructure is needed. This paper presents the results of a literature review, organized by the Milstein and Cotton (2000) evaluation capacity framework, detailing and analyzing the contextual factors and system features that impact the evaluation capacity of school counseling programs. The paper fills a need in the literature for more substantive work focused on evaluation capacity, particularly with respect to the school counseling field. The analytic approach is a test case that could be applied to the analysis of evaluation capacity in other settings.
Using Continuous System Level Assessment to Build School Capacity
Authors: Smith Christopher L. , Freeman Rachel L.
Published: August 2005

The purpose of this article is to introduce a conceptual model for internal assessment and professional development planning. The continuous systems-level assessment (CSLA) is a model that can provide school and other professionals with a method for making data-based decisions, and is intended to foster local expertise by utilizing assessment information to design effective professional development strategies. The CSLA model includes three phases: needs assessment and problem identification; designing interventions and building staff capacity; and implementing and evaluating interventions. The authors outline the values associated with the CSLA model, the major phases involved in the process, and provide an example of how the process is currently being implemented in an urban school in Kansas.

School-based programs are complex, dynamic, and usually involve long-term commitments from a variety of partners. In addition, program implementation often occurs within the context of organic, changing organizations and therefore, requires a flexible model of evaluation. Several bottom-up program evaluation models have been developed that can be used to guide an assessment of the commitments, values, and capacities of schools and other organizations (Fawcett et al., 1996; Fetterman, 1996; Greenwood, Whyte, & Harkavy, 1993; Knoff, 1996; Levin, 1996; Millett, 1996; Scriven, 1967; Stake, 1967, 197 6). The approach described here is similar to these bottom-up program evaluation models, and is being used to assess ongoing improvement efforts and identify professional development needs for schools.
Evaluator Roles and Strategies for Expanding Evaluation Process Influence
Authors: Morabito Stephen M.
Published: August 2005

This article explores various evaluator roles and strategies that have the potential to increase the likelihood that the evaluation process will have an influence on an organization and its members. These roles are (1) educator, (2) consultant, (3) facilitator, and (4) counselor. A brief case study presents the discussion dynamics of an evaluation workgroup. This workgroup resulted in programmatic changes based primarily on the discussion of developing indicators for measuring student outcomes. Practical implications for evaluators seeking to enhance process influence may involve the evaluator using a “client-centered” approach. Facilitating collaborative, open environments and managing discussion dynamics are fundamental in establishing a community of organizational learners. These dynamics may take the form of both task-focused and task-related discussions.
Performance Measurement Redux
Authors: Feller Irwin
Published: August 2005

Recent developments in the United States in the use of performance measurement in science policy and higher education are used to comment further on the Perrin-Bernstein-Winstondebate about the effective use and misuse of performance measurement. Particular attention is given to the influence of political/organizational factors and the production processes of agencies on how performance measures are constructed and used. The analysis points to further limitations in the use of performance measurement. In both cases, long-gestating, probabilistic linkages between outputs and outcomes limit the usefulness of mainstream indicators as a measure of current agency performance and as a guide to major, discontinuous resource allocation decisions. Conspicuously absent from many performance measurement undertakings are provisions for evaluating the impact of the undertakings themselves. An updated account of the status of the Government Performance and Results Act (GPRA) indicates that the Act has not had the impacts predicted for it.
A Three-Step Approach to Teaching Logic Models
Authors: Renger Ralph , Titcomb Allison
Published: August 2005

Developing a logic model is an essential first step in program evaluation. Our experience has been that there is little guidance to teach students how to develop a logic model that is true to its intended purpose of providing a clear visual representation of the underlying rationale that is not shrouded by including the elements of evaluation. We have developed a three-step approach that begins with developing the visual representation of the underlying rationale, central to which is the identification of Antecedent conditions. Step 2 ensures that program activities Target antecedent conditions, while Step 3 focuses on Measurement issues, depicting indicators and objectives for outcomes being included in the evaluation plan. We have coined this method of teaching the ATM approach. We hope that teachers of evaluation will find the ATM approach useful in the form presented here or at least stimulate thought as to how to adapt the approach to meet individual teaching needs.
Youth Participation in Community Evaluation Research
Authors: Checkoway Barry , Richards-Schuster Katie
Published: August 2005

Should young people participate in community evaluation research and, if so, what roles should they play? We believe that youth participation is desirable, but that it remains relatively undeveloped as a field of practice or subject of study. For this reason, this paper provides perspective on the increasing involvement by young people, identifies the participation patterns and evaluation roles of young people, and draws conclusions from empirically-based practice. It draws on the available literature, including our own participatory evaluation of a project intended to increase youth participation in organizational development and community change.
Reflections on the Use of Narrative in Evaluation
Authors: Costantino Tracie E. , Greene Jennifer C.
Published: August 2005

In this article, we tell the story of our evaluation of an intergenerational storytelling program. The story offers our reflections on the evolution of our practice, from an initial evaluation approach of interpretive responsiveness to an ending framework of narrative for this evaluation. The paper is structured so that it reflects the interwoven quality of narrative, as we tell the story of the program within the story of our practice. We conclude with a discussion of several theoretical anchors for our premise that narrative can offer valuable contributions to program evaluation, along with important challenges such a framework poses for our craft.
A Comparison of Three Retrospective Self-reporting Methods of Measuring Change in Instructional Practice
Authors: Lam Tony C. M. , Bengo Priscilla
Published: August 2005

In the post + retrospective pretest method of measuring change, evaluators ask the respondents to recall pre-intervention status at posttest time. Research has produced strong evidence in support of this approach over the pretest-posttest approach to measuring change. However, no research has yet to examine and compare different forms of retrospective methods. We compared three retrospective methods of measuring elementary grade teachers’ self-reported change in mathematics instructional practices: the post + retrospective pretest method (reporting current practices and earlier practices), the post + perceived change method (reporting current practice and the amount and direction of change), and the perceived change method (reporting only the amount and direction of change). Teachers in the post + retrospective pretest condition reported least change, followed by teachers in the post + perceived change condition; teachers in the perceived change condition reported the greatest change. We can explain our findings in terms of differential satisficing (the tendency to exert minimal effort in responding) caused by differences in cognitive demands among the three methods. Greater task difficulty leads to greater satisficing, which causes respondents to resort more to socially desirable responses. A greater tendency to provide socially desirable responses leads to relying on expected implicit theory of change and subsequently reporting greater change in instructional practices.
Accountability and Performance in Charter Schools in Florida: A Theory-Based Evaluation
Authors: Crew Robert E. , Anderson Mary Ruggiero
Published: August 2005

This paper utilizes the framework of program theory evaluation to assess the operation of charter schools in Florida in the 1999-2000 school year. Charter school theory is modeled and a series of hypotheses are developed about the linkages expected to produce the primary program outcomes, improved student academic performance and change in the operations of regular public schools. The hypotheses are evaluated with quantitative and qualitative data. The analysis raises questions about the theory. Student performance in charter schools is not better than in regular public schools and, as might be expected, regular schools have not changed their behavior in response to their challenge. The outcome is attributed primarily to shortcomings in establishing accountability for the actions of charter schools.
Program Staff Perceptions of Barriers to Evaluation Implementation
Authors: Taut Sandy M. , Alkin Marvin C.
Published: August 2005

This study investigated the nature of barriers to the implementation of (external) program evaluation. The purpose of the study was to expand the scarce empirical research on this topic by adding program staff perspectives. Eighteen staff members of a university outreach program that had been subjected to external evaluation were interviewed (1) about their general thoughts on barriers to evaluation implementation, and (2) about their views on the explanatory value of factors found in research related to evaluation utilization and which we believed to be theoretically relevant for barriers to evaluation implementation. These factors are summarized as human, evaluation, and context factors. Asked what they considered to be barriers to evaluation implementation, the interviewees mentioned human factors most frequently. They talked about evaluation and context factors much less frequently. With regard to human factors, interviewees focused on the evaluator’s social competence and program staff’s lack of trust in evaluators and evaluation process. Taking into account the context of the study, particularly the participants’ prior experiences with evaluation, the findings reinforce our understanding that in order to avoid barriers to evaluation implementation, it is important to create a trusting relationship with those affected by the evaluation through continuous participation and communication, and to conduct carefully planned, methodologically appropriate evaluations.
Adapting the Empowerment Evaluation Model: A Mental Health Drop-in Center Case Example
Authors: Sullins Carolyn D.
Published: August 2005

Empowerment evaluation involves a program’s stakeholders in designing and implementing an evaluation of their own program, thus contributing to the program’s improvement and self-determination (Fetterman, 1994, 1996). It appeared to be an appropriate approach for evaluating a mental health drop-in center, which had congruent goals of collaboration and self-sufficiency. However, encouraging ongoing participation among the stakeholders was challenging. The evaluation process was adapted, with the evaluator designing and conducting most of the evaluation but incorporating substantial input from the stakeholders throughout the process. This involved careful attention to the evaluator’s roles, and decisions concerning the depths of stakeholder versus evaluator participation at each step of the evaluation. Adaptations to empowerment evaluation must consider the values of the program, as well as balance the stakeholders’ needs for resources with their rights for autonomy.
Rapid Assessment with Qualitative Telephone Interviews: Lessons from an Evaluation of California’s Healthy Families Program & Medi-Cal for Children
Authors: Sobo Elisa J. , Simmes Diana R. , Landsverk John A. , Kurtin Paul S.
Published: August 2005

California’s Healthy Families/Medi-Cal for Children (HF/MCC) program provides low- and no-cost health insurance to low-income children. In December 1999 and January 2000, 72 community-based organizations (CBOs) were contracted by California’s Department of Health Services (DHS) to supply culturally appropriate HF/MCC outreach and enrollment services. This article describes the incorporation of a qualitative telephone interview component for rapid assessment purposes after the planned quantitative analysis of program records faltered in 2001 due to missing, invalid, and unreliable data. We describe the rapid interview protocol and discuss benefits and barriers. The protocol can serve as a model for others interested in using rapid qualitative telephone interviews within their own evaluation efforts.
Building a Better Body of Evidence: New Opportunities to Strengthen Evaluation Utilization
Authors: Ginsburg Alan , Rhett Nancy
Published: August 2005

A useful evaluation of an education program is one that adds to the body of timely, relevant evidence to increase the likelihood that policy decisions improve program performance. This definition focuses on performance rather than change and recognizes that studies can’t guarantee the use of their findings but can only increase the likelihood of such use. Adding to the body of sound program evidence is the evaluators’ main method for influencing programs. Unprecedented congressional demand for scientifically sound evaluation evidence will lead investigators to conduct experiments and systematic reviews to rigorously answer causal program questions. But not all questions related to improving education program performance are causal, and other useful methodologies utilize scientifically valid, large-scale surveys and case studies to evaluate program outcomes and better understand implementation. The experience of the U.S. Department of Education demonstrates that well-designed, timely evaluations using scientifically valid methodologies tailored to important questions can make important contributions to legislative, budget, and administrative decisions to improve the performance of government-funded programs.
What Makes an Evaluation Useful? Reflections from Experience in Large Organizations
Authors: Grasso Patrick G.
Published: August 2005

Evaluation units found in large organizations confront a number of impediments to the use of their findings and recommendations. These derive from both the formal roles they play and their own operating styles and processes. Drawing on experience at the U.S. General Accounting Office and the World Bank, this paper suggests that to improve the usefulness—and use—of their work, such units need to pay careful attention to the multiplicity of audiences they necessarily address, the divergent information needs of those audiences, and the timeliness of their work.
Shoestring Evaluation: Designing Impact Evaluations under Budget, Time and Data Constraints
Authors: Bamberger Michael , Rugh Jim , Church Mary , Fort Lucia
Published: August 2005

The paper discusses two common scenarios in which evaluators must conduct impact evaluations when working under budget, time, or data constraints. Under the first scenario the evaluator is not called in until the project is already well advanced, and there is a tight deadline for completing the evaluation, frequently combined with a limited budget and without access to baseline data. Under the second scenario the evaluator is called in early, but for budget, political, or methodological reasons it is not possible to collect baseline data on a control group and sometimes not even on the project population. As a result of these constraints, many of the basic principles of impact evaluation design (comparable pretest-posttest design, control group, instrument development and testing, random sample selection, control for researcher bias, thorough documentation of the evaluation methodology, etc.) are often sacrificed. We describe the “Shoestring Evaluation” approach which is being developed to provide tools for ensuring the highest quality evaluation possible under constraints of limited budget, time and data availability. While most of the data collection and analysis techniques will be familiar to experienced evaluators, what is new is the combination of these techniques into an integrated six-step approach which covers: (1) planning and scoping the evaluation, (2-4) options for dealing with constraints related to costs, time and data availability (which could include reconstructing baseline conditions and control groups), (5) identifying the strengths and weaknesses (threats to validity and adequacy) of the evaluation design, and (6) taking measures to address the threats and strengthen the evaluation design and conclusions. When necessary, many of these corrective measures can be introduced at a very late stage, even when the draft evaluation report has already been produced. Although the approach was designed to assist evaluators working in developing countries, the principles are equally applicable for conducting evaluations under similar constraints in industrial nations. Case studies are included to illustrate the different Shoestring approaches, drawing from the authors’ experience in Eastern Europe, Africa, Asia, and Latin America.
A Low-Cost, Post Hoc Method to Rate Overall Site Quality in a Multi-Site Demonstration
Authors: Barth Michael C.
Published: August 2005

Demonstration programs and social experiments are often subject to sophisticated, controlled evaluations. An important factor that is not subject to control, and sometimes even goes unobserved, is overall program site quality. Site quality can be observed in process evaluations, but these tend to be expensive. This paper describes an alternative approach to site quality measurement. With observations elicited from national program staff of a large demonstration project, we developed an approach that uses Concept Mapping to identify dimensions for rating program site quality and compared our results with ratings based on site visits developed during program implementation. The site quality ratings and rankings resulting from our approach are plausible, have desirable psychometric properties, and are developed at much lower cost than is possible for even relatively modest process analyses.
Using Corporate Data in Workplace Program Evaluation
Authors: Spicer Rebecca , Nelkin Valerie , Miller Ted , Becker Les
Published: August 2005

Corporate data fall under the general category of administrative data and provide varied information on the worker and workplace. Studies of worker health, including evaluations of workplace programs, are often based on corporate data. With the exception of health care claims data, which are usually administered by a third party, the corporation collects these data and grants permission to use them. In light of new federal regulations, gaining access to corporate data is more complicated and intimidating than ever before. This paper discusses the impact of federal regulations on the use of corporate data in research, the potential of corporate data in evaluation studies, the challenges to obtaining and using these data, and finally, study designs that make use of corporate data.
Theory-Based Evaluation: The Case of Social Funds
Authors: Carvalho Soniya , White Howard
Published: August 2005

The theory-based evaluation approach documents the assumptions implicit in program design and points to the data required to test these assumptions. Collecting and analyzing such data through quantitative and qualitative techniques enhances understanding of the validity of the assumptions and the relevance of key program processes. This article lays out the application of a theory-based approach to the evaluation of social fund projects, with a focus on the issues of subproject sustainability and institutional development impact. It shows the processes at work especially at the community level and tests the assumptions underlying social fund subproject sustainability and institutional development impact.
Using Concept Mapping to Develop a Logic Model and Articulate a Program Theory: A Case Example
Authors: Yampolskaya Svetlana , Nesman Teresa M. , Hernandez Mario , Koch Diane
Published: August 2005

Using a case example of one of the mental health agencies in Florida, we demonstrate the utility of concept mapping for developing a program logic model and articulating a program theory for program assessment. The results of the concept mapping procedure enabled identification of 100 program services as described in statements by staff. Moreover, results revealed four major categories of those services and the main strategies used. This information was used to develop the first three components of a logic model. Concept maps were also used to reveal underlying assumptions built into the agency’s theoretical approach.
Creating a Culture of Data Use for Continuous Improvement: A Case Study of an Edison Project School
Authors: Sutherland Stephanie
Published: August 2005

In this article, I outline how, with the appropriate mix of external requirements and internal motivation, structure and capacity, a school can promote and maintain a culture of evaluative practices, specifically data use, for continuous improvement. I draw upon qualitative data conducted in a study of an Edison Project school. The findings from this paper are the result of a larger 4-year study examining Comprehensive School Reform in 12 Title 1 schools in three states. Interviews were conducted with principals, teachers, other staff, parents and students, as well as with state officials, the superintendent, assistant superintendent and other high level administrators in the district. Observations were undertaken at the classroom level. Key findings include the importance of obtaining a multidimensional view of educational change. That is, we need to obtain a picture of the school reform environment (i.e., interconnections between the state, district, and school levels), and simultaneously investigate the interplay among critical school levers that are required to promote and maintain a culture of data use for school improvement. From this multidimensional view, we are better positioned to investigate and evaluate school change efforts.
Improving Design Sensitivity through Intervention-Sensitive Measures
Authors: Meier Scott T.
Published: August 2005

Despite evidence that the choice of dependent measures can significantly influence design sensitivity, many evaluators default to traditional measures that may be insensitive to intervention effects. This paper describes an innovative set of test development guidelines designed to select items and create aggregate scales that are better able to detect program effects. The application of these Intervention Item Selection Rules (IISRs) is illustrated during the initial development of an outcome measure, completed by teachers, for elementary age children receiving psychosocial services from community mental health agencies. The major scale formed with these change-sensitive items displayed a larger effect size and an adequate reliability estimate.
Teaching Evaluation through Modeling: Using the Delphi Technique to Assess Problems in Academic Programs
Authors: Garavalia Linda , Gredler Margaret
Published: August 2005

The Delphi technique permits the collection of rich evaluation data. Benefits of using the Delphi include the collection of data that is generated by participants with minimal leading by researchers, an iterative process that demonstrates to participants how their ideas and opinions are being utilized in the research process, and a planned interaction to share results with participants, which usually creates good will between the participants and research team. This paper describes two cases in which the Delphi technique was implemented in order to teach students how to use the method. In both cases, students identified problems in their particular programs as the focus of the data collection activity.
Tales of Refusal, Adoption, and Maintenance: Evidence-Based Substance Abuse Prevention Via School-Extension Collaborations
Authors: St. Pierre Tena L. , Kaltreider D. Lynne
Published: August 2005

Despite availability of empirically supported school-based substance abuse prevention programs, adoption and implementation fidelity of such programs appear to be low. A replicated case study was conducted to investigate school adoption and implementation processes of the EXSELS model (Project ALERT delivered by program leaders through Cooperative Extension). Interviews with school personnel revealed: (1) schools were not aware of evidence-based programs until Extension approached them; (2) schools dared not eliminate DARE; (3) teachers are unlikely to implement with fidelity; (4) implementation of theory-based prevention is not consistent with school views of curriculum delivery; and (5) schools believed Project ALERT via EXSELS was an advantage over teacher delivery, but only three of the eight schools sustained the model. Discussed are potential for Extension as a national implementation system, the value of qualitative inquiry to study processes of adoption, and issues related to the selection and implementation of evidence-based programs.
Multilevel Evaluation Alignment: An Explication of a Four-Step Model
Authors: Yang Huilan , Shen Jianping , Cao Honggao , Warfield Charles
Published: August 2005

Using the evaluation work on the W.K. Kellogg Foundation’s Unleashing Resources Initiative as an example, in this article we explicate a general four-step model appropriate for multilevel evaluation alignment. We review the relevant literature, argue for the need for evaluation alignment in a multilevel context, explain the four-step model, discuss the benefits of the model, and address the issues that one should pay attention to when implementing our alignment model.
Establishing Essential Competencies for Program Evaluators
Authors: Stevahn Laurie , King Jean A. , Ghere Gail , Minnema Jane
Published: August 2005

This article presents a comprehensive taxonomy of essential competencies for program evaluators. First, the authors provide a rationale for developing evaluator competencies, along with a brief history of the initial development and validation of the taxonomy of essential evaluator competencies in King, Stevahn, Ghere, and Minnema (2001). Second, they present a revised version of that taxonomy and describe the revision process. Third, a crosswalk accompanying the taxonomy indicates which competencies address standards, principles, and skills endorsed by major evaluation associations in North America. Finally, the authors identify future needs related to the taxonomy, including the need for validation research, a shared understanding of terms, and the construction of descriptive rubrics for assessing competence.
The Centrality of Practice to Evaluation
Authors: Schwandt Thomas A.
Published: August 2005

There is a strong tendency for evidence-based approaches to social practices to view these practices as imperfect devices for delivering social services. Practices are regarded as in need of repair by evaluation (and research) that can deliver the necessary science-based solution to the problems of practice. This article presents a different view of practices as material and linguistic events in which activities and relationships are constituted and unfold in interaction and in which people change and develop, and it argues for restoring this view of practice to evaluation. The article discusses two different ways in which notions of evidence based, practice, and evaluation are related and suggests what a genuinely practice-oriented approach to evaluation entails.
The Missing Treatment Design Element
Authors: Barnette J. Jackson , Wallis Anne Baber
Published: August 2005

We rely a great deal on the schematic descriptions that represent experimental and quasi-experimental design arrangements, as well as the discussions of threats to validity associated with these, provided by Campbell and his associates: Stanley, Cook, and Shadish. Some of these designs include descriptions of treatments removed, removed and then reintroduced, or reversed with use of multiple postobservations. The focus of this article is to consider what happens to the intervention between and among postobservations and how that relates to threats to validity and data modeling and analysis. Evaluation designs must consider such questions as “Has the intervention been removed, or is it still in place? If so, has it changed in intensity, or are there persistence or carryover effects?” This article adds to the evaluation methodology literature by filling a gap regarding the issue of treatment or intervention continuity between and among postobservations.
Minimalist Theory: The Least Theory That Practice Requires
Authors: Scriven Michael
Published: August 2005

This paper has two parts. The first provides a general discussion of the nature of theories-and the reasons why some understanding of this abstract subject is of value in the practice of evaluation. The second part looks more closely at the dependence of good practice on a minimalist theory of evaluation, and at what such a theory comprises.
Professional Reasons for Declining an Evaluation Contract
Authors: Smith Nick L.
Published: August 2005

Professional evaluation is a market-based activity in which economic considerations play a dominant role. But are there ever professional reasons for turning down an evaluation contract? Considering how evaluations can promote either guild maintenance interests or societal improvement interests illuminates several dilemmas in the proper conduct of evaluation and suggests the need for greater explicit attention to what it means for the evaluation profession to contribute to the public interest.