# Evaluation Theories/Greene, Henry, Donaldson, King Comparison/Greene's "Value-Engaged Approach"
### Article Metrics

* Name: Jennifer C. Greene, 'A value-engaged approach for evaluating the Bunche–Da Vinci Learning Academy' - New Directions for Evaluation Special Issue: Theorist's Models in Action, pp. 27–45.
* Original Work Metrics: 7555 words, 48515 characters, 19 pages; Skim Highlights and Comments: 2945 words, 19443 characters
* Notes to Original Work Ratio: 2.6 to 1
What follows is a raw dump of Skim comments and highlights.
A Value-Engaged Approach for Evaluating the Bunche–Da Vinci Learning Academy
Jennifer C. Greene
active engage- ment with values that are drawn principally from democratic and culturally responsive traditions in evaluation (Hood, 1999; House and Howe, 1999; MacDonald, 1977).
Because value-engaged evaluation is fundamentally responsive to the particular character of an evaluation context and the programmatic and policy issues that are present in that context, it is not possible to develop an a priori or preordinate (Stake, 1987) evaluation design using this approach.
concentrates on envisioning these front-end aspects of evaluation in this context
front-end
Assumptions will also be made along the way to enable a more complete story to be told

## Developing an Understanding of the Context and the Program to Be Evaluated

(J: Brilliant: Greene starts with an email. This is helpful because working with "real working data" (or as close as possible to it) can be better for people wanting to learn a process than working with descriptions of those things.)
Principal
District Superintendent
Dear Ms. García:
I am
keenly (This is a filler that adds spice to the email: if it wasn't there, some enthusiasm would be lacking, and it could be read to be something other. . . I think this is indicative that we expect higher-energy interaction with others on first meeting)
committed
eager
would like to learn more
This will enable me to develop an evaluation plan that can yield the kinds of informa- tion likely to be meaningful and useful to your school community (J: sets up deliverable)
meaningful and useful (J: Primary values?)
your school community (J: Primary stakeholder?)
For this purpose, I am hoping that I can visit Bunche–Da Vinci several times this spring and (J: Sets up activities)
review relevant materials (for example, annual reports and previous evaluation studies),
i
talk with various members of your commu- nity
l
simply spend some time at your school.
learn more about the kinds of evaluation priorities that various members of your school community have. (J: Speaks to her value of the democratic process / stakeholder involvement)
I will follow up this email with a phone call to pursue this idea further and hopefully identify particular days for my visits. (J: next-steps) /COVER LETTER/

### The Bunche–Da Vinci Community

A very important first step in developing an evaluation plan for the Bunche–Da Vinci Learning Academy is to better understand the particular characteristics and diversities of this school community
especially important not to make assumptions about this particular context based on urban stereotypes.
Learning about this particular school community—its uniqueness, its complexities, and its continuing and dy- namic evolution—is fundamental to developing an evaluation plan
/REITERATION OF CASE/
\TEAM ENGAGEMENT\
Estella Marquez <emarquez@uiuc.edu>, Hashid Robinson <hjr3@uiuc.edu (J: STAKEHOLDERS: EVALUATION TEAM MEMBERS)
/TEAM ENGAGEMENT/
\REITERATION OF CASE\

### The Da Vinci School Reform Program
## The Bunche–Da Vinci Learning Academy

/REITERATON OF CASE/

### Just What Is Being Evaluated?

a more complete understanding of the Da Vinci program and its implementa- tion in the Bunche–Da Vinci Learning Academy is a critical initial step in a value-engaged approach to evaluation. (J: "value-engaged" sounds nebulous here; like marketing speak. It would seem to be a critical initial step in ANY approach to evaluation...)
value-engaged
Identification of the evaluand, or just what is being evaluated, is not assumed in this evaluation approach, but rather is a matter of discussion among as many stakeholder groups as possible in this front-end part of the evaluation
The evaluator con- tributes to this discussion her or his expertise and perspectives, including ideas from relevant research literature.
/DESCRIPTION OF EVAL PHILOSOPHY/
\REITERATION OF CASE\
/REITERATION OF CASE/
value commitments of this value-engaged evaluation approach
understanding the character of the pro- gram (and/or institutional structure, like a partnership) to be evaluated in its particular and unique context
contextuality
inclusion of all legitimate stakeholder views and perspec- tives
with special efforts to include the more marginalized people in the context (House and Howe, 1999
(J: i.e.) discouraged teachers and tran- sient families

## Identifying the Key Evaluation Questions

Principal
we brought to that meeting key evaluation priorities identified during our con- versations
discussion at
faculty meeting last Thursday
w
our review of relevant documents
over the past month
and
we invited further faculty comment on these priorities during the meeting
the most important clusters of evaluation priorities at this time con- cern the following issues:
the demands of the curriculum
Teachers
feel overloaded with
the long hours
the extra courses
Teachers also feel underappreciated as professionals
With constant teacher turnover, it is hard to develop and sustain a sense of community in the school.
\GOAL\ sustain a sense of community in the school /GOAL/
The Da Vinci curriculum is being questioned by teachers and parents alike for its relevance
power to enable meaning- ful learning
and
concerned about the rele- vance and effectiveness of the curriculum in meeting the needs of English lan- guage learners and students with special needs.
concerns about the integration of most special services into the regular classroom and curriculum
wonder why so many families don’t stay at Da Vinci, but rather transfer after just one or two years.
committed to raising the test scores of participating students
concerned that the school leadership does not have the authority to make changes in the Da Vinci pro- gram as might be needed,
We will use these evaluation priorities as the basis for developing our key evaluation questions and our evaluation plan, (J: What "PRIORITIES"? So far you've talked about concerns, one commitment, and one "wondering")
attached please find a draft of our description of the Bunche–Da Vinci Learning Academy and its surrounding communities. When
we would welcome your feedback on this draft, especially its accu- racy and completeness.
We also intend to share this draft with several teach- ers and community members who have been involved in this school for many years and who also volunteered to provide feedback.
teach- ers
community members
aug- ment our understanding of the quality and effectiveness of the evaluand
\DESCRIPTION OF EVAL PHILOSOPHY\ The major purpose of evaluation in a value-engaged approach is to
This is
educative
requires that the evaluation include assessments of the program
and experienced
as implemented
assessments of what important differences the pro- gram has made in the lives of its participants
evaluation questions attend specifically to the interests of people traditionally underserved in our country, including racial and ethnic minorities, low-income people, immigrants, and people with disabilities.
These frames for evaluation questions
reflect additional value commitments underlying this value-engaged approach to evaluation
A commitment to learning in and through evaluation
to contribute to bet- ter understanding of how well the program being evaluated meets impor- tant needs and interests of participants in the particular setting at hand
in addition to how well participants perform in the program)
A commitment to engaging with difference and diversity
as manifested in the context at hand
to assess how well the program being evaluated meets the particular needs and interests of people traditionally underserved
key eval- uation questions

#### Overall Question

In what ways and to what extent does the educational program offered at the Bunche–Da Vinci Learning Academy meet the important educational needs of the children and families served by this school, in particular the distinctive needs of English language learners, children from racial and ethnic minority groups, children from low-income families, and children with special needs? And in what ways and to what extent does the struc- ture of the partnership support this primary educational mission?

### Subquestions
#### Quality

"1. What is the quality of the educational program offered at Bunche–Da Vinci Learning Academy for this particular community of children and families?" (p. 44)

#### Educational Outcomes for Children

"2. To what extent and in what ways are the children at the Bunche-Da Vinci Learning Academy attaining meaningful and valued educational outcomes?" (ibid.)

## Establishing the Criteria for Making Judgments of Program Quality

Principal
Sandra Martin
lead teacher
We have completed our analysis of what various members of your school community said when we asked them, “What matters to you most in an educational program for children at this school? And how would you know that the program in this school is a good one?” (J: Double-barreled question)
share this draft among the Bunche–Da Vinci staff
Bunche–Da Vinci staff
Criteria for judging educational quality in this context are complicated by
the presence of the Da Vinci program, which likely has its own criteria of quality (which are not yet known),
the diversity of program stake- holders
draft set of general criteria envisioned as possible in this context includes
high quality, based on relevant the- ory and research (for example, research on effective teachers of minority children) and
curriculum and the pedagogy
perspectives and experiences of the Bunche–Da Vinci staff and families.
offer relevant and valuable approaches to learning
curriculum and pedagogy
for all of Bunche–Da Vinci’s particular population of students.
well aligned with state stan- dards in all subject areas,
curricular program and its assessments
especially language arts and mathematics
students at Bunche–Da Vinci
Bunche–Da Vinci Learning Academy community
public responsibility and authority

## The Evaluation Design

rich description of the program and its context (including relevant history)
three parts
begin with
a clear statement of the primary eval- uation purposes and audiences (including the various stakeholder interests in the evaluation),
presentation of the key evaluation questions and critical criteria for judging quality.
Second, the more technical evaluation design section
would spec- ify the overall inquiry design to be used, along with
data-gathering meth- ods and samples,
analysis procedures,
technical quality considerations,
informed-consent procedures
third section of the plan would present information on
reporting plans.
staffing, and
evaluation management
budget

### Overall Design.

an interpretive case study with multiple and mixed methods implemented for purposes of understanding selected phe- nomena more comprehensively and with greater insight and value con- sciousness (Greene, Benjamin, and Goodyear, 2001).

### Methods and Samples.

emphasize
assessments of the quality of the program experience for students
and teachers
also include analyses of relevant achieve- ment test data

#### Curriculum Review.

reviewed by approximately five experts in curriculum and the teaching of children like those in the Bunche–Da Vinci school population
address
questions of
curricular quality
using care- fully identified criteria.
curricular fit to the Bunche–Da Vinci school population
elements representing the Da Vinci program’s underlying theoretical rationale
likely to include
principal developers of this program at the Da Vinci Learning Corporation
factors drawn from the research literature
interviewed (J: what about documents? Documents -> Interview)

#### Classroom and School Observations

One class at each grade level
iden- tified as “representative or typical”
Classes
require teacher and parent con- sent.
observations will be made about once a week for about half of the school day during the months of October, November, February, March, and April, scheduled so that all components of the curriculum are observed mul- tiple times.
observations will generate rich descriptions of the learning activities, interactions, and environments of these classrooms.
focuses
include
engagement and learning of different kinds of children in the various components of the curriculum
character of the integrated in-class services provided
how
time
is used
the evaluation team will record field notes of unstructured observations conducted at other sites in the school, including
lunchroom
playground
fif- teen teachers will be individually interviewed

#### Teacher and Administrator Interviews

selected to be collectively representative of the school’s faculty, along dimensions of
sample
education
classroom or specialist assignment
experience
longevity at the school
Very new teachers and teachers new to the school will be excluded
race and ethnicity
commitment to the school
(J: less likely to provide "rich" information)
With the consent of the teachers’ union and

#### Information on Teacher Quality.

the district and school administrators
group-level evaluative information about the Bunche–Da Vinci teaching staff over the past three years
profile of strengths and weaknesses
weaknesses
If access to this information is denied, the eval- uation team will consult with the administration and faculty at Bunche–Da Vinci regarding other avenues to secure data on teaching quality.

#### Parent Interviews.

Six group interviews of parents
five to ten parents
in each group
formed primarily around the demographics of the school
will include
1) one group of Latino parents of children who have good English fluency
2) one group of Latino parents of children in the Structured English Immersion classes
(3) one group of African American parents
5) one group of parents of children in special education (J: Short Essay Prompt 1: What percentage of the possible combination of named parent groups is this? Short Essay Prompt 2: What percentage of students does this combination reach?)
(4) one group of parents of children in Title I
For all groups, parents will be sampled for representativeness, along
longevity in the commu- nity
age
gender
work
income status
number of children in the school
longevity in the Bunche–Da Vinci school
(J: LOGISTICS:)
Child care, transportation, and food will be provided for all group interviews.
interviews will be conducted in Spanish if the Spanish-speaking parents prefer this
major focus of these interviews will be to
gather parent descriptions of and evaluative reflections on their children’s learning experiences in this school
sense of connectedness and commitment to the school.
their

#### Test and Test Score Analysis

full content analysis
comparison of
teacher- developed assessments
Da Vinci program tests
state tests and standards
with the consultation of
additional content experts as needed
two experts in psychometrics
stu- dent performance on all of these tests over the past four years, using the first year before the Da Vinci partnership as a baseline, will be analyzed by stu- dent subgroup and skill area.
analysis will endeavor to identify the spe- cific areas of relative strength and weakness displayed by various subgroups of students
for sub- groups and specific skill areas
patterns in performance over time
summary of methods as connected to evaluation questions
Table 3.1. (J: Communication note: Put links to summary tables (or the tables themselves) at the BEGINNING of documents)
= Reporting = ("June 5, 2004")
reporting section of our evaluation plan
periodic brief reports on the evaluation’s progress first in writing and then in person, including time for questions, comments, and dis- cussion
We believe that ongoing engagement in the evaluative issues being pursued, especially by your staff and interested parents, can enhance the potential power of the evaluation to be a meaningful and valuable learning activity.
for this first year of the evaluation, can we reserve one of your faculty meetings for evaluation reporting and discussion in mid- November, one in early March, and then one in late May?
thought that it could be useful to have a joint pub- lic forum for all interested members of the Bunche–Da Vinci community to discuss with one another the emerging findings and continuing issues being engaged in the evaluation.
we
\LOGISTICS\ We could organize a pasta and salad dinner for the occasion, and we would have interpreters there for the Spanish-speaking parents /LOGISTICS/
every three to four months, the evaluation team will present written progress reports to
leadership of the Bunche–Da Vinci Learning Academy (principal and lead teachers), the district leadership (superintendent and identified others), interested parent and community groups, and designated representatives of the Da Vinci Learning Corporation.
outline in brief, non- technical form the major evaluation activities accomplished since the last progress report
reports will
activ- ities upcoming
highlights of recent evaluation findings
third, the value-engaged evaluator will seek opportunities for meaningful dialogues with stakeholders and especially opportunities for stakeholders to engage A VALUE-ENGAGED APPROACH 43 44 THEORISTS’ MODELS IN ACTION in dialogue with one another about
Resources permitting, a Spanish version of all progress and annual highlights reports will be prepared for the Spanish- speaking members of the community.

## The Evaluation Process

responsive to the issues and concerns of all legiti- mate stakeholder groups
requires an on-site presence and an ear keenly tuned to the multiple and diverse rhythms of lived experiences in this particular community
provide spaces and places for thoughtful, data-informed reflections on practice (J: How do you measure this??)
also fundamentally educative
attend
to the substantive and methodological dimensions of the evalua- tion, but also its relational dimensions
to how the evaluator is present in the context at hand
to evaluation itself as a moral and ethical practice (Schwandt, 2004).
(Schwandt, 2004).
enacted in three principal ways.
ry to establish respectful, reciprocal, partnering relationships with all stakeholders
Respect and reciprocity can be communicated, for example, by assuming that all stakeholders have something valuable to contribute to the evaluation and by listening well to each one (Greene, 2003).
Second, the evaluation pro- cess will be an open and transparent one that invites participation by mul- tiple stakeholders in all phases, but especially at the front end, when the evaluation agenda is being established, and at the back end, when inter- pretations of results and action implications are discussed.
inclusion of all legitimate stakeholder perspectives in the evaluation and anchoring of the evaluative work in the particularities of this context
Less important is actual stakeholder partici- pation in the nitty-gritty activities of data gathering and analysis, although stakeholders would be welcome in these processes as well
in dialogue with one another

## Meta-Evaluation

near the end of the two years of this evaluation project, a modest meta-evaluation will be commissioned for this value-engaged evaluation of the Bunche–Da Vinci Learning Academy
Two well-regarded evaluators will be contracted to evaluate this evaluation.
evaluator who advocates responsiveness and learning in evaluation practice
evaluator with a different evaluative philosophy and practical approach
evaluate this evaluation on the basis of
1) criteria intrinsic to this approach, (2) the widely accepted evaluation standards (utility, feasibility, propriety, accuracy), and (3) her or his own ideas about what constitutes “good” eval- uation practice