**San Diego Striving Readers Assessment System**

**An assessment system that uses the**

**Berkeley Evaluation & Assessment Research (BEAR) Center**

**approach to embedded classroom assessment.**

The San Diego Unified School District Striving Readers Assessment System
was developed using the Berkeley Evaluation & Assessment Research (BEAR)
Center approach to embedded classroom assessment, known as the BEAR
Assessment System (BAS). The BAS is an integrated approach to developing
assessments that provide meaningful interpretations of student work
relative to the cognitive and developmental goals of a curriculum (NRC,
2001). It consists of four building blocks: progress variables, an items
design, an outcome space, and a measurement model (Wilson, 2005).

# [Progress Variables]{.underline}

Progress variables are the particular concepts and skills that form the
core learning goals of the curriculum. They are organized according to
the principle that assessments should reflect a developmental
perspective of student learning in which deeper understandings are
developed from, and take the place of, earlier understandings as
students progress toward higher levels of sophistication and competence
(Wilson, 2005).

In the San Diego Striving Readers Assessment System, progress variables
were derived in part from the critical literacy skills identified in
past research (McDonald & Thornley, 2002, 2005) and in part from
empirical research on how students performed on pilot versions of the
SLIC assessments. Importantly, they map onto the scope and sequence of
the curriculum itself such that they should mirror the learning and
development expected to take place in the classroom. The curriculum is
focused on orienting students to texts using the form and surface
features of expository, persuasive text and narrative texts. Teachers
alert students to and question them about the ways in which authors
provide information in headings, sub headings, tables, maps, etc., and
of the ways that grouping this information can provide the reader with
background knowledge of the information to be conveyed in the running
text. In narrative text the focus moves into the structure of the plot
and the author's use of language and literary features. The SLIC
Progress Variables describe the development of literacy skills as
progressing from a foundation of using surface features to navigate and
predict the content of unfamiliar text, to a deeper understanding of
content within and across texts, to an ability to infer meaning of
unfamiliar vocabulary from context, to a critical appreciation of
authorial intent and the choices authors make in crafting text. These
variables were developed in partnership with the San Diego Unified
School District and the curriculum developers.

# [Items Design]{.underline}

The items design is a framework for designing tasks to elicit specific
kinds of evidence about student knowledge that can be interpreted in
terms of the progress variables. This building block is grounded in the
principle that assessment should be embedded in normal classroom
activity and based upon authentic instructional tasks.

In the Striving Readers Assessment System, assessment items were derived
from the skills and strategies of proficient readers described in the
progress variables and correspond to specific teaching points and
instructional activities in the SLIC curriculum. Examples of such
activities include anticipating the content of a text from the text's
surface features, determining the main ideas of a paragraph, and
inferring an author's implicit meaning. Each of these activities is
closely linked with an item type, a template for a suite of assessment
items associated with different texts that each measure the same skill
or strategy.

Sixteen full-length assessments have been developed by applying these
item types to three types of texts---expository, persuasive, and
narrative---across 4 grade levels (7^th^ through 10^th^). The assessment
system includes a pre-test, post-test, and several intermediate
benchmark assessments at grades seven, eight, nine, and ten (sixteen
assessments; 4 per grade level). The goal is for teachers to monitor the
progress of their students throughout the year and to allow the tracking
of individual students across grade levels.

# [Outcome Space]{.underline}

The outcome space describes the qualitatively different kinds of student
response elicited by the items and maps these classes of response to the
levels of the progress variables, operationalizing the principle that
teachers should be the primary managers of assessment in the classroom.
The outcome space is the evidentiary foundation for teachers to use on a
daily basis, in both formal and informal instructional contexts, to
judge the progress of their students along the progress variables and
make informed decisions about future instruction.

In the Striving Readers Assessment System, the outcome space was
informed by previous research (McDonald & Thornley, 2002, 2005) and
further refined by analyzing student responses to pilot versions of the
assessments. Each activity described above, such as determining the main
ideas of a paragraph, is a complex skill requiring attention to and the
coordination of multiple considerations. Rather than relying on single
clues, proficient readers consider multiple aspects of the text and
cross-check their meaning; we call this applying and cross-checking
multiple *tactics* in the service of a particular skill or strategy. The
Striving Readers Outcome Space is a general rubric that describes more
and less proficient application of tactics. This outcome space is
applied to each item, resulting in a collection of item-specific scoring
guides that share a common framework.

In 7^th^ grade, for example, students were required to read a persuasive
text extolling the benefits of exercise and describing an exercise
program. Items were developed to assess student understanding of the
progress goals. For example, in order to fully understand the main idea
of the paragraph, the students had to be able to pull together
information from several different sentences within the paragraph and
discard those sentences which were irrelevant to the main idea. These
tactics were identified earlier, and the progress goals developed from
the tactics. Once pilot data was collected, the next step of the process
was to go through student responses and decide qualitatively when
students reached a particular level of progress. The process was
repeated across items, instruments, and grades.

# [Measurement Model]{.underline}

The measurement model provides estimates of person proficiency and item
difficulty calibrated onto an interval scale using a multidimensional
item response model (Adams, Wilson, & Wang, 1997), upholding the
principle of sound standards of validity and reliability. The creation
of a calibrated, quantitative version of the progress variables allows
teachers to track and map the progress of individual students and groups
as they undergo instruction; this is facilitated with an integrated
suite of technological tools (Kennedy, Wilson, & Draney, 2005) that
allow the teacher to focus on her crucial roles as evaluator and
interpreter of student work.

In the Striving Readers Assessment System, the measurement model allows
us to both quantitatively measure the progress of students across all
four years of the study along a common interval scale and to interpret
those numerical estimates of proficiency in terms of the qualitative,
meaningful levels of the progress variables. Furthermore, we can easily
map change over time as the student progresses through the curriculum
and across grades. These findings will also inform the evaluation piece
of the project.

[Conclusion]{.underline}

The BEAR assessment system starts from the position that the assessment
instrument is always secondary---there is always a purpose for which the
instrument is needed and the context in which it is being used (Wilson,
2005). It entails defining a construct---a theoretical or practical
object of interest---prior to developing any item or any method of
scoring responses. Furthermore, the process is collaborative. The
development of the assessment is designed with the goals of the
curriculum and teachers in mind.

For this project, assessments were developed, piloted, and designed in
partnership with school district leaders and educational researchers and
refined based on feedback from teachers and coaches. While effective,
the process was not always easy. The project has gone through several
iterative stages of defining progress goals, creating items, and
specifying what a specific response in the outcome space means
substantively and how it can be reliably scored by teachers. For
example, rater effects were discovered across teachers---in the pilot
test teachers had very different ways of scoring particular student
responses. This has led us to revise our progress goals, outcome spaces,
and the best way to score the responses. Ultimately, however, we are
confident that this will lead to an assessment system for the SLIC
program that will serve the purposes of practitioners and researchers
alike.

**References**

Adams, R. J., Wilson, M., & Wang, W. (1997). The multidimensional random
coefficients multinomial logit model. Applied Psychological Measurement,
21, 1-23.

Belzer, A. (2005). Improving professional development systems:
Recommendations from the Pennsylvania adult basic and literacy education
professional development system evaluation \[Electronic Version\].
*Adult Basic Education, 15*(1), 33-56.

Kennedy, C. A., Wilson, M., & Draney, K. (2005). GradeMap 4.1 (computer
program). Berkeley, CA: University of California-Berkeley, BEAR Center.

McDonald, T., & Thornley, C. (2002). Unlocking meaning in text: some
thoughts on the literacy challenges our students face. *English in
Aotearoa*(48), 54-60.

McDonald, T., & Thornley, C. (2005). Literacy teaching and learning
during the secondary years: Establishing a pathway for success to NCEA
and beyond. *set: Research Information for Teachers, 2*, 9-14.

Mizell, H. (2003). Facilitator: 10 refreshments: 8 evaluation: 0
\[Electronic Version\]. *Journal of Staff Development, 24*(4).

National Research Council (2001). Knowing what students know: The
science and design of educational assessment. Washington, D.C.: National
Academy Press.

Poulson, L., & Avramidis, E. (2003). Pathways and possibilities in
professional development: case studies of effective teachers of
literacy. British Educational Research Journal, 29(543-569.

Thornley, C., & McDonald, T. (2002). Reading across the curriculum:
Secondary students talk about themselves as readers. *set: Research
Information for Teachers*(1), 19-24.

Wilson, M. (2005). Constructing measures: An item response modeling
approach. Mahwah, NJ: Erlbaum.

Wilson, M. & Scalise, K. (2003). Reporting progress to parents and
others: Beyond grades. In J. M. Atkin & J. E. Coffey (Eds.), Everyday
assessment in the science classroom. NSTA Press: Arlington, VA.

Wilson, M. & Sloane, K. (2000). From principles to practice: An embedded
assessment system. Applied Measurement in Education, 13, 181-208.