----

##  State Standards, Assessments and Instruction. ERIC Digest.                   

Most states have put in place curriculum standards and                          state-developed assessments to monitor the implementation of those              standards. Most state standards define expected outcomes, that is,              what students need to know and be able to do, but do not mandate                specific strategies or pedagogy used by local districts. As                     researchers Paul Black and Dylan William (1998) note, standards are             raised only by changing what happens in the classroom, beginning                with teachers and students. This Digest describes a program used by             two educators to help teachers improve instruction through a deeper             understanding of state standards and test                                       specifications. DEVELOPING A KNOWLEDGE BASEUnderstanding how              standards-based state tests are constructed is the first step in                being able to use them to guide and improve instruction. A test is              essentially a sample of questions or activities that reflect a large            body of knowledge and mental processes associated with an academic              subject area  such as language arts, mathematics, science, etc.                 School districts are under pressure to perform well on state tests              and often use a test preparation strategy of giving students sample             tests from commercially prepared workbooks or state-released items              to get ready for state tests. Although this strategy can be useful              for providing general information regarding student strengths and               weaknesses as related to the samples, it should not be the only                 method used by teachers because it does little to educate them about            how to understand and use  state tests, standards, and test                     specifications. This article recommends a three-part process of                 delineation, alignment, and calibration for developing an                       understanding of state assessments and using that understanding to              improve instruction. The New Jersey Core Curriculum Content                     Standards are used to model the process, but it can be applied to               any state's or district's standards. DELINEATIONDelineation is            the process of  identifying all aspects or dimensions of a                      particular subject domain. It involves using state testing documents            that describe each content area of the assessment, including test               specifications, specific skill cluster information, subject area                frameworks, assessment examples and exemplars, and the state                    standards. Delineation requires an examination of these documents               for assessment dimensions such as content, cognitive level and                  complexity. A thorough delineation might also include analysis of               additional factors such as the test format and difficulty level of              the questions. 
The New Jersey Standards include macro, or big                 picture, statements and cumulative progress indicators that provide             details about general performance expectations.  The standards                  include knowledge specifications, which describe the specific                   processes and content that all students must know by the end of                 fourth grade (also known as content standards) as well as                       problem-solving specifications, which describe what students should             be able to do with the content knowledge (also known as process                 standards).  The following example is excerpted from the 4th grade              New Jersey mathematics standards and test specification                         manuals. 
Macro Standard 4.1:  All students will develop the                   ability to pose and solve mathematical problems in mathematics,                 other disciplines, and everyday experiences. 
Cumulative Progress              Indicator 4.1.2: Recognize, formulate, and solve problems arising               from mathematical situations and everyday experiences. Test                     Specification Manual - Cluster IV Discrete Mathematics:                         Knowledge (content standards): Students should have a conceptual                understanding of: Tree diagram 
Problem Solving (process                       standards):  In problem solving settings, students should be able               to:  Draw and interpret networks and tree diagrams 
After reviewing            the entire body of standards and test specifications for                        fourth-grade mathematics in New Jersey, a teacher would be able to              identify seven distinct mathematics strands or dimensions:                      Numeration and Number Theory, Whole Number Operations, Fractions and            Decimals, Measurement/Time/Money, Geometry, Probability/ Statistics,            and Pre-algebra. The test specifications for that exam imply that               the mathematics test questions are primarily composed of                        problem-solving tasks. Therefore, it is safe to assume that test                questions will require thinking in the application, analysis, and               perhaps synthesis and evaluation levels of cognition. 
Once                    teachers and administrators specify all of the subject area                     dimensions, the following activities can begin: 
* Selecting and               designing classroom assessments and practice questions. 
* Revising            and designing curriculum that is congruent with the content                     identified in the state standards and the district's delineation of             the state-designed exams 
* Designing teacher training using                   instructional techniques that support these                                     dimensions. ALIGNMENTDuring the alignment phase,                          administrators and teachers work to identify, analyze, generalize,              and describe the links between the various elements associated with             the subject area previously delineated and the sample questions                 selected for practice or classroom activities to assess student                 progress. The sample questions and student assessments can be                   derived from several sources, including state-released test items,              commercially manufactured test preparation materials, or                        teacher-made activities. Teachers and administrators examine                    linkages in the materials, organization, textbooks, instructional               strategies and other elements described in the curricula and used in            daily instructional activities to ensure consistency with the                   district's delineation of the state assessment. 
Using and                     understanding the test specifications become even more important at             this stage.  Returning to the example above, a teacher seeking to               ensure that students would be able to understand tree diagrams and              solve problems using them would complete several alignment                      tasks: 
1. Review classroom resources, curriculum, textbooks,                  teacher activities, student thinking strategies, and tests to ensure            that the  test specifications and macro standards are addressed on              the knowledge and problem solving level.  Do the teacher resource               materials and classroom instruction address the proper skills? 
2.             Review the above factors to ensure congruency between the level o               difficulty required by the standards and specifications, and the                difficulty of the actual teacher resources and activities.  Do the              teacher's tests, lessons, and activities match the difficulty level             required by the standards and specifications? 
3. Consider format.             Although less important than skills and difficulty,  the teacher                resources, activities, and tests should familiarize the students                with state test question formats. Teachers must align classroom                 assignments and activities to the subject area delineation to ensure            congruency. CALIBRATIONAfter completing the delineation and               beginning the alignment processes, calibration begins. Calibration              is the act of conducting communications and interactions with                   teaching staff based on the information identified in delineation               and used in alignment. The calibration process ensures that the                 conceptualization of content, cognitive process, complexity,                    formats, etc., is consistently understood for each subject area.                Calibration, in its simplest form, is designing classroom                       instruction, activities, and assessments that are congruent with                content area delineation and alignment. Using the mathematics                   vignette as an example, one can begin to see how the process takes              place. 
Figure 1.  Represents the sequence of events leading up to             Calibration. 
Delineation 
1. Teacher performs content area                   delineation and chooses a unit of focus. In this case Standard 4.1,             Cluster IV-Discreet Mathematics-tree diagrams 
Alignment 
2.                  Teacher examines a compares classroom resources, local curriculum,              activities, skills level of difficulty, format, and tests to the                Standards and test specifications to ensure                                     congruency. 
Instruction 
3. Teacher uses and implements                      calibrated activities, tasks, resources and                                     lessons. 
Calibration 
4. Teacher designs lessons and activities,             gathers resources, and creates tests that are congruent with the                skills and level of difficulty for the Standards, test                          specifications and curriculum. 
Figure 1. Delineation, Alignment,              and Calibration Flow of Events 
Matt has four channels on his                  television. He has channels 2,3,4 and 5. If Matt watches only two               channels each night, haw many different combinations of channels can            he watch? Show all your work, and explain your answer. Matt can                 watch (2,3), (2,4), (2,5) or (3,2), (3,4), (3,5) etc. 
A 4th grade             teacher completing delineation and alignment and discovering that               her/his program was missing a unit on discrete mathematics would                develop objectives related to understanding, using, and interpreting            tree diagrams. Figure 2 is a sample activity/test question created              by 4th grade teacher Terry Maher to begin addressing the aspect of              discrete math noted in the Cluster IV test                                      specification. 
Calibration is any action that helps teachers                  design activities and construct assessments based on the dimensions             of state assessments and standards. This process helps to foster a              collective understanding and agreement of the dimensions and domains            of each content area.  It should be a team effort based on group                inquiry. USING SCORE REPORTS TO IMPROVE CALIBRATIONAs teachers            gain a better understanding of how student work reflects the                    standards and test specifications through delineation, alignment,               and calibration, their efficiency and accuracy at identifying which             students are meeting the standards should increase. Herein lies the             usefulness of score reports sorting students into categories of                 varying proficiency.  A student who scores partially proficient,                proficient, or advanced proficient on a state language arts test may            also show some congruency in the level of achievement in his/her                well-aligned school work and classroom assessments. As teachers                 become better calibrated, they will be able to answer questions such            as: What level of proficiency does this student show on class                   assessments?  Is the difficulty level of the class work comparable              to the state exam?  What can I do to help this student meet the                 state standards?  Is my program meeting the standards? PREDICTING             OUTCOMESTeachers can reflect upon their level of calibration                accuracy by attempting to predict student results on state                      assessments. Teachers should be aware, however, that 100% agreement             should not be expected between a student's performance on                       well-calibrated classroom tests and on the state assessments, based             on many factors of test design. 
To begin the prediction process,              the teacher uses a list of the students taking the test.  Beside                each name, the teacher enters a predicted score level.  When the                state assessment scores arrive, the teacher can compute the level of            accuracy as shown below. 
Name Prediction                                      Score 
Allan          Proficient   Adv. Proficient 
Ann                       Proficient            Proficient 
Tamika         Adv.                          Proficient       Proficient 
Bronson Partial Proficient    Partial             Proficient 
The list above shows a 50% level of success in the                 predictions made. 
The teacher making the predictions returns to               each student's work and compares the successful predictions with the            unsuccessful ones to gain a better idea of how the assessment                   performances reflect the aligned student work. Student work                     associated with actual test scores can form the basis for subsequent            calibration discussions.  Student work connected to state assessment            score levels can also function as scoring examples that students                refer to when judging their own work. FINAL THOUGHTSThere is a            distinct difference between traditional notions of test preparation             and aligning and calibrating instruction and assessments with the               content, cognition, difficulty, and format of state assessment                  instruments, specifications, and standards.  As educators, we are               trying to link the classroom activities to the standards and skills             set by the state. The aim is to ensure that teachers understand, and            calibrate their classrooms with respect to, the entire process and              do not simply focus on how to answer specific types of test                     questions. The questions will change, but the underlying skills and             concepts will not. One must be careful not to wallow in the mire of             test prep. 
Delineation, alignment, and calibration are academic               endeavors that demand unending commitment. Do not expect to                     accomplish alignment or calibration at an in-service day, or even               during the course of a school year. The administration must provide             the time and resources to conduct frequent calibration meetings to              examine such things as classroom work and student assessment                    samples.  Beware, it is easy to fall out of alignment and                       calibration and into test prep. REFERENCEBlack, Paul, and                 Wiliam, D. ( 1998 ). Inside the black box: Raising standards through            classroom assessment, Phi Delta Kappan October, pp.                             139-148. 
----- 
This publication was prepared with funding from              the Office of Educational Research and Improvement, U.S. Department             of Education, under contract ED99CO0032.  The opinions expressed in             this report do not necessarily reflect the positions of policies of             OERI or the U.S. Department of Education.  Permission is granted to             copy and distribute this ERIC/AE Digest.                                     **Title:** State Standards, Assessments and Instruction. ERIC Digest.                   **Document Type:** Information Analyses---ERIC Information Analysis Products (IAPs) (071); Information Analyses---ERIC Digests (Selected) in Full Text (073); **Available From:** ERIC Clearinghouse on Assessment and Evaluation, 1129 Shriver                   Laboratory, University of Maryland, College Park, MD 20742-3742.                Tel: 800-464-3742 (Toll Free). Web site: http://ericae.net.                  **Descriptors:** Educational Assessment,  Educational Improvement, Elementary                    Secondary Education, Information Dissemination,  Instruction,  State            Standards,  Test Construction, Test Use                                      **Identifiers:**  Calibration, ERIC Digests, New Jersey,  Test Specifications                 
###
[Return to ERIC Digest Search Page]