# ![](media/image1.wmf){width="4.986111111111111in" height="0.84375in"}

**Assessment**

**Task Group**

**Chicago Meeting**

**Progress Report**

**April 20, 2007**

**Task Group Members**

**Camilla Benbow, Susan Embretson, Francis (Skip) Fennell**

**Bert Fristedt, Tom Loveless, and, Sandra Stotsky**

**Ida Kelley, Staff**

**RESEARCH QUESTION**

**Determine the correspondence of NAEP 4th and 8th grade tests to
selected state accountability tests for validity in assessing
mathematics proficiency.**

**FOUR ASPECTS OF VALIDITY**

***Aspects are particularly relevant for comparing NAEP and state
accountability tests.***

**- Content**

**- Substantive**

**- Consequential**

**- Generalizability**

**POSSIBLE DIFFERENCES**

**BETWEEN NAEP & ACCOUNTABILITY TESTS**

**1 - The content may be weighted very differently between and within
strands.**

**2 -- The cognitive complexity of test items, testing the same content
may vary broadly.**

**3 -- The empirical difficulty of items measuring the same content may
vary broadly between NAEP and state accountability tests and between
different states.**

**POSSIBLE DIFFERENCES**

**BETWEEN NAEP & ACCOUNTABILITY TESTS**

**4 -- Tool Inclusion; several modes of testing may have an impact on
assessed proficiency and test validity.**

**5 -- Test delivery mode, particularly computer-based versus paper and
pencil tests.**

**6 -- Item formats (true/false, multiple choice, constructed response,
worked problems, etc) should be examined for impact on proficiency.**

**COMPARISION VARIABLES**

1 - The **proportional representation of content**, as available from
test blueprints, to examine the content aspect of validity.

2 - **Cognitive complexity and conceptual skill level** of items with
the same purported content from an inspection of actual test items, to
examine the substantive aspect of validity.

3 - **Empirical item difficulty between items** with the same content on
NAEP and year-end tests.

4 -- **Tool inclusion.**

5 -- **Test delivery mode.**

6 - **Item format representation,** particularly as crossed with
content.

+--------------+-----------+-----------+----------------+-------------+
|              | **        | **Subs    | **Gene         | **Cons      |
|              | Content** | tantive** | ralizability** | equential** |
+--------------+-----------+-----------+----------------+-------------+
| **           | **X**     |           |                | **X**       |
| Proportional |           |           |                |             |
| Repr         |           |           |                |             |
| esentation** |           |           |                |             |
+--------------+-----------+-----------+----------------+-------------+
| **Complexity | **X**     | **X**     |                | **X**       |
| &**          |           |           |                |             |
|              |           |           |                |             |
| **Skill**    |           |           |                |             |
+--------------+-----------+-----------+----------------+-------------+
| **Item**     |           | **X**     | **X**          | **X**       |
|              |           |           |                |             |
| **           |           |           |                |             |
| Difficulty** |           |           |                |             |
+--------------+-----------+-----------+----------------+-------------+
| **Tool**     |           | **X**     | **X**          | **X**       |
|              |           |           |                |             |
| *            |           |           |                |             |
| *Inclusion** |           |           |                |             |
+--------------+-----------+-----------+----------------+-------------+
| **Test       |           |           | **X**          | **X**       |
| Delivery     |           |           |                |             |
| Mode**       |           |           |                |             |
+--------------+-----------+-----------+----------------+-------------+
| **Item**     |           |           | **X**          | **X**       |
|              |           |           |                |             |
| **Formats**  |           |           |                |             |
+--------------+-----------+-----------+----------------+-------------+

**CONSEQUENTIAL VALIDITY**

Impact of Varying Test Properties on Observed Proficiency Levels of
Significant Groups

\- Gender

\- Racial-Ethnicity

\- English Language Learners

\- Various Disabilities

**CONSEQUENTIAL VALIDITY**

\- If available, group differences compared under different test
preparations

\- Research literature should be examined as well