# Presentation: 037096

## MAPP as an Accountability Assessment 

- Brent Bridgeman
- Educational Testing Service
- Center for Validity Research

**Notes:**
Notes

## Overview

- What is MAPP
- Content validity of MAPP
- MAPP norms
- Reliability and relationships among MAPP scores
- Relationship of MAPP to other measures
- Sensitivity to instruction
- MAPP in Europe

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Measure of Academic Proficiency and Progress (MAPPTM) and the Academic Profile

- MAPP was introduced in 2006 as a successor to the Academic Profile
- MAPP was created to the same specifications as the Academic Profile and assesses the same skills (reading, critical thinking, writing, and mathematics) and contexts (humanities, social sciences, and natural sciences)
- Scores on MAPP are equated to Academic Profile
- Two versions
- Standard form—108 questions in 2 hours—ok for individual scores
- Abbreviated form—36 questions in 40 minutes—matrix sample for group scores only
- Two formats—paper-and-pencil and online (proctored or non-proctored)
- Optional essay and option to add locally authored questions

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Validity of the Measure of Academic Proficiency and Progress as an Accountability Measure

- Tests cannot be validated, but rather “the extent to which the proposed uses and interpretations are plausible and appropriate” (Kane)
- MAPP has several uses
- Counseling tool to identify students with skill deficits
- Track growth of individual students
- Track trends over time in skills of freshmen and seniors
- Curriculum improvement
- Serve as one component of a value added assessment, assessing general education core curriculum skills in critical thinking and writing

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Content Validity--What Does MAPP Measure?

- “Academic skills developed through general education courses,” and NOT subject matter specifically taught in those courses
- No attempt to measure such areas as:
- Graphic artist thinking critically about use of negative space
- Structural engineer thinking critically about how to maximize both safety and efficiency in a bridge design

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Content Validity--What Does MAPP Measure?

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Content Validity--What Does MAPP Measure?

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## MAPP Content Validity

- Ultimately, content validity determination should be made locally--from the MAPP web site:
-     While the MAPP test is developed to measure a common core set of skills, judging the appropriateness of a test for a particular department or program is an important first step in considering the use of an outcomes assessment. Faculty members should review the test content and skill area coverage to determine whether it is consistent with your institution’s expectations.
- Hundreds of institutions have determined that MAPP assesses relevant general education outcomes

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Does MAPP have Meaningful Norms?

- Banta is rightly concerned that norms not be based on small samples
- MAPP norms are based on over 375,000 students from more than 380 institutions
- MAPP norms are provided separately by Carnegie Classification, and institution may customize a peer group for comparison

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Sample Norms

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## 					...

- 					**...**

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Construct Validity—Are the General Academic Skills in MAPP a Single Construct?

- Reliabilities of all four skills in the .80’s
- True correlations among skills:

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Importance of the Skills Assessed by MAPP

- Core academic skills in critical reading (thinking), quantitative reasoning, and writing are strongly related to high levels of success in college and beyond

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Relationship of Reading and Quantitative Reasoning Skills to College Success 

- Define success as 3.5 or higher GPA over 4 years

## Are the Critical Thinking Skills Assessed by MAPP Similar to SAT and ACT Scores?

- For 100 students from a “large Midwestern university” who had completed most of their general education requirements, correlation of Academic Profile (short form) with:
- PSAT-V              .61
- PSAT-M             .46
- ACT-Composite .87
- ACT-COMP       .62
- (Hendel, D. D. [1991]. Evidence of convergent and discriminant validity in three measures of college outcomes.  *Educational and Psychological Measurement, 51*, 351-357.)
- For 810 seniors from the University of Tennessee, correlation of Academic Profile with ACT-Composite:  .71
- (Pike, G. R. [1988].  A comparison of the College Outcome Measures Program [COMP] exam and the ETS Academic Profile.  University of Tennessee technical report, ERIC ED 348 369).

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Are the Critical Thinking Skills Assessed by MAPP Similar to SAT and ACT Scores? (con’t)

- For 1,823 seniors from 23 colleges
- Correlations MAPP with SAT V+M:
- 	Reading             .72
- 	Math                  .63
-    Writing               .54
-    Critical thinking  .59
- 	**TOTAL              .77**

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Do High Correlations with SAT/ACT Scores Invalidate Value-Added Estimates?

- High within-college correlations are irrelevant for judgments of value added
- High across-college correlations (with college as the unit of analysis) indicate most, BUT NOT ALL, of the variance among colleges in outcomes can be predicted from pre-admission tests
- With college as unit of analysis, MAPP correlation with SAT V+M is .93
- Good news or bad news?

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Does a High Correlation of MAPP Scores and SAT/ACT Scores Imply that MAPP Could Not be used for a Value-Added Assessment?

- If MAPP scores and SAT/ACT scores measure essentially the same construct, and *if that construct is a fixed, inherited capability*, then MAPP could not measure value added by any instructional program
- The second part of the above statement is clearly not true
- SAT scores increase substantially with increased education
- Scores of seniors substantially above scores of 7th and 8th grade students
- “the verbal and mathematical reasoning skills measured by the SAT may be enhanced to a measurable degree by long-term and intensive instruction”  (Messick, S., & Jungeblut, A. [1981].  Time and method in coaching for the SAT.  *Psychological Bulletin, 89*, 191-216).

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Are Academic Profile Scores Related to College-Defined Success (Grades)?

- Mean Critical Thinking scores:
- For students with GPA below 2.50 = 110 (n = 1145)
- For students with GPA above 3.49 = 117 (n = 837)
- SD = 6, so *d* =  1.2
- Similar findings for Reading, Writing, and Mathematics
- (Marr, D. [1995].  *Validity of the Academic Profile*.  Princeton, NJ: Educational Testing Service)

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Are Academic Profile Scores Sensitive to Instruction?

- Mean Critical Thinking score:
- for students starting core curriculum = 109 (n=214)
- for students completing core curriculum = 114 (n=1739)
- SD = 6.0, so *d *= .8
- (Marr, D. [1995].  *Validity of the Academic Profile*.  Princeton, NJ: Educational Testing Service)
- BUT this result could simply be the result of less able students dropping out

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Are Academic Profile Scores Sensitive to Instruction (continued)

- Compare Academic Profile scores of a *longitudinal* sample of freshman and seniors
- Sample is 112 students at a small Midwest university
- Freshman mean = 448
- Senior mean = 453
- SD = 17, so *d* = .4
- Must assume equal motivation of freshmen and seniors, so may be an underestimate
- (Weimer, D. [1999].  Dealing with messy data: Analyzing pre- and post-test assessment results in the real world.  Paper presented at the Annual Forum of the Association for Institutional Research, Seattle)

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Are MAPP Scores Sensitive to Instruction?

- Freshmen and Seniors from 23 colleges; college is unit of analysis

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## What are Threats to Validity of Value Added Scores?

- Construct irrelevant variance
- Motivation
- Non-representative sample
- Construct underrepresentation
- Banta and Pike (2006) argue, “testing students’ generic as well as discipline specific knowledge and skills in their major fields of study provides a much more promising avenue for assessing growth and development in college than do today’s most commonly discussed tests of generic skills alone”

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## How to Address the Motivation Problem

- Screen unmotivated examinees
- Self-reports
- Timing
- Monetary incentives *for performance*
- Appeal to value to students of having recognized high-performing school
- Despite these concerns, high correlations with SAT scores suggest motivation not currently a major problem

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## How to Address Sampling Problem

- Random sample--avoid volunteer sample
- Test all seniors
- Avoids self-selection problem
- Permits comparisons among different majors/programs

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## How to Address Construct Underrepresentation

- In addition to a general skills test, such as MAPP, assess knowledge and skills that are specific to particular majors (e.g. Major Field Tests)
- Field-specific tests are likely to show much stronger growth than general skills test

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Issues in Using MAPP in Other Countries

- Institution must evaluate extent to which skills assessed *in English* are relevant to the institutional mission
- Norm groups not comparable
- Translating words will not necessarily create a test in another language that assesses the same construct as MAPP

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.

## Review and Next Steps

- MAPP is:
- Content valid
- Reliable
- Correlated with related constructs
- Sensitive to instruction
- Future research
- FIPSE study relating scores on CLA, CAAP, and MAAP currently underway
- Initial value added study of 23 current MAPP users will be expanded as more institutions provide relevant data
- ETS will work with individual institutions on local validity studies

- Confidential and Proprietary. Copyright ©2008 Educational Testing Service.