# Presentation: 503252

## The Health Literacy Component (HLC) 

- The Health Literacy Component (HLC)
- of the 2003 NAAL

- Sheida White
- Project Officer National Assessment of Adult Literacy
- (NAAL)

## Topics covered by this presentation

- Performance of literacy tasks
- Skills required to perform literacy tasks
- Assessment design and administration
- Data analysis and reporting

## Background, Key Features, and Major Goals

## NAAL background

- The 2003 National Assessment of Adult Literacy (NAAL) builds on and expands previous NCES adult literacy assessments
- NAAL is the first assessment of the nation’s progress in adult literacy since the 1992 National Adult Literacy Survey (NALS)

## Key features of NAAL

- Provides data for the United States as a whole, individual states choosing to participate in a state-level assessment, and the nation’s prison population
- Emphasizes the use of printed everyday materials (such as bills, prescriptions, newspapers) needed to function adequately in one’s environment
- Provides data on background characteristics, performance of everyday literacy tasks, and skills underlying task performance

## Major goals

- Report on trends between 1992 and 2003
- Identify relationships between literacy and selected characteristics of adults
- Provide new information about skills underlying adult literacy, including basic reading skills of the least-literate adults
- Facilitate use of NAAL data by diverse audiences, including policymakers, researchers, and educators

## Performance of Literacy Tasks

## NAAL’s task-based literacy definition

*	**Literacy is the ability to use printed and written* *infor-mation *[prose, document, and quantitative] *to function in society, to achieve one’s goals, and to develop one’s knowledge and potential*

- S. White 8-04

## Definitions of three key literacy areas

## Measurement of prose, document, and quantitative literacy

- NAAL’s prose literacy, document literacy, and quantitative literacy scale scores are comparable to those of the 1992 NALS
- Scores are based on completion of items that are all
- Classified as prose, document, or quantitative
- Open-ended to represent everyday tasks
- Placed before stimulus materials to set a purpose for reading
- Most items require
- Short written responses
- Searching the text for specific information

## Topics of NAAL stimulus materials

## Sample prose item used in 1992

- S. White 8-04

## Sample document item used in 1992

- S. White 8-04

## Sample quantitative item used in 1992

## A new health literacy score based on health-related items

- All participants receive some health-related items
- All health-related items are also classified as prose, document, or quantitative items
- The health-related items
- Are included with other items when calculating the prose, document, and quantitative scale scores
- Are used to calculate a separate health literacy score, based solely on health-related items

- A new health literacy score based on health-related items

## Development and focus of the NAAL Health Literacy Component

- NAAL focuses on one key aspect of health literacy—the ability to use printed and written materials to accomplish a broad range of health-related tasks
- NAAL’s health-related tasks include
*Clinical tasks,* having to do with the health care itself
*Preventive tasks,* having to do with healthy habits and prevention of illness
*Navigation tasks,* having to do with bureaucratic demands to get to the health care

- Development and focus of the NAAL Health Literacy Component

## Skills Required to Perform Literacy Tasks

- Skills Required to Perform Literacy Tasks

## Purpose of the Adult Literacy Supplemental Assessment (ALSA)

- The 1992 NALS provided little information about the literacy abilities of the least-literate adults, who were not able to complete the assessment
- The purpose of the new Adult Literacy Supplemental Assessment (ALSA) is to get a clearer picture of the skills of the least-literate adults
- ALSA is intended only for adults who would not be capable of meaningful participation in the “main NAAL” assessment, because they lack the skills to perform even the easiest tasks with a high degree of success
- These adults take ALSA *instead of* the main NAAL

## Selection of ALSA participants

- All NAAL participants take seven easy core screening items
- Interviewers score participants’ answers using the scoring guides provided on their computers
- Computer selects ALSA participants using an empirically derived algorithm that predicts very low performance
- The algorithm predicts which participants would get a total of five items or fewer correct (including the core items)
- Five items, spread across the three NAAL scales, would not be sufficient to allow accurate estimates of performance on any of the scales

## Important features of ALSA

- ALSA instructions and responses are given orally and can be in either English or Spanish
- ALSA materials are
- Contextualized (with logos, pictures, etc.)
- Familiar (as measured by background questions)
- Tangible (e.g., food boxes, medicine bottles, utility bills)
- ALSA tasks, although simple and contextualized, still require participants to read at least parts of the words

## The need for additional data about skill deficits

- In addition to the 6% of adults who lacked the skills to complete the 1992 NALS, an additional 15% of adults were able to perform only the easiest NALS tasks (classified as “Level 1” tasks)
- For adults with low scores, more information is needed about the nature of the skill deficits
- Lack of basic (word-level) reading skills?
- Lack of fluency in basic skills?
- Lack of higher level literacy skills?

## Range of skill deficits and their potential implications

## New data on the skills of all adults

- The Fluency Addition to NAAL (FAN) provides new data on the skills of *all* adults
- After completing either the main NAAL or ALSA, all participants take FAN, which
- Consists of timed oral reading tasks
- Describes the oral reading skills of adults (e.g., words read correctly per minute)
- Provides a measure of basic (word-level) reading skills
- FAN scores can be compared with main NAAL scores at various performance levels, ALSA scores, and health literacy scores

## FAN oral reading materials and associated measures

*Pseudoword lists,* consisting of possible but nonoccurring English forms (e.g., “wike”), provide a measure of adults’ ability to “decode” (or identify the sounds of) words with which they are not familiar
*Word lists,* consisting of English words arranged in increasing order of difficulty, provide a measure of adults’ ability to recognize familiar words (often referred to as “sight words”) as well as to decode
*Text passages,* consisting of 150–200 words each, provide a measure of adults’ ability to read words in connected texts

## Initial measures and scoring challenges for FAN

- Initial FAN target measures are
- Total words read aloud—whether correctly or not—per unit of time*
- Words read correctly per unit of time*
- Words read correctly as a percentage of total words read
- Challenges in scoring FAN data include ensuring
- That correctness can be measured reliably
- That speakers of nonstandard varieties of English are not unfairly penalized
- *Unit of time: 60 seconds for text passages or 20 seconds for lists

## Key similarities and differences between FAN and ALSA

- Similarities include the following:
- Both measure basic reading skills
- Both have oral instructions in either English or Spanish
- Both require reading words and connected text
- Differences include the following:
- Only ALSA permits the use of compensatory strategies, while FAN does not provide any nonlinguistic clues (such as pictures)
- ALSA connected texts are fewer, easier, and shorter (12 sentences each in ALSA, compared with 150200 words each in FAN)

## The functionality of word-level skills and higher level literacy skills

- NAAL defines literacy as the ability to *use* printed and written information
- As defined by NAAL, *all* literacy is functional; therefore, NAAL does not differentiate between “literacy” and “functional literacy”
- Certain skills, including word-level reading skills, are needed to successfully perform NAAL tasks
- The ability to read words *is* a functional skill, but not the only one
- In addition to word-level skills, NAAL recognizes six types of literacy skills

## The six types of literacy skills

**NAAL literacy skills**

**Language comprehension skills**

- Understanding the structure and meaning of sentences

**Text ****search skills**

- Searching textefficiently

**Inferential ****skills**

- Drawing appropriate text-based inferences

**Application skills**

- Identifying the calculations required to solve quantitative problems

**Computation identification skills**

**Computation performance skills**

- Performing any required calculations

- Using newly searched, inferred, or computed information to accomplish a variety of goals

## Analysis of main NAAL data in terms of underlying skills 

- Recently developed multidimensional analysis methods will be applied to 2003 main NAAL data in order to
- Characterize the proficiency of adults in terms of the NAAL literacy skills
- Characterize effects of task demands and written materials on the difficulty of literacy tasks

## Assessment Design and Administration

## Selection of the national sample for NAAL 

## Key features of the NAAL sample

- Adults age 16 and over living primarily in households
- Nationally representative sample of 10,006 adults, augmented by
- Samples from the six states that participated at the state level (an additional 7,166 adults)
- Prison sample of inmates (about 1,000 more adults)
- Incentive payment of $30 to increase representativeness of the sample and response rate
- Oversampling of Blacks and Hispanics

## Matrix sampling

- The main NAAL has a total of 153 items (including the 7 core screening items that are given to all participants and used to select ALSA participants)
- Matrix sampling—administering only a portion of the non-core items to each respondent—ensures broad coverage while limiting respondent burden
- Each respondent takes 1 of 26 booklets
- Each booklet includes 3 of 13 blocks of items (for about 40 items per booklet, including the core items)
- Each block is spiraled in the 26 booklets (i.e., paired with every other block)

## 13 blocks total

- Allowing comparisons with the 1992 NALS while introducing new features

- 1992 assessment

- 2003 assessment

**Six of the blocks from 1992 are used again in 2003.**
**Seven new (2003) blocks replace all six of the blocks from 1985 plus one of the blocks from 1992.**

- S. White 8-04

## The seven blocks that were newly created for 2003

- Are similar to the 1992 blocks in terms of the distribution of skills required by tasks
- Have approximately the same average task difficulty as the 1992 blocks
- Are spiraled with the 1992 blocks in the 26 booklets
- Item response theory will be used to link the 1992 and 2003 scales using the tasks common to both assessments

- Allowing comparisons with the 1992 NALS while introducing new features (continued)

## New performance levels for 2003

- In 1992, five performance levels were developed by grouping together tasks of similar difficulty
- The 1992 levels proved to have some limitations
- Scores that are the cut points between levels vary depending on the measure of task difficulty used
- No distinctions are made within the lowest level of performance (Level 1)
- For 2003, new performance levels are being developed for the main NAAL by the National Academy of Sciences
- Goals of the new levels include increased clarity and better differentiation among adults at the lower end of the scale

## 	                                                      Main NAAL	ALSA	

- 	Background questionnaire	28 min	28 min
- 	Core screening items	  8 min	  8 min
- 	Main NAAL or ALSA assessment	39 min	25 min
- 	FAN	_14 min_	_12 min_
**Total****	****89 min****	****73 min**
- Respondents may take as long as they need to perform main NAAL or ALSA tasks, but are encouraged to move to a new task when stuck or frustrated

- NAAL total average assessment time

## Participants with a native language other than English or Spanish may take the core screening items even if they cannot complete the background questionnaire

- The assessment is conducted in the participant’s home
- The assessment is administered one on one
- All participants receive additional time to complete the main NAAL or ALSA if they need it
- The background questionnaire is administered orally in either English or Spanish
- Instructions for the core screening items and for ALSA are given orally in either English or Spanish

- Accommodations for adults with special needs

## Expanded NAAL background questionnaire

- For 2003, the background questionnaire was expanded to better reflect
- Economic and technological developments in the past decade
- The background of low-performing adults
- Health-related activities of adults
- Literacy-related activities of parents and their children

## Characteristics of the NAAL scoring rubrics:

- Statistical considerations

- Responses to each item are analyzed for
*Frequency *with which a particular response is given
*Interrater reliability *in scoring responses
*P-value *(percentage of the population responding correctly)
*Biserial (and polyserial) correlation *between each response (and partially correct response) and the respondent’s overall score
*Range findings *for delineation of the range of responses received
*Differential item functioning (DIF) *in favor of certain  groups

## Seek evidence that adults can indeed use printed materials

- Allow for partially correct responses if the information provided is still useful in accomplishing the task
- Score most responses as either correct or incorrect, since often a partially completed task is not an accomplished task
- Allow for writing errors as long as the overall meaning of the response is correct
- Distinguish responses providing specific information from those providing generalizations

-  Development of the NAAL scoring rubrics:Content considerations

## Data Analysis and Reporting

## -

- The new software will reduce the amount of time required to analyze data by several months
- The software will allow the technical report and the Data File Users Manual to be produced on a more timely schedule than in 1992
- The software will also make it easier for states and researchers to do their own data analyses to supplement the information in the state and national reports

-  New software for data analysis

## Average scores and distributions

- Average scale scores and score distributions will be
- provided
- For the population as a whole
- For population groups
- But not* *for individuals

## Inappropriateness of individual scores

- No respondent takes all of the assessment
- It would not be appropriate to calculate individual scores based on part of the assessment because
- The individual scores would be unreliable (a reliable or consistent measure requires many items)
- The individual scores would not be a valid representa-tion of the domain of adult literacy (valid representation requires many types of tasks, genres, etc.)

## Trend analysis

- Results from the 1992 NALS and the 2003 main NAAL will be compared in terms of
- Average scale scores for prose, document, and quantitative literacy
- Performance levels set by the National Academy of Sciences
- P-values (the percentage of the population giving the correct answer to each item)

## New release of items and p-values

- About 90 assessment items used in 1992 assessment will soon be released
- All released items will be available on the NAAL website (_http://nces.ed.gov/naal_)
- Each item’s p-value will be published
- This will be the first publication of p-values for 1992 assessment items
- P-values will be published for all items used in 1992—even items that are not released

## Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #1

- Test Questions Search Tool on the NCES website (_http://nces.ed.gov/naal_)—Sample screen #1

## Test Questions Search Tool on the NCES website (http://nces.ed.gov/naal)—Sample screen #2

- Test Questions Search Tool on the NCES website (_http://nces.ed.gov/naal_)—Sample screen #2

## New types of results in 2003

- Health literacy scale scores measuring the performance of American adults on health-related tasks
- Skill-based analyses inferring the literacy skills associated with the ability to perform main NAAL tasks
- ALSA scores providing information on the literacy skills and deficits of America’s least-literate adults
- FAN oral reading and basic skills data providing information about the skills of all adults

## NAAL mailing list

- On the form, you can indicate up to 16 areas of interest
- Adult basic education
- Adult secondary education
- Citizenship/civics education
- College prep
- Computer literacy
- Correctional education
- Early childhood/children’s literacy
- ESL/ESOL (English as a second language)

-   Family literacy
-   Health literacy
-   Homeless education
-   Learning disabilities
-   Mathematical literacy
-   Native language literacy   (i.e., literacy in Spanish)
-   Policy and legislation
-   Workplace literacy

## National Center for Education Statistics

**Sheida White***NAAL Project Officer*(202) 502-7473*Sheida.White@ed.gov*
**Andrew Kolstad***Senior Technical Adviser*(202) 502-7374*Andrew.Kolstad@ed.gov*
**Website***http://nces.ed.gov/naal*

**National Center for Education Statistics**