**NAAL September, 2004**

Sheida White: Welcome to today\'s webcast and thank you for your
interest in NAAL, the National Assessment of Adult Literacy.

Today I\'ll present a few highlights about NAAL. It is a rich source of
information about adult literacy in America. But first, allow me to
introduce myself. I\'m Sheida White, the project officer for NAAL at the
National Center for Education Statistics, which is one of three centers
within the U.S. Department of Education\'s Institute of Education
Sciences.

And I am delighted to have my colleague Andrew Kolstad here. Andrew was
the project officer for the 1992 assessment and is currently the senior
technical advisor for NAAL and I\'m most honored to have with us today
Cheryl Keenan on this important International Adult Literacy Day. Cheryl
is the Director, Division of Adult Literacy, at the Office of Vocational
and Adult Education, also known as OVAE, which is also part of the U.S.
Department of Education and I\'ll let you say a few words.

Cheryl Keenan: Thank you, Sheida. It\'s a pleasure to be here today with
you, especially since today is International Literacy Day and I think
it\'s a very fitting day to be holding this webcast to talk about the
NAAL, the new adult literacy assessment, and to talk about the
importance of it to this country. As you may know, this is the third
webcast that is being held to brief folks on NAAL and it\'s very fitting
that it is being held today on International Literacy Day.

The importance of benchmarking literacy skills of Americans is\-- is
something that I think you all agree is critical to the success of this
country. I know that the importance of the data is certainly profound
when we look at the Division of Adult Education and Literacy and OVAE,
because so much of the policy that we make, national policy that we make
around literacy is based on the findings of that 1992 study.

OVAE was a partner in the 1992 study and is also a partner with the
National Center of Educational Sciences\-- or Educational Statistics in
the new NAAL study and so we are very happy to be here today and to talk
to you about the importance of literacy and the importance of this study
to our country on International Literacy Day.

According to the 1992 results of the NAAL, as you know, we learned that
there were tens of millions of adults who do not have the skills to be
self-sufficient or the tools that they need to adapt to the changing
demands of a global economy and we know that the upwards figures of 40
million adults who lack these skills have profound effects on our
workforce and our ability to compete in a global economy. We know from
some of those projections that in the next decade we could face a
shortage of 12 million workers based on this data alone and so having
this information is so important to us as a country in terms of how we
make policy and how we address the educational needs of the adults who
live in America.

The United States Department of Education, particularly the Office of
Vocational and Adult Education, is committed to improving access to
quality education services for these adults, including those who are new
to this country and do not possess command of the English language. The
Office of Vocational and Adult Education appropriates and obligates
money to the tune of something like a half a billion dollars every year
to provide programs in all of the states and outlying areas and
territories of the United States to improve the prospects of education
for our adults.

We also use a portion of our money to make critical investments at the
national level and the NAAL is one such investment that we make with the
money that is appropriated through the Adult Education and Family
Literacy Act. It is critical for\-- these funds are critical for many
things, including investments in research such as the NAAL and other
critical investments that relate to reading and how well our adults
function in the United States.

For example, many of our critical investments are in the area of
research so that once the results become available, of studies such as
these, that we can use our national investments to remedy the problems
that we find through these descriptive studies. Those investments\--
lately we have been able to invest a significant amount of money in
partnership with the National Institute of Health and the National
Institute for Literacy and are investing those funds in studying ways
that are effective in teaching adults who do not read well.

So our investments in adult reading are geared to eventually find their
way into practice and into the classrooms of teachers and one such
project that we have this year is referred to as Student Achievement in
Reading or Project Star. We are working with five states and have
translated some of the latest findings of the reading research into
teacher toolkits so that practice in the classroom\-- at the classroom
level can be based on research-driven practices and, as we invest more
in studies such as the NAAL, that we hope to see an increase in the
literacy trends in the United States rather than a stagnation of not
making any progress in improving the literacy skills.

So in these ways, our investments work together, not only to discover
what the baseline skills are of the adult population, to look at trends
and whether that improves, to look at ways that people learn how to
read, to apply research to practice and to give teachers new tools and
strategies to become effective in providing high-quality services to the
adults that we seek to improve their education.

You know, also, when we look at the numbers that the first NAAL provided
to us, we know that there are tens of millions of adults that can
benefit from adult literacy services and on this International Literacy
Day, I think it\'s very important that each and every one of us think
about not only the adults that we are serving in our programs and in our
adult education classes, but also think about those that we are not
serving and the ways that we have available to us to assist in expanding
access of services to more individuals who could use our assistance. And
we are doing that through many different kinds of partnerships,
partnerships with states to expand access through the use of technology
in distance learning, through partnerships with businesses to expand
services to people who are in the workforce and at the worksite and in
partnership with other federal, state and local agencies, including the
folks who operate the One Stops and the people in your communities that
are partnering with their human resources and social services to help
those who are dependent right now on welfare.

So when you think about all of these things on International Literacy
Day, the numbers of people that we have yet to serve, those that we are
serving and, perhaps, can serve better and the contributions that these
individuals can make to the United States, to our economy, to our
workforce and to our schools and our communities, I, indeed, am thankful
that you are here to learn more about a very interesting study that can
provide very powerful data to you at your state levels, at your local
levels, in advocating for those individuals that you serve.

Thank you, Sheida.

Sheida White: Thank you. Before we get started, I would like to say a
few words about people who have contributed to this assessment. We have
multiple contractors who have helped us develop the 2003 NAAL.

Westat helped design the sample and conducted the data collection and
scoring. American Institutes for Research, also known as AIR, they
developed the items. They developed the scoring guides. They developed
the background questions and currently, as I\'m speaking, they are
working very hard to analyze our data.

We also have the Education Statistics Support Institute\--

Andrew Kolstad: Services.

Sheida White: \--also known, as ESSI\-- I\'m sorry, Education Services
Support\--

Andrew Kolstad: No, Statistics Services Institute. Support isn\'t in
there. OK.

Sheida White: Thank you.

Andrew Kolstad: When we go by acronyms all the time, sometimes we forget
what they stand for.

Sheida White: In any case, they helped us enormously with the planning,
dissemination and a whole lot of other activities.

We also have multiple advisory groups and panels and consultants who
provide a very, very, broad range of input.

We are going to have a Q&A session at the end of this presentation and
at the end of the Q&A session, I would like to ask you to go online and
tell us three things that you liked about this presentation and one
thing that you\'d like us to do differently. You can e-mail your
comments to me at sheida \-- s-h-e-i-d-a \-- .white \-- w-h-i-t-e \--
\@ed.gov, e-d.gov or you can send it to NIFL, who I should have really
cited for organizing this and for inviting to make this presentation. In
any case, you can also send your comments to Jaleh Behroozi at NIFL and
her e-mail is j-b-e-h-r-o-o-z-i \@nifl \-- n-i-f-l \-- .gov. I got that
one right.

In any case, your input is\-- we take your input very, very seriously
because we want to do more of these sessions, hopefully, in the future.

We have\-- we have made the slides available for you a few days ago so
you could have a chance to go over them and formulate your questions or
comments. I hope you had a chance to do that. They will remain on the
website and I\'ll give you the website address at the end of this
session, just in case.

I\'m going to go over the agenda, the topics that we\'re going to cover
today. We have\-- OK, I can\'t see the screen, so I guess I\'m going to
take your word for it.

The first topic is background, key features and major goals. Then we\'re
going to talk about performing literacy tasks. We\'re going to talk
about the skills that are required to perform those tasks. This is where
some of our new instruments and features come into play. I\'m going to
say a few words about the assessment design and administration and data
analysis and reporting.

So let\'s move to our first topic, which is really NAAL background and
I\'m going to ask my colleague Andrew, since he was the project officer
in 1992, I think it\'s appropriate for you to say things about the
background.

Andrew Kolstad: Well, this is the third in a series of surveys where we
can provide comparability over time. In 1985 we at NCES sponsored the
first survey, which was administered under a grant to Educational
Testing Service. At that time, it was part of the general grant that
went to the National Assessment of Educational Progress at ETS. In
this\-- in the 1985 assessment, the relationship between the National
Assessment of Educational Progress and the young adult literacy
assessment was so close that one of the 11th grade reading blocks was
used in the young adult literacy assessment at that time.

In 1992 we had a slightly different arrangement. It was a contract done
independently of the National Assessment of Educational Progress,
directly to ETS with Westat as a subcontractor to ETS. It used the same
framework or recovered the same content as the 1985 assessment and we
carried over most, if not all, of the test questions that were used in
1985. There were seven blocks of test questions used in 1985 and in 1992
we used those, plus some new ones.

In this\-- in 1992 we had a national sample of 13,600 adults who were
ages 16 to 99. In 1985 the age range was much narrower. It had been only
21\-- adults ages 21 to 25. So that part of the sample, the 21 to 25
year olds in 1992 were comparable and we did publish some measures of
change over that period for our\-- for that group.

We also supplemented the sample with 11 states\' samples. In fact,
Cheryl was helpful in that because she was in Pennsylvania at the time
when Pennsylvania was one of the participating states in that survey. We
also had a prison population sample in 1992 where we measured\-- we took
our survey into the prisons and added them into our estimates for the
population of the United States, which in 1992 was 191 million Americans
16 years old and older.

Now this survey, the National Assessment of Adult Literacy, is the first
assessment since\-- that would be comparable and could show change since
the 1992 survey. Our sample now is a little bit smaller than it was
then, but it still includes a state sample and a prison sample to
supplement them and it now represents the 225 million adults 16 years
old and older. So between 1992 and 2003 our population grew by more than
30 million people.

We do intend to continue to do studies like this in the future, although
planning that far down the road is still getting started. It\'s a long
time between surveys because it\'s a very expensive kind of survey to
conduct and we don\'t think the population changes very rapidly.

Sheida White:: Now the key features of NAAL on bullet one, NAAL is the
most comprehensive national measure of the literacy of U.S. adults. It
is really the largest NCES assessment that includes in-person interviews
of adults in the household.

Bullet two \-- in addition to national results, as Andy said, the NAAL
provides results at the state level for participating states and for the
prison population.

Bullet three \-- NAAL emphasize the performance of literacy tasks that
are similar to those people encounter in everyday life. I\'ll provide
more details later on about the NAAL functional, as opposed to
school-based, view of literacy and I\'ll do that in the next section.

Bullet four \-- the 2003 NAAL provides a lot of data about how
performance relates to background characteristics. We\'re going to be
looking at key population subgroups such as adults at the lowest level
of literacy. You\'ll see that those are very important to pick up people
who are the least literate. We\'re going to look at non-native speakers
of English, also of concern to you, and we\'re going to be looking at
the relationship between proficiency and older adults. We\'re going to
talk about key issues such as health literacy, family literacy,
job-related training and technology.

The next slide on major goals, bullet one we are going to\-- NAAL is
really designed to take a snapshot of the status of literacy at the
national level, at the prison level and at the state level for the six
participating states.

Bullet two has to do with trends. By comparing results from 1992 to 2003
we\'ll be able to provide, as you said, the first indicator in a decade
of the nation\'s progress. We have actually six blocks of items, which
translates to 65 items, that are exactly the same between 1992 and 2003
and that allows us to create this trend and link the two assessments to
one another.

As I said, bullets three and four, the 2003 assessment provides expanded
information on the relationship between performance and tasks\-- and
literacy tasks and adult characteristics and skills.

Bullet five, partnership \-- one of the important objectives of the 2003
NAAL is to develop partnerships with other government agencies like OVAE
and HHS, Health and Human Services. For example, we worked with HHS to
provide a new measure of health literacy and I\'ll say more about that
later on.

Also bullet five, we provide technical support to those of you who are
researchers. We have a history of providing training for data analysts
who work with NCES data, but we will expand that training to include the
NAAL data, as well. We\'ll provide data analysts with professional
development training. We\'ll help them analyze our data and also
interpret our data. We will produce what we call a NAAL public use data
file. We\'re going to have an accompanying code book and some
documentation of how they can use the code book and the user file\--
data file.

All right. Now let\'s talk about the, really, the definitions of adult
literacy that we used in our assessment. This definition is really
grounded in what we call task-based theories of literacy as opposed to
skill-based theories of literacy.

The definition that you see on the screen, it emphasizes the use of
printed everyday material to function adequately in one\'s environment.
It reflects adults\' ability to achieve both internal goals \-- these
are goals that they\'ve set for themselves \-- and to respond to
external demands, those that are\-- demands that are made by their jobs
and demands that are made by the society.

So in the definition I\'d like to draw attention to a few phrases here.
The phrases, "to achieve one\'s goals," and "to develop one\'s knowledge
and potential." These phrases emphasize goals that come from individual
people. These goals might include, for example, running for political
office or it could be just practical goals like paying the bills.

At the same time, there is another phrase in there and that is, "to
function in society." And that phrase suggests that there is a minimum
level of literacy that all adults must achieve, regardless of their own
personal goals. As you can see by the graphic, this literacy definition
implies the ability to meet literacy demands at home, in the work place
and in the community by performing prose, document or quantitative
tasks.

The next slide is the\-- provides the definition of the three key
literacy areas. I\'m not going to repeat those definitions, but I would
like to say a few things here. Prose and document literacy makes use of
stimulus materials that have differences in terms of their structure and
in terms of their structural elements. Prose stimulus materials are sort
of characterized by being continuous text, whereas document stimuli are
characterized by non-continuous text. Documents also contain multi-media
elements like side bars and boxes and stuff like that.

Now these different structural elements also require different skills on
the part of the respondent. For example, and we have data to support
this based on some coding that I\'ve been doing in the past few months,
inferential processes, for example, are much more prevalent to prose
literacy than to document literacy. On the other hand, searching the
text for information, that particular skill is more relevant to document
than the prose.

Now quantitative literacy requires all of the skills that are necessary
for prose and document literacy plus more and that is plus computational
skills. So you need to be able to read and make inferences, all of that,
but also be able to do calculations that are embedded within either a
document or prose. And it turns out to be that the quantitative tasks
are, generally, the most difficult tasks that we have.

Measurement of prose, document and quantitative literacy, the first
bullet \-- I said earlier comparability of the scale scores would make
sure that we can accurately measure trends and we have\-- and that link
between the 2003 and the \'92 scales is provided by the 65 items that
were in common between the two assessments.

Now let\'s look at bullet two, sub-bullet one in bullet two, I would be
showing you an example of each type of prose, document and quantitative
experience in just a moment. Bullet two, sub-bullet two, all of our
taxes, 100 percent of our taxes are\-- have\-- I\'m sorry.

Did I say taxes?

Andrew Kolstad: Yes.

Sheida White: I\'m sorry. They\'re open-ended. In other words, we have
no multiple-choice items in NAAL.

Bullet two, sub-bullet three, participants read in order to\-- in NAAL,
people read in order to do something with the information. In other
words, in this sense, we may call this\-- this is the only way\-- the
only sense of the word functional that I\'m willing to use on this
assessment\--

Andrew Kolstad: OK.

Sheida White: \--to differentiate NAAL from school-based assessment and
school-based tasks.

Andrew Kolstad: Right. In school-based tasks, they often read the
passage and then find out what they\'re to be asked about the passage.

Sheida White: Exactly. And in terms of quantitative items, they\'re
often\-- mathematics in school they\'re often sort of decontextualized
and they have to do tasks that are not\-- don\'t have immediate purpose
and goals that are real. And so it\'s\-- I think it\'s appropriate to
use the term functional in here\--

Andrew Kolstad: OK.

Sheida White: \--but people use the term functional in many, many
different ways that we always agree with. But I think this is one way to
use the term functional that applies to NAAL tasks but not to
school-based assessments.

Bullet three \-- take a look at sub-bullet one. While most of our items
require short written responses, many of our tasks also require just
underlining or circling the part of the text\-- part of the text that is
relevant to successful performance.

The last bullet under bullet three, for some tasks the task demands
include such actions as making text-based inferences or performing
computations on numbers that are embedded in the text.

Here are some topics that our stimulus materials cover. As you see, the
NAAL stimulus materials were evaluated to make sure that they were
distributed across the content areas that were used in 1992 and these
were the content areas that were used, also, in 1992. Of course, not
every adult will be familiar with every one of these topics, but we
expect that most people who take NAAL use at least some of these topics
in their everyday lives.

OK. As promised, here is a sample prose task. This is a released item
from the 1992 assessment.

Andrew Kolstad: What\'s the task here?

Sheida White: OK. The\-- the question is to copy three food sources
named in the almanac that contain vitamin E. Remember, this little
excerpt was taken from an almanac so they have to\-- before they respond
they have to search the almanac to come to the right\--

Andrew Kolstad: An almanac is a book about that thick, right?

Sheida White: Yeah.

Andrew Kolstad: So they have to find the place in the book?

Sheida White: That\'s right. So that\'s another element of difficulty.

So here we have a paragraph about vitamin E. This is a very typical and
relatively easy task.

Andrew Kolstad: It\'s easy once you\'ve found this passage.

Sheida White: That\'s right. That\'s right. We assume that you are going
to get to that passage.

It asks respondents to search a very short text here to locate some
easily identifiable information. Now in some of the more\-- in some of
our more difficult prose items people may need to make inferences, may
need to compare and contrast information, sometimes to have\-- to
synthesize some piece of information from the long and complex passages,
but in this case it\'s very simple. All they have to do is identify
these three words.

Here is a sample of a document task. Now first of all, this is a
document. It\'s non-continuous text, although I must say that reading
the question itself does require practicing connected text, but the
document itself is\-- does not require knowledge of discourse.

This is a moderately difficult document item. This is what we call a
nested document.

Andrew Kolstad: So this document is\-- would this be like a poster or a
pamphlet that they would see in a hardware store?

Sheida White: This\-- I think\--

Andrew Kolstad: Where would you find an abrasive selection guide?

Sheida White: Yes, exactly what you said in a hardware store.

Andrew Kolstad: Right. OK and so once\-- they wouldn\'t look at this
unless they had a question in mind and the question is\-- so what is the
question?

Sheida White: OK. The question is \-- I was going to say that.

Andrew Kolstad: OK.

Sheida White: All right.

Andrew Kolstad: Good.

Sheida White: Now to determine the right answer, this is\-- the answer
that the question is asking, the question is asking, you need to smooth
wood in preparation for sealing and plan to buy garnet sand paper. What
type of sand paper should you buy?

All right. Now to determine the right answer, our participant must first
identify the correct column here and that is garnet. Correct?

Andrew Kolstad: Now that\'s a set of four columns.

Sheida White: So among those four columns you have to identify the right
column. Then they need to identify the correct row and that is,
preparation for sealing. It\'s a sub\-- it\'s a sub-category under wood.

Then they have to connect the abbreviation "F" with the word "fine" in
the footnote. Right? And then select that individual cell among all
other\-- among all other types of sand paper as the right one that
they\'d find.

Now there is a lot of work in here, a lot of cognitive processes that
have to take place. Some of the simplest document tasks that we have
require only actions like signing a form in the right place or
appropriately filling in blanks. More difficult tasks have requirements
such as comparing, contrasting, drawing high-level inferences from
multiple pieces of information and more complex documents.

Now when you go to our website and look at these released items,
hopefully in a month or so, you will also be able to see the difficulty
level of this item in terms of the percent correct, percent of
individuals who responded correctly to this item.

And finally we have an example of a quantitative item. As you know,
quantitative tasks require the person to identify\-- to identify or to
perform\-- notice that there are two different quantitative skills,
identify or perform, computations on numbers that are embedded in either
prose or documents.

In this case, the numbers are embedded in a document. Now in order to
perform this \-- I hope that you are with me. In order to perform this
task, the respondent must subtract the mileage on March 2nd, which was
42,775, from the mileage on March 9th, which was 43,083. Then person has
to divide the result, 808 miles, they have to divide that result by the
number of gallons used on March 9th, which is 12.5. In other words, the
car got 25 miles per gallon since the person filled it up with gas on
March 2nd. Now if any of these computation processes goes wrong, the
correct answer will not be obtained.

Now a simpler quantitative task might involve solving a single equation
using only numbers that actually appear on the document. So, for
example, one of the key quantitative items we have from the 1992
assessment asks participants simply to figure out the amount of a bank
deposit by adding the two amounts that would be shown on the deposit
slip.

All right. OK. This is very exciting. We have a new health literacy
component. This is entirely new. This component is the first-ever
national assessment designed specifically to measure American adults\'
ability to use literacy skills to read, understand and apply
health-related information, first ever. The NAAL health literacy data
can help guide development of health-related materials so that these
materials are more appropriate for the literacy levels of the target
audiences, whatever they are.

OK. Let\'s look at bullets one and three. The item on vitamin E that I
showed you earlier, that item would be considered a health literacy\--
health-related item. In this case, we would classify that as a prose\--
as a health prose item.

Bullet four \-- look at sub-bullet two. We are going to provide you with
a new health literacy score that will be based only on health literacy
items. There are 28 of those.

It is not possible to establish a health literacy trend between 1992 and
2003. That we cannot do and the reason for it is\-- You want to say it?

Andrew Kolstad: No, go ahead.

Sheida White: The reason for it is because the 1992 assessment did not
contain enough health-related items. We only had six health-related
items\--

Andrew Kolstad: Oh, that\'s right.

Sheida White: \--out of 166 items in 1992.

Andrew Kolstad: And how many\--

Sheida White: And only two of those health-related items from the 1992
were reused in the 2003 assessment.

Andrew Kolstad: So the linkage would have to be done with those two
items.

Sheida White: And that\'s\--

Andrew Kolstad: It\'s just not enough.

Sheida White: It\'s not possible to do that.

Andrew Kolstad: Yeah, right.

Sheida White: We look forward to comparing the 2003 health literacy data
with\-- hopefully with health literacy data that will be provided,
hopefully, in future administrations of NAAL.

OK. Next I\'ll say a few words about the development of the health
literacy component. Again, as I said earlier, one of our objectives is
to develop partnerships with other government agencies. We worked very
hard with the U.S. Department of Health and Human Services to provide
them with a measure\-- with a baseline of health literacy for what they
call the Healthy People 2010 objective and this objective itself\-- that
included another health literacy objective, to improve the health
literacy of persons with inadequate or marginally adequate skills.

Bullet two \-- the Healthy People 2010 defines health literacy as the
degree to which individuals have the capacity to obtain, process and
understand health-related information and services needed to make
appropriate health decisions.

Now this important. Although the NAAL health literacy component\-- let
me put it this way. Although the health literacy involves some of the
factors that NAAL cannot measure \-- for example, we cannot measure the
ability to communicate orally in a health setting.

Andrew Kolstad: Right. Right.

Sheida White: We cannot do that. However, we are able to provide a
portrait of one important aspect of health literacy and that\'s written
communication\-- in a certain area of written communication.

Andrew Kolstad: That\'s right. And doesn\'t the HHS definition of
literacy include an oral component? Isn\'t that part of their Health
People 2010 definition? I know there\'s some part\--

Sheida White: Yes. It is much broader\--

Andrew Kolstad: It\'s broader than we\'re able to measure in our survey.

Sheida White: Exactly. Exactly. But that\'s an important area that we
are measuring\--

Andrew Kolstad: Oh, yeah.

Sheida White: \--written communication, being able to communicate about
health issues in writing.

Bullet three \-- the clinical task\-- let me give you an example of a
clinical task and that would be filling out a patient information form
for an office visit or taking a medication prescription as it was
prescribed and intended.

Bullet three on preventive \-- an example of a preventive health task
would be following guidelines for age-appropriate preventive health
measures or understanding warnings on labels about health risks
associated with consumer products.

The navigation, the third bullet, has to do with navigating the
healthcare system. Examples include understanding eligibility for and
benefits of health insurance plans.

So clinical has to do with disease, preventive has to do with
non-disease and navigation is to get to the health\-- more the
bureaucratic part of it.

Andrew Kolstad: Does it also have to\-- the navigation also have to do
with paying for it afterwards?

Sheida White: Yes.

Andrew Kolstad: Sort of the insurance part?

Sheida White: Right. Right.

All right. We are going to begin a whole new section in here and that
has to do with skills that are required to perform literacy tasks. These
are very new features of NAAL. We did not have these features in the
1992 assessment.

The first bullet, ALSA, which stands for Adult Literacy Supplemental
Assessment, is very important. It\'s important because it provides data
on the skill levels and deficits of the least-literate adults. Previous
assessments did not give us much data on the least-literate adults
because these adults were unable to perform the assessment tasks. Is
that correct? It also uses easier tasks to measure these adult literacy
skills.

Bullet two \-- FAN uses oral reading tasks to obtain data on the basic
reading skills of all adults. Because all adults take FAN, it allows us
to compare the basic reading skills of adults across the entire spectrum
of task performance.

Bullet three \-- in addition to basic \-- and by basic, I mean
word-level reading skills. In addition to basic, NAAL identifies six
types of higher-level literacy skills. We have recently developed and
are still refining a multi-dimensional analysis method which will\-- we
will be applying to NAAL performance data in order to characterize the
proficiency of adults in terms of these six skills.

Now let\'s look at each of these new features individually.

Andrew Kolstad: The first\--

Sheida White: You want to talk about it?

Andrew Kolstad: Yeah. Yeah.

Sheida White: OK.

Andrew Kolstad: The first of these new features is the Adult Literacy
Supplemental Assessment, which was developed, as Sheida said, because
the 1992 survey provided not enough information about the literacy
abilities of the very low level adults, the ones who were not able to
complete the assessment.

Now here\'s the problem. As\-- people who develop tests are interested
in\-- are very capable of measuring how well someone performs when they
take the test. The problem for a literacy assessment is you have to
be\-- you have to be able to read well enough to take the test and if
you\'re not able to have that level of literacy, then you\'ll sort of
fall below the floor, how low a level of performance the test can
measure. And in that case, we\'re very interested in those people and I
think in the 1992 survey we didn\'t really set the floor of the test
down low enough to have \[inaudible\] people to be able to answer.

So the purpose of the supplemental assessment is to get a clearer
picture of the least literate adults, the ones who fail to complete the
assessment and we were particularly interested in those who fail to
complete the assessment for literacy-related reasons, namely having a
general difficulty with reading or writing or having difficult with
reading and writing in English or a mental or learning disability that
interfered with their ability to read. And we generally found out about
these things because when the survey participant stopped taking the
survey or didn\'t take it and they might tell the interviewer and, in
fact, the interviewer would ask them what the reason was. And that\'s
how we would find out this kind of information.

So if they\-- they may have broken off the interview without completing
very many questions and that\-- without having enough test questions
\[inaudible\] So that\'s really the motivation behind developing this
assessment.

Sheida White: And one of the assumptions that you\'re making is that
even those adults in 1992 who were not able to respond, they would have
been able to show us something about their skills\--

Andrew Kolstad: Right, if we had a test question\--

Sheida White: If we had easier\--

Andrew Kolstad: \--for it.

Sheida White: If we had easier items for them.

Andrew Kolstad: Right.

Sheida White: And that\'s why we have these easy items so we can\--

Andrew Kolstad: Yeah, on the other hand\--

Sheida White: \--collect useful information about these people.

Andrew Kolstad: These very easy items, however, are so easy that we
didn\'t want to give them to the general population because they would
have been\--

Sheida White: Inappropriate.

Andrew Kolstad: \--inappropriate for them. So we\'ve divided the test
into two parts and that\'s what this next slide is about is choosing
which interview form the person takes. So if their skills are very low,
then they would take the supplemental assessment and if they were
average or even well below average, they could take the main assessment.

In order to decide which path they were to take, the participants
started out with seven very easy screening questions that we called our
core. The interviewers in the field would score the answers using the
scoring guide that was provided for them on the computer, so they\-- and
they had been trained in scoring these things to make sure that they
knew exactly what they were looking for in the answers and the
questions.

The criterion was, I think, two or three answers\-- they had to get at
least two or three answers on this question\-- on this set of core
questions right in order to take the main assessment and if they
couldn\'t get that many right, then they would be routed off into the
alternative assessment.

Sheida White: It\'s also important to remember that we\-- we have
quality control. Even though these were scored on the spot by the
computer by interviewers, we do quality control afterwards and we have
all of the\-- all of these seven core items are re-scored by Pearson,
who is subcontractor to Westat\--

Andrew Kolstad: That\'s the scoring contractor?

Sheida White: Yes, the scoring\--

Andrew Kolstad: Subcontractor.

Sheida White: And they re-score\-- re-score all of them and if there is
a discrepancy we go with the Pearson scores.

Andrew Kolstad: Yeah. Yeah, the thing is, with seven questions you\'re
bound to make a few mistakes in choosing which path to take and it\'s
kind of inevitable. We tried\-- we reduced it to a very minimal level
through quality control. Throughout the training we did in advance and
the testing of the questions we tried to reduce that misclassification
error to as small a level as we could and I think we did.

And we\-- we\'re going to do\-- since we didn\'t have this procedure,
this alternative assessment, in 1992, we do have to do some work to try
and make sure that we\-- when we do try to compare the two years that we
use the same procedures both times and for that reason\-- that\'s one of
many reasons that we will be re-analyzing the old data according to the
new methods.

Sheida White: Exactly. OK. Here are some important features of ALSA.

Let me draw your attention to bullet three. For example, some ALSA tasks
ask participants to read or to point to a word or to a letter. ALSA
starts with the easiest tasks and moves to more difficult ones.

Now here an example of a relatively difficult ALSA item would be one
that asks them, for example, to\-- to tell us\-- a question might be
something like this. What does the label say people should do if they
took too much of this, you know, drug or whatever? So they have to be
able to understand the label, read and understand the label, which is a
short text, nonetheless it\'s a continuous text.

The need for additional data about skills deficits \-- as I said before,
ALSA will provide valuable information about adults and this information
will be analogous to those who failed to complete the 1992 assessment.
For example, we would know whether poor performance is due to just
simply basic reading skills or is it due to higher-level inferential
skills. At the moment, we don\'t know why people performed poorly, but
we would be able with this instrument to identify the deficits, the
areas of deficit.

Andrew Kolstad: Sheida, in this first bullet on that page, I\'m not sure
that I agree with that division of the 21 percent in level one between 6
percent and 15. We\-- in level one, some of the people in level one
could perform those literacy tasks with an 80 percent probability and
some of them couldn\'t, but I\'m not sure where the dividing line is
between those who couldn\'t \[inaudible\] and that. So I\'m not sure
that\'s quite the right division.

Sheida White: Well, the 6 percent\-- the 6 percent is, of course, is our
weighted sample. But we couldn\'t get more information from your
technical\--

Andrew Kolstad: Yes. We\'ll be working on that. I think we may find out
the answer to that when we publish our report. There\'s a lot of
analysis left yet to be done.

Sheida White: That\'s right. We\'re going to make sure that our
technical report this time is much more specific.

All right. The\-- here on this next slide, I\'d really like to
underscore the importance of the implications of poor basic reading
skills. Adults who cannot\-- I\'m just going to read this to you because
we have spent a whole lot of time developing these slides.

Adults who cannot decode or recognize words in a list or paragraph \--
these are basic reading skills. \[inaudible\] Adults who cannot do that,
they obviously cannot access the meaning. OK?

Adults who can decode and recognize words but do so slowly and with
effort, they may struggle to understand the text.

Adults who can decode and recognize words fluently but do not have
higher-level literacy skills, they may struggle with drawing appropriate
inferences and may draw upon irrelevant background knowledge and myths.

Andrew Kolstad: \[inaudible\]

Sheida White: That\'s what we\'re going to find out at the end of our
analysis, we\'re going to find out. But this is what we want to be able
to communicate to you, whether it\'s absence of word meaning, are they
struggling to comprehend the text and are they having difficulty with
the higher level literacy.

New data on the skills \-- bullet three, we think that the FAN data
should provide insight the role that basic reading skills play in the
performance of all the literacy tasks.

I\'m going to move on a little bit faster, because I don\'t think that
\[inaudible\]

Next slide, oral\-- on FAN oral reading materials and associated
measures, bullet one, here\-- in a nutshell here we\'re trying to get a
measure of the ability to read unfamiliar words. In bullet two, we\'re
trying to get a measure of the ability to read familiar words. And, of
course, then we have the text passages.

Initial measures and scoring challenges for FAN, bullet one \-- these
measures are easily understood by potential users of the data and we
believe will provide a concise summary description of each participant.
Bullet two \-- in preparation for scoring the FAN task, we did extensive
work \-- I went to the site and I observed the work. It was extensive
work that our subcontractors have done to ensure that correctness\--
correctness is measured reliably and that speakers of non-standard
varieties of English would not be unfairly penalized.

In a nutshell, the scoring rules consider non-standard pronunciations
acceptable as long as they are consistent with participants\' general
speak pattern. So, for example, a regional pronunciation of the word
"torque," who would you pronounce that?

Andrew Kolstad: Torque?

Sheida White: Yeah, for torque\-- that\'s a regional\--

Andrew Kolstad: Oh? Where?

Sheida White: I believe it\'s in the Northeast. Torque for torque. That
would be acceptable if every other post-vocalic r would be unpronounced,
as well.

Now here are some key similarities and differences between FAN and
between ALSA. Let\'s go to bullet two, sub-bullet one. Low-literacy
adults rely heavily on context and familiarity to comprehend the text,
as many of you know. These are often called compensatory strategies.
ALSA materials facilitate the use of these compensatory strategies. On
the other hand, FAN materials do not allow participants to make use of
compensatory strategies to partially offset the deficits that people
have in reading skills.

Andrew Kolstad: \[inaudible\]

Sheida White: Oh, now, well, that\'s one way, but also, as I said
earlier, the ALSA materials are very highly contextualized. They\'re
very familiar; they\'re tangible. You see? So people can get the meaning
of the word, conceivably, from looking at the pictures and diagrams and
all these other contextual factors, although we make it difficult for
them to do that because we want them to be able to tell us whether or
not they can read the words, the letters on the page.

Nonetheless, they are very contextualized, but FAN doesn\'t allow us
to\-- doesn\'t allow respondents to do that.

Here is a slide on the functionality of world-level skills and
higher-level literacy. This is one of your favorite topics is the term
functionality, so let\'s get into that.

Bullet two \-- now, bullet two, OK. We say that all literacy is
functional, therefore NAAL does not differentiate between literacy and
functional literacy. This may comes as news to some of you viewers\--
our viewers.

Some people\-- we say that because some people may think of functional
literacy as the general ability to solve problems. I\'ve heard people
say, like figuring out signs without necessarily having the skills to
read the words is a functional literacy.

But even in ALSA, where we do have, you know, contextual information
that are non-linguistic, we intended for the less-literate adult, these
people, to not be able to perform the task simply by looking at the
non-linguistic context. They have to deal with the printed and written
material.

Now we said earlier\-- we talked about possibly we can say functionality
applies in terms of genuine purpose, to accomplish a task that is\--
that one is encountering in the real world as opposed to school-based
tasks. We said that maybe that\'s OK to use the term functional, but I
hope that people don\'t think we are making\-- we have a contradiction
here when we say\-- on the one hand, we say all literacies are
inherently functional, but on the other hand we say that in NAAL people
use these functional skills to apply them solely to real-world tasks
whereas these functional or literacy tasks which we call\-- which we
think all of them are functional, may be used\-- may not be used in a
school-based\--

I\'m sorry. Let me start again. I\'m getting myself in here confused.

OK. I\'m hoping that you don\'t make\-- you don\'t think we are having a
contradiction here because earlier\-- on the one hand, here, we are
saying that all literacy skills are inherently functional. OK? Yeah,
anything\-- anything, reading, vocabulary, fluency \-- they\'re all
functional. We\'re calling them all functional skills. OK?

But in NAAL, as opposed to school-based tasks, these skills are applied
to real-world tasks. OK? The functional skills in school-based
assessment situations are applied to not-real-world texts, but more
decontextualized, let\'s say, mathematical exercises.

All right. The multi-dimensional analysis that I talked about earlier
will be compared with the FAN scores. We will not compare the
multi-dimensional analysis, the skill-based assessment with the
task-based assessment, in other words, with the results from the main
NAAL scores because the main NAAL and the multi-dimensional analysis
results are both\-- are both based on the same set of item response
data. So they will be correlated, but the substantive meaning of the
correlation will not be interpreted.

All right. A few words about the assessment design and administration.

How are we doing in terms of time? Is that OK to move on?

All right. We have\-- OK, in NAAL we cannot accept adults to volunteer
their participation.

Andrew Kolstad: Right. This is a scientific\--

Sheida White: OK. Well, some people call up and they say, "Could we
participate in NAAL." We cannot do that.

This is because if we did that, it\'s possible that higher-performing
adults would, let\'s say, volunteer and this would result in a sample
that is not representative of the entire population.

Andrew Kolstad: OK. So here\'s a question, Sheida. If we have 225
million people who are eligible to be in our survey, how did we choose
the 13,000 we got.

Sheida White: OK. A good question, Andy. So let\'s answer that.

What we did is we used a multi-stage sampling method to select these
households.

Andrew Kolstad: What\'s the first stage?

Sheida White: I am\-- OK. You\'re going to move on. Good. The first
stage is the selection of the geographical unit, which we call\-- this
is on the upper left hand of your screen. It\'s called\--

Andrew Kolstad: Primary sampling units?

Sheida White: \--primary sampling units or PSUs and PSUs consist of
counties or adjacent counties. Now the main NAAL sample included 100
PSUs. We selected these from almost 2000 PSUs, using the Census data.
And we have some PSUs that are very, very small, in fact, some that are
only 16,000 and we have some that are over 9 million, which is very,
very large.

And the second phase is the selection of segments, of area segments
within those PSUs. That\'s on the upper right hand corner of your
screen. Then we select households and within households we select
age-eligible individuals.

Andrew Kolstad: Might I say something about this?

Sheida White: Please.

Andrew Kolstad: Yeah, this is not strictly a random sample. These are
done in clusters. Even though people who live in the same neighborhood
are more similar to one another than they would be if you randomly
selected people all over the country, the difference is\-- it\'s really
driven by cost. You\-- we send an interviewer to these segments, to the
block neighborhoods, and then they walk around and find a few households
in this neighborhood. It\'s a lot cheaper than sending them to one
place\-- one household here and one house across the city\--

Sheida White: Good.

Andrew Kolstad: \--and one house in another city.

Sheida White: Right.

Andrew Kolstad: That\'s not possible.

Sheida White: Yeah, this assessment is very, very expensive because it
involves interviewers going into adults\' homes.

Here are some key features of the NAAL sample. These are adults aged 16
and over. I must say that we do not go to group quarters, for example.
However we do go to college dormitories if at the time of data
collection college students are living at home.

Andrew Kolstad: Well, no\--9

Sheida White: Rather\-- I\'m sorry, are living in the dorm as opposed to
home.

Andrew Kolstad: But our period of interviewing is seven months.

Sheida White: Yes.

Andrew Kolstad: So what happens when the college students are at home?
We try to interview them at home then.

Sheida White: But if we can\'t and if they\'re in school, we would go to
dormitories and interview them in the dormitories if it is within that
window of opportunity.

Andrew Kolstad: There\-- there are very few people in group quarters
like this. This might be a caretaker living on the property and\--

Sheida White: Right. Right. Bullet two \-- we combine, as Andrew said,
we combine the state and prison samples with the national sample, but we
do flag them to allow separate estimates for the prison population and
for individual states.

Bullet two, response rate \-- we are\-- I have to say, unfortunately,
that we started out with a much larger sample in mind, about 21,000, but
we had difficulty achieving the response rate that we were aiming for
and this is not\-- this is not unique to NAAL. This is, in recent years,
surveys in general have been experiencing this difficulty, so the
screening response rate was 82 percent and the background questionnaire
response rate was 78 percent. It\'s a little bit lower than what we
would like to see it, but we\'re going to do analysis to\--

Well, you go ahead.

Andrew Kolstad: Well, it was lower than we experienced in 1992, but
consistent with what all other surveys find.

Sheida White: Exactly. Exactly.

Andrew Kolstad: The public seems to be less trusting of survey
organizations and less willing to let them in.

Sheida White: That\'s right. Today that is true, right. OK.

Andrew Kolstad: And the third bullet?

Sheida White: The third bullet \-- we did offer a \$30 incentive payment
to boost the rate of response and then we offered more to improve it
when we realized that the response rate was going to be\--

Andrew Kolstad: We didn\'t pay that \$30 to prison inmates because
generally that would violate the rules of the prison.

Sheida White: Right.

Andrew Kolstad: So we found that prisoners were probably more
cooperative in participating than people living in households.

Sheida White: That\'s right.

Andrew Kolstad: Perhaps because they weren\'t as busy.

Sheida White: That\'s right. We do over-sample Blacks and we did
over-sample Hispanics and that\'s because it\'s unlikely that we will
have a large enough sample of Blacks and Hispanics to be able to conduct
meaningful analysis. We over-sample these groups and then we would
adjust for them statistically.

Andy, you want to talk? You want to\--

Andrew Kolstad: I want to say a little bit about the design of the
survey. Our\-- we had\-- we developed 153 test questions. To take all of
the test questions would have taken three hours and we were just talking
about how much difficulty we were having recruiting people for an hour
and a half survey. If we wanted to give people three hours of testing, I
think our response rate would have been a lot lower.

So we designed our surveys so that the testing part of it takes only 45
minutes rather than three hours and in order to carry this off we give
only a portion of the non-core items to each adult. Each\-- we divide
the 153 test questions into 13 blocks of items and then each person gets
three of those. So everybody gets about 40-45 questions per person.
That\'s about what it is.

Sheida White: Yeah.

Andrew Kolstad: And we pair these blocks in all possible combinations so
that every item\-- every test question is paired with every other test
question. That\'s important for us to do the statistical analysis
afterwards. And in order to get all possible combinations of booklets,
we wound up having 26 different test forms.

So that\'s one of the complications with the science part of this. Well,
it gets technical, but the idea is we\'re trying to see how well the
population as a whole does on the test as a whole and in doing that,
different people get different parts of the test.

Sheida White: OK. We can go on now.

Andrew Kolstad: Oh, yes. The next slide has kind of a picture of how the
blocks are divided up. In 19\-- in both surveys there were 13 blocks,
but in 1992 six of the blocks came from the earlier study, from 1985,
and seven were developed newly in 1992. Now in 2003 we again had those
13 blocks, but six of them were from the previous survey and seven were
newly developed. Now we would have taken all seven from the previous
survey, but we were limited by the budget and sample size because each
test form has a certain minimum number of cases that we have to have in
order to be able to use the information properly.

So we do\-- when we reuse the test questions one of the things we do as
part of the scaling of the test is make sure that the item is
functioning in the same way in both time periods and so we try to fit
item parameters to the same item parameter in both years and in one case
there was an item that didn\'t fit in both years, so we treated it as if
it were a separate item in each year and not part of the trend.

So those are some of the technical issues involved with comparing\--
developing the test, administering it and doing some of the analysis.

This next slide is also on the things we have to do to compare the \'92
survey with the newer one while introducing new features. The seven
blocks that were newly created were similar. We tried to make them as
similar as possible to the 1992 test questions so that we could be sure
that we were measuring the same thing.

They have\-- we were able to discover through our analysis that they
have approximately the same average task difficulty as the ones had in
1992 and we mixed them together. There would be some test booklets that
had all \'92 items, some test booklets have all 2003 items, but most of
the test forms are a mixture of the \'92 and the 2003 items. And we then
used item response theory in the scaling to link them together and
measure, essentially, the same scale in both time periods.

On the next slide we see\-- we\'re talking about performance levels in
19\-- in 2003. In the previous assessment in 1992, we used reporting
levels, level one, level two, level three, level four, level five. They
were developed by grouping together tasks of similar difficulty.

But these 1992 levels proved to have some limitations in reporting. The
cut points between the levels were not fixed but depended on what
measure of task difficulty we used. In 1992 we used a response
probability of 80 percent to decide how difficult the task was and where
the cut points fell on the scale and if we were to change that response
probability convention to be more consistent with either the National
Assessment of Educational Progress or many state testing programs, the
cut points would be in a different place.

So that was one problem. Another problem is that there were no
distinctions made within the lowest level, level one because in level
one the people who were able to do level one tasks with an 80 percent
probability were combined with those who weren\'t able to do it and so
there was no lower bound to that level one. And that sort of confused
many people who wanted to know who were the illiterate adults and we
really weren\'t able to say anything about that.

So in 2003 we commissioned a study done by the National Academy of
Sciences, which has a Board on Testing and Assessment consisting of\--
well luminaries is probably the right term in the field of educational
testing. They set up a committee to study the reporting levels of the
1992 survey and what they\'re going to do is to review those reporting
levels and if they think they can do a better job or think it should be
done differently, they\'re going to set standards on the adult literacy
test. They haven\'t told us whether they will recommend to us a new set
of reporting standards or keep the old ones and if they do provide new
reporting standards, they haven\'t told us how many categories. Will
there be five or six or three? We don\'t know that.

They are planning to release their\-- they\'re planning to tell us what
their recommendation is in January, which we will then use in writing
the report. Our report will be due out in July.

Back to you, Sheida.

Sheida White: OK. As you can see from this next slide, the total
interview takes approximately 90 minutes. The amount of time\-- the
respondents, then, have an unlimited time\--

Andrew Kolstad: Could I interject something?

Sheida White: Yes.

Andrew Kolstad: This is a little bit longer than we did in 1972\-- in
1992. It was about one and a quarter hours rather than one and a half
hours the way we\'re doing it now and the idea was to give a little more
time for the background questionnaire, which was a little too short last
time to get the information we needed.

Sheida White: OK. And the reason we did not give respondents an
unlimited time is because we think that the rate of efficiency in
responding is important in many everyday tasks and also unlimited time
is very expensive.

Andrew Kolstad: We\'re paying these interviewers to listen\-- to watch
them take the test.

Sheida White: That\'s right. So interviewers encourage respondents to
move on if it appears that they\'re struggling. The interviewers will
say something along the following lines. They may say, "Some of the
tasks in this section can be fairly difficult. Many people do not
complete all of them. Why don\'t you go on to the next section?"

Here is a visual of the interview flow. All participants begin the
interview by responding to an oral background questionnaire.

Andrew Kolstad: Now oral, that means that they don\'t have to write or
read anything at this point, right?

Sheida White: Yes.

Andrew Kolstad: Yes.

Sheida White: The background questionnaire is administered using a
computer-assisted personal interview, also known as CAPI, system.

Andrew Kolstad: It doesn\'t assist the respondent, right? It assists the
interviewer.

Sheida White: Administered using\--

Andrew Kolstad: Yeah, so the interviewer is using the computer, not the
respondent.

Sheida White: Exactly.

Andrew Kolstad: Right.

Sheida White: Exactly. We use the\-- we believe that the use of the CAPI
system has several advantages. For example, it reduces interview error
and it speeds the production of the data.

In terms of\-- we\'ve been talking about the seven core items and how we
use them to select the ALSA participants. The main NAAL participants,
once they\'re determined that they\'re main NAAL participants, they read
the assessment questions from the printed booklet and they write their
responses in pencil in the booklet. ALSA respondents, on the other hand,
give their responses orally to oral questions.

Andrew Kolstad: But they can\'t get it right unless they read something,
right?

Sheida White: Yes, exactly. They have to refer to printed material in
order to find the answer.

Andrew Kolstad: Right and is that printed material in English or
Spanish?

Sheida White: That is in English because this is an assessment of
English literacy\--

Andrew Kolstad: OK.

Sheida White: \--not Spanish literacy.

Andrew Kolstad: OK.

Sheida White: At the end of the interview, all participants, whether
they\'re ALSA or whether they\'re main NAAL participants, they all take
the Fluency Addition to NAAL, also known as FAN, which requires them to
read passages aloud from printed booklets. The participants\' responses
to the FAN are recorded using a special CAPI software that is
incorporated\-- that incorporates, actually, an automatic speech
recognition technology in it. It\'s a very fascinating system and I have
seen how it operates.

Andrew Kolstad: And, Sheida, you didn\'t mention the company that is
providing us this oral recognition.

Sheida White: The company is called Ordinate and they have developed
this automatic speech recognition technology. They are a subcontractor
to our data collection contractor, who is Westat.

So we don\'t really deal with them directly. We deal with them through
our contractor.

Now here are some accommodations. I know many of you out there are
interested in accommodations for adults with special needs.

In 1990 there was a passage of the Americans with Disabilities\-- that
was passed, Americans with Disabilities Act and then in 1991 we had
Individuals with Disabilities Education Act. Because of these two Acts,
assessment programs at all levels have been encouraged \-- they have
been, actually, forced \-- to consider issues of inclusion and
accommodation.

I\'m happy to say that accommodations are built into the design of NAAL.
It\'s an inherent part of our assessment. We offer two kinds of
accommodations. We\'re on page 40.

We offer two kinds of accommodations, administrative accommodations.
These are the first three bullets on this slide that you see and we
offer language accommodations. These are the last three bullets.

Andrew Kolstad: Those first three are things\-- well, most educational
testing programs administer one-on-one and allowing additional time are
really some of the major accommodations that are done in most testing
programs\--

Sheida White: That\'s correct.

Andrew Kolstad: \--although they don\'t do it in quite as friendly an
environment as people\'s own homes.

Sheida White: Including NAEP. That\'s correct.

Andrew Kolstad: That\'s right.

Sheida White: Yeah, the reason we do not offer instructions in other
languages other than Spanish is the practical one. It would be too
costly. It\'s just very expensive to offer assessments in Chinese and
Japanese and other languages.

Andrew Kolstad: The Census Bureau may provide something like support in
80 different languages, but they\'re surveying everyone and we just
don\'t have the resources to do that.

Sheida White: I also want to underscore the fact that the assessment
itself is always in English because this is an assessment of English
language literacy.

Next slide on the background questionnaire. I think I\'m going to just
skip this one because we\'re running a little short of time. I\'m going
to skip this one. It\'s very self-explanatory. You can take a few
moments to look at it.

Andy, do you want to\--

Andrew Kolstad: I was thinking of skipping this\-- No, this is a
technical thing having to do with what do we look at when we first get
the questions back from the survey and these are some of the preliminary
things. We want to know how reliable it is to score the item. We want to
know what percentage of the population gets it correct and how
correlated is the item performance with the rest of the items in the
test form that people take.

Sheida White: OK.

Andrew Kolstad: So this is preliminary work for analysis.

Sheida White: So this last section in this\-- in this section, it\'s
really important that you understand this and I think it\'s a very
exciting aspect of our scoring system. When we score the NAAL items we
think it is really important that we establish criteria that are
consistent with this larger message of what adult literacy represents.

So the principles behind the NAAL scoring rubrics are based on the
conceptual information about literacy which we use in NAAL and that is
the ability to use printed information to function in one\'s life.

So, for example, bullet two \-- so, for example, if a respondent makes a
minor error in a catalog, in a shopping catalog, it probably is not
going to have a whole lot of implications because the information is so
redundant in catalogs. There\'s a product number, there\'s a price and
then there is a description. It is intended\-- it\'s designed that
way\--

Andrew Kolstad: To be redundant.

Sheida White: \--to be redundant. As a result, even if a respondent
makes a minor error in an order form, it will probably not result in an
incorrect order being placed.

On the other hand, if a respondent makes a minor error in a Social
Security number, that would count as an error because it is an error\--
that application is just not going to be processed. You\'re not going to
get your benefits.

Andrew Kolstad: Right.

Sheida White: So, again, this goes back to our definition of adult
literacy, which is using information to function.

Andrew Kolstad: Sheida, wouldn\'t it have been easier to just give them
multiple-choice questions instead of worrying about these subtle
differences in the answers in scoring?

Sheida White: Why don\'t we give\-- Because we\'re going to get a whole
lot more information from open-ended responses than in a multiple
choice. And, in fact, while we\'re at this\-- while we\'re at this let
me say that we are planning to in the future\-- we are planning for it
now, to offer assessments in three new areas and that\'s vocabulary,
writing and computer literacy.

So going back to the open-ended responses, we can actually
multi-score\-- multi-score the responses. We can look at the response as
an open-ended task and score\-- not just score the ability to comprehend
what they have read, but also get some measure of their writing skills
and that needs to be looked at. So\--

Andrew Kolstad: I thought you\'d say that multiple choice are easy for
us as testing people to handle, but it\'s less reflective of what people
actually have to do in the real world.

Sheida White: Right. OK. Let\'s move on to\-- I think it\'s the last
section. Andy, do you want to take over on the analysis?

Andrew Kolstad: Is this the one on the software? Software analysis?

Sheida White: Yeah, um-hmm \[affirmative\].

Andrew Kolstad: OK. We\'re\-- the slide is on new software. Well, the
American Institutes for Research, our contractor for the analysis, has
actually developed another form of software to handle the data that we
get, the partial scores that we get for each person and this will be
available to the public. It\'s not a proprietary piece of software and
it\'s actually already available on the web at am.air.org where it\'s
freely downloadable and usable for analyzing either the old 1992 data or
for other assessment programs like the trends in International
Mathematics and Science or the National Assessment of Educational
Progress.

AIR believes that this is faster, but I think it\'s not going to speed
up our report because of all the other things that are going on that
also are delaying our report. So it may make it easier to do analysis by
other people.

Sheida White: OK.

Andrew Kolstad: Now when\-- when they do the analysis what we get is we
get average scale scores and score distributions. By score distributions
we mean what percentage of this group lies along the scale and above
whatever cut points we\'re using or what the percentiles of the scores
are.

So for the population as a whole and for population groups, but we
don\'t really\-- we don\'t get scores for individuals because they
haven\'t taken enough of the test to give them a score. They only have
taken a part of the test so we can only give them a part score.

Sheida White: It\'s unreliable.

Andrew Kolstad: Yes. So it would be\-- this is\-- The next slide says
what I just said over again, so let\'s move on to the trend analysis
slide.

We can compare the average scores for prose, document and quantitative
literacy between the \'92 and the 2003 in terms of average scores or in
terms of these new performance levels, if they do recommend new
performance levels and the Department of Education accepts them. And
then we also can compare\--

Sheida White: Two big ifs.

Andrew Kolstad: Two big ifs, yes.

And then we will also report on the percentage of the population giving
the correct answer to each item. I think that this may actually surprise
people, because in 1992 we never reported how many people got each of
those sample questions right. And I think the numbers may be higher than
people think. So that\'s\-- I\'m really looking forward to getting that
far.

Now it may not be terribly meaningful if we\'re still keeping the test
question secret to know that 55 percent of the population got this
question right when you don\'t know what the question is, but, still,
for the ones that we have released the item, it\'ll be very informative.

Sheida White: I think you were going to\--

Andrew Kolstad: We will. We will. But the thing is, and we\'ll have
those P values for the ones that we release, but it\'s not meaningful
unless you can see what the question is, I think. OK.

Sheida White: I see what you mean.

Andrew Kolstad: So all of these released items will be available on the
new\-- on the adult literacy website in its reformed version, which
Sheida and her team is working on right now, trying to get a new\-- a
new version ready. We will publish the P values, the percent correct for
all the items. And even for items that are not released and you don\'t
know exactly what it means, you\'ll still know whether people did the
same or better.

Sheida White: Absolutely. Yeah, as Andy says, we are in the final stages
of\-- I\'m right now looking at the test question search tool on the
NCES website. We are in the final stages of developing a test question
search tool, which is what we call it, on the NAAL website. This tool
will allow you, the users, to search for a question item and for
stimulus material that meets\-- that meets your criteria.

At the moment, the search tool includes 92 items, all of which were used
in the 1992 assessment. Most of these items were carried over from the
1985 assessment, actually.

So as you can see from this\-- from your screen, the tool allows you to
search by assessment year. It allows you to search by test type, but
literacy scale, by stimulus materials and for each of these categories
there will be a pull-down menu list, a pull-down menu that would tell
you all of the available options.

In addition, you will be able to type in a word or a phrase. Let\'s say
that you\'re interested in looking at the phrase "health literacy." The
next slide would display an item having to do with health literacy that
we have searched for. So here you can also click on the\-- Are we OK?
OK. We can also click on the links that would allow you to see the
correct answer, which is really a nice option. It would allow you to see
the major task demands. This is a new area that we are working on like
whether or not it demanded the task to search a text or to draw
inferences.

OK. Below the question the item itself is displayed, along with its
corresponding stimulus material. So, actually, when you\-- when you do
use the website, you will be able to scroll down to view more of the
stimulus material.

So, anyway, new types of results in 2003 \-- this is my last slide. I
must say that we\'ve never before had this opportunity to conduct such
analysis. Although the trends between 1992 and 2003 are available for
prose, document and quantitative literacies, the trends between 2003 and
future assessments will be available for the new type of data that we
have.

I think that, perhaps, you can also say something, Cheryl. I think this
enormous amount and the variety of information that we\'re going to be
providing by NAAL will be able to serve multiple, multiple purposes and
benefit various audiences. And I\'m not talking about the traditional\--
the conventional audiences, the researchers and analysts and the
practitioners, but also people like in business where they have
workplace literacy training. I\'m talking about medical groups. I\'m
talking about practitioners, policy makers, a whole host of audiences
who benefit because we have so many new features and types of data that
we can provide.

And you will be able to\-- if you are interested in getting more
information about NAAL that will be tailored to your\-- and this is very
interesting, we are tailoring our data to your particular interest. This
is also a very new phenomenon for NAAL. We have identified 16 areas of
interest. So, for example, if you are interested in, say, family
literacy you can circle that and we will be able to send you any
information that pertains to family literacy automatically in the
future.

We have\-- on the last page we\'re telling our websites, our telephone
numbers. You are welcome to call us or write us e-mail at any time with
your questions. As I said before, we are always thinking of new ways to
enhance the assessment. Right now we are planning for writing,
vocabulary and computer literacy. We would like to hear what you think
of these new additions.

Again, I would like to thank all of you for joining us and I would like
to thank NIFL for organizing this, again. I\'d like to thank them for
inviting us and thank you, Cheryl, for participating.

Andrew Kolstad: Cheryl, your name and phone number and e-mail address is
not on this slide. Maybe you could just let people know how to reach
you, as well?

Cheryl Keenan: Yes. My e-mail is cheryl \-- c-h-e-r-y-l \-- .keenan \--
k-e-e-n-a-n \-- \@ed.gov, which is really the best way to reach me.
Thank you.

Sheida White: So we took up almost two hours\--

Andrew Kolstad: We have a question already. And we have a question for
us. Were any of the 26 forms specifically tailored to measure the
literacy levels of adults with disabilities such as a visual impairment?

I think people were allowed to use magnifying glasses if they have them
and normally use them in their reading. So I don\'t think we
specifically tailored them for that purpose, but the difficulties of
these graphics and their text was taken into account so that people who
had visual difficulties were able to use glasses or magnifying glasses
if they normally did.

Sheida White: And let me say that we went to a great extent\-- to great
pains to make sure that we\-- that the letters and the figures were
large enough for most people to be able to read.

Andrew Kolstad: I\'m glad to hear that. We have another question here
from North Carolina. Was the difficulty of each test question determined
in advance, based on the complexity of the task, or afterwards, based on
the performance of how well the adults did? And I think the answer to
that question is, it\'s a little of both. Because the state of the art
in testing is not that good at predicting just exactly how difficult a
test question will be. We had a sense of that and we can sort of
predict, but we\'re not always sure about that.

Sheida White: But we also use the field test to help us.

Andrew Kolstad: And it says a related question from Amy Trawick \[ph\]
in North Wilkes-Barre, North Carolina, is do the levels refer to the
tasks or to the participants? And that\'s a good question. I think
the\-- in terms of defining what the levels were, it was done in terms
of the tasks, but once those definitions are used to set cut points on
the scale, then you could apply those cut points to the distribution of
participants and say how many or what proportion of the adult population
performs in each of those categories.

So\-- Well, Cheryl, we\'re running out of questions that have been
submitted in advance. Did you have any questions for us in listening to
us because you listened to us some of the time?

Cheryl Keenan: Yes. You know, actually, I learned quite a bit and it\'s
really been interesting being here. You see this information so many but
every time I hear it new light bulbs go on.

Andrew Kolstad: That\'s right.

Cheryl Keenan: One of the things that I had the question coming in
today, but of course I had the opportunity to talk to you before and get
the answer to is, on the timeline, on where we\'re at in the study now
and when do we think that the information will become available to the
general public?

Andrew Kolstad: OK. Let me say\-- take a stab at it. We finished our
field work for the main assessment in January\-- at the end of January
and we completed the scoring, Pearson Educational Measurement completed
the scoring in May, right?

Sheida White: Um-hmm \[affirmative\].

Andrew Kolstad: And while that was going on the Westat sampling people
were working on preliminary weights. So they finished the weights and
turned them over to AIR. AIR, then, did the scaling and at the end of
July turned over their preliminary scaling work and data to the National
Academy of Sciences to work on the performance levels and when they
report back to us at the end of January, we\'ll, then, start writing our
report to NCES.

Sheida White: But the first report is going to come out in July. That\'s
a very highlight report. That\'ll be July 15 of 2005. The comprehensive
report will come out in December of 2005 and the technical report will
come a month or so later in January or February of 2006.

But I wanted to say something. I know that a lot of you out there think
that we have waited too long for this report to come out and it
certainly has taken a long time for the reports to come out, but I think
that it has been well worth it\-- the wait because we have developed
enormous amounts of new instruments for us to be able to give you more
variety of information and also we have commissioned the National
Academy of Sciences to develop new performance levels and, as you know,
those performance levels in 1992 were very controversial.

So I think that we are going to have a very, very solid and very
thorough and comprehensive assessment, even though we have to wait a bit
long for that.

Cheryl Keenan: OK. Thanks.

Andrew Kolstad: OK. Do we\-- Laura \[ph\] in New Jersey asks, "When will
we be able to request the tailored information from the adult literacy
survey?"

Sheida White: The tailored, what does she mean?

Andrew Kolstad: What you talked about it in the last slide, the
various\--

Sheida White: The 16 areas?

Andrew Kolstad: Yes.

Sheida White: Oh, OK. Yes, we are actually planning a number of issue
briefs. It will be eight or 10 issues briefs that concentrate on
particular topics.

Andrew Kolstad: What is an issue brief for those who don\'t work at
NCES?

Sheida White: OK. An issue brief is a very, very brief publication.
It\'s usually three or four pages only, but nonetheless, it goes through
a very, very rigorous adjudication process, more so than our regular
reports because we are taking the information out of its natural context
and making it salient\--

Andrew Kolstad: So it\'s on a very small facet of performance, each one
of these briefs\--

Sheida White: On a topic.

Andrew Kolstad: \--would be on a small topic.

Sheida White: A small topic\--

Andrew Kolstad: Right.

Sheida White: \--that we found interesting data.

Andrew Kolstad: Right.

Sheida White: So we\-- And also, on some of the topics we may choose to
have, actually, a brief\-- a larger report but still brief, maybe
25-30-40, even 100 pages. For example, on health literacy, on prison
population, perhaps something on low basic reading skills. We may have a
special report on that.

So there will be many different ways to\-- and also, in addition to all
of this, we are in the process of completing frameworks. We\'re going to
have several types \-- \[inaudible\], Andy. We\'re going to have
three\-- at least three different types, or four different types of
frameworks.

One is going to be a more theoretical framework on the skill-based
literacy, you know, talking about all those skills that we talked about
earlier in the presentation. And then we\'re going to have a less
technical version of the same report, a briefer and less technical
version, and then we\'re going to have a framework on the background
questionnaire and one on health literacy.

So we\'re going to have a number of those frameworks coming out, a
number of reports, a number of issue briefs, specialty reports.

Andrew Kolstad: OK. Well, does the audience have to rely on the reports
that we produce?

Sheida White: No. We encourage them\-- You want to say something on
that? We encourage them very much to be secondary analysts and use our
data. We will be happy, as I said earlier, to provide them with the
training that they need.

Andrew Kolstad: Do we have a date when we will publishing\-- making the
data available for secondary analysts? Public-use data or even
restricted-use data? We have\--

Sheida White: That should come right after the technical report, which
is\-- the current date is February of 2006.

Andrew Kolstad: I should say that because of the privacy and
confidentiality rules of the federal government we are going to have two
versions of the data file. One will be sort of cleaned and carefully
screened to make sure no disclosure can take place, no matter who
analyzes the data, and for that you won\'t need a license.

For the more detailed data where it\'s possible that a disclosure could
happen, we will make people sign a license\-- apply for a license to use
the data and that restricts the use to certain things and the
restrictive use\-- one of the main things it restricts people to do is
that they would promise that if they release any data they would be
subject to prosecution. So that is not any data, I mean, any data that
would disclose the identify of an individual taking part in this survey.
That\'s the main thing that the government is worried about.

So we\'ll have two forms of the data file.

Sheida White: I also get a lot of phone calls and e-mails from people
asking us for\-- these are analysts, secondary analysts, asking us for
the use of our instrument, like the FAN instrument or the assessment.

Andrew Kolstad: Or the assessment form. Suppose I want to know what\'s
the literacy level of the people in my program, can I use the adult
literacy survey?

Sheida White: What would you say?

Andrew Kolstad: No.

Sheida White: Do you want to explain?

Andrew Kolstad: Well, the reason is that it\'s not designed to measure
the abilities of individuals, on the one hand, so you\'d be better off
using the old Test of Adult Literacy Skills that used to be published by
ETS \-- I don\'t know if it\'s still in business or not \-- or one of
the other commercial tests. But ours is not designed to measure the
ability of a person but rather the ability of a population since we have
26 different test forms.

Sheida White: But what if they wanted\-- what if they do want to measure
the ability of a population?

Andrew Kolstad: Then we also have a problem with that, in that we don\'t
permit the use of the blocks that are being retained for future\-- use
in a future survey for comparability purposes. We want to measure how
literacy has changed from 2003 \'til the next time, a decade from now,
when we do it again and we want to use the same test questions. If a
portion of the population has been exposed to these questions they might
not work in the same way. So we retain for\-- and keep secret the
questions that we\'re planning to use again.

Sheida White: OK. And we have written procedures for all of these
requests.

Andrew Kolstad: Well, the questions that have been released will be much
more readily available now than they ever were in the past because of
this web tool that you\'ve been developing. I think people will get to
see more\-- a greater variety of test questions than they\'ve seen
before.

Cheryl Keenan: There\'s another question, will this webcast be available
for viewing on a DVD?

Sheida White: Yes, I believe Armand \[ph\], who is the person running
this studio, he told me that it would be available on a DVD.

Andrew Kolstad: From whom? From NIFL, probably? I don\'t know. We may
have to address this question to the people running this.

Sheida White: Well, we have\-- I certainly can have access to it, but I
don\'t know how the general public could access to it. We need to
probably pose that question to Armand \[ph\].

Andrew Kolstad: Or to NIFL. The National Institute For Literacy would
probably be the ones distributing it, I would think, since they\'re the
sponsors of this webcast.

Sheida White: Right. Yeah, you\'d probably have to send a request and
purchase it.

Cheryl Keenan: So anyone listening at the NIFL office, if you want to
weigh in, we\'d appreciate it.

Andrew Kolstad: Well, we have no more questions on the screen. Cheryl,
did you have more questions or things you wanted to query us about?

Cheryl Keenan: Well, I\'m very interested on the 16 areas of interest
that was on one of your last slides. Are you\-- is the purpose of this
online form at Ed Pubs to get an idea of where\-- which of these 16
areas are more popular so that you could produce these issue briefs.

Andrew Kolstad: That might be a good idea. I don\'t know if we\'ve
thought of it.

Sheida White: That\'s not really the purpose of those. We want to have a
dissemination list, an ongoing dissemination list. And we want to know
who our audiences are and what their interests are so that we can
better\-- so we can better serve them, so we can tailor our reports to
their needs.

Andrew Kolstad: I have a question about one of these on the list.

Sheida White: Let me just finish.

Andrew Kolstad: OK.

Sheida White: If we have a report on prison populations and we know all
individuals who are interested in incarceration, OK? Then we can
communicate with that particular group. We want to reach as many people
as we can and as many interested people as we can to really make our
dissemination as efficient as possible. So it\'s not\--

Andrew Kolstad: Sheida what can we tell people about homeless education,
because the homeless were basically not part of our population? They
don\'t live in households and so we wouldn\'t have interviewed them.

Sheida White: OK, but keep in mind that these areas of interest, we are
collaborating with NIFL.

Andrew Kolstad: OK.

Sheida White: We are combining our databases\--

Andrew Kolstad: OK.

Sheida White: \--our dissemination databases. Some of these areas we are
interested more in, some they are more interested in.

Andrew Kolstad: Oh, I see. So this is combined with NIFL.

Sheida White: This a joint effort with NIFL.

Andrew Kolstad: OK. I didn\'t understand that, because\-- so the
National Institute might well know something\-- know a lot about
education of the homeless\--

Sheida White: Exactly.

Andrew Kolstad: \--even though we don\'t in our survey.

Sheida White: Exactly, even though we don\'t. Yes.

Andrew Kolstad: OK, good.

Andrew Kolstad: OK, Joe \[ph\] from NIFL has responded to your request
by saying, let them know that the webcast will be archived and
accessible on l-i-n-c-s, lincs, which is, Cheryl?

Cheryl Keenan: Which is NIFL\'s launch technology database. I was able
to go on to the lincs website and go into the\-- I think it\'s called
webcasts, I\'m not sure, but there\'s a certain area on that page and if
you click on it you can see the archived videos on there.

Sheida White: By date.

Cheryl Keenan: By date. And so it will be available on there and it\'s
very easy to access, really much easier than getting their hands on an
actual DVD.

Sheida White: That\'s right. And the address for that\-- it\'s long, but
I\'ll tell you anyway\--

Andrew Kolstad: It\'s on the last page of the slides.

Sheida White: Yeah, it\'s
www.nifl.gov/nifl/webcast/2004/0908webcast09-08.html.

Andrew Kolstad: Wow. OK. So that\'s for this one.

Sheida White: There\'s a lot of webcasts and a lot of dates, 9-8-2004.

Andrew Kolstad: Right.

Sheida White: So\--

Cheryl Keenan: Oh, there\'s one new question up.

Andrew Kolstad: OK.

Cheryl Keenan: When will certain 2003 questions that will not be used
again in the next assessment be released on the web tool?

Andrew Kolstad: Well, I don\'t think there are any questions that we
plan not to use.

Sheida White: We would\-- we will make that determination after the\--
all of the data are analyzed and after we complete the reports we will
sit down and determine which of those questions\--

Andrew Kolstad: So you think we are going to release some questions?

Sheida White: Definitely.

Andrew Kolstad: OK.

Sheida White: Yeah, definitely we will be releasing some questions from
the 2003.

Andrew Kolstad: Not very many.

Sheida White: We don\'t know how many or which ones.

Andrew Kolstad: Well, we have seven blocks of questions.

Sheida White: Yes.

Andrew Kolstad: If we release them, we\'ll probably release them all
from the same block.

Sheida White: Yes.

Andrew Kolstad: So that block wouldn\'t be able to\-- Yeah. So they
would have\-- one block has maybe 15 questions in it at the most?

Sheida White: About 15 questions.

Andrew Kolstad: So there might be five on each scale, at the most.

Sheida White: Right. It would be at the scale of what we have been\--
what we did before.

Andrew Kolstad: Yeah, right. Right. Well, that was a good question. OK.
I\'ve sort of run out of things. Did you have more that you wanted?

Sheida White: Erin \[ph\], do you have any questions for us? She\'s our
new employee.

Andrew Kolstad: OK.

Sheida White: All right. So, again, thank you very much for joining us
today on International Literacy Day and thank you, NIFL. Thank you,
Jaleh. Thank you, Sandra \[ph\].

Andrew Kolstad: OK.