NAAL -- December, 2003

Sheida White: We appreciate your interest in NAAL and are pleased to
welcome you today. I\'m Sheida White, the NCES project officer for NAAL.
My educational background and doctoral degree are in linguistics. Prior
to joining NCES in 1991, I worked as a full-time reading researcher for
six years. During my first eight years at NCES I monitored item
development for NAEP, which stands for National Assessment of
Educational Progress, and during the past four years, I have been
monitoring the NAAL project.

I have invited my colleague, Andrew Kolstad, to join me today. I am so
glad he has had the time to do that. Andy was the project officer for
the 1992 assessment. He is now the senior technical advisor on NAAL. He
will be talking to you about the NAAL analysis and reporting. We are
also very fortunate to have Westat \[sp\]; and AIR, which stands for
American Institute for Research; and ESSI, which stands for Education
Statistics Services Institute, to help pus with data collection, data
analysis, and general planning.

As we will now see, NAAL is the result of a decades-long trend towards
more effective literacy assessment. I would like to say a couple of
other things. We have created an evaluation form for you to complete, if
you will. This is the first time that we are doing a webcast
presentation, so your constructive feedback would be very helpful to us.
You can e-mail your feedback to <sheidawhite@ed.gov>. That\'s
s-h-e-i-d-a, that\'s white, at ed.gov.

One more thing before we start \-- I was asked by the staff at NIFL and
thinking of Jean Laba Lusee \[sp\] and Sandra Stein, in particular, to
whom I am very, very grateful for this opportunity. Whether or not I
have a question for you, the viewers, and I do. I have one question, and
that is do you think that we should provide an estimate of illiteracy in
our assessment? And, if so, what do you think the definition of
illiteracy ought to be? So think about that question.

Here are the major topics that we will be covering today. First, we are
going to give you an overview of NAAL. Then new features of NAAL will
give you a base description of some new instruments that we have
developed called FAN, Fluency Addition to NAAL; also, adult literacy
supplemental assessment, and a new health literacy component. We also
will talk to you about NAAL\'s secondary analysis and how we can support
you, the researchers, with your efforts.

I understand there will be questions. We will stop at each of these
major topics and respond to your questions. You can send your questions
via e-mail or by telephone, and if we don\'t get to answer your
questions here, we will be happy to respond to those later. You can
e-mail us, and we are generally fairly prompt in answering questions. We
enjoy it when people are interested in our survey.

OK, let\'s start with a brief history of NAAL. We have had two
assessments of adult literacy in the past, and we are now embarking on
the third one. Here is why NAAL is important. I think this last bullet
is especially important. For example, at the federal level, the Office
of Vocational and Adult Education, which is known as OVAE, has, in its
objectives, developing indicators of the extent to which programs
sponsored by the agency actually improves literacy in the U.S. The 2003
NAAL is among the better sources that OVAE and states who participate in
NAAL use and, thus, and indicator of the improvement and as an indicator
of the relationship between adult literacy and key issues such as
citizenship.

Here is our task-based definition of literacy. The key words in this
definition are \"use to function.\" Some key features of the definition
\-- I think bullet three requires a little bit of an explanation. The
phrases in the definition \"to achieve one\'s goals and to develop
one\'s knowledge and potential\" emphasize that the appropriate goals
come from the individual people. These goals include running for
political office or simply accomplishing everyday tasks like paying for
an electric bill. At the same time, the phrase, \"to function\" in
society suggests that there is a minimum level of literacy that all
adults must achieve regardless of their own personal goals. And that\'s
why I asked you early on if you please let us know what you think of a
definition of illiteracy.

Is it, for example, the ability to decode and recognize words smoothly
and fluently and with little effort or is it the ability to understand
connected text or is it both? We would appreciate your feedback on that.

Here are three areas of literacy \-- there will be a separate scale
score for each measure of prose document and quantitative literacy.
Every task in NAAL is classified as prose document and quantitative. We
have very few hybrid tasks. In fact, there are only four out of 149
assessment items that we have \-- only four of those are what we call
hybrid. These tasks have their answer either in a prose that is embedded
in the document or in a document that is embedded in prose. Now, if the
response requires processing the prose segment, it is coded \-- the task
is coded as a prose task and by \[inaudible\].

Let\'s look at a sample prose task. First, prose tasks are tasks for
which the correct response requires processing continuous text. In this
example, here we have a brief newspaper article that is titled \"Swimmer
Completes Manhattan Marathon.\" The question asks readers to underline
the sentence that contains information on what Miss Chan ate during the
swim. Now, only one sentence gives any information about food in this
text. This is a typical, relatively easy task. The text is short, and
the task demand is to locate a single piece of information that is
easily identifiable.

We have several types of prose \-- expository, procedural, persuasive,
narrative, and poetry. Now, let\'s look at the example of a document
task. These are tasks for which the correct response requires processing
non-continuous text, although I must say that we think the question
actually requires processing connected text. The question here reads,
\"What is the gross pay for this year to date?\" The respondent simply
has to locate a number in this task.

> These are the types of lists and tables that are included in the
> assessment. Other document tasks, like maps and forms and bills
> obviously have different structures.
>
> OK, let\'s consider here a quantitative task and this is an example of
> it. These are tasks that require respondents to either identify
> computations or perform computations on numbers that are embedded in
> text, either in prose or in documenting text. In this sense, they are
> different from school-based or academic mathematics, which tend to be
> decontextualized. Now, the question here reads, \"You need to borrow
> \$10,000. Explain how you would compute a total amount of interest
> charges you would pay under this loan plan?\" Now, I would like to
> explain what the respondents have to do to get this question right.
> The quantitative tasks tend to be, in general, more difficult than
> prose and document tasks, and I think it would be helpful if I take
> you through the processes that the reader has to go through.
>
> First \-- that\'s OK, you don\'t have it. First, to get the correct
> answer, the respondents must locate the \$10,000 in the text, in the
> document. Then the respondent has to locate the monthly payment, which
> is \$156.77. Then the respondent has to infer the statement \"120
> months, 5.25% APR\" means that the monthly payment will be due for 120
> months resulting in an annual interest rate of 5.25%. Then \-- this is
> the first process \-- then they have to identify the actual
> computation, the formula, and that is \$156.77, which is the monthly
> payment, they have to multiply by 120 months, and then subtract
> \$10,000 principal from it to get the total interest. Actually, this
> example did not ask the respondents to perform the computation, only
> to explain how they would get to that answer.
>
> I want to say a word about calculator use. The calculator is available
> to all respondents throughout the assessment, and they are free to use
> it or not use it, as they see fit. Unfortunately, the 1992 and the
> 2003 actually main assessment did not assess skill at performing
> computation with a calculator and skill at performing computation by
> hand. So a preliminary analysis of the sample textbook makes us feel
> that respondents did not always use the calculator, even when
> performing multi-digit multiplications or divisions. So, for the
> now-present population, the interviewers will record when the
> respondent actually uses the calculator. In addition, we are
> considering a special study on calculator use, so we\'ll give more
> information on that.
>
> The next \-- I\'m going to switch again here and move to sampling \--
> metric sampling. I\'ll give you a moment to look at the slide. I would
> like to elaborate on the second bullet for a moment. We wanted to
> capture the full content domain of adult literacy in the U.S. In order
> to do this, a very large number of items, 149 to be exact, is needed
> to do this. It would be unreasonable, I think it would be very
> exhausting to both the interviewer and to the interviewee to
> administer all 149 items to every single respondent. It would take
> over three hours. So instead of administering all of the items to each
> respondent, we do what is called \"metric sampling,\" which
> administers only a portion of the total number of items to any one
> participant.
>
> Here is our block design. Each respondent takes one of 26 booklets.
> Each booklet contains the same seven, easy core items, and each
> booklet contains three of 13 blocks of tasks that we have.
>
> Here is our individual booklets, how they are designed \-- actually,
> individual blocks, I am sorry \-- take a look at this slide for a
> moment. Regarding the last bullet, most of these 26 booklets \--
> actually, 21 of the 26 booklets, include at least one 2003 block and
> at least one 1992 block. The remaining five are either all 1992 or all
> 2003 blocks.
>
> Here is a slide on a NAAL sample. Let me take you through this slide,
> a bullet at a time. First bullet, I saw primarily in households,
> because while we don\'t go to group quarters, we do administer to
> those living in college dormitories if, at the time of data
> collection, college students happen not to be living at home. In other
> words, we don\'t wait for them to come home. With respect to bullets
> two, three, and four, theoretically we should have about 20,600 adults
> in our sample. This is counting the national, state, and the prison
> samples but, in reality, this number may actually look more like
> 18,000 or 19,000 after the data collection is complete, hopefully by
> the end of January.
>
> Bullet number five, I use the field test sample was partly random and
> partly purposes, and by purposes I mean non-representative but still
> cross-sectional so the results are useful and meaningful. As you know,
> the field test is an important dress rehearsal for the assessment. It
> provides an opportunity to test the materials, the approaches, and the
> duration of the assessment.
>
> Bullet number six \-- we over-sampled black and Hispanics because it
> is unlikely that we will have a large enough sample of blacks and
> Hispanics to be able to conduct meaningful, so we over-sample minority
> groups, and then we adjust for it statistically.
>
> OK, a few words about the administration of the main NAAL \-- the
> sample designed for the 2003 NAAL is based on four stages, as you can
> see. First, to your upper left, to the upper left, you will see the
> selection of what we call \"primary sampling units.\" They are also
> called PSUs. They consist of counties or blocks of counties. Then
> there is the selection of area segments, which consist of census
> blocks or groups of blocks. Then they have, obviously, the selection
> of households and individuals living in those households.
>
> Now, this is important for you to know, because the sample is selected
> proportional to the population in the United States, some smaller
> states do not have any adults selected in the NAAL sample. However,
> the residents of these small states are represented in the NAAL
> through the samples adults with similar literacy-related
> characteristics who reside in other states. In other words, all adults
> were given a chance of selection, and a sample was drawn with higher
> chance of the selection given to areas with no population, and this
> was done to select an optimal sample. Now, consequently, states that
> have more population end up with larger samples and states with
> smaller population end up with no PSUs in the sample, correct, Andy?

Andrew Kolstad: That\'s right.

Sheida White: OK. Next slide, I know some of you are interested in how
we address issues of learning disabilities. We have several questions on
disability in our rather extensive \-- very extensive \-- background
questionnaire. The reason we have only several and not very many is that
\-- is the concern that the more detailed questions about disability may
set interview expectations for respondent performance because the
assessment follows the background questionnaire. Also based on available
research \-- there isn\'t a huge variety of interventions for specific
learning disabilities. So the four questions that we ask are \-- have
you been diagnosed or identified with having a learning disability? Do
you have any difficulty seeing the words in ordinary newspaper, even
when wearing glasses or contact lenses? Do you have any difficulty
hearing what is said in a normal conversation with another person, even
when using a hearing aid? And do you have any other health problems?

We offer two kinds of accommodations. One is administrative
accommodation and one is language accommodation. Here in this slide you
see our administrative accommodations. As you can see here,
administrative accommodations for the disabled adults are wider inherent
in the design of NAAL.

All right, here are the NAAL goals. Essentially, the first two bullets
tell you that NAAL is designed to take a snapshot of the status of
literacy, both at the national and state levels for those states who
participate in NAAL.

> With respect to the last bullet, for example, we work with the
> Department of Health and Human Services to provide a new measure of
> health literacy, which I will be talking more about that later. Also,
> later, I\'ll discuss how NAAL is going to facilitate the use of NAAL
> data by many of you viewers who understand our researchers. OK, I
> guess, Andy, you can take away from here reporting.

Andrew Kolstad: Yes, I am going to talk for a while on the reporting of
data from the National Assessment of Adult Literacy. There are several
ways we plan to report the data. One is in terms of scale scores, as
before, we expect our scales to be reported with numbers that range from
zero to 500. The scale score summarizes performance across all the
literacy tasks on the assessment. So if we have 40 questions on prose
literacy, the single scale score summarizes how well people did on all
40 tasks. We would report average scores and score distributions for
different groups of people. We will also be reporting in terms of
performance levels, but what those levels are haven\'t been decided yet.
We will publish sample test questions or assessment items and with some
responses to show how we scored the answers.

Reading the report, the p values are \-- that\'s a technical term for
the proportion of the American population that gets a test question
correct. In our 1985 and 1992 surveys, we just \-- I guess we felt this
was too simple and never recorded it, but I think people may well be
surprised by the fairly substantial proportions of adults that get these
questions right, because their literacy assessment is actually a fairly
easy test for most people. We will also show items maps, which show the
relative difficulty of the assessment items and how they relate to the
scale scores.

We will provide scale scores for groups of people but not for individual
students, because the assessment isn\'t really designed for that. We
will prepare these scores in a way that they are comparable between the
1992 survey and the 2003. We will be able to show how scale scores have
changed or improved on the prose document and quantitative literacy
scales over the period. The performance levels that we will use in
recording are under consideration right now at the National Academy of
Sciences. They have commissioned a group within the Board on Testing and
Assessment to set achievement-level goals for America and allow us to
report the proportion that meet those goals and once we have these
goals, we can go back and re-report the 1992 data in terms of those
categories of performance.

Now, as I mentioned, we don\'t have scores for individual people. It
wouldn\'t be appropriate to calculate them because the individual scores
are so unreliable. Each person gets such a small sample of the test
questions on the prose scale around the document scale that it\'s not
enough to provide a score, and it\'s not enough to cover the full domain
of the test questions for any one person. Nevertheless, we can get
average scores for groups by combining the results from many different
test forms taken by many different people.

This next slide shows a sample table in which we show the average scale
score on the 1992 prose literacy scale for five categories of adult
education, the first being less than high school, high school
graduation, or a GED equivalency; less than college, college graduation
or more than college. The second line of this table shows the standard
deviation of scores within each of these categories and while
researchers know what standard deviations are, if people are not
familiar with it, the next picture on the next graph illustrates the
distribution of scores around the central tendency. On the left we see
those who score the lowest, the less than high school group, and the
other four categories are the other ones we had in the table.

What is kind of interesting is that they all overlap. There are people
within each education level that score above the average for all the
other groups. So there is \-- while the average group certainly goes up,
there is a range of scores within each category that is quite
substantial, and that\'s what\'s captured by a standard deviation.
Another interesting feature of this graphic image is the tail at the low
end. For people with the least education, there may well be some people
who are illiterate in that category, and they seem to be pulling that
tail down to the bottom.

This graphic image was generated by AM software. It\'s a new feature
that was just added in the latest revision, and we\'ll talk later on how
people can obtain copies of it. It was written at AIR with support from
NCES, and since it was publicly supported, it\'s available to the public
for free. So people can download the AM software and generate a chart
like this themselves.

Now I want to talk a little bit about the performance levels we\'ll be
using. In 1992, we recorded five literacy levels, and I think all
researchers who have looked at the 1992 data or read reports from the
1992 data are familiar with those levels. They were defined by grouping
together items with similar task demands, and then the boundaries
between these groups of items formed the cut points on the scale. The
problem is comes in the point where we tied all the items to the scale,
because the location of the items on the scale was dependent on our sort
of abstruse aspect of IRT theory, which, in fact, response \[inaudible\]
criteria, and I\'ve written a number of papers on this, and I can refer
you to those papers, but we\'ll just talk a little bit about them, and
I\'ll answer questions if people want to hear about that.

But one of the basic problems we had with the recording levels from
before is that the Level 1 is quite broad, and it doesn\'t really make
enough useful distinctions for the literacy community within Level 1. We
hope that the new levels developed by the National Academy of Sciences
will address this issue.

We would be releasing many examples of items. There are some of the
assessment questions used in 1992 that we\'re not using again and all
those that we will not be using again will be released, and we will also
be re-recording the proportion of the population answering all of the
1992 questions correctly as well as the proportions for the new items in
the 2003 assessment.

Now I just want to talk a little bit about item maps. There are several
ways one could show the relative difficulty of items, and I can think of
three, and that\'s what I put in the slide. One could map items in terms
of the percent correct with the easiest items at one end of the scale
and the hardest at the other, but then you would have to show the
relationship to a proportion from zero to 100, and what you might want
to see is the relationship of the difficulty of the items to the scale
we use in recording. One way to do this would be to map using the IRT
difficulty parameter, the one that comes right out of the model we\'re
using for scaling. We didn\'t do that in 1992. Instead we used an
arbitrary response probability convention of 80%.

> Other conventions give you a different relationship of the items to
> the scales, and on this next slide, you probably can\'t read this, but
> you see the four gray bands across, which track the distinction
> between the groups of items. At the top of the page, you see above the
> gray line, you see items in Level 1. Above the second gray line from
> the top you see the items in Level 2, and above the third gray line
> you see the items in Level 3. On the left side of this slide with RP80
> at the top, we see \-- that is the actual \-- that\'s a segment of the
> actual item map that we showed to the public in publishing all of our
> reports. The other two columns of this figure show how the same item
> maps would have appeared had we used a different mapping criteria \--
> a response probability of 65% or 50%. And I have to say there is no
> consensus in the assessment field for what the proper value is. Our
> National Assessment of Educational Progress reports \-- maps items
> using a 65% criteria; many testing programs map items using a 50%
> criteria. When the NAEP program selects exemplar items for this scale,
> they choose any item above 50%, and the bookmark procedure used in
> setting standards for many state testing programs, tend to use a 65%
> or 67% criterion. So there is really no consensus in the field as to
> what the right value is. I think the adult literacy program is
> probably unique in choosing such a high level.
>
> Let\'s move on \-- I\'ll answer questions on that issue if people have
> them. Let\'s move on to talk about what we can and can\'t do with the
> adult literacy assessment. We actually can provide a snapshot of
> literacy status and trends from 1992 for the population as a whole and
> for various subjects of the population, and we can describe how the
> skills relate to the backgrounds of the adults that we found out from
> the background questionnaire. What we can\'t do with the adult
> literacy data is to provide evidence of program effectiveness. We
> can\'t provide instructional practices and interventions based on this
> survey, and we can\'t provide cause-and-effect relationships. And
> there is one other thing not on this slide that we can\'t do, and that
> is we can\'t provide the test forms used in adult literacy for use by
> literacy practitioners to find out what their score is of their
> clients, because we don\'t have a single test form. It wasn\'t
> designed to measure the literacy of any one person. So it\'s not
> suitable for that.
>
> The next section is on the new features of the adult literacy survey.
> Oh \-- are we going to take a question break here?

Sheida White: We are going to take a little \-- yeah. OK, let\'s start
with \--

Andrew Kolstad: \-- we have gotten some \--

Sheida White: \-- some questions \--

Andrew Kolstad: \-- questions submitted from the audience already.

Sheida White: All right. There is a question asking what are the six
participating states and those are Kentucky, Maryland, Massachusetts,
Missouri, New York, and Oklahoma. There is a question \-- are
respondents asked about their language or cultural background or are
non-native English speakers eliminated at earlier points?

OK, yes, the answer to the first part of the question is that we ask
extensively about the respondent\'s background \-- linguistic background
\-- and the answer to the second part is no, that we do not eliminate
non-native speakers of English at an early point.

Andrew Kolstad: Can I add to that, Sheida?

Sheida White: Yes.

Andrew Kolstad: In the 1992 survey, we provided a Spanish language
translation of the background questionnaire. We do that again in 2003,
and the field procedures permit a household member to translate the
English form into whatever language is needed in a household, if such a
household member is there. But if we can\'t communicate with a person
because we only hired Spanish and English language interviewers, then we
are not able to deal with that.

Sheida White: Well, I wanted to add one more thing. As Andy said, we
accommodate Hispanic adults by what Andy said, but also this is a new
feature in the 2003 NAAL \-- the core screening items, the seven core
screening items that I talked to you about \-- those are, along with the
accompanying instructions, are offered in Spanish and in English, and
also the two new instruments that we developed \-- what we call FAN and
also the \-- all instructions to those two instruments are also
available in Spanish. But one thing that \--

Andrew Kolstad: Well, Sheida, you have to point out that the text that
they have to read is still in English. It\'s only the instructions and
questions that could be in another language.

Sheida White: Exactly, exactly, this is an assessment of English
literacy, not an assessment of Spanish literacy. But I want to add one
more point to what the caller asked \-- OK, we are accommodating
Hispanic adults, but also we are accommodating, in a way, non-Hispanic
ESL adults in the following way \-- in 1992 if the language spoken at
home was other than English or Spanish, the background questionnaire
could not have been administered, and the case would have been closed as
language problem, and the remaining assessment, the core and the main
NAAL, could not be administered. Therefore, there was no way of getting
literacy data for non-Hispanic adults who did not speak English well
enough to be interviewed for the background questionnaire.

Now, even if they could read English well enough to complete the core or
the cognitive assessment, they would not be part of the assessment. Now,
in 2003 NAAL, the non-Hispanic ESL adults have the opportunity to take
the core items, even if they haven\'t finished the background
questionnaire. So this is one way of accommodating, let\'s say, Asians
or other groups.

So, let\'s see, we have one more question. Do you want to attempt this?
Since our state is a small agricultural state, our individuals were
probably not sampled. Where can we find information on the populations
whose results we should look to when analyzing NAAL data in relation to
our students?

Andrew Kolstad: I\'m not sure that I understand that question. That\'s
\-- if we don\'t have information from a geographical region, it\'s \--
from a small area \-- then we are bound to have a problem recording on
it. One way \-- we report on each region, and we report on urbanicity,
so we would be able to report on rural areas, and we might be able to
report on the South or maybe even rural areas in the South, but we might
not be able to report on a small county in the South. Do you think that
answers the question?

Sheida White: Perhaps the individual wants to call us. Could we ask the
individual to call us?

Andrew Kolstad: If they want to follow up on that.

Sheida White: If they want to follow up, perhaps they can send us
another e-mail or call us. I just want to add something to what you just
said \-- we are considering a special study on small-area estimations,
which is also called \"synthetic\" estimation to provide us with more
information about certain geographical areas not covered under NAAL, and
that would be like non-participating states, larger cities, or large
counties.

Andrew Kolstad: Right. Steve Reeder \[sp\] gave such a project for the
Office of Adult and Vocational Education \--

Sheida White: \-- in 1992 \--

Andrew Kolstad: \-- for the 1992 data, and he based his results on the
relationship between literacy scores in areas where we did have
information and the census data in those same areas, and then did a
projection based on census data for other areas where we had no data on
how the literacy scores might have been if we\'d conducted a survey
there. So it\'s possible that those projections could be helpful,
although they are less accurate than they may appear.

We have another question here \-- how were the levels defined by the
National Academy of Science, relate to the current level definition in
our reporting?

> I would say that we don\'t know the answer to that, because that\'s up
> to the National Academy of Science.

Sheida White: Yeah, but whatever levels they come up with, we analyzed
the 1992 data using those new levels. So we would be able to compare the
results between the two assessment years.

Andrew Kolstad: So \-- basically, we don\'t know yet.

Sheida White: We don\'t know yet.

Andrew Kolstad: OK, shall we go on?

Sheida White: No phone calls, I guess.

Andrew Kolstad: OK.

Sheida White: Now, this next section is very exciting, I think, at least
to me. These are the new features that we have developed for NAAL 2003.
OK, if you look at your slide that is titled \"New Features of NAAL,\"
let\'s look at \-- let\'s start with participants. In 1992, the state
Adult Literacy Assessment, I believe it\'s there for Florida, did not
include adults over 65. In 2003, they will. The percentage of population
over age 65 is increasing, and many state programs are aimed at this
population.

OK, another \-- the second bullet under participants \-- I think, as I
mentioned earlier, we are also giving the non-Hispanic ESL adults the
option of taking the assessment, and this is partly because we now have
a supplemental assessment for the missed literate adults.

OK, the second item, the second major feature \-- and this, indeed, is
the most important new feature that we\'ve added to the assessment, and
that\'s instrumentation. We had expanded the information at the lowest
literacy level, and I will be talking in more detail about the
instrument after this set is over. In terms of \-- let\'s see \--
performance level you talked about, and in terms of block design, our
primary goal in developing new items was to update the NAAL pool of
items, many of which had become dated. But still we wanted the items to
be modeled after the 1992 items, allowing us to measure trends.

We eliminated, for example, a task that required all responses, because
now we have all reading fluency. We eliminated, I believe, 13
multiple-choice items, and now the assessment is entirely 100% open. And
then we had one item that required respondents to write a letter and,
really, there wasn\'t much we could do with one item, so we eliminated
that as well. Other than those, the new items that we have replaced
those with are very similar to the 1992 items.

In terms of assessment tasks, we have a new assessment \-- our health
literacy assessment, and there will be a separate score associated with
that. Again, later on I will say more about this assessment in detail.
In terms of ESL adults, as I said earlier, the core screening items are
in Spanish, and we are trying to accommodate not just Hispanic people
but also other ethnicity and cultural backgrounds.

In terms of administration, and these are completely new in 2003, we
have CAPI system, which is Computer-Assisted Personal Interview; we have
cognitive laboratories; the incentives has gone up by \$10 per
individual from 1992. We are offering partial scoring that we did not
have in \--

Andrew Kolstad: You mean scoring of partial credit items?

Sheida White: Yes, yes.

Andrew Kolstad: The items we scored either correct or \--

Sheida White: \-- or incorrect \--

Andrew Kolstad: \-- and this allows for a correct \--

Sheida White: \-- partially correct and \[inaudible\] \-- yes, thank
you.

Andrew Kolstad: And the scale is now able to take care of that.

Sheida White: Exactly right. In terms of reporting, our reports are
going to be more timely, and, again, we are going to say more about
these two bullets later on. This is just a little overview.

Now, this is important. I\'d like you to really pay attention to this
next slide. Why did we decide to add all of these instruments and revise
the performance levels and so on? As Andy said, in 1992, Level 1 was
very broad and very diverse. The 21% in 1992 represented approximately
one-fifth of the population, which is \-- what \-- 50 million people or
something?

Andrew Kolstad: Forty to 44 is what we said in the report.

Sheida White: That was in 1992, right, OK \-- a large number of people
it represents. And 20% of the 21% was about 5% of the sample population,
and that would have represented approximately 8 million or 10 million
\--

Andrew Kolstad: \-- are you talking about the second bullet here?

Sheida White: Yes, individuals \-- now, many, at the time, in 1992,
considered this group to be illiterate. OK, and then in 1992 this 5% was
treated as if they had taken the assessment but got in the wrong answer.
In other words, their scores were imputed.

Andrew Kolstad: Well, some people took it and got them all wrong; some
people gave up and said, \"I can\'t read,\" or \"I don\'t speak English
well enough.\" And in those cases, where we had clear evidence that they
couldn\'t do the task, we gave them the wrong answer as if they had
taken it. That\'s the principle involved there.

Sheida White: That is correct. OK, there is also potential increase in
low-scoring adults in 2003 and the nature of their deficit was very
unknown; whether they couldn\'t decode or recognize words or whether
that they could decode and recognize words, but they did it with effort
and laboriously or was it that they simply didn\'t have higher-level
reasoning, you know, interpretation skills? So the lack of distinction
made within Level 1 around other possible factors led, at the time, in
1992, led to headlines declaring, like, 20% of America\'s adults were
functionally illiterate. This contradicted adults \[inaudible\] report.
Seventy-one percent of the respondents in Level 1 reported that they
read English well or very well, but we didn\'t see this contradiction in
other levels. It was only at Level 1.

OK, so given the size and the diversity in performance in Level 1 in
1992, given the unknown nature of the deficit, given the growth of low
literate population, something had to be done in order to identify the
scales and characteristics of this particular low-performing population.
So one approach was to revisit the performance level, and another one
was to develop a measure of basic reading skills, and that\'s how FAN,
which is Fluency Addition to NAAL, and also which is Adult Literacy
Supplemental Assessment were created.

OK, Andy talked about the performance levels. Generally, people like
performance levels because they are easy for people to understand but,
as Andy said, they have been subject to controversy in the past, and we
are very fortunate to have the National Academy of Sciences help us with
the new levels. Just a little bit about the Academy and how they are
working \-- the performance levels will be set, of course, after the
assessment data are analyzed. The levels will take account of
stakeholder feedback, they will be outlined in a preliminary report that
is due to us in December of 2004, and, as I said earlier, they would be
reflected in the re-analysis of the data.

OK, now here is perhaps the most important feature of all of the new
features that we have developed for now, and that is FAN and ALSA. Now,
why it is important to look at basic reading skills that these two
instruments measure, is adults cannot decode or recognize words in a
list or in a paragraph. They obviously cannot access the meaning of the
words. If they can decode and recognize words, but they do so very
slowly and with effort, they will certainly struggle to comprehend the
text. If they can decode fluently, but if they don\'t have the
higher-level literacy scale, they will struggle in their effort to draw
inferences and make the proper connections to the background
experiences.

So \-- I said early on that \-- and I showed you the NAAL task-based
definition of literacy, which was the ability to use printed information
to do something with. Now you see, in this slide, you see the
complementary skill-based definition of literacy, which suggests that
successful use of printed information is a product of both word-level
reading skills and higher-level reading skills.

The next chart, the flowchart shows you that when NAAL is administered,
every participant receives the same background questionnaire, receives
the same course cleaning items, and the same FAN tasks. Now, very low
performance on the seven easy core screener items identifies also
participants. Now, based on the data we have collected so far, and this
is as of maybe a month ago, approximately 4.8% of the assessment
participants have taken also. Of this approximately 5%, 63% are
Hispanic, 18% are black, and 17% white, and only 4% other races.

Andrew Kolstad: I did want to add something here.

Sheida White: Let me finish my \--

Andrew Kolstad: \-- OK, go ahead \--

Sheida White: \-- OK, now, I want to say that most of the ALSA folks
cannot read connective text, which means that they can\'t even read the
question itself.

Andrew Kolstad: Well, what I was going to add is that this is not
designed to replace what was done last time in terms of the adult
literacy. I think all the people who are being routed into the
supplemental assessment are people who would not have been able to take
the NAAL in 1992 or in 2003, because they\'re sort of off the bottom of
the scale, and this is designed to provide some measurement of how those
people can perform and what it is they can do.

Sheida White: Exactly, and I would say a few more words about each of
these two assessments after this set of questions is over, and we answer
some of your questions. But let me finish first, and then we\'ll get to
the questions. We have already gotten some questions.

The entire interview, as you can see from the next slide, takes
approximately 19 minutes, and even though NAAL is not timed, it\'s
really not the case that the respondent has unlimited time, and this is
because the rate or efficiency is important for many everyday tasks and
unlimited time is time-consuming and therefore expensive.

OK, as I said earlier, we have not only expanded information at the very
low end, but we have also expanded the NAAL background questionnaire. We
did this in order to better reflect the economic and technological
development and better understand the background of low-performing
adults and health-related activities and family literacy.

Now, the expanded \[inaudible\] questionnaire was hoped to reflect, at
least in part, the action agenda for literacy, which was released in
2000, the year 2000, and it addressed these four domains that was
perceived to be of great concern to adult literacy.

OK, here are the background categories. I don\'t know if you have access
to our brochure. We also have a brochure on our website or the NAAL
brochure. The NAAL brochure also includes these variables, these
background variables. We have two versions of the NAAL background
questionnaire. There is a general version, which you see on your screen,
and then there is a version that is administered to prison populations,
and the numbers of questions that you see in parentheses \-- they are
the number of questions for each category. Now, some questions may have
sub-parts to them. Technology questions cut across most of these
categories. I guess that\'s all about that slide.

This is just a brief description of our health literacy. It provides
information on health literacy. We have a separate score for health
literacy. We look at the relationship between health literacy scores and
a variety of background variables as well as to other scale scores.

Andrew Kolstad: Sheida, before you go on, we have a question on health
literacy, which I thought we could fit in right now.

Sheida White: Could I see it?

Andrew Kolstad: Yes, I\'ll read it to you. It says, \"The health
literacy component is very interesting to me. Please explain whether the
health literacy component is measuring participants\' knowledge of
health and health-related issues or rather if it measuring
participants\' ability to read and process information text about
health.\"

Sheida White: It\'s the second part \-- we are not measuring adults\'
knowledge of, for example, what to do if they have hypertension, for
example. This is not knowledge of health, per se. It\'s the ability to
use their \-- or to apply \-- their literacy skills to health-related
materials, and there will be more on that, on the health literacy, in a
little while.

Andrew Kolstad: OK.

Sheida White: OK, as I mentioned earlier, we have a CAPI system, which
is new. We didn\'t have that in 1992. As in 1992, the background
questionnaire items are read aloud to the respondents. However, instead
of reading from a printed questionnaire, as was done in 1992, the
interviewers in 2003 read the items from laptop computer screens, and
this computerizing the questionnaire makes it easier to target items at
particular populations. It makes the production of usable data files
much more speedy, speedier, I should say, and it results in fewer
interviewer errors, we feel, in selecting the appropriate questions.

Andrew Kolstad: Excuse me, of interviewer errors \-- it\'s so good at
checking on what the interviewers do that we found interviewers who were
actually falsifying data and replaced them with new interviewers who
redid the interviews.

Sheida White: That\'s right, and we wouldn\'t have known that in \--

Andrew Kolstad: \-- we wouldn\'t have known that without the computers
\--

Sheida White: \-- in 19982, that\'s right. So, in a way, the data are
more \--

Andrew Kolstad: \-- cleaner and better \--

Sheida White: \-- cleaner and better data \--

Andrew Kolstad: \-- than we did last time, yes.

Sheida White: Yeah, mm-hm. OK, also, the cognitive laboratories, which
we did not have in 1992 are extremely helpful in making the assessment
test better tasked, because we can clean up any problems with respect to
these four bullets that you see. Cognitive laboratories, as you may
know, are one-on-one interviews.

OK, I\'m going to skip the new software, which is also a new feature,
because Andy is going to say a whole lot more about that when we get to
the section on secondary analysis.

I wanted to say one thing \-- that even though we are adding all of
these new, wonderful features to the new assessment, that we want to
make sure that we are still able to compare our results to the 1992
survey, and these are the ways we do that.

OK, the questions \-- can we get to the questions now?

Andrew Kolstad: All right, we have several questions. Let\'s start with
the first one, which was this one, which is a question about how the
assessment is used for non-native speakers. And the question is \-- \"If
they actually do try the core items, will the results be compared to
those of native English speakers or at least will they figure in the
picture of overall literacy? And if this is so, doesn\'t this one
counter to the current wisdom that giving non-native speakers
assessments designed for native speakers merely test their English not
their actual literacy skills?\" Non-native speakers can be quite
literate in their first language in core and English skills, and we
wouldn\'t be measuring their actual literacy skills, just their English
skills. So this would be a different animal. So how would you respond to
that?

Sheida White: Well, I would say that this goes back to a definition of
literacy.

Andrew Kolstad: Which is literacy in English.

Sheida White: Yes, which is literacy in English and which is being able
to \-- the ability to use printed information to function in society. So
using printed recent information is our task-based definition of
literacy.

Andrew Kolstad: Yeah, but nevertheless, while for the overall skills,
they will be lumped in together. We could, with the background data,
distinguish the non-Native speakers and report separately on what their
skills are, what their literacy skills and maybe when you get into the
skill area, we\'ll find where their skill deficits might be.

Sheida White: And we had a minority report in 1992, and we can do the
same \--

Andrew Kolstad: \-- on English language minorities \--

Sheida White: \-- minorities, so we can report separately \--

Andrew Kolstad: \-- right \--

Sheida White: \-- for this group.

Andrew Kolstad: OK.

Sheida White: I would be interested in a follow-up with this individual
to understand what other aspects of literacy you have in mind that we
ought to look at. If you care to expand on this, that would be very
informative, and we can even have a dialog on this later on, if you
like.

Andrew Kolstad: Well, another person writes in to ask, \"Can you say
more about the process the National Academy of Sciences will use and how
literacy stakeholders and others may participate in the process?\" We
have two bits of information for you here. One is the project
monitor/director at the Board of Testing and Assessment is Judy Koneig,
and her email address is <j-k-o-n-e-i-g@nas.edu>. And they \-- the next
meeting of this committee will be February 27th, and at that meeting
they will be seeking input from the stakeholders. So if you want to have
an input to this process, you should communicate \-- send an e-mail to
Judy Koneig. I can also give her phone number \-- 202-334-7879? 2879?
Which is it? Oh, this is the project support person. But you could get
close to Judy.

Sheida White: The meeting is open to public.

Andrew Kolstad: Right, to the public. So contact Judy Koneig at the
National Academy of Sciences to get information about how to reach the
committee.

Now, the next question is \-- \"Given the important association of
literacy practices to proficiency, what\'s the rationale for having so
few items on the background questionnaire assessing literacy
practices?\" What literacy practices do we query in our background
questionnaire?

Sheida White: Oh, a lot. Actually, we are going to put the background
questionnaire, in its entirety, both in its regular and CAPI form on the
website, on our NAAL website, as soon as the data collection is complete
and that is the end of January. But just to give you an answer now, we
are \-- for example, we are asking questions like \-- of people who are
parents \-- we ask them whether or not they read with their children and
what kind of \-- and if they do read to their children, what kind of
reading activities they do. We get very specific information in that
area. But I\'ll be happy to give you more information on all of the
family literacy questions that we have, if you care to send me an
e-mail.

Andrew Kolstad: The person who asked about the health literacy component
submitted a second, unrelated question. She is interested in the
assessment of the prison sample, and says, \"In regard to my state, I am
wondering if the prison sample scores will be included in all the scores
of the participants in my state, thus driving our scores down, or if
they will be separated?\" And the answer is \--

Sheida White: \-- both \--

Andrew Kolstad: \-- the prison sample will not play any role in the
state data. That\'s considered to be part of the national sample. We
don\'t have a state representative prison sample.

Sheida White: For the states who participate.

Andrew Kolstad: For the states that participate, the prison sample would
not be part of that \-- of those \-- for those six states. They would
only be part of a national result. In the 1992 survey, they were
included in the national results but not in the separate state samples.
So we\'re following that practice.

Sheida White: It seems like that would be important to be included in
the national, because that was the reason we wanted to have a prison
sample.

Andrew Kolstad: The prisons are highly concentrated, we don\'t go to
every state, I think we are going to something, like, 80 to 90 \-- or is
it 100 \-- prisons? Somewhere around that order of magnitude, and they
are very concentrated in certain areas, and they would draw down the
score if they were considered to be part of that state or locality.

Sheida White: OK, so we are ready for our third set of \-- and that\'s
the instrumentation. I am going to start with the first new instrument
that we developed, and that\'s Fluency Addition to NAAL. These are sort
of units of analysis. We\'re going to look at words read per minute,
words read accurately per minute, and possibly \-- we\'re still
negotiating this \-- words read accurately and smoothly per minute, and
by smoothly I mean aspects of reading \-- we\'ll be looking at aspects
of reading, like hesitation, both intra-word hesitations and inter-word
hesitation.

> Alex, I think I am going to go quickly over the next few slides,
> because they are various self-explanatory, and we don\'t have a whole
> lot of time. Except for the last graph, that I would like to talk
> about. Maybe you can just take it at a pace where people can read and
> just move on.

Andrew Kolstad: This is the one \-- on word-level reading skills and
text-level reading skills?

Sheida White: Yes, uh-huh, and so that\'s one and let them read.

Andrew Kolstad: So also respondents get an easier version of the fluency
assessment.

Sheida White: They get the one-page \-- the one passage \-- the shorter,
simpler passage, yes.

Andrew Kolstad: Are you going to describe the administration of this to
the \-- well, maybe I\'m going too fast.

Sheida White: The technology?

Andrew Kolstad: Yes, the technology and the fluency assessment is quite
interesting. It couldn\'t have been done in \'92.

Sheida White: OK, I\'ll talk about the technology a little bit. Do we
have this slide? OK.

Andrew Kolstad: Computer-assisted \--

Sheida White: OK, let me just say a few words about the technology, that
is, if you\'re interested in technology. The interviewer tells the
respondent to read each passage quickly, but at a speed where they can
understand what they are reading, because they will be asked a question
about the passage, but that\'s really only to keep the respondent
honest. We are not scoring that question.

The interviewer then \-- did they have this in there? No, they didn\'t?
OK. Well, the interviewer then clicks a \"record\" button, which tells
the respondents to start reading. And they start the computer\'s timer
and recorder for a minute and other measures then are computer after the
recordings are processed. Again, at one minute, the computer beeps and
\-- but only the interviewer can hear it \-- and the interviewers
terminate the recording at that point and tell the respondents to stop
because the computer has recorded a long enough sample for them.

Administration of NAAL \-- I just want to say, during the field test
operations, the interviewers actually manually went and marked the last
words on \-- the last words that the respondents read.

Andrew Kolstad: That is if they didn\'t get all the way through it.

Sheida White: No, this was actually \-- we used this manual mark by the
interviewers to check the accuracy of the computer software to make sure
that \--

Andrew Kolstad: I would have used the computer to check the accuracy of
the interviewer, but that\'s because I\'m a technology person.

Sheida White: Right. Now, if a respondent completes the reading of a
passage in less time than one minute, then the words-per-minute is
pro-rated to one minute.

This next slide challenges in scoring the FAN data, we are in the \-- as
we speak, we are in the midst of discussing what the definition of
accuracy should be. So maybe this is another question for you. Earlier,
I said if you want to tell us what your definition of illiteracy is and
whether or not it should be included in our data, consider this another
question \-- how would you define accuracy of reading if you were to
measure reading in terms of speed?

Andrew Kolstad: And what\'s hard about that?

Sheida White: Measuring accuracy?

Andrew Kolstad: Yes.

Sheida White: Because you want to be fair to people with dialect
variation and pronunciation difficulties, and you don\'t want to count
as an error systematic differences in ESL pronunciation and dialect, and
we are looking into black dialects, and we are looking into Southern and
New England dialects \--

Andrew Kolstad: And the computer can recognize all that stuff?

Sheida White: Hopefully.

Andrew Kolstad: That\'s great.

Sheida White: Feel free to send us any questions. We didn\'t get any
phone calls. You can call us or you can send us e-mails. I\'m sure this
provokes a lot of questions in your mind. We are really very interested
in being able to create a basic reading skills scale score, and we are
working towards that.

OK, let me take you to this more difficult slide \-- the hypothetical
analysis. As an example this graph shows the relationship between oral
reading rate, oral reading \-- that\'s right, oral reading rate and
quintiles of, say, literacy levels. For each literacy quintile, the mean
is shown as a diamond \-- I hope you can see that \-- is shown as a
diamond, and the range \-- 75th and 25th percentiles, are shown as
arrows and squares. So, for example, for quintile 1, the average reading
rate might be around 40, and the 25th and 75th percentile may be around
20 and 60 words per minute \-- this is all hypothetical, of course.

> Now, the relationship here may be curvilinear, as this graph suggests.
> Reading words may increase sharply, say, between quintile 1 and
> quintile 2, and, again, between quintile 2 and quintile 3, and
> increase somewhat less between quintile 3 and quintile 4. Now, after
> that, the relationship may be flat. Reading rates may be about the
> same as quintile 4 and quintile 5. In other words, many adults, even
> at quintile 2 and even possibly 3, may be lacking reading skills,
> especially where they are encountered with more difficult passages.
>
> Now, adults in quintile 1 may read, let\'s say, 40 words \-- this is
> important, I\'d like you to pay attention to this \-- and adults in
> quintile 1 may read about 40 words per minute on average, OK? Now, if
> this turns out to be the case, or something like that, then now to
> announce it as in 3, will provide significant new information. I think
> perhaps a revelation.

Andrew Kolstad: Is that because people who read at 40 words per minute
are often not able to understand what they are reading? They\'re going
so slow it interferes with their understanding.

Sheida White: Exactly, exactly, it would suggest that the basic skills
do, indeed, play a role in limiting the functional literacy of some
adults. And that\'s the point of all of this.

OK, let\'s move on to a next new instrument, and that is Adult Literacy
Supplemental Assessment, also call this ALSA. Again, I am going to
really let you read through this, Alex, is fairly self-explanatory. The
slides on abilities assessed by ALSA \-- low-literacy adults rely,
really, very heavily on context and familiarity to comprehend text and
is often referred to compensatory strategies.

Andrew Kolstad: People need to keep in mind that the only ones who are
taking this part of the assessment are the ones who have been through
the core and didn\'t do very well, and so they would not have the
ability to take the rest of the assessment.

Sheida White: Is this on?

Andrew Kolstad: Oh, you said that already?

Sheida White: Is this on \-- \[inaudible\] OK.

Here are the assumptions underlying ALSA. The next slide shows our goals
for ALSA. Now, as we analyze the ALSA data, we will have a greater sense
as to how high someone can score on ALSA. I want to say a couple of
words about the second bullet on the goals of ALSA. I think it is
important that you keep in mind that the ALSA scale and the NAAL scale
are not linked together. In other words, we have no connection \-- in
other words, no one takes both ALSA and NAAL that, let us say that, say,
a score of \"x\" on ALSA will correspond to a score of \"y\" on the
NAAL.

The next slide \-- selection of ALSA respondents. The scoring guides are
designed in such a way to make sure that no one who is in ALSA, as you
said earlier, that no one who is ALSA belongs to the main NAAL. There
are individuals who are incapable of meaningfully interacting with
demand \-- with the main NAAL. And we have very elaborate algorithms
that we have developed to make sure that this does not happen. So the
likelihood that someone who is taking ALSA is also capable of taking
NAAL is really very, very, very slim. The algorithm was intended to be
very conservative in the number of people who can go on to ALSA.

The \[inaudible\] are fairly self-explanatory as is the similarities \--
let\'s look at the similarities between FAN and ALSA. ALSA, here, as you
can see, permits compensatory strategies because the written material is
contextualized, it is familiar, it\'s tangible, but FAN does not. Here
are some \-- the next slide shows the differences between FAN and ALSA.
Notice that, for example, there are no pseudowords in ALSA. The ALSA
tasks are followed by genuine comprehension questions, and I said
earlier the FAN question is just to keep people honest.

Andrew Kolstad: We have a question to follow-up on that \-- someone
asked, \"What is the reason\" \--

Sheida White: \-- \[inaudible\] \--

Andrew Kolstad: Yeah, but it\'s so related to what you\'re talking about
just then. \"What is the reason for not scoring or checking
comprehension on the FAN? Isn\'t comprehension an important aspect of
reading?\"

Sheida White: We have a lot of other measures of comprehension. We have
\--

Andrew Kolstad: A lot of other measures?

Sheida White: We have a lot of other measures of comprehension in the
NAAL assessment, and we really don\'t think that this one question is
going to add a whole lot more information to what we already have.

And, also, we need to keep in mind that all oral reading is really not
something that adults do very often. It\'s not something that\'s part of
their habit, and it\'s possible that they would be a little bit
distracted in terms of the meaning of what they are reading. When they
are reading for an audience, it becomes more of a performance. So it\'s
not appropriate, I would say, oral reading is more appropriate for
measuring basic reading skills like decoding skills and word recognition
skills, at least in this context.

I also wanted to say that \-- notice that in ALSA, we provide direct
measures of respondents\' prior knowledge. We ask them whether or not
they have experience with some of the tasks that they get to do.

The next slides show the difference between the stimulus materials in
FAN and in ALSA. Notice here that the ALSA stimulus materials are highly
contextualized for this before the very familiar \-- and also notice \--
this was very interesting to me when I created these charts, to see
that, really, the ALSA passages are very few and much shorter in length.
They are only one or two sentences long compared to FAN passages that
are 150 to 180 words long.

Here are some other differences between the two instruments in terms of
sample. The ALSA sample is a whole lot smaller than FAN sample. It\'s
approximately 5% of the population, so we expect it to be around 1,000
people. In terms of duration, it also takes a lot longer. FAN is only 15
minutes. In terms of administration, FAN is highly computerized. ALSA is
paper and pencil and, in terms of scoring, ALSA is scored on the spot,
FAN is scored after the completion of the interviews. So this is the
last slide in this \-- of ALSA, I guess, is it? And that is overview of
how ALSA differs from FAN, and you can glance at those differences.

Andrew Kolstad: We have a follow-up question to try and clarify what you
were just talking about. This viewer says, \"I do not understand your
statement that the people taking ALSA are not also taking the FAN?\"

Sheida White: I hope I didn\'t say that. ALSA people also take FAN. FAN
is given to everybody. Everybody in the assessment takes the background
questionnaire, responds to the background questionnaire; takes the core
screening items, the seven items; and the FAN. The branching occurs at
the core level, and the people who cannot perform well on the core items
are branches to ALSA, and the rest go to the main NAAL, but everybody
takes the FAN, because the reason is that the FAN is intended to measure
primarily basic reading skills. We were very interested in the basic
reading skills of the least literate population. The only difference
that I can see is that, as you know, there are two passages in the FAN.
One is short, and the more difficult one is at the third-grade level,
and the other one is at the eighth-grade level, and obviously all the
participants cannot take the more difficult passage. They cannot read
the more \-- and we are very lucky if they are able to attempt even the
short, simple passage at the third-grade level, because this population
cannot read connected text. They are more able to read the word level or
even letters and numbers and so forth.

Andrew Kolstad: We have another question on the general issue you raised
at the beginning. Should we report an illiteracy rate, and author here
says, \"That if you do, it should be a measure of functional illiteracy
and should not include limited English-proficient adult immigrants. It
would also be useful to have an accompanying measure, the percentage of
American adult residents who lack the basic skills we now expect of high
school graduates.\" So I think this should be communicated or maybe you
will tell Judy Koneig at the National Academy of Sciences about this.

Sheida White: I have a question for the author of this e-mail. I\'d like
to know what you mean by functional illiteracy.

Andrew Kolstad: Our definition of literacy includes the ability to
function.

Sheida White: Well, but that\'s a definition of literacy.

Andrew Kolstad: Right.

Sheida White: We are interested here in the definition of illiteracy,
and I\'d like to know what the viewer thinks of functional \-- what does
it add to illiteracy that just simply illiteracy doesn\'t have. Isn\'t
illiteracy just illiteracy and what is functional illiteracy? So that
would be helpful to know.

Andrew Kolstad: And it sounds as though this author wants us to report
separately on illiteracy among native-born U.S. residents and
foreign-born or those for whom English is not their first language \--
what their illiteracy rate would be and possibly not combine them, as I
understand it.

Sheida White: Well, again, it depends on what we mean by illiteracy.
Yeah, let\'s just hypothetically assume that illiteracy is lack of
ability to decode and recognize words automatically and fluently and
slightly. If that\'s our definition of illiteracy, and this is
hypothetical, then I don\'t see how you could have a difference \--
apply that differently to different populations, because this is English
illiteracy.

Andrew Kolstad: I think we should encourage other people to submit their
opinions as well.

Sheida White: Yes, please do that.

Andrew Kolstad: OK, we have another follow-up question on the FAN and
ALSA. It said, \"Does he understand \-- or \-- does the author properly
understand that FAN and ALSA are oral assessments?\" Does the main NAAL
have a silent reading component?

Sheida White: Oh, this is a question you\'re asking?

Andrew Kolstad: Yes, that\'s the question.

Sheida White: OK. NAAL has a silent reading component, and that\'s the
main NAAL. That is a silent reading. When we give our assessment, our
main assessment, other than FAN and ALSA, which are supplemental
assessments, it\'s all silent reading. In response to the first part,
the FAN \-- OK, FAN is oral, is an oral assessment in that participants
have to read orally. ALSA is not an oral assessment; it\'s a
paper-and-pencil assessment.

Andrew Kolstad: Well, in the sense that they read \-- whatever we give
them to read, they read it to themselves, and they may provide oral
responses, but they\'re not reading the text out loud. They\'re
explaining what they read to us.

Sheida White: Yes. They are demonstrating their English literacy skills
silently as opposed to orally.

Andrew Kolstad: OK. Now, the next question is a kind of general question
\-- \"How does ALSA compare to the main?\"

Sheida White: To the main NAAL?

Andrew Kolstad: Yes.

Sheida White: OK, well, the main difference between ALSA and NAAL is
that NAAL requires reading connected text. ALSA has, as I said earlier,
has very few passages that go beyond one or two sentences. And ALSA
materials are very highly, highly contextualized. They have logos, they
have pictures, and they are very tangible, and you can actually hold a
food can or a newspaper or whatever. They are very manipulable \[sp\],
if you will, whereas not all FAN material are that way, and \-- what
else \-- they are familiar, they are tangible, they are very
contextualized, and not all NAAL \--

Andrew Kolstad: Well, this is not exactly a question.

Sheida White: I don\'t know if I have answered completely to this
individual. If you would like to have more dialogue on this, I\'ll be
happy to talk to you more on this, definitely.

Andrew Kolstad: Well, you asked earlier, why is it we\'re not getting
questions by telephone? The answer is we haven\'t given the phone
number. Here is the number that people can call to ask questions. The
number is 888-366-5122. So just to repeat that in case people weren\'t
ready to write that phone number down, it\'s 888 \-- which is a
toll-free number \-- 366-5122.

Sheida White: But don\'t call until I finish this next set, because
it\'s part of the third set, and that\'s health literacy \-- if you can
just hold off for five minutes.

NAAL health literacy \-- and you see a definition of that on your
screen. The addition of health literacy was in response to the
Department of Health and Human Services request to provide them with
some quantitative measures and a baseline for literacy.

The next slide on health literacy tasks \-- the health literacy tasks
are associated with all context and settings that are relevant to home,
relevant to community, is relevant to workforce.

Andrew Kolstad: As are the rest of the questions.

Sheida White: Exactly, exactly. Regarding the third bullet, all but one
of the new blocks and, as you know, there are seven new blocks and there
are a total of 13 blocks, seven new blocks \-- all but one of the new
blocks of the new seven blocks had one stimulus material that has two or
three health-related questions associated with it. There is only one
block that has two stimulus materials on health.

These are the three health literacy domains. As I said earlier, health
literacy tasks are really embedded throughout the assessment.

Here are the scales that are assessed by the NAAL health tasks. As I
said earlier, I want to re-emphasize that health literacy scores reflect
the application or the use of literacy skills to materials. We do not
measure health knowledge.

And, actually, searching text, drawing appropriate inferences,
identifying or performing computations, using search material and
inferred material and all of that \-- these are also the kind of skills
\-- this is what we call the higher-level literacy skills that are
assessed by \-- in our tasks as well, not just the health literacy
tasks.

We have 12 health literacy materials \-- stimulus materials, that is \--
and that\'s out of a six-phase total, and these are the areas that they
cover.

The health items will be analyzed in two different ways in NAAL
analysis. We expect that the health-related items will be very similar
to the general health literacy score. Now, if it turns out that the
health items function differently from the rest of the items, we will
\-- how would we deal with that? Would we at least wait for the \--

\[crosstalk\]

Andrew Kolstad: We\'ll cross that bridge when we come to it, because we
really don\'t expect that to happen.

Sheida White: We don\'t expect \-- based on some preliminary results
that we have seen, they should be very similar.

And this is the last slide on the role of processor task. OK, now we are
done with the third set. We have saved the best for last, because that
deals with secondary \-- we are not going to go to the last set yet, but
I just wanted to remind you that the last set will deal with secondary
analysis and should be of interest to those of you who are secondary
analysts. Did we get any questions?

Andrew Kolstad: We have a telephone call?

Unidentified Speaker: Yes.

Andrew Kolstad: But in the meantime, let\'s say again that the phone
number where you can reach us is 888 \-- that\'s a toll-free area code
\-- 366-5122. Who speaks first on this phone call? They or \--

Sheida White: Let\'s take the call.

Andrew Kolstad: As soon as we get these technical details worked out,
we\'ll have a phone call.

\[off mike conversation\]

We\'ll just go on to the next segment and sort out the technical details
soon.

Sheida White: I did mention in the beginning that we have an evaluation
form. They have access to the evaluation forms? OK. So when we finish
with all of this, we really appreciate you completing that form. That
would really help us in future presentations \-- webcast presentations.

OK, I guess you didn\'t have any questions, because you are eager to
start with the next set.

Andrew Kolstad: They didn\'t have questions so we couldn\'t figure how
to handle it.

Sheida White: OK. What is that?

Andrew Kolstad: It\'s the timing.

\[off mike conversation\]

Andrew Kolstad: Yes, we have a question.

Sheida White: Yes? What\'s going on?

Unidentified Speaker: Hello? We\'ve got the call, hello?

Andrew Kolstad: We hear nothing.

Sheida White: Momentarily.

Andrew Kolstad: OK. They\'re still getting the technical connections
worked out.

Sheida White: We can take a few seconds back.

Caller: Are you still on the line?

Andrew Kolstad: Yes, we hear you, we hear you, speak up. Is the caller
there?

Unidentified Speaker: Andy, is the caller there, too?

Sheida White: She\'s here.

Unidentified Speaker: Is she speaking?

Sheida White: Yes, she is.

Unidentified Speaker: OK, what\'s your question, ma\'am?

Caller: I had a question.

Sheida White: Yes.

Andrew Kolstad: OK.

Caller: The question is \--

Unidentified Speaker: Hello?

Andrew Kolstad: You\'ll have to repeat that.

Caller: I am a basic reading researcher interested in adult \--

Sheida White: Oh, we lost \-- can you speak louder?

Caller: Yes. I am a basic researcher interested in adult literacy, and I
was wondering whether the FAN and also will be available. Those of us
who would want to administer those tasks as part of our protocol in an
effort to then be able to slide our sample who might be administering
other tests, but we want to be able to describe the sample relation to
the sample \[audio fade\].

Sheida White: Right. We, really, very seldom give access to the
assessment items themselves.

Andrew Kolstad: Don\'t we have multiple forms of each piece?

Sheida White: These are all brand-new assessments.

Andrew Kolstad: Yes, do we have multiple forms on FAN?

Sheida White: That we have not yet field-tested and haven\'t used?

Andrew Kolstad: Well, in the actual administration is there only one
version or are there four versions?

Sheida White: Well, we have eight passages, but we use all of those in
the assessment.

Andrew Kolstad: In different forms?

Sheida White: Yeah, they\'re in different forms, yeah.

Andrew Kolstad: The different forms is part of the answer to this
question, which is that we don\'t have a single FAN instrument. We give
pieces of the FAN to each of our respondents. So we don\'t have an
overall test form.

Basically, since we\'re trying to measure populations, we give pieces of
our test to each person in our survey. We don\'t give the whole test to
anybody because it\'s too long. So for that reason it\'s not really
suitable. We have not provided that in the past. In the 1992 survey, the
educational testing service, who was the contractor that created the
instrument then created a second version for a book publisher that would
provide individual-level \-- sort of \-- a client diagnostic instrument
that was 30 minutes for each of the three scales. I think that may still
be available if you contact Educational Testing Service and ask about
the \--

Caller: Is that for FAN or ALSA?

Andrew Kolstad: No, it\'s not for FAN and ALSA. We haven\'t created an
individual version of it.

Sheida White: And you said you also have a comment?

Caller: Yes, there was a question about the difference between \-- or
the definition of illiteracy?

Sheida White: Of illiteracy, yeah.

Caller: \[audio difficulties\] OK, off the top of my head, for me, is an
illiterate person could not do anything at all \[audio fade\]. Maybe
they can identify the logos, but they absolutely \[audio fade\].

Sheida White: Right.

Caller: \-- in our country that represents a very, very small \[audio
fade\].

Sheida White: Can you speak louder, please?

Caller: \[audio difficulties\] Sure. \[inaudible\] that person wouldn\'t
have any of the advanced skills but have enough reading ability to
function day-to-say with print requirements that we all have to deal
with as adults is to read street signs, fill out applications,
\[inaudible\]. That\'s how I distinguish in my mind illiterate,
low-literate, and functional illiterate.

Sheida White: Thank you very much.

Caller: You\'re welcome.

Sheida White: OK, we have three e-mails. Two are fairly long. I\'m going
to tackle the short ones. \"Are health literacy items taken from
materials currently used or are they specifically designed for this
test?\"

The health literacy materials are specifically designed for this test.
As I said earlier, there are 26 health literacy tasks. In 1992 we only
had two.

Andrew Kolstad: Yeah, but the question was \-- were the materials
designed for this test? No. They are authentic materials used in
ordinary \-- in government publications and in other places to do the
things that adults normally have to do with health materials. It\'s only
the tasks that were specially designed for the test. The materials are
common.

Sheida White: No, but it says in here, \"Are the materials taken from
materials currently used.\"

Andrew Kolstad: Well, they may be \--

Sheida White: \-- what do you mean by \"currently used\"?

Andrew Kolstad: They may have updated it since we designed the form, but
they were currently used at the time we designed our instrument. They
are real materials. They weren\'t written specifically for us.

Sheida White: OK. In other words, will the questions reflect the type of
literature an individual would currently find in the health environment?
Yes, OK. Yes, that\'s true.

OK, this question has to do with \-- (inaudible) \-- \"The Professional
Organization of Teachers of English as a Second Language recognize
almost a decade ago that functional illiteracy and non-literacy in
English were two very different things\" \-- functional illiteracy and
non-literacy in English. \"For example, when I was recently in Hungary,
I was able to read any signs in Hungarian and would score in your lowest
ranges on the FAN in that language. However, being a, I hope, highly
literate person, I could figure out that certain signs pointed to exits
and entrances indicated danger in a subway \-- meant \"open\" or
\"closed\" indicated prices of things, et cetera. A functional
illiterate person cannot do that. Many of the international students I
taught at a large university have similar low levels of functioning in
their English classes. Yes, we are able to comprehend subject matter
rather well, a phenomenon well recognized in the literature of second
language acquisition and \[inaudible\]. But unless there is a
distinction made between English participants \-- between English
literacy and functional literacy, the information obtained of
non-English-speaking participants can be quite misleading.\"

OK, I think we have another question that\'s related to the same thing.

Andrew Kolstad: That isn\'t a question, that\'s more a statement.

Sheida White: That\'s right. \"The information obtained on
non-English-speaking participants can be quite misleading, unless there
is a distinction between English literacy and functional literacy.\" So
let\'s look at the other question, which also is \-- has to do with ESL
\-- \"As someone who works with native speakers of English reading at or
below the fourth-grade level, I think it is important to separate the
results of adults from native speakers. Often, adults who can\'t decode
words or have problems with reading, have learning disabilities\" \--
which is true \-- \"which have never been fully addressed.
Unfortunately, once people leave the K-12 system, there is limited, if
any, support services and/or testing available to them. Since this study
was used to influence policy, I would be interested in knowing the true
picture of illiteracy in the United States. In other words, how many
native speakers can read?\"

Andrew Kolstad: Yeah, that\'s been exceptionally difficult for our
testing, because our test is designed to measure the literacy skills of
those who can read and those who can\'t read well enough to access the
test. We don\'t really get answers to the questions fully.

Sheida White: Well, actually, that\'s not quite true, because we have so
many different tasks, as one of the callers \-- I think the caller said
\-- we have lesser identification, we have digit identification, we have
words that are tasks, we have sentence-level tasks, we have
\[inaudible\]. We have a variety of tasks that would allow us to explain
to policymakers and to the laymen, the general public, how many native
speakers cannot read words, cannot read smaller \--

Andrew Kolstad: \-- pieces of words \--

Sheida White: \-- chunks of words or letters and how many cannot read
connected text. So because of the ALSA and FAN, we are able to really
provide very detailed and distinct information about reading skills of
adults.

Now, what do we call reading, you know, that is, again, open to
interpretation. To me, reading is being able to decode and recognize
words not only to do that but to do that effortlessly so you can focus
on other things.

Andrew Kolstad: I was just going to add that I think this literacy study
does a better job of doing that than most testing programs that are in
the K-12 system.

Sheida White: Exactly, yeah.

Andrew Kolstad: The National Assessment of Educational Progress, for
example, can\'t tell how many fourth graders aren\'t able to read.

Sheida White: Right. So I think we are in a very unique position to
provide detailed information at the low end.

Oh, good, we have another phone call.

Andrew Kolstad: OK, we\'re ready to hear your question.

Sheida White: Or comment.

Andrew Kolstad: That is, if the system were working.

Unidentified Speaker: Is it green?

Andrew Kolstad: No \-- yes.

Sheida White: It\'s green.

Unidentified Speaker: Hello, caller, could you ask your question?

Andrew Kolstad: Can you hear us?

Unidentified Speaker: Hello, are you there? Is the caller still there?
Could you ask your question, sir?

Unidentified Speaker: Just speak loudly.

Caller: My question is \[inaudible\].

Sheida White: The \[inaudible\] questionnaire does have a question on
whether or not they have high school diplomas.

Caller: How about GED?

Sheida White: Yes, GED, don\'t they have that? I believe we have GED.

Andrew Kolstad: We do ask about GED.

Caller: \[inaudible\].

Sheida White: Yes. We ask those separately.

Caller: \[inaudible\]

Sheida White: What?

Andrew Kolstad: When will he get the data? When will the data be
released?

Sheida White: Oh, the data is expected to be released in June of 2005.

Andrew Kolstad: That\'s our current plan.

Sheida White: That\'s our current plan.

Andrew Kolstad: OK.

Sheida White: You\'re welcome.

Andrew Kolstad: OK, we have a question or two \-- a statement, really, a
comment on the definition of functional literacy.

Sheida White: Illiteracy or literacy?

Andrew Kolstad: Well, it says literacy. \"Functional literacy in the
U.S. includes reading, writing, and numeracy \[sp\] skills which adults
need in order to perform basic tasks in modern society; that is, to read
maps and directions, complete employment applications, read charts and
graphs, read and navigate Web pages, read bus and train schedules, read
basic health brochures, and so forth. This definition, of course, is
affected by how clearly, simply, and directly the materials are written.
One might argue that leases, for example, should be included in the
definition of functional literacy but legal documents and some
government documents are often written at an unnecessarily high level.
Although this is beyond the scope of the National Assessment of Adult
Literacy, perhaps in recording the results, the analysis might include
that part of the literacy problem is that Americans need to learn to
write in plain language.\" That\'s the first comment.

The second comment on another topic is the writer believes that we were
referring to the tests of applied literacy skills \-- TALS. If so, when
I tried to purchase it a year ago, I was informed that it is no longer
being published. So the author is reporting to us that the
ETS-originated test is no longer available. Sorry about that.

Sheida White: Let me refer to the first two parts of your question. The
average p value, that means the average difficulty of the NAAL main
tasks is something like 0.61%. That means, on average, 61% of
respondents can handle the NAAL tasks. So they are a little bit on the
easy side, overall. And in terms of the range, they vary anywhere
between 0.2 to 0.9% in terms of p value. So we really do not have tasks
that are like some of those government documents have that are very
complicated. Nobody can get. We don\'t include those in the assessment
tasks that only the top 2% of adults can handle. They are not included
in the assessment.

In terms of writing, you are correct in that we chose to define literacy
in terms of being able to read, to read recent text, and I have come
across many definitions of literacy that include writing and, in fact,
definitions also include being able to speak and listen effectively. But
because of budget considerations, we think we cannot include all
measures of literacy, such as writing, in the assessment. However, if
you are a secondary analyst, and you want to come up with some clever
ways of us being able to measure writing ability within the context of
our current assessment, let us know. We would like to hear that.

Andrew Kolstad: We have another question from a viewer in Ottawa,
Canada. This question is a follow-up on health literacy and how we\'re
measuring it. This writer says, \"I am unable to tell whether there is
anything distinctive about the scale, given that the definition concerns
generic literacy skills applied to health-related materials. Am I right
to understand that your assumption is that there is nothing distinctive
about health literacy practices?\"

Sheida White: Yes, that is true in the sense that they have to apply
their generic literacy skills. That\'s why I think it\'s so, so
important that we measure basic reading skills, because they come, you
know, they are everywhere. You need to have basic reading skills to be
able to do \-- to read health-related materials; to do all kinds of
things. So, yes, we are only measuring a core, a generic core reading
skills that we hope that it applies to understanding health-related
materials as well.

Andrew Kolstad: OK, and let me announce one more time the toll-free
number to ask questions \-- it\'s 888-366-5122. And now we\'ll go on
with the next section on the \--

Sheida White: \-- this is the section \--

Andrew Kolstad: \-- secondary analysis \--

Sheida White: \-- this, really, I developed this \-- Andy and I
developed this slide specifically for you viewers who are secondary
analysts for this very presentation. So this is for you.

OK, these are the \-- let me back up a little bit and give you a little
bit of a context in here. Every time I go to AERA, I am really struck by
the relatively few sessions on adult literacy \-- or any other
conference \-- very few sessions on adult literacy and also by the
relatively few number of people who attend those sessions on adult
literacy. And this really suggests to me that not many people are doing
research on adult literacy, and so I ask myself, \"What is it that NCES
can do to change that?\" And here are a few actions that we\'ve taken to
try to stimulate more research.

One area \-- and I\'m going to talk more in detail about each one of
these areas in terms of reporting \-- I added another bullet here \--
and that\'s released items. We have more released items in terms of
background variables, data variety, in terms of technical assistance,
and information exchange. So let me begin with reporting.

Now, to give access to technical specifications sooner \-- in 2003 the
technical report and the users\' manual will be released much sooner
than in 1992. This way you will have access to data much sooner. I do
need to say that actually the main report, the first main report, will
not be any speedier than in 1992, and that\'s because of the performance
level process that \--

Andrew Kolstad: \-- well, there are a whole series of things. First,
once they complete the data, they have to construct the weights, the
sampling weights.

Sheida White: Right.

Andrew Kolstad: And that takes a couple of months. And then, once they
have the sampling weights, they use them in doing the scaling. That\'s
how it creates \-- find out what the item characteristics are and create
the scale scores. Then they have to \-- we may even have to wait a
little bit for the reporting levels to be finished, because they need
that information for that. And then they have to create all the tables,
look at the results and write the reports, and then get the reports
checked for accuracy and consistency. So it\'s \-- last time, we
produced our report in about 13 months after the finish of data
collection. And what\'s the target this time?

Sheida White: This is going to be about maybe 15 months or something
like that, but it\'s not going to be any faster.

OK, there would be more items released and, Alex, when do you think
those would go off on the \[audio fade\]? Early in January \-- in a
month or so, we are going to put many more released items on our
website.

Our next slide, as I said earlier, we\'ve added many new variables to
the 2003 NAAL assessment in response to the trends in the past decade
including the democratic, technological, and economic. This way you will
have more material properly to work with. We are enhancing our technical
assistance to you, the researchers. The AM software \-- Andy will say
more in just a couple of minutes. He has quite a lot to say about that.
Actually, he has used that recently for some analysis and enjoyed doing
that.

> In terms of the NCES summer training institute, NCES has a history of
> providing summer training institutes for data analysts both at the
> national and state levels. These are people who, for example, work
> with NAEP, in particular. They\'ve had that opportunity. NCES will
> expand this program to include the NAAL, and this will provide data
> analysts with the training they need in how to analyze and use and
> interpret the NAAL data. We hope to also provide some seminars
> specifically designed for the research community as we get closer to
> the data release time.
>
> I guess, Andy, you can \--

Andrew Kolstad: Oh, OK, this is where it gets technical. This is where
it gets technical, so that\'s where I come in. Well, the basic problem
with a technical issue is that each respondent receives about a quarter
of the assessment items, maybe fewer than 10, on each of the scales \--
the three literacy scales. And such a small number of items is too few
to create a valid score for individuals. So that means when we\'re
computing average scores, we don\'t have individual scores to average.
So that\'s sort of a problem. How do we compute group averages without
having scores for the group \-- for the members that compose the group?

What we use is a specialized statistical approach called
\"marginal/maximum likelihood,\" and we use the item response theory as
the basis for that.

The idea of marginal/maximum likelihood is to look at the joint
probability of having the same test score underlie the responses to the
test questions and be a sample from the group from which the person is
taken. So we evaluate these joint probabilities across all possible
scores and choose the average group score that best fits the answer to
all the questions for all the members of the group.

Now, this method is incorporated in \-- or this basic idea is
incorporated all the different ways to process the data, whether in
international surveys of assessments in \-- national assessments for
children as well as our assessment for adults. The Educational Testing
Service was the innovator in developing this method and has been using
it for many years, both for the National Assessment of Educational
Progress and for the Adult Literacy Survey. More recently, American
Institutes for Research has been developing a general-purpose tool for
analyzing data from complex samples and data from surveys based on
assessments.

While ETS has also developed software, it has been proprietary, and ETS
has been, really, the only people who use that, although they provide a
form of the data that people can use without the proprietary software.

Let\'s get back to AM. AM software was originally developed to estimate
regression models for data like this \-- large-scale assessment data.
And rather than \-- well, I explained the principle already, but this AM
software has been supported by NCES, the National Center for Education
Statistics, and by AIR themselves, and they have been providing it free
to the public. So it\'s available to download at the website \-- easy to
remember \-- am.air.org. It\'s easy to access and fairly easy to use.

There are really two ways to process the data. One is to represent \--
well, for reach individual there is a probability distribution of
possible scores that could represent that person\'s performance, and the
problem is that many of these possible scores really work fairly well.
You don\'t distinguish the ones that fit well from the ones that don\'t,
given a small number of answers to questions. So the AM software can
represent each individual\'s proficiency as a probability distribution
over all possible scores and work with the whole distribution, or the
way ETS does it is to samplify \[sp\] values from this probability
distribution and treat them as if they were scores, and you can almost
treat them as if they were scores, but not quite. It\'s still a little
more complicated.

Both of these options have been implemented in AM and works pretty well.
So as input data, you can either use the possible values that are
available on the files, or you can work with the answers to all the
questions that each person gives and operate directly on those
questions. They provide the same answers no matter which way you go.

AM calculates standard errors, which are appropriate to complex samples.
By complex sample, we mean that we use stratification to choose certain
areas with known probabilities, like, for example, very small \-- areas
with low population density are chosen with a lower probability, and
then we compensate for them with weights, but that\'s \-- the density of
the population might be one of the stratification variables. Within each
of the strata, we choose clustered samples; that is, people who are
geographically located close together in order to save travel time of
having the interviewer go from one place to another.

> The downside of that, the reason it matters, is that people who live
> close together are more similar to one another than are people who
> live far apart, and that similarity provides a sort of a lack of
> independence between observations that complicates the statistical
> analysis, because most normal analysis assumes that cases are
> independent of one another.
>
> So what the AM software does, to take the complexity of the sample
> design into account, is to use either the Taylor \[sp\] Series
> approximation or a replication technique, such as balance repeated
> replication or the Jackknife. These are kind of technical issues, but
> I refer you to the Help Manual within AM software to understand
> what\'s done. With our adult literacy data set, you can use either
> approach \-- either the Taylor Series or the Jackknife, and you can
> use either the possible values representation of the scores or the
> answers to all the questions that people give.
>
> There are other possibilities. ETS, Educational Testing Service, has
> produced a NAEP data toolkit, which is suitable for this kind of data
> and might be adaptable to the Adult Literacy Survey, and they have
> also promised to release a public version of their proprietary
> software that does the marginal/maximum likelihood estimate. So this
> slide talks about how to get it from the NAEP website to get the
> component that ETS used to produce, which was an add-on to the SPSS
> software. So that may well be usable.
>
> And then I think it\'s back to you, Sheida. I\'ve done the technical
> part that\'s on what kind of improved information do we have on \--
> for exchanging data among users and reaching out to people who might
> be using our data. So that\'s where we are.

Sheida White: All right.

Andrew Kolstad: So what is it that NCES is going to be doing?

Sheida White: OK, well, for one thing \-- let\'s just go through this.

Andrew Kolstad: OK, go through the lists, right.

Sheida White: One bullet at a time \-- we hope to have more conferences
via videotape. We are all very busy and traveling can be very expensive,
so this may be a method of communicating.

Andrew Kolstad: Do you mean webcasts or videotapes?

Sheida White: No, just videotapes of conference presentations as well.
And so your comments on the evaluation form will be very, very helpful
to us as we plan to communicate to more people about the NAAL
assessment.

In terms of \-- Alex, do you want to say something about the list
server? Is there something that we have developed or are in the process
or something we are considering?

Andrew Kolstad: She\'s not on microphone.

Sheida White: Just tell me and \--

Alex: \[inaudible\]

Sheida White: Will be developed, and we will let you know about that. We
have \-- actually, we have a consultant whose job is specifically to
broaden the dissemination list and our outreach program. That\'s her
full-time job, which we didn\'t have before in 1992. So she is going to
give us a presentation next week on different ways that we are
broadening our dissemination and outreach, and I will be happy to share
that with you at a later time, and so forth. That\'s that. And you can
read the next two slides on your own.

Essentially, in a nutshell, we like to be able to communicate with
different constituencies on a more regular basis; to have a dialog and
to get feedback to let you know what we are doing, the work in progress,
and so forth. That\'s the idea of the new \-- let me \--

Andrew Kolstad: Are the secondary users going to be able to download the
data, as they do now, for the Adult Literacy Survey? Will we be
distributing it on CD? Do you know that yet? How will we be distributing
the new datafiles?

Sheida White: I don\'t know that yet.

Andrew Kolstad: OK, well, we can say that the rest of the National
Center for Education Statistics often distributes the data in both ways
\-- either by having it downloadable from the website or on CD. Right
now our \'92 data are available for download but not on CD.

Sheida White: Speaking of reports and materials that we have for
downloading, the next slide will show you the reports that we have from
the 1992 assessment. These reports are all available on our website. You
have the URL \-- nces.ed.gov/naal. There are limited hard copies, and I
get a lot of phone calls from people asking \-- or emails \-- asking for
hard copies of this and that. We have only limited hard copies of the
1992 publication.

Andrew Kolstad: By limits do you mean about three or four, right?

Sheida White: I don\'t know how many there are. There are a few at the
Ed Pub.

Andrew Kolstad: Oh, OK, some left for distribution.

Sheida White: Yes, some level, and the number there is 1-877-4ED-PUBS,
p-u-b-s. However, I need to let you know that there are no longer copies
of the technical report or the labor force report. But all of these
reports are also available in PDF and can be printed in their entirety,
and there are links that can be sent on our NAAL website.

Andrew Kolstad: We could mention also \"Literacy of Older Adults in
America,\" another report, and \"Literacy Behind Prison Walls\" \-- the
prison literacy report. Those are available as well as the ones you have
listed here.

Sheida White: Is that true, Alex?

Alex: \[off mike\]

Sheida White: That\'s right, those are selected reports.

Andrew Kolstad: Right, but they\'re also available.

Sheida White: The next slide tells you about the types of reports that
will be released. We will soon, before the assessment results are
released, we will be able to \-- maybe in less than a year, we will be
able to release our assessment frameworks. We are working on new
framework for the assessment because of all the new instruments that we
have, and the new features of the assessment that requires new
framework. We will have a framework on the background questionnaire.
There will be the regular main report, technical report, we would have a
report and a re-analysis of the 1992 data, we will have individual
states\' reports, a special report for the prison and health and
literacy and many, many, many issue-based \--

Andrew Kolstad: What about the reporting on the National Academy of
Science\'s setting of literacy \--

Sheida White: I think they are going to give us their report.

Andrew Kolstad: Will they publish it or will we publish it?

Sheida White: I don\'t know, but \--

Andrew Kolstad: But there will be a report.

Sheida White: There will be a report, but most likely we will be
publishing, but it would certainly not go through our adjudication
system.

OK, here is a timeline, and I am told to make several announcements, and
please do not leave without hearing these announcements. Again, the NIFL
staff, this staff, has asked me to make sure that you are aware of the
webcast evaluation form, which can be accessed through the NIFL gateway
page. So this webcast \--

Andrew Kolstad: That\'s where they already are if they\'re watching
this.

Sheida White: That\'s right \-- gateway page, and that is
[www.nifl.gov](http://www.nifl.gov/) \-- n-i-f-l. gov, and I would
appreciate if you address your evaluation emails to me, and that\'s
sheida white \-- <s-h-e-i-d-a.white@ed.gov>. That\'s one announcement.
And, of course, we have time to take more questions. Let me go through
the other \-- oh, the phone number is 888-366-5122. Also, I was asked to
let you know that there is a link to the PowerPoint presentation on the
NIFL website. We disagree on whether or not they can \[inaudible\] this
material. It is up to their discretion, I guess. Also, the third
announcement is that the entire webcast will be online on the NIFL site
as well.

Those are my announcements.

Andrew Kolstad: We have a question here from a user, and the question is
\"Do some of the respondents in your study possess a high school diploma
or GED? Do you have any idea what that percentage might be?\" And the
answer is, well, we will certainly find out once we know what the
weights are, but until we\'ve completed the survey, we don\'t know what
that percentage will be, but we certainly will be able to answer that
question in the future.

Sheida White: That\'s right.

Andrew Kolstad: And we could, in the past. I just don\'t have those
figures with me.

OK, we\'re ready for more questions, but we\'re sort of running out of
questions. What do we do if we finish early? I think we \--

Sheida White: Again, the number is 888-366-5122. Let\'s just wait a
couple of more minutes to see if \--

Andrew Kolstad: If we have any more questions or comments from our
users.

Alex: Do you have pictures?

Andrew Kolstad: Pictures of our grandchildren? No, we\'ll skip that
part.

Well, I think we should thank the listeners for tuning in and thank the
National Institute for Literacy for sponsoring this webcast and
providing this information to the literacy community, because the
National Assessment of Adult Literacy will be a very valuable resource
for the next decade in the literacy field.

Sheida White: That\'s right, and we hope that you found this form of
presentation helpful.

Andrew Kolstad: So turn in your evaluation form.

Sheida White: Turn in your evaluation and let us know if you\'d like us
to have more of this in the future as we move forward with the
assessment. There are always new things coming up.

Andrew Kolstad: OK.

\[end of recording\]