**August 2004 NAAL**

Sheida White: We appreciate your interest in the Health Literacy
Component of NAAL and we're pleased to welcome you. I also want to
express my appreciation to NIFL for giving us this opportunity again. My
name is Sheida White and I am the Project Officer for the National
Assessment of Adult Literacy, also known as NAAL. And I have my
colleague, Andrew Kolstad here. Andy is the Senior Technical Advisor to
NAAL and he is the former Project Officer for the 1992 assessment. In
addition to us, we have multiple contractors who have helped us develop
this health literacy components for the NAAL. We have Westadt, who
designed the sampling and conducted the data collection. We have the
American Institute for Research for developed the items, the assessment
items, the background questions, and they are currently doing all the
analyses as we speak. And in addition to the contractors, we have
multiple advisory panels and we have many, many consultants who help us
on a wide variety of issues.

I'm sure that - - yes, sorry, you go ahead and interrupt.

Andrew Kolstad: I was just going to say - - some of the important groups
that work with us are also the Education and Statistics Services
Institute that's been working on our website and the National Academy of
Sciences that are working on developing new reporting categories for us.

Sheida White: Exactly, thank you. Now I'm sure that there are many
health literacy professionals and experts along with the viewers today
and we would like to learn from you as well. So please do call us, send
us e-mail. And we are going to stop at the end of major sections of this
presentation to give you an opportunity to ask questions and of course
we can have an extended Q&A session at the end. Now at the end of the
Q&A section, we would like you to go on line and answer some questions
for us. We would like you to tell us three things that you liked, really
liked about this presentation and one thing that you think we ought to
change next time. Or if there is two things that you'd like to see
changed for the next time. You can use the form that is on line, or if
you like, you can call me. My name again is
[Sheida.white@ed.gov]{.underline} or you can call \[inaudible\] at NIFL,
which by the way it is \[inaudible\] [JVEHROODI@nifl.gov]{.underline}.
We take your input very seriously and want to do more of these web casts
in the future, so your input would be very helpful to us. So I think we
are ready. You may want to follow along using - - the slides were made
available a few days ago - - you could use your hard copies or use your
computer monitor and follow along with us.

> Here are the topics that are going to be covered today. HLC stands for
> the Health Literacy Component of NAAL. We're going to tell you a
> little bit about its background, some definitions and limitations.
> We're going to talk about task types. We use the terms task and item
> interchangeably. And we're going to talk about some skill
> requirements. We're going to talk about stimulus materials and
> background questions and Andy is going to talk about the analysis that
> we have found and actually is being conducted right now. Then we're
> going to summarize the unique features of this assessment and how it
> could be used by professionals, by practitioners, by researchers, by
> policy makers, and so forth.
>
> Okay, we are going to start with the background definitions and
> limitations. The importance of health literacy. Research indicates you
> can - - you can follow along with yours.

Andrew Kolstad: Okay, I was looking at yours.

Sheida White: Use your own. Research indicates that adults with limited
health literacy report poorer health status and are less likely to use
preventive services. At the same time, we live in a society with a
growing focus on prevention. There are more tests, there are more
procedures and there are more self care requirements. Several studies
suggest that inadequate health literacy is related to higher healthcare
costs, approximately \$1,000 more a year. I realize that there is no
robust measures for the effect of limited literacy on health
expenditures, but these studies, I believe, underscore the importance of
addressing limited literacy from a financial perspective.

> NAAL is the first large scale national assessment in the United States
> to have a health literacy component, not bad. I wanted to say a few
> words about NAAL just to give you a broader context here. NAAL is a
> project of the National Center for Education Statistics, which is one
> of three centers within the Institute of Education Statistics, also
> known as IES.

Andrew Kolstad: Education Sciences.

Sheida White: I'm sorry.

Andrew Kolstad: Institute for Education Sciences.

SW Institute for Education Sciences, not Statistics, yes. And IES, in
turn, is part of the U.S. Department of Education. It used to be
statistics, now it's the National Institute \[inaudible\]. All right,
NAAL is the first assessment of the nation's progress since 1992. We
have prose, we have document and quantitative literacy and we even have
a special score on health literacy. Here are some key features of the
National Assessment of Adult Literacy. There are approximately 18,000
participants in the assessment. We have 6 states participating in the
assessment. The emphasis is on printed everyday materials and we're
going to collect data on a variety of background characteristics.

> Partnership between IES and HHS. We developed the health literacy
> component to a response to a request by the U.S. Dept. of Health &
> Human Services, also known as HHS. They wanted to have a measure and a
> baseline for their Healthy People 2002, which is turn has an objective
> of improving the health literacy of adults with inadequate and
> marginal literacy skills. We worked with them very, very closely in
> developing the health literacy and stimulus materials and the items
> themselves.

Andrew Kolstad: Sheida, before you go on, you might mention the office,
because HHS is a very large organization.

Sheida White: Yes, I'm sorry. It's on the slide. I'm talking about the
Office of Disease Prevention and Health Promotion.

Andrew Kolstad: And they're responsible for the initiative - - the
Healthy People 2010 initiative, right?

Sheida White: Yes, exactly. Now here is the definition of health
literacy by Healthy People 2010. The degree to which adults have the
capacity to obtain, process and understand basic health information and
services needed to make appropriate health decisions. Now HHS recognizes
the importance of improving the health literacy skills of adults as well
as the way information is communicated to them. Here is our definition
of health literacy which focuses on only one key aspect of health
literacy, and that's the ability to use printed and written information
associated with a broad range of health related tasks in order to
accomplish one's goal at home, in the workplace, and in the community.
Here is a graphic representation which was just completed last week, I
hope you like it. The graphic shows that like other literacy tasks, the
health related tasks relates to accomplishing goals at home, in the
workplace, or in the school. If you are over age 16 you can still be in
school. And in the community which we include healthcare settings in the
community. Now depending on the type of healthy specific task that is
required, each health related task can be classified as either clinical,
preventive or navigation and I'll discus this in greater detail later
on. And depending on the kind of stimulus material, each health related
task can also be classed in three categories that apply to all of NAAL
tasks. These are prose, document and quantitative. So for example a
document task may also serve as a clinical task. So for example, knowing
how much medicine to give to a child based on a drug label, that would
be using a clinical task and a document task.

Andrew Kolstad: But there's no necessary relationship or one to one
between clinical, navigation and preventive, between those three and the
prose, document and quantitative, right? Those are completely
independent.

Sheida White: I wouldn't say completely They are different but related
and we're going to get into that later. Okay, here is how the Health
Literacy data may be used by a variety of audiences. The data can assist
in identifying the health literacy skills of target audiences. It may
guide the development of health information so that it doesn't exceed
the literacy skills of audiences. We feel like when the healthcare
materials are written at the levels that exceed their reading skills,
patients may fail to take medications as directed because they don't
understand the instructions.

> Now I'd like to underscore the limitations of the NAAL health measures
> because I think it's important. One limitation is the inability of
> NAAL to determine skills that are associated with listening and
> speaking. Now of course all comprehension of what a doctor says or
> what a nurse says or what a pharmacist says, being able to understand
> their communication and why it is important and what to do about the
> problem - - these are all very, very important. However, we are not
> measuring those things. We are only measuring written and printed
> comprehension of this aspect. And the reason is that it's very
> expensive and it's very difficult to measure.

Andrew Kolstad: So if it doesn't have anything printed or a graphical
material that's on the page, then it's not really literacy from our
point of view.

Sheida White: And even if it is graphical, it has to be associated with
words, not just pictures. It would have no relationship to NAAL. Really,
the three - - even a very highly educated person like yourself, you may
not be able to understand a report, a lab report - -

Andrew Kolstad: Right, with a specialized vocabulary and - -

Sheida White: It's a specialized vocabulary, so independent of education
and intelligence, we try to make sure that our assessment materials are
free of jargon and technical terms because that's not what we are after.
So in short, what the NAAL health literacy scores do measure is the
ability of individuals to perform literacy tasks when the task does not
require listening and speaking and when the material does not require
special knowledge of the content or lexicon. So I have finished this
section. Should we see if there are any questions up to now or shall I
move forward?

Andrew Kolstad: I think this would be a good time to have some questions
if - -

Sheida White: Is it possible?

Andrew Kolstad: If any have come in. Maybe the thing to do is call for
questions and then proceed with your talk a little bit and then if some
questions come in we can interrupt and sort of cover the reactions to
those. We need to give people time to get their thoughts together and
transmit them to us.

Sheida White: That's right. Okay. Now, I'd like to talk about the task
types and the skill requirements. This is a little complex slide in
here. The first rule is clinical tasks. One example that I've shown here
in the slide is taking medication as prescribed, that's a clinical task.
Other examples of clinical tasks would include, for example, following
hospital instructions, discharge instructions, or reading about a
disease on the Internet. Now this one is very tricky because if you get
on the Internet and want to read let's say about diabetes and if you
were just recently diagnosed with diabetes, that would be considered a
clinical task.

Andrew Kolstad: Oh, right, if you had the disease.

Sheida White: If you had the disease. Now if you went on the Internet
and wanted to know how to prevent diabetes by doing the right things in
terms of diet and exercise and so forth, that would be in the preventive
category.

Andrew Kolstad: Right, okay.

Sheida White: So the same task could be different depending on the
purpose in which you're trying to achieve.

Andrew Kolstad: Right, okay, I understand.

Sheida White: The second rule is prevention. Now the category of
prevention includes both disease prevention and healthy habits like
being able to read nutrition labels - -

Andrew Kolstad: Wearing your safety belt?

Sheida White: Yeah, I guess that would be. Now the third category,
navigation, an example of that would be understanding eligibility for
and benefits of health insurance plans. This category to me it relates
more to bureaucratic demands of the healthcare system. So another
example of navigation task would be being able to understand, for
example, that the third floor labeled nephrology is the correct
destination. Do you know what nephrology means?

Andrew Kolstad: Kidney illnesses, right?

Sheida White: Yes, good. Okay. So \[inaudible\] that all of this
activity is focused only on the written and printed text and not on the
oral dialogue with healthcare providers. So briefly I'm going to go over
the definitions of these three categories. Clinical tasks require - -
these are tasks that require the ability to use printed and written
information associated with interactions between the healthcare provider
and the patient. Clinical encounter is diagnosis and treatment of
illness and medication. The preventive tasks require the ability to use
printed and written information associated with maintaining and
improving health, preventing disease, intervening early in emerging
health problems, and engaging in self care and healthy behaviors.

The next slide, navigation task, again, these require the ability to use
printed and written information associated with the way the healthcare
system works, an individual's rights and responsibilities. Okay. In a
nutshell, next slide, clinical tasks have to do with the healthcare
itself.

Andrew Kolstad: Right.. So when you have an illness or an injury, what
do you do about it, right?

Sheida White: Yes, it's the healthcare itself. Preventive tasks have to
do with healthy habits and prevention of illness. You can think of it
that way, okay?

Andrew Kolstad: Yes, getting your shots and eating well and all that
stuff.

Sheida White: That's right, getting a mammogram. Not for you but - -
navigation tasks have to do with bureaucratic demands to get to the
healthcare. Okay? Okay, here is a breakdown of the health related tasks
that we have. We have 28 health related tasks. As you can see here, the
NAAL tasks in this domain vary in type although they are rather scant on
clinical tasks. However, I think it is important to bear several things
in mind and that's the subject of the next slide. The reason for
preponderance of preventive and navigation tasks. Most clinical
materials and tasks are targeted towards a specific disease or
condition. Therefore, they tend to include specialized vocabulary that
is not consistent with the NAAL framework. We included clinical tasks
that are generally more applicable, like reading a drug label.

Andrew Kolstad: So if you didn't have a particular illness or injury,
then you might not be interested in knowing what to do about it.

Sheida White: That's right. Rule two, by definition, preventive tasks
apply to all people in a population or at least a wide range of people
in a population. Similarly, navigation materials, such as medical
insurance forms, also apply to a wide range of people in the population.
Clinical tasks, on the other hand, relate to the healthcare itself.
Because these tasks generally relate to the specifics, as you said, to
the specifics of an individual patient's treatment, they often do not
apply to as large a proportion of the population. This is another reason
why the non-health related tasks are more heavily weighted towards
navigations and preventive tasks. And finally, we're just going to have
a single scale.

Andrew Kolstad: Right, we don't have enough questions really to do three
scale.

Sheida White: We don't have enough questions to do three scales, so it
was more important to get enough questions in the categories. Now this
is sometimes very confusing to people when we talk about health literacy
tasks and we talk about health literacy skills. This is the relationship
between tasks and skills. The relationship is that in order to perform a
clinical, a preventive, or a navigation task that we discussed already,
one needs to bring to bear, to apply literacy skills. General literacy
skills now, okay? Now, we are not going to get into that, that's a whole
new subject. We are not going to get into the general literacy skills in
this web cast. But we are going to soon hopefully have the framework. We
are working on the framework and we are soon going to be able to release
the framework and discuss those skills in the next few months. Okay,
should I stop for more questions now?

Andrew Kolstad: Have some questions come in?

Sheida White: No questions?

Andrew Kolstad: Well, we invite you to submit your questions to us on
these health literacy issues. Or maybe in the end we could talk about
the literacy study in general if people have questions about that.

Sheida White: Yes, that's right. I'll just go on to the next section and
that is health related items and stimulus materials. I just hope that
the system works. Is the telephone system working? Okay. All right.
Health related items in the 2003 NAAL, I mentioned earlier that there
are 28 health related items in the assessment, this is out of 149 items
that we have altogether. Now let me say that these 28 health related
tasks are associated with all contexts and settings that are relevant to
NAAL. They are associated with home, they are associated with workplace,
they are associated with community, healthcare settings and so forth.
And they can function, these 28 health literacy tasks, they can also
function as a prose item, they can function as a document or they can
function as a quantitative task. And don't forget the quantitative tasks
because that's pretty important, too. Bullet two, the two health related
tasks that we carried over from the 1992 NAAL, we wanted to make sure
that they remained intact in the original order and in their original
context, whereas the new 26 items are sort of spread throughout the
assessment. We wanted to say something about the importance of
maintaining the context and the order - - we didn't pick and choose
items from the 6 blocks that we borrowed from the 1992, because they
have to be - -

Andrew Kolstad: Yes, we brought over as a group.

Sheida White: As a group, exactly. Okay. Bullet three is very important.
It's important to note that the 26 new tasks were created specifically,
I'm saying specifically, to measure health literacy. We didn't take
these items, we didn't take these whole items from - - items that we had
already developed for the 2003 task and they didn't come from items that
were already developed for the 1992 assessment. They were specifically
designed to measure health literacy.

Andrew Kolstad: So in consultation with the HHS people?

Sheida White: Exactly. We collaborated very, very closely with experts
and their experts, the HHS people and their own experts, to identify the
stimulus materials and design the tasks. Okay. The health related items
in the 1992 NAAL, we had 6 items out of 166 items in the 1992 assessment
that fit in the definition of our health literacy. This number was not
enough to provide a separate health literacy score, so of these six
items, we reused 2 in the 2003 assessment and we are going to release 4
of these. Now in addition to these 4 items, these 4 health related items
from the 1992, we are going to release 88 more items from 1992 that were
not reused in the 2003 assessment. I want to say this - - even if we
were to use all of the 6 items that we had, health related items from
the 1992, it still would not have been enough to establish trend.

Andrew Kolstad: A solid measure of change in health literacy from '92 to
2003, yes.

Sheida White: Exactly. This next slide is a graphic summary of the
health related items. Now you see the last row I reiterated what I said
earlier, we have 4 items that we are going to release and the reason
that we didn't use these 4 health literacy items is that they were not
part, as I said earlier, they were not part of this total package of
items that we have chosen to repeat in the 2003 assessment. So we wanted
to maintain the contextual integrity of the items so we couldn't use
them. All right, this is kind of exciting, the next slide. This is just
to give you a little taste of what you can expect in the next couple of
months or maybe less. Before you study the sample screen, let me say
that we are in the final stages of developing what we call a test
questions search tool. And we are going to put this on the NAAL website.
This tool will allow users to search for assessment items and stimulus
materials that meet various criteria. At the moment, the search tool
includes 92 items, all of which were used, as I said, in 1992 NAAL. Some
of these items were actually used in 1985 and were carried over.

Andrew Kolstad: In the young adult literacy assessment that was done in
1985.

Sheida White: And then they were incorporated in the other 1992 NAAL.
Let's see - - I guess in the future we're going to add more once this
assessment, the 2003 assessment is over, we're going to add more.

Andrew Kolstad: We're planning in general, - - who knows what till
happen in the next decade, but in general the plan is to replace about
half of the test questions in each survey and release the half that are
no longer being used. So we'll develop another set of items and release
the old ones.

Sheida White: Okay, and the test question search tool would allow us to
just bring those and incorporate them in the same tool that we have
developed. Now take a look at the sample screen, the same sample screen,
as you can see the tool allows you to search by the assessment year. The
first one you'll see it says year. It would allow you to search by task
type and that relates to the skills that we haven't talked about yet. It
allows you to search by whether or not they are prose, document or
quantitative by the literacy scale, by the stimulus material type,
whether it's \[inaudible\] or whatever. Now for each of these
categories, a pull down menu will list the available options. Now in
addition, they can type in a word or a phrase. So on this particular
sample screen, we may want to conduct a search, say health literacy and
be able to get to all the health literacy items.

Andrew Kolstad: All the health literacy items that have been released,
not the new ones.

Sheida White: Yes. Okay, let's look at the results of this search. We
assume that you've done the search - -

Andrew Kolstad: In this case there would be 4 items, right?

Sheida White: Yes, 4 items. But of course when we release the 2003 items
in a year, in less than a year, it would be a lot more. Okay. This
screen displays an item that you have searched for along with
information about that item. So you can also click on the links that
will allow you to see the correct answer up in the left hand corner or
the major task event like being able to locate information, text search.
And any related items that relates to the same stimulus materials. Now
information about the item itself - - directly above the item itself is
a description of the item. In this case, it says Almanac vitamins, this
is correct information from almanac, this is just a brief description.
Followed by the main literacy scale for the item. In this case it would
be a prose literacy scale. The first year the item was used, the type of
stimulus material and the item number. Below the question, the item
itself is displayed along with its corresponding stimulus material. Now
when using the website, you should be able to scroll down to view more
of this stimulus material, not just the scores, the presentation. Now
I'm going to show you another slide which will show the question more
closely.

Andrew Kolstad: So this is the part of the stimulus material that
relates to Vitamin E that's an excerpt from a larger piece, right?

Sheida White: Exactly. The question says copy three food sources named
in Almanac that contain Vitamin E. Now this example again is one of 6
health literacy items that were included in 1992. This is an example of
a prose task that functions as a preventive task because it requires the
ability to use printed material associated with maintaining and
improving health.

Andrew Kolstad: Now I just I might point out that this is contained in
an Almanac that was published in what? 1985? Or 1983. So the Almanac is
21 years old which is why we didn't want to keep using it, because it
was too old and worn out and dated.

Sheida White: Yes, we have another almanac.

Andrew Kolstad: It might have been good to keep the item around because
we needed health literacy items, but basically, it was just too old.

Sheida White: Yes, I see what you mean. Maybe the same thing applies to
Vitamin E. Did you know that? Did you read it? About Vitamin E?

Andrew Kolstad: Oh, that the material itself from the almanac might have
become dated? It may or may not be dated but the books were getting worn
out from use in several different surveys and we just couldn't keep
using them.

Sheida White: That's right. We made many, many copies, thousands of
copies. All right. The next slide has to do with the health related
stimulus materials. I like to say that these materials are authentic. In
other words, we didn't design them specifically for this assessment.

Andrew Kolstad: So they naturally occurred and other adults would have
to deal with these in their daily lives?

Sheida White: That's right. They represent and reflect the type of
materials that people find in their everyday environment. Is that a
question?

Andrew Kolstad: Oh, we have a question, but I don't quite understand it.

Sheida White: We have several questions.

Andrew Kolstad: Oh, here it is - - why does it include only two items
from the 1992 survey? And as I was just starting to explain, some of the
old items were sort of - - while they might have been good on their own,
they had other things that made them no longer usable and the main thing
was, in that case, was that the almanac was too old and the books were
wearing out and they couldn't be replaced because it's not in print. The
almanac company publishes their new almanac, they don't publish their
1983 almanac anymore.

Sheida White: Yes, but I think there was another reason as important and
that is there were 13 blocks of items in 1992. We took 6 of those items,
6 blocks of those items and it happened that in those 6 blocks of items
there were only 2 health related items. We couldn't take the other
health related items out of context because, as I said earlier, it's
important to maintain the context where the items appear. I think there
was another question that we missed before. Are we going to get this
in - - wow, lots of questions.

Andrew Kolstad: The problem is, it's hard for us to read.

Sheida White: Can we get it in paper?

Speaker: He's working on it.

Andrew Kolstad: Okay, well we'll deal with these questions in a little
while. They have been on a display on the screen in front of us and
they're not quite legible yet, so once they get the technical issues
worked out, we'll come back to answering these questions.

Sheida White: We'll get them in hard copy and we'll go over it like we
did last time. Okay, now let's talk about the health related background
questions that we have in the assessment. NAAL includes 10 health
related background questions out of a total of 118. They cover a whole
bunch of areas. Bullet two, these background questions would help us,
would allow us to report the degree to which say adults with various
linguistic, racial, economic and other demographics receive their health
information. For example from the Internet as opposed to television,
magazines, talking to people and other means. Bullet three - - actually
it's not bullet three, it's bullet two sub bullet three. So for example
here, we would be able to report the percentage of adults ages 50 and
older let's say who have colon cancer. I'm sorry, who had colon cancer
screening in 2003. This background information is from the year before
the assessment. The assessment was given in 2003, so the question they
are asking about is 2002 practices. As well as their sort of information
for this sort of preventive task for colon cancer screening.

> Let me also add that not all of these analyses will be reported on in
> the main comprehensive report which comes out in December of 2005.
> Some of this analysis we will do in house at NCES and we will publish
> as issue briefs. Now the next slide - - here are some samples - -
> they're actually not sample background questions, they are the
> background questions. They cover all the background questions that we
> have. They are sort of abbreviated in here to fit into the slides. The
> arrows that you see on this slide and on the next slide show, is
> intended to show, draw your attention to the health related background
> questions that were specifically designed for the 2003 NAAL. So you
> see there are, in fact the majority of the background questions were
> developed in 2003.
>
> I'm going to let you look through these questions yourself. The next
> slide, the continuation of this slide, let me say that these
> questions, these new questions that we've added to the assessment,
> will put us in a position to provide new information that were not
> available in 1992. Health status, as you all know, may be associated
> with many, many factors, not just literacy. We will call it
> performance on health literacy tasks with new health variables
> including lack of health insurance which are bullets or arrows one and
> two. Inability to use the Internet which is bullet 4. And that's these
> days, being able to use the Internet to get health information is one
> of the main sources that people use. Or simply be able to relate it to
> lack of information about the value of engaging in healthy habits like
> having regular mammograms and that's bullet 5.

(Cross talk)

Sheida White: Because you need to correct me sometimes.

Andrew Kolstad: Well, you finished your section on the background
questions, right?

Sheida White: And now it is your section.

Andrew Kolstad: But I wanted to respond to some of the questions that
people have submitted. The first question from Daniel Lee - - I can't
read the first part of it because it starts in mid-question. But it asks
something about the reliability of the health literacy scale and have we
assessed the reliability of it? Well, no, in fact we haven't yet because
we're still in the process of analyzing the survey results that came in.
We've just completed very recently the scaling of the questions with
respect to the prose, the document and the quantitative literacy scales.
But we have not yet started the analysis of the results with respect to
the health literacy scale.

Sheida White: But one thing that we did do at NCES was to look at the
inter-reliability between diets who were not familiar at all with the
health literacy component. We gave them definitions of clinical,
preventive and navigation and asked them to code all of the 28 items. We
had two diets coding those items and the inter-reliability for one diet
was 92% - - this is percent agreement - - and for the other it was 96.
And these are people who had never been involved in this project, so he
have at least - - we know that the definitions of health literacy and
its components that we are using are very clear and reliable.

Andrew Kolstad: Well, the question was what are the psychometric
attributes of the health literacy scale. And basically we don't know yet
because we haven't done the analysis. We're in the process of doing that
now. The second question from Daniel Lee of Chapel Hill was how should
we, how can users obtain a copy of the questionnaire. And I have to
break our response into two parts. The background questionnaire, I
think, will be available on the website.

Sheida White: It's already on the website.

Andrew Kolstad: It's on the website, so you can look at the background
questionnaire.

Sheida White: Yes, there are two versions of it on the web. One is the
CAPI version, computer assisted personal interview version, which is
adaptive questionnaire and then the other one is a regular, one of each.

Andrew Kolstad: So that would just be a printed version?

Sheida White: A printed version.

Andrew Kolstad: So that would show you the exact text of the background
questions. But often when people ask that question, what they really
mean is can they look at the test questions and use them for their own
purposes. But no, we don't put the test questions up on the web or make
them available to users because we are concerned about security of the
test questions. We intend to reuse half of these questions in the next
assessment of adult literacy and in order to do that, we want to keep
those questions from being published anywhere and becoming widely known.
So we don't make available copies of the test questions themselves.

Sheida White: Unless they're released.

Andrew Kolstad: Right. And those are the ones that we have used in the
past and don't plan to use again. And as Sheida said, there will
something like almost 90 questions from the earlier surveys that will
become available so people can see how they're used. But on the other
hand, even if you have the set of test questions, it sill doesn't
constitute a test that you can use of individuals. Because our test is
designed to measure how well a population does and we don't give the
whole test to anybody. So we have not what fatigue of taking the whole,
of all of the questions, would do to performance and so it can't really
be used for that.

Sheida White: It's not reliable at the individual level, it's only
reliable at the group level.

Andrew Kolstad: Right. So the next question is, "Why did you include
only two items from the 1992 assessment?"

Sheida White: We answered that already.

Andrew Kolstad: Okay, we did answer that, and also, a large part of the
population is engaged in managing diseases such as diabetes or asthma,
so his comment is that it seems that more items could have been included
under clinical. Well let me say, one of the constraints on adding more
test questions is we need to get about 1,500 to 2,000 persons answering
each test question in order to make them part of the scale. And to get
more people answering the questions costs more and at some point gets
beyond even NCES' government size budget. It's expensive to add new
questions. So we could either have taken out other questions to make
room for them or increase the sample size as a whole and neither one of
those were appetizing options. Let me ask a question though, Sheida. Did
we understand this distinction among these three categories at the point
when we were developing the items? The navigation and clinical and
stuff? Or is that a question that came after we were done?

Sheida White: No, I believe that we had the definitions at the time, and
that was a decision that was made to have enough items in the category
of prevention and navigation.

Andrew Kolstad: Okay, so we sacrificed some clinical on the sense of
priorities of what was most important to measure?

Sheida White: Yes.

Andrew Kolstad: Okay.

Sheida White: As I said, we worked very, very closely with the HHS staff
and their experts to develop this item.

Andrew Kolstad: Within the limits of the budget?

Sheida White: Yes.

Andrew Kolstad: Now, we have another question - - will the health
literacy responses be evaluated within the context of literacy? For
example, how will the responses of the marginally literate be
interpreted? Maybe we should answer that after we go through the
analysis section. Okay, we have - -Sheida developed some slides here to
give us an indication of how in general we expect to analyze the health
literacy information that will come out of these test questions. Now
since they were developed to specifically measure prose, document and
quantitative literacy, these items will be included in the prose
literacy scale and the document literacy scale and in the quantitative
literacy scale. But in addition, we will lump the health literacy items
together and provide sort of a composite health literacy scale, so the
same health item would show up in either the health literacy scale or
one of the three generic literacy scales.

Sheida White: We're hoping that we keep this scale separate. Most likely
the general NAAL - - the three literacy scales would be from zero - -
the scale would be from, probably - - if the scale is still going to be
from 0 - 500 - -

Andrew Kolstad: It would be from 0 - 500.

Sheida White: If it will continue to be from 0 - 500, then the health
literacy scale, we would want it to be maybe 0 - 300 so we don't want
people to confuse the two.

Andrew Kolstad: Yes, we haven't made those decisions yet, but that
sounds like a reasonable thing to do. So this next slide says that
there - - each literacy task is classified as prose, document and
quantitative and is included in those scale scores. Now I'm hoping that
everyone in the audience knows what a scale score is, but basically the
idea of a scale score is that a single thing explains performance in all
the health literacy tasks. That single thing is a scale that measures
how much health literacy people have and the more you have, the better
you do on each of the test questions. And the same thing applies to
prose, document and quantitative. So that's what we're going to do on
that. The single health literacy score will measure the performance on
health literacy tasks for the entire U.S. adult population, that is,
people 16 years old and older living in households or prisons but not
living in other places that are hard to find. We will establish a
baseline for future assessments of adult literacy in 2003 and 1992
doesn't really establish a baseline because there's too little
information there to prepare a generic scale. Two items would allow you
to classify people into probably 3 or 4 categories and that's it and it
wouldn't be very generalizable beyond the two items. So we can analyze
how people do on the 2 scales and sort of do a correlation across them,
but there is kind of a built in, because both PDQ scales and the health
literacy scales are composed of some of the same items, there will be a
built in correlation among them. So it's not an entirely independent
thing to try and show how they relate to one another.

Sheida White: That's why I said they are different but related.

Andrew Kolstad: Related, yes. We'll be showing how various population
groups perform on the different measures. I know in 1992, we saw some
racial ethnic differences among the subscales that the Asian adults
tended to do somewhat better on the quantitative scale than they did on
the prose scale. So there are some differences that show up among groups
in performance on one scale versus another. It may be that older adults
may perform better on the health literacy than younger adults since
they're closer to health problems than have to do more navigation. But
that's a prediction. Who know what will happen there. But that's the
kind of thing that we'll be doing.

Sheida White: I would have though just the opposite.

Andrew Kolstad: Well the general literacy tends to be a little lower
among the older adults, but it may be that on health, they do a little
better. We'll find that out, that will be kind of interesting. Now the
question was, in terms of how well people do on the literacy scale for
low performing adults. Now here is kind of a problem. In the 1992
survey, the scale didn't really go down quite low enough. There was kind
of a floor effect, and below a certain level of literacy we couldn't
really measure what people could do because we didn't have enough really
easy questions. And one of the things we did in the 2003 survey was to
develop a way of distinguishing those adults who didn't perform very
well and provide an alternative assessment for them of literacy
questions that were considerably easier than that given to the rest of
the population.

(Cross talk).

Andrew Kolstad: You hear about the adult literacy alternative
assessment.

Sheida White: But I think in this case we are going to correlate these
items with the FAN.

Andrew Kolstad: Well we can do with both - - oh, you're right, because
the health literacy components required a certain level of basic
literacy to understand, to respond to them at all, the people who were
below the floor on the test wouldn't be given the health literacy
component and therefore we can't really see how well they did for people
who are really not literate enough to be tested with the regular
assessment.

Sheida White: So the correlation would be with the oral / written.

Andrew Kolstad: And their alternative assessment for the below the floor
people didn't really contain health literacy questions, right?

Sheida White: Yeah, there is one.

Andrew Kolstad: There is one?

Sheida White: Yeah, drug labels.

Andrew Kolstad: Oh, a drug label? Good. Well that's nice to know. It's
not the whole scale but we can see how well we did on that one item.
Good. In fact, what we were just talking about gets at the first
question from Pat Campbell from Rochester. The question was, will the
health literacy responses be evaluated within the context of literacy?
For example, how will the responses of the marginally literate be
interpreted? Well now, the question is, which side of the margin do they
fall? Do they fall below the floor of the test that we can measure? And
in that case we can't say very much about them. Or are they above the
floor and perform well enough to be tested with the regular assessment?

Sheida White: We also said that we were going to report on adults who
score at certain percentiles. Let's say people who score at the 20^th^
percentile, low literate adults, we would be able to say whether or not
these people, for example, can read a drug label or not. And we can also
look at the relationship of the performance of the low performing adults
in relation to the performance levels, the standards, that are being
developed. We're going to say a little but more about that in a moment,
by the National Academy of Sciences. So we can say people who are at
level one - - let's say if we have 5 levels - - again, we can say people
who are in the general NAAL are at level one, they score between 150-175
on the health literacy scale. So there are many ways that we can relate,
we can evaluate the responses of the very low performing adults, both in
relationship to the main NAAL and in relationship to the kind of tasks
that they can do actually, and in relationship to a new instrument that
we have developed called fluency addition to NAAL, or FAN, which is
based on oral reading which intends to measure basic reading skills like
decoding and word recognition. So this is one of - - I think this is one
of the beauties of the 2003 assessment is that we have many, many more
options of looking at the performance of the low performing groups and
be able to interpret that in light of health literacy scores or other
scores.

Andrew Kolstad: Well Daniel Lee has thrown us another difficult
question - - how is the measure of health literacy used in our survey
different from other health literacy instruments such as the TOFLA and
LOREL? Are they compatible? One thing I can say is we're not going to
know very much about that because we haven't given the two different
tests to the same person and can't compare how well that person did on
one test versus the other. That's the normal way of equating tests. But
another thing is you can look at the test questions. I know some of
those questions in those indexes are fill in the blank or multiple
choice kinds of questions. And our questions are not multiple choice
questions. The respondent always has to volunteer an answer and can't
rely on options from a list to choose from.

Sheida White: Yeah, but apart from the type of answer, the format, you
can look at the question between the two assessments. We can look at
the - - this is something, now that you brought it up, this is something
that we may want to do, we may want to look at TOFLA, for example,
because we have a coding system developed for coding, for classifying
our own health literacy items. We can tell whether they are on one
dimension whether they are clinical or preventive or navigation. On
another dimension we can tell whether they are prose, whether they are
document or whether they are quantitative. And we can also classify
these health related items based on the kind of skills that they require
the person to bring to bear in order to perform those tasks. Which you
haven't seen here. That's why I said it's a whole new topic. So we can
look at the coding system that we have already developed for classifying
the NAAL tasks and scales and applying that coding scheme to any other
assessment of health literacy and see how they compare, just the
questions themselves.

Andrew Kolstad: Here comes another question from an Evenstark person,
Aminia. Will adolescents or adolescent parents, ages 16-24, be one of
the subgroups that we report on? Now I know that 16-24 tends to be one
of the groups that we report on. Whether we break that out separately by
parentage, I'm not sure that we would do that ourselves. It may be that
the data would have to - - the questioner might have to acquire the data
and do their analysis themselves. The data will be available to the
public for other analyses than what we publish, because we can't
possible analyze everything that people want to know about. It's just
too many different possibilities. So the National Center for Education
Statistics publishes all our data and makes it available for secondary
use. So we - - in fact, the better technology of another 10 years - - it
should be easier to analyze these data than it has been in the past.

I thought it might also be interesting to point out that one of the
things that we want to report on in terms of the analysis is some very
simple statistics. What is the percentage of the adult population that
answered these questions correctly? How many got it right? Because I
think the impression that was portrayed from the 1992 data was that
people performed rather poorly and didn't get very many of the questions
right. But that in fact is not the case. People, I think, would be
surprised to know just how large the percentage is of people answering
these questions correctly. So one of the things we're going to publish
is the percent who answer each question correctly and how it differed
from 1992.

Sheida White: So as I said earlier, in addition to reporting on task
performance, we will provide information on general literacy skills and
we have identified seven categories of general literacy skills. Text
search has been able to locate information that's in a document, that's
one of those. Now I want to say that the NAAL literacy scales, the NAAL
literacy scales, I'm not talking about the health component. The NAAL
literacy scales are general. They're general in that they are relevant
to all literacy tasks and all written materials. In health literacy,
people apply these general skills to materials that require knowledge of
health, everyday health vocabulary and issues, like Vitamin E or a
health insurance form. Now if we were to say if we were to create an
assessment on financial literacy, for example, the same general skills
would also apply to tasks and materials that require financial related
vocabulary, like interest rates or to document structures that are
relevant to finances like bank statements. So the skills themselves are
general, but they can be applied to different domains.

Andrew Kolstad: You know I found in the years after 1992, people used
literacy a lot. There was - - I had mentioned a financial literacy,
emotional literacy, kitchen literacy - - a variety of literacies that -
-

Sheida White: Well, as long as they have to do with the written text,
with the written and printed text, these literacy skills apply. Like
ability to read the words. You've got to read the words. You have to be
able to recognize the words, you know, whatever the other things you
have to do with it. All right. Now, we said earlier that NAAL and the
health literacy components are related yet they are different. This is
how they are related. They are related because of those general literacy
skills that we talked about. Now regardless of the topic of the stimulus
material, an adult must be able to perform basic arithmetic operations
on materials whatever. Whether it's finance, whether it's computer,
whether it's health related or what. So that's why the distribution of
health related literacy scores cannot be independent of the distribution
of general literacy scores. That's how the two scales are going to have
to be dependent, related.

Andrew Kolstad: Right.

Sheida White: So we can never say they are completely independent
scales. Now here is how they are different. Don't look at the question
until we get to the end of this, there's just one more slide.

Andrew Kolstad: Okay.

Sheida White: The two types of scores differ in that health related
tasks are associated with health related materials requiring knowledge
of health related vocabulary, abbreviations, and forms, like
abbreviations like knowing the difference between milligram and
microgram. It's a huge difference. Mg and Mcg. Those are important
things, knowing those abbreviations. Another way to look at the two
scales actually, is the NAAL, the three scales in NAAL, the prose,
document and quantitative - - they are more structurally based. They're
based on whether or not the material is continuous text in which case it
would be prose. Or whether or not the material is non-continuous text
which would be a document format. Now the way we look at the scale in
health literacy - - it's more driven by the content. So in that sense,
that's another way to look at the two scales, being one is more content
driven and one is more structurally driven.

> It is important to mention that even though the vocabulary of
> health - - there is everyday vocabulary related to health in our
> health related items, we made sure that we don't have jargons and
> technical terms. And there's a difference between using jargon - -
> what would be a jargon? Think of a highly - - term that would fall
> into a - -

Andrew Kolstad: I know that jargon about testing - - item and task
difficulty and task discrimination. They're technical terms from the
testing industry.

Sheida White: I'm sure you can think of many technical items that do not
appear in every day written material. But we don't think things like
diabetes or blood cholesterol - - these are health related items, but
they are also everyday health related items, so we included - - that's
what we define health related material to me. So I'm going to stop here
because I'm at the end of this set.

Andrew Kolstad: Well, we have a couple of questions that have come in.
These are kind of - - the first one is kind of general - - from Monica
Cabrera of Los Angeles, California who asks, does your sample include
limited English proficient participants or adult speakers of English as
a second language? And well, you have to talk about different degrees.
To the extent that they do speak English, we try to administer it in
English. For those people who speak Spanish, we hired bilingual
interviewers and were able to conduct the background questionnaire in
Spanish and the initial questions, the 7 easiest questions that began
the test question part, were the material they were to read was English
but the questions were asked orally by the interviewers in Spanish. So
they could be given the task in Spanish but they had to read something
in English. Now if they were not Spanish speakers, then we're kind of
out of luck. Or out of money shall we say. The census bureau has the
resources to provide speakers of many different languages, but we did
not.

Sheida White: Not completely out of - - -

Andrew Kolstad: Okay, what happened to people who were not Spanish
speakers but showed up in the survey.

Sheida White: Okay. In the past, in 1992, if they couldn't get past the
background questionnaire, they would not be included in the assessment.
Now, if they don't complete the background questionnaire, they can still
participate in the assessment.

Andrew Kolstad: Oh, all right.

Sheida White: So you could be a Chinese native speaker of Chinese and be
able to read better than you can speak, which tends to be the case. And
the background questionnaire you need to have spoken English ability to
be able to respond to the questions orally, all right? Now in the past,
if you couldn't pass that phase, you wouldn't get to the assessment in
order to display your knowledge of the written text. You would have been
screened out.

Andrew Kolstad: So we gave people the opportunity to do it written
without the interaction in English.

Sheida White: Without completing the background questions.

Andrew Kolstad: But - - what I was saying though was we didn't have the
resources to have bilingual interviewers in languages other than
Spanish.

Sheida White: We still don't.

Andrew Kolstad: We still don't, yeah. So some kinds of speakers of
English as a second language can be accommodated, others can't. It sort
of depends on how well they do in English.

Sheida White: And if they don't there's also that supplemental
assessment.

Andrew Kolstad: Well, the supplemental assessment is also Spanish
oriented, and so it's easy English, and it's discussed with them by a
Spanish speaking interviewer, but if they're not Spanish, then we don't
have interviewers in other languages. So I hope that takes care of it.
The next question come from James of the Pfizer Company and he asks
about corporate responsibility. What is corporate responsibility in
reference to health literacy, especially with recent targeted marketing
practices among the medical fields? I wish he had spoken this, explained
this to us orally because I'm not sure I entirely understand the
question.

Sheida White: I understand.

Andrew Kolstad: Oh, why don't you discuss it?

Sheida White: Yeah. In fact, if you don't mind, I'm going to move
forward with my slides, or maybe I eliminated it - - yes, the last part
is contribution of the health literacy components to written health
communication.

Andrew Kolstad: Oh, okay.

Sheida White: So I think that will be relevant to that. Anything else?

Andrew Kolstad: Well, so you're saying that what we have learned in
terms of being able to present information or the way we look at the
skills involved in understanding written health communications, can help
people in designing forms to fill out or drug labels or instructions or
things like that, is that what you're saying?

Sheida White: What I'm saying is that knowing the literacy level of the
various target audiences would allow you to match the literacy skill of
your audiences to the materials that you're developing so that your
materials do not exceed the level of your targeted audiences.

Andrew Kolstad: There's more to it than just the materials. You also
have to think about how the audience, how the reader - -

Sheida White: Let me get into the next few slides and then - -

Andrew Kolstad: Okay, another question has come up but we'll come back
to that. Go ahead.

Sheida White: Can I go ahead? Okay. Now let me summarize the unique
features of NAAL. The health literacy component of NAAL is based on
current data. It is based on 2003 data. It's not based on 1992 data. So
it's very current, that's a plus, it's a unique feature. It is based on
a nationally represented sample of 18,000, approximately 18,000 adults
randomly selected. The tasks are designed specifically to measure health
literacy. The scale is based on 28 health related tasks that very
tightly fit the definition of health literacy that we have. The
background questionnaire includes new health related questions in
addition to a few questions from 1992. We are going to correlate, as I
said earlier, we are going to correlate the health literacy score with
an instrument that we developed based on oral reading fluency, and let
me just say a little bit about that.

As I said earlier - - I keep saying as I said earlier - - as we said
earlier, one of the objectives of Healthy People 2010 is to improve the
health literacy of persons with inadequate or marginal literacy skills,
okay? So we have this new instrument called Fluency Addition to NAAL,
which is designed to focus on just this particular type of people who
don't have adequate literacy skills. So what we are going to do is we
are going to provide data on the percentage of people with inadequate or
marginal skills who can perform specific health tasks. This is based on
the FAN, the Fluency Assessment, as well as on the data, on the health
literacy performance of the entire U.S. population, if that makes any
sense.

> The last bullet, I'd like to say that we have commissioned the
> National Academy of Sciences to set new standards in an open public
> way. In 1992, as you know, I'm sure Andy will be happy to say more on
> this, level one in 1992 was very, very large and very diverse.
> Therefore, we didn't know the nature of deficit. We didn't know if
> people didn't perform well because they didn't have the basic reading
> skills, they couldn't recognize and identify words, or maybe they
> didn't have higher level reasoning skills. We didn't know because it
> was so large and so diverse. So the Academy of Sciences is going to
> create new levels for us and they will report these levels in January
> of 2005. We will re-analyze the 1992 data based on the new levels.
>
> Now it's important, I think, to mention that the new performance
> levels that the Academy is going to set is going to be for the entire
> assessment. They're not going to set performance levels for healthy
> literacy items, okay? However, we might be able to say, for example,
> that people, as I said earlier, people whose score is between 150 to
> 175 on the health scale, they may be classified as level one. And that
> would be - - the performance levels would be set separately for each
> of the scales. There would be performance levels for document and one
> for prose and one for quantitative. So we would be able to say people
> with score at a certain point on the health literacy scale would be
> able to perform at level one of document literacy on NAAL. So in other
> words - - you seem puzzled a little bit.

Andrew Kolstad: No, no, go ahead.

Sheida White: So in other words, we would relate health literacy to some
known standards, which is the NAAL standards. Okay, that's the end of
this section and I was going to get to - -

Andrew Kolstad: Let me respond to another user questions. We have a sort
of a complicated question here. Someone - - Winston Lawrence from New
York City, wants to return to an issue about the role of speaking skills
in health literacy. He said that you mentioned that one of the
limitations of our assessment is that we're not measuring oral and
listening skills. But Winston says if we consider those who are
described as marginally literate, many may be able to talk about aspects
of their health and they may be able to tell you quite a lot. So is the
health literacy assessment then really valid? And the question of
validity in the testing industry really comes down to valid for what
kind of inference? Now the Department of Health & Human Services defines
literacy as the degree to which individuals have the capacity to
process, to obtain and understand basic health information and services.
But it doesn't say contained in printed and written material, and that's
the part of the definition that the Department of Education is most
interested in - - the using of printed and written materials to
understand these things. And now if HHS were to commission their own
survey with their National Center for Health Statistics, they could
define literacy exactly the way they want, and they could put that oral
literacy component int. But here we have two different departments with
two different, slightly different ways of understanding what health
literacy is. And in the education Department we wanted to have this
literacy tied to printed and written materials. That's what makes
literacy to us is the use of printed and written materials. So the oral
part is really left out by us on purpose because - -

Sheida White: But it is still valid. To answer the caller, it is valid
for the purpose for which it was intended.

Andrew Kolstad: For the purpose of measuring the use of printed and
written materials in the health field. But not necessarily their ability
to understand, obtain and process basic health information in general.

Sheida White: In general, exactly. Okay. The next slide talks about the
key factors in recent health communications. The three key factors that
affect (cross talk)

Andrew Kolstad: We're going to pause for a moment due to technical
adjustments.

Sheida White: We are almost done. I only have two more slides.

Andrew Kolstad: Yes, we're going to be ahead of schedule here unless get
more questions.

Sheida White: I have only two more slides so you may want to start
thinking about questions.

Andrew Kolstad: They may put up another question here, they just erased
those. Ah, we do have another questions.

Sheida White: What is the timeframe for - -

Andrew Kolstad: They're releasing the conclusions. Of course, we needed
to know this - - well, we have an interesting question. A question which
no doubt most of our users have been asking themselves and me. I get a
lot of calls on my answering machine \-- what is the timeframe for
releasing the conclusions from the various parts of this study? I can
tell you where things stand now. We have finished collecting the data.
We have figured out what the sampling weights were. We have completed a
preliminary scoring and sent the data to the National Academy so that
they can set the new reporting standards. Then the National Academy will
give us their results in January and right after that, we will be
scrambling to process the data and develop the tables we need to write
our report. And when do we expect the first report?

Sheida White: July of '05.

Andrew Kolstad: July of '05 is when the first report comes out.

Sheida White: Let me just say one thing, that the delay in the
assessment was for good cause. One of the reasons why the results are
delayed is that the Academy, the National Academy of Sciences, is
developing performance levels for the assessment so that we don't have
to use the old levels which didn't seem to be quite accurate.

Andrew Kolstad: And also FAN.

Sheida White: Yes. And I was going to say. In addition to new
performance levels, we have developed two new instruments, assessment
instruments designed specifically to get at the skills of individuals at
the very low end among many other new features. So it's a trade off
there. So let me just complete my slides in here. The three key factors,
and I think this should related to the callers, earlier callers
question, the three factors that affect all health communications and
those are the literacy skills of the individual, the appropriateness of
the text of the written material to their literacy level material. And
then this is what you were going to say, and that is the task to be
performed using the materials. So the difficulty applies not only to the
language, which is exactly what you were saying - - the difficult
applies not only to the language and presentation of the material, but
also to the task itself. That is, what one is going to do with the
material.

Andrew Kolstad: Right. And in many cases, we can anticipate what kind of
thing the user, who has concern with health issues, is going to do with
the material and to the extent we can anticipate what they have to do
with it, we can design the presentation of the materials to make that
accomplishing that easier.

Sheida White: Exactly. So let's just to be - - let's make it very
concrete using an example. We're going to make this very concrete.
Something that is very difficult for me - - I'm speaking from personal
experience - - sometimes I read food labels and the food label talks
about the number of grams of fat per serving and then it tells you how
many grams of fat you need per day. But there's nothing difficult about
grams. I grew up in a country where that was the system. Grams I
understand very well and there's nothing difficult about fat and there's
nothing difficult about the presentation of the information. What is
difficult is what you have to do to make sense out of that in terms of
deciding whether or not you should be eating more of it or less of it
for yourself. You try that. It is a lot of mathematics that you have to
do in order to figure out whether or not that particular serving is
something that fits in your daily regimen or not. So that's why the task
is also a factor.

> So the health literacy component, we hope that this component will
> have a positive impact on the recent health communications by helping
> adult literacy practitioners address deficiencies in health literacy.
> Not just practitioners, this also applies to policy makers and so
> forth. And to provide input to help health communicators create more
> effective materials and tasks. So that concludes my slides.

Andrew Kolstad: Can you put the contact information up on the screen?

Sheida White: Yes, just leave that on the screen.

Andrew Kolstad: Yeah, so if people want to reach us they can either call
or e-mail. E-mail works better given how many meetings we have to
attend.

Sheida White: Yes, that's right.

Andrew Kolstad: All right. Have we gotten anymore questions or is this
pretty much it? I don't see any new questions, so I think unless people
submit one right now, I think we'll wind it up. Do you have any final
words?

Sheida White: No. Just one more time I want to thank NIFL for, again,
for giving us this opportunity to talk with you and we hope to be able
to do more of this in the future. So please remember to go online and
complete your evaluation form. This time we made the evaluation form
very, very simple. The last time it was a page long. This is just a
couple of questions. Just tell us the three things you liked about
this - -

Andrew Kolstad: And send those to me, and the ones you didn't like send
to Sheida, right?

Sheida White: Yeah. Just tell us three things you liked about this
presentation. Tell us one, two, okay three things you didn't like about
this presentation or you think we - - or if you could make a
constructive - -

Andrew Kolstad: I like the original ratio - - three things they like,
one thing they don't like. So if they have two things they don't like,
they should have six things that they like.

Sheida White: That's right. And let us use that information and input to
improve our presentations in the future since we would like to do more
of these web casts in the future.

Andrew Kolstad: Yes. Thank you.

Sheida White: Thank you.

\[end of recording\]