NATIONAL INSTITUTE FOR LITERACY

**ADVISORY BOARD MEETING**

**\* \* \***

**SUMMARY MINUTES**

**\* \* \***

**May 19-20, 2005**

**\* \* \***

The Advisory Board met at 8:43 a.m. in the Board Room of the National
Institute for Literacy, 1775 I Street, NW, Washington, D.C. M. Carmel
Borders, Board Chair, presided.

**[PRESENT]{.underline}:**

M. Carmel Borders Advisory Board Chair

Douglas W. Carnine, Ph.D. Advisory Board

Carol Gambill Advisory Board Secretary

William Hiller, Ph.D. Advisory Board

Phyllis Hunter Advisory Board

Jean Osborn Advisory Board

Richard K. Wagner, Ph.D. Advisory Board

Mark Yudof, LL.B. Advisory Board

Sandra L. Baxter, Ed.D. Institute's Interim Director

[ABSENT]{.underline}:

Juan R. Olivarez, Ph.D. Advisory Board Vice Chair

Blanca Enriquez, Ph.D. Advisory Board

**[ALSO PRESENT]{.underline}:**

Marty Dannenfelser, *Deputy Asst. Secretary for Policy & External
Affairs, Administration for*

*Children & Families, U.S. Department Of Health And Human Services*

Maria Flynn, *Administrator, Office of Policy Development & Research for
Employment &*

*Training Administration, U.S. Department Of Labor*

Cheryl Keenan, *Director, Division of Adult Education & Literacy, Office
of Vocational and Adult*

*Education, U.S. Department of Education*

G. Reid Lyon, Ph.D., *Chief, Child Development & Behavior Branch,
National Institute for Child*

*Health and Human Development, National Institutes of Health*

**[PRESENTERS]{.underline}:**

# Chris Schatschnieder, Ph.D., Associate Professor of Psychology, Florida State University

Jay Diskey, *Principal, Diskey and Associates*

Barbara Wauchope, Ph.D*., RMC Research Corp.*

Ron Perkins, *Consultant*

Brian Shea, Ph.D.*, KPMG*

Annette Zehler, Ph.D., *Senior Associate, Development Associates, Inc.*

Paul Hopstock, Ph.D., *Senior Associate, Development Associates, Inc.*

Elliott Medrich, Ph.D*., Director Policy Analysis & Development, MPR
Inc.*

**[THURSDAY, May 19, 2005]{.underline}**

**Approval of Minutes**

Chairperson Borders called for a motion to approve the minutes from the
previous meeting, and Ms. Osborn so moved. Dr. Hiller seconded, and the
motion carried unanimously.

**Board Chair's Report**

Chairperson Borders elected to relinquish the time allotted to
accommodate Dr. G. Reid Lyon.

## Presentation:

G. Reid Lyon, Ph.D., Chief, Child Development & Behavior Branch,
National Institute for Child Health and Human Development, National
Institutes of Health

Dr. Lyon acknowledged the new leadership at the Dept. of Education,
which he said recognizes the Institute as a "significant partner and cog
in the wheel" of educational development. Referring to the Institute's
Advisory Board, he also said that the department "clearly recognizes a
very substantial Board intellectually, experientially, and one that can
guide a lot of us."

He said President Bush, Education Secretary Margaret Spellings, and
Health and Human Services Secretary Leavitt have presented his agency
with four initiatives over the past six months. He said these endeavors
might be instructive to the Board.

The first endeavor, which Doug Carnine has led, is the new flexibility
in special education. Previously, 1 percent of students were exempt from
testing for grade-level standards. Now the standard is 2 percent. While
this lower bar might be perceived as "political caving" to cries from
Texas, Connecticut, Utah, and other states for more flexibility, he
said, the truth is that several months ago the Secretary asked education
leaders to examine whether it is "realistic to ask all kids to reach
grade-level standards, even with the best instruction." Upon reviewing
the literature, the researchers found that when schools implement the
best instruction possible, 2 percent of the children still do not reach
grade level. He said scientific argument, rather than pressure from
rebellious states, demanded the new flexibility.

At the request of the First Lady, leaders have developed the Helping
America's Youth Initiative, designed to guide youth who may read well
but are at risk because of "substance abuse, aggression, and violent
behavior," among other problems. Leaders have visited 12 states, looking
at programs that could serve as models. Many are evidence-based; others
are promising. In the fall, the Mrs. Bush will host a major conference
that will introduce a tool that communities can use to assess the needs
in their area. The tool will "superimpose resources on top of target
sites" and then help communities develop programs to address these
matters.

Lyon said the project that is close to the Institute is the Early
Childhood Initiative. At the behest of President Bush, Secretary
Spellings and Secretary Leavitt have asked leaders to survey what the
various agencies are doing and identify the gaps and overlaps in
investment and technical assistance. He said Doug Carnine and Jennifer
Tchantz have been working with him on this project, which he described
as "pretty far along." Dr. Lyon said the next step is to gather this
information and make sure the investments made are toward "targeted
outcomes."

Related to this project, he said, is the National Early Literacy Panel
Report. He was emphatic that the report "serve as the core for a
dissemination effort about what we know and what we don't know." He
suggested that the Institute begin to develop a plan to distribute the
report.

Dr. Lyon said that as an advisor to Mrs. Bush, he is uncomfortable
traveling around the country discussing the Striving Readers Initiative
for adolescent students because the scientifically based research about
this group of readers is "scattered" and "unclear." He said, "the
Institute could do us a great service by helping to collate, organize,
and bring the reports together, then let us know the quality of the
information."

He cited the importance of identifying effective Reading First and Early
Reading First sites. He said the Dept. of Education's Institute of
Education Sciences is doing some of the work but added that the findings
need to be available as soon as possible.

Dr. Lyon noted that the Commission on Reading Research is "almost ready
to go," that the National Reading Panel is in place, and that Sandra
Baxter and the Institute have been working with his agency to develop an
Early Reading Panel and a Second Language Panel. He said these panels
are scientifically valuable because "we run the data through a screen,
see what is good, what is not good, and how it converges."

He acknowledged the "really nice" publications the Institute is
developing under the Partnership for Reading program but wondered how
leaders could create a technical assistance framework that would better
distribute them to the parents and educators.

Lyon briefly touched on other areas he described as critical. He asked
whether the Advisory Board could help leaders assess the quality of math
instruction. He supported the Board's plan to move cautiously in
exploring the use of foundations. Finally, he said, leaders need to do a
better job of identifying good examples of Early Reading First and
Reading First programs.

# 

# 

# 

# 

# **[Board Discussion with Dr. G. Reid Lyon]{.underline}**

Sandra Baxter, in response to Mark Yudof, said the Institute has not
conducted surveys of teachers concerning their knowledge of
scientifically based reading research. She noted that such a project was
proposed in the first year of the Partnership for Reading program, but
surveys require clearance by the Office of Management and Budget and the
process is often time-consuming. She said such an effort would take six
to 12 months from development to response. She and Mr. Yudof noted that
a private group could conduct the survey faster if the Institute could
partner with a private entity that did not use federal funds.

Prompted by Mr. Yudof's concerns, Dr. Lyon discussed the Institute's
brand identity. Dr. Lyon said the Institute's visibility on publications
should be stronger. After Ms. Hunter stressed that she had made this
point at this Board's first meeting, Dr. Lyon replied that a "purely
bureaucratic bungle" among the Institute's federal partners impeded
progress in this area. Dr. Baxter said specific recommendations would
come later in the day from a consultant who has been working on this
issue.

Dr. Carnine then led a discussion on the implications of Dr. Lyon's
presentation. Dr. Carnine said it is critical that the literacy leaders
have the draft report of the Early Childhood Initiative, expected by the
end of the summer, so that "we can play a meaningful part" in the
initiative. Dr. Lyon, Dr. Carnine and Dr. Baxter discussed the editing
process and concluded that the leaders should send the report directly
through peer review to expedite matters.

Dr. Baxter noted that leaders have a plan to distribute the report but
must work on a method of explaining to the field how the information can
be used. Dr. Carnine suggested partnering with other organizations as a
means of achieving this goal. Dr. Hiller and Mr. Yudof emphasized the
importance of clarity in a field that bombards listeners and readers
with information. Mr. Yudof and Dr. Lyon suggested that the Institute
invite David Dunn of the Dept. of Education to the Advisory Board
meeting in Florida to discuss implementation at the state level.

In response to Mr. Yudof's question about the status of the Striving
Readers Initiative, Dr. Baxter said the Institute is working with the
Dept. of Education's Office of Vocational Education on an interagency
working group that is focusing on adolescent literacy. She said the
group is developing a synthesis of 20 to 25 pieces of literature on the
subject. She noted that the studies are not all scientifically based;
some are review articles.

Some Board members expressed concerns about the methodology used in
developing the report. Dr. Baxter said that Donna Alverman and Dorothy
Strickland have been the informal reviewers during the development,
which is being handled through a contract with George Washington
University. Tanya Shuy, the Institute's adolescent literacy specialist,
committed to providing a list of the university staff members working on
the project. Dr. Carnine and Dr. Lyon said the Board members should
review the timeline for this initiative. Ms. Osborn noted that while the
Carnegie report is reflected in the GW study, the Carnegie report does
not touch on vocabulary, which leaves a huge void in the study. Dr.
Carnine and Dr. Baxter agreed on the wisdom of expeditiously hiring a
contractor who would identify high-performing schools.

Dr. Carnine then turned the discussion to math, saying the Board members
need "to decide if we want to consider spending a small amount of money
on a document for math, scientifically based research that would grow
out of what Russ Whitehurst has put together." Carnine also asked
"whether we would want to bring together the people who already have
expertise in studying high-performance schools in the area of math to
see what data they have and what models they have identified," which
could be used in a math initiative.

Dr. Hiller, however, raised concerns about a "new pile of initiatives."
He stressed that Board members must not lose sight of the crosswalk they
have developed and suggested that new initiatives should fall into that
framework.

Board members then discussed the lack of math proficiency among
American-born students and the implications of such deficiencies for
academia and the workforce.

## Election of Board Members

Two offices were up for reelection: the Institute's Board Chair and
Board Secretary. Mr. Yudof nominated Ms. Borders for chairperson. Ms.
Osborn seconded. The motion passed unanimously by voice vote.
Chairperson Borders, in turn, nominated Ms. Gambill as secretary. Mr.
Yudof seconded, and that motion passed unanimously by voice vote.

**[Institute's Branding/Products]{.underline}**

Board members launched into an unscheduled discussion of branding. They
directed Dr. Baxter to inform the Institute's grantees that the Board
requires that the National Institute for Literacy's name be prominently
displayed on all products funded by the Institute, including websites.

Several Board members raved about the draft versions of the Partnership
for Reading products. They agreed that the Partnership name should
appear on the backside, with the Institute's name in large type on the
front, and suggested that the family-oriented material be sent to widely
read newspapers, magazines and civic organizations for Mother's Day and
Father's Day 2006. Dr. Baxter committed to sharing with the new
committee the distribution plan once the material is finalized.

## Interagency Management Group Members' Report

## 

## Cheryl Keenan

## U.S. Department of Education Update

Ms. Keenan appeared on behalf of Susan Sclafani to provide an update on
activities of the U.S. Dept. of Education's Division of Adult Education
and Literacy. She began by talking about one of her office's major
initiatives: Project STAR (Student Achievement in Adult Reading).

The director said her office has developed a reading toolkit based on
the work of the National Institute for Literacy. Taking the research
synthesis John Kruidenier conducted for the Reading Partnership, her
unit hired a team of experts led by Mary Beth Curtis. She said Curtis
"painstakingly identified the strongest findings" in Kruidenier's work
and developed a teacher strategy tool kit in an effort to convert
research into practice.

Ms. Keenan said the toolkit targets intermediate-level readers, many of
whom speak English as a second language. She said the researchers are
working from the Adult Reading Component Study profiles on the LINCS
site. She noted that the tool kit, vetted by Peggy McCardle and the
Reading Partnership, is getting "very good reviews."

The division director said researchers are piloting the tool kit in
California, Connecticut, Illinois, Maine, Ohio and South Dakota. In
addition to field-testing the tool kit with teachers, the planners are
engaging the states in "systemic reform in reading practices, including
the organization and development of professional development and the
structuring of programs and assessment practices.

Ms. Keenan said more than 120 teachers and 40 to 50 local programs are
participating in the study, which will run for about nine months. Along
with the toolkit, she said the researchers are creating a professional
development package that will explain how the toolkit can be used at the
policy level, the professional development level, and the teacher level.

Keenan updated the group on the content standards, which determine the
content of what adults need to know and be able to do to function
effectively as adults. She said the Dept. of Education is working with
14 states on a technical assistance project that will help them
implement the standards. One group is working broadly on ABE standards,
while another group is working broadly on ESL standards. Once the
standards are developed, the creators must align a means of assessing
those standards.

Ms. Keenan said the department has developed a "standards warehouse"
with a searchable database that will house all of the standards that
have been developed in other places. She said the department is nearly
ready to launch a live assessment warehouse.

She also discussed another endeavor: an interagency project designed to
help states form partnerships with "sister and brother agencies" at the
state level to improve the scope and quality of services in adult
education programs.

To put the three major projects in perspective, she said six states are
engaged in reading reform, 14 are involved in content standards and six
are working to leverage activities with other agencies.

She then talked about another initiative: the Center for Adult English
Language Acquisition, which is designed to address the growing ESL needs
in adult education. Since 43 percent of enrollees in adult education
programs are ESL learners, the center is working with 25 states to help
teachers get the training to work with this population. The center also
serves as a clearinghouse and resource center and does briefs and other
activities further explained in the packet she distributed.

Keenan touched on the Transition Project, a best practices study
designed to prepare students for postsecondary education in Florida,
Kentucky, Washington and Wisconsin. Four local programs operate in each
state.

In terms of mathematics, she said her office is working with the
Department of Education's Institute of Education Sciences on a two-year
project that would convene a panel of experts on numeracy, conduct a
review of literature on the subject and environmental scans and perhaps
fund the development of some interventions that might have some
potential for further study.

The Office of Vocational and Adult Education is developing an initiative
that seeks to improve teachers' proficiency in using technology in the
classroom. Another goal, she said, is to help states better use distance
learning in adult education.

In response to Ms. Hunter's questions regarding the reading toolkit, Ms.
Keenan said some states allowed programs to apply to participate in the
pilot. Others selected the pilots. The tool kit is being tested in high
schools as well as adult education programs. She said practitioners are
facing serious challenges in using the toolkit because it is forcing
programs to restructure the way in which students are placed in
multi-level classes.

**Maria Flynn**

**U.S. Department of Labor Update**

Ms. Flynn spoke on behalf of Labor Dept. Assistant Secretary Emily
DeRocco. Ms. Flynn is the administrator of the Office of Policy
Development and Research at Labor's Employment and Training
Administration.

Flynn began by noting that half of the Labor Department's \$6 billion
funding comes under the Workforce Investment Act, which is tied up in
Congress. Although the House of Representatives has passed its
reauthorization package that is close to the Administration's proposal,
the White House strongly opposes the Senate Health, Education, Labor and
Pensions Committee version. "We are unsure of how things are going to
move forward" on the Senate floor and in the conference committee that
will eventually reconcile any variations between the House and Senate
bills, she added.

She said her agency is proposing to change the way literacy services are
rendered. Right now, clients receive such services only if they are in
occupational training. Agency leaders want literacy to be a stand-alone
activity, regardless of whether clients are in occupational training.

Flynn outlined another proposed change. Under current law, local areas
must use 30 percent of their youth money on out-of-school youth, but the
Administration wants to devote all of the youth funds to that group. She
said her agency is concerned that of 100 African-Americans who enter
kindergarten, the number who graduate high school "is low." Ms. Hunter
pointed out that the number is 18. The graduation rate for Hispanics and
Natives Americans is also low, Ms. Flynn said. Ms. Keenan added that of
the 2.7 million students in adult education programs under her agency, 1
million of them are ages 16 to 24.

Ms. Flynn highlighted some planned grant opportunities. The \$125
million community-based job training initiative will award grants to
partnerships composed of community colleges, the public workforce
system, and employers. Four-year colleges and faith-based organizations
can be included in the partnership, she said. The money can be used for
training in high-growth occupations and for building the capacity of the
community colleges to train in those occupations.

The agency has released competitive announcements for a migrant and
seasonal farm worker program, known as the National Farm Worker Jobs
Program, which will award grants to entities that will run these
programs at the local level. This year, she said, labor leaders are
looking to award about \$50 million in grants that focus on high-growth
industries such as healthcare and computers.

A new area of planned funding is \$15 million in grants that would
prepare young offenders to enter high-growth and high-demand industries,
she said. The services will include reading and math remediation,
on-the-job training, the granting of industry-recognized credentials,
and linkages between community colleges and four-year institutions.
Another reentry effort will award grants to faith-based organizations
that work with young offenders.

The \$5 million Hispanic Worker Initiative that will award seven to 10
grants that focus on innovative ESL approaches to training workers in
high-growth industries. She said leaders regard this plan as a means of
addressing local, state and regional labor-market issues that stem from
increased immigration.

Ms. Flynn informed the group of some interagency collaboration. She said
the departments of Labor, Education, Health and Human Services, and the
Interior are developing common performance measures. She said Labor's
Employment and Training Administration would be implementing those
measures by July 1 of this year. She noted that a literacy and numeracy
gains measure for the youth population is being developed.

Flynn said leaders are struggling to define what role the Labor Dept.
should play in developing content standards. She noted that the National
Skills Standards Board that ran from the mid 1990s to 2001 "did not work
very well" and was scuttled. The federal government has been encouraging
industries to develop their own competency models, and, as a result, the
Retail Federation has adopted a set of customer-service skill standards
that other industries are using.

Ms. Flynn announced that the U.S. Dept. of Labor is hosting its annual
conference July 11-13 in Philadelphia. This year's session will be
called "Workforce Innovations" and will focus on alternative education
such as online learning and other non-traditional training models. She
said the four plenary sessions would feature Innovation Economy with
economist Michael Mandel of Business Week, Manufacturing and Workforce
Issues, Energy, and Education led by Secretary Margaret Spellings.

Prompted by Ms. Hunter's request, she said her staff would supply a list
of "dying jobs," information that BLS tracks. With the abundance of
travelers making their own arrangements, she cited travel agent as a
good example of a "dying job."

**Marty Dannenfelser**

**U.S. Department of Health and Human Services Update**

Mr. Dannenfelser, the Deputy Assistant Secretary for Policy and External
Affairs in the Administration for Children and Families at the U.S.
Dept. of Health and Human Services, briefed the group on his agency's
activities. He did the same for Windy Hill, the Associate Commissioner
of the Head Start Bureau, who was unable to attend.

Mr. Dannenfelser highlighted the efforts of the Good Start, Grow Smart
interagency workgroup. He said the group was organized in the summer of
2003 by Shannon Christian, the Child Care Bureau's Associate
Commissioner. The group has developed a tracking tool to identify and
link the federally funded research that is contributing to the work with
states and communities in implementing Good Start, Grow Smart.

He said the workgroup members and guests participated in a research
briefing April 28, and that a related report from a roundtable on early
learning guidelines would be available within a month on the Child Care
Bureau's web site,
[www.acf.hhs.programs/ccb](http://www.acf.hhs.programs/ccb) This report
highlights approaches, challenges and lessons shared by 10 states
implementing early learning guidelines across early childhood programs.

On March 20, Assistant Secretary Ray Simon of the Education Dept.'s
Office of Elementary and Secondary Education and Health and Human
Services Assistant Secretary Wade Horn testified before the Senate
Health, Education, Labor and Pensions Committee on the two agencies'
collaborative efforts on education and early childhood development. Mr.
Dannenfelser said both officials highlighted the work of the Good Start,
Grow Smart interagency workgroup in promoting the president's goals for
school readiness.

He said a design team has developed training modules that will be
delivered to regions and states. On-site training has been provided to
Illinois, Delaware, Ohio and Palm Beach County in Florida on
cross-sector professional development and strategic planning with
partners. Training has been provided to Maine on implementing early
learning guidelines and to Ohio on developing early learning guidelines
to infants and toddlers.

Dannenfelser said states are increasingly using quality-rating systems
as "mechanisms to align state policies to ensure the overall quality of
care to strengthen efforts to inform parents and others on the elements
of quality early care in education." Ten states have such systems, he
said, and 19 states are investigating or developing proposals to
implement them. More information is available at
[www.ncccic.org/populartopics/tieredquality](http://www.ncccic.org/populartopics/tieredquality)

Turning his attention to Head Start, Dannenfelser said the bureau has
developed the Head Start Management Initiative. The agency also has
focused on improving Head Start's monitoring, training and technical
assistance.

To assist in monitoring, the bureau has developed a fiscal checklist,
established a process to address executive-level compensation,
implemented a process to assess the performance of reviewers, and
established standards to improve the quality of monitoring reports. Last
year the bureau introduced a new system to improve training and
technical assistance.

He said Head Start's involvement with Good Start, Grow Smart has
included new steps to strengthen the Head Start program, particularly in
the area of the national reporting system on child outcomes, which
involve training nearly 50,000 Head Start teachers in early literacy
techniques, developing an electronic learning center and launching a
parent-mentor training program.

On the Congressional front, he said Head Start reauthorization "is
starting to see some movement in the House committee, and we are
expecting some action in the Senate Health Committee in the foreseeable
future." He noted that there were "some controversies" on the House
floor in the previous Congress, which narrowly reauthorized the program.
"We're not sure if some of those areas are going to be in contention
again this year," he said.

Ms. Osborn asked about the connection between the upcoming Head Start
report and the National Early Literacy Panel Report (NELP), which is
being produced by the National Institute for Literacy and its partners.
She raised concerns about the two reports causing confusion in a field
that she said is often characterized as disorganized. Dr. Baxter replied
that the two groups are working closely to avoid major discrepancies.
Dr. Hiller added that readers would accept different recommendations
from separate reports as long as they are acknowledged.

## Adjournment

The meeting adjourned for the day at 5:09 p.m.

**[NOTE]{.underline}: Day one presentations can be found in attached
Appendix.**

#### FRIDAY, May 20, 2005

**Strategic Planning and Quality Evaluation**

**Dr. William Hiller**

Dr. Hiller opened the strategic planning and quality evaluation section
of the meeting by commenting that the National Institute for Literacy
had made great progress toward linking these two distinct concepts
through constantly thinking about methodologies and ideas. He also
emphasized the importance of keeping track of projects, looking at the
gap analyses, and understanding past discussions.

Board members viewed a chart Dr. Hiller referred to during his
presentation. He identified the national research agenda as the first
column of the chart on which the Board needed to maintain a strong
handle. He next identified the research agenda as it relates to early
childhood and adolescence, then adult through the lifespan, since
attention to the lifespan is a critical part of attention to the mission
of the Institute based on the lifespan mission. He then stated that the
research agenda is tied to the crosswalk, and emphasized the importance
of using the crosswalk as a reference tool. In short, he said that the
research agenda is tied to the lifespan, which in turn is tied to the
crosswalk and the goals established by the Board. He then commented on
the importance of channeling through this system so as not to lose sight
of certain goals in favor of others for which there is perhaps greater
pressure for movement or greater preference on the part of the staff.

The next strategic item Dr. Hiller discussed was tying the interagency
work with the research agenda, lifespan, crosswalk, and goals. He said
that, although the Interagency Group is separate from the organization's
own program committee, the Interagency Group has a legal obligation to
make suggestions to the National Institute for Literacy, which, however,
do not necessarily equate to program goals or the crosswalk.

The next line of thinking Dr. Hiller addressed was product
dissemination, including decisions on what to disseminate and which
products to give customers, such as training products, presentation
products, and executive summary products.

He then turned the proceedings over to Dr. Baxter, who reviewed a chart
that outlines various research activities, in particular those that are
relevant to Goal 1 in the crosswalk, which is to support rigorous
research on literacy acquisition. The chief areas in this section are
early childhood, K through 5, adolescent literacy, and adult literacy.
She explained that the purpose of this mapping is to allow the Board to
see which interagency recommendations were being fulfilled through this
work, to identify stakeholder recommendations and to examine the roles
the Institute would play in each of these areas.

Dr. Baxter then identified the functions being fulfilled through this
work in the area of early literacy as follows: coordinating across
agency offices, convening, and communicating both to internal and
external audiences. She then said that the goals were addressed through
a general-purpose statement along with specific research questions. She
said that these research questions would be important to the next step
in the analysis \-- looking at these questions in terms of the
information gathered as well as the information already available, in
order to begin to consider the gaps and to build a future research
calendar, and, eventually, anticipated products.

Dr. Baxter commented that the interagency recommendations for the
Institute were less specific with regard to the Interagency Group's
expectations for the Institute than they were with regard to the work
that the Interagency Group is expected to take on. This is favorable for
the Institute because the Board remains free to decide how best to
contribute. She further commented that the way in which the Institute
contributes depends upon the Institute's authorized activities as
outlined in the Workforce Investment Act and No Child Left Behind. She
then noted that Good Start, Grow Smart is an exception, but that it is
important to include it because of the coordinated interagency emphasis
on early childhood products. Furthermore, this group will be essential
in extracting the core messages from the report, and in developing,
coordinating and disseminating these messages.

Dr. Baxter next said that the Institute is involved in the Early Reading
First national evaluation, which invited the Board to look at the
relevant list of goals. She also noted that the money the Institute had
contributed to that evaluation had allowed for an extension of the
design to include identification of specific sites that have effective
programs, defining "effectiveness" in terms of specific student
outcomes, as defined by the Institute for Education Sciences. She then
said that the results of the Early Reading First national evaluation
would be the foundation of the dissemination work in terms of the
Institute's mandate in No Child Left Behind to disseminate information
about effective early reading first programs.

Dr. Baxter then moved on to the efforts in K through 12. She said that
the types of topics the Commission on Reading Research panelists would
be looking at and the types of methodologies to be reflected in the
studies they reviewed would be determined in large part by the
Commission on Reading Research. She said that the Institute's one
expectation of the Commission was that it update the research in the
areas of the National Reading Panel Report, but that the Commission
would otherwise be autonomous in deciding where it wishes to conduct
further inquiry and synthesis.

Ms. Osborn asked about the "anticipated products" found in the next
column, to which Dr. Baxter replied that it had been placed in the wrong
column and should be placed instead under "interagency recommendation."
Dr. Baxter also noted that an effort would be made to "go in and fill in
these products for each of these things," but that the leaders were in a
wait position, and that it was expected that the Commission would take
two years, once it has convened, to actually conduct the review and
issue a report. She said, however, that it might be possible to receive
an interim product. Dr. Baxter also said that the report of the National
Literacy Panel on Language Minority Children and Youth has been finished
and is in review by IES. Once the report has passed the review process,
its dissemination will, in part, be carried out by the Institute. Ms.
Hunter expressed a desire that this report carry the National Institute
for Literacy's logo. Dr. Baxter replied that it is largely an IES
project, but that, depending on the role that the Institute plays in
dissemination, there may be some conversation with regard to the
inclusion of the Institute's logo.

With regard to adolescent literacy, Dr. Baxter noted again that the
anticipated products should be in the interagency recommendation column.
She then explained that the intended products are a summary document
which will be published formally as a report, along with a number of
practitioner friendly documents based on that summary that will be
developed for educators, parents, and others interested in the issue.

With regard to adult literacy, Dr. Baxter began by explaining that she
had not included in the chart all of the activities that had been
approved by the White House Office of Management and Budget for the 2004
spending plan. Instead, what is reflected is the \$2 million dollars
that the Institute provided to the National Institute for Child Health
and Human Development (NICHD) for the Adult Reading Research Network,
the jointly sponsored national research program between NICHD, the U.S.
Dept. of Education, and the Institute. Reflected also are the
Institute's adult reading components interactive website and the Adult
Literacy and Life Skill Survey. She also said that literacy leaders
would be exploring the possibility of conducting some secondary analyses
of the results of the National Assessment of Adult Literacy (NAAL) in
the coming months. She said that the Institute had reconvened the Adult
Literacy Research Working Group with an eye towards updating the adult
reading document if enough literature was available to accomplish this
and to create other appropriate products. She then stated that a grantee
in the Adult Reading Research Network had requested supplemental funding
that would allow the grantee to increase the number of subjects in the
study and increase the number of measures that are used, which would
allow the Institute to compare results across the six studies that
comprise that research program.

Dr. Baxter concluded by saying that the Institute had excellent coverage
on lifespan literacy issues from the research perspective, but that some
of these efforts would begin to wind down as the panels finished their
work and moved into the implementation phase in terms of dissemination.
She also encouraged the Board to bear in mind the importance of moving
"towards a model that says research findings are not stagnant. They
change. We are building a knowledge base." She also encouraged the Board
to think about follow-up activities in terms of research for each of the
above-listed major reports and efforts, whether it be simple periodic
updating, targeting important additional topics.

The floor was then opened for questions. Ms. Hunter asked if there is a
time line associated with these projects, and Dr. Baxter answered that
they all had individual time lines. Dr. Hiller asked if this would be a
useful tool to increase their knowledge of their products, to which Ms.
Hunter replied that it would be easier to scan it. Dr. Hiller then
asked, "Do you think this is useful?" Ms. Osborn replied, "It gives a
very concrete realization of what is happening." Dr. Hiller said that he
had been hoping that the Strategic Planning Committee would provide one
tool that could be used at every meeting \-- something that could be
shared with the literacy organizations as it develops. He then said that
it would perhaps be a consistent tool to the foundations, the literacy
organizations, and the researchers, and that it could be updated by
staff.

Ms. Osborn asked what the anticipated problems were for the National
Early Literacy Panel report, and Dr. Baxter answered that they had not
identified all of the specific practitioner documents, although they
could generally say that they would have one document for early
childhood educators and another for caregivers. Ms. Reddy added that
they would also write a document for policy makers and commented on the
issue of whether the initial spin-off documents need to be targeted to
particular audiences, or whether they can be used more generally. She
then commented that they were grappling with the question of whether it
would be appropriate, due to the diversity of settings in the pre-school
community, to produce a single document, like "Put Reading First." Ms.
Osborn added that it would also be helpful to find some way of
designating products that "are going to be done here as compared to the
report itself, which you're just going to funnel through. So that
doesn't involve enormous amounts of work for the Institute, where these
kind of products, they either are contracted out or much more work."

Dr. Hiller commented that the next phase in the refining of the report
would involve products that require measurement, such as survey data. He
said that he hoped that this would eventually be used as a monitoring
tool. Chairperson Borders wondered if this discussion answered Mr.
Yudof's question from the previous meeting concerning the research
agenda, to which Dr. Hiller replied that it not only answered this
question, but also provided easier access to research questions, which
helps when talking to constituencies. Dr. Baxter suggested that they
also list the goals of other major research agendas, to which Dr. Hiller
replied that "that's where your interagency issues come in."

Ms. Hunter suggested more delineation under national research agenda,
which led to a brief discussion of the format of the chart and how to
read it. Ms. Osborn suggested that it be noted that the research
questions are "as identified by the panel," and Dr. Baxter concurred.

Dr. Hiller said that he would like to see a tool which allowed the
Institute to take credit where credit is due, since it is important for
other agencies to see what the Institute is doing.

Dr. Hiller next explained that he had been working to ensure that the
Institute is always prepared for its PART performance measure
evaluation. To this end, he read several sample questions that might be
asked in the evaluation, such as "Is the program purpose clear?" and
"Does the program address a specific interest, problem, or need?" Dr.
Hiller then said, "so you see the kinds of questions that I believe if
they're formatted well on this other document, we can answer." Dr.
Hiller then noted that the White House said that around 10 of the listed
projects would be removed from the budget because of poor PART
performance. Thus, he explained, it is very important to do well on the
next evaluation. He suggested that the Institute use the Institute for
Education Sciences' PART evaluation as a reference point and example of
how to deal with the questions. He stressed the importance of providing
evidence such as customer satisfaction surveys, of taking a proactive
approach with strategic planning opportunities, and of remaining clear
on performance result requirements. In conclusion, he said that the key
to performing well on the PART evaluation is evidence and products, or
"evidence products."

Dr. Hiller next identified several questions on the PART, which carry
greater weight than others: "Has the program demonstrated adequate
progress in achieving its long-term outcome goals?" and "Does the
program demonstrate improved effectiveness in achieving program goals
each year?" With regard to the latter question, he said that he was
working with staff to try to improve efficiencies, but that customers
can also help with efficiencies if they know the agenda.

Ms. Osborn asked if these reports would be monitored. Dr. Hiller replied
that the monitoring came with the chart, and that staff would also
provide documentation to support the answers given in the self-report.

Dr. Hiller expressed his desire for the Institute to stay ahead of the
performance requirement document and to retain its position in the top 5
percent of agencies in terms of preparation for the evaluation. Ms.
Osborn asked when the Institute might be required to complete this
report. Dr. Hiller replied that the Institute would not be given much
warning. He also commented on how stringent the PART requirements are
with regard to budget reauthorization, and again emphasized the
importance of being prepared and of being aware of the performance
measures. Dr. Hiller concluded by saying that the staff members had done
a great job so far, and that, by preparing ahead of time, they were
saving themselves from doing a great deal of work in the future.

## Public Comments

**Leila Plassey, Director, National Coalition for Literacy**

Ms. Plassey introduced herself as the executive director of the National
Coalition for Literacy. She then explained that the Coalition is a
membership organization representing some of the largest literacy, ESOL
and family literacy, adult education and family literacy organizations
across the country. She then thanked the Institute for inviting her
organization to make public comment, and expressed her interest in the
Institute's work. She also thanked the Institute for advertising for
senior program positions related to adult education, wished the agency
luck in filling those positions, and commented that she might be
assisting in that process. She then communicated the coalition's strong
support for hiring a permanent executive director, since having a
strong, permanent staff is the first step to success.

Ms. Plassey next said that the coalition would most likely support the
Senate version of the WIA bill, since it would be the strongest for both
the Institute and for the interests of her organization. She also
communicated that the coalition is advocating level funding for adult
education through the appropriations process and asked that the
Institute request level funding for adult education as the legislation
moves forward. To Ms. Hunter's question of why the coalition was so
intent on level funding, Ms. Plassey replied that this interest was
primarily because the two-thirds cut, proposed by the Administration,
which had the potential to bankrupt many programs, and in turn leave up
to 500,000 students without program support. Secondly, Ms. Plassey said
that the PART evaluation of the adult education programs was inaccurate.

Ms. Plassey next stated that the coalition was taking an active role in
working with the American Institute for Research to disseminate the NAAL
survey results. She said that her group would be working through a
funder to develop a media strategy around the release of the data,
which, to her knowledge, would occur on September 8 \-- International
Literacy Day. She then stressed that this opportunity is very important
to the coalition, since it occurs only once every 10 to 12 years.

## 

## 

## 

## Jean Stevens, Director, Center for Literacy Studies at the University of Tennessee

Ms. Stevens introduced herself as the director of the Center for
Literacy Studies at the University of Tennessee and thanked the Board
for the opportunity to attend the meeting. She then said that she would
like to speak briefly on two topics: LINCS and the Equipped for the
Future project.

With regard to LINCS, she said that she spoke for many in the field in
expressing her gratitude that the Institute had undertaken the
evaluation that they had in the past several years, since it is
important to have projects reviewed by outside evaluators. She also
said, however, that many people in the field are concerned about what
will happen to LINCS, since LINCS is important to the field, and it is
beginning to be known that the contracts with the special collections
and regional technology centers will all be ending on September 30. She
therefore requested that the Institute move as quickly as possible to
make the necessary decisions on the direction of LINCS to prevent gaps
in service.

With regard to the Equipped for the Future project, Ms. Stevens stated
that it was a major, 10-year project through the Institute, and that its
federal funding had ended last September. She said that the Center had
done training and implementation for the project over the past several
years, and had, therefore, worked hard over the past year to maintain
services to the field and the usage of materials developed by the
Institute. She said that they were accomplishing this through two means:
the use of outside funding to help with internal strategic planning and
work done with local programs on one of the project's products, and
contracts with agencies and states to provide specific training and
technical assistance to implement many of the products, training,
concepts and framework of the project.

Ms. Stevens next commented that, in collaboration with the National
Center for Family Literacy, the Center for Literacy Studies had used
National Institute for Literacy and Reading Partnership funds to develop
a curriculum to train teachers to teach reading according to research
that's been done in the four major areas, and she expressed pleasure
that John Kuidenier had spent several days last May helping with a
training session for this product. She then said that the group had two
major initiatives involving the implementation of teacher training: one
in Oklahoma and one in Washington, D.C. She said that the team would
continue to use both of these states to expand training. In addition,
she said that planners were also doing training with Literacy New York,
as well as at the Pro Literacy Southeast Regional Conference and the Pro
Literacy Annual Conference. To Ms. Hunter's question of whether they had
limited the training to teachers of adult education students, Ms.
Stevens replied that this was, in fact, their focus, and concluded by
saying that they were pleased to continue this work for the National
Institute for Literacy.

## Budget Committee Report

Dr. Baxter began the Budget Committee Report by saying that, as
continuing grant and contract activities were winding down and new
dollars were competing for new activities, certain information should
not be released to the public as it could affect the outcomes of a
competition. Therefore, she explained, they had not included figures for
new competitions from the report.

Dr. Baxter then stated that the spending plan for the Workforce
Investment Act for fiscal year '04 had been approved in late April, and
that, although some funds had been obligated to existing activities, the
bulk of the money was for new competitions. She said that there would be
some reprogramming of funds, and that at least one of the activities \--
the English Language Acquisition project \-- is now being funded
entirely out of Dept. of Education funds. She also stated that the
Institute had been slated to jointly fund the White House Initiative on
Educational Excellence for Hispanics conference along with the
Department, but that the group no longer needed those funds.

In terms of discretionary activities, Dr. Baxter stated that the
Institute was in good shape with regard to its payroll, and that the
budget would be more than adequate to cover any hiring that might take
place before the end of the fiscal year, as well as travel expenses for
the Board members. Dr. Baxter next highlighted non-pay Item B-1: rent
and Education Dept. services. She advised the Board to think of this as
overhead money for services provided, and commented that Institute
managers knew that there would be increased costs in this area on
account of the increase in responsibilities taken on by the department.
She said that final figures would not available until September.
Finally, she said that if significant funds are left over in any of
these discretionary accounts, managers could easily transfer the money
into the program budget.

With regard to gift funds, Dr. Baxter said that they came from three
sources. She then said that the Verizon grant money was reserved for
America's Literacy Directory and so would not be used until the next
year when the Institute begins to make changes to the Directory.

With regard to the Partnership for Reading budget, Dr. Baxter said her
office had only obligated money from this budget to cover current costs
for distribution, hot line, and storage costs associated with the
publications. She then said that this spending plan was very close to
being approved by the OMB, and that once it is approved, the Institute
leaders will be able to move forward. She commented that their most
focused effort in this plan will be on Item C-1, the K through 3
successful programs, on which they are working very closely with NICHD
and the Dept. of Education on developing a strategy.

Dr. Baxter concluded by saying that, although Institute managers had not
yet met their goals in terms of obligation of funds, they were on the
right track, and she did not anticipate any major problems.

**Review of Action Items**

**Dr. William Hiller**

Dr. Hiller began by saying that, because of our recent discussion of
personnel and organizational frameworks, Ms. Hunter and Ms. Osborn would
be talking to staff regarding these issues. He then suggested that the
Program Committee take a close look at Dr. Lyon's analysis in order to
better determine how it factors into the Institute's work, and Dr.
Baxter said that she would provide a summary of Dr. Lyon's comments and
suggestions to each of the Board members. Dr. Hiller then said that
Strategic Planning Committee would continue to refine the tracking chart
related to PART and use it as a tool with the Interagency Group. He also
said that planners would continue with the LINCS information, as well as
move along with the high performing program methodology related to
criteria. Ms. Hunter asked if anyone had received an email on the report
from the National Center for Family Literacy, to which Dr. Baxter
replied that she would make sure that Ms. Hunter got it.

Dr. Hiller congratulated Ms. Borders and thanked her for being Chair
again. He then announced the scheduling of a conference call for July 25
at 2:00 p.m., as well as a meeting for November 2 and 3 in Tallahassee.
One of the focuses of the November meeting will be high-performing
programs. He also reminded the Board to keep in mind Dr. Catherine
Snow's recommendation for the establishment of a well-integrated data
system for states that aggregated data from the district level to the
state level. He then said that he had found the priority sheet very
helpful and requested that Dr. Baxter send it out along with the chart
and the IES evaluation, in order to better make sense of what they were
doing and to bring a larger percentage of the Board up to speed.

Dr. Baxter noted that the meeting would adjourn earlier than planned
because of Board members' changing travel arrangements due to weather
conditions.

Chair Borders thanked everyone for attending, and said that it had been
a productive meeting.

## Adjournment

The meeting adjourned at 12:06 p.m.

##### APPENDIX

## Presentation by: 

## Chris Schatschnieder, Ph.D., Associate Professor of Psychology, Florida State University

## National Early Literacy Panel

Dr. Schatschnieder used a series of slides to discuss the methodology
used in developing the National Early Literacy Panel (NELP) report. He
called the methodology rigorous and described the report as
"well-aligned with the What Works Clearinghouse in terms of their
criteria for evaluating studies."

## Board Discussion with Dr. Chris Schatschnieder

## Board members then asked Dr. Schatschnieder a series of questions and discussed his presentation.

## 

## When Ms. Hunter asked whether the What Works Clearinghouse has produced any deliverables related to his research, Dr. Schatschnieder said officials at the What Works Clearinghouse are reviewing their standards as a step toward producing deliverables.

He said NELP was established to "summarize the scientific evidence on
early literacy development and the home and family influence on that
development." He added that the panel is a subset of a larger project
being conducted by the Family Partnership in Reading. He said the panel
was "charged to conduct a research synthesis that would contribute to
the decisions and ED policy that affect early literacy development and
how teachers and families support young children's language and
literacy."

## Dr. Schatschnieder said the panel focuses on children at birth through age 5, examining what they need to know in order to get an early start in learning how to read, write and spell. Dr. Schatschnieder said the report is not simply a literature review. He said his colleagues regard their work as a "systematic empirical study in and of itself, where the individual studies become sort of subsets in a study."

"It's an independent research study in its own right," he said. "It uses
existing studies as data for the analyses. It also includes predefined
selection criteria for identifying relevant research, standards for
judging quality, operational definitions and reparability of methods."
He said the benefit of research synthesis is that it allows researchers
to systematize the evidence in support of a research question, though he
acknowledged that such a study is limited to the information available.
He said researchers keep track of the gaps in knowledge.

Dr. Schatschnieder said having presented research data more than 10
times at national conferences he has discovered that there is a "lot of
anger" about scientifically based research. He contended that in
addition to providing clarity, education leaders must convince the field
that "these are good things to do." He said in presenting NELP results,
telling listeners that a finding is based on an aggregation of 10,000
subjects carries no weight. He said listeners want an example. They want
concrete, anecdotal evidence, preferably via a videotaped testimonial.

He noted that Keith Stanovich's book, "Using Research and Reason in
Education," discusses this trend. Dr. Schatschnieder said scientists
call it the difference between the "correspondence and a coherence
theory of truth." That is, "people either believe something is true
based on evidence or whether it rings true for them inside." He said the
only way scientifically based research will ring true is if researchers
have specific examples. Otherwise, he said, "We are going to be in big
trouble. We are going to be fighting an uphill battle."

Several Board members agreed. Some consider the climate even worse. Ms.
Hunter pointed out that regulations require that schools meet adequate
yearly progress, a tough standard for many educators. She said
simplicity has worked for administration leaders who are in the field
outlining the five components of the President's No Child Left Behind
program. "We have to do more of that," she said.

Dr. Schatschnieder then discussed in detail the research questions used
in the NELP study. He said the first task, research question 1, was to
develop correlates for reading, which are the factors researchers can
measure in children from birth to age 5 that later can be linked to
reading, writing or spelling. "We need to know what's related in early
childhood that then translates into what is going on in schools," he
said.

He then skipped to question 4, in which the researchers targeted factors
related to reading, writing and spelling and looked to see if there are
any programs or interventions that have shown positive impact upon those
correlates of reading. He later described question 4 as the "bread and
butter" of the report. He said this question focuses on "what can we do
to get kids to be better-prepared for reading once they get into
schools."

He described questions 2 and 3 as "moderator variables." "That is, you
get this outcome, you get this overall estimate of how effective it is,"
he said. "And then you try to figure out who it is most effective for
\[and\] who is it least effective for."

Dr. Schatschnieder explained the criteria for including research into
the NELP. He said the panelists decided to examine only articles
published in English. They defined the age group as zero to 5-year-olds
in kindergarten. The data they collected had to be based on empirical
research; qualitative reviews were not included. He said the researchers
boiled their examination down to 234 studies.

After he and several Board members discussed the methodology, Dr.
Schatschnieder focused on some of the results. He showed Board members a
table of predictors of decoding -- that is, word reading or non-word
reading in kindergarten or beyond. "The biggest thing we have learned is
that reading predicts reading," he said.

He identified his fellow NELP panelists as: Susan Landry, Kathy
Escamilla, Dorothy Strickland, Anne Cunningham, Chris Lonigan, Tori
Molfese, Janet Fischel and Tim Shanahan, the chairman.

Board members discussed strategies to publicize information about the
report in USA Today and other general-interest media. Mr. Yudof, Dr.
Carnine and Ms. Hunter agreed to form a committee that would coordinate
the release of the Preschool Curriculum Evaluation Research (PCER) in
the NELP report.

## Presentation by:

## Barbara Wauchope, Ph.D., RMC Research Corp.

## LINCS Program Review

Dr. Wauchope gave an overview of the comprehensive review and analysis
she and her team are conducting of LINCS. She said the goal of the RMC
Research Corp. Project is to describe and assess the implementation on
LINCS. "We want to get a picture of what it is, what is trying to do,
where it is going, how it operates, its structure, and its activities."
She said her team is examining the "use, value, quality and
effectiveness of LINCS." She called the review a qualitative analysis of
the information the Institute has produced, including the Institute web
site, the LINCS web site and the central structure and infrastructure of
LINCS.

The researchers are also examining the discussion lists, the regional
technology centers, and special collections. At the Institute's request,
she said, the team examined the assessment strategies and reading
profiles web site developed by researchers at Harvard University's
NCSALL, based on the center's Assessment of Reading Components study.

Responding to Chairperson Borders' question, she said RMC has done a
preliminary analysis of user surveys. She said the surveys include
special collections in the Regional Technology Centers and the technical
assistance component of the project. Dr. Wauchope added that the surveys
attempt to gather information about what users regard as valuable to
them about LINCS, how they use it, who is using it, and how often.

Dr. Wauchope said RMC conducted a mail survey of people who work in
adult education programs to capture LINCS users and non-users. She said
her team wants to find who is not using LINCS and why. The team also is
examining document reviews and financial analyses of the programs,
products, and services.

A study is being conducted that is reviewing the "use quality and value
and effectiveness" of the discussion lists. She said this study is
primarily analyzing the content but is also looking at user
satisfaction. She said her team is also querying moderators to get their
input. Researchers want to find the extent of advocacy on the discussion
list and the impact of delivery on services.

Wauchope's team has interviewed directors and partners of the Regional
Technology Centers who are mainly employees of state agencies. The RMC
group is reviewing the structure, organization management, operation,
and activities of these centers.

The reading profiles study, like the LINCS study, focuses on use,
quality, value, and effectiveness.

Wauchope then gave a status report on the project. She said RMC has
submitted some interim reports to the Institute but must collect and
analyze the surveys and interviews before writing a final report. She
then took questions from Board members.

## Board Discussion with Dr. Barbara Wauchope

In response to Chairperson Borders' question, Dr. Wauchope said the
Institute would have to decide whether it wants recommendations from
RMC. She said, "everyone agrees that the quality of LINCS varies
tremendously. There is not a lot of evidence-based material accessible
through LINCS." She noted one reason for the range of quality is that
LINCS was designed to help people access free information. "It is not
supposed to connect you with the Institute or commercial web sites out
there or those products that people can sell to you." According to Dr.
Wauchope, "the original goal was more to provide what is publicly
available."

She said directors of special collections have complained that they do
not have access to scholarly information and that it is very expensive
to get access. Mr. Yudof suggested that Institute leaders investigate
the use of the Google Scholars Program, but Dr. Wauchope countered that
research articles don't appear to be a top priority for LINCS users. She
said instructors and administrators appear to be the primary LINCS
users, and that preliminary research indicates that 25 percent of LINCS
users want facts and statistics, 19 percent seek lesson plans for their
classrooms, while 16 percent want news and announcements and 16 percent
want articles, journals and digests.

## Presentation by:

## Ron Perkins, Consultant

## LINCS Usability/Navigation

Mr. Perkins, president of Design Perspectives, was under contract with
RMC to study the usability and navigation of LINCS. He explained the
methodology of his study. He said his team gathered 10 educators and
administrators and asked them to perform key functions that they would
normally do with the web site. Over two days, examiners observed them
from a separate room, listening to them think aloud as they attempted to
complete the evaluation. They wanted to see who succeeded, who failed
and why.

The objectives of the study were to identify and explore the usability
issues in navigation, browsing and searching the site and to determine
whether the graphic design helps or hinders using it. Of the 10 subjects
in the study, 40 percent were administrators and 60 percent were
teachers or tutors. Forty percent identified themselves as advanced
computer users, and 60 percent said they were novices or intermediate
users. Mr. Perkins said 30 percent were experienced LINCS users; 70
percent were not. He said the examiners wanted to bias the study with
inexperienced users because studying them would yield more insight. He
said a moderator gave the participants 16 tasks on a set of flash cards.

Perkins then relayed some key findings. While users are very interested
in the information on LINCS and value it, the information is not easy to
find. Educators had more trouble than administrators. "They were looking
for more practical things. They might have been less experienced with
doing things like searches, which is one of the major issues here," he
said.

He said users would browse for information but fail to figure out how
LINCS is organized. Then they would try to search but fail that, too,
and generally give up and look elsewhere. He said users had trouble
finding local information on the regional sites, which are organized by
region rather than by topic. He said users had difficulty returning to
the home page, and that color coding for the regional sites is
confusing. To fix the browsing problem, he suggested linking to the
Google search engine.

Overall, he said, the structure, graphic design, and manner in which the
search tools work make LINCS difficult to use. He characterized the
LINCS home page as a "good first step," but once users get started, the
navigation and architecture intermesh. Mr. Yudof asked whether it would
be better to revise the site or start over. Mr. Perkins said, because he
does not know how LINCS is built, he could not answer. He said changing
some of the graphics and fixing the search would solve some minor
problems.

Board members, Dr. Baxter, and Mr. Perkins then discussed the
feasibility of short-term fixes and ways to place LINCS higher on the
Google search engine. Mr. Yudof and Dr. Wauchope debated the content of
the material -- its popularity versus its scientific legitimacy. For
example, users may favor workbooks, lesson plans and other documents
useful for practitioners, but they may be of little scientific value.

## Presentation by:

## Brian Shea, Ph.D., KPMG

## LINCS/Discussion Lists

Dr. Shea said his team had two major tasks: to examine the incidence of
"misuse" of the discussion lists for advocacy-related postings and to
determine the extent to which the lists offer the literacy field an
effective and topical communication network.

Dr. Shea began with a dictionary definition of advocacy, but Mr. Yudof
objected vehemently to the term, asserting that his scope was too broad.
Mr. Yudof contended that the issue that sparked this study was that some
discussion-list participants were "using government equipment or
government avenues" for "partisanship" purposes.

Dr. Shea explained the methodology that was used for evaluation. He said
his team focused on the ESL and Family Literacy lists, which make up
nearly 30 percent of all activity on the discussion list, for 1995-2003
and more recently 2001-2003. The researchers used the search terms
"Senate," "Congress," "House," "Representatives," "contact," "call,"
"fax," "action," "amendment" or "bill." From 1995-2003, they found that
0.2 percent of the postings on the ESL discussion list contained these
terms. For the same period, 0.5 percent of the postings on the family
literacy list contained these terms. For the period 2001-2003, 0.3
percent of the postings contained these terms. For the same time frame,
0.4 percent of the postings had these terms. In essence, according to
Dr. Shea's slide, "casting the net as wide as possible over the entire
history or just the most recent years reveals a level of advocacy
related to the postings that is negligible." He told the group, "It
turns out \[advocacy\] really isn't an issue. That's the bottom line."

As for the impact of the discussions on program outcomes, he said the
lists do not contain information that allowed the researchers to
determine this variable qualitatively or quantitatively. He recommended
an "Impact Month" during which moderators focus people's attention and
report on the discussion they have had and what they have learned from
each other.

## Presentation by:

## Barbara Wauchope, Ph.D., RMC Research Corp.

## Regional Technology Centers

Dr. Wauchope discussed the key findings of this element of the overall
LINCS review. The first is that the RTC grant program has tried to fill
the need for technology and resources in the adult education field,
resulting in a broad, diverse program of activities. The second is that
the primary work of the RTCs has been to develop the technological
capacity and infrastructure and provide the technical assistance and
training to use it. The third is that the centers did not receive the
guidance to develop an effective system for monitoring and assessing
program outcomes.

In explaining the reasons for the first finding, she said that the
National Literacy Act of 1991 mandated the development of LINCS and that
it was constructed as a centralized, national database and technical and
policy repository. But in 1998 the focus shifted to expanding the
national database to an electronic one with regional hubs. The other job
was to integrate LINCS into classrooms and staff development and to
enhance the knowledge database.

Elaborating on the first finding, she said the past four or five years
have produced another shift; this time, a federal emphasis on reading
research, reading instruction and professional development in reading.
"That has been translated by the LINCS staff into encouraging the
grantees when they reapply for funds to look at the broader
implications, broader kinds of aspects of resources, services that they
can do that are focused not only on adult ed but on family literacy as
well, and a shift from locally developed resources to research-based
resources," she explained.

As for the second finding, she said LINCS grantees have been building
partners, expanding the system, building web sites, purchasing servers,
loading software and training users to access and operate LINCS.

Overall, she said, the LINCS RTCs function well. The grantees have done
what they have been instructed to do, "but they have really had a lack
of clear direction or vision in terms of where to go next." She noted a
"lack of accountability in the evaluation" of grantees.

She posed several questions for the Board, the Institute's leaders, and
RMC to ponder. "What are the Institute's priorities for information to
be collected and disseminated by LINCS? What kind of organizational
structure is appropriate for LINCS? Should users provide or obtain
information through a single web site or through a network of sites? How
much access should they have to the database and the information that
we, the Institute, provide to them? In terms of technical assistance,
how much support is required to collect and disseminate this
information?"

## Presentation by:

## Jay Diskey, Principal, Diskey and Associates

## Institute Identity

Mr. Diskey's appearance was a follow-up to the preliminary report he
delivered to the Board in February. He described his endeavor as a
re-branding project. Renaming, conversely, would require an act of
Congress.

He said participants in focus groups and 30 to 35 people he has
interviewed have told him that the Institute is not widely known and
understood by K-12 organizations and associations. There is very little
recognition of the acronym NIFL, known primarily by the adult education
community. He suggested that the Institute use the full name because it
sounds more impressive: "People think that they ought to know about a
national institute for literacy," he said.

Overall, he recommended that the organization refrain from branding
individual products and, instead, put the Institute's logo on all the
products it funds. He added that the agency should develop taglines
based on its various activities regarding research, policy development,
practice and dissemination, and it should create activities to extend
its brand.

Board members critiqued several revised Institute logos he developed and
decided that any new one should be phased in on the Institute's website,
letterhead and other products. They agreed that a slightly different
logo could be used to accommodate the name of the Institute's projects,
such as the Partnership for Reading and the Commission on Reading
Research. But even in those cases, the Institute's name should take
precedence. They discussed changes to the Partnership logo. Mr. Diskey
committed to working on some revisions. They also briefly discussed the
LINCS logo but agreed to await the completed evaluation of the program
before making any major changes.

He recommended that the Institute develop a standards guide to govern
the uses of logos.

## Presentation by:

## Annette Zehler, Ph.D./Paul Hopstock, Ph.D., Senior Assoc., Development Associates, Inc. 

## Methodology for Indentifying High-Performance Programs

Dr. Zehler said the goal of the Institute's contract with her company,
Development Associates, is to develop a methodology for identifying
high-performing programs serving individuals across the lifespan:
children, youth and adults. She defined high-performing programs as
those that "produce notable outcomes on selected educational and
employment criteria."

The original RFP called for the establishment of a technical methodology
panel comprised of experts in research methodology and literacy program
evaluation, she said. The panel's goal is to guide development of the
methodology and provide specific guidance and recommendations regarding
its definition. The Institute also sees a role for the panel in helping
distribute and support use of the methodology.

The parties agreed to identify a methodology for adult literacy programs
and provide recommendations for how that might be adapted for children
and youth. Concerned that this approach might be inadequate, however,
the contractors decided to develop a methodology framework and apply it
to separate age groups.

The lifespan panel, then, would include three core methodologists: one
for pre-K, a second for K-12, and a third for adult literacy. To provide
consistency, these core methodologists would be members of the
individual panels, which would include two experts in each particular
subgroup.

Dr. Hopstock said the team created a list of 22 issues that needed to be
addressed for a methodology to be useful. He said the researchers
re-examined the literature to get a sense of how others had addressed
those issues and what kinds of answers they had reached. Most of the
focus was on adult literacy programs, he said, but they collected much
information on other types of programs as well.

Among the 22 issues were six key issues that cut across the lifespan, he
said. The first key issue was whether the panel agreed that there should
be more than one methodology for the different age groups.

The second matter was the definition of the term "program." The field
defines a program as an administrative unit, such as a school district,
that provides services. A second definition refers to a specific set of
instructional activities or approaches.

A third issue that the panel would need to discuss, he said, are the
potential users of the methodologies and how they might be used. "For
example," he explained, "the methodology to identify high-performing
programs might be used for recognition or support of high-performing
programs or it might be used to identify such programs so that further
research can be conducted to better understand the reasons for success."

Fourth, the panel will have to examine whether the methodology should
focus almost or completely on outcome measures or whether other
indicators should be included, he said.

He said the panel will also need to address the scope of the definition
of literacy and the extent to which writing and other reading skills
should be examined. For example, some definitions of literacy include
numeracy, occupational literacy, and financial literacy.

Dr. Zehler then gave an update on the status of the panel's work. She
said the researchers are focusing on developing the technical
methodology panel. They have nearly confirmed five members and hope to
meet in early to mid-summer. She said her team has approved a panel that
will define the methodology for adult literacy programs. In addition,
the researchers have conducted the foundation paper and have begun some
work toward the K-3 methodology research review.

# **[Board Discussion with Dr. Zehler and Dr. Hopstock]{.underline}**

Dr. Hiller raised concerns about use of the phrase "age groups" when, in
his estimation, the researchers are referring to "grade levels." He
urged the researchers to ponder the notion further and cautioned them to
"be careful of mixing ages and grades" in defining the scope of the
panels.

In response to Dr. Hiller's and Dr. Wagner's comment that a five-member
technical methodology panel "sounds small," Drs. Zehler and Hopstock
said the team is exploring ways to get more input from the field,
including assembling an expert peer review group and allowing presenters
at the panel meeting to provide additional information.

Ms. Hunter asked for clarity on what the deliverable of the project
would be. Dr. Zehler said the core outcome would be the definition of a
methodology by which a user could identify high-performing programs. She
also asked how this project aligns with the What Works Clearinghouse.
Dr. Hopstock replied that What Works identifies programs by comparing
them with a control group. This methodology project might compare
program to program.

## Presentation by:

**Elliott Medrich, Ph.D., Director, Policy Analysis & Development, MPR
Inc.**

## Literature Reviews for Program Planning

Lynn Reddy introduced Dr. Medrich of MPR Associates. MPR is working on
program planning in the area of English language acquisition with the
goal of increasing the use of evidence-based practices in adult
education. She said nearly all of the planning contracts include a
literature review. She said that Dr. Medrich has done much work with the
What Works Clearinghouse and she, Dr. Medrich and Kathy Chernus of MPR
have been exploring ways to develop standards for more rigorous
literature reviews.

Dr. Medrich said that while his goal is to trigger discussion, "the real
hard work" is for the Board and the Institute staff to create a
framework for the reviews. He said the timing of this inquiry is
fortuitous "because there has been a real change in what we expect and
what we want from literature reviews."

In the past 10 to 30 years, he explained, literature reviews were casual
documents based solely on the author's criteria, which he called the
"author-determined model." But the Institute of Education Sciences has
raised the quality standard of the reviews. This "external model" uses
more objective criteria. Dr. Medrich said the Institute recognizes that
the bar has been raised. In today's climate, "We need some process that
is going to ensure that the things we include in the review . . . are
appropriate," he said.

He said the Board and the staff need to consider how to establish
criteria for selecting the content for literature reviews for two
reasons: one, to help the reviewers form a systematic basis for judging
what to include in the reviews; and two, to help readers of literature
reviews understand the basis on which items are included or excluded.

"The What Works Clearinghouse strategy is one way to assure
high-quality, well-structured literature reviews," he said. "The problem
is that in the world of education research this process leads to "a
barren evidentiary trail."

"You can start with 1,000 pieces of research," he explained, "and by the
time you bring it through all of the criteria that are a part of the
process . . . you \[end\] up with two items that make it through the
sieve."

He encouraged the Board and the staff to develop their own process, and
urged full disclosure in reporting. "We need to be able to spell out to
readers why they are seeing what they are seeing. In other words,
describe the criteria being used for what has been included. Doing so
will help defend the review against challenges regarding inclusion and
exclusion of material. Sound criteria should have three components --
categories that describe the design, population and results of the
review.

"The mandate," he concluded, "is a clear set of criteria that the
reviewers can use when they do their work, a clear set of criteria that
the readers can find so that they can understand why they're seeing what
they are seeing in the literature review, and an extraordinary
supplementary document that really does parse out for the readers how it
is that the work that is cited fits the criteria that have been
established."

## Board Discussion with Dr. Elliott Medrich

Dr. Wagner, a psychologist and researcher, cautioned against using
criteria that are too strict. "When you rigidly have this bar that only
a few things get by," he warned, "not only are you sort of not taking
advantage of the literature out there, but you also are just opening
yourself up to so much carping that you are excluding this, this and
this and for sometimes somewhat arbitrary reasons."

Dr. Medrich replied that the What Works Clearinghouse model is, in part,
an example of this problem. The formulation "was so thoroughly
exclusionary that very few items could ever be reported with
confidence," he said. But Dr. Medrich reiterated that the key is full
disclosure, informing readers of the nature of the study: what is
included, excluded, and why.

But Dr. Wagner countered, "I am not sure that is going to be a very
effective approach in the long run. I think it is better to be very
clear in sort of describing the studies and their characteristics and
doing that quantitatively and actually being explicit about the specific
criteria that you are making these judgments on."

He added: "I just want to make the point that there is at least one area
out there that has gone in a real different approach than this one."

Dr. Medrich responded that there might be "sort of an intermediate zone
between the more rigid approach and the whatever approach."

Ms. Reddy jumped into the debate. She noted that since the literature
reviews are in a variety of areas with varying research bases, one set
of standards that would work across all subjects is "probably just not
possible."

Dr. Medrich replied that the standards don't have to be defined in rigid
terms, but that some standards have to be set.

Several Board members discussed the What Works Clearinghouse. Dr.
Medrich said most of the work that the Clearinghouse "believes to be of
high value is work that takes sometimes five to 10 years. They are major
intervention, random assignment studies" that are costly.

Dr. Medrich said education research is an important matter for the
Institute to consider. "The thing is that right now it has the
Institute's implications because the world is looking at your
literature, the literature that you're using in different ways than it
did 10 years ago, five years ago," he said. "And one not only has to be
cognizant of that, one has to, in some way, be responsive to it. And the
question is: What is the appropriate way to respond? And that is a issue
of the Institute."

Barbara Milton, a contractor, Dr. Medrich and Board members discussed
details regarding the methodology of literature reviews.