# EXECUTIVE SUMMARY

Twenty-six participants in the dissemination workshop generated 238
unique comments and suggestions during the daylong workshop. Eleven NASA
insiders and fifteen outsiders attended. The day was broken into three
hour long brainstorming session with the participants divided into three
groups. Between break out sessions, there were informative and topical
presentations and plenary group reports.

A number of themes are identifiable based on the emergence of related
suggestions cutting across groups and the three sessions:

1.  One educational portal

2.  Include user input from the ground up to design products that really
    meet the needs and are within the ability range of the target
    audience.

3.  Brand NASA supported/approved products and pages.

4.  Resolve hard questions up front.

5.  Change the mentality that the product is done and will disseminate
    itself.

6.  Partner.

7.  Make better use of existing dissemination mechanisms.

8.  Retire educational resources that are no longer useful.

9.  There was a general sense that NASA needs to move toward a more
    unified and cohesive educational effort.

Forty-five of the more promising specific recommendations are presented
in the paper.

# INTRODUCTION

The Leading Educators to Applications, Research, and NASA-Related
Educational Resources in Science (LEARNERS) initiative sponsored a
one-day Dissemination Workshop to generate ideas and suggestions about
how to maximize the impact of NASA funded e-based educational resources.
The sense was that there is room to improve the current level of
dissemination and that bringing together NASA insiders and
representatives from education and industry could generate the ideas to
fuel such improvement. The workshop was conducted at University of
Maryland Baltimore County, South Campus Technology Center, on August 1,
2002

Twenty-four invited participants attended with the last minute addition
of two informal attendees. Nine NASA insiders participated from
Headquarters, Goddard, Ames, and Langley, representing the Space
Science, Human Exploration and Development of Space and Earth Science
Enterprises and a variety of educational areas. Fifteen outside
participants represented schools, universities, libraries, the online
gaming industry, software companies, and educational organizations. Two
of the LEARNERS principal investigators gave demonstrations of their
projects and participated as representatives of the developer community.

The format of the workshop was to emphasize the brainstorming and idea
generating portions of the day with short, informative presentations
distributed through the schedule. With three hours of breakout groups
and two hours of plenary discussion time, the bulk of the day went to
soliciting input.

# WORKSHOP

The day started with a brief welcome from Dr. Robert Gabrys (NASA GSFC)
as LEARNERS manager. Mark Leon (NASA ARC), Learning Technologies Project
manager, gave a short talk on the importance of educational outreach to
NASA. Susan Hoban (LEARERS, UMBC) provided an overview and context for
LEARNERS. Peter Coppin (Carnegie Mellon University) demonstrated
Eventscope ([www.eventscope.org](http://www.eventscope.org/)), and Glen
Schuster (U.S. Satellite Laboratory) gave an overview of Signals of
Spring ([www.signalsofspring.net](http://www.signalsofspring.net/)) to
give the participants a taste of some of the LEARNERS projects.

Much of the rest of the day was spent with participants divided into
three groups of eight or nine members and a facilitator from the
LEARNERS team. Each group met three times for one hour each to
brainstorm. Each session had a theme topic about which participants were
charged to answer questions. Group participants stayed with the same
group and facilitator for all three session. It was hoped that such an
arrangement would contribute to deeper and more open communications, as
the group members became more familiar with their group mates and
facilitator. Each participant received a worksheet with the questions
for that session at the start of each session. They were encouraged to
write responses on the worksheet. The facilitators prompted their groups
to brainstorm responses through discussion. In two groups, the group
responses were recorded on flip chart pads. In the third, the group
decided to have the facilitator take notes on worksheets while they
group brainstormed the topic.

At the end of each session, the worksheets were collected from all
participants. After each session the facilitators reported the their
group's responses to the all of the participants. This gave individual
participants a chance to correct misrepresentations and share the input
with the other groups.

# SESSIONS

## Session I: Delivery Systems

Prior to breaking into groups for Session I, Robert Gabrys gave a
ten-minute presentation on Educational Resource Centers. Larry Bilbrough
(NASA HQ) gave a short overview of the Aerospace Education Services
Program.

Participants were asked to respond to the questions:

1.  What delivery systems are successful?

2.  What delivery systems do not work?

## Session II: Measuring Success

Prior to breaking into groups for Session II, Patrick Hogan (NASA ARC),
Learning Technologies Project Deputy, gave a ten-minute talk on what
information NASA used to judge successful dissemination.

Participants were asked to address the questions:

1.  How do we currently measure success?

```{=html}
<!-- -->
```
3.  Are there other methods of measuring success that would be useful to
    implement?

4.  What are or ought to be our goals?

5.  What factors (technical, political) affect our goals?

## Session III: Designing for Success

Prior to breaking into groups for Session III, David Anderson of NSTA,
and Troy Cline of Raytheon, gave an overview of NSTA's SciLinks program.

Participants responded to the questions:

1.  How do people search for educational products?

```{=html}
<!-- -->
```
6.  What can we do to improve the chances of them finding and adopting
    of NASA products?

7.  Are there things we can do up front?

8.  Are there add-on enhancements that can be 'retrofitted' to existing
    products to extend their appeal?

# INPUT

The format of the workshop was to emphasize the brainstorming, idea
generating portions of the day with short, informative presentations
mixed in mostly for the benefit of the outside attendees. With three
hours of breakout groups and ninety minutes of plenary discussion time,
the bulk of the day by far went to soliciting input. The workshop
generated sixty-eight individual pages of comments in the form of the
worksheets handed out in each session and twenty-four flip chart pages.
These contained almost 500 individual comments, notes, suggestions, and
ideas. Of these, 238 were sufficiently unique to warrant listing as
distinct remarks. Most fell broadly in to range of the questions that we
developed before the workshop. Some dozen comments did not fit any
category and were placed in a separate category of their own.

There were insufficient controls put in place to treat the ideas and
information generated in the workshop in any sort of mathematical way,
so there are no graphs or formulae used in this assessment. Instead, the
feedback is presented based on a qualitative assessment. Weight is given
to those areas where multiple participants have noted the same or very
similar points regardless of the area or the portions of the workshop in
which they were derived. Those points that were repeated over and over
again by at least several participants are identified here as over
arching themes that emerged in the workshop. Bearing in mind that the
same twenty-six participants were the source of all the material, trends
are to be expected. Concepts that were important to a group member in
the first session were likely to still be important in the last session.

The vast majority of comments, whether from individual worksheets or
flip charts, came in the form of single words or short phrases. There
were also several dozen longer comments ranging from significant
sentence fragments to, in one case, an entire back page of a worksheet
filled with numbered related points.

# THEMES

A number of overarching themes emerged from the input. These were areas
in which participants made comments or recommendations throughout the
day and across all three groups. Some of these themes address systemic
issues while others are matters of attitude.

## One educational portal

There was consensus that there needs to be a single point from which the
online community can find all of NASA's educational products. Currently
there are several sites that say they are the gateway to NASA education
but none link to all the resources according to participants.

According to the NASA home page there are more than 4.1 million NASA web
pages. As the warning there says it can be difficult to find what
someone is looking for in all those pages. To an outsider and many
insiders, the profusion of NASA education sites does not intuitively
present itself in an organized fashion. Among the participants that have
experience with NASA education, there was a sense of confusion of how
the parts fit together. Several participants said that the ability to
search was either not always present or not comprehensive. Others noted
that while a number of pages said they were the gateway to NASA
education, they did not provide useful access to resources the
participants knew to exist.

There is a persistent tension between the resources that NASA or any
similar organization has to offer and the challenge of meeting the needs
of potential end users.

One member of the LEARNERS team describes the meeting of a teacher and
scientist that ground to a fruitless halt when the scientist said, "tell
me what you need" and the teacher could only say, "tell me what you have
that I can use." The heart of that conflict is the interface between the
potential end user and the mass over of resources available. Basically,
without knowing what is out there, the teacher is limited in their
ability to know what to ask for. This tension is at the core of the
National Science Digital Library initiative. It is a matter of having an
interface that is useful to both the novice and the experienced user so
they can find useful resources without having to know they exist ahead
of time.

Part of the solution is a good, semantic search engine that finds
resources intelligently using the users own words. The other part is a
coherent, intuitive "portal" to NASA's educational resources. Such a
portal, gateway, or 'front door' needs to be able to get users to all of
NASA educational resources online smoothly and intuitively. The OneNASA
Portal RFP (RFP #GLW-0822) is heading in this direction; NASA education
needs a similar approach.

It would be impractical to take down all of NASA's education related web
pages to reorganize them in some kind of a systematic, hierarchical
latticework. A realistic solution has to deal with practical approaches.
The easiest approach to simulate the effect of having one portal put a
link to the main NASA education (<http://education.nasa.gov/>) on all of
the exiting education related pages. By insuring the main page does have
the best possible links and pointers to NASA's varied educational
resources, such a mandated linking program would insure that any user
was only one step away from the NASA educational portal no matter where
they started. It would also help keep the main page at the top of
searches for NASA educational resources since search engines use number
of links to a page as one determinant in ranking pages identified in a
search.

## Include user input from the ground up to design products that really meet the needs and are within the ability range of the target audience

The persistent slogan that is supposedly shunned by developers is the
"build it an they will come" mentality. Experience shows that simply
designing from an expert's perspective tends to create products that,
while perhaps elegantly designed, fail to fully address the needs of the
supposed targeted users. Too often only lip service is paid to including
the input of proposed users. At the LEARNERS CAN Opener session, a
member of one of the project teams vehemently expressed distaste for
including teachers in the development process because he felt they would
just get in the way and not know enough to be helpful.

Funded proposals need to include input from members of the proposed
community of end users in order to appreciate fully the needs and
abilities of that community. It is very unlikely that a quality product
can be designed without adequately assessing the needs of the user
community. Requests for proposals and proposal guidelines routinely
solicit information about the need the proposed project is expected to
fulfill, but in an environment where both the proposers and reviews tend
to be from similar backgrounds, it is too easy for vague answers to be
enough to satisfy the requirements. If end users are involved in the
project, it is often not until the testing phase. At that point it can
be difficult, costly and time consuming to modify the product to meet
the test users needs.

To minimize risks of both mismatching products with targeted users and
wasted time and effort, members of the proposed user community should be
included in the project from the earliest stages. In the case of
products targeted at formal education, which means including both
students (as actual end users) and teachers (as gatekeepers and 'pre-end
users' since teachers aren't likely to use products they can't
understand or work with well). Ideally these students and teachers
should be recruited because their interest and skill levels reflect the
average members of the target end user community. Teachers and students
with established connections to NASA or similar programs are likely to
have greater skill and motivation than the average user and thus not
provide fair input and feedback on the project.

To facilitate the process of getting input from novice potential end
users, LEARNERS, LTP, or NASA Education could put together a guide for
developers to give them ideas on how to find volunteers. It may seem
like a simple concept, but the idea of driving to a nearby school of the
appropriate grade level and going to the front office to start a search
may not readily occur to a developer. The notion of recruiting a pool of
volunteers to fulfill the role of end user 'experts' would eventually be
self-defeating. As the volunteers became more experienced, they would
grow more sophisticated and become less like the population of users
they were intended to represent.

## Brand NASA supported/approved products and pages

Brand NASA supported/approved products with a clearly identifiable logo
preferably with the URL to the main educational portal. A template
exists with the logo in the middle and URL and program name circling it,
so this goal seems to be fairly easy to reach from an internal,
logistical perspective. It is simply a matter of applying an existing
mechanism more broadly.

Every web page that is funded by NASA and is educational should have the
NASA logo and a link to NASA's main education page or the funding
programs education page that in turn links to the main education page.
The same rule should also apply any place there is an opportunity to
make a link to NASA's educational efforts. Participants recommended
pens, pencil, and key chain ornaments. There are toys on shelves in toy
stores that use the NASA logo as a selling point that would be an ideal
place to put the education URL. One mother, when asked in an extremely
informal study, said she would jump at the chance to get educational
follow up on a toy or game that got her sons' interest. No opportunity
to put a link to NASA education site in the hands of potential users
should be allowed to slip by.

## Resolve hard questions up front

-   What are the evaluation standards?

-   Who is the audience?

-   What is the goal?

The impression of many participants was that the answers to these sorts
of questions are either too vague or too sweeping to be concretely
meaningful. At least one of these issues can be clearly addressed by
putting forward existing information for at least two enterprises. The
evaluation criteria for Earth Science and the Space Science Enterprises
can be found online:

-   <http://www.strategies.org/NASA_Reviews_Forms/EvaluationCriteria.html>

-   <http://ssibroker.colorado.edu/Broker/Eval_criteria/Guide/Default.htm>

The other areas of concern can be addressed by not accepting vague or
overly broad answers to standard questions. If proposal reviews expect
and demand concrete answers to questions about the target audience of a
product, they can improve the chances of a product successfully reaching
its target audience. Requiring solid examples of how the product can be
used to attain educational goals as part of the proposal can also help
avoid products that are function but of not particular educational
value.

## Change the mentality that the product is done and will disseminate itself

There must be strong strategies in place to disseminate resources. That
means pushing resources that NASA stands behind instead of putting them
on the shelf and hoping someone finds and uses them. That assumption was
at the heart of the workshop. There was a wide range of suggestions
throughout the workshop that all involved actively promoting or products
rather than taking a passive approach. Whatever approaches are taken,
there has to be a shift away from the notion that the product is done
and can sit and wait for users to find it.

It was suggested that NASA should reconsider the model under which it
funds projects and plan longer projects that factor in more evaluation
and support time. That would give products a longer lifetime and
presumably insure the active involvement of the developing team. Another
possible approach would be for NASA to take a more active role in the
product's life. In the case of the current LEARNERS products, that could
mean members of the LEARNERS team taking on the role of promoter for the
projects. That could include giving demonstrations and training NASA
personnel as noted in the theme on making better use of existing
dissemination mechanisms.

A related set of suggestions was to open Legacy Projects for revision
and enhancement. This might lend new life to a product that was
otherwise inactive.

## Partner

At all levels of the workshop partnering was suggested as a means of
enchancing NASA's ability to impact education. Partnerships with
educational organizations, other Federal agencies, educational industry
members, computer manufactures and textbook companies were all
recommended. The assumption built into NASA's educational efforts is
that the organization offers unique assets that can improve education in
science, technology, engineering, and mathematics. Partnering with other
entities in education is the best way to maximize the impact of those
assets.

## Make better use of existing dissemination mechanisms

The LEARNERS initiative, among other producers of electronic resources
for education within NASA, needs to put the time and energy into
training ERC, AESP, and Space Grant personnel on their products.
Representatives of both Space Grant and AESP were emphatic that this was
the case. If NASA resources sponsors do not make them aware and
knowledgeable, members of those groups cannot possibly help disseminate
products effectively. Each NASA development group needs to identify time
when such training can be provided.

## Retire educational resources that are no longer useful

The phrase "let them die a natural death" was used. When a product is no
longer supported, unless it is exemplary, it has limited usefulness.
There should be an ongoing review of Legacy Projects to retire those
that are not longer in use. There should also be reviews of web sites to
identify and remove links that are no longer valid. Users who are short
on time need reliable resources that are going to be available when they
need them. A link that no longer functions or educational sites that are
not maintained can cause frustration and sends the message that online
resources may not be reliable. Teachers seldom run in to the problem of
telling students to turn to the next chapter and suddenly discovering
that part of the textbook missing. NASA needs to make every effort to
keep educational resources reliable and up to date no matter what the
medium.

## There was a general sense that NASA needs to move toward a more unified and cohesive educational effort

This argues for finding and reducing redundant efforts. Clearly
identifying what skills are necessary to be successful is an essential
first step in this process. Given the current focus on education as an
organizational priority and the broad nature of the task, this paper may
not be the best platform for addressing those issues beyond the
recommendations in other sections.

**Recommendations**

Listed below are the five most promising recommendations and suggestions
to come out of each of the brainstorming sessions.

1.  What delivery systems work?

-   Those that partner with established and successful groups in
    education.

```{=html}
<!-- -->
```
-   Those that partner with textbook companies.

-   Those that target audiences' preferred means of communication to
    reach them. For example, if a group used newsletters, reach them
    through their newsletters.

-   Those systems that choose a target audience and design for them.

-   Those systems that do needs studies and pilot surveys.

10. What delivery systems do not work?

-   Avoid jargon-ridden products by consulting novice, targeted users.

-   Do not do passive dissemination like mass mailing or pure web based
    projects.

-   Do not rely on teachers to figure out what they need alone. They
    lack the knowledge of what is available to make the choices.

-   Do not disseminate products without a target audience.

-   Eliminate broken web links.

11. Are there other methods of measuring success that would be useful to
    implement?

-   Hire experts to build evaluation tools.

-   Collect as much data as COPPA will allow giving information on use
    of websites.

-   Do rigorous clinical trials. Mimic NIH testing.

-   Develop rubrics to measure success.

-   Target small groups that can be more easily monitored for measurable
    changes in performance. (One participant recommended targeting
    tribal universities since they are under served community and
    improvement in math, science, and engineering scores and enrollment
    would be measurable.)

12. What ought to be our goals?

-   To develop educational products that reach 53 million children.

-   To have a significant impact on math, science and engineering
    education.

-   Take time and resources to study and learn as you go.

-   To focus on things NASA can do that no one else can or is doing.

-   To stem the attrition rate in the natural sciences.

13. What factors (technical, political) affect our goals?

-   Some people think NASA should not be in education.

-   Changing administrations change priorities.

-   Out of date computers in schools.

-   Fear of change.

-   Lack of emphasis on science in early grades.

14. How do people search for educational products?

-   Online by grade and by subject.

-   Conferences and trade shows.

-   Clearinghouses like ERIC.

-   By asking people they know.

-   Educational journals and newsletters.

15. What can we do to improve the chances of them finding and adopting
    of NASA products?

-   Create a true single portal to NASA educational resources: "NASA
    Wal-Mart."

-   Translate NASA jargon to more accessible language.

-   Put NASA's logo on NASA developed products.

-   Push products and information to the National Education Association
    and American Federation of Teachers. The teachers who teach science
    in the lower grades tend not to think of themselves as science
    teachers and do not join NSTA.

-   Make compelling educational games with NASA content. (One
    participant pushed for partnering to create NASA SIMS.)

16. Are there things we can do up front?

-   Provide strong evaluation guidelines.

-   Require developers to commit to training ERC, AESP, and Space Grant
    personnel.

-   Conform to national and state education standards.

-   Do not fund projects from scratch. (No betting on horses.)

-   Include educators in the development process of projects.

17. Are there add-on enhancements that can be 'retrofitted' to existing
    products to extend their appeal?

-   Created a 'lessons learned' database from developers' experiences.

-   Apply a NASA logo/seal of approval.

-   Update existing products and accept proposals to modify them.

-   Have a librarian or archivist for existing products.

-   Let products die a natural death.

18. Random input that did not fit anywhere else.

-   Retool NASA staffing to include marketers. Marketing is a
    profession.

-   Understand where computer-applied technology is and is not better
    than textbooks and use it appropriately.

-   Have reasonable models of success. "Don't shoot for the moon" every
    time.

-   Use benchmarking.

-   Tap in to the computer game market to develop compelling educational
    games.

# OBSERVATIONS

Despite the fact that the majority of participants were NASA outsiders,
the participants as a group seemed to develop a sense of having a real
interest in improving dissemination of NASA-funded educational products.
They appeared to take the charge to come up with suggestions very
seriously and most expressed a sense of accomplishment and that the
workshop had been a good use of their time. The hope that staying with
the same people through the day would build rapport seems to have been
realized. Each group seemed to develop a bond through the day.

The opened-ended nature of the brainstorming process may have generated
many ideas that were not immediately suitable for the dissemination
purposes. Overall, it seems to have been a fair tradeoff for the
creative input that was derived.

# CONCLUSIONS

This workshop was about generating ideas. The premise behind it was that
getting enough intelligent, educated people in one place and letting
them interact could produce valuable results. Channeling that intellect
toward the issue of dissemination produced almost three hundred
different comments and ideas. Many of these ideas are worth of further
investigation. Some are self evidently useful. Some should be thought
provoking. There is a tendency in education as well as other fields, to
take ideas that sound good and either borrow their terminology or
reinterpret them to mean some thing an organization already does.
Particular care should be paid to avoid those pitfalls.

If anything substantial is to come from this workshop, there must be
follow-up on the major themes and at least some of the individual points
presented. The prevalent belief that more can be done to accomplish
NASA's educational mission hinges at least partly on improving the
dissemination of NASA funded educational products. At this time of
revitalizing education, the results of this workshop should be
considered to alter and improve existing dissemination practices. Four
suggestions have already been adopted and worked into the upcoming
LEARNERS II solicitation. Those proposing under the new CAN will know be
committed to train ERC, AESP and Space Grant personnel, can see the
evaluation standards against which their progress will be measured, will
be urged to include end user representatives from early on and have to
brand all published material with a credit and a URL to LEARNERS web
site. A website has been established to provide access to the material
generated by the workshop. If concrete improvements are to come out of
this process, detailed action plans should be developed based on the
most promising ideas here.

# ADDITIONAL INFORMATION

Websites:

A list of all of the unique input generated by the workshop can be found
here:

-   <http://learn.arc.nasa.gov/ltpmgmt/reports/dissemination.html>

A list of all the participants can be found here:

-   <http://learners.gsfc.nasa.gov/workshop/workshop.html>