# Delivery Systems

## What delivery systems are successful?

1.  Educational events with compelling content

2.  Disseminating content with follow up.

3.  Incentive based systems

4.  Enroll the gatekeepers

5.  Funded/commercial

6.  Thing that target end user

7.  Web based products

8.  Personal contacts programs.

9.  NASA funds then industry pushes into society

10. Those that encourage users to share ideas.

11. Teacher training workshops

12. Needs Survey (pilot studies)

13. Targeting early adopters

14. Giving out materials at state fairs

15. Nurturing individual teacher and compensating them in a meaningful
    way.

16. Cable television.

17. Partnering with established successful groups

18. Textbooks (partnerships)

19. Benchmarking

20. Target audience: users must feel it is for them.

21. NASA TV/PBS

22. Training with follow up training built in.

23. Push instead of pull

24. Continually advertised products.

25. Non-school venues like museums.

26. Space Grant Colleges

27. Systems that allow for updating material.

28. ERCs

29. AESP

30. CORE

31. Spacelink

32. Other digital libraries

33. NASA Quest

34. Web-mission

35. Those guided by on going evaluation (formative).

36. Teachers love Space Camp.

37. Email Q&A

38. Live webcasts

39. NASA stuff on 'other' sites

40. Online communities

41. User audiences preferred communication methods: newsletters, email,
    training.

42. Associate of secondary school principals

## What delivery systems do not work?

1.  Mass mailing

```{=html}
<!-- -->
```
1.  Dissemination without follow up

2.  Webcasts without follow up

3.  Non-funded materials

4.  Mistreated info: top down marketing

5.  "What worked in the past is no longer adequate"

6.  Giving it away lowers perceived value.

7.  Passive means (pure web, booths, mass mailing)

8.  Rapid fire short term workshops

9.  Word of mouth is too slow.

10. Untargeted delivery

11. Jargon-ridden products.

12. Web based activities: too unstable, installation problems, loss of
    control.

13. Dead web links

14. Over reliance on teachers as a delivery mechanism.

15. Big monolithic projects.

16. Self-ID by teachers, they really don't know.

17. Those developed too late, in isolation, unevaluated.

18. Those that are cost-free.

19. Teachers contributing to web gratis.

# Measuring Success

## How do we currently measure success?

1.  Test scores

```{=html}
<!-- -->
```
20. Web hits, kb downloaded.

21. Papers published

22. Conferences attended

23. 3Rs (reading, writing and mathematics)

24. CD distributed

25. Teachers reached

26. Percentage increase in people reached

27. Retention rate of customer base

28. Drop out rate

29. Continued logins

30. Teachers saying what is good.

31. Peer review (self/peer evaluation)

## Are there other methods of measuring success that would be useful to implement?

1.  In immersive technology

-   Hire experts that can build evaluation tools.

-   Develop ideas during RFP process.

2.  SRI comparison group

-   Memorization: content no collaborative of communications

-   Multimedia: content collaborative communications

3.  Rubrics

4.  Change the ***test*** by changing state curriculum

5.  Clinical trials.

6.  Tracking studies.

7.  Evaluate ability to visualize a concept

8.  Metrics which are not number based

9.  Long term programs to measure skills (2-6 years)

-   Asks students what they think they will do for a career each year
    and monitor.

10. Measure discoveries made by percentages.

11. Track behavior of what they are doing

12. User message boards: feedback

13. Statistics of navigation/IP

14. COPPA Requirements: collect non-personal info

15. Unique user logins

16. Use evaluation handbook: The User-friendly Handbook for Evaluation
    of NASA Education Programs.

17. Meeting deadlines.

18. Base further funding on success.

19. More students go into SMT.

20. More students in AP courses.

21. Mimic NIH testing

## What ought to be out goals?

1.  Create new measurements now to link learning with outcomes

```{=html}
<!-- -->
```
1.  Partner but avoid stepping on toes (DoEd, AAAS, Project 61, states,
    etc.)

2.  No child left behind: Individualize.

3.  Impact (results oriented)

4.  Take the time and resources to study and learn.

5.  Apply learning to improve program design and results.

6.  Focus on niches where you can make a difference because the need is
    great and the competition minimal.

7.  Every outreach activity matters and should be counted.

8.  Meta-mission: MOUs

9.  Encourage learning at home (cars = physics, baking = chemistry)

10. Testing has a life span (law suits)

11. Ability for students to communicate in their own words.

12. Quality product for marginal population leads to quality project for
    all.

13. Develop metrics that are not number based.

14. Need criteria to measure against

15. That 53 million kids use it (Simpsons, daily use)

16. Less attrition in Natural Sciences.

17. Define product lifetime.

18. Define success

19. Have a 10-year outlook.

20. Target Congress

21. NASA SIMS

## Factors

1.  Some people think NASA should not be in the ed business.

2.  Homework not being graded

3.  Do sampling

4.  Lack of longitudinal data.

5.  Holding back games as learning tools

6.  Doing same stuff as 30 years ago.

7.  Narrowness of usual tests.

8.  Lack of science testing.

9.  Lack of emphasis on science in elementary grades.

10. Linear vs. open design-choices (\$14/month)

11. Out of date computers

12. Interactivity necessary

13. Schools do not want to be the control in experiments.

14. Public/private mix/ partnership

15. Funding things that cost too much.

16. Fear

17. Change in Administration at NASA changes priorities.

18. Culture? Who cares?

19. \$- Longevity, stability of purpose

20. Not adequately measuring understanding of phenomena.

21. Television used to "be" bad for schools now it's good.

22. Game market is big- e-learning: CD, web, downloadable

# Designing for Success

## How do people search?

1.  Online by grade and subject

```{=html}
<!-- -->
```
22. Don't care about reviews

23. Brick and mortar stores

24. Ed journals

25. Catalogs

26. Newspapers

27. Exposure: In-service training

28. Training and promotional videos

29. ERCs

30. Spacelink

31. Word of mouth

32. Clearinghouses (like ERC)

33. List servers

34. Search engines

35. Expert recommendations

36. SciLinks

37. Amazon

38. Textbook reps

39. NSTA newsletter

40. Enclosed materials

41. Conferences

42. 'Toys' with branding/promo tricks

43. Libraries

44. Internet

45. GEMS

46. EBay

47. Ads

48. Conferences and trade shows

49. Gaming

50. Direct mail: mail, email, news groups

## What can do to improve finding and adopting?

1.  Have a central portal for ed stuff (Spacelink and others don't have
    it all-unsearchable.)

```{=html}
<!-- -->
```
51. "NASA Wal-Mart"

52. Use popular search engines like Google and AltaVista

53. Nasa has a great space image but other areas are unknown.

54. Hold conferences at NASA "Do you know...you work at NASA"

55. Rebranding NASA as more than just space- rogue products.

56. Put NASA logo on NASA products.

57. Translate NASA jargon to more accessible language.

58. Consider factors of Browse vs. Search dominant users.

59. Improve usability

60. Register keywords with search engines

61. Make EDCATS browsable.

62. Everyone can have their own web page-updating is time consuming

63. SCORM- adl.org advance distribute learning, Digital Library
    projects, military, DoD, WestEd.

64. Rules for how many sites link to your site.

65. Amazon rating, relevance

66. NEA, AFT (not just NSTA)- lower level teachers don't think of
    themselves as science teachers!

67. Go through Associate of secondary school principals

68. Build in interactivity

69. Telepresence

70. Make ed games with

-   Choices

-   Addictive- constant new content

-   Emotional involvement

71. NASA should adopt an industry search engine (e.g., Yahoo)

72. Make all learners the audience.

73. Partner with entertainment industry (Apollo 13, Mission to Mars)

74. Trustworthy, peer reviewed (by the target group)

75. Parental involvement.

76. Think about national parks as dissemination sites.

77. Have reasonable model (don't always shoot for he moon)

78. Study Spacecamp: kids love it. Can you replicate?

79. Need to reach ordinary teachers, not just early adopters, and
    experts.

80. Partner with big partners: Dell, Intel, Blackboard- tap into their
    distribution.

81. Input from master teachers during development and design

82. Provide opportunities for kids to find their own sense of wonder.

83. CDs in cereal boxes.

84. Don't limit to 2 or 3 year projects

85. Smithsonian search in progress

86. NASA seal of approval.

87. Build a 'learning support system'

```{=html}
<!-- -->
```
1.  Course design tools

2.  Tutor/mentor network

3.  Feedback tools- assessment

4.  Constructional tools

5.  Instructional tools (facts/principals/simulations)

6.  Software industry

```{=html}
<!-- -->
```
1.  Often builds on the same engine

2.  Common modular tools

3.  Pairing customer w/engineering during spec phase

4.  Build in pedagogy from the beginning

5.  Journal notes.

```{=html}
<!-- -->
```
88. Mock up usability testing.

89. Reuse code

90. Start with concept map.

91. Reduce URLs in NASA ed domain

92. One person accountable for NASA web ed.

93. Navy Blue and Gold guys as model (retires visiting schools to
    recruit by presentation)

94. State Science Supervisors

## Are there thing we can do up front?

1.  PR strategy: "NY Times Teachers Section

```{=html}
<!-- -->
```
95. Plug in meta data hooks

96. Publish metric priorities

97. Conform to ed standards

98. Market research: ss, forums, needs gap analysis (who, Edmark) it's
    own field: not cheap

99. Providing tools for developers for reuse (\$1 million to develop
    quake engine)

100. No betting on horses: don't start from scratch.

101. Nature of teaching/learning needs to be incorporated in the model
     (don't need to go through teachers, students as customers)

102. Control? Not desirable any more!?

103. Provide evaluation guidelines.

104. Open source.

105. Connections and partnerships

106. Narrow the box (what do we want to develop?)

107. Provide access to data.

108. Templates for webpages.

109. Don't be too specific in RFP, kills creativity.

110. Require developers to train ERC, AESP and Space Grant personnel.

111. Include educators in development stages.

112. Ask for models of good use.

113. Publish development criteria online.

114. Cross link products.

## Are there add-on enhancements that can be "retrofitted" to existing products to extend their appeal?

1.  Let them die a natural death

```{=html}
<!-- -->
```
43. Create a lessons learned from developers database.

44. Have a librarian/archivist

45. Cross accessibility.

46. Thesaurus of information terms.

47. Accept proposals for product modification.

48. Update products.

49. NASA seal of approval

50. Evaluate and pay attention to results.

51. Cross-link "you might also be interested...." Do follow ups
    depending on what was selected.

# Random Input

## Comments not directly fitting existing categories.

1.  Develop umbrella product strategy and segmentation that RFPs should
    respond to -- including in RFP factors such as: communications plan,
    usage and measurement plans.

```{=html}
<!-- -->
```
52. Develop list of US state curriculum administrators and tie NASA
    product criteria development to match requirements as well as
    federal level as part of criteria.

53. Redefine what doing to develop teachers in science given shortage
    with the Department of Education.

54. Partner with industry to incubate businesses and technologies for ed
    tech.

55. Evaluate what dissemination must be direct to end-user and what can
    be third party in terms of effectiveness.

56. Retooling schools and teachers will be an on going capitalization
    cost.

57. Retool MASA staffing to include marketers- train them on product
    knowledge- not product experts expected to do marketing well.
    Marketing is a profession!

58. Do follow up marketing and customer service support. Don't just
    distribute in a vacuum- follow up!

59. Develop a holistic integrated product and marketing plan- similar to
    industry paired with product quality of a non-profit to get the
    benefits of effectiveness in the end user market place- systemic
    idea.

60. Consider AP course development to further accelerate the gifted and
    talented student 'scientists'.

61. Understand where the computer-applied technology ***is*** and where
    it is not beyond a textbook. For example: Computers can teach
    fractions ***intuitively*** vs. not. Whereas computers are
    equivalent to teaching vocabulary with paper- just in a computer
    format. Then use marketing messaging to position the difference for
    adaptation and use.

62. Develop integrated marketing communications plans as well as
    individual product plans.

63. Consider product development programming for K-12 schools private
    cable TV networks that is interactive and fun!

64. Develop vision for scenario planning brand technology adaptation and
    new technologies to come- for example, broadband application may
    arrive soon. Immersive technology with multi-sensory environments.

65. Show incremental change with specifically targeted group. Example:
    Target tribal universities (35 of them). They are neglected and
    improvement would be easy to measure.

66. Coordinate efforts with other government agencies. When partnering,
    make sure to maintain attribution of our distinctive contributions
    to it is traceable to NASA.

67. Need to pay more attention to informal educational opportunities.

68. Game vs. learning tool (preconception against)

-   Is learning occurring?

-   Sim-Earth & Sim-City (resource oriented game)

```{=html}
<!-- -->
```
-   Black box-don't know fundamentals algorithms-\> if could expose
    those and change-\>

-   More acceptable.

69. Why no strong ed games?

70. Holding back games as learning tools.

71. NASA SIMS (looking into partnering with the SIMS people)

72. Homes schoolers like technology

73. Plan early. Involve representatives of the users. Integrate with
    others.

74. Have a model of success in mind. Do you see a broad repertoire of
    products and services? Not all will be smash hits that in demand by
    all teachers in all classrooms. Targets: minorities, women, poverty,
    others not well served. These targets may imply limited markets.

75. Need to take time.

76. Learn from others (and ever stop learning and applying knowledge).
    Known as 'benchmarking'.

77. Know what business you are in- your niche- what makes you special-
    what "as only NASA can" really means.

78. Have a deep and deepening understanding of your clients, mission,
    content, processes, strengths, weaknesses, needs, emerging (and
    future) issue.

79. Delivery systems must provide context and process with lots of
    professional development, spread over time, with technical
    assistance.

80. Take the time and resources to study and learn.

81. Apply learning to improve program design and results.

82. Focus on niches where you can make a difference because the need is
    great and the competition minimal.