# Presentation: 457006

## NASA CUSTOMER SURVEYSUMMARY REPORTPrepared by Science Applications International Corporation (SAIC)Arlington, VirginiaAugust 1998

## Organization of Report

- Population
- Approach
- Introductory Questions
- ASTT Program Three Pillars and Ten Goals
- Level of Satisfaction with NASA’s ASTT Program
- Facilities and Services
- Technology Transfer
- Program Planning
- Level of Satisfaction with NASA
- Resource Management
- Effectiveness in Meeting National Needs
- Closing Questions – Executive Level Only

## Survey Population Data

- 	         				          Executive	    Mid-Level Manager
- Survey Population			               101		72
- _Customer Groups_
- 	Direct Industry				63		65
- 	Indirect Industry			  	38		  7
- 	Benchmark Companies		  	50		59
- 	Aviation Advisory Committee Members	11	 	  0
- 	Universities				18		  3
- 	Aviation Associations			  8		  3

## Industry Groups

**DIRECT INDUSTRY            **
- Large Civil Transport Manufacturers
- High Performance Aircraft Manufacturers
- Aircraft Engine Manufacturers
- Rotorcraft Manufacturers
- GA/Commercial/Sport/Business/Other Aircraft Manufacturers
- Launch Vehicle Manufacturers
- Launch Vehicle Engine/Propellant Manufacturers
- Aircraft Instrument/Avionics Manufacturers
- Specialized Aerospace Services
- Department of Defense Entities
- Other Federal Agencies

**INDIRECT INDUSTRY            **
- Aviation Metal/Metal Component Manufacturers
- Non-Metallic Aviation Materials Manufacturers
- Air Carriers
- Satellite Manufacturers and Other Launch Service Customers
- Universities (Major ASTT Grantees)
- Aviation Associations
- Aerospace Analysts and Advocacy Groups

## Survey Approach

- Quantitative Questions
- Statistical Analysis (Mean, Standard Deviation)
- 		Ratings from 1 to 10
- 1 to 4 - low
- 5 to 7 - medium
- 8 to 10 - high
- Qualitative Questions (open-ended)
- Followed several Quantitative Questions
- YES/NO Questions
- Sorting Items into Categories
- Comparisons with 92 and 95 Surveys

## Survey Approach Summary

- 				         Executive           Mid-Level Manager
- Method of Interview
- 	Personal Visit			86		14
- 	Telephone			15		58
- Average Interview Duration (Minutes)	56		41
- Level of Understanding (1 to 10 scale)	7.7		7.9
- 	(Interviewer Assessment)
- Level of Interest (1 to 10 scale)	8.2		7.2
- 	(Interviewer Assessment)
- ASTT Mission Area to Which Answers Apply
- 	Aeronautics			63		47
- 	Space			14		15
- 	Aeronautics and Space		24		10

## Overall Ratings by Customer Group

- _Category__	__	__	_               _Mean__	_             _Rank_
- Aeronautics Advisory Committee 	7.02		1
- Universities			6.85		2
- Government			6.69		3
- Associations/Publications		6.58		4
- Mid-Level Managers		6.55		5
- Benchmark Companies		6.53		6
- Direct Industry			6.50		7
- Executives			6.41		8
- Indirect Industry			6.36		9
- Overall				6.47

## 1.  What is your overall level of interest in the ASTT Program?

**1.  What is your overall level of interest in the ASTT Program?**

- 96 percent of the total respondents said they had a high or medium level of interest in the ASTT Program.  This level was up from the 1995 survey when the total was 92 percent.
- 79 percent of the total said they had a high level of interest.  This compares to 75 percent in the 1995 survey.
- NASA’s Aviation Advisory Committee (AAC) had the highest level of interest at 100 percent in the high category.
- University interest dropped from 100 percent in 1995 to 90 percent in 1998.
- The group that had the lowest level of interest in ASTT was Indirect Industry.
- This question had the highest mean value of all quantitative questions in the 1998 survey.

- 8.9

- 7.6

- 8.2

- Mean

## 1b.  Select all the NASA mission areas that greatly interest you.

**1b.  Select all the NASA mission areas that greatly interest you.**

- Critical Technology (Base R&T) was selected as the area of greatest interest by 73 percent of the respondents.
- Other areas of high interest were Advanced Subsonic Transport (57percent) and High-Speed Research       (51 percent).
- High Performance Aircraft (45 percent), High Performance Computing (45 percent), Space Transportation Technology   (41 percent) and General Aviation (41 percent) all had moderate interest.
- Hypersonics (33 percent ) and Rotorcraft  (33 percent) had the lowest level of interest in the specifically named categories.

- a.  Advanced Subsonic Transport
- b.  High Speed Research
- c.  High Performance Aircraft
- d.  Hypersonics
- e.  High Performance Computing
- f.  Critical Technology (R&T Base)

- g.  Facilities and Services
- h.  Space Transportation Technology
- i.  General Aviation
- j.  Rotorcraft
- k.  Other

## 2.  How well do you understand the mission of the ASTT Program?

**2.  How well do you understand the mission of the ASTT Program?**

- The level of understanding of the ASTT Program for all respondents dropped from 55 percent in 1995 to 44 percent in 1998 in the “well” category.  The mean value also dropped from 7.0 in 1995 to 6.4 in 1998.
- AAC Members had the highest level of understanding with a mean value of 8.8.
- Benchmark Customers had the second highest level of understanding with a mean value of 6.8.
- The group with the lowest level of understanding was Association/Publication with a mean value of 4.9.
- This question’s overall mean score of 6.4 ranked 14th of 30 quantitative questions.

- Mean

## 3.  How well are the current mission areas of NASA’s ASTT Program      aligned to your organization’s needs?

**3.  How well are the current mission areas of NASA’s ASTT Program****      aligned to your organization’s needs?**

- Government rated NASA’s ASTT Program mission areas best aligned to their needs with a mean value of 7.0.
- The Benchmark Companies and Direct Industry rated the mission areas next well aligned with a mean value of 6.1.
- Total scores were close to 1995’s scores, only dropping one percent from 26 percent to 25 percent in well aligned, but the poorly aligned category grew from 23 percent to 27 percent.
- Associations/Publications rated the goals least aligned with none scoring in the “well” category.
- This question’s overall mean score of 5.8 ranked 27th of 30 quantitative questions.

- Mean

## 3b.  What current mission areas of NASA’s ASTT Program are most        important to your organization? 

**3b.  What current mission areas of NASA’s ASTT Program are most****        important to your organization? **

- Organizational interest closely paralleled individual interest (Question 1b) but at a level approximately 8 to 10 percent less.
- Critical Technology (Base R&T) showed the most interest with 61 percent of the respondents.
- Another area of significant interest was Advanced Subsonic Transport (49 percent).
- High Speed Research (39 percent), High Performance Computing (38 percent), High Performance Aircraft (34 percent), Space Transportation Technology (34 percent), Facilities and Services (32 percent), General Aviation (30 percent), and Rotorcraft               (30 percent) all had moderate interest.
- Hypersonics (23 percent ) had the lowest level of interest in the specifically named categories.

## 4.  How well is NASA’s ASTT program accomplishing its mission?

**4.  How well is NASA’s ASTT program accomplishing its mission?**

- 63 percent of the total respondent group rated their general satisfaction in the middle range and most individual groups followed this pattern.
- The mean value for the 98 Survey (6.0) dropped by 0.3 as compared to the 95 Survey (6.3).
- Government Customers gave the highest rating with a mean value of 6.7.
- Mean values for Benchmark Customers was 6.1 which is slightly higher than the total population.
- Lower scores for accomplishing mission are consistent with lower scores for understanding of ASTT mission (Question 2).
- This question’s overall mean score of 6.0 ranked 24th of 30 quantitative questions.

## 4b.  What current mission areas is NASA’s ASTT Program accomplishing         the best?

**4b.  What current mission areas is NASA’s ASTT Program accomplishing****         the best?**

- Respondents typically only selected one or two items, unlike the previous selection questions where they picked several choices.
- Four mission areas were the most common selection; High Speed Research (31 percent), Advanced Subsonic Transport (29 percent), Critical Technology (R&T Base) (22 percent), and Space Transportation Technology            (21 percent).
- High Performance Computing (12 percent) received a moderate rating.
- General Aviation (8 percent); High Performance Aircraft, Facilities and Services and Rotorcraft (all at 7 percent) and Hypersonics (5 percent) are accomplished least well according to the respondents.

## 5.  How familiar were you with the Pillars and Goals before this survey?

**5.  How familiar were you with the Pillars and Goals before this survey?**

- Over two-thirds of the respondents were somewhat to very familiar with the Pillars and Goals before the survey.
- Those most familiar were the AAC Members (91 percent) followed by Benchmark Companies at 49 percent.
- The groups least familiar were Associations/ Publications with 62 percent being unfamiliar and Indirect Industry with 51 percent.
- This question’s overall mean score of 6.1 ranked 21st of 30 quantitative questions.
- There was no comparable question on the 95 or 92 surveys.

## 5b.  How did you learn about the Pillars and Goals? 

**5b.  How did you learn about the Pillars and Goals? **

- Most respondents learned about the Pillars and Goals through interaction with NASA, either through working with NASA or from Committee work.  These are indicated in the “Other” column.
- The next most frequent way of learning about the Pillars and Goals was from the NASA Administrator’s Speech.
- The least frequent way of learning about the Pillars and Goals was from the Internet.

- a.  NASA Administrator’s Speech
- b.  Pamphlet Mailed to You
- c.  Letter Introducing the Survey
- d.  From the Internet
- e.  Other

## 6.  How well are the Goals aligned to your organization’s needs?

**6.  How well are the Goals aligned to your organization’s needs?**

- Mean values of ratings for this question were closely bunched ranging from a high of 7.1 for AAC Members to a low of 6.1 for Associations/ Publications
- 42 percent of respondents ranked the Goals as well-aligned to their organization.
- This question’s overall mean score of 6.6 ranked 10th of 30 quantitative questions.

## 6b.  Which goals are best aligned to your organization’s needs?

**6b.  Which goals are best aligned to your organization’s needs?**

**Many respondents identified specific Pillars or Goals; some identified subsets of Pillars or Goals.**

## 6c.  Which important Goal, from your organization’s perspective, is missing?

**6c.  Which important Goal, from your organization’s perspective, is missing?**

- Work with DoD where commonality applies -- [need] Goal to share and integrate cost-effective R&D with DoD
- Greatly concerned about future of subsonic transport R&T.  Basic is being neglected.
- Development of future rotorcraft concepts..., working with FAA to develop infrastructure for civil transportation.
- No materials research for airframe technology for general aviation.
- Aeronautics and Earth Science not connected -- NASA ignores internal customers.
- High performance military aircraft.
- Next generation avionics.
- FANS, satellite communications technology, composite wing, wet wing.
- Reduce cost to lower Earth orbit applies to only part of mission -- need Goals that cover launch to orbit mission, transfer orbits and on issues with space probe.
- Future engineering education should be in the Goals; important to next generation.
- Economic viability of the contemplated technologies.
- Human factors research.
- Product quality.
- Fundamental flow physics.
- Infrastructure development.
- There isn’t anything missing.

**Respondents provided various ideas regarding goals that should be/could be added.**

## 7.  How well are the Goals aligned to national needs?

**7.  How well are the Goals aligned to *****national***** needs?**

- Total respondents overwhelmingly ranked NASA’s Goals as being very well aligned to national needs scoring 66 percent in the “well” category.  Total “well” and “moderately well” rating was 97 percent.
- Benchmark Customers and Universities ranked this question very high with 70 percent in the “well” category.
- The lowest “aligned well” rating was Government at 54 percent, which also had the second highest “poorly aligned” rating of 15 percent.
- Mean values closely followed the percentile ratings.
- This question’s overall mean score of 7.9 ranked 2nd of 30 quantitative questions.

## 7b.  Which goals are best aligned to national needs?

**7b.  Which goals are best aligned to *****national***** needs?**

**7c.  What important Goal, from a *****national***** perspective, is missing?**

- Many of the suggestions for National Goals were similar to those provided in Question 6c in response to missing Goals from the Customer’s organizational needs.

**Most respondents identified specific Pillars or Goals; some respondents identified subsets of Pillars or Goals.  These results closely parallel the results of Question 6b regarding the alignment to organizational needs.**

## 8.  Overall, how well are NASA’s current programs aligned to the Goals?

**8.  Overall, how well are NASA’s current programs aligned to the Goals?**

- Overall, the respondents thought that NASA’s current programs were moderately well aligned to the Goals.  However, within the Customer Groups, the rankings for this question were nearly the reverse of those observed in most other questions.
- The highest rankings were given by the Associations/Publications and the Universities with a mean value of 6.9.
- The lowest rankings within the customer groups were given by the Benchmark Customers and the Direct Industry Customers.  They each had a mean value of 6.3.
- This question’s overall mean score of 6.4 ranked near the middle at 16th of 30 quantitative questions.

- Mean

- 13

- 64

- 61

- 58

- 23

- 40

- 28

- 11

- 10

- 15

- 15

- 42

- 50

- 47

- 24

- 0%

- 20%

- 40%

- 60%

- 80%

- 100%

- 98 Benchmark

- 98 Direct

- Industry

- 98 Universities

- 98 Assoc/Pub

- 98 All

- Poorly

- Moderately

- Well

## 8b.  Which current ASTT programs are best aligned to the goals?

**Most respondents provided answers that related to specific Pillars, Goals, or ASTT Programs.**

## 9.  What is your level of satisfaction with NASA’s ASTT Facilities and      Services?

**9.  What is your level of satisfaction with NASA’s ASTT Facilities and****      Services?**

- The level of satisfaction with ASTT facilities and services dropped significantly as compared to the 1995 survey.  The mean value in 1995 was 6.8, in this survey it dropped to 6.0
- Government Customers showed the greatest satisfaction with facilities and services with a mean value of 7.0; the AAC had the second highest mean value of 6.5.
- Associations and Publications had the lowest mean value of 4.9; Indirect Industry had the next lowest mean value at 5.4.
- This question’s overall mean score of 6.0 ranked moderately low at 22nd of 30 quantitative questions.

- Mean

## 9b.  Which NASA facility and service issues are most important to        your organization?

-    _Facility or Service Issue__	_                  _Percent_
- 1. Quality of the Data	 	75
- 2. Level of Technical Expertise	68
- 3. Timeliness of the Results	63     High
- 4. Availability of Facility or Service	60
- 5. State-of-the-Art Facilities	58
- 6. Protection of the Data		42
- 7. Ease of Technology Transfer	39
- 8. Familiarity and/or Trust		38     Moderate
- 9. Charge for Using Facility or Service	36
- 10. Ownership by the U.S. Government	35
- 11. Computational Quality		32
- 12. Convenience of Facility or Service	19
- 13. Ownership by U. S. Government	10     Low
- 14. Location of the Facility or Service	  6

## 10.  Which methods of technology transfer work best for your        organization?

- 7.49

**10.  Which methods of technology transfer work best for your****        organization?**

- These results differ somewhat from results of the 1992 survey.  In that survey, customers were asked to give the various technology transfer methods a ranking from 1 to 10 with 1 being unfavorable and 10 being  favorable.  These results are shown in the right-hand columns.

- _92 Survey_

- 7.01

- _Mean_

- _Rank_

-   4

-   1

- 6.81

-   6

- 7.07

-   2

- 6.85

-   5

- 7.03

-   3

- 5.98

-   7

## 11.  How would you rate NASA’s program planning?

**11.  How would you rate NASA’s program planning?**

- Program planning was rated higher in this survey (mean = 6.0) than in the 95 survey (mean = 5.6).
- Program planning was rated much higher in this survey than in the 92 survey when 53 percent of the respondents indicated that NASA’s program planning was inadequate.
- This question was rated higher by Associations and Publications (mean = 7.3) and Universities (mean = 6.7).  It was rated lower by Benchmark Customers (mean = 5.9) and lowest  by Government Customers (5.8)
- 58 percent of respondents answered “Yes” to the question, “Does NASA adequately involve your organization in its program planning for ASTT?”
- 62 percent of respondents answered “Yes” to the question, “Does NASA plan research programs to provide results in a form, structure and format useful to you?”
- This question’s overall mean score of 6.0 ranked quite low at 23nd of 30 quantitative questions.

- Mean

## 12.  How would you rate NASA’s Outreach and External Communication?

**12.  How would you rate NASA’s Outreach and External Communication?**

- Mean

- Outreach and External Communication was rated slightly higher in this survey (mean = 6.3) than in the 95 survey (mean = 6.2).
- The AAC (mean = 7.4) and University groups (mean = 7.3) rated this question high.  Benchmark Customers (mean = 6.4) and Direct Industry Customers (mean = 6.3) rated the question lower than the overall mean of 6.3.
- 68 percent of respondents answered “Yes” to the question, “Does NASA ask for your input on plans and issues important to your organization?”
- 67 percent of respondents answered “Yes” to the question, “Does NASA use your input?”
- 65 percent of respondents answered “Yes” to the question, “Does NASA provide feedback on plans/issues important to your organization?”
- This question’s overall mean score of 6.0 ranked near the middle at 17th of 30 quantitative questions.

## 13.  What is your overall level of satisfaction with NASA?

**13.  What is your **_**overall**_** level of satisfaction with NASA?**

**13b.  In what areas are you most satisfied with NASA?**
- Increasing attentiveness to the needs of U. S. industry; improved communication ... in executing contracts.
- Good job getting input from industry; good facilities and people.
- Quality work, quality reports, results and availability.
- Headquarters' interest in having a positive national impact through technology development.
**13c.  In what areas are you **_**least**_** satisfied with NASA?**
- Inability to do meaningful multi-year planning.
- Inability to change with changing needs of industry.
- Technology transfer - cost to industry; complex paperwork; “nightmare” multi-page licenses.
- NASA competition with industry.

- Mean

- The Customer’s Overall Level of Satisfaction with NASA was rated slightly higher in this survey (mean = 6.6) than in the 95 survey (mean = 6.4).
- There was a significant difference (p<0.05) in the mean values of the Executives (mean = 6.3) and the Mid-Level Managers (mean = 7.0) on this question.
- This question’s overall mean score of 6.6 ranked near the top one-third of the questions at 11th of 30 quantitative questions.

## 14.  What is your overall level of satisfaction with NASA’s ASTT products?

**14.  What is your **_**overall**_** level of satisfaction with NASA’s ASTT products?**

**14b.  With what products were you **_**most**_** satisfied?**
- 					            	     _Responses__	_	                      _Responses_
- High Speed Research/Travel & High Speed Civil Transport        23          Technical Reports/Symposia      13 Computational Fluid Dynamics/Computational Analysis	             17          Use of NASA Facilities               12
- Research and Technology Base		             15          Design and Analysis Tools         10
- Advanced Subsonic Transport/Subsonic Research	             14          Reduce Noise	      10
- Experimental Aircraft (X-33, X-34, X-36, Hyper-X, Future-X)       13          Rotorcraft/Tiltrotor	        9
**14c.  What products were you **_**least**_** satisfied with?**
- Those that don’t have a cooperative character...
- Full-cost [funding] resulting in budget cuts and diminished expectations.
- Technology transfer between larger corporations and NASA and government (as on HSR program).

- 6.4

- 7.1

- 6.9

- 6.3

- Mean

- The Customer’s Overall Level of Satisfaction with NASA’s ASTT Products was rated moderately high.  Overall, 85 percent of the respondents rated their satisfaction as “medium” or “high.”
- University Customers (mean = 7.1) and AAC Customers (mean = 6.9) had a higher rating than the overall population.  Direct Industry Customers (mean = 6.3) had a lower rating than the overall population.
- This question’s overall mean score of 6.4 ranked in the middle at 15th of 30 quantitative questions.

## 15.  How does your organization’s overall level of satisfaction with NASA compare with its satisfaction 3 to 6 years ago (before the last surveys)?

**15.  How does your organization’s overall level of satisfaction with NASA compare with its satisfaction 3 to 6 years ago (before the last surveys)?**

**15b.  In what area(s) has your level of satisfaction **_**increased**_**?**
- Alignment of research to the Pillars; six years ago there was inefficiency and lack of focus.
- Shift from micro-management to a partnership relationship.
- Involvement and visibility of NASA plans and goals.
- Just in being part of their activities; we’re in partnership with them; our relationship has grown by leaps and bounds.
**15c.  In what area(s) has your level of satisfaction **_**decreased**_**?**
- Systems integration emphasis, computational resources, and test facility readiness.
- Availability of wind tunnel test facilities; reduced emphasis on R&T base for aeronautics.
- I think the most important is the condition of NASA’s deteriorating facility support.
- Difficulty in dealing with NASA as a customer.

- 6.5

- 6.6

- 6.3

- The Customer’s felt that their satisfaction with NASA had improved over the last 3 to 6 years.  They rated this question slightly higher in the current survey (mean = 6.5) than in the 95 survey (mean = 6.3).
- 85 percent  of respondents rated this question as “medium” or “high.”  Benchmark Customers (mean = 6.6) provided slightly higher ratings, and University Customers (mean = 6.1) provided slightly lower answers than the overall population.
- This question’s overall mean score of 6.5 ranked slightly above the middle at 13th of 30 quantitative questions.

## 16.  NASA provides key laboratory, computing, and test facilities.

**16.  NASA provides key laboratory, computing, and test facilities.**

**16b.  What NASA facilities do you view as being essential to the conduct of leading edge R&D?**
**The overwhelming answer to this question was wind tunnels, followed by computational resources.  The comments below characterize these and other comments.**
- Modern wind tunnel test facilities and world class computational resources.
- Space test facilities at Stennis Center.
- Long duration test and space exposure tests.
- Airborne test beds, simulation and flight facilities.

- 7.6

- 8.2

- 7.5

- 8.1

- Mean

- The respondents strongly felt that NASA provides key facilities.
- 93 percent  of respondents rated this question as “partially agree” or “agree” and 62 percent rated it “agree.”  Government Customers (mean = 8.2) and University Customers (mean = 8.1) provided high ratings, and Benchmark Customers (mean = 7.5) provided slightly lower answers than the overall population.
- This question’s overall mean score of 7.6 ranked near the top at 3rd of 30 quantitative questions.

## 17.  NASA downsizing has affected its ability to support its customers.

**17.  NASA downsizing has affected its ability to support its customers.**

**17b.  In what ways, if any, has NASA downsizing affected your organization?**
**Some respondents felt that NASA downsizing had a negative effect:**
- Foundational research is not getting done.
- Primarily, loss of key NASA expertise and missing expertise is not being replaced.
- We are  starting to reduce our alliance on technologies developed by NASA.
**Other respondents felt that NASA downsizing had positive effects:**
- Very positive effect - less micro-management - more of a partnership.
- Haven’t downsized enough.
- Helped since NASA is now less competitive with industry.
- Its been positive; it’s made them take a hard look at programs and ... it helped in that they are more focused.

- 5.8

- 7.0

- 5.9

- 6.4

- Mean

- The respondents were equally divided in response to this question.
- 65 percent  of respondents rated this question as “partially agree” or “agree” and 33 percent rated it “agree.”  AAC Customers (mean = 7.0) and University Customers (mean = 6.4) provided high ratings, while Benchmark Customers (mean = 5.9) were more inclined to disagree than the overall population.
- This question’s overall mean score of 5.8 ranked near the bottom at 26th of 30 quantitative questions indicating significant disagreement that downsizing has affected NASA’s ability to support customers.

**18.  NASA is a key contributor to the education of scientists and engineers.**

**18b.  In what ways can NASA improve the education and training of our next generation of scientists and engineers? **
**The main themes were more funding, cooperative programs, and more internships.  Also NASA involvement in university, secondary and elementary schools was identified as being important.**
- Strengthen university relationships that include industry.
- Put more money into foundational research in universities best equipped to do that type of research.
- Re-instituting university programs programs where NASA provides seed money to university engineering programs.
- Be more involved in [recommending] curricula of engineering schools.  Have more open house events to stimulate younger students.
- Public relations programs aimed at younger folks to kindle their interest early on.
- Target younger children, even in elementary grades.

- 7.0

- 8.0

- 6.7

- 6.9

- Mean

- The respondents strongly felt that NASA is a key contributor to the education of scientists and engineers.  The respondents of the current survey (mean = 7.0) gave higher ratings to this question than did the respondents of the 95 survey        (mean = 6.7).
- 84 percent  of respondents rated this question as “partially agree” or “agree” and 52 percent rated it “agree.”  University Customers (mean = 8.0) provided high ratings, and Benchmark Customers (mean = 6.9) provided slightly lower ratings than the overall population.
- This question’s overall mean score of 7.0 ranked near the top at 4th of 30 quantitative questions

**19.  NASA cooperates with other organizations to make cost-effective use of limited R&D resources, skills and facilities.**

**19b.  In what ways can NASA improve cooperation with other organizations?**
**More communication and coordination was the primary theme of the responses.**
- Talk to and coordinate with DoD -- and don’t just talk -- [perform] some deeds.
- Expand levels of cooperation with other government agencies as well as corporate world; develop and participate in [joint] technical programs.
- By helping its personnel to understand the realities and mechanisms of the commercial marketplace.
- Improve communication; become more aware of industry programs to create cooperative programs.
- [Be] more pro-active in searching for organizations to cooperate with.

- 6.3

- 7.0

- 6.1

- 6.3

- Mean

- The respondents were mixed in their rating of NASA’s cooperation with other organizations.  The respondents of the current survey (mean = 6.3) gave slightly higher ratings to this question than did the respondents of the 95 survey (mean = 6.1).
- 78 percent  of respondents rated this question as “partially agree” or “agree” and 35 percent rated it “agree.”  AAC Customers (mean = 7.0) provided higher ratings than the overall population and Benchmark Customers (mean = 6.3) provided ratings equal to the overall population.
- This question’s overall mean score of 6.3 ranked in the lower half at 19th of 30 quantitative questions

**20.  The ASTT program effectively balances fundamental (R&T base) research with its focused-program research.**

- Fundamental research; if fundamental research is done, the focused-program research will be done better.
- [NASA] should be looking at 5 to 10 year technology; universities should be looking at 10 to 15 year technology; industry should look at 2 to 5 year technology.
- More fundamental research; industry better understands market need; leave focused-program to industry.
- More focused-program [research] because it is a more effective use of resources.
- Focused-program research to become more efficient -- pull research rather than push research.
- More focused-program research, because it offers more incentive for industry involvement that would lead to commercial products.

- 5.9

- 7.0

- Mean

- The respondents were mixed in their rating of NASA’s balance of fundamental versus focused-program research.  Government Customers     (mean = 7.0) provided much higher ratings than the overall population.
- This question’s overall mean score of 5.9 ranked in the lower half at 25th of 30 quantitative questions.
- Respondents of the current survey called for significantly more fundamental research than did the respondents of the 92 survey.

**20b.  Should NASA pursue more fundamental research or focused-program research, and why? **
**Some thought balance was good; others presented arguments for both fundamental and focused research.**

**21.  The ASTT program effectively balances analytical, computational, ground-based experimental and flight research.**

**21b.  Should NASA change its mix of analytical, computational, ground-based experimental and flight research, and why? **
**Most respondents felt the balance was good now, but the mix needs to be constantly scrutinized.  There was some call for more flight testing.**
- Yes, not enough flight research for aeronautics; maintain ground-based experimental facilities.
- More emphasis on flight research [ will produce] more rapid progress.
- In general, they are pretty well balanced, but caution is necessary on flight research to make sure the costs/benefits are scrutinized.
- The mix is fine and doesn’t need to be changed.

- 6.8

- 7.6

- 5.9

- 6.9

- Mean

- The respondents generally approved of NASA’s mix of research methods.  The overall mean was quite high at 6.8.
- There was some variation within the customer groups.  AAC Customers (mean = 7.6) provided higher ratings than the overall population; Benchmark Customers (mean = 6.9) provided slightly greater than the ratings of the overall population; Associations and Publications gave a low rating.
- This question’s overall mean score of 6.8 ranked quite high (in the upper quarter) at 6th of 30 quantitative questions

**22.  The ASTT engineers and scientists play a key role in the development of innovative or enhanced technical solutions.**

**22b.  In what areas have NASA engineers & scientists played a key role?**
- Human factors, structures, propulsion and computational fluid dynamics -- Space travel is the biggest
- Atmospheric physics; earth physical science -- Development of high performance rocket propulsion
- Lifting-body ceramic and metallic thermal protection systems; supercritical wing; coke-bottle design of aircraft
- Composite technology; computer fluid analytical (CFA) tools; and noise research -- Atmospheric modeling
**22c.  In what area have they **_**not**_** played a key role?**
- They’ve tried to play a role in system integration, but unsuccessfully -- Solution of critical operational problems
- General aviation and airline safety -- Development of cost effective launch systems
- Have not achieved leadership in information technology, for example, redundant/fault-tolerant information systems
- Reducing cost of bringing advanced technology to market and reducing the time to certification of key technologies

- 7.0

- 7.3

- 6.9

- 7.3

- Mean

- The respondents generally felt that NASA’s engineers and scientists play a key role in finding technical solutions.  The overall mean was quite high at 7.0.
- There was only slight variation within the customer groups as the mean values ranged from 6.9 for Benchmark Customers to 7.3 for University and Government Customers.
- This question’s overall mean score of 7.0 ranked quite high (in the upper quarter) at 5th of 30 quantitative questions

**23.  How effective is NASA’s ASTT program in performing fundamental research?**

- 6.3

- 7.8

- 6.0

- 6.7

- Mean

- As compared with the 95 survey(which contained two question relating to performance of fundamental research) the 98 survey showed much lower mean values.  The 98 survey had a mean of 6.3.  The 98 survey had a mean score of 7.1 for effectiveness and a 7.5 score for contributions to national needs.
- The customer groups presented varied results in rating the effectiveness NASA’s ASTT Program in performing fundamental research.  The mean values ranged from 6.0 for Benchmark Customers to 7.8 for Associations and Publications.
- This question’s overall mean score of 6.3 ranked in the lower half at 18th of 30 quantitative questions.

- 7.1

- 7.5

**24.  How effective is NASA’s ASTT Program in improving national security?**

- 5.1

- 5.4

- 5.5

- 4.8

- Mean

- As compared with the 95 survey the 98 survey scored much lower mean values.  The 98 survey had a mean of 5.1.  The 98 survey had a mean score of 6.0 for contribution to national needs.
- The customer groups presented very low results in rating the effectiveness NASA’s ASTT Program in improving national security.  The mean values ranged from 4.5 for Mid-Level Managers to 5.4 for AAC Members.
- The statistical data showed there was a significant difference in the mean values of the Executive Customers (mean = 5.5) and the Mid-Level Managers (mean = 4.5).  For this question, there is a high likelihood that the two customer groups represent different population groups.
- This question’s overall mean score of 5.1 ranked lowest in the entire survey at 30th of 30 quantitative questions.

- 4.5

- 6.0

**25.  How effective is NASA’s ASTT Program in developing/ transferring technologies for economic competitiveness?**

- 5.8

- 6.5

- 5.8

- 6.4

- Mean

- As compared with the 95 survey the 98 survey scored much lower mean values.  The 98 survey had a mean of 5.8 for effectiveness.  The 95 survey had a mean score of 6.9 for effectiveness and 6.6 for contribution to national needs.
- The customer groups had generally low mean values in rating the effectiveness NASA’s ASTT Program in developing/transferring technologies for economic competitiveness.  The mean values ranged from 5.8 for Benchmark Customers to 6.5 for University Customers.
- This question’s overall mean score of 5.8 ranked in the lowest quarter of the survey at 28th of 30 quantitative questions.

- 6.6

- 6.9

**26.  How effective is NASA’s ASTT Program in maintaining superiority of US aircraft and engines?**

- 6.7

- 7.4

- 6.7

- 7.1

- Mean

- As compared with the 95 survey the 98 survey scored slightly higher mean value.  The 95 survey had a mean of 6.6 for national goals achievement.  The 98 survey had a mean score of 6.7 for effectiveness.
- The customer groups had moderately high mean values in rating the effectiveness NASA’s ASTT Program in maintaining the superiority of U. S. aircraft and engines.  The mean values ranged from 6.1 for Government Customers to 7.4 for University Customers.
- This question’s overall mean score of 6.7 ranked in the highest third of the survey at 9th of 30 quantitative questions.

- 6.1

- 6.6

**27.  How effective is NASA’s ASTT Program in developing a superior, affordable, global air transportation system?**

- 5.7

- 6.7

- 6.4

- 6.5

- Mean

- As compared with the 95 survey the 98 survey scored a lower mean value.  The 95 survey had a mean of 6.2 for national goals achievement.  The 98 survey had a mean score of 5.7 for effectiveness.
- The customer groups had very low mean values in rating the effectiveness NASA’s ASTT Program in developing a superior, affordable global transportation system.  The mean values ranged from 5.6 for Benchmark Customers to 6.7 for AAC Members.
- This question’s overall mean score of 5.7 ranked very low in the survey at 29th of 30 quantitative questions.

- 5.6

- 6.2

**28.  How effective is NASA’s ASTT Program in developing superior, affordable, space launch systems**?

- 6.1

- 6.2

- 6.6

- 5.1

- Mean

- As compared with the 95 survey the 98 survey scored a slightly lower mean value.  The 95 survey had a mean of 6.4 for contributions to national needs.  The 98 survey had a mean of 6.1 for effectiveness.
- The customer groups had very mixed mean values rating the effectiveness in developing superior, affordable space launch systems.  The mean values ranged from 5.1 for AAC Members to 6.6 for Mid-Level Managers.  The AAC Members gave this question their lowest rating  in the 98 survey.
- The statistical data showed there was a significant difference in the mean values of the Executive Customers (mean = 5.8) and the Mid-Level Managers (mean = 6.6).  For this question, there is a high likelihood that the two customer groups represent different population groups.
- This question’s overall mean score of 6.1 ranked low in the survey at 20th of 30 quantitative questions.

- 5.8

- 6.4

- There was one question on the 95 survey regarding contributions to national needs in space.  The 95 survey question and the 98 survey question are slightly related in that regard and a comparison is presented herein.

**29.  How effective is NASA’s ASTT Program in ensuring long-term environmental compatibility of aerospace systems?**

- 6.7

- 7.4

- 6.6

- 7.1

- Mean

- There were two questions on the 95 survey dealing with environmental technology.  One related to contribution to national needs and the other related to national goals achievement.
- The 98 survey scored between the two 95 survey questions.  The 95 survey had a mean of 6.4 for national goals achievement 6.8 for contribution to national needs.  The 98 survey had a mean score of 6.7 for effectiveness.
- The customer groups had generally high mean values in rating the effectiveness NASA’s ASTT Program in ensuring long-term environmental compatibility of aerospace systems.  The mean values ranged from 6.6 for Benchmark Customers to 7.4 for University Customers.
- This question’s overall mean score of 6.7 ranked in the upper third of the survey at 8th of 30 quantitative questions.

- 6.4

- 6.8

**30.  How effective is NASA’s ASTT Program in making partnerships with government, industry, and universities?**

- 6.8

- 6.9

- 6.7

- 6.9

- Mean

- There were two questions on the 95 survey dealing with partnerships.  One related to effectiveness and the other related to national goals achievement.
- The 98 survey scored between the two 95 survey questions.  The 95 survey had a mean of 6.3 for national goals achievement and 7.0 for effectiveness.  The 98 survey had a mean score of 6.8 for effectiveness.
- The customer groups had generally high mean values in rating the effectiveness NASA’s ASTT Program making partnerships with government, industry and universities.  The mean values were tightly bunched and ranged from 6.5 for Indirect Industry Customers Customers to 6.9 for several customer groups.
- This question’s overall mean score of 6.8 ranked in the upper quarter of the survey at 7th of 30 quantitative questions.

- 6.3

- 7.0

**31.  How effective is the **_**overall**_** ASTT contribution to US R&D needs?**

**31b.  What can ASTT do to increase its overall effectiveness in contributing to US R&D needs?**
**The main themes were more funding, more fundamental research, and working more closely with customers.**
- More stable, long term programs; more emphasis on validation.
- NASA needs to excel in the use of critical research and development methods with both industry and other branches of the government, especially DoD and FAA.
- Maintain a high level of communication in trying to understand industry’s needs.
- Perform better market research to validate the need  for the research.
- Continue looking at business-based metrics; watch U. S. competitive needs.

- 6.6

- 6.9

- 7.3

- 6.6

- Mean

- The respondents felt that NASA is an effective contributor to U. S. research and development needs.  However, the respondents of the current survey (mean = 6.6) gave lower ratings to this question than did the respondents of the 95 survey        (mean = 7.3).
- 88 percent  of respondents rated this question as “partially agree” or “agree” and 31 percent rated it “agree.”  AAC Members (mean = 6.9) provided slightly higher ratings than the overall population. Benchmark Customers (mean = 6.6) provided ratings consistent with the overall population.
- This question’s overall mean score of 6.6 ranked near the middle at 12th of 30 quantitative questions

**32.  Has NASA’s planning worsened or improved over the last few years?**
**(Asked to Executive Customers only)**

**32b.  What can NASA do to improve its planning?**
- Provide a forum that goes beyond workshops and actually sets up programs -- smaller group of key industry participants to boil down ideas into programs; do more feasibility studies with industry before launching a program.
- Be more responsive to industry’s needs -- not so much planning as it is attitude.
- More planning in cooperation with DoD elements in aeronautics and space.
- Better execution of plans.
- Match planning to budget; avoid stop - starts in programs due to budget fluctuations.
- Maintain commitments once established.
- Narrow focus -- be better at fewer things.

- 6.4

- 7.3

- 5.9

- 6.6

- Mean

- Most respondents (51 percent) felt that NASA’s planning had stayed the same over the last few years.  However, more respondents (32 percent) felt that planning had improved as compared to those who felt that it had worsened (18 percent).
- AAC Members (mean = 7.3) provided significantly higher ratings than the overall population 	(mean = 6.4).  Benchmark Customers (mean = 6.6) provided ratings slightly higher than the overall population.  Indirect Industry Customers 	(mean = 5.9) rated this question much lower than the overall mean.
- This question was not included in the count of 30 quantitative rating questions because it was asked only of Executive Customers.  However, the overall mean score of 6.4 ranked near the middle of the 30 quantitative questions

## 33.  Have you participated in a previous NASA Aero Survey?

- 21 Executive Level respondents (out of a total of 101) participated in the 1992 survey.
- 35 Executive Level respondents participated in the 1995 survey.
- 13 Executive Level  respondents participated in both prior surveys.

**33b.  If you participated in the 1992 and/or the 1995 survey, did the results of the survey in any way change your organization’s relationship with NASA?  If so, how?**
**The responses were mixed concerning changes in relationships with NASA.**
- Yes, between 1992 and 1995 industry became more involved in planning through workshops.
- Yes, helped put more relevancy in NASA which led to Pillars and Goals.
- Yes, increased efforts in joint industry/NASA planning.
- Created an awareness of the advantages of working with NASA.
- No apparent impact -- however, NASA should be praised for attempt.
- No, I don’t know if I ever saw results of the survey.

**34.  Compared to previous NASA Aero Surveys (1992 and 1995), how would you rate this one?  ****(Asked to Executive Customers only)**

**34b.  How can future surveys be improved? **
- Offer opportunity to get into specifics in different areas of ASTT.
- Have Pillars and Goals ahead of time.
- Need more questions on how NASA measures their effectiveness against  customer need.
- ... A few weeks before the survey, provide a list of fundamental areas that will be discussed.
- Survey well done; it can’t be improved.
- By dividing them into specific areas, such as:  global civil aviation, space transportation, etc.
- Continue the in-person interviews.
- Emphasize more narrative questions and answers.  There are techniques to analyze narrative answers.

- 6.9

- 7.1

- 6.6

- 6.8

- Mean

- Most respondents (95 percent) rated the 98 survey as “medium” or “high.”
- Indirect Industry Customers (mean = 7.1) provided slightly higher ratings than the overall population	(mean = 6.9).  Benchmark Customers (mean = 6.8) provided ratings slightly lower than the overall population.  Government Customers (mean = 6.6) also rated this question lower than the overall mean.
- This question was not included in the count of 30 quantitative rating questions because it was asked only of Executive Customers.  However, the overall mean score of 6.9 ranked in the top quarter of the 30 quantitative questions

## 35.  For the next survey, what new question(s) should be asked?

**35.  For the next survey, what new question(s) should be asked?**

**There are a wide range of suggested questions, but respondents were most interested in rating NASA’s performance and follow through.  They also wanted to be involved in future NASA planning and strategy development processes.**
- Specific questions within the mission areas — a two-level questionnaire.
- What changes would you make to improve NASA effectiveness in achieving its mission?  What diverts or interferes with NASA being effective?
- Break into space and aeronautical segments.
- Are NASA's expenditures properly balanced between programs?
- More questions aimed at university programs, training, and engineers.
- Rate performance of different NASA levels: Center, HQ Staff, and Administration
- A lot of the questions from this survey are the right ones.  Maybe some about how we are doing on the implementation of the recommendations would improve and smooth the survey effort.  Kind of the "big stick" approach.
- Pretty complete.  Presupposes that interviewee knows the breakdown of the budget.
- Ask some fundamental questions on US aeronautics policy and what would affect the customer's predilection to using technology.
- Questions about individual center's performance, especially, performance at headquarters.
- What specific areas do you work with or have worked with in the past year with NASA?  How frequently do you interact with NASA?

## 36.  What do you want to say to NASA administrator about ASTT Program?

**36.  What do you want to say to NASA administrator about ASTT Program?**

- Hold the course, balance is good, good support from industry -- don't overreact to public statements.
- That the ASTT Program is the most efficient dollars NASA spends.  The ASTT industry/NASA roles are a more appropriate approach for space development (not including exploration) than currently allocated.  For all the criticism, it’s much more efficient when industry contributes funds and resources collaboratively.
- Don't sacrifice aeronautics work due to pressure of financing space transportation or other space activities.
- Strengthen university programs to develop next generation scientists and engineers.
- The subsonic section has inadequate advocacy with the public and is not on par with the space activity.
- Thank you for covering the wide range in ASTT from general aviation to high-speed aeronautics.
- We are on a good course.  Things have certainly been improving.
- NASA should consider and address specific needs in industry, especially increased competitiveness.  Keep up research in propulsion area and noise reduction area.  Emphasize creature comforts and safety in GA.
- It needs a heavier emphasis on space technology.
- Goals are good.  NASA needs to open up as going through goals.  Allows for realistic programs to achieve goals.
- I'm very happy to see that the ASTT is stronger focused and more dynamic.  I like the way the program is now and not the way it was before.
- More cooperation with military, there's been an attempt but not "tight" enough.
- NASA should focus on space and let the aeronautics industries handle aeronautics research.
- DoD is interested in being more closely related with NASA than in past.  Could use a true personnel exchange.
- Loss of key personnel, key capability, need to recruit the best personnel to stay on top of programs.
- Quit using taxpayers’ money to fund competition because you are discouraging private investment ...
- Don't let large industry drive high-speed civil.  Follow the recommendations of National Research Council on HSCT. Is it necessary for NASA to be so broad? Aeronautics should stick to aeronautics ...
- Job well done!!
- Keep providing results to hold confidence of Congress to hold the program.  But, reach out to industry and identify specific programs to which those funds should be targeted.