# Presentation: 150932

## NASA AERONAUTICS AND SPACE TRANSPORTATION TECHNOLOGY

**1998 CUSTOMER SURVEY**
**Science Applications International Corporation**
**Arlington, Virginia**
**September 1998**

## PRESENTATION OBJECTIVES

**Familiarize NASA Management with Survey Process**
**Review Key 1998 Survey Results and Findings**
** **
**Compare the 1992, 1995 and 1998 Survey Results**
**Discuss OASTT Actions Resulting from Survey**

## SURVEY APPROACH - HISTORICAL PERSPECTIVE

**   **_**1992**__**	**__**1995**__**	**__**1998**_
_**Performed by**_
**   ****Gallup Associates****	****Arthur D. Little****	****Science Applications**
**   ****Nysmith Associates****	****Associates****	****International Corporation**
**	****(sub to SAIC) ****	****(SAIC), Mason-Dixon**
**	****	****Market Research (sub)**
_**Organizations Surveyed**_
**   ****25 Direct Customers****	****44 Direct Customers****	****33 Direct Customers**
**   ****3 Indirect Customers****	****34 Indirect Customers****	****30 Indirect Customers**
_**Interviews Conducted**_
**   ****37 Executive Interviews****	****121 Executive Interviews****	****101 Executive Interviews**
**     ****(all face-to-face)****	****  (103 face-to-face, 18 phone)****	****  (86 face-to-face, 15 phone)**
**   ****242 Mid-Level Interviews****	****No Mid-Level Interviews****	****72 Mid-Level Interviews**
**     ****(all phone)****	****	****  (14 face-to face, 58 phone)**

## SURVEY APPROACH - TYPES OF QUESTIONS

_**1992 Survey**_
**Customer Perceptions of NASA Aeronautics**
**Rating of Research Centers**
**Technology Transfer**
**Future Technological Developments**
**Customer Relationships**

_**1995 Survey**_
**Customer Understanding and Interest**
**Customer Level of Satisfaction**
**Effectiveness of Program**
**Contributions to National Needs**
**Achievement of National Goals**
**Economic Impact**
**Policy on Foreign Technology Transfer**

_**1998 Survey**_
**Customer Understanding and Interest**
**Alignment of Mission Areas**
**Understanding and Effectiveness of Pillars and Goals**
**Customer Level of Satisfaction**
**Research Methodology (Facilities, Services, Contributions, Balance)**
**Program Effectiveness**
**Program Planning**
**Survey Suggestions**
**Inputs for the Administrator**

## SURVEY POPULATION

- Large Civil Transport Manufacturers
- High Performance Aircraft Manufacturers
- Aircraft Engine Manufacturers
- Rotorcraft Manufacturers
- GA/Commercial/Sport/Business/Other Aircraft Manufacturers
- Launch Vehicle Manufacturers
- Launch Vehicle Engine/Propellant Manufacturers
- Aircraft Instrument/Avionics Manufacturers
- Specialized Aerospace Services
- Department of Defense Entities
- Other Federal Agencies
_**Indirect Industry**_
- Aviation Metal/Metal Component Manufacturers
- Non-Metallic Aviation Materials Manufacturers
- Air Carriers
- Satellite Manufacturers and Other Launch Service Customers
- Universities (Major ASTT Grantees)
- Aviation Associations
- Aerospace Analysts and Advocacy Groups

**SURVEY POPULATION**

**UTC/Sikorsky(B)**
**Textron/Bell (B)**
**Boeing (B)**
**Robinson (B)**
- Enstrom

- Executive Level
- Mid-Level Manager

- ASTTAC Member

_**Population Sub-Categories**_
**Direct Industry** - Aerospace Organizations That Directly Use NASA Products
**Indirect Industry** - Aerospace Organizations That Are Indirectly Involved with NASA
**Benchmark Companies (B) **- Market Leaders in Their Industry Segment
**Executive** - Person Who Makes or Influences Company R&D Policy
**Mid-Level Manager** - Person Who Implements Company R&D Policy
**ASTTAC Member**

## SURVEY POPULATION COMPARISON

_**DIRECT CUSTOMERS**__**	**__**	**__**	**_**             **_**1992**_**          **_**1995**_**      **_**1998**_
- 1.   Large Civil Transport Manufacturers			35	  8	23
- 2.   Military Aircraft Manufacturers				84	14	24
- 3.   Aircraft Engine Manufacturers				40	11	20
- 4.   Rotorcraft Manufacturers				20	  5	16
- 5.   GA, Commuter, Sport, Business and Other Aircraft Mfgrs.	10	10	14
- 6.   Aircraft Instrument and Avionics Manufacturers		  9	  9	  8
- 7.   Launch Vehicle Manufacturers				  --	 --	  3
- 8.   Launch Vehicle/Engine/Propellant Manufacturers		  --	 --	  5
- 9.   Specialized Aviation Service Companies		  --	  --	  4 	  2
- 10. Computer System/Component Manufacturers		  --	  4	  --
- 11. Department of Defense Entities			19	12	  6
- 12. Other Government Agencies (FAA, NTSB, DOE, NIST)	19	  7	  7
- _	__	__	__	__	_          Subtotal        236	84             128
_**INDIRECT CUSTOMERS**_
- 13. Aviation/Metal/Metal Component Manufacturers		  --	  1	  2
- 14. Non-Metallic Aviation Materials Manufacturers		  --	  3	  4
- 15. Air Carriers					  6	  5	  3
- 16. Satellite Manufacturers and Other Launch Service Customers	  --	  --	  2
- 17. Automobile/Auto Engine Manufacturers			  --	  2	  --
- 18. Universities (Major ASTT Grantees)			  --	12	21
- 19. Aviation Associations				  --	12	11
- 20. Aerospace Analysts and Advocacy Groups		  --	  2	  2
- _	__	__	__	__	_          Subtotal	  6	37	45
**	****	****	****	****	**

## SURVEY RESULTS -- “GOOD NEWS”

**Most respondents were very positive in their attitude and response to ASTT possibly indicating that customers feel ASTT is generally doing the right things.  Opportunities to take the survey and obtain results were generally felt to be indicators of ASTT’s interest in good relations/communications with customers.**
** **
**Pillars and Goals are very well-aligned to national needs and generally well-aligned to customer’s organization needs.**
**Ratings related to satisfaction with ASTT (overall satisfaction, program planning, outreach and communication, and satisfaction compared to 3-6 years ago) were higher than in the 1995 survey.**
** **
**NASA  facilities (laboratory, computing and test facilities) continue to be perceived as important national assets and are greatly appreciated by most customers, **_**but**_** there is concern with long term facility support and improvements.**

## SURVEY RESULTS -- “OPPORTUNITIES FOR IMPROVEMENT”[Customers/Areas where average ratings were less than 5]

**Associations and Publications Customer Group had consistently lower ratings than other groups (One half of the total “poor’ ratings were given by this group)**
**	**
**	****Understand Mission of ASTT (4.9)**
**	****Goals Aligned to Your Organization’s Need (4.5)**
**	****Familiarity with Pillars and Goals (4.4)**
**	****Satisfaction with ASTT Facilities and Services (4.9)**
**	**
**Indirect Industry Group (one rating below 5 compared to 3 in 1995)**
**	****Familiarity with Pillars and Goals (4.8)**
**Question 24: Rate NASA ASTT effectiveness in improving national security.**
** **
**Direct Industry (4.7)**
**Government (4.8)**
**Benchmark (4.8)**
**[Department of Defense (4.8)]**

## SURVEY RESULTS -- ASTTAC MEMBERS

**Ratings from the ASTTAC Members continue to be consistently higher than those from all other groups.  (Mean score of 7.0 compared to 6.5 overall.)**
** **
**Possible Conclusion:  More informed customers generally better understand NASA planning and processes and consequently give higher ratings.**
** **
**Ratings from the ASTTAC Members were somewhat different than those from the overall population.**
** **
**Possible Conclusion:  Need a broader picture than just ASTTAC to represent the overall Customer Base.**
** **
**ASTTAC rated one question very low compared to overall population:**
**“****How effective is NASA’s ASTT Program in developing superior, affordable space launch systems?”**

## SURVEY RESULTS - PILLARS AND GOALS

_**Mean**__**	**__**Question**_
** ****7.9****	****How well are Goals aligned to national needs?**
** ****6.6****	****How well are Goals aligned to your organization’s needs? ****	**
**	****	****    ****	****	****	****	****(Overall Survey Mean - 6.47)**
** ****6.4****	****How well are NASA’s programs aligned to the Goals?**
** ****6.1****	****How familiar were you with the Pillars&Goals before Survey?**

**Notes:**
Objective of slide is to present survey results of Pillars and Goals questions.
1.  Ratings given to alignment with national needs were very high (mean=7.9,
second highest rating in survey.)
2.  Ratings given to alignment with respondents organizational needs was somewhat lower but still above the mean for all questions in the survey(mean = 6.6, ranked 10th in 30 questions).  This result is not surprising because organizations represented in the survey had many product/services that were not related to NASA ASTT technology.
3.  Ratings given to alignment of goals with NASA’s programs was slightly below the mean for all survey questions (mean = 6.4, ranked 16th of 30 questions).  May indicate NASA needs to realign some programs to better meet Goals.
4.  Rating given to familiarity with Pillars & Goals before survey were not good (mean = 6.1, ranked 21st in 30 questions).  Evidence is shown in left-hand graph -- ASTTAC and benchmark customers very familiar -- Indirect industry and associations/publications were very unfamiliar.  Indicates need to communicate better with these organizations.
5.  12 areas best aligned to Pillars and Goals is shown in right-hand graph for national and organizational needs.  Results are generally consistent for national and organizational needs but some differences arise, particularly in design tools and revolutionary technology leaps where organizational needs rate higher.  Answers were not forced - respondents answered with combination of pillars, goals and programs. 

## HIGHEST AVERAGE RATINGS (On a scale of 1 to 10)

_**Mean Score**_
**8.2  Level of interest in ASTT Program**
- 96 percent rated their level of interest in the ASTT Program as “high” or “medium”
**7.9  Goals aligned to national needs**
- 97 percent rated the Pillars and Goals as “well-aligned” or “moderately well-aligned” with national needs
**7.6  Provides key facilities**
- Common responses were, “Modern wind tunnel facilities and world-class computational resources are essential.”
**7.0  Contributes to education of scientists and engineers**
- Common theme for improvement was, “More funding, cooperative programs, more internships, and involvement at all levels of education.”
**7.0  NASA scientists provide innovative technical solutions**
- Common responses were, “Human factors, structures, propulsion, computational fluid dynamics and space travel.”
**6.8  Balances analytical, computational, experimental and test research**
- Most felt the balance was good, but the mix constantly needs to be scrutinized; some wanted more flight testing.
**6.8  Partnerships with government, industry and universities**
- 90 percent rated ASTT as “effective” or “moderately effective” at forming partnerships
**6.7  Environmental compatibility of aerospace systems  **
- 91 percent rated ASTT as “effective” or “moderately effective” in ensuring aerospace environmental compatibility
**6.7  Superiority of U. S. aircraft and engines**
- 88 percent rated ASTT as “effective” or “moderately effective” in maintaining superiority of U.S. aircraft and engines
**6.6  Goals aligned to customer’s organizational needs**
- 80 percent rated the Pillars and Goals as “well-aligned” or “moderately well-aligned” to their organizational needs

## LOWEST AVERAGE RATINGS(On a scale of 1 to 10)

_**Mean Score**_
**6.1  Familiarity with Pillars and Goals**
- 62 percent of Associations/Publications and 51 percent of Indirect Industry were unfamiliar with Pillars and Goals.
**6.0  Satisfaction with NASA’s ASTT facilities**
- Mean score for satisfaction with ASTT facilities has dropped from 6.8 in 1995 to 6.0 in 1998
**6.0  Rate program planning**
- Although the mean score is somewhat low, it rose from 4.7 in 1992 to 5.6 in 1995 to 6.0 in 1998.
**6.0  How well is ASTT accomplishing its mission?**
- High speed research, advanced subsonic transport, critical technology and space transportation were rated highest.
- High performance computing was rated medium.  General aviation, high performance aircraft, facilities and services, rotorcraft, and hypersonics were rated lowest.
**5.9  Balances R&T and focused-program research**
- 33 percent called for more fundamental research in the 1998 survey as compared to 16 percent in the 1992 survey.
**5.8  NASA downsizing has affected its customers**
- Major effects to NASA downsizing were loss of foundational research and loss of key NASA personnel.  Others felt downsizing had a positive effect, “... now  less micro-management and more of a partnership with NASA.”
**5.8  Mission areas are aligned to customers needs**
- Government and benchmark customers rated this question high (means = 7.0 and 6.1); all others rated it low.
**5.8  Developing technology for economic competitiveness**
- Mean score dropped from 6.9 in 1995 to 5.8 in this survey.
**5.7  Developing a superior, affordable global air transportation system**
- Only 20 percent rated ASTT as effective in developing a superior, affordable air transportation system.
**5.1  Improves national security**
- 42 percent rated ASTT as ineffective in improving national security.

## OTHER NOTABLE RATINGS (On a scale of 1 to 10)

_**Mean Score**_
**6.6  Overall level of satisfaction with NASA**
- Slightly improved since 1995 survey.  Increase:  “Good job getting inputs from industry; good facilities, good people.” Decrease:  “Inability to do meaningful multiyear planning; inability to change with changing needs of industry.”
**6.6  Contribution to U. S. R&D needs**
- Main themes were more funding, more fundamental research, and working more closely with customers.
**6.5  Compare with satisfaction 3 to 6 years ago**
- Slightly improved since 1995 survey.  Increase:  “Alignment of research with the Pillars; shift from micro-management to a partnership relationship.”  Decrease:  “...condition of NASA’s deteriorating facility support.”
**6.4  Understand the mission of ASTT**
- Significant drop in mean score from 1995 (7.0 to 6.4).  Possible confusion regarding addition of space transportation.
**6.4  Overall satisfaction with ASTT products**
- Highest:  High speed research, computational products, advanced subsonic research.  Least: those canceled due to lack of funding; and those that do not have a cooperative character.
**6.4  Are programs aligned to Goals?**
- Best aligned:  High speed research; advanced subsonic research; experimental aircraft; and safety.
**6.3  Rate outreach and communications**
- Slight increase in mean score since 1995 survey (6.3 in 1998 versus 6.2 in 1995).
**6.3  NASA performs fundamental research**
- Significant drop in mean score since 1995 survey (7.1 in 1995 versus 6.3 in 1998).
**6.3  Cooperates with other organizations**
- Suggestions for improvement:  Be proactive in finding organizations to cooperate with; understand realities of industry.
**6.1  Developing superior, affordable space launch systems**
- Drop in mean score since the 1995 survey (6.4 in 1995 versus 6.1 in 1998).  1995 question related to national needs.

## OVERALL LEVEL OF SATISFACTION WITHTHE NASA ASTT PROGRAM

**Generally Unsatisfied**
**(“Poor”) (1-2-3-4)**
**Generally Satisfied**
**(“Average”) (5-6-7)**
**Highly Satisfied**
**(“Good”) (8-9-10)**

- Mean

-   N

- Percentage of Responding Customers

**Continued improvement from 1992 and 1995 surveys; however, improvement is less dramatic since 1995 survey.**
**Possible New Goal:  95 percent rank ASTT at 5 or above; 45 percent rank ASTT at 8 or above.**
- FY92 Question Asked:  Overall, the extent to which NASA Aeronautics is meeting your needs (Good-Average-Poor).
- FY95 Question Asked:  Overall, on a scale of 1 to 10, how satisfied are you with the NASA Aeronautics Program?
- FY98 Question Asked:  On a scale of 1 to 10, what is your overall level of satisfaction with NASA ASTT?

## OVERALL LEVEL OF SATISFACTION WITH THE NASA ASTT PROGRAM COMPARED WITH 3 TO 6 YEARS AGO

**Generally Unsatisfied**
**(“Poor”) (1-2-3-4)**
**Generally Satisfied**
**(“Average”) (5-6-7)**
**Highly Satisfied**
**(“Good”) (8-9-10)**

- Mean

-   N

- Percentage of Responding Customers

**Significant improvement in the “Unsatisfied” category since 1995; some improvement in the overall mean score from 6.3 to 6.5.**
**Possible New Goal:  90 percent rank ASTT at 5 or above; 40 percent rank ASTT at 8 or above.**
- FY95 Question Asked:  On a scale of 1 to 10, overall how does the [Aeronautics] Program’s recent performance compare to the performance 5 to 10 years earlier?
- FY98 Question Asked:  On a scale of 1 to 10, how does your organization’s overall level of satisfaction with NASA’s ASTT Program compare with its satisfaction 3 to 6 years ago?

## CUSTOMER SURVEY ACTION PLAN

_**Immediate Action**_
**Disseminate Survey Data**_** **_
**Provide copies of survey report and detailed data to the Centers**
**Provide copies of summary report to survey participants**
**Discuss survey results with ASTTAC**
**Put article on survey in NASA Agency and Center publications**
**Put information on survey on NASA OASTT home page**
**Encourage Centers to conduct follow-up surveys at project levels**
_**Follow-on Actions**_
**Increase Customer Awareness of NASA ASTT**
**Target Associations, Publications, and Indirect Industry Customers with ASTT information and accomplishments**
**Provide similar information to other customer groups**
**Continue efforts to develop better contacts with customers**
**Improve Key NASA ASTT Management Processes**
**Develop awareness within ASTT of importance of obtaining customer input**
**Develop processes for getting and using additional customer input**
**Consider policy changes based on survey results (e.g., more fundamental research)**

## WHAT DO YOU WANT TO SAY TO THE NASA ADMINISTRATOR?

- Hold the course, balance is good, good support from industry -- don’t overreact to public statements.
- ASTT Program is the most efficient dollars NASA spends.  ASTT industry/NASA roles are appropriate approach for space development.  Much more efficient when industry contributes funds and resources...
- Don’t sacrifice aeronautics work due to pressure of financing space transportation or other space activities.
- Strengthen university programs to develop next generation scientists and engineers.
- Subsonic section has inadequate advocacy with the public and is not on a par with the space activity.
- Thank you for covering the wide range in ASTT from general aviation to high-speed aeronautics.
- We are on a good course.  Things have certainly been improving.
- NASA should ... address specific needs in industry, especially ... competitiveness.  Keep up research in propulsion and noise reduction.  Emphasize creature comforts and safety in general aviation.
- [ASTT Program] needs heavier emphasis on space technology.
- ... Happy to see that ASTT is stronger focused and more dynamic.  I like the way the program is now ...
- [We need] more cooperation with the military; there has been an attempt but not “tight” enough.
- NASA should focus on space and let aeronautics industries handle aeronautics research.
- DoD is interested in being more closely related to NASA than in past.  Could use a true personnel exchange.
- [Due to] loss of key personnel and capability, [NASA] needs to recruit best personnel to stay on top of programs.
- Quit using taxpayers’ money to fund competition, because you are discouraging private investment ...
- Don’t let large industry drive high-speed civil.  Follow recommendations of National Research Council on HSCT.
- Keep providing results to hold confidence of Congress to hold the [ASTT] Program.  But, reach out to industry and identify specific programs to which those funds should be targeted.

## CLOSING

**The ASTT and Aero Customer Surveys are viewed by the Customers as positive communication efforts on NASA’s part to get feedback from customers.**
** **
**	****Most survey respondents were pleased that NASA ASTT made the effort to ask them for their input; several stated that the survey was evidence that NASA ASTT was communicating effectively with the customers.**