# Presentation: 402504

## An Overview of MSFC Quantitative

- Risk Assessment (QRA) Practices
- Fayssal Safie/MSFC
- October 25, 2000

## Agenda

- Quantitative Risk Assessment System (QRAS)
- Other PRA-Related Practices
- Reliability Prediction
- Probabilistic Structural Analysis
- Similarity Analysis
- Reliability Demonstration

## MSFC Propulsion ElementsQRAS

## QRAS

- _Background_

- Space Shuttle probabilistic risk assessment (PRA) studies
- 1988 - Space Shuttle PRA for Galileo mission (PRC)
- 1993 - Galileo PRA update (SAIC)
- 1995 - Space Shuttle PRA (SAIC)
- 1997/2000 - Space Shuttle PRA (NASA/Code Q)

## QRAS

- _Background (cont’d)_

- 1997/2000 NASA QRA study
- In July 1996, the NASA Administrator directed NASA Headquarters to develop a software system to quantitatively assess the overall shuttle risk and serve as a tool to estimate risk changes due to proposed shuttle upgrades.
- At the request of NASA Headquarters, MSFC and JSC, supported by their prime contractors, are modeling their respective elements.
- The software system, called QRAS (Quantitative Risk Assessment System), is designed and developed by NASA Headquarters Code Q.

## QRAS

- Assess the reliability/risk of the overall shuttle vehicle, its major elements, and their components
- Evaluate risk reduction due to proposed shuttle upgrades
- Rank shuttle failure modes
- Perform trade studies/sensitivity analyses

- _QRAS_
- _Objectives_

## QRAS

- _QRAS_
- _Model Requirements_

- Model builds on work done by SAIC 1993-1995 Shuttle PRA model.
- Model is modular, reflecting shuttle modularity with its discrete elements, subsystems, and components (flexible to accommodate upgraded components and additional details).
- Model must be most detailed in high risk areas to allow sensitivity analysis and trade studies to be performed.
- Model/tool must be user-friendly and easily updateable.
- Model must be capable of identifying, quantifying, and prioritizing the major risk contributors.
- Model must support NASA decision-making process (evaluating shuttle upgrades and supporting flight issues).

## QRAS

- _QRAS_
- _Modeling Approach_

- Space
- Shuttle

- ORBITER

- MCC

-     HEX

- HPFTP

- LPFTP

-   - - -

**Products**

- 1. Space Shuttle Risk
- 2. Element Risk
- 3. Subsystem  Risk
- 4. Risk Ranking
- 5. Sensitivity Analysis
-     etc..

- Turbine Blade Porosity

- Turn-Around Duct Fail.

- Housing Retaining Lug Fail.

- FLIGHT/TEST DATA
- PROBABILISTIC STRUCTURAL  MODELS
- SIMILARITY ANALYSIS
- ENGINEERING JUDGMENT

**System Hierarchy**

- Turbine
- Blade Porosity

- Inspection
- Not Effective

- Porosity Present
- in Critical
- Location

- ET

- SSME

- ISRB

- UNCERTAINTY DISTRIBUTION FOR LOV DUE
- TO TURBINE
- BLADE POROSITY

**Event Tree**

**   **RISK AGGREGATION
-     OF BASIC EVENTS

**Functional Event Sequence Diagram (FESD)**

- End State
- or Transfer

- Porosity Present in Critical Location Leads to Crack in <4300 sec

- Scenario
- Number

- 1

- LOV

- 3

-    MS

- 4

- MS

- 2

- MS

- Turbine
- Blade
- Porosity

- Inspection
- Not
- Effective

- Porosity
- Present in
- Critical
- Location

- QUANTIFICATION
- OF FESD
- INITIATING &
- PIVOTAL EVENTS

- UNCERTAINTY DISTRIBUTION FOR EVENT PROBABILITY

- EVENT PROBABILITY
- DISTRIBUTION

- BASIC/INITIATING  EVENTS

- Porosity in Critical Location Leads to Crack in <4300 sec

- Loss of
- Vehicle
- (LOV)

- Blade
- Failure

- 5

- MS

## QRAS

- _QRAS_
- _MSFC Team Participants_

- MSFC
- Safety & Mission Assurance (S&MA)
- Chief engineer & project offices
- Engineering
- Prime contractors
- Reliability engineering
- Design & manufacturing engineering
- Hernandez Engineering Inc. (HEI)
- Reliability engineering and simulation

## QRAS

- _QRAS_
- _Databases_

- Problem Reporting and Corrective Action (PRACA)
- Automated Configuration data Tracking System (ACTS)
- Logbooks
- Engineering data/analyses
- Generic data
- Lessons learned
- SAIC study

## QRASPropulsion Element ModelsSignificant Observations

- _Strength:_
- QRAS modeling effort has contributed towards drawing management attention in using statistical and probabilistic information in the decision making process.
- Event Sequence Diagram (ESD) provides a better understanding of the failure mode risk and an excellent way to address risk mitigation.
- Data contained in the individual ESD packages are an excellent source of reference material and lessons learned.
- QRAS models constitute:
- The best source of failure rate data for the shuttle program to evaluate upgrades.
- The best source of information to understand the risk mitigation in place.
- The best source to understand the physics of failure for critical failure modes/events.

## QRASPropulsion Element ModelsSignificant Observations (cont’d)

- _Considerations:_
- QRAS is a large scale QRA study which is very complex and require extensive knowledge of the system, a large amount of data, and extensive modeling.
- Use of engineering judgment introduces significant amount of uncertainty.
- Quantification methods, in most cases, are not robust. Overlooking one piece of data may dramatically change the probability of loss of vehicle.

- _QRAS__Propulsion Element Models__Significant Observations (cont’d)_

## Considerations (cont’d):
Modeling of human error/process error is a big challenge. 
Human error/process error has been incorporated implicitly where flight and test data exist.
For structural failures which are modeled using design information, the human error/process error has been incorporated explicitly using placeholders based on historical data.
The QRAS modeling effort has shown that developing explicit models for the human error/process error is extremely difficult because of lack of adequate data. 

- Modeling of human error/process error is a big challenge.
- Human error/process error has been incorporated implicitly where flight and test data exist.
- For structural failures which are modeled using design information, the human error/process error has been incorporated explicitly using placeholders based on historical data.
- The QRAS modeling effort has shown that developing explicit models for the human error/process error is extremely difficult because of lack of adequate data.

- _QRAS__Propulsion Element Models__Significant Observations (cont’d)_

## Considerations (cont’d):
QRAS/PRA failure probabilities are imbalanced
Some failure probabilities are derived using mainly design information ( P&W Turbopumps), while others are derived using mainly test and flight data (RKDN SSME hardware). Generic data are also used in other cases. 
Some failure probabilities are derived using limited data (solid propulsion elements), while others are derived based on a lot of data (liquid propulsion elements).
Difficult to model common cause failures
Incomplete interface models

- QRAS/PRA failure probabilities are imbalanced
- Some failure probabilities are derived using mainly design information ( P&W Turbopumps), while others are derived using mainly test and flight data (RKDN SSME hardware). Generic data are also used in other cases.
- Some failure probabilities are derived using limited data (solid propulsion elements), while others are derived based on a lot of data (liquid propulsion elements).
- Difficult to model common cause failures
- Incomplete interface models

- _QRAS__Propulsion Element Models__Significant Observations (cont’d)_

## QRASConclusions

- Following a well defined and documented systematic procedure, involving the appropriate disciplines (reliability, design, and manufacturing engineering), and using the appropriate data are the key elements for a successful QRA study.
- Information derived from QRA studies are most accurate and useful at lower levels (within components and failure modes).
- QRAS tool is the best QRA tool available to support the shuttle program management decisions.

## Other PRA-Related PracticesReliability Prediction

## Reliability Prediction

- Reliability prediction techniques are dependent on the degree of the design definition and the availability of historical data. Two commonly used techniques are:
- Probabilistic design techniques: Reliability is predicted using engineering failure models.
- Similarity analysis techniques:  Reliability of a new design is predicted using reliability of similar parts.

## Reliability Prediction

- _Probabilistic Structural Analysis_

- It is a tool to probabilistically characterize the design and analyze its reliability using engineering failure models.
- It is a tool to evaluate the expected reliability of a part given the structural capability and the expected operating environment.
- It is used when failure data is not available and the design is characterized by complex geometry or is sensitive to loads, material properties,  and environments.

**FRACTURE**
**LOCATION**

- During rig testing the AT/HPFTP Bearing 	experienced several cracked races.
- Summary of 440C race fractures / tests: 3 of 	4 Fractured

- _Reliability Prediction_
- _Probabilistic Structural Analysis (cont’d)_Turbo-Pump Bearing Example

## Reliability PredictionProbabilistic Structural Analysis (cont’d)Turbo-Pump Bearing Example

- _Reliability Prediction__Probabilistic Structural Analysis (cont’d)_Turbo-Pump Bearing Example

## Reliability PredictionProbabilistic Structural Analysis (cont’d)Turbo-Pump Bearing Example

- Using rig fits and clearances
- Crack size data from actual cut-ups
- Stresses associated with manufacturing (ideal)
- Materials properties and their variations
- Failure mode being analyzed is over-stress

- _Reliability Prediction__Probabilistic Structural Analysis (cont’d)_Turbo-Pump Bearing Example

## Reliability PredictionProbabilistic Structural Analysis (cont’d)Turbo-Pump Bearing Example

- HPFTP Roller Bearing Inner Race - Model Flow

- Randomly select values for inner race material properties

- 

- 

- 

- Randomly select values for shaft and sleeve material properties

- 

- 

- 

- Tolerance fits of rig test bearing

- Inner race hoop stress contribution at given conditions

- Shaft and sleeve hoop stress contribution at given conditions.

- Total hoop stress

- Stress due to Manufacturing

- Stress > Allowable Load

- Iterate and compute Failure Probability

- Variation in:
- o  Fracture Toughness
- o  Yield Strength
- o  No. of Cracks
- o  Crack Depth
- o  Crack Length

- Compute Allowable
- Load for each crack

- Compute Allowable
- Load (worst crack)

- _Reliability Prediction__Probabilistic Structural Analysis (cont’d)_Turbo-Pump Bearing Example

## Reliability PredictionProbabilistic Structural Analysis (cont’d)Turbo-Pump Bearing Example

-    _At Test_
- 3 of 4 failed
-      ---
-      ---
- In 15+ tests
- never had a
- through ring
- fracture

-        _Race Configuration_
- 440C w/ actual manufacturing
- stresses (ie ideal + abusive
- grinding)
- 440C w/no manf. stresses
- 440C w/ideal manf. stresses
- 9310 w/ ideal manf stresses

- _Probabilistic Structural Analysis_

- 68,000 fail/100k firings
- 1,500 fail/100k firings
- 27,000 fail/100k firings
- 10 fail/100k firings

- It is estimated that 50% of the through ring fractures would result in an engine shutdown.  The shutdown 9310 HPFTP Roller Bearing Inner Race Failure Rate is then: 0.50 X 10/100k = 5 fail/100k firings

- _Reliability Prediction__Probabilistic Structural Analysis (cont’d)_Turbo-Pump Bearing Example

## Reliability Prediction

- _Similarity Analysis_

- Similarity Analysis is a technique for predicting the reliability of a new design based on historical data of similar designs (heritage hardware).
- Failure rates derived from historical data are modified to reflect the design and environment of the new hardware.
- Similarity Analysis is best performed at the lowest level possible, where more data is available and more appropriate judgment is made.

## Reliability Prediction

- _Similarity Analysis (cont’d)_ Fuel Turbo Pump Example

- Assume a Fuel Turbo Pump (FTP) has a historical failure rate of:
- 	50 per 100k firings
- Assume also the failure mode break down is:

-    Then the Cracked/Fractured Failure rate is: .35 X 50 = 17.5/100k firings

## Reliability PredictionSimilarity Analysis (cont’d) Fuel Turbo Pump Example 

- Then the Thermal Stress Failure Rate is:
- 		0.57 X 17.5 = 10/100k firings

- 100%

- _Reliability Prediction__Similarity Analysis (cont’d)_ Fuel Turbo Pump Example

## Reliability PredictionSimilarity Analysis (cont’d)  Fuel Turbo Pump Example

- Test Results
- Preliminary Analyses
- Integrated Product Team (IPT) Input

- Address "high hitters" - Using Thermal Stress failure rate of 10.0/100k firing
- Design changes to improve reliability
-                                                                                                        Cum
-                                                                  Percent                    Failure Rate
-                                                              Improvement                 Reduction
- Lower Operating Temperatures               20%                              2.00
- (Test)
- Hollow Blades                                          30% (additional)          4.40
- (Analysis, Expert Opinion)
- Material Change                                      20%  (additional)          5.52
- (Analysis)

- _Reliability Prediction__Similarity Analysis (cont’d)_  Fuel Turbo Pump Example

## Reliability PredictionSimilarity Analysis (cont’d) Fuel Turbo Pump Example 

- 		50 - 5.52 =  44.48 / 100k firings

- _Reliability Prediction__Similarity Analysis (cont’d)_ Fuel Turbo Pump Example

## Other PRA-Related PracticesReliability Demonstration

## Reliability Demonstration

- Reliability Demonstration is a reliability estimation method that primarily uses test data (objective data) to calculate demonstrated reliability with some statistical confidence.
- Some commonly used models and techniques for reliability demonstration include Binomial, Exponential, and Weibull models. Reliability growth techniques, such as the U.S. Army Material Systems Analysis Activity (AMSAA) and Duane models can also be used to calculate demonstrated reliability.

## SFR Criteria is an optimization tool based on the demonstrated reliability of SSME hardware.
SFR is used by the SSME Program as a quantitative probabilistic risk management tool for SSME critical hardware.
SFR Criteria:
Extensive fleet hot-fire experience
No failures or MR history
No periodic inspection
Use discrete optimization for life limit determination
Extend life limit up to 50% fleet leader but not to exceed the minimum run time of the six leading samples
New life limit should not be less than 25% of the fleet leader
Advantages include:
Maximize hardware usage
Use of all operational history

- SFR is used by the SSME Program as a quantitative probabilistic risk management tool for SSME critical hardware.
- SFR Criteria:
- Extensive fleet hot-fire experience
- No failures or MR history
- No periodic inspection
- Use discrete optimization for life limit determination
- Extend life limit up to 50% fleet leader but not to exceed the minimum run time of the six leading samples
- New life limit should not be less than 25% of the fleet leader
- Advantages include:
- Maximize hardware usage
- Use of all operational history

- _Reliability Demonstration_

- Example
- SSME Single Flight Reliability (SFR) Criteria

## Reliability Demonstration

- Example
- SSME Single Flight Reliability (SFR) Criteria – Powerhead Assembly Example

- Serial
- Number
- 4873937
- 4876118
- 4881840
- 4101464
- 4103704
- 4887803
- 4886959
- 4107624
- 4881353
- 4891093
- 4883915
- 4882395
- 4892855
- 4887018
- 4881159
- 4101420
- 4106454
- 4889036
- 4885515
- 4873558
- 4915695
- 4889794
- 4886294
- 4878165
- 4891738
- 4876216
- 4102590
- 4881664
- 4876351
- 4877733

- Seconds
- 33329
- 24716
- 21017
- 20327
- 20046
- 19908
- 16473
- 16444
- 15346
- 12194
- 11843
- 11125
-   9338
-   9230
-   9137
-   8821
-   8199
-   8070
-   7797
-   6583
-   5893
-   5577
-   5031
-   4989
-   4980
-   4643
-   4391
-   4376
-   3716
-   3636

- (Partial Listing)

- Beta = 2.08

- Powerhead Assembly LRU Code A050

**25% F/L – 8332            50% F/L - 16665**