Memorandum U.S. Department of Transportation Federal Aviation Administration Subject: INFORMATION: Risk Assessment for Reciprocating Engine Airworthiness Directives Date: 5-24-99
From: Manager, Engine and Propeller Standards Staff, ANE-110 Reply to Attn. of: Mark Rumizen, ANE-110 781-238-7113 FAX 781-238- 7199 To: Manager, Aircraft Manufacturing Division, AIR-200 Manager, Brussels Aircraft Certification Staff, AEU-100 Manager, Engine Certification Office, ANE-140 Manager, Engine Certification Branch, ANE-141 Manager, Engine Certification Branch, ANE-142 Manager, Boston Aircraft Certification Office, ANE-150 Manager, New York Aircraft Certification Office, ANE-170 Manager, Airframe and Propulsion Branch, ANE-171 Manager, Rotorcraft Directorate, ASW-100 Manager, Rotorcraft Standards Staff, ASW-110 Manager, Airplane Certification Office, ASW-150 Manager, Rotorcraft Certification Office, ASW-170 Manager, Special Certification Office, ASW-190 Manager, Small Airplane Directorate, ACE-100 Manager, Small Airplane Standards Office, ACE-110 Manager, Atlanta Aircraft Certification Office, ACE-115A Manager, Propulsion Branch, ACE-140A Manager, Chicago Aircraft Certification Office, ACE-115C Manager, Propulsion Branch, ACE-118C Manager, Wichita Aircraft Certification Office, ACE-115W Manager, Propulsion Branch, ACE-140W Manager, Anchorage Aircraft Certification Office, ACE-115N Manager, Transport Airplane Directorate, ANM-100 Manager, Transport Standards Staff, ANM-110 Manager, Airframe and Propulsion Branch, ANM-112 Manager, Seattle Aircraft Certification Office, ANM-100S Manager, Propulsion Branch, ANM-140S Manager, Denver Aircraft Certification Office, ANM-100D Manager, Los Angeles Aircraft Certification Office, ANM-100L
1. INTRODUCTION This memo provides guidance for Aircraft Certification Offices (ACOs) to use when evaluating reciprocating engine service problems for determination of appropriate FAA action. Airworthiness Directives (AD’s) are required for unsafe conditions, but the determination of which types of engine service problems should be considered unsafe conditions is dependent upon the type of airplane in which the engine is installed. Reciprocating engines are typically installed in small airplanes intended forpersonal use, and the regulations governing the design and operation of these airplanes incorporate "mitigating features" to lessen the criticality of the engine. These mitigating features include low stall speeds, handling and stability criteria, emergency landing procedures, crashworthiness, and pilot training. These mitigating factors don’t guarantee safety when an engine service problem occurs, but instead provide a level of assurance that a pilot can reasonably fly the airplane to a safe landing. Using loss of engine power as measure of an airplane’s ability to accommodate engine failures, actual service data indicates that total aircraft power losses on turbine powered transport aircraft are ten times more likely to result in fatalities than on small piston powered GA aircraft. Therefore, it can be substantiated that General Aviation (GA) aircraft equipped with reciprocating engines differ from turbine powered transports relative to the criticality of the engine. This uniqueness of the GA fleet has resulted in inconsistent bases for issuance of ADs related to reciprocating engine service problems. In some cases, ADs have been issued where other, less burdensome forms of FAA action would have been more appropriate. And, conversely, in some cases where no FAA action was taken, an AD was warranted based on the potential safety risk. The FAA and the turbine engine industry have addressed similar continued airworthiness inconsistencies by instituting formalized, quantitatively-based risk assessment methodologies for evaluation of service problems. Risk assessment methodologies can also be applied to the GA reciprocating engine fleet, but must be modified to accommodate the less sophisticated technical resources and the incompleteness and inaccuracies of service data that is typical of the GA industry. The risk assessment methodology presented below should be considered a general guideline, rather than a specific procedure, to use for the evaluation of GA reciprocating engine service problems. It must be emphasized that, because each service problem presents its own unique set of circumstances, the risk assessment methodology will need to be customized to accommodate each analysis. 2. RISK ASSESSMENT METHODOLOGY A risk analysis utilizes data and information on a service problem to quantify the expected number of future events over a specified time period. The risk analysis should consider the consequences of the service problem relative to safety of flight, the probability of that service problem occurring, and the exposure of the current GA fleet to the problem. The following procedure is provided to assist in development of a risk analysis for a GA engine service problem. Because the particular details of any given service problem vary, this procedure can only be considered a starting point; evaluation methods will likely require customization to fit the specific data. It should also be noted that in many cases, all of the necessary data may not be available, and estimates must be used in place of the actual data. If necessary, engineers or flight test pilots can be consulted regarding the characteristics of airplane response to a given engine problem. An example based on an actual service problem will be provided to parallel each step of the following risk assessment process. Each subparagraph will contain its respective step from the example at the end of the descriptive text. The example will be based on the service problem evaluated for issuance of recent AD, which addressed failures of engine crankshafts. a. Consequences of the Engine Service Problem The first step in the process involves evaluation of the engine service problem to determine the potential effect on flight safety. For the purpose of this Guidance Memo, engine service problems that are being considered for AD action can typically be grouped in one of the three following hazard levels:
1

. Hazardous : Engine service problems that cause fire, uncontainment or other problems that could result in immediate collateral damage to the aircraft. These require minimal evaluationas they represent a direct safety hazard to the aircraft and they should be considered an unsafe condition that warrants an AD. However, a risk analysis should still be performed to help determine compliance times for the AD.
2. Major : Engine Service Problems that cause a significant power loss. These events pose an indirect hazard to the aircraft and do not necessarily require an AD. As discussed above, the design of GA airplanes incorporate mitigating features that contribute to lessening the severity of an engine service problem. Other factors, such as probability of the event occurring and fleet exposure, need to be considered for these service problems before initiating an AD.
3. Minor: Other types of service problems that do not result in a significant power loss, such as a partial power loss, rough running, pre-ignition, backfire, single magneto failures. These are potential AD candidates only if the probability of the event is very high. Information on the consequences of the service problem should be obtained from the production approval holder (PAH), which includes the engine manufacturer, STC holder, or PMA holder. EXAMPLE: Manufacturing defects in a certain population of engine crankshafts had experienced numerous failures resulting in 13 accidents over a six year time period. Failure of the crankshaft resulted in immediate engine shutdown, but did not result in uncontained engine destruction, failure of the engine mounting system, fire, or other collateral damage. Therefore, the failure mode posed an indirect hazard to the airplane and was classified as "major". b. Identification of Suspect Population The suspect population consists of all engines on which the service problem might occur. This could include the entire fleet of a particular engine model, or a subset of that fleet. For example, a quality escape might only impact a range of engine serial numbers shipped over a certain time period. Identification of the suspect lot requires input from the PAH. The suspect population can be defined in the following terms:

• Direct Population: this represents the engines that are confirmed to have the suspect part or condition and on which the service problem might occur. The direct population can be defined only if records exist that specifically define engine serial numbers, or a range of engine serial numbers, on which the risk of the service problem exists. However, the number of engines in the direct population can be determined based on the number of parts shipped. The conversion of the number of suspect spare parts to an equivalent number of engines must take a conservative approach, and assume that a minimum number of the suspect parts were installed in each engine.

• Indirect Population: this represents the engines that require further inspection or maintenance action to determine if they have the suspect part or condition. This would apply if, for example, a suspect lot of spare parts were shipped to various third party repair facilities, and records are not available to identify which engine serial numbers the parts were installed in. Or, if the failure condition results from an improper repair or maintenance procedure, and it is not known which engines underwent the repair or action, then all engines of the particular model must be considered suspect. Determination of the total number of engines of a particular model that are currently in service can be obtained from the engine manufacturer, or from the FAA aircraft registry in Oklahoma City.EXAMPLE: Data from the engine manufacturer and from the FAA indicates that the suspect crankshafts are installed on approximately 10,100 engines.

• Because the FAA/APO GA Survey presents operating hours for airplanes, not engines, the number of equivalent airplanes needs to be calculated: o assume 13% aircraft are twin engine (FAA/APO GA Survey)

o 10,100 engines = 87% N + 2 x (13% N), where N = total no. of airplanes o N = 8938 airplanes, (1162 twins + 7775 singles)
• this is the direct population because this is an estimate of the number of engines equipped with the suspect crankshafts
c. Event Rate The event rate is expressed as the number of service problem events per operating hour. The rate can be based on actual service experience, test data, or analysis. The rate may change with time; for example, for a fatigue-related problem, the rate may increase as a part or engine accumulates more total time. In many cases, only data on the number of accidents is available, not the number of total events. The event rate will then need to be estimated from the available data. To accomplish this, the following relationship between shutdowns, accidents and fatal accidents was derived from an analysis of FAA SDR data and NTSB accident data:

• Shutdowns/power losses:>1 every 10,000 hours
• Accidents: 1 every 100,000 hours
• Fatal Accidents: 1 every 1,000,000 hour For the purpose of the risk assessment, the event rate is assumed to be equivalent to the shutdown/power loss rate. The following formulas can then be used to estimate the number of events from available accident data:

• No. of events = (No. of accidents) X 10, or

• No. of events = (No. of fatal accidents) X 100
EXAMPLE:

• NTSB accident data indicated 13 accidents due to failures of engine crankshafts over the period from 1986 to 1992
• The event rate needs to estimated from the accident rate
• It is assumed that the event rate will not change in the future.

• estimate applicable airplane flight hours over relevant time period

• piston fleet est’d at 198,335 aircraft (FAA/APO GA Survey)
• applicable airplanes estimated as 8938 (step b above)
• applicable airplanes as % of piston fleet = 8938/198335 = 4.5% of fleet
• 189,947,000 hours for total fleet over ‘86-’92 time period

• 4.5% of total fleet hrs for applicable population = 8,559,036 aircraft hours• calculate event rate

• 13 accidents/incidents over ‘86-’92 time period
• 13 accidents/ 8,559,036 hrs = 1.52 x 10-6 accident rate
• 10 x (accident rate) = 15.2 x 10-6 event rate d. Exposure to Failure Condition The exposure to the service problem is a function of the suspect population, and the number of hours those engines can be expected to operate over a specified time period.

• Determination of the appropriate time period to use for the analysis depends on the characteristics of the service problem. In some cases, for high utilization aircraft, it may be appropriate to use the overhaul period and assume that maintenance is not performed between overhauls. A one year specified time period may be used if no other basis exists for the estimate.

• The number of hours per engine must be estimated. Manufacturer’s data can be used, or the General Aviation and Air Taxi Activity Survey, published by the FAA Office of Aviation Policy and Plans, provides GA fleet utilization hours to estimate the number of hours the suspect population of engines are operated.

• The total hours of exposure of the suspect population can then be found by multiplying the direct population by the number of hours per engine per year, multiplied by the specified time period. EXAMPLE: A one-year time period was chosen for this analysis and the utilization rate was estimated as 130 hour/airplane/year (based on FAA/APO GA Survey).

• Exposure = (130 hrs/airplane/yr) x (8938 airplanes) = 1.16x106 hours e. Expected Events The expected number of events can then be found by multiplying the event rate by the number of hours of exposure over the specified time period. The expected number of events can then be compared to historical data or FAA safety objectives for the respective event criticality level (hazardous, major or minor) to determine the appropriate form of FAA action, if any. However, for small populations of at-risk engines, the risk exposure may be unacceptable even if the analysis forecasts a low number of expected events. In those cases, further analysis may be required. The following table illustrates possible alternative courses of FAA action based on the risk assessment results. It is provided as a recommended guideline, and as previously stated, each service problem will have unique aspects that may require modifications to this process.Recommended FAA Action 1 Expected Number of Events 2 Minor Failure Consequences Major Failure Consequences Hazardous Failure Consequences
Low None ANPRM GA Alert AC 43-16 Or SAIB Airworthiness Directive (AD)

Medium GA Alert AC 43-16 Or SAIB 3 Airworthiness Directive (AD)

(EXAMPLE) Airworthiness Directive (AD)
High Airworthiness Directive (AD) Airworthiness Directive (AD) Airworthiness Directive(AD)

1. This assumes that company actions such as Service Letters, Service Bulletins, and Type Club or other association publications will be taken. If not, then FAA action may be required to compensate for the lack of company action.
EXAMPLE:

• Expected events = (event rate) x (exposure) = (15.2 x 10-6 events/hour) x (1.16x106 hours) = 18 expected events

• For the purposes of the table shown below, 18 expected events are assumed to represent a "medium" value, and for a major failure condition, an AD is recommended. f. Other Considerations The following additional factors should be considered when evaluating the need to issue an AD:

• If the suspect parts are installed on an identifiable group of engines (i.e., by engine serial number), or if only a small fleet of the suspect engine model exists, then the per flight risk, or risk exposure of any individual aircraft, to the service problem is higher for a given event probability. In these cases, an AD would be more likely to be required.

• Service problem occurrence rates that change over time must be considered in the analysis. These service problems are typically fatigue-related and are more likely to occur as the part or engine accumulates more operating hours. Additional data is often required to properly assess these conditions.

• In some instances, where the indirect population greatly exceeds the direct population (those engines with the suspect part), the number of expected events will be low relative to the size of the fleet. If an AD is required, the AD compliance section should structured to limit the burden on the indirect population of engines.• Other sources of data that can be used to support the risk analysis include FAA Service Difficulty Report (SDR) and Accident/Incident data, and data from GA organizations such as Airplane Owners and Operators Association (AOPA) or Aeronautical Repair Station Association (ARSA). These organizations can conduct surveys of their members to obtain specific information.
Original signed by: Thomas A. Boudreau, Manager ANE-110