Source: https://www.ntsb.gov/investigations/_layouts/ntsb.recsearch/Recommendation.aspx?Rec=A-14-119
Timestamp: 2019-10-18 22:15:55
Document Index: 774684430

Matched Legal Cases: ['§ 25', '§ 25', '§ 25', '§ 25', 'art 23', 'art 23']

Safety Recommendation A-14-119
Recommendation: TO THE FEDERAL AVIATION ADMINISTRATION: Provide your certification engineers with written guidance and training to ensure that (1) assumptions, data sources, and analytical techniques are fully identified and justified in applicants’ safety assessments for designs incorporating new technology and (2) an appropriate level of conservatism is included in the analysis or design, consistent with the intent of Advisory Circular 25.1309 (Arsenal draft).
Addressee(s) and Addressee Status: FAA (Open - Acceptable Response)
Response: We note that, on May 11, 2010, you received the final recommendations of your Aviation Rulemaking Advisory Committee (ARAC) for revisions to 14 CFR sections 25.671 and 25.1309, and in the 7½ years since then, you have been preparing a notice of proposed rulemaking (NPRM) for these revisions. We further note that your ARAC recommended concurrent issuance of a revision to AC 25.1309-1A, “System Design and Analysis.” The revised AC will provide guidance for showing compliance with the proposed revisions to sections 25.671 and 25.1309, as well as guidance to justify assumptions, data sources, and analytical techniques in safety assessments, and it will advise that extra conservatism be built into the analysis or design when the analytical process is variable or uncertain. The revised AC will also emphasize the importance of including safety critical assumptions in safety assessments to address new technologies that involve intrinsically hazardous equipment. Although you originally intended to publish the NPRM in December 2012, assessing the costs and benefits of the proposed rule has taken considerably longer than you originally expected; as a result, the NPRM and AC revisions have been delayed. We note from your current letter that you are not able to provide an estimated publication date for the NPRM and draft changes to AC 25.1309-1A and that, as an interim measure, you will continue to use issue papers, as you have for many years, to apply the analysis techniques that will be in the NPRM and revised AC to assess failure conditions for all new type certification programs. Although we believe that the NPRM, the resulting final rule, and the revisions to AC 25.1309 1A will satisfy Safety Recommendations A-99-22 and -23, we remain disappointed with the slow pace of progress. Using issue papers to apply the revisions has been an effective interim step, but these recommendations are now 19 years old and there is still no NPRM scheduled for publication to address them, let alone a final rule. Pending publication of the NPRM and a resulting final rule, and of revisions to AC 25.1309 1A that include the recommend requirements, Safety Recommendations A 99 22 and 23 remain classified “Open—Unacceptable Response.” Similarly, although the revisions to AC 25.1309-1A will likely satisfy Safety Recommendation A 02 51, it is 15 years old. Pending prompt issuance of the final rule and appropriate revisions being made to the AC, Safety Recommendation A-02-51 remains classified “Open—Unacceptable Response.” In your February 24, 2015, letter about Safety Recommendation A-14-119, you indicated that paragraph 11.h of AC 25.1309 (Arsenal draft) provides guidance to justify assumptions, data sources, and analytical techniques in safety assessments and also advises that extra conservatism be built into the analysis or design when there is variability or uncertainty in the analytical process. You also indicated that the draft AC did not specifically emphasize the importance of including all safety critical assumptions in a safety assessment. At that time, you planned to review the guidance to determine what revisions were necessary to specifically address new technologies that also involve intrinsically hazardous equipment, and you planned to train your engineers and engineering designees about the revised guidance once the revision was finished. Pending release of the revised AC with guidance on including the safety-critical assumptions in safety assessments and on new technologies that also involve intrinsically hazardous equipment, and the provision of adequate training on the revisions for all of your certification engineers and engineering designees, Safety Recommendation A 14 119 remains classified OPEN--ACCEPTABLE RESPONSE.
Response: -From Michael P. Huerta, Administrator: As noted in our previous letters to the Board, the Federal Aviation Administration (FAA) tasked the Aviation Rulemaking Advisory Committee (ARAC) with recommending refinements to safety rules and guidance concerning these four recommendations. In 2010. the ARAC made recommendations to the FAA to revise Title 14. Code of Federal Regulations§§ 25.671and25.1309, and associated guidance material to further mitigate the risk from latent failures, including control systems jams, as well as the risk of wear-related failures. The ARAC recommendations address the issues raised in all four safety recommendations for future new airplane designs. For existing airplane designs, we have addressed these systems failures through airworthiness directives and continuously monitor this fleet to correct any new issues that arise through our continued operational safety process. For future airplanes and those currently being certified, the FAA applies the analysis of failure conditions to all new type certification programs through issue papers, and will continue this practice on new certification projects, as applicable, until publication of our fin al rule and guidance. We initiated a rulemaking project in 20 11 to incorporate the ARAC recommendations for revising §§ 25.671 and 25.1309, applicable to new airplane designs and significant changes to existing designs. Assessing costs and benefits of the proposed rule took longer than expected and issuance of the Notice of Proposed Rulemaking (NPRM), entitled System Safety Assessments, and associated guidance was delayed. As a result, we cannot provide an estimated publication date for the NPRM and draft changes to Advisory Circular (AC) 25.1309- l A at this time. We continue to work toward issuance of a NPRM to revise §§ 25.67 l and 25.1309. The NPRM is in internal executive level coordination. The ARAC has recommended concurrent issuance of a revision to AC 25.1309-1 A, System Design and Analysis. In addition to providing guidance for showing compliance with the proposed revisions to §§ 25.671 and 25.1309. the revised AC will include guidance to justify assumptions. data sources, and analytical techniques in safety assessments, and advise that extra conservatism be built into the analysis or design when there is variability or uncertainty in the analytical process. The AC will also emphasize including safety-critical assumptions in the safety assessment to address new technologies that involve intrinsically hazardous equipment. The FAA will train our engineers and engineering designees accordingly once there is a final published version of the revised AC. I will keep the Board informed of the FAA's progress on rulemaking and the AC to address these recommendations and provide an update by October 31, 2018.
Response: In your February 24, 2015, letter about this recommendation, you indicated that paragraph 11.h of AC 25.1309 (Arsenal draft) provides guidance to justify assumptions, data sources, and analytical techniques in safety assessments and also advises that extra conservatism be built into the analysis or design when there is variability or uncertainty in the analytical process. You also indicated that the draft AC did not specifically emphasize including all safety critical assumptions in a safety assessment. At that time, you planned to revise the guidance after a review to determine what revisions may be necessary to specifically address new technologies that also involve intrinsically hazardous equipment, and you planned to train your engineers and engineering designees about the revised guidance once the revision was finished. We note that you continue to work towards issuing the revised AC, which you plan to publish for public comment later this year. Pending your release and our review of the revised AC with guidance on including the safety-critical assumptions in safety assessments and on new technologies that also involve intrinsically hazardous equipment, Safety Recommendation A 14 119 remains classified OPEN—ACCEPTABLE RESPONSE.
Date: 7/6/2016
Response: -Michael P. Huerta, Administrator: As noted in our previous response, draft Advisory Circular (AC) 25.1309 Arsenal provides guidance to justify assumptions, data sources, and analytical techniques in safety assessments and also advises that extra conservatism be built into the analysis or design when there is variability or uncertainty in the analytical process. This guidance will be included in a revised version of AC 25.1309. This AC will also emphasize including safety-critical assumptions in the safety assessment to address new technologies that involve intrinsically hazardous equipment. The FAA continues to work towards issuance of this revised AC, which we plan on publishing for public comment in late 2016. We will train our engineers and engineering designees accordingly once there is a final published version of the revised AC.
Response: CC# 201600220, dated May 12, 2016: The National Transportation Safety Board (NTSB) has reviewed the Federal Aviation Administration’s (FAA) notice of proposed rulemaking (NPRM) titled “Revision of Airworthiness Standards for Normal, Utility, Acrobatic, and Commuter Category Airplanes,” which was published at 81 Federal Register (FR) 13451 on March 14, 2016. The NPRM proposes to amend airworthiness standards for normal, utility, acrobatic, and commuter-category airplanes by removing prescriptive design requirements and replacing them with performance based airworthiness standards. The proposed standards would also replace the current weight and propulsion divisions in small airplane regulations with performance- and risk based divisions for airplanes with a maximum seating capacity of 19 passengers or less and a maximum takeoff weight of 19,000 lbs. or less. In addition, the NPRM proposes to adopt additional airworthiness standards to address certification for flight in icing conditions, enhanced stall characteristics, and minimum control speed to prevent departure from controlled flight for multiengine airplanes. On April 8, 2016, the FAA also published on its website draft Advisory Circular (AC) 23.10 titled “FAA Accepted Means of Compliance Process for 14 [Code of Federal Regulations] CFR Part 23.” The draft AC provides guidance on how to submit a proposed means of compliance (MOC) with Part 23 for acceptance by the Administrator in accordance with proposed section 23.10, “Accepted Means of Compliance,” in the NPRM. This letter provides comments on the NPRM. We will submit separate comments regarding draft AC 23.10. Related Safety Recommendations and Investigations We issued safety recommendations to the FAA and investigated accidents/incidents related to the following topics discussed in the NPRM: Certification One of the FAA’s stated goals in this rulemaking is to encourage the introduction of new and innovative technology. Our investigation of the January 7, 2013, fire in the aft cabin of a Japan Airlines Boeing 787-8, JA829J, which was parked at a gate at General Edward Lawrence Logan International Airport, Boston, Massachusetts, focused on issues related to the safety of new technology introduced to aviation (specifically, new lithium-ion battery applications). Manufacturers and subtier vendors developed design and oversight criteria to be applied and then presented the criteria to the FAA. We found that there was insufficient guidance and education for FAA certification engineers during the type certification process to ensure compliance with applicable requirements. As a result, on December 1, 2014, we issued the following safety recommendations to the FAA: A-14-119 Provide your certification engineers with written guidance and training to ensure that (1) assumptions, data sources, and analytical techniques are fully identified and justified in applicants’ safety assessments for designs incorporating new technology and (2) an appropriate level of conservatism is included in the analysis or design, consistent with the intent of [AC] 25.1309 (Arsenal draft). [Classified “Open—Acceptable Response”] A-14-121 Develop written guidance for your certification engineers and engineering designees about the use of traceability principles to verify that the methods of compliance proposed by type certification applicants for special conditions involving new technology are correct and complete. [Classified “Open—Acceptable Response”] A-14-122 Once the guidance requested in Safety Recommendation A-14-121 has been issued, provide training to your certification engineers and engineering designees on the subjects discussed in the guidance. [Classified “Open—Acceptable Response”] A-14-123 Require applicants to discuss key assumptions related to safety-significant failure conditions, their validation, and their traceability to requirements and proposed methods of compliance during certification planning meetings for type designs involving special conditions. [Classified “Open—Acceptable Response”] We are concerned that with the revisions proposed in the NPRM, and procedures proposed in draft AC 23.10, the FAA’s certification engineering staff will face increased demands to evaluate new technologies, and the FAA may face challenges similar to those encountered with the certification of the lithium-ion batteries in the Boeing 787.
Response: We acknowledge that paragraph 11.h of AC 25.1309 (Arsenal draft) contains guidance to justify assumptions, data sources, and analytical techniques, and that extra conservatism should be built into the analysis or the design when there is variability or uncertainty in the analytical process. We agree with you that the draft AC does not specifically emphasize including the safety-critical assumptions in the safety assessment. We note that you plan to revise the guidance after a review to determine what revisions may be necessary to specifically address new technologies that also involve intrinsically hazardous equipment, and that you plan to train your engineers and engineering designees about the revised guidance, once the revision is finished. Pending completion of all these actions, Safety Recommendation A-14-119 is classified OPEN—ACCEPTABLE RESPONSE.
Response: -From Michael P. Huerta, Administrator: We understand and concur with the Board's rationale leading to this recommendation. As the Board observed, paragraph 1l.h of draft Advisory Circular (AC) 25.1309-Arsenal does contain guidance to justify assumptions, data sources, and analytical techniques. The draft AC also advises that extra conservatism should be built into the analysis or the design when there is variability or uncertainty in the analytical process. However, the draft AC does not specifically emphasize capturing in the safety assessment the safety critical assumptions and validation thereof. Further guidance may be necessary to specifically address new technologies that also involve intrinsically hazardous equipment such as large energy supply devices. We plan to review our guidance materials to determine any needed improvements, and to provide training to our engineers and engineering designees accordingly.