Table of contents 1.Introduction 1.1.Purpose and scope 1.1.1.Purpose of the evaluation 1.1.2.Scope of the evaluation 2.Background to the intervention 2.1.Transition towards a proactive and evidence-based aviation safety management system 2.2.Description of the intervention logic and objectives of the initiative 2.3.Baseline and points of comparison 3.Implementation / state of Play 4.Method 4.1.Methodology and sources of information 4.2.Limitations and robustness of findings 5.Analysis and answers to the evaluation questions 5.1.EFFECTIVENESS Establishment of mandatory and voluntary reporting systems Occurrence collection rates Occurrence reporting flow Quality and completeness of data

Protection of information source and use of information for safety purposes “Just Culture” 5.2.RELEVANCE 5.3.EFFICIENCY 5.4.COHERENCE 5.5.EU ADDED VALUE 6.Conclusions Annex 1: Procedural information Annex 2: Stakeholder consultation Annex 3: Methods and analytical models Glossary Term or acronym Meaning or definition ADREP Accident/Incident Data Reporting Programme AMC Acceptable Means of Compliance ANS Air Navigation Services AOC Air Operator Certificate ATC Air Traffic Control ATM Air Traffic Management CAA Civil Aviation Authority CAG Collaborative Analysis Group CAT Commercial Air Transport DG JUST Directorate-General for Justice and Consumers DG MOVE Directorate-General for Mobility and Transport EASA European Union Aviation Safety Agency EC European Commission ECCAIRS European Coordination Centre for Accident and Incident Reporting Systems ECR European Central Repository ECR-ECCAIRS European Central Repository for Occurrences ECR-SRIS European Central Repository for Safety Recommendations ENCASIA European Network of Civil Aviation Safety Investigation Authorities EPAS European Plan for Aviation Safety ERCS European Risk Classification Scheme EU European Union FDM Flight Data Monitoring FTE Full-time Equivalent GDPR General Data Protection Regulation GM Guidance Material ICAO International Civil Aviation Organisation IFR Instrument Flight Rules iSTARS Integrated Safety Trend Analysis and Reporting System ITS Interrupted Time Series JRC Joint Research Centre MOR Mandatory Occurrence Report NoA Network of Analysts RMT Rulemaking task SCM Standard Cost Model SIA Safety Investigation Authority SMS Safety Management System SMTeB Safety Management Technical Body SSP State Safety Programme UAS Unmanned Aircraft System VOR Voluntary Occurrence Report 1.Introduction 1.1.Purpose and scope 1.1.1.Purpose of the evaluation This Commission Staff Working Document presents the ex-post evaluation of Regulation (EU) No 376/2014 on the reporting, analysis and follow-up on occurrences in civil aviation carried out to support the preparation of the Commission’s Report on the implementation of this Regulation pursuant to Article 24 of Regulation (EU) No 376/2014. It assesses whether the main objectives of this Regulation have been achieved, in particular whether the Regulation improved aviation safety by ensuring that relevant safety information relating to civil aviation is reported, collected, stored, protected, exchanged, disseminated and analysed. This Staff Working Document evaluates whether the Regulation contributed to reducing the number of aircraft accidents and related fatalities. It compares the initial expectations with the current situation. The evaluation highlights the improvements made in the area of occurrence reporting and analysis thanks to the Regulation and identifies some deficiencies and possible gaps. The assessment takes account of the following criteria: relevance, effectiveness, efficiency and the added value of the EU intervention. It also reviews the internal coherence of the Regulation as well as its coherence with other EU aviation safety regulations. The main objective of this evaluation is to determine whether the Regulation has had the expected effect on safety and whether it has contributed to the reduction of accidents and related fatalities in aviation throughout the EU. 1.1.2.Scope of the evaluation

The evaluation covers the period from 2015 to 2019. All EU Member States and their stakeholders were consulted through the public consultation, online survey and cost-benefit questionnaire. Except for one Member State, Member States were represented by at least one stakeholder. Non-EU countries, including Norway and Switzerland, were also consulted, as well as other third countries not directly affected by the Regulation (i.e. Turkey, Canada, United States of America, Japan, Montenegro, and Albania). 2.Background to the intervention 2.1.Transition towards a proactive and evidence-based aviation safety management system Regulation (EU) No 376/2014 is applicable as from 15 November 2015. It aims to prevent aircraft accidents and related fatalities through feedback and lessons learnt from occurrences. An occurrence is a safety-related event which endangers or which, if not corrected or addressed, could endanger an aircraft, its occupants or any other person. The Regulation seeks to ensure that relevant safety information relating to civil aviation is reported, collected, stored, protected, exchanged, disseminated and analysed. The industry and the competent authorities have to take safety action in a timely manner based on the analysis of the information collected. To ensure the continued availability of safety information, the Regulation provides rules on confidentiality of the reporting source and on the appropriate use of information. To this end, the Regulation grants, under certain conditions, the protection to the persons reporting the occurrences and other persons mentioned in occurrence reports and ensures that the information is used for safety improvement purposes. Finally, the Regulation ensures that aviation safety hazards are identified and appropriate risk-mitigation actions are taken at industry, national and Union levels. This proactive and evidence-based approach is implemented by the relevant aviation safety authorities of Member States, by organisations as part of their safety management system and by the European Union Aviation Safety Agency (EASA). Regulation (EU) No 376/2014 is complemented by an implementing regulationestablishing a list of occurrences in civil aviation to be mandatorily reported. Before the establishment of the occurrence reporting system in Europe, the advancements in aviation safety relied mainly on lessons learned from the investigation of aircraft accidents. However, whilst the ability to learn lessons from accident investigation is a crucial element in ensuring aviation safety and preventing the accidents from reoccurring, such reactive approach had shown its limits in bringing safety improvements, notably in the context of continuous air traffic growth (which in 2014 was expected to almost double by 2030). On the other hand, the experience had shown that accidents are often preceded by safety-related incidents and deficiencies revealing the existence of safety hazards. Safety information is therefore an important resource for the detection of potential safety hazards. Consequently, the reactive system of accident investigation should be complemented by a proactive system, which uses other types of safety information to achieve effective improvements in aviation safety. At the European level, the transition towards a more proactive aviation safety management system started with the adoption of Directive 2003/42/EC which required each Member State to set up a mandatory occurrence reporting system (MORS). Member States had to collect, store, protect and disseminate between themselves information on certain civil aviation incidents. The Directive, however, did not include provisions related to the analysis of the data collected, for the benefit of aviation safety. Considering other reporting requirements existing in parallel, both at the European and international level, the adoption of Directive 2003/42/EC also led the Member States to adopt rather divergent approaches. These diverging approaches resulted, for instance, in the existence of multiple occurrence databases at the European level. The clarification of reporting obligations had to be addressed in order to improve the accessibility, the quality, the analysis and the exchange of occurrence-related information. Additionally, the Impact Assessment for the proposal of a new Regulation on occurrence reporting envisaged that the future legislative should address the growing importance of a “Just Culture” , encouraging reporting of essential safety-related information. 2.2.Description of the intervention logic and objectives of the initiative The 2012 Impact Assessment for the proposal for the occurrence reporting regulation identified the following four main problem drivers related to the previous legal framework established by Directive 2003/42/EC: ·The collection of occurrences was not optimal At the time, the Impact Assessment concluded that the optimal collection of the safety-related information had not been achieved due to the following reasons: oThe scope of reporting differed between the Member States, which resulted in discrepancies in Member States reporting levels oIndividual reporters were afraid of repercussions for submitting occurrence reports oNo obligation to establish voluntary reporting schemes was laid down in the Directive oThere was an insufficient clarity in reporting obligations and in the flow of information Consequently, the specific objective of the Regulation is to ensure that all occurrences which endanger or would endanger aviation safety are collected and are providing a complete and clear picture of safety risks in the EU and its Member States. To achieve this objective, following operational objectives were set: oEstablishment of the mandatory reporting systems oEstablishment of the voluntary reporting systems

oReaching a higher occurrence collection rate through harmonisation and classification of reporting obligations oClarifying and development of the legislative requirements on “Just Culture” oClarifying the flow of information To achieve these operational objectives, the Regulation sets out the specific provisions on the mandatory and voluntary reporting, establishes a regime on the protection of information source and sets the related requirements for the Member States to ensure the protection of reporters and it provides for the specific rules facilitating the exchange of safety information between the stakeholders. ·Suboptimal data integration, low quality and incomplete data High quality safety-related data is crucial for identifying and assessing safety hazards posed by occurrences. The Impact Assessment identified low data quality and incompleteness of data provided in the occurrence reports as two main causes of the data integration, needed for comprehensive understanding of potential safety deficiencies, being suboptimal. To remedy the situation, the specific objective was set for the Regulation to make sure that occurrence reports stored in the national databases and in the European Central Repository (ECR) are complete and contain high quality data. Establishment and management of the ECR, standardised data entry processes, establishment of mandatory data fields and establishment of quality checking processes are the operational objectives leading to the achievement of this specific objective. ·Legal and organisational obstacles for ensuring adequate access to information contained in the ECR The regime established by the Directive on the access to the data contained in the ECR led to a situation where due to the requirement to de-identify the data contained therein, Member State authorities and EASA did not have an access to crucial occurrence data. At the same time, Member States would also refrain from uploading the data into the ECR due to fears that such data could be used by other stakeholders for other purposes than safety (e.g. benchmarking). Consequently, the specific objective of the Regulation is to ensure that all safety-critical information stored in the ECR is accessed adequately by the competent authorities and that they are used strictly for safety-improvement purposes. Hence, at the operational level, the Regulation provides for the granting of access rights to the ECR information to appropriate safety authorities and establishes rules on confidentiality when data is transferred to the interested parties and safeguards against misuse of data. The Regulation therefore contains the detailed rules on the access to the ECR, granting the full online access to the ECR to any entity entrusted with regulation of civil aviation safety or any safety investigation authority within the Union and further specifying the conditions under which the interested parties listed in Annex II to the Regulation can request the information contained in the ECR. The Regulation lays down the requirements on confidentially and protection of data gathered via the occurrence reporting systems. ·Lack of occurrence analysis at Member State level and at European level and of appropriate corrective and preventive actions Even where the data was collected under the previous legal framework, the Directive did not specify how the information collected should be used in order to contribute to safety improvements. Therefore, the Regulation aims at ensuring that reported occurrences are effectively analysed, that safety hazards are identified and addressed, where relevant, and that the safety effectiveness of actions taken is monitored. To achieve this objective, following operational objectives were set: oTo analyse occurrence data and to identify actual or potential safety hazards oTo adopt preventive or corrective actions where appropriate oTo oversee the effectiveness of the actions taken oTo create a common EU risk classification scheme for classifying occurrences To this end, the Regulation contains the provisions on the occurrence analysis and follow up at both Member States and EU levels. The Regulation also provides for the development of the European risk classification scheme and confers upon the Commission the necessary delegated and implementing powers to define the scheme in the EU legislation and to set the arrangements for its implementation.

Figure 1: Intervention logic of Regulation (EU) No 376/2014 2.3.Baseline and points of comparison Baseline analysis was prepared as part of the Support Study to this evaluation. The baseline analysis was built on the qualitative analysis conducted in the 2012 Impact Assessment by using quantitative methods to forecast how the selected indicators would have evolved without the Regulation (EU) No 376/2014 in place. The methodology used is the Interrupted Time Series (ITS) methodology, relying on comparison of time-series data before and after intervention, built upon the basic assumption that the outcome variable would not be altered if there were no intervention. The baseline scenario for the selected indicators was forecasted through time-series, capturing the historic data evolution of these indicators themselves and other independent variables with time, and projecting this same behaviour into the future as per the actual independent variables evolution. In the issue at hand, time-series were used to forecast selected indicators from 2015 up to 2019, based on their historic evolution (from 2009 to 2015) and the evolution of other variables. After that, the Baseline results were compared with the ones actually recorded to derive conclusions on the intervention impact. To model the counterfactual scenario under which the Regulation had not been adopted, the study analysed a number of parameters and their evolution over a period of time. The parameters included the recorded growth in the level of reporting, the levels of reporting per type of operation, stakeholder and Member State, as well as percentage of completeness of ECR fields, the amount of civil aviation accidents and incidents, and other variables such as traffic movements or a number of safety recommendations and safety actions. In the counterfactual scenario, due to the increasing air traffic in the EU, the occurrence reporting would have continuously increased at a modest pace for some stakeholders. However, since the entry into force of the Regulation higher number of occurrences than forecasted is reported. Such improvement in occurrence reporting is evidenced across all aviation domains which are subject to reporting requirements. Without the Regulations’ provisions on data quality, the level of completeness of the occurrence reports in the ECR would be considerably lower. Thanks to the Regulation, occurrence reports are approximately 10% more complete.

Finally, while impossible to conclude with certainty, the increased reporting may have contributed to the overall improvement of aviation safety at the European level especially by improving safety awareness, identification of safety risks and increased safety promotion. This might have had an indirect effect on the reduction of the total number of accidents compared to the counterfactual scenario. On a more general level, without the Regulation, the voluntary reporting systems in organisation established in the EU Member States would have not existed. The Regulation allowed also for the protection of identity of the reporters of occurrences in civil aviation. 3.Implementation / state of Play The provisions on the establishment of mandatory and voluntary reporting, set out in Articles 4 and 5 of the Regulation, were fully implemented by both the Member States and EASA. In practice, this means that each Member State has established both mandatory and voluntary reporting mechanisms at the level of its aviation authority. Similarly, EASA has a functioning mandatory reporting system in line with the requirement set out in Article 4(4) and voluntary reports can be submitted directly to the Agency and the Member States via the European Reporting Portal. Some gaps have been identified in the implementation of these requirements by the organisations. While some organisation were already subject to the mandatory reporting system requirement prior to the Regulation’s entry into force, some other organisations had to develop it. This resulted in some organisations reporting less than others. Similarly, in the field of voluntary reporting, some organisations indicated that they did not have a voluntary reporting system in place. Finally, some organisations do not distinguish between voluntary and mandatory reporting systems, having only one reporting system in place. The Regulation lays down the requirements for the collection, storage and exchange of information in Articles 6 and 9. Those requirements were implemented in full by the organisations as well as by the Member States and EASA. Member States established mechanisms for the exchange of information with other Member States and with EASA, and for the transfer of such information into the ECR. The Regulation also lays down specific requirements for the management of the ECR, standardised data entry processes and quality checking processes. Majority of those provisions were fully implemented by the organisations, Member States and EASA. Nonetheless, some organisations do not have in place a formal quality checking process as required by Article 7(3). In addition, the development of a common European risk classification scheme was implemented partially. While the Commission developed the scheme by the deadline of 15 May 2017, as envisaged in Article 7(5), it still has to define the scheme in accordance with Article 7(6). The definition of ERCS together with the arrangements for its implementation is ongoing. Articles 10, 11 and 12 stipulate the arrangements for access to the information stored in the ECR. Those provisions are fully implemented by the Commission, which is responsible for the processing data stored in the ECR and by the Member States (so called “points of contact”) who assist the Commission in relation to the requests for information stored in the ECR. The implementation of the Regulation’s provisions related to “just culture” and the protection and appropriate use of the source of information was closely analysed. There are few shortcomings in the implementation of those provisions. While it was found that all Member States and majority of the organisations put in place processes to ensure anonymization of occurrence reports, the issue with traceability – i.e. the possibility to infer the persons concerned in an occurrence report based on the circumstances described in the report – remain, especially in the smaller organisations with less employees. Article 16(11) lays down an obligation for the organisations to adopt internal “just culture” rules. The available information is not conclusive on whether this requirement has been fully implemented by all organisations. The most significant shortcoming in the implementation of the Regulation was found in relation to the obligation laid down in Article 16(12) for the Member States to designate a body responsible for the implementation of the “just culture” principles. Based on the reviews of the reports on the functioning of the “just culture bodies” submitted by the Member States in accordance with Article 16(13) and additional information received from citizens, the Commission identified 11 Member States that failed to designate a “just culture body” responsible for the implementation of paragraphs 6, 9 and 11 of Article 16. As a consequence, the Commission opened infringement cases and sent out 10 Letters of Formal Notice and one Reasoned Opinion to the non-compliant Member States. The Commission also sent out a number of EU Pilots to the Member States that stated in their reports to have designated a “just culture body” but where there were still doubts as to whether such body has the competencies to discharge the function envisaged in the Regulation. Similarly, there has been a limited progress in the implementation of Article 15(4) of the Regulation, which aims to ensure the cooperation of the competent authorities responsible for the collection, evaluation, processing, analysis and storage of the occurrence reports and the judicial authorities in the Member States via advanced arrangements. It appears that only limited number of Member States put such arrangements in place and some of those arrangements do not comprehensively cover the details of the cooperation. Finally, the Regulation also lays down in Article 13 rules on the analysis of occurrence reports and on the adoption of preventive or corrective actions. EASA and the organisations across the EU fully implemented that provision. The Member States do use the analysis of the occurrence reports to identify potential or actual safety hazards. However, some Member States upload reports into the ECR that are relevant to other Member States, without duly informing them. Consequently, the concerned Member States are not aware of those occurrence reports and cannot make use of the information contained therein. When it comes to the adoption of preventive or corrective actions, those are fully implemented by EASA and Member States who take preventive actions identified through the analysis of the occurrences. At the organisations side, the situation depends on the maturity of the organisation’s occurrence reporting and analysis systems. Organisations with more mature systems are capable of identifying and acting upon potential safety hazards, other organisations, with less mature systems, identify less safety hazards and take less preventive or corrective actions. 4.Method 4.1.Methodology and sources of information A support study was carried out by an external contractor to provide input to this evaluation. In accordance with the Terms of Reference, the study was structured along eight consecutive Tasks, as depicted in the figure below: Figure 2: Support Study Methodology Structure The methodology for this support study was based on:

·Desk-based research, including a range of legislative documents as well as previous studies and other relevant reports that were coded and analysed to form the basis of the study and informed the assessment of the current state of implementation; ·Stakeholder consultation via an online survey, cost-benefit questionnaires, in-depth interviews, Member State experts and industry stakeholders meetings, and an open public consultation to supplement secondary evidence with views and opinions from the wide range of stakeholders; and ·In-depth analysis of the primary and secondary data collected, including an assessment of the state of implementation, an analysis of the baseline, a cost assessment, and five analytical case studies presenting results for a range of thematic areas within civil aviation. All findings and results were triangulated and synthesised to provide evidence-based answers to the evaluation questions. The Study was structured according to an evaluation matrix, presented in Annex 3. This matrix operationalised the study’s evaluation questions, setting them out against the sub-questions, indicators and judgement criteria that will be used to answer them. It was developed on the basis of an in-depth understanding of the Regulation’s intervention logic, depicting what it aims to achieve and how, as presented in Section 2.2 above. The field research consisted of a public consultation as well as a range of targeted consultation activities. Seventeen stakeholder groups were consulted in the context of the study by different means. ·An online survey was launched on 13 November 2019. The survey remained open for three weeks and formally closed for responses on 9 December 2019. It was targeted to a range of different stakeholder groups, including national competent authorities, safety investigation authorities, EASA, but also all different industry sectors, and associations representing staff in the aviation field. The online survey included questions on the effectiveness, relevance, and EU added value of Regulation (EU) No 376/2014, as well as few questions pertaining to the degree to which the Regulation has been implemented in the Member States. A total of 67 respondents fully completed the online survey, and 32 respondents partially completed it (i.e. started completing the survey but did not answer all questions). Considering the efforts made to reach as broad a respondent group as possible, including through reminders/follow-ups, this is considered a decent number of respondents. ·Targeted interviews aimed to complement the questionnaires by collecting more details on why stakeholders hold certain opinions, as well as gather additional evidence in relation to evaluation questions for which qualitative data was judged to be an important source. A total of 40 interviews were planned out of which 39 were effectively carried out with a sample of key stakeholders per main stakeholder type. ·In terms of expert meetings, three workshops were organised, two with competent authorities (September and October 2019 in Cologne, Germany) and one with the aviation industry representatives (November 2019 in Brussels, Belgium), with the aim to gather data to feed into the answers to the evaluation questions. ·Finally, the Commission launched an open public consultation on 7 November 2019 to gather the opinions on the Regulation from the broader stakeholder community, specifically relating to the relevance, effectiveness, coherence, efficiency and EU added value of the Regulation, and on specific questions related to the concept of “just culture”. The public consultation was open for responses from the general public until 30 January 2020. A total of 198 responses were received, Furthermore, in order to complement the study with factual information, desk research on several related issues was carried out. The aim of the desk research was to collect, organise and analyse relevant information from relevant secondary sources. This included statistical data, legislative documents and their related implementing acts, and relevant reports and studies, with the intention of collecting both quantitative and qualitative evidence to complement the primary evidence collected through field research. ·EU Legislation ·ECCAIRS cases ·EASA documentation ·Existing studies and literature. ·Five thematic case studies, namely Air traffic control, commercial air transport (CAT) operations of large aeroplanes and related design and production, general aviation manufacturer of Non-Complex aeroplanes, Unmanned Aircraft Systems (UAS) with a Maximum Take-Off Mass below 150 kg and ground handling. The proposed sample of case studies was well balanced in order to cover instances of occurrence reporting, analysis of the reported occurrences, its quality as well as the follow-up actions taken by the relevant stakeholders. The case studies illustrate the instances of both voluntary and mandatory occurrence reporting and were selected to also provide insights into exchange and dissemination of information, confidentiality, and protection of the information source. The objective of the case studies was to take findings from stakeholder consultations (including surveys and interviews) and provide an insight on the impacts of the Regulation in specific operational scenarios. The data collected was used to respond to the evaluation questions. All the analytical findings constitute the basis for the assessment on how the Regulation has scored on the evaluation criteria. Each of these criteria was addressed through evaluation questions, as presented in the evaluation question matrix presented in Annex 3.

In the Commission, an Inter-Service Steering Group provided advice and monitored the progress of the exercise. Being composed of members from different Commission services and having the necessary mix of knowledge and experience, the Steering Group brought together a range of different perspectives and provided the necessary input, in particular where the evaluation touched different policy areas. 4.2.Limitations and robustness of findings Even though the evaluation was designed to ensure the robustness of the data supporting the findings presented in this Staff Working Document, some limitations to the robustness of certain data were identified while drafting the support study. First, the responses from the online survey came mostly from the competent authorities in the Member States (34 out of 99), which biased the results of the survey towards that stakeholder group. To mitigate this bias, the results of the stakeholder groups were analysed separately, where relevant. Secondly, there was a limited response rate to the cost-benefit questionnaires, especially from the industry stakeholders. This made the assessment of costs incurred by the industry brought by the Regulation more difficult. Thirdly, the contractor initially encountered some difficulties in scheduling the targeted interviews with the selected representatives from various stakeholder groups, in essence due to a lack of availability by the persons and/or the entities targeted for such interviews. The impact of such situation on the data gathered was however mitigated by extending the period in which the interviews were scheduled. 5.Analysis and answers to the evaluation questions This section provides the analysis and the results for the five evaluation criteria of effectiveness, efficiency, relevance, coherence, and EU added value. The findings presented are based on the results from desk research, as well as results obtained through stakeholder consultations. 5.1.Effectiveness This subsection assesses whether the Regulation has been effective in achieving the intended objectives and in particular the improvement of aviation safety in Europe. It in particular assesses how the specific objectives and the operational objectives of the intervention have been implemented. Figure 3: Steps in the occurrence reporting, analysis and follow-up process The first specific objective of Regulation (EU) No 376/2014 is to ensure that all occurrences that endanger or would endanger aviation safety are collected and are providing a complete and clear picture of safety risks in the EU and in the Member States. Establishment of mandatory and voluntary reporting systems Requirements for mandatory occurrence reporting already existed prior to the adoption of Regulation (EU) No 376/2014 and they did not therefore need to be re-established. The Regulation has however filled gaps by developing the common EU standards for reporting, across different industry domains and Member States. As such, the Regulation’s requirements are broader in terms of scope than of its predecessor Directive 2003/42/EC. Mandatory and voluntary occurrence reporting systems are indeed in place at the Member State level, at EASA level and in the majority of concerned organisations. In addition, the European Reporting Portal was established by the Commission services. The majority of Member States do not distinguish between the two types of reports in their respective databases, as they rather prefer encouraging organisations to submit all reports to them, regardless of their status. As a consequence, it is difficult to assess the number of voluntary reports submitted, as there is no easy way to distinguish between MOR and VOR stored in the ECR.

The Study concluded that there are no major concerns in terms of coverage of who has to report. Moreover, EASA noted that the Regulation omits ‘declared organisations’. A related problem for EASA is that it also oversees organisations established in third countries, which are subject to reporting requirements under Regulation 2018/1139 but not Regulation 376/2014, which raises questions in terms of a level playing field and what creates a burden on EASA who has to define ‘equivalent’ occurrence reporting, analysis and follow-up requirements for those organisations. Occurrence collection rates An expected outcome from the effective implementation of Regulation (EU) No 376/2014 is the increased number of reports submitted and thus a greater amount of aviation safety information. One of the key objectives of Regulation (EU) No 376/2014 is to increase the available data on occurrences. This objective has been met. The total number of occurrence reports stored in the ECR has increased significantly since the entry into force of the Regulation. In 2015, there were just over 200,000 occurrence reports in the ECR in the given year. As shown in the Figure 4 below, in 2019, the total number of reports transferred to the ECR in one year reached 291,458. Cumulatively, the number of occurrence reports contained in the ECR more than doubled during this period: from 1,125,264 reports at the end of 2014 to 2,548,578 reports in 2019. Figure 4: Number of records in the ECR (source: JRC overview of ECR-ECCAIRS content) As shown in the Figure 5 below, there would have been an increase in the amount of collected occurrences even in the counterfactual scenario, without the Regulation in place. This would have been directly correlated with the annual traffic increase. However, since entry into force of the Regulation, the collection rate has been increased significantly compared to the expected trends. This therefore suggests that the Regulation facilitated the increase in the occurrence reporting across the EU. Figure 5: Number of occurrences reports issued in the evaluation baseline at European level However, according to the views expressed by stakeholders in the online survey, no conclusion can be made on whether all relevant occurrences are reported and whether the reported occurrences provide a complete and clear picture of safety risks across the EU. This can in part be attributed to the difficulty to measure whether some occurrences pose an immediate, significant safety risk and hence whether they should be reported. There is a divergence in the views expressed by the Member States and the industry stakeholders. The authorities in the Member States believe that there has been a clear improvement in understanding safety risks through reporting. Although they acknowledge that it is likely that not all occurrences are reported, they have seen a clear improvement, not just in numbers but also in the quality of the occurrence reports. They believe that the picture of safety risks is adequate. Conversely, the stakeholders representing organisations or associations stated that not all occurrences are reported and hence the general understanding of safety risk might not be fully adequate. Occurrence reporting flow The reporting flow (i.e. who reports to whom and when) is well understood by all stakeholders involved, although some issues have been identified. Sometimes the details of occurrences are uploaded to the ECR by one Member State with no notification to the Member States concerned. However, there is a general expectation that the new ECCAIRS 2 software will help on this aspect. Furthermore, the industry stakeholders complained about the lack of transparency on the processing of occurrence reports after they have been submitted to the competent authority and stored in the ECR. 23 out of 25 industry representatives in the workshop expressed their view that it is not very clear or not clear at all what happens with the occurrence reports once they are submitted to the competent authorities. The industry also expressed disappointment on the lack of a subsequent feedback from the authorities. Furthermore, some stakeholders also expressed the view that the reporting timelines provided in the Regulation are too tight. The 72-hour time limit set out in the Regulation seems in particular to be an issue for the air navigation service providers and commercial air transport operators. However, it should be noted that in exceptional circumstances the Regulation gives possibility to submit the occurrence reports after the 72-hour time limit. The second specific objective of Regulation (EU) No 376/2014 is to ensure that the occurrence reports stored in the national databases and the ECR are complete and contain high quality data. Quality and completeness of data Regulation (EU) No 376/2014 underlines the importance of high-quality and complete data. Overall, there has been a considerable improvement in the quality and completeness of the occurrence data contained in the ECR as well as in the national databases. This has been achieved thanks to the key requirements mandated in the Regulation such as a standardised data entry process, minimum information to be included and quality checking processes. An important EU contribution is the ECR that stores all occurrence reports collected in the EU. Member States and EASA are required to transfer to the ECR all information relating to safety stored in their respective databases in a timely manner. The national data is integrated in the ECR and checked in accordance with quality criteria. The ECR consists of two physical databases – the European Central Repository for Occurrences (ECR-ECCAIRS) and the European Central Repository for Safety Recommendations (ECR-SRIS). Access to the ECR is subject to strict rules and mainly limited to the European competent authorities.

Throughout the period under review, the European Commission’s Joint Research Centre (JRC) provided the operational services related to the ECR-ECCAIRS. As identified by the JRC, the ECR-ECCAIRS is fed with data originating from any of the 31 contributing States as well as from EASA on a scheduled basis, with a maximum timespan of 1 month. Following a change of strategy and after an impact study, the Commission services decided to replace the current ECCAIRS platform by a fully re-engineered platform called ECCAIRS 2, and to transfer the management of the ECCAIRS system from the JRC to EASA as of 2020. ECCAIRS 2 was released in October 2020 and it is expected to solve many of the issues reported by competent authorities in terms of user-friendliness, system performance and training. It is a web/cloud-based solution with a user-friendly interface. It will integrate the local national repositories (without duplication/copying) into a regional view and integrate into the Data4Safety (D4S) programme of the Commission/EASA, subject to the strict rules on protection of information contained in the ECR imposed by Regulation (EU) No 376/2014 and applicable data protection rules. ECCAIRS 2 will allow the competent authorities to exploit advanced analytics, insights and auto-improved data quality. ECCAIRS 2 is likely to have a further positive impact on the effectiveness of Regulation 376/2014 as a whole. To furthermore ensure the operational objective as regards the completeness of the data contained in the occurrence reports, Annex I to the Regulation sets out a list of mandatory data fields to be filled when uploading an occurrence report. The list of mandatory data fields can be updated by means of delegated acts, thus giving the Commission the flexibility to adapt those data fields, when appropriate and to reflect developments that may occur in the aviation sector. There are diverging opinions about that list. Some stakeholders think that not all mandatory data fields are relevant for all the reporters or reporting organisations. Furthermore, both organisations and Member States noted that the requirement of completing the mandatory data fields has produced an additional workload for both reporters and authorities. This is not the view of EASA, as it considers that all eight (8) common mandatory data fields are relevant, however those should be completed only if applicable. Quality checking procedures should be implemented for checking the data quality of the occurrence reports thus avoiding inconsistencies and completing missing parts. Based on the collected evidence, it can be concluded that quality checking processes have at least partially been implemented by organisations and competent authorities, and they are implemented in the ECR with feedback being sent on the success/failure to the submitting body. It cannot be concluded with certainty that this was as a result of the Regulation as these processes already existed. In terms of resources, there are strong concerns in the Member States about the resources needed to do this task, as it is performed mostly manually. The use of digital tools (e.g. ECCAIRS 2, D4S) would help. The third specific objective of Regulation (EU) No 376/2014 addresses the competent authorities’ adequate access to the safety-critical information contained in the occurrence reports, and the strict use of such information for safety-related purposes, only. Protection of information source and use of information for safety purposes Regulation (EU) No 376/2014 mandates for all organisations and competent authorities in the Member States to take the necessary measures to ensure the appropriate confidentiality of the details of occurrences obtained by them (i.e. mandatory and voluntary reports, as well as the data taken from the ECR) and to process this data in accordance with applicable European rules on data protection. It requires that competent authorities in the Member States as well as EASA, do not record any personal details in their respective databases to prevent a transfer of personal data to the ECR. Based on the evidence collected for this evaluation, it can be concluded that organisations and all competent authorities in the Member States have internal processes in place to ensure that occurrence reports are anonymised before they are transferred to interested parties, notwithstanding the high workload this creates. Despite these efforts, mistakes happen and occasionally personal data is transferred due to a human error when inputting the occurrence data into the ECR. With regard to the operational objective that the information contained in the ECR is used solely for the improvement of the safety situation in aviation and not for apportioning blame and liability, the Regulation sets out a possibility for the competent aviation authorities to conclude advance administrative arrangements with their counterpart judicial authorities setting out the terms of using the occurrence data in judicial proceedings while seeking “to ensure the correct balance between the need for proper administration of justice, on the one hand, and the necessary continued availability of safety information, on the other”. However, almost no Member State has been able to conclude such advance arrangements with their respective judicial authorities. In fact, only seven Member States have some type of arrangement in place. Hindering factors are the lack of guidance as to the content or format of such an arrangement or lack of resources and the fact that those arrangements are only facultative. Despite the lack of advance administrative arrangements, the information available suggests that so far the data contained in the ECR is used strictly for safety improvement purposes only. This is one of the objectives to which the Regulation has contributed the most. “Just Culture” Regulation (EU) No 376/2014 is the first legal act to incorporate “just culture” provisions, thus ensuring its application across all the domains of the aviation at industry as well as at the competent authority level. The Regulation requires organisations to establish specific rules and procedures for the application of the Just Culture principles to the reporting of occurrences, thus ensuring that the reporters will not be subject to any form of punishment or prejudice for the occurrences reported, unless their conduct amounts to gross negligence or wilful misconduct. This should motivate the reporting at every level of the organisation. From the evidence collected, there are indications that such rules are being formally established by the concerned organisations, however the evidence is inconclusive as regards the actual compliance with those rules by the organisations. Some representatives interviewed during the stakeholder workshop of November 2019 spoke of few instances, which appeared to be in direct contradiction to the “just culture” principles. At the national level, the Member States are also required to put in place measures ensuring the protection of reporters and ensuring that the reporters will not be prosecuted in relation to the information shared in the occurrence report, unless their conduct amounts to gross negligence or wilful misconduct. To ensure such protection, the Member States are required to designate a body specifically responsible for the implementation of these “just culture” principles, as contained in paragraphs 6, 9 and 11 of Article 16. However, only a limited number of Member States have complied with this requirement, which led the Commission to opening a number of infringement proceedings. It should also be added that many industry stakeholders are unaware of the existence of a “just culture” body in their State, with only 4 out of 31 participants in the industry workshop answering positively when asked about their awareness of the existence of such body. At the industry level, the European social partners, members of the sectoral social dialogue committee for civil aviation (CANSO, ETF, ATCEUC), have jointly agreed on the “ATM Just culture tool box”. It targets staff and managers within organisations providing air traffic management / air navigation services and sets guiding principles for implementing Just Culture and fostering a healthy and open reporting culture. Ensuring that the all reported occurrences are effectively analysed, safety hazards are identified and addressed, where relevant, and the safety effectiveness of such actions is monitored, is the fourth specific objective of the Regulation. Analysis of occurrences and identification of actual or potential hazards

The Regulation set outs the obligation for the organisations, Member States and for EASA to develop processes to analyse the information contained in the occurrence reports in order to identify safety hazards associated with those occurrences. Furthermore, the competent authorities and EASA are mandated to exchange and analyse safety information contained in the occurrence reports at the EU level. Whereas the Study concluded that organisations, generally speaking, do have such processes in place, it should be noted that majority of the organisations already had such processes in place before the Regulation, as part of the requirements related to safety management system (SMS) contained in the implementing rules in the area of aviation safety. Nonetheless, there are organisations (e.g. ground handling), which subsequent to the Regulation’s entry into force had to introduce the processes to analyse occurrences. However, the study noted the problems experienced by such organisations in setting up such processes as the provisions of the Regulation in this regard are rather general. Additionally, many industry stakeholders noted that the timeline of 30 days for providing the preliminary analysis of the occurrences to the competent authority is too short. Moreover, the Regulation also sets a deadline of three months from the initial report for the organisations to submit their final analysis. The majority of industry stakeholders indicated that this deadline is too tight. The Regulation allows however to extend that deadline if this is justified. Similar to organisations, the competent authorities in the Member States all have processes in place to analyse occurrences. However, Member States differ in their opinions as regards the usefulness of such analyses and their content. Some authorities believe that the Regulation is not sufficiently clear on this aspect. Consequently, there are differences in the degree to which analyses are carried out and the depth of these analyses. This in turn can have an impact on the identification of safety hazards and on appropriate follow up actions being taken in a timely manner. Other reasons for those differences derive also from a lack of resources (in terms of time, budget and skills) to carry out the analytical work. The authorities noted also the “imbalance” between the requirements for such analysis and the degree of details included in the requirements for reporting, collecting, processing and storing occurrences. This imbalance leads to a high workload associated with the collecting and processing of occurrence reports and leaves little time for the analysis and follow-up. At the EU level, there are two streams of analysis: EASA’s analysis of occurrences reported directly to the Agency by organisations for which EASA acts as a competent authority (based on occurrences stored in EASA’s own database) and a collaborative analysis done together with Member States. Those analyses result in the publication of the EASA Annual Safety Reviews and in the development of Safety Risk Portfolios (SRPs), which are used in the publication of the European Plan for Aviation Safety (EPAS). The EASA Annual Safety Review is produced by the Safety Intelligence and Performance Department of EASA and is based on the data contained both in EASA’s own Occurrence Database as well as data contained in the ECR. As part of its risk management process, the Agency also defines and implements appropriate actions, which are subject to a continuous monitoring. EASA works with a range of collaborative partners through two main platforms: the Network of Analysts (NoA) and the Collaborative Analysis Groups (CAGs). The NoA is set up by Regulation 376/2014 and pulls together data from different Member States to get a more general picture of safety hazards faced by the EU as a whole. The CAGs are expert groups set up by EASA that are responsible for analysing the safety of European aviation for specific aviation domains. They review available safety information, arrange in depth safety analyses and identify emerging issues. They also monitor the safety performance of their domain and provide feedback on the effectiveness of actions taken. Both systems are complementary because they provide analysis at different levels and they contribute to the Safety Risk Portfolios and to the safety actions which may be adopted in the EPAS. Adoption and monitoring of preventive and corrective actions At the EU level, the analyses are carried out by EASA in collaboration with the authorities in the NoA and CAGs. The outcomes of those analyses drive the development of safety actions for the EPAS and allow the setting of safety actions and priorities, not only for EASA, but also for all the different actors in the aviation supply chain. The Regulation has therefore contributed largely to the development of analyses at the EU level. It is difficult to conclude with certainty to what extent organisations implemented preventive or corrective actions as a result of the Regulation or whether those actions derive from the pre-existing SMS requirements. Although the analyses are not necessarily resulting in a safety action, they always contribute to the greater understanding of the potential safety risks. In addition, the analyses also contribute to further staff training, safety awareness and other safety-related purposes. The study identified that the high workload and unclear definition of the term “analysis” has resulted in a limited number of corrective and preventive actions by Member States. Regarding the monitoring of the actions, the study confirmed that Member States and organisations put in place such monitoring. It appears, however, that those processes are largely based on the pre-existing SMS requirements and Regulation (EU) No 376/2014, did not change them. This is because the latter does not concretely suggest how the monitoring of actions should be conducted, limiting itself to establishing the obligation to conduct such monitoring. There is a certain lack of clarity on the side of national authorities responsible for the oversight of the Regulation. While the Regulation allows for the Member States to assign one or more competent authorities responsible for establishing a mechanism to “independently collect, evaluate, process, analyse and store details of occurrences”. However, the Regulation does not provide an explicit mandate to a competent authority nominated under the Regulation to oversee the implementation of the Regulation. Furthermore, in some Member States the competent authority under Article 6(3) is a Safety Investigation Authorities (SIA), or another independent body. Those bodies however are often not empowered to conduct oversight at the level comparable to the CAA (who have an experience in the certification, oversight and enforcement in the context of national regulations and Regulation (EU) 2018/1139). At the EU level, part of the EPAS concerns the monitoring of the implementation of EPAS actions. The implementation of safety actions identified through the domain-specific analysis carried out through the EASA CAGs is directly monitored by the relevant group. For the actions where Member States are not competent and where EASA acts as a competent authority, feedback on the implementation is regularly provided during advisory body meetings. It can be concluded that organisations, Member States and EASA put in place processes for the analysis and follow-up of occurrences. Whereas the Regulation introduced some changes in the areas where such requirement did not exist, it nevertheless appears that some stakeholders would welcome improvements in terms of cataloguing and monitoring the corrective and preventive actions adopted as the result of the occurrence reports’ analyses. Reduction in the number of aircraft accidents and fatalities The general objective of the Regulation is to contribute to the reduction in the number of aircraft accidents and fatalities through a proactive approach resulting from the specific objectives detailed in the preceding sections.

It is difficult to measure the degree to which the Regulation has contributed to an actual decrease in the number of accidents and fatalities in civil aviation, both quantitatively and qualitatively. Figure 5 shows the evolution of key statistics from commercial aircraft accidents and serious incidents of the European operators recorded by EASA. Between 2010 and 2018 the number of accidents has decreased in average terms. The decrease is more evidenced in the rate of accidents by 1 million flights (high severity). The rate of accidents has continuously decreased since 2014, while the rate of serious incidents stabilised after a peak in 2016. That peak was not an actual increase in incidents but rather a result of technical adjustments and improvements in the classification. Figure 6: Annual safety-related occurrences in total numbers and rate per 1M movements (Source: EASA Annual Safety Reviews (2019, 2018, 2017) After the entry into force of the Regulation, there has been a decrease in the accident rate, from 3.5 accidents per 1 million flights and 1.3 accidents per 1 million flights post-intervention (see Figure 6). Figure 7: Number of civil aviation accidents in the evaluation baseline (Source: results from the analysis of the baseline) The support study concluded that the Regulation contributed to the increase of the number of occurrence reports stored in the ECR, which in turn might have had contributed to the improvement of safety awareness and identification of safety risks. Consequently, Regulation (EU) No 376/2014 has likely helped to reduce the total number of accidents. Although it is almost impossible to model the exact number of accidents and other safety related occurrences in the absence of the Regulation, it can be concluded that, to some extent, the Regulation has contributed to that reduction. Conclusions on Effectiveness The Regulation has made a significant step towards achieving its specific objectives. It has led to improvements in the reporting across the stakeholder groups and across the EU. There has also been an improvement in the completeness and quality of the occurrence reports. However, the associated high workload related to filling in the mandatory data fields and the lack of feedback from the authorities affect to some extent the motivation to report. There has been an increase in the number of reports stored in the ECR, suggesting that the Regulation has been effective in meeting its objectives in the area of data integration. The complexity of the use of the previous version of the ECCAIRS software and the noted gaps in the flow of information between the Member States should be solved by the new and improved ECCAIRS 2 software. As regards the Regulation’s objectives related to the protection of information sources and the “just culture”, the Regulation provides an adequate level of protection of the reporters. However, the Regulation has not produced yet the desired progress in terms of advance administrative arrangements being concluded with the judicial authorities and the “just culture” bodies in the Member States. Likewise, whilst the Regulation caters for improved occurrence analysis and follow-up requirements, nevertheless no improvements in terms of analysis and follow up seem to be noted. The evaluation has not revealed any issues with the regulatory provisions themselves, but points to some difficulties with the availability of adequate resources for such analysis. In addition, the implementation of Regulation (EU) No 376/2014 and its Implementing Regulation 2015/1108 has caused increased workload for both organisations and competent authorities (the increased requirements for quality, content and timing of the occurrence reporting). Finally, no firm conclusions can be drawn as to whether the Regulation has succeeded in achieving its overall objective to contribute to the reduction of aircraft accidents and related fatalities. Nonetheless, the data presented above show the decrease in the amount of accidents when compared to the increasing level of traffic. Due to the improved collection and analysis of the safety data from the occurrence reports and the elaboration and implementation of relevant corrective and preventive measures, it can be concluded that the Regulation has to some extent contributed to this decrease in the accident rate, along with other safety measures put in place. 5.2.Relevance The core objective of the Regulation is to ensure that the information relevant to occurrences which endanger or could endanger aviation safety is reported, collected, processed, stored, and analysed with a view to identifying safety hazards that can then be adequately addressed through preventive or corrective actions. This proactive and predictive approach is increasingly important as there are fewer fatal accidents in the EU than globally making the traditional reactive approach of lessons learnt from such accidents unsuitable for the continuous maintenance of high levels of aviation safety. The aviation sector has experienced important developments since the entry into force of the Regulation. In view of the emerging technology advancements, as well as advancements in digitalisation, the study assessed whether Regulation is relevant and fit-for-purpose, notably given the upcoming rapid proliferation of the Unmanned Aircraft Systems (UAS) and the growing importance of cybersecurity. Unmanned Aircraft Systems

Thanks to the recent amendment to the Regulation brought by Regulation (EU) 2018/1139, reporting of occurrences and other safety-related information involving unmanned aircraft falls within its scope. However, in case of occurrences and other safety-related information involving unmanned aircraft for which a certificate or declaration is not required pursuant to the Regulation (EU) 2018/1139, the Regulation (EU) No 376/2014 only applies as long as the occurrence results in a fatal or serious injury to a person or it involves aircraft other than unmanned aircraft. As said, the occurrences involving the UAS fall within the scope of the Regulation as amended by Regulation (EU) 2018/1139. However, given the novel nature of the UAS operations and the recent entry into force of the specific regulatory framework regulating such operations there might be a need to adjust the list of mandatory data fields in the Annex to the Regulation to reflect the specificities of UAS operations. An issue to be closely monitored once reporting of occurrences with UAS aircraft advances. The Regulation however offers relatively simple solutions in the possibilities to update both the list of occurrences to be mandatorily reported and the Annex containing the mandatory data fields in order to make the Regulation more fit for this type of operations. It follows that the Regulation continues to be relevant in the area of UAS operations. Cybersecurity Cybersecurity is an important new challenge in the aviation safety sector. The growing risks emanate from the use of new technologies and new devices connected to the aircraft systems and for which there are currently no guarantees that there won’t be any interferences . It is expected that in the context of further smart and sustainable mobility developments, new technologies will continue to develop. Though the cyber aspect is not directly addressed in Regulation (EU) No 376/2014, nor in its Implementing Regulation 2015/1018, it is possible to address such aspects under the mandatory occurrence reporting categories of ‘meteorological conditions or security-related occurrences’ , ‘failure of ATM system security’ or ‘aerodrome security related occurrences’ . Cybersecurity should figure as one of the Key Risk Areas in the common European Risk Classification Scheme (ERCS), currently under development. It is expected that, similarly to UAS, a possible review of the mandatory data fields in light of potential cybersecurity threats will be needed when more experience is gathered. Relevance to other recent developments Some EU legislation adopted after the entry into force of Regulation (EU) No 376/2014 has introduced a number of provisions that might impact Regulation (EU) No 376/2014. This concerns Regulation (EU) 2018/1139, the General Data Protection Regulation (GDPR) , Regulation (EU) 2018/1725 on the protection of natural persons with regard to the processing of personal data by the Union institutions, bodies, offices and agencies and the Directive (EU) 2019/1937 on the protection of persons who report breaches of Union law (the “Whistleblowing Directive”) . In particular, the Whistleblowing Directive contains a number of provisions detailing the manner in which reporting channels should be set-up, and how breaches of the Regulation should be followed-up as well as measures of protection of reporting persons. Nevertheless, none of the above instruments affect the continuing relevance of Regulation (EU) No 376/2014. Furthermore, no concerns have been identified in relation to the environmental developments that might impact the continued relevance of the Regulation. Market and social developments have also been considered. The air transport sector has grown by 26% between 2014 and 2018 in terms of the number of passengers transported and its growth is expected to continue with a commensurate rise in competitiveness and commercial pressures to reduce costs. This could have an impact on the employment patterns (e.g. such as hiring aircrews via intermediaries or as self-employed or so-called "pay-to-fly"). Under such context people might feel less safe to report or sharing safety experiences being afraid that such reporting might put their jobs at risk. Therefore, the concept of “Just Culture” introduced by the Regulation is even more relevant. Conclusions on relevance Even in the light of the recent developments in the aviation sector, involving the rapid increase in the UAS operations and the increasing threats related to cybersecurity, the Regulation continues to be relevant and able to capture those trends. Where some adjustments to the Regulation’s provisions could be desirable, e.g. the updates to the list of mandatorily reported occurrences to better reflect the UAS occurrences or mandatory data fields to gather the UAS-specific data, the Regulation offers enough flexibility to introduce those changes in a timely and efficient manner. The Regulation also continues to be relevant in a broader socio-economic context. With its rules on the “just culture”, it offers an adequate level of protection to the reporters to foster reporting even in the increasingly more competitive environment. 5.3.Efficiency The global average annual financial impact of Regulation (EU) No 376/2014 for the European aviation industry as a whole has been estimated to be around €15.3 million in 2018. The apparent increase in costs due to Regulation (EU) No 376/2014 is mainly caused by a sustained increase in the workload associated with its implementation, mostly the increasing number of occurrence reports collected. Over 2014–2020 period, the administrative burdens borne in the implementation of Regulation vary significantly across Member States, industry key players and years, ranging from €7.32 million in 2015 to more than €15 million in 2018. The total annual cost of Regulation (EU) No 376/2014 for national competent authorities is estimated to be €7.7 million. There is, however, significant variation between Member States and years, ranging from estimated annual costs of €50,000 to more than €400,000. Over time, the total costs incurred by Member States due to Regulation (EU) No 376/2014 have increased from about €2 million in 2014 to almost €8 million in 2019, indicating a significant increase resulting from the need to process the large volume of reports. As a result, the average increase in workload is estimated at 3 full-time employees per Member State. The annual costs incurred for the exchange of information, analysis of occurrences and dissemination are estimated to represent the largest share of total costs for competent authorities in the Member States (53%), closely followed by the collection and processing of occurrences (47%). EASA has experienced an annual cost increase of €700,000 as a result of the implementation of the Regulation, merely in collecting and processing of occurrences (63%), analysis of occurrences, exchange of information and dissemination (16%). In addition, infrastructure costs relative to ECCAIRS updates to accommodate occurrence reporting developments have been estimated as €100,000 per year.

On the benefits side, the Regulation has introduced a benefit of approximately €66.48 million over the 2015-2019 period. The model employed, taking account of all monetised costs and benefits for the 2014-2019, results in a cost-benefit ratio of 1.14, indicating a positive return on investment. Conclusions on efficiency Calculations made in the context of the Support Study show that the benefits derived from the Regulation outweigh the implementation costs incurred by the various stakeholders. It follows that the Regulation has been efficient in bringing considerably more benefits than costs. 5.4.Coherence The evaluation did not identify any contradiction, overlap, duplication or inconsistency relative to the regulatory provisions set out in Regulation (EU) No 376/2014. The various components of the Regulation complement each other to achieve its objectives. Similarly, no major issues were identified in terms of coherence with its Implementing Regulation 2015/1018. In terms of external coherence, the Regulation has been assessed against the following EU Regulations: -Regulation (EU) 996/2010 No major inconsistencies or a lack of coherence has been identified. However, safety investigation authorities raised concerns that due to the lack of clarity between the two reporting systems serious incidents and accidents would be reported only to the competent authority under Regulation (EU) No 376/2014. Recital 2 to Regulation (EU) No 376/2014 explains that this Regulation should not interfere with the process of accident and incident investigations managed by national safety investigation authorities as defined in Regulation (EU) No 996/2010. In the event of an accident or a serious incident, notification of the occurrence is also subject to Regulation (EU) No 996/2010. It seems therefore, that there is a need for better coordination between the relevant national authorities, so that the serious incidents and accidents are not reported only to the competent authority under Regulation (EU) No 376/2014. -Regulation (EU) 2018/1139 Regulation (EU) 2018/1139 and its implementing rules establish obligations on certain organisations to establish occurrence reporting systems in the context of their safety management systems. A safety management system (SMS) aims to mitigate safety risks in a proactive way before they result in aviation accidents and incidents. For this, it is important to identify those risks and control them in a systematic and regulated way. When Regulation (EU) No 376/2014 was proposed, as reflected in its fourth recital, a need to avoid duplication of reporting efforts was highlighted. The Impact Assessment for Regulation (EU) No 376/2014 identified however a need for a specific regulation that would cover all occurrence reporting requirements, while at the same time establishing firm rules protecting the reporters and fostering the ‘just culture. At the time Regulation (EU) No 376/2014 was adopted it covered areas for which EASA had no parallel competence. That situation has changed over time with the extension of EASA’s areas of activities, although it has not yet produced any parallel coverage. The large majority stakeholders considered that both Regulations are consistent and coherent, and no specific problems were highlighted. EASA highlighted some practical difficulties. Although there are no legal constraints, it is often very difficult to maintain and update a coherent and effective set of safety management and oversight systems when (1) the personal and material scope of those legal acts vary, and (2) the related occurrence reporting provisions are spread across various legal acts, thus rendering their understanding difficult and hence possibly hampering effective safety management. Although the majority of stakeholders did not raise any issue in terms of coherence, consistency or duplication resulting from the co-existence of occurrence reporting provisions in those acts, a compilation of the existing Regulations into one act could be contemplated. It has to be born in mind however that certain fundamental aspects dealt with by Regulation (EU) No 376/2014 do not fall under EASA’s remit. Whereas the coexistence of various regulatory provisions dealing with occurrence reporting in different EU legislative acts has not been identified as a major hurdle or an impediment in the evaluation, a consolidation of those existing regulatory provisions by creating a one-stop-for all system which still guarantees protection of reports and a Just Culture provisions could be considered. -Other relevant EU legislation As regards coherence of Regulation (EU) No 376/2014 with other legislation, the assessment included, primarily, a range of legislative acts in the field of aviation security , where no inconsistencies, duplications or concerns were identified. As regards employment and social matters, no relevant EU legislation has been identified that would require coherency assessment. As regards protection of personal data, the coherence review concluded that there are potential inconsistencies between the General Data Protection Regulation and the data protection rules specific to EU institutions and agencies contained in Regulation (EU) 2018/1725 on the one hand and Regulation (EU) No 376/2014 on the other hand. The relevant competent authorities, organisations, EASA and the Commission while storing data on the occurrences must comply with the respective requirements laid down in Regulations (EU) 2016/679 and (EU) 2018/1725. This means, that if a report includes personal data, a relevant consent for the processing of such data as required by the GDPR must be obtained from all persons identified in the report.

However, when it comes to removal of personal data, the Regulation distinguishes between two forms of such removal. On the one hand, when uploading occurrence reports into the national databases and subsequently transferring them into the ECR, or when disseminating the information within the organisation concerned, the Regulation requires the disidentification of the data. Disidentified information is defined as “information arising from occurrence reports from which all personal data such as names or addresses of natural persons have been removed”. However, such information may still contain other contextual information, which may lead to the identification of the persons involved in the reported occurrence. Such data might however be essential for the meaningful reporting, the analysis and a follow-up of occurrences. At the same time, disidentified information can still contain personal data and hence compliance with the GDPR rules must be ensured. On the other hand, the Regulation requires a full anonymization, i.e. removal of all personal data when it comes to access to the occurrence data by the interest parties listed in Annex II to the Regulation or when the occurrence data are published by the Member States either as individual reports or in the annual safety reports. Finally, the Regulation’s coherence with Directive (EU) 2019/1937 on the protection of persons who report breaches of Union law was also assessed. The Directive lays down rules for common minimum standards to ensure the establishment of internal and external reporting channels and the adequate protection of reporting persons (“whistleblowers”). The Directive explicitly states that it shall complement the existing rules on the reporting of occurrences under Regulation (EU) No 376/2014, which at the same time constitute breaches of European law. Consequently, once transposed in 2021, the Directive is anticipated to have some impacts on the regime established by the Regulation. The Directive establishes, amongst other provisions, specific details on how reporting channels should be set-up, how reports should be followed-up, including an obligation to provide feedback to the reporting person. This is expected to complement the Regulation’s regime and potentially partly solve some of the shortcomings related to the lack of feedback from the competent authorities as identified in the section on effectiveness above. The Directive also provides for a number of enhanced measures that the Member States shall introduce to protect the reporters and penalties for breaching the confidentiality of the reports as well as for a retaliation against the reporter, which could, in some aspects, complement the Regulation’s own regime on the protection of reporters. In conclusion, the Regulation is both coherent with and complemented by the Directive. Conclusions on coherence The Regulation is internally coherent and contains no overlaps, contradictions or inconsistencies. Overall, the Regulation is coherent with other European legislation on aviation safety. The Regulation’s reporting requirements however partially overlap with the requirements contained in Regulation 2018/1139 and its implementing rules. Further analysis on the impacts of removing those overlaps, possibly through a consolidation process, could be considered. The Regulation is also coherent with other relevant EU legislation. 5.5.EU Added Value The Regulation has been found to bring EU added value in the following areas: First, the Regulation has standardised the approach to occurrence reporting and analysis throughout the EU, unifying criteria and leading to increased rates of reporting and higher quality reports. Results from the baseline analysis show a clear improvement in the reporting rates and completeness of reports. The Regulation has created a common framework at the EU level, which has facilitated the harmonisation of occurrence reporting, analysis and follow-up actions across Member States and across all industry domains. Second, the EU involvement has enabled the pooling of information and experience at the European level. This improves coordination among Member States, EASA and the Commission, for example via the Network of Analysts. All Member States are involved in the proceedings and have full access to the ECR data. It is expected that the exchange of safety information will improve with the help of modern and efficient e-tools (e.g. ECCAIRS 2, Data4Safety). The Regulation has enabled an EU-wide understanding on aviation safety that could not have been achieved by the Member States themselves and has in turn contributed to the further maturation of the EPAS. Third, the Regulation has supported the development of the “Just Culture” and helped to increase the awareness of the importance of occurrence reporting for safety, which in turn led to an increase in the number of occurrence reports submitted. Finally, with the upcoming replacement of the current ECCAIRS platform with a more modern and efficient software platform developed and managed by EASA and the adoption of a common risk classification scheme as required by Regulation (EU) No 376/2014 further improvements in the collection and analysis of the occurrences could be expected. On the basis of all those elements it can be concluded that the EU added value of Regulation (EU) No 376/2014 is very high. 6.Conclusions The Regulation has met all evaluation criteria. As the Regulation is a part of a wider aviation safety framework, it is difficult to isolate its effects. Nonetheless, the evaluation suggests that the Regulation has contributed to the overall improvement of aviation safety in the EU. While it is impossible to draw conclusions on the exact impact of Regulation (EU) No 376/2014 on the reduction of accidents and fatalities in civil aviation, there are some indications that the Regulation played a positive role in decreasing aviation accidents in Europe. The Regulation has made a significant step towards achieving its specific objectives. It has been particularly successful in increasing the number of occurrences reported and collected and it had a positive impact on the level of reporting. Nevertheless, some shortcomings relating to the big amount of mandatory data fields or too short reporting deadlines were identified, which could have led to an increased workload both at organisational and Member State levels and might have had an impact on the quality and completeness of the reports.

The Regulation has contributed to better collection and storage of the occurrence reporting. There has been a considerable improvement in the quality and completeness of the occurrence data contained in the ECR and national databases. This however implies a considerable workload for the involved stakeholders and especially for the authorities. Digital processing offered by the recently introduced ECCAIRS 2 software should mitigate some of those difficulties. Concerning data protection and exchange of information, as occurrence reporting uses no IT technology which would ensure full disidentification of the reports, personal details occasionally are registered in the national databases and the ECR. In this regard better enforcement of GDPR requirements is needed, and as of December 2021, the competent authorities should also ensure the confidentiality of within the obligations established by the Whistleblowing Directive. The evaluation concluded that access to the ECR is well protected and that ECR data have been used for safety purposes only. The Regulation however only partially achieved its objectives with regard to the protection of reporters and the "just culture”. Only a fraction of Member States established advance administrative arrangements with their respective judicial authorities to manage the access to and the use of occurrence reports contained in their national databases. Moreover, a number of Member States failed to designate a “just culture body” with the necessary competences to implement its duties and guarantees that reporters are not punished for reporting. To address this problem the Commission opened several infringement proceedings against Member States that are lagging behind. At the EU level, the Regulation has facilitated the analysis of occurrence reports through the standardisation of requirements what resulted in the increase of quality of reported data. The annual publication of the EPAS, including key risk areas and actions to mitigate safety concerns is a good example. At a national level, however, the evaluation has identified an apparent lack of analysis and follow up capabilities. A lack of expertise, both in terms of adequate resources and procedures seem to be the root causes. Furthermore, industry stakeholders have highlighted the lack of feedback from the competent authorities on the analysis of the reported occurrences. The Regulation’s main deficiency is the lack of clarity on the competent authority responsible for the oversight of the entire Regulation. The Regulation allows the Member States to nominate up to three ‘competent authorities’ to run the process from collection to analysis, with no further obligations on oversight. Some of those authorities seem not to be properly equipped to oversee the follow up actions. In terms of relevance, the evaluation shows that Regulation (EU) No 376/2014 and its objectives remain relevant also in face of the latest developments in aviation, allowing for taking account of the upcoming increase of UAS traffic. Regulation (EU) No 376/2014 has resulted in costs for both competent authorities and organisations, but the identified benefits clearly outweigh the costs incurred. In terms of coherence, Regulation (EU) No 376/2014 is fully coherent internally and overall, it is consistent with other EU instruments, notably with Regulation (EU) 2018/1139, Regulation (EU) No 996/2010 on accident and incident investigation, Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, Regulation (EU) on the protection of natural persons with regard to the processing of personal data by the Union institutions, bodies, offices and agencies and on the free movement of such data and the Directive (EU) 2019/1937 on the protection of persons who report breaches of Union law. The interaction between Regulation (EU) 2018/1139 and Regulation (EU) No 996/2010 warrant further consideration whether reporting and analysis obligations should be streamlined into one set of obligations. Finally, based on the evidence analysed, it can be concluded that the Regulation brought a clear added value in the area of occurrence reporting, analysis and follow-up. It has contributed to a clear improvement in reporting rates, completeness of reports, amount of safety-relevant data available across the EU as well as to the protection of reporter and introduction of “just culture” principles. Regulation (EU) No 376/2014 has fostered the coordination amongst Member States, EASA and the Commission via different groups (e.g. Network of Analysts). Exchange of safety information is possible via the access to the European Central Repository, set up by the EU by all Member States’ authorities (including safety investigation authorities). Annex 1: Procedural information 1.Lead DG, Decide Planning/CWP references Lead DG: DG MOVE Planning reference: PLAN/2019/5269 2.Organisation and timing Article 24 of Regulation (EU) No 376/2014 establishes the legal obligation to submit an evaluation report on the implementation of this Regulation to the Parliament and to the Council. The evaluation was launched in April 2019 by establishing an inter-service steering group (ISSG). The first meeting of the ISSG took place on 12 April 2019 and discussed the detailed planning of the evaluation exercise as well as the content of a draft roadmap outlining the procedural and methodological steps to be taken. The roadmap was published for feedback for a period of four weeks. The roadmap was finalised on 22 May 2019. The Commission launched a call for tenders for a support study on “Ex-post evaluation of Regulation (EU) No 376/2014 on the reporting, analysis and follow-up of occurrences in civil aviation”. A specific contract No MOVE/E4/SER/2019-422/SI2.811176 was signed with an external consultant Ramboll Management Consulting A/S on 23 May 2018 under the framework contract No MOVE/A3/2017-257. The external consultant delivered the first draft of the support study on 24 February 2020. The final delivery of the final report was sent on 12 May 2020 and the ISSG subsequently approved the final report and it was published by the Commission. The evaluation study is based on the assessment of the effectiveness, efficiency, coherence, relevance and EU added value of Regulation 996/2010, and reviews the objectives of the original regulatory intervention and the performance of the intervention as compared to the initial expectations and the current situation. The study also determines whether there are overlaps with other safety-related regulatory tools.

Based on the answers received in the OPC and based on the support study produced by the external consultant, the Commission proceeded with drafting this Commission Staff Working Document (CSWD). The ISSG was consulted on the draft CSWD on 16 and 30 September 2020. 3.Exceptions to the better regulation guidelines No exceptions to the Better Regulation Guidelines. 4.Consultation of the RSB (if applicable) N/A 5.Evidence, sources and quality The evaluation of Regulation 376/2014 was based on the intervention logic of Regulation (EU) No 376/2014 and a comprehensive analytical framework comprising the evaluation questions and their respective judgement criteria, indicators and information sources. The data collection tools used to gather the relevant information consisted of desk research, targeted online survey, targeted cost-benefit questionnaires, targeted interviews, expert meetings, public consultation and five case studies. The documents reviewed consisted of the EU legislation, reports and other communications, ENCASIA reports, ICAO documents, advanced arrangements concluded between the SIAs and national judicial authorities, safety recommendations, documents and reports by the national investigation authorities, and other sources, including academic articles and publications. The targeted online survey covered a representative selection of stakeholders involved in the or affected by the reporting, analysis and follow-up of occurrences in civil aviation including national competent authorities, safety investigation authorities, EASA, but also all different industry sectors, and associations representing staff in the aviation field. A total of 67 respondents fully completed the online survey, and 32 respondents partially completed it. The targeted cost-benefit questionnaire dissemination followed the same approach as that used for the online survey. In total 17 respondents completed the targeted cost-benefit questionnaire. Targeted interviews were conducted with a range of different stakeholder groups, and a wide geographical scope, namely with European Commission officials and representatives of the following organisations: ICAO, EUROCONTROL, EASA, competent authorities in EU Member States, Safety Investigation Authorities, just culture bodies / judicial authorities, commercial aircraft operators, manufacturers, maintenance industry, airports / ground handling operators, aviation personnel, General aviation / non-commercial operations / UAS, ANSPs, ATC service providers, research centre, and passengers. A total of 39 targeted interviews were carried out. In the context of the expert meetings, the study team facilitated two workshops and attended a third one as an observer with support from EASA and the European Commission. The first workshop was a full day workshop with competent authorities; the second workshop was a half-day workshop with the Network of Aviation Safety Analysts (NoA); and, the third workshop was a full-day workshop with a range of industry stakeholders. In addition, the European Commission launched an online public consultation on 7 November 2019 to gather the opinions of the broader stakeholder community on Regulation 376/2014. A total of 198 responses were received. Five case studies were conducted on the application of the Regulation 376/2014. The case studies were selected based on the criteria of the relevance vis-à-vis the evaluation questions, falling within the scope of the Regulation, being related to Member States compliance and based on the diversity and the availability of sufficient amount of details/information. Then, an assessment of the state of implementation of the Regulation 376/2014 was carried out based on data collected through desk-based research, stakeholder consultations, data collection tools, the online survey and interviews. This information was then triangulated to enable a judgement to be made on the extent to which the requirement has been implemented. Furthermore, an analysis of the evaluation baseline was carried out. In particular, the baseline analysis covered the effectiveness of the Regulation, particularly pertaining to the extent to which the Regulation has impacted the levels of safety in the EU. The methodology used concerns the Interrupted Time Series (ITS) methodology, which was considered the strongest quasi-experimental method to derive causal inference from the data available. Finally, a cost-assessment was carried out. The targeted cost-benefit questionnaires formed the basis of this assessment, supplemented by estimations where needed to fill the gaps. The methodology that was applied for this assessment is the Standard Cost Model (SCM), which allows to address the costs and burdens of Regulation 376/2014 for both Member States (competent authorities) and industry stakeholders. Annex 2: Stakeholder consultation

Introduction The objective of this synopsis is to provide an overview of the results of the stakeholder consultation carried out in order to evaluate the Regulation 376/2014 on the reporting, analysis and follow-up of occurrences in civil aviation. It includes a general analysis of the several methods used to reach the stakeholders, which were: ·An online survey launched on the 13th November 2019 and extended until the 9th December 2019; ·A cost-benefit questionnaire launched on the 13th November 2019 and extended until the 9th December 2019; ·Targeted interviews; ·Two workshops with competent authorities held on the 19th September 2019 and 30th October 2019; ·A workshop with industry stakeholders held on the 25th November 2019; ·An open public consultation launched on the 7th of November 2019 until the 30th of January 2020. The goal was to assess to what extent the aim of the Regulation was achieved, through an independent and evidence-based investigation. It also gave the possibility to stakeholders to express themselves and share their opinion on the progress made in accident and occurrence reporting in civil aviation since the Regulation entered into force. Consultation activities Online survey Launched on the 13th November 2019, the survey was targeted to a range of different stakeholder groups, including national competent authorities, safety investigation authorities, EASA, but also all different industry sectors, and associations representing staff in the aviation field. A total of 67 respondents fully completed the online survey, and 32 respondents partially completed it. Stakeholders were often unresponsive to the launch of the survey and consequently the deadline had to be extended and reminders sent. No answers were collected from three groups: judicial or law enforcement authorities, EFTA surveillance authority, and aerodrome operators. All other groups were represented, at least by a few respondents (Figure 1). The respondents were from 28 Member States (Belgium, Croatia, Cyprus, Czechia, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden, Switzerland and United Kingdom). The online survey included questions on the effectiveness, relevance, and EU added value of Regulation 376/2014, as well as a few questions pertaining to the degree to which the Regulation has been implemented in the Member States.

Cost-benefit questionnaire Launched on the 13th November 2019, the cost-benefit questionnaire was targeted to national authorities on one hand, and industry stakeholders on the other. On the authority side, a total of 14 completed questionnaires were returned (Figure 2), covering 14 different Member States On the industry side, however, only 3 completed questionnaires were returned: from a ground handling service provider, an aircraft parts maintenance organisation, and an aviation manufacturer. The cost-benefit questionnaire contained questions pertaining to the efficiency of Regulation 376/2014, including specific questions regarding the costs incurred as a result of the Regulation, and how these costs compare to the benefits achieved. Targeted interviews A total of 40 targeted interviews were carried out in the context of the evaluation support study. They cover a range of different stakeholder groups, and a wide geographical scope (Table 1). Some of the interviews conducted were group interviews involving two or more stakeholders, so the total number of organisations covered is higher than the foreseen number of interviews. Issues encountered in scheduling interviews (e.g. because of the Christmas holidays, and aviation professionals having limited time to talk to us) caused delays compared to the initially foreseen timeline. However, the deadline was extended to mitigate this, and the foreseen number of interviewees was attained. Overall, the study team judges the sample consulted to be representative of the breadth of stakeholders concerned by Regulation 376/2014. Stakeholder Planned Conducted ICAO 1 1 EUROCONTROL 1 1 European Commission 2 2 EASA 2 2 Competent authorities in EU Member States 5 5 Safety Investigation Authorities (SIA) 3 3 Just culture bodies / Judicial authorities 2 2 Commercial aircraft operators 4 4 Manufacturers 2 2 Maintenance industry 2 2 Airports / ground handling operators 3 3 Aviation personnel 3 3 General aviation / non-commercial operations / UAS 3 3 ANSPs 3 3 ATC service providers 2 2 Research centre 1 1 Passengers 1 1 Total 40 40 The targeted interviews aimed to complement the questionnaires by collecting more details on why stakeholders hold certain opinions, as well as gather additional evidence in relation to evaluation questions for which qualitative data was judged to be an important source. First workshop with competent authorities A first workshop with competent authorities was held at EASA premises in Cologne on 19th September 2019, after the SMTeB meeting. Hence, it was attended by representatives of the Member States’ competent authorities, some of which are SMTeB members. A total of 17 Member States were present at the meeting and six non-EU Member States (Moldova, Norway, Serbia, Montenegro, Iceland, and Switzerland), and EASA. The first workshop was a full day workshop with competent authorities, which aimed to gather insights into:

1.the effectiveness, efficiency and coherence of Regulation 376/2014; 2.reporting and collecting; 3.data protection, confidentiality and just culture; 4.exchange of information, dissemination and actions. The authorities were split up into discussion groups (ensuring a balanced geographical and gender representation in each group), where they discussed each of the above topics, following a list of predetermined questions. After two discussion sessions, participant rapporteurs were asked to present their key findings in plenary. Second workshop with competent authorities The second workshop with competent authorities was held on 30th October 2019 in Cologne. It constituted of a half-day workshop with members of the network of analysts (NoA). The topics covered were therefore primarily related to analysis and follow up. Considering the limited time and space available, live audience polling software was used to gather data from participants, specifically covering questions on four broad topics: 1.storage; 2.risk classification; 3.analysis; 4.actions and dissemination. The results of given questions were discussed by a participant panel, after which the floor was opened to additional comments from all other participants. This data was subsequently analysed in Excel, and completed with notes taken by a member of the study team who attended the workshop as an observer. Workshop with industry stakeholders The third workshop was a full-day workshop and was held on 25th November 2019 in the Albert Borschette conference centre in Brussels. The study team, with support from EASA and the European Commission, facilitated the workshop. A range of different industry stakeholders were invited to attend, to get as broad a stakeholder coverage as possible (Table 2). However, it proved difficult to get some groups on board, e.g. large airlines, airports, large aircraft manufacturers, ground handling service providers, maintenance organisations. Yet, a decent balance