# Presentation: 855638

## Development of an Industry Standardized Web-based Surveillance & Auditing Tool (WebSAT) to minimize maintenance Errors Prior to Dispatch by Airlines

- Department of Industrial Engineering
- Clemson University

**Clemson WebSAT Team:****Dr. Anand K. Gramopadhye ****Dr. Joel S. Greenstein**
**Pallavi Dharwada****Kunal Kapoor****Nikhil Iyengar**

**FAA Program Manager: Dr. William “Kip” Krebs****Industry Partner: FedEx**

## Outlinehttp://www.hf.faa.gov/docs/508/docs/maintTool051904.ppt

- Research Requirement and Execution Plan
- Premise
- Project Background
- First Year Objectives
- Methods
- Results and Deliverables
- Problems Encountered and Best Accomplishment
- Expected Conclusion
- Dissemination to-date

## Aviation Maintenance TCRG Requirement

- Develop existing PC /paper based auditing/surveillance tool to web-based application in performing auditing/surveillance/monitoring and validation of oversight of maintenance to ensure a consistent level of oversight is maintained.
- This system can proactively identify contributing factors of improper maintenance before aircraft is dispatched once work is complete.
- In addition, portions of this web-based surveillance auditing tool can be used by aircraft manufacturers before delivery of aircraft to their customers.

## Aviation Maintenance TCRG Execution Plan

- Execution plan states:
- identify an exhaustive list of impact variables that affect aviation safety and transcend across various aircraft maintenance organizations;
-  develop data collection/reduction and analysis protocol to analyze errors for the identified set of impact variables; and
- using the results of the aforementioned activity develop and implement a surveillance/monitoring tool to ensure all appropriate data is used to identify safety improvements in the maintenance environment.
- _FY03 _		_FY04_		_FY05_		_FY06_
- 	$64.4K	     $142.3K	     $163.7K        $129.4K

## Premise (1 of 3)

- Maintenance Error is a *crucial factor* in aircraft accidents
- In a recent study:
- 23% of accidents studied involved incorrect removal or installation of components
- 28% involved a manufacturer or vendor maintenance/inspection error
- 49% involved error due to an airline’s maintenance policy
- 49% involved poor design that led to maintenance errors

**Source: Rankin, W., Hibit, R., Allen, J., Sargent, R., Development and evaluation of the maintenance error decision aid (MEDA) process, International Journal of Industrial Ergonomics, 26, 2000**

## Premise (2 of 3)

- Maintenance accidents are analyzed ***after*** they have occurred
- HFACS
- There is opportunity for improvement of maintenance procedures
- Lack of standardization of data collection

**American Airlines Flight 1420 (June 1, 1999) **

**Notes:**
Lack of Standardization : Inconsistencies in maintenance within airline and across the industry

## Premise (3 of 3)

**Analysis of aircraft maintenance**
-  Closely monitoring and evaluating aircraft maintenance and inspection activities
**Surveillance and auditing activities**
** **Level of compliance
-  Objectives
-  Findings
-  Importance

**Notes:**
Analyzing Aircraft Maintenance: Closely monitoring and evaluating aircraft maintenance and inspection activities
Surveillance and Auditing activities
- Level of Compliance: This group constantly monitors and evaluates the flight procedures to determine their level of compliance
- Objectives: Objective of these activities is achieved through effective functioning of the quality assurance representatives and auditors who perform these activities
- Findings: Their findings help in the evaluation and assessment of the internal and external organizations (e.g., vendors) associated with the airline which influence the safety and airworthiness of aircraft
   - Importance: The surveillance and auditing activities are of foremost importance in ensuring adherence to the quality assurance requirements and also maintaining a consistent level of supervision over maintenance operations

## WebSAT – Project Background

- Addresses the need for a system that ensures quality performance of surveillance and auditing activities
- Identifies *proactively* the various factors that contribute to improper maintenance
- Develops and promotes a standardized format within and across all airlines for
- Data collection
- Data reduction
- Data analysis

## First Year Objectives (09/03 to 09/04) 

**Data collection **** **
**Identify customer needs**** **
**Identification of relevant process measures *****(previously referred to as impact variables)****** ***
**Submit quarterly reports periodically**** **
**Submit annual report**

**Notes:**
Data Collection: Collection of data from maintenance personnel, auditing and air worthiness directives group
Identification of relevant process measures: Develop variables which will ensure efficient running and easy monitoring of the processes within Surveillance, Auditing and Airworthiness DirectivesNote: WebSAT team and FedEx are contemplating on using the term impact variable which measures the process. Some of the other options being considered are “process measures” and “performance metrics”. 

## Methods (1 of 2)

**Task analytic & user centered software lifecycle methodology guides through the following Phases**
- I: Identifying process measures
- II: Design and develop WebSAT prototype
- III: Develop data analysis module

## Methods (2 of 2)

**Phase I**: Identifying process measures
***Discover***
- Understand surveillance and auditing functions
- Conduct interviews, focus groups and observations
***Identify***
- Determine process measures
***Validate***
- Ensure measures are representative of those used across maintenance facilities
- Survey other stakeholders to obtain consensus on process measures
***Construct***
- Identify WebSAT modules
- Develop module goals

**Notes:**
Stakeholders:
 Alaska, American, Delta and other airlines
 FAA Team

## Results and Deliverables (1 of 5)

***Discover:***
- Three Interview visits with the Quality Assurance departments at FedEx in Memphis, TN.
- Two Observation visits at Mobile Aerospace Engineering in Mobile, AL.
- Three trip reports documenting the information gathered.
***Identify:***
- Translated voice of the user to user need statement
- Developing metrics corresponding to user needs
- Defining process measures with FedEx Quality Assurance departments.
***Validate:***
- Developing web-based survey to validate process measures with other stakeholders

## Results and Deliverables (2 of 5)

***Results:***
- Developed preliminary version of WebSAT framework:

**Notes:**
WebSAT 3-tier framework: 
Data collected from the three modules (surveillance, auditing and airworthiness directives) will be analyzed using the identified process measures as shown in tier-1 which allows us to evaluate the efficiency of each respective module.
Further analysis of data will lead us to the categories in tier-2 which evaluates the performance of the airline across the three modules. 
WebSAT hopes to conduct analysis of tier-2 and estimate safety index of the airline by identifying the risk-causing factors represented in tier-3. 

## Results and Deliverables (3 of 5)

***Results:*** Identified probable process measures

***Currently in the process of developing a consensus on these aforementioned variables***

**Surveillance:  **(i) In-process Surveillance(ii) Verification Surveillance(iii) Safety/Fire protection(iv) Shelf Life

**Airworthiness Directives Control Group: ****	**(i) Compliance	(ii) Publication Changes(iii) Verification(iv) Applicability(v) Authorization(vi) EO/WIC Readability

**Auditing:  **(i) Administration(ii) Safety(iii) Manuals(iv) Records

## Results and Deliverables (4 of 5)

***Deliverables:***
- Two Quarterly reports – December’03 and March’04
- Three Trip Reports – October’03, January’04 and March’04
- Four conference papers submitted to IERC, SAHI and HFES presenting the ongoing research initiative
- Two research posters displaying the WebSAT research presented at Clemson University’s Research Forum poster competitions
***Work-in-progress:***
- Web-based survey to validate process measures
- Identify WebSAT modules
- Visit FedEx headquarters at Memphis, TN
- Visit FedEx maintenance facility, Greensboro, NC

**Notes:**
SAHI – Safety Across High-Consequence Industries

IERC – Industrial Engineering Research Conference

HFES – Human Factors and Ergonomics Society

## Results and Deliverables (5 of 5)

- Screen Shot of Web-based survey tool which is in process of development

## Problems Encountered &Best Accomplishments

**Problems**
- Developing and sustaining interests of other airlines
- Agreeing upon terminology
**Accomplishments**
- Identified probable process measures
-  Translated customers’ voice to need   statements

**Notes:**
Problems
Developing and sustaining interests of other airlines
Agreeing upon terminology: For example, WebSAT team and FedEx are contemplating on using the term impact variable which measures the process. Some of the other options being considered are “process measures” and “performance metrics”. 
Accomplishments
Identified probable process measures
 Translated customers’ voice to need   statements

## Expected Conclusion

**WebSAT is expected to achieve the following:**
- Support of maintenance related surveillance, auditing and airworthiness directive activities
- Identification of maintenance related process measures and related safety indices
- Identification of potential problems at geographically dispersed maintenance sites through data standardization
- Accessible display of trends in data

## Backup Slides

## Dissemination to-date

- Dharwada, P., Iyengar, N., Kapoor, K., Gramopadhye, A.K., and Greenstein, J.S., Web-Based Surveillance and Auditing Tool (WebSAT): A Proactive System to Capture Maintenance Errors, Proceedings of Safety Across High-Consequence Industries, St. Louis, Missouri, 2004.
- Iyengar, N., Kapoor, K., Greenstein, J.S., and Gramopadhye, A.K., Selection of Data Gathering Methodologies for the Aviation Maintenance Industry, Proceedings of the Industrial Engineering Research Conference, Houston, 2004.
- Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J.S., and Gramopadhye, A.K., Standardized Auditing and Surveillance of the Aircraft Maintenance Operations, Proceedings of the Industrial Engineering Research Conference, Houston, 2004.
- Kapoor, K., Dharwada, P., Iyengar, N., Greenstein, J.S., and Gramopadhye, A.K., A Strategy for the Development of a Web-based Tool to Reduce Aviation Maintenance Errors, Proceedings of the Human Factors and Ergonomics Society, New Orleans, 2004.