# Presentation: 243205

## Data Quality Metric Based on Time-out CNX 

- CDM A&D Meeting
- 5 June 2001
- Dennis Gallus

## ORD GDP 3-25-01

## Purpose of Briefing

- Idea:  Encourage improvement in data quality
- Perhaps through permission to use slot-hold flag
- Choose airline-controllable performance measure:  Time-out cancellations
- Define the metric
- Set acceptable standards
- Determine fall-out for unacceptable performance
- Other possible metrics (later):
- CNX flights that fly
- Pop-up flights (without CDM messages)

## “First Principles”

- The number of wasted slots is what we want to reduce (through better data)
- A TO CNX flight during a GDP is potentially a wasted slot
- Metric should allow comparison across many airports (but not average over airports)
- Normalization required

## Normalization colors the result

**3/25/01****	****ORD****	**
**Metric:  #TO CNX by airline / Total # arrivals by that airline, expressed as a percent  ****	**
**Looks like:****	****	****	****	****	****	****	****	****	****	****	**
**	****	****AAL****	****ACA****	****AWE****	****COA****	****DAL****	****FDX****	****NWA****	****Other****	**** TWA****	****UAL****	****UPS****	****USA****	**
**	****	****0.2****	****4.3****	****7.7****	**** ****	**** ****	**** ****	**** ****	**** 5.5****	**** ****	****  1.6****	**** ****	**** ****	**

**The above metric implies that AWE contributed heavily to the TO CNX problem...**

## Normalization colors result--badly

**3/25/01****	****ORD****	**
**Metric:  #TO CNX by airline / ****Total # arrivals by that airline****  (%)****	****	****	****	****	****	****	****	****	****	****	****	**
**	****	****AAL****	****ACA****	****AWE****	****COA****	****DAL****	****FDX****	****NWA****	****Other****	****TWA****	****UAL****	****UPS****	****USA****	**
**	****	****0.2****	****4.3****	****7.7****	**** ****	**** ****	**** ****	**** ****	**** 5.5****	**** ****	****1.6****	**** ****	**** ****	**
_**The above is based on this data**_**:**
**	****	****AAL****	****ACA****	****AWE****	****COA****	****DAL****	****FDX****	****NWA****	****Other****	****TWA****	****UAL****	****UPS****	****USA**
_**total arrivals**_**	**
- 		**489****	****23****	****13****	****19****	****28****	****4****	****24****	**** 91****	****  11****	**** 571****	****    1****	****  21****	**
_**T/O CNX**_**	**
**	****	**** ****1****	**** 1****	**** ****1****	****0****	****0****	****0****	****0****	****   5****	****    0****	****   ****9****	****   0        0****	**

**In fact, AWE was no worse than three other airlines.  UAL was the biggest contributor to TO CNX.**

## Total Arrivals by all airlines at that airport is a better normalization

**Metric: #TO CNX by airline / ****Total # arrivals at that airport on that day**
**T/O CNX:****	**
**AAL****	****ACA****	****AWE****	****COA****	****DAL****	****FDX****	****NWA****	****Other****	****TWA****	****UAL****	****UPS****	****USA**
** ****1****	**** 1****	**** ****1****	****	****	****	****	****   5****	****	****   ****9****	**
**total arrivals at ORD on 3/25/01:  1295****	**
**Do division and multiply by 1000; round to a single digit to get a nice reference number that translates to “TO CNX per 1000 arrivals”**
**Metric now looks like:**
**AAL****	****ACA****	****AWE****	****COA****	****DAL****	****FDX****	****NWA****	****Other****	****TWA****	****UAL****	****UPS****	****USA****	**
** ****.4****	**** .4****	**** ****.4****	**** 0****	**** 0****	**** 0****	**** 0****	****  2****	**** 0****	**** ****4****	**** 0****	**** 0****	**

**This metric more accurately depicts each airline’s contribution to TO CNX problem.**

## Trend--All TO CNX at ORD, Apr

**Normalized by total # arrivals at ORD each day**

## Trend--All TO CNX at LGA, Apr

**Normalized by total # arrivals at LGA each day**

**Some airlines have near-perfect days between bad ones...**

## Controlled flights are most important

**Normalized by # slots in GDP**

**Compare to the previous metric:  Normalized by total # Arr at ORD**

## We can eliminate CNX-but-flew...

**Should CNX-but-flew flights count against airline data quality score?**

## Recommendations

- Track TO CNX for controlled flights
- Phase in statistical limit
- Perhaps 2 std.dev. at the start
- Running calculation, permit x bad days per month?
- Collectively decide where to put the bar

**2 std dev.**

**1 std dev.**

**Average**

## BACKUP SLIDES

## Desirable characteristics of data quality metric

- Simple
- Airline-controllable
- Reflect only the carrier being measured
- Operation-independent
- Minimally disputable

## We can eliminate TO CNX flights removed at first ADL update

**Qualitatively the same as that for all controlled flights**

## Data Quality issues

- Stale data
- Time-out delay and Time-out CNX
- Spurious data--airlines
- Substitution flights given inflated ETE
- CNX-that-flew (without proper reinstatement)
- Spurious data--system
- Duplicate ACIDs generated by ETMS
- Bad ETEs used in modeling
- Missing data
- No DZ or AZ msgs
- OAG flights (FS msg but no FZ)
- Bad operations on good data
- FSM algorithm flaws
- Specialist’s failure to enter end time of previous GDP in a revision, causing flights to be sorted by IGTA vice CTA
- Other GDP setup panel errors that sub-optimize result
- Other
- Effects of MIT during GDP

## Metron Aviation Under-delivery studies

- ERTA submissions
- MIT/GDP analysis
- C-Flow
- SMS/ETA predicting
- OOOI & ETMS
- Algorithm changes—flights with ETA <= GDP start
- Algorithm changes—flights with ARTA <= CTA
- Compliance
- Arrivals—Actual vs. AAR
- GS/GDP Order of Operations
- Training
- T0 CNX and data quality metrics

## Add a bar to indicate tolerable limit

- average

- 1 std.dev.

- 2 std.dev