# Presentation: 621096

## Anomaly Detection from ASRS Database of Textual Reports

- Hanhuai Shan
- shan@cs.umn.edu

- Arindam Banerjee
- banerjee@cs.umn.edu

*University of Minnesota, Twin Cities*

- Conference on Intelligent Data Understanding

**This work was supported by NASA under award NNX08AC36A**

## ASRS Database

- I WAS FLYING THE KATANA WITH A STUDENT AND ON DOWNWIND **THE FUEL PRESSURE DROPPED TO ZERO, AND THE ENG WAS CUTTING OFF**. I VERIFIED FUEL PUMP WAS ON AND IT WAS ON. BY THE TIME WE TURNED SHORT FINAL, THE PROP STOPPED AND WE LANDED THE AIRPLANE SAFELY. THEN WE CALLED CASTLE UNICOM TO SEND THE FUEL TRUCK

**Narratives report an anomaly:**

**Notes:**
ASRS Database interface and a sample report.

## Goal

- Automatically discover various types (categories) of anomalies from textual reports.
- e.g. Maintenance, Weather...
- Why?
- Put each report into a certain category/categories.
- Which report reports which problem(s).
- Correct the reports that are in wrong categories in the database.

- ...RPTR FURTHER STATED THAT **THIS HAS BEEN A PROBLEM FOR SEVERAL YEARS** **WITH VERY LITTLE DONE BY THE ARPT**...

## Mixture of von Mises Fisher Distribution [Banerjee et al, 2005] 

- Data points (reports) lie on a unit hyper-sphere.
*     **- **mean direction*
*     **- **concentration parameter*
-  Example: Three types of reports could be represented by three vMF distributions (red, green, yellow) – mixture of vMF.

**Notes:**
Method I: Mixture of vMF (movMF)

## Latent Dirichlet Allocation [Blei et al, 2003]

- document1

- document2

- For each document,
- Choose *π* ~Dirichlet(**)
- For each word *x*n:
- Choose a topic *z*n ~Discrete(*π*)
- Choose a word *x*n from *p*(*x**n**|z**n**,***)*,* a Discrete distribution conditioned on the topic *z*n.

**Notes:**
Method II: Latent Dirichlet Allocation –A three-level Bayesian Model.

## Confusion Matrix and Topic Lists for a Three-category Dataset

- Flight crew human performance.
- Passenger.
- Maintenance human performance

|  | 1 | 2 | 3 |
| --- | --- | --- | --- |
| 1 | 1185 | 45 | 35 |
| 2 | 12 | 1150 | 49 |
| 3 | 169 | 42 | 1538 |

| Flight Crew | Passenger | Maintenance |
| --- | --- | --- |
| rwy
apch
acft
dep
alt
turn
time
atc
flt
twr  | pax
flt
attendant
capt
seat
told
asked
back
attendants
acft  | acft
maint
eng
zzz
flt
mel
chk
fuel
time
gear  |

- Numbers on the diagonal –number of correctly clustered reports

**Notes:**
The dataset contains 4226 reports with three causes of the problem.

Confusion matrix is at bottom left. Row denotes the true clusters. Column denotes the result clusters from our algorithm. So the entry (i,j) is the number of reports whose true cluster is i being put into the jth cluster buy our algorithm. The numbers on the diagonal show the number of reports correctly clustered. 

The bottom right table shows the word list for each topic (cluster). In general, the algorithm gives the result that each topic is represented by a distribution over all words in corpus. We pick the top ten words as the representatives for each topic.

## Each point represents one report.
The color of the point – the report’s label.
The location of the point – mixed membership from LDA +ISOMAP.
Focusing on:  Points having different colors with the neighbours
     Isolated points

- The color of the point – the report’s label.
- The location of the point – mixed membership from LDA +ISOMAP.
- Focusing on:  Points having different colors with the neighbours
-      Isolated points

- Two-Dimensional Visualization for Reports

**Red: Flight Crew****                  **
**Blue: Passenger****            **
**Green: Maintenance**

**Notes:**
For each report, our algorithm gives a k-dimensional representation, where k is the number of clusters. We further project them to a two dimensional space via ISOMAP to visualize the points.
The colors show the true clusters: Red: Flight crew, Blue: Passenger, Green: Maintenance.

We marked several points that are not consistent with their true clusters and take a closer look at these reports. For example, 680’s true cluster label is Passenger, but the algorithm puts it into the flight crew cluster. Similar with the next three slides

## Two-Dimensional Visualization for Reports

**The pilot flies an owner's airplane with the owner as a passenger. Lose contact with the center during the flight.**

**While performing a sky diving, a jet approaches at a same altitude, but the accident is avoided finally.**

- Red: Flight Crew                  Blue: Passenger            Green: Maintenance

## Two-Dimensional Visualization for Reports

**Altimeter has a problem, but the pilot overcomes the difficulty during the flight**.

**During acceleration, a flap retraction issue happens. The pilot then returns to base and landing. The mechanician finds out the problem.**

- Red: Flight Crew                  Blue: Passenger            Green: Maintenance

## Two-Dimensional Visualization for Reports

**The pilot has a landing gear problem. Maintenance crew joins radio conversation to help. **

**The captain has a medical emergency. . **

- Red: Flight crew                   Blue: Passenger             Green: Maintenance

## Two-Dimensional Visualization for Reports

**After landing, there is a trouble with getting the jet bridge into position. Flight crew evacuate the passenger**.

**A passenger seems  to have a mental problem.**

- Red: Flight Crew                  Blue: Passenger            Green: Maintenance

## Mixed Membership of Reports

- Red: Flight Crew                  Blue: Passenger            Green: Maintenance

**Flight Crew: 0.7039**
**Passenger: 0.0009**
**Maintenance: 0.2953**

**Flight Crew: 0.1405**
**Passenger: 0.0663**
**Maintenance: **** 0.7932**

**Flight Crew: 0.2563**
**Passenger: 0.6599**
**Maintenance: **** 0.0837**

**Flight Crew: 0.0013**
**Passenger: 0.0013**
**Maintenance:  0.9973**

**Notes:**
The algorithm gives mixed membership for each reports.
Compare the membership between those “mis-clustered” with that of a typical Maintenance report.  

## Fast LDA - A More Efficient Algorithm for LDA

- Perplexity is a monotonically decreasing function of log-likelihood, evaluating how the model fits the data –the lower the better.
- Fast LDA is much faster than LDA, with a similar perplexity.

**Time comparison**

**Perplexity comparison**

## Word Lists for Topics

- Word lists from LDA and Fast LDA are similar.

## Future Work

- Automatically detect the outliers.
- Taking advantage of existent labels to reports, develop a semi-supervised version of LDA and moVMF.
- One report could belong to multiple categories. Develop an overlapping clustering algorithm.