Reducing Commercial Aviation Fatalities | Kaggle Competition
By: Vijay Maurya
Tags: Kaggle, Aviation Fatalities, Kaggle Competition, Lightgbm

From Google Image

I was working on Kaggle competition which is organized by Booz Allen Hamilton company on Kaggle. From the title, we can understand competition is about aviation fatalities we have to predict fatalities based on given data in competition. So let’s discuss data provided for competition and how I solve this.

Table Content

Overview About Kaggle Problem All about Data Problem Type EDA -Exploratory Data Analysis Feature Engineering Model Training Conclusion Improvement

1. Overview

Any medium of travel one of import thing is safety which may occur due to mechanical or human error. There are many reasons why a plane might crash, including bad weather, imprecise navigation and pilot error. In fact, pilot error is the single biggest factor leading to a CFIT(Controlled flight into terrain) incidents (according to aircraft manufacturer Boeing). Pilots who may be distracted, sleepy or in other dangerous cognitive states.

2. About Kaggle Problem

In this dataset, we have given 18 pilot’s physiological data with their cognitive state and events. Our goal is to build a model which can detect their events, So pilots could then be alerted when they enter a troubling state, preventing accidents and saving lives. The data acquired from actual pilots in test situations and our models should be able to run calculations in real-time to monitor the cognitive states of pilots.

3. All about Data

Data is collected in Line Oriented Flight Training consists of a full flight (take off, flight, and landing) in a flight simulator (non-flight environment, outside of a flight simulator).

The pilots experienced below distractions

1. Channelized Attention (CA): the state of being focused on one task.

2. Diverted Attention (DA): the state of having one’s attention diverted by actions.

3. Startle/Surprise (SS): the state of surprising by when one event is occurring.

Variables with the EEG prefix are electroencephalogram recordings.

id — (test.csv and sample_submission.csv only) A unique identifier for a crew + time combination. we must predict probabilities for each id.

crew — unique id for a pair of pilots. There are 9 crews in the data.

experiment — One of CA, DA, SS or LOFT. The first 3 comprise the training set. The latter the test set.

time — seconds into the experiment

seat — is the pilot in the left (0) or right (1) seat

EGG- eeg_fp1,eeg_f7,eeg_f8,eeg_t4,eeg_t6,eeg_t5,eeg_t3,eeg_fp2,eeg_o1,eeg_p3,eeg_pz,eeg_f3,eeg_fz,eeg_f4,eeg_c4,eeg_p4,eeg_poz,eeg_c3,eeg_cz,eeg_o2 : Electroencephalogram recordings (EGG) which have different parameters.

ECG — 3-point Electrocardiogram signal. The sensor had a resolution/bit of .012215 µV and a range of -100mV to +100mV. The data are provided in microvolts.

R— Respiration, a measure of the rise and fall of the chest. The sensor had a resolution/bit of .2384186 µV and a range of -2.0V to +2.0V. The data are provided in microvolts.

GSR — Galvanic Skin Response, a measure of electrodermal activity. The sensor had a resolution/bit of .2384186 µV and a range of -2.0V to +2.0V. The data are provided in microvolts.

Event — The state of the pilot at the given time: one of A = baseline, B = SS, C = CA, D = DA

4. Problem Type

This is a classification problem in that we have to predict any event out of 4 events-based on given data.

A = baseline, B = SS, C = CA, D = DA

5. EDA -Exploratory Data Analysis

Let’s explore the data for that we have to do exploratory data analysis. So I had loaded both train and test data.

Let’s see train data sample

training file sample data

5.1. The first step to explore train data is to check data is balanced or imbalanced. So I plotted a bar plot based on the label value.

Bar plot of the Target value in training data

As we can see in above plot Event A, B, C, D have the completely different count percentage, we can understand from above plot D and B are in very less number like D=4.8 % and B is 2.7% which very less in compare to A and C. So if labels count in the dataset is not equal or about to equal that dataset is called imbalanced dataset. For make data balanced we need to oversample and under-sample after that we can train the model.

5.2 Experiment count plot

Compare between experiment and Event

when CA experiment happen A event is occurred on 0.1%, B=0%, C=34% and D=0% When DA experiment Event A is 29%,B=0,C=0%,D=4.8% When SS experiment event A is 29%, B=2.7%,C=0%,D=0%

Let’s see data in the test dataset

Test Dataset experiment data

We can see in Test dataset we have a completely different experiment which is LOFT — Line-oriented flight training

5.3 Time

Time for every record

Time taken for event A, C, D is about to equal. if we see mean of time taken for these 3 events is about 175 where event B meantime is 100. Event B,50% data is taken 60 to 100 and the remaining 50% data time is between 100 to 310 millisecond. Other 3 event A, C, D time taken of the experiment is about equal, we can see the plot having very small changes in that.

Let’s compare time distribution in train and test data set

Train and Test time distribution

From the above plot, we can understand that there is a huge difference in Train and test time of the experiment. So the use of time in training time may make our prediction makes the wrong direction. time in flight simulator nothing to do with time in the experiment

5.4 Electrogastrogram Reading:

EGG Box plot

I have plotted Box plot for EGG feature.Box plot show the quartile of data like 0,25,50,75,100 quartile .

1. In every plot off EGG feature we can see that event B have 0 to 100% shape larger than other which can distinguish from others event.

2. Using Box Plot we can see every plot looks about to same so we can’t get more details from box plot.

I have plotted EGG reading for every event.you can visit EDA ipython notebook to check this information I am putting final observation of EGG over here

Observation is as below.

1. As we know EGG reading taking with 20 electrode which is putting on different different position on scalp of head.

2. Based on every event reaction There is electrode getting more fluctuate . You can see in below image

Event A = Fp1,Fp2,F3,C3,Cz

Event B = F8,T4,T3,F4,P4,C2

Event C = Fp1,F8,T4,O1,F2,F4,F3,C2,C3

Event D = Fp1,Fp2,F3,C4,C3,C2

Now Let’s check train and test data distribution

Train and Test data distribution

We can see above both Train and Test EGG data follow about to normal distribution but test data have a bigger peek at zero(0) and more variance in test data. In the end, we can say train and test EGG data is coming from the same distribution. If train and test are coming from the same distribution model better because the model will find train type data in test dataset which is easy for the model to predict it.

5.6 ECG

Based on Labels I have plot violin plot of ECG features

ECG plot bases of Labels

In ECG violin plot we can see that all event plot looks about the same and there is 50% data range is also the same for all event. The event we can see data density between 700 mV to 1000 mV is high in all event.

I have plotted every Event frequency, from this plot we can say that every event has followed a different frequency range in ECG data. So ECG may play a big role in event classification. If we notice Event B has the clean frequency and Event A have noisy frequency based on that separation of both event is very easy.

5.7 Respiration:r

From respiration violin plot we can understand that A, C, D Event violin plot about same there 50% value is also coming about same on about 740μV event B There 50% data is coming on about 760μV. And if you Notice on Event A data is between 450 to 850 μV but other event data is about 540μV to 850μV

5.8 Galvanic Skin Response( GSR )

GSR

GSR Violin plot is almost the same as all other feature we have same B feature little difference. Every event data distribution for GSR is also about the same because if we see bottom side all violin plot is the same expect then B.

If we see the dark line of violin plot about the same range but the event is little larger then others.

Conclusion from EDA

Data is completely imbalanced so I have to oversample or under-sample dataset before the train. Time in Train and Test dataset is completely different distribution so the use of this in the train may our prediction in the wrong direction. EGG Reading violin plot is about same for every event, if we see EGG signal plot we will see different between every event on the different -different electrode on the scalp(see in scalp image). If we see the frequency plot of ECG, respiration,r we can see all three feature follow about the same pattern of frequency for every event.

Now let's come to Feature engineering part

6. Feature Engineering

We know that we training dataset is imbalanced so we have to make balanced with oversampling. So I have used the SMOT technique to oversample data I found one article about physiological parameters in that he has mentioned how EGG data in clinically reading form so that we can found something meaningful. So based on that we did feature engineering.

We can see below image how the electrode is set on the head scalp. For clinically reading we need to subtract from one electrode to another electrode for example eeg_fp1 — eeg_f7. It doesn’t matter which way you do it, as long as it’s consistent. I did this from front to back.

electrode setup on head scalp

Now our training data is ready for the train. Let’s talk about the model training part

7. Model Training

In the end, we need to train the training dataset using different machine learning algorithms and find the best result for test data because at the end we need to get the best score on Kaggle, the basis of test data prediction.

Approach 1

We have seen in EDA, time and experiment data in train and test dataset is completely different.so basic idea is drop those feature because they lead us in a different direction.

For check worst log loss I had created a random model which predict randomly event I found that on this dataset worst log -loss is about 1.64 that means our machine learning model must be less than this log loss.

Now I have tried first model Logistic regression which log loss is about 1.37. So this was not much improved so I understand any linear model will perform badly.

So I had tried Random forest and LightGBM in that I found there is much improvement in log loss, it is about 0.144 of Random Forest and 0.133 for LightGBM.I found a very good log loss so I have submitted test data basis on these models.

I found Test result on Kaggle completely shocking

Approach 2

I thought maybe I have to try with time and experiment features also so I have trained and submitted test data I found the result is about to same not change much. I am still far from the top score which is 0.300

Approach 3

If you notice first count plot of labels we can see only about 50% of train data is event A and other 3 are reaming 50%.So if we doing oversampling then we training event mostly B and D on mock data. which may cause our model prediction wrong. So I have tried model prediction without oversampling using LightGBM. I found a huge improvement in my Kaggle score.

Now I am on 12th position on the basis of the leader board score.

Approach 4

We know that we have time feature in the training dataset if you notice time is given interval of 0.003906. so I have treated as time-series data and did not split the training data randomly and train the model.

I found this model score on Kaggle test data has huge improvement it took my score from 12th position to about 3 rd position.

8. Conclusion

From model training part you can understand, Machine learning is all about the experiment. According to EDA, I thought time and experiment may cause prediction in the wrong direction but they play a big role in prediction. So my approach 4th on train dataset with time, experiment and without shuffling train_test_split giving very good result on test data.

9. Improvement

In this dataset about all features are biological reading so if we use the biological library and extract meaning full data then we can improve results. As we have seen time feature made a huge improvement in the Kaggle score. So we improve model using time series analysis using LSTM. We can also try Deep Learning Model on this dataset.

You can find my complete solution on Github link if have any suggestion or you like this article please comment your view on this article.

Reference