Titanic dataset-Advanced data exploration in Python | Towards Data Science

Unique vignettes tumbled out during the course of my discussions with the Titanic dataset. Some of them well documented in the past and some not. A few examples:
Would you feel safer if you were traveling Second class or Third class? Contrary to popular opinion made famous by Hollywood movies, you had DOUBLE the chance of surviving the Titanic as an adult male if you travelled THIRD class instead SECOND class
“What’s in a Name? That which we call a rose, By any other name would smell as sweet” — Act II, Scene II. Unfortunately the Titanic data set seems to violently disagree with Juliet (& the old Bard) for people with short names had extremely high mortality rate on the Titanic compared to people with long names
A case for Altruism on the high seas — Darwin was proved wrong that night, but does the data speak of any other acts of altruism that went undocumented? 3 separate pieces of evidence present themselves on closer inspection
Oops Errata on the Titanic dataset tutorial blogs — One original mistake, several copies
Females on the Titanic may have been charged a little more — But males paid with a lot more
Correlation is not Causation — The fallacy of the ‘Survival of the Groups’ conclusion
What is the single biggest indicator of survival on the Titanic data set — No it is none of the 11 columns listed in the dataset
Fill in the blanks — Is there a vital clue regarding the female-children that seems to have overlooked while filling the missing ages?
No matter what the leaders say — going solo does not necessarily mean you had lower chances of survival. If you were a female, you WOULD want to go solo in order to survive the Titanic. Upon probing why, the data narrates a heart-wrenching tale involving the ultimate act of sacrifice that played out on that fateful night more than 100 years back...
It is almost a decade since the Titanic competition was hosted by Kaggle. It is like the proverbial ‘Rite of Passage’ for any aspiring data scientist. You can see some of today’s grandmasters at Kaggle who are amongst the world’s best data scientists leave their indelible stamp on this dataset sometime in the past. It is a ridiculously simple and small dataset but provides a surprisingly good exercise in almost all aspects of data science and Machine learning — starting from data exploration/visualization to guessing missing values, feature engineering and to testing out a number of ML models.
The details of the competition may already be well known and I summarize it in just 2–3 sentences — It involves predicting whether passengers on the Titanic would survive or not..given a set of characteristics like their age, Sex, class of travel, port Embarked etc. The aspiring data scientist is supposed to make his model ‘learn’ from the training data set (which contain an additional column on whether the passenger survived or not) and apply these ‘learnings’ to the test data set and predict which of the passengers in the test dataset survives.
Though many years have passed, there is still a surprising amount of information hidden in the dataset. With a bit of coaxing though, it is more than willing to give up its secrets. So without much ado, we jump into transcripts of the discussions I had with this fascinating dataset in Python — a language I learnt just to converse with it. We will solely use the Kaggle dataset to come to our conclusions and will not refer to any external source of data. So this analysis is to be looked at from that perspective only and may/may not be a reflection on the actual events.
import pandas as pdimport seaborn as snsimport matplotlib.pyplot as plttest_data = pd.read_csv (‘/kaggle/input/titanic/test.csv’)train_data = pd.read_csv(“/kaggle/input/titanic/train.csv”)##These survival rates will be used as a referenceprint(round(train_data [['Pclass','Survived']].groupby(['Pclass']).mean()*100,1))print(round(train_data [['Sex', 'Pclass','Survived']].groupby(['Pclass', 'Sex']).mean()*100,1))
The basic observations are well known and are henceforth summarized in couple of quick sentences. The Sex of the passenger and the Class in which they travelled (Pclass) seem to have the highest correlation with survival. Age seems to play a role. Other factors like Port Embarked seem to play a small role. The numbers above seem to be in accordance with the general expectations as well.
We start our analysis with a Violen plot.
sns.violinplot(x=’Pclass’, y=’Age’, hue=’Survived’, split=True, data=train_data, palette={0: “r”, 1: “g”});
Straight away we see some puzzling things that need to investigated. Pclass 2 and 3 have a very sharp pronounced peaks of expiry near the 20–35 age range. This is not the case with Pclass 1 though. PClass 1 has far more survivors across all age levels except maybe after 50 or so. After 60 there is a sharp jump in expiry. This is not as sharp in Pclass 2 or 3. This is a curious phenomenon. Let us investigate a bit.
print('Pclass 1 survivors above Age 60:', round(len(train_data[(train_data['Pclass']==1) & (train_data['Age']>59) & (train_data['Survived']==True)])/len(train_data[(train_data['Pclass']==1) & (train_data['Age']>59)])*100,1), '%')print('Pclass 2 survivors above Age 60:', round(len(train_data[(train_data['Pclass']==2) & (train_data['Age']>59) & (train_data['Survived']==True)])/len(train_data[(train_data['Pclass']==2) & (train_data['Age']>59)])*100,1), '%')print('Pclass 3 survivors above Age 60:', round(len(train_data[(train_data['Pclass']==3) & (train_data['Age']>59) & (train_data['Survived']==True)])/len(train_data[(train_data['Pclass']==3) & (train_data['Age']>59)])*100,1), '%')print('Pclass1 survivors between 20-30 Age:',round(len(train_data[(train_data['Pclass']==1) & (train_data['Age']>19) & (train_data['Age']<31) & (train_data['Survived']==True)])/len(train_data[(train_data['Pclass']==1) & (train_data['Age']>19) & (train_data['Age']<31)])*100,1),'%')print('Pclass2 survivors between 20-30 Age:',round(len(train_data[(train_data['Pclass']==2) & (train_data['Age']>19) & (train_data['Age']<31) &(train_data['Survived']==True)])/len(train_data[(train_data['Pclass']==2)&(train_data['Age']>19) &(train_data['Age']<31)])*100,1),'%')print('Pclass3 survivors between 20-30 Age:',round(len(train_data[(train_data['Pclass']==3) & (train_data['Age']>19) & (train_data['Age']<31) &(train_data['Survived']==True)])/len(train_data[(train_data['Pclass']==3) & (train_data['Age']>19) &(train_data['Age']<31)])*100,1),'%')
Indeed in Pclass=1, there is nearly 2.5 times more survival chance if one is between 20–30 age as compared to 60. As the Pclass changes we can see how the survival gap narrows between the 2 age groups. What could explain this? Is this an act of altruism — letting the younger generation live? Why is this limited only to the first class? Is it just a quirk of data and does not mean much? A quick analysis of data shows that Pclass 2 and Pclass 3 have statistically insignificant count of older passengers. Any survivals would possibly skew the percentages. But Pclass1 has 17 people greater than 60 which is not an insignificant number.
One possibility is it that there were more males among the elderly and more females among the middleage group in Pclass 1..so that is the reason for the disparity. A quick series of commands reveal that only 2 males > 60 out of 14 in pclass 1 survive as compared to 9 out of 19 for middle age group. Thus even after accounting for all parameters we see that this difference of 14% survival versus a near 50% which is a mystery. It is not that the middle age folks pushed around the older lot. In fact much of the evacuation was orderly except for the very last few unfortunate minutes. Most of the folks who exipred were able bodied middle aged males..which is why I talked about Darwin beng proved wrong that night as the first act of altruism. But looks like there is a twist to this theory as far as first class passengers are concerned. It does look like some of the older males voluntarily made way for the younger males..Could this be the 2nd act of altruism on the Titanic? The data is very small but not entirely insignificant and it does point towards that direction.
Another stunning data presents itself when we further add the ‘Sex’ column to the filter. We see that NONE of the 33 Pclass 2 males in the middle age group (20–30) age group survives. The survival rate for Pclass2 Male passengers is 16% as we have seen earlier, so this is strange. In fact it is so curious that I extended the age ranges and found that only 4 out of 72 males between 20–60 age group survived. This is just 5% compared to the survival rate for Pclass2 males which is 16%. This could mean either of 2 things: Either in Pclass2 more kids survived than normal or there were more %age of kids in Pclass2 males. This means that adult males in Pclass 2 had actually less survival chance compared to Pclass 3 and kind of turns popular theories around the head. Let us check this...
Unbelievably fascinating and a wonder this has not been documented so far. To put it bluntly if you were an adult male, you had double the chances of survival if you got yourself booked in 3rd class on the Titanic rather than 2nd class! This does turn popular theories on the head.. We have discovered 3 acts of altruism so far on the Titanic. The first one was well known. The second is not very conclusive due to lack of enough data but is not statistically insignificant. The third was a big surprise to me. A huge chunk of Pclass 2 males voluntarily gave up their lives so that more females and children (from all classes) survive. Are there any more acts left? Let us see.
But before that let us analyze the Embarked column with a facetgrid plot.
fg = sns.FacetGrid(train_data, row='Embarked', aspect=2)fg.map(sns.pointplot, 'Pclass', 'Survived', 'Sex')fg.add_legend()
Did you notice something ridiculously strange happening on Embarked=C port? Those passengers who embarked on port C have totally reversed the odds for survival as far as Sex is concerned. Most of the males live and the females expire! This is FALSE info & can quickly be verified by a query to gather the %age of survivors grouped by Sex for Port C embarkation. Obviously the colors for Male & Female have got mixed up in the middle row. If your console warnings are suppressed you may not even notice it.. Luckily the warning is clear. Please use the ‘order’ parameter else you may get incorrect ordering
fg = sns.FacetGrid(train_data, row='Embarked', aspect=2)fg.map(sns.pointplot, 'Pclass', 'Survived', 'Sex', hue_order=['female','male'], order=[1,2,3],palette={'female':"r",'male': "g"})fg.add_legend()
Better! One very popular blog post (& at least a dozen other copies) make this erroneous observation and go on to other topics as if it were perfectly normal data behaviour. My sole intention in making this limited point is to request young readers not to go about the data exploration phases as if it were a formality that needed to be quickly gotten over with before jumping into the real thing — the ML modelling. Please dig into the data especially if you see patterns or anamolies which need further investigation.
Let us take very small deviation now.. Just for fun, I tried analyzing fares for women passengers to see if they differed from men. This was merely a distraction... and laying out (possible) proof of gender discrimination as far as fees were concerned was never on my agenda but here were the observations:
Looks like on an average, women’s tickets were priced higher by about 20% in Pclass 1, 8% in Pclass 2 and 4% in Pclass 3. I rest my case. Maybe the men bargained more to get better deals? In any case, the fares are clearly more favorably disposed to males but they got more than what they bargained for at the end. On a separate note, I sometimes feel we take the chivalry of these gentlemen for granted and assume that the same would repeat under any circumstances but this was clearly not the case in a few other parallel tragedies (women and children had lower mortality than males in those incidents). So there was definitely something special about these gentlemen on the Titanic..or maybe it was the crew and the leadership team that made it happen. We can only speculate...
Let us now turn our attention to a popular notion — that people in groups of 2–4 have good survival chances and groups >5 have lower chances.
train_data[‘FamilyCount’]=train_data[‘Parch’]+train_data[‘SibSp’]+1sns.countplot(x=’FamilyCount’, hue=’Survived’, data=train_data)
It appears as if solo travellers are at high risk. From 2–4 groups though the survival rate is high and then 5 onwards the survival rate dips completely. This observation is made by almost all Kaggle posts and indeed some of the very tall leaders have also made this observation in their analysis. Intuitively it may seem true — If you are travelling in a larger group, you would perhaps be made aware faster of the impeneding tragedy because of the sheer size of the group. On the other hand solo travellers, may have been fast asleep in their bunks and may have gotten the news late. As group size increases, perhaps it became more and more difficult to collect everyone & make a dash for safety and so maybe with larger groupsize the trend turns negative and survivals dip.
My initial conjecture was while this could be true, I felt that this could be a case of correlation instead of causation. In other words having larger groups did not ‘cause’ the survival rate to plunge. What I felt was actually happening is that large group sizes were more prevelant in Pclass 3 who anyway had a lower chance of survival based on the class of travel. In other words Pclass=3 was the causation of high mortality and groupsize>4 was just corelated to higher mortality. However I did feel that being solo DID increase mortality rate. Let us quickly see if all this is true.
It is indeed a corelation, not a causation! It is the Pclass which is playing a hidden role here. Let us replot the graph but do so at a common Pclass.
sns.countplot(x=’FamilyCount’,hue=’Survived’,data=train_data[(train_data.Pclass==3) & (train_data.FamilyCount >1)])
Situation looks gloomy now & quite the opposite from earlier graph. So groupsize does not seem to have a very major role to play in survival. Being solo or not does but beyond that the size of group does not seem to matter. It could unnecessarily give some false positives and one must think whether this information is relevant enough to be fed to the final ML model.
Even the ‘solo’ feature is worth a discussion. The general concession is that solo travellers have higher mortality rate. Let us check this.
Solo females buck the trend. But you may say this is not a fair comparision as females anyway have better survival rate. Let us do a final graph comparing mortality for Pclass 3 females between solo travellers & non-solo travellers.
plt.figure(figsize=(16, 6))sns.countplot(x='FamilyCount',hue='Survived',data=train_data[(train_data.Sex=='female') & (train_data.Pclass==3)])
We now see that the mortality rate percentage for solo females in Pclass 3 (or any Pclass for that matter) is much lower that mortality rates percentage for non-solo females. So basically if you were a solo woman traveller, you had much much better chance of survival compared to women with groups or families...and mind you the data is not statistically insignificant. What could be the reason for this extremely peculiar trend?
There were 60 women solo travellers in Pclass 3 compared to 84 non-solo woman travellers and it is here that perhaps we see the 4th and the greatest act of altruism seen aboard the the Titanic. While there can be other reasons for this, in my view this happened because the women here travelling with their husbands and male children(>12 years of age) did not possibly want to leave them even though they had a chance to do so!! Some individual acts of chivalry and altruism have been recognized on the Titanic, but this perhaps is the greatest of them all and it is a pity that this has not been called out so far.
We havent still analyzed the Name and the Ticket columns. When checking the correlation between variables, I found something unbelievable..Those people with lengthy Names had clear high chances of survival! Then it stuck me that maiden names are part of the name for married females. Thus if we sort by the length of the name, we get mostly all the women first and then later the men. Since it is undeniable that Woman across all Pclasses have clearly higher mortality than men, this explained the correlation. A number of other kernels too have made this interesting observation.
Another interesting aspect is that Names combined with Tickets allow us the ability to Group people. They will provide us valuable hints to who all are travelling together...which brings us to the last and most important point. I realized this while plotting the group size versus survival. While analyzing the data I noticed that while there is no connection between large groups and survival, there was a definite connection between survival of members within a group. This — more than Sex, Pclass and Age — determines if a person survives or not for a vast majority of the population (barring Pclass 1 and 2 females who almost always survive). More specifically within a group, if we do a sub-grouping based on Sex and age, we can get to a fine degree of prediction of survival. Grouping can be enhanced by building even maternal side relationships (using the maiden surname of the female passenger).
The Titanic dataset has a lot of missing ages. We end this blog by analyzing various perspectives from basic to advanced on how to do this. The simplest approach is to use the mean or median to fill all the missing ages. The mean age is around 30. However, one would be more accurate in imputing missing ages by calculating mean age of a group of ‘similar passengers’. For e.g., one could choose to calculate mean of Pclass1 passengers and assign it to all Pclass1 missing ages..and likeways for Pclass2 and Pclass3.
But we can further improvise on this. Because it is quite likely that (say)females on Pclass3 have a diff mean age than females on Pclass 1, so an even better approach (the one that most Kaggle leaders have taken) is to group people based on Sex, PClass. If one wants to further refine it, one could make use of the ‘Title’ and the PClass and it should give a slightly better result than Sex and Pclass. The ‘Title’ can be extracted from the name easily.
One can further improvise by building relationships and calculate missing values of ages. For e.g. say you have a husband-wife traveling and husband’s age is missing whereas the wife age is given. You could ‘guess’ the husband’s age as wife age+some number (like 5). Similarly you could guess a missing Sibling’s ages based on the other Sibling or a chid’s age from the parents.
Also, one could even use an ML model to predict the missing value of ages like a few kernels have done. But a pertinent question presents itself at this point. How much does the numerical value of age contribute to the survival? Will it make a lot of difference to the survival? The answer is NO. There are other factors which are far more important. HOWEVER the fact whether the person is a child or an adult or a senior citizen DOES play a critical role in survival. So long as one can categorize the missing ages as belonging to one of this group it is fine! In particular, we should be spending a lot more time worrying about whether the person in question (with the missing age) was a child or not? This makes a lot of difference to the survival chance. How do we do that? Well it is easy for male children because of the Title ‘Master’ which is prefixed to their names. But how do we identify a female child among the missing ages. There is no Title specific to female children. All unmarried females across all ages had the ‘Miss’ Title.
So here is a simple clue to identify such folks and impute their missing ages. I haven’t seen it being used in any kernel so far (at least the ones I have gone thru’ though). We can identify the such cases by checking the Parch flag. If Parch flag is >0 then they are most likely female children. And so it would be good to add this important heuristic after we are done with all the other steps...
train_data['Title'] = train_data.Name.str.extract(' ([A-Za-z]+)\.', expand=False) print ("Avg age of 'Miss' Title", round(train_data[train_data.Title=="Miss"]['Age'].mean()))print ("Avg age of 'Miss' Title travelling without Parents", round(train_data[(train_data.Title=="Miss") & (train_data.Parch==0)]['Age'].mean()))print ("Avg age of 'Miss' Title travelling with Parents", round(train_data[(train_data.Title=="Miss") & (train_data.Parch!=0)]['Age'].mean()), '\n')Avg age of ‘Miss’ Title 22Avg age of ‘Miss’ Title travelling without Parents 28Avg age of ‘Miss’ Title travelling with Parents 12 !!
See the HUGE difference! If we had used the average value without considering the Parch, we would have gone horribly wrong. Even here there is a huge gap between Pclasses and we need to take that into consideration before imputing the final values.
The Titanic dataset continue to surprise and inspire even a decade after it was made available. Top scores on the Titanic follow a pattern of waves. Every once in a few years, there is a renewed interest and the next generation of data scientists push the top score ever so slightly. It has been a couple of years since Chris Deotte — the current number 1 on Kaggle in terms of discussions and number 2 in terms of Notebooks — upped the ante with a score of 84% (Let us ignore the 100% scores and even the kernels with pretuned hyperparameters). I do hope the new budding data scientists reading this blog are motivated to take a re-look at this extremely captivating dataset and push the scores even higher. Knowing Chris, he would be the first one to clap if this were to happen!
All the best!
Do check my notebook titled — Captivating Conversations with the Titanic dataset in Python. https://www.kaggle.com/allohvk/captivating-conversations-with-the-titanic-dataset which has a lot of documentation and is extremely easy to follow even if one is a beginner to both ML and Python.