Welcome back in this previous section we downloaded the Boston Housing data set performed some exploratory data analysis and just to get ourselves familiar with the features that has been provided. Just as a reminder the features that's been provided will start from the bottom here. The last column being the medium house price you have the lower stages of the population the population of blacks. The pupil teacher ratio the property text rate per 10,000 the also the accessibility or how close it is from the highway the distance from the employment center the age of the buildings the number of rooms per dwelling the nitric oxides concentration. Childs river dummy variable whether it's actually tracked along it or not. The proportion of non-retail business acres per town the proportion of residential land zone for lots of 25,000 and finally the per capita crime rate by town. So it's necessary for us to be familiar with them so the next thing that we are going to do is just get another sense another scan through a bit like X-ray. Do you understand the correlation between the actual individual features meaning each column how do they relate to each other and more importantly how does it actually relate to the so-called median price that we are trying to predict here. The beauty of it is that we in pandas it has a function or method call call and basically that is correlation analysis all you have to do is take the pandas data frame dot call and run it and you will get a correlation matrix like this. This is probably one of the key features that you can use to get a really quick appreciation and understanding of how the underlying data or the features interact with each other. Alright I made some modification because it was rather difficult to read before with four decimals and just look too cluttered. So I've just set the using the pandas option set the flow format to just three decimals makes it a bit easier to read now. Now because we are trying to model a basic simple linear regression with only one variable on the medium value we would like to look at something that actually is high. So if you scan through this whole column here and basically this is relating to each and every one of these. Obviously to itself is really high. Also has a high correlation with L-stat. The other one is RM and this is one way that you can select your features using RM and using L-stat which is actually directly opposite. So L-stat is basically the percentage of lower status of the population. So the greater the proportion, the more it has a direct correlation they are inversely correlated. The other way to do this is to just plot a figure, a heat map using the C-bond. So if we plot this out this is what it looks like. What I've done is I've set the figure a little bit larger so that we can actually visualize this easily and the other one is just annotated so that we can actually see the value here. So here you can see the RM is fairly high. Of course to itself is 100% and it's really blue here for the lower stats. So looking through this if you're gonna do multiple linear regression you probably will choose this. Anything that is larger than 0.5 is probably worth while looking at it. At least that's the first cut as you add. And there are quite a few that's negative correlated. So you have the text distance from the road. This is the PT ratio which is the parent teacher ratio and also the lower stats. And finally this is the industry percentage that is allocated to non-retail. And also not surprisingly it's inversely correlated to crime as well. So there's quite a few things that explain why house prices go down. There's only really one that explain why house prices goes up which is the number of room. Okay so that's really the end of what I want to show you here in terms of the correlation analysis and feature selection. I like to give you an opportunity to try this out because it's necessary for you to play around with this just to actually get yourself familiar with it. The first thing is that you might want to try the float to see what it looks like in terms of the presentation. This is really crucial especially when you're presenting to management. If it's like this is they're gonna find it really difficult to understand and most people have a four-be-hour numbers. So if you present it with this this way again on a table or 14 by 14 it's just too much for people to take in and one go. However if you actually change the so-called to a heat map and present it this way it's much more palatable to people. There are a couple of features that comes with the sepon heat map. I just want to show you heat map and the one is one of the beauty of the Jupyter notebook is if you just press question it brings up all of the dock string or the document tation. There are few things that you can actually modify here. You can actually change the color setting to make it well choose the so-called the color type that you like. Obviously chosen to annotate. There are a few other things to actually beautify the whole charts that is suitable for your use. So I highly encourage you to actually play around with it. So I'm gonna pause the video, let you try typing the actual code in first of all and play around with it and better still. See if you can actually select let's just say these four column. Cream, ZN, Indus and cars rather than plotting such a big heat map just plot these first four against MedV. MedV. Okay and see what does the actual heat map look like. So that will be your challenge. I'm gonna encourage you to pause the video. I'll come back and wrap this up. Welcome back. Did you manage that? How did that work out? I really hope that it went well and obviously the probably will encounter some problems. What I want to illustrate here is that when it comes to the pender selection in terms of the columns you do need to put a double quote. So I've selected cream ZN, Indus, CHAS as well as MedV. So let's just plot this heat map out and this is what it looks like. So if we narrow down to just these four variables along with the medium value this is what it looks like. So you have cream and again this is actually negatively inversely correlated. The number of non-retail also inversely correlated which is not surprising because most people prefer to not live nearby a factory. Okay especially a non-retail factory. It could be steel plan that gives out fuel man and a noise right throughout the night or manufacturing plant that runs throughout the day in 24 hours. So people usually don't like that. So that's understandable. So that's really the end of this video. I hope you have found that really useful. In the next video we will move into the actual linear regression itself using the variable that we've selected and we're going to use psychic learn to perform our modeling and also just to see what the actual looks like. Okay what the chat looks like and also do some visualization as well. So I look forward to seeing you in the next video.