2 Hours of ML a Day — Day 4: 5:30 p.m. to 7:30 p.m.
By: Pratibha Ur
Tags: Outliers, Sweetviz, Autogluon, Pandas Profiling, Exploratory Data Analysis

Background: 2 Hours of ML a day

Intro: I stumbled upon an ongoing a Kaggle competition — Tabular Playground Series — Jul 2021, which does not have images or audio related data (those are too advanced for me). My plan is to understand the goal or the end result. Following this, my first step is to perform EDA (exploratory data analysis), which will help me understand the data better. During the working of the project, I will refer to — already submitted codes for this competition.

Yesterday’s link

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Starting a bit early today, to get into a better time to do this. During the night, I just struggle a lot more when I get stuck. Hope this time is a better one!

Outlier Detection (contd)

I had talked about a function yesterday, that could be used to identify an outlier. The output of that function is a list of outliers in the form of a list. I googled a lot to identify how to filter a DataFrame based on a list, and its just not working! So instead, I am filtering every variable based on the upper and lower bound identified in the function.

For example: for one of the target variables —

outliers = []

outliers1 = pd.DataFrame()

sorted(df[‘target_carbon_monoxide’])

quantile1, quantile3= np.percentile(df[‘target_carbon_monoxide’],[25,75])

iqr = quantile3-quantile1

lower_bound_val = quantile1 -(1.5 * iqr)

upper_bound_val = quantile3 +(1.5 * iqr) print(“Upper Bound = “, upper_bound_val)

print(“Lower Bound = “, lower_bound_val) df = df[(df[‘target_carbon_monoxide’] >= lower_bound_val) & (df[‘target_carbon_monoxide’] <= upper_bound_val)]

df

For one of the sensor variables

outliers = []

outliers1 = pd.DataFrame()

sorted(df[‘sensor_1’])

quantile1, quantile3= np.percentile(df[‘sensor_1’],[25,75])

iqr = quantile3-quantile1

lower_bound_val = quantile1 -(1.5 * iqr)

upper_bound_val = quantile3 +(1.5 * iqr) print(“Upper Bound = “, upper_bound_val)

print(“Lower Bound = “, lower_bound_val) df = df[(df[‘sensor_1’] >= lower_bound_val) & (df[‘sensor_1’] <= upper_bound_val)]

df

Yes, the initial data stack had 7111 records, but after removing the outliers, it stands at 5850 records — around 17% of the data was removed as outliers.

Outliers are a bit tricky. I dont know if this was an essential step or not, but will keep it for now. If my model results are not significant, I will revisit this part.

For further EDA, I want to try to use a library. I have been browsing through a few recently, and was interested in trying one at least that I might like using.

EDA Libraries exploration

I am following this link, to explore the EDA libraries for now.

a. pandas-profiling

Installing the library

pip install pandas-profiling

Then two lines of code. Here I will be using the dataframe for which the outliers are not removed, to see the results from the library.

from pandas_profiling import ProfileReport

profile = ProfileReport(df, title=”Pandas Profiling Report”)

Exploring the results —

Overview is pretty straightforward, with just the details of the dataset, with additional features like pointing to missing values or duplicate rows.

Next is Variables. Very detailed overview — some points that caught my attention were Min/Max, percentiles & skewness. Since the skewness is between -0.5 and 0.5, the data are fairly symmetrical. Not sure what three times HIGH CORRELATION in red means though — highly correlated with three variables? may be…

Similarly, I will explore other variables as well. I found that sensor_1, sensor_2, sensor_3, sensor_5 and all the three target variables are highly skewed, based on similar summary as shown for deg_C variable

Moving on to Interactions, I personally did not like the representation of this. I would have preferred a scatter plot for easier interpretation. The diagram below shows the interaction between sensor_1 and target_carbon_monoxide.

Correlation is a standard heatmap. I would prefer to list down the highly correlated independent variables at this point —

deg_C is highly correlated to relative_humidity, moderately correlated with absolute_humidity, relative_humidity is moderately correlated to the work_hours. absolute_humidity is highly correlated to sensor_3 and sensor_4 all the 5 sensors are highly correlated to one another!

b. sweet-viz

pip install sweetviz import sweetviz as sv my_report = sv.analyze(df_orig)

my_report.show_html() # Default arguments will generate to “SWEETVIZ_REPORT.html”

The results are quite similar to pandas-profile, however it looks very different

c. autoviz

May be, I am bored at this point, so did not want to resolve this error that I got using autoviz

pip install autoviz #importing Autoviz class

from autoviz.AutoViz_Class import AutoViz_Class#Instantiate the AutoViz class

AV = AutoViz_Class()

error:

d. D-Tale

Unfortunately, I could not make it work as well. Urghh… But I want to move on….

blank output

Autogluon exploration

With half an hour more on my plate. I was looking to explore the autogluon which is an Automated ML tool, that evaluates different models. I have just read about it, but let’s use and see the initial results. Will give me something to think about for the next few hours.

https://pypi.org/project/autogluon/

pip install -U setuptools wheel

pip install -U “mxnet<2.0.0”

pip install autogluon # autogluon==0.2.0

I will reconvene tomorrow! installing autogluon took sometime, as it required restarting the kernel.

Not a bad day!