How ‘Big’ should be your Data?. An experimental study on how size of... | by Monik Raj | Towards Data Science

A simple question, that comes to all data scientists’ mind while designing a machine learning model is, “How big should be the data set? How many features should be used for modelling?”. This article will walk you through an experimental study of how you can make decisions on data set size and number of features required for model.
Below is the link to my github repository. It contains the python codes used for the analysis. You can fork or clone the repository and get your hands dirty with code :-P
github.com
Below is the link to the data repository from UCI Machine Learning Repository. It contains the information related to data set and attributes. Here it mentions that, the problem is a regression problem.
https://archive.ics.uci.edu/ml/datasets/Physicochemical+Properties+of+Protein+Tertiary+Structure
Description of DatasetDataset inititally has 10 columns. Column with name 'RMSD' is the target column of our data setRest 9 columns with name F1, F2, ... , F9 are the features columns of the data set. m = Number of observations in the data setDimension of Y : m X 1Dimension of X : m X 9
Now, I will show you the correlation matrix between every column of the data frame, including RMSD (Data frame is a terminology to denote the data set read by Pandas module of Python).
From the above figure, we can conclude that RMSD is pretty much independent with other columns, since theory says if, correlation(x,y)~0, then ‘x’ and ‘y’ are independent of each other (where ‘x’ and ‘y’ are vectors of same dimension).
In cases, where we don’t know which column is to be selected as the target column, we can find the correlation matrix and find the column which is mostly independent with all the other columns.
Here is the Correlation matrix plot between all the feature columns in X.
For column ‘i’ and column ‘j’ of X, it shows the scatter plot between the two variabls, when ‘i’ != ‘j’. If ‘i’==’j’, then it shows the frequency distribution of the variable.
The above figure tells about the relationship between the variables with each other. From the same, we can graphically infer about the correlation between matrix for better analysis.
Frequency distribution of the variables shows about the kind of distribution it follows, and if follows a known and standard distribution like ‘Gaussian’ distribution or ‘Rayleigh’ distribution, we can predict and analyze the behaviour of the variable, which can be handy in further investigation of the data.
Note: Frequency distribution of the variable or Histogram are generally used for fitting a variable to a standard distribution.
Now I would like to draw your attention towards feature selection. Here I will throw light on method to decide which features to keep. I am going to use SelectKBest module of Sci-kit learn module from Python to do so. They use F-test estimates to find the best features from the data set. More information on the module and theory can be found in the reference.
from sklearn.feature_selection import SelectKBest, f_regression#Assuming X and Y are numpy array of dimension (m,9) and (m,1) #respectively#We use f_regression function since our's is a regression problemX_best = SelectKBest(f_regression, k=5).fit_transform(X,Y)'''After the code snippet is run, X_best keeps top k=5 features from X.'''
I will now show you the effect of best ‘k’ features on ‘R2-score’ metric of the model using linear regression. Here you can see that by increasing the value of ‘k’, we get better R2-score for our linear regression model. I have also stored the descending order of features with respect to ‘SelectKBest’ method. The code for same can be found in my github link above.
With this, we can select a suitable value of ‘k’ for a corresponding value of R2-score of the model, which is acceptable for our problem statement (I may go with k=5, but for you, it may differ as per the demand of the problem). Here is the list of ‘k’ features in descending order : [F3, F4, F2, F9, F6, F1, F5, F7, F8]
One of the major application of Feature Selection is to reduce dimensionality of the data, for better computation performance.
Principal Component Analysis is another method to decompose the dimensions (features) into smaller dimension spcace. Major benefits of using Principal Component Analysis are:
Dimensionality reduction
Visualization of higher dimension space data by converting it to lower dimension space data
Below are the visualizations of the data after decomposing higher dimension space into lower dimension space (for ‘X’).
I will now show you the effect of projecting data onto ‘p’ components feature space, on ‘R2-score’ metric of the model using linear regression. Here you can see that by increasing the value of ‘p’, we get better R2-score for our linear regression model.
Looking at the graph, we can select a suitable value of ‘p’ for our solution. The value of ‘p’ should be selected by corresponding acceptable value of R2-score, as per the requirement of the problem.
Now I will discuss the effect of the size of data set, ‘m’ = Number of observations in data set. I have kept the 9 features, as originally present in the data set, I have varied the ‘m’, trained on train set(80%) and calculated the R2-score for each ‘m’ on test set(20%) using linear regression model.
Analyzing the above graph, we can infer the following things:
Inital spike in R2-score is because of the bias in data because of the lower ‘m’.
Following the spike, the R2-score saturates to some extent (ignoring the intermediate spikes, which is because of the random selection of data points while splitting into ‘m’ %), which says gradually increasing the data set size, eventually increases the R2-score, till saturation.
After a certain point, it saturates and increasing the ‘m’, doesn’t affect the R2-score.
Lower ‘m’ results in either bias in model or lower R2-score.
Designing a machine learning model includes many factors like selection of features, deciding on size of data for better performance of model, computation costs, feasibility in achieving real time data (for some cases), and many more as per the requirement of the problem. It’s always a exploratory analysis of the data, which leads to a better model design, because
Every problem is different in Machine learning
Reference:
Sci-kit Module : http://scikit-learn.orgSelectKBest Module : http://scikit-learn.org/stable/modules/feature_selection.htmlPandas Module : https://pandas.pydata.org/pandas-docs/stable/Seaborn Module for visualization : https://seaborn.pydata.org/Principal Component Analysis : https://en.wikipedia.org/wiki/Principal_component_analysis
Sci-kit Module : http://scikit-learn.org
SelectKBest Module : http://scikit-learn.org/stable/modules/feature_selection.html
Pandas Module : https://pandas.pydata.org/pandas-docs/stable/
Seaborn Module for visualization : https://seaborn.pydata.org/
Principal Component Analysis : https://en.wikipedia.org/wiki/Principal_component_analysis