Introduction to Statistical Methods in AI | by Atul Agarwal | Towards Data Science

Statistical Learning is a set of tools for understanding data. These tools broadly come under two classes: supervised learning & unsupervised learning. Generally, supervised learning refers to predicting or estimating an output based on one or more inputs. Unsupervised learning, on the other hand, provides a relationship or finds a pattern within the given data without a supervised output.
Let, suppose that we observe a response Y and p different predictors X = (X1, X2,...., Xp). In general, we can say:
Y =f(X) + ε
Here f is an unknown function, and ε is the random error term.
In essence, statistical learning refers to a set of approaches for estimating f.
In cases where we have set of X readily available, but the output Y, not so much, the error averages to zero, and we can say:
¥ = ƒ(X)
where ƒ represents our estimate of f and ¥ represents the resulting prediction.
Hence for a set of predictors X, we can say:
E(Y — ¥)2 = E[f(X) + ε — ƒ(X)]2=> E(Y — ¥)2 = [f(X) - ƒ(X)]2 + Var(ε)
where,
E(Y — ¥)2 represents the expected value of the squared difference between actual and expected result.
[f(X) — ƒ(X)]2 represents the reducible error. It is reducible because we can potentially improve the accuracy of ƒ by better modeling.
Var(ε) represents the irreducible error. It is irreducible because no matter how well we estimate ƒ, we cannot reduce the error introduced by variance in ε.
Variables, Y, can be broadly be characterised as quantitative or qualitative( also known as categorical). Quantitative variables take on numerical values, e.g., age, height, income, price, and much more. Estimating qualitative responses is often termed as a regression problem. Qualitative variables take on categorical values, e.g., gender, brand, parts of speech, and much more. Estimating qualitative responses is often termed as a classification problem.
There is no free lunch in statistics: no one method dominates all other over all possible data sets.
Variance refers to the amount by which ƒ would change if we estimated with different training data sets. In general, when we over-fit a model on a given training data set(reducible error in training set is very low but on test set is very high), we get a model that has higher variance since any change in the data points would results in a significantly different model.
Bias refers to the error introduced by approximating a real-life problem, which may be extremely complicated by a much simpler model — for example, modeling non-linear problems with a linear model. In general, when we over-fit, a model on given data set it results in very less bias.
This results in the variance bias trade-off.
As we fit the model over a given data set, the bias tends to decrease faster than the variance increases initially. Consequently, the expected test error(reducible) declines. However, at some point, when over-fitting starts, there is a little impact on the bias, but variance starts to increase rapidly when this happens the test error increases.
Linear regression is a statistical method belonging to supervised learning used for predicting quantitative responses.
Simple Linear Regression approach predicts a quantitative response ¥ based on a single variable X assuming a linear relationship. We can say :
¥ ≈ β0 + β1X
Our job is now to estimate β0 and β1, the parameters/coefficients of our model based on the training data set, such that the hyperplane(in this case a line) is close to the training data set. Many criteria can estimate the closeness, the most common being least square.
The sum of the square of the difference between all observed response and the predicted response formulates to Residual Sum Of Squares(RSS).
Problems in Linear Regression
Non-linearity of the response-predictor relationships.
Correlation of error terms.
The non-constant variance of error terms.
Outliers: when the actual prediction is very far from the estimated one, can arise due to inaccurate recording of data.
High-leverage points: Unusual values of the predictors impact the regression line known as high leverage points.
Collinearity: where two or more predictor variables are closely related to each other, it may be challenging to weed out the individual effect of a single predictor variable.
KNN Regression
KNN Regression is a non-parametric approach towards estimating or predicting values, which do not assume the form of ƒ(X). It estimates/predicts ƒ(x0) where x0 is a prediction point by averaging out all N0 responses closest to x0. We can say:
Note: If K is small, the fit would be flexible and any change in the data would result in a different fit, hence for small K the variance would be high and bias low; conversely, if K is large, it might mask some structure in the data hence the bias would be high.
The responses as we discussed till now, may not always be quantitative, it can be also qualitative, predicting these qualitative responses is called classification.
We will discuss various statistical approaches to classification including:
SVM
Logistic Regression
KNN Classifier
GAM
Trees
Random Forest
Boosting
SVM or support vector machine is the classifier that maximizes the margin. The goal of a classifier in our example below is to find a line or (n-1) dimension hyper-plane that separates the two classes present in the n-dimensional space. I have written a detailed article explaining the derivation and formulation of SVM. In my opinion, it is one of the most powerful techniques in our tool box of statistical methods in AI.
Logistic model models the probability of output response ¥ belonging to a particular category.
We can say:
Applying componendo dividendo we get:
which is nothing but the odds.
For estimating the beta coefficients, we can use maximum likelihood. The basic idea is to estimate the betas such that the estimated value and observed value of the results are as close as possible. In a binary classification, with observed classes as 1 and 0, we can say the likelihood function would look like:
KNN(K nearest neighbors) Classifier is a lazy learning technique, where the training data set is represented on a Euclidean hyperplane, and test data is assigned the labels based on the K Euclidean distance metrics.
Practical Aspects
K should be chosen empirically and preferably odd to avoid tie situation.
KNN should have both discrete and continuous target functions.
Weighted contribution(e.g. distance based) from different neighbors can be used computing the final label.
Note: Performance of KNN degrades when the data is high dimensional. This can be avoided by providing weights to the features itself.
Effect Of K on the decision boundary
Advantages Of KNN
We can learn a complex target function.
Zero loss of any information.
Disadvantages of KNN
Classification cost of new instances is very high.
Significant computation takes place at classification time.
GAM provides a generalized framework extending standard multivariable linear regression with the nonlinear function of each variable while maintaining its additive nature. Thus, all nonlinear functions can be independently calculated and added later.
Note: GAM like linear regression can be applied to both quantitative and qualitative responses.
Trees or decision trees are useful and straightforward methods for both regression and classification involving segmenting the predictor space into simple regions.
Typically decision trees are drawn upside down meaning the leaves are at the bottom of the tree. The points where the predictor space is split are known as internal nodes, and the leaf nodes or terminal nodes are the ones which given the predictions. Segments joining the nodes are known as branches.
For prediction, we take a top-down(at the first point all the observation belongs to just one region), greedy(best split is made in the particular step) approach known as recursive binary fitting.
There are strategies like tree pruning that solves the over-fitting problem of trees by cutting some of the branches to get a small sub-tree.
For a classification problem, we either use Gini index,
or entropy
to represent the purity of a node, where Pmk is the proportion of samples in the mth region from kth class.
Decision trees still suffer from high variance and are not competitive with other supervised approaches. Therefore, we introduce random forest boosting and bagging.
Bagging
Bagging is a general-purpose method to reduce variance in a statistical learning method. The core idea is that averaging a set of observations reduces variance. Hence we do a random sampling of our data multiple times, and for each sample, we construct a tree and average out all the predictions to give a low variance result.
Random Forest
When in the collection of bagged trees, a fix k predictors are chosen at random from each tree having total m predictors (k < m), then bagging becomes a random forest.
This is done because most of the bagged trees would look more or less the same. Hence, the predictions of individual bag trees would be highly co-related. Therefore, there would not be much reduction in the variance of our inferences. Random forests can be thought of as the process of de-correlating bagged trees.
Boosting
Boosting approach is a slow learning statistical method, where classifiers are learned on modified data set sequentially. In the context of decision trees, each tree is grown using information from the previous trees. This way, we do not fit a single large tree.
All the above methods had some form of annotated data set. But when we want to learn patterns in our data without any annotations unsupervised learning comes into the picture.
The most widely used statistical method for unsupervised learning is K-Means Clustering. We take k random points in our data set and map all other points to one of the K regions based on their closeness to K chosen random points. Then we change the K random points to the centroid of the clusters thus formed. We do that until we observe a negligible change in the cluster formed after each iteration.
There are other techniques like PCA in unsupervised learning that are used a lot, but for now, we end here.
Next: Introduction to Artificial Neural Nets