Myth of Regression
By: Swagatika Ai
Tags: Regression, ML

I thought why not share what I understood about “Regression”. Why I added the myth here as there is a long history behind this Regression. If you are in the field of Data Science or Analytics then you must be aware of the keyword “regression”. Regression is used for investigating the relationships between variables.

So, what wiki says is:-

In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the ‘outcome variable’) and one or more independent variables (often called ‘predictors’, ‘covariates’, or ‘features’).

The simplest form of regression is “linear regression”. Why the term “linear” because it gives you a linear relationship. But I am not here to write about linear regression. I am here to write about multiple linear regression vs. multi-variate linear regression. When I first read about that multiple and multivariate, I was confused. What is this multiple linear regression? How it differs from multivariate regression? Is it actually two different concepts or was it due to two person’s rivalry to stick to their own concepts? Let’s go back to the history of Regression.

Let’s talk about who is the father of “Regression”.

Sir Francis Galton is the father of regression. Galton, a cousin of Charles Darwin and an accomplished 19th century scientist in his own right. The term regression was first applied to statistics by the polymath Francis Galton. Galton is a major figure in the development of statistics and genetics.

Refer this wiki link for further details about Francis Galton.

Karl Pearson did the association between the least squares method of prediction with Regression by late 1800s. He was the colleague of Francis Galton and member of the founders of mathematical statistics. He noticed that if you plotted the height of parents on the x-axis and their children on the y-axis, the line that best fit the data according to least squares had a slope of less than one. A slope of less than one is essentially the mathematical representation of “regression to the mean.” Pearson referred to this slope on a graph as the “regression line.” And thus, the method of least squares and regression became somewhat interchangeable. The mathematicians Carl Friedrich Gauss got the credit for the invention of “the method of least squares”.

How multiple regression comes in to the picture????

Galton started the regression analysis using sweet pea. Galton studied everything from fingerprint classification to meteorology, but he gained recognition primarily for his work on inheritance. His most important insight came to him while he was studying the inheritance of one of the most obvious of all human characteristics: height. In order to understand how the characteristic of height was passed from one generation to the next, Galton collected data on the heights of individuals and the heights of their parents, grandparents etc. He somewhat realised from the collected data that the generations prior to the immediate parents could also influence individual characteristics. He even noticed that certain characteristics occasionally skipped one or more generations; a man may appear more similar to his grandfather than to his father in certain respects.

With this realisation of this diminishing effect, Galton had come up with the idea of multiple regression. A characteristic or variable may be influenced not simply by a single important cause but by a multitude of causes of greater and lesser importance. In later publications Galton listed some mathematical formulae that captured this same basic idea, but he was never able to develop a complete mathematical treatment of the matter.

“The somewhat complicated mathematics of multiple correlation, with its repeated appeals to the geometrical notions of hyperspace, remained a closed chamber to him. (Pearson 1930)”

Galton’s ground work gives the foundation to Pearson later developed a rigorous mathematical treatment. Pearson’s subsequent work included further development of multiple regression as well as innovative progress on other statistics such as chi-square (Pearson 1938). Several years later Student gave the correct distribution of the sample variance and his now famous T-statistic. Student was the pen-name of William S Gossett.

Now we got to know about the history of multiple regression. Let’s look for multivariate regression. Who developed this and what is the concepts?

In simplest form multivariate regression deals with multiple outcome values (i.e. multiple dependent/predicted values)

The foundations of multivariate statistical analysis were laid by pioneering work of the British statistician, geneticist, eugenicist, and professor Sir Ronald Aylmer Fisher in short R.A. Fisher (1890–1962). Fisher himself wrote 395 statistical papers and 8 books. He entered Gonville and Caius College, Cambridge in 1909, and passed the Mathematical Tripos Part II in 1912 as a Wrangler, his interest area was mathematical physics. Shortly before world war in 1914 Fisher first came in contact with Karl Pearson, during this time he read Karl Pearson’s “Mathematical Contribution to the theory of evolution”.

Now let’s dig into the history to find out how R.A. Fisher established his analysis on multivariate regression.

Fisher worked at Rothamsted on sampling distributions, and in 1921 completed the classic memoir on Mathematical Foundations of Theoretical Statistics which was intended to supply the framework of current statistical theory. In Rothamsted he analyzed its huge data from crop experiments since the 1840s, and developed the analysis of variance (ANOVA).

The pioneering work of “Student” (W. S. Cosset) on small-sample distributions led to R. A. Fisher’s distributions of simple and multiple correlation coefficients.

Fisher introduced the term variance and proposed its formal analysis in a 1918 article,” The Correlation Between Relatives on the Supposition of Mendelian Inheritance”.

His first application of the analysis of variance was published in 1921. Analysis of variance became widely known after being included in Fisher’s 1925 book “Statistical Methods for Research Workers” .

As per wiki:-

“The ANOVA is based on the law of total variance, where the observed variance in a particular variable is partitioned into components attributable to different sources of variation. In its simplest form, ANOVA provides a statistical test of whether two or more population means are equal, and therefore generalises the t-test beyond two means.”

Three conceptual models of ANOVA are

1)Fixed-effects model — An independent variable is a fixed effect if the levels are deliberately chosen, and are the only levels of interest. In a replication of the study, the same levels would be used.,

2) Random-effects model — An independent variable is a random effect if the levels are randomly selected (the subjects in an experiment, for example). In a replication, a new set of randomly selected levels would be used.

3) Mixed effects — Depending most on the number of treatments and levels of the experiment,which contain both random and fixed effects.

Multivariate analysis of variance (MANOVA) is an extension of common analysis of variance (ANOVA). In ANOVA, differences among various group means on a single-response variable are studied. In MANOVA, the number of response(dependent) variables is increased to two or more. The hypothesis concerns a comparison of vectors of group means. When only two groups are being compared, the results are identical to Hotelling’s T² procedure.

The advantage of a MANOVA over many univariate ANOVA tests is that a MANOVA is able to account for correlation between dependent variables. Instead of using an F-statistic, MANOVA uses an analogous value known as Wilk’s lambda which calculates the amount that factors other than the measured independent variable would have affected the experimental sample.

Most of the concepts and methods introduced by R.A Fisher are fundamental and continue to provide the framework for the discussion of statistical theory.

In the 1920s R. A. Fisher created modern regression analysis out of two nineteenth century theories: the theory of errors by Gauss and the theory of correlation by Pearson.

The innovation was based on Fisher’s realisation that the distribution associated with the regression coefficient was unaffected by the distribution of X. Subsequently Fisher interpreted the fixed X assumption in terms of his notion of ancillarity.

As per Fisher’s “the independent and dependent variables bear to the regression line. If errors occur in the former, the regression line will be altered; if they occur in the latter, the regression line will not be altered, provided the errors “balance in the averages”; so, the errors in variables case was not covered. Second, “the regression function does not depend on the frequency distribution of the independent variable, so that a true regression line may be obtained even when the age groups are arbitrarily selected. . ..” On the other hand, a selection of the dependent variate will “change the regression line altogether.”

He also mentioned that the partial correlation depends on the “assumption that the variates correlated are normally distributed”. He was investigating the existence of dependence between variables by testing hypotheses about correlation coefficient rather than regression coefficient.

Regression analysis as we know it today is primarily the work of R.A. Fisher, one of the most renowned statisticians of the 20th Century. Fisher combined the work of Gauss and Pearson to develop a fully realised theory of the properties of least squares estimation. Due to Fisher’s work, regression analysis is not just used for prediction and understanding correlations, but also gives the inference about the relationship between a factor and an outcome.

I think I have just given you a glance of history about this evolution of regression. If we go back to the history, we can learn a lot.

So, in a nutshell we can explain the multiple/multivariate regression from a layman point of view like let’s say independent variable X1, ⋯ , Xm for dependent variable Y1,⋯,Yn

When m > 1, it is usually called “multiple regression” or “multi-variable regression”.

When n > 1, it is usually called “multi-variate regression”.

Of course, you can have “multiple/multi-variable multi-variate regression,” i.e., m > 1, n > 1.

Hope you enjoy reading this blog.

References:-