Dimensions and degrees of freedom
By: Assaad Moawad
Tags: Machine Learning, Dimensionality Reduction, Degrees Of Freedom, Data Analytics, Data Science

Dimensions and degrees of freedom

Number of features vs number of dimensions

The dimension of a mathematical object is the number of independent variables needed to fully describe it. A point has 0 dimensions. A line has 1 dimension, a square has 2 dimensions and a cube has 3 dimensions. On a line we need one variable, let’s say the distance from a starting point in order to pinpoint our position. On a square we need at least 2 pieces of information (x and y). In a cube, we need 3 coordinates (x,y,z)

The dimension of an object is an intrinsic property of the object independent of the space in which it is embedded.

Basically a line has intrinsically 1 dimension, even if we put it in 2-D, 3-D, or 10-D spaces. For example, if we have points on a straight line (y=2x) in a 2-D plane, we still need one information, the distance from the origin for example, to uniquely identify any point on this line. Even if these points have 2 Cartesian coordinates, the coordinates x and y are dependent by the equation of the line (y=2x), thus they are redundant since any of them can be deduced given the other one (x=y/2, or y=2x).

A big misconception in machine learning and data analytics is the confusion between 2 totally different concepts: number of features and number of dimensions. The best way to illustrate the difference is to imagine 3 pipes of different sizes, all controlled by the same valve. Each having a different fluid flowing in it with different properties and a meter counting the volume. Thus the number of features we have in our dataset is 3 (Volume of fluid 1, Volume of fluid 2, Volume of fluid 3).

3 pipes controlled by the same valve. Image rendering by Alexy Frangieh

+---------+---------+---------+

| Meter 1 | Meter 2 | Meter 3 |

+---------+---------+---------+

| 1 | 10 | 100 |

+---------+---------+---------+

| 2 | 20 | 200 |

+---------+---------+---------+

| 3 | 30 | 300 |

+---------+---------+---------+

| 4 | 40 | 400 |

+---------+---------+---------+

| 5 | 50 | 500 |

+---------+---------+---------+

| 2.5 | 25 | 250 |

+---------+---------+---------+

| 0 | 0 | 0 |

+---------+---------+---------+

A sample of the data collected

Although we have 3 features in this dataset, we can easily see that there is a proportionality or a correlation between them (namely 1x: 10x: 100x). This correlation exists because we have only one control point. Mainly one valve controlling all the 3 flows simultaneously, at the same rate. So even seemingly we have 3 different variables, 3 meters and 3 features, what really matters for machine learning and data analytics is the degrees of freedom in the dataset. How many different pipes can we control independently of each others. Can we have the first pipe totally opened while the second totally closed? In this system we can’t, by design. So even if we collect the data of the 3 flows, it doesn’t matter, it’s a redundant information. One flow information would be enough. The rest can be deduced by proportionality/correlation.

Only when we use an independent valve on each pipe, we can say that the number of dimension is equal to the number of features. But then, we need to try all the different combinations between all the states of all the valves on all the pipes, before we can say that we “covered” the whole exploration space.

What matters is how many independent knobs we have.

If we consider only 2 states per valve: totally open and totally closed. with each additional valve added to the system, we have 2 times more combinations to explore. For instance, with 3 valve we get the following 8 combinations:

[closed, closed, closed]

[closed,closed,opened]

[closed, opened, closed]

[closed, opened, opened]

[opened, closed, closed]

[opened,closed,opened]

[opened, opened, closed]

[opened, opened, opened]

Each additional pipe, with an additional valve (Or knob) will add an additional dimension to the space, and double the data needed. We would need exponentially more data to cover all the possible combinations. That’s a well-known problem in data analytics called “The curse of dimensionality”.

Too many variables!

I was going to write about the mathematical tools that allow to discover the number of independent dimensions from the different features (like Principal Component Analysis PCA). But I found this excellent blog post about the topic.