by godelski | 2025-08-30 20:05:32+00:00
I highly encourage you to talk to a mathematician about this stuff. In high dimensions everything is all about caveats and edge cases. Your {1,2,3}D intuition leads you astray rather than helps you closer to the answer. In fact, math is specifically an overly pedantic because nuances are so critical.

I'll use an example that we can reason from low dimensions but that many people make the wrong connections. What is the probability that in N dimensions two i.i.d randomly drawn vectors are orthogonal? We can easily intuit in low dimensions that this is a pretty unlikely event. But in high dimensions exactly the opposite is true. An easy mistake to make is to not think about this question from a continuous perspective, by reframing the question to ask what the probability that the angle between to i.i.d drawn vectors are within some epsilon bound. If we don't, then the answer is trivial in all cases: 0. A good student would be able to reason through this from looking at the 2D case and 3D case but this is only a sample of 2 and it would be naive to generalize such a notion without doing the proof.

So let's look at something a little harder. A classic problem is looking at the volume of a n-ball. Our 2D and 3D understanding make this problem look easy. The volume of the 3D ball is much larger than the 2D ball. We can even verify this my analytically solving the 4D case. Same with the 5D case! But that's when things go awry. For the unit n-ball, 5D is where the maximum volume is. By 10 dimensions you have less volume than our 2D case and by 20 dimensions your volume is quickly converging to 0.

Now let's look at something wild. Consider a hypersphere with radius r inscribed within a hypercube with sides of length 2r, what proportion of the hypersphere resides within the hypercube for a given dimension D? This sounds like an absolutely dumb question if we use or 2D and 3D logic. What amount of the sphere is inside the cube? 100%! It's right there in the problem description! Right? But this is entirely wrong. As dimensionality increases, very quickly, ~0% of the hypersphere resides within the hypercube and this is all due to the fact that hypercubes and hyperspheres have very different representations in high dimensions. Yes, there is still visual understanding we can draw to help reason, but again, this requires us to look at the problem from a different vantage point (the key thing I stressed in my original post btw). We have to look at the dual of the problem and instead ask "what is the ratio of the volume of a hypersphere to the volume of the hypercube it is inscribed within?" This will help but there's also a reason I put this question after the previous one because we know that there is not a clean linear relationship here and that at least with the sphere the volume increases and then decreases. Our visual intuition only helps if we can recognize that the relationship has everything to do with the corners of the cube.

It is easy to look at these 3 cases and see how visual intuition from low dimensions can help, but be careful about post hoc trivialization. It's easy when you know these things and after they have been told to you but they are not so clear when you're being presented with the problem. All 3 of these problems are deeply related to the nearest neighbor problem and the concentration of measure. Where we run into the problem that it becomes nearly impossible to distinguish the nearest point to the furthest point. Leveraging our lower dimensional visualization can provide help but it is important to remember that they are a small part of the much larger story here. Which of course should make sense because as the dimensionality grows our 2D/3D slices of those are a smaller and smaller portion.

This is why I referenced the blind man and the elephant. Not because you can't use visualization to aid you, but because to see the elephant there you have to sample that elephant at many different places. And just like the blind man, you don't know how big that elephant is and where things are changing.

But this stuff isn't going to help with even more abstract concepts. Like how there is no division algebra in 3D and how the largest is represented via the Octonion. How by moving to quaternions we lose commutativity and how moving to octonions we lose associativity. These are very fundamental features of math that we are losing while moving to higher dimensions.

So I'm not sure why you're pointing to Farin and Hansford's book. Their highly concentrated on low dimensions and only briefly discuss generalization at the very end. They miss many important concepts. Not because the book is poorly written but because they are far out of scope from what is being taught. My caution about high dimensional spaces does not mean I don't absolutely love Needham's books on Visual Differential Geometry and Forums or his book Visual Complex Analysis or even Carter's inspired Visual Group Theory. These are masterpieces and books I highly recommend. But it is about what is being communicated. We work frequently in lower dimensions and there is a wealth of information there. But this does not mean higher dimensions are not absolutely littered with pitfalls and paradoxes. The true study of high dimensional spaces is relatively new and drove people like Cantor and Boltzmann insane. Every small mistake that is easy to make in low dimensions becomes exponentially more important in high dimensions. Yes, there are dimensional reduction techniques and these help with varying degrees (utility being critically dependent upon the latent dimensionality of the data) but every single one of these comes with major concessions. There is often even very bad science performed due to this lack of understanding. Techniques like t-SNE and UMAP are routinely unknowingly abused to draw inaccurate conclusions about these spaces, as it is possible to perform nearly any clustering you wish (see Lior Pachter's Picasso). Even PCA is frequently abused in this area due to there being subtle assumptions being invalid on different problems.

My point is: you absolutely should warn people that there are many pitfalls when dealing with high dimensions. Anyone brushing these differences off as inconsequential nuances is just ignorant of this rich space. I don't blame anyone for being unaware of these things because it requires some very complex mathematics to truly understand, and we're talking about math that the average scientist is never going to be introduced to. You likely won't even see this in a given science PhD. Not because it doesn't matter, but because it is just very difficult math. And we learn math through a game of telephone that is often hyper focused (not a great way to learn it tbh)