Visualizing beyond 3 Dimensions. Peeking into unseeably complex data | by Alec Morgan | Towards Data Science

Vision is arguably one of our greatest strengths as humans. Even the most tremendously complex ideas tend to become easy (or at least, much easier) to understand as soon as you find a way to visualize them — our occipital lobes have done us well. That’s why when you’re working with datasets that have more than 3 dimensions, it can be a challenge just to understand what’s going on inside of them, much less to present your findings to others. That’s the challenge we’ll be tackling today.
To this end we’ll be using a King County house sales data set which contains information on over 21,000 real house sales from 2014 and 2015. This dataset also happens to contain 21 pieces of info in every single record, such as price, square footage, condition of the property, and many more — in other words, this dataset is 21-dimensional. I actually had to transpose it just to get it to fit horizontally for this next shot.
df.head(10).T
Where do we even start? One of the easiest ways is to plot histograms for all metrics, or to plot all features against your choice of target (e.g. price) in scatterplots.
df.hist() 
This tells us something about all of our pieces of data individually, but nothing about how they relate to each other. Enter scatterplots — yes, I just said scatterplots aren’t too much help for this either, but bear with me.
For visibility’s sake, lets engineer a price_thousands feature and take a random sample of roughly 3% of our data points. 3% is still ~600 in this case, any more than than that and the plot would get too crowded in this case.
df['price_thousands'] = df['price'] / 1000np.random.seed(25565)df = df.sample(n=round(len(df)/33))plt.figure(figsize=(9, 9))sns.scatterplot(x='price_thousands',                 y='sqft_living',                 palette='Blues',                 data=df);
The result looks like this:
However, just by adding a few new parameters, we can make it look like this...
sns.scatterplot(x='price_thousands',                 y='sqft_living',                 hue='grade',                 size='sqft_living',                 palette='Blues',                 data=df);
By essentially adding hue as a dimension, we’re able to essentially able to represent 3 dimensions in 2 — not a bad deal! Notice that we’ve also represented sqft_living twice here: once as an axis, then once more as the scale of the points. If you’re thinking to yourself it’d be much better to use the scale of the dots to represent sqft_living and the y-axis to represent some other feature of the data, then in principle I agree with you wholeheartedly. In practice however, with this particular dataset such attempts get so messy that the data becomes visually intractable. At least we can increase the readability of this data by representing sqft_living twice though, so it’s not as though nothing has been gained. I would say the takeaway from this tangent is that data visualization is still more of an art than a science.
Back to the topic at hand: how to visualize beyond 3 dimensions. Maybe we can combine our new tricks of changing color gradients and point sizes with an actual 3D scatterplot? Well, yes, but actually no.
from mpl_toolkits.mplot3d import Axes3Dfig = plt.figure(figsize=(9, 9))ax = fig.add_subplot(111, projection='3d')x_param = 'price_thousands'y_param = 'sqft_living'z_param = 'bedrooms'x = df[x_param]y = df[y_param] * 25z = df[z_param]ax.set_xlabel(x_param)ax.set_ylabel(y_param)ax.set_zlabel(z_param)ax.scatter(x, y, z,            c='b', s=50,           edgecolors='w',            marker='o',            alpha=0.4);
The shifting hues and point sizes are a feature of Seaborn, but Seaborn does not support 3D scatter plots as far as I know. Conversely, Matplotlib may be able to plot in 3-dimensions, but it does not support those shifting hues and point sizes that are so lovely and useful. In summary, there is no solution (that I know of) which comes right out of the box ready to let us have our cake and eat t too. If this does exist though, please do share in the comments!
Lets try one more approach: surface plots.
fig = plt.figure(figsize=(9, 9))ax = fig.gca(projection='3d')ax.plot_trisurf(y, x, z, cmap=plt.cm.viridis, linewidth=0.2)ax.view_init(30, 145)plt.show();
Hmm. As it turns out, surface plots are very messy unless we sort their values. However, while sorting the values will give us a clearer picture, it won’t necessarily give us a more useful one.
x = np.sort(x)y = np.sort(y)z = np.sort(z)fig = plt.figure(figsize=(9, 9))ax = fig.gca(projection='3d')ax.plot_trisurf(y, x, z, cmap=plt.cm.viridis, linewidth=0.2)ax.view_init(35, 140)plt.show();
All these variables correlate very strongly with each other, so what we end up with is this very narrow and linear shape that’s difficult to make heads or tails of. In the end, it seems that the scatterplots provided a more effective visualization, at least for this particular dataset. Others translate somewhat better to this medium however.
iris = sns.load_dataset('iris')fig = plt.figure(figsize=(9, 9))ax = fig.gca(projection='3d')ax.plot_trisurf(iris['sepal_length'],                 iris['sepal_width'],                 iris['petal_length'],                 cmap=plt.cm.viridis,                  linewidth=0.2)ax.view_init(30, 145)plt.show();
There are some more tricks we haven’t tried yet (such as different point shapes or animating our plot to change over time), but the fact of the matter is we’re still not close to visualizing 21 dimensions simultaneously — it’s not exactly possible. But that doesn’t mean we can’t glean meaningful insights from our super-dimensional data. We probably care less about all 21 dimensions of this data and more about what all that data can actually do for us, such as allowing us to more accurately predict price. To that end, we can simply plot our error values in two dimensions.
residuals = model_1st.residfig = sm.graphics.qqplot(residuals, dist=stats.norm, line='45', fit=True)plt.xlabel('Quantiles of Standard Normal Distribution')plt.ylabel('Quantiles of Residuals')plt.title('Distribution of Residuals (Best Model)')fig.show()
Even if we can’t see all 21 dimensions simultaneously, there are plenty of ways that we can take slices of that data and glean meaningful insights from it. If you know of any more, please let me know!