Loss landscapes and the blessing of dimensionality
By: Javier Ideami
Tags: Artificial Intelligence, Machine Learning, Data Science, Editors Pick, Deep Learning

Loss landscapes and the blessing of dimensionality

Explore visualizations of deep learning loss landscapes, the blessing of dimensionality and other visuals including GANs & Geometric DL

Loss landscape visualizations by Javier Ideami | losslandscape.com

In this article, you will:

Understand how the loss landscapes of neural networks can be visualized.

how of neural networks can be visualized. Explore high resolution visualizations (both static and in movement ) of loss landscapes using real data captured from a variety of networks, from convolutional nets to GANs.

(both ) of loss landscapes captured from a variety of networks, from convolutional nets to GANs. Explore ways to analyze these visualizations with examples.

with examples. Explore other visualizations that go into Geometric deep learning and Bayesian deep learning .

that go into and . Explore the “blessing of dimensionality” concept.

Visualizing the loss landscape. Infographic by Javier Ideami | losslandscape.com

Optimization and life. It’s all about movement

“Life requires Movement” — (Aristotle, 4th century BC)

“Life is movement. The more life there is, the more flexibility there is. The more fluid you are, the more you are alive.” — Arnaud Desjardins

“Life is movement, movement is change” — beginning of a quote by Neale Donald Walsch

“If life boils down to one thing, it’s movement. To live is to keep moving” — Jerry Seinfeld

As many well known people have told us throughout history, life is movement. Life is changing your state in a proactive way, going from A to B. Life is also an expensive process and that makes the process of going from A to B a delicate, fascinating process that needs to be optimized. And that’s where we begin this article. And although we will focus throughout the following sections on deep learning and A.I, we will be touching simultaneously on universal themes and principles that go to the core of what means being alive. Moving. Going from A to B. Doing so in an optimized way. In a process that depends on a very large number of parameters, which makes it multidimensional. Which makes it hard to visualize for beings that operate in only 3 dimensions (4 with time). Which is the whole point of this article. So let’s begin this ride where it all begins, with movement.

Say we want to go from A to B in regards to some objective.

We want to learn to draw a human face.

We want to become a successful medical doctor.

We want to detect financial fraud when it happens.

We want to wake up early tomorrow to do exercise.

We want to lead a satisfying life.

We want to eat healthier this week.

We want to automate the process of detecting intruders in homes.

Some of these challenges will take minutes, other hours, others days and some of them years. Some of them depend on a moderate number of factors, others depend on a massive number of them.

We want to optimize these and infinite other challenges and the objective is always going from A to B. You may also combine many challenges and see life itself as a massive fractal made of optimization processes at different scales.

Going from A to B could be tackled in different ways. We could do it very systematically, trying lots of possibilities. Or we could try to find the most efficient way to get there as soon as possible. It turns out that life is all about optimization because, as mentioned earlier, life is expensive. Time is expensive, glucose is expensive, keeping us alive, and life in general, is expensive. So it’s all about optimizing the process of going from A to B. Finding as fast as possible the most efficient way to get to the objective.

Now, if we want to optimize any challenge, we first need to understand how many factors, how many parameters are involved in the process of solving that challenge. So, how many factors are involved in each specific challenge?

Some challenges we face depend on 1 or 2 key factors. Others in 10 or 20. Some of them in 100 or more. And some, like our objective of becoming a successful medical doctor may depend on thousands of factors whose values have to combine in the right way over many years to take you to that objective.

Deep learning and neural networks also deal with complex challenges. They also require an optimization process to find the optimum combination of factors that help the network go from A to B: from a series of images to predicting what those images represent. From a sequence of words in English to the same sequence in Spanish, etc.

Now, in order to tackle complex challenges, neural networks use architectures composed of millions, sometimes billions of parameters. Their optimization processes depend on millions or billions of factors. Their parameter space, or weight space, is very high dimensional.

To find out how we can combine the different values of those parameters to reach our objective, we cannot systematically try different combinations randomly. We would be trying forever. Instead, there are different optimization algorithms that are used to find the optimum combination of those parameters. One of them, very used in deep learning, is gradient descent.

Graphic by Javier Ideami | losslandscape.com

Furthermore, not only we want to find a combination of values of our parameters that takes us efficiently to the objective. But it would be great to find one that generalizes as well.

This means to find a combination of values that could take the network not only from A to B, but also from A2, A3, and other variations of A (within the same statistical distribution space) to B.

Taking the network from D, E, F, that is, data that is out of the distribution space of A, to B, is also an objective, one that we can do as humans, but that is very difficult for deep learning at the moment.

In general, we want to avoid overfitting, that is, producing a combination of parameters that creates a mapping function from A to B that adapts too closely to the training data (A), but whose performance degrades when applied to other unseen input data points.

We also want to avoid underfitting, producing a mapping function from A to B which is too simple and fails to capture the existing variation in our source data.

Therefore, we want to optimize (avoid underfitting), but also generalize (avoid overfitting).

But in this article we focus not on how we find the values of those 10, 100, 1000 or 1 billion parameters. Instead, we focus on how we may visualize that optimization process. The process of moving from A to B.

Visualization has been a favourite tool of great creative geniuses throughout history. Visualizing allows us to study and access simultaneously a lot of information at low levels of abstraction (rather than accessing information sequentially at high levels of abstraction like when we, for example, use language). Visualization can help us gain insights that may take us much longer to access through other means (think of Albert Einstein visualizing that he was a photon travelling through space, as he, in combination with other strategies, visualized his way to his theory of relativity).

So, let’s say that we would like to create a visualization that showed us how we went from A to B, from being a student to becoming a medical doctor, from being an unhealthy eater to becoming a person that eats a balanced diet, from having a financial data series to being able to extract fraud signals from that data. We may then bump into trouble when one of these challenges depends on too many parameters:

Visualizing the optimization process seems impossible. We can only visualize up to 3 dimensions , and yet these challenges depend on many more, sometimes millions of dimensions/factors/parameters.

, and yet these challenges depend on many more, sometimes millions of dimensions/factors/parameters. If we want to explain to somebody how we went from A to B, this also becomes complicated due to the high number of parameters involved in the process.

Our brain is not even designed to make sense of more than 3 dimensions (4 with time). If we could access somehow a massive number of dimensions, we may perceive nothing, or we may perceive a big mess. And the more we went within, the more nonsensical it may all feel.

This messy “something” represents how we may perceive a high number of dimensions. Maybe as something invisible to us that we cannot even notice, or maybe as a big mess, incomprehensible in every way. Graphic by Javier Ideami | losslandscape.com

Our brain is not prepared to interpret more than 3 dimensions at once. Graphic by Javier Ideami | losslandscape.com

And yet, do we give up? No, of course not. What do we do instead in all the above cases?

We perform a dimensionality reduction process. And this is something that we are doing constantly in our lives.

Simplifying complexity

If I ask you: how did you go from being a student to becoming a successful medical doctor? You don’t stop and tell me: “Oh sorry, that process depended on so many parameters that I cannot really explain to you how I went from A to B”.

Instead, you reduce the high complexity/dimensionality to something more manageable. With your incredible brain, you identify the most essential vectors/directions/factors/parameters that have influenced that process of going from A to B. And you reduce that high dimensionality from maybe thousands of factors to just 2, 3, 4 or 7, for example. And then you proceed to explain that going from A to B was possible because of this, and this, and that. Voila.

In the same way, if you want to do an infographic, or a chart or a drawing or whatever that displays that A to B journey, you will do a similar thing. You will reduce the dimensionality/complexity and then proceed to create that visualization using just the reduced number of dimensions.

As we continue reflecting on these visualizations, we begin to realize that apart from creating visual representations of the different factors involved, it would be fantastic to be able to chart the actual progress towards the objective.

That is, to visualize how far from the objective we were at different parts of the optimization process.

And we realize that at every step of the process, we can combine the different parameters involved, after assigning a numerical value to them, and produce a result that expresses where we are at that point in time.

And then we can compare that result with the objective and estimate how far we are from that objective.

Reaching the objective will imply that there is no difference between the value produced by combining our different parameters and the intrinsic value of the objective. The difference will be 0. We will have reached B.

Conversely, at the beginning of the process, the difference between the result of combining our parameters and our objective will be large.

This difference, by the way, is what we call in deep learning, the loss value. The difference between what we should obtain (our target objective) and what we are obtaining at the moment (our current state).

If we plot this process we find out that we are moving from a high area (a high loss value) to a low area (a low loss value). And so, our objective is, in fact, to go from a high position (when the way we combine the different factors involved produces a result that is far away from our final objective), to a low position, one that is as low as possible (when the way we combine the different factors involved produces a result that is very close to our final objective).

If we plot only our state, we will follow a point that moves from a high position to a low position.

But, if we plot not only our position, but also other nearby combinations, a range of other combinations in parameter space, we will then create a surface, a landscape.

The loss landscape

We can describe this loss landscape in many different ways.

On the one hand, it is a visual representation of the performance (loss value) of different areas of our parameter or weight space, of how different combinations of our parameters perform.

It can also be used, when following the state of our agent/minimizer through the optimization process, as a visual representation of the process of looking for different ways of combining the parameters involved in our challenge, to produce a result that is as close as possible to our final objective.

A visual representation that shows not only where we are, but also other possibilities that exist around our state, within the parameter space that surrounds our state, allowing us to study the morphology and dynamics of a surface that expresses this fundamental and constant principle in life: movement. Going from A to B. And looking for the best possible way to do that.

In a recent conversation with Tom Bilyeu, Donald Hoffman, professor of cognitive science at the University of California, Irvine, tells us, as mentioned previously, that one of the defining features of life is precisely that, movement, directed movement. As such, visualizing these processes goes way beyond neural networks and deep learning. We are touching an essential part of what defines life, intentional proactive movement from here to there, in search of the optimum way to get to those low valleys as soon as possible, because our survival is in the game, because computing is expensive, glucose is expensive, life in general, is expensive. And that’s why we may view our entire existence as a massive fractal of optimization processes.

And when we work with deep learning processes and neural networks, we are working with optimization processes that depend on millions of parameters.

In such cases, plotting these surfaces and landscapes becomes a bigger challenge. And that’s what we are going to tackle in the next sections. So let’s now go to the specific case of plotting these intriguing landscapes within deep learning training processes.

The usefulness of loss landscapes

Most deep learning processes have the objective of learning a complex function (the mapping from A to B that I wrote about previously). And how can we evaluate the performance of those learning processes? We can do it with what we call a loss function. Such function will tell us how far apart are the current output of our network and what that output ideally should be. That difference is the loss. And as the training process advances, our objective is to minimize that loss, to make it as small as possible. A loss value of 0 means that the network performs perfectly.

Note: What is described in this section applies mainly to supervised learning, the most used by far type of learning nowadays in these kinds of systems.

Typically, at the beginning of the training process, the network won’t perform very well and our loss value will be pretty high. As training progresses, loss values gradually become smaller.

As mentioned previously, visualization has been a favourite tool of scientists, explorers and creators throughout history. As opposed to the abstract and sequential nature of language, visualization allows us to simultaneously access a lot of information at much lower and more detailed levels of abstraction.

If we are able to visualize the dynamics and morphology of these loss landscapes as the training process progresses, and if we can do that in as much detail as possible, we increase our chances of generating valuable insights in connection with deep learning and its optimization processes.

Loss Landscape generated with real data: Convnet, imagenette dataset, sgd-adam, bs=16, bn, lr sched, train mod, 1 million pts, 0.5 w range, 20p-interp, log scaled (orig loss nums) & vis-adapted | losslandscape.com

What about using just maths instead of visualizations?

Maths and visualizations complement each other. They are both important in their own right. The value of visualizing is in accessing simultaneously a very rich and detailed amount of data at a low level of abstraction, which can potentially accelerate the discovery of insights that may take us longer to extract from numerical calculations.

Let’s go deeper into those landscapes. And those loss values, what are they related to in terms of the network itself?

The loss of a network, let’s recap, is related to the difference between the current output of the network and what that output should be. And the output of the network is calculated by performing operations with the weights of the network. Therefore, the loss value depends on those network weights.

All right, so that’s what the loss value is. But what about the entire loss landscape? The loss landscape represents the loss value calculated not only with the specific weight values that the network may have currently, but also in a certain range around the current position within the weight space of the optimizer.

So the loss landscape helps us visualize the loss values within the weight space of a network (which includes the current weight values but also many other combinations of them), and is typically centered in the current state of our optimizer. Therefore, in the center of the landscape we can see what the loss value is for our current set of weights. And around that central point, we can see loss values calculated with other combinations of the weights of the network.

Now, remember that we said that to compare the current output of the network with its ideal output we use a function to calculate the loss value. And that function is the loss function (or cost function). Because neural networks have lots of parameters, this function is a multidimensional and multivariable function. This will become important shortly, because it’s easy for us humans to visualize a 1 dimensional function, or a 2D or 3D function. But what about a 1 million dimensional function? or a 1 billion dimensional function? Stand by, we are getting there.

Loss Landscape generated with real data: Convnet, imagenette dataset, sgd-adam, bs=16, bn, lr sched, train mod, 1 million pts, 0.5 w range, 20p-interp, log scaled (orig loss nums) & vis-adapted | losslandscape.com

The loss landscape, does it change as training evolves, or does it have a fixed shape?

This is a typical question that often comes up. Wait, I see the landscape change in some videos. Shouldn’t the landscape be fixed? Let’s explain this in detail.

The shape of the loss landscape is dependent on the weights (parameters) of the neural network and on the loss values generated by that network. In the full high dimensional weight space that covers all possible combinations of values of those weights, the landscape is fixed in relation to that specific network architecture, to the different variations of those weights (weight space) and to the calculated loss values derived from the previous two factors.

However, things change when we create practical representations of these landscapes:

First of all, we cannot visualize the entire loss landscape . What we do, instead, is to center the landscape on the current position of our minimizer (so the center of the landscape shows the loss value calculated with the current weight values of the network).

. What we do, instead, is to (so the center of the landscape shows the loss value calculated with the current weight values of the network). And then we display as well a range of the landscape around that central point ( sometimes the confusion arises because we call “loss landscape” to both the full high dimensional entire landscape, and also to the partial nearby-region reduced dimensionality version of it ).

). As the training process evolves, the position of our minimizer (calculated from its weight values within weight space) changes. If the central point moves, it means that the range we visualize around that central point is also going to displace and move within the entire full landscape. And that’s one of the reasons why we see the landscape change. We are riding alongside the current position of the minimizer as it moves around weight space during the training process. And as we ride alongside it, its surroundings, the loss values of the part of the landscape immediately surrounding the current state of the minimizer, change as well. This gets even trickier , because changes happen in the high dimensional space, not in the reduced dimensionality interpretation of those changes, and that has an impact on the dynamics of the landscape in the visual representation as well (more about this in a later section).

and move within the entire full landscape. And that’s one of the reasons why we see the landscape change. during the training process. And as well. This , because interpretation of those changes, and that has an impact on the dynamics of the landscape in the visual representation as well (more about this in a later section). As mentioned earlier, another key point is that we cannot visualize 1 million dimensions . That’s why, as I will explain a bit later in more detail, we do a dimensionality reduction process to reduce that huge amount of dimensions to just 2 (which adding the third dimension, which is the loss value, produces a 3D visualization).

. That’s why, as I will explain a bit later in more detail, to reduce that huge amount of dimensions to just 2 (which adding the third dimension, which is the loss value, produces a 3D visualization). Therefore, even though the entire full high dimensional landscape , which includes the entire weight space related to our network, does not change, what changes is the partial, relative, low dimensional interpretation of the nearby-landscape located around our moving and evolving minimizer.

, which includes the entire weight space related to our network, located around our moving and evolving minimizer. This low-dimensional landscape, which expresses the nearby loss values located around the present position of our optimizer, represents a dynamic changing system, and studying it, we have the chance to access interesting insights about the impact that different network architectures, different hyperparameter values and other factors have on the training process.

It is essential, therefore, to clarify that our changing landscape is representing a partial region of the full landscape , located around our moving, evolving minimizer, and visualized through a dimensionality reduction technique .

, . As the training process advances, the weights of our network change (our position in weight space changes), therefore the position of our minimizer evolves, and the nearby region around it does also, within the lower dimensionality visualization we are creating.

You may now ask: if we apply such a dimensionality reduction process, are we still preserving enough useful information? The answer is yes, and I will explain why in detail in the next few sections.

COLUMN DESCENT. External view of the area leading to a minima. The area reminds of the top part of a column. Loss Landscape created with real data from the training process of a convolutional network. | losslandscape.com

So what about that high dimensional space? How do we visualize it?

Easy. We cannot. We, human beings, cannot visualize data in 100 million dimensions. But no worries, because math gives us tricks and ways to do what, in fact, we are doing constantly in our lives.

When we take a picture with a camera , our camera cannot capture and represent 3 dimensions on its flat screen, or on its square sensor. So the device captures the information and transforms it into 2 dimensional data.

, our camera cannot capture and represent 3 dimensions on its flat screen, or on its square sensor. So the device captures the information and transforms it into 2 dimensional data. When we work on complex projects that depend on a large number of variables, we simplify this complexity, reducing it to a smaller number of variables that still express the essence of the complex scenario.

Whenever we do any of the above: is the new lower dimensional representation still useful? Does the 2D picture still communicate valuable and useful information about the 3D world? Yes, of course. Let’s now focus on how we go about this in relation to our loss landscapes.

It’s clear that we need to reduce the high dimensional data to a lower number of dimensions. There are many dimensionality reduction techniques used in machine learning, which include, for example, PCA (Principal component analysis), LDA (Linear discriminant analysis), MDS(Multidimensional scaling), Spectral Embedding, t-SNE, Isometric Feature Mapping, Factor Analysis, etc. We can even use autoencoders (AE neural nets).

In our case, we are looking for a technique that gives us a way to reduce the number of dimensions while preserving enough information about our state and about a range of possibilities and values around where we are.

There are different strategies to do this. One of these strategies was introduced by the excellent paper Visualizing the Loss Landscape of Neural Nets by Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, Tom Goldstein. With this strategy, what we are going to do, thinking of an analogy, is to create a plane and kind of slice the high dimensional space with that plane to transform it into a 2D surface on which we can build our 3D visualization.

To do that, we are going to firstly find and select a couple of random directions in weight space. These are random vectors and they should have the same dimension of the weight vectors in our network. With those 2 random directions we can now form a plane (more about the orthogonality of those 2 random directions a bit later).

Then, we can take that plane and slice the high dimensional space with it, revealing its structure projected in 2 dimensions. At that point, we can add a third dimension, which will be the loss value calculated at each point within that plane.

Let’s remember that we will position the current state of our minimizer at the center of that projection and then look around a certain range within that projected 2D plane. And with that, voila, we can represent the nearby region around our optimizer in 3D (as well as its current state).

Slicing high dimensional space. Graphic by Javier Ideami | losslandscape.com

Slicing high dimensional space. Graphic by Javier Ideami | losslandscape.com

Cool, but what about using PCA?

PCA (principal component analysis) is a dimensionality-reduction method that could in principle be applied here. However, PCA will help us in visualizing the landscape along the most optimized directions. And that is going to help us see those parts of the landscape that are more optimized. We, instead, want to see a more diverse range within weight space, other parts of the landscape further away that are not so optimized. Using other kinds of directions (like the method previously described) allows us to do that.

All right, so we use two random directions. But if they are random, how do you know that they are orthogonal (perpendicular) to each other so that they may form a plane?

Math to the rescue! Math experts have proven that within high dimensional spaces, if you choose random vectors, they tend to be orthogonal to each other (there are proofs of this online at various places).

Let’s make an analogy to explore this a bit. With more dimensions, we get more complexity, and also more possible directions that can be accessed.

Let’s consider a random vector in a 1 dimensional world. Nothing orthogonal to it can exist.

Let’s move to a 2 dimensional world. Orthogonal vectors to our random vector will form a 1 dimensional line. Therefore, the orthogonal vectors live in 2–1=n-1= 1 dimension.

Now let’s go to a 3 dimensional world. Orthogonal vectors in that world will compose a 2 dimensional plane. Therefore, the orthogonal vectors live in 3–1 = n-1 = 2 dimensions, producing 2 dimensional planes.

As we keep increasing the number of dimensions, the orthogonal vectors that we may find are present in a subspace of that space which has n-1 dimensions. Now we can see that the larger n is, the larger is as well the size of that subspace, the share of the complete space that is taken by the possible orthogonal vectors.

In summary, even though it cannot be guaranteed that two random vectors within a high dimensional space will be with all precision orthogonal to each other, we can however predict that they will be very possibly and very nearly so.

All right, so we reduced the dimensions, how do we then combine them to plot the landscape?

Let’s recap. We have created a plane from two random orthogonal vectors. And now we have to decide on the coordinates of our surface and on the range that we will explore around the central point of that plane (which, let’s remember, will be the current state of our minimizer).

So we create a 2D grid, which will include the x and y coordinates of our visualization. The center of that grid will be the 0,0 coordinate and in there we will position the loss value that corresponds with the current state of our minimizer (calculated with the current weight values of our network).

We then explore around that central point within a range. We may pick for example a range that goes from -1 to 1. Picking that range means exploring a portion of the landscape that has a similar magnitude to that of our weights. Basically, we would explore the landscape within a range of variation in weight space similar to the magnitude of our weights, and that’s pretty reasonable.

Then, we decide the resolution of the landscape. We can break down that -1 to 1 range into as many points as we like. For example, a 50x50 visualization will use 50 points in the first axis, and another 50 in the other axis, for a total of 2500 points. For each of those 2500 points, we have to calculate the loss value, which gives us the third axis, the vertical z coordinate.

And how are the loss values calculated at each of the points in the grid?

Time to calculate the loss values. What we do is to take the current weight values of our minimizer, that correspond to the center of our grid (0,0) and change them along the random directions we previously selected. And we modify them by the amount that corresponds to the point on the grid whose loss value we want to know. Let’s go into the details of the equation itself:

Theta

Theta: It represents the current weights of our network, the parameters of our minimizer. The loss value calculated with these parameters will appear at the center of the grid (0,0)

Delta and Eta

Delta and Eta: These 2 represent our two random vectors, our random directions in weight space.

Alpha and Beta

Alpha and Beta: They are the x and y coordinates of each of the points in the grid plane.

To calculate the loss at one specific point of the grid plane, we will first calculate a new set of parameters(weights) which are the result of adding the original ones of our minimizer to the result of adding the product of the coordinate values of that point by each of our random directions. And then we will calculate the loss with that new set of parameters.

Projecting our parameters along the plane

Finally, the coordinates of each of the points of our loss landscape will be:

Coordinates of each point of our loss landscape

Slicing high dimensional space. Graphic by Javier Ideami | losslandscape.com

Visualizing the loss landscape. Graphic by Javier Ideami | losslandscape.com

What about zooming in and out?

We are looking at a certain range of the full landscape around our minimizer. So modifying that range, we can zoom more or less around our current position. If instead of projecting our weights through a -1,1 range, we use a -0.2, 0.2 range, we will be focusing our detail (whatever number of points we are calculating) on a much tighter range around our minimizer. Conversely, using a -3,3 range, we will zoom out to encompass a way larger area that shows areas of the landscape further away from our minimizer.

How far can we go around our minimizer?

As much as we like. Of course the further you go, the more you should increase the number of points calculated by axis in order to maintain a similar amount of detail.

The issue, however, is that typically, as we go further and further away from the current position in the weight space of our minimizer, we find that loss values become really high and the landscape moves up and up so we stop seeing interesting features. When facing extreme contrasts in the visualization, we also tend to visualize the log of the loss values. This helps us in visualizing the extreme contrasts that appear on the edges of the visualization.

Therefore, typically, the area we explore is more around the -1,1 range.

As we move away from a local minima, we may wonder if we could find paths that connect that minima with other local minima around our position. In recent years researchers have uncovered ways to find shortcuts and connections between local minima, paths through which loss values remain very low. These paths are typically not straight lines but curved paths, polygonal paths, etc. Through a collaboration with Pavel Izmailov (@Pavel_Izmailov) and Timur Garipov (@tim_garipov), we created Icarus, a visualization based on the NeurIPS 2018 paper by Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson: https://arxiv.org/abs/1802.10026 where we visualize mode connectivity (paths connecting different local minima) using real data. Through these paths (a bezier curve in this specific case), training and test accuracy remain nearly constant.

ICARUS Mode Connectivity. Real Data. Resnet-20 No-Skip. Train Mode. CIFAR10. Batchnorm. SGD MOM, BS=128, WD=3e-4, M=0.9. NeurIPS 2018 ARXIV/1802.10026. Optima of complex loss functions connected by simple curves over which training and test accuracy are nearly constant. Icarus uses real data and showcases the training process that connects two optima through a pathway generated with a bezier curve. To create ICARUS, 15 GPUs were used over more than 2 weeks to produce over 50 million loss values. The entire process end to end took over 4 weeks of work | losslandscape.com

MODE CONNECTIVITY. Visualization data generated through a collaboration between Pavel Izmailov (@Pavel_Izmailov), Timur Garipov (@tim_garipov) and Javier Ideami (@ideami). Based on the NeurIPS 2018 paper by Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson: https://arxiv.org/abs/1802.10026 | Creative visualization and artwork produced by Javier Ideami | losslandscape.com

GRADIENT X-RAY. Mode Connectivity. Visualization data generated through a collaboration between Pavel Izmailov (@Pavel_Izmailov), Timur Garipov (@tim_garipov) and Javier Ideami (@ideami). Based on the NeurIPS 2018 paper by Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson: https://arxiv.org/abs/1802.10026 | Creative visualization and artwork produced by Javier Ideami. This artwork shows the area of the two minima and also higher loss areas which contain irregular morphology | losslandscape.com

ICARUS. Mode Connectivity. Real Data. Resnet-20 No-Skip. Train Mode. CIFAR10. Batchnorm. SGD MOM, BS=128, WD=3e-4, M=0.9. Visualization data generated through a collaboration between Pavel Izmailov (@Pavel_Izmailov), Timur Garipov (@tim_garipov) and Javier Ideami (@ideami). Based on the NeurIPS 2018 paper by Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson: https://arxiv.org/abs/1802.10026 | Creative visualization and artwork produced by Javier Ideami | losslandscape.com

GRADIENT THEATER. Mode Connectivity. Visualization data generated through a collaboration between Pavel Izmailov (@Pavel_Izmailov), Timur Garipov (@tim_garipov) and Javier Ideami (@ideami). Based on the NeurIPS 2018 paper by Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson: https://arxiv.org/abs/1802.10026 | Creative visualization and artwork produced by Javier Ideami. This artwork shows the area of the two minima and also higher loss areas which contain irregular morphology | losslandscape.com

FROM ABOVE. Mode Connectivity. Visualization data generated through a collaboration between Pavel Izmailov (@Pavel_Izmailov), Timur Garipov (@tim_garipov) and Javier Ideami (@ideami). Based on the NeurIPS 2018 paper by Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson: https://arxiv.org/abs/1802.10026 | Creative visualization and artwork produced by Javier Ideami. This artwork shows the area of the two minima and also higher loss areas which contain irregular morphology | losslandscape.com

Later on in this article, I will write about the blessing of dimensionality, as more and more researchers are considering that the great results we get using deep learning may be related to how these high dimensional spaces allow gradient descent to easily find good minima in regions that are local, close to our starting position, regardless of where we initialize the optimizer within the loss landscape.

Simulated representation of the Blessing of Dimensionality (it’s easy to find nearby paths to convergence). Graphic by Javier Ideami | losslandscape.com

Great, but how can I be sure that these visualizations provide useful data? and how precise are these visualizations?

Great questions! Let’s go for it.

First of all, applying dimensionality reduction techniques, transforming a very large number of dimensions into just 2, implies being realistic. In the same way that a 2D photograph cannot express the full richness of the 3D world, also our 3D loss landscape representations cannot communicate the full complexity of the high dimensional space.

Also, in the same way that taking a photograph from different angles will provide us with different perspectives of the 3D world, using a different set of random directions may also generate different perspectives of the loss landscape.

So, let’s make one thing clear. If your objective is to get a fully precise idea of the shape of the loss function and landscape, you need to analyze these 3D representations carefully. In the same way that looking at 2D photographs requires some careful analysis if we are to interpret with precision some of the features of the 3D world.

However, the key question is:

Can we obtain from these lower dimensional representations useful information ? Yes , absolutely.

? , absolutely. And could we obtain information that may trigger useful insights? It will depend on our specific objectives, but in general, yes.

Going back to the photography analogy. When we take photos, not only we transform a 3 dimensional world into a 2 dimensional representation. But we often apply filters and other treatments that further distort the original 3D data source. As mentioned earlier, our position and the angle we use when taking the picture have a lot of influence as well on the final result. And we could continue reviewing other ways in which our interpretation keeps moving further and further away from the original data source.

So, is the photograph a precise representation of that 3D world? No, surely not.

And yet, is the photograph giving us useful data about that 3D world? information that could even potentially trigger interesting insights? Yes, of course. Definitely.

That’s why we take pictures. And combining pictures from different angles, we gain new insights that deepen our understanding of that 3D world.

A similar thing happens when astronomers take pictures of galaxies, planets and stars. And a similar thing happens when we explore these loss landscapes, studying them from different perspectives and also in movement.

In a certain sense, everything is relative and if we begin to scratch the surface of the different interpretations and dimensionality transformations that we apply in life, we could go on for a long time. Our perception of the “3D world” out there is of course yet another interpretation-distortion generated by our brain. But each of these transformations provides useful information, data that can inform, inspire and even potentially generate deep insights.

Therefore, going back to these landscapes. Are they precise representations of the high dimensional “reality”? Definitely not. But are they giving us useful data about that underlying high dimensional space, data that could potentially trigger new interesting insights? Definitely yes.

And can we actually prove that? Yes, we can. For example, deep learning experts have demonstrated that the key convexities and non-convexities that appear in these kinds of reduced dimensionality representations match with those indicated by numerical analysis processes that are performed using the main related eigenvalues and eigenvectors.

And the shape, the morphology of these surfaces, is just the beginning. We may also study the dynamics of these landscapes as our minimizer moves through them. For example, studying the way these representations change over time as we modify various hyperparameters of our network, can help us understand in new ways the dynamics of our training processes and the impact that these hyperparameters have in them.

Therefore, even though we cannot visualize the entire high dimensional complexity, we can still gather many useful insights as we visualize these landscapes from different perspectives, just as we do when we take photographs.

And just as capturing movies = pictures in movement, deepen our understanding of the 3D world (because static pictures can sometime produce confusion in our interpretation of that world), in the same way, loss landscapes in movement, representations in which we ride alongside our minimizer as the training process progresses, give us new ways to deepen our understanding of the dynamics of these processes and their behaviour and state at each of the steps of the process.

And going back to what we mentioned earlier about eigenvalues. Math helps validate these representations by allowing us to verify if the convexity distribution we observe matches with what numerical analysis reports. And for that, as mentioned, we may use the Hessian and its related eigenvalues.

Hessian matrices contain second order derivatives. Through them, we can understand the curvature of the loss landscape. To get that data, the simplest way is to study the essence of the Hessian, which can be expressed through its eigenvalues and eigenvectors.

Fully convex functions have positive semi-definite Hessians with curvature values which are non-negative. Conversely, the opposite happens in non-convex functions.

The academic paper arXiv:1712.09913 tells us that the key curvatures of these low dimensional representations we are creating, are in fact averages, weighted averages of those present at the high dimensional original space and function. And that’s why by analyzing the eigenvalues of the hessian at each of the points of our landscape, we can estimate that on average, these visualizations are correct.

Calculating the maximum and minimum eigenvalues of the hessian at each of the points in our visualization, we can then use their absolute ratio to study and map their distribution. In this way, we can see which areas have more or less convexity. With this data, we can verify that those areas that look very convex in our representation, match with areas that contain very small negative eigenvalues. And those areas in our visualization that are non-convex match with areas that have large negative eigenvalues.

And why do we use eigenvalues?

Hessian matrices are complex entities and computing them requires a lot of resources. Many complex entities and systems can be better understood by working with some of their properties which capture their essential qualities.

That’s why we use eigenvalues and eigenvectors. They help us capture the essential and key features of systems, in this specific case, of the hessian matrix.

We are extracting the key data we need through a series of vectors and numbers, which is much simpler and faster than dealing with the full and often very large square hessian matrix.

Furthermore, eigenvalues and eigenvectors give us insights about the properties of the hessian matrix that are not so easy to understand from the full matrix representation. That’s why eigenvectors and eigenvalues are considered very useful in a variety of fields when we need to capture and extract the essence of a system. They are used, as an example, within image compression systems, through the process of discarding the small eigenvalues and preserving the ones that express the most important information of that entity.

So when we look at the eigenvalues of a specific point, what information do they give us?

Let’s go for it. When looking at the hessian at a specific point:

If the hessian at that point is positive definite (all eigenvalues > 0), that point is a local minimum of the function, a point of convexity .

at that point is (all eigenvalues > 0), that point is of the function, a point of . If the hessian is negative definite (all eigenvalues < 0), that point is a local maximum , a point of negative curvature ).

(all eigenvalues < 0), that point is , a point of ). If we have a mixture of positive and negative eigenvalues, that point is a saddle point.

By plotting the different combinations of eigenvalues, numerical analysis helps us understand and verify the convexity patterns of our landscape

Captured by Javier Ideami | losslandscape.com

And do we have to extract all the eigenvalues of the Hessian?

The hessian of a typical neural network is really large. Its size is proportional to the square of the number of weights of the network. And as we know, typical neural nets have a very large number of parameters.

Therefore, computing with the complete hessian is not feasible in relation to any network of a significant size. It is not practical at all to try to gather all the eigenvalues.

Instead, we use, for example, some algorithmic tricks and the Scipy library to extract the smallest and the largest eigenvectors and eigenvalues, and that is enough for us to get the data we need in order to study the convexity distribution of the landscape.

What are some examples of areas in which these visualizations can help us?

There are so many possibilities. These visualizations help us deepen our understanding of the morphology and dynamics of the loss function as we move through the training process. The study of the dynamics of these optimization processes is one of the interesting areas. The study of the impact of changes in the values of different hyperparameters is another. But you could pick many others, for example: weight pruning and subnetworks. Research tells us that pruning weights, and finding good subnetworks can give us results that use less time and resources and yet are as good as those we obtain when using the full network. Visualizations can help us track and gain insights on the impact of these pruning processes on the loss landscape and function.

Interesting. Can a complex function be approximated by a simpler one?

Yes indeed. For example, the Taylor series (a way of expressing a function as an infinite sum of terms) shows us how we can prune some of the complexity of a function while still retaining its key features.

So how strong is the impact of different network hyperparameters in the morphology of these landscapes?

It can be pretty strong across the training phase. As an example, in training mode, the use of batchnorm or dropout can have a significant impact on the morphology of the landscape.

EDGE HORIZON APPROACH. Loss Landscape created with data from the training process of a convolutional network. Imagenette Dataset. Sgd-Adam, train mode, 1 million points, log scaled. Captured and visualized by Javier Ideami | losslandscape.com

Analyzing changes in morphology related to the use of batchnorm, we find morphological variatons in the landscape when it is active (we keep other parameters like batch size, dropout, mode, etc stable to be able to establish comparisons). Batchnorm can have a regularizing effect, reducing generalization error.

When the mean and the standar deviation are calculated in batch norm (calculations which are performed across batches), these values, these estimates, are noisy. The noise produced in the process of calculating the values is coming from other parts of the selected batch. That noise is then combined with the weights when the batchnorm operations are performed, and could potentially be helping the generalization capabilities of the network, producing a regularization effect.

Therefore, currently under study, one of the hypotheses in connection with the morphology of the related landscape is that it may be related to a higher level of resilience, the previously described level of noise that in a certain sense helps the network generalize better. Just an example of how we can analyze changes in morphology to search for new insights.

When we look at dropout, the impact on the training mode morphology is very clear, but the perturbations have a different character.

EDGE OF NOISE. Loss Landscape created with data from the training process of a convolutional network that uses dropout. The use of dropout produces this distinctive morphology and sharp patterns during the training process. Captured and visualized by Javier Ideami | losslandscape.com

Dropout introduces high frequency noise in the morphology of the landscape. These high frequency perturbations may be related to the way dropout helps the networks generalize better.

More examples

Let’s go into more examples about how we can analyze these visualizations. Due to the limited space available in this article, the explanations will be quite generic, but we will still cover different variations.

Let’s pick a specific moment of the process and show how we can use static high resolution captures to reflect about different aspects of the network (the same can be done with the dynamic moving visualizations which allow us in addition to study the dynamics of the process)

Captured and visualized by Javier Ideami | losslandscape.com

Captured and visualized by Javier Ideami | losslandscape.com

Captured and visualized by Javier Ideami | losslandscape.com

Looking at these captures, let’s compare and analyze the difference between using a batch size of 16 and 128 in regards to the apparent width of the minima . As explained somewhere else in this article, there are different opinions regarding the connection between flat wide minima and generalization. In this specific example, we find a connection in terms of the morphology between higher batch sizes, sharper minima and more overfitting. Lower batch sizes have often been connected with better regularization capabilities and better generalization, possibly because of the perturbations they introduce in the process.

in regards to . As explained somewhere else in this article, there are different opinions regarding the connection between flat wide minima and generalization. In this specific example, we find a connection in terms of the morphology between higher batch sizes, sharper minima and more overfitting. Lower batch sizes have often been connected with better regularization capabilities and better generalization, possibly because of the perturbations they introduce in the process. As mentioned earlier, both dropout and batchnorm introduce perturbations of different kinds. Dropout introduces higher frequency perturbations in the morphology. These perturbations evolve in different ways throughout the process (for example, the way they evolve may depend on the specific degree of dropout and the specific network architecture being used). As opposed to the potentially destructive perturbations caused by a very small batch size (if the learning rate is not adequate), the perturbations created by dropout are like an extra layer on top of the underlying morphology, possibly contributing in this way to keep the training process away from overfitting.

introduce perturbations of different kinds. introduces in the morphology. These perturbations evolve in different ways throughout the process (for example, the way they evolve may depend on the specific degree of dropout and the specific network architecture being used). As opposed to the potentially destructive perturbations caused by a very small batch size (if the learning rate is not adequate), the perturbations created by dropout are like an extra layer on top of the underlying morphology, possibly contributing in this way to keep the training process away from overfitting. Looking at the example with a batch size of 2 , we see how in this specific case (because of the set of parameters being used), inestable morphological features are produced (emphasized, in between other things, by a learning rate that is too high), producing many perturbations that obscure the access to a good minima. This kind of instability could be compensated by using smaller learning rates to move more slowly through the landscape . Analysing this aspect in combination with the learning rate could allow us to find the sweet spot that opens access to good minima while keeping the benefits of using low batch sizes.

we see how in this specific case (because of the set of parameters being used), inestable morphological features are produced (emphasized, in between other things, by a learning rate that is too high), producing many perturbations that obscure the access to a good minima. This kind of . Analysing this aspect in combination with the learning rate could allow us to while keeping the benefits of using low batch sizes. Let’s see another example, in this case related to the kind of architecture known as Resnet. First, we examine the morphology of the same resnet with and without skip connections. We verify that the lack of skip connections produces a more rugged landscape. We proceed to study the dynamics of the comparison, verifying that over time, the version with skip-connections stabilizes at a lower loss value.

WALTZ-RES visualizes the difference in morphology and dynamics between two resnet-25 networks, one with skip connections and one without. In this fragment of the visualization, we can see the first 2 and a half epochs of the training process. We ride along the minimizer while exploring its nearby surroundings | losslandscape.com

As mentioned earlier, high resolution visualizations can provide a detailed view on the impact of hyperparameter changes on the loss function and landscape. On this visualization, we can see the dynamics of the landscape in relation to different changes in the value of the learning rate. This is shown through a learning rate stress test, in which we begin with a moderate stable rate and then increase it suddenly and decrease it again through different cycles.

LR COASTER visualizes a learning rate stress test during the training of a convnet. We ride along the minimizer while exploring its nearby surroundings. I use extreme changes in the learning rate to illustrate how the morphology and dynamics of the loss landscape change in response to the changes in the learning rate. The resolution (300K loss values calculated per frame) allows us to explore the change in morphology | losslandscape.com

The creation of a 3D visualization allows us to study the landscape from any angle and perspective, being able to analyze the structure of minima from a diversity of angles and in connection with various network architectures and hyperparameters.

GOBLIN takes us on a journey from above the edge horizon of the loss landscape of a convnet, during its training process, through the edge horizon (laterally) and to the perspective from below its dynamic convexity. We ride along the minimizer while exploring its nearby surroundings | losslandscape.com

As I explored more and more landscapes, I began to identify key features of these landscapes which I continue investigating in more depth. These are a few of them:

Edge horizon: I call edge horizon to the distinctive sudden change of convexity that happens at the entrance of a minima. This change of inclination is of a different degree in different cases and various features of this area may hold insights related to the performance of the network.

I call edge horizon to the distinctive sudden change of convexity that happens at the entrance of a minima. This change of inclination is of a different degree in different cases and various features of this area may hold insights related to the performance of the network. Ruggedness index: the degree of ruggedness vs smoothness of the area around the minimizer. This feature, as happens in the case of batchnorm and dropout, expresses sometimes a connection to the degree of overfitting/generalization of the networks. In what other ways could we create different kinds of ruggedness and smoothness in the landscape? Research continues on this and related topics.

the degree of ruggedness vs smoothness of the area around the minimizer. This feature, as happens in the case of batchnorm and dropout, expresses sometimes a connection to the degree of overfitting/generalization of the networks. In what other ways could we create different kinds of ruggedness and smoothness in the landscape? Research continues on this and related topics. Ruggedness adaptive index: the nature of the rugged terrain in the landscape can vary. It can have a more stable or unstable nature. High learning rates combined with very low batch sizes can produce highly unstable patterns in the landscape. Conversely, dropout, under the right conditions, produces morphological patterns that adapt to the general morphology of the surface.

the nature of the rugged terrain in the landscape can vary. It can have a more stable or unstable nature. High learning rates combined with very low batch sizes can produce highly unstable patterns in the landscape. Conversely, dropout, under the right conditions, produces morphological patterns that adapt to the general morphology of the surface. Downfall: the area that goes from the edge horizon to the bottom of the minima. Research is ongoing to understand this area in more depth.

the area that goes from the edge horizon to the bottom of the minima. Research is ongoing to understand this area in more depth. Noise factor: using small vs large batch sizes, the use of dropout and other strategies introduce more or less noise/instability/randomness into the process. This can be detected visually in the dynamics of the landscape as well as in its morphology.

More of these features and related analyses will be published and shared in more depth in the future.

EDGE HORIZON. Loss Landscape created with data from the training process of a convolutional network. Imagenette Dataset. Sgd-Adam, train mode, 1 million points, log scaled. Graphic by Javier Ideami | losslandscape.com

So, do flatter minima generalize better?

There is active research about this matter and this question has been answered in different ways, sometimes opposing ways, in recent years.

The paper “Sharp Minima Can Generalize For Deep Nets” by Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio, tells us that sharp minima can generalize as well as flat ones. It also states that flat ones can generalize poorly as well on occasions. (note: the paper defines flatness as the size of a region that has roughly constant error).

Other papers describe a different perspective on the matter. For example, the paper “Understanding Generalization through Visualizations” verifies that generalization error correlates well with the flatness of the minimizer. It also verifies that differences between flat and sharp minima are emphasized by the multidimensional nature of these loss landscapes. The paper also concludes that the high dimensionality nature of these landscapes keeps the minimizer away from bad minima (more about this a bit later in this article).

In general, the trend is to state that flat minima generalize better than sharp ones. Intuitively this makes sense, given that flat minima should allow for more deviations from a certain point, while maintaining a similar loss value.

Can we apply these analyses to other deep learning architectures like GANs for example?

Yes, it’s possible to analyze the loss landscape with certain types of GANs. The issue with GANs, traditionally, has been that there is not a clear correlation between the loss value and how well the network is performing. In supervised learning, we know that if loss values are low, the network is performing well. But in GANs, we may be getting great results and yet have loss values all over the place. As the generator and discriminator compete with each other, loss values can have patterns of behaviour that are rather non-intuitive.

To the rescue came a type of GAN called a Wasserstein GAN (with gradient penalty in this case).

In most GANs, the loss value is telling us how well the generator is fooling the discriminator, rather than what the quality or performance of the output result is. So the quality of our output may be improving and yet the loss of the generator may not go down.

So by analyzing the loss value in most GANs it is difficult to ascertain how well we are doing.

On the other hand, with Wasserstein GANs, the loss value communicates the performance and quality of the output result, which therefore makes it possible to build visualizations in which we use the loss values to analyze the network’s performance and other metrics around it.

Below I show a visualization that uses the Celeb A dataset and a Wasserstein GP Gan. As seen in the video, we can track the performance of our generator’s loss landscape, which starts as a flat surface that then begins to evolve as our minimizer converges towards different minima.

We can also sense in the visualization the inestable nature of the training processes of GANs, that often behave like wild beasts that need to be tamed through very careful picking of the parameters and timings of their training processes.

Loss Landscape generated with real data: wasserstein GP Gan, celebA dataset, sgd-adam, bs=64, train mod, 300k pts, 1 w range, latent space dimensions: 200, generator is sometimes reversed for visual purposes, critic is log scaled (orig loss nums) & vis-adapted | losslandscape.com

Can we apply these kinds of visualizations to other areas of deep learning?

Yes, they can be applied to many other areas of deep learning and to other fields.

For example, an area of deep learning where visualizations can help a lot is Geometric deep learning. Geometric deep learning intends to adapt neural networks to non-euclidean domains, which could be manifolds, graphs, etc.

In collaboration with Neural Concept SA, the following video shows the activations of the convolutional layers inside a Geometric CNN, extracting features from the surface of a drone, while the network is being trained to predict aerodynamic properties of the aircraft. We visualize features near the beginning of the network and others near the end of the network.

Real Data visualization of a geometric convnet. L2 Loss, ADAM, BS=1, LR=0.0001 / 24588 Vertices. The video shows the activations of the convolutional layers inside a Geometric CNN extracting features from the surface of a drone, while the network is being trained to predict aerodynamic properties of the aircraft. This project is ongoing and new updates will come later on | losslandscape.com

Another area is Bayesian deep learning. For example, in this collaboration with NYU researchers Pavel Izmailov and Andrew Gordon Wilson, we focus on their SWAG paper NeurIPS 2019:1902.02476: “A simple baseline for Bayesian Uncertainty in Deep Learning” by Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, Andrew Gordon Wilson, a paper related to Bayesian deep learning, in which they take a pre-trained solution produced through SGD, Adam or any other optimizer, and then starting from it, they use a high learning rate and SGD to find a set of solutions.

Then, they build a gaussian distribution that captures the different solutions traversed by the stochastic gradient descent training process. We can then use this distribution as an approximation to the posterior.

This visualization uses real data to plot the optimized part of the loss landscape (using PCA) as well as the positions of the solutions and the related gaussian distribution produced.

UNCERTAIN DESCENT. NeurIPS 2019, ARXIV:1902.02476 / swa-gaussian (swag). a simple baseline for bayesian uncertainty in deep learning | losslandscape.com

UNCERTAIN DESCENT. swa-gaussian (swag). a simple baseline for bayesian uncertainty in deep learning. Based on the paper by wesley maddox, timur garipov, pavel izmailov, dmitry vetrov, andrew gordon wilson. Visualization is a collaboration between pavel izmailov, timur garipov and javier ideami@losslandscape.com. NeurIPS 2019, ARXIV:1902.02476 | losslandscape.com

UNCERTAIN DESCENT. swa-gaussian (swag). a simple baseline for bayesian uncertainty in deep learning. Based on the paper by wesley maddox, timur garipov, pavel izmailov, dmitry vetrov, andrew gordon wilson. Visualization is a collaboration between pavel izmailov, timur garipov and javier ideami@losslandscape.com. NeurIPS 2019, ARXIV:1902.02476 | losslandscape.com

UNCERTAIN DESCENT. swa-gaussian (swag). a simple baseline for bayesian uncertainty in deep learning. Based on the paper by wesley maddox, timur garipov, pavel izmailov, dmitry vetrov, andrew gordon wilson. Visualization is a collaboration between pavel izmailov, timur garipov and javier ideami@losslandscape.com. NeurIPS 2019, ARXIV:1902.02476 | losslandscape.com

These visualizations allow us to study all kinds of surfaces from different perspectives as in the following study that looks at the convexity dynamics from below, as a minimizer progresses towards a minima.

Study on convexity dynamics from a below perspective. Real data. Captured and visualized by Javier Ideami | losslandscape.com

As mentioned earlier, the level of detail in these visualizations can be taken as far as we like. Of course the more data, the more compute time and resources we need. More data allows us to study the morphology of these surfaces in more detail.

Detailed study of the edge horizon of a training process. Real data. Captured and visualized by Javier Ideami | losslandscape.com

The blessing of dimensionality

Many researchers are finding that as deep learning networks increase their dimensionality, finding good minima gets easier, not harder. This connects with what some call “The blessing of dimensionality”.

Recently, Babak Hassibi, who is the inaugural Mose and Lillian S. Bohn Professor of Electrical Engineering and Computing and Mathematical Sciences at the California Institute of Technology, gave a talk titled “Deep learning and the blessing of dimensionality” (joint work with Navid Azizan Ruhi and Sahin Ali Lale).

The opposite of the blessing of dimensionality is what is called “The curse of dimensionality”, which refers to the fact that as dimensionality grows, the number of possible combinations and configurations grows as well, exponentially, whereas the number of those combinations that are covered by our data gets lower. This should theoretically cause different issues and yet, the opposite seems to be happening.

As our networks grow larger in dimensionality, their number of parameters dwarf the number of data points in the system (overparameterized networks). As such, they should overfit the data, not generalize. However, what we find is that they definitely generalize, and the larger the network, the better!

It is common to see cases where we have millions of parameters and thousands of data points, and yet the network won’t overfit.

Babak makes a great analogy in his talk in connection with finding needles in haystacks. In relation to under parameterized systems, we can imagine that we are trying to find a few needles (combinations of weight values that take us to good minima) in the haystack. Conversely, in relation to overparameterized networks, there are infinitely more needles available in the haystack, and that’s why it may be much easier to find them.

When we have 1 million parameters, for example, and 1000 data points, any point from which we begin our training process within the loss landscape, is going to be close to a combination that interpolates our data.

And due to this blessing of dimensionality, convergence becomes an easier challenge to solve, it becomes easy to converge to a low minima within the landscape. On top of that, with a good regularizer we may also get the generalization capability that we want. Furthermore, Babak tells us that there is an implicit regularization capability embedded in our deep learning SGD algorithm.

In summary, extra dimensions facilitate our optimization challenges. In high dimensional networks, the set of interpolating solutions is really large. Because of that, wherever your starting position is on the loss landscape, chances are very high that there is a local path somewhere close to your starting point that will take you to a good minima.

Would it be possible to keep the global loss landscape static and visualize the moving optimizer in action?

Good question, let’s explore it.

In these visualizations, what we are doing could be described as “following the minimizer while examining its nearby surroundings”. This is feasible because we know at all times the current weights of our network, so we can take those as the center / reference point of our low dimensional plane (the one we produced from the random orthogonal directions), and from that point we can explore its surroundings. Those nearby surroundings are constantly changing because the position of our minimizer (calculated from the changing weights of our network) varies as the training progresses (and as we move through weight space), all of this taking place within the non-changing, full high dimensional landscape, a region of which is expressed visually through our low dimensional interpretation.

Let’s now consider the alternative. If we want to fix in place the landscape while the training progresses, and at the same time we try to follow the path of the minimizer around that landscape, we face a challenge.

The landscape we visualize in our representation is a low dimensional interpretation of a portion of the full high dimensional one. To create that representation, that lower dimensional version, we need to make use of a reference point, which happens to be, naturally, the current position of our minimizer, its current state and position in weight space.

So, if we want to represent both the static landscape, and the path followed by the minimizer, we are missing something. What portion of the high dimensional landscape should we show at each step? We would naturally say that we want to show the area near to the current position of the minimizer. Sure, let’s do that. But then, when the minimizer moves, what do we do? How does the trajectory relate to the area of the landscape that we are representing? The key thing to consider is that the minimizer is moving within the high dimensional space, not within the low dimensional interpretation. Therefore, the low dimensional interpretation of the “next” position of the minimizer, may have little to do with the previous one (because the change happens in the high dimensional space). By fixing the landscape in place, we fix a low dimensional representation projected onto a plane that corresponds to a specific state and position in weight space of the minimizer.

We can try to deal with this issue in various ways. The paper mentioned earlier, “Visualizing the Loss Landscape of Neural Nets”, makes use of PCA to analyze the trajectory of the minimizer, but as discussed earlier, PCA directions extract the optimized parts of the landscape. And by doing that, we won’t have a way to match that data with the rest of the landscape with accuracy (in relation to the wider and more diverse areas that surround the minimizer).

If the movement of our minimizer had a very low dimensional nature, we could take a single fixed landscape state and attempt to match a minimizer’s trajectory to it (using PCA), by using projections. However, as Tom Goldstein once told me, the minimizer’s trajectory will not be perfectly planar so we have to project it onto a plane, which means that the contours projected may not be the ones that exist off the plane. Beyond that, PCA limits us in other ways also in relation to the surrounding landscape.

In summary, matching trajectory and landscape with accuracy remains a challenge. On top of that, to combine across the entire process an extensive and wide area of the surrounding landscape and the minimizer’s trajectory is yet another obstacle on top of the previous one.

A core issue here is that we are at all times visualizing and working with low dimensional interpretations of the higher dimensional structure. Let’s consider a variation of our photography analogy.

Say you take a series of pictures (2 dimensional interpretations of the 3d world) of any subject (existing in the higher 3 dimensional space), from various angles. And say that you then try to match them with each other. You could use photogrammetry (https://en.wikipedia.org/wiki/Photogrammetry). Photogrammetry can extract the underlying 3d structure from a combination of 2d sources of data. Could something similar be done with these loss landscapes? The challenge is that photogrammetry makes transformations from a mix of 2d sources to a 3d space. In our case, we are dealing with very high dimensional spaces, spaces with millions or billions of dimensions. There may be a way of doing such a thing in the future, but at the moment it remains an area of research.

In summary, in these visualizations, we are joining the high dimensional ride by attaching ourselves to the minimizer itself. As we ride alongside the minimizer, we make use of a low dimensional 2d plane to analyze and study its surroundings from a certain perspective.

If we didn’t ride on the shoulders of the minimizer, and instead we stayed in place, analyzing a portion of the landscape through the lens of a fixed low dimensional interpretation, the minimizer would then move in the following steps to different parts of the higher dimensional structure, whose low dimensional representation is going to be different.

Unfortunately for us, we cannot visualize those higher dimensional structures, and as we always need to slice them somehow with lower dimensional representations, fixing these low dimensional representations becomes possible if the movement of the minimizer took place in a very constrained number of dimensions. As such things cannot be guaranteed, finding an accurate way to put together those 2 representations (the minimizer’s trajectory and the fixed landscape) becomes a challenge.

The alternative is what we see here, riding safely on the shoulders of the minimizer, while analyzing and studying the changing surroundings as the training process progresses and we move around weight space. And this is enough to potentially give us useful information and insights about the morphology and dynamics of these processes.

When calculating loss values, do we have to iterate through our entire training set?

Ideally yes. But as usual, Pareto often beats perfection. It really depends on your objective. For some objectives, you may iterate on a subset of the entire set as long as you keep using the same subset. For example, the paper “Negative eigenvalues of the Hessian in deep neural networks” states that while studying the eigenvalues of the hessian, using 5% of the training set (and repeating always the use of the same 5%) generated results close enough to those produced by using the complete set.

What’s next?

More related writings and more visualizations and audiovisual creations in the intersection of research and art, as well as collaborations with researchers around the world. These types of collaborations help us push forward the field of deep learning.

Pavel Izmailov at the Broad Institute / Harvard University

Let’s keep going further and deeper into the fascinating world of loss landscapes.

You may find more deep learning visualizations here:

https://www.youtube.com/playlist?list=PLXRmG-SrXaHg4XsOAlZ3zbsYvwYhyXycJ

Contact me on ideami@ideami.com