user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: Critical Percolation as a Framework to Analyze the Training of Deep Networks

Paper content:
In this paper we approach two relevant deep learning topics:
i) tackling of graph structured input data and
ii) a better understanding and analysis of deep networks and related learning algorithms.
With this in mind we focus on the topological classification of reachability in a particular subset of planar graphs (Mazes).
Doing so, we are able to model the topology of data while staying in Euclidean space, thus allowing its processing with standard CNN architectures.
We suggest a suitable architecture for this problem and show that it can express a perfect solution to the classification task.
The shape of the cost function around this solution is also derived and, remarkably, does not depend on the size of the maze in the large maze limit.
Responsible for this behavior are rare events in the dataset which strongly regulate the shape of the cost function near this global minimum.
We further identify an obstacle to learning in the form of poorly performing local minima in which the network chooses to ignore some of the inputs.
We further support our claims with training experiments and numerical analysis of the cost function on networks with up to $128$ layers.
Deep convolutional networks have achieved great success in the last years by presenting human and super-human performance on many machine learning problems such as image classification, speech recognition and natural language processing ).
Importantly, the data in these common tasks presents particular statistical properties and it normally rests on regular lattices (e.g. images) in Euclidean space BID3 ).
Recently, more attention has been given to other highly relevant problems in which the input data belongs to non-Euclidean spaces.
Such kind of data may present a graph structure when it represents, for instance, social networks, knowledge bases, brain activity, protein-interaction, 3D shapes and human body poses.
Although some works found in the literature propose methods and network architectures specifically tailored to tackle graph-like input data BID3 ; BID4 ; BID15 ; BID22 ; BID23 b) ), in comparison with other topics in the field this one is still not vastly investigated.Another recent focus of interest of the machine learning community is in the detailed analysis of the functioning of deep networks and related algorithms BID8 ; BID12 ).
The minimization of high dimensional non-convex loss function by means of stochastic gradient descent techniques is theoretically unlikely, however the successful practical achievements suggest the contrary.
The hypothesis that very deep neural nets do not suffer from local minima BID9 ) is not completely proven BID36 ).
The already classical adversarial examples BID27 ), as well as new doubts about supposedly well understood questions, such as generalization BID43 ), bring even more relevance to a better understanding of the methods.In the present work we aim to advance simultaneously in the two directions described above.
To accomplish this goal we focus on the topological classification of graphs BID29 ; BID30 ).
However, we restrict our attention to a particular subset of planar graphs constrained by a regular lattice.
The reason for that is threefold:
i) doing so we still touch upon the issue of real world graph structured data, such as the 2D pose of a human body BID1 ; BID16 ) or road networks BID25 ; BID39 );
ii) we maintain the data in Euclidean space, allowing its processing with standard CNN architectures;
iii) this particular class of graphs has various non-trivial statistical properties derived from percolation theory and conformal field theories BID5 ; BID20 ; BID34 ), allowing us to analytically compute various properties of a deep CNN proposed by the authors to tackle the problem.Specifically, we introduce Maze-testing, a specialized version of the reachability problem in graphs BID42 ).
In Maze-testing, random mazes, defined as L by L binary images, are classified as solvable or unsolvable according to the existence of a path between given starting and ending points in the maze (vertices in the planar graph).
Other recent works approach maze problems without framing them as graphs BID37 ; BID28 ; BID33 ).
However, to do so with mazes (and maps) is a common practice in graph theory BID2 ; BID32 ) and in applied areas, such as robotics BID11 ; BID7 ).
Our Mazetesting problem enjoys a high degree of analytical tractability, thereby allowing us to gain important theoretical insights regarding the learning process.
We propose a deep network to tackle the problem that consists of O(L 2 ) layers, alternating convolutional, sigmoid, and skip operations, followed at the end by a logistic regression function.
We prove that such a network can express an exact solution to this problem which we call the optimal-BFS (breadth-first search) minimum.
We derive the shape of the cost function around this minimum.
Quite surprisingly, we find that gradients around the minimum do not scale with L. This peculiar effect is attributed to rare events in the data.In addition, we shed light on a type of sub-optimal local minima in the cost function which we dub "neglect minima".
Such minima occur when the network discards some important features of the data samples, and instead develops a sub-optimal strategy based on the remaining features.
Minima similar in nature to the above optimal-BFS and neglect minima are shown to occur in numerical training and dominate the training dynamics.
Despite the fact the Maze-testing is a toy problem, we believe that its fundamental properties can be observed in real problems, as is frequently the case in natural phenomena BID31 ), making the presented analytical analysis of broader relevance.Additionally important, our framework also relates to neural network architectures with augmented memory, such as Neural Turing Machines BID13 ) and memory networks BID40 ; BID35 ).
The hot-spot images FIG9 , used to track the state of our graph search algorithm, may be seen as an external memory.
Therefore, to observe how activations spread from the starting to the ending point in the hot-spot images, and to analyze errors and the landscape of the cost function (Sec. 5) , is analogous to analyze how errors occur in the memory of the aforementioned architectures.
This connection gets even stronger when such memory architectures are employed over graph structured data, to perform task such as natural language reasoning and graph search ; BID17 ; BID14 ).
In these cases, it can be considered that their memories in fact encode graphs, as it happens in our framework.
Thus, the present analysis may eventually help towards a better understanding of the cost functions of memory architectures, potentially leading to improvements of their weight initialization and optimization algorithms thereby facilitating training BID26 ).The
paper is organized as follows: Sec. 2 describes in detail the Maze-testing problem. In
Sec. 3 we suggest an appropriate architecture for the problem. In
Sec. 4 we describe an optimal set of weights for the proposed architecture and prove that it solves the problem exactly. In
Sec. 5 we report on training experiments and describe the observed training phenomena. In
Sec. 6 we provide an analytical understanding of the observed training phenomena. Finally
, we conclude with a discussion and an outlook.
Despite their black-box reputation, in this work we were able to shed some light on how a particular deep CNN architecture learns to classify topological properties of graph structured data.
Instead of focusing our attention on general graphs, which would correspond to data in non-Euclidean spaces, we restricted ourselves to planar graphs over regular lattices, which are still capable of modelling real world problems while being suitable to CNN architectures.We described a toy problem of this type (Maze-testing) and showed that a simple CNN architecture can express an exact solution to this problem.
Our main contribution was an asymptotic analysis of the cost function landscape near two types of minima which the network typically settles into: BFS type minima which effectively executes a breadth-first search algorithm and poorly performing minima in which important features of the input are neglected.Quite surprisingly, we found that near the BFS type minima gradients do not scale with L, the maze size.
This implies that global optimization approaches can find such minima in an average time that does not increase with L. Such very moderate gradients are the result of an essential singularity in the cost function around the exact solution.
This singularity in turn arises from rare statistical events in the data which act as early precursors to failure of the neural network thereby preventing a sharp and abrupt increase in the cost function.In addition we identified an obstacle to learning whose severity scales with L which we called neglect minima.
These are poorly performing minima in which the network neglects some important features relevant for predicting the label.
We conjectured that these occur since the gradual incorporation of these important features in the prediction requires some period in the training process in which predictions become more noisy.
A "wall of noise" then keeps the network in a poorly performing state.It would be interesting to study how well the results and lessons learned here generalize to other tasks which require very deep architectures.
These include the importance of rare-events, the essential singularities in the cost function, the localized nature of malfunctions (bugs), and neglect minima stabilized by walls of noise.These conjectures potentially could be tested analytically, using other toy models as well as on real world problems, such as basic graph algorithms (e.g. shortest-path) BID14 ); textual reasoning on the bAbI dataset ), which can be modelled as a graph; and primitive operations in "memory" architectures (e.g. copy and sorting) BID13 ).
More specifically the importance of rare-events can be analyzed by studying the statistics of errors on the dataset as it is perturbed away from a numerically obtained minimum.
Technically one should test whether the perturbation induces an typical small deviation of the prediction on most samples in the dataset or rather a strong deviation on just a few samples.
Bugs can be similarly identified by comparing the activations of the network on the numerically obtained minimum and on some small perturbation to that minimum while again looking at typical versus extreme deviations.
Such an analysis can potentially lead to safer and more robust designs were the network fails typically and mildly rather than rarely and strongly.Turning to partial neglect minima these can be identified provided one has some prior knowledge on the relevant features in the dataset.
The correlations or mutual information between these features and the activations at the final layer can then be studied to detect any sign of neglect.
If problems involving neglect are discovered it may be beneficial to add extra terms to the cost function which encourage more mutual information between these neglected features and the labels thereby overcoming the noise barrier and pushing the training dynamics away from such neglect minimum.

assistant: text: A toy dataset based on critical percolation in a planar graph provides an analytical window to the training dynamics of deep neural networks