Towards understanding deep learning with the natural clustering prior

The prior knowledge (a.k.a. priors) integrated into the design of a machine learning system strongly influences its generalization abilities. In the specific context of deep learning, some of these priors are poorly understood as they implicitly emerge from the successful heuristics and tentative approximations of biological brains involved in deep learning design. Through the lens of supervised image classification problems, this thesis investigates the implicit integration of a natural clustering prior composed of three statements: (i) natural images exhibit a rich clustered structure, (ii) image classes are composed of multiple clusters and (iii) each cluster contains examples from a single class. The decomposition of classes into multiple clusters implies that supervised deep learning systems could benefit from unsupervised clustering to define appropriate decision boundaries. Hence, this thesis attempts to identify implicit clustering abilities, mechanisms and hyperparameters in deep learning systems and evaluate their relevance for explaining the generalization abilities of these systems. Our study of implicit clustering abilities exploits hierarchical class labels to show that the subclasses (e.g., orchids, poppies, roses, sunflowers, tulips) associated to a class (e.g., flowers) are differentiated in deep neural networks that generalize well, even though only class-level supervision is provided. We then look for clustering mechanisms through the study of neuron-level training dynamics in multilayer perceptrons trained on a synthetic dataset with known clusters. Our experiments reveal a winner-take-most mechanism: training progressively increases the average pre-activation of the most activated clusters of a class and decreases the average pre-activation of the least activated clusters of the same class. Remarkably, this implicit mechanism leads neurons to differentiate some clusters from the same class more strongly than clusters from different classes. These studies indicate the emergence of a neuron-level training process that is critical for implicit clustering to occur. We propose to capture the extent by which the neurons of each layer have been effectively “trained” during the global training process through the amount of layer rotation, i.e. the cosine distance between the initial and final flattened weight vectors of each layer. Equipped with tools to monitor and control the amount of layer rotation during training, we demonstrate that this implicit hyperparameter exhibits a consistent relationship with model generalization and training speed. Moreover, we show that the impact of layer rotation on training seems to explain the effect of several explicit hyperparameters such as the learning rate, weight decay, and the use of adaptive gradient methods. Overall, our work thus provides a collection of experiments to support the relevance of the natural clustering prior for explaining generalization in deep learning. Additionally, it highlights the potential of using explicit clustering algorithms for training deep neural networks, as this would facilitate the integration of natural clustering-related priors into the design of deep learning systems.