Research Project Proposal: Structured Learning

In recent years, deep learning models outperformed state-of-the-art techniques in many machine learning tasks, bringing ground-shaking advances in research and industrial environments. Deep learning models involve powerful function approximators to be trained over huge sets of data. As an example, the review [21] collects deep learning advances in common computer vision tasks, such as motion tracking, action recognition, human pose estimation, and semantic segmentation. Neural networks are a computational tool that computer scientists know since the middle of the last century. They implement a "threshold logic", loosely inspired by biological neurons of the human brain. Despite their early invention, deep learning based on neural networks became popular only since 2012, when AlexNet [12], a deep convolutional neural network, won several computer vision competitions. This advance was due to the great improvements in GPU design, which sped significantly the training time of the networks. Deep learning techniques are heavily used today in research and industry because they build data-driven features from data. Before deep learning, features were hand-crafted by domain experts. This approach was applicable up to a satisfactory rate of success only in some areas and not feasible in others. The ability of building meaningful features in an automated way is extremely powerful, as we do not need to rely on domain knowledge and we can discover meaningful patterns without any human supervision. Even if the literature is overflowing with incredibly successful applications of deep learning architectures based on neural networks, they come with some structural drawbacks. The first one is explainability [7]: it is very difficult to delve into their complex structure and reconstruct the reasoning, in a human-understandable sense, that brings the network to a specific decision. Explainability is mandatory in large-scale, safety-critical and mission-critical applications, as requirements impose hard constraints on model verifiability. The second drawback is the size of training set necessary for the learning procedure, called backpropagation. Since deep neural networks have a huge set of parameters, their tuning requires a high cost in terms of data, computational time, and memory. Recently, many researchers started experimenting with embedding algorithmic priors directly in the neural network structure. Their aim is to insert algorithmic modules with a predefined behavior in the architecture while maintaining the end-to-end training procedure. This approach increases explainability, as algorithmic priors force the network to adapt to the algorithms’ interfaces. It also helps training, because it reduces the variance of the model by lowering the number of tunable parameters. Finally, it provides a stabler solution during training, being it theoretically guided. This emerging field is called Structured Learning: algorithms provide structure and neural networks provide flexibility to learn and adapt from data. The backpropagation algorithm, in order to train a structured system end-to-end, requires all the architecture components to be differentiable. End-to-end training is a desirable property, as it requires a single input-output dataset. A non-end-to-end training process would require separate tuning of each architectural component. This approach may be costly or not feasible in general. So, in order to embed an algorithm in a deep neural architecture and maintain end-to-end learning, it must be differentiable. This property can be achieved by using automatic differentiation, a technique common in fluid dynamics or engineering design optimization, but still underused in the machine learning area. Many experiments show the potential of Structured Learning in deep learning architectures and compiler experts are moving towards providing automatic differentiation as a default feature of programming languages.