On Deep Multiscale Recurrent Neural Networks

Deep learning is a study of deep artificial neural networks that has led to several breakthroughs in many machine learning applications. In this thesis, a subgroup of deep learning models, known as recurrent neural networks is studied in depth. Recurrent neural networks are special types of artificial neural networks that possess more strength in modelling temporal structures of sequential data such as text and speech. Recurrent neural networks are used as the core module of many practical applications including speech recognition, text-to-speech, machine translation, machine comprehension, and question and answering. However, our understanding of recurrent neural networks is still limited, and some inherent problems with recurrent neural networks remain unresolved. This thesis includes a series of studies towards deep multiscale recurrent neural networks and novel architectures to overcome the inherent problems of recurrent neural networks. In the first article, we introduce a deep recurrent neural network that can adaptively control the connectivity patterns between layers at consecutive time steps. The recurrent connections between time steps are not only restricted to self-connections as the conventional recurrent neural networks do, but a higher-level layer can connect to the lower-level layers, and vice-versa. A set of parametrized scalar gating units is learned in order to open or close the connections that carry the feedback from the layers at the previous time step. We investigate how the top-down information can be useful for modelling temporal dependencies. In the second article, we study a neural machine translation system that exploits a character-level decoder. The motivation behind this work is to answer a fundamental question: can we generate a character sequence as translation instead of a sequence of words? In order to answer this question, we design a naive two-level recurrent neural network and a more advanced type of recurrent neural network that tries to capture faster and slower components separately with its layers. This proposed model is based on an idea of modelling time dependencies with multiple components that update with different timescales. In the third article, we investigate a framework that can discover the latent hierarchical structure in sequences with recurrent neural networks. The proposed framework introduces a set of boundary detecting units that are used to detect terminations of meaningful chunks. The recurrent neural network updates each hidden layer with different timescales based on the binary states of these boundary detecting units. The inclusion of the boundary detectors enables us to implement a novel update mechanism using three types of different operations. Each layer of the recurrent neural network can choose either to completely copy the dynamic state, to update the state or to flush the state to the upper-level layer and reset the context. Finally, in the fourth article, we study an inclusion of latent variables to recurrent neural networks. The complexity and high signal-to-noise ratio of sequential data such as speech make it difficult to learn meaningful structures from the data. We propose a recurrent extension of the variational auto-encoder in order to introduce high-level latent variables to recurrent neural networks and show performance improvements on sequences modelling tasks such as human speech signals and handwriting examples.

Keyphrases: Deep learning, Neural networks, Recurrent neural networks, Hierarchical recurrent neural networks, Multiscale recurrent neural networks, Language modelling, Machine translation, Speech generation, Handwriting generation, Variational auto-encoders