Stabilizing and Enhancing Learning for Deep Complex and Real Neural Networks

Dans cette these nous proposons un ensemble de contributions originales sous la forme de trois articles relatifs aux reseaux de neurones profonds reels et complexes. Nous abordons a la fois des problemes theoriques et pratiques lies a leur apprentissage. Les trois articlestraitent des methodes concues pour apporter des solutions aux problemes de l’instabilite observee au cours de l’entrainement des reseaux, notamment le probleme notoire de dilution et d’explosion des gradients ou «vanishing and exploding gradients » lors de l’entrainement des reseaux de neurones profonds. Nous proposons dans un premier temps la conception de modules d’entrainement appropries, designes par «building blocks», pour les reseaux de neurones profonds a valeurs complexes. Notre proposition comporte des methodes d’initialisation et de normalisation ainsi que des fonctions d’activation des unites neuronales. Les modules concus sont par la suite utilises pour la specification d’architectures profondes a valeurs complexes dediees a accomplir diversestâches. Ceci comprend des tâches de vision par ordinateur, de transcription musicale, de prediction du spectre de la parole, d’extraction des signaux et de separation des sources audio. Finalement nous procedons a une analyse detaillee de l’utilite de l’hypothese contraignanted’orthogonalite generalement adoptee pour le parametrage de la matrice de transition a travers les couches des reseaux de neurones reels recurrents.----------ABSTRACT : This thesis presents a set of original contributions in the form of three chapters on real and complex-valued deep neural networks. We address both theoretical issues and practicalchallenges related to the training of both real and complex-valued neural networks. First, we investigate the design of appropriate building blocks for deep complex-valued neural networks, such as initialization methods, normalization techniques and elementwise activation functions. We apply our theoretical insights to design building blocks for the construction of deep complex-valued architectures. We use them to perform various tasks in computer vision, music transcription, speech spectrum prediction, signal retrieval and audio source separation. We also perform an analysis of the usefulness of orthogonality for the hidden transition matrix in a real-valued recurrent neural network. Each of the three chapters are dedicated to dealing with methods designed to provide solutions to problems causing traininginstability, among them, the notorious problem of vanishing and exploding gradients during the training of deep neural networks. Throughout this manuscript we show the usefulness of the methods we propose in the context of well known challenges and clearly identifiable objectives. We provide below a summary of the contributions within each chapter.At present, the vast majority of building blocks, techniques, and architectures for training deep neural networks are based on real-valued computations and representations. However, representations based on complex numbers have started to receive increased attention. Despitetheir compelling properties complex-valued deep neural networks have been neglected due in part to the absence of the building blocks required to design and train this type of network. The lack of such a framework represents a noticeable gap in deep learning tooling.