Evolutionary regularization of deep neural networks

Se propone una estrategia innovadora de regularizacion para redes neuronales basada en la nocion de que, al quitar conexiones entre neuronas en la topologia de una red neuronal, no solo se obtendra una estructura menos compleja sino que tambien tendra mejores capacidades de generalizacion que si fuera una estructura totalmente conexa. Esta estructura parcial se generara mediante un algoritmo de programacion genetica guiada por gramatica que buscara en el conjunto de las topologias validas hasta encontrar una adecuada. Para realizar esta tarea, se ha desarrollado una metagramatica que, dada una estructura maxima, esta metagramatica produzca subestructuras correctas que formen redes neuronales parcialmente conexas contenidas en la superestructura definida. Una vez instanciada, la gramatica producira arboles de derivacion que contienen diferentes caminos desde una neurona de la capa de entrada a una neurona de la capa de salida pasando por una neurona de cada capa oculta. La topologia final se obiene finalmente superponiendo todos los caminos usando unicamente una conexion en aquellos casos en los que haya repeticiones en la coleccion de caminos definidos por el arbol de derivacion. El objetivo de desarrollar esta estrategia es proponer una alternativa a la tecnica de dropout que, no solo reduzca el numero de parametros a estimar en una red, sino que tampoco requiera de ajustar hiperparametros. Para probar esta tecnica, se entrenara una red neuronal sobredimensionada para el conjunto de datos para confirmar que hay over- fitting, despues se aplicara regularizacion para observar los resultados y finalmente se aplicara nuestra estrategia para comparar los resultados con los obtenidos en los pasos previos.---ABSTRACT---We propose a novel strategy of performing regression for deep neural networks based on the notion that by dropping connections in the topology of the net not only will the structure be less complex but it will also generalize better than if it were fully connected. This partially connected structure will be created by a grammar-guided genetic algorithm which will search through different topologies until a suitable one is found. To this end, a metagrammar has been developed so that given a maximum possible structure, the grammar will produce substructures forming partiallyconnected neural networks that are contained in said structure. The grammar produces derivation trees that contain different paths from a neuron in the input layer towards a neuron in the output layer going through exactly one neuron on each of the hidden layer. The final topology is then obtained by overlaying all paths using only one connection in case they are repeated in the path collection defined by each derivation tree. The goal of developing this strategy is to propose an alternative to the standard regularization techniques such as dropout that not only reduces the number of parameters in the net but it also doesn't require the tunning of hyperparameters. To test this approach, an overgrown neural network will be trained for a specific dataset to confirm that there is overfitting, then a dropout technique will be used to avoid said overfitting and finally our strategy will be applied to the overgrown net to compare the results with those obtained via dropout.