# An Overview of Open Source Deep Learning-Based Libraries for Neuroscience
## 2 Background  
### Deep Learning  
Deep learning has contributed many of the best solutions to problems in its parent field, machine learning, thanks to its theoretical and technological achievements that unlocked its intrinsic versatility. Machine learning is the study of computer algorithms that tackle problems without complete access to predefined rules or analytical, closed-form solutions. The algorithms often require a training phase to adjust parameters and satisfy internal or external constraints (e.g., of exactness, approximation, or generality) on dedicated data for which solutions might be already known. Machine learning comprises a wide array of statistical and mathematical methods, including Artificial Neural Networks (ANNs), biologically inspired systems that connect inputs and outputs through simple computing units (neurons), which act as function approximators. Each unit implements a nonlinear function of the weighted sum of its inputs; thus, the output of the whole ANN is a composite function, as formally intended in mathematics. The networks of neurons are most often layered and \"feed-forward\", meaning that units from any layer only output results to units in subsequent layers. The width of a layer refers to its neuron count, while the depth of a network refers to its layer count. The typical architecture instantiating the above characteristics is the MultiLayer Perceptron [1] (MLP). Universal approximation theorems [2; 3] ensure that whenever a nonlinear network, such as the MLP, is either bound in width and unbound in depth or vice versa, its weights can then be set to represent virtually any function (i.e., a wide variety of function families). The training problem thus consists of building networks with sets of weights so to instantiate or approximate the function that would solve the assigned task or that represents the input-output relation. This search is not trivial: it can be framed as the optimization problem for a function over the ANN weights. Such functions, typically called \"loss function\", associates the \"errors\" made on the training data to the neural net parameters (its weights), acting as a total performance score. Approaching local minima of the loss function and improving the network performance on the training data is the prerequisite to generalizing on real-world and unseen data. Deep learning is concerned with the use of deep ANNs, namely characterized by depth, stacking several intermediate (hidden) layers between input and output units. As mentioned above, with other dimensions being equal, the depth increases the representational power of ANNs and, more specifically, aims at modeling complicated functions as meaningful compositions of simpler ones. As with their biological counterparts [4], depth is supposed to manage hierarchies of features from larger input portions, capturing characteristics often inherent to real-world objects and effective in modeling actual data. Overall, depth is one of the key features that allowed us to overcome historical limits [5] of simpler ANNs such as the Perceptron. At the same time, depth comes with numerical and methodological hardships in model training. Part of the difficulties arises as the search space for the optimal set of parameters grows considerably with the number of layers (and their width as well). Other issues are strictly numerical since the training algorithms include long computation chains that may affect the stability of training and learning. Hence, new or rediscovered ideas in training protocols and mathematical optimization (e.g., applying the \"backpropagation of errors\" algorithm to neural nets [6]) played an important role through times when the scientific interest and hopes in ANNs faded (so-called \"AI winters\"), paving the way for later advancement. The main drivers for the latest success of deep neural networks are of varied nature and can be schematized as technical and human-related factors.