# An Overview of Open Source Deep Learning-Based Libraries for Neuroscience
## 2 Background
### Bringing Deep Learning to the Neurosciences  
The deep learning community is accustomed to open science, as many datasets, models, programming frameworks, and scientific outcomes are publicly released by both academia and companies continuously. However, while deep learning can openly provide state-of-the-art models for old and new problems in neuroscience, theoretical understanding, formalization, and standardization are often yet to be achieved, which may prevent adoption in other research endeavors. From a technical standpoint, deep networks are a viable tool for many tasks involving data from the brain sciences. Image classification has arguably been the task in which deep neural networks have had the highest momentum in terms of pushing the state of the art forward. This translates now into a rich taxonomy of architectures and pre-trained models that consistently maintain interesting performances in pattern recognition across a number of image domains. Pattern recognition is indeed central for diagnostic purposes, in the form of classification of images with pathological features (e.g., types of brain tumors or meningiomas), segmentation of structures (such as the brain, brain tumors, or stroke lesions), classification of signals (e.g., classification of electromyography or electro encephalopathy data), as well as for action recognition in Human-Computer Interfaces (HCIs) and Brain-Computer Interfaces (BCIs), where the complex systems underlying human behavior and mind must be interpreted, processed and used by artificial systems (see [32] for a larger review of BCIs). The initiatives BRain Tumor Segmentation (BRATS) Challenge ([[https://www.med.upenn.edu/cbica/brats/](https://www.med.upenn.edu/cbica/brats/)]([https://www.med.upenn.edu/cbica/brats/](https://www.med.upenn.edu/cbica/brats/))) [33], Ischemic Stroke LEsion Segmentation (ISLES) Challenge ([[https://www.isles-challenge.org/](https://www.isles-challenge.org/)]([https://www.isles-challenge.org/](https://www.isles-challenge.org/))) [34; 35], and Ninapro ([[http://ninaweb.hevs.ch/node/7](http://ninaweb.hevs.ch/node/7)]([http://ninaweb.hevs.ch/node/7](http://ninaweb.hevs.ch/node/7))) [36] are examples of data releases for which above-mentioned tools proved effective. There are models learning image-to-image functions capable of enhancing data, preprocessing it, correcting artifacts and aberrations, allowing smart compression as well as super-resolution, and even expressing cross-modal transformations between different acquisition apparatuses. In the related tasks of object tracking, action recognition, and pose estimation, research results from the automotive sector or crowd analysis have inspired solutions for behavioral neuroscience, especially in animal behavioral studies. When dealing with sequences, deep networks success in computer vision has inspired CNN-based approaches to EEG and EMG studies [37; 38], either with or without relying on 2D data, given that mathematical convolution has a 1D version, and 1D signals have 2D spectra. Other architectures more directly instantiate temporal and sequential aspects, e.g., Recurrent Neural Networks (RNNs) such as the Long Short Term Memory (LSTM) [39] and Gated Recurrent Units (GRUs) [40], and they too can be applied to sequence problems and sub-tasks in neuroscience, such as decoding time-dependent brain signals. Although deep neural networks do not explicitly model the nervous system, they are inspired by biological knowledge and mimic some aspects of biological computation and dynamical systems. This has inspired new comparative studies and analogous approaches to learning and perception in a unique way among machine learning algorithms [41]. Many neuroinformatic studies demonstrate how novel deep learning concepts and methods apply to neurological data [12]. However, they often showcase new advanced achievements in performance metrics that do not translate directly to new accepted neuroscience discoveries or clinical best practices.