Package: elektronn2
About ELEKTRONN2IntroductionWhat is ELEKTRONN2?Use casesInstallationDesign principlesFeaturesOperationsLoss functionsOptimisersTrainerTraining Examples for CNNsUtilitiesDocumentation and usage examplesContributorsLicenseIntroductionWhat is ELEKTRONN2?ELEKTRONN2 is a flexible and extensiblePython toolkitthat facilitatesdesign,training and application of neural networks.It can be used for general machine learning tasks, but its main focus is onconvolutional neural networks (CNNs) for high-throughput 3D and 2D image analysis.ELEKTRONN2 is especially useful for efficiently assessing experimental
neural network architectures thanks to itspowerful interactive shellthat can be
entered at any time during training, temporarily pausing all calculations.The shell interface provides shortcuts and autocompletions for
frequently used operations (e.g. adjusting the learning rate)
and also provides a complete python shell with full read/write access to the network model, the
plotting subsystem and all training parameters and hyperparameters.
Changes made in the shell take effect immediately, so you canmonitor, analyse andmanipulate your training sessions directly during their run time, without losing
any training progress.Computationally expensive calculations areautomatically compiled and transparentlyexecuted as highly-optimized CUDAbinaries on your GPU if a CUDA-compatible
graphics card is available[1].ELEKTRONN2 is written in Python (2.7 / and 3.4+) and is a complete rewrite of the
previously publishedELEKTRONNlibrary. The largest
improvement is the development of afunctional interface that allows easy creation ofcomplex data-flow graphswith loops between arbitrary points in contrast to
simple “chain”-like models.
Currently, the only supported platform is Linux (x86_64).[1]You can find out if your graphics card is compatiblehere.
Usage on systems without CUDA is possible but generally not recommended
because it is very slow.NoteELEKTRONN2 is being superceded by the more flexible, PyTorch-basedelektronn3library. elektronn3
is actively developed and supported, so we encourage you to use it instead
of ELEKTRONN2 (if elektronn3’s more experimental status and currently less
extensive documentation are acceptable for you).Use casesAlthough other high-level libraries are available (Keras, Lasagne), they all
lacked desiredfeaturesand flexibility for our work,
mostly in terms of an intuitive method to specify complicated computational
graphs and utilities for training and data handling, especially in the domain
of specialised large-scale 3-dimensional image data analysis forconnectomicsresearch
(e.g. tracing, mapping and segmenting neurons in in SBEM[2]data sets).Although the mentioned use cases are ELEKTRONN2’s specialty, it can be used and
extended for a wide range of other tasks thanks to its modular object-oriented
APIdesign(for example, new operations can be implemented as subclasses of theNodeclass and seamlessly integrated into neural network models).[2]Serial Block-Face Scanning Electron Microscopy,
a method to generate high resolution 3D images from small samples, such as
brain tissue.Example visualisation of ELEKTRONN2’s usage on a 3D SBEM data set
(blending input to output from left to right).Left (input: raw data)Right (output: predictions by ELEKTRONN2, color-coded)3D electron microscopy images of a zebra finch
brain (area X dataset j0126 by Jörgen Kornfeld).Probability of barriers (union of cell boundaries and
extracellular space, marked inwhite) and mitochondria
(marked inblue) predicted by ELEKTRONN2.InstallationSee the installation instructions athttps://elektronn2.readthedocs.io/en/latest/installation.html.Design principlesELEKTRONN2 adds another abstraction layer to Theano. To create a model, the
user has to connect different types of node objects and thereby builds a graph
as with Theano. But the creation of the raw Theano graph, composed of symbolic
variables and trainable model parameters, is hidden and managed through usage
of sensible default values and bundling of stereotypical Theano operations into
a single ELEKTRONN2 node.
For example, creating a convolution layer consists of initialising weights,
performing the convolution, adding the bias, applying the activation function
and optional operations such as dropout or batch normalisation. Involved
parameters might be trainable (e.g. convolution weights) or non-trainable but
changeable during training (e.g. dropout rates).FeaturesOperationsPerceptron / fully-connected / dot-product layer, works for arbitrary
dimensional inputConvolution, 1-,2-,3-dimensionalMax/Average Pooling, 1,2,3-dimensionalUpConv, 1,2,3-dimensionalMax Fragment Pooling (MFP), 1,2,3-dimensionalGated Recurrent Unit (GRU) and Long Short Term Memory (LSTM) unitRecurrence / Scan over arbitrary sub-graph: support of multiple inputs
multiple outputs and feed-back of multiple values per iterationBatch normalisation with automatic accumulation of whole data set statistics
during trainingGaussian noise layer (for Variational Auto Encoders)Activation functions: tanh, sigmoid, relu, prelu, abs, softplus, maxout,
softmax-layerLocal Response Normalisation (LRN), feature-wise or spatiallyBasic operations such as concatenation, slicing, cropping, or element-wise
functionsLoss functionsBernoulli / Multinoulli negative log likelihoodGaussian negative log likelihoodSquared Deviation Loss, (margin optional)Absolute Deviation Loss, (margin optional)Weighted sum of losses for multi-task trainingOptimisersStochastic Gradient Descent (SGD)AdaGradAdaDeltaAdamTrainerAutomatic creation of training directory to which all files (parameters,
log files, previews etc.) will be savedFrequent printing and logging of current state, iteration speed etc.Frequent plotting of monitored states (error samples on training and
validation data, classification errors and custom monitoring targets)Frequent saving of intermediate parameter states and history of monitored
variablesFrequent preview prediction images for CNN trainingCustomisable schedules for non-trainable meta-parameters (e.g. dropout rates,
learning rate, momentum)Fully functional python command line during training, usable for
debugging/inspection (e.g. of inputs, gradient statistics) or for changing
meta-parametersTraining Examples for CNNsRandomised patch extraction from a list of of input/target image pairsData augmentation trough histogram distortions, rotation, shear, stretch,
reflection and perspective distortionReal-time data augmentation through a queue with background threads.UtilitiesArray interface forKNOSSOSdata sets with
caching, pre-fetching and support for multiple data sets as channel axis.Viewer for multichannel 3-dimensional image arrays within the Python runtimeFunction to convert ID images to boundary imagesUtilities needed for skeltonisation agent training and applicationVisualisation of the computational graphClass for profiling within loopsKD Tree that supports append (realised through mixture of KD-Tree and
brute-force search and amortised rebuilds)Daemon script for the synchronised start of experiments on several hosts,
based on resource occupation.Documentation and usage examplesThe documentation is hosted athttps://elektronn2.readthedocs.io/(built automatically from the sources in thedocs/subdirectory of the
code repository).ContributorsMarius Killinger(main developer)Martin DrawitschPhilipp SchubertELEKTRONN2 was funded byWinfried Denk’s labat the Max Planck Institute of Neurobiology.Jörgen Kornfeldwas academic advisor to this project.LicenseELEKTRONN2 is published under the terms of the GPLv3 license.
More details can be found in theLICENSE.txtfile.