Neuron is trainable but there seems to be a bug. Starting from sample 53 in mnist training set, it goes wrong. The output rises quickly and overflows.