Before revision:  For vanilla SGD training, the choice ReLU  (0) = 0 seems to be the most efficient.
After revision:  For vanilla SGD training, the choice ReLU ' (0) = 0 seems to be the most efficient.