Context Dependent Modulation of Activation Function
We propose a modification to traditional Artificial Neural Networks, which provides the ANNs with new aptitudes motivated by biological neurons. Biological neurons work far beyond linearly summing up synaptic inputs and then transforming the integrated information. A biological neuron change firing modes accordingly to peripheral factors as well as intrinsic ones. Our modification connects a new type of ANN nodes, which mimic the function of biological neuromodulators and are termed modulators, to enable other traditional ANN nodes to adjust their activation sensitivities in run-time based on their input patterns. In this manner, we enable the slope of the activation function to be context dependent. This modification produces statistically significant improvements in comparison with traditional ANN nodes in the context of Convolutional Neural Networks and Long Short-Term Memory networks.
We propose a modification to traditional Artificial Neural Networks motivated by the biology of neurons to enable the shape of the activation function to be context dependent.A method to scale the activations of a layer of neurons in an ANN depending on inputs to that layer that reports improvements above the baselines.Introduction of an architectural change for basic neurons in a neural network, and the idea to multiply neuron linear combination output by a modulator prior to feeding it into the activation function.