Title: Difference back propagation with inverse sigmoid function
Primary area: optimization
Keywords: Machine Learning, AI, Algorithm, Back Propagation

Abstract: Since the proposal of neural network, the derivative-based back propagation algorithm has been the default setting. However, the derivative for a non-linear function is an approximation for the difference of the function values, and it would be a more precise way to do back propagation using the difference directly instead of the derivative. While the back propagation algorithm has been the rule-of-thumb for neural networks, it becomes one of the bottleneck in modern large deep learning models. With the explosion of big data and large-scale deep learning models, a tiny change in the back propagation could lead to a huge difference. Here we propose a new back propagation algorithm based on inverse sigmoid function to calculate the difference instead of derivative, and verified the effectiveness with basic examples.

TL;DR: We propose a new back propagation algorithm that calculates the back propagatiion updates using the difference instead of the derivative from the activation function