SGD: Deep Learning’s Foundational Steps. Gradient Descent is a common optimization strategy in machine learning and deep learning, and it may be used for nearly all learning algorithms.