Some of the key aspects that we should monitor during training of a neural network are – Loss function Validation / training accuracy Ratio of weights : updates Activation / gradient distributions per layer First layer visualizations The first quantity that is useful to track during training is the loss, as it is evaluated on the individual batches during the forward pass. Below is a diagram showing the loss over time, and especially what the shape might tell us about the learning rate. The second important quantity to track while training a classifier is the validation/training accuracy. This plot can give us valuable insights into the amount of overfitting in our model: The last quantity we might want to track is the ratio of the update magnitudes to the value magnitudes. Note: updates, not the raw gradients (e.g. in vanilla sgd this would be the gradient multiplied by the learning rate). We might want to evaluate and track this ratio for every set of parameters independently. A rough heuristic is that this ratio should be somewhere around 1e-3. If it is lower than this then the learning rate might be too low. If it is higher then the learning rate is likely too high. Here is a specific example: