A Guide to Neural Network Loss Functions with Applications in Keras | by Andre Ye | Towards Data Science

Loss functions are an essential part in training a neural network — selecting the right loss function helps the neural network know how far off it is, so it can properly utilize its optimizer. This article will discuss several loss functions supported by Keras — how they work, their applications, and the code to implement them.
Binary Cross-Entropy is defined mathematically as —
— given a correct target t and a predicted value p.
Given values of p for a correct target 0, the binary cross entropy value can be graphed as —
Given values of p for a correct target 1, the binary cross entropy value can be graphed as —
Entropy is the measure of uncertainty in a certain distribution, and cross-entropy is the value representing the uncertainty between the target distribution and the predicted distribution.
#FOR COMPILINGmodel.compile(loss='binary_crossentropy', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.binary_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0)
Categorical Cross Entropy and Sparse Categorical Cross Entropy are versions of Binary Cross Entropy, adapted for several classes. Categorical cross entropy should be used when one sample has several classes or labels are soft probabilities, and sparse categorical cross entropy should be used when the classes are mutually exclusive.
Categorical cross-entropy:
#FOR COMPILINGmodel.compile(loss='categorical_crossentropy', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.categorical_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0)
Sparse categorical cross-entropy:
#FOR COMPILINGmodel.compile(loss='sparse_categorical_crossentropy', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.sparse_categorical_crossentropy(y_true, y_pred, from_logits=False, axis=-1)
Cosine similarity is a measure of similarity between two vectors. The mathematical representation is —
— given two vectors A and B, where A represents the prediction vector and B represents the target vector.
A higher cosine proximity/similarity indicates a higher accuracy. Perfectly opposite vectors have a cosine similarity of -1, perfectly orthogonal vectors have a cosine similarity of 0, and identical vectors have a cosine similarity of 1.
Cosine Proximity can be implemented in Keras:
#FOR COMPILINGmodel.compile(loss='cosine_proximity', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.cosine_proximity(y_true, y_pred, axis=-1)
Hinge Loss is mathematically defined as —
— given a prediction y and a target value t of ±1. Note y should be a probability and not a single class label.
Below is a plot of hinge loss, which is linearly negative until it reaches an x of 1.
#FOR COMPILINGmodel.compile(loss='hinge', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.hinge(y_true, y_pred)
Hinge has another deviant, squared hinge, which (as one could guess) is the hinge function, squared.
#FOR COMPILINGmodel.compile(loss='squared_hinge', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.squared_hinge(y_true, y_pred)
Huber Loss is defined mathematically as
...for a coefficient c, where t represents the difference between the target value and the predicted value, which can be graphed as
...for various values of c.
This can implemented in Keras as
#FOR COMPILINGmodel.compile(loss='huber_loss', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.huber_loss(y_true, y_pred, delta=1.0)
The hyperbolic cosine function is log(cosh(x)), graphed as
...where x indicates the difference between the predicted value and the target value.
#FOR COMPILINGmodel.compile(loss='logcosh', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.logcosh(y_true, y_pred)
The logarithm of the mean absolute error, or log(x2), is graphed below.
The logarithm of the error squared is special in that a decrease of, say, 0.1 in the error would have a bigger drop in the loss function if it is closer to 0 (say, 0.2 to 0.1) than if it were farther (say, 1.2 to 1.1).
This can be implemented in Keras as:
#FOR COMPILINGmodel.compile(loss='mean_squared_logarithmic_error', optimizer='sgd')# optimizer can be substituted for another one#FOR EVALUATINGkeras.losses.mean_squared_logarithmic_error(y_true, y_pred)
If you enjoyed, check out other posts:
A Guide to Neural Network Layers with Applications in Keras
A Guide to Neural Network Optimizers with Applications in Keras