# Optimizing Contrastive Learning with Semi-Online Triplet Mining
## 1 Introduction  
This article presents a method of optimizing contrastive learning by controlling the ordering of examples in a dataset which we devised while solving a real-life industry problem. Our findings can be applied to most of the contrastive learning methods available, although this article focuses on triplet learning. Contrastive learning is a technique that creates a model that seeks to distinguish between the \"same\" and \"different\" examples. It has been used to solve classification-like problems with a variable number of classes: verification, similarity computation, or clustering problems [1]. In the process, an embedding transformation is learned to transform original data points into a compact vector representation (typically hundreds or thousands of dimensions) with the desired properties. Such a process often entails the \"same\" examples ending close to each other in the embedding space and \"different\" ones ending far away from each other. Our real-life industry problem is similar, as we must identify pairs in a large set of second-hand shoes for our client, VIVE Textile Recycling. A shoe is hung on a dedicated danger and travels through the firm's facility via an automated overhead transport (AOT) system. It gets photographed from three angles inside one of two photobooths during this process. The solution we develop together with VIVE must achieve the highest possible pairing accuracy solely due to the scale of the operation. The processing of over two million shoes per month makes even a small fraction of the errors worth correcting, as they can easilyamount to considerable losses. The shoe stream is large and diverse, as it comprises used footwear collected from across Europe. This volume and variance render the model's training even more difficult, as it is challenging for similar-but-different pairs to be found, retrieved, and used in contrastive learning. Our previous work [2] describes how we used triplet learning with a three-image-input VGG-inspired convolutional deep neural network to achieve high pairing accuracy. We opted to name shoes that belong to the same pair as \"same\" and shoes that do not as \"different\". This decision has the drawback of making the embedding transformation laterality agnostic, which means that our main network cannot differentiate left shoes from right ones. Triplet loss [2] pulls the embeddings of anchor A (e.g., a left shoe) and positive example P (e.g., a right shoe) closer together, while pushing the embedding of negative example N (a nonmatching shoe) away. In other words, we force the network to transform left- and right-shoe images into the same region of the embedding space. Laterality is handled separately by a dedicated network, a simple binary image classifier. Later, we use this information to avoid pairing two left shoes together. In classical machine learning problems, such as classification, examples are independent. Triplet learning is different because a single input to the network comprises three distinct dataset examples, one reference (anchor) example, one positive example (being \"the same\" as the anchor), and one negative example (\"different\" from the anchor). Therefore, an implicit relation between different examples in the dataset exists. Due to this relation, alongside the properties of triplet learning and triplet loss itself, training in this regime might be more challenging than classical problems like classification. We experienced this problem during the training of our pairing network. As the training progressed, relatively soon into the process, triplet loss started to become zero on progressively more batches. The zeroing of the loss function meant that there were no gradient and no updates to the model parameters, despite quite many errors continuing to exist on the test dataset. This happens because randomly selected triplets tend to be relatively easy, which means that even simple (or untrained) models' embeddings already satisfy the triplet loss equation.