# Optimizing Contrastive Learning with Semi-Online Triplet Mining
## 2 Related Work  
A growing number of examples exist of the use of AI-based methods in textile recycling--or, more broadly, the apparel industry [3]. Many of them are, at base, examples of material classification with the support of machine learning. Such classification is often performed using spectral imagery, which can exploit the fact that different electromagnetic wavelengths interact with particular materials in particular ways [4; 5; 6]. Other applications in the field include clothing classification by garment type (e.g., jeans, shirts, and jackets) [7] and the color classification of wool fabrics [8]. In the footwear domain, prior research focuses on brand identification [9] and shoe style classification [10].  
Surprisingly little research has been produced on the shoe pairing process. Our previous study [2] established the base model that we continue to use in this work. We use a convolutional neural network [11; 12] as an embedding generator. We tested many modern architectures, including CLIP [13], dense networks [14], EfficientNet [15], Inception [16],and ResNet [17]. We settled for a relatively old VGG16 [18] network, which yielded the best tradeoff between pairing accuracy and computational overhead for our application. We define pairing accuracy as in Equation (1). It remains the master metric of performance throughout the research process because it is crucial in business. It gets maximized indirectly by the minimization of triplet loss (2) using SGD methods.  
\[\textit{pairing accuracy}(S)=\frac{|\textit{shoes correctly assigned matching shoe}|+|\textit{shoes correctly left unpaired}|}{|all\textit{shoes in shoe set}|} \tag{1}\]  
Three input images are processed with independent VGG16 networks (stripped of the dense layers); global average pooling [19] is then performed on each branch before they get concatenated and processed with final dense layers. We use transfer learning [20] for initializing convolutional features from ImageNet pretraining. The architecture of our network is described in our previous work in great detail [2].  
To train effective embeddings, we use the triplet loss function as defined in [1], which is given by Equation (2). \(T\) is a set of triplets in the form \((x^{a},x^{p},x^{n})\) (anchor, positive, and negative) of size \(N\). \(E\) is our neural network model that creates the embeddings, while \(d\) is a metric; we use Euclidean distance as in [1]. \(m\) is the parameter responsible for the separation margin. We later use \(D=\{\{x_{1}^{1},x_{1}^{2}\},\{x_{2}^{1},x_{2}^{2}\},\{x_{3}^{1},x_{3}^{2}\},...\}\) to denote a dataset. The subscript denotes the identity of an example, and the superscript describes the order within an identity.  
\[L(T)=\sum_{i=1}^{N}max(0,d(E(x_{i}^{a}),E(x_{i}^{p}))-d(E(x_{i}^{a}),E(x_{i}^{ n}))+m) \tag{2}\]  
As the authors of [1] suggest, searching for hard or at least semi-hard examples is helpful because easy examples satisfy the triplet inequality early in the training process. Hard triplets are those for which the model (at some point in training) places the embedding of a negative example closer to the anchor than a positive one. In a semi-hard triplet, the negative embedding distance from the anchor is greater than the positives but less than the margin. The authors also distinguish two methods of searching. The first is offline mining, which depends on preparing the triplets every \(N\) training steps with the current state of the embedding network on a subset of data. The disadvantage of the offline approach is that the triplets used in training result from an old, now nonexistent model (as its parameters have changed). This can make them suboptimal for learning. Online mining searches for hard triplets only within mini-batches, which results in a smaller population of examples being considered for hard triplets simultaneously, with the possibility of valuable ones being missed. The authors focus on the online approach, as they could afford mini-batches of a few thousand examples.