# Optimizing Contrastive Learning with Semi-Online Triplet Mining
## 3 Method  
Our method strives to maintain the best of both worlds while limiting the adversarial effects. The name \"semi-online\" stems from the combination of online, batch-level search for the best triplets with offline, periodic dataset reordering. High-level comparison of these is shown on Figure 1. To preserve the principle of using embeddings inferred by the latest state of the trained model, the online triplet mining stage remains unchanged. The proposed novelty lies in the addition of an extra step before batching: the (offline) reordering of the dataset in a manner that favors similar examples (but with different identities) being closer together, thus increasing the probability of them getting into the same mini-batch. The rationale is that such a regime creates better-performing models by enabling online triplet mining to find more (semi-)hard negative examples. One way of formalizing this involves seeking batch \(B=\{x_{1}^{1},x_{1}^{2},x_{1}^{1},x_{2}^{2},x_{3}^{1},x_{3}^{2},...\}\) with minimal \(\sum_{1\leq i<i\leq n}d(E(x_{i}^{1}),E(x_{j}^{1}))\). Whether this is the optimal formulation of that idea remains unknown. It is possible that a batch that has a single but very hard example for every identity (i.e., \(\sum_{i}d(E(x_{i}^{1}),E(\arg\min_{x_{j}}d(E(x_{i}^{1}),E(x_{j}^{1}))))\)) would be better for training the embedding network (in terms of pairing accuracy) than a batch in which, for every identity, every other identity creates a moderately hard triplet. The relation between batch properties and pairing accuracy seems intractable; therefore, no single best criterion on the batch itself can be formed. Figure 2 presents a visual representation of the process. This method is akin to using a larger mini-batch size without needing more memory on the GPU or accelerator. It differs from the gradient accumulation technique [21], however, because the gradient is still calculated based on a smaller mini-batch. Only the triplet mining aspect is upgraded. Figure 2: A visual representation of regular triplet learning (**left**) and triplet learning with our semi-online reordering method (**right**). A single training example (one shoe) is represented as a letter concatenated with a digit. Examples that contain the same letters have the same identity. Color corresponds to the region in the embedding space. The more similar the colors are, the less Euclidean distance between the embeddings there is. Therefore, triplets with a negative example (the third element of the triplet) with a similar color to the anchor (the first element) are harder triplets, which is desirable. Ordering the dataset into a “rainbow” yields batches on which triplet mining can find more hard triplets, visible as more uniformly colored triplets. This is a conceptual, nonquantitative image only. In our method, similarly to offline triplet mining, the frequency of the performance of the extra work and the number of data searched can be selected arbitrarily. Regarding frequency, reordering examples too frequently leads to considerable computational overhead but can also cause the order to align better with the current state of the model (less outdated). Using high frequencies is not as crucial as it is in offline mining because the subsequent online mining step will use the current model to calculate the most recent embeddings. The fraction of the dataset being searched and ordered simultaneously is also a tradeoff: using the small buffer of identities will not improve the performance and will ultimately collapse into simple online mining (no meaningful reordering occurs when the buffer size is equal to the batch size). Although reordering the whole dataset entails the heaviest computational overhead (the complexity of this process is higher than linear), it is also capable of finding and exaggerating errors in the dataset. We discovered the reordering of the whole dataset once every five epochs to be a reasonable tradeoff. A running buffer of carefully ordered examples is maintained to enable online triplet mining to work best.