# Optimizing Contrastive Learning with Semi-Online Triplet Mining
## 5 Experiments  
We conducted a series of experiments that compared regular online (semi-)hard-triplet mining with the proposed method, which contains the additional step of ordering the dataset periodically. A single experiment involved training an embedding model multiple times and evaluating each instance on ten test sets. The training and test datasets were identical for all runs. Each test dataset comprised 1000 identities with two realizations per identity and 2000 identities with a single realization per identity. For the shoe dataset, that translated into 2000 paired and 2000 unpaired shoes. Shoe pairs in our system are marked either \"training pairs\" or \"test pairs.\" We used all training pairs for training but random samples (of the mentioned size) for testing. The main dataset comes in two versions: regular and cleansed. The generated dataset has six versions: some versions have HP and WP errors added, with three error levels for each. The levels of error considered were 0.1%, 1%, and 10%. Training commences with VGG16 layers being initialized with pretrained weights from ImageNet [27]. Downstream dense layers are initialized with uniform Glorot [28] initialization for kernels and zeros for biases. The loss function is set to the sum of Triplet-SemiHardLoss and TripletHardLoss. These are TensorFlow (v.2.4.1) [29, 30] Addons implementations of the triplet loss function as defined [1]. Both are complex functions responsible for the triplet mining process, not the mere implementation of the triplet loss function. As they are very similar, they share many computationally intensive operations, such as calculating a distance matrix. To prevent the overhead of performing these calculations twice, a new implementation is created to reuse intermediate results, but which still returns the sum of the losses mentioned. The margin parameter of triplet loss is set to 0.2. This loss is minimized using the Adam [31] optimizer. The learning rate for the optimizer starts with 0.001 and is scheduled to decay exponentially ten times per thirty epochs. Training is performed for 300 epochs with early stopping after 30 consecutive epochs with no progress. Our server with one A6000 per VGG16 branch (see [2] for detailed architecture) takes approximately two to three days to complete training. We perform dataset reordering every five epochs. We use a reordering buffer that fits the whole dataset. That does not mean that we fit the whole dataset into memory at any given point. The current state of the trained embedding model is used to infer the embeddings of the buffer (in our case, the whole training dataset) in batches. Then, our reordering method is applied, and training proceeds normally for another five epochs. Figure 6 shows how reordering frequency affects pairing accuracy and training time. The validation metric is pairing accuracy, described in [2]. It is a fraction of shoes from a testing set that were paired correctly with another shoe or correctly left unpaired, as the test set can contain single shoes (or unpaired shoes). This metric is test-dataset-size-sensitive, which is unusual. For accuracy or mean squared error, the dataset size is irrelevant because no dependency between examples exists. We may also consider the growing number of all possible pairings in datasets of growing size, from which a pairing method must select one to score 100%. The test sets for the generated data comprise identities generated consecutively; therefore, they are similar, which makes this a particularly challenging task--even for a human. We hypothesized in the Section 3 of this article that the efficiency of our method is based on increased ability to identify hard triplets. As we only have two examples per identity, selecting a positive example for any triplet is obvious (there is only one candidate), so identifying a hard negative is the only degree of freedom and the only challenge. The harder the negative, the closer its embedding is to the anchor example's embedding.