Label-aware Hard Negative Sampling Strategies with Momentum Contrastive Learning for Implicit Hate Speech Detection (2024)

Abstract: Detecting implicit hate speech that is not directly hateful remains a challenge. Recent research has attempted to detect implicit hate speech by applying contrastive learning to pretrained language models such as BERT and RoBERTa, but the proposed models still do not have a significant advantage over cross-entropy loss-based learning. We found that contrastive learning based on randomly sampled batch data does not encourage the model to learn hard negative samples. In this work, we propose Label-aware Hard Negative sampling strategies (LAHN) that encourage the model to learn detailed features from hard negative samples, instead of naive negative samples in random batch, using momentum-integrated contrastive learning. LAHN outperforms the existing models for implicit hate speech detection both inand cross-datasets. The code is available at https://github.com/Hanyang-HCC-Lab/LAHN

heading: 1 Introduction
text: Online hate speech has become one of the major social problems as it leads to discrimination against certain groups and social conflict (Howard, 2019; Matamoros-Fernández and Farkas, 2021). Hate speech can be either explicit, which directly uses hateful language, or implicit, which metaphorically implies hateful language (Waseem et al., 2017). While explicit hate speech can be addressed relatively easily through the use of automated filters (Watanabe et al., 2018; Xiang et al., 2012; Rodríguez-Sánchez et al., 2020), dealing with implicit hate speech is highly challenging. For example, identity term bias (ElSherief et al., 2021; Dixon et al., 2018), where identity terms (e.g., black, jew) frequently appear in hateful contexts, can over-bias the model and cause false positives in implicit hate speech detection. Given the characteristics of implicit hate speech, it will be important to encourage
*Corresponding author
models to learn subtle differences between similar sentences that might otherwise confuse them. Research has built implicit hate speech datasets of implicit hate speech (ElSherief et al., 2021; Sap et al., 2020; Hartvigsen et al., 2022; Vidgen et al., 2021) and proposed detection models (Kim et al., 2022, 2023) using contrastive learning; however, the model showed limited performance improvement in in-dataset evaluation or have limitations that require external knowledge or additional computational resources for pre-training.
Figure 1 shows an easy negative (middle) and a hard negative (bottom) for an anchor sentence targeting black. Easy negatives have opposite labels and are relatively distinct from the anchor, meaning that for the model it is not difficult to distinguish from the anchor. On the other hand, hard negatives are semantically similar to the anchor but have opposite labels, meaning that the model may have difficulty distinguishing them from the anchor. The key task is to effectively identify hard negatives and train the model accordingly. If the negative sample trained along with the anchor is similar to the anchor, the model can be trained to better distinguish relatively indistinguishable data (Robinson et al., 2021; Jiang et al., 2022; Wu et al., 2021). However, existing work has used naive contrastive learning, which encourages contrast between data in randomly sampled mini-batches to learn repre-
16177
sentations and fails to guarantee the learning of hard negative samples.
In this work, we propose a novel approach to implicit hate speech detection, namely Labelaware Hard Negative sampling strategies (LAHN). LAHN focuses more on distinguishing between the anchor and hard negatives, mitigating overfitting to the context of the text or specific words. Inspired by MoCo (He et al., 2020), LAHN employs a momentum queue to effectively expand the negative samples that are candidates for hard negative samples, and compactly performs contrastive learning by extracting the top-k hard negatives that require sophisticated disentangling from the anchor. A key difference from previous research is that LAHN extracts hard negatives from the momentum queue in contrastive learning based on the similarity between the anchor and the negatives and the level of ambiguity for each negative (Figure 2).
In summary, our contributions are as follows:
• We propose LAHN, a novel method that focuses on hard negatives that should be disentangled from the anchor to promote the effective learning of hate speech representations with implicit characteristics.
• Contrary to previous studies, we observe that LAHN can improve performance in both inand cross-dataset evaluation with a simple dropout noise augmentation without external knowledge and additional cost.
• We validate the generalized learning effect of LAHN by achieving state-of-the-art performance in both in- and cross-dataset evaluation on four representative public benchmark datasets for implicit hate speech detection.

heading: 2 Related work

heading: 2.1 Implicit Hate Speech Detection
text: Implicit hate speech refers to expressions of hatred or discrimination that are not directly stated or overtly aggressive but are conveyed through subtle, indirect language, insinuations, or coded messages (ElSherief et al., 2021; Kim et al., 2023). According to prior studies (Ocampo et al., 2023; Yang et al., 2023), it is challenging to identify implicit hate speech due to its reliance on background knowledge, cultural context, and the ability to infer the implied meanings behind seemingly neutral or ambiguous expressions. For instance, Ocampo et al. (2023) explores the absence of consistent hate-speech-specific prosody across languages, indicating the importance of linguistic and cultural nuances in understanding hate speech.
For this reason, the existing pre-trained language models in the hate speech domain, such as HateBERT (Caselli et al., 2021) or fBERT (Sarkar et al., 2021), may have a spurious correlation issue that classifies an input text as hateful due to the presence of specific identity terms (e.g., Black, Asian, etc.) and this hinders generalized performance. This body of work underscores the need for sophisticated, context-aware approaches in hate speech detection systems to effectively address the subtleties of implicit hate speech, considering its varied manifestations and significant societal impact.
Contrastive learning methods (Wu et al., 2020; He et al., 2020; Khosla et al., 2020; Giorgi et al., 2021; Gao et al., 2021) has been increasingly recognized as a necessary and effective approach for detecting implicit hate speech due to its ability to handle the context-dependent nature of such content by leveraging the subtle differences and similarities between hateful and non-hateful content, enhancing model sensitivity to the nuanced expressions of hate speech. Kim et al. (2022) proposed ImpCon, which uses the external knowledge (i.e., the implication of anchor sentences) as positive samples using contrastive loss for implicit hate speech detection. The recently proposed ConPrompt (Kim et al., 2023) utilized machine-generated statements to improve implicit hate speech performance by applying contrastive learning using example sentences from the original prompt as positive samples. However, these previous methods still have limitations due to reliance on pre-defined external knowledge or text generation costs.
Recently, there has been research in using Large
non-hate speech
hate speech
Label-Aware
(b)
(c)
Hard Negative (d)
non-hate speech hate speech
Momentum queue
......
Weighting matrix
Language Models (LLMs) for hate speech detection. Zhang et al. (2024) investigated the sensitivity and limitations of LLMs for performing hate speech detection, and Roy et al. (2023) explored several prompting strategies to improve the detection capabilities of LLMs. Yang et al. (2023) proposed the HARE framework, which incorporates explanations generated using chain-of-thoughts (CoT) into the language model learning process. Despite these efforts, the performance of LLMs in hate speech detection is still limited, and costly compared to the language models such as BERT and RoBERTa. Our experimental results on implicit hate speech detection using LLMs are presented in Appendix A.

heading: 2.2 Hard Negative Mining
text: Hard negative mining is an important technique in various areas of machine learning that significantly improves a model’s performance by carefully selecting negative samples that are less obviously distinguishable from positive samples (Robinson et al., 2021; Gunel et al., 2020; Schroff et al., 2015; Wu et al., 2021; Ge et al., 2021; Kalantidis et al., 2020). In contrastive learning, many studies have found that focusing on hard negative samples among negative samples learns better representations and improves performance (Robinson et al., 2021; Gunel et al., 2020; Schroff et al., 2015; Kalantidis et al., 2020).
In the field of computer vision, many studies related to hard negatives have been proposed. Ge et al. (2021) identified the impact of hard nega-
tives in learning strategies to generate appropriate negative samples, and Robinson et al. (2021) proposed H-SCL, which uses the label information of samples to introduce hard negative sampling in supervised contrastive learning. Wu et al. (2021) proposed a ring method to step-wise retrieve hard negative samples that are neither too hard nor too easy, and experiments showed that dynamically changing the hard negative sampling range can help to learn.
In NLP, hard negative mining can help models learn finer distinctions between textual data and better understand the nuances of language, context, and semantic meaning. The supervised SimCSE for learning text representations proposed by Gao et al. (2021) considered predefined contradiction pairs as hard negatives for the first time in the field of NLP and encouraged learning to distinguish hard negatives in contrastive learning. However, supervised SimCSE relies on predefined in-batch hard negatives and only utilizes samples within the batch in contrastive learning, so it is limited in considering negative samples broadly. ESimCSE (Wu et al., 2022) and MoCoSE (Cao et al., 2022), which were proposed to compensate for these limitations of SimCSE, integrated momentum contrastive learning with SimCSE, but hard negative samples were not considered in the contrastive learning process.
To the best of our knowledge, no research has explored how to effectively integrate hard negatives with supervised contrastive learning using label information without relying on external knowledge. In this paper, we propose LAHN, which integrates
Algorithm 1: Pseudocode of LAHN # E, E_m: main encoder end momentum encoder # queue: momentum contrast queue # m, t: momentum and temperature parameter # k: hard negative sampling size # sim: cosine similarity function
# load a batch x with N samples for x in loader:
x_anc, pred = E.forward(x) x_aug, _ = E_m.forward(x) x_aug = x_aug.detach() # no gradient
# enqueue the current batch embeddings enqueue(queue, x_aug) # dequeue the earliest batch embeddings dequeue(queue)
# hard negative sampling for anchor _, weights = E_m.forward(queue) negs = sort(sim(x_anc, queue) * weights) hard_neg = topk(negs, k) # top-k sampling # extract pos and neg simialrity pos = sim(x_anc, x_aug)/t pos = diag(pos).view(-1, 1) # extract pos neg = sim(x_anc, hard_neg)/t logits = concat([pos, neg], dim=1)
# Contrastive loss and CE loss labels = Zeros(logits)[:, 0] = 1 loss_1 = CrossEntropy(logits, labels) loss_2 = CrossEntropy(pred, x.labels) (loss_1 + loss_2).backward() update(E.params) # momentum update E_m.params = m*E_m.params+(1-m)*E.params
momentum contrastive learning and label-aware hard negative sampling strategies to effectively handle implicit hate speech data.

heading: 3 Method
text: As shown in Figure 2-(a), the existing method performs contrastive learning on all other samples with equal weight, including those of the same class, for a given anchor in a mini-batch. This characteristic makes it difficult for the model to focus on distinguishing between the anchor and hard negatives, which are semantically similar and embedded close to the anchor.
On the other hand, our LAHN first uses label information to identify negatives with labels different from the anchor’s, which should be embedded in opposite directions. Next, the momentum encoder calculates the probability that the negative is in the same class as the anchor, and this probability is used to multiply the similarity value between the negative and the anchor. Finally, LAHN extracts the top-k hard negatives based on the calculated values and performs compact contrastive learning
for the given anchor (Figure 2-(b)). Through this, LAHN better captures the detailed differences between implicit hate speech and non-hate speech by focusing on hard negatives.
Figure 3 illustrates the overview of our LAHN. Random mini-batch samples are used as the input to the main and momentum encoders, and the output embedding of the momentum encoder is inserted into the momentum queue. Hard negative sampling for each anchor is performed after the momentum queue is filled with at least a quarter of the size. As in MoCo (He et al., 2020), we use an exponential moving average (EMA) to slowly update the momentum encoder based on the weights of the main encoder. Algorithm 1 presents the pseudocode implementation.

heading: 3.1 Label-aware Hard Negative Sampling
text: In this section, we propose a label-aware hard negative sampling strategy to effectively integrate label information and hard negative mining methods in supervised contrastive learning. We assume that focusing on contrastive learning to distinguish the anchor from the hard negatives that are semantically similar to the anchor can help learn disentangled representations of implicit hate speech data.
In supervised contrastive learning, false negatives, where samples with the same label as the anchor are used as negative samples, can hinder the representation learning of the model (Kalantidis et al., 2020). To exclude false negative cases in the hard negative sampling process, we identified the true negatives of the momentum queue using the label information of each anchor in the batch and then sorted them by finding the cosine similarity between anchors and true negatives.
To sample hard negatives, which should be primarily included for successful compact contrastive learning, we considered not only the contextual similarity to the anchor, but also the ambiguity of the representation to classify the class of the hard negative. First, we calculated the probability that all true negatives in the momentum queue, which are hard negative candidates for contrastive learning, are predicted as the class of the anchor through the prediction head of the momentum encoder. Second, we applied hard negative sampling based on the assumption that the closer the probability of being predicted as the anchor class for true negatives is to 1 (more ambiguity), the more sophisticated representation learning is required. Finally,
we multiplied the prediction probabilities by the similarity values with the anchor, sampled the topk hard negatives based on the multiplied values, and assigned them as negatives of each anchor in compact contrastive learning.

heading: 3.2 Training objective for Implicit Hate Speech Detection
text: In this section, we conduct contrastive learning for disentangled representation learning and classification training for implicit hate speech detection. First, we employ momentum contrastive learning (He et al., 2020) to maximize the effectiveness of our proposed hard negative sampling strategy. In the implicit hate speech domain, text augmentations, such as the most commonly used token replacement (e.g., synonym substitution), can alter a sentence’s hateful or non-hateful nature. Therefore, we utilized the augmentation technique that employs dropout noise, as used in SimCSE (Gao et al., 2021). In Section 5, the experiments using external knowledge employed the augmentation techniques (i.e., implication on hate speech and synonym substitution on non-hate speech) used in (Kim et al., 2022). After hard negative sampling in Section 3.1, we used InfoNCE (Oord et al., 2018) loss for the contrastive learning as follows:
LCL = − log exp(sim(xi, x p i ))/τ∑N
j=1 exp(sim(xi, x n j ))/τ
(1)
where xi is the anchor sample of the batch, and every anchor has one positive sample xp. The proposed method uses only the batch’s positive samples and the hard negatives sampled from the momentum queue as negatives. N denotes the number of selected hard negatives from the queue, xn denotes the negative samples, τ is a temperature parameter, and sim is the cosine similarity function. To minimize the loss in the corresponding contrastive learning process, the similarity between anchor and positive samples should be maximized, and the similarity between anchor and negative samples should be minimized. The loss function for implicit hate speech detection is as follows.
LCE = − 1
N
N∑
i=1
[yi log ŷi + (1− yi) log (1− ŷi)]
(2)
where ŷi ∈ {0, 1} is one-hot encoded ground truth. The final loss that we use for training is as follows:
L = (1− λ)LCL + λLCE (3)
where λ is a loss scaling parameter and we empirically found and set the parameter λ as 0.1.

heading: 4 Experiment

heading: 4.1 Datasets
text: Similar to the approach in (Kim et al., 2022; Hartvigsen et al., 2022), we used the implicit hate speech benchmark datasets—IHC, SBIC, DynaHate, and ToxiGen—for model evaluation.
• Implicit Hate Speech Corpus (ElSherief et al., 2021): The dataset focuses on implicit hate speech collected from Twitter. It comprises 19,112 tweets, with 4,909 labeled as implicit hate and 933 labeled as explicit hate.
• Social Bias Inference Corpus (Sap et al., 2020): The dataset with hierarchical categories of social biases and stereotypes. It supports large-scale modeling and evaluation with 150k structured annotations of social media posts, covering over 34k implications about demographic groups.
• Dynamically Generated Hate Speech Dataset (Vidgen et al., 2021): The dataset comprises 41,255 entries generated by a humanand-model-in-the-loop process. It captures hate speech against the ten most frequently targeted groups, including black people, women, and Jews, among others.
• ToxiGen (Hartvigsen et al., 2022): Created using GPT-3, this dataset contains 274,186 machinegenerated statements, with over 135k toxic and 135k benign statements. It targets 13 minorities, such as Blacks, Jews, and LGBTQ+ people.

heading: 4.2 Implementation Details
text: We used a pre-trained language model (RoBERTabase (Liu et al., 2019) and BERT-base-uncased (Devlin et al., 2019)) as a sentence encoder in all experiments and used the Adam optimizer with a learning rate of 2e-5, batch size of 16, and dropout of 0.1 during the fine-tuning process. We used NVIDIA RTX 4090 GPU (24GB) for training all models, and the hyper-parameters were set to loss scaling parameter λ ∈ {0.1}, momentum parameter m ∈ {0.999} based on previous studies, and the temperature parameter τ ∈ {0.05, 0.07, 0.1}, momentum queue size q ∈ {512, 1024, 2048}, and hard negative k ∈ {16, 32, 64} for further hyper-parameter search. We chose the best model score with macro F1-score in the validation.

heading: 4.3 Baseline Models
text: • Cross-Entropy (CE) loss: CE loss is widely
adopted as a general approach to classification tasks and hate speech detection.
• Supervised Contrastive Learning (SCL) with CE loss (Gunel et al., 2020): This method enhances CE loss by combining supervised contrastive learning. This is effective for complex tasks such as hate speech detection, where distinguishing subtle class differences.
• Contrastive Learning using Implication (ImpCon) with CE loss (Kim et al., 2022): This method improves CE loss by integrating implication-based contrastive learning. ImpCon improves the model to understand contextual relationships by injecting common implications of implicit hate speech.

heading: 5 Results and Analysis

heading: 5.1 Implicit Hate Speech Detection Results
text: Table 1 shows the evaluation results in the four datasets for the training model on the IHC and SBIC datasets. External knowledge means that additional training knowledge beyond the existing data is used, such as implication and synonym substitution. In the case without external knowledge,
the encoder dropout noise (Gao et al., 2021) is used as the positive sample. In the cases with external knowledge (Ours, ImpCon), implication was used as the positive sample for hate speech, and synonym substitution was used as the positive sample for non-hate speech, the same as Kim et al. (2022).
In-dataset evaluation (i.e., IHC training set → IHC test set, SBIC training set → SBIC test set) and cross-dataset evaluation (i.e., IHC training sett → all test sets except IHC test set, SBIC training sett → all test sets except SBIC test set), LAHN achieves the best performance in 29 out of 32 comparison cases. Furthermore, unlike previous studies (Kim et al., 2022) that found inferior or equal performance to single CE loss learning in the indataset case, LAHN achieves better performance than CE loss learning in all in-dataset evaluations.
LAHN is also robust in the absence of external knowledge. We observe that SCL loss without external knowledge performs poorly relative to CE loss, and Kim et al. (2022) shows the same trend for SCL with augmentation based on synonym substitution. In contrast, LAHN, without external knowledge, which uses only dropout noise applied to anchors as a positive sample, achieves the best performance in 15 out of 16 cases without external knowledge comparison.
Hate speech Non-hate speech
Hate speech Non-hate speech
In the case using the SBIC training dataset, the average score of LAHN without external knowledge (Average column) outperforms the average score of LAHN with external knowledge in both BERT and RoBERTa models. This indicates that the implication used as external knowledge is of poor quality or that the implication may lead to negative bias in model training. The average scores for the cases without external knowledge using BERT and RoBERTa are 69.75 and 71.28, respectively, while the cases with external knowledge are lower (68.85 and 70.76). This shows that the implication information used as external knowledge for SBIC’s implicit hate speech data or the synonym substitution augmentation used as external knowledge for non-hate speech is likely to be of poor quality.

heading: 5.2 Qualitative Analysis
text: Figure 4 shows a visualization of some implicit hate speech data from the IHC validation set using t-SNE (Van der Maaten and Hinton, 2008) on a model trained on the IHC dataset by three targets (Immigrants, Blacks, Jews).
SCL (Figure 4-b) shows more dense clusters compared to CE (a) but still has some samples mixed in with each other. ImpCon and LAHN, on the other hand, form clearer cluster boundaries. We note that LAHN forms different sub-clusters within the immigrant cluster. SCL and ImpCon were trained to cluster similar classes using label information and implication, respectively. They push
or pull semantically similar or dissimilar samples within randomly sampled mini-batches, depending on the training strategy. In this process, contrastive loss can degrade the semantic similarity of a representation by further attracting positive samples that are already close enough or pushing negative samples that are already far enough away.
In contrast, our method assumes that the pretrained language model already has a high-quality representation and encourages the model to use only hard negatives for training in order to increase the classification performance of the model without compromising the semantic information as much as possible. As shown in Figure 4-d, LAHN ensures the margin between sentences with different target information while maintaining the margin of data with different semantic characteristics within the immigrants.
Figure 5 shows a visualization of the extracted embeddings for the part of the ToxiGen dataset in the zero-shot setting with the model fine-tuned on the IHC dataset. We randomly sampled both hate speech and non-hate speech about black people. The embeddings using LAHN form a relatively sharper boundary between hate speech and nonhate speech in compared to the other three models. This result shows that our LAHN leads the model to form a more generalizable representation compared to the other methods.

heading: 5.3 Effect of Hyperparameters
text: We evaluate the effects of the hyper-parameters q (queue size) and k (hard negative sample size). Cao et al. (2022) found that for NLP tasks where the model has a fast update rate, a queue size as high as 4096 can cause performance degradation. This shows that, unlike MoCo, queue expansion does not inevitably lead to better performance. This finding is consistent with our results, and Figure 6 shows the variation of the model’s performance with queue size and hard negative sample size. The x-axis is the number of hard negatives sampled for each anchor, and the y-axis is the size of features that can be candidates for hard negatives.
In contrast to previous work (Cao et al., 2022; Wu et al., 2022), we did not use all of the queue features as negative samples but only sampled some of them, meaning that an increase in queue size does not imply the use of outdated information. Therefore, an increase in queue size does not necessarily lead to a linear decrease in performance. Nevertheless, our ablation results show that performance degradation occurs at a certain point in the queue size. That is, as the queue size increases, outdated samples may also become candidates for hard negative sampling, which can negatively affect performance. This result shows that maintaining a large number of hard negative sample candidates does not always improve performance.
Similarly, we found that increasing the number of hard negative samples does not result in a linear performance improvement. Since our method determines the hard negative samples based on the sorted similarity between the anchor and the negative sample, selecting a large number of hard negative samples implies that a high similarity to the anchor cannot be guaranteed. Therefore, our results show that it is necessary to experimentally select the appropriate number of hard negative samples and queue size.

heading: 5.4 Ablation Study
text: We conducted the ablation studies by removing each component of our methods on the IHC dataset and the SBIC dataset with RoBERTa-based model.
• MoCo: Momentum Contrast means that all samples in the Momentum Contrast queue with label information are used during contrastive learning.
• HN-Samp: Hard Negative Sampling means to use the Label-aware Hard Negative Sampling strategy with the momentum queue.
• S-Weight: Similarity Weight means using the prediction probability from the momentum encoder to induce the model to select features with a high probability of confusion as hard negatives.
The results are shown in Table 2. Because of the interdependence of our methods, a total of four progressive ablation studies were performed (i.e., HN-Samp cannot be used alone without SupMoCo, and S-Weight cannot be used alone without HNSamp). These results show that LAHN can significantly increase all performance compared to the MoCo, including false negatives to increase the number of negative samples, or HN-Samp which only similarity-based hard negative sampling without S-weight.

heading: 6 Conclusion and Future Work
text: In this paper, we propose LAHN that incorporates the momentum queue to extract hard negatives, which are more likely to be confused by the model as compared to anchors. We demonstrated the effectiveness of LAHN in both in-dataset and crossdataset performance evaluations, compared to existing methods. In the future, we will focus on finding new sampling metrics that are more advanced than similarity-based hard negative sampling for implicit hate speech detection and finding ways to exploit better semantic features from hard negatives.

heading: 7 Limitations
text: While our proposed LAHN has demonstrated its effectiveness in implicit hate speech detection, our research has some limitations.
First, LAHN relies on supervision, which reduces the advantage of MoCo in allowing a large amount of unlabeled data to be used for supervised learning. In addition, the dual encoder framework, which uses an additional MoCo encoder, still has cost limitations compared to other deep learning methods.
Second, our strategy is limited in that it requires the exploration of a large number of hyperparameters. As we have seen in Section 5.4, LAHN exhibits dynamic performance depending on hyperparameters such as q, k, and τ , which force the cost of hyperparameter exploration to vary across the dataset.
Third, since we sampled only a fraction of the MoCo queue, we may lose the benefit of MoCo’s use of large, consistent negative samples for training. In addition, since we are not using a large queue size like MoCo, it may be more advantageous to adopt an in-batch hard negative sampling strategy. However, we emphasize that our methodology overcomes the constraints of limited computational resources, which may inspire future work.
In future work, we can explore methods that are robust to multiple hyper-parameters, and investigate strategies for effectively sampling out-ofbatch hard negatives for training without introducing MoCo.

heading: 8 Ethics Statement
text: Our work aims to extend previous research on implicit hate speech detection and contribute to solving social conflicts caused by hateful content. We use publicly available open datasets containing implicit hate speech without compromising user privacy.
Implicit hate speech detection models can still be biased, and our method may cause the risk of increasing false predictions that have not been observed in previous studies and are not revealed by the performance. However, we believe that our ongoing efforts to improve implicit hate speech detection can help mitigate this risk.

heading: Acknowledgements
text: This research was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2020-II201373, Artificial Intelligence Graduate School Program(Hanyang University)), and the National Research Foundation of Korea (NRF) (2018R1A5A7059549, 2021S1A5A2A03065899).

heading: Future Work
text: However, we emphasize that our methodology overcomes the constraints of limited computational resources, which may inspire future work. In future work, we can explore methods that are robust to multiple hyper-parameters, and investigate strategies for effectively sampling out-ofbatch hard negatives for training without introducing MoCo