SlowGuess/NAACL_2022_OCR huggingface dataset:
To make a fair comparison, the hyperparameter setting of the backbone module is adopted explicitly from (Xiong et al., 2019), such as implementing the 300-d GloVe embeddings for question entities, maximal number of neighboring entity as 50, maximal question length as 10. The matching loss has been implemented using binary cross-entropy loss with 0.1 smoothing factor. In addition, the mini-batch size is 16, the Adam optimizer with a learning rate setting of 0.001, and the number of training epoch is set as 100. For the implemented knowledge extension module, factual descriptions
about entity and relation are available publicly<sup>4</sup>. Meanwhile, we generate three additional QA pairs for one single QA input. For the sMoCo, the momentum rate is 0.99,  $\tau = 0.07$ ,  $\lambda_{sMoCo} = 0.2$ ,  $\lambda_Q = 0.95$ ,  $\lambda_U = 0.4$ , and the negative queue capacity is set as 10000. The proposed model is trained on a machine with four Tesla V100 GPUs. The Hits@1 score is used to measure the performance.
We compare the proposed with different methods, including the baseline model (i.e. KDReader (Xiong et al., 2019)) and state-of-the-arts (i.e. PullNet (Sun et al., 2019), 2HR-DR (Han et al., 2020), EmbedKGQA (Saxena et al., 2020), and RecHyperNet (Yadati et al., 2021)). The comparison results over 10 trails are shown in Table 3.
Table 3: Results in Hits@1 obtained by sMoCo and existing methods re-implemented for four test sets. The number within the bracket indicates the original result reported by the paper.
Algorithm	KB0.1	KB0.3	KB0.5	KBFull
KDReader	33.5(33.6)	42.6(42.6)	52.8(52.7)	67.4(67.2)
PullNet	33.7	42.8	52.1(51.9)	68.0(68.1)
2HR-DR	33.5	42.5	52.0(52.2)	66.9(67.0)
EmbedKGQA	34.3	41.5	53.2(53.2)	67.0(66.6)
RecHyperNet	34.5	43.1	53.6(53.7)	68.4(68.4)
sMoCo	36.1	44.2	54.1	69.2
sMoCo shows superior performance compared to the state-of-the-arts via achieving a considerable margin. For instance, competing with the strongest baseline RecHyperNet, the proposed method outperforms by  $2.55\%$ ,  $0.93\%$ ,  $1.17\%$  with respect to KB0.3 KB0.5 and KBFull. It is also worth noting that sMoCo obtains the highest performance boost with KB0.1 ( $7.76\%$  and  $4.63\%$  compared to the baseline and RecHyperNet), which demonstrates its strong capability of handling the extreme low-resource KB.
In addition, to further evaluate the improvement from sMoCo, we also compare ours with cutting-edge contrastive learning techniques, including SimCLR (Chen et al., 2020a), MoCoV1 (He et al., 2020), MocoFT (Zhu et al., 2021), MoCoV3 (Chen et al., 2021), and SimSiam (Chen and He, 2021). Again, the backbone module from KDReader is
4Description about entities and relations can be found https://developers.google.com/freebase and https://freepal.appspot.com/, respectively
3There are another two training strategies, including pre-train and alternate. The former is to update the model first using  $\mathcal{L}_{sMoCo}$  before fine-tuning with  $\mathcal{L}_{\text{Matching}}$ , while the latter is to train the model with  $\mathcal{L}_{\text{Matching}}$  for  $(N_t - 1)$  iterations and switch to  $\mathcal{L}_{sMoCo}$  once, for every  $N_t$  iterations. We leave these as the future work.
implemented for all contrastive methods for a fair comparison. In addition, hyperparameters for employed contrastive methods are set to similar as sMoCo: the momentum rate is 0.99,  $\tau = 0.07$ ,  $|\mathcal{Q}| = 10000$ , while the prediction head is implemented as a 2-layer MLP with a 512-hidden dimension.
Table 4: Average Hits@1 results (10 trials) obtained by sMoCo and other contrastive learning methods.
Algorithm	KB0.1	KB0.3	KB0.5	KBFull
SimCLR	34.5	42.9	52.9	67.3
MoCoV1	34.5	43.2	52.8	67.8
MocoFT	35.4	43.6	53.3	67.9
MoCoV3	35.0	43.4	53.3	68.1
SimSiam	34.5	43.2	52.9	67.5
sMoCo	36.1	44.2	54.1	69.2
Table 4 clearly demonstrates the superiority of applying contrastive based methods, as averaged results (over 10 trials) show all methods achieve better performance compared to the baseline model (i.e., KDReader). In particular, contrastive methods perform specifically well with low-resource KBQA, as they lead to from  $2.98\%$  to  $7.76\%$  averaged improvement for KB0.1 (higher than other three cases). This result indicates the advantage of using contrastive learning for extremely low-resource KBQA tasks. Additionally, the proposed sMoCo outperforms existing methods, by achieving a notable accuracy (50.9 for overall averaged Hits@1) compared to that of SimCLR(49.4), MoCoV1(49.6), MocoFT(50.1), MoCoV3(50.0), and SimSiam(49.5), respectively.
Experiments are conducted to validate contributions from proposed modules, mainly the knowledge extension (KE) and sMoCo module. To highlight the low-resource nature of KG, we are particularly interested in the performance on datasets of KB0.1 and KB0.3. Again, all the results are reported as an averaged Hits@1 over 10 trials.
On knowledge extension module. To begin with, we consider the impact from the proposed knowledge extension (or data argumentation) by treating additional QA pairs as training samples, while no contrastive learning (sMoCo) is applied.
Results in terms of different numbers of additional pairs  $(|Q|_a)$  are then summarized in Table 5. Compared to the baseline (KDRreader), the advantage of KE is observed from the performance
Table 5: Performance against different numbers  $\left( {\left| Q\right| }_{a}\right)$  of additional QA pairs (without sMoCo).
|Q|a	KB0.1	KB0.3
baseline	33.5	42.6
1	33.9	42.9
3	33.7	43.1
improvement. With  $|Q|_a = 3$ , for instance, the model produces 38.4 for Hits@1 on average. Not surprisingly, the model performance is enhanced by the proposed KE (via providing additional training samples). In the following, we fix  $|Q|_a = 3$  for KE and analyze the contrastive module.
Notably, compared to traditional method (such as MoCoV1), sMoCo is different from three aspects: 1) adopting hard positive and 2) negative samples for estimating the contrastive loss, and 3) queue updating with hard positives (anchors). We accordingly perform ablation study on individual aspect to manifest their efficacy and the results are summarized in Table 6. In particular, for comparison purposes we take "KE" and "MoCoV1" to represent the result from the knowledge extension module and normal contrastive learning, as the baseline; "+hard positive" differs from MoCoV1 by using one hard positive; "+hard negative" considers a subset of hard negatives while maintaining a  $95\%$  of total sum, in addition to "+hard positive"; "+queue updating" further applies the proposed combination strategy to update negatives iteratively.
Table 6: Ablation study on hard positive, hard negative, and queue updating from the proposed sMoCo.
Model Variants	KB0.1	KB0.3
KE	33.7	43.1
MoCoV1	34.5	43.2
+hard positive	34.8(↑ 0.3)	43.4(↑ 0.2)
+hard negative	35.7(↑ 0.9)	43.8(↑ 0.4)
+queue updating	36.1(↑ 0.4)	44.2(↑ 0.4)
Results from Table 6 show contributions from individual aspect to the final performance, which evidently states their effectiveness. At first, not surprisingly, all four contrastive variants improve the overall performance compared to KE, which again demonstrate the superiority of forming discriminate features to separate positives and negatives. Additionally, we observe the step of “+hard negative” brings the biggest performance boost, followed by
“+queue updating”. The result highlights the significance of maintaining and utilizing hard negatives for contrastive learning, instead of employing a large negative queue as existing methods.
On hard positives. We then consider the individual impact from hard positives by fixing the rest setting of sMoCo, such as screening hard negatives  $(\lambda_{\mathcal{Q}} = 0.95)$  and updating  $\mathcal{Q}$  with  $\lambda_U = 0.4$ . Precisely, four utilization, including "1-pos" (employing only one positive like MoCoV1/2), "3-pos" (with three positives), "1-easy" (picking up the most similar/easy positive like SimCLR), and "1-hard" (the proposed), are considered, while their comparison is illustrated in Fig. 4.
Figure 4: Comparison among four different ways of utilizing positives in sMoCo.
Notably, applying more positives (i.e., "3-pos") performs competitively than cases with only one positive (either "1-pos" or "1-easy"). One reason could be the variety of positives helps in avoiding the model collapsing (that is, positives become very similar to anchors). Additionally, focusing on the hardest one achieves the best result, as shown in "1-hard", as it forces the model to pay attention to the positive far away from the anchor.
On hard negatives. Next we fix to use "1-hard" and evaluate the impact from negatives (by testing different  $\lambda_{Q}$ , in line with the analysis presented in Section 3.4). Note that with a large  $\lambda_{Q}$ , more negatives are included in the loss calculation. In particular, with  $\lambda_{Q} = 100\%$ , sMoCo applies all negatives (same as MoCoV1/2/FT).
Table 7: Performance comparison in terms of hard negatives as a function of  $\lambda_{\mathcal{Q}}$ .
λQ	100%	99%	97%	95%	90%
KB0.1	34.5	34.7	35.2	36.1	35.5
KB0.3	43.5	43.6	43.6	44.2	43.8
Table 7 shows the improvement by utilizing hard negatives, from which the model with the full queue  $(\lambda_{Q} = 100\%)$  achieves the worst result.
Additionally, the best performance is observed with  $\lambda_{\mathcal{Q}} = 95\%$ , approximately 2000 negatives. This findings clearly suggest that it is unnecessary to have a large number of negatives for a performance gain, not to mention its computational cost with a huge queue.
On updating negatives. At last, by fixing "1-hard" and  $\lambda_{\mathcal{Q}} = 95\%$ , we investigate the impact of updating negatives via changing  $\lambda_{U}$ . As discussed before, will a small  $\lambda_{U}(= 0)$ , sMoCo adopts positives directly to replace previous negatives like MoCoV1/2; on the other hand, sMoCo behaves similar to SimCLR or MoCoV3 if  $\lambda_{U} = 1$ .
Figure 5: Model accuracy obtained from sMoCo as a function of  $\lambda_U$  via updating  $\mathcal{Q}$ .
Fig. 5 shows the model accuracy based on different settings of  $\lambda_U$ . Results indicate that the proposed combination of positives and anchors (with  $\lambda_U = 0.4$ ) could bring in the performance boost (than that of  $\lambda_U = 0$  or 1). Note that for KBQA, there exists QA pairs requiring same entities and relations. Using previous positives or anchors to replace previous negatives could mislead the model to separate them into different feature spaces. By contrast, the proposed updating maintains a good balance of forming hard but informative negatives.
We present a novel KBQA model that particularly tackles the low-resource (incompleteness) nature of the context knowledge bases (KBs). The proposed model is characterized by a knowledge extension and a sMoCo module, that is motivated by the learn-by-analogy capability of human readers. Precisely, the former extends existing knowledge via producing additional question-answer pairs, which are further utilized by the sMoCo module. The latter appropriately learns informative representations that grouping hard positives and pushing away hard negatives. Empirically, in comparison to existing approaches, the proposed algorithm produces the state-of-the-art performance on the WebQues
tionsSP benchmark, in particular with the extreme incomplete KBs. In future work, we will extend the proposed idea to explore more contrastive behavior of utilizing hard positives and negatives. More importantly, sMoCo is agnostic to the downstream tasks, i.e., we could incorporate it into other applications.
This work was partially supported by the Australian Research Council Discovery Project (DP210101426).
Sheng Bi, Xiya Cheng, Yuan-Fang Li, Yongzhen Wang, and Guilin Qi. 2020. Knowledge-enriched, Type-constrained and Grammar-guided question generation over Knowledge Bases. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2776-2786, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Xinlei Chen and Kaiming He. 2021. Exploring simple Siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750-15758.
Xinlei Chen, Saining Xie, and Kaiming He. 2021. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9640-9649.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. 2020b. Toward subgraph guided Knowledge Graph question generation with Graph Neural Networks. arXiv preprint arXiv:2004.06015.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. 2020. Bootstrap Your Own Latent - a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271-21284. Curran Associates, Inc.
Jiale Han, Bo Cheng, and Xu Wang. 2020. Two-phase hypergraph based reasoning with dynamic relations
for multi-hop KBQA. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3615-3621. International Joint Conferences on Artificial Intelligence Organization.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-9738.
Sen Hu, Lei Zou, and Xinbo Zhang. 2018. A state-transition framework to answer complex questions over Knowledge Base. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2098-2108, Brussels, Belgium. Association for Computational Linguistics.
Apoory Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over Knowledge Graphs using Knowledge Base embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4498-4507, Online. Association for Computational Linguistics.
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on Knowledge Bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2380-2390, Hong Kong, China. Association for Computational Linguistics.
Yawei Sun, Lingling Zhang, Gong Cheng, and Yuzhong Qu. 2020. Sparqa: Skeleton-based semantic parsing for complex questions over Knowledge Bases. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8952-8959.
Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Improving question answering over incomplete KBs with knowledge-aware reader. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4258-4264, Florence, Italy. Association for Computational Linguistics.
Naganand Yadati, Dayanidhi R S, Vaishnavi S, Indira K M, and Srinidhi G. 2021. Knowledge Base Question Answering through recursive hypergraphs. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 448-454, Online. Association for Computational Linguistics.
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for Knowledge Base Question Answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics
(Volume 2: Short Papers), pages 201-206, Berlin, Germany. Association for Computational Linguistics.
Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. 2021. Improving contrastive learning by visualizing feature transformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10306-10315.