SlowGuess/NAACL_2022_OCR huggingface dataset:
Junping Liu†, Shijie Mei†, Xinrong Hu†, Xun Yao†, Jie Yang‡, Yi Guo†  
†School of Computer Science and Artificial Intelligence, Wuhan Textile University  
‡School of Computing and Information Technology, University of Wollongong  
°School of Computer, Data and Mathematical Sciences, Western Sydney University  
{jpliu, hxr, yaoxun}@wtu.edu.cn, jackeymeis1@gmail.com  
jiey@uow.edu.au,  
y.quo@westernsydney.edu.au
Given a context knowledge base (KB) and a corresponding question, the Knowledge Base Question Answering task aims to retrieve correct answer entities from this KB. Despite sophisticated retrieval algorithms, the impact of the low-resource (incomplete) KB is not fully exploited, where contributing components (i.e. key entities and/or relations) may be absent for question answering. To effectively address this problem, we propose a contrastive regularization based method, which is motivated by the learn-by-analogy capability from human readers. Specifically, the proposed work includes two major modules: the knowledge extension and sMoCo module. The former aims at exploiting the latent knowledge from the context KB and generating auxiliary information in the form of question-answer pairs. The later module utilizes those additional pairs and applies the contrastive regularization to learn informative representations, that making hard positive pairs attracted and hard negative pairs separated. Empirically, we achieved the state-of-the-art performance on the WebQuestionsSP dataset and the effectiveness of proposed modules is also evaluated.
The task of Knowledge Base Question Answering (KBQA) refers to answering a question given a background knowledge base (KB). A large number of studies on KBQA can be cast into two mainstream categories: semantic parsing (SP) and information retrieval (IR) based. The former focuses on parsing questions into symbolic logic forms, such as query graph (Hu et al., 2018) and skeleton grammar (Sun et al., 2020), before identifying final answer(s) from the KB. The IR approach, on the other hand, aims to perform semantic matching between topic entities from questions and candidate answers within the KB (Xiong et al., 2019; Sun et al., 2019; Saxena et al., 2020; Yadati et al.,
2021). This paper is on a novel method for information retrieval based KBQA.
Approaches to IR-based KBQA usually follow a three-step process, including question analysis, subgraph reasoning and answer matching. At first, the question analysis is to understand reasoning instructions behind questions for extracting topic entities and involved relations, etc. The second step is performed to retrieve relevant entities and relations (as subgraphs of the context KB) according to reasoning instructions, and further formulate candidate answers. The last step is to identify the best answer by estimating and ranking the semantic-relationship matching score between the given question and candidates.
However, existing IR-based matching pays insufficient attention to the low resource (incompleteness) nature of KB, where contributing components (i.e. neighboring entities, key relations and/or reasoning path) may be absent. The limit or lack of enough information poses challenges for the subsequent effective question answering. Some work have been proposed to utilize auxiliary information, such as extra question-related texts (Sun et al., 2019) and pre-trained KB embeddings (Saxena et al., 2020), which unfortunately could introduce noisy and misleading facts, not to mention the extra computational cost.
Notably, when human performs QA reasoning, one could infer other cases from one similar instance. Even with a new question, experienced readers could still make a guess using similar concepts or facts from their current knowledge, and potentially discriminate candidate answers from different aspects. For instance, given a question of "What state did Al Gore represent?", one could infer from his working experience (i.e. The Tennesean (News)) and/or the graduate university (i.e. Vanderbilt University), to further predict the correct answer of Tennessee.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1085 - 1094 July 10-15, 2022 ©2022 Association for Computational Linguistics
Figure 1: Overview of the proposed learn-by-analogy framework for KBQA, where latent knowledge is exploited and represented as the form of QA pairs. The sMoCo module is then applied to extract informative features for question answering.
Inspired from the aforementioned learn-by-analogy process, we propose a novel KBQA framework as illustrated in Fig. 1, which includes two major modules: knowledge extension and sMoCo. The former aims to expand knowledge from the existing KB to discover latent information. That is, hidden knowledge is exploited and manipulated in a form of question-answering pairs via making use of existing relations and entities (simulating human readers to use similar concepts/facts). The sMoCo adopts the contrastive learning mechanism, with the ultimate aim of capturing discriminative features from correct and misleading question-answering pairs (simulating human's inference skill). Furthermore, different from traditional contrast methods, the proposed sMoCo is particularly designed to utilize hard positive and negative pairs, which further improves the model generalizability.
- To the best of our knowledge, we are the first to advance the contrastive learning-based algorithm to resolve the KBQA task; the proposed method is motivated by the learn-by-analogy capability from human readers;
- Simulating the human capability of utilizing similar facts, the proposed knowledge extension module explores the context KB to capture latent knowledge and further manipulates them as the task-related question-answer pairs;
- A novel contrast algorithm (sMoCo) is further introduced to simulate human inference ca
pability; compared to existing methods, the proposed one mainly relies on hard positive and negative pairs during the training process. Related theoretical analysis is also provided for the discussion of its applicability and sensitivity;
- Experimental evaluations show that the proposed algorithm outperforms state-of-the-arts, in particular with the low-resource cases. We also conduct comprehensive ablation studies to characterize the proposed algorithm, by investigating the impact from the knowledge extension and hard positives/negatives.
2.1 KBQA
Knowledge Base Question Answering (KBQA) is one of the most popular and challenging research topics for machine reading comprehension (MRC). Aiming at determining correct answer(s) given the background knowledge base (KB) and one question, a large amount of research efforts have been put forward to supplement the KBQA task via either semantic parsing (SP) or information retrieval (IR) based strategies. The work from (Hu et al., 2018; Sun et al., 2020), for instance, belongs to the former, which introduced the state-transition and skeleton-based parsing approach to convert the target question into a semantic graph and structural tree, respectively, before querying answers.
Another line of studies aims to retrieve answers by following a more end-to-end training style, that is, to learn representations of the target question and candidate answers. KDReader from (Xiong et al., 2019) performed an attention-based fusion to combine the question and answer features. PullNet (Sun et al., 2019) employed the question-related content as a guideline to extract supporting components (entities and relations from KB). The work of Saxena et al., 2020) utilized the pre-trained KB embeddings. In addition, Han et al. proposed a hypergraph-based reasoning strategy with dynamic relation and entity embeddings (Han et al., 2020). Similarly, RecHyperNet (Yadati et al., 2021) applied recursive hypergraphs to form groups (relations and entities with similar semantic) in the KB. With well-represented features, a matching step is usually followed to identify/rank the best candidate(s).
Despite some promising results from the afore
<sup>1</sup>The source codes is publicly available at https://github.com/JakeyMei/sMoCo
mentioned feature learning and matching, less work has been put forward to explore underlying/hidden knowledge from the context KB. We argue that making full use of those existing but latent knowledge is beneficial to the subsequent QA task.
Contrastive learning (CL) has attracted a lot of attention in the recent several years, which utilizes input data itself as the additional supervision signal for training (Chen et al., 2020a; Grill et al., 2020; Chen and He, 2021; He et al., 2020; Chen et al., 2021; Zhu et al., 2021). Specifically, for an input sample (anchor)  $\boldsymbol{x}_i$  and an encoder  $f(\cdot)$ , the overall contrastive loss is formulated as follows:
$$
\operatorname {s i m} \left(f \left(\boldsymbol {x} _ {i}\right), f \left(\boldsymbol {x} _ {i} ^ {+}\right)\right) \gg \operatorname {s i m} \left(f \left(\boldsymbol {x} _ {i}\right), f \left(\boldsymbol {x} _ {i} ^ {-}\right)\right), \tag {1}
$$
where  $sim(\cdot, \cdot)$  is a user-defined similarity function, and  $\boldsymbol{x}_i^+$  and  $\boldsymbol{x}_i^-$  are contrastive pairs of positive and negative, respectively. The training purpose via the contrastive loss is to form informative features (i.e.  $z_i^{+/ - } = f(\boldsymbol{x}_i)$ ) such that positives stay close to anchors and negatives are pushed away. Yet, existing CL methods fail to differentiate the pair significance, while the proposed method focuses on hard contrastive pairs.
The illustration of our framework is shown in Fig 2. The KBQA task has been formulated as searching for optimal entities, given the question  $\mathbf{q}$  and the external KB (i.e. a set of triples  $(\mathbf{e}_h, \mathbf{r}, \mathbf{e}_t)$  representing head entity, semantic relationship and tail entity, respectively).
3.1 Knowledge extension module
Most existing KBQA work, unfortunately, focuses on the inference accuracy by offering sophisticated reasoning models. They have neglected the low resource nature of KB, from which key reasoning paths could be absent. Note that, experienced human readers could still utilize knowledge from limit but similar circumstances to infer correct answers. As such, this paper argues that it is beneficial to leverage hidden knowledge, from the context KB (even incomplete), for the subsequent QA process. That is the main aim for the proposed knowledge extension module.
Notably, knowledge extension serves the same role as data argumentation. Yet, this knowledge
Figure 2: Overview of the proposed framework for KBQA which consists of four modules. First, the knowledge extension module is introduced to explore hidden knowledge from the context KB and produce new QA pairs. Second, a backbone module is applied to extract latent representations, after which the sMoCo module is employed, utilizing only hard contrastive pairs, to ensure that the similarity (of feature representations) between correct QA pairs is maximized by contrasting to that of inaccurate ones. At last, an overall identification module is utilized to identify the best-matching answer.
extension task is nontrivial. In the domain of image processing, argumentation can be carried out by using randomized operations, such as image rotation, cut-off, and scaling. However, in the context of KB, it is impractical to simply randomize knowledge, which could lead to meaningless or inaccurate information.
Intuitively, we propose to create new KB-relevant QA pair(s) from available triple(s) using an unsupervised manner. By doing so, we can alleviate the aforementioned challenge by not only effectively utilizing existing knowledge facts, but also providing additional samples for training the QA model. In this regard, we implement the knowledge-extension module using a template-oriented strategy<sup>2</sup>.
Figure 3: Example of producing the template-oriented QA pairs.
Given an known triple of  $(e_h,r,e_t)$ , as the example illustrated in Figure 3. The relation of fb: geography. river.origin is associated with a factual description (Desc. [r]), that can be interpreted as [X] originates from
This strategy can also be viewed as the task of question generation. As such, some general generation techniques (Chen et al., 2020b; Bi et al., 2020) may help if complicated questions (with multiple hops) are preferred and we leave the investigation to future work.
[Y] ([X] and [Y] is the placeholder for  $e_h$  and  $e_t$  respectively). Then, we produce a template question “[Wh] +  $[e_h] + \text{Desc.} [r]$ ?” where Desc. [r] is the relation description. The [Wh] component belongs to one of the types (such as who, where, and when), depending on the tag of  $e_t$  (such as person, location, and time).
3.2 Backbone module
We adopt KDReader from (Xiong et al., 2019) as the backbone module. For a given question  $q$ , KDReader firstly retrieves relevant entities from the context KB, according to topic entities  $e_{q}$  from  $q$ , and further formulates entity embeddings  $\mathcal{G}_{e}$ . Secondly, with the question embedding  $Q_{q}$ , a classifier  $C$  is trained via optimizing the following loss function:
$$
\mathcal {L} _ {\text {M a t c h i n g}} = \operatorname {S o f t M a x} \left(C \left(\mathcal {G} _ {\boldsymbol {e}}, Q _ {\boldsymbol {q}}\right)\right). \tag {2}
$$
3.3 sMoCo module
Learning distinguishable features to match correct entities with the given question lies at the heart of KBQA, for which we accordingly propose a new contrastive learning algorithm in this module. Yet, majority contrast methods are characterized by a slow momentum encoder  $(E_{p})$  and an independent queue  $(\mathcal{Q})$  with thousands of negative samples. Despite its simplicity, the existing design poses the following problems: (i) the slow encoder  $E_{p}$  could produce easy positive samples (highly similar to anchors); similarly (ii) the lengthy queue  $\mathcal{Q}$  may contain easy negative samples (highly different to anchors).
Notably, existing methods fail to differentiate easy and hard positives/negatives. We argue that the contribution of easy ones to the contrastive loss is trivial, while the significance from hard ones is the success toward the contrastive learning. As a result, we propose a screening-based Momentum Contrast method (termed sMoCo) via screening hard contrastive pairs.
To illustrate this idea, let  $\mathbf{z}_i$  and  $\mathcal{P}$  represent the  $i$ -th anchor feature and the set of corresponding positives. We propose to select only one hard positive from  $\mathcal{P}$ , say  $\mathbf{z}_i^{h, + }$ , with the lowest similarity to  $\mathbf{z}_i$ . On the other hand, we select hard negatives according to their similarity to  $\mathbf{z}_i$  to form a subset  $\mathcal{I}_{-}$  (with  $\mathcal{I}_{-} \in \mathcal{Q}$ ). More precisely,  $\forall \mathbf{z}_i^{h, - } \in \mathcal{I}_{-}$
and  $\forall \mathbf{z}_n\in \mathcal{Q}$  we have
$$
s \left(\mathbf {z} _ {i} ^ {h, -}\right) > s \left(\mathbf {z} _ {i} ^ {-}\right), \forall \mathbf {z} _ {i} ^ {-} \in \mathcal {Q}, \mathbf {z} _ {i} ^ {-} \notin \mathcal {I} _ {-},
$$
$$
\text {a n d} \sum_ {\mathbf {z} _ {i} ^ {h, -} \in \mathcal {I} _ {-}} s \left(\mathbf {z} _ {i} ^ {h, -}\right) = \lambda_ {\mathcal {Q}} \sum_ {\mathbf {z} _ {n} \in \mathcal {Q}} s \left(\mathbf {z} _ {n}\right), \tag {3}
$$
where  $s(\cdot)$  is the normalized inner product (for simplicity say  $s(\mathbf{z}_i^{h, - }) = \langle \mathbf{z}_i^{h, - },\mathbf{z}_i\rangle$  ), and  $\lambda_{\mathcal{Q}}$  is a screening term. As such, the proposed contrastive loss for  $\mathbf{z}_i$  is formulated as follows:
$$
\mathcal {L} _ {s M o C o} = - \log \frac {e ^ {\langle \mathbf {z} _ {i} ^ {h , +} , \mathbf {z} _ {i} \rangle}}{e ^ {\langle \mathbf {z} _ {i} ^ {h , +} , \mathbf {z} _ {i} \rangle} + \sum_ {j \in \mathcal {I} _ {-}} e ^ {\langle \mathbf {z} _ {j} , \mathbf {z} _ {i} \rangle}} \tag {4}
$$
In addition to utilizing the proposed hard positive  $\mathbf{z}_i^{h, + }$  and hard negatives  $\mathcal{I}_{-}$ , we further consider a liner combination of updating  $\mathbf{z}_n$  from the negative queue  $\mathcal{Q}$  via:
$$
\mathbf {z} _ {n} = \lambda_ {U} \mathbf {z} _ {i} + (1 - \lambda_ {U}) \mathbf {z} _ {i} ^ {h, +}, \tag {5}
$$
where  $\lambda_U\in [0,1]$  is a hyperparameter.
Comparison with existing contrastive methods. The mainstream approaches for implementing contrastive learning include SimCLR (Chen et al., 2020a), MoCoV1/2 (He et al., 2020), MoCoFT (Zhu et al., 2021), MoCoV3 (Chen et al., 2021), SimSiam (Chen and He, 2021). Although our approach shares similar ideas of utilizing positives and/or negatives as the above, our algorithm is different in the following: (i) the majority existing work neglects the difference of easy and hard contrastive pairs; for instance, MoCoV1/2/3 and SimSiam only considers one positive, while SimCLR selects the easiest one (with the highest similarity with anchors) with the presence of multiple positives; meanwhile, they take all available negatives into account. By contrast, ours is particularly interested in hard positives and negatives; (ii) MoCoFT applies feature-level transformation (FT) to generate hard positives/negatives, which in fact has no direct impact for/from encoders; another drawback then lies in the sensitivity of those FT hyperparameters, that further reduces its generalization capability; Similarly, SimSiam is also significantly impacted by the hyperparameters to produce positives, as no negative exists; and (iii) for updating the negative queue, the proposed update can be cast as a linear combination of SimCLR and MoCos, where other methods (such as MoCoFT) require one entire queue for each single anchor, that is computationally expensive. The aforementioned difference is also summarized in Table 1.
Table 1: Comparison between sMoCo and existing methods, where  $|b|$ ,  $|\mathcal{Q}|$ , 1-E and 1-H represents the batch size, queue size, one easy and one hard positive, respectively.
P	Q	Update Q
SimCLR	1-E	|b|-2	zn = zi
MoCoV1/2	1	|Q|	zn = zi+
MoCoFT	1-H	|Q|	zn = λzi + (1 - λ)Q
MoCoV3	1	|b|-2	zn = zi
SimSiam	1	×	×
sMoCo	1-H	I_{n,-}	zn = λUzi + (1 - λU)zh,+
We present the sensitivity analysis hereafter to justify the choice of the screening parameter (or  $\lambda_{\mathcal{Q}}$  from Eq. (3)). Note that the traditional contrastive loss, for the  $i$ -th anchor, is defined as follows
$$
\mathcal {L} _ {c} = - \log \frac {e ^ {\langle \mathbf {z} _ {i} ^ {h , +} , \mathbf {z} _ {i} \rangle}}{e ^ {\langle \mathbf {z} _ {i} ^ {h , +} , \mathbf {z} _ {i} \rangle} + \sum_ {j} e ^ {\langle \mathbf {z} _ {j} ^ {-} , \mathbf {z} _ {i} \rangle}}. \tag {6}
$$
Denoting  $s_i^+ = e^{\langle \mathbf{z}_i^{h, + },\mathbf{z}_i\rangle}$  and  $s_i^{-} = \sum_je^{\langle \mathbf{z}_j^{-},\mathbf{z}_i\rangle}$  Eq. (6) can be written more concisely as
$$
\mathcal {L} _ {c} = - \log \frac {s _ {i} ^ {+}}{s _ {i} ^ {+} + s _ {i} ^ {-}} = \log \left(1 + \frac {s _ {i} ^ {-}}{s _ {i} ^ {+}}\right). \tag {7}
$$
We then quantify the screened proportion as  $\epsilon \in [0,1)$  for negative samples, and note that
$$
\epsilon = 1 - \lambda_ {\mathcal {Q}}.
$$
By removing a given proportion of negative samples, it is equivalent to reduce  $s_i^-$  to  $\tilde{s}_i^-$  by a suitable value of  $\epsilon$  such that  $\tilde{s}_i^- = (1 - \epsilon)s_i^-$ . Hence the perturbed loss function  $\tilde{\mathcal{L}}_c$  becomes
$$
\tilde {\mathcal {L}} _ {c} (\epsilon | r _ {i}) = \log (1 + (1 - \epsilon) r _ {i}), \tag {8}
$$
where  $r_i = \frac{s_i^-}{s_i^+}$ . The perturbed loss is caused by  $\epsilon$  and  $\tilde{\mathcal{L}}_c(0|r_i)$  recovers the original loss in Eq. (7). Therefore the sensitivity of the function in Eq. (8) at  $(\epsilon = 0)$  determines the change to the loss by introducing screening. The first order Taylor expansion shows that:
$$
\tilde {\mathcal {L}} _ {c} (\epsilon | r _ {i}) = \tilde {\mathcal {L}} _ {c} (0 | r _ {i}) + \frac {\partial \tilde {\mathcal {L}} _ {c}}{\partial \epsilon} | _ {\epsilon = 0} [ r _ {i} ] \cdot \epsilon + O (| \epsilon | ^ {2}),
$$
where  $O(|\epsilon|^2)$  is the negligible higher order term when  $\epsilon$  is approximate zero. Note that  $\tilde{\mathcal{L}}_c(0|r_i) = \mathcal{L}_c$ , i.e. the original loss. Plugging in the partial
derivatives evaluated at  $\epsilon = 0$  with  $r_i\left(\frac{\partial\tilde{L}_c}{\partial\epsilon}\big|_{\epsilon = 0}[r_i]\right)$  i.e.
$$
\frac {- r _ {i}}{r _ {i} + 1},
$$
$$
\tilde {\mathcal {L}} _ {c} (\epsilon | r _ {i}) \approx \mathcal {L} _ {c} - \frac {r _ {i} \epsilon}{r _ {i} + 1}. \tag {9}
$$
Remark 1. The perturbation (or the difference in loss aroused by screening) is approximately  $\frac{-r_i\epsilon}{r_i + 1}$ , which is an apparent reduction to the original loss  $\mathcal{L}_c$  as all variables in Eq. (9) are positive.
Furthermore, it is worth noting that normalized similarities are between  $[-1,1]$ , that is,  $\langle \mathbf{z}_i^{h, + },\mathbf{z}_i\rangle \to 1$ , and  $\langle \mathbf{z}_j^-, \mathbf{z}_i\rangle \to -1$  at the convergence of the optimization, and hence
$$
r _ {i} \rightarrow e ^ {- 2} | \mathcal {Q} | > \frac {| \mathcal {Q} |}{9},
$$
where  $|\mathcal{Q}|$  is the size of the negative queue or number of negative samples. In many contrast methods (He et al., 2020; Zhu et al., 2021; Chen et al., 2021),  $Q$  consists of thousands of negatives, which leads to  $r_i \gg 1$ .
At last, combining  $\frac{\partial\tilde{\mathcal{L}}_c}{\partial\epsilon} |_{\epsilon = 0}[r_i]$  with the value of  $r_i$ , we obtain
$$
\frac {\partial \tilde {L} _ {c}}{\partial \epsilon} | _ {\epsilon = 0} [ r _ {i} ] \approx - 1,
$$
and the perturbation is approximately  $-\epsilon$  at convergence.
Remark 2. The perturbation indicates that if we screen out a small proportion of negative samples, say  $1 - \lambda_{\mathcal{Q}}$  (or  $\epsilon$ ), the function loss is reduced by approximately that much. In other words, the sample screening connects to the loss reduction directly. Although we discussed only ideal situation where the contrast reaches maximum, similar result exists in expectation sense as well because a large value of  $|\mathcal{Q}|$  outweighs less contrast in negative and positive similarities.
Notably, the above perturbation analysis is performed at 0 requiring  $\epsilon$  to be small or even close to zero (say  $\epsilon \in [0.01, 0.1]$ ), and hence satisfies this condition. The loss function change caused by screening will eventually transfer to gradients to the model parameters. Our results indicate the scaling factor in front of the final gradients. Nonetheless, when  $r_i$  is too large, it saturates the loss function to the "plateau" stage where gradients are small and hence the aforementioned analysis results hold.
3.5 Overall identification
With the matching and contrastive loss from the Backbone and sMoCo modules, we propose the following joint loss for the overall identification:
$$
\mathcal {L} _ {\text {I d e n t i f i c a t i o n}} = \mathcal {L} _ {\text {M a t c h i n g}} + \lambda_ {s M o C o} \mathcal {L} _ {s M o C o}, \tag {10}
$$
where  $\lambda_{sMoCo}$  is a penalty term<sup>3</sup>.
The WebQuestionsSP datasets is employed (Yih et al., 2016) with a total of 4737 questions, that are answerable through the Freebase KB. This entire KB consists of 601,145 distinct entities, 568 different relations, 1,261,849 unique triples. Followed by the work from (Xiong et al., 2019; Saxena et al., 2020), the low-resource KB settings have been constructed by down-sampling a percentage of facts in the background KB (we randomly retain a triple with probability of 0.1, 0.3, and 0.5). Accordingly, the resultant datasets are referred as KB0.1, KB0.3, KB0.5 and KBFull (original), respectively, and Table 2 shows the statistics of those four datasets.
Table 2: Summary of four adopted datasets, where  $\# e$ ,  $\# r$  and  $\# triples$  is the averaged number of entities, relations, and triples per question, respectively.
Dataset	#e	#r	#triples
KB0.1	152.5	51.3	184.8
KB0.3	182.3	54.8	567.1
KB0.5	183.5	56.1	837.3
KBFull	191.2	56.7	1484.2