SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3 shows most of the supervised models outperform full-scale PLMs by a large margin, which suggests the inefficacy of directly using PLMs to perform link prediction. However, the subnetworks identified by our pruning procedure can acquire performance on par with or better than the state-of-the-art supervised models, which shows the potential of language models as neural knowledge base that is underestimated by previous studies. Surprisingly, pruned DistilBERT get the highest MRR, outperforming other larger and more advanced PLMs. RoBERTa struggles to predict correct objects, perhaps due to its larger vocabulary size compared to WordPiece (50,265 vs 30,522) and less lexicon overlap (53% vs. 59%) with the dataset.
Two-hop extrapolation. Sometimes, a pair of commonsense relations can be combined to derive a new relational knowledge triple. For example, the two triples  $\langle s1, IsA, o1 \rangle$  and  $\langle s2, HasProperty, o2 \rangle$ , where  $o1$  equals to  $s2$ , can be combined into a new test triple  $\langle s1, HasProperty, o2 \rangle$ . Based on this rule, we construct 5,151 new test triples (absent in
ConceptNet-100K) with HasProperty relation, which allows us to compare the two-hop extrapolation ability of pruned subnetworks with original PLMs. Table 4 shows that pruned subnetworks exhibit significantly better ability of extrapolating from known relational knowledge to novel knowledge, with an average improvement of 23.2/24.8/24.0 in terms of  $\mathrm{P@1 / P@2 / P@3}$ .
Model	P@1	P@2	P@3
DistilBERT-base	10.6	17.8	23.0
DistilBERT-base (pruned)	28.2	36.5	44.6
BERT-base	11.9	18.5	21.9
BERT-base (pruned)	42.9	52.1	58.4
RoBERTa-base	16.8	24.1	28.0
RoBERTa-base (pruned)	21.5	28.7	31.0
MPNet-base	16.6	24.8	29.3
MPNet-base (pruned)	56.2	64.2	67.7
Table 4: Two-hop extrapolation results  $(\%)$
Model	< linen, IsA, ? >	< sing, Causes, ? >
DistilBERT	surname	death
commodity	disease
profession	trouble
DistilBERT(pruned)	cloth	happiness
fabric	joy
garment	peace
Case Study. We present a case study of link prediction examples from both pre-trained and pruned DistilBERT in Table 5. In both examples, the pre-trained DistilBERT makes completely unrelated predictions (e.g., surname, profession, and death) and only two predictions (i.e., commodity and trouble) can be hardly considered as plausible. In contrast, the model pruned for IsA/Causes are specialized in accurately representing these relations and can even produce reasonable answers in addition to the ground-truth.
3.3 Multi-relation Scenario
In this section, we compare the transfer ability of pruned subnetworks against original PLMs in a scenario that implicitly requires knowledge of multiple commonsense relations, i.e., commonsense question answering (CQA).
Stand-alone fine-tuning. We conduct standalone fine-tuning using BERT/DistilBERT and their pruned subnetworks on 6 widely
adopted CQA tasks: RTE (Dagan et al., 2009), COPA (Roemmle et al., 2011), CommonsenseQA (Talmor et al.), SWAG (Zellers et al., 2018), aNLI (Bhagavatula et al., 2020) and CosmosQA (Huang et al., 2019). For each task, we identify the commonsense knowledge that might be required with a simple yet effective heuristic: we obtain the five most frequent relations measured by how many times the subject and object holding certain relation in ConceptNet appear in the context or the answer. Then we take the union of masks for each relation and apply the resultant mask to BERT/DistilBERT as the initialization for fine-tuning. Intuitively, such union operation would preserve all relational knowledge of interest. We repeat training three times with different random seeds for each task using hyperparameters suggested in the original papers. The detail of mask combination for each task is in Appendix B.
Table 5: Top-3 predictions of two triples sampled from CKBC dev set. The predictions are ranked by probability in descending order with ground-truth marked in green and other plausible answers underlined.
Task	BERT	DistilBERT
RTE	69.2±2.3⇒69.8±2.0	61.2±1.2⇒62.1±1.2
COPA	62.4±5.0⇒63.1±4.7	54.0±2.0⇒56.0±2.0
CommonsenseQA	53.1±0.6⇒54.1±0.7	47.9±0.7⇒48.6±0.7
SWAG	73.9±0.3⇒74.2±0.1	70.1±0.3⇒70.4±0.1
aNLI	63.7±0.4⇒64.0±0.4	60.1±0.4⇒60.4±0.4
CosmosQA	61.3±1.0⇒61.8±0.2	56.4±0.8⇒57.2±0.4
Table 6: Stand-alone fine-tuning accuracy (original  $\Rightarrow$  pruned) of BERT and DistilBERT.
Figure 5: Results on COPA with varying portion of data.
Table 8 shows that, when initialized with proper subnetworks, the model can better transfer to commonsense question answering tasks via more relevant prior knowledge. We further analyze the change of performance under the low-resource regime. Figure 5 shows that the pruned BERT exhibits a notable advantage when training data is extremely scarce. As more training data is seen, the benefit of the pruned model is reduced but still significant.
Integrated fine-tuning. We also integrate the pruned subnetworks to QA-GNN (Yasunaga et al.,
Model	COPA-Tra.	COPA-Val.	CSQA	CA	WSC	SM	ARCT1	ARCT2	Avg.	Δ
DistilBERT-base	58.3	60.0	29.6	84.6	53.3	71.6	48.6	50.4	57.0	-
DistilBERT-base (pruned)	61.5	69.0	31.5	89.6	56.9	72.1	53.4	51.6	60.7	+3.7
BERT-base	60.2	54.0	26.5	89.0	57.3	69.7	46.8	50.3	56.7	-
BERT-base (pruned)	63.0	64.0	28.5	91.8	59.0	71.7	50.0	52.0	60.0	+3.3
RoBERTa-base	60.7	59.0	39.9	90.1	61.8	73.1	48.6	53.1	60.7	-
RoBERTa-base (pruned)	65.3	72.0	40.4	93.4	62.9	74.4	53.2	55.1	64.6	+3.9
MPNet-base	66.5	69.0	40.0	94.5	64.3	75.8	52.9	56.7	64.9	-
MPNet-base (pruned)	71.0	74.0	41.7	97.3	66.4	77.5	56.1	57.7	67.7	+3.2
Table 7: Zero-shot accuracy (%) for commonsense question answering. Better results of each pair is in bold.
2021), a state-of-the-art hybrid question answering system in which a PLM and GNN are employed for joint reasoning over text and knowledge graph. We follow their official implementation with the only modification on the PLM part. With the same set of masks as in the stand-alone fine-tuning on CommonsenseQA, the pruned BERT achieves an accuracy of  $60.9\%$  versus  $60.1\%$  of original model, suggesting a generally stronger knowledge transfer ability not only in stand-alone but also in the integrated settings.
Zero-shot learning. We then assess the ability of pruned subnetworks to perform zero-shot commonsense reasoning, a setting where the knowledge relied on to complete the task is solely determined by the model itself. We focus on the following 8 multiple-choice CQA datasets: training set of COPA (COPA-Tra.), validation set of COPA (COPA-Val.), CommonsneseQA, conjunction Acceptability (CA) (Zhou et al., 2020), Winograd Schema Challenge (WSC) (Levesque et al., 2012), SenseMaking (SM) (Wang et al., 2019), ARCT1 (Habernal et al., 2018) and ARCT2 (Niven and Kao, 2019). Each sample in the above datasets can be formulated as  $\{[CLS]$  context [SEP]  $\text{choice}_i$  [SEP] $\}\_{i=1}^N$ , where  $N$  is the number of choices. We compute the plausibility score of each choice using the MLM head. The choice of the maximum plausibility score is chosen as the answer.
Since multiple types of knowledge are typically required for effective commonsense reasoning, We employ the same heuristic used in many-shot learning setting for determining the set of the most important relations for each task as well as the same mask union operation to obtain parameter initialization. It can be observed from Table 7 that the pruned models can actually surpass their full-size counterparts in all tasks considered in our experiments. Our explanation is that knowledge irrelevant to the specific task in the original PLMs hurt
the in-domain zero-shot reasoning capability. It also demonstrates that the most important relational knowledge vary from task to task.
Since the emergence of large pre-trained language models, much work has focused on understanding their internal contextual representations. Most prior work (Shi et al., 2016; Belinkov et al., 2017) pays attention to either using extraneous probing tasks to examine whether certain linguistic properties can be identified from those representations, or ablating the models to observe how behavior changes. More recently, some studies (Goldberg, 2019; Tenney et al., 2019) have shown the existence of linguistic knowledge (e.g., syntax) in various but generally lower layers of pre-trained transformers.
To shed more light on how PLMs memorize abstract knowledge rather than statistical co-occurrence patterns, we extend previous probe (Petroni et al., 2020) on relational knowledge. Specifically, we are concerned with commonsense knowledge that is grounded on ConceptNet relations. Our work differs in that we focus on not only probing but also bringing latent relational knowledge to the surface and unleashing more potential for better relation reasoning.
Another relevant line of research is network pruning (Liu et al., 2019b; Lin et al., 2020) and lottery ticket hypothesis (Frankle and Carbin, 2019; Prasanna et al., 2020; Chen et al., 2020). The former aims at reducing the size of model parameters without compromising accuracy and the latter reveals subnetworks whose initializations made them capable of being trained effectively comparable to the original model. In contrast, we seek to uncover subnetworks in over-parametrized PLMs that specializes on commonsense knowledge tailored for downstream tasks rather than focusing on good global initialization, and achieve good results.
This study investigated specializing PLMs for better relational reasoning via network pruning. In the pilot experiment we find evidence of latent sparse subnetworks capable of representing grounded commonsense relations in various PLMs. Further experiments revealed that such subnetworks possess stronger relational reasoning capability than original PLMs. Our work provides a new vantage point about the internal mechanism as well as practical utilization of relational knowledge in PLMs, opening up avenues to better understanding and adapting pretrained language representations.
Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. In Proceedings of EMNLP, pages 5185-5194.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861-872, Vancouver, Canada.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre-trained BERT networks. In Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020.
Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. *Natural Language Engineering*, 15(Special Issue 04):i-xvii.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Proceedings of AAAI 2018, pages 1811-1818. AAAI Press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019.
Yoav Goldberg. 2019. Assessing bert's syntactic abilities. CoRR, abs/1901.05287.
Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1930-1940.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of EMNLP, pages 2391-2401.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks. In Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, pages 4107-4115.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, May 7-9, 2015, Conference Track Proceedings.
Jason D. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo. 2020. Predicting what you already know helps: Provable self-supervised learning. CoRR, abs/2008.01064.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR'12, pages 552-561. AAAI Press, Rome, Italy.
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1445-1455.
Zhuohan Li, Di He, Fei Tian, Wei Chen, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018. Towards binary-valued gates for robust LSTM training. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3001-3010. PMLR.
Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. 2020. Dynamic model pruning with feedback. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2019b. Rethinking the value of network pruning. In 7th International Conference on Learning Representations, ICLR 2019.
Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of EMNLP, pages 1499-1509, Brussels, Belgium.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. Language models as knowledge bases? EMNLP-IJCNLP 2019, pages 2463-2473.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. 2020. When BERT Plays the Lottery, All Tickets Are Winning. In Proceedings of EMNLP, pages 3208-3229.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning.
Victor Sanh. 2019. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. 2020. A mathematical exploration of why language models help solve downstream tasks. CoRR, abs/2010.03648.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019. End-to-end structure-aware convolutional networks for knowledge base completion. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, Hawaii, January 27 - February 1, 2019, pages 3060-3067.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proceedings of EMNLP, pages 1526-1534, Austin, Texas.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of EMNLP, pages 4222-4235.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2020. Mpnet: Masked and permuted pretraining for language understanding. arXiv preprint arXiv:2004.09297.
Robyn Speer and Catherine Havasi. 2012. Representing general relational knowledge in Concept-Net 5. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3679-3686, Istanbul, Turkey. European Language Resources Association (ELRA).
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. *CommonsenseQA: A question answering challenge targeting commonsense knowledge.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, Volume 1 (Long and Short Papers), pages 4149-4158.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanian Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, May 6-9, 2019.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. CoRR, abs/1606.06357.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. 2020. Interacte: Improving convolution-based knowledge
graph embeddings by increasing feature interactions. In Proceedings of AAAI2020, pages 3009-3016. AAAI Press.
Jesse Vig. 2019. Visualizing attention in transformer-based language representation models. CoRR, abs/1904.02679.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki-data: A free collaborative knowledgebase. Commun. ACM, 57(10):78-85.
Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. Does it make sense? and why? a pilot study for sense making and explanation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020-4026, Florence, Italy.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015, May 7-9, 2015, Conference Track Proceedings.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In North American Chapter of the Association for Computational Linguistics (NAACL).
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of EMNLP, pages 93-104, Brussels, Belgium.
Tianyi Zhang and Tatsunori Hashimoto. 2021. On the inductive bias of masked language modeling: From statistical to syntactic dependencies. CoRR, abs/2104.05694.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020. Masking as an efficient alternative to finetuning for pretrained language models. In Proceedings of EMNLP, pages 2226-2241.
Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pretrained language models. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, February 7-12, 2020, pages 9733-9740. AAAI Press.
A Derivation for Stochastic Pruning
To re-parametrize the discrete binary Bernoulli variable  $m_{i,j}^{l}\sim B(\sigma (g_{i,j}^{l}))$  , denote the approximate differentiable variable as  $\tilde{m}_{i,j}^{l} = \sigma \left(\frac{g_{i,j}^{l} + \log U - \log(1 - U)}{\tau}\right)$  where  $\tau$  is a real-valued temperature value, we have the following derivation holds for arbitrary  $\epsilon \in (0,0.5)$  ..
$$
P (m _ {i, j} ^ {l} = 1) - P (\tilde {m} _ {i, j} ^ {l} \geq 1 - \epsilon) \leq (\frac {\tau}{4}) \log \frac {1}{\epsilon}
$$
Specifically, when temperature  $\tau$  approaches 0,  $\tilde{m}_{i,j}^{l} = m_{i,j}^{l}$
Lemma 1:  $\sigma^{-1}(x) = \log \frac{x}{1 - x}$ .
Lemma 2:  $\frac{\sigma(x) - \sigma(y)}{x - y} \leq \frac{1}{4}$ .
$$
\begin{array}{l} P (\tilde {m} _ {i, j} ^ {l} \geq 1 - \epsilon) \\ = P (\sigma (\frac {g _ {i , j} ^ {l} + \log U - \log (1 - U)}{\tau}) \geq 1 - \epsilon) \\ = P \left(\frac {g _ {i , j} ^ {l} + \log U - \log (1 - U)}{\tau} \geq \log \left(\frac {1}{\epsilon} - 1\right)\right) \\ = P \left(g _ {i, j} ^ {l} - \tau \log \left(\frac {1}{\epsilon} - 1\right) \geq \log \left(\frac {1}{U} - 1\right)\right) \\ = P \left(e ^ {g _ {i, j} ^ {l} - \tau \log \left(\frac {1}{\epsilon} - 1\right)} \geq \frac {1}{U} - 1\right) \\ = P (U \geq \frac {1}{1 + e ^ {g _ {i , j} ^ {l} - \tau \log \left(\frac {1}{\epsilon} - 1\right)}}) \\ = \sigma \left(g _ {i, j} ^ {l} - \tau \log \left(\frac {1}{\epsilon} - 1\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} P (m _ {i, j} ^ {l} = 1) - P (\tilde {m} _ {i, j} ^ {l} \geq 1 - \epsilon) \\ = \sigma \left(g _ {i, j} ^ {l}\right) - \sigma \left(g _ {i, j} ^ {l} - \tau \log \frac {1}{\epsilon} - 1\right) \\ \leq \frac {\tau}{4} \log \left(\frac {1}{\epsilon} - 1\right) \\ \leq \frac {\tau}{4} \log \frac {1}{\epsilon} \\ \end{array}
$$
The process for deriving  $P(m_{i,j}^{l} = 0) - P(\tilde{m}_{i,j}^{l}\leq \epsilon)\leq (\frac{\tau}{4})\log \frac{1}{\epsilon}$  can be analogously obtained.
B Implementaiton Details
B.1 Templates
The templates we used in single-relation scenario for different commonsense relations are defined as follows:
AtLocation: Something you find at  $\langle obj\rangle$  is  $\langle subj\rangle$
CapableOf:  $\langle \text{subj} \rangle$  can  $\langle \text{obj} \rangle$ .
Causes:  $\langle \text{subj}\rangle$  causes  $\langle \text{obj}\rangle$ .
CausesDesire:  $\langle \text{subj} \rangle$  would make you want to  $\langle \text{obj} \rangle$ .
Desires:  $\langle \text{subj} \rangle$  wants to  $\langle \text{obj} \rangle$ .
HasPrerequisite:  $\langle \text{subj} \rangle$  requires  $\langle \text{obj} \rangle$ .
HasProperty:  $\langle \text{subj} \rangle$  can be  $\langle \text{obj} \rangle$ .
HasSubevent: when  $\langle \text{subj} \rangle, \langle \text{obj} \rangle$ .
HasA:  $\langle \text{subj} \rangle$  contains  $\langle \text{obj} \rangle$ .
IsA:  $\langle \text{subj} \rangle$  is a  $\langle \text{obj} \rangle$ .
MadeOf:  $\langle \text{subj} \rangle$  can be made of  $\langle \text{obj} \rangle$ .
MotivatedByGoal: you would  $\langle \text{subj} \rangle$  because  $\langle \text{obj} \rangle$ .
NotDesires:  $\langle \text{subj} \rangle$  does not want  $\langle \text{obj} \rangle$ .
PartOf:  $\langle \text{subj} \rangle$  is part of  $\langle \text{obj} \rangle$ .
ReceivesAction:  $\langle \text{subj} \rangle$  can be  $\langle \text{obj} \rangle$ .
UsedFor:  $\langle \text{subj} \rangle$  may be used for  $\langle \text{obj} \rangle$ .