SlowGuess/NAACL_2022_OCR huggingface dataset:
When performing CKBC task, we first fetch the template based on the relation of the triple to be complete and fill in the  $\langle subj\rangle$  and let the model predict the missing  $\langle obj\rangle$ . Concretely, the  $\langle obj\rangle$  placeholder is replaced by the mask token corresponding to different pre-trained language models.
B.2 Notation for Knowledge Type
HasSubevent: 0
MadeOf: 1
HasPrerequisite: 2
MotivatedByGoal: 3
AtLocation: 4
CausesDesire: 5
IsA: 6
NotDesires: 7
Desires: 8
CapableOf: 9
PartOf: 10
HasA: 11
UsedFor: 12
ReceivesAction: 13
Causes: 14
HasProperty: 15
In the remainder of this section, we use  $\cup$  to indicate mask union operation upon multiple commonsense relations.
B.3 Stand-alone Fine-tuning
For fine-tuning on commonsense reasoning tasks, we only experiments with BERT-base due and perform hyper-parameter search only in terms of batch size in the range of  $\{8,16,32\}$  and learning rate in the range of  $\{3e^{-5},4e^{-5},5e^{-5}\}$  due to computational budget. We also adopt early stopping based
Model/Task	RTE	COPA	CSQA	SWAG	HellaSWAG	aNLI	CosmosQA
BERT	0 ∪ 6 ∪ 14	5 ∪ 8 ∪ 14	3 ∪ 4 ∪ 8 ∪ 14	1 ∪ 6 ∪ 10 ∪ 11	0 ∪ 3 ∪ 8 ∪ 14	3 ∪ 5 ∪ 8 ∪ 14	3 ∪ 5 ∪ 8
Table 8: Optimal fine-tuning knowledge type combination for BERT-base on commonsense reasoning tasks.
Model/Task	COPA (Dev.)	CSQA	CA	WSC	SM	ARCT1	ARCT2
DistilBERT	1 ∪ 6 ∪ 14	2 ∪ 3 ∪ 13	0 ∪ 1 ∪ 9	6 ∪ 7 ∪ 10	2 ∪ 8 ∪ 13	2 ∪ 3 ∪ 14	1 ∪ 2 ∪ 7
BERT	4 ∪ 11 ∪ 15	1 ∪ 2 ∪ 15	6 ∪ 8 ∪ 12	2 ∪ 9 ∪ 14	6 ∪ 12 ∪ 15	1 ∪ 9 ∪ 10	1 ∪ 5 ∪ 8
RoBERTa	2 ∪ 3 ∪ 8	0 ∪ 2 ∪ 5	0 ∪ 1 ∪ 8	1 ∪ 2 ∪ 4 ∪ 5 ∪ 11	8 ∪ 11 ∪ 12	2 ∪ 5 ∪ 11 ∪ 13	0 ∪ 8 ∪ 11
MPNet	1 ∪ 6 ∪ 8	6 ∪ 12 ∪ 13	2 ∪ 3 ∪ 10	1 ∪ 3 ∪ 4	6 ∪ 10 ∪ 13	2 ∪ 5 ∪ 6	5 ∪ 6 ∪ 7
Table 9: Optimal zero-shot knowledge type combination for each PLM on each commonsense reasoning tasks.
Model	P@1	P@2	P@3	Sparsity	\(l_b - l_t\)	#Param.
BERT-large w/o pruning	15.1	20.9	24.6	0%	-	336M
BERT-large w/ stochastic pruning	22.1	30.1	35.4	~30%	17-24	336M
BERT-large w/ deterministic pruning	69.2	74.1	76.3	~50%	17-24	284M
Table 10: Macro-averaged precision metrics of BERT-large on C-LAMA test set
on accuracy on the development set. The combination achieving highest accuracy is shown in Table 8.
B.4 Zero-shot Learning
In constraint with fine-tuning, zero-shot evaluation is deterministic as long as the model does not involve any stochastic module, thereby averting extensive hyperparameter tuning. Instead we perform exhaustive search over knowledge combinations for each pretrained language model with number of knowledge types in  $\{3,4,5\}$ . The ConceptNet-grounded knowledge type combination achieving highest accuracy is listed in Table 9.
C Extracted Commonsense Triples
Here we present the additional experiment result of extracting novel relaitonal triples based on our specialized relation-specific knowledge models. Applying the pruned DistilBERT-base model to predict missing objects for triples in ConceptNet-100K test set, we obtain commonsense triples deemed to be novel by three human annotators with Flessi's Kappa score  $\kappa$  of 0.65. We further filtered out triples that are included in the training or development set of ConceptNet-100K. Here we show some representative cases categorized by their relations:
CapableOf:
(computer, crash), (computer, communicate)
IsA:
(sex,relationship), (submarine,weapon),
(submarine, vessel)
AtLocation:
(knife, war), (knife, dinner), (crab, dinner)
UsedFor:
(  $stage,fun)$  ， (stage,performance),
(literature, education), (literature, research)
HasA:
(book,index),(book,information)
HasProperty:
(music, loud)
Future work involves using seed triples beyond ConceptNet-100K dataset, e.g., the whole ConceptNet knowledge graph, and mining more novel and plausible commonsense knowledge.
The major limitations of our work lie in how do we choose and combine the representation subspaces/subnetworks in multi-relation scenario. We proposed a simple heuristic (i.e., based on statistics of dataset and union operation upon masks) in the paper and it empirically works well, but more principled and optimal method should be further studied. Another potential limitation is that we limit our scope to commonsense relations only in this paper. We leave other binary relations (e.g., factual relations defined in WikiData) as future work.