SlowGuess/NAACL_2022_OCR huggingface dataset:
Figure 7: Illustration of the zero-shot fusion architecture with parameters. Each colored circle represents expert adapters, except the black circle which denotes KG-Classifier adapter. * indicates the fixed layer.
G Commonsense Reasoning Benchmarks
SocialIQA (SIQA) (Sap et al., 2019b) requires reasoning for emotional and social intelligence in everyday situations. Each QA consists of a context that comes from ATOMIC, a question which is based on the relations in ATOMIC, and 3 answer candidates. It contains 38,000 multiple-choice
questions, which is generated by crowdsourcing.
CommonsenseQA (CSQA) (Talmor et al., 2018) evaluates a broad range of concept-level commonsense reasoning. Each multiple-choice question, answer and distractors are designed by crowdsourcing based on the ConceptNet.
Abductive NLI (a-NLI) (Bhagavatula et al., 2020) asks to infer the most plausible explanation based on the given causal situation to test abductive reasoning in narratives. Each sample consists of the beginning and the end of the story with two possible options to be an explanation for the given situation.
PhysicalIQA (PIQA) (Bisk et al., 2020) requires physical commonsense reasoning to select the most sensible solution for the given goal among the two choices. Its dataset is comprised of over 16,000 training samples, 2K validation samples, and 3K test samples.
HellaSWAG (HSWAG) (Zellers et al., 2019) is an evolved version of SWAG (Zellers et al., 2018), which asks to infer the most proper story based on the given situation. The dataset consists of 70K questions with four answer options.
H Implementation Details
In all our experiments, we use max sequence length 128, batch size 32, weight decay 0.01, adam  $\beta_{1}$  0.9,adam  $\beta_{2}$  0.99,adam epsilon  $1e^{-8}$  ,warm-up proportion 0.05,and margin 1.0.The experiments are conducted split across NVIDIA GeForce 3090 and NVIDIA RTX A5000.
H.1Baselines
The baseline models for STL-PLM and MTL are trained with learning rate  $1e^{-5}$  for single epoch.
H.2 Adapter
For expert adapters, we use learning rate  $8e^{-5}$  after tuning in  $\{5e^{-6}, 8e^{-6}, 1e^{-5}, 5e^{-5}, 8e^{-5}, 1e^{-4}\}$ . For KG-Classifier adapter, we use learning rate  $1e^{-5}$ , batch size 64 for five epochs.
H.3 Zero-shot fusion
After experiment with learning rates  $\{1e^{-5}, 8e^{-5}\}$ , we empirically find that a learning rate of  $1e^{-5}$  works well on zero-shot fusion without/with KGClassifier adapter, respectively. Here, we set the attention drop probability 0.1. As we used extremely smaller subset of the synthetic QA dataset, zero-shot fusions are trained for five epochs.
I Knowledge aggregation of zero-shot fusion
In order to validate the efficacy on knowledge aggregation of zero-shot fusion over the STL, we present the results of each framework with various combination of KGs in Table 9 and Table 10.
Model	KG	a-NLI	CSQA	PIQA	SIQA	WG	Avg.
STL-PLM	AT	71.6	64.0	72.2	63.2	60.5	66.3
CN	67.9	68.5	72.6	54.6	58.6	64.4
WD	68.4	64.7	72.0	53.7	58.6	63.5
WN	67.2	61.4	71.7	53.5	58.9	62.5
MTL	AT, CN	70.5	68.4	72.2	60.1	58.2	65.9
AT, WD	69.9	66.4	72.0	60.1	59.3	65.5
AT, WN	69.1	62.7	71.6	59.1	59.1	64.3
CN, WD	69.6	67.8	72.0	54.3	59.5	64.6
CN, WN	69.8	66.3	71.7	53.8	56.4	63.6
WD, WN	67.5	62.0	71.7	53.7	59.0	62.8
MTL	AT, CN, WD	70.4	66.8	71.5	62.4	61.0	66.4
AT, CN, WN	68.5	65.7	72.1	62.7	59.1	65.6
AT, WD, WN	71.0	65.1	71.1	63.2	60.8	66.2
CN, WD, WN	69.6	67.3	72.5	52.0	57.2	63.7
MTL	AT, CN, WD, WN	69.8	67.1	72.0	61.9	59.3	66.0
Table 9: STL-PLM and MTL performance across five commonsense tasks in various combination of KGs. AT, CN, WD and WN represent ATOMIC, ConceptNet, WikiData and WordNet, respectively. We run our experiment with seed 42.
Model	KG	a-NLI	CSQA	PIQA	SIQA	WG	Avg.
STL-Adapter	AT	71.3	66.5	71.1	64.4	60.3	66.7
CN	70.6	67.2	72.4	55.5	58.7	64.9
WD	66.8	61.6	69.9	51.8	58.5	61.7
WN	67.6	60.0	70.3	54.0	57.0	61.8
AT,CN,WD,WN	71.5	66.7	72.1	64.7	59.0	66.8
zero-shot fusion w/KGC-adapter	AT, CN	71.9	68.1	72.8	65.4	59.7	67.6
AT, WD	71.5	66.3	71.4	65.3	61.2	67.1
AT, WN	72.5	67.5	73.1	66.4	59.5	67.8
CN, WD	70.8	68.1	72.1	55.3	59.3	65.1
CN, WN	71.0	67.6	73.0	54.8	59.1	65.1
WD, WN	67.8	62.6	71.3	52.9	57.1	62.3
zero-shot fusion w/KGC-adapter	AT, CN, WD	72.3	68.0	72.9	66.2	60.5	68.0
AT, CN, WN	72.5	68.7	73.8	66.8	60.4	68.4
AT, WD, WN	71.9	67.6	73.0	66.0	59.7	67.6
CN, WD, WN	69.6	67.6	73.1	53.7	59.5	64.7
zero-shot fusion w/KGC-adapter	AT, CN, WD, WN	72.4	68.3	73.0	66.7	60.9	68.3
Table 10: STL-Adapter and zero-shot fusion w/ KG-C adapter performance across five commonsense tasks in various combination of KGs. AT, CN, WD and WN represent ATOMIC, ConceptNet, WikiData and WordNet, respectively. Whole represents the combination of AT, CN, WD and WN. We run our experiment with seed 42.