SlowGuess/NAACL_2022_OCR huggingface dataset:
We implement RCL based on Transformers package<sup>4</sup> (Wolf et al., 2020), where we use Bert-base-uncased as backbone. We set the maximum input length to 96 for SemEval2010 Task8 and 80 for FewRel. The epoch is set to 6 for training and we use an Adam optimizer (Kingma and Ba, 2014) with a batch size of 32. The learning rate is set to 5e-5 and the weight decay is set to 0.1. Same as SimCSE (Gao et al., 2021), the dropout probability of data augmentation is set to 0.1. The temperature  $\tau$  is set to 0.05 across two datasets and we set  $\alpha$  to 0.4 and 0.6 on SemEval2010 Task8 and FewRel respectively. The hidden size of fully-connected layer is set to 1536. All experiments are conducted by using a GeForce RTX 3090Ti with 24 GB memory.
C Standard F1 score Results on Two Datasets
To comprehensively compare the performance of baselines and our method, we report the standard F1 score results in Table 5 and Table 6. We follow the setting in the previous work (Simon et al., 2019) to convert pseudo labels predicted by clustering to relation labels. In each cluster, the relation label
Table 4: Statistics of two datasets. "Avg.len." means the average length of sentences.
SemEval2010 Task8
Model	P	R	F1
CNN	46.83	48.53	47.52
Att-BiLSTM	47.46	51.87	49.41
Supervised RSN	32.87	40.12	36.07
MTB-BERT	54.42	58.93	56.29
MTB-RoBERTa	53.28	55.09	54.13
ZS-BERT	34.0	41.88	37.25
RCL w/o RC	61.41	60.91	61.13
RCL	79.6	79.91	79.72
Table 5: Experimental results(%) on SemEval2010 Task8 in terms of standard precision, recall, F1 score.
FewRel
Model	m=5	m=10	m=15	m=30	m=40	Avg.
CNN	83.88	68.19	67.97	49.3	45.51	62.97
Att-BiLSTM	88.11	80.10	77.19	59.29	54.5	71.84
Supervised RSN	83.28	49.43	44.24	23.22	14.68	42.97
MTB-BERT	92.82	84.76	83.43	68.99	65.46	79.09
MTB-RoBERTa	94.57	86.25	85.21	69.66	66.49	80.43
ZS-BERT	82.83	77.57	70.93	53.68	55.75	68.15
RCL w/o RC	78.63	72.6	67.63	49.71	41.5	62.02
RCL	94.08	87.46	85.95	75.2	72.64	83.07
Table 6: Experimental results(%) on FewRel in terms of standard F1 score.
with the largest proportion among the cluster is assigned to all samples as the prediction label. From the standard F1 score results, we can see that the performance of our method is much better than the baselines, especially in SemEval2010 Task8. For FewRel, we find that the improvement of RCL gets larger when  $m$  is larger, especially when  $m = 40$ . It is obvious that it becomes more difficult for prediction since the number of unseen relations increases leading to more seriously similar problems. It proves that our method can effectively mitigate two types of similar problems.
The effects of hyper-parameters are shown in Table 7 and Table 8. For hyper-parameter  $\alpha$ , we vary  $\alpha$  in the list of [0.0, 0.2, 0.4, 0.6, 0.8, 1.0] and find RCL can achieve the best performance when  $\alpha$  is set to 0.4 on SemEval2010 Task8 or 0.6 on Fewrel. For temperature hyper-parameter  $\tau$ , we vary  $\tau$  in the list of [0.001, 0.01, 0.05, 0.1, 1.0] and find  $\tau = 0.05$  can achieve the best performance across two datasets.
E Different Clustering methods for Zero-shot Prediction
Figure 5 shows the results of different clustering methods for RCL, including Mini-Batch KMeans (Sculley, 2010), Gaussian Mixture Model
α	0.0	0.2	0.4	0.6	0.8	1.0
SemEval2010 Task8	45.71	65.45	68.02	67.08	66.00	64.46
FewRel	81.08	82.32	82.94	84.52	83.35	83.11
Table 7: Experimental results(%) with different  $\alpha$  in term of  ${\mathrm{B}}^{3}\mathrm{\;F}1$  score. For FewRel,we report the results on the unseen relation number  $m = {10}$  .
τ	0.001	0.01	0.05	0.1	1.0
SemEval2010 Task8	44.99	47.18	68.02	61.85	42.23
FewRel	82.23	83.51	84.52	83.11	77.29
Table 8: Experimental results  $(\%)$  with different temperatures over two datasets in term of  $\mathrm{B}^3$  F1 score. For FewRel, we report the results on the unseen relation number  $m = 10$
(GMM), Hierarchical Agglomerative Clustering (HAC), Birch (Zhang et al., 1996), K-Means. We can find that the performance of K-Means is much better than other clustering methods on two datasets. Moreover, Mini-Batch K-Means still outperforms MTB-BERT on SemEval2010 Task8, even its performance is worse than other clustering methods, showing the effectiveness of our model.
Figure 5: Different clustering methods for our proposed RCL model on two datasets. For FewRel, the number of unseen relations is set to 10.
To intuitively show how RCL helps to solve two types of similar problems (similar relations and similar entities), we conduct some case studies on two datasets. As shown in Figure 6, it is clear to see that RCL effectively solves these two problems under the multi-task learning structure. Specifically, RCL can better represent two sentences which have similar relations or similar entities, and then make their euclidean distance closer to the cluster corresponding to their ground truth.
SemEval 2010 Task8	Sentence	Label	Cluster Center	Euclidean Distance
ZS-BERT	MTB-BERT	RCL
Similar Relations	The envelope contained an important intelligence discovery of the war.	Content-Container	Member-Collection	12.08	9.92	8.55
Content-Container	12.18	12.42	8.36
The kitchen holds patient drinks and snacks.	Content-Container	Member-Collection	10.41	8.68	8.52
Content-Container	11.71	10.66	8.41
Similar Entities	Group1: China has tested Barack Obama early in his presidency, with a flotilla of naval vessels surrounding and harassing a US spy ship in the South China Sea.	Member-Collection	Member-Collection	11.59	7.02	6.83
Instrument-Agency	11.83	12.41	8.30
Group1: Until 1864 vessels in the service of certain UK public offices defaced the Red Ensign with the badge of their office.	Instrument-Agency	Member-Collection	11.84	8.62	8.20
Instrument-Agency	13.74	11.02	7.70
Group2: The puppy was inside a sealed garbage bag lying in vomit and near death.	Content-Container	Content-Container	9.65	7.83	7.00
Entity-Origin	16.97	13.69	8.93
Group2: The puppy was born in a barn where Layla made a soft, bed out of hay in an empty horse stall.	Entity-Origin	Content-Container	13.49	9.29	10.14
Entity-Origin	15.49	11.78	9.37
FewRel	Sentence	Label	Cluster Center	Euclidean Distance
ZS-BERT	MTB-BERT	RCL
Similar Relations	The Doctor tries to restore the universe with the help of River and the alternative universe versions of his companions Amy Pond (Karen Gillan) and Rory Williams (Arthur Darvill).	part_of	member_of	16.55	26.13	28.76
part_of	17.90	26.42	26.06
Later in the game, she joins Snake in rescuing Dr Marv, but dies when Jaeger (as Gray Fox in Metal Gear D) destroys the bridge she is on.	part_of	member_of	16.10	25.63	31.34
part_of	16.85	26.10	27.19
Similar Entities	Group1: In May 2015, Vienna hosted the Eurovision Song Contest following Austria's victory in the 2014 contest.	follows	follows	13.57	7.52	11.41
part_of	16.11	26.01	26.01
Group1: Thus, the song was succeeded as Romanian representative at the 2002 Contest by Monica Anghel & Marcel Pavel with "Tell Me Why".	part_of	follows	14.89	25.46	28.22
part_of	18.11	26.54	27.78
Group2: On 1 September 1939, the Second World War began with the German Invasion of Poland, and two days later the United Kingdom declared war on Germany.	part_of	part_of	15.16	22.98	21.81
follows	18.64	30.77	28.90
Group2: During the War of 1812, Rolette, like many other French-Canadian Fur Traders in the Old Northwest, was an active supporter of the British Empire against the United States.	follows	part_of	15.15	24.63	25.00
follows	18.13	25.28	23.54
Figure 6: Case study of similar relations and similar entities on two datasets. "Euclidean Distance" is the euclidean distance between the relation representation of input sentence and the cluster center of the relation. The target entities of input sentence are marked in orange.