## 2 Materials and Methods
### Evaluation Metrics  
In this study, we employed the degrees of agreement between labels of a query and labels of similar texts as measures of similar text retrieval accuracy. We focused not on ranking the extracted similar texts but rather on how many of the top-\(K\) similar instances had labels that matched the query. In multilabel machine learning [21], there are three possible states of label agreement: complete match, partial match, and complete mismatch. First, we defined the exact match ratio as a measure of complete match. Here the top-\(K\) instances similar to a query instance \(\big{(}s_{q},Y_{q}\big{)}\) found by search engines in dataset \(\mathcal{D}\) are \(\big{(}s_{k}(\mathcal{D},s_{q}),Z_{k}\big{(}\mathcal{D},s_{q}\big{)}\big{)},k= 1,2,\cdots,\ K\). The multilabel \(Z_{k}\big{(}\mathcal{D},s_{q}\big{)}\) is given by a one-hot vector of \(Z_{k}\big{(}\mathcal{D},s_{q}\big{)}=\big{(}z_{q,k,1},z_{q,k,2},\cdots,z_{q,k,L }\big{)},\ z_{q,k,l}\in\{0,1\},\ 1\leq l\leq L\). Then, the exact match ratio is given by  
\[\mathit{ExactMatchRatio}\big{(}\mathcal{D},\big{(}s_{q},Y_{q}\big{)}\big{)}= \frac{1}{K}\sum_{k=1}^{K}1\big{(}Y_{q}=Z_{k}\big{(}\mathcal{D},s_{q}\big{)} \big{)}, \tag{4}\]  
where \(1(x)\) is the indicator function: \(1(x)=\{1\ \text{if}\ x\ \text{is}\ \text{true},\ 0\ \text{otherwise}\}\).  
In some of the extracted similar texts, one label matches that of the query, but the other label is different (e.g., surgical vs. dental anesthesia accidents). Therefore, we defined the Hamming score in the following equation as a measure of partial match:  
\[HammingScore\big{(}\mathcal{D},\big{(}s_{q},Y_{q}\big{)}\big{)}=\frac{1}{K}\sum_ {k=1}^{K}\frac{\big{|}Y_{q}\cap Z_{k}\big{(}\mathcal{D},s_{q}\big{)}\big{|}}{ \big{|}Y_{q}\cup Z_{k}\big{(}\mathcal{D},s_{q}\big{)}\big{|}}. \tag{5}\]  
The Hamming score is sometimes referred to simply as accuracy in multilabel classification tasks; we also defined the Hamming loss as a measure of mismatch:  
\[HammingLoss\big{(}\mathcal{D},\big{(}s_{q},Y_{q}\big{)}\big{)}=\frac{1}{KL} \sum_{k=1}^{K}\sum_{l=1}^{L}1\big{(}y_{q,l}\
eq z_{q,k,l}\big{)}. \tag{6}\]  
Finally, we averaged the above evaluation measures defined for individual queries for all queries. Let \(\mathcal{D}^{train}\) be a training dataset and \(\mathcal{D}^{test}\) be a testing dataset, and we defined the averaged metrics with the following equations:  
\[AveragedExactMatchRatio\Big{(}\mathcal{D}^{train},\mathcal{D}^{test} \Big{)} =\frac{1}{|\mathcal{D}^{test}|}\sum_{(s_{q},Y_{q})\in\mathcal{D} ^{test}}ExactMatchRatio\Big{(}\mathcal{D}^{train},\big{(}s_{q},Y_{q}\big{)} \Big{)}, \tag{7}\] \[AveragedHammingScore\Big{(}\mathcal{D}^{train},\mathcal{D}^{test} \Big{)} =\frac{1}{|\mathcal{D}^{test}|}\sum_{(s_{q},Y_{q})\in\mathcal{D} ^{test}}HammingScore\Big{(}\mathcal{D}^{train},\big{(}s_{q},Y_{q}\big{)}\Big{)},\] (8) \[AveragedHammingLoss\Big{(}\mathcal{D}^{train},\mathcal{D}^{test} \Big{)} =\frac{1}{|\mathcal{D}^{test}|}\sum_{(s_{q},Y_{q})\in\mathcal{D} ^{test}}HammingLoss\Big{(}\mathcal{D}^{train},\big{(}s_{q},Y_{q}\big{)}\Big{)}. \tag{9}\]  
Larger averaged exact match ratios and Hamming scores and smaller averaged Hamming losses are better. In the results discussed below, \(K\), the number of extracted similar instances, was fixed at 10.