# Appendix D

# Preprocessing Statistics

In this Appendix, we present the full results of our preprocessing approaches, as explained in Chapter 5.

## D.1 Preprocessing the VSI dataset

<table><caption>TABLE D.1: Unique Entries per content source before and after preprocessing</caption><thead><tr><th>Unique entries per Content Source</th><th>Raw</th><th>After preprocessing</th><th>Percentage</th></tr></thead><tbody><tr><td>Title</td><td>24037</td><td>16222</td><td>67.49%</td></tr><tr><td>Abstract</td><td>19939</td><td>16974</td><td>85.13%</td></tr><tr><td>Full text</td><td>26346</td><td>22535</td><td>85.53%</td></tr><tr><td>Translated Title</td><td>23829</td><td>16359</td><td>68.65%</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>4712</td><td>4050</td><td>85.95%</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>19798</td><td>16833</td><td>85.02%</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>16473</td><td>14359</td><td>87.17%</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>24447</td><td>20880</td><td>85.41%</td></tr></tbody></table>

<table><caption>TABLE D.2: Token Length statistics after preprocessing the VSI dataset</caption><thead><tr><th>Content Source</th><th>Size</th><th>Mean</th><th>STD</th><th>Min</th><th>25%</th><th>50%</th><th>75%</th><th>max</th></tr></thead><tbody><tr><td>Title</td><td>16222</td><td>12.464</td><td>29.867</td><td>1</td><td>8</td><td>11</td><td>15</td><td>1565</td></tr><tr><td>Abstract</td><td>16974</td><td>50.549</td><td>73.982</td><td>1</td><td>19</td><td>27</td><td>46</td><td>1537</td></tr><tr><td>Full text</td><td>22535</td><td>1097.756</td><td>5763.482</td><td>1</td><td>187</td><td>411</td><td>800</td><td>240318</td></tr><tr><td>Translated Title</td><td>16359</td><td>12.821</td><td>30.943</td><td>1</td><td>8</td><td>11</td><td>15</td><td>1565</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>4050</td><td>34.919</td><td>33.765</td><td>1</td><td>18</td><td>25</td><td>39</td><td>537</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>16833</td><td>35.768</td><td>44.700</td><td>1</td><td>17</td><td>25</td><td>38</td><td>1518</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>14359</td><td>115.655</td><td>429.369</td><td>1</td><td>26</td><td>52</td><td>114</td><td>40700</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>20880</td><td>217.711</td><td>614.478</td><td>1</td><td>32</td><td>71</td><td>201</td><td>40700</td></tr></tbody></table>

<table><caption>TABLE D.3: Label Distribution after preprocessing the VSI dataset</caption><thead><tr><th>Content Source</th><th>Irrelevant</th><th>Relevant</th><th>Size</th><th>% Irrelevant</th><th>% Relevant</th></tr></thead><tbody><tr><td>Title</td><td>13917</td><td>2305</td><td>16222</td><td>85.8%</td><td>14.2%</td></tr><tr><td>Abstract</td><td>14625</td><td>2349</td><td>16974</td><td>86.2%</td><td>13.8%</td></tr><tr><td>Full text</td><td>19794</td><td>2741</td><td>22535</td><td>87.8%</td><td>12.2%</td></tr><tr><td>Translated Title</td><td>14084</td><td>2275</td><td>16359</td><td>86.1%</td><td>13.9%</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>2962</td><td>1088</td><td>4050</td><td>73.1%</td><td>26.9%</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>14553</td><td>2280</td><td>16833</td><td>86.5%</td><td>13.5%</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>12130</td><td>2229</td><td>14359</td><td>84.5%</td><td>15.5%</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>18344</td><td>2536</td><td>20880</td><td>87.9%</td><td>12.1%</td></tr></tbody></table>
<figure><img src="image_75.png" /></figure>
# Appendix E

## Training Setup

### E.1 GPU Specifications

<table><caption>TABLE E.1: Specifications of Lab-IA Nodes</caption><thead><tr><td>Node Names</td><td>CPU</td><td>RAM</td><td>GPU</td></tr></thead><tbody><tr><td>n[1-5]</td><td>2 x Intel Xeon E5-2620 v4 8 cores / 16 threads @ 2.40 GHz (Haswell)</td><td>128 GiB</td><td>2 x NVIDIA RTX A6000 with 48 GiB GDDR6 (PCIe)</td></tr><tr><td>n[51-55]</td><td>2 x Intel Xeon Gold 5120 14 cores / 28 threads @ 2.2GHz (Skylake)</td><td>192 GiB</td><td>3 x NVIDIA Tesla V100 with 32 GiB of RAM (PCIe)</td></tr><tr><td>n[101-102]</td><td>2 x Intel Xeon Gold 6148 20 cores / 40 threads @ 2.4 GHz (Skylake)</td><td>384 GiB</td><td>4 x NVIDIA Tesla V100 with 32 GiB of RAM (NVLink)</td></tr></tbody></table>
<table><caption>TABLE E.2: Split Statistics for Unbalanced Fine-tuning</caption><thead><tr><th>Content Source</th><th>Train-Size</th><th>Positives</th><th>Negatives</th><th>Dev-Size</th><th>Positives</th><th>Negatives</th><th>Test-Size</th><th>Positives</th><th>Negatives</th></tr></thead><tbody><tr><td>Title</td><td>12980</td><td>1845</td><td>11135</td><td>1621</td><td>230</td><td>1391</td><td>1621</td><td>230</td><td>1391</td></tr><tr><td>Abstract</td><td>13582</td><td>1881</td><td>11701</td><td>1696</td><td>234</td><td>1462</td><td>1696</td><td>234</td><td>1462</td></tr><tr><td>Full text</td><td>18031</td><td>2193</td><td>15838</td><td>2252</td><td>274</td><td>1978</td><td>2252</td><td>274</td><td>1978</td></tr><tr><td>Translated Title</td><td>13091</td><td>1821</td><td>11270</td><td>1634</td><td>227</td><td>1407</td><td>1634</td><td>227</td><td>1407</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>3242</td><td>872</td><td>2370</td><td>404</td><td>108</td><td>296</td><td>404</td><td>108</td><td>296</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>13469</td><td>1826</td><td>11643</td><td>1682</td><td>227</td><td>1455</td><td>1682</td><td>227</td><td>1455</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>11491</td><td>1785</td><td>9706</td><td>1434</td><td>222</td><td>1212</td><td>1434</td><td>222</td><td>1212</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>16706</td><td>2030</td><td>14676</td><td>2087</td><td>253</td><td>1834</td><td>2087</td><td>253</td><td>1834</td></tr></tbody></table>

<table><caption>TABLE E.3: Split Statistics for Balanced Fine-tuning</caption><thead><tr><th>Content Source</th><th>Train-Size</th><th>Positives</th><th>Negatives</th><th>Dev-Size</th><th>Positives</th><th>Negatives</th><th>Test-Size</th><th>Positives</th><th>Negatives</th></tr></thead><tbody><tr><td>Title</td><td>12976</td><td>6488</td><td>6488</td><td>1622</td><td>811</td><td>811</td><td>1621</td><td>230</td><td>1391</td></tr><tr><td>Abstract</td><td>13578</td><td>6789</td><td>6789</td><td>1696</td><td>848</td><td>848</td><td>1696</td><td>234</td><td>1462</td></tr><tr><td>Full text</td><td>18028</td><td>9014</td><td>9014</td><td>2252</td><td>1126</td><td>1126</td><td>2252</td><td>274</td><td>1978</td></tr><tr><td>Translated Title</td><td>13086</td><td>6543</td><td>6543</td><td>1634</td><td>817</td><td>817</td><td>1634</td><td>227</td><td>1407</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>3240</td><td>1620</td><td>1620</td><td>404</td><td>202</td><td>202</td><td>404</td><td>108</td><td>296</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>13466</td><td>6733</td><td>6733</td><td>1682</td><td>841</td><td>841</td><td>1682</td><td>227</td><td>1455</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>11486</td><td>5743</td><td>5743</td><td>1434</td><td>717</td><td>717</td><td>1434</td><td>222</td><td>1212</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>16704</td><td>8352</td><td>8352</td><td>2088</td><td>1044</td><td>1044</td><td>2087</td><td>253</td><td>1834</td></tr></tbody></table>

## E.2 Statistics for Splits

<table><caption>TABLE E.4: Split Statistics for PET 50</caption><thead><tr><th>Content Source</th><th>Train-Size</th><th>Positives</th><th>Negatives</th><th>Dev-Size</th><th>Positives</th><th>Negatives</th><th>Test-Size</th><th>Positives</th><th>Negatives</th><th>Unlabeled</th></tr></thead><tbody><tr><td>Title</td><td>100</td><td>47</td><td>53</td><td>1622</td><td>811</td><td>811</td><td>1621</td><td>230</td><td>1391</td><td>12876</td></tr><tr><td>Abstract</td><td>100</td><td>55</td><td>45</td><td>1696</td><td>848</td><td>848</td><td>1696</td><td>234</td><td>1462</td><td>13478</td></tr><tr><td>Full text</td><td>100</td><td>53</td><td>47</td><td>2252</td><td>1126</td><td>1126</td><td>2252</td><td>274</td><td>1978</td><td>17928</td></tr><tr><td>Translated Title</td><td>100</td><td>56</td><td>44</td><td>1634</td><td>817</td><td>817</td><td>1634</td><td>227</td><td>1407</td><td>12986</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>100</td><td>42</td><td>58</td><td>404</td><td>202</td><td>202</td><td>404</td><td>108</td><td>296</td><td>3140</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>100</td><td>59</td><td>41</td><td>1682</td><td>841</td><td>841</td><td>1682</td><td>227</td><td>1455</td><td>13366</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>100</td><td>50</td><td>50</td><td>1434</td><td>717</td><td>717</td><td>1434</td><td>222</td><td>1212</td><td>11386</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>100</td><td>51</td><td>49</td><td>2088</td><td>1044</td><td>1044</td><td>2087</td><td>253</td><td>1834</td><td>16604</td></tr></tbody></table>
<table><caption>TABLE E.5: Split Statistics for PET 100</caption><thead><tr><th>Content Source</th><th>Train-Size</th><th>Positives</th><th>Negatives</th><th>Dev-Size</th><th>Positives</th><th>Negatives</th><th>Test-Size</th><th>Positives</th><th>Negatives</th><th>Unlabeled</th></tr></thead><tbody><tr><td>Title</td><td>200</td><td>102</td><td>98</td><td>1622</td><td>811</td><td>811</td><td>1621</td><td>230</td><td>1391</td><td>12776</td></tr><tr><td>Abstract</td><td>200</td><td>97</td><td>103</td><td>1696</td><td>848</td><td>848</td><td>1696</td><td>234</td><td>1462</td><td>13378</td></tr><tr><td>Full text</td><td>200</td><td>109</td><td>91</td><td>2252</td><td>1126</td><td>1126</td><td>2252</td><td>274</td><td>1978</td><td>17828</td></tr><tr><td>Translated Title</td><td>200</td><td>100</td><td>100</td><td>1634</td><td>817</td><td>817</td><td>1634</td><td>227</td><td>1407</td><td>12886</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>200</td><td>78</td><td>122</td><td>404</td><td>202</td><td>202</td><td>404</td><td>108</td><td>296</td><td>3040</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>200</td><td>115</td><td>85</td><td>1682</td><td>841</td><td>841</td><td>1682</td><td>227</td><td>1455</td><td>13266</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>200</td><td>105</td><td>95</td><td>1434</td><td>717</td><td>717</td><td>1434</td><td>222</td><td>1212</td><td>11286</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>200</td><td>96</td><td>104</td><td>2088</td><td>1044</td><td>1044</td><td>2087</td><td>253</td><td>1834</td><td>16504</td></tr></tbody></table>

<table><caption>TABLE E.6: Split Statistics for PET 200</caption><thead><tr><th>Content Source</th><th>Train-Size</th><th>Positives</th><th>Negatives</th><th>Dev-Size</th><th>Positives</th><th>Negatives</th><th>Test-Size</th><th>Positives</th><th>Negatives</th><th>Unlabeled</th></tr></thead><tbody><tr><td>Title</td><td>400</td><td>201</td><td>199</td><td>1622</td><td>811</td><td>811</td><td>1621</td><td>230</td><td>1391</td><td>12576</td></tr><tr><td>Abstract</td><td>400</td><td>206</td><td>194</td><td>1696</td><td>848</td><td>848</td><td>1696</td><td>234</td><td>1462</td><td>13178</td></tr><tr><td>Full text</td><td>400</td><td>226</td><td>174</td><td>2252</td><td>1126</td><td>1126</td><td>2252</td><td>274</td><td>1978</td><td>17628</td></tr><tr><td>Translated Title</td><td>400</td><td>203</td><td>197</td><td>1634</td><td>817</td><td>817</td><td>1634</td><td>227</td><td>1407</td><td>12686</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>400</td><td>170</td><td>230</td><td>404</td><td>202</td><td>202</td><td>404</td><td>108</td><td>296</td><td>2840</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>400</td><td>206</td><td>194</td><td>1682</td><td>841</td><td>841</td><td>1682</td><td>227</td><td>1455</td><td>13066</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>400</td><td>200</td><td>200</td><td>1434</td><td>717</td><td>717</td><td>1434</td><td>222</td><td>1212</td><td>11086</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>400</td><td>184</td><td>216</td><td>2088</td><td>1044</td><td>1044</td><td>2087</td><td>253</td><td>1834</td><td>16304</td></tr></tbody></table>

<table><caption>TABLE E.7: Split Statistics for PET 500</caption><thead><tr><th>Content Source</th><th>Train-Size</th><th>Positives</th><th>Negatives</th><th>Dev-Size</th><th>Positives</th><th>Negatives</th><th>Test-Size</th><th>Positives</th><th>Negatives</th><th>Unlabeled</th></tr></thead><tbody><tr><td>Title</td><td>1000</td><td>481</td><td>519</td><td>1622</td><td>811</td><td>811</td><td>1621</td><td>230</td><td>1391</td><td>11976</td></tr><tr><td>Abstract</td><td>1000</td><td>512</td><td>488</td><td>1696</td><td>848</td><td>848</td><td>1696</td><td>234</td><td>1462</td><td>12578</td></tr><tr><td>Full text</td><td>1000</td><td>513</td><td>487</td><td>2252</td><td>1126</td><td>1126</td><td>2252</td><td>274</td><td>1978</td><td>17028</td></tr><tr><td>Translated Title</td><td>1000</td><td>482</td><td>518</td><td>1634</td><td>817</td><td>817</td><td>1634</td><td>227</td><td>1407</td><td>12086</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>1000</td><td>481</td><td>519</td><td>404</td><td>202</td><td>202</td><td>404</td><td>108</td><td>296</td><td>2240</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>1000</td><td>524</td><td>476</td><td>1682</td><td>841</td><td>841</td><td>1682</td><td>227</td><td>1455</td><td>12466</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>1000</td><td>499</td><td>501</td><td>1434</td><td>717</td><td>717</td><td>1434</td><td>222</td><td>1212</td><td>10486</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>1000</td><td>492</td><td>508</td><td>2088</td><td>1044</td><td>1044</td><td>2087</td><td>253</td><td>1834</td><td>15704</td></tr></tbody></table>

<table><caption>TABLE E.8: Split Statistics for PET 1000</caption><thead><tr><th>Content Source</th><th>Train-Size</th><th>Positives</th><th>Negatives</th><th>Dev-Size</th><th>Positives</th><th>Negatives</th><th>Test-Size</th><th>Positives</th><th>Negatives</th><th>Unlabeled</th></tr></thead><tbody><tr><td>Title</td><td>2000</td><td>956</td><td>1044</td><td>1622</td><td>811</td><td>811</td><td>1621</td><td>230</td><td>1391</td><td>10976</td></tr><tr><td>Abstract</td><td>2000</td><td>1000</td><td>1000</td><td>1696</td><td>848</td><td>848</td><td>1696</td><td>234</td><td>1462</td><td>11578</td></tr><tr><td>Full text</td><td>2000</td><td>1002</td><td>998</td><td>2252</td><td>1126</td><td>1126</td><td>2252</td><td>274</td><td>1978</td><td>16028</td></tr><tr><td>Translated Title</td><td>2000</td><td>975</td><td>1025</td><td>1634</td><td>817</td><td>817</td><td>1634</td><td>227</td><td>1407</td><td>11086</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>2000</td><td>987</td><td>1013</td><td>404</td><td>202</td><td>202</td><td>404</td><td>108</td><td>296</td><td>1240</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>2000</td><td>1026</td><td>974</td><td>1682</td><td>841</td><td>841</td><td>1682</td><td>227</td><td>1455</td><td>11466</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>2000</td><td>963</td><td>1037</td><td>1434</td><td>717</td><td>717</td><td>1434</td><td>222</td><td>1212</td><td>9486</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>2000</td><td>996</td><td>1004</td><td>2088</td><td>1044</td><td>1044</td><td>2087</td><td>253</td><td>1834</td><td>14704</td></tr></tbody></table>
<figure><img src="image_76.png" /></figure>
# Appendix F

## Training Results

In this Appendix we present the numerical results and metrics from our training experiments.

### F.1 VSI Optimal Epochs

<table><caption>TABLE F.1: Optimal epochs for Unbalanced Fine-tuning</caption><thead><tr><th>Model/Content Source</th><th>Title</th><th>Abstract</th><th>Full text</th><th>Translated Title</th><th>Phrases with (Abstract)</th><th>Phrases with (Abstract)</th><th>Phrases with (Full text)</th><th>Phrases with (Full text)</th></tr><tr><th></th><th></th><th></th><th></th><th></th><th>Keywords</th><th>Keywords + O.C</th><th>Keywords</th><th>Keywords + O.C</th></tr></thead><tbody><tr><td>BERT-base</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>BiBert</td><td>4</td><td>2</td><td>4</td><td>4</td><td>2</td><td>2</td><td>4</td><td>10</td></tr><tr><td>Bio-Link-BERT</td><td>7</td><td>10</td><td>2</td><td>3</td><td>4</td><td>6</td><td>10</td><td>4</td></tr><tr><td>SciBERT</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>9</td></tr><tr><td>RoBERTa-base</td><td>8</td><td>7</td><td>7</td><td>4</td><td>4</td><td>9</td><td>3</td><td>4</td></tr><tr><td>XLM-RoBERTa</td><td>5</td><td>10</td><td>6</td><td>8</td><td>4</td><td>9</td><td>9</td><td>6</td></tr></tbody></table>
<table><caption>TABLE F.2: Optimal epochs for Balanced Fine-tuning</caption><thead><tr><th>Model/Content Source</th><th>Title</th><th>Abstract</th><th>Full text</th><th>Translated Title</th><th>Phrases with (Abstract)</th><th>Phrases with (Abstract)</th><th>Phrases with O.C</th><th>Phrases with (Full text)</th><th>Phrases with Keywords + O.C (Full text)</th></tr></thead><tbody><tr><td>BERT-base</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>3</td><td>1</td><td>1</td></tr><tr><td>mBERT</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Bio-Link-BERT</td><td>1</td><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>SciBERT</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>RoBERTa-base</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td><td>1</td></tr><tr><td>XLM-RoBERTa</td><td>3</td><td>1</td><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td></tr></tbody></table>

<table><caption>TABLE F.3: Optimal epochs for PET 50</caption><thead><tr><th>Model/Content Source</th><th>Title</th><th>Abstract</th><th>Full text</th><th>Translated Title</th><th>Phrases with (Abstract)</th><th>Phrases with (Abstract)</th><th>Phrases with O.C</th><th>Phrases with (Full text)</th><th>Phrases with Keywords + O.C (Full text)</th></tr></thead><tbody><tr><td>BERT-base</td><td>3</td><td>1</td><td>2</td><td>3</td><td>4</td><td>1</td><td>3</td><td>2</td><td>2</td></tr><tr><td>mBERT</td><td>1</td><td>2</td><td>2</td><td>1</td><td>3</td><td>1</td><td>2</td><td>2</td><td>2</td></tr><tr><td>Bio-Link-BERT</td><td>1</td><td>1</td><td>3</td><td>2</td><td>2</td><td>1</td><td>3</td><td>3</td><td>3</td></tr><tr><td>SciBERT</td><td>2</td><td>1</td><td>2</td><td>5</td><td>3</td><td>2</td><td>1</td><td>2</td><td>2</td></tr><tr><td>RoBERTa-base</td><td>4</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>3</td><td>2</td><td>2</td></tr><tr><td>XLM-RoBERTa</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td><td>2</td><td>2</td><td>1</td><td>1</td></tr></tbody></table>

<table><caption>TABLE F.4: Optimal epochs for PET 100</caption><thead><tr><th>Model/Content Source</th><th>Title</th><th>Abstract</th><th>Full text</th><th>Translated Title</th><th>Phrases with (Abstract)</th><th>Phrases with (Abstract)</th><th>Phrases with O.C</th><th>Phrases with (Full text)</th><th>Phrases with Keywords + O.C (Full text)</th></tr></thead><tbody><tr><td>BERT-base</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>mBERT</td><td>2</td><td>1</td><td>1</td><td>1</td><td>4</td><td>3</td><td>2</td><td>1</td><td>1</td></tr><tr><td>Bio-Link-BERT</td><td>2</td><td>4</td><td>1</td><td>4</td><td>2</td><td>3</td><td>1</td><td>1</td><td>1</td></tr><tr><td>SciBERT</td><td>2</td><td>2</td><td>3</td><td>1</td><td>3</td><td>1</td><td>1</td><td>3</td><td>1</td></tr><tr><td>RoBERTa-base</td><td>1</td><td>1</td><td>1</td><td>2</td><td>3</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>XLM-RoBERTa</td><td>3</td><td>1</td><td>1</td><td>4</td><td>3</td><td>1</td><td>1</td><td>2</td><td>2</td></tr></tbody></table>

<table><caption>TABLE F.5: Optimal epochs for PET 200</caption><thead><tr><th>Model/Content Source</th><th>Title</th><th>Abstract</th><th>Full text</th><th>Translated Title</th><th>Phrases with (Abstract)</th><th>Phrases with (Abstract)</th><th>Phrases with O.C</th><th>Phrases with (Full text)</th><th>Phrases with Keywords + O.C (Full text)</th></tr></thead><tbody><tr><td>BERT-base</td><td>4</td><td>3</td><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td><td>1</td><td>2</td></tr><tr><td>mBERT</td><td>2</td><td>1</td><td>1</td><td>3</td><td>2</td><td>1</td><td>2</td><td>2</td><td>2</td></tr><tr><td>Bio-Link-BERT</td><td>3</td><td>4</td><td>4</td><td>2</td><td>3</td><td>3</td><td>4</td><td>4</td><td>4</td></tr><tr><td>SciBERT</td><td>3</td><td>3</td><td>2</td><td>1</td><td>1</td><td>4</td><td>1</td><td>2</td><td>2</td></tr><tr><td>RoBERTa-base</td><td>2</td><td>4</td><td>2</td><td>2</td><td>1</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>XLM-RoBERTa</td><td>3</td><td>3</td><td>1</td><td>5</td><td>1</td><td>4</td><td>1</td><td>1</td><td>1</td></tr></tbody></table>

<table><caption>TABLE F.6: Optimal epochs for PET 500</caption><thead><tr><th>Model/Content Source</th><th>Title</th><th>Abstract</th><th>Full text</th><th>Translated Title</th><th>Phrases with (Abstract)</th><th>Phrases with (Abstract)</th><th>Phrases with O.C</th><th>Phrases with (Full text)</th><th>Phrases with Keywords + O.C (Full text)</th></tr></thead><tbody><tr><td>BERT-base</td><td>4</td><td>1</td><td>3</td><td>2</td><td>1</td><td>3</td><td>2</td><td>3</td><td>3</td></tr><tr><td>mBERT</td><td>3</td><td>3</td><td>3</td><td>1</td><td>1</td><td>3</td><td>1</td><td>3</td><td>3</td></tr><tr><td>Bio-Link-BERT</td><td>3</td><td>4</td><td>2</td><td>3</td><td>1</td><td>3</td><td>3</td><td>3</td><td>3</td></tr><tr><td>SciBERT</td><td>3</td><td>2</td><td>1</td><td>1</td><td>1</td><td>2</td><td>4</td><td>2</td><td>2</td></tr><tr><td>RoBERTa-base</td><td>5</td><td>1</td><td>4</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td><td>2</td></tr><tr><td>XLM-RoBERTa</td><td>2</td><td>2</td><td>3</td><td>2</td><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td></tr></tbody></table>

<table><caption>TABLE F.7: Optimal epochs for PET 1000</caption><thead><tr><th>Model/Content Source</th><th>Title</th><th>Abstract</th><th>Full text</th><th>Translated Title</th><th>Phrases with (Abstract)</th><th>Phrases with (Abstract)</th><th>Phrases with O.C</th><th>Phrases with (Full text)</th><th>Phrases with Keywords + O.C (Full text)</th></tr></thead><tbody><tr><td>BERT-base</td><td>1</td><td>1</td><td>3</td><td>1</td><td>1</td><td>2</td><td>1</td><td>1</td><td>1</td></tr><tr><td>mBERT</td><td>1</td><td>1</td><td>5</td><td>2</td><td>1</td><td>3</td><td>1</td><td>1</td><td>1</td></tr><tr><td>Bio-Link-BERT</td><td>2</td><td>3</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td><td>4</td><td>4</td></tr><tr><td>SciBERT</td><td>2</td><td>2</td><td>2</td><td>2</td><td>1</td><td>1</td><td>1</td><td>1</td><td>1</td></tr><tr><td>RoBERTa-base</td><td>1</td><td>5</td><td>1</td><td>1</td><td>1</td><td>4</td><td>3</td><td>1</td><td>1</td></tr><tr><td>XLM-RoBERTa</td><td>1</td><td>2</td><td>5</td><td>1</td><td>1</td><td>5</td><td>1</td><td>3</td><td>3</td></tr></tbody></table>
F.2 VSI Metrics at Optimal Epochs
<table><caption>TABLE F.8: BERT-base $F_2$ on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>68.22%</td><td>64.97%</td><td>49.6%</td><td>38.3%</td><td>65.7%</td><td>71.14%</td><td>73.92%</td><td>Title</td></tr><tr><td>Abstract</td><td>61.15%</td><td>67.6%</td><td>62.74%</td><td>32.01%</td><td>61.64%</td><td>66.78%</td><td>74.35%</td><td>Abstract</td></tr><tr><td>Full text</td><td>68.94%</td><td>69.76%</td><td>53.02%</td><td>70.37%</td><td>73.16%</td><td>81.35%</td><td>83.1%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>69.91%</td><td>70.72%</td><td>78.96%</td><td>75.88%</td><td>76.32%</td><td>74.07%</td><td>74.43%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>65.22%</td><td>71.22%</td><td>47.74%</td><td>49.12%</td><td>44.06%</td><td>62.44%</td><td>65.59%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>64.58%</td><td>67.59%</td><td>69.47%</td><td>76.91%</td><td>70.52%</td><td>77.68%</td><td>80.16%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>63.96%</td><td>64.55%</td><td>52.23%</td><td>67.21%</td><td>75.84%</td><td>71.83%</td><td>71.21%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>62.25%</td><td>70.31%</td><td>70.28%</td><td>70.16%</td><td>70.95%</td><td>74.63%</td><td>75.47%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>

<table><caption>TABLE F.9: mBERT $F_2$ on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>60.05%</td><td>68.5%</td><td>53.03%</td><td>67.66%</td><td>67.81%</td><td>81.06%</td><td>82.44%</td><td>Title</td></tr><tr><td>Abstract</td><td>0%</td><td>71.31%</td><td>51.25%</td><td>48.59%</td><td>69.07%</td><td>77.38%</td><td>79.71%</td><td>Abstract</td></tr><tr><td>Full text</td><td>61.29%</td><td>75.24%</td><td>66.17%</td><td>75.67%</td><td>83.29%</td><td>84.72%</td><td>82.75%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>0%</td><td>75.57%</td><td>68.61%</td><td>79.5%</td><td>78.36%</td><td>80.32%</td><td>82.87%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>73.48%</td><td>77.15%</td><td>19.74%</td><td>55.67%</td><td>54.88%</td><td>72.77%</td><td>77.48%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>65.49%</td><td>69.45%</td><td>64.76%</td><td>79.35%</td><td>81.28%</td><td>79.34%</td><td>81.64%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>61.53%</td><td>76.07%</td><td>33.46%</td><td>71.9%</td><td>77.79%</td><td>78.15%</td><td>80.25%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>0%</td><td>67.81%</td><td>66.93%</td><td>68.66%</td><td>70.16%</td><td>79.26%</td><td>82.29%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>
<table><caption>TABLE F.10: Bio-Link-BERT $F_2$ on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>65.93%</td><td>79.97%</td><td>79.11%</td><td>74.01%</td><td>77.01%</td><td>81.94%</td><td>81.2%</td><td>Title</td></tr><tr><td>Abstract</td><td>41.85%</td><td>74.63%</td><td>22.3%</td><td>54.82%</td><td>65.14%</td><td>73.95%</td><td>76.57%</td><td>Abstract</td></tr><tr><td>Full text</td><td>69.93%</td><td>75.49%</td><td>66.94%</td><td>73.6%</td><td>76.35%</td><td>81.94%</td><td>83.25%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>68.61%</td><td>74.59%</td><td>83.77%</td><td>81.31%</td><td>80.02%</td><td>78.83%</td><td>80.65%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>70.29%</td><td>77.67%</td><td>58.07%</td><td>35.71%</td><td>59.58%</td><td>73.34%</td><td>72.89%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>53.88%</td><td>74.05%</td><td>70.33%</td><td>81.95%</td><td>77.02%</td><td>81.2%</td><td>79.35%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>0%</td><td>75.78%</td><td>55.44%</td><td>68.61%</td><td>75.76%</td><td>77.37%</td><td>80.19%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>61.97%</td><td>73.53%</td><td>63.54%</td><td>67.28%</td><td>71.73%</td><td>79.5%</td><td>79.74%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>

<table><caption>TABLE F.11: SciBERT $F_2$ on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>64.18%</td><td>69.04%</td><td>58.89%</td><td>60.75%</td><td>71.78%</td><td>75.91%</td><td>80.37%</td><td>Title</td></tr><tr><td>Abstract</td><td>57.89%</td><td>64.95%</td><td>58.02%</td><td>48.16%</td><td>67.4%</td><td>78.08%</td><td>75.47%</td><td>Abstract</td></tr><tr><td>Full text</td><td>68.18%</td><td>70.13%</td><td>59.93%</td><td>73.05%</td><td>71.87%</td><td>82.43%</td><td>83.95%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>70.89%</td><td>73.72%</td><td>78.28%</td><td>78.12%</td><td>76.74%</td><td>74.26%</td><td>78.27%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>53.6%</td><td>71.43%</td><td>55.61%</td><td>32.45%</td><td>40.86%</td><td>72.84%</td><td>67.11%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>64.44%</td><td>70.34%</td><td>74.67%</td><td>78.99%</td><td>77.94%</td><td>78.04%</td><td>78.93%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>64.38%</td><td>71.96%</td><td>55.79%</td><td>73.47%</td><td>70.7%</td><td>71.55%</td><td>75.03%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>54.55%</td><td>61.84%</td><td>73.97%</td><td>75.8%</td><td>75.86%</td><td>76.82%</td><td>80.13%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>
<table><caption>TABLE F.12: RoBERTa-base $F_2$ on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>0%</td><td>77.14%</td><td>48.3%</td><td>36.17%</td><td>57.09%</td><td>77.73%</td><td>76.28%</td><td>Title</td></tr><tr><td>Abstract</td><td>0%</td><td>79.77%</td><td>31.98%</td><td>35.32%</td><td>49.85%</td><td>68.98%</td><td>79.66%</td><td>Abstract</td></tr><tr><td>Full text</td><td>64.15%</td><td>79.41%</td><td>42.48%</td><td>73.41%</td><td>79.01%</td><td>82.72%</td><td>83.18%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>64.66%</td><td>74.07%</td><td>83.54%</td><td>71.96%</td><td>71.55%</td><td>80.25%</td><td>81.02%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>67.61%</td><td>68.03%</td><td>31.28%</td><td>55.11%</td><td>54.27%</td><td>75.48%</td><td>82.78%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>0%</td><td>72.03%</td><td>57.81%</td><td>77.08%</td><td>73.51%</td><td>77.25%</td><td>80.51%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>0%</td><td>77.08%</td><td>46.61%</td><td>70.47%</td><td>70.85%</td><td>69.86%</td><td>74.53%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>0%</td><td>71.84%</td><td>67.83%</td><td>59.86%</td><td>69.82%</td><td>73.4%</td><td>78.52%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>

<table><caption>TABLE F.13: XLM-RoBERTa $F_2$ on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>0%</td><td>70.92%</td><td>22.25%</td><td>62.33%</td><td>70.07%</td><td>80.11%</td><td>83.12%</td><td>Title</td></tr><tr><td>Abstract</td><td>48.66%</td><td>75.74%</td><td>53.67%</td><td>63.15%</td><td>72.72%</td><td>78.69%</td><td>82.34%</td><td>Abstract</td></tr><tr><td>Full text</td><td>0%</td><td>83.62%</td><td>61.73%</td><td>82.62%</td><td>80.51%</td><td>85.64%</td><td>85.73%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>48.99%</td><td>80.16%</td><td>82.89%</td><td>80.27%</td><td>81.22%</td><td>78.36%</td><td>79.91%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>0%</td><td>76.44%</td><td>38.38%</td><td>58%</td><td>52.36%</td><td>78.04%</td><td>82.85%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>0%</td><td>77.82%</td><td>75.58%</td><td>81.94%</td><td>78.55%</td><td>80.79%</td><td>81.13%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>48.65%</td><td>74.94%</td><td>43.41%</td><td>73.3%</td><td>80.04%</td><td>79.59%</td><td>79%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>59.01%</td><td>63.35%</td><td>67.21%</td><td>69.72%</td><td>77.95%</td><td>82.49%</td><td>80.51%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>
<table><caption>TABLE F.14: BERT-base AUC on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>87.58%</td><td>84.33%</td><td>60.55%</td><td>66.45%</td><td>68.61%</td><td>73.41%</td><td>78.43%</td><td>Title</td></tr><tr><td>Abstract</td><td>84.6%</td><td>84.01%</td><td>62.84%</td><td>63.3%</td><td>68.22%</td><td>70.49%</td><td>77.93%</td><td>Abstract</td></tr><tr><td>Full text</td><td>90.23%</td><td>87.46%</td><td>60.5%</td><td>66.04%</td><td>72.51%</td><td>79.66%</td><td>85.09%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>88.8%</td><td>87.35%</td><td>61.03%</td><td>67.21%</td><td>73.89%</td><td>80.89%</td><td>82.94%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>76.56%</td><td>74.04%</td><td>58.04%</td><td>57.61%</td><td>62.62%</td><td>63.03%</td><td>61.13%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>85.87%</td><td>84.02%</td><td>64.03%</td><td>67.59%</td><td>69.42%</td><td>76.22%</td><td>80.2%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>86.38%</td><td>83.49%</td><td>64.86%</td><td>69.48%</td><td>70.67%</td><td>74.13%</td><td>78.18%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>86.47%</td><td>85.65%</td><td>63%</td><td>68.14%</td><td>70.01%</td><td>75.19%</td><td>79.16%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>

<table><caption>TABLE F.15: mBERT AUC on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>89.58%</td><td>85.08%</td><td>62.69%</td><td>70.39%</td><td>71.14%</td><td>80.93%</td><td>83.46%</td><td>Title</td></tr><tr><td>Abstract</td><td>59.16%</td><td>85.66%</td><td>64.84%</td><td>65.77%</td><td>70.88%</td><td>79.8%</td><td>82.81%</td><td>Abstract</td></tr><tr><td>Full text</td><td>92.01%</td><td>90.38%</td><td>62.79%</td><td>71.3%</td><td>80.29%</td><td>86.75%</td><td>88.82%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>63.71%</td><td>86.85%</td><td>66.74%</td><td>70.19%</td><td>73.87%</td><td>81.88%</td><td>85.27%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>82.77%</td><td>80.77%</td><td>59.16%</td><td>59.64%</td><td>65.64%</td><td>68.93%</td><td>67.59%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>86.77%</td><td>85.77%</td><td>61.12%</td><td>69.23%</td><td>74.41%</td><td>81.8%</td><td>84.51%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>87.56%</td><td>87.23%</td><td>62.65%</td><td>70.19%</td><td>73.28%</td><td>80.17%</td><td>82.43%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>66.75%</td><td>86.51%</td><td>61.35%</td><td>68.66%</td><td>74.04%</td><td>81.62%</td><td>83.35%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>
<table><caption>TABLE F.16: Bio-Link-BERT AUC on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>87.92%</td><td>84.18%</td><td>63.14%</td><td>70.27%</td><td>69.32%</td><td>76.65%</td><td>79.8%</td><td>Title</td></tr><tr><td>Abstract</td><td>83.07%</td><td>83.17%</td><td>61.32%</td><td>64.91%</td><td>71.08%</td><td>76.67%</td><td>79.93%</td><td>Abstract</td></tr><tr><td>Full text</td><td>90.3%</td><td>87.55%</td><td>62.36%</td><td>69.01%</td><td>74.07%</td><td>82.41%</td><td>85.16%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>89.25%</td><td>87.81%</td><td>67.29%</td><td>71.45%</td><td>73.33%</td><td>81.22%</td><td>83.03%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>76.2%</td><td>74.15%</td><td>57.97%</td><td>57.97%</td><td>64.61%</td><td>68.24%</td><td>64.57%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>77.04%</td><td>83.74%</td><td>59.48%</td><td>67.77%</td><td>69.91%</td><td>77.49%</td><td>79.01%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>69.36%</td><td>82.23%</td><td>67.15%</td><td>69.43%</td><td>72.1%</td><td>76.44%</td><td>77.69%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>87.07%</td><td>84.14%</td><td>58.85%</td><td>66.96%</td><td>71.86%</td><td>78.92%</td><td>82.23%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>

<table><caption>TABLE F.17: SciBERT AUC on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>85.92%</td><td>85.31%</td><td>60.48%</td><td>66.94%</td><td>71.19%</td><td>77.66%</td><td>80.88%</td><td>Title</td></tr><tr><td>Abstract</td><td>83.87%</td><td>84.7%</td><td>61.99%</td><td>62.9%</td><td>67.7%</td><td>76.74%</td><td>80.41%</td><td>Abstract</td></tr><tr><td>Full text</td><td>89.14%</td><td>88.28%</td><td>60.01%</td><td>65.82%</td><td>71.05%</td><td>79.82%</td><td>84.83%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>90.07%</td><td>88.25%</td><td>63.6%</td><td>71.77%</td><td>72.58%</td><td>78.79%</td><td>83.33%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>60.81%</td><td>74.59%</td><td>56.9%</td><td>57.95%</td><td>63.58%</td><td>65.37%</td><td>62.92%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>85.99%</td><td>83.41%</td><td>62.9%</td><td>68.86%</td><td>70.51%</td><td>78.24%</td><td>79.8%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>87.06%</td><td>85.17%</td><td>63.77%</td><td>68.39%</td><td>70.87%</td><td>75.59%</td><td>79.12%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>85.58%</td><td>83.59%</td><td>64.93%</td><td>69.68%</td><td>73.13%</td><td>78.18%</td><td>81.2%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>
<table><caption>TABLE F.18: RoBERTa-base AUC on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>70.16%</td><td>84.16%</td><td>61.59%</td><td>68.73%</td><td>71.31%</td><td>77.09%</td><td>80.67%</td><td>Title</td></tr><tr><td>Abstract</td><td>61.35%</td><td>85.35%</td><td>61.03%</td><td>60.8%</td><td>66.09%</td><td>73.05%</td><td>81.69%</td><td>Abstract</td></tr><tr><td>Full text</td><td>90.61%</td><td>89.72%</td><td>61.17%</td><td>66.13%</td><td>73.06%</td><td>81.03%</td><td>84.94%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>88.78%</td><td>86.82%</td><td>64.43%</td><td>70.05%</td><td>74.71%</td><td>82.01%</td><td>84.2%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>77.87%</td><td>70.52%</td><td>58.25%</td><td>58.35%</td><td>62.3%</td><td>68.92%</td><td>63.71%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>64.25%</td><td>84.43%</td><td>63.24%</td><td>68.62%</td><td>70%</td><td>77.66%</td><td>81.8%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>67.9%</td><td>82.78%</td><td>64.45%</td><td>70.02%</td><td>71.33%</td><td>74.8%</td><td>80.06%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>65.28%</td><td>85.97%</td><td>65.87%</td><td>69.46%</td><td>72.09%</td><td>77.1%</td><td>81%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>

<table><caption>TABLE F.19: XLM-RoBERTa AUC on Development split</caption><thead><tr><th>Content Source</th><th>Unbalanced Fine-tuning</th><th>Balanced Fine-tuning</th><th>PET 50</th><th>PET 100</th><th>PET 200</th><th>PET 500</th><th>PET 1000</th><th>Content Source</th></tr></thead><tbody><tr><td>Title</td><td>45.39%</td><td>84.85%</td><td>61.54%</td><td>69.62%</td><td>73.51%</td><td>82.48%</td><td>83.55%</td><td>Title</td></tr><tr><td>Abstract</td><td>84.81%</td><td>86.26%</td><td>62.55%</td><td>66.29%</td><td>74.06%</td><td>81.92%</td><td>84.15%</td><td>Abstract</td></tr><tr><td>Full text</td><td>41.19%</td><td>90.57%</td><td>60.9%</td><td>70%</td><td>78.04%</td><td>86.66%</td><td>88.85%</td><td>Full text</td></tr><tr><td>Translated Title</td><td>76.76%</td><td>84.24%</td><td>63.84%</td><td>70.52%</td><td>74.28%</td><td>82.48%</td><td>84.19%</td><td>Translated Title</td></tr><tr><td>Phrases with Keywords (Abstract)</td><td>61.8%</td><td>73.74%</td><td>59.34%</td><td>61.82%</td><td>61.68%</td><td>69.38%</td><td>70.57%</td><td>Phrases with Keywords (Abstract)</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>54.71%</td><td>86.08%</td><td>64.83%</td><td>72.02%</td><td>75.57%</td><td>82.09%</td><td>84.82%</td><td>Phrases with Keywords + O.C (Abstract)</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>64.07%</td><td>78.83%</td><td>68.78%</td><td>72.97%</td><td>74.48%</td><td>79.52%</td><td>80.14%</td><td>Phrases with Keywords (Full text)</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>85.01%</td><td>85.27%</td><td>62.15%</td><td>69.31%</td><td>73.44%</td><td>80.73%</td><td>83.46%</td><td>Phrases with Keywords + O.C (Full text)</td></tr></tbody></table>
<table><caption>TABLE F.20: Development F<sub>2</sub> and AUC of top performing classifiers by BERT Model</caption><thead><tr><th>Model</th><th>Content Source</th><th>Training Method</th><th>Development F<sub>2</sub></th><th>Development AUC</th></tr></thead><tbody><tr><td>BERT-base</td><td>Full text</td><td>PET 500</td><td>81.35%</td><td>79.66%</td></tr><tr><td>BERT-base</td><td>Full text</td><td>PET 1000</td><td>83.1%</td><td>85.09%</td></tr><tr><td>BERT-base</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 1000</td><td>80.16%</td><td>80.2%</td></tr><tr><td>mBERT</td><td>Abstract</td><td>PET 1000</td><td>79.71%</td><td>82.81%</td></tr><tr><td>mBERT</td><td>Full text</td><td>PET 200</td><td>83.29%</td><td>80.29%</td></tr><tr><td>mBERT</td><td>Full text</td><td>PET 500</td><td>84.72%</td><td>86.75%</td></tr><tr><td>mBERT</td><td>Full text</td><td>PET 1000</td><td>82.75%</td><td>88.82%</td></tr><tr><td>mBERT</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 100</td><td>79.35%</td><td>69.23%</td></tr><tr><td>mBERT</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 200</td><td>81.28%</td><td>74.41%</td></tr><tr><td>mBERT</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 500</td><td>79.34%</td><td>81.8%</td></tr><tr><td>mBERT</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 1000</td><td>81.64%</td><td>84.51%</td></tr><tr><td>mBERT</td><td>Phrases with Keywords (Full text)</td><td>PET 1000</td><td>80.25%</td><td>82.43%</td></tr><tr><td>mBERT</td><td>Phrases with Keywords + O.C (Full text)</td><td>PET 500</td><td>79.26%</td><td>81.62%</td></tr><tr><td>mBERT</td><td>Phrases with Keywords + O.C (Full text)</td><td>PET 1000</td><td>82.29%</td><td>83.35%</td></tr><tr><td>mBERT</td><td>Title</td><td>PET 500</td><td>81.06%</td><td>80.93%</td></tr><tr><td>mBERT</td><td>Title</td><td>PET 1000</td><td>82.44%</td><td>83.46%</td></tr><tr><td>mBERT</td><td>Translated Title</td><td>PET 100</td><td>79.5%</td><td>70.19%</td></tr><tr><td>mBERT</td><td>Translated Title</td><td>PET 500</td><td>80.32%</td><td>81.88%</td></tr><tr><td>mBERT</td><td>Translated Title</td><td>PET 1000</td><td>82.87%</td><td>85.27%</td></tr><tr><td>Bio-Link-BERT</td><td>Title</td><td>Balanced Fine-tuning</td><td>79.97%</td><td>84.18%</td></tr><tr><td>Bio-Link-BERT</td><td>Title</td><td>PET 50</td><td>79.11%</td><td>63.14%</td></tr><tr><td>Bio-Link-BERT</td><td>Title</td><td>PET 500</td><td>81.94%</td><td>76.65%</td></tr><tr><td>Bio-Link-BERT</td><td>Title</td><td>PET 1000</td><td>81.2%</td><td>79.8%</td></tr><tr><td>Bio-Link-BERT</td><td>Full text</td><td>PET 500</td><td>81.94%</td><td>82.41%</td></tr><tr><td>Bio-Link-BERT</td><td>Full text</td><td>PET 1000</td><td>83.25%</td><td>85.16%</td></tr><tr><td>Bio-Link-BERT</td><td>Translated Title</td><td>PET 50</td><td>83.77%</td><td>67.29%</td></tr><tr><td>Bio-Link-BERT</td><td>Translated Title</td><td>PET 100</td><td>81.31%</td><td>71.45%</td></tr><tr><td>Bio-Link-BERT</td><td>Translated Title</td><td>PET 200</td><td>80.02%</td><td>73.33%</td></tr><tr><td>Bio-Link-BERT</td><td>Translated Title</td><td>PET 1000</td><td>80.65%</td><td>83.03%</td></tr><tr><td>Bio-Link-BERT</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 100</td><td>81.95%</td><td>67.77%</td></tr><tr><td>Bio-Link-BERT</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 500</td><td>81.2%</td><td>77.49%</td></tr><tr><td>Bio-Link-BERT</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 1000</td><td>79.35%</td><td>79.01%</td></tr><tr><td>Bio-Link-BERT</td><td>Phrases with Keywords (Full text)</td><td>PET 1000</td><td>80.19%</td><td>77.69%</td></tr><tr><td>Bio-Link-BERT</td><td>Phrases with Keywords + O.C (Full text)</td><td>PET 500</td><td>79.5%</td><td>78.92%</td></tr><tr><td>Bio-Link-BERT</td><td>Phrases with Keywords + O.C (Full text)</td><td>PET 1000</td><td>79.74%</td><td>82.23%</td></tr><tr><td>SciBERT</td><td>Title</td><td>PET 1000</td><td>80.37%</td><td>80.88%</td></tr><tr><td>SciBERT</td><td>Full text</td><td>PET 500</td><td>82.43%</td><td>79.82%</td></tr><tr><td>SciBERT</td><td>Full text</td><td>PET 1000</td><td>83.95%</td><td>84.83%</td></tr><tr><td>SciBERT</td><td>Phrases with Keywords + O.C (Full text)</td><td>PET 1000</td><td>80.13%</td><td>81.2%</td></tr><tr><td>RoBERTa-base</td><td>Abstract</td><td>Balanced Fine-tuning</td><td>79.77%</td><td>85.35%</td></tr><tr><td>RoBERTa-base</td><td>Abstract</td><td>PET 1000</td><td>79.66%</td><td>81.69%</td></tr><tr><td>RoBERTa-base</td><td>Full text</td><td>Balanced Fine-tuning</td><td>79.41%</td><td>89.72%</td></tr><tr><td>RoBERTa-base</td><td>Full text</td><td>PET 200</td><td>79.01%</td><td>73.06%</td></tr><tr><td>RoBERTa-base</td><td>Full text</td><td>PET 500</td><td>82.72%</td><td>81.03%</td></tr><tr><td>RoBERTa-base</td><td>Full text</td><td>PET 1000</td><td>83.18%</td><td>84.94%</td></tr><tr><td>RoBERTa-base</td><td>Translated Title</td><td>PET 50</td><td>83.54%</td><td>64.43%</td></tr><tr><td>RoBERTa-base</td><td>Translated Title</td><td>PET 500</td><td>80.25%</td><td>82.01%</td></tr><tr><td>RoBERTa-base</td><td>Translated Title</td><td>PET 1000</td><td>81.02%</td><td>84.2%</td></tr><tr><td>RoBERTa-base</td><td>Phrases with Keywords (Abstract)</td><td>PET 1000</td><td>82.78%</td><td>63.71%</td></tr><tr><td>RoBERTa-base</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 1000</td><td>80.51%</td><td>81.8%</td></tr><tr><td>XLM-RoBERTa</td><td>Title</td><td>PET 500</td><td>80.11%</td><td>82.48%</td></tr><tr><td>XLM-RoBERTa</td><td>Title</td><td>PET 1000</td><td>83.12%</td><td>83.55%</td></tr><tr><td>XLM-RoBERTa</td><td>Abstract</td><td>PET 1000</td><td>82.34%</td><td>84.15%</td></tr><tr><td>XLM-RoBERTa</td><td>Full text</td><td>Balanced Fine-tuning</td><td>83.62%</td><td>90.57%</td></tr><tr><td>XLM-RoBERTa</td><td>Full text</td><td>PET 100</td><td>82.62%</td><td>70%</td></tr><tr><td>XLM-RoBERTa</td><td>Full text</td><td>PET 200</td><td>80.51%</td><td>78.04%</td></tr><tr><td>XLM-RoBERTa</td><td>Full text</td><td>PET 500</td><td>85.64%</td><td>86.66%</td></tr><tr><td>XLM-RoBERTa</td><td>Full text</td><td>PET 1000</td><td>85.73%</td><td>88.85%</td></tr><tr><td>XLM-RoBERTa</td><td>Translated Title</td><td>Balanced Fine-tuning</td><td>80.16%</td><td>84.24%</td></tr><tr><td>XLM-RoBERTa</td><td>Translated Title</td><td>PET 50</td><td>82.89%</td><td>63.84%</td></tr><tr><td>XLM-RoBERTa</td><td>Translated Title</td><td>PET 100</td><td>80.27%</td><td>70.52%</td></tr><tr><td>XLM-RoBERTa</td><td>Translated Title</td><td>PET 200</td><td>81.22%</td><td>74.28%</td></tr><tr><td>XLM-RoBERTa</td><td>Translated Title</td><td>PET 1000</td><td>79.91%</td><td>84.19%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords (Abstract)</td><td>PET 1000</td><td>82.85%</td><td>70.57%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 100</td><td>81.94%</td><td>72.02%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 500</td><td>80.79%</td><td>82.09%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords + O.C (Abstract)</td><td>PET 1000</td><td>81.13%</td><td>84.82%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords (Full text)</td><td>PET 200</td><td>80.04%</td><td>74.48%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords (Full text)</td><td>PET 500</td><td>79.59%</td><td>79.52%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords (Full text)</td><td>PET 1000</td><td>79%</td><td>80.14%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords + O.C (Full text)</td><td>PET 500</td><td>82.49%</td><td>80.73%</td></tr><tr><td>XLM-RoBERTa</td><td>Phrases with Keywords + O.C (Full text)</td><td>PET 1000</td><td>80.51%</td><td>83.46%</td></tr></tbody></table>
<table><caption>TABLE F.21: Development $F_2$, AUC, and $F_1$ of top performing classifiers by Content Source</caption><thead><tr><th>Content Source</th><th>Model</th><th>Training Method</th><th>Development F<sub>2</sub></th><th>Development AUC</th><th>Development F<sub>1</sub></th></tr></thead><tbody><tr><td>Title</td><td>Bio-Link-BERT</td><td>Balanced Fine-tuning</td><td>79.97 %</td><td>84.18 %</td><td>78.45 %</td></tr><tr><td>Title</td><td>XLM-RoBERTa</td><td>PET 500</td><td>80.11 %</td><td>82.48 %</td><td>76.25 %</td></tr><tr><td>Title</td><td>SciBERT</td><td>PET 1000</td><td>80.37 %</td><td>80.88 %</td><td>75.77 %</td></tr><tr><td>Title</td><td>mBERT</td><td>PET 500</td><td>81.06 %</td><td>80.93 %</td><td>74.84 %</td></tr><tr><td>Title</td><td>Bio-Link-BERT</td><td>PET 1000</td><td>81.2 %</td><td>79.8 %</td><td>74.99 %</td></tr><tr><td>Title</td><td>mBERT</td><td>PET 1000</td><td>82.44 %</td><td>83.46 %</td><td>77.55 %</td></tr><tr><td>Title</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>83.12 %</td><td>83.55 %</td><td>77.3 %</td></tr><tr><td>Abstract</td><td>RoBERTa-base</td><td>PET 1000</td><td>79.66 %</td><td>81.69 %</td><td>76.12 %</td></tr><tr><td>Abstract</td><td>mBERT</td><td>PET 1000</td><td>79.71 %</td><td>82.81 %</td><td>76.53 %</td></tr><tr><td>Abstract</td><td>RoBERTa-base</td><td>Balanced Fine-tuning</td><td>79.77 %</td><td>85.35 %</td><td>79.16 %</td></tr><tr><td>Abstract</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>82.34 %</td><td>84.15 %</td><td>77.41 %</td></tr><tr><td>Full text</td><td>RoBERTa-base</td><td>Balanced Fine-tuning</td><td>79.41 %</td><td>89.72 %</td><td>81.81 %</td></tr><tr><td>Full text</td><td>BERT-base</td><td>PET 500</td><td>81.35 %</td><td>79.66 %</td><td>74.28 %</td></tr><tr><td>Full text</td><td>Bio-Link-BERT</td><td>PET 500</td><td>81.94 %</td><td>82.41 %</td><td>74.62 %</td></tr><tr><td>Full text</td><td>SciBERT</td><td>PET 500</td><td>82.43 %</td><td>79.82 %</td><td>72.87 %</td></tr><tr><td>Full text</td><td>RoBERTa-base</td><td>PET 500</td><td>82.72 %</td><td>81.03 %</td><td>74.7 %</td></tr><tr><td>Full text</td><td>mBERT</td><td>PET 1000</td><td>82.75 %</td><td>88.82 %</td><td>81.52 %</td></tr><tr><td>Full text</td><td>BERT-base</td><td>PET 1000</td><td>83.1 %</td><td>85.09 %</td><td>79.13 %</td></tr><tr><td>Full text</td><td>RoBERTa-base</td><td>PET 1000</td><td>83.18 %</td><td>84.94 %</td><td>77.87 %</td></tr><tr><td>Full text</td><td>Bio-Link-BERT</td><td>PET 1000</td><td>83.25 %</td><td>85.16 %</td><td>77.82 %</td></tr><tr><td>Full text</td><td>mBERT</td><td>PET 200</td><td>83.29 %</td><td>80.29 %</td><td>74.04 %</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>Balanced Fine-tuning</td><td>83.62 %</td><td>90.57 %</td><td>83.95 %</td></tr><tr><td>Full text</td><td>SciBERT</td><td>PET 1000</td><td>83.95 %</td><td>84.83 %</td><td>77.98 %</td></tr><tr><td>Full text</td><td>mBERT</td><td>PET 500</td><td>84.72 %</td><td>86.75 %</td><td>79.78 %</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>PET 500</td><td>85.64 %</td><td>86.66 %</td><td>78.34 %</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>85.73 %</td><td>88.85 %</td><td>81.6 %</td></tr><tr><td>Translated Title</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>79.91 %</td><td>84.19 %</td><td>77.17 %</td></tr><tr><td>Translated Title</td><td>XLM-RoBERTa</td><td>Balanced Fine-tuning</td><td>80.16 %</td><td>84.24 %</td><td>79.23 %</td></tr><tr><td>Translated Title</td><td>RoBERTa-base</td><td>PET 500</td><td>80.25 %</td><td>82.01 %</td><td>75.93 %</td></tr><tr><td>Translated Title</td><td>mBERT</td><td>PET 500</td><td>80.32 %</td><td>81.88 %</td><td>76.1 %</td></tr><tr><td>Translated Title</td><td>Bio-Link-BERT</td><td>PET 1000</td><td>80.65 %</td><td>83.03 %</td><td>76.67 %</td></tr><tr><td>Translated Title</td><td>RoBERTa-base</td><td>PET 1000</td><td>81.02 %</td><td>84.2 %</td><td>78.47 %</td></tr><tr><td>Translated Title</td><td>mBERT</td><td>PET 1000</td><td>82.87 %</td><td>85.27 %</td><td>78.81 %</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>mBERT</td><td>PET 500</td><td>79.34 %</td><td>81.8 %</td><td>75.42 %</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>Bio-Link-BERT</td><td>PET 1000</td><td>79.35 %</td><td>79.01 %</td><td>74.14 %</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>BERT-base</td><td>PET 1000</td><td>80.16 %</td><td>80.2 %</td><td>75.63 %</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>RoBERTa-base</td><td>PET 1000</td><td>80.51 %</td><td>81.8 %</td><td>76.41 %</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>XLM-RoBERTa</td><td>PET 500</td><td>80.79 %</td><td>82.09 %</td><td>76.15 %</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>81.13 %</td><td>84.82 %</td><td>78.3 %</td></tr><tr><td>Phrases with Keywords + O.C (Abstract)</td><td>mBERT</td><td>PET 1000</td><td>81.64 %</td><td>84.51 %</td><td>78.53 %</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>79 %</td><td>80.14 %</td><td>74.17 %</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>XLM-RoBERTa</td><td>PET 500</td><td>79.59 %</td><td>79.52 %</td><td>73.32 %</td></tr><tr><td>Phrases with Keywords (Full text)</td><td>mBERT</td><td>PET 1000</td><td>80.25 %</td><td>82.43 %</td><td>76.63 %</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>mBERT</td><td>PET 500</td><td>79.26 %</td><td>81.62 %</td><td>75.84 %</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>Bio-Link-BERT</td><td>PET 1000</td><td>79.74 %</td><td>82.23 %</td><td>75.58 %</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>SciBERT</td><td>PET 1000</td><td>80.13 %</td><td>81.2 %</td><td>75.16 %</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>80.51 %</td><td>83.46 %</td><td>77.22 %</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>mBERT</td><td>PET 1000</td><td>82.29 %</td><td>83.35 %</td><td>78.19 %</td></tr><tr><td>Phrases with Keywords + O.C (Full text)</td><td>XLM-RoBERTa</td><td>PET 500</td><td>82.49 %</td><td>80.73 %</td><td>74.44 %</td></tr></tbody></table>
<table><caption>TABLE F.22: Development and Test metrics for Best Performing classifiers</caption><thead><tr><th>Content Source</th><th>Model</th><th>Training Method</th><th>Method</th><th>Development F2</th><th>Development AUC</th><th>Development F1</th><th>Development Accuracy</th><th>Development Precision</th><th>Development Recall</th><th>Test F2</th><th>Test AUC</th><th>Test Precision</th><th>Test Recall</th><th>Test F1</th><th>Test Accuracy</th></tr></thead><tbody><tr><td>Title</td><td>nBERT</td><td>PET 1000</td><td>nBERT</td><td>82.44 %</td><td>83.46 %</td><td>77.63 %</td><td>77.3 %</td><td>74.3 %</td><td>80.8 %</td><td>86.06 %</td><td>60.54 %</td><td>92.5 %</td><td>85.75 %</td><td>82.83 %</td><td>69.53 %</td></tr><tr><td>Title</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>XLM-RoBERTa</td><td>82.12 %</td><td>83.55 %</td><td>77.3 %</td><td>77.3 %</td><td>74.3 %</td><td>69.22 %</td><td>87.52 %</td><td>58.4 %</td><td>80.8 %</td><td>26.13 %</td><td>84.18 %</td><td>39.92 %</td></tr><tr><td>Abstract</td><td>nBERT</td><td>PET 1000</td><td>nBERT</td><td>82.34 %</td><td>84.15 %</td><td>77.41 %</td><td>74.91 %</td><td>70.4 %</td><td>85.08 %</td><td>60.57 %</td><td>84.6 %</td><td>28.74 %</td><td>83.76 %</td><td>42.79 %</td><td>61.24 %</td></tr><tr><td>Full text</td><td>nBERT</td><td>PET 1000</td><td>nBERT</td><td>82.75 %</td><td>85.82 %</td><td>84.52 %</td><td>84.06 %</td><td>79.46 %</td><td>83.58 %</td><td>60.97 %</td><td>89.61 %</td><td>40.21 %</td><td>79.24 %</td><td>35.44 %</td><td>86.13 %</td></tr><tr><td>Full text</td><td>RoBERTa-base</td><td>PET 1000</td><td>RoBERTa-base</td><td>83.1 %</td><td>85.09 %</td><td>79.13 %</td><td>77.33 %</td><td>73.3 %</td><td>85.98 %</td><td>59.65 %</td><td>84.84 %</td><td>40.38 %</td><td>69.08 %</td><td>26.47 %</td><td>86.86 %</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>XLM-RoBERTa</td><td>83.18 %</td><td>84.94 %</td><td>77.87 %</td><td>75.24 %</td><td>70.39 %</td><td>87.13 %</td><td>57.83 %</td><td>85.96 %</td><td>38 %</td><td>64.82 %</td><td>24.18 %</td><td>88.69 %</td></tr><tr><td>Full text</td><td>Bio-Link-BERT</td><td>PET 1000</td><td>Bio-Link-BERT</td><td>83.25 %</td><td>85.16 %</td><td>77.82 %</td><td>75.11 %</td><td>70.19 %</td><td>87.31 %</td><td>59.15 %</td><td>85.66 %</td><td>39.23 %</td><td>66.33 %</td><td>25.13 %</td><td>89.42 %</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>Balanced F1-tuning</td><td>XLM-RoBERTa</td><td>83.62 %</td><td>90.57 %</td><td>83.95 %</td><td>84.06 %</td><td>84.52 %</td><td>83.39 %</td><td>87.22 %</td><td>82.24 %</td><td>51.1 %</td><td>82.24 %</td><td>38.42 %</td><td>76.28 %</td></tr><tr><td>Full text</td><td>SciBERT</td><td>PET 1000</td><td>SciBERT</td><td>83.95 %</td><td>84.83 %</td><td>77.98 %</td><td>75.02 %</td><td>69.72 %</td><td>88.46 %</td><td>56.23 %</td><td>85.13 %</td><td>36.40 %</td><td>62.78 %</td><td>23.02 %</td><td>87.96 %</td></tr><tr><td>Full text</td><td>nBERT</td><td>PET 500</td><td>nBERT</td><td>84.72 %</td><td>86.75 %</td><td>79.78 %</td><td>77.6 %</td><td>72.7 %</td><td>88.38 %</td><td>61.25 %</td><td>87.61 %</td><td>41.6 %</td><td>69.48 %</td><td>27.1 %</td><td>89.42 %</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>PET 500</td><td>XLM-RoBERTa</td><td>85.64 %</td><td>86.66 %</td><td>78.34 %</td><td>74.76 %</td><td>68.6 %</td><td>91.3 %</td><td>56.84 %</td><td>87.59 %</td><td>36.31 %</td><td>61.09 %</td><td>22.67 %</td><td>91.24 %</td></tr><tr><td>Full text</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>XLM-RoBERTa</td><td>85.73 %</td><td>88.85 %</td><td>81.6 %</td><td>79.99 %</td><td>75.53 %</td><td>88.73 %</td><td>63.79 %</td><td>89.86 %</td><td>44.34 %</td><td>72.49 %</td><td>29.4 %</td><td>90.15 %</td></tr><tr><td>Translated Title</td><td>RoBERTa-base</td><td>PET 1000</td><td>RoBERTa-base</td><td>0.8102069614</td><td>84.2 %</td><td>78.47 %</td><td>77.28 %</td><td>74.57 %</td><td>82.81 %</td><td>50.18 %</td><td>83.72 %</td><td>31.41 %</td><td>75.98 %</td><td>44.44 %</td><td>73.86 %</td></tr><tr><td>Translated Title</td><td>nBERT</td><td>PET 1000</td><td>nBERT</td><td>82.87 %</td><td>85.27 %</td><td>78.81 %</td><td>76.92 %</td><td>72.86 %</td><td>85.82 %</td><td>68.81 %</td><td>84.2 %</td><td>31.86 %</td><td>85.15 %</td><td>46.37 %</td><td>0.7289663462</td></tr><tr><td>Phrases with (Abstract)</td><td>XLM-RoBERTa</td><td>PET 1000</td><td>Keywords + O C</td><td>81.13 %</td><td>84.82 %</td><td>0.78290977620</td><td>76.96 %</td><td>74 %</td><td>83.14 %</td><td>58.48 %</td><td>83.78 %</td><td>29.18 %</td><td>78.07 %</td><td>42.48 %</td><td>71.38 %</td></tr><tr><td>Phrases with (Abstract)</td><td>nBERT</td><td>PET 1000</td><td>Keywords + O C</td><td>81.64 %</td><td>84.51 %</td><td>78.53 %</td><td>77.08 %</td><td>73.85 %</td><td>83.85 %</td><td>58.69 %</td><td>83.37 %</td><td>28.73 %</td><td>79.39 %</td><td>42.19 %</td><td>70.55 %</td></tr><tr><td>Keywords + O C</td><td>nBERT</td><td>PET 1000</td><td>nBERT</td><td>82.29 %</td><td>83.35 %</td><td>78.19 %</td><td>76.22 %</td><td>72.2 %</td><td>85.26 %</td><td>56.41 %</td><td>82 %</td><td>25.31 %</td><td>81.42 %</td><td>38.61 %</td><td>08.65 %</td></tr><tr><td>Keywords + O C</td><td>nBERT</td><td>PET 1000</td><td>nBERT</td><td>82.29 %</td><td>83.35 %</td><td>78.19 %</td><td>76.22 %</td><td>72.2 %</td><td>85.26 %</td><td>56.41 %</td><td>82 %</td><td>25.31 %</td><td>81.42 %</td><td>38.61 %</td><td>08.65 %</td></tr></tbody></table>