SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: Test set results for baseline and proposed methods on sub-tasks included in the SCIDOCS benchmark. Our approaches outperform a prior strong model, SPECTER, by 1-1.5 points on 3 of 4 sub-tasks. Metrics (MAP, NDCG) are computed based on averages over three re-runs of models. SPECTER uses model parameters as part of the Huggingface library. Here, TSASPIRE: Text supervised single-match method, OTASPIRE: Optimal Transport multi-match method and TS+OTASPIRE: Multi-task multi aspect method.
CSFCUBE Agg.	MAP	NDCG%20
ABSASPIRESpec	37.03	59.57
±1.39	±0.76
TSASPIRESpec	40.26	60.71
±0.93	±0.67
TRECCOVIDRF	RELISH
MAP	NDCG%20	MAP	NDCG%20
ABSASPIRESpec	25.42±0.9	55.34±0.55	58.78±0.69	75.80±0.57
TSASPIRESpec	26.24±0.45	56.55±0.65	61.29±0.51	77.89±0.42
A2. Can multi-aspect matching use attention aggregation instead of optimal transport? Since our multi-aspect match model uses a soft sparse matching with optimal transport we examine contributions of this component by comparing performance of a model (ATTASPIRE) trained with soft-alignment using an attention mask,  $\mathbf{A}$  - attention is also a popular choice in prior work Humeau et al. (2020); Zhou et al. (2020). Here,  $f_{\mathrm{Att}}(p,p^{\prime}) = \langle \mathbf{D},\mathbf{A}\rangle$  with,  $\mathbf{A} = \mathrm{softmax}(-\mathbf{D} / \tau)$ . Note that OT imposes specific inductive bias via the structure of the transport plan in ensuring it to be a permutation matrix - a desirable property in computing multiple alignments between a set of points. Table 5 examines performance of these model variants. Broadly, ATTASPIRE sees performance comparable or worse than OTASPIRE. While ATTASPIRE
Table 4: Results for Ablation A1. Performance of TSASPIRE trained with textual supervision from co-citation contexts ablated for the effect of the text vs. influence of the text encoder  $(\mathsf{BERT}_{\mathcal{E}} = \mathsf{CoSentBert};$  in §3.2.2) used to compute alignments to the co-citation contexts. Standard deviation across 3 model re-runs under mean performance.
CSFCUBE Agg.	MAP	NDCG%20
ATTASPIRESpec	41.85	61.67
±1.52	±0.82
OTASPIRESpec	40.79	61.41
±0.53	±0.52
TRECCOVIDRF	RELISH
MAP	NDCG%20	MAP	NDCG%20
ATTASPIRESpec	29.51±0.78	60.96±0.51	61.92±0.52	78.54±0.50
OTASPIRESpec	30.92±0.53	62.23±0.67	62.57±0.29	78.95±0.26
Table 5: Results for Ablation A2. Performance for an alternative method, ATTASPIRE, for modeling multiple matches with an attention mechanism instead of optimal transport in the proposed method. Standard deviation across 3 model re-runs under mean performance.
sees improved performance in CSFCUBE it sees much larger variation across runs. In our abstract retrieval datasets, where we expect gains from modeling multiple matches, we see better or similar performance from OTASPIRE over ATTASPIRE.
A3. Can single-match models be learned without co-citation contexts? While our model for single matches leverages weak textual supervision from co-citation contexts, we ask if these models can be learned in the absence of this supervision. We answer this by training a simpler model, MAXASPIRE, which finds the maximally aligned aspects between documents using the representations from  $\mathrm{BERT}_{\theta}$  alone, giving us  $f_{\mathrm{Max}}(p,p^{\prime}) = \max_{i,j}\mathbf{D}$ . To examine the role of  $\mathrm{BERT}_{\theta}$  we compare performance with different initializations, with SPECTER presenting a initial model fine-tuned for similarity vs SCIBERT which isn't fine-tuned for text similarity.
CSFCUBE Aggregated	MAP	NDCG%20
MAXASPIRESciB	36.66 ±1.37	57.68 ±0.86
MAXASPIRESpec	39.42 ±1.38	60.63 ±1.53
TSASPIRESciB	40.10 ±0.76	60.92 ±0.61
TSASPIRESpec	40.26 ±0.93	60.71 ±0.67
TRECCOVIDRF	RELISH
MAP	NDCG%20	MAP	NDCG%20
MAXASPIRESciB	24.87	54.33	61.36	78.10
±1.15	±1.49	±0.31	±0.24
MAXASPIRESpec	25.84	56.52	61.20	78.00
±0.85	±1.21	±0.97	±0.36
TSASPIRESciB	27.68	58.42	61.45	78.12
±0.71	±0.75	±0.31	±0.33
TSASPIRESpec	26.24	56.55	61.29	77.89
±0.45	±0.65	±0.51	±0.42
Table 6: Results for Ablation A3. Performance of a simpler single-match model, MAXASPIRE, trained using only  $\mathrm{BERT}_{\theta}$  representations while also varying encoder initialization between SPECTER and SCIBERT (indicated as subscripts for models). Standard deviation across 3 model re-runs under mean performance.
We note the following from the results in Table 6: MAXASPIRE sees a dependence on the underlying encoder, a SCIBERT initialization nearly always sees poorer performance - only seeing performance competitive with TSASPIRE when initialized with SPECTER. This is reasonable given that this model must bootstrap fine-grained similarity while only relying on the encoder induced similarity. In cases where MAXASPIRE matches performance of TSASPIRE it sees larger performance differences across runs which may also be explained by the dependence on the initialization. Finally, TSASPIRE consistently sees similar or better performance with varying initialization, indicating the value of our text supervised method.
I Extended Results
Tables 1, 2 in §4.1 omit presentation of standard deviations across runs for the proposed approaches for brevity. We include these in Tables 7 and 8.
J Extended Related Work
A range of modeling approaches in multi-instance learning, models leveraging textual supervision, and optimal transport resemble elements of our approach. We describe these next.
Multi-instance Learning: Our work applies MIL for learning fine-grained similarity, while prior work has most often been applied to classification
or regression tasks (Hope and Shahaf, 2016, 2018; Ilse et al., 2018; Angelidis and Lapata, 2018). Our work bears resemblance to an application of MIL in content based image retrieval (Song and Soleymani, 2019), where MIL is applied to learn alignments between image and text aspects.
Textual Supervision: Our use of co-citation text as a source of textual supervision draws on other work leveraging textual justifications of labels as a source of supervision for classification tasks (Hancock et al., 2018; Murty et al., 2020; Hanjie et al., 2022), with recent concurrent work of Hanjie et al. including an overview of this line of work. Our co-citation contexts may be considered justifications for similarity of co-cited papers. Nie et al. (2020) presents work in a biomedical literature recommendation task, where human justifications of a relevance label are used to identify unigram features indicative of the label and train a recommendation model.
Optimal Transport: Our use of optimal transport draws on other recent work in learning sparse alignments between texts (Swanson et al., 2020; Tam et al., 2019). Work of Swanson et al. (2020) learns sparse binary alignments for sentence and document similarity tasks to rationalize decisions and Tam et al. (2019) leverage sparse soft alignments between characters for string similarity. Kusner et al. (2015) uses alignment based on word embeddings for document classification tasks using a K-nearest neighbors method. However, applying OT at the word level in scientific documents would lead to a large increase in computational complexity.
CSFCUBE facets→	Aggregated	background	method	result
MAP	NDCG%20	MAP	NDCG%20	MAP	NDCG%20	MAP	NDCG%20
MPNET-1B	34.64	54.94	41.06	65.86	27.21	42.48	36.07	54.94
SENTBERT-PP	26.77	48.57	35.43	60.80	16.19	33.40	29.16	48.57
SENTBERT-NLI	25.23	45.39	30.78	54.23	15.02	31.10	30.27	45.39
UNSIMCSE-BERT	24.45	42.59	30.03	51.59	14.82	31.23	28.76	42.59
SUSIMCSE-BERT	23.24	43.45	30.52	55.22	13.99	30.88	25.58	43.45
CoSentBert	28.95	50.68	35.78	61.27	19.27	38.77	32.15	50.68
ICTSENTBERT	28.61	48.13	35.93	59.80	15.62	35.91	34.42	48.13
OTMPNET-1B	36.41	56.91	43.23	67.60	28.69	43.49	37.76	60.30
SPECTER	34.23	53.28	43.95	66.70	22.44	37.41	36.79	56.67
SCINCL	39.37	59.24	49.64	70.02	27.14	46.61	41.83	61.70
SPECTER-COCITEscib	37.90 ±1.48	58.16 ±1.9	48.40 ±2.51	68.71 ±2.71	26.95 ±0.96	46.79 ±0.74	38.93 ±2.17	59.68 ±3.58
SPECTER-COCITEspec	37.39 ±0.73	58.38 ±0.86	49.99 ±1.2	70.03 ±1.16	25.60 ±0.53	45.99 ±1.35	37.33 ±0.86	59.95 ±1.02
TSASPIRESpec	40.26 ±0.93	60.71 ±0.67	49.58 ±1.59	70.22 ±1.74	28.86 ±1.71	48.20 ±1.72	42.92 ±0.54	64.39 ±0.28
OTASPIRESpec	40.79 ±0.53	61.41 ±0.52	50.56 ±1.52	71.04 ±1.42	27.64 ±0.92	46.46 ±0.1	44.75 ±1.57	67.38 ±0.99
TS+OTASPIRESpec	40.26 ±0.71	60.86 ±0.58	51.79 ±1.18	70.99 ±1.28	26.68 ±3.21	47.60 ±2.45	43.06 ±0.21	64.82 ±0.19
Table 7: Test set results for baseline and proposed methods on CSFCUBE, an expert annotated fine-grained similarity dataset of computer science papers. Our approaches outperform strong prior models OT/MPNET-1B and SPECTER by 5-6 points, and the concurrently introduced SCINCL model by 1.5-2 points aggregated across queries. Metrics (MAP, NDCG $_{20}$ ) are computed based on a 2-fold cross-validation and averaged over three re-runs of models. Standard deviations are below run averages. Here, TSASPIRE: Text supervised single-match method, OTASPIRE: Optimal Transport multi-match method and TS+OTASPIRE: Multi-task multi aspect method.
TRECCOVIDRF	RELISH
MAP	NDCG%20	MAP	NDCG%20
MPNET-1B	17.35	43.87	52.92	69.69
SENTBERT-PP	11.12	34.85	50.80	67.35
SENTBERT-NLI	13.43	40.78	47.02	63.56
UNSIMCSE-BERT	9.85	34.27	45.79	62.02
SUSIMCSE-BERT	11.50	37.17	47.29	63.93
CoSentBert	12.80	38.07	50.04	66.35
ICTSENTBERT	9.80	33.62	47.72	63.71
OTMPNET-1B	27.46	58.70	57.46	74.64
SPECTER	28.24	59.28	60.62	77.20
SCINCL	28.73	59.16	62.09	78.72
SPECTER-COCITEscib	30.60 ±0.87	62.07 ±0.95	61.43 ±0.32	78.01 ±0.1
SPECTER-COCITEspec	28.59 ±0.25	60.07 ±0.36	61.43 ±0.24	77.96 ±0.23
TSASPIRESpec	26.24 ±0.45	56.55 ±0.65	61.29 ±0.51	77.89 ±0.42
OTASPIRESpec	30.92 ±0.53	62.23 ±0.67	62.57 ±0.29	78.95 ±0.26
TS+OTASPIRESpec	30.90 ±0.71	62.18 ±0.7	62.71 ±0.16	79.18 ±0.15
Table 8: Test set results for baseline and proposed methods on  $\mathrm{TRECCOVID}_{\mathrm{RF}}$  and RELISH, expert annotated abstract similarity datasets of biomedical papers. Our approaches outperform or match a strong prior model, SPECTER, and the concurrently introduced SCINCL by 2-3 points across metrics (MAP, NDCG%20). These are computed as averages over three model re-runs. Standard deviations are below run averages. Method names map similarly to Table 7.