SlowGuess/ACL_2020_OCR huggingface dataset:
Controllable Sparsity. Table 1 summarizes the differences between vanilla OT and the constrained variants. The freedom to select the type of constraint and the value of  $k$  gives fine-grained control over the level of sparsity. We evaluate the performance of all these variants in our experiments.
Datasets. We evaluate our model and all baselines on four benchmarks: two document similarity tasks, MultiNews and StackExchange, and two classification tasks, e-SNLI and MultiRC. The e-SNLI and MultiRC tasks come from the ERASER benchmark (DeYoung et al., 2019), which was created to evaluate selective rationalization models. We chose those two datasets as they are best suited for our text matching setup.
$\text{StackExchange}^2$  is an online question answering platform and has been used as a benchmark in previous work (dos Santos et al., 2015; Shah et al., 2018; Perkins and Yang, 2019). We took the June 2019 data dumps<sup>3</sup> of the AskUbuntu and SuperUser subdomains of the platform and combined them to form our dataset.
MultiNews (Fabbri et al., 2019) is a multi-document summarization dataset where 2 to 10 news articles share a single summary. We consider
Table 1: Summary of constrained alignment construction and sparsity. #R is the number of replicas, #D is the number of dummy points, R one-to-  $k$  is the relaxed one-to-  $k$  assignment,and  $n = \left| X\right|  \leq  \left| Y\right|  = m$  .
Metric	Exchange	MultiNews
# docs	730,818	10,130
# similar doc pairs	187,377	22,623
Avg sents per doc	3.7	31
Max sents per doc	54	1,632
Avg words per doc	87	680
Vocab size	603,801	299,732
Table 2: Statistics for the document ranking datasets.
every pair of articles that share a summary to be a similar document pair. Table 2 shows summary statistics of the two document ranking datasets.
e-SNLI (Camburu et al., 2018) is an extended version of the SNLI dataset (Bowman et al., 2015) for natural language inference where the goal is to predict the textual entailment relation (entailment, neutral, or contradiction) between premise and hypothesis sentences. Human rationales are provided as highlighted words in the two sentences.
MultiRC (Khashabi et al., 2018) is a reading comprehension dataset with the goal of assigning a label of true or false to a question-answer pair depending on information from a multi-sentence document. We treat the concatenated question and answer as one input and the document as the other input for text matching. Human rationales are provided as highlighted sentences in the document.
For StackExchange and MultiNews, we split the documents into  $80\%$  train,  $10\%$  validation, and  $10\%$  test, while for e-SNLI and MultiRC, we use the splits from DeYoung et al. (2019).
Metrics. We evaluate models according to the following three criteria.
1. Sparsity. To evaluate sparsity, we compute the average percentage of active alignments produced by each model, where an alignment is active if it exceeds a small threshold  $\lambda$ . This threshold is necessary to account for numerical imprecision in alignment values that are essentially zero. We set  $\lambda = \frac{0.01}{n \times m}$  unless otherwise specified, where  $n$  and  $m$  are the number of text spans in the two documents.
2. Sufficiency. If a model makes a correct prediction given only the rationales, then the rationales are sufficient. We evaluate sufficiency by providing the model only with active alignments and the aligned text representations and by masking non-active inputs (using the threshold  $\lambda$ ).
<sup>2</sup>https://stackoverflow.com/sites
<sup>3</sup>https://archive.org/details/
5614
Figure 4: An illustration of our constrained OT model applied to two text documents. The final output of the model depends on a combination of the encodings, the cost matrix, and the alignment matrix.
3. Relevance. The relevance of rationales is determined by whether a human would deem them valid and relevant. We compute relevance using the token-level F1 scores of model-generated rationales compared to human-selected rationales on the e-SNLI and MultiRC datasets. We also perform a qualitative human evaluation.
Baselines and Implementation Details. We use the decomposable attention model (Parikh et al., 2016) as our baseline attention model. In addition, we compare our model to two attention variants that are designed to encourage sparsity. The temperature attention variant applies a temperature term  $T$  in the softmax operator (Lin et al., 2018). The sparse attention variant adopts the sparsemax operator (Martins and Astudillo, 2016) in place of softmax to produce sparse attention masks.
Our constrained OT model operates as illustrated in Figure 4. After splitting the input documents into sentences, our model independently encodes each sentence and computes pairwise costs between the encoded representations<sup>4</sup>. Dummy and replica encodings are added as needed for the desired type of constrained alignment. Our model then applies OT via the Sinkhorn algorithm to the cost matrix  $\mathbf{C}$  to produce an optimal alignment matrix  $\mathbf{P}$ . For the document ranking tasks, the final score is simply  $\langle \mathbf{C}, \mathbf{P} \rangle$ . For the classification tasks, we use the alignment  $\mathbf{P}$  as a sparse mask to select encoded text representations, and we feed the aggregated representation to a shallow network to predict the output label, similar to our baseline attention models.
For a fair comparison, our models and all baselines use the same neural encoder to encode text spans before the attention or OT operation is applied. Specifically, we use RoBERTa (Liu et al., 2019), a state-of-the-art pre-trained encoder, for
Figure 5: Attention or alignment heatmaps generated by different methods on a synthetic  $30 \times 20$  cost matrix.
the StackExchange and MultiRC dataset. We use use bi-directional recurrent encoders (Lei et al., 2018) for the MultiNews and e-SNLI datasets<sup>5</sup>. The value of  $k$  for the OT constraints is chosen for each dataset by visually inspecting alignments in the validation set, though model performance is robust to the choice of  $k$ . In order to compare our models' rationales to human annotations, we use a binary thresholding procedure as described in Appendix C. We report results averaged over 3 independent runs for each model. Additional implementation details are provided in Appendix C.
Synthetic Visualizations. Before experimenting with the datasets, we first analyze the alignments obtained by different methods on a synthetic cost matrix in Figure 5. As shown in the figure, all attention baselines struggle to produce sufficiently sparse alignments, even with the use of a small temperature or the sparsemax operator. In contrast, our methods are very sparse, as a result of the provable sparsity guarantees of the constrained alignment
5The input text in the MultiNews dataset is too long for large BERT models. The e-SNLI dataset in ERASER contains human-annotated rationales at the word level while BERT models use sub-word tokenization.
For the e-SNLI dataset, where documents are single sentences, we use the contextualized token representations from the output of the sentence encoder following previous work (Thorne et al., 2019).
5615
StackExchange	MultiNews
Model	AUC	MAP	MRR	P@1	# Align.	AUC	MAP	MRR	P@1	# Align.
OT	98.0	91.2	91.5	86.1	8	97.5	96.8	98.1	97.2	48
OT (1:1)	97.7	89.7	90.0	83.9	4	97.8	96.7	97.9	96.8	19
OT (relaxed 1:1)	97.8	88.5	88.9	81.8	3	93.1	93.2	96.0	94.1	19
OT (exact k)	98.1	92.3	92.5	87.8	2	96.4	96.3	97.7	96.6	6
Attention	98.2	92.4	92.5	88.0	23	97.8	96.4	97.6	96.3	637
Attention (T=0.1)	98.2	92.4	92.5	87.7	22	98.0	97.0	98.1	97.1	634
Attention (T=0.01)	97.9	89.7	89.9	83.5	8	97.9	96.9	98.0	97.0	594
Sparse Attention	98.0	92.5	92.6	88.3	19	98.2	97.7	98.1	97.1	330
Table 3: Performance of all models on the StackExchange and MultiNews datasets. We report ranking results and the average number of active alignments (# Align.) used. For our method with the exact  $k$  alignment constraint, we set  $k = 2$  for StackExchange and  $k = 6$  for MultiNews, respectively.
problem. For instance, the relaxed one-to-  $k$  assignment produces fewer active alignments than either the number of rows or columns, and the exact-  $k$  assignment finds exactly  $k = 4$  alignments.
StackExchange & MultiNews. Table 3 presents the results of all models on the StackExchange and MultiNews datasets. We report standard ranking and retrieval metrics including area under the curve (AUC), mean average precision (MAP), mean reciprocal rank (MRR), and precision at 1  $(\mathrm{P}@\mathrm{1})$ . The results highlight the ability of our methods to obtain high interpretability while retaining ranking performance comparable to strong attention baselines. For example, our model is able to use only 6 aligned pairs to achieve a  $\mathrm{P}@\mathrm{1}$  of 96.6 on the MultiNews dataset. In comparison, the sparse attention model obtains a  $\mathrm{P}@\mathrm{1}$  of 97.1 but uses more than 300 alignment pairs and is thus difficult to interpret. Model complexity and speed on the StackExchange dataset are reported in Table 7 in Appendix C.
e-SNLI. Table 4 shows model performance on the e-SNLI dataset. As with document similarity ranking, we evaluate classification accuracy when the model uses only the active alignments. This is to ensure faithfulness, meaning the model truly and exclusively uses the rationales to make predictions. Since attention is not explicitly trained to use only active alignments, we also report the accuracy of attention models when using all attention weights.
As shown in the table, the accuracy of attention methods decreases significantly when we remove attention weights other than those deemed active by the threshold  $\lambda$ . In contrast, our model retains high accuracy even with just the active alignments since sparsity is naturally modeled in our contrained optimal transport framework. Figure 6 visualizes the
Figure 6: Model accuracy on the e-SNLI dataset when using different percentages of tokens as rationales. The attention model values are obtained using different thresholds  $\lambda$  to clip the attention weights while the values for our exact-  $k$  model correspond to  $k = 1,2,3,4$ .
change to model accuracy when different proportions of tokens are selected by the models.
Table 4 also presents the token-level F1 scores for the models' selected rationales compared to human-annotated rationales. Note that the rationale annotations for this task are designed for token selection rather than alignment and are sometimes only on one of the input sentences. Nevertheless, our model obtains F1 scores on par with recent work (DeYoung et al., 2019; Thorne et al., 2019).
MultiRC. Table 5 presents the results on the MultiRC dataset. Compared to attention models, our OT-based models achieve similar task performance with a higher rationale F1 score, despite selecting fewer rationales. The model variants from DeYoung et al. (2019) in general achieve higher task F1 performance. However, their unsupervised model suffers from degeneration due to the challenges of end-to-end training without rationale supervision.
We also create supervised versions of our models that learn from the human-annotated rationales
Model	Accuracy	Task F1	% Token	Premise F1	Hypothesis F1	P&H F1
OT (relaxed 1:1)	82.4	82.4	69.1	25.1	43.7	34.6
OT (exact k = 4)	81.4	81.4	38.7	24.3	45.0	35.4
OT (exact k = 3)	81.3	81.4	29.6	28.6	50.0	39.8
OT (exact k = 2)	81.3	81.3	21.6	24.8	30.6	27.8
Attention	76.3 (82.1)	76.2	37.9	26.6	37.6	32.2
Attention (T = 0.1)	73.9 (81.5)	73.9	33.0	28.4	44.1	36.5
Attention (T = 0.01)	70.2 (81.4)	69.9	30.6	26.1	38.0	32.2
Sparse Attention	63.5 (75.0)	63.1	12.5	8.8	24.5	17.2
Thorne et al. (2019)	- (81.0)	-	-	22.2	57.8	-
†Lei et al. (2016)	-	90.3	-	-	-	37.9
†Lei et al. (2016) (+S)	-	91.7	-	-	-	69.2
†Bert-To-Bert (+S)	-	73.3	-	-	-	70.1
Table 4: e-SNLI accuracy, macro-averaged task F1, percentage of tokens in active alignments, and token-level F1 of the model-selected rationales compared to human-annotated rationales for the premise, hypothesis, and both (P&H F1). Accuracy numbers in parentheses use all attention weights, not just active ones. (+S) denotes supervised learning of rationales.  $\dagger$  denotes results from DeYoung et al. (2019).
Model	Task F1	% Token	R. F1
OT (1:1)	62.3	21.6	33.7
OT (relaxed 1:1)	62.0	23.1	32.1
OT (relaxed 1:2)	62.2	24.0	35.9
OT (exact k = 2)	62.5	25.8	34.7
OT (exact k = 3)	62.0	24.6	37.3
Attention	62.6	44.7	21.3
Attention (T = 0.1)	62.6	34.7	18.2
Attention (T = 0.01)	62.7	30.1	17.3
Sparse Attention	59.3	31.3	21.2
†Lei et al. (2016)	64.8	-	0.0
OT (1:1) (+S)	61.5	19.0	50.0
OT (relaxed 1:1) (+S)	60.6	19.4	45.4
OT (relaxed 1:2) (+S)	61.5	28.7	46.8
OT (exact k = 2) (+S)	61.0	18.9	51.3
OT (exact k = 3) (+S)	60.9	23.1	49.3
†Lei et al. (2016) (+S)	65.5	-	45.6
†Lehman et al. (2019) (+S)	61.4	-	14.0
†Bert-To-Bert (+S)	63.3	-	41.2
Table 5: MultiRC macro-averaged task F1, percentage of tokens used in active alignments, and token-level F1 of the model-selected rationales compared to human-annotated rationales (R. F1). (+S) denotes supervised learning of rationales.  $\dagger$  denotes results from DeYoung et al. (2019).
during training. These supervised models achieve comparable task performance to and better rationale F1 scores than models from DeYoung et al. (2019), demonstrating the strength of a sparse rationale alignment. Supervised training details can be found in Appendix C.
Qualitative Studies. We performed a human evaluation on documents from StackExchange that reveals that our model's alignments are preferred to attention. The results of the human evaluation,
along with examples of StackExchange and e-SNLI alignments, are provided in Appendix A.
Balancing performance and interpretability in deep learning models has become an increasingly important aspect of model design. In this work, we propose jointly learning interpretable alignments as part of the downstream prediction to reveal how neural network models operate for text matching applications. Our method extends vanilla optimal transport by adding various constraints that produce alignments with highly controllable sparsity patterns, making them particularly interpretable. Our models show superiority by selecting very few alignments while achieving text matching performance on par with alternative methods. As an added benefit, our method is very general in nature and can be used as a differentiable hard-alignment module in larger NLP models that compare two pieces of text, such as sequence-to-sequence models. Furthermore, our method is agnostic to the underlying nature of the two objects being aligned and can therefore align disparate objects such as images and captions, enabling a wide range of future applications within NLP and beyond.
We thank Jesse Michel, Derek Chen, Yi Yang, and the anonymous reviewers for their valuable discussions. We thank Sam Altschul, Derek Chen, Amit Ganatra, Alex Lin, James Mullenbach, Jen Seale, Siddharth Varia, and Lei Xu for providing the human evaluation.
5617
David Alvarez-Melis and Tommi Jaakkola. 2018a. Gromov-Wasserstein alignment of word embedding spaces. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1881-1890, Brussels, Belgium. Association for Computational Linguistics.
David Alvarez-Melis and Tommi Jaakkola. 2018b. Towards robust interpretability with self-explaining neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7775-7784. Curran Associates, Inc.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations.
Joost Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963-2977, Florence, Italy. Association for Computational Linguistics.
Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python. O'Reilly Media Inc.
Garrett Birkhoff. 1946. Tres observaciones sobre el algebras lineal. Universidad Nacional de Tucumán Revista Series A, 5:147-151.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Yann Brenier. 1987. Décomposition polaire et réar- rangement monotone des champs de vecteurs. C. R. Acad. Sci. Paris Sér I Math., 305:805-808.
Richard A. Brualdi. 1982. Notes of the birkhoff algorithm for doubly stochastic matrices. Canadian Mathematical Bulletin, 25:191-199.
Richard A Brualdi. 2006. Combinatorial Matrix Classes, volume 108. Cambridge University Press.
Luis A. Caffarelli and Robert J. McCann. 2010. Free boundaries in optimal transport and monge-ampère obstacle problems. Annals of Mathematics, 171:673-730.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 9539-9549. Curran Associates, Inc.
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi Jaakkola. 2019. A game theoretic approach to class-wise selective rationalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 10055-10065. Curran Associates, Inc.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. 2018a. Learning to explain: An information-theoretic perspective on model interpretation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 883-892, Stockholm, Sweden. PMLR.
Kan Chen, Jiang Wang, Liang-Chieh Chen, Haoyuan Gao, Wei Xu, and Ram Nevatia. 2015. Abccnn: An attention based convolutional neural network for visual question answering. arXiv preprint arXiv:1511.05960.
Liqun Chen, Shuyang Dai, Chenyang Tao, Haichao Zhang, Zhe Gan, Dinghan Shen, Yizhe Zhang, Guoyin Wang, Ruiyi Zhang, and Lawrence Carin. 2018b. Adversarial text generation via feature-mover's distance. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 4666-4677. Curran Associates, Inc.
Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 551-561, Austin, Texas. Association for Computational Linguistics.
Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 2292-2300. Curran Associates, Inc.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. 2019. Eraser: A benchmark to evaluate rationalized nlp models. arXiv preprint arXiv:1911.03429.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th
5618
Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Alessio Figalli. 2010. The optimal partial transport problem. Archive for Rational Mechanics and Analysis, 195:533-560.
Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. arXiv preprint arXiv:1902.10186.
Leonid Kantorovich. 1942. On the transfer of masses (in russian). Doklady Akademii Nauk, 37:227-229.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262, New Orleans, Louisiana. Association for Computational Linguistics.
Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2018. Structured attention networks. International Conference on Learning Representations.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference on Learning Representations.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 957-966, Lille, France. PMLR.
Anirban Laha, Saneem Ahmed Chemmengath, Priyanka Agrawal, Mitesh Khapra, Karthik Sankaranarayanan, and Harish G Ramaswamy. 2018. On controllable sparse alternatives to softmax. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 6422-6432. Curran Associates, Inc.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C. Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3705-3717, Minneapolis, Minnesota. Association for Computational Linguistics.