SlowGuess/ACL_2020_OCR huggingface dataset:
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117, Austin, Texas. Association for Computational Linguistics.
Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. 2018. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4470-4481, Brussels, Belgium. Association for Computational Linguistics.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220.
Qiuchi Li, Benyou Wang, and Massimo Melucci. 2019. CNM: An interpretable complex-valued network for matching. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4139-4148, Minneapolis, Minnesota. Association for Computational Linguistics.
Junyang Lin, Xu Sun, Xuancheng Ren, Muyu Li, and Qi Su. 2018. Learning when to concentrate or divert attention: Self-adaptive attention temperature for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2985-2990, Brussels, Belgium. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Chaitanya Malaviya, Pedro Ferreira, and André F. T. Martins. 2018. Sparse and constrained attention for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 370-376, Melbourne, Australia. Association for Computational Linguistics.
Andre Martins and Ramon Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1614-1623, New York, New York, USA. PMLR.
Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. 2018. Learning latent permutations with gumbel-sinkhorn networks. International Conference on Learning Representations.
Gaspard Monge. 1781. Mémoire sur la théorie des déblais et des remblais. Histoire de l'Académie Royale des Sciences, pages 666-704.
5619
Vlad Niculae and Mathieu Blondel. 2017. A regularized framework for sparse and structured neural attention. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3338-3348. Curran Associates, Inc.
Vlad Niculae, André F. T. Martins, Mathieu Blondel, and Claire Cardie. 2018. Sparsemap: Differentiable sparse structured inference. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 3799-3808. PMLR.
Ankur Parikh, Oscar Tackström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249-2255, Austin, Texas. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. NIPS 2017 Autodiff Workshop.
Hugh Perkins and Yi Yang. 2019. Dialog intent induction with deep multi-view clustering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4014-4023. Association for Computational Linguistics.
Gabriel Peyre and Marco Cuturi. 2019. Computational optimal transport. Foundations and Trends in Machine Learning, 11:335-607.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, page 1135-1144, New York, NY, USA. Association for Computing Machinery.
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočisky, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664.
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez. 2017. Right for the right reasons: Training differentiable models by constraining their explanations. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2662-2670.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Cicero dos Santos, Luciano Barbosa, Dasha Bogdanova, and Bianca Zadrozny. 2015. Learning hybrid representations to retrieve semantically equivalent questions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 694-699, Beijing, China. Association for Computational Linguistics.
Bernhard Schmitzer. 2016. Stabilized sparse scaling algorithms for entropy regularized transport problems. SIAM Journal on Scientific Computing, 41:A1443-A1481.
Darsh Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov. 2018. Adversarial domain adaptation for duplicate question detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1056-1063, Brussels, Belgium. Association for Computational Linguistics.
Richard Sinkhorn and Paul Knopp. 1967. Concerning nonnegative matrices and doubly stochastic matrices. Pacific J. Math, 21:343-348.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 3319-3328. JMLR.org.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. Generating token-level explanations for natural language inference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 963-969, Minneapolis, Minnesota. Association for Computational Linguistics.
Sarah Wiegrefe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Qizhe Xie, Xuezhe Ma, Zihang Dai, and Eduard Hovy. 2017. An interpretable knowledge transfer model for knowledge base completion. In Proceedings of the 55th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 950-962, Vancouver, Canada. Association for Computational Linguistics.
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. 2019. Gromov-Wasserstein learning for graph matching and node embedding. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6932-6941, Long Beach, California, USA. PMLR.
Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, page 2048-2057. JMLR.org.
Mo Yu, Shiyu Chang, Yang Zhang, and Tommi Jaakkola. 2019. Rethinking cooperative rationalization: Introspective extraction and complement control. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4094-4103, Hong Kong, China. Association for Computational Linguistics.
Ye Zhang, Iain Marshall, and Byron C. Wallace. 2016. Rationale-augmented convolutional neural networks for text classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 795-804, Austin, Texas. Association for Computational Linguistics.
A Qualitative Study
Human Evaluation. We performed a human evaluation of rationale quality on the Stack-Exchange dataset. We asked 8 annotators to rate 270 rationale examples selected from three models including OT (exact  $k = 2$ ), Attention ( $T = 0.01$ ), and Sparse Attention. For each example, we presented the human annotator with a pair of similar documents along with the extracted alignment rationales. The annotator then assigned a score of 0, 1, or 2 for each of the following categories: redundancy, relevance, and overall quality. A higher score is always better (i.e., less redundant, more relevant, higher overall quality). For attention-based models, we selected the top 2 or 3 aligned pairs (according to the attention weights) such that the number of pairs is similar to that of the OT (exact  $k = 2$ ) model. The results are shown
(a) Redundancy
(b) Relevance
(c) Overall quality
Figure 7: Human evaluation of rationales extracted from StackExchange document pairs using metrics of redundancy, relevance, and overall quality. Scores are either 0 (red), 1 (gray), or 2 (blue) and higher is better. The length of each bar segment indicates the proportion of examples with that score, and the number to the right of each bar is the average score.
in Figure 7. Attention models have more redundancy as well as higher relevance. This is not surprising since selecting redundant alignments can result in fewer mistakes. In comparison, our OT-based model achieves much less redundancy and a better overall score.
Example Rationales. Figure 8 shows examples of rationales generated from our OT (exact  $k = 2$ ) model on the StackExchange dataset. Our extracted rationales effectively identify sentences with similar semantic meaning and capture the major topics in the AskUbuntu subdomain. Figure 9 similarly shows example rationales on the e-SNLI dataset.
MultiRC Experiments with Recurrent Encoder. Table 6 shows the experimental results on the MultiRC dataset when we replace the RoBERTa encoder (results shown in Table 5) with the bi-directional simple recurrent unit (SRU) encoder (Lei et al., 2018) that we used for the MultiNews and e-SNLI datasets. In the unsupervised rationale learning setting, the
5621
Figure 8: Examples of extracted rationales from the StackExchange dataset using the OT (exact  $k = 2$ ) model. Each rationale alignment is displayed visually as lines connecting pairs of sentences from the two text documents.
Figure 9: Examples of extracted rationales from the e-SNLI dataset using the OT (exact  $k = 3$ ) model. We show two examples of entailment (left column), neutral (middle column) and contradiction (right column).
Model	Task F1	% Token	R. F1
OT (1:1)	59.5	20.3	24.2
OT (1:2)	60.1	28.0	26.5
OT (relaxed 1:1)	59.7	13.6	19.5
OT (relaxed 1:2)	60.2	24.7	29.1
OT (exact k = 2)	61.0	15.2	22.7
Attention	61.4	33.2	15.7
Attention (T = 0.1)	61.0	34.7	17.5
Attention (T = 0.01)	61.0	34.4	18.5
Sparse Attention	60.7	37.5	25.0
OT (1:1) (+S)	62.1	20.5	48.1
OT (1:2) (+S)	60.0	31.3	46.0
OT (relaxed 1:1) (+S)	60.3	18.2	46.2
OT (relaxed 1:2) (+S)	60.6	25.2	44.9
OT (exact k = 2) (+S)	61.2	16.7	48.7
Table 6: MultiRC macro-averaged task F1, percentage of tokens used in active alignments, and token-level F1 of the model-selected rationales compared to human-annotated rationales (R. F1). (+S) denotes supervised learning of rationales. All models use a simplified recurrent unit (Lei et al., 2018) encoder.
SRU alignment models achieve lower task F1 score and lower rationale token F1 score than the RoBERTa counterpart. Nevertheless, our models still outperform attention-based models, the unsupervised rationale extraction baseline (Lei et al., 2016) implemented in DeYoung et al. (2019), and even one supervised rationale model (Lehman et al., 2019) implemented in DeYoung et al. (2019). In the supervised rationale learning setting, the SRU alignment models achieve performance comparable to that of the RoBERTa alignment models. Both alignment models achieve higher rationale F1 score than the baseline models, regardless of the encoder architecture, demonstrating the strength of our model for learning rationales.
Text Span Extraction. Sentences are extracted from the documents using the sentence tokenizer from the nltk Python package (Bird et al., 2009).
Text Embeddings. For the bi-directional recurrent encoder, we use pre-trained fastText (Bojanowski et al., 2017) word embeddings, while for the RoBERTa encoder, we use its own pre-trained BPE embeddings.
OT Cost Functions. We use negative cosine similarity as the cost function for our OT (relaxed 1:1) model to achieve both positive and negative values in the cost matrix. For all the other OT variants, we use cosine distance, which is non-negative. We found that cosine-based costs work better than euclidean and dot-product costs for our model.
Sinkhorn Stability. To improve the computational stability of the Sinkhorn algorithm, we use an epsilon scaling trick (Schmitzer, 2016) which repeatedly runs the Sinkhorn iterations with progressively smaller values of epsilon down to a final epsilon of  $10^{-4}$ .
Loss Function. For the document ranking tasks, MultiNews and StackExchange, we train our model using a contrastive loss based on the difference between the optimal transport costs of aligning similar and dissimilar documents. Given a document  $D$ , if  $\mathbf{C}^{+}$  is the cost matrix between  $D$  and a similar document and  $\{\mathbf{C}_i^{-}\}_{i=1}^l$  are the cost matrices between  $D$  and  $l$  dissimilar documents, then the loss is defined as
$$
\max _ {i \in [ [ l ] ]} \left[ \max (\langle {\bf C} ^ {+}, {\bf P} ^ {+} \rangle - \langle {\bf C} _ {i} ^ {-}, {\bf P} _ {i} ^ {-} \rangle + \Delta , 0) \right],
$$
where  $\mathbf{P}^{+}$  and  $\mathbf{P}_i^-$  are the OT alignment matrices computed by the Sinkhorn algorithm for  $\mathbf{C}^{+}$  and  $\mathbf{C}_i^-$ , respectively, and where  $\Delta$  is the hinge margin.
For the classification tasks, e-SNLI and MultiRC, we use the standard cross entropy loss applied to the output of a shallow network that processes the cost and alignment matrices. Specifically, our model implementation is similar to the decomposable attention model (Parikh et al., 2016), in which the attention-weighted hidden representation is given to a simple 2-layer feed-forward network to generate the classification prediction. We similarly use the alignment output  $\mathbf{P}$  from OT as the weight mask (which will be sparse) to select and average over hidden representations.
Comparison to Human-Annotated Rationales. The e-SNLI and MultiRC datasets from the ERASER benchmark provide human rationale annotations, enabling a comparison of model
$^{6}$ https://www.nltk.org/
selected rationales to human-annotated rationales. However, the rationales are provided independently for each of the two input documents without alignment information. Therefore, in order to compare our models' rationales to the human annotations, we need to convert our pairwise alignments to independent binary selection rationales for each of the two input documents. This can be accomplished via thresholding, as described below.
Given an alignment matrix  $\mathbf{P} \in \mathbb{R}_+^{n \times m}$  aligning documents  $X = \{\mathbf{x}_i\}_{i=1}^n$  and  $Y = \{\mathbf{y}_i\}_{i=1}^m$ , the goal is to determine two binary rationale selection vectors  $\mathbf{R}^{\mathbf{x}} \in \{0,1\}^n$  and  $\mathbf{R}^{\mathbf{y}} \in \{0,1\}^m$  indicating which text spans in  $X$  and  $Y$  are selected. Each entry of  $\mathbf{R}^{\mathbf{x}}$  and  $\mathbf{R}^{\mathbf{y}}$  is computed as  $\mathbf{R}_i^{\mathbf{x}} = \mathbb{1}[\sum_{j=1}^m \mathbb{1}[\mathbf{P}_{i,j} > \delta] > 0]$  and  $\mathbf{R}_j^{\mathbf{y}} = \mathbb{1}[\sum_{i=1}^n \mathbb{1}[\mathbf{P}_{i,j} > \delta] > 0]$ , where  $\mathbb{1}[\cdot]$  is an indicator function. Intuitively, this means that  $\mathbf{R}_i^{\mathbf{x}} = 1$  if  $\mathbf{P}_{i,j} > \delta$  for any  $j = 1, \ldots, m$ , i.e., if at least one text span in  $Y$  aligns to text span  $\mathbf{x}_i$ , and  $\mathbf{R}_i^{\mathbf{x}} = 0$  otherwise. The meaning is the equivalent for  $\mathbf{R}_j^{\mathbf{y}}$ .
The binary selection rationales  $\mathbf{R}^{\mathrm{x}}$  and  $\mathbf{R}^{\mathrm{y}}$  can then be compared against the human-annotated rationales as measured by the F1 score. The threshold  $\delta$  is selected based on the  $\delta$  which produces the greatest F1 score on the validation set.
Supervised Rationale Training. Our models are designed to learn alignments in an unsupervised manner, but it is possible to alter them to learn from human-annotated rationales in a supervised way.
We do this by constructing a soft version of the independent binary rationale selections described in the previous section. First, we compute  $\widetilde{\mathbf{R}}_i^{\mathbf{x}} = \sum_{j=1}^{m} \mathbf{P}_{i,j}$  and  $\widetilde{\mathbf{R}}_j^{\mathbf{y}} = \sum_{i=1}^{n} \mathbf{P}_{i,j}$  as soft rationale indicators. We then compute the cross entropy loss  $\mathcal{L}_r$  between these soft predictions and the human-annotated rationales. This loss is combined with the usual task classification cross entropy loss  $\mathcal{L}_c$  to form the total loss
$$
\mathcal {L} = \alpha \cdot \mathcal {L} _ {c} + (1 - \alpha) \cdot \mathcal {L} _ {r},
$$
where  $\alpha$  is a hyperparameter. In our experiments, we set  $\alpha = 0.2$ .
Model	# Parameters	Train time (s)	Infer time (s)
OT	2.0M	600	8.0e-3
Attention	2.4M	180	4.9e-3
Table 7: Number of parameters, training time, and inference time for models on the StackExchange dataset. Training time represents training time per epoch while inference time represents the average time to encode and align one pair of documents. All models use an NVIDIA Tesla V100 GPU.
Model Complexity and Speed. Table 7 compares the model complexity and model speed between OT-based and attention-based models with bi-directional recurrent encoders (Lei et al., 2018). Our model does not add any trainable parameters on top of the text encoder, making it smaller than its attention-based counterparts, which use additional parameters in the attention layer. Our model is 3.3 times slower than attention during training and 1.6 times slower than attention during inference due to the large number of iterations required by the Sinkhorn algorithm for OT.
Additional Details. We use the Adam (Kingma and Ba, 2014) optimizer for training. Hyperparameters such as the hinge loss margin, dropout rate, and learning rate are chosen according to the best validation set performance. All models were implemented with PyTorch (Paszke et al., 2017). Table 7 shows model complexity, training time, and inference time for the StackExchange dataset.
D Obtaining Permutation Matrix Solutions to Optimal Transport Problems
Our goal in this paper is to create an optimal transport problem that results in an assignment between two sets  $X$  and  $Y$ . The core idea is to create an expanded optimal transport problem between augmented sets  $X'$  and  $Y'$  such that  $|X'| = |Y'| = n$ . Then Proposition 2 implies that the optimal transport problem with  $\mathbf{a} = \mathbf{b} = \mathbb{1}_n / n$  has a permutation matrix solution. This permutation matrix represents a one-to-one assignment between  $X'$  and  $Y'$  from which we can extract an assignment between  $X$  and  $Y$ .
However, a problem with this approach is
5624
that the permutation matrix solution might not be the only solution. In general, linear programming problems may have many solutions, meaning we are not guaranteed to find a permutation matrix solution even if it exists. Since we require a permutation matrix solution in order to obtain our desired sparsity bounds, we are therefore interested in methods for identifying the permutation matrix solution even when other solutions exist. Although these methods were not necessary for our experiments, since the Sinkhorn algorithm almost always found a permutation matrix solution for our inputs, we present these methods to ensure that the techniques presented in this paper can be used even in cases with degenerate solutions.
One option is to avoid the problem altogether by using cost functions that are guaranteed to produce unique solutions. For example, Brenier (1987) showed that under some normality conditions, the cost function  $c(\mathbf{x}, \mathbf{y}) = ||\mathbf{x} - \mathbf{y}||^2$ , i.e., the Euclidean distance, produces OT problems with unique solutions. However, it is sometimes preferable to use cost functions with different properties (e.g., bounded range, negative cost, etc.) which may not guarantee a unique OT solution.
To find unique solutions for general cost functions, one method is to first find any solution to the optimal transport problem (e.g., by using the Sinkhorn algorithm) and then to use Birkhoff's algorithm (Brualdi, 1982) to express that solution as a convex combination of permutation matrices. Since the original solution is optimal, every permutation matrix that is part of the convex combination must also be optimal (otherwise the cost could be reduced further by removing the suboptimal matrix from the combination and rescaling the others). Thus we can pick any of the permutation matrices in the convex combination as our optimal permutation matrix solution. However, since Birkhoff's algorithm is not differentiable, these procedures cannot be used in end-to-end training and can only be applied at inference time.
An alternate method, which preserves the differentiability of our overall approach, is to solve a modified version of the linear programming problem that is guaranteed to have
a unique permutation matrix solution that closely approximates the solution the original problem. Theorem 1 demonstrates that by adding random iid noise of at most  $\epsilon$  to each element of the cost matrix  $\mathbf{C}$  to create a new cost matrix  $\mathbf{C}^{\epsilon}$ , then with probability one, the resulting linear programming problem on  $\mathbf{C}^{\epsilon}$  has a unique permutation matrix solution  $\mathbf{P}^{\epsilon*}$  which costs at most  $\epsilon$  more than the true optimal solution  $\mathbf{P}^*$ . Thus, we can obtain a permutation matrix solution for  $\mathbf{C}$  that is arbitrarily close to optimal. Furthermore, Corollary 1 implies that if we know that the difference in cost between the optimal permutation matrix and the second best permutation matrix is  $\delta$ , then we can choose  $\epsilon < \delta$  to ensure that we actually find an optimal permutation matrix.
Theorem 1. Consider  $L_{\mathbf{C}}(\mathbf{a}, \mathbf{b}) = \operatorname*{argmin}_{\mathbf{P} \in \mathbf{U}(\mathbf{a}, \mathbf{b})} \langle \mathbf{C}, \mathbf{P} \rangle$ , where  $\mathbf{C} \in \mathbb{R}^{n \times n}$  is ar-