SlowGuess/ACL_2020_OCR huggingface dataset:
Kyle Swanson*
ASAPP, Inc.
kswanson@asapp.com
liliyu@asapp.com
Tao Lei
tao@asapp.com
Selecting input features of top relevance has become a popular method for building self-explaining models. In this work, we extend this selective rationalization approach to text matching, where the goal is to jointly select and align text pieces, such as tokens or sentences, as a justification for the downstream prediction. Our approach employs optimal transport (OT) to find a minimal cost alignment between the inputs. However, directly applying OT often produces dense and therefore uninterpretable alignments. To overcome this limitation, we introduce novel constrained variants of the OT problem that result in highly sparse alignments with controllable sparsity. Our model is end-to-end differentiable using the Sinkhorn algorithm for OT and can be trained without any alignment annotations. We evaluate our model on the StackExchange, MultiNews, e-SNLI, and MultiRC datasets. Our model achieves very sparse rationale selections with high fidelity while preserving prediction accuracy compared to strong attention baseline models.
The growing complexity of deep neural networks has given rise to the desire for self-explaining models (Li et al., 2016; Ribeiro et al., 2016; Zhang et al., 2016; Ross et al., 2017; Sundararajan et al., 2017; Alvarez-Melis and Jaakkola, 2018b; Chen et al., 2018a). In text classification, for instance, one popular method is to design models that can perform classification using only a rationale, which is a subset of the text selected from the model input that fully explains the model's prediction (Lei et al., 2016; Bastings et al., 2019; Chang et al., 2019). This selective rationalization method, often
Can I find duplicate songs with different names?
I have so many duplicate songs but they have different names.
Is there an application I can use to find and delete the duplicates?
How to find (and delete) duplicate files?
I have a largish music collection and there are some duplicates in there. Is there any way to find duplicate files. At a minimum by doing a hash and seeing if two files have the same hash. ... I'm happy using the command line if that is the easiest way.
Figure 1: An illustration of a text matching rationale for detecting similar forum posts.
trained to choose a small yet sufficient number of text spans, makes it easy to interpret the model's prediction by examining the selected text.
In contrast to classification, very little progress has been made toward rationalization for text matching models. The task of text matching encompasses a wide range of downstream applications, such as similar document recommendation (dos Santos et al., 2015), question answering (Lee et al., 2019), and fact checking (Thorne et al., 2018). Many of these applications can benefit from selecting and comparing information present in the provided documents. For instance, consider a similar post suggestion in a tech support forum as shown in Figure 1. The extracted rationales could provide deeper insights for forum users while also helping human experts validate and improve the model.
In this work, we extend selective rationalization for text matching and focus on two new challenges that are not addressed in previous rationalization work. First, since text matching is fundamentally about comparing two text documents, rationale selection should be jointly modeled and optimized for matching. Second, the method should produce an interpretable alignment between the selected rationales showcasing their relations for the downstream prediction. This is very different from rationaliza
$^{\dagger}$ Our code is publicly available at https://github.
com/asappresearch/rationale-alignment.
5609
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5609-5626
tion for text classification, where the selection is performed independently on each input text and an alignment between rationales is unnecessary.
One popular method for aligning inputs is attention-based models (Bahdanau et al., 2015; Rocktäschel et al., 2015; Rush et al., 2015; Xu et al., 2015; Kim et al., 2018). However, a limitation of neural attention is that the alignment is rarely sparse, thus making it difficult to interpret how the numerous relations among the text spans lead to the model's prediction. Recent work has explored sparse variants of attention (Martins and Astudillo, 2016; Niculae and Blondel, 2017; Lin et al., 2018; Malaviya et al., 2018; Niculae et al., 2018), but the number of non-zero alignments can still be large (Laha et al., 2018). Additionally, because of the heavy non-linearity following most attention layers, it is difficult to truly attribute the model's predictions to the alignment, which means that attention-based models lack fidelity.
We propose to address these challenges by directly learning sparse yet sufficient alignments using optimal transport (OT). We use OT as a building block within neural networks for determining the alignment, providing a deeper mathematical justification for the rationale selection. In order to produce more interpretable rationales, we construct novel variants of OT that have provable and controllable bounds on the sparsity of the alignments. Selecting and aligning text spans can be jointly optimized within this framework, resulting in optimal text matchings. Our model is fully end-to-end differentiable using the Sinkhorn algorithm (Cuturei, 2013) for OT and can be used with any neural network architecture.
We evaluate our proposed methods on the StackExchange, MultiNews (Fabbri et al., 2019), e-SNLI (Camburu et al., 2018), and MultiRC (Khashabi et al., 2018) datasets, with tasks ranging from similar document identification to reading comprehension. Compared to other neural baselines, our methods show comparable task performance while selecting only a fraction of the number of alignments. We further illustrate the effectiveness of our method by analyzing how faithful the model's predictions are to the selected rationales and by comparing the rationales to human-selected rationales provided by DeYoung et al. (2019) on the e-SNLI and MultiRC datasets.
Selective Rationalization. Model interpretability via selective rationalization has attracted considerable interest recently (Lei et al., 2016; Li et al., 2016; Chen et al., 2018a; Chang et al., 2019). Some recent work has focused on overcoming the challenge of learning in the selective rationalization regime, such as by enabling end-to-end differentiable training (Bastings et al., 2019) or by regularizing to avoid performance degeneration (Yu et al., 2019). Unlike these methods, which perform independent rationale selection on each input document, we extend selective rationalization by jointly learning selection and alignment, as it is better suited for text matching applications.
Concurrent to this work, DeYoung et al. (2019) introduce the ERASER benchmark datasets with human-annotated rationales along with several rationalization models. Similarly to DeYoung et al. (2019), we measure the faithfulness of selected rationales, but our work differs in that we additionally emphasize sparsity as a necessary criterion for interpretable alignments.
Alignment. Models can be made more interpretable by requiring that they explicitly align related elements of the input representation. In NLP, this is often achieved via neural attention (Bahdanau et al., 2015; Chen et al., 2015; Rush et al., 2015; Cheng et al., 2016; Parikh et al., 2016; Xie et al., 2017). Many variants of attention, such as temperature-controlled attention (Lin et al., 2018) and sparsemax (Martins and Astudillo, 2016), have been proposed to increase sparsity within the attention weights. However, it is still debatable whether attention scores are truly explanations (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Distance-based methods of aligning text have also been proposed (Li et al., 2019), but they similarly cannot guarantee sparsity or explainability. In this work, we explicitly optimize rationale selection and alignment as an integral part of the model and evaluate the degree to which the alignment explains the model's predictions.
Optimal Transport. The field of optimal transport (OT) began with Monge (1781), who explored the problem of determining a minimal cost assignment between sets of equal sizes. Kantorovich (1942) relaxed Monge's problem to that of determining an optimal transport plan for moving probability mass between two probability distributions.
Since the introduction of a differentiable OT solver by Cuturi (2013), OT has seen many applications in deep learning and NLP, such as topic embedding (Kusner et al., 2015), text generation (Chen et al., 2018b), cross-lingual word embedding alignment (Alvarez-Melis and Jaakkola, 2018a), graph embedding (Xu et al., 2019), and learning permutations (Mena et al., 2018). Peyre and Cuturi (2019) provides an overview of the computational aspects of OT. Unlike prior work, we develop novel additional constraints on the OT problem that produce particularly sparse and interpretable alignments.
Consider two related text documents  $D^{x}$  and  $D^{y}$ . These documents are broken down into two sets of text spans,  $S^{x}$  and  $S^{y}$ , where the text spans can be words, sentences, paragraphs, or any other chunking of text. These text spans are then mapped to vector representations using a function  $g(\cdot)$  (e.g., a neural network), which produces two sets of vectors representing the inputs,  $X = \{\mathbf{x}_i\}_{i=1}^n = \{g(S_i^x)\}_{i=1}^n$  and  $Y = \{\mathbf{y}_i\}_{i=1}^m = \{g(S_i^y)\}_{i=1}^m$ , where  $\mathbf{x}_i, \mathbf{y}_i \in \mathbb{R}^d$ .
We define an interpretable text matching as an alignment between the text spans in  $X$  and  $Y$  that explains the downstream prediction. Following common practice for previous self-explaining models (Lei et al., 2016; Alvarez-Melis and Jaakkola, 2018b), we specify that a desirable model must produce alignments satisfying the following criteria of interpretability.
Explicitness. The alignment between text spans generated by the model should be an observable and understandable component of the model. Our model explicitly encodes the alignment between  $X$  and  $Y$  as a matrix  $\mathbf{P} \in \mathbb{R}_{+}^{n \times m}$  where  $\mathbf{P}_{i,j}$  indicates the degree to which  $\mathbf{x}_i$  and  $\mathbf{y}_j$  are aligned.
Sparsity. In order for the alignment to be interpretable, the alignment matrix  $\mathbf{P}$  must be sparse, meaning there are very few non-zero alignments between the text spans. A sparser alignment is easier to interpret as fewer alignments between text spans need to be examined.
Faithfulness. An interpretable text matching is only meaningful if the model's predictions are faithful to the alignment, meaning the predictions are directly dependent on it. Similarly to previous work, our model achieves faithfulness by using
only the selected text spans (and their representations) for prediction. That is, the selected rationales and alignment should be sufficient to make accurate predictions. In addition to sufficiency, faithfulness also requires that the model output should be easily attributed to the choice of alignment<sup>1</sup>. For simple attribution, we define our model output as either a linear function of the alignment  $\mathbf{P}$  or a shallow feed-forward network on top of  $\mathbf{P}$ .
In the following sections, we introduce optimal transport as a method to produce interpretable text matchings satisfying all three desiderata.
4 Background: Optimal Transport
An instance of the discrete optimal transport problem consists of two point sets,  $X = \{\mathbf{x}_i\}_{i=1}^n$  and  $Y = \{\mathbf{y}_i\}_{i=1}^m$ , with  $\mathbf{x}_i, \mathbf{y}_i \in \mathbb{R}^d$ . Additionally,  $X$  and  $Y$  are associated with probability distributions  $\mathbf{a} \in \Sigma_n$  and  $\mathbf{b} \in \Sigma_m$ , respectively, where  $\Sigma_n$  is the probability simplex  $\Sigma_n := \{\mathbf{p} \in \mathbb{R}_+^n : \sum_{i=1}^n \mathbf{p}_i = 1\}$ . A cost function  $c(\mathbf{x}, \mathbf{y}) : \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$  specifies the cost of aligning a pair of points  $\mathbf{x}$  and  $\mathbf{y}$ . The costs of aligning all pairs of points are summarized by the cost matrix  $\mathbf{C} \in \mathbb{R}^{n \times m}$ , where  $\mathbf{C}_{i,j} = c(\mathbf{x}_i, \mathbf{y}_j)$ .
The goal of optimal transport is to compute a mapping that moves probability mass from the points of  $X$  (distributed according to a) to the points of  $Y$  (distributed according to b) so that the total cost of moving the mass between points is minimized according to the cost function  $c$ . This mapping is represented by a transport plan, or alignment matrix,  $\mathbf{P} \in \mathbb{R}_{+}^{n \times m}$ , where  $\mathbf{P}_{i,j}$  indicates the amount of probability mass moved from  $\mathbf{x}_i$  to  $\mathbf{y}_j$ . The space of valid alignment matrices is the set
$$
\mathbf {U} (\mathbf {a}, \mathbf {b}) := \left\{\mathbf {P} \in \mathbb {R} _ {+} ^ {n \times m}: \mathbf {P} \mathbb {1} _ {m} = \mathbf {a}, \mathbf {P} ^ {\mathrm {T}} \mathbb {1} _ {n} = \mathbf {b} \right\}
$$
since  $\mathbf{P}$  must marginalize out to the corresponding probability distributions  $\mathbf{a}$  and  $\mathbf{b}$  over  $X$  and  $Y$ .
Under this formulation, the optimal transport problem is to find the alignment matrix  $\mathbf{P}$  that minimizes the sum of costs weighted by the alignments:
$$
L _ {\mathbf {C}} (\mathbf {a}, \mathbf {b}) := \min  _ {\mathbf {P} \in \mathbf {U} (\mathbf {a}, \mathbf {b})} \left\langle \mathbf {C}, \mathbf {P} \right\rangle = \sum_ {i, j} \mathbf {C} _ {i, j} \mathbf {P} _ {i, j}.
$$
Note that this optimization is a linear programming problem over the convex set  $\mathbf{U}(\mathbf{a},\mathbf{b})$ . As a result, one of the extreme points of  $\mathbf{U}(\mathbf{a},\mathbf{b})$  must be an optimal solution.
For example, a linear model achieves strong attribution because the importance of each input feature is a constant parameter.
5611
4.1 Sparsity Guarantees
Optimal transport is known to produce alignments that are especially sparse. In particular, the following propositions characterize the extreme point solution  $\mathbf{P}^*$  of  $L_{\mathbf{C}}(\mathbf{a},\mathbf{b})$  and will be important in designing interpretable alignments in Section 5.
Proposition 1 (Brualdi (2006), Thm. 8.1.2). Any extreme point  $\mathbf{P}^*$  that solves  $L_{\mathbf{C}}(\mathbf{a},\mathbf{b})$  has at most  $n + m - 1$  non-zero entries.
Proposition 2 (Birkhoff (1946)). If  $n = m$  and  $\mathbf{a} = \mathbf{b} = \mathbb{1}_n / n$ , then every extreme point of  $\mathbf{U}(\mathbf{a},\mathbf{b})$  is a permutation matrix.
In other words, while the total number of possible aligned pairs is  $n \times m$ , the optimal alignment  $\mathbf{P}^*$  has  $\mathcal{O}(n + m)$  non-zero entries. Furthermore, if  $n = m$ , then any extreme point solution  $\mathbf{P}^*$  is a permutation matrix and thus only has  $\mathcal{O}(n)$  non-zero entries. Figure 2 illustrates two alignments, including one that is a permutation matrix.
Note that the optimal solution of  $L_{\mathbf{C}}(\mathbf{a}, \mathbf{b})$  may not be unique in degenerate cases, such as when  $\mathbf{C}_{i,j}$  is the same for all  $i, j$ . In such degenerate cases, any convex combination of optimal extreme points is a solution. However, it is possible to modify any OT solver to guarantee that it finds an extreme point (i.e., sparse) solution. We provide a proof in Appendix D, although experimentally we find that these modifications are unnecessary as we nearly always obtain an extreme point solution.
4.2 Sinkhorn Algorithm
$L_{\mathbf{C}}(\mathbf{a},\mathbf{b})$  is a linear programming problem and can be solved exactly with interior point methods. Recently, Cuturi (2013) proposed an entropy-regularized objective that can be solved using a fully differentiable, iterative algorithm, making it ideal for deep learning applications. Specifically, the entropy-regularized objective is
$$
L _ {\mathbf {C}} ^ {\epsilon} (\mathbf {a}, \mathbf {b}) := \min  _ {\mathbf {P} \in \mathbf {U} (\mathbf {a}, \mathbf {b})} \left\langle \mathbf {C}, \mathbf {P} \right\rangle - \epsilon \mathbf {H} (\mathbf {P}),
$$
where  $\mathbf{H}(\mathbf{P})$  is the entropy of alignment matrix  $\mathbf{P}$  and  $\epsilon > 0$  controls the amount of entropy regularization. In practice,  $\epsilon$  can be set sufficiently small such that the solution to  $L_{\mathbf{C}}^{\epsilon}(\mathbf{a}, \mathbf{b})$  is a good approximation of the solution to  $L_{\mathbf{C}}(\mathbf{a}, \mathbf{b})$ .
Conveniently,  $L_{\mathbf{C}}^{\epsilon}(\mathbf{a},\mathbf{b})$  has a solution of the form  $\mathbf{P}^{*} = \mathrm{diag}(\mathbf{u})\mathbf{K}\mathrm{diag}(\mathbf{v})$  , where  $\mathbf{K} =$ $e^{-\mathbf{C} / \epsilon}$  and  $(\mathbf{u},\mathbf{v})\in \mathbb{R}_+^n\times \mathbb{R}_+^m$  . The vectors  $\mathbf{u}$  and  $\mathbf{v}$  can be determined using the Sinkhorn-Knopp matrix scaling algorithm (Sinkhorn and Knopp, 1967),
Figure 2: An illustration of two different alignments between the points of  $X$  and  $Y$ , displayed both as a graph (top) and as an (unnormized) alignment matrix  $\mathbf{P}$  (bottom). Alignment 2 (right) corresponds to the special case where  $\mathbf{P}$  is a permutation matrix, which produces an assignment between points in  $X$  and  $Y$ .
which iteratively computes
$$
\mathbf {u} \leftarrow \mathbf {a} \oslash \mathbf {K v} \quad \text {a n d} \quad \mathbf {v} \leftarrow \mathbf {b} \oslash \mathbf {K} ^ {\mathrm {T}} \mathbf {u}
$$
where  $\oslash$  denotes element-wise division.
Since each iteration consists only of matrix operations, the Sinkhorn algorithm can be used as a differentiable building block in deep learning models. For instance, in this work we take  $\mathbf{C}$  as the distance between hidden representations given by a parameterized neural network encoder. Our model performs the Sinkhorn iterations until convergence (or a maximum number of steps) and then outputs the alignment  $\mathbf{P}$  and the total cost  $\langle \mathbf{C},\mathbf{P}\rangle$  as inputs to subsequent components of the model.
5 Learning Interpretable Alignments
Using "vanilla" OT produces sparse alignments as guaranteed by Proposition 1, but the level of sparsity is insufficient to be interpretable. For instance, Alignment 1 in Figure 2 still has a significant number of non-zero alignment values. Motivated by this limitation, we propose to encourage greater sparsity and interpretability by constructing OT problems with additional constraints.
General Recipe for Additional Constraints. Intuitively, an interpretable alignment should be sparse in two ways. First, each text span should be aligned to one or a very small number of spans in the other input text. Second, the total number of
5612
Figure 3: An illustration of the process of computing a one-to-two assignment between the points of  $X$  and  $Y$ . (a) The original points of  $X$  and  $Y$ . (b)  $\hat{X}$  and  $\hat{Y}$  are constructed so that  $\hat{X}$  has two copies of each point in  $X$  and one dummy point and  $\hat{Y} = Y$ . (c) OT is applied to  $\hat{X}$  and  $\hat{Y}$  using uniform distributions a and b, which produces a one-to-one assignment between  $\hat{X}$  and  $\hat{Y}$ . (d) A one-to-two assignment between  $X$  and  $Y$  is extracted from the one-to-one assignment between  $\hat{X}$  and  $\hat{Y}$ .
aligned pairs should be small enough so that the alignment can be easily examined by a human. We modify the OT problem in several ways to guarantee both aspects of sparsity.
We start by forcing the solution to be an assignment, which is a one-to-one (or one-to-few) alignment such that every non-zero entry in the alignment matrix is equal, thereby simplifying interpretability. Alignment 2 in Figure 2 is an example of a one-to-one assignment. We also consider two other constructions, one that makes every text span in the alignment optional and another that directly limits the total number of aligned pairs.
At the core of our construction are two types of auxiliary points that are added to the input point sets  $X$  and  $Y$ :
- Replica points are exact copies of the original points in  $X$  or  $Y$  and can be used to control the sparsity of each point's alignment.
- Dummy points, also known as tariff-free reservoirs in prior work, are points that can be aligned to with 0 cost. Dummy points are used for absorbing unused probability mass in partial transport, where the constraints are relaxed to  $\mathbf{P}\mathbb{1}_m\leq \mathbf{a}$  and  $\mathbf{P}^{\mathrm{T}}\mathbb{1}_n\leq \mathbf{b}$  (Caffarelli and McCann, 2010; Figalli, 2010).
The idea is to add an appropriate number of replica points and dummy points to create  $\hat{X}$  and  $\hat{Y}$  with  $|\hat{X}| = |\hat{Y}| = N$  for some  $N$ . Then by using uniform probability distributions  $\mathbf{a} = \mathbf{b} = \mathbb{1}_N / N$ , Proposition 2 implies that one of the solutions to the OT problem will be a permutation matrix, i.e., a one-to-one assignment between the points in  $\hat{X}$  and  $\hat{Y}$ . Since the points of  $X$  and  $Y$  are included in  $\hat{X}$  and  $\hat{Y}$ , we can directly extract an assignment between  $X$  and  $Y$  from the assignment between  $\hat{X}$  and  $\hat{Y}$ . Figure 3 illustrates the procedure. Note that
the same solution can be attained without explicitly replicating any points by adjusting the probability distributions  $\mathbf{a}$  and  $\mathbf{b}$ , but we use replication for ease of exposition. Also note that the Sinkhorn algorithm is compatible with replica and dummy points and the model remains differentiable.
We now describe three specific instances of this procedure that produce interpretable assignments with different sparsity patterns. Without loss of generality, we assume that  $n = |X| \leq |Y| = m$ .
One-to-  $k$  Assignment. In this assignment, every point in the smaller set  $X$  should map to  $k$  points in the larger set  $Y$ , where  $k \in \{1, 2, \ldots, \lfloor \frac{m}{n} \rfloor\}$ . This will result in a sparsity of  $kn \leq \lfloor \frac{m}{n} \rfloor n \leq m$ .
To compute such an assignment, we set  $\hat{Y} = Y$  and we construct  $\hat{X}$  with  $k$  copies of every point in  $X$  along with  $m - kn$  dummy points. Since  $|\hat{X}| = |\hat{Y}| = m$ , applying OT to  $\hat{X}$  and  $\hat{Y}$  produces a one-to-one assignment between  $\hat{X}$  and  $\hat{Y}$ . As  $\hat{X}$  contains  $k$  replicas of each point in  $X$ , each unique point in  $X$  is mapped to  $k$  points in  $Y$ , thus producing a one-to- $k$  assignment. The remaining  $m - kn$  mappings to dummy points are ignored.
Relaxed One-to-  $k$  Assignment. In a relaxed one-to-  $k$  assignment, each point in  $X$  can map to at most  $k$  points in  $Y$ . As with the one-to-  $k$  assignment, we use  $k$  replicas of each point in  $X$ , but now we add  $m$  dummy points to  $X$  and  $kn$  dummy points to  $Y$ , meaning  $|\hat{X}| = |\hat{Y}| = m + kn$ . Because of the number of replicas, this will produce at most a one-to-  $k$  assignment between  $X$  and  $Y$ . However, since there is now one dummy point in  $\hat{Y}$  for every original point in  $\hat{X}$ , every original point has the option of aligning to a dummy point, resulting in at most  $k$  alignments. Note that in this case, the cost function must take both positive and negative values to prevent all original points from
Constraint	#R of X	#D in X'	#D in Y'	Sparsity (s)
Vanilla	1	0	0	s ≤ n + m - 1
One-to-k	k	m - kn	0	s = kn ≤ m
R one-to-k	k	m	kn	s ≤ kn ≤ m
Exact-k	1	m - k	n - k	s = k ≤ n
mapping to the zero-cost dummy points.
Exact-  $k$  Assignment. An exact-  $k$  assignment maps exactly  $k$  points in  $X$  to points in  $Y$ , where  $k \leq n$ . An exact-  $k$  assignment can be constructed by adding  $m - k$  dummy points to  $X$  and  $n - k$  dummy points to  $Y$ , meaning  $|\hat{X}| = |\hat{Y}| = n + m - k$ . In this case, the cost function must be strictly positive so that original points map to dummy points whenever possible. This leaves exactly  $k$  alignments between original points in  $X$  and  $Y$ .