SlowGuess/ACL_2020_OCR huggingface dataset:
Ryosuke Kohita kohl@ibm.com
Issei Yoshida Hiroshi Kanayama Tetsuya Nasukawa {issei,hkana,nasukawa}@jp.ibm.com
We propose a methodology to construct a term dictionary for text analytics through an interactive process between a human and a machine. The interactive approach helps the creation of flexible dictionaries with precise granularity required in text analysis. This paper introduces the first formulation of interactive dictionary construction to address this issue. To optimize the interaction, we propose a new algorithm that effectively captures an analyst's intention starting from only a small number of sample terms. Along with the algorithm, we also design an automatic evaluation framework that provides a systematic assessment of any interactive method for the dictionary creation task. Experiments using real scenario based corpora and dictionaries show that our algorithm outperforms baseline methods, and works even with a small number of interactions. Also, we provide our dataset for future studies<sup>1</sup>.
Since the emergence of practical interests in text analytics that finds insights from massive documents (Nasukawa and Nagano, 2001), there are several requirements for enhancing valuable discoveries. The one critical issue we tackle in this paper is an effective construction of a term dictionary (Godbole et al., 2010). The term dictionary, which is an arbitrary set of terms, is used in text analytics to represent interesting analysis perspectives (Nasukawa and Nagano, 2001; Nasukawa, 2009); for example, dictionaries of "product names" and "evaluative description" are required for mining customer reputations about products. The motivation of this paper is how to reduce the human workload for the dictionary con
Typical dictionary in previous studies
Synonym of U.S.A U.S.,U.S.A., America, ...
Medicine name Aspirin, Odpivo, Tylenol, ...
Flexible and fine-grained dictionary in this work
Figure 1: Typical dictionaries in previous works (upper) and fine-grained dictionaries in this work (lower)
struction as much as possible. To this end, we establish a methodology of interactive dictionary construction that incrementally captures an analyst's intention starting from a small number of sample terms and enables him/her to effortlessly expand terms in the intended dictionary through suggestions by a machine.
The term dictionary for text analytics is expensive to be constructed because we need to focus more on terms with flexible granularity for in-depth analysis (Takeuchi et al., 2009; Godbole et al., 2010; Mostafa, 2013). For instance, if the analyst wants to examine product evaluation from both its function and appearance, he/she then needs to separately create those dictionaries whose boundaries are vague and overlapped (Figure 1). In short, we need to group any terms the analyst wants together depending on documents and the objective of analysis, which forces an ad hoc construction of the term dictionary. This situation is rather severe in the real-world tasks because the vocabulary size for an exhaustive search of the texts is vast, and the analyst will go through re
https://github.com/kohilin/ IDC-evalset.git
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 789-799 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
peated trial and error of creating dictionaries until he/she reaches findings.
At present, there is a demand for a machine that decreases the cost of the ad hoc dictionary construction. As the dictionary construction can be considered as a type of collecting terms, there is a related research field — set expansion that expands a small set of terms by means of bootstrapping (Pantel and Pennacchiotti, 2006). This approach automatically finds new terms for the given set from documents in accordance with a predefined exploration strategy (Pantel et al., 2009; He and Xin, 2011). Although such an automatic procedure is advantageous for reducing the human workload, the quality of the collected terms is suspicious for a term dictionary. For example, a good analysis requires more fine-grained dictionaries than the original targets in set expansion such as distinct ontological terms (e.g., country name, Shen et al. 2017, 2018).
Several studies have incorporated a human in the term collection process (Godbole et al., 2010; Coden et al., 2012). Specifically, dictionaries are built in an interactive process where the human gives feedback to the machine and the machine suggests candidates based on the given feedback (Alba et al., 2017, 2018). Such a human-in-the-loop approach has been an active topic in other fields as well, for instance, image classification (Cui et al., 2016), dialogue system (Li et al., 2017), and audio annotation (Kim and Pardo, 2018). We can generally expect that a reliable feedback provided by human makes a system more accurate. With respect to dictionary construction, however, experimental results in this vein are limited due to the empirical evaluation by just a few participants and the use of a coarse dictionary as the test items. In short, it is a still open question — what is a critical issue for interactive construction of fine-grained term dictionary for text analytics?
Moving in the same promising direction of leveraging both a human and a machine, we establish a well-defined and effective methodology for constructing the term dictionary. In summary, our contribution in this paper is fourfold: (i) We formulate the interactive process of a term collection, which brings clarity to the problem to be solved (§2). (ii) We develop a method that captures an analyst's intention from a small number of samples with our formulation as the basis (§3). (iii) We
Figure 2: Interactive dictionary construction
propose an automatic evaluation framework that provides a systematic assessment for interactive methods (§4). (iv) Our experimental results show that the proposed method surpasses baseline methods such as set expansion, word embedding and a linear classifier on the crowdsourced dataset. The dataset emulates the real-world scenario of flexible and fine-grained dictionary construction, and we distribute the dataset to the public (§5).
In this section, we provide the definitions and notations used throughout this paper.
First, a term is a string representation of a certain notion such as "apple" and "New York". A dictionary is a collection of terms. A user denotes the person who wants to construct a dictionary, and system denotes the machine that helps the user. Let  $W$  be the whole set of terms in documents. Our objective is to rapidly find as many terms of the user's interest  $U \subset W$  as possible.
As seen in Figure 2, interactive dictionary construction is defined as an iterative process in which each iteration consists of the following steps: 1) User feedback in which the user selects terms for the dictionary from the current candidate terms, and 2) Candidate selection in which the system finds candidate terms for the next user feedback. For the  $i$ -th iteration  $(i = 0,1,2,\ldots)$ , let  $C_i$  be the set of terms that the system finds in the candidate selection step and  $U_i$  be the set of terms that the user selects from  $C_{i-1}$  in the user feedback step as positive examples. Here,  $U_0$  is a special feedback we call seed terms that are directly given by the user first. Note that, because we wish to expand the dictionary, each term in  $C_i$  should be new to the user in the  $(i+1)$ -th iteration.
In the  $i$ -th step of the user feedback  $(i \geq 1)$ , we assume that the user can annotate which terms in  $C_{i-1}$  are in  $U$  without being aware of the whole
Figure 3: Task definition
$U$ . So,  $U_{i} \subset U$  for each  $i$ . Let  $\widetilde{U}_{i} := \cup_{m=0}^{i} U_{m}$  be the set of words of the user's interest that is found by the end of the  $i$ -th iteration.
However, it is impractical to define our objective as an optimization problem for the asymptotic convergence of  $\widetilde{U}_i$  because the user feedback is done by a human, and  $i$  cannot be large. Hence, we try to maximize  $|C_i \cap U|$ , the number of suggested terms that match the user's interest. Also, since  $C_i$  is manually selected by a human user, the proper size of  $C_i$  is practically limited to  $5 \sim 10$ .
Figure 3 shows the steps from setting the seed terms to giving the first feedback to the first candidates. Using the example in Figure 2,  $U_{0}$  is {Formal},  $C_{0}$  is {Nice, Traditional},  $U_{1}$  is {Traditional}, and  $C_{0} \backslash U$  is {Nice}. The system then next selects  $C_{1}$  based on  $U_{0}$  and  $U_{1}$  (i.e.,  $\widetilde{U}_{1}$ ) from  $W$  except for the shown terms  $C_{0} \cup \widetilde{U}_{1}$ . It is important that we design the system to be effective so that the overlapped area of  $C_{i}$  and  $U$  becomes larger.
There are two major challenges for this problem; one is number of seed terms, and the other is term overlaps of different dictionaries. In terms of the first issue, we have only a few seed terms for the target dictionary at the first iteration. If the system requires more seed terms, the advantage of the system drops because it contradicts our purpose to decrease the human workload in constructing the dictionary. Therefore, we need a method that captures the user's intention from a smaller number of samples. In terms of the second issue, identifying terms of user's interest is difficult because boundaries between dictionaries are often overlapped in text analytics as seen in Figure 1. In other words, the system need to be more sensitive to subtle semantic differences only with a few feedbacks.
In this section, we first describe a previous candidate selection model, SetExpan algorithm (Shen et al., 2017) that inspired our method (§3.1). Subsequently, we introduce our method as the weighted version of SetExpan with improvements in dealing with interactive settings (§3.2~). Throughout this section, we discuss the  $i$ -th step of candidate selection for a certain  $i$ . For simplicity,  $C_i$  and  $\widetilde{U}_i$  are denoted as  $C$  and  $\widetilde{U}$ , respectively.
3.1 Candidate Selection: Similarity Scoring based on Feature Collection
As we stated in §2, the objective of the task is to suggest  $C$  that contains as many terms in  $U$  as possible. Recall that  $\widetilde{U}$  is a set of positive examples for terms of the user's interest that are found in previous steps. Following the strategy taken in set expansion (Shen et al., 2017), a straightforward and reasonable approach to determine  $C$  is to define  $Sim(e, e'|F)$  which returns a similarity score for two terms  $e$  and  $e'$  based on a set of features  $F$ , and then to select terms that are most similar to the positive terms in  $\widetilde{U}$ .
The issue is how to obtain the ideal  $F$  that assigns a higher score to terms potentially included in  $U$ . Shen et al. (2017) formulates this feature selection problem as choosing features with the number of fixed-size  $Q$  so that the positive terms are most similar to each other:
$$
F ^ {*} = \arg \max  _ {| F | = Q} \sum_ {1 \leq i \leq j \leq n} S i m \left(e _ {i}, e _ {j} \mid F\right), \tag {1}
$$
where  $\widetilde{U} := \{e_1, \ldots, e_n\}$ . They propose using the Jaccard coefficient for  $Sim(e_i, e_j | F)$ , which narrows the optimization problem to a binary decision on whether to use each feature. This combinatorial problem is NP-hard; hence, they use heuristics to choose an approximation of  $F^*$ .
3.2 From Feature Selection to Feature Weighting with Predefined Similarity
Instead of explicitly choosing features to use in the similarity calculation, we consider using all of the possible features  $\{f_1,\ldots ,f_L\}$  with the weight  $w_{k}\in \mathbb{R}$  for each feature  $f_{k}$ . In addition, we define our optimization problem as finding the best  $w_{k}$  for  $f_{k}$ $(k = 1,\dots ,L)$ .
Let us develop a formula that extends (1) and takes  $w_{k}$  into consideration. First, in such a formula,  $Sim(e_i,e_j|F)$  should be a weighted sum of
the similarity score for each feature  $f_{k}$  , denoted as  $Sim(e_i,e_j|f)$  . By replacing  $F$  with  $\pmb{w}$  in the expression of the similarity function, we have
$$
\operatorname {S i m} \left(e _ {i}, e _ {j} \mid \boldsymbol {w}\right) = \sum_ {k = 1} ^ {L} w _ {k} \cdot \operatorname {S i m} \left(e _ {i}, e _ {j} \mid f _ {k}\right). \tag {2}
$$
Next, to define the similarity between a term  $e$  and  $\widetilde{U}$ , we assume that the similarity is the average of similarities between  $e$  and  $e_i \in \widetilde{U}$ , that is,
$$
\operatorname {S i m} (e, \widetilde {U} | \boldsymbol {w}) := \frac {1}{n} \sum_ {i = 1} ^ {n} \operatorname {S i m} (e, e _ {i} | \boldsymbol {w}). \tag {3}
$$
The initial formulation of our optimization problem is thus as follows:
$$
\boldsymbol {w} ^ {*} = \arg \max  _ {\boldsymbol {w}} \sum_ {1 \leq i \leq n} \operatorname {S i m} \left(e _ {i}, \widetilde {U} | \boldsymbol {w}\right). \tag {4}
$$
We show in the Appendix that our formulation of (4) can be considered as the weighted version of (1) under the natural condition that  $\text{Sim}(e_i, e_i | f_k) = \text{Sim}(e_j, e_j | f_k)$  for any  $i$ ,  $j$ , and  $k$ , and  $\sum_{k=1}^{L} w_k = 1$ . It is easy to set  $\text{Sim}(e, e'|f_k)$  satisfying this condition. For a feature  $f_k$ , we define a vector  $\mathbf{v}_{f_k}(e)$  of an  $e$  and define  $\text{Sim}(e, e'|f_k)$  as the standard inner product of  $\mathbf{v}_{f_k}(e)$  and  $\mathbf{v}_{f_k}(e')$ . Then by normalizing all these vectors,  $\text{Sim}(e_i, e_i | f_k) = \| \mathbf{v}_{f_k}(e_i) \| = 1$  holds for any  $i$ ; hence, the condition is satisfied, and that is a conventional cosine similarity of word vectors (Levy et al., 2015). Thus, any mapping from  $W$  to a vector space is available as a feature such as the tfidf of terms and discrete features (Manning et al., 2008), word2vec (Mikolov et al., 2013), or GloVe (Pennington et al., 2014). Note that the dimension of the vector space may be different among the features.
Hence, we assume  $\pmb{v}_{f_k}(e)$  is defined for each feature  $f_{k}$  and any  $e$ . When we use  $Sim(e,e^{\prime}|f_{k})$ $= \pmb{v}_{f_k}(e)\cdot \pmb{v}_{f_k}(e')$ , (2) is computed by
$$
\operatorname {S i m} \left(e _ {i}, e _ {j} \mid \boldsymbol {w}\right) = \sum_ {k = 1} ^ {L} w _ {k} \cdot \boldsymbol {v} _ {f _ {k}} \left(e _ {i}\right) \cdot \boldsymbol {v} _ {f _ {k}} \left(e _ {j}\right), \tag {5}
$$
and by a simple calculation, (3) is equal to
$$
\operatorname {S i m} (e, \widetilde {U} | \boldsymbol {w}) = \sum_ {k = 1} ^ {L} w _ {k} \cdot \boldsymbol {v} _ {f _ {k}} (e) \cdot \boldsymbol {v} _ {f _ {k}} (\widetilde {U}), \tag {6}
$$
where  $\pmb{v}_{f_k}(\widetilde{U})\coloneqq \frac{1}{n}\sum_{i = 1}^{n}\pmb{v}_{f_k}(e_i)$  is the centroid vector for  $\{\pmb{v}_{f_k}(e_i)\}_{i = 1,\dots,n}$  in the feature space of
Figure 4: Feature weighting puts weights on feature spaces by placing terms of user's interest nearby.
$f_{k}$ . We simply call  $\pmb{v}_{f_k}(\widetilde{U})$  the centroid of  $\widetilde{U}$ . Formulas (5) and (6) demonstrate that the similarity between any two terms can be measured by combining the characteristics of the  $L$  different feature spaces. We "select" the feature spaces in which terms in  $\widetilde{U}$  become similar to each other by adjusting the weights, as shown in Figure 4.
Note that our feature weighting formulation is categorized as a conventional linear regression that finds  $f_{k}$  characterizing  $\widetilde{U}$  via the weights. Instead of calculating the weights for bare features of each term, our method estimates those for differently predefined feature spaces (i.e., the similarity scores in these spaces). It aims to mitigate the difficulty of finding optimal weights for the vast number of features only from few labeled samples. However, the drawback is that this sacrifices a model's degree of freedom; therefore, we test the effectiveness of our proposed model compared to an ordinary linear classifier in the experiment.
3.3 Optimization by User Feedback
Although the initial formulation (4) proved to be a natural extension of the discrete version of feature selection, it does not always work as expected. In this section, we discuss the reason for this and how we can improve the initial formulation of our optimization problem.
By substituting (2) and (3) into (4), the objective  $\sum_{1\leq i\leq n}Sim(e_i,\widetilde{U} |w)$  is a linear function of  $w$ . Assuming that  $\sum_{k = 1}^{L}w_{k} = 1$ , the optimal  $w$  is determined by putting all the weight values on a particular feature space which has the highest score in the averaged similarity between the terms in  $\widetilde{U}$  and the centroid of  $\widetilde{U}$ . This is equivalent to selecting only one feature space for the similarity computation. Such extreme optimization is not suitable for our interactive setting because the target dictionary is obscure, especially in earlier iter
ations. We want the system to diversify the candidate terms to broadly cover the user's interests and allow the user to discover related vocabularies for a customized dictionary. To address this issue, we modify our formulation of (4) as
$$
\boldsymbol {w} ^ {*} = \arg \max  _ {\boldsymbol {w}} \min  _ {1 \leq i \leq n} \operatorname {S i m} \left(e _ {i}, \widetilde {U} | \boldsymbol {w}\right). \tag {7}
$$
We maximize the minimum similarity score between a term in  $\widetilde{U}$  and the centroid of  $\widetilde{U}$ . The idea here is to reduce the distance between the farthest positive term and the centroid. This strategy is analogous to those used in active learning, where examples near the separating hyperplane are actively leveraged (Schohn and Cohn, 2000). Our objective function  $\min_{1\leq i\leq n}\operatorname {Sim}(e_i,\widetilde{U} |w)$  is a concave function of  $w$  (see Appendix); therefore, we can solve it by (for example) gradient descent.
We can also leverage negative feedback, i.e., unselected terms in  $C$ , to make the system more sophisticated. Let  $N := C \setminus U = \{z_1, \ldots, z_m\}$ , then we can extend (7) by
$$
\begin{array}{l} \boldsymbol {w} ^ {*} = \arg \max  _ {\boldsymbol {w}} \left\{\min  _ {1 \leq i \leq n} \operatorname {S i m} \left(e _ {i}, \widetilde {U} | \boldsymbol {w}\right) \right. \\ - \max  _ {1 \leq j \leq m} S i m \left(z _ {j}, \widetilde {U} | \boldsymbol {w}\right) \}. \tag {8} \\ \end{array}
$$
The second term on the right-hand side of (8) increases the distance between the closest negative term and the centroid of  $\widetilde{U}$ . Again the objective function of (8) is a concave function of  $w$ ; thus, the information of both positive and negative examples is taken into consideration to learn the optimal  $w^{*}$ .
3.4 Feedback Denoising
Although our min-maximize optimization strategy diversifies candidates, it may be disadvantageous in terms of the system being affected by outliers. It happens that several terms in  $\widetilde{U}$  (especially for manually fed terms such as seeds) distribute differently in possessing feature spaces compared to the rest of the positive terms. Such a case holds up the learning because the maximum similarity score of the outliers to the centroid is low. The left side of Figure 5 shows an example of this problem: specifically, the system cannot put a higher weight value on  $f_{1}$  because the optimization target, which is the most distant one from the centroid ("watermelon" in this case), is biased to  $f_{2}$ .
Feedback denoising is a simple solution to this problem. We apply a clustering algorithm (e.g.,
Figure 5: The difference in terms used in learning (blue) with/without feedback denoising.
K-Means) to terms in  $\widetilde{U}$ , and obtain  $K$  term sets  $\widetilde{U}_{(0)},\widetilde{U}_{(1)},\dots,\widetilde{U}_{(K)}$ . Then, we conduct the optimization by replacing  $\widetilde{U}$  in (7) and (8) with  $\widetilde{U}_{(K^*)}$  where  $K^{*} = \arg \max_{K}|\widetilde{U}_{(K)}|$ , that is, the majority class among terms in  $\widetilde{U}$  as shown in the right side of Figure 5. This is effective for denoising irregular terms with respect to feature distribution, and for guiding the system to a promising  $\boldsymbol{w}^{*}$ .
In this section, we explain an automatic evaluation framework for interactive dictionary construction. By using a predefined dictionary as the oracle dictionary  $U^{*}$ , we emulate the manual feedback process and apply a new evaluation metric to estimate the effectiveness of building a dictionary with consideration of the human interaction.
4.1 Human Emulation
We describe the emulation process with  $U^{*}$ , and the entire flow of the emulation procedure is in Algorithm 1. At the beginning of the emulation process, a small number of seed terms are randomly chosen from  $U^{*}$ , and  $U_{0}$  is initialized with them (l.1). The number of iterations  $I$  (l.2) and the number of suggested terms per iteration  $|C|$  (l.3) are also determined. The iteration consisting of user feedback and candidate selection is then launched. In every  $i$ -th iteration, the system first suggests the  $C_i$  based on the known positive terms  $\widetilde{U}_{i-1}$  (l.5). After receiving the suggested  $C_i$ , the automatic evaluation process takes the intersection of  $C_i$  and  $U^{*}$ , and records the overlapped terms as  $U_i$  (l.6). It also takes the difference set of  $C_i$  and  $U^{*}$  as the negative terms  $N_i$  (l.7). If the system is trainable, its training process runs before moving to the next iteration (l.8 - 10).
Algorithm 1 Human emulation with oracle dictionary
1: SET seed terms  $U_{0}$  from  $U^{*}$
2: SET number of iterations  $I$
3: SET number of suggested terms per iteration  $|C|$
4: for  $i = 1$  to  $I$  do
5:  $C_i \gets$  Suggest from  $\widetilde{U}_{i-1}$
6:  $U_{i}\gets C_{i}\cap U^{*}$
7:  $N_{i}\gets C_{i}\backslash U^{*}$
8: if System is trainable then
9: Run training with  $\widetilde{U}_i$  (and  $\widetilde{N}_i$ )
0: end if
11: end for
4.2 A Metric for Effectiveness Estimation
In addition to the automatic evaluation process, we introduce a new metric that takes the interaction quality into account when evaluating the accuracy of the candidate selection.
The final goal of dictionary construction is to obtain a complete set of terms consistent with  $U^{*}$ ; however, there is a limitation stemming from a user's workload in real scenarios. Given that an effective system should suggest terms of user's interest in earlier iterations, we propose weighted coverage per iteration (WCpI) as the evaluation metric for interactive dictionary construction:
$$
W C p I = \frac {\sum_ {i = 1} ^ {I} (1 - \alpha) ^ {i - 1} \frac {| \widetilde {U} _ {i} |}{\min \{i | C | , | U ^ {*} | \}}}{\sum_ {i = 1} ^ {I} (1 - \alpha) ^ {i - 1}}, \tag {9}
$$
where  $\alpha$  is the hyperparameter to adjust the importance of the iteration number. We illustrate the intuition of  $WCpI$  in Figure 6.  $WCpI$  is an area ratio of accumulated positive terms from system suggestions to its upper bound in each iteration. In short, it measures how many correct suggestions the system can provide in the comparison with a "perfect" system that never suggests unrelated terms.
We can also regulate the importance of iteration number by adjusting  $\alpha$ . Specifically, a larger value of  $\alpha$  underestimates the importance of terms found in the later iterations, in other words, it attaches importance to terms found in the earlier iterations. As an intuitive explanation based on an actual scenario,  $\alpha$  is like representing a constant probability for the user to quit dictionary construction midway through. The graphs in Figure 6 compare the calculation of  $WCpI$  for the same system suggestions. The right one with  $\alpha = 0.1$ , in which we as
Figure 6: Weighted coverage per iteration  $WCpI$ . TheThe x- and y-axes are the number of iterations and accumulated positive terms, respectively. The blue and red areas represent the upper bound and system performance, respectively. The left side is for  $\alpha = 0.0$  and the right side is for  $\alpha = 0.1$  when  $|C| = 10$ ,  $|U^{*}| > 100$ ,  $|U_{i}| = 10 - i$ , and  $I = 10$ .