SlowGuess/NAACL_2021_OCR huggingface dataset:
Karine Chubarian $^{\S}$  = Abdul Rafae Khan $^{\dagger}$  = Anastasios Sidiropoulos $^{\S}$  = Jia Xu $^{\dagger}$  *
$^{\dagger}$ Department of Computer Science, Stevens Institute of Technology
$^{\S}$ Department of Computer Science and Technology, University of Illinois at Chicago
{kchuba2, sidiropo}@uic.edu
Deep Learning-based NLP systems can be sensitive to unseen tokens and hard to learn with high-dimensional inputs, which critically hinder learning generalization. We introduce an approach by grouping input words based on their semantic diversity to simplify input language representation with low ambiguity. Since the semantically diverse words reside in different contexts, we are able to substitute words with their groups and still distinguish word meanings relying on their contexts. We design several algorithms that compute diverse groupings based on random sampling, geometric distances, and entropy maximization, and we prove formal guarantees for the entropy-based algorithms. Experimental results show that our methods generalize NLP models and demonstrate enhanced accuracy on POS tagging and LM tasks and significant improvements on medium-scale machine translation tasks, up to +6.5 BLEU points. Our source code is available at https://github.com/abdulrafae/dg.
Natural Language Understanding has seen remarkable success with the rise of Deep Learning. However, human languages' variety and richness result in high-dimensional inputs to NLP models, increasing learning complexity and error rates. First, open-vocabulary inputs inevitably bring rare and out-of-Vocabulary words (OOVs). Second, network complexity increases with input dimension, specifically the "curse of dimensionality" makes learning difficult on medium and small datasets.
This paper addresses these limitations by introducing new grouping methods to compute alternative language representations that simplify textual inputs. We currently have alternative language representations, such as Pinyin, Metaphone, logogram,
and Emoji, that exist in natural languages and that have been shown to improve various NLP applications (Du and Way, 2017; Liu et al., 2018; Khan et al., 2020). While these representations can help, they are not developed for NLP performance. Our goal is to design algorithms for computing new language representations specifically to enhance NLP performance in this work. We ask:
"Can we compute a generalized language representation to improve NLP applications?"
An intuitive approach to answering this question is to group similar words in training and test sets and replace each word with its group. A word grouping viewed as a many-to-one mapping function can significantly reduce the vocabulary size that lowers the input feature dimensions leading to a generalized NLP model learning.
For example, let us take two sentences: (a) "you ask me"; (b) "she tells me." There are five words "ask", "she", "tells", "me", and "you" in the vocabulary. Grouping words into "A" and "B" will reduce the vocabulary size to two, resulting in a simplified language representation. We can apply conventional word clustering to group words after embedding words into a vector space and measuring their distances with cosine similarity. However, clustering can map different sentences to the same sequence of groups, making them indistinguishable. In our example, we cluster similar pronouns "you", "she" and "me" into one group indicated by "A", and verbs "ask", "tells" by "B". Then both sentences are rewritten as "A B A." The distinct meanings of the two original sentences are lost. However, if we group diverse semantic words, namely, "you", "tell" as "A"; "she", "me", "ask" as "B", then we maintain two samples of (a) "A B B." and (b) "B A B." So, the distinct meanings of the two sentences are retained. This example illustrates the need to group words so that each sentence is uniquely represented. Now, to generalize this idea,
"How can we design an algorithm that simplifies
Author ordered alphabetically. Jia Xu is the corresponding author of this paper.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3217-3228 June 6-11, 2021. ©2021 Association for Computational Linguistics
language representation while preserving meaning expressiveness?
Our key observation is that the context of semantically diverse words varies more than that of semantically close words. In our approach, we measure semantic similarity using the cosine of word embeddings, learned based on context, see (Mikolov et al., 2013; Bojanowski et al., 2017; Pennington et al., 2014). Thus, similar contexts indicate semantic similarity and vice versa. In this way, our diverse grouping uses context to distinguish words from the same group, leading to a more expressive representation.
In this paper, we introduce five novel algorithms in three types that group semantically diverse words together. We develop novel theoretical methods for diverse grouping and port them in our NLP context. We begin by considering random sampling grouping. Next, we develop a grouping algorithm based on geometric distances by designing an algorithm that computes a partition of a set of points in some metric space to maximize the sum of intra-group distances. This approach is essentially the opposite of the objectives used in clustering problems, such as  $k$ -means (Forgy, 1965) and  $k$ -medians (Jain and Dubes, 1988), where one seeks to minimize a monotonic function of intra-group distances. Finally, we present a grouping algorithm to maximize diversity by maximizing the unigram entropy of the representation.
We show that the unigram entropy algorithm is  $\frac{C - 1}{4C + 4\varepsilon C}$ -approximations of the optimal solutions of maximizing the entropy of the new representation, where  $C$  is the number of groups, and  $\varepsilon$  is a small positive real number. This bound means that in the worst case, our algorithm is about 1/4 away from the optimal, while in typical cases, it could be very close to the optimal. Importantly, our theoretical results' outcomes show their usefulness in NLP tasks after we appropriately adjust them. In our experiments, each of the above methods significantly enhances the NMT accuracy by up to 6.5 BLEU points (36.9% relatively).
1. Diversity Grouping Algorithms. We introduce various algorithms that group semantically diverse words together based on random sampling, geometric distances, and entropy maximization (§3).
2. Formal guarantees. We provide provable guarantees for our entropy-based algorithm (§4).
3. Applications in NLP. Importantly, we apply the above algorithms to NLP applications, and we show that they significantly enhance prediction accuracy. (§5).
While typical word clustering (Baker and McCallum, 1998; Martin et al., 1998; Feng et al., 2020) (or word class (Halteren et al., 2001)) methods collect similar words together, while our method groups semantically diverse words together. However, unlike the common use of clustering to smooth unseen words, our goal is to deduce the input sentence's dimension by grouping diverse words so that a word-group sequence uniquely represents a word sequence.
Our diverse grouping approach is also close to the sparse representation (Wright et al., 2008), which makes the network parameter matrices sparse without changing its dimension. Our methods reduce the Neural Network (NN) input dimension.
Such a dimension reduction can be seen as a kind of regularization on NNs. There have been many types of NN regularization methods. (Louizos et al., 2018) adds a parameter norm penalty to the objective function, (Bertsekas, 2014) adds constrained optimization, and many works exploit the sub-structure of network models, such as dropout, early stopping, and weight decay. Those approaches are very popular but may limit the capacity of models, while our methods benefit from indomain linguistic knowledge. Work such as (Wang et al., 2018) adds augmented data (e.g., noisy data, pseudo data, etc.) but is a domain-dependent approach that inventively increases the training time.
Our work is perpendicular to the successful research in word embedding, whereby a word is mapped one-one onto a real number vector trying to preserve word pair distances. In contrast, our methods map many words into one group in a discrete space. Also, our systems build on BPE, but we do not decompose and recombine words. Therefore, our methods are additive to any improved word embedding (May et al., 2019), or BPE (Provilkov et al., 2020) versions.
Different from the inspiring work that uses Soundex, NYSIIS, Metaphone, logogram (Khan et al., 2020), Pinyin (Du and Way, 2017; Liu et al., 2018), skip-ngram (Bojanowski et al., 2017), and Huffman coding (Chitnis and DeNero, 2015; Khan
Algorithm 1 Random Grouping
Input: Vocabulary of words  $\mathbb{V}$ , a phonetic encoding
Parameter:
Output: Grouping  $\gamma$
1: Perform a phonetic encoding (e.g. Metaphone) as baseline
2: Initialize the current group  $i\gets 1$
3: for each unique phonetic encoding do
4:  $k \gets$  "how many words are mapped"
5: Sample  $\{v_{j}\}_{j = 1}^{k}$  from  $\mathbb{V}$  uniformly at random
6: Set  $\gamma(v_1) \gets \gamma_i, \dots, \gamma(v_k) \gets \gamma_i$
7: Set  $\mathbb{V} \gets \mathbb{V} \setminus \{v_1, \ldots, v_k\}$
8:  $i\gets i + 1$
9: end for
10: return  $\gamma$
et al., 2020), our study aims to develop new artificial algorithms that lower the dimensions of the textual inputs with smaller vocabularies.
3 Algorithms for Diverse Grouping
We now present our algorithms. We denote the set of words by  $\mathbb{V}$ , and the set of groups by  $\nu$ , i.e.,  $\nu$  is a subset of the powerset of  $\mathbb{V}$ . Each grouping can be encoded as a function  $\gamma$  that maps each word  $w \in \mathbb{V}$  to some group  $\gamma(w) \in \mathcal{V}$ .
3.1 Random Grouping
Our first approach computes a random grouping, as shown in Algorithm 1. This algorithm's complexity is  $\mathcal{O}(\mathbb{V})$ , where  $V$  is vocabulary size.
We map each word to a group chosen uniformly at random.  $C$  is a hyperparameter indicating the total number of groups. Because it is expensive to tune  $C$  with exhaustive search, we set  $C$  as the total number of Metaphones in English, inspired by previous work (Du and Way, 2017; Liu et al., 2018) in which phonetics improves NMT in specific languages. Furthermore, each group's size follows the natural phonetic encoding distribution (e.g., Metaphone (Philips, 1990)) by considering each phonetic encoding as a group. For example, each Metaphone is considered as a group, and the number of groups in the random grouping is set to the number of unique Metaphone.
Algorithm 2 extends Algorithm 1 by learning a Poisson or Gaussian model for the distribution of group sizes. We fit the distribution of the Meta
Algorithm 2 Poisson/Gaussian-based Random Grouping
Parameter: Groups  $[\gamma_1,\dots ,\gamma_C],C\in \mathbb{N}$
1: for  $1 \leq i \leq C$  do
2: Randomly sample the group size  $k$  from Poisson/Gaussian distribution (which is trained on the English Metaphone distribution)
3: Sample  $\{v_{j}\}_{j = 1}^{k}$  from  $\mathbb{V}$  uniformly at random
4: Set  $\gamma(v_1) \gets \gamma_i, \dots, \gamma(v_k) \gets \gamma_i$
5: Set  $\mathbb{V} \gets \mathbb{V} \setminus \{v_1, \ldots, v_k\}$
6: end for
7: return  $\gamma$
phone group sizes into a Poisson or Gaussian distribution. Then, we sample the group size according to this Poisson or Gaussian distribution. Finally, we sample words for each group uniform randomly.
3.2 Distance-Based Diverse Grouping
We now introduce our grouping algorithm, which uses distances on the vector representations of words. The complexity of this algorithm is  $\mathcal{O}(\mathbb{V}^2)$ .
Our approach is described in Algorithm 3, which is inspired by the classical 2-approximation algorithm for  $k$ -center clustering (Gonzalez, 1985). Our algorithm works as follows: randomly pick a word from the vocabulary and add it to the list  $L'$ . Pick the second word that is the furthest from the first word, pick the third word which is furthest from the closest of the two selected words, and so on. Finally, for each group size  $k$  that follows a Metaphone encoding size distribution, group the top  $k$  words into group one and remove those  $k$  words from the list. This process is performed iteratively until all words are assigned. We use cosine-similarity to measure the pairwise distance between words. Figure (1) illustrates the work of the algorithm.
3.3 Maximum Entropy-Based Unigram Diverse Grouping
We now present our grouping algorithm, which maximizes unigram entropy. Our ultimate goal is to maximize the information kept (or reduce the infor
1: Embed  $\mathbb{V}$  in  $\mathbb{R}^N$  using e.g. word2vec
2:  $\mathbb{W}\gets$  resulting embedding
3: Randomly pick  $w_0 \in \mathbb{W}$ , append  $w_0$  to the ranked list  $L'$
4: for  $1 \leq j \leq |\mathbb{V}|$  do
5: maxmin = 0
6: for all  $w_{i}\in \mathbb{W}\setminus L^{\prime}$  do
7: Find mindist  $i\gets \min_{v\in L^{\prime}}\| w_{i} - v\|_{2}$
8: if mindist  $i >$  maxmin then
9: Set  $\text{maxmin} \gets \text{mindist}_i$ ,  $W \gets w_i$
10: end if
11: end for
12: Append  $W$  in  $L^{\prime}$
13: end for
14: Perform a phonetic encoding (e.g. Metaphone)
15:  $i\gets 0$
16: for each encoding  $\& i + +$  do
17: Assign the encoding size as  $k_{i}$
18: end for
19: for  $1 \leq i \leq C$  do
20: Set  $\gamma(v) \gets \gamma_i$  for the top  $k_i$  points of  $L'$ .
21: Remove the top  $k_{i}$  points from  $L^{\prime}$
22: end for
23: return  $\gamma$
Figure 1: An example of Distance-Based Diverse Grouping
mation loss) from the original input sentences in the newly coded sentences with a reduced vocabulary. Distance-based diverse grouping does not consider the probability (relative frequency) of each element (original word), i.e., the input distribution. For example, if a word occurs very frequently, e.g., "the", which can be followed by many words (different nouns), then the context of "the" cannot help much to distinguish its meaning. Therefore, it is less ambiguous to assign a frequent word ("the") than an
infrequent word to a unique codeword without sharing the codeword. Shannon entropy provides the quantitative measure on information considering such an input distribution.
Importantly, Maximum Entropy-Based Unigram Diverse Grouping (Entropy) is a more efficient algorithm, with a complexity of  $N^{\mathcal{O}(1)}$ , where  $N$  is the number of running words in training. Furthermore, we provide a provable guarantee of about  $\frac{1}{c}$ -approximation.
The entropy-based diverse grouping aims to maximize the diversity of group assignments in the given text with respect to its entropy. Because the entropy is maximal when the underlying distribution is as close to uniform as possible, this objective captures the diversity requirement. As an illustration, consider the following text: (1) "she is running very fast"; (2) "he is running very fast"; (3) "running is very popular today." We want to form three groups. This text has a length fourteen; thus, a grouping with high entropy aims to keep the frequency of each group around  $\frac{5}{14}$ . Therefore, the frequent words like "running" and "is" are likely to be grouped apart; infrequent words will be spread among groups uniformly. For instance, the grouping  $\{1$ : running, fast, late\},  $\{2$ : is she today\},  $\{3$ : he very popular\} has a group frequency of  $\frac{5}{14}$ ,  $\frac{5}{14}$ ,  $\frac{4}{14}$  hence achieving high entropy. Furthermore, the grouped words appear to be diverse enough so that each pair of groups  $\{11, 12, \ldots, 33\}$  appears exactly once or twice after we perform the grouping. We consider the entropy with respect to a distribution induced by the relative frequencies of group unigrams.
Formally, for any group  $\gamma_{i}$  we can define a relative frequency of a group as
$$
c _ {\gamma_ {i}} = \sum_ {w \in \mathbb {V}: \gamma (w) = \gamma_ {i}} F _ {w}
$$
where  $F_{w}$  is a relative frequency of a word  $w$ . If the group is empty, its relative frequency is 0. The unigram entropy of a grouping with  $\mathcal{V} = [\gamma_1,\dots ,\gamma_C]$  is
$$
H (\gamma) = - \sum_ {i = 1} ^ {C} c _ {\gamma_ {i}} \log c _ {\gamma_ {i}} \tag {1}
$$
We are interested in a grouping that maximizes (1).
3.3.1 Algorithms for Unigram Entropy Diverse Grouping
We show how to compute a grouping for (1) by adapting the approximation algorithm for submod
ular maximization under matroid constrains due to (Lee et al., 2009). In our terminology, their algorithm applies three operations to all possible pairs  $(w,\gamma_{i})$  where  $w\in \mathbb{V}$  and  $\gamma_{i}\in \mathcal{V}$ . It terminates when for every  $(w,\gamma_{i})$  each operation is either impossible to perform or the resulting entropy gain is below  $(1 + \varepsilon /(C|\mathbb{V}|)^4)H_{\mathrm{old}}$  where  $H_{\mathrm{old}}$  denotes an entropy of the grouping before the operation. These operations are:
1. Put a word  $w$  into a group  $\gamma_i$
2. Remove a word  $w$  from a group  $\gamma_{i}$
3. Remove a word  $w$  from a group  $\gamma_{i}$  and then put another word  $v$  into a group  $\gamma_{j}$  (we allow either  $w = v$  or  $\gamma_{i} = \gamma_{j}$ ).
After we find the initial grouping, some of the words may remain unassigned. We note that in general, adding new words to a grouping may decrease the entropy. As an example, assume that we have two groups  $\gamma_{1},\gamma_{2}$  with  $c_{\gamma_1} = 0.25,c_{\gamma_2} = 0.5$  and an ungrouped word  $w$  with  $F_{w} = 0.25$ . The current entropy of a grouping is  $-0.25\log 0.25 - 0.5\log 0.5$ . Setting  $\gamma (w) = \gamma_{2}$  means that the contribution of  $c_{\gamma_2}$  is now  $-0.75\log 0.75 < -0.5\log 0.5$  hence the total entropy decreased. To minimize the potential entropy loss, we map ungrouped words to a group  $\gamma_{j}$  with the smallest partial entropy  $G(c_{\gamma_j}) = -c_{\gamma_j}\log (c_{\gamma_j})$ .
Algorithm 5 explains the detail of the unigram entropy diverse grouping algorithm, respectively. We give proofs to this algorithm with the main result stated as follows:
Theorem 1. Given any precision parameter  $\varepsilon >0$ , Algorithm 5 runs in polynomial time, and computes a grouping that is a  $\frac{C - 1}{4C + 4\varepsilon C}$ -approximation to the maximum unigram entropy.
Roughly speaking, our algorithms are about 1/4 away from optimal of maximizing the entropy. In typical cases, our algorithms could be very close to the optimal. Section 4 describes the details of the proofs.
4 Entropy Maximization as Submodular Maximization
In order to apply the optimization techniques from (Lee et al., 2009), which we briefly describe in Section 3.3.1, we need to use a different representation of grouping. We view a grouping  $\gamma$  as the set of all pairs  $(w,\gamma_{i})$  where  $\gamma (w) = \gamma_{i}$ . For
Algorithm 4 Initial grouping (Lee et al., 2009)
Input: Vocabulary of words  $\mathbb{V}$ , relative frequencies  $F_{w}$
Parameter: Groups  $[\gamma_1,\dots ,\gamma_C],C\in \mathbb{N}$  precision parameter  $\varepsilon$
Output: Grouping  $\gamma': \mathbb{V} \to \mathcal{V}$
1: Brute-force search for  $w_{0}$  with the biggest partial entropy
$$
w_{0}\in \operatorname *{arg  max}_{w\in \mathbb{V}}\left\{-F_{w}\log F_{w}\right\}
$$
2: Assign  $\gamma'(w_0) \gets \gamma_1$
3: Set threshold  $t\gets 1 + \varepsilon /(C|\mathbb{V}|)^4$
4: until no update is possible do:
5: Try all possible updates and all pairs  $(w, \gamma_i)$ :
6: Update 1 Add  $w$  to  $\gamma_{i}$ , compute entropy of the update  $H_{1}$
7: Update 2 Remove  $w$  from  $\gamma_{i}$ , compute entropy of the update  $H_{2}$
8: Update 3 Remove arbitrary  $v$  from  $\gamma(v)$ , add  $w$  to  $\gamma_i$ , compute entropy of the update  $H_3$
9: if Update  $j$  can be used on  $(w, \gamma_i)$  then
10: if Updated entropy  $H_{j} > tH(\gamma)$  then
11: Perform update j
12: end if
13: end if
14: return  $\gamma$
a vocabulary  $\mathbb{V}$  and groups  $\nu$  we denote a set of all possible pairs  $(w,\gamma_{i})$  as  $\mathbb{V}\times \mathcal{V}$ . For instance, let  $\mathbb{V} = \{\text{"she", "tells", "me"}\}$  and  $\mathcal{V} = [\gamma_1,\gamma_2]$ . Then  $\mathbb{V}\times \mathcal{V} = \{\text{"she", 1},\text{"she", 2},\text{"tells", 1), ("tells", 2), ("me", 1), ("me", 2)}\}$ . Consider a grouping  $\gamma$  such that  $\gamma (\text{"she"}) = \gamma (\text{"tells"}) = 1$  and  $\gamma (\text{"me"}) = 2$ . Then  $\gamma$  can be described as a set is  $\{\text{"she", 1},\text{"tells", 1},\text{"me", 2)}\}$ . We say that such set defines a grouping and refers to a family of all such sets as grouping set family.
Note that an arbitrary set in  $\mathbb{V} \times \mathcal{V}$  may not define a grouping. For instance, the set  $\{(\text{"she", 1}), (\text{"tells", 1}), (\text{"tells", 2}), (\text{"me", 2})\}$  does not, as it maps "tells" to more than one group.
To apply the results of (Lee et al., 2009), we need to show that the grouping set family forms a matroid (Lee et al., 2009) on  $\mathbb{V}\times \mathcal{V}$ .
Lemma 1. The grouping set family defines a matroid on  $\mathbb{V} \times \mathcal{V}$ .
To satisfy the conditions of a matroid, we need
Algorithm 5 Unigram Entropy Diverse Grouping
Output: Grouping  $\gamma : \mathbb{V} \to \mathcal{V}$
1: Compute initial grouping  $\gamma^\prime$  using Algorithm (4)
2: if  $\gamma'(w)$  is undefined for some  $w$  then
3: Let  $\mathbb{W} \gets \{w \in \mathbb{V} : \gamma'(w)$  is undefined\}
4: Create a new grouping  $\gamma \gets \gamma'$
5: Find a group  $\gamma_{i_0}$  with the lowest partial unigram entropy:
$$
\gamma_{i_{0}}\in \operatorname *{arg  min}_{\gamma^{\prime}_{j}\in \mathcal{V}}\left\{G\left(c_{\gamma^{\prime}_{j}}\right)\right\}
$$
6: for all  $w\in \mathbb{W}$  do
7: Set  $\gamma (w)\gets \gamma_{i_0}$
8: end for
9: end if
10: return  $\gamma$
two properties. As an example<sup>1</sup>, consider  $\mathbb{V}$  and  $\mathcal{V}$  as in Figure (2). Firstly, let  $Q \subseteq \mathbb{V} \times \mathcal{V}$  be a set that defines a grouping of  $\mathbb{V}$ . For instance,  $Q = \{(\text{point}, 1), (\text{graph}, 1), (\text{noun}, 1), (\text{text}, 2), (\text{science}, 2)\}$ . Then every  $R \subset Q$  such as  $R = \{(\text{point}, 1), (\text{graph}, 1), (\text{text}, 2)\}$  must define a grouping as well. Secondly, take two sets  $S, T \subseteq \mathbb{V} \times \mathcal{V}$  that both define groupings. Then if  $|T| < |S|$ , we should always be able to find a pair  $(w, \gamma_i) \in S \setminus T$  such that adding  $(w, \gamma_i)$  to  $T$  results in a grouping. In Figure (2), this pair is (point, 1) in  $S$ ;  $T \cup (\text{point}, 1)$  does define a new groping.
Moreover, the algorithm from (Lee et al., 2009) requires an objective function to be submodular (Lee et al., 2009). Intuitively, submodularity means that a function value changes less for larger inputs.
Lemma 2. The function  $H:\mathbb{V}\times \mathcal{V}\to \mathbb{R}$  is nonnegative and submodular.
To see that  $H$  is submodular, consider  $\mathbb{V}$  and  $\nu$  as pictured in Figure (2). Consider groupings  $\gamma$  and  $\gamma^\prime$  induced by  $R$  and  $Q$  from Figure (2). Assume that we add (word, 1) to  $\gamma$  and  $\gamma^\prime$ . Relative frequencies of  $\gamma_{2}$  remains unchanged for  $\gamma, \gamma^\prime$ . Then the entropy gain for  $\gamma^\prime$  and  $\gamma$  depends only on the
Figure 2: Properties 2) and 3) of a matroid for a grouping with  $\mathbb{V} = \{\text{graph, point, noun, word, text, science}\}$  and  $\mathcal{V} = [1,2]$ . Blue color denotes group 1, red color denotes group 2.
partial entropies of a group indexed by 1:
$$
\begin{array}{l} - \left(c _ {\gamma_ {1} ^ {\prime}} + F _ {\text {w o r d}}\right) \log \left(c _ {\gamma_ {1} ^ {\prime}} + F _ {\text {w o r d}}\right) + c _ {\gamma_ {1} ^ {\prime}} \log \left(c _ {\gamma_ {1} ^ {\prime}}\right) \\ - \left(c _ {\gamma_ {1}} + F _ {\mathrm {w o r d}}\right) \log \left(c _ {\gamma_ {1}} + F _ {\mathrm {w o r d}}\right) + c _ {\gamma_ {1}} \log \left(c _ {\gamma_ {1}}\right) \\ \end{array}
$$