SlowGuess/NAACL_2021_OCR huggingface dataset:
Every pair  $(w, \gamma_i)$  grouped by  $\gamma$  is also grouped by  $\gamma'$  thus  $c_{\gamma_1} < c_{\gamma_1'}$ . Because the function  $L(x) = -x \log (x + F_{\mathrm{well}}) + x \log x$  is monotone decreasing for all real non-negative values  $x$ , we have  $L(c_{\gamma_1}) > L(c_{\gamma_1'})$ . Hence, the larger grouping gains less in entropy than the smaller one.
Now we give a sketch of the proof of Theorem 1.
Proof. We claim that  $H(\gamma) \geq \frac{C - 1}{4C + 4\varepsilon C} H^*$  where  $H^*$  is the largest unigram entropy among all groupings. We should consider the case  $H(\gamma) < H(\gamma')$ . The groupings  $\gamma$  and  $\gamma'$  differ only in index  $i_0$ . Thus the difference  $H(\gamma) - H(\gamma')$  is equal to the difference in the partial entropies
$$
G \left(c _ {\gamma_ {i _ {0}}}\right) - G \left(c _ {\gamma_ {i _ {0}} ^ {\prime}}\right).
$$
We note that the group  $\gamma_{i_0}'$  with the smallest partial entropy contributes at most  $H(\gamma') / C$  to the total entropy of  $\gamma'$ . Moreover, partial entropy of  $\gamma_{i_0}$  is always non-negative. We obtain  $H(\gamma) - H(\gamma') \geq -H(\gamma') / C$ . Our bound follows by plugging in the estimation  $H(\gamma') \geq H^* / (4 + 4\varepsilon)$  which is the approximation guarantee for the Algorithm 4 from (Lee et al., 2009).
Combination Methods Below, we will discuss how to incorporate our new representation using any of our grouping methods in NLP tasks. Firstly, we group each word independently. Applying a grouping function  $\gamma (\cdot)$  in Section 3 on each word  $x_{1},x_{2},x_{3},\dots ,x_{i},\dots ,x_{I^{\prime}}$  in an input sentence
The full proof is provided in Appendix.
(b)
Figure 3: Combination methods for different NN architectures: (a) concatenation for ConvS2S and XLM; (b) linear interpolation on encoder outputs for Bi-LSTM with attention.  $\gamma (\cdot)$  is a word grouping function.
one by one generates a sequence of word groups  $\gamma(x_1), \gamma(x_2), \gamma(x_3), \dots, \gamma(x_i), \dots, \gamma(x_{I'})$  in the same length  $I'$ . Note that we use the term "word" loosely here; it can mean a word or a subword (of a BPE token), or even a character.
The first combination method is concatenation, see Figure 3a. We apply this method in NMT. First, we concatenate two input sources. Next, we apply the Byte-Pair-Encoding (Sennrich et al., 2015) (BPE) and word embeddings implemented by Rehurek and Sojka (2010) on each word  $\epsilon(x)$  and its codeword  $\epsilon_{\gamma}(\gamma(x))$ . We separately train word embedding on groups and on words. Thus,  $\epsilon_{\gamma}(\cdot)$  and  $\epsilon(\cdot)$  are different functions. As shown in Figure 3, the input to the NLP system is the embedded words of a sentence,  $\tilde{\epsilon}(x_1), \tilde{\epsilon}(x_2), \tilde{\epsilon}(x_3), \dots, \tilde{\epsilon}(x_i), \dots, \tilde{\epsilon}(x_I)$ , where  $\tilde{\epsilon}(x_i)$  is the concatenation of the embedded words  $\epsilon(x_i)$  and their groups  $\epsilon_{\gamma}(\gamma(x_i))$ :
$$
\tilde {\epsilon} \left(x _ {i}\right) = \left[ \epsilon \left(x _ {i}\right); \epsilon_ {\gamma} (\gamma \left(x _ {i}\right)) \right] \tag {2}
$$
The second method is linear combination on encoder outputs, see Figure 3b. We use this method in part-of-speech (POS) tagging. The input to the linear combiner is the grouped sentence, represented by a sequence of hidden states
$\tilde{h}^1 (\epsilon (x_I)),\dots ,\tilde{h}^j (\epsilon (x_I)),\dots ,\tilde{h}^J (\epsilon (x_I))$  of the last position  $I$  in each of the encoder layers  $j\in [1,2,\dots ,J]$  .  $J$  is the number of nodes at each decoder layer. Recall that each hidden state is a real vector  $\mathbb{R}^d$  , which is why we can use the vector space operations such as addition on it. For convenience, we denote the last hidden state of the  $j$  -th encoder layer, which we take as the input to the decoder,  $\tilde{h}^j (\epsilon (x_I))$  , by  $\tilde{h}_I^j$  , the last hidden state of the  $j$  -th encoder layer of the original textual sentence  $h^j (\epsilon (x_I))$  by  $h_I^j$  , and the last hidden state of the  $j$  -th encoder layer of the grouped sentence  $h^{j}(\epsilon_{\gamma}(\gamma (x_{I})))$  by  $h_{\gamma I}^{j}$  . The combined encoder hidden state  $\tilde{h}^j$  is a linear interpolation of the hidden states of the textural input and its group input:
$$
\tilde {h} ^ {j} = (1 - \alpha) h _ {I} ^ {j} + \alpha h _ {\gamma I} ^ {j} \tag {3}
$$
As shown in Figure (3b), each layer's combined last hidden state is fed into the baseline decoder with the operator of  $+$ .  $\alpha$  is the encoder weight of the grouped sentence, and here,  $\alpha = 0.5$ .
In the following context, we will show the our methods' evaluation results on three representative NLP tasks: (1) Machine translation as a recognition and generation problem; (2) Language modeling as a regression problem; and (3) POS tagging as a typical sequence labeling problem. We will show that our methods have the potential to improve any NLP application with textual inputs.
Dataset We empirically verify our method on the IWSLT'17 dataset containing 226 thousand sentences. Table 1 shows the vocabulary statistics before and after the pre-processing on the original and the concatenated data. We carry out experiments on the English-to-French (EN-FR) language direction.
We also carry out experiments on additional medium and small NMT tasks. For medium-sized tasks, we use the IWSLT'17 dataset with language directions including English to German (EN-DE), German to English (DE-EN), and English to Chinese (EN-ZH). We use the MTNT'18 dataset with language directions English to French (MTNT EN-FR) and French to English (MTNT FR-EN) for the small-sized task.
Baseline and Setup As a filter in pre-processing, every sentence is restricted to 250 characters and 1.5 length ratio between source and target sentences using Moses tokenizer (Koehn et al., 2007).
Running Words	Method	EN	FR
Before BPE	Baseline	4.8	5.1
+Entropy	9.5	5.1
After BPE	Baseline	5.2	5.6
+Entropy	10.2	5.8
Vocabulary	Method	EN	FR
Before BPE	Baseline	63	81
+Entropy	67	81
After BPE	Baseline	11	13
+Entropy	12	11
The Byte-pair encoding model (with  $16K$  BPE operations) is jointly trained on the source textual word inputs, cluster ID inputs, and target outputs. The baseline NMT model is the Convolutional Sequence to Sequence (Gehring et al., 2017) (ConvS2S), with the following parameter setting: the embedding dimension as 512, the learning rate as 0.25, the gradient clipping as 0.1, the dropout ratio as 0.2, and the optimizer as NAG. The training is terminated when the validation loss does not decrease for five consecutive epochs. For Chinese translations, we use the IWSLT post-processing script (IWSLT, 2021). Finally, the translation accuracy is measured with the BLEU score using SacreBLEU (Post, 2018).
Distance Measure To empirically compare across random and distance-based grouping algorithms, we measure the intra-group average distance of group pairs as follows: For each group in the source side vocabulary of the training and test set, compute the sum of the cosine distance  $1 - \frac{A \cdot B}{||A|| \cdot ||B||}$  of the embedding of each word pair, then divide it by the total number of word pairs in this group to get the group diversity. Then, average the distance of all groups in the vocabulary. Each algorithm generates the same number (63992) of word groups. The average distance of Poisson/Gaussian-based Random Grouping is 0.1286, and that of Rank-based Diverse Grouping is 0.1291. This finding is consistent with the translation BLEU score in Table 2. The greater the intra-group distance, the higher the accuracy. Maximum entropy approaches cannot be compared with this measure because it takes the entropy as an objective function. We have provided its provable guarantee in Section 4.
Table 1: IWSLT'17 English-French statistics of running words in millions  $(M)$  and vocabulary words in thousands  $(K)$ .
Method	Test
Baseline	17.6
Word Grouping
Poisson R.G. (Alg.2)	22.4
Gaussian R.G. (Alg.2)	23.4
R.G. (Alg.1)	23.3
Distance D.G. (Alg.3)	23.6
Entropy (Alg.4)	24.1 (+36.9%)
Table 2: Translation results in BLEU[%] on IWSLT'17. EN-FR. Baseline is (Gehring et al., 2017) on words. Dev: IWSLT test 2013, test 2014, test 2015; Test: IWSLT test 2017. Our methods include Random Grouping (R.G.), Poisson/Gaussian-based Random Grouping (Poisson/Gaussian R.G.), Distance-based Diverse Grouping (Distance D.G.), and Entropy-based Diverse Grouping (Entropy), respectively. BPE:  ${16k}$  .
Dataset	IWSLT	MTNT
Method	EN-DE	DE-EN	EN-ZH	EN-FR
Baseline	19.4	22.6	18.6	19.4
Entropy	21.0	24.0	19.2	23.9 (+18.8%)
Table 3: Left: Translation results in BLEU[%] on IWSLT'17 (EN-DE, DE-EN, EN-ZH; Dev: test 2013, test 2014, test 2015; Test: test 2017). Right: Translation results in BLEU[%] on MTNT'18 (EN-FR; Dev: MTNT dev; Test: MTNT test 2018) datasets. Baseline: (Gehring et al., 2017) on words. BPE operations:  ${16k}$  .
Figure 4: Histogram of Sentence-level BLEU Scores for baseline, Distance-based D.G. & Entropy.
Results For IWSLT'17 task, Table 2 shows the improvement when applying each of our methods on the ConvS2S baseline. All of our methods significantly enhance the accuracy of the NMT systems. Among them, the entropy-based diverse grouping achieves the greatest improvement, i.e., +6.5 BLEU points, which is +36.9% relative improvement.
Analysis Figure 4 compares entropy, distance-based D.G. and the baseline method with respect to the sentence-level BLEU score in a histogram (Neubig et al., 2019). The baseline method generates
Source	Qu'est-ce que cela signifie pour le procrastinateur?
Reference	Now, what does this mean for the procrastinator?
Baseline	Well, there's an issue of what it means for procrastinator?
Entropy	Now, what does this mean for procrastinator?
Source	Pendant queça pousse, ils font des changements.
Reference	While it's growing, they make changes.
Baseline	It's going to push it, they make change.
Entropy	So, i think it's growing, they make change.
Source	Mes parents ne se sont jamais plaints.
Reference	And you know, my parents never complained.
Baseline	It's never complained.
Entropy	So, my parents never complained.
almost double low-quality translations (347) compared to the distance D.G. and entropy methods (178 and 179), while the latter two methods generate many more high-quality translations with BLEU above  $20\%$ .
Table 4 shows the FR-EN baseline and entropy translation outputs, respectively. We observe that our entropy method is particularly better than the baseline when the baseline fails in: (1) performing a reasonable translation; (2) missing phrases; (3) mis-translating phrases.
5.2 POS Tagging
We evaluate our approach in POS Tagging on Brown Corpus (Francis and Kucera, 1979). Brown corpus is a well-known English dataset for POS and contains 57341 samples. We uniform randomly sample  $64\%$  data as the training set,  $16\%$  as the validation set, and  $20\%$  as the test set. Our baseline is a Keras (Chollet, 2015) implementation (Joshi, 2018) of Bi-LSTM POS Tagger (Wang et al., 2015). We train word embedding (Mikolov et al., 2013) implemented by Rehulek and Sojka (2010) with 100 dimensions. Each of the forward and the backward LSTM has 64 dimensions. We use a categorical cross-entropy loss and RMSProp optimizer. We also use early stopping based on validation loss.
Table 4: Comparison of translation outputs for baseline, Distance D.G. & Entropy.
Dev	Test
Loss	Accuracy	Error Rate
Baseline	5.16	5.39	98.61	1.39
R.G.	5.32	5.07	98.69	1.31
Poisson R.G.	5.56	5.27	98.62	1.38
Entropy	5.48	5.22	98.66	1.34 (-3.60%)
Table 5: POS accuracy and error rate in [%] on Brown corpus. Baseline is Bi-LSTM POS Tagger (Joshi, 2018) on words.
Test PPL
Baseline	22.85
Entropy	21.99 (-3.76%)
Table 6: LM PPL on the English part of IWSLT'17 ENFR. Baseline: XLM (Lample and Conneau, 2019).
5.3 Language Modeling (LM)
We train and evaluate the English part of EN-FR IWSLT'17 dataset. We use 256 embedding dimensions, six layers, and eight heads for efficiency. We set dropouts to 0.1, the learning rate to 0.0001, and BPE operations to  $32k$ . We used Adam optimizer with betas of 0.9. As shown in Table 6, Entropy-based diverse grouping reduces PPL of the baseline system, i.e.,  $3.76\%$  relatively.
We introduce a novel approach that generalizes Deep Learning models by grouping input words to maximize their semantic diversity. To this end, we design a family of algorithms based on random sampling, geometric distance, and entropy, and provide provable guarantees to the entropy-based diverse grouping. Our methods reduce the number of low-quality translation outputs ( $< 10\%$  in BLEU) to half and greatly increase the high-quality translation ( $>20\%$  in BLEU) ratio. Experiments show that our approach significant improves over state-of-the-art baselines in Neural Machine Translation (i.e., up to  $+6.5$  BLEU points) and achieves higher accuracy in POS Tagging and Language Modeling.
We appreciate the National Science Foundation (NSF) Award No. 1747728 and the National Science Foundation of China (NSFC) Award No. 61672524 to fund this research. We are also thankful for the support of the Google Cloud Research Program.
L Douglas Baker and Andrew Kachites McCallum. 1998. Distributional clustering of words for text classification. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval.
Dimitri P Bertsekas. 2014. Constrained optimization and Lagrange multiplier methods. Academic press.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics.
Rohan Chitnis and John DeNero. 2015. Variable-length word encodings for neural translation models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
François Chollet. 2015. Keras. https://github.com/keras-team/keras.git.
Jinhua Du and Andy Way. 2017. Pinyin as subword unit for Chinese-sourced neural machine translation. In Proceedings of Irish Conference on Artificial Intelligence and Cognitive Science.
Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, and Manabu Okumura. 2020. A simple and effective usage of word clusters for CBOW model. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing.
Edward W Forgy. 1965. Cluster analysis of multivariate data: efficiency versus interpretability of classifications. Biometrics, 21:768-769.
W. N. Francis and H. Kucera. 1979. Brown corpus manual. Technical report, Department of Linguistics, Brown University, Providence, Rhode Island, US.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of International Conference on Machine Learning.
Teofilo F Gonzalez. 1985. Clustering to minimize the maximum intercluster distance. Theoretical Computer Science, 38:293-306.
Hans van Halteren, Jakub Zavrel, and Walter Daelemans. 2001. Improving accuracy in word class tagging through the combination of machine learning systems. Computational Linguistics, 27(2):199-229.
IWSLT. 2021. IWSLT post-processing toolkit. In http://hltshare.fbk.eu/IWSLT2015/chineseText2Chars.pl.
Anil K Jain and Richard C Dubes. 1988. Algorithms for clustering data. Prentice-Hall, Inc.
Aneesh Joshi. 2018. LSTM POS tagger. https://github.com/aneesh-joshi/LSTM_POSTAGGER.git.
Abdul Rafae Khan, Jia Xu, and Weiwei Sun. 2020. Coding textual inputs boosts the accuracy of neural networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of Association for Computational Linguistics.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. Advances in Neural Information Processing Systems.
Jon Lee, Vahab S. Mirrokni, Viswanath Nagarajan, and Maxim Sviridenko. 2009. Non-monotonic submodular maximization under matroid and knapsack constraints. In Proceedings of the 41st Annual ACM Symposium on Theory of Computing, STOC 2009, Bethesda, MD, USA, May 31 - June 2, 2009, pages 323-332. ACM.
Hairong Liu, Mingbo Ma, Liang Huang, Hao Xiong, and Zhongjun He. 2018. Robust neural machine translation with joint textual and phonetic embedding. In Proceedings of Association for Computational Linguistics.
Christos Louizos, Max Welling, and Diederik P Kingma. 2018. Learning sparse neural networks through  $l_{-}0$  regularization. In International Conference on Learning Representations.
Sven Martin, Jörg Liermann, and Hermann Ney. 1998. Algorithms for bigram and trigram word clustering. Speech Communication, 24(1):19-37.
Avner May, Jian Zhang, Tri Dao, and Christopher Ré. 2019. On the downstream performance of compressed word embeddings. Advances in Neural Information Processing Systems, 32:11782-11793.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems.
Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang. 2019. compare-mt:
A tool for holistic comparison of language generation systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations).
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.
Lawrence Philips. 1990. Hanging on the metaphone. Computer Language.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2020. BPE-dropout: Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Radim Šehůrek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the Conference on Language Resources and Evaluation 2010 Workshop on New Challenges for NLP Frameworks.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
Peilu Wang, Yao Qian, Frank K. Soong, Lei He, and Hai Zhao. 2015. Part-of-speech tagging with bidirectional long short-term memory recurrent neural network. arXiv preprint arXiv:1510.06168.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. SwitchOut: an efficient data augmentation algorithm for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
John Wright, Allen Y Yang, Arvind Ganesh, S Shankar Sastry, and Yi Ma. 2008. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2):210-227.
Appendix A Used notation
We list the notation used throughout the paper
$\mathbb{V}$  : vocabulary of words
$\nu$  : vocabulary of groups
$w, v$ : a word
$F_{w}$ : relative frequency of a word  $w$
$\gamma_{i}, \gamma_{j}$ : a group
$\mathbb{V}\times \Gamma$  : set of all possible pairs  $(w,\gamma_{i})$
$c_{\gamma_i}$ : relative frequency of a group  $\gamma_i$
$\gamma$  : an assignment (grouping)
$H(\gamma)$  : unigram entropy of a grouping  $\gamma$
$G\left(c_{\gamma_j'}\right)$ : partial entropy of a group  $\gamma_i$
$C$  : number of groups
$[1, \ldots, C]$  - natural numbers from 1 to  $C$
N - natural numbers
Appendix B Omitted proofs
Definition 1 (Matroid). Let  $\Omega$  be a finite set (universe) and  $\mathcal{I} \subseteq 2^{\Omega}$  be a set family (independent sets). A pair  $\mathcal{M} = (\Omega, \mathcal{I})$  is called a matroid if
1.  $\emptyset \in \mathcal{I}$
2. If  $Q \in \mathcal{I}$  and  $R \subseteq Q$  then  $R \in \mathcal{I}$
3. For any  $Q,R\in I$  with  $|R| < |Q|$  there exists  $\{x\} \in Q\setminus R$  such that  $R\cup \{x\} \in \mathcal{I}$
Let us denote a family of all grouping sets of  $\mathbb{V}\times \mathcal{V}$  as  $\mathcal{G}$
Proof of Lemma 1. We have to show that  $(\mathbb{V} \times \mathcal{V}, \mathcal{G})$  satisfies three conditions from the Definition 1.
1. An empty grouping is a grouping.
2. Consider an arbitrary  $Q \in \mathcal{G}$  and  $R \subset Q$ . Since  $Q$  defines a grouping, for any  $(w, \gamma_i) \in Q$  we have  $(w\gamma_j) \notin Q$  for all  $\gamma_j \neq \gamma_i$ . Therefore, for all  $(w, \gamma_i) \in R$  we have  $(w\gamma_j) \notin R$  given  $\gamma_j \neq \gamma_i$  and thus  $R$  defines a grouping as well.
3. Consider two arbitrary  $R, Q \in \mathcal{G}$  with  $|R| < |Q|$ . Let us denote  $\{w \in \mathbb{V} : (w, \gamma_i) \in Q$  for some  $\gamma_i\}$  as  $\pi(Q)$ . We claim that  $|Q| = |\pi(Q)|$ . Otherwise, there must exist  $w$  such that  $(w, \gamma_i), (w, \gamma_j) \in Q$  and  $\gamma_i \neq \gamma_j$ . However, this is forbidden for a set which defines a grouping. Analogously,  $|R| = |\pi(R)|$ . Since both  $R, Q$  are finite, we have  $0 < |Q \setminus R| = |\pi(Q)| - |\pi(R)| = |\pi(Q) \setminus \pi(R)|$ . Consider
an arbitrary  $w^{\prime}\in \pi (Q)\setminus \pi (R)$  and its group  $\gamma_{i'}$  in  $Q$ ; we have  $(w^{\prime},\gamma_{i^{\prime}})\in Q\backslash R$ . Moreover, since  $w^{\prime}$  is ungrouped by  $R$ , we conclude that  $R\cup \{(w^{\prime},\gamma_{i^{\prime}})\} \in \mathcal{G}$  and finish the proof.
Definition 2 (Submodular function). A function  $f: 2^{\Omega} \to \mathbb{R}$ , where  $\Omega$  is finite, is submodular if for any  $X \subseteq Y \subseteq \Omega$  and any  $x \in \Omega \setminus Y$  we have
$$
f (X \cup \{x \}) - f (X) \geq f (Y \cup \{x \}) - f (Y).
$$
For any non-negative real  $x$  and fixed  $a > 0$ , we denote  $-(x + a)\log_2(x + a) + x\log x$  as  $L_{a}(x)$ .
Proof of Lemma 2. First, we show that  $H(Q) \geq 0$  for all  $Q \subseteq \mathbb{V} \times \mathcal{V}$ . By definition, we have  $H(\emptyset) = 0$ . Consider an arbitrary non-empty  $Q \subseteq \mathbb{V} \times \mathcal{V}$ . For any  $\gamma_i \in \mathcal{V}$  we have
$$
0\leq c_{\gamma_{i}} = \sum_{\substack{w\in \mathbb{V}:\\ (w,\gamma_{i})\in Q}}F_{w}\leq \sum_{w\in \mathbb{V}}F_{w} = 1.
$$
Therefore,  $-c_{\gamma_i}\log c_{\gamma_i}\geq 0$  and
$$
\sum_ {i = 1} ^ {C} L \left(c _ {\gamma_ {i}}\right) \geq 0.
$$
Now we establish submodularity. Consider an arbitrary  $Q \subseteq \mathbb{V} \times \mathcal{V}$ ,  $R \subset Q$  and any  $(w', \gamma_{i'}) \notin Q$ . Let  $Q' := Q \cup \{(w', \gamma_{i'})\}$ ,  $R' := R \cup \{(w', \gamma_{i'})\}$ . We need to show that
$$
H \left(R ^ {\prime}\right) - H (R) \geq H \left(Q ^ {\prime}\right) - H (Q). \tag {4}
$$
Let us denote the frequency of the unigram  $\gamma_{j}$  in  $Q$ ,  $Q^{\prime}$  as  $c_{\gamma_j}(Q)$ ,  $c_{\gamma_j}(Q')$ . Since  $Q$  and  $Q^{\prime}$  differ only in the group  $\gamma_{i'}$  we have
$$
\begin{array}{l} H (Q ^ {\prime}) - H (Q) = \\ - c _ {\gamma_ {i ^ {\prime}}} \left(Q ^ {\prime}\right) \log c _ {\gamma_ {i ^ {\prime}}} (Q) + c _ {\gamma_ {i ^ {\prime}}} (Q) \log c _ {\gamma_ {i ^ {\prime}}} (Q) \tag {5} \\ \end{array}
$$
Similarly, (5) holds for  $H(R') - H(R)$ . Thus, to proof (4) it is enough to show