Before revision:  However, due to expensive trial costs, most studies only conduct simple trials with dominant approaches (e.g BPE) and commonly used vocabulary sizes. In this paper, we find an exciting relation between an information-theoretic feature and BLEU scores. With thisobservation , we formulate the quest of vocabularization -- finding the best token dictionary with a proper size -- as an optimal transport  problem.
After revision:  This paper aims to figure out what is a good vocabulary and whether one can find the optimal vocabulary without trial training. To answer these questions, we first provide an alternative understanding of the role of vocabulary from the perspective of information theory. Motivated by this , we formulate the quest of vocabularization -- finding the best token dictionary with a proper size -- as an optimal transport problem.