Zipfian Whitening
Authors: name: Sho Yokoi, name: Han Bao, name: Hiroto Kurita, name: Hidetoshi Shimodaira

The word embedding space in neural models is skewed, and correcting this can
improve task performance. We point out that most approaches for modeling,
correcting, and measuring the symmetry of an embedding space implicitly assume
that the word frequencies are uniform; in reality, word frequencies follow a
highly non-uniform distribution, known as Zipf's law. Surprisingly, simply
performing PCA whitening weighted by the empirical word frequency that follows
Zipf's law significantly improves task performance, surpassing established
baselines. From a theoretical perspective, both our approach and existing
methods can be clearly categorized: word representations are distributed
according to an exponential family with either uniform or Zipfian base
measures. By adopting the latter approach, we can naturally emphasize
informative low-frequency words in terms of their vector norm, which becomes
evident from the information-geometric perspective, and in terms of the loss
functions for imbalanced classification. Additionally, our theory corroborates
that popular natural language processing methods, such as skip-gram negative
sampling, WhiteningBERT, and headless language models, work well just because
their word embeddings encode the empirical word frequency into the underlying
probabilistic model.