SlowGuess/NAACL_2021_OCR huggingface dataset:
Ioannis Tsochantaridis, T. Joachims, Thomas Hofmann, and Y. Altun. 2005. Large margin methods for structured and interdependent output variables. J. Mach. Learn. Res., 6:1453-1484.
Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 384-394, Uppsala, Sweden. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
Dilin Wang, Chengyue Gong, and Qiang Liu. 2019c. Improving neural language modeling via adversarial training. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6555-6565, Long Beach, California, USA. PMLR.
Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. 2020. Improving neural language generation with spectrum control. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
J. Weston and C. Watkins. 1999. Support vector machines for multi-class pattern recognition. In ESANN.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's
neural machine translation system: Bridging the gap between human and machine translation. In arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. In International Conference on Learning Representations.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Canlin Zhang, Xiuwen Liu, and Daniel Bis. 2019. An analysis on the learning rules of the skip-gram model. 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8.
George KingsleyZipf.1949.Human behaviour and the principle of least effort:an introduction to human ecology.
A Cosine Regularization as Mean Direction Minimization
In this section we show the equivalence of Cosine Regularization and mean direction minimization.
$$
\begin{array}{l} \operatorname {C o s R e g} = \frac {1}{N ^ {2}} \sum_ {i} ^ {N} \sum_ {j \neq i} ^ {N} \hat {\boldsymbol {w}} _ {i} ^ {\top} \hat {\boldsymbol {w}} _ {j} (12) \\ = \frac {1}{N ^ {2}} \left(\sum_ {i} ^ {N} \hat {\boldsymbol {w}} _ {i}\right) ^ {\top} \left(\sum_ {i} ^ {N} \hat {\boldsymbol {w}} _ {i}\right) - N (13) \\ = \frac {1}{N ^ {2}} \left| \left| \sum_ {i} ^ {N} \hat {\boldsymbol {w}} _ {i} \right| \right| _ {2} ^ {2} - N, (14) \\ \end{array}
$$
as  $\mathrm{N}$  is a constant, minimizing  $\frac{1}{N}\left\| \sum_{i}^{N}\hat{\boldsymbol{w}}_i\right\| _2$  is equivalent to minimizing the CosReg term.
B Token Frequency Validation
Figure 4 validates that the word frequency imbalance is preserved in a corpus tokenized with Word-Piece (Schuster and Nakajima, 2012; Wu et al., 2016).
In Table 3, we provide expanded results on the embedding benchmarks (see Section 5.2 for details).
Our experiments reveal a negative 0.61 correlation between the average norm of the embedding vectors and their isotropy. Additionally, the ratio of the  $L_{2}$  norm of the mean vector to the average of the  $L_{2}$  norms of embeddings tends to be larger for less isotropic embeddings. Figure 5 shows distributions of  $L_{2}$  norms of embeddings from the models studied in this paper. Moreover, Figure 6 compares the effect of centering on the  $L_{2}$  norms of embeddings from the BERT-large-cased and RoBERTa-large. We leave the relationship between the norms of the vectors, their isotropy, and the pretraining details (e.g., corpus size, number of training steps, weight decay) for future studies.
(a) Words
(b) Tokens
Figure 4: Comparison of word and token frequencies on CNN-DailyMail corpus.
Model	MEN
{·,·}/cos	RW
{·,·}/cos	Simlex-999
{·,·}/cos	WordSim-Sim
{·,·}/cos	WordSim-Rel
{·,·}/cos
Bert-base-cased	65.53 / 67.07	61.22 / 63.72	52.52 / 52.37	75.03 / 75.57	50.27 / 52.70
+ Centered-Scaled	69.32 / 68.36	63.57 / 63.94	47.34 / 47.96	78.58 / 77.36	53.23 / 53.96
+ Post-Process	72.20 / 70.91	67.71 / 68.20	53.60 / 52.53	79.38 / 78.15	57.63 / 58.04
Bert-large-cased	64.64 / 67.20	58.94 / 61.89	52.68 / 52.91	73.44 / 76.32	47.40 / 51.16
+ Centered-Scaled	68.63 / 67.69	61.67 / 62.19	48.66 / 48.53	77.47 / 77.45	48.81 / 50.02
+ Post-Process	72.12 / 71.07	67.22 / 68.07	55.41 / 53.77	78.80 / 78.88	55.90 / 56.79
GPT-2-small	65.55 / 71.16	58.08 / 63.99	50.43 / 51.45	72.98 / 78.32	50.19 / 58.63
+ Centered-Scaled	75.55 / 74.10	64.05 / 64.94	51.60 / 50.36	81.41 / 80.61	63.98 / 62.86
+ Post-Process	76.00 / 75.34	65.30 / 66.56	53.74 / 53.15	80.70 / 80.79	62.62 / 63.40
GPT-2-medium	65.01 / 71.55	58.55 / 65.22	51.13 / 52.51	72.67 / 78.48	48.65 / 59.90
+ Centered-Scaled	76.10 / 75.02	64.14 / 65.34	54.22 / 53.30	80.85 / 80.56	65.79 / 65.51
+ Post-Process	75.92 / 75.25	64.80 / 66.11	54.65 / 54.19	79.99 / 80.25	63.71 / 64.43
RoBERTa-base	72.83 / 72.70	64.72 / 66.61	55.17 / 54.84	78.00 / 78.62	59.31 / 61.27
+ Centered-Scaled	74.18 / 73.33	65.38 / 66.35	54.16 / 53.77	79.35 / 79.10	61.66 / 62.53
+ Post-Process	74.22 / 73.13	65.45 / 66.11	53.87 / 53.35	79.64 / 79.08	62.58 / 62.70
RoBERTa-large	42.92 / 63.44	49.72 / 64.31	45.75 / 54.54	58.09 / 73.46	27.47 / 50.69
+ Centered-Scaled	70.55 / 70.95	63.09 / 65.63	55.24 / 54.18	75.72 / 77.34	56.19 / 59.35
+ Post-Process	72.01 / 70.55	63.11 / 63.64	52.48 / 50.93	77.87 / 77.09	60.36 / 59.71
Table 3: Performance (Pearson's  $r \times  {100}$  ) of the models on the non-contextual benchmarks. Centered-Scaled stands for embedding matrix with mean direction, scaled by the norm of each word embedding, subtracted; Post-Process corresponds to the method defined in eq. 11. Results from different models are not directly comparable due to different tokenization.
Figure 5: Kernel density estimation of the  $L_{2}$  norms of embeddings from different models. Models trained on larger corpora and with an increased number of pretraining steps exhibit larger embedding norms.
(a) BERT-large-cased
(b) RoBERTa-large
Figure 6: The effect of mean subtraction on the distribution of  $L_{2}$  norms of embeddings in BERT-large-cased and RoBERTa-large.