SlowGuess/NAACL_2022_OCR huggingface dataset:
Natural Language Processing: Tutorial Abstracts, pages 15-20, Online. Association for Computational Linguistics.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. 2019. Fair resource allocation in federated learning. In International Conference on Learning Representations.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. 2020. Three approaches for personalization with applications to federated learning.
Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2775-2779, Brussels, Belgium. Association for Computational Linguistics.
Fatemehsadat Mireshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. 2020. Privacy in deep learning: A survey. In ArXiv, volume abs/2004.12254.
Ali Rahimi, Benjamin Recht, et al. 2007. Random features for large-scale kernel machines. In NIPS, volume 3, page 5. Citeseer.
Johannes Schneider and M. Vlachos. 2019. Mass personalization of deep learning. *ArXiv*, abs/1909.02803.
Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1422-1432, Lisbon, Portugal. Association for Computational Linguistics.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. 2021. A field guide to federated optimization. arXiv preprint arXiv:2107.06917.
Weichao Wang, Shi Feng, Wei Gao, Daling Wang, and Yifei Zhang. 2018. Personalized microblog sentiment classification via adversarial cross-lingual multi-task learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics.
Yi Yang and Jacob Eisenstein. 2017. Overcoming language variation in sentiment analysis with social attention. Transactions of the Association for Computational Linguistics, 5:295-307.
A.1 Federated Learning as an Application
Federated learning is a form of distributed learning where data never leaves each user's device (Wang et al., 2021; Konečný et al., 2018; Mireshghallah et al., 2020). Instead, the user trains a model on their device locally and then shares the gradients (model updates) with a centralized server, which aggregates the gradients from different users and sends the updated model back to all of them, for further training. We target this setup since it is a good candidate for personalization, given how a conventionally trained global model often fails to accommodate all users (Kulkarni et al., 2020; Mansour et al., 2020). Table 5 shows the performance gain of applying UserIdentifier, in a federated setup. UserIdentifier can be readily applied in federated learning, by assigning identifiers to each user and then asking them to append it to all their samples. We have used the Rand. All type of user identifier for this experiment, since we observed in previous sections that it was the most effective. In general, the baseline performance and the performance gain in the federated setup is slightly lower than in centralized learning, which is due to the distributed nature of FL, and the fact that only the average of multiple gradient updates are shared with the server for aggregation.
Table 5: Performance of UserIdentifier for sentiment classification in a federated learning setup.
Dataset	Conventional	User Identifier
BERT RoBERTa	IMDB	44.30	47.23
Yelp	68.40	70.60
Sent140	84.40	86.30
Sent140 (Skewed)	86.50	90.00
A.2 Further User-level Accuracy Studies
Figure 3 shows the change in user accuracy, when we use UserIdentifier for training, instead of conventional training for each user. In other words, the horizontal axis shows conventional  $UID_{acc}$  for each user, and the vertical axis shows the count of users.
As the plots show, on average across the two datasets,  $32.1\%$  of the users see improvements in accuracy, whereas  $54.2\%$  don't see any change.
A.3 Maximally Distant User Identifiers
To better understand the effect of edit distance between user identifiers, We also experimented with maximally distanced identifiers (for the Rand. All setup), where the maximum distance would be
(b) Yelp
Figure 3: Distribution of test accuracy change across users.
the length of the identifier here since each token in the identifier can take a substantially large number of values. For this experiment, we used rejection sampling for user ids, as in if a new random sample had any token overlaps with existing user ids, we would reject it and sample a new one. We observed results very similar to the ones with the random identifiers, which we hypothesize is because the random identifiers are already highly distanced and rarely overlap (less than  $10\%$  of the users have non-maximal distance).