SlowGuess/ACL_2020_OCR huggingface dataset:
Amirreza Shirani†, Franck Dernoncourt‡, Jose Echevarria‡, Paul Asenté‡, Nedim Lipka‡ and Thamar Solorio†
†University of Houston ‡Adobe Research
$\dagger$  {ashirani, tsolorio}@uh.edu
$\ddagger$  {franck.dernoncourt,echevarr,asente,lipka}@adobe.com
In this paper, we aim to learn associations between visual attributes of fonts and the verbal context of the texts they are typically applied to. Compared to related work leveraging the surrounding visual context, we choose to focus only on the input text as this can enable new applications for which the text is the only visual element in the document. We introduce a new dataset, containing examples of different topics in social media posts and ads, labeled through crowd-sourcing. Due to the subjective nature of the task, multiple fonts might be perceived as acceptable for an input text, which makes this problem challenging. To this end, we investigate different end-to-end models to learn label distributions on crowd-sourced data and capture inter-subjectivity across all annotations.
In visual designs, textual information requires the use of fonts with different properties. Whether it is books, magazines, flyers, ads or social media posts, different typefaces are commonly used to express non-verbal information and add more dimensions to the text. An appropriate font usually embodies information about character, context and usage of the design (Doyle and Bottomley, 2006). This motivates us to explore font associations with regular users in a crowd-sourced setting. In other words, we investigate how users relate fonts to different characteristics of the input text.
Current font selection interfaces such as O'Donovan et al. (2014) and commercial online services (e.g., MyFonts<sup>1</sup> and Typekit<sup>2</sup>) assist users in selecting fonts by taking into account font similarity. However, they do not consider the verbal
context of the input text. Having a better understanding of the input text, users can benefit from a font recommendation system during authoring, saving time and avoiding tedious exploration of long lists of fonts.
Most graphic designers agree that there is no strict or universally-accepted rule for choosing fonts. Different social and personal factors can be involved in typeface selection, which makes this process subjective. However, there seems to be enough agreement among human opinions to build reasonably effective models of font properties (O'Donovan et al., 2014; Shinahara et al., 2019). Several empirical studies have directly explored the relationship between fonts and texts (Shinahara et al., 2019; Henderson et al., 2004; Mackiewicz, 2007). For example, Brumberger (2003a) indicates that readers have strong opinions about the appropriateness of particular typefaces for particular text passages, and they can differentiate typeface/text mismatches.
In this study, we aim to model for the first time the associations between visual font attributes and textual context, with the final goal of better font recommendation during text composition. Our main contributions are: 1) We propose and formulate a new task: "font recommendation from written text." 2) We introduce a new dataset, Short Text Font Dataset, containing a variety of text examples annotated with ten different representative fonts. 3) We compare different end-to-end models that exploit contextual and emotional representations of the input text to recommend fonts. These models are able to capture inter-subjectivity among all annotations by learning label distributions during the training phase. We show that emotional representations can be successfully used to capture the underlying characteristics of sentences to suggest proper fonts.
1www.myfontsc.com
$^{2}$ https://fonts.adobe.com/
8607
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8607-8613 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Font-related studies have been extensively explored in graphic design literature. Shinahara et al. (2019) performed an empirical study on collections of book titles and online ads, showcasing trends relating typographic design and genre. Several previous studies have attempted to associate personality traits and fonts (O'Donovan et al., 2014; Brumberger, 2003b; Juni and Gross, 2008; Mackiewicz and Moeller, 2005; Amare and Manning, 2012). They support the idea of typefaces consistently perceived to have particular personas, emotions, or tones. More recently, FontLex (Kulahcioglu and De Melo, 2018) was the first to find the association between fonts and words by utilizing font-emotion and word-emotion relationships. Instead of focusing on independent words, our proposed model suggests fonts by considering the broader context of the whole text.
Task Subjectivity In some tasks, aggregated annotations always correspond to the correct answer (Brew et al., 2010). Therefore, to fully utilize the crowd's knowledge, different approaches have been proposed to aggregate labels, from simply applying majority voting to more sophisticated strategies to assess annotators' reliability (Yang et al., 2018; Srinivasan and Chander, 2019; Rodrigues et al., 2014). All of these methods rely on the assumption that only one answer is correct and should be considered as ground truth (Nguyen et al., 2016). Whereas in tasks like ours, sentiment analysis (Brew et al., 2010) or facial expression (Barsoum et al., 2016), the answer is likely to be more subjective due to its non-deterministic nature (Urkullu et al., 2019). We follow previous studies that successfully employed label distribution learning to handle ambiguity in the annotations (Geng et al., 2013; Shirani et al., 2019; Yang et al., 2015).
3 Font Dataset
The proposed dataset includes 1,309 short text instances from Adobe Spark<sup>3</sup>. The dataset is a collection of publicly available sample texts created by different designers. It covers a variety of topics found in posters, flyers, motivational quotes and advertisements.<sup>4</sup>
Choice of Fonts A vast number of fonts and typefaces are used in contemporary printed literature. To narrow down the task, we had a font expert select a set of 10 display fonts that cover a wide range of trending styles. These fonts display enough differentiation in visual attributes and typical use cases to cover the topics in our text samples. Figure 1 shows several examples from the dataset, each rendered with the most congruent font (font with the highest agreement).
Keep Calm and 25% OFF EVERYTHING ONLINE
Visit London AND IN-STORE
Donut love Ejoy the Last Bit of Suawer!
Money The Ultimate Carrot
Management Cake
EMPIRE STATE OF MIND
Figure 1: Examples from our collected dataset visualized through fonts with the highest annotation agreements.
Annotation Process In an MTurk experiment, we asked nine annotators to label each sample text by selecting their top three fonts (Figure 2). Workers were asked to choose suitable fonts after reading the sentence. We included carefully-designed quality questions in 10 percent of the hits to monitor the quality of our labeling. We also needed to ensure workers selected fonts based on the comprehension of the text rather than just personal preference. Therefore, we removed the annotations of workers who selected the same font more than 90 percent of the time, resulting in six to eight annotations per instance (we removed instances with fewer than six annotations).
As we mentioned earlier, we asked annotators to rank their top three font choices for each text in our dataset. We decided to treat the first, second, and third choices differently as they represent the workers' priorities. Therefore, we give the highest weight to the first choices (1.0) and lower weights (0.6) and (0.3) to the second and third choices, respectively. Figure 3 shows three examples with label distributions over 10 fonts. By comparing the label distributions of these examples, we can observe that 'formal' fonts like F0, F2, and F5 are often selected in business contexts (left). 'modern/display' fonts like F1, F3, and F8 are favored in more casual settings (center), and 'script' fonts like
<sup>3</sup>https://spark.adobe.com.
4The dataset along with the annotations can be found online: https://github.com/RiTUAL-UH/Font-prediction-dataset
8608
F0: Let's ring in the new year together!
F1: LET'S RING IN THE NEW YEAR TOGETHER!
F2: Let's ring in the new year together!
F3: Let's ring in the new year together!
F4: Let's ring in the new year together!
F5: Let's ring in the new year together!
F6: LET'S RING IN THE NEW YEAR TOGETHER!
F7: Let's ring in the new year together!
F8: Let's ring in the new year together!
F9: Let's ring in the new year together!
Figure 2: A text sample from the dataset rendered using the available 10 fonts for labelling. F0) Source Sans Pro, F1) Blakely, F2) FF Ernestine Pro, F3) FF Market Web, F4) Bickham Script Pro 3, F5) Burbank Big, F6) Fresno, F7) Sneakers Script Narrow, F8) Felt Tip Roman, F9) Pauline
Figure 3: Label distributions for three examples
Figure 4: Average label distribution of the entire corpus
F4, F8, and F9 are preferred for more emotional contexts (right). We observe that some fonts are more popular than others. Figure 4 shows the average label distribution over all instances. F3, F2, and F1 are the most popular, while F4, F8, and F9 are the least popular among all 10 fonts.
Statistics The dataset contains 8,211 tokens. The mean and standard deviation number of tokens per instance is 6.27 and 4.65, ranging from 1 to 27 tokens. We obtained a Fleiss kappa agreement (Fleiss, 1971) of 0.348 by taking into account all three choices. This value is reasonable for a task such as this since previous subjective tasks have also reported low inter-rater agreement scores (Salminen et al., 2018; Alonso et al., 2014). We split up the data randomly into training  $(70\%)$ , development
(10%) and test (20%) sets for further experimentation and evaluation.
Task Definition Given a piece of text  $X$ , we want to determine which font(s)  $y = \{y_0, \ldots, y_9\}$  are more appropriate or congruent with the properties of the input text. We formulate this problem as a ranking problem where the model assigns each font a real value  $d_y^x$ , representing the degree to which  $y$  describes  $X$ . In other words,  $d_y^x$  represents the degree of congruency of font  $y$  with input  $X$ . The values for all the labels are summed up to 1 to fully describe the instance (Geng, 2016).
We explore transfer learning from pre-trained models to improve the performance of our task. We investigate four different deep learning-based architectures to learn font distributions of examples in our dataset. Inspired by previous works, which supported the relationship between font and emotion (Section 2), we compare the effectiveness of emotional embeddings in our models to contextual embeddings like BERT.
GloVe-BiLSTM Model In this model, we use GloVe embeddings (Pennington et al., 2014) as input and a BiLSTM layer to encode word sequence information in forward and backward directions. Subsequently, we pass the encoded-words to two dense layers for prediction.
NRC Model Similar to the GloVe-BiLSTM Model, this model is LSTM-based. The difference is that instead of GloVe embeddings, we use the emotional representations of words from NRC
5The implementation is available online: https://github.com/RiTUAL-UH/Font_LDL_2020
8609
Figure 5: Font-Emoji Pearson Correlation Coefficient Heatmap
Emotion (Mohammad and Turney, 2013), Intensity (Mohammad, 2018b) and Valence, Arousal, and Dominance (VAD) (Mohammad, 2018a) lexicons as input to the model. To efficiently look up the emotion value of words, we search for the stemmed and synonym versions of out-of-vocabulary words.
BERT Model We use pre-trained BERT sequence classification model (Devlin et al., 2018) to obtain contextual embeddings as features. Then the output is fed to two dense layers yielding the class predictions. We implement our model based on the Hugging Face's BERT implementation (Wolf et al., 2019).
**Emoji Model** In this model, we use the Deep-Moji pre-trained model (Felbo et al., 2017) to generate emoji vectors by encoding the text into 2304-dimensional feature vectors. We treat these features as embedding and pass them to the model with two dense layers. Deepmoji is a sentence-level model containing rich representations of emotional content which is trained on a 1,246 million tweet corpus in the emoji prediction task.
5 Experimental Settings and Results
The Kullback-Leibler Divergence (KL-DIV) (Kullback and Leibler, 1951) is used as the loss function to train the models. KL-DIV measures how the predicted probability distribution is different from the ground truth probability distribution. To train all the models, we use Adam optimizer (Kingma and Ba, 2014) to optimize the model parameters. We run all models over four runs with different random seeds and report the averaged score to ensure stability. The reported test results correspond to models with the best accuracy on the validation set.
5.2 Evaluation Settings
We evaluate the performance by using two different evaluation metrics for this new task.
Font Recall (FR) Less popular fonts could be underrepresented by the models. Therefore we need an evaluation metric that measures the performance of models in learning individual labels. Since we are dealing with an unbalanced dataset, motivated by evaluation methodology used in previous recommendation systems like Kar et al. (2018); Carneiro et al. (2007), we compute Font Recall, i.e. the average recall per font, to measure the performance of the models in learning individual labels.
$$
\operatorname {F R} := \frac {\sum_ {i = 1} ^ {| F |} | R _ {i} |}{| F |}
$$
Where  $|F|$  represents the number of labels and  $R_{i}$  is the recall for the  $i^{th}$  font.
F-score For each instance  $X$  from the test set, we select the top  $k = \{1,3\text{and} 5\}$  fonts with the highest probabilities from both ground truth and prediction distributions. Then we compute weighted averaged F1-score for each  $k$ .
Note that there are many cases where two or more fonts have the exact same probability. In this case, if the model predicts either one of the labels, we consider it as a correct answer in both metrics.
Model/Evals	FR Top3	FR Top5	F-Top1	F-Top3	F-Top5
Majority Baseline	30.00	50.00	12.44	43.72	62.24
NRC Model	30.78	51.60	23.10	47.27	66.16
GloVe Model	32.71	53.74	25.95	51.29	68.29
Emoji Model	33.17	54.06	26.00	51.43	68.53
BERT Model	33.54	56.00	26.97	51.91	69.38
Table 1: Experimental results for all five models. FR represents Font Recall and F represents F-1 score. The results in bold are statistically significant compared to the Majority Baseline.
Table 1 compares different models in terms of five evaluation settings. The first two columns of the results show FR for the top 3 and 5 fonts. The other three columns show F-score for the top 1, 3
$^{6}$ Our implementation is based on the Hugging Face Torch-moji implementation, https://github.com/huggingface/torchMoji
and 5 fonts. Comparing to the Majority Baseline, the results from the Emoji and BERT models are statistically significant under paired t-test with  $95\%$  confidence interval. Although the BERT model performs slightly better than the rest, the Emoji model performs just as well, which suggests two things: (1) the font recommendation task is highly related to what emojis represent and 2) a simpler model like Emoji model can perform similarly to a complex solution like BERT.
We analyze the reason behind the effectiveness of the Emoji model by visualizing the Font-Emoji Pearson Correlation Coefficient Heatmap (Figure 5) in the training set. Interestingly, fonts F4 and F9 with a 'Script' style are highly correlated by 'Heart' and 'Love' emojis. Also, F3 with a 'Playful' style is negatively correlated with emojis with discomfort and mild irritation expressions.
Data Augmentation A well-established technique for automatic data augmentation is leveraging machine translation to find meaning-equivalent phrases in a single language (Mallinson et al., 2017; Coulombe, 2018). To mitigate the highly imbalanced class distribution in our data set, we tried over- and under-sampling techniques. We selected examples with high values in underrepresented classes and translated them to four non-English languages using Google Translate<sup>7</sup>. We then translated these examples back to English, resulting in 170 more examples. We also removed 50 instances with high values in the popular classes. We observed that the data augmentation process has marginal improvements (up to  $1\%$ ) in some models. We leave the exploration of more sophisticated data augmentation approaches for future work.
In this paper, we associated font with written text and tackle the problem of font recommendation from the input text. We collected more than 1,300 short written texts and annotated them with ten fonts. We formulated this task as a ranking problem and compared different models based on emotional and contextual representations that exploit label distribution learning to predict fonts.
The current approach covers a fixed number of fonts, but it can be extended to support a larger set of fonts. For example, we can use font similarity techniques and enable users to pick a group of
fonts, or to provide increased flexibility for the fonts available to users.
This research began during an internship at Adobe Research, and was sponsored in part by Adobe Research. We thank the reviewers for their thoughtful comments and efforts towards improving our work. We also thank Tim Brown for his help with font set selection.
Omar Alonso, Catherine Marshall, and Marc Najork. 2014. Crowdsourcing a subjective labeling task: a human-centered framework to ensure reliable results. *Microsoft Res.*, Redmond, WA, USA, Tech. Rep. MSR-TR-2014-91.
Nicole Amare and Alan Manning. 2012. Seeing typeface personality: Emotional responses to form as tone. In 2012 IEEE International Professional Communication Conference, pages 1-9. IEEE.
Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang. 2016. Training deep networks for facial expression recognition with crowdsourced label distribution. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 279-283. ACM.
Anthony Brew, Derek Greene, and Pádraig Cunningham. 2010. Using crowdsourcing and active learning to track sentiment in online media. In *ECAI*, pages 145-150.
Eva R Brumberger. 2003a. The rhetoric of typography: The awareness and impact of typeface appropriateness. Technical communication, 50(2):224-231.
Eva R Brumberger. 2003b. The rhetoric of typography: The persona of typeface and text. Technical communication, 50(2):206-223.
Gustavo Carneiro, Antoni B Chan, Pedro J Moreno, and Nuno Vasconcelos. 2007. Supervised learning of semantic classes for image annotation and retrieval. IEEE transactions on pattern analysis and machine intelligence, 29(3):394-410.
Claude Coulombe. 2018. Text data augmentation made simple by leveraging nlp cloud apis. arXiv preprint arXiv:1812.04718.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
John R Doyle and Paul A Bottomley. 2006. Dressed for the occasion: Font-product congruity in the perception of logotype. Journal of consumer psychology, 16(2):112-123.
<sup>7</sup>https://cloud.google.comtranslate/docs apis
8611
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Xin Geng. 2016. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 28(7):1734-1748.
Xin Geng, Chao Yin, and Zhi-Hua Zhou. 2013. Facial age estimation by learning from label distributions. IEEE transactions on pattern analysis and machine intelligence, 35(10):2401-2412.
Pamela W Henderson, Joan L Giese, and Joseph A Cote. 2004. Impression management using typeface design. Journal of marketing, 68(4):60-72.
Samuel Juni and Julie S Gross. 2008. Emotional and persuasive perception of fonts. *Perceptual and motor skills*, 106(1):35-42.
Sudipta Kar, Suraj Maharjan, and Thamar Solorio. 2018. Folksonomication: Predicting tags for movies from plot synopses using emotion flow encoded neural network. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2879-2891.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Tugba Kulahcioglu and Gerard De Melo. 2018. Fontlex: A typographical lexicon based on affective associations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018).
Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86.
Jo Mackiewicz. 2007. Audience perceptions of fonts in projected powerpoint text slides. Technical communication, 54(3):295-307.
Jo Mackiewicz and Rachel Moeller. 2005. Why people perceive typefaces to have different personalities. In International Professional Communication Conference, 2004. IPCC 2004. Proceedings., pages 304-313. IEEE.
Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 881-893.
Saif M. Mohammad. 2018a. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of The Annual Conference of the Association for Computational Linguistics (ACL), Melbourne, Australia.
Saif M. Mohammad. 2018b. Word affect intensities. In Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018), Miyazaki, Japan.
Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436-465.
An Thanh Nguyen, Matthew Halpern, Byron C Wallace, and Matthew Lease. 2016. Probabilistic modeling for crowdsourcing partially-subjective ratings. In Fourth AAAI Conference on Human Computation and Crowdsourcing.
Peter O'Donovan, Janis Libeks, Aseem Agarwala, and Aaron Hertzmann. 2014. Exploratory font selection using crowdsourced attributes. ACM Transactions on Graphics (TOG), 33(4):92.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2014. Sequence labeling with multiple annotators. Machine learning, 95(2):165-181.
Joni O Salminen, Hind A Al-Merekhi, Partha Dey, and Bernard J Jansen. 2018. Inter-rater agreement for social computing studies. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS), pages 80-87. IEEE.
Yuto Shinahara, Takuro Karamatsu, Daisuke Harada, Kota Yamaguchi, and Seiichi Uchida. 2019. Serif or sans: Visual font analytics on book covers and online advertisements. arXiv preprint arXiv:1906.10269.
Amirreza Shirani, Franck Dernoncourt, Paul Asente, Nedim Lipka, Seokhwan Kim, Jose Echevarria, and Thamar Solorio. 2019. Learning emphasis selection for written text in visual media from crowd-sourced label distributions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1167-1172.
Ramya Srinivasan and Ajay Chander. 2019. Crowdsourcing in the absence of ground truth-a case study. arXiv preprint arXiv:1906.07254.
A Urkullu, A Perez, and B Calvo. 2019. On the evaluation and selection of classifier learning algorithms with crowdsourced data. Applied Soft Computing, 80:832-844.
8612