Introduction
Author profiling is the task of discovering latent user attributes disclosed through text, such as gender, age, personality, income, location and occupation BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . It is of interest to several applications including personalized machine translation, forensics, and marketing BIBREF7 , BIBREF8 .
Early approaches to gender prediction BIBREF9 , BIBREF10 are inspired by pioneering work on authorship attribution BIBREF11 . Such stylometric models typically rely on carefully hand-selected sets of content-independent features to capture style beyond topic. Recently, open vocabulary approaches BIBREF12 , where the entire linguistic production of an author is used, yielded substantial performance gains in online user-attribute prediction BIBREF13 , BIBREF14 , BIBREF15 . Indeed, the best performing gender prediction models exploit chiefly lexical information BIBREF16 , BIBREF17 .
Relying heavily on the lexicon though has its limitations, as it results in models with limited portability. Moreover, performance might be overly optimistic due to topic bias BIBREF18 . Recent work on cross-lingual author profiling has proposed the use of solely language-independent features BIBREF19 , e.g., specific textual elements (percentage of emojis, URLs, etc) and users' meta-data/network (number of followers, etc), but this information is not always available.
We propose a novel approach where the actual text is still used, but bleached out and transformed into more abstract, and potentially better transferable features. One could view this as a method in between the open vocabulary strategy and the stylometric approach. It has the advantage of fading out content in favor of more shallow patterns still based on the original text, without introducing additional processing such as part-of-speech tagging. In particular, we investigate to what extent gender prediction can rely on generic non-lexical features (RQ1), and how predictive such models are when transferred to other languages (RQ2). We also glean insights from human judgments, and investigate how well people can perform cross-lingual gender prediction (RQ3). We focus on gender prediction for Twitter, motivated by data availability.
Profiling with Abstract Features
Can we recover the gender of an author from bleached text, i.e., transformed text were the raw lexical strings are converted into abstract features? We investigate this question by building a series of predictive models to infer the gender of a Twitter user, in absence of additional user-specific meta-data. Our approach can be seen as taking advantage of elements from a data-driven open-vocabulary approach, while trying to capture gender-specific style in text beyond topic.
To represent utterances in a more language agnostic way, we propose to simply transform the text into alternative textual representations, which deviate from the lexical form to allow for abstraction. We propose the following transformations, exemplified in Table TABREF2 . They are mostly motivated by intuition and inspired by prior work, like the use of shape features from NER and parsing BIBREF20 , BIBREF21 , BIBREF22 , BIBREF23 :
Experiments
In order to test whether abstract features are effective and transfer across languages, we set up experiments for gender prediction comparing lexicalized and bleached models for both in- and cross-language experiments. We compare them to a model using multilingual embeddings BIBREF24 . Finally, we elicit human judgments both within language and across language. The latter is to check whether a person with no prior knowledge of (the lexicon of) a given language can predict the gender of a user, and how that compares to an in-language setup and the machine. If humans can predict gender cross-lingually, they are likely to rely on aspects beyond lexical information.
Lexical vs Bleached Models
We use the scikit-learn BIBREF26 implementation of a linear SVM with default parameters (e.g., L2 regularization). We use 10-fold cross validation for all in-language experiments. For the cross-lingual experiments, we train on all available source language data and test on all target language data.
For the lexicalized experiments, we adopt the features from the best performing system at the latest PAN evaluation campaign BIBREF17 (word 1-2 grams and character 3-6 grams).
For the multilingual embeddings model we use the mean embedding representation from the system of BIBREF27 and add max, std and coverage features. We create multilingual embeddings by projecting monolingual embeddings to a single multilingual space for all five languages using a recently proposed SVD-based projection method with a pseudo-dictionary BIBREF28 . The monolingual embeddings are trained on large amounts of in-house Twitter data (as much data as we had access to, i.e., ranging from 30M tweets for French to 1,500M tweets in Dutch, with a word type coverage between 63 and 77%). This results in an embedding space with a vocabulary size of 16M word types. All code is available at https://github.com/bplank/bleaching-text. For the bleached experiments, we ran models with each feature set separately. In this paper, we report results for the model where all features are combined, as it proved to be the most robust across languages. We tuned the INLINEFORM0 -gram size of this model through in-language cross-validation, finding that INLINEFORM1 performs best.
When testing across languages, we report accuracy for two setups: average accuracy over each single-language model (Avg), and accuracy obtained when training on the concatenation of all languages but the target one (All). The latter setting is also used for the embeddings model. We report accuracy for all experiments.
Table TABREF13 shows results for both the cross-language and in-language experiments in the lexical and abstract-feature setting.
Within language, the lexical features unsurprisingly work the best, achieving an average accuracy of 80.5% over all languages. The abstract features lose some information and score on average 11.8% lower, still beating the majority baseline (50%) by a large margin (68.7%). If we go across language, the lexical approaches break down (overall to 53.7% for Lex Avg/56.3% for All), except for Portuguese and Spanish, thanks to their similarities (see Table TABREF16 for pair-wise results). The closely-related-language effect is also observed when training on all languages, as scores go up when the classifier has access to the related language. The same holds for the multilingual embeddings model. On average it reaches an accuracy of 59.8%.
The closeness effect for Portuguese and Spanish can also be observed in language-to-language experiments, where scores for ES INLINEFORM0 PT and PT INLINEFORM1 ES are the highest. Results for the lexical models are generally lower on English, which might be due to smaller amounts of data (see first column in Table TABREF13 providing number of users per language).
The abstract features fare surprisingly well and work a lot better across languages. The performance is on average 6% higher across all languages (57.9% for Avg, 63.9% for All) in comparison to their lexicalized counterparts, where Abs All results in the overall best model. For Spanish, the multilingual embedding model clearly outperforms Abs. However, the approach requires large Twitter-specific embeddings.
For our Abs model, if we investigate predictive features over all languages, cf. Table TABREF19 , we can see that the use of an emoji (like ) and shape-based features are predictive of female users. Quotes, question marks and length features, for example, appear to be more predictive of male users.
Human Evaluation
We experimented with three different conditions, one within language and two across language. For the latter, we set up an experiment where native speakers of Dutch were presented with tweets written in Portuguese and were asked to guess the poster's gender. In the other experiment, we asked speakers of French to identify the gender of the writer when reading Dutch tweets. In both cases, the participants declared to have no prior knowledge of the target language. For the in-language experiment, we asked Dutch speakers to identify the gender of a user writing Dutch tweets. The Dutch speakers who participated in the two experiments are distinct individuals. Participants were informed of the experiment's goal. Their identity is anonymized in the data.
We selected a random sample of 200 users from the Dutch and Portuguese data, preserving a 50/50 gender distribution. Each user was represented by twenty tweets. The answer key (F/M) order was randomized. For each of the three experiments we had six judges, balanced for gender, and obtained three annotations per target user.
Inter-annotator agreement for the tasks was measured via Fleiss kappa ( INLINEFORM0 ), and was higher for the in-language experiment ( INLINEFORM1 ) than for the cross-language tasks (NL INLINEFORM2 PT: INLINEFORM3 ; FR INLINEFORM4 NL: INLINEFORM5 ). Table TABREF22 shows accuracy against the gold labels, comparing humans (average accuracy over three annotators) to lexical and bleached models on the exact same subset of 200 users. Systems were tested under two different conditions regarding the number of tweets per user for the target language: machine and human saw the exact same twenty tweets, or the full set of tweets (200) per user, as done during training (Section SECREF14 ).
First of all, our results indicate that in-language performance of humans is 70.5%, which is quite in line with the findings of BIBREF6 , who report an accuracy of 75% on English. Within language, lexicalized models are superior to humans if exposed to enough information (200 tweets setup). One explanation for this might lie in an observation by BIBREF6 , according to which people tend to rely too much on stereotypical lexical indicators when assigning gender to the poster of a tweet, while machines model less evident patterns. Lexicalized models are also superior to the bleached ones, as already seen on the full datasets (Table TABREF13 ).
We can also observe that the amount of information available to represent a user influences system's performance. Training on 200 tweets per user, but testing on 20 tweets only, decreases performance by 12 percentage points. This is likely due to the fact that inputs are sparser, especially since the bleached model is trained on 5-grams. The bleached model, when given 200 tweets per user, yields a performance that is slightly higher than human accuracy.
In the cross-language setting, the picture is very different. Here, human performance is superior to the lexicalized models, independently of the amount of tweets per user at testing time. This seems to indicate that if humans cannot rely on the lexicon, they might be exploiting some other signal when guessing the gender of a user who tweets in a language unknown to them. Interestingly, the bleached models, which rely on non-lexical features, not only outperform the lexicalized ones in the cross-language experiments, but also neatly match the human scores.
Related Work
Most existing work on gender prediction exploits shallow lexical information based on the linguistic production of the users. Few studies investigate deeper syntactic information BIBREF9 , BIBREF2 or non-linguistic input, e.g., language-independent clues such as visual BIBREF29 or network information BIBREF3 , BIBREF5 , BIBREF19 . A related angle is cross-genre profiling. In both settings lexical models have limited portability due to their bias towards the language/genre they have been trained on BIBREF30 , BIBREF31 , BIBREF32 .
Lexical bias has been shown to affect in-language human gender prediction, too. BIBREF6 found that people tend to rely too much on stereotypical lexical indicators, while BIBREF13 show that more than 10% of the Twitter users do actually not employ words that the crowd associates with their biological sex. Our features abstract away from such lexical cues while retaining predictive signal.
Conclusions
Bleaching text into abstract features is surprisingly effective for predicting gender, though lexical information is still more useful within language (RQ1). However, models based on lexical clues fail when transferred to other languages, or require large amounts of unlabeled data from a similar domain as our experiments with the multilingual embedding model indicate. Instead, our bleached models clearly capture some signal beyond the lexicon, and perform well in a cross-lingual setting (RQ2). We are well aware that we are testing our cross-language bleached models in the context of closely related languages. While some features (such as PunctA, or Frequency) might carry over to genetically more distant languages, other features (such as Vowels and Shape) would probably be meaningless. Future work on this will require a sensible setting from a language typology perspective for choosing and testing adequate features.
In our novel study on human proficiency for cross-lingual gender prediction, we discovered that people are also abstracting away from the lexicon. Indeed, we observe that they are able to detect gender by looking at tweets in a language they do not know (RQ3) with an accuracy of 60% on average.
Acknowledgments
We would like to thank the three anonymous reviewers and our colleagues for their useful feedback on earlier versions of this paper. Furthermore, we are grateful to Chloé Braud for helping with the French human evaluation part. We would like to thank all of our human participants.

Question: What are the evaluation metrics used?
Answer: average accuracy over each single-language model (Avg), and accuracy obtained when training on the concatenation of all languages but the target one (All)