Venue: NAACL 2018

# Community Member Retrieval on Social Media using Textual Information

 Aaron Jaech, Shobhit Hathi, Mari Ostendorf

University of Washington

{ajaech, shathi, ostendor}uw.edu

###### Abstract

This paper addresses the problem of community membership detection using only text features in a scenario where a small number of positive labeled examples defines the community. The solution introduces an unsupervised proxy task for learning user embeddings: user re-identification. Experiments with 16 different communities show that the resulting embeddings are more effective for community membership identification than common unsupervised representations.

## 1 Introduction

Active users of social media often like identifying other users with common interests and values. Or, a user may want to find other users that share characteristics with specific accounts that they follow, e.g. cartoonists or local food trucks. Members of such communities of interest are often identifiable via their social network connections, and shared social connections are clearly important in recommendations. However, shared connections often reflect a subset of a person's interests, and there may be users of interest where any shared connections are distant. In addition, there may be scenarios where there is no explicit social graph, or the full graph is expensive to obtain. In such cases, the language of tweets, blogs, etc. is helpful in identifying users with particular interests.

In this paper, we represent users in terms of the text in their communications and introduce a scenario where a user can define a "community" by providing a small number of example accounts that are used to train a system for retrieving similar users. Note that our use of the term "community" differs from other online contexts, where members explicitly self-identify with a community (e.g. by joining a discussion forum or using a specific hashtag). The community is in the eye of the user issuing the query.

We frame the task of community membership detection as a retrieval problem. A small set of representative accounts selected by the user forms the query, and the system retrieves additional community members from a large index of accounts. The task is loosely related to entity set expansion Pantel et al. (2009). We make no assumptions about the type of communities that can be handled, and no labeled data is available other than the query. Because the training set (query) is minimal, unsupervised learning is useful for the text representation. We propose the proxy task of person re-identification for learning a user embedding, where the goal is for two embeddings from the same user to be closer to each other than to the embedding of a random user. The hypothesis is that a representation useful for detecting similarities between posts from the same person made at different times will also do well at identifying similarities between people in the same community. This hypothesis stems from observations that people with shared interests often talk about topics related to these interests, and that they tend to have shared jargon and other similarities in language us Nguyen and Rose (2011); Danescu-Niculescu-Mizil et al. (2013); Tran and Ostendorf (2016).

In this paper, we demonstrate experimentally that the re-identification proxy task is useful with simple models that are suited to the retrieval scenario, and present analyses showing that the approach learns to emphasize words associated with individual interests and polarizing issues.

## 2 Model

The model for community detection includes: i) a mapping from a user's text (a collection of tweets) to a \(k\)-dimensional embedding, and ii) a binary classifier for detecting whether a candidate user belongs to the target community. The novel con

tribution of the work is the proxy re-identification task for learning the user embedding.

User Embedding Model.The mapping from text to an embedding could leverage any document-level representation. We focus on a simple weighted bag-of-words neural model for direct comparison to other popular methods, motivated by the fact that many virtual communities form around shared interests in particular topics. Specifically, let \(c_{p,i}\) denote the number of times person \(p\) uses word \(v_{i}\in V\), where \(V\) is the vocabulary, and \(w_{p,i}=\log(c_{p,i}+1)\) be the log-scaled word count. Then the user embedding is

\[u_{p}=\frac{w^{T}_{p}\mathbf{E}}{||w^{T}_{p}\mathbf{E}||} \tag{1}\]

where \(w_{p}=[w_{p,1}\cdots w_{p,|V|}]\) and \(\mathbf{E}\in\Re^{|V|\times k}\) is the matrix of word embeddings.

Person Re-identification Learning.The embedding matrix \(\mathbf{E}\) is learned using a person re-identification objective that encourages embeddings from the same person to be closer than embeddings from different people. We build on the triplet loss function taken from Schroff et al. (2015) used to train a face recognition system. Specifically:

\[\mathbf{E}=\operatorname*{argmin}_{\mathbf{E}}\sum_{p_{1},p_{2}\in\mathcal{P} }\mathrm{cost}(p_{1},p_{2}), \tag{2}\]

\[\mathrm{cost}(p_{1},p_{2})=(1+\mathrm{d}(u_{p_{1}^{1}},u_{p_{1}^{2}})-\mathrm{ d}(u_{p_{1}^{1}},u_{p_{2}^{1}}))^{+},\]

xwhere \(\mathrm{d}(x,y)\) is the cosine distance between \(x\) and \(y\). \(u_{p_{1}^{1}}\) and \(u_{p_{2}^{1}}\) are embeddings made from distinct subsets of a single person's Tweets, and \(u_{p_{2}^{1}}\) is an embedding made from a subset of another person's Tweets. In practice, we estimate the loss function randomly sampling triplets (\(p_{1}^{1}\), \(p_{1}^{2}\), \(p_{2}^{1}\)) from a large training set.

Classifier.A logistic regression model with L2 regularization is used for the classifier, because it is simple but powerful and our scenario has little training data. Simplicity is important because the classifier should be trainable in real-time after receiving the query. The classifier objective is to discriminate the embeddings from the users in the query from a set of user embeddings from the general collection. For the \(i\)-th user, let \(y_{i}\in\{0,1\}\) be the binary label indicating whether the user belongs to a particular community and \(u_{i}\) be the user embedding. The logistic regression model computes the probability that the user belongs to the community according to:

\[p(y_{i}=1|u_{i})=\sigma(w^{T}u_{i}+b), \tag{3}\]

where \(\sigma(x)=1/(1-e^{-x})\). During evaluation, the users in the index are ranked according to the maximum log probability ratio

\[\operatorname*{argmax}_{i}\log\frac{p(y_{i}=1|u_{i})}{p(y_{i}=0|u_{i})}= \operatorname*{argmax}_{i}w^{T}u_{i}. \tag{4}\]

Because the classifier is linear, we can quickly retrieve the top matching users from the index using approximate nearest-neighbor search (Kushilevitz et al., 2000). The technique is scalable up to hundreds of millions of users and beyond.

## 3 Data

All data was collected using the Twitter API.1 We used 1,035 randomly selected items from the list of trending topics in the USA during the period April-June 2017 to query for users and collected their most recent 2,000 tweets. Example trending topics are #Quantico, RonaldoCristiano, and #MayDay2017. (The full list is available with the data.) Each user had at least one Tweet that mentioned a trending topic but their other Tweets could be on any topic.

Footnote 1: [http://developer.twitter.com/en/docs/api-reference-index](http://developer.twitter.com/en/docs/api-reference-index)

We refer to this collection as the "general population," because it was not targeted towards any particular community. In total, we collected around 80,000 such users and used roughly 36,000 for learning user embeddings, 1,000 for learning the community classifiers, and 43,000 for evaluation. The text is mostly in English, but some of it is in Spanish, French, or other languages. A list of the tweet IDs is available.2

Footnote 2: [http://github.com/ajaech/twittercommunities](http://github.com/ajaech/twittercommunities)

To support evaluation with the community detection task, we conducted a second collection (contemporaneous with the first) targeting members that we had identified as belonging to one of 16 communities (Table 2). To define a "community," volunteers manually selected a set of users that fit with a theme that they had familiarity with. Thus, the specific 16 communities were determined based on themes of interest to the authors and their friends and colleagues, where we could 

be reasonably confident about membership decisions. In addition, we tried to avoid themes that might be biased towards well-known celebrities, and we made an effort to have diversity in the characteristics of the communities. The communities were selected to span a range of topics, sizes (6-130 accounts), individuals vs. organizations, and other characteristics. A few of the communities are comprised of organizations rather than individuals such as the high school drama departments and the Pittsburgh food truck communities. (The community names are invented by the authors for purposes of describing the data in this paper; they are not part of the retrieval task.)

Footnote 2: [http://www.cds.org/](http://www.cds.org/)

The text is lower-cased and some punctuation is removed using regular expressions. Words are formed by splitting on white space. While this strategy will not work for languages that do not delimit words by spaces, these make up a negligible portion of the data. A 174k vocabulary was created by extracting the unique types that were seen in the tweets from the general population, as well as selected bigrams extracted using the open source Gensim library using a point-wise mutual information criteria Rehurek and Sojka (2010). The vocabulary included roughly 49k bigrams, 36k usernames and 17k hashtags. Usernames, hashtags, and URLs are not treated specially and can be part of the vocabulary just like any other word if they occur frequently enough.

## 4 Experiments

### Experiment Configuration

The experiments involved comparing different methods of learning user embeddings, all with a weighted bag-of-words modeling assumption:

* Weighted word2vec (W2V) using default3 skip-gram training Mikolov et al. (2013); Footnote 3: The default configuration uses a window of \(\pm 7\) words. We also tried using a window of 50 words, which roughly matches the context used in other methods, but community detection performance was significantly worse.
* Latent Dirichlet allocation (LDA) Blei et al. (2003), using default settings from the Scikit Learn library Pedregosa et al. (2011);
* Person re-identification with random initialization (RE-ID); and
* Person re-identification with W2V initialization (RE-ID, W2V init).

Both count-weighted W2V and LDA have been used as unsupervised representations in Twitter classification tasks, as noted in Section 5. Default configurations are used because there is insufficient data to have a separate validation set.

For all methods, the same vocabulary, final dimension (128), unit vector normalization strategy, and logistic regression model training were used. The embeddings are trained on the 36k user general data, randomly sampling pairs of users \(p_{1}\) and \(p_{2}\) and then sampling 50 tweets at a time without replacement to create \(u^{\text{${}_{p_{1}^{1}}$}}_{\text{$n_{p_{1}^{2}}$}}\), and \(u^{\text{${}_{p_{2}^{1}}$}}_{\text{$n_{2}^{1}$}}\). The logistic regression models are trained on the 1K user general training pool, using the 50 most recent tweets for each user. Because there are so few labeled examples for most communities, training and evaluation is done using a leave-one-out strategy with the positive samples but including all of the 1K negative samples. For each of the \(N\) classifiers (corresponding to \(N\) labeled samples), the test set is the left-out positive example and the 43K general user test pool. Also because of training limitations, there is no tuning of the regularization weight; the default weight of 1.0 is used. Tuning may be useful given a collection of training and testing communities. Performance is averaged over the \(N\) classifiers (corresponding to the \(N\) labeled samples). Two evaluation criteria are used: a retrieval metric (inverse mean reciprocal rank or 1/MRR) Voorhees et al. (1999) and a detection metric (area under the curve or AUC).

### Results

Table 1 shows retrieval results averaged across all communities. The RE-ID model outperforms the W2V and LDA baselines for both criteria, with substantial gains in 1/MRR (lower is better). Further, the version of RE-ID initialized with word2vec did better than the one that was initialized randomly even though the randomly initialized version was trained for twice as long.

A breakdown of the best model performance by community is given in Table 2. Sample size does not seem to be a good indicator of performance: the two smallest communities Cartoonists Fresno City Council 

of the best results, respectively. Anecdotally, we observed that the sample of cartoonists were more likely to Tweet about topics outside their main interest (e.g., politics or sports). We hypothesize that the diversity of interests of the members of a community affects the difficulty of the retrieval task, but our test set is too small to confirm this hypothesis.

Footnote 3: [https://www.facebook.com/news/article-research/2018/09/2017/04/2018/05/2](https://www.facebook.com/news/article-research/2018/09/2017/04/2018/05/2) 

are relevant across tasks. Our task is perhaps more similar to using social media text to predict author characteristics such as personality (Golbeck et al., 2011), gang membership (Wijeratne et al., 2016), geolocation (Han et al., 2014), political affiliation (Makazhanov et al., 2014), occupational class (Preotiuc-Pietro et al., 2015), and more. Again, a commonality across tasks is the frequent use of unsupervised representations of textual features.

In representing text, a common assumption is that community language reflects topical interests, so representations aimed at topic modeling have been used, including LDA (Pennacchiotti and Popescu, 2011) and tf-idf weighted word2vec embeddings (Boom et al., 2016; Wijeratne et al., 2016). Yu et al. (2016) compute a user embedding by averaging tweet embeddings. Other work investigates methods for learning embeddings that integrate text and social network (graph or text-based) features (Benton et al., 2016).

The work closest to ours is by Fani et al. (2017), which learns embeddings that are close for like-minded users, where like-minded pairs are identified by a deterministic algorithm that leverages timing of related posts. Our approach requires no additional heuristics for defining user similarity, but instead relies on an objective that maximizes self-similarity and minimizes similarity to other users randomly sampled from a large general pool.

Our person re-identification proxy task makes use of the triplet loss used to learn person embeddings for face recognition (Schroff et al., 2015). In image processing, person re-identification refers to the task of tracking people who have left the field of view of one camera and are later seen by another camera (Bedagkar-Gala and Shah, 2014). It is different from our proxy task and the methods are not the same.

## 6 Conclusion

In summary, this paper defines a task of community member retrieval based on their tweets, introduces a person re-identification task to allow community definition with a small number of examples, and shows that that the method gives very good results compared to word2vec and LDA baselines. Analyses show that the user embeddings learned efficiently represent user interests. The text embeddings are largely complementary to the social network features used in other studies, so performance gains can be expected from feature combination. While our experiments use a bag-of-words representation, as in most related work, the re-identification training objective proposed here can easily be used with other methods for deriving document embeddings, e.g. (Le and Mikolov, 2014; Kim, 2014).

## Acknowledgements

The authors thank the anonymous reviewers for their feedback and helpful suggestions.

## References

* Bedagkar-Gala and Shah (2014) Apurva Bedagkar-Gala and Shishir Shah. 2014. A survey of approaches and trends in person re-identification. _Image and Vision Computing_, 32(4):270-216.
* Benton et al. (2016) Adrian Benton, Raman Arora, and Mark Dredze. 2016. Learning multiview embeddings of Twitter users. In _Proc. ACL_, volume 2, pages 14-19.
* Blei et al. (2003) David Blei, Andrew Ng, and Michael Jordan. 2003. Latent dirichlet allocation. _Journal of Machine Learning Research_, 3:993-1022.
* De Boom et al. (2016) Cedric De Boom, Steven Van Canneyt, Thomas Demeester, and Bart Dhoedt. 2016. Representation learning for very short texts using weighted word embedding aggregation. _Pattern Recognition Letters_, 80:150-156.
* Chen et al. (2009) Jiyang Chen, Osmar R. Zaiane, and Randy Goebel. 2009. Local community identification in social networks. In _International Conference on Advances in Social Network Analysis and Mining_, pages 237-242.
* Danescu-Niculescu-Mizil et al. (2013) Cristian Danescu-Niculescu-Mizil, Robert West, Dan Jurafsky, Jure Leskovec, and Christopher Potts. 2013. No country for old members: User lifecycle and linguistic change in online communities. In _Proc. WWW_.
* Di (2011) Ying Di. 2011. Community detection: topological vs. topical. _Journal of Informatics_, 5(4):489-514.
* Fani et al. (2017) Hossein Fani, Ebrahim Bagheri, and Weichang Du. 2017. Temporally like-minded user community identification through neural embeddings. In _Proc. ACM Conference on Information and Knowledge Management_, pages 577-586.
* Golbeck et al. (2011) Jennifer Golbeck, Cristina Robles, Michon Edmondson, and Karen Turner. 2011. Predicting personality from twitter. In _Privacy, Security, Risk and Trust (PASSAT) and 2011 IEEE Third International Conference on Social Computing (SocialCom), 2011 IEEE Third International Conference on_, pages 149-156. IEEE 

Pankaj Gupta, Ashish Goel, Jimmy Lin, Aneesh Sharma, Dong Wang, and Reza Zadeh. 2013. Wtf: The who to follow service at Twitter. In _Proc. WWW_, pages 505-514. ACM.
* Han et al. (2014) Bo Han, Paul Cook, and Timothy Baldwin. 2014. Text-based Twitter user geolocation prediction. _Journal of Artificial Intelligence Research_, 49:451-500.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proc. EMNLP_, pages 1746-1751.
* Kushilevitz et al. (2000) Eyal Kushilevitz, Rafail Ostrovsky, and Yuval Rabani. 2000. Efficient search for approximate nearest neighbor in high dimensional spaces. _SIAM Journal on Computing_, 30(2):457-474.
* Le and Mikolov (2014) Quo Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In _Proc. ICML_, pages 3104-3112.
* Makazhanov et al. (2014) Aibek Makazhanov, Davood Rafiei, and Muhammad Waqar. 2014. Predicting political preference of twitter users. _Social Network Analysis and Mining_, 4(1):1-15.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Proc. NIPS_, pages 3111-3119.
* Nguyen and Rose (2011) Dong Nguyen and Carolyn P. Rose. 2011. Language use as a reflection of socialization in online communities. In _Proceedings of the Workshop on Languages in Social Media_, LSM '11, pages 76-85. Association for Computational Linguistics.
* Pantel et al. (2009) Patrick Pantel, Eric Crestan, Arkady Borkovsky, Ana-Maria Popescu, and Vishnu Vyas. 2009. Web-scale distributional similarity and entity set expansion. In _Proc. EMNLP_, pages 938-947. Association for Computational Linguistics.
* Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. _Journal of Machine Learning Research_, 12:2825-2830.
* Pennacchiotti and Popescu (2011) Marco Pennacchiotti and Ana-Maria Popescu. 2011. A machine learning approach to Twitter user classification. In _ICWSM_.
* Preotiuc-Pietro et al. (2015) Daniel Preotiuc-Pietro, Vasileios Lampos, and Nikolaos Aletras. 2015. An analysis of the user occupational class through Twitter content. In _Proc. ACL-IJCNLP_, pages 1754-1764.
* Rehurek and Sojka (2010) Radim Rehurek and Petr Sojka. 2010. Software framework for topic modelling with large corpora. In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_, pages 45-50, Valletta, Malta. ELRA. [http://is.muni.cz/publication/884893/en](http://is.muni.cz/publication/884893/en).
* Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In _Proc. CVPR_, pages 815-823.
* Tran and Ostendorf (2016) Trang Tran and Mari Ostendorf. 2016. Characterizing the language of online communities and its relation to community reception. In _Proc. EMNLP_.
* Voorhees et al. (1999) Ellen M Voorhees et al. 1999. The TREC-8 question answering track report. In _Trec_, volume 99, pages 77-82.
* Wijeratne et al. (2016) Sanjaya Wijeratne, Lakshika Balasiuriya, Derek Doran, and Amit Sheth. 2016. Word embeddings to enhance Twitter gang member profile identification. In _Proc. IJCAI Workshop on Semantic Machine Learning_.
* Yu et al. (2016) Yang Yu, Xiaojun Wan, and Xinjie Zhu. 2016. User embedding for scholarly microblag recommendation. In _Proc. ACL_, pages 449-453.

 



## Appendix: Supplementary Tables

\begin{table}
\begin{tabular}{l l}
**Interpretation** & **Top Words** \\ \hline Languages \& & \(\bullet\) a, ca, j’ai, quand, c’est, avec, sur, dans\_le \\ Dialects & \(\bullet\) c, não, melhor, tem, mesmo, so, mais, hoje, luma, tá, ja \\  & \(\bullet\) es\_un, más, jugar, en\_el, muduro, jajajaja \\  & \(\bullet\) bruh, dawg, @iamakademiks, black\_women, @chancetherapper, Imaooo, y’all, tryna \\ \hline Sports & \(\bullet\)@mlb, baseball, bullpen, @angels, mets, mlb \\  & \(\bullet\) arsenal, mate, liverpool, @manutd, moruinho, \#murofc \\  & \(\bullet\)@nhl, hockey, nhl, leafs, @nhlblackhawks, @nhlonnhcsports \\  & \(\bullet\) xd, @playoverwatch, \#ps4share, anime, @keemstar, overwatch, twitch, @intendoamericta, gaming \\ \hline Music & \(\bullet\)@nialofficial, @harry\_styles, @louis.tomlinson, @ashton5sos, @shawnmmedes, @ethanololan, @graysondolan, @michael5sos, @danisonotonfire \\ \hline Political & \(\bullet\) @indivisibleteam \#resist, \#trumpcare, @ezlaszig, @kurteichenwald, @george-takei, @sarahkendzior, @repadamschiff, @malcolmnance, @lawrence \\  & \(\bullet\) @mitchellvii, @prisonplanet, @realjameswoods, @jackposobiec, @bfraser747, @cernovich, @ten\_gop, \#maga \\ \hline Other & \(\bullet\) tories, labour, corbyn, \#auspol, torsy, mum, rhs, lads scotland \\ \end{tabular}
\end{table}
Table 4: Clusters of words that change the most between Word2Vec and the re-identification objective.

\begin{table}
\begin{tabular}{p{108.3pt} p{196.2pt}}
**Community** & **Selected Tweet** \\ \hline Chess Stars & @chescom ype karpov well\_done twittersphere \\ Professional Economists & \#china real\_estate as long as liquidity remains ample this will continue \\ Fashion Commentators & rihamana’s fenty corp creative\_director jahleel weaver styles the collection on 3 muses \\ Fresno City Council & gr8 resource developed by our local @citdfresno on how to export @city-offresno @fresnoCountyedce lee\_ann eager \\ High School Drama & we’re looking\_forward to seeing you opening\_night may 24th love the cast of high\_school musical \\ Mathematicians NLP Researchers & forms of knowledge of advanced mathematics for teaching (i wrote a thing) recurrent neural\_network grammars simplified and analyzed \\ Police Dogs & when a trained police dog is placed with another handler they complete a re handling course to be licensed normally 2\_weeks \\ SCOTUS Reporters & as supreme\_court throws out two gop-drawn congressional\_districts as unconsutriational racial gerrymanders \\ Ultramarathon Runners & we’re covering the lake sonoma 50 mile live on saturday tell your friends spread the word and get ready \\ \end{tabular}
\end{table}
Table 3: Top tweets for selected communities. Underscore is used to join bigrams.