Evaluation	We evaluate the performance of using word embeddings to discover gang member profiles on Twitter. We first discuss the dataset, learning algorithms and baseline comparison models used in the experiments. Then we discuss the 10-fold cross validation experiments and the evaluation matrices used. Finally we present the results of the experiments.

Overall, out of the five vector operations that we used to train machine learning classifiers, four gave us classifier models that beat baseline Model(1) and two vector based operations gave us classifier models that either achieved very similar results to baseline Model(2) or beat it. This evaluation demonstrates the promise of using pre-trained word embeddings to boost the accuracy of supervised learning algorithms for Twitter gang member profile classification.

10-fold cross validation	We conducted 10-fold cross validation experiments to evaluate the performance of our models. We used all Twitter profiles in the dataset to conduct experiments on the five methods we used to combine word embedding vectors. For each of the five vector combination methods (as mentioned in Section SECREF9 ), we trained classifiers using each learning algorithm we considered. In each fold, the training set was used to generate the word vectors, which were then used to compute features for both the training set and test set. For each 10-fold cross validation experiment, we report three evaluation metrics for the `gang' (positive) and `non-gang' (negative) classes, namely, the Precision = INLINEFORM0 , Recall = INLINEFORM1 , and INLINEFORM2 -score = INLINEFORM3 , where INLINEFORM4 is the number of true positives, INLINEFORM5 is the number of false positives, INLINEFORM6 is the number of true negatives, and INLINEFORM7 is the number of false negatives. We report these metrics for the `gang' and `non-gang' classes separately because of the class imbalance in the dataset.

Conclusion and Future Work	This paper presented a word embeddings-based approach to address the problem of automatically identifying gang member profiles on Twitter. Using a Twitter user dataset that consist of 400 gang member and 2,865 non gang member profiles, we trained word embedding models based on users' tweets, profile descriptions, emoji, images, and videos shared on Twitter (textual features extracted from images, and videos). We then use the pre-trained word embedding models to train supervised machine learning classifiers, which showed superior performance when compared to the state-of-the-art baseline models reported in the literature. We plan to further extend our work by building our own image classification system specifically designed to identify images commonly shared by gang members such as guns, gang hand signs, stacks of cash and drugs. We would also like to experiment with automatically building dictionaries that contain gang names and gang-related slang using crowd-sourced gang-related knowledge-bases such as HipWiki. We also want to experiment with using such knowledge-bases to train word embeddings to understand whether having access to gang-related knowledge could boost the performance of our models. Finally, we would like to study how we can further use social networks of known gang members to identify new gang member profiles on Twitter.

Word Embeddings to Enhance Twitter Gang Member Profile Identification	Gang affiliates have joined the masses who use social media to share thoughts and actions publicly. Interestingly, they use this public medium to express recent illegal actions, to intimidate others, and to share outrageous images and statements. Agencies able to unearth these profiles may thus be able to anticipate, stop, or hasten the investigation of gang-related crimes. This paper investigates the use of word embeddings to help identify gang members on Twitter. Building on our previous work, we generate word embeddings that translate what Twitter users post in their profile descriptions, tweets, profile images, and linked YouTube content to a real vector format amenable for machine learning classification. Our experimental results show that pre-trained word embeddings can boost the accuracy of supervised learning algorithms trained over gang members social media posts.

Despite its public nature, gang members post on social media without fear of consequences because there are only few tools law enforcement can presently use to surveil social media  . For example, the New York City police department employs over 300 detectives to combat teen violence triggered by insults, dares, and threats exchanged on social media, and the Toronto police department teaches officers about the use of social media in investigations  . From offline clues, the officers monitor just a selected set of social media accounts which are manually discovered and related to a specific investigation. Thus, developing tools to identify gang member profiles on social media is an important step in the direction of using machine intelligence to fight crime. To help agencies monitor gang activity on social media, our past work investigated how features from Twitter profiles, including profile text, profile images, tweet text, emjoi use, and their links to YouTube, may be used to reliably find gang member profiles  . The diverse set of features, chosen to combat the fact that gang members often use local terms and hashtags in their posts, offered encouraging results. In this paper, we report our experience in integrating deep learning into our gang member profile classifier.

Evaluation setup	We consider a dataset of curated gang and non-gang members' Twitter profiles collected from our previous work  . It was developed by querying the Followerwonk Web service API with location-neutral seed words known to be used by gang members across the U.S. in their Twitter profiles. The dataset was further expanded by examining the friends, follower, and retweet networks of the gang member profiles found by searching for seed words. Specific details about our data curation procedure are discussed in  . Ultimately, this dataset consists of 400 gang member profiles and 2,865 non-gang member profiles. For each user profile, we collected up to most recent 3,200 tweets from their Twitter timelines, profile description text, profile and cover images, and the comments and video descriptions for every YouTube video shared by them. Table 1 provides statistics about the number of words found in each type of feature in the dataset. It includes a total of 821,412 tweets from gang members and 7,238,758 tweets from non-gang members. To build the classifiers we used three different learning algorithms, namely Logistic Regression (LR), Random Forest (RF), and Support Vector Machines (SVM).

We ignore the words that occur less than 5 times in our training corpus. We investigated how well the local language has been captured by the word embedding models we trained. We used the `most similar' functionality offered by Word2Vec tool to understand what the model has learned about few gang-related slang terms which are specific to Chicago area. For example, we analyzed the ten most similar words learned by the word embedding model for the term BDK (Black Desciples Killers). We noticed that out of the 10 most similar words, five were names of local Chicago gangs, which are rivals of the Black Disciples Gang, two were different syntactic variations of BDK (bdkk, bdkkk) and the other three were different syntactic variations of GDK (gdk, gdkk, gdkkk). GDK is a local gang slang for `Gangster Disciples Killer' which is used by rivals of Gangster Disciples gang to show their hatred towards them. We found similar results for the term GDK. Out of the ten most similar words, six were showing hatred towards six different Gangster Disciples gangs that operate in Chicago area.

We used version 0.17.1 of scikit-learn machine learning library for Python to implement the classifiers. An open source tool of Python, Gensim  was used to generate the word embeddings. We compare our results with the two best performing systems reported in  which are the two state-of-the-art models for identifying gang members in Twitter. Both baseline models are built from a random forest classifier trained over term frequencies for unigrams in tweet text, emoji, profile data, YouTube video data and image tags. Baseline Model(1) considers all 3,285 gang and non-gang member profiles in our dataset. Baseline Model(2) considers all Twitter profiles that contain every feature type discussed in Section SECREF2 . Because a Twitter profile may not have every feature type, baseline Model(1) represents a practical scenario where not every Twitter profile contains every type of feature. However, we compare our results to both baseline models and report the improvements.

Specifically, we investigate the effect of translating the features into a vector space using word embeddings  . This idea is motivated by the recent success of word embeddings-based methods to learn syntactic and semantic structures automatically when provided with large datasets. A dataset of over 3,000 gang and non-gang member profiles that we previously curated is used to train the word embeddings. We show that pre-trained word embeddings improve the machine learning models and help us obtain an INLINEFORM0 -score of INLINEFORM1 on gang member profiles (a 6.39% improvement in INLINEFORM2 -score compared to the baseline models which were not trained using word embeddings). This paper is organized as follows. Section SECREF2 discusses the related literature and frames how this work differs from other related works. Section SECREF3 discusses our approach based on word embeddings to identify gang member profiles. Section SECREF4 reports on the evaluation of the proposed approach and the evaluation results in detail. Section SECREF5 concludes the work reported while discussing the future work planned.

showed that simple algebraic operations can be performed on word embeddings learned over a text corpus, which leads to findings such as the word embedding vector of the word “King” INLINEFORM0 the word embedding vectors of “Man” INLINEFORM1 “Woman” would results in a word embedding vector that is closest to the word embedding vector of the word “Queen”  . Recent successes in using word embeddings to improve text classification for short text  ,  , encouraged us to explore how they can be used to improve gang and non-gang member Twitter profile classification. Word embeddings can be performed under different neural network architectures; two popular ones are the Continuous Bag-of-Words (CBOW) and Continuous Skip-gram (Skip-gram) models  . The CBOW model learns a neural network such that given a set of context words surrounding a target word, it predict a target word. The Skip-gram model differs by predicting context words given a target word and by capturing the ordering of word occurrences. Recent improvements to Skip-gram model make it better able to handle less frequent words, especially when negative sampling is used  .

Classification approach	Figure FIGREF11 gives an overview of the steps to learn word embeddings and to integrate them into a classifier. We first convert any non-textual features such as emoji and profile images into textual features. We use Emoji for Python and Clarifai services, respectively, to convert emoji and images into text. Prior to training the word embeddings, we remove all the seed words used to find gang member profiles and stopwords, and perform stemming across all tweets and profile descriptions. We then feed all the training data (word INLINEFORM0 in Figure FIGREF11 ) we collected from our Twitter dataset to Word2Vec tool and train it using a Skip-gram model with negative sampling. When training the Skip-gram model, we set the negative sampling rate to 10 sample words, which seems to work well with medium-sized datasets  . We set the context word window to be 5, so that it will consider 5 words to left and right of the target word (words INLINEFORM1 to INLINEFORM2 in Figure FIGREF11 ). This setting is suitable for sentences where average sentence length is less than 11 words, as is the case in tweets  .

Word Embeddings	A word embedding model is a neural network that learns rich representations of words in a text corpus. It takes data from a large, INLINEFORM0 -dimensional `word space' (where INLINEFORM1 is the number of unique words in a corpus) and learns a transformation of the data into a lower INLINEFORM2 -dimensional space of real numbers. This transformation is developed in a way that similarities between the INLINEFORM3 -dimensional vector representation of two words reflects semantic relationships among the words themselves. These semantics are not captured by typical bag-of-words or INLINEFORM4 -gram models for classification tasks on text data  ,  . Word embeddings have led to the state-of-the-art results in many sequential learning tasks  . In fact, word embedding learning is an important step for many statistical language modeling tasks in text processing systems. Bengio et al. were the first ones to introduce the idea of learning a distributed representation for words over a text corpus  . They learned representations for each word in the text corpus using a neural network model that modeled the joint probability function of word sequences in terms of the feature vectors of the words in the sequence. Mikolov et al.

Further, we noticed that gang members mainly use Twitter to discuss drugs and money using terms such as smoke, high, hit, money, got, and need while non-gang members mainly discuss their feelings using terms such as new, like, love, know, want, and look. Twitter profile description: We found gang member profile descriptions to be rife with curse words (nigga, fuck, and shit) while non-gang members use words related to their feelings or interests (love, life, music, and book). We noticed that gang members use their profile descriptions as a space to grieve for their fallen or incarcerated gang members as about INLINEFORM0 of gang member Twitter profiles used terms such as rip and free. Emoji features: We found that the fuel pump emoji was the most frequently used emoji by gang members, which is often used in the context of selling or consuming marijuana. The pistol emoji was the second most frequently used emoji, which is often used with the police cop emoji in an `emoji chain' to express their hatred towards law enforcement officers.

Experimental results	Table TABREF22 presents 10-fold cross validation results for the baseline models (first and second rows) and our word embeddings-based models (from third row to seventh row). As mentioned earlier both baseline models use a random forest classifier trained on term frequencies of unigram features extracted from all feature types. The two baseline models only differs on the training data filtering method used, which is based on the availability of features in the training dataset as described in  . The baseline Model(1) uses all profiles in the dataset and has a INLINEFORM0 -score of 0.7364 for `gang' class and 0.9690 for `non-gang' class. The baseline Model(2) which only uses profiles that contain each and every feature type has a INLINEFORM1 -score of 0.7755 for `gang' class and INLINEFORM2 -score of 0.9720 for `non-gang' class. Vector sum is one of the basic operations we can perform on word embedding vectors. The random forest classifier performs the best among vector sum-based classifiers where logistic regression and SVM classifiers also perform comparatively well ( INLINEFORM0 ).

Introduction	Street gangs are defined as “a coalition of peers, united by mutual interests, with identifiable leadership and internal organization, who act collectively to conduct illegal activity and to control a territory, facility, or enterprise”  . They promote criminal activities such as drug trafficking, assault, robbery, and threatening or intimidating a neighborhood  . Today, over 1.4 million people, belonging to more than 33,000 gangs, are active in the United States  , of which 88% identify themselves as being members of a street gang. They are also active users of social media  ; according to 2007 National Assessment Center's survey of gang members, 25% of individuals in gangs use the Internet for at least 4 hours a week  . More recent studies report approximately 45% of gang members participate in online offending activities such as threatening, harassing individuals, posting violent videos or attacking someone on the street for something they said online  ,  . They confirm that gang members use social media to express themselves in ways similar to their offline behavior on the streets  .

Features considered	Gang member tweets and profile descriptions tend to have few textual indicators that demonstrate their gang affiliations or their tweets/profile text may carry acronyms which can only be deciphered by others involved in gang culture  . These gang-related terms are often local to gangs operating in neighborhoods and change rapidly when they form new gangs. Consequently, building a database of keywords, phrases, and other identifiers to find gang members nationally is not feasible. Instead, we use heterogeneous sets of features derived not only from profile and tweet text but also from the emoji usage, profile images, and links to YouTube videos reflecting their music preferences and affinity. In this section, we briefly discuss the feature types and their broad differences in gang and non-gang member profiles. An in-depth explanation of these feature selection can be found in  . Tweet text: In our previous work, we observed that gang members use curse words nearly five times more than the average curse words use on Twitter  .

showed that short text categorization can be improved by word embeddings with the help of a neural network model that feeds semantic cliques learned over word embeddings in to a convolutions neural network  . We believe our corpus of gang and non-gang member tweets, with nearly 64.6 million word tokens, could act as a rich resource to train word embeddings for distinguishing gang and non-gang member Twitter users. Our investigation differs from other word embeddings-based text classification systems such as  ,  due to the fact that we use multiple feature types including emojis in tweets and image tags extracted from Twitter profile and cover images in our classification task.

Using vector mean ( INLINEFORM1 ) improves all classifier results and SVM classifier trained on the mean of word embeddings achieves very close results to the baseline Model(2). Multiplying vector sum with corresponding word counts for each word in word embeddings degrades the classifier accuracy for correctly identifying the positive class ( INLINEFORM2 ). When we multiply words by their corresponding INLINEFORM3 - INLINEFORM4 values before taking the vector sum, we again observe an increase in the classifiers' accuracy ( INLINEFORM5 ). We achieve the best performance by averaging the vector sum weighted by term frequency ( INLINEFORM6 ). Here we multiply the mean of the word embeddings by count of each word, which beats all other word embeddings-based models and the two baselines. In this setting, logistic regression classifier trained on word embeddings performs the best with a INLINEFORM7 -score of 0.7835. This is a 6.39% improvement in performance when compared to the baseline Model(1) and a 1.03% improvement in performance when compared to baseline Model(2).

The money bag emoji, money with wings emoji, unlock emoji, and a variety of the angry face emoji such as the devil face emoji and imp emoji were also common in gang members' but not in non-gang members' tweets. Twitter profile and cover images: We noticed that gang members often pose holding or pointing weapons, seen in a group fashion which displays a gangster culture, show off graffiti, hand signs, tattoos, and bulk cash in their profile and cover images. We used Clarifai web service to tag the profile and cover images of the Twitter users in our dataset and used the image tags returned by Clarifai API to train word embeddings. Tags such as trigger, bullet, and worship were unique for gang member profiles while non-gang member images had unique tags such as beach, seashore, dawn, wildlife, sand, and pet. YouTube videos: We found that 51.25% of the gang members in our dataset have a tweet that links to a YouTube video. Further, we found that 76.58% of the shared links are related to hip-hop music, gangster rap, and the culture that surrounds this music genre  .

We believe that those who used the term GDK to show their hatred towards Gangster Disciples gangs might be also having rivalry with the six gangs we found. We obtain word vectors of size 300 from the learned word embeddings. To represent a Twitter profile, we retrieve word vectors for all the words that appear in a particular profile including the words appear in tweets, profile description, words extracted from emoji, cover and profile images converted to textual formats, and words extracted from YouTube video comments and descriptions for all YouTube videos shared in the user's timeline. Those word vectors are combined to compute the final feature vector for the Twitter profile. To combine the word vectors, we consider five different methods. Letting the size of a word vector be INLINEFORM0 , for a Twitter profile INLINEFORM1 with INLINEFORM2 unique words and the vector of the INLINEFORM3 word in INLINEFORM4 denoted by INLINEFORM5 , we compute the feature vector for the Twitter profile INLINEFORM6 by: Sum of word embeddings INLINEFORM0 . This is the sum the word embedding vectors obtained for all words in a Twitter profile: INLINEFORM1  Mean of word embeddings INLINEFORM0 .

A data collection process involving location neutral keywords used by gang members, with an expanded search of their retweet, friends and follower networks, led to identifying 400 authentic gang member profiles on Twitter. Our study discovered that the text in their tweets and profile descriptions, their emoji use, their profile images, and music interests embodied by links to YouTube music videos, can help a classifier distinguish between gang and non-gang member profiles. While a very promising INLINEFORM0 measure with low false positive rate was achieved, we hypothesize that the diverse kinds and the multitude of features employed (e.g. unigrams of tweet text) could be amenable to an improved representation for classification. We thus explore the possibility of mapping these features into a considerably smaller feature space through the use of word embeddings. Previous research has shown word embeddings-based methods can significantly improve short text classification  ,  . For example, Lilleberget et al. showed that word embeddings weighted by INLINEFORM0 - INLINEFORM1 outperforms other variants of word embedding models discussed in  , after training word embedding models on over 18,000 newsgroup posts. Wang et al.

Moreover, we found that eight YouTube links are shared on average by a gang member. The top 5 terms used in YouTube videos shared by gang members were shit, like, nigga, fuck, and lil while like, love, peopl, song, and get were the top 5 terms in non-gang members' video data.

Related Work	Researchers have begun investigating the gang members' use of social media and have noticed the importance of identifying gang members' Twitter profiles a priori  ,  . Before analyzing any textual context retrieved from their social media posts, knowing that a post has originated from a gang member could help systems to better understand the message conveyed by that post. Wijeratne et al. developed a framework to analyze what gang members post on social media  . Their framework could only extract social media posts from self identified gang members by searching for pre-identified gang names in a user's Twitter profile description. Patton et al. developed a method to collect tweets from a group of gang members operating in Detroit, MI  . However, their approach required the gang members' Twitter profile names to be known beforehand, and data collection was localized to a single city in the country. These studies investigated a small set of manually curated gang member profiles, often from a small geographic area that may bias their findings. In our previous work  , we curated what may be the largest set of gang member profiles to study how gang member Twitter profiles can be automatically identified based on the content they share online.

This is the mean of the word embedding vectors of all words found in a Twitter profile: INLINEFORM1  Sum of word embeddings weighted by term frequency INLINEFORM0 . This is each word embedding vector multiplied by the word's frequency for the Twitter profile: INLINEFORM1  where INLINEFORM0 is the term frequency for the INLINEFORM1 word in profile INLINEFORM2 . Sum of word embeddings weighted by INLINEFORM0 - INLINEFORM1 INLINEFORM2 . This is each word vector multiplied by the word's INLINEFORM3 - INLINEFORM4 for the Twitter profile: INLINEFORM5  where INLINEFORM0 is the INLINEFORM1 - INLINEFORM2 value for the INLINEFORM3 word in profile INLINEFORM4 . Mean of word embeddings weighted by term frequency INLINEFORM0 . This is the mean of the word embedding vectors weighted by term frequency: INLINEFORM1

Question: Which supervised learning algorithms are used in the experiments?
Answer: Logistic Regression (LR), Random Forest (RF), Support Vector Machines (SVM)