Venue: NAACL 2022

# CTM - A Model for Large-Scale Multi-View Tweet Topic Classification

Vivek Kulkarni

Twitter Cortex

vkulkarni@twitter.com&Kenny Leung

Twitter Cortex

kennyleung@twitter.comAria Haghighi

Twitter Cortex

ahaghighi@twitter.com

###### Abstract

Automatically associating social media posts with topics is an important prerequisite for effective search and recommendation on many social media platforms. However, topic classification of such posts is quite challenging because of (a) a large topic space (b) short text with weak topical cues and (c) multiple topic associations per post. In contrast to most prior work which only focuses on post classification into a small number of topics (\(10\)-\(20\)), we consider the task of large-scale topic classification in the context of Twitter where the topic space is \(10\) times larger with potentially multiple topic associations per Tweet. We address the challenges above by proposing a novel neural model, CTM that (a) supports a large topic space of \(300\) topics and (b) takes a holistic approach to tweet content modeling - leveraging multi-modal content, author context, and deeper semantic cues in the Tweet. Our method offers an effective way to classify Tweets into topics at scale by yielding superior performance to other approaches (a relative lift of **20%** in median average precision score) and has been successfully deployed in production at Twitter.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

On many social media platforms like Twitter, users find posts that they are interested in through two mechanisms: (a) search and (b) recommendation. Both mechanisms typically use the topics associated with posts to identify potential candidates that are displayed to the user. Therefore, automatically associating a post with topics is important for effective search and recommendation. Furthermore, due to the diverse nature of social media content, for such topic association to be useful in practice, it is important to (a) support classification into a large number of topics (potentially hundreds or thousands of topics) and (b) allow for a post to have multiple topics or no topic at all.

Traditionally, there has been a long line of work on classifying documents (like news articles, movie reviews etc.) into topics [46; 79; 88; 10]. Additionally, there have been attempts to leverage known label hierarchy to perform hierarchical classification of documents. Most of these approaches learn a model per node of the hierarchy with potentially some form of hierarchy-based regularization in-order to assign labels to a document at each level in the label taxonomy [108; 119; 120]. With the rise of social media platforms, researchers noted that classification of social media content poses several unique challenges [106]. First, such posts can be very short and noisy with very weak cues provided by the linguistic context [88]. Second, content may be multi-modal with associated images, videos, and hyperlinks. Approaches for classifying documents tend to ignore this multi-modal nature [106]. Several works do explore classification of social media posts (like Tweets) [107; 108; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 172; 173; 174; 175; 176; 177; 178; 179; 180; 181; 182; 183; 184; 185; 186; 187; 188; 189; 190; 191; 192; 193; 194; 195; 196; 197; 198; 199; 200; 201; 202; 203; 204; 205; 206; 207; 208; 209; 210; 211; 212; 213; 214; 215; 216; 217; 218; 219; 223; 224; 225; 226; 227; 228; 229; 230; 231; 232; 233; 234; 235; 236; 237; 238; 239; 240; 241; 242; 243; 244; 245; 246; 247; 248; 249; 250; 251; 252; 253; 254; 255; 256; 257; 258; 259; 260; 261; 262; 263; 264; 265; 266; 267; 268; 269; 270; 271; 272; 273; 274; 275; 276; 277; 278; 279; 280; 281; 282; 283; 284; 285; 286; 287; 288; 289; 290; 291; 292; 293; 294; 295; 296; 297; 298; 299; 300; 301; 302; 303; 304; 305; 306; 307; 308; 309; 310; 311; 312; 313; 314; 315; 316; 317; 318; 319; 320; 321; 322; 323; 324; 325; 326; 327; 328; 329; 333; 334; 335; 336; 337; 338; 339; 340; 341; 342; 343; 344; 345; 346; 347; 348; 349; 350; 351; 352; 353; 354; 355; 356; 357; 358; 359; 360; 361; 362; 363; 364; 365; 366; 367; 368; 369; 370; 371; 372; 373; 374; 375; 376; 377; 378; 379; 380; 381; 382; 383; 384; 385; 386; 387; 388; 388; 389; 390; 388; 389; 391; 388; 389; 392; 393; 394; 395; 396; 397; 398; 399; 400; 401; 402; 403; 404; 405; 406; 407; 408; 409; 411; 412; 413; 414; 415; 416; 417; 418; 419; 420; 421; 422; 433; 444; 451; 423; 452; 453; 454; 46; 461; 462; 463; 464; 474; 475; 476; 477; 478; 479; 480; 481; 482; 483; 484; 485; 486; 487; 488; 489; 490; 491; 492; 493; 494; 495; 496; 497; 498; 499; 500; 501; 502; 503; 504; 505; 506; 507; 508; 511; 512; 513; 514; 515; 516; 517; 518; 529; 530; 519; 540; 513; 528; 541; 531; 542; 543; 555; 56; 571; 572; 573; 58; 58; 593; 594; 514; 544; 545; 595; 515; 529; 546; 547; 558; 596; 560; 517; 581; 597; 598; 599; 618; 620; 631; 648; 651; 66; 67; 68; 690; 649; 652; 66; 691; 66; 692; 67; 693; 68; 694; 695; 70; 696; 71; 72; 73; 74; 75; 76; 77; 78; 79; 80; 81; 82; 83; 84; 85; 86; 87; 88; 89; 90; 82; 84; 86; 88; 89; 91; 85; 87; 89; 92; 88; 89; 93; 94; 89; 95; 96; 97; 98; 99; 99; 100; 99; 111; 112; 113; 114; 115; 116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132; 133; 133; 134; 135; 136; 137; 138; 139; 140; 141; 142; 143; 144; 145; 146; 147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159; 160; 161; 162; 163; 164; 165; 166; 167; 168; 169; 170; 171; 173; 174; 175; 176; 177; 178; 179; 180; 181; 183; 184; 185; 186; 187; 188; 189; 190; 188; 191; 192; 189; 193; 194; 181; 187; 188; 189; 195; 196; 189; 197; 198; 199; 200; 210; 222; 233; 244; 256; 267; 27

author features, entity features thus moving beyond shallow Tweet text features and (c) supports multiple topics to be associated per Tweet. Our method offers an effective way to classify Tweets into topics at scale and is superior in performance to other approaches yielding a significant relative lift of \(\mathbf{20}\%\) in median average precision score. CTM has been successfully deployed at Twitter where online A/B experiments have also shown increased engagement and improved customer experience.

## 2 Related Work

Early works on Tweet classification used bag-of-words features constructed from Tweet text and classifiers like Rocchio classifiers, logistic regression, and support-vector machines Lee et al. (2011); Genc et al. (2011); Tao et al. (2012); Stavrianou et al. (2014); Selvaperumal and Suruliandi (2014). Follow-up work investigated using increasingly rich features for topic classification including graph-based features of term-co-occurrence graphs, hyperlink information, and distributed representations derived from deep learning models Cordobes et al. (2014); Kataria and Agarwal (2015); Li et al. (2016a, b, c, d); Ive et al. (2018); Kang et al. (2019); Gonzalez et al. (2021).

However, one notes at-least one of the following limitations in all of the above works: (a) focus on a very small number of topics (\(5-20\)) (b) do not support multiple topic labels per Tweet (c) do not consider or discuss how to model content beyond the raw Tweet text (d) do not capture label constraints. A sole exception to some of the above limitations is the work of Yang et al. (2014) which performs large-scale Tweet topic classification focusing on \(300\) topic labels in a real-time setting using only n-gram based features derived from the Tweet text, but ignores other cues. We revisit their large-scale setting after a decade and propose a vastly improved model for large-scale Tweet topic classification modeling Tweets holistically.

## 3 Data

Similar to Yang et al. (2014), we consider a set of \(300\) popular Twitter Topics 1. While Yang et al. (2014) construct data by only using weak labels obtained from a rule-based system using keyword matches, we employ both high precision human-labeled annotations and weakly-labeled data from a rule-based system using keyword matches 2 to construct the following datasets:

Footnote 1: We focus on only English Tweets. See the Appendix for the full list of topics considered.

* **Human Labeled Data (HCOMP Dataset)**: We closely follow the procedure outlined by Yang et al. (2014) which first samples Tweets based on topic priors to obtain Tweets that are weakly relevant to a topic, and then seeks label confirmation from trained human annotators. Specifically, we consider Tweets originating from users that are known to tweet mostly about a given topic (for example: Tweets authored by CNN are almost certainly about the "News" topic). We collect \(100\)K such Tweets with at-least \(200\) Tweets per topic. We then sought label confirmation from trained human annotators with each Tweet-topic pair being independently rated by \(3\) annotators and use a majority vote to determine the final labels (see Appendix for details). Finally, we create training, validation, and test splits of this dataset disjoint at both the Tweet and the user level.3

Footnote 2: See the Appendix for a brief description of this rule-based system for yielding weak labels.

Footnote 3: We do this because as we will see later, we use author level features in our model.
* **Weakly Labeled Data (WLD Dataset)**: We also construct a large-scale data-set of weakly labeled Tweets (**WLD** dataset) for task-specific pre-training (see Section 4). Specifically, we use the rule-based system to obtain a random sample of \(250\) million weakly labeled Tweets that is disjoint from the **HCOMP** dataset both in terms of time-span and Tweets.
* **Chatter Data (CHT Dataset)**: To ensure that our model does not incorrectly assign topics to what is termed "Twitter chatter" -Tweets that are largely about daily status updates, greetings and clearly non-topical content, we closely follow Yang et al. (2014) and construct a dataset of weakly labeled non-topical Tweets by sampling Tweets that trigger none of the topical rules in the rule-based system. We verify that a random sample (\(N=150\)) (denoted by **CHT-test**) are indeed non-topical through independent human annotators which we set-aside for model evaluation. The remaining portion also user-disjoint (\(N=100000\)) is used as training data.

## 4 Models and Methods

Problem Formulation.We formulate our problem as one of standard multi-label classification.

 

Formally, let \({\cal S}\) denote the given set of topics. Given \(\mathbf{X}\), a set of Tweet features and a set of topics \(\mathbf{L}\in 2^{\mathcal{S}}\), we seek to model \(\Pr(\mathbf{L}|\mathbf{X})\). We encode the topic labels \(\mathbf{L}\) as a binary vector \(\mathbf{Y}\) of length \(|{\cal S}|\) using multi-hot encoding. We consider a simple approach to multi-label classification4 - a neural architecture parameterized by \(\mathbf{\Theta}\) that outputs a vector \(\hat{\mathbf{Y}}\) of length \(|{\cal S}|\) where \(\hat{\mathbf{Y}_{i}}\in[0,1]\) is the probability of the Tweet belonging to topic \(i\).

Footnote 4: We largely consider a flat classification setting given the absence of well-defined, comprehensive and highly agreed-upon topic taxonomy for Twitter topics, and also because this formulation is better aligned with model deployment constraints.

Model Overview.CTM has three components:

* **Tweet Feature Encoder**: This component encodes features of the Tweet holistically. Specifically, it encodes the Tweet text, hyperlink features, named entity mention features, as well as features of associated media. This encoder outputs a vector of topic logits (one for each topic) based on these input features which we denote by \(\hat{\mathbf{Y}}^{t}\).
* \(\hat{\mathbf{Y}}^{c}\) which can be converted to probability scores using a sigmoid transformation.
* **Topic Constraint Model**: The topic constraint model encourages the predictions to reflect known constraints among the topic labels. For example, Tweets about "Soceer" are almost certainly also about "Sports" but very unlikely to also be about "Basketball". We encode such pre-specified label constraints in the output space via a factor-graph. Performing inference on the factor-graph re-calibrates the raw probabilities given by \(\hat{\mathbf{Y}}^{c}\) to better reflect the output label constraints yielding the final predicted probabilities for each topic \(\hat{\mathbf{Y}}^{f}\).

### Tweet Feature Encoder

The Tweet feature encoder is a standard Bert encoder with a linear classification head where all layers are trainable. Each individual Tweet feature is modeled as follows:

* **Tweet Text**: We simply pass the Tweet text as an input string to Bert after standard preprocessing (case-folding, stripping hyperlinks and user mentions).
* **Hyperlink Features**: For each hyperlink in the Tweet text, we obtain the raw HTML content of the web-page being referenced, and extract the web-page title and the first \(100\) characters of the web-page description. These features are simply concatenated with the Tweet text using a pre-defined separator token.
* **Media**: To incorporate topical cues from any attached media (images, gifs, and videos), we obtain media annotations for the given media. These media annotations are broad categories that summarize the content of the media. We then simply concatenate all of these media annotations to the current input string using a pre-defined token as a delimiter. The media annotations themselves are predicted by a media-annotations classifier that learns to assign each media to zero or more categories from a set of pre-defined categories. 5 Footnote 5: See the Appendix for details on the media categories classifier.
* **Entity Features**: Noting that mentions of named entities provide strong topical cues, we extract such mentions in the Tweet text using an off-the-shelf Twitter NER model Mishra et al. (2021) and link each extracted named entity to their entry in WikIData where available. We use the WikIdata descriptions of e

Figure 1: **Overview of our CTM model for large-scale topic classification of Tweets. Our model consists of \(3\) components: (a) A tweet feature encoder encoding Tweet features (b) an Author feature encoder encoding author features thus capturing author-topic affinity and (c) a constraint model that encourages the topic scores to respect prior constraints.**



tional inputs to the Tweet feature encoder. As an example, this enables CTM to infer that Tweets which mention "Steve Waugh" are likely about "Cricket".

Pretraining the Tweet Feature Encoder.Noting that the weights of the standard Bert encoder are not reflective of the domain of Tweets and may represent a poor initialization point during subsequent finetuning, we pretrain the Bert encoder on the task of predicting topics using the **WLD** dataset only using the raw Tweet text as the input feature. As we will show empirically, this large-scale pretraining improves generalization performance by better adapting the model to Twitter data.

### Author Feature Encoder

The author feature encoder is also identical to a standard Bert encoder with a linear classification head, with all layers being trainable. We use the following features of the author (all of which are simply concatenated together as input to Bert): (a) **Author Biography**: We use the self-reported publicly available author-profile description of the author posting the Tweet. (b) **Author Name**: We also use the author's display name. We hypothesize that all of these features may be indicative of the topics that the author likely tweets about. For example, an author name containing the string "FashionNews" strongly suggests that Tweets made by that author will likely be about Fashion.

### Topic Constraint Model

The topic constraint model encodes output label constraints in the topic prediction and captures correlations among topics. We encode such dependencies via a factor graph. Given a vector of topic predictions (probabilities) \(\mathbf{\hat{Y}^{c}}\), for each topic \(T_{i}\), we associate a discrete binary random variable with that topic \(v_{i}\), and a corresponding unary factor with potential function \(f_{i}\) such that \(f_{i}(0)=1.0-\mathbf{\tilde{Y_{i}}}^{c}\) and \(f_{i}(1)=\mathbf{\hat{Y_{i}}}^{c}\). For every constraint between a pair of topics \((i,j)\), we construct a binary factor with potential function \(\phi_{i,j}(v_{i},v_{j})\). This potential function encodes the compatibility between prediction scores for topic \(i\) and topic \(j\). Domain experts can craft their own potential functions to reflect positive or negative compatibility between topic pairs or alternatively even learn these from correlation data. CTM considers two types of constraints:

* **Broader Topic Inclusion**: If a Tweet is about a specific topic \(c\), then it is very likely that the Tweet is also about topic \(p\) where \(p\) subsumes topic \(c\). Other cases are a "don't-care". For example, if a Tweet is about "Basketball", it is almost certainly about "Sports". We use the following potential matrix6 for encoding this type of constraint: Footnote 6: The potential matrices are not necessarily unique and other equivalent matrices may exist.
* **Topic Pair Exclusion**: At-most one among topic \(a\) and \(b\) can be active at any time. For example, it is very unlikely to have a Tweet which is about both Cricket and Basketball. We use the following potential matrix for encoding this type of constraint: Footnote 7: See the Appendix for more details on this procedure.

After constructing a factor graph encoding the specified output constraints, we perform belief propagation8 on the factor graph to obtain the final marginal probabilities \(\mathbf{\hat{Y}^{f}}\) which reflect the encoded output constraints. In our experiments, we impose the above constraint types on specific topics falling under (and including) the broad topics of Sports, Music, Animation, Science, Animals, Anime & Manga.

Footnote 8: footnotemark:

## 5 Experiments

### Quantitative Evaluation

Baselines and Evaluation Setup.We consider two baselines: (a) A bag-of-words logistic regression (LR) model - our best-effort attempt to reproduce the decade old setup of Yang et al. (2014) and (b) a standard Bert model using only the Tweet text thus replacing logistic regression in (a) with a current state of the art deep-learning model. We train all models on the training data set using class weighted binary cross entropy loss, and evaluate them on the two held-out test sets:

* **HCOMP Test Set**: We evaluate model performance on the held out test split from the **HCOMP** dataset. We report the median average precision score over all topics. We consider the average precision score, since unlike the F1 score, it summarizes model performance over all operating thresholds.

 

* **CHT Test Set:** In order to measure the ability of our models to effectively reject assigning topics to "non-topical" Tweets (chatter), we evaluate our models on the held-out chatter test set. Here, we report the number of predictions made by the model over a given probability threshold (lower scores are better). We perform a systematic feature ablation study of our proposed CTM model to quantify the effect of feature sets considered. Table 1 shows the results of our evaluation where model suffixes represent different ablation settings. Note that our full model significantly outperforms the logistic regression and BERT baselines (**Median APS**: \(67.0\) vs \(54.8\)) and yields a relative improvement of \(20\%\) thus underscoring the effectiveness of our approach. We also make the following additional observations:

* [noitemsep]
* **Including non-topical tweets in training improves performance of rejecting chatter** Note that including non-topical tweets in the training data improved the performance of the BERT baseline on the **CHT** dataset (from \(254\) to \(135\) where lower is better).
* **Media features have a focused impact.** Adding media annotations overall does not affect the median average precision score significantly (compare row **CTM-A**: \(54.4\) to row above: \(54.8\)). However, we observe that many tweets in the evaluation may not contain media annotations. When we restricted our evaluation to only the tweets containing media, we observed a significant improvement where the corresponding average precision scores are \(71.0\) vs \(58.4\) respectively. By further computing per-topic performance improvement due to media annotations, we note that media features significantly boost the performance of Automotive, US national news, Anime, and Movies which indeed tend to be media rich, suggesting their focused impact.
* \(56.7\) vs \(54.4\)).
* \(57.2\) vs \(56.7\))**. However as with media features, when we restricted our evaluation to only those instances with hyperlinks we indeed observe a significant performance gain where the corresponding scores are \(92.67\) vs \(83.4\). Similar to our analysis of media features, a per-topic improvement analysis reveals that hyperlink features most improve the performance on Travel, Movies, Gaming, and US national news which tend to be hyperlink heavy.
* \(63.3\) vs \(57.2\))** thus reaffirming the importance of user-level modeling in NLP tasks.
* \(66.5\) vs \(63.3\))**. Drilling down, we noted that entity linking features most improve the performance on Rap, American football, K-pop, Entertainment News, and Cricket
- all topics whose Tweets are likely to mention sport players, movie stars, and musicians that are suggestive of the topic.
* especially violations of the broader topic inclusion constraint (see Table 2).8 Footnote 8: This slight degradation on CHT is due to error propagation of high confidence false positives which occurs to respect the constraints.

### Qualitative Evaluation

In addition to evaluating our CTM quantitatively, we also inspected the model predictions qualitatively to identify instances which (a) reveal the benefits of holistic tweet modeling and (b) highlight challenging cases. Table 3 shows a few instances that illustrate the benefit of holistically modeling Tweet content. Note that in "Power hitter joins #yellowstorm", only the attached media (which displays a cricket apparel) is indicative of the topic. Similarly, our model correctly predicts that "Re 

vealed: Australia's stars set to be pulled from IPL" is about "Cricket" by leveraging topical cues extracted from the linked website's content. Finally, CTM correctly infers that the Tweet referencing "Cody Ko and Noel Miller" is about "Digital Creators" by leveraging named entity cues. Finally, we also noted a few systematic failure modes (see Table 4). In particular, our model does not pick up on (a) metaphorical usage of topical words like "running" or "cheer-leading" (b) sarcasm and irony (c) NSFW senses of certain topical phrases (d) topical content in conversational threads since this requires modeling conversational context.

### Online Evaluation

Finally, we also evaluated CTM online by performing an A/B test comprising of \(25\) million users in each bucket. To summarize the results of the A/B test briefly, we observed that CTM relatively increased: (a) the size of the topic Tweet inventory online by about \(4\%\). This translates to about \(600\)K additional topical Tweets daily that could be surfaced to users based on their topical interests to improve their user experience. (b) precision by \(5\%\)

\begin{table}
\begin{tabular}{l l c}  & **Setting** & **Median APS \(\uparrow\)** & **CHT \(\downarrow\)** \\ \hline LR(baseline) Yang et al. (2014) & Tweet text (trained on only HCOMP) & 33.0 & 108 \\ BERT(baseline) & Tweet text (trained on only HCOMP) & 54.5 & 254 \\ BERT (baseline) & Tweet text (trained on HCOMP + CHT) & 54.8 & 135 \\ \hline CTM-A & Tweet text + media annotation (trained on HCOMP + CHT) & 54.4 & 121 \\ CTM-B & CTM-A + pretraining & 56.7 & 107 \\ CTM-C & CTM-B + Hyperlink features & 57.2 & 101 \\ CTM-D & CTM-C + Author features & 63.3 & 75 \\ CTM-E & CTM-D + Entity Linking features & 66.5 & 80 \\
**CTM-F** (Full model) & **CTM-E + Constraint model** & 67.0 & 90 \\ \end{tabular}
\end{table}
Table 1: **Performance of CTM on the test sets. The median APS is the median average precision on the HCOMP test set (_higher is better_, \(N=10000\)) where as CHT column shows the number of model predictions exceeding a probability score of \(0.9\) (noting robustness to other thresholds) on the CHT test set (_lower is better_). CTM significantly outperforms baselines and demonstrates the effectiveness of modeling content beyond the immediate Tweet text.**

\begin{table}
\begin{tabular}{p{113.8pt}|p{113.8pt} p{113.8pt}}
**Tweet Content** & **Predicted Label** & **Helpful feature** \\ \hline In times of trouble, regression models come to me, speaking words of wisdom Power hitter joins flyellowstorm att:Attached media of cricket bat and gloves Cameras in USC vs UT stopped working, so it is a podcast now Revealed: Australia’s stars set to be pulled from IPL URL to fox.sports domain & Data Science Cricket & Tweet text Media Annotations \\ codoy ko and noel miller are just ... & & **American Football** & Author Bio Hyperlink \\ \end{tabular}
\end{table}
Table 2: **Performance improvements due to the constraint model. The constraint model yields significant improvements on broader topics (as large as \(20\) points). Performance on narrower topics do not change significantly.**

\begin{table}
\begin{tabular}{p{113.8pt}|p{113.8pt}|p{113.8pt}}
**Tweet Content** & **Predicted Label** & **Error Reason** \\ \hline In life, you have not seen your best days, you have not run your best race ... Cheerleading the mob is not going to save ... I am going to have very large drink tonight not sure if whisky or cyanide I need my **** 

and (c) user engagement by \(5.5\%\). In a nut-shell, our online experiments suggested that CTM significantly improves the user experience of the Topics product surface in Twitter and has consequently been deployed in production.

## 6 Conclusion

We revisited the problem of large scale Tweet topic classification posed by Yang et al. (2014) and proposed a model for classifying Tweets into a large set of \(300\) topics with improved performance. In contrast to prior work we take a holistic approach to modeling Tweets and model not only the immediate Tweet text, but also associated media, hyperlinks, author context, entity mentions, and incorporate domain knowledge expressed as topic constraints in a principled manner. Our model showed significantly increased engagement and improved customer experience in several online A/B experiments, and it has been deployed into production at Twitter with millions of active users. Finally, while our model and approach has been restricted to Tweet classification, our proposed methods and observations may benefit other social media platforms seeking to classify content into a large number of topics effectively.

## Ethical Considerations

This paper and the data used within was reviewed as part of Twitter's standard privacy and legal review processes. No data has been publicly released in relation to this paper. While there is a possibility that the model could be misused, we do not anticipate any new or increased risks over those already present in established prior work and prior models on topic classification.

## Acknowledgments

We would like to acknowledge the contributions of Inom Mirzaev, Chenguang Yu, Laleh Soltan Ghoraie, Kovas Boguta, Jun Ng, Andy Aitken, Olivia Ifrim, Shubhanshu Mishra, Sneha Mehta, Aman Saini, Sijun He, Wayne Krug, and Ali Mollahosseini in supporting this work. We would also like to thank the anonymous reviewers for their comments and suggestions.

## References

* Adhikari et al. (2019) Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. Docbert: Bert for document classification. _arXiv preprint arXiv:1904.08398_.
* Balabanovic and Shoham (1995) Marko Balabanovic and Yoav Shoham. 1995. Learning information retrieval agents: Experiments with automated web browsing. In _On-line Working Notes of the AAAI Spring Symposium Series on Information Gathering from Distributed, Heterogeneous Environments_, pages 13-18.
* Baldwin et al. (2013) Timothy Baldwin, Paul Cook, Marco Lui, Andrew MacKinlay, and Li Wang. 2013. How noisy social media text, how diffrrt social media sources? In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_, pages 356-364.
* Bender and Friedman (2018) Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. _Transactions of the Association for Computational Linguistics_, 6:587-604.
* Borko and Bernick (1963) Harold Borko and Myrna Bernick. 1963. Automatic document classification. _Journal of the ACM (JACM)_, 10(2):151-162.
* Chang et al. (2015) Shuo Chang, Peng Dai, Jilin Chen, and Ed H Chi. 2015. Got many labels? deriving topic labels from multiple sources for social media posts using crowdsourcing and ensemble learning. In _Proceedings of the 24th International Conference on World Wide Web_, pages 397-406.
* Cordobes et al. (2014) Hector Cordobes, Antonio Fernandez Anta, Luis F Chiroque, Fernando Perez, Teofilio Redondo, and Agustin Santos. 2014. Graph-based techniques for topic classification of tweets in spanish. _International Journal of Interactive Multimedia and Artificial Intelligence_.
* Genc et al. (2011) Yegin Genc, Yasuaki Sakamoto, and Jeffrey V Nickerson. 2011. Discovering context: classifying tweets through a semantic transform based on wikipedia. In _International conference on foundations of augmented cognition_, pages 484-492. Springer.
* Gonzalez et al. (2021) Jose Angel Gonzalez, Liuis-F Hurtado, and Ferran Pla. 2021. Twilbert: Pre-trained deep bidirectional transformers for spanish twitter. _Neurocomputing_, 426:58-69.
* Gopal and Yang (2013) Siddharith Gopal and Yiming Yang. 2013. Recursive regularization for large-scale classification with hierarchical and graphical dependencies. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 257-265.
* Ive et al. (2018) Julia Ive, George Gkotsis, Rina Dutta, Robert Stewart, and Sumithra Velupillai. 2018. Hierarchical neural model with attention mechanisms for the classification of social media text related to mental health. In _Proceedings of the Fifth Workshop on Computational 

Linguistics and Clinical Psychology: From Keyboard to Clinic_, pages 69-77.
* Joachims (1998) Thorsten Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. In _European conference on machine learning_, pages 137-142. Springer.
* Kang et al. (2019) Jaeyong Kang, HongSeok Choi, and Hyunju Lee. 2019. Deep recurrent convolutional networks for inferring user interests from social media. _Journal of Intelligent Information Systems_, 52(1):191-209.
* Kataria and Agarwal (2015) Saurabh Kataria and Arvind Agarwal. 2015. Supervised topic models for microblog classification. In _2015 IEEE International Conference on Data Mining_, pages 793-798. IEEE.
* Koller and Sahami (1997) Daphne Koller and Mehran Sahami. 1997. Hierarchically classifying documents using very few words. Technical report, Stanford InfoLab.
* Lee et al. (2011) Kathy Lee, Diana Palsetia, Ramanathan Narayanan, Md Mostofa Ali Patwary, Ankit Agrawal, and Alok Choudhary. 2011. Twitter trending topic classification. In _2011 IEEE 11th International Conference on Data Mining Workshops_, pages 251-258. IEEE.
* Li et al. (2016a) Quanzhi Li, Sameena Shah, Mohammad Ghassemi, Rui Fang, Armineh Nourbakhsh, and Xiaomo Liu. 2016a. Using paraphrases to improve tweet classification: Comparing wordnet and word embedding approaches. In _2016 IEEE International Conference on Big Data (Big Data)_, pages 4014-4016. IEEE.
* Li et al. (2016b) Quanzhi Li, Sameena Shah, Xiaomo Liu, Armenheh Nourbakhsh, and Rui Fang. 2016b. Tweet topic classification using distributed language representations. In _2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI)_, pages 81-88. IEEE.
* Li et al. (2016c) Quanzhi Li, Sameena Shah, Xiaomo Liu, Armineh Nourbakhsh, and Rui Fang. 2016c. Tweetsift: Tweet topic classification based on entity knowledge base and topic enhanced word embedding. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_, pages 2429-2432.
* Li et al. (2016d) Quanzhi Li, Sameena Shah, Armineh Nourbakhsh, Xiaomo Liu, and Rui Fang. 2016d. Hashtag recommendation based on topic enhanced embedding, tweet entity data and learning to rank. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_, pages 2085-2088.
* Mishra et al. (2020) Shubhanshu Mishra, Sijun He, and Luca Belli. 2020. Assessing demographic bias in named entity recognition. _arXiv preprint arXiv:2008.03415_.
* Mishra et al. (2021) Shubhanshu Mishra et al. 2021. Improved multilingual language model pretraining for social media text via translation pair prediction. In _Proceedings of the Seventh Workshop on Noisy User-generated Text (WNUT 2021)_, pages 381-388, Online. Association for Computational Linguistics.
* Rojas et al. (2020) Kerry Rivas Rojas, Gina Bustamante, Arturo Oncevay, and Marco A Sobrevilla Cabezudo. 2020. Efficient strategies for hierarchical text classification: External knowledge and auxiliary tasks. _arXiv preprint arXiv:2005.02473_.
* Sandler et al. (2018) Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 4510-4520.
* Selvaperumal and Suruliandi (2014) P Selvaperumal and A Suruliandi. 2014. A short message classification algorithm for tweet classification. In _2014 International Conference on Recent Trends in Information Technology_, pages 1-3. IEEE.
* Stavianou et al. (2014) Anna Stavianou, Caroline Brun, Tomil Silander, and Claude Roux. 2014. Nlp-based feature extraction for automated tweet classification. _Interactions between Data Mining and Natural Language Processing_, 145.
* Tao et al. (2012) Ke Tao, Fabian Abel, Claudia Hauff, and Geert-Jan Houben. 2012. What makes a tweet relevant for a topic? ln # MSM, pages 49-56. Citeseer.
* Tsutsumi et al. (2007) Kimitaka Tsutsumi, Kazutzka Shimada, and Tsutomu Endo. 2007. Movie review classification based on a multiple classifier. In _Proceedings of the 21st pacific Asia conference on language, information and computation_, pages 481-488.
* Yang et al. (2014) Shuang-Hong Yang, Alek Kolcz, Andy Schlaikjer, and Pankaj Gupta. 2014. Large-scale high-precision topic modeling on twitter. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1907-1916.

 

Appendix

### Details Regarding Off the Shelf Components Used in CTM

#### a.1.1 Media Annotations Classifier

The media annotations classifier takes as input an image and classifies the image into one or more of \(45\) media categories listed in Table 5. The classifier is essentially a standard MobileNet V2 model Sandler et al. (2018) further fine-tuned on a human-labeled curated dataset of \(100\)K images from Twitter. The operating threshold of the media classifier is set to achieve a precision of about \(90\%\) on each topic.9

Footnote 9: For videos, and GIF’s each frame is analyzed by the model with the prediction scores being aggregated using the max operator.

#### a.1.2 Twitter Named Entity Recognizer

The Twitter NER model is a standard bi-directional LSTM with a CRF layer and detects mentions of persons, places, organizations, and products in a Tweet. The model has been trained on \(100\)K human annotated labeled tweets Mishra et al. (2020) and has a precision of \(85\%\) with a recall of \(70\%\) on a held-out test set. We link the extracted mention to a potential WikiData candidate as follows: (a) we first construct a set of potential WikiData entity candidates - the set of all entities whose label or alias has a match with the extracted mention (b) link the mention to the top entity candidate obtained by sorting the candidate set in descending order of page view count as the primary key breaking ties using page rank as the secondary key. We use this approach as an expedient choice noting that more sophisticated entity linking approaches can be used.

#### a.1.3 Rule Based System for Generating Weakly Labeled Examples.

We employ a rule-based system consisting of tens of thousands of rules based on key-words to generate weakly labeled examples. All rules are manually curated and added by domain experts and data specialists.

### Hyper-parameter Tuning

As is standard practice, we use the validation set (\(N=10000\)) to perform hyper-parameter tuning. We explored several hyper-parameter settings for the baseline models namely Logistic Regression and BERT to make baseline comparisons strong and compare CTM against only the best performing baseline settings. In particular, we explored training for different epochs \((1-10)\) for the BERT baseline. For the logistic regression baseline, we also tried various settings for the maximum number of iterations of the optimizer (\(100-1000\)) as well as various values for the strength of the L\(2\) regularizer (\(C=[0,1,10,100]\)).

For our proposed model CTM, we did not do any specific hyper-parameter tuning and just trained all models for \(5\) epochs using 1 A\(100\) GPU.

### Details on the Human Labeled Annotation Task

In this section, we briefly describe the human annotation task used for obtaining topic label confirmation used in the construction of the **HCOMP** dataset. Each annotator is shown a Tweet, topic pair and asked to judge whether the topic is relevant to the Tweet or not. The instructions are:

``` Tasks: In this task, you will be shown a tweet and a topic and asked whether the tweet is 'relevant' for a topic. Topics:You will be asked to determine if a tweet is relevant for a given topic. A "Topic" is a potential subject of conversation that can be identified with a commonly held definition, where mass interest in the subject is not likely to be temporary, e.g. 'Comedy' or 'Knitting' is a topic as it is non-subjective and has a commonly held definition. Purely social tweets like "are you doing okay?" or personal remarks like "I'm having a bad day" are not topical. A Tweet can be popular without being topical. Question: The primary question you will be asked is "Is this tweet about a topic?", the possible responses are: Yes - This tweet is primarily about this topic. Somewhat - This tweet is related to this topic, but it is not a primary topic of this tweet. No - This tweet is unrelated to this topic. Unsur - I don't understand this tweet. Guidelines: You will first want to make sure you understand the presented topic. If you are unfamiliar with the topic presented in this question, please click on the topic which will take you to a Google search result page. Feel free to click on a few links (news articles or a Wikipedia page) to familiarize yourself with the topic. When elements of the tweet can I use to make a judgment? It can sometimes be challenging to tell what a tweet is about from tweet text alone. In order to determine what the tweet is about you may need to do the following: Look at replies of a tweet, which might provide additional context by clicking on the tweet. (NOTE: If you can understand the tweet by relying just on the body or author of the tweet, it is fine to not designate replies as being used to make a judgment.) Google phrases in the tweet text if you are unfamiliar with a mentioned entry or phrase that will help you understand the tweet. Look at the image, video, or click on any link (including a hashtag) associated with the Tweet, since it may be commenting on this media. If the media is primarily about the topic, the tweet is as well. Look at the tweet author's name, profile, public timeline, or linked website if it helps disambiguate tweet content. (NOTE: Please don't use the author alone in making determination, without some other element of the tweet.) 

Each HIT was judged by \(3\) independent highly reliable annotators. Finally, we noted that two-way (majority) agreement rate was \(86\%\), unanimous agreement was \(66\%\) and the topic precision overall was \(70\%\) (with "somewhat" ratings being counted towards a precision error).

### Data Statement

Here, we outline other aspects of our data as per recommendations outlined in Bender and Friedman [2].

Summary - We collect a set of Tweet, topic pairs focusing on only English Tweets which we use for predictive modeling and evaluation.

Curation Rationale - The rationale for the setup used in data collection was primarily driven by our task (large scale topic classification) and the need for data to a build a predictive model. The size of the data collected was thus influenced by task, available budget, and time available.

Language variety - The tweets were restricted to English only and are from the time range between September 2020 and May 2021. More fine-grained information is not available.

Speaker demographic - We do not have any demographic information of the users in this data. One would expect the demographic information to be similar to the demographics of Twitter users around the time of data collection.

Annotator demographic - Human Annotators are primarily native English speakers. No other information is available.

Text Characteristics - Tweets are short, informal and have at-most \(280\) characters. Tweets are generally meant to be engaged with by other Twitter users.

### Details on Belief Propagation

In this section, we provide more details on the procedure of belief propagation used in the topic constraint model component. In belief propagation, messages are alternately passed between variable nodes and factor nodes (until convergence is achieved or a finite number of iterations is completed). A message is simply a vector \(\mu\) where the individual components denote the probability of the random variable taking a specific value \(x\in\{0,1\}\). The message from a variable \(v\) to neighboring factor \(f\) on taking a specific value \(x\) is given by the following equation:

\[\mu_{v\longrightarrow f}(x)\propto\prod_{g\in\mathcal{N}(v)\backslash f}\mu_{ g\longrightarrow v}(x) \tag{1}\]

, where \(g\) belongs to the set of factor nodes connected to \(v\) excluding \(f\). Similarly, the message from a factor node \(f\) to the variable \(v\) on the variable taking a specific value \(x\) is given by the following:

\[\mu_{f\longrightarrow v}(x)\propto\sum_{\mathbf{x}:\mathbf{x}_{u}=x}\phi( \mathbf{x})\prod_{u\in\mathcal{N}(f)\backslash v}\mu_{u\longrightarrow f}( \mathbf{x}_{u}) \tag{2}\]

, where \(u\) belongs to the set of variable nodes connected to \(f\) excluding \(v\).

Finally, after convergence (or a finite number of iterations), the updated marginal probability of variable \(v\) taking on a value \(x\) is given by \(\Pr(v=x)\propto\prod_{g\in\mathcal{N}(v)}\mu_{g\longrightarrow v}(x)\).

 

* [25] M. C. Bader, A. K. Goyal, and R. H. Snoek. The role of the model for predicting the age of a person's age distribution. In _Proceedings of the 10th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 349-368, 2007.
* [26] J. D. Lee, Y. Zhang, and L. Zisserman. Predicting age distributions in a social network. In _Proceedings of the 1st International Conference on Machine Learning_, volume 1, pages 1-10, 2007.
* [27] T. Lin, X. Liu, and W. Wang. An efficient approach to predict age distribution using a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2007.
* [28] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 1-10, 2007.
* [29] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [30] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [31] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [32] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [33] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [34] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [35] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [36] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [37] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [38] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [39] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [40] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [41] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [42] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [43] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [44] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [45] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [46] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [47] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [48] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [49] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [50] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [51] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [52] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [53] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [54] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [55] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [56] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [57] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [58] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [59] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [60] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [61] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [62] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [63] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [64] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [65] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [66] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [67] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [68] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2007.
* [69] P. Li, F. Chen, and L. Zisserman. Predicting age distribution by an effective model for predicting age distribution via a deep learning framework. In _Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining_, pages 1-10, 2 



\begin{table}
\begin{tabular}{l l l l} \hline
2D animation & Country music & Horns & Rock climbing \\
3D animation & Cricket & Hotels & Reduce \\ Accounting & Cruise travel & Houston & Rollplaying games \\ Action and adventure films & Cuttle classes & Independent films & Romance books \\ Adventure travel & Cutting & Index rock & Rowing \\ Advretiative & Cybencutivity & Information security & Rugby \\ Agriculture & Cycling & Interior design & Running \\ Air travel & Dance & Internet of things & Sailing \\ Alternative rock & Darts & Investigne & Saxophone \\ American football & Data science & J-pop & Sci-fi and fantasy films \\ Animals & Databases & Jazz & Sci-fi and fantasy films \\ Aulimated films & Dating & Jewely & Science \\ Animation & Digital creator & Job searching and networking & Science news \\ Automation software & Documentary films & Judo & Screenwriting \\ Anime & Dogs & K-hp hop & Sculpting \\ Anime \& nanga & Dreams flints & K-cop & Sharks \\ Antiques & Drawing and illustration & Kavian & Shoes \\ Archaeology & Drumus & Knitting & Showping \\ Archuncture & EDM & Lactrose & Stateboarding \\ Art & Economics & Language learning & Slasing \\ Artificial intelligence & Education & Latin pop & Skin care \\ Artik culture & Electronic music & MMA & Small business \\ Arts \& culture neas & Entertainment & Makeup & Snakers \\ Arts and crafts & Entertainment news & Murine life & Snowdar \\ Attrophy & Environmentalism & Marketing & Soap operats \\ Astronants & Egorusta & Martial arts & Soccer \\ Athletic: apparel & Europe travel & Mathematics & Soccer stats \\ Augmented reality & Everyday style & Men’y booking & Soccer transfers \\ Australian rules football & Experimental music & Men’s gift & Soft rock \\ Auto racing & Fanous quotes & Men’s style & Sofitual \\ Automotive & Fantany baseball & Motorcycle tracking & Space \\ Aviation & Fantany basketball & Motorcycles & Sporting goods \\ Backpacking & Fantany football & Movie news & Sports \\ Baltimore & Fantany sports & Movies & Sports grows \\ Ballite & Fashion & Movies \& TV & Sports stats \\ Baseball & Faúhion and beauty & Museums & Startups \\ Basketball & Fraiboin business & Music & Storyboarding \\ Bountry & Faith magazines & Music feativls & Street art \\ Bigographies and memoirs & Fabian models & Music industry & Sterewear \\ Biology & Fast food & Music news & Supernarenal \\ Biotech and biomedical & Friction & Music production & Surfiting \\ Birdwitching & Pabling games & Musical & Suimming \\ Black Lives Matter & Figure skating & Mystery and crime books & Table tennis \\ Blues music & Financial services & National parks & Takeltop gaming \\ Board games & Fittech & Nature & Telephone eye-playing games \\ Hollywood dance & Fishing & Nature photography & Tancos \\ Hollywood films & Fitness & Nskall & Tech news \\ Hollywood music & Folk music & Norprofits & Technology \\ Hollywood news & Food & Olympics & Television \\ Books & Food inspiration & Online education & Teonis \\ Bowring & Futrology & Open store & Theater \\ Booring & Game development & Opera & Theme parts \\ Brazilian funk & Gaming & Organic & Thriller films \\ Business \& finance & Gaming news & Organic foods & Track \& field \\ Business media & Gardening & Outdoor aprnet & Trading card games \\ Business news & Geneology & Outdoors & Traditional games \\ Business personalities & Geography & Painting & Travel \\ C-rop & Geology & Parenting & Travel guides \\ Careers & Golf & Pets & Travel leaves \\ Cartons & Graduate school & Philosophy & Thiahloni \\ Cats & Grammy Awards & Photography & Us national news \\ Cheredeling & Graphic design & Physics & Vegamism \\ Chemistry & Guitar & Packets \& radio & Vegetantiansim \\ Chess & Gymnastics & Peter & Venturte capital \\ Classic rock & Har care & Post & Video games \\ Classical music & Hallocene films & Pep Pink & Visual arts \\ Cloud computing & Handpass & Pore rack & Valleball \\ Cloud platforms & Hard rock & Progressive rock & Watches \\ College life & Health areas & Psychology & Weather \\ Combut sports & Heavy metal & Punish music & Web development \\ Comedy & Historical fiction & Funk & Wedings \\ Comedy films & History & ReB and soul & Weight training \\ Comics & Hockey & Rap & Women’s bashing \\ Computer programming & Home \& family & Reality TV & Women’s golf \\ Concept Art & Home improvement & Reggae & Women’s gymnatics \\ Contraction & Microscope & Registration & World news \\ Cooking & Horner films & Road styles & Westline \\ Cosplay & Hover rising and cytok & Rock & Yogg \\ \hline \end{tabular}
\end{table}
Table 6: List of topics making up our topic space.