Venue: NAACL 2019

# Microblog Hashtag Generation via Encoding Conversation Contexts

 Yue Wang  Jing Li\({}^{2}\)1 Irwin King\({}^{1}\) Michael R. Lyu\({}^{1}\) Shuming Shi\({}^{2}\)

Footnote 1: This work was mainly done when Yue Wang was an intern at Tencent AI Lab.

###### Abstract

Automatic hashtag annotation plays an important role in content understanding for microblog posts. To date, progress made in this field has been restricted to phrase selection from limited candidates, or word-level hashtag discovery using topic models. Different from previous work considering hashtags to be inseparable, our work is the first effort to annotate hashtags with a novel _sequence generation_ framework via viewing the hashtag as a short sequence of words. Moreover, to address the data sparsity issue in processing short microblog posts, we propose to jointly model the target posts and the _conversation contexts_ initiated by them with bidirectional attention. Extensive experimental results on two large-scale datasets, newly collected from English Twitter and Chinese Weibo, show that our model significantly outperforms state-of-the-art models based on classification.1 Further studies demonstrate our ability to effectively generate rare and even unseen hashtags, which is however not possible for most existing methods.

Footnote 1: To obtain our datasets, please contact Yue Wang and Jing Li.

+
Footnote †: The _Proceedings of NAACL-HLT 2019_, pages 1624–1633

Mineapolis, Minnesota, June 2 - November 7, 2021

## 1 Introduction

Microblogs have become an essential outlet for individuals to voice opinions and exchange information. Millions of user-generated messages are produced every day, far outpacing the human being's reading and understanding capacity. As a result, the current decade has witnessed the increasing demand for effectively discovering gist information from large microblog texts. To identify the key content of a microblog post, hashtags, user-generated labels prefixed with a "#" (such as "#NAACL" and "#DeepLearning"), have been widely used to reflect keyphrases Zhang et al. (2016, 2018) or topics Yan et al. (2013); Hong et al. (2012); Li et al. (2016). Hashtags can further benefit downstream applications, such as microblog search Efron (2010); Bansal et al. (2015), summarization Zhang et al. (2013); Chang et al. (2013), sentiment analysis Davidov et al. (2010); Wang et al. (2011), and so forth. Despite the widespread use of hashtags, there are a large number of microblog messages without any user-provided hashtags. For example, less than \(15\)% tweets contain at least one hashtag Wang et al. (2011); Khabiri et al. (2012). Consequently, for a multitude of posts without human-annotated hashtags, there exists a pressing need for automating the hashtag annotation process for them.

Most previous work in this field focuses on extracting phrases from target posts Zhang et al. (2016, 2018) or selecting candidates from a pre

\begin{table}
\begin{tabular}{|l|} \hline
**Target post for hashtag generation** \\ \hline This _Azarenka_ woman needs a talking to from the \\ umpire her werid noises are tones inapproposes professionally. _#AnsOpen_ \\ \hline \multicolumn{2}{|c|}{**Replying messages forming a conversation**} \\ \hline
**[T1]** How annoying is she. I just worked out what \\ she sounds like one of those turbo charged cars \\ when they change gear or speed. \\
**[T2]** On the topic of noises, I was at the _Nadal-Tomic_ game last night and I loved how quiet \\ _Tomic_ was compared to _Nadal_. \\
**[T3]** He seems to have a shiftoad of talent and the postnatal press conf. He showed a lot of maturity \\ and he seems nice. \\
**[T4]**_Tomic_ has a fantastic _tennis_ brain... \\ \hline \end{tabular}
\end{table}
Table 1: A post and its conversation snippet about “Australian Open” on Twitter. “_#AusOpen_” is the human-annotated hashtag for the target post. _Words indicative of the hashtag_ are in blue and italic type.



defined list (Gong and Zhang, 2016; Huang et al., 2016; Zhang et al., 2017). However, hashtags usually appear in neither the target posts nor the given candidate list. The reasons are two folds. For one thing, microblogs allow large freedom for users to write whatever hashtags they like. For another, due to the wide range and rapid change of social media topics, a vast variety of hashtags can be daily created, making it impossible to be covered by a fixed candidate list. Prior research from another line employs topic models to generate topic words as hashtags (Gong et al., 2015; Zhang et al., 2016). These methods, ascribed to the limitation of most topic models, are nevertheless incapable of producing phrase-level hashtags.

In this paper, we approach hashtag annotation from a novel _sequence generation_ framework. In doing so, we enable phrase-level hashtags beyond the target posts or the given candidates to be created. Here, hashtags are first considered as a sequence of tokens (e.g., "_#DeepLearning_" as "_deep learning_"). Then, built upon the success of sequence to sequence (seq2seq) model on language generation (Sutskever et al., 2014), we present a neural seq2seq model to generate hashtags in a _word-by-word_ manner. To the best of our knowledge, _we are the first to deal with hashtag annotation in sequence generation architecture_.

In processing microblog posts, one major challenge we might face is the limited features to be encoded. It is mostly caused by the data sparsity exhibited in short and informal microblog posts.2 To illustrate such challenge, Table 1 displays a sample Twitter post tagged with "_#Au-sOpen_", referring to Australian Open tennis tournament. Only given the short post, it is difficult to understand why it is tagged with "_#AusOpen_", not to mention that neither "_aus_" nor "_open_" appear in the target post. In such a situation, how shall we generate hashtags for a post with limited words?

Footnote 2: For instance, the eligible length of a post on Twitter or Weibo is up to 140 characters.

To address the data sparsity challenge, we exploit conversations initiated by the target posts to enrich their contexts. Our approach is benefited from the nature that most messages in a conversation tend to focus on relevant topics. Content in conversations might hence provide contexts facilitating the understanding of the original post (Chang et al., 2013; Li et al., 2015). The effects of conversation contexts, useful on topic modeling (Li et al., 2016, 2018) and keyphrase extraction (Zhang et al., 2018), have never been explored on microblog hashtag generation. To show why conversation contexts are useful, we display in Table 1 a conversation snippet formed by some replies of the sample target post. As can be seen, key content words in the conversation (e.g., "_Nadal_", "_Tomic_", and "_tennis_") are useful to reflect the relevance of the target post to the hashtag "_#AnsOpen_", because Nadal and Tomic are both professional tennis players. Concretely, our model employs a dual encoder (i.e., two encoders), one for the target post and the other for the conversation context, to capture the representations from the _two sources_. Furthermore, to capture their joint effects, we employ the bidirectional attention (**bi-attention**) (Seo et al., 2016) to explore the interactions between two encoders' outputs. Afterward, an attentive decoder is applied to generate the word sequence of the hashtag.

In experiments, we construct two large-scale datasets, one from English platform Twitter and the other from Chinese Weibo. Experimental results based on both information retrieval and text summarization metrics show that our model generates hashtags closer to human-annotated ones than all the comparison models. For example, our model achieves \(45.03\)% ROUGE-1 F1 on Weibo, compared to \(25.34\)% given by the state-of-the-art classification-based method. Further comparisons with classification-based models show that our model, in a sequence generation framework, can better produce rare and even new hashtags.

To summarize, our contributions are three-fold:

\(\bullet\) We are the first to approach microblog hashtag annotation with _sequence generation_ architecture.

\(\bullet\) To alleviate data sparsity, we enrich context for short target posts with their _conversations_ and employ a bi-attention mechanism for capturing their interactions.

\(\bullet\) Our proposed model outperforms state-of-the-art models by large margins on two large-scale datasets, constructed as part of this work.

## 2 Neural Hashtag Generation Model

In this section, we describe our framework shown in Figure 1. There are two major modules: a dual encoder to encode both target posts and their conversations with a bi-attention to explore their interactions, and a decoder to generate hashtags 

Input and Output.Formally, given a target post \({\bf x}^{p}\) formulated as word sequence \(\langle x_{1}^{p},x_{2}^{p},...,x_{|{\bf x}^{p}|}^{p}\rangle\) and its conversation context \({\bf x}^{c}\) formulated as word sequence \(\langle x_{1}^{c},x_{2}^{c},...,x_{|{\bf x}^{c}|}^{c}\rangle\), where \(|{\bf x}^{p}|\) and \(|{\bf x}^{c}|\) denote the number of words in the input target post and its conversation, respectively, our goal is to output a hashtag \({\bf y}\) represented by a word sequence \(\langle y_{1},y_{2},...,y_{|{\bf y}|}\rangle\). For training instances tagged with multiple gold-standard hashtags, we copy the instances multiple times, each with one gold-standard hashtag following Meng et al. (2017). All the input target posts, their conversations, and the hashtags share the same vocabulary \(V\).

Dual Encoder.To capture representations from both target posts and conversation contexts, we design a dual encoder, composed of a post encoder and a conversation encoder, each taking the \({\bf x}^{p}\) and \({\bf x}^{c}\) as input, respectively.

For the post encoder, we use a bidirectional gated recurrent unit (Bi-GRU) Cho et al. (2014) to encode the target post \({\bf x}^{p}\), where its embeddings \(e({\bf x}^{p})\) are mapped into hidden states \({\bf h}^{p}=\langle{\bf h}_{1}^{p},{\bf h}_{2}^{p},...,{\bf h}_{|{\bf x}^{p}|}^ {p}\rangle\). Specifically, \({\bf h}_{i}^{p}=[\overrightarrow{{\bf h}_{i}^{p}};\overleftarrow{{\bf h}_{ i}^{p}}]\) is the concatenation of forward hidden state \(\overrightarrow{{\bf h}_{i}^{p}}\) and backward hidden state \(\overleftarrow{{\bf h}_{i}^{p}}\) for the \(i\)-th token:

\[\overrightarrow{{\bf h}_{i}^{p}}=GRU(e({\bf x}_{i}^{p}),\overrightarrow{{\bf h }_{i-1}^{p}}), \tag{1}\]

\[\overleftarrow{{\bf h}_{i}^{p}}=GRU(e({\bf x}_{i}^{p}),\overleftarrow{{\bf h }_{i+1}^{p}}). \tag{2}\]

Likewise, the conversation encoder converts conversations into hidden states \({\bf h}^{c}\) via another Bi-GRU. The dimensions of both \({\bf h}^{p}\) and \({\bf h}^{c}\) are \(d\).

Bi-attention.To further distill useful representations from our two encoders, we employ the bi-attention to explore the interactions between the target posts and their conversations. The adoption of bi-attention is inspired by Seo et al. (2016), where the bi-attention was applied to extract query-aware contexts for machine comprehension. Our intuition is that the content concerning the key points in target posts might have their relevant words frequently appearing in their conversation contexts, and vice versa. In general, such content can reflect what the target posts focus on and hence effectively indicate what hashtags should be generated. For instance, in Table 1, names of tennis players (e.g., "_Azarenka_", "_Nadal_", and "_Tomic_") are mentioned many times in both target posts and their conversations, which reveals why the hashtag is "#_AusOpen_".

To this end, we first put a _post-aware_ attention on the conversation encoder with coefficients:

\[\alpha_{ij}^{c}=\frac{\exp(f_{score}({\bf h}_{i}^{p},{\bf h}_{j}^{c}))}{\sum_{ j^{\prime}=1}^{|{\bf x}^{c}|}\exp(f_{score}({\bf h}_{i}^{p},{\bf h}_{j^{\prime}}^ {c}))}, \tag{3}\]

where the alignment score function \(f_{score}({\bf h}_{i}^{p},{\bf h}_{j}^{c})={\bf h}_{i}^{p}{\bf W}_{bi-att}{\bf h }_{j}^{c}\) captures the similarity of the \(i\)-th word in the target post and the \(j\)-th word in its conversation. Here \({\bf W}_{bi-att}\in\mathbb{R}^{d\times d}\) is a weight matrix to be learned. Then, we compute a context vector \({\bf r}^{c}\) conveying post-aware conversation representations, where the \(i\)-th value is defined as:

\[{\bf r}_{i}^{c}=\sum_{j=1}^{|{\bf x}^{c}|}\alpha_{ij}^{c}{\bf h}_{j}^{c}. \tag{4}\]

Analogously, a _conversation-aware_ attention on post encoder is used to capture the conversation-aware post representations as \({\bf r}^{p}\).

Merge Layer.Next, to further fuse representations distilled by the bi-attention on each encoder, we design a _merge_ layer, a multilayer perceptron (MLP) activated by hyperbolic function:

\[{\bf v}^{p}=\tanh({\bf W}_{p}[{\bf h}^{p};{\bf r}^{c}]+{\bf b}_{p}), \tag{5}\]

\[{\bf v}^{c}=\tanh({\bf W}_{c}[{\bf h}^{c};{\bf r}^{p}]+{\bf b}_{c}), \tag{6}\]

where \({\bf W}_{p},{\bf W}_{c}\in\mathbb{R}^{d\times 2d}\) and \({\bf b}_{p},{\bf b}_{c}\in\mathbb{R}^{d}\) are trainable parameters.

Note that either \({\bf v}^{p}\) or \({\bf v}^{c}\) conveys the information from both posts and conversations, but with a

Figure 1: Our hashtag generation framework with a dual encoder, including a post encoder and a conversation encoder, where a bi-attention (bi-att) distills their salient features, followed by a merge layer to fuse them. An attentive decoder generates the hashtag sequence.

 

different emphasis. Specifically, \({\bf v}^{p}\) mainly retains the contexts of posts with the auxiliary information from conversations, while \({\bf v}^{c}\) does the opposite. Finally, vectors \({\bf v}^{p}\) and \({\bf v}^{c}\) are concatenated and fed into the decoder for hashtag generation.

Decoder.Given the representations \({\bf v}=[{\bf v}^{p};{\bf v}^{c}]\) produced by our dual encoder with bi-attention, we apply an attention-based GRU decoder to generate a word sequence \({\bf y}\) as the hashtag. The probability to generate the hashtag conditioned on a target post and its conversation is defined as:

\[Pr({\bf y}|{\bf x}^{p},{\bf x}^{c})=\prod_{t=1}^{|{\bf y}|}Pr(y_{t}|{\bf y}_{< t},{\bf x}^{p},{\bf x}^{c}), \tag{7}\]

where \({\bf y}_{<t}\) refers to \((y_{1},y_{2},...,y_{t-1})\).

Concretely, when generating the \(t\)-th word in hashtag, the decoder emits a hidden state vector \({\bf s}_{t}\in\mathbb{R}^{d}\) and puts a global attention over \({\bf v}\). The attention aims to exploit indicative representations from the encoder outputs \({\bf v}\) and summarizes them into a context vector \({\bf c}_{t}\) defined as:

\[{\bf c}_{t}=\sum_{i=1}^{|{\bf x}^{p}|+|{\bf x}^{c}|}\alpha_{ti}^{d}{\bf v}_{i}, \tag{8}\]

\[\alpha_{ti}^{d}=\frac{\exp(g_{score}({\bf s}_{t},{\bf v}_{i}))}{\sum_{i^{ \prime}=1}^{|{\bf x}^{p}|+|{\bf x}^{c}|}\exp(g_{score}({\bf s}_{t},{\bf v}_{ i^{\prime}})}, \tag{9}\]

where \(g_{score}({\bf s}_{t},{\bf v}_{i})={\bf s}_{t}{\bf W}_{att}{\bf v}_{i}\) is another alignment function (\({\bf W}_{att}\in\mathbb{R}^{d\times d}\)) to measure the similarity between \({\bf s}_{t}\) and \({\bf v}_{i}\).

Finally, we map the current hidden state \({\bf s}_{t}\) of the decoder together with the context vector \({\bf c}_{t}\) to a word distribution over the vocabulary \(V\) via:

\[Pr(y_{t}|{\bf y}_{<t},{\bf x}^{p},{\bf x}^{c})=softmax({\bf W}_{v}[{\bf s}_{t };{\bf c}_{t}]+{\bf b}_{v}), \tag{10}\]

which reflects how likely a word to be the \(t\)-th word in the generated hashtag sequence. Here \({\bf W}_{v}\in\mathbb{R}^{V\times 2d}\) and \({\bf b}_{v}\in\mathbb{R}^{V}\) are trainable weights.

Learning and Inferring Hashtags.During the training stage, we apply stochastic gradient descent to minimize the loss function of our entire framework, which is defined as:

\[\mathcal{L}(\Theta)=-\sum_{n=1}^{N}\log(Pr({\bf y}_{n}|{\bf x}_{n}^{p},{\bf x }_{n}^{c};\Theta)). \tag{11}\]

Here \(N\) is the number of training instances and \(\Theta\) denotes the set of all the learnable parameters.

In hashtag inference, based on the produced word distribution at each time step, word selection is conducted using beam search. In doing so, we generate a ranking list of output hashtags, where the top \(K\) hashtags serve as our final output.

## 3 Experiment Setup

Here we describe how we set up our experiments.

Datasets and Statistic Analysis.Two large-scale experiment datasets are _newly collected_ from popular microblog platforms: an English Twitter dataset and a Chinese Weibo dataset. The Twitter dataset was built based on the TREC 2011 microblog track.3 To recover the conversations, we used Tweet Search API to fetch "in-reply-to" relations in a recursive way. The Weibo dataset was collected from January to August 2014 using Weibo Search API via searching messages with the trending queries4 as keywords. For gold-standard hashtags, we take the user-annotated hashtags, appearing before or after a post, as the reference.5 The statistics of our datasets are shown in Table 2. We randomly split both datasets into three subsets, where \(80\)%, \(10\)%, and \(10\)% of the data corresponds to training, development, and test sets, respectively.

Footnote 3: [https://treec.nist.gov/data/tweets/](https://treec.nist.gov/data/tweets/)

Footnote 4: [http://open.weibo.com/wiki/Trends/](http://open.weibo.com/wiki/Trends/)

Footnote 5: Hashtags in the middle of a post are not considered here as they generally act as semantic elements Zhang et al. (2016, 2018).

To further investigate how challenging our problem is, we show some statistics of the hashtags in Table 3 and the distributions of hashtag frequency in Figure 2.

In Table 3, we observe

\begin{table}
\begin{tabular}{|l|r|r|r|r|r|} \hline
**Datasets** & \# of & Avg len & Avg len & Avg len & \# of tags \\  & posts & of posts & of confs & of tags & per post \\ \hline Twitter & 44,793 & 13.27 & 29.94 & 1.69 & 1.14 \\ Weibo & 40,171 & 32.64 & 70.61 & 2.70 & 1.11 \\ \hline \end{tabular}
\end{table}
Table 2: Statistics of our datasets. Avg len of posts, convs, tags refer to the average number of words in posts, conversations, and hashtags, respectively.

\begin{table}
\begin{tabular}{|l|r|r r r|} \hline
**Datasets** & \(|\)Tagset\(|\) & \(\mathcal{P}\) & \(\mathcal{C}\) & \(\mathcal{P}\cup\mathcal{C}\) \\ \hline Twitter & 4,188 & 2.72\% & 5.58\% & 7.69\% \\ Weibo & 5,027 & 8.29\% & 6.21\% & 12.52\% \\ \hline \end{tabular}
\end{table}
Table 3: Statistics of the hashtags. \(|\)Tagset\(|\): the number of distinct hashtags. \(\mathcal{P}\), \(\mathcal{C}\), and \(\mathcal{P}\cup\mathcal{C}\): the percentage of hashtags appearing in their corresponding posts, conversations, and the union set of them, respectively.

 

the large size of hashtags in both datasets. Moreover, Figure 2 indicates that most hashtags only appear a few times. Given such a large and imbalanced hashtag space, hashtag selection from a candidate list, as many existing methods do, might not perform well. Table 3 also shows that only a small proportion of hashtags appearing in their posts, conversations, and either of them, making it inappropriate to directly extract words from the two sources to form hashtags.

Preprocessing.For tokenization and word segmentation, we employed the tweet preprocessing toolkit released by Baziotis et al. (2017) for Twitter, and the Jieba toolkit6 for Weibo. Then, for both Twitter and Weibo, we further take the following preprocessing steps: First, single-character hashtags were filtered out for not being meaningful. Second, generic tags, i.e., links, mentions (@username), and numbers, were replaced with "URL" "MENTION", and "DIGIT", respectively. Third, inappropriate replies (e.g., retweet-only messages) were removed, and the remainder were chronologically ordered to form a sequence as conversation contexts. Last, a vocabulary was maintained with the \(30K\) and \(50K\) most frequent words, for Twitter and Weibo, respectively.

Footnote 6: [https://pypi.python.org/pypi/jieba/](https://pypi.python.org/pypi/jieba/)

Comparisons.For experiment comparisons, we first consider a weak baseline Random that randomly ranks hashtags seen from training data. Two unsupervised baselines are also considered, where words are ranked by latent topics induced with the latent Dirichlet allocation topic model (henceforth LDA), and by their TF-IDF scores (henceforth Tf-Idf). Here for TF-IDF scores, we consider the \(N\)-gram Tf-Idf (\(N\leq 5\)). Besides, we compare with _supervised_ models below:

\(\bullet\)Extractor: Following Zhang et al. (2018), we extract phrases from target posts as hashtags via sequence tagging and encode conversations with memory networks (Sukhbaatar et al., 2015).

\(\bullet\)Classifier: We compare with the state-of-the-art model based on classification (Gong and Zhang, 2016), where hashtags are selected from candidates seen in training data. Here two versions of their classifier are considered, one only taking a target post as input (henceforth Classifier (_post only_)) and the other taking the concatenation of a target post and its conversation as input (henceforth Classifier (_post+conv_)).

\(\bullet\)Generator: A seq2seq generator (henceforth Seq2Seq) (Sutskever et al., 2014) is applied to generate hashtags given a target post. We also consider its variant augmented with copy mechanism (Gu et al., 2016) (henceforth Seq2Seq-copy), which has proven effective in keyphrase generation (Meng et al., 2017) and also takes the post as input. The proposed seq2seq with the bi-attention to encode both the post and its conversation is denoted as Our model for simplicity.

Model Settings.We conduct model tunings on the development set based on grid search, where the hyper-parameters that give the lowest objective loss are selected. For the sequence generation models, the implementations are based on the OpenNMT framework (Klein et al., 2017). The word embeddings, with dimension set to \(200\), are randomly initialized. For encoders, we employ two layers of Bi-GRU cells, and for decoders, one layer of GRU cell is used. The hidden size of all GRUs is set to \(300\). In learning, we use the Adam optimizer (Kingma and Ba, 2014) with the learning rate initialized to \(0.001\). We adopt the early-stop strategy: the learning rate decreases by a decay rate of \(0.5\) till either it is below \(1e^{-6}\) or the validation loss stops decreasing. The norm of gradients is rescaled to \(1\) if the \(L2\)-norm \(>1\) is observed. The dropout rate is \(0.1\) and the batch size is \(64\). In inference, we set the beam-size to \(20\) and the maximum sequence length of a hashtag to \(10\).

For Classifier and Extractor, lacking publicly available codes, we reimplement the models using Keras.7 Their results are reproduced in their original experiment settings. For LDA, we employ an open source toolkit lda.8

Footnote 7: [https://keras.io/](https://keras.io/)

Footnote 8: [https://pypi.org/project/lda/](https://pypi.org/project/lda/)

Evaluation Metrics.Popular _information retrieval_ evaluation metrics F1 scores at K (F1@K)

Figure 2: Distribution of hashtag frequency. The horizontal axis refers to the occurrence count of hashtags (shown with maximum \(50\) and bin \(5\)) and the vertical axis denotes the data proportion.

 

and mean average precision (MAP) scores Manning et al. (2008) are reported. Here, different \(K\) values are tested on F1@K and result in a similar trend, so only F1@1 and F1 @5 are reported. MAP scores are also computed given the top \(5\) outputs. Besides, as we consider a hashtag as a sequence of words, ROUGE metrics for _summarization_ evaluation Lin (2004) are also adopted. Here, we use ROUGE F1 for the top-ranked hashtag prediction computed by an open source toolkit pythonrouge,9 with Porter stemmer used for English tweets. For Weibo posts, scores calculated at the Chinese character level following Li et al. (2018). We report the average scores for multiple gold-standard hashtags on ROUGE evaluation.

Footnote 9: [https://github.com/tagucci/pythonrouge](https://github.com/tagucci/pythonrouge)

## 4 Experimental Results

In this section, we first report the main comparison results in Section 4.1, followed by an in-depth comparative study between classification and sequence generation models in Section 4.2. Further discussions are then presented to analyze our superiority and errors in Section 4.3.

### Main Comparison Results

Table 4 reports the main comparison results. For Classifier, their outputs are ranked according to the logits after a \(softmax\) layer. For Extractor, it is unable to produce ranked hashtags and thus no results are reported for F1@5 and MAP. For LDA, as it cannot generate bigram hashtags, no results are presented for ROUGE-SU4. In general, we have the following observations:

\(\bullet\)_Hashtag annotation is more challenging for Twitter than Weibo_. Generally, all models perform worse on Twitter measured by different metrics. The intrinsic reason is the essential language difference between English and Chinese microblogs. English allows higher freedom in writing, resulting in more variety in Twitter hashtags (e.g., abbreviations are prominent like "_aus_" in "_#AusOpen_"). For statistical reasons, Twitter hashtags are more likely to be absent in either posts or conversations (Table 3), and have a more severe imbalanced distribution (Figure 2).

\(\bullet\)_Topic models and extractive models are ineffective for hashtag annotation_. The poor performance of all baseline models indicates that hashtag annotation is a challenging problem. LDA sometimes performs even worse than Random due to its inability to produce phrase-level hashtags. For extractive models, both Tf-Idf and Extractor fail to achieve good results. It is because most hashtags are absent in target posts, as we see in Table 3 that only \(2.72\)% hashtags on Twitter and \(8.29\)% on Weibo appear in target posts. This confirms that extractive models, relying on word selection from target posts, cannot well fit the hashtag annotation scenario. For the same reason, copy mechanism fails to bring noticeable improvements for the seq2seq generator on both datasets.

\(\bullet\)_Sequence generation models outperform other counterparts._ When comparing Generators with other models, we find the former uniformly achieve better results, showing the superiority to produce hashtags with sequence generation framework. Classification models, though as

\begin{table}
\begin{tabular}{|l|c c c c c||c c c c c|} \hline \multirow{2}{*}{**Model**} & \multicolumn{6}{c||}{Twitter} & \multicolumn{6}{c|}{Weibo} \\ \cline{2-11}  & F1@1 & F1@5 & MAP & RG-1 & FG-4 & F1@1 & F1@5 & MAP & RG-1 & RG-4 \\ \hline
* & 0.10 & 0.86 & 0.94 & 3.89 &
- \\ Tf-Idf & 0.02 & 0.02 & 0.03 & 0.54 & 0.14 & 0.85 & 0.73 & 1.30 & 8.04 & 4.29 \\ Extractor & 0.44 &
- & 1.14 & 0.14 & 2.53 &
- & 7.64 & 5.20 \\ \hline \hline
**State of the arts** & & & & & & & & & & \\ Classifier (_post only_) & 9.44 & 6.36 & 12.71 & 10.75 & 4.00 & 16.92 & 10.48 & 22.29 & 25.34 & 21.95 \\ Classifier (_post+conv_) & 8.54 & 6.28 & 12.10 & 10.00 & 2.47 & 17.25 & 11.03 & 23.11 & 25.16 & 22.09 \\ \hline \hline
**Generators** & & & & & & & & & & \\ Seq2Seq & 10.44 & 6.73 & 14.00 & 10.52 & 4.08 & 26.00 & 14.43 & 32.74 & 37.37 & 32.67 \\ Seq2Seq-copy & 10.63 & 6.87 & 14.21 & 12.05 & 4.36 & 25.29 & 14.10 & 31.63 & 37.58 & 32.69 \\ Our model & **12.29*** & **8.29*** & **15.94*** & **13.73*** & **4.45** & **31.96*** & **17.39*** & **38.79*** & **45.03*** & **39.73*** \\ \hline \end{tabular}
\end{table}
Table 4: Comparison results on Twitter and Weibo datasets (in %). RG-1 and RG-4 refer to ROUGE-1 and ROUGE-SU4 respectively. The best results in each column are in bold. The “*” after numbers indicates significantly better results than all the other models (\(p<0.05\), paired t-test). Higher values indicate better performance.

 

the state of the art, expose their inferiority as they select labels from the large and imbalanced hashtag space (reflected in Table 3 and Figure 2).

\(\bullet\)_Conversations are useful for hashtag generation._ Among the sequence generation models, Our model achieves the best performance across all the metrics. The observation indicates the usefulness of bi-attention in exploiting the joint effects of target posts and their conversations, which further helps in identifying indicative features from both sources for hashtag generation. However, interestingly, incorporating conversations fails to boost the classification performance. The reason why Our model better exploits conversations than Classifier (_post+conv_) might be that we can attend the indicative features when decoding each word in the hashtag, which is however not possible for classification models (considering hashtags to be inseparable).

### Classification vs. Generation

From Table 4, we observe that the classifiers outperform topic models and extractive models by a large margin but exhibit generally worse results than sequence generation models. Here, we present a thorough study to compare hashtag classification and generation. Four models are selected for comparison: two classifiers, Classifier (_post only_) and Classifier (_post+conv_), and two sequence generation models, Seq2Seq and Our model. Below, we explore how they perform to predict rare and new hashtags.

Rare HashtagsAccording to the hashtag distributions in Figure 2, we can see a large proportion of hashtags appearing only a few times in the data. To study how models perform to predict such hashtags, in Figure 3, we display their F1@1 scores in inferring hashtags with varying frequency. The lower F1 score on less frequent hashtags indicates the difficulty to yield rare hashtags. The reason probably comes from the overfitting issue caused by limited data to learn from.

We also observe that sequence generation models achieve consistently better F1@1 scores on hashtags with varying sparsity degree, while classification models suffer from the label sparsity issue and obtain worse results. The better performance of the former might result from the word-by-word generation manner in hashtag generation, which enables the internal structure of hashtags (how words form a hashtag) to be exploited.

New HashtagsTo further explore the extreme situation where hashtags are absent in the training set, we experiment to see how models perform in handling new hashtags. To this end, we additionally collect instances tagged with hashtags absent in training data and construct an external test set, with the same size as our original test set. Considering that classifiers will never predict unseen labels, to ensure comparable performance, we only adopt summarization metrics here for evaluation and report ROUGE-1 F1 scores in Table 5.

As can be seen, creating unseen hashtags is a challenging task, where unsurprisingly, all models perform poorly on this task. Nevertheless, sequence generation models perform much better on both datasets, e.g., at least 6.5x improvements over classification models observed on Weibo dataset. For Twitter dataset, the improvements are not that large, which confirms again that hashtag annotation on Twitter is more difficult due to the noisier data characteristics. In particular, compared to seq2seq, our model achieves an additional performance gain in producing new hashtags by leveraging conversations with the bi-attention.

### Further Discussions on Our Model

To further analyze our model, we conduct a quantitative ablation study, a qualitative case study, and an error analysis. We then discuss them in turn.

Figure 3: F1@1 on Twitter (the left subfigure) and Weibo (the right subfigure) in inferring hashtags with varying frequency. In each subfigure, from left to right shows the results of Classifier (_post only_), Classifier (_post+conv_), Seq2Seq, and Our model. Generation models consistently perform better.

 

Ablation Study.We report the ablation study results in Table 6 to examine the relative contributions of the target posts and the conversation contexts. To this end, our model is compared with its five variants below: Seq2Seq (_post only_), Seq2Seq (_conv only_), and Seq2Seq (_post+conv_), using standard seq2seq to generate hashtags from their target posts, conversation contexts, and their concatenation, respectively; Our model (_post-att only_) and Our model (_conv-att only_), whose decoder only takes \({\bf v}^{p}\) and \({\bf v}^{c}\) defined in Eq. (5) and Eq. (6), respectively. The results show that solely encoding target posts is more effective than modeling the conversations alone, but exploring their joint effects can further boost the performance, especially combined with a bi-attention mechanism over them.

Case Study.We further present a case study on the target post shown in Table 1, where the top five outputs of some comparison models are displayed in Table 7. As can be seen, only our model successfully generates "_aus open_", the gold standard. Particularly, it not only ranks the correct answer as the top prediction, but also outputs other semantically similar hashtags, e.g., sport-related terms like "_bbc football_", "_arsenal_", and "_murray_". On the contrary, Classifier and Seq2Seq tend to yield frequent hashtags, such as "_just saying_" and "_jan 25_". Baseline models also perform poorly: LDA produces some common single word, and TF-IDF extracts phrases in the target post, where the gold-standard hashtag is however absent.

To analyze why our model obtains superior results in this case, we display the heatmap in Figure 4 to visualize our bi-attention weight matrix \({\bf W}_{bi-att}\). As we can see, bi-attention can identify the indicative word "_Azarenka_" in the target post, via highlighting its other pertinent words in conversations, e.g., "_Nadal_" and "_tennis_". In doing so, salient words in both the post and its conversations can be unveiled, facilitating the correct hashtag "_aus open_" to be generated.

Error Analysis.Taking a closer look at our outputs, we find that one type of major errors comes from the unmatched outputs with gold standards, even as a close guess. For example, our model predicts "_super bowl_" for a post tagged with "_#steelers_", a team in super bowl. In future work, the semantic similarity should be considered in hashtag evaluation. Another primary type of error is caused by the non-topic hashtags, such as "_#fb_" (indicating the messages forwarded from Facebook). Such non-topic hashtags cannot reflect any content information from target posts and should be distinguished from topic hashtags in the future.

## 5 Related Work

Our work mainly builds on two streams of previous work -- microblog hashtag annotation and neural language generation.

We are in the line of microblog hashtag annotation. Some prior work extracts phrases from target posts with sequence tagging models Zhang et al. (2016, 2018). Another popular approach is to apply classifiers and select hashtags from a candidate list Heymann et al. (2008); Weston et al. (2014); Sedhai and Sun (2014); Gong and Zhang (2016);

Figure 4: Visualization of bi-attention given the input case in Table 1. The horizontal axis denotes a snippet of a truncated conversation. The vertical axis shows the target post. Salient words are highlighted.

\begin{table}
\begin{tabular}{|l|r|r|} \hline
**Model** & **Twitter** & ** Weibo** \\ \hline Seq2seq (_post only_) & 10.44 & 26.00 \\ Seq2seq (_conv only_) & 6.27 & 18.57 \\ Seq2seq (_post + conv_) & 11.24 & 29.85 \\ Our model (_post-att only_) & 11.18 & 28.67 \\ Our model (_conv-att only_) & 10.61 & 28.06 \\ Our model (_full_) & **12.29** & **31.96** \\ \hline \end{tabular}
\end{table}
Table 6: F1@1 scores (%) for our variants.

 

Huang et al., 2016; Zhang et al., 2017). Unlike them, we generate hashtags with a language generation framework, where hashtags in neither the target posts nor the pre-defined candidate list can be created. Topic models are also widely applied to induce topic words as hashtags Krestel et al. (2009); Ding et al. (2012); Godin et al. (2013); Gong et al. (2015); Zhang et al. (2016). However, these models are usually unable to produce phrase-level hashtags, which can be achieved by ours via generating hashtag word sequences with a decoder.

Our work is also closely related to neural language generation, where the encoder-decoder framework Sutskever et al. (2014) acts as a springboard for many sequence generation models. In particular, we are inspired by the keyphrase generation studies for scientific articles Meng et al. (2017); Ye and Wang (2018); Chen et al. (2018, 2019), incorporating word extraction and generation using a seq2seq model with copy mechanism. However, our hashtag generation task is inherently different from theirs. As we can see from Table 4, it is suboptimal to directly apply keyphrase generation models on our data. The reason mostly lies in the informal language style of microblog users in writing both target posts and their hashtags. To adapt our model on microblog data, we explore the effects of conversation contexts on hashtag generation, which has never been studied in any prior work before.

## 6 Conclusion

We have presented a novel framework of hashtag generation via jointly modeling of target posts and conversation contexts. To this end, we have proposed a neural seq2seq model with bi-attention over a dual encoder for capturing indicative representations from the two sources. Experimental results on two newly collected datasets have demonstrated that our proposed model significantly outperforms existing state-of-the-art models. Further studies have shown that our model can effectively generate rare and even unseen hashtags.

## Acknowledgements

This work is supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 14208815 and No. CUHK 14210717 of the General Research Fund). We thank NAACL reviewers for their insightful suggestions on various aspects of this work.

## References

* Bansal et al. (2015) Piyush Bansal, Somay Jain, and Vasudeva Varma. 2015. Towards semantic retrieval of hashtags in microblogs. In _World Wide Web Conference_.
* Baziotis et al. (2017) Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis. 2017. Datastories at semeval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In _North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
* Chang et al. (2013) Yi Chang, Xuanhui Wang, Qiaozhu Mei, and Yan Liu. 2013. Towards twitter context summarization with user influence models. In _International Conference on Web Search and Data Mining_.
* Chen et al. (2018) Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018. Keyphrase generation with correlation constraints. In _Empirical Methods in Natural Language Processing_.
* Chen et al. (2019) Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R. Lyu. 2019. Title-guided encoding for keyphrase generation. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence_.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Empirical Methods in Natural Language Processing_.
* Davidov et al. (2010) Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Enhanced sentiment learning using twitter hashtags and smileys. In _International Conference on Computational Linguistics_.
* Ding et al. (2012) Zhuoye Ding, Qi Zhang, and Xuanjing Huang. 2012. Automatic hashtag recommendation for microblogs using topic-specific translation model. In _International Conference on Computational Linguistics_.
* Efron (2010) Miles Efron. 2010. Hashtag retrieval in a microblogging environment. In _Conference on Research and Development in Information Retrieval_.
* Godin et al. (2013) Frederic Godin, Viktor Slavkovikj, Wesley De Neve, Benjamin Schrauwen, and Rik Van de Walle. 2013. Using topic models for twitter hashtag recommendation. In _World Wide Web Conference_.
* Gong et al. (2015) Yeyun Gong, Qi Zhang, and Xuanjing Huang. 2015. Hashtag recommendation using dirichlet process mixture models incorporating types of hashtags. In _Empirical Methods in Natural Language Processing_.
* Gong and Zhang (2016) Yuyun Gong and Qi Zhang. 2016. Hashtag recommendation using attention-based convolutional neural network. In _International Joint Conference on Artificial Intelligence_ 

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Association for Computational Linguistics_.
* Heymann et al. (2008) Paul Heymann, Daniel Ramage, and Hector Garcia-Molina. 2008. Social tag prediction. In _Conference on Research and Development in Information Retrieval_.
* Hong et al. (2012) Liangjie Hong, Amr Ahmed, Siva Gurumurthy, Alexander J. Smola, and Kostas Tsioutsiouliklis. 2012. Discovering geographical topics in the twitter stream. In _World Wide Web Conference_.
* Huang et al. (2016) Haoran Huang, Qi Zhang, Yeyun Gong, and Xuanjing Huang. 2016. Hashtag recommendation using end-to-end memory networks with hierarchical attention. In _International Conference on Computational Linguistics_.
* Khabiri et al. (2012) Elham Khabiri, James Caverlee, and Krishna Yeswanth Kamath. 2012. Predicting semantic annotations on the real-time web. In _ACM Conference on Hypertext and Social Media_.
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In _International Conference on Learning Representations_.
* Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. In _Association for Computational Linguistics_.
* Krestel et al. (2009) Ralf Krestel, Peter Fankhauser, and Wolfgang Nejdl. 2009. Latent dirichlet allocation for tag recommendation. In _ACM Conference on Recommender Systems_.
* Li et al. (2015) Jing Li, Wei Gao, Zhongyu Wei, Baolin Peng, and Kam-Fai Wong. 2015. Using content-level structures for summarizing microblog repost trees. In _Empirical Methods in Natural Language Processing_.
* Li et al. (2016) Jing Li, Ming Liao, Wei Gao, Yalan He, and Kam-Fai Wong. 2016. Topic extraction from microblog posts using conversation structures. In _Association for Computational Linguistics_.
* Li et al. (2018) Jing Li, Yan Song, Zhongyu Wei, and Kam-Fai Wong. 2018. A joint model of conversational discourse and latent topics on microblogs. _Computational Linguistics_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text Summarization Branches Out: Proceedings of the Association for Computational Linguistics-04 Workshop_.
* Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze. 2008. _Introduction to information retrieval_. Cambridge University Press.
* Meng et al. (2017) Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017. Deep keyphrase generation. In _Association for Computational Linguistics_.
* Sedhai and Sun (2014) Surendra Sedhai and Aixin Sun. 2014. Hashtag recommendation for hyperlinked tweets. In _Conference on Research and Development in Information Retrieval_.
* Seo et al. (2016) Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. In _International Conference on Learning Representations_.
* Sakhbaatar et al. (2015) Sainbayar Sakhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In _Neural Information Processing Systems_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Neural Information Processing Systems_.
* Wang et al. (2011) Xiaolong Wang, Furu Wei, Xiaohua Liu, Ming Zhou, and Ming Zhang. 2011. Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach. In _Conference on Information and Knowledge Management_.
* Weston et al. (2014) Jason Weston, Sumit Chopra, and Keith Adams. 2014. #tagspace: Semantic embeddings from hashtags. In _Association for Computational Linguistics_.
* Yan et al. (2013) Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2013. A biterm topic model for short texts. In _World Wide Web Conference_.
* Ye and Wang (2018) Hai Ye and Lu Wang. 2018. Semi-supervised learning for neural keyphrase generation. In _Empirical Methods in Natural Language Processing_.
* Zhang et al. (2017) Qi Zhang, Jiawen Wang, Haoran Huang, Xuanjing Huang, and Yeyun Gong. 2017. Hashtag recommendation for multimodal microblog using co-attention network. In _International Joint Conference on Artificial Intelligence_.
* Zhang et al. (2016) Qi Zhang, Yang Wang, Yeyun Gong, and Xuanjing Huang. 2016. Keyphrase extraction using deep recurrent neural networks on twitter. In _Empirical Methods in Natural Language Processing_.
* Zhang et al. (2013) Renxian Zhang, Wenjie Li, Dehong Gao, and Ouyang You. 2013. Automatic twitter topic summarization with speech acts. _IEEE Trans. Audio, Speech & Language Processing_.
* Zhang et al. (2018) Yingyi Zhang, Jing Li, Yan Song, and Chengzhi Zhang. 2018. Encoding conversation context for neural keyphrase extraction from microblog posts. In _North American Chapter of the Association for Computational Linguistics: Human Language Technologies_