言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Predict Emoji Combination with Retrieval Strategy
Weitsung LinTinghsuan ChaoTianhuang SuJianmin Wu
EBG, Baidu Inc, Shenzhen, 518052{v liweicong, v zhaotingxuan, wujianmin01, sutianhuang}@baidu.com

Abstract

dicting emojis and emoji categories to improve prediction accuracy. In this paper, we extend the single
emoji prediction task to a setup much closer to realworld usage. First, we extend emoji classes to predict to 500, which cover 95% of emoji usage. Second,
we predict multiple emojis for a given text since inreal world user often use multiple emojis.
There is huge variation in people’s usage of emoji.
It could have many different but reasonable target
emojis conditions on the same textual input which
increases the difficulty of predicting. This problem
becomes more critical in the multiple emojis task
when possible candidates increase exponentially by
combining multiple emojis. To tackle this difficult
problem, we propose a state of the art classification
NLP model to predict the emojis from a Twitter message. This model is based on BERT [6] with retrieval
strategy we proposed.

As emojis are widely used in social media,
people not only use an emoji to express
their emotions or mention things but also
extend its usage to represent complicate
emotions, concepts or activities by combining multiple emojis. In this work, we
study how emoji combination, a consecutive emoji sequence, is used like a new language. We propose a novel algorithm called
Retrieval Strategy to predict what emoji
combination follows given a short text ascontext. Our algorithm treats emoji combinations as phrase in language, ranking setsof emoji combinations like retrieving words
from dictionary. We show that our algorithm largely improves the F1 score from0.141 to 0.204 on emoji combination prediction task.

12Introduction

Emojis are very popular and widely used in large social media platforms like Twitter, Facebook and Instagram. These visual symbols can be used to convey
emotions and underlying information and strengthened its power by combining multiple ones (Figure1). In Natural Language Processing (NLP) perspective, they provide extra information for the semanticsof sentences.

Related Works

Previous works on emoji prediction focus on predicting single emoji by textual inputs. This is done bymodeling the input text into a sentiment represent
and predicting only popular emojis. The model architecture includes SVM, LSTM and LSTM with attention have been explored [2, 9, 4].
The recent approaches add other input modalities
and predict target base on this setting. [1, 5] consider both textual and visual inputs and build a multimodal emoji prediction. [3] uses both emojis andemoji categories from keyboard session as predicting targets to train a multi-task model and furtherincreases the emoji labels to 300 which is closer topractical usage.
Our task setting is most similar to [3], which uses
only text input. The difference is in targets, we use
multiple emojis other than single emoji. This target
setting is more closer to the real world use case.

Figure 1: Emoji combination use cases

3
Recent work [2] has shown that textual information
can be used to predict single emoji associated to text.
[3] further extended the emoji prediction task to 300
emoji classes and used a multi-task approach, pre-Dataset and Task

In this work we retrieved 10 million tweets withemojis in text from Twitter. Tweets is restricted tohave geo-location in the United States of America― 132 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

4.1

Emoji Prediction Model: BERT

BERT is a multi-layer bidirectional Transformer
encoder based on the original implementation
described in [8]. It obtains state-of-the-art results
on NLP benchmark GLUE. We follow the training
procedure described in paper with pre-training stage
and fine-tuning stage.
Pre-training: In the pre-training stage, there
are two differences between our model and theoriginal BERT model. The first one is training data.
Instead of using huge corpus like Wikipedia, weonly use corpus from Twitter to learn a language
model because we want the language model to be
more related to twitter usage and restriction ofcomputation resource.
The second difference is that we only use Masked
LM as pre-training target, because it is hard todetermine next sentence between tweets and mosttweets contain only one sentence. In model architecture, we use not only word level embedding andalso character level embedding in order to capture
meaning of out of vocabulary words since tweets
contain lots of new words and misspellings.
Figure 2: 500 emoji classes in this work

and written in English. In the preprocessing stage,
we removed all hyperlinks, mentions and hashtagsfrom tweet. Also we normalize emoji by removing
skin tone modifier to focus on meaning of emoji. To
build the dataset, whose sample consists of a pairof textual context (feature) and emoji combination(target), we process Tweets as follow: (1) Selecttweets include emojis in the 500 most frequent ones
(2) Find consecutive sequence emojis as target andall text before it including emojis as textual contextand (3) Keep only samples with targets included
one to three emojis. In the end, we collect a final
dataset contained about 9 million samples.

Fine-tuning: In the fine-tuning stage, we selectfirst time step vector in output sequence as input text
representation like original BERT. The objective ofemoji combination task is similar to language modeling to maximize following conditional probability:
P (Y |W ) = P (y0 |W )nY

P (yi |W, y0 , ..., yi−1 )

(1)

i=1

Where W is the input text, Y is emoji combinationand yi is an individual emoji in Y . Because our task
selects emoji combination with size between one tothree and ease of computation, we approximate then-gram conditional probability by using the following
unigram models:
P (Y |W ) ≈nY

P (yi |W )

(2)

i=04

We apply softmax on the model output to computethe probability distribution and minimize the crossentropy between prediction andQlabel distribution to
maximize the log-likelihood of P (yi |W ).

MethodIn this Section, we describe our method to predict
emoji combination given a textual context. The
method consists of two parts: (1) A model to predict
probability of emojis given a textual context and(2) a strategy to consume probability and generatefinal emoji combination.

4.2

Retrieval Strategy

The naive way to produce emoji combination fromprobability distribution is picking top ones and finding a good threshold to produce emoji combination
with variable length. However, this strategy tends
to predict common emojis and produce unreasonable― 133 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

combinations, resulting bad performance. Moreover,
the ordering of emoji combination can’t be carefullymodeled, hence visual meaning lost. To tackle thisproblem, we treat emoji as a language: taking emoji
as word then emoji combination naturally becomesphrase. Instead of predicting emoji combination directly, we use retrieve strategy to rank the existed
emoji combinations which are mined from corpus. In
other word, we predict the emoji combination froma collection like predicting phrase from a dictionary.
The ranking score we used is cross-entropy between
the emoji combination candidate and the prediction
probability distribution:
X
Sj = −
Candidatej (yi ) log P (yi |W )(3)

Table 1: Combination strategies evaluation resultson test dataset. The higher threshold for Greedy
Top-3 results in higher recall but lower precision.
The penalty in Retrieval strategy penalizes less emoji
number in combination. Fewer emoji in combination
results bigger penalizing. The highest F1 is Retrieval
Strategy with penalty at 0.3.
Combination
Strategy
Naive Top-3
Greedy Top-3 thr=0.4
Greedy Top-3 thr=0.3
Greedy Top-3 thr=0.2
Retrieval
Retrieval pen=0.2
Retrieval pen=0.3
Retrieval pen=0.4iWhere Sj is the ranking score, Candidatej (yi ) isemoji distribution of yi in candidate j, P (yi |W ) is
the model prediction for emoji distribution. Becauseof the approximation we used in Eq.2, emoji combination with the same emoji set but different orderings would have the same ranking score. We use
frequency of emoji combination in training data to
break a tie. As a result, our method can produce
meaningful emoji combination contains rare emojis.

5Experiment

To evaluate the quality of emoji combination prediction, we randomly select 10000 samples from theTwitter dataset for testing and use precision, recalland F1 score as evaluation metrics. We compared theemoji combination prediction task with three combination strategies: Naive Top-3: Select first threeemojis with top probability, Greedy Top-3: Selectemojis with top probability greedily until cumulative
probability reaches threshold or size exceeds threeand Retrieval Strategy described in section 4.2.
In Retrieval Strategy, we use Eq.3 to rank 30kmost frequently used emojis in dataset and add anemoji size penalty to trade off between recall andprecision.
For data preprocessing, we simply use whitespace as
separator for processing input text and treat eachemoji in input text as a word token. The emoji
is normalized by cleaning the different skin tones
and only used which is in 1 Unicode Full Emoji List
v11.0. In this work, the emoji prediction model
has smaller architecture than BERT base model tosave computation power. We denote Transformer
blocks as L, hidden size as H and self-attention
heads as A. The model size is: L=10, H=256,
A=4, feed-forward/filters=3H and the maximum sequences length is 24 tokens.
1 https://unicode.org/emoji/charts/full-emoji-list.html

recall
(%)
29.4
28.0
25.6
21.4
18.6
24.3
26.6
28.3

precision
(%)
9.3
13.4
14.5
16.0
22.5
21.9
21.2
20.1

F1
(×100)
14.1
18.1
18.5
18.3
20.4
23.0
23.6
23.5

In the pre-training stage, the training sequence isgenerated from all 10M tweets mention in Section
3. We train with batch size of 64 sequences for
1,500,000 steps, which is approximately 10 epochs
over the 10M tweets data. We set hyperparameters,
including Adam, L2 weight decay, linear decay oflearning rate, dropout and activation function thesame as the original BERT model.
In the fine-tuning stage, we take the first time step oftransformer output sequence and denote this vector
as C ∈ RH . In order to calculate label probability, we add a fully-connection layer with parameters
W ∈ RK×H and b ∈ RK , where K is the number
type of emojis which in our case is 500. The label
probabilities P ∈ RK are computed with a standard
softmax, P = sof tmax(CW T + b). The training setting is batch size 64, Adam learning rate 5e-5 andtrain for 5 epochs. We also add a weighted object
Mask LM from pre-trained as auxiliary object like
OpenAI GPT [7].
The comparison result of combination strategies islisted in Table 1. All strategies use the same emoji
prediction model weights and only different in thecombination strategy which processes emoji probability from the same model.
We add a value for compromising between recall andprecision for both strategies. The highest F1 score isbetween the most balance recall and precision. The
result shows our retrieval algorithm largely improved
the F1 score in emoji combination prediction task.

― 134 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

6Conclusion

As emojis getting widely used in social media, people are continuing inventing new emoji combinations
to express different meanings. The emoji combinations are used more and more like a new language,
constructed in different ordered even with structure
to express new meanings. To tackle the emoji combination prediction problem, we propose a novel algorithm consists of a state-of-the-art language modeling to predict emoji probability distribution anda retrieval strategy which approximates complex ngram model with unigram fashion to produce reasonable emoji combination. We showed that our
method beats other methods on emoji combination
prediction task with large margin. In addition, weadd a penalty in emoji combination scoring by compromising between recall and precision for the retrieval strategy to further improve F1 score. To ourbest knowledge, this is the first work to study emojicombination and achieve decent result on Twitterdataset.

[8] Ashish Vaswani, Noam Shazeer, Niki Parmar,
Jakob Uszkoreit, Llion Jones, Aidan N. Gomez,Lukasz Kaiser, and Illia Polosukhin. Attention is
all you need. In NIPS, 2017.
[9] agri öltekin and Taraka Rama. Tübingen-oslo atsemeval-2018 task 2: Svms perform better thanrnns in emoji prediction. In SemEval@NAACLHLT, 2018.

References[1] Francesco
Barbieri,MiguelBallesteros,
Francesco Ronzano, and Horacio Saggion.
Multimodal emoji prediction. In NAACL-HLT,2018.
[2] Francesco Barbieri, Miguel Ballesteros, and Horacio Saggion. Are emojis predictable? In EACL,2017.
[3] Francesco Barbieri, Lús Marujo, Pradeep Karuturi, and William Brendel. Multi-task emoji
learning. 2018.
[4] Christos Baziotis, Nikos Athanasiou, Georgios
Paraskevopoulos, Nikolaos Ellinas, Athanasia
Kolovou, and Alexandros Potamianos. Ntuaslp at semeval-2018 task 2: Predicting emojis using rnns with context-aware attention. In
SemEval@NAACL-HLT, 2018.
[5] Spencer Cappallo, Stacey Svetlichnaya, Pierre
Garrigues, Thomas Mensink, and Cees Snoek.
The new modality: Emoji challenges in prediction, anticipation, and retrieval.
CoRR,abs/1801.10253, 2018.
[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018.
[7] Alec Radford. Improving language understanding by generative pre-training. 2018.

― 135 ―

Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.