Question: Do they pay special attention to emoji when comparing their model to a baseline?
Evidence: The prevalent use of emoji—and their text-based precursors—is mostly unaddressed in current natural language processing (NLP) tasks. The support of the Unicode Standard BIBREF0 for emoji characters in 2010 ushered in a wide-spread, international adoption of these graphical elements in casual contexts. Interpreting the meaning of these characters has been challenging however, since they take on multiple semantic roles BIBREF1.
Whether or not emoji are used depends on the context of a text or conversation, with more formal settings generally being less tolerating. So is the popular aligned corpus Europarl BIBREF2 naturally devoid of emoji. Technical limitations, like no Unicode support, also limit its use. This in turn affects commonly used corpora, tokenizers, and pre-trained networks.
Take for example the Ubuntu Dialog Corpus by BIBREF3, a commonly used corpus for multi-turn systems. This dataset was collected from an Internet Relay Chat (IRC) room casually discussing the operating system Ubuntu. IRC nodes usually support the ASCII text encoding, so there's no support for graphical emoji. However, in the 7,189,051 utterances, there are only 9946 happy emoticons (i.e. :-) and the cruelly denosed :) version) and 2125 sad emoticons.
Word embeddings are also handling emoji poorly: Word2vec BIBREF4 with the commonly used pre-trained Google News vectors doesn't support the graphical emoji at all and vectors for textual emoticons are inconsistent. As another example with contextualized word embeddings, there are also no emoji or textual emoticons in the vocabulary list of BERT BIBREF5 by default and support for emoji is only recently added to the tokenizer. The same is true for GPT-2 BIBREF6. As all downstream systems, ranging from multilingual résumé parsing to fallacy detection BIBREF7, rely on the completeness of these embeddings, this lack of emoji support can affect the performance of some of these systems.
Another challenge is that emoji usage isn't static. Think of shifting conventions, different cultures, and newly added emoji to the Unicode list. Several applications also use their own custom emoji, like chat application Slack and streaming service Twitch. This becomes an issue for methods that leverage the Unicode description BIBREF8 or that rely on manual annotations BIBREF9.
Our contribution with this paper is two-fold: firstly, we argue that the current use—or rather non-existing use—of emoji in the tokenizing, training, and the datasets themselves is insufficient. Secondly, we attempt to quantify the significance of incorporating emoji-based features by presenting a fine-tuned model. We then compare this model to a baseline, but without special attention to emoji.
Section SECREF2 will start with an overview of work on emoji representations, emoji-based models and analysis of emoji usage. A brief introduction in conversational systems will also be given. Section SECREF3 will then look into popular datasets with and without emoji and then introduce the dataset we used.
Our model will then be discussed in Section SECREF4, including the tokenization in Subsection SECREF4, training setup in Subsection SECREF6 and evaluation in Subsection SECREF10. This brings us to the results of our experiment, which is discussed in Section SECREF5 and finally our conclusion and future work are presented in Section SECREF6.
Secondly, we attempt to quantify the significance of incorporating emoji-based features by presenting a fine-tuned model. We then compare this model to a baseline, but without special attention to emoji.
Answer: No.