Abstract: The role of social media, in particular microblogging platforms such as Twitter, as a conduit for actionable and tactical information during disasters is increasingly acknowledged. However, time-critical analysis of big crisis data on social media streams brings challenges to machine learning techniques, especially the ones that use supervised learning. The Scarcity of labeled data, particularly in the early hours of a crisis, delays the machine learning process. The current state-of-the-art classification methods require a significant amount of labeled data specific to a particular event for training plus a lot of feature engineering to achieve best results. In this work, we introduce neural network based classification methods for binary and multi-class tweet classification task. We show that neural network based models do not require any feature engineering and perform better than state-of-the-art methods. In the early hours of a disaster when no labeled data is available, our proposed method makes the best use of the out-of-event data and achieves good results.

Introduction: Time-critical analysis of social media data streams is important for many application areas. For instance, responders to humanitarian disasters (e.g., earthquake, flood) need information about the disasters to determine what help is needed and where. This information usually breaks out on social media before other sources. During the onset of a crisis situation, rapid analysis of messages posted on microblogging platforms such as Twitter can help humanitarian organizations like the United Nations gain situational awareness, learn about urgent needs of affected people at different locations, and decide on actions accordingly BIBREF0 , BIBREF1 . Artificial Intelligence for Disaster Response (AIDR) is an online platform to support this cause BIBREF2 . During a disaster, any person or organization can use it to collect tweets related to the event. The total volume of all tweets can be huge, about 350 thousand tweets per minute. Filtering them using keywords helps cut down this volume to some extent. But, identifying different kinds of useful tweets that responders can act upon cannot be achieved using only keywords because a large number of tweets may contain the keywords but are of limited utility for the responders. The best-known solution to address this problem is to use supervised classifiers that would separate useful tweets from the rest. Classifying tweets to identify their usefulness is difficult because: tweets are short – only 140 characters – and therefore, hard to understand without enough context; they often contain abbreviations, informal language and are ambiguous; and, finally, determining whether the tweet is useful in a disaster situation and identifying required actions for relief operations is a hard task because of its subjectivity. Individuals differ on their judgement about whether a tweet is useful or not and sometimes whether they belong to one topical class or another especially when there is information in a tweet that would be classified into multiple topical classes. Given this ambiguity, a computer cannot agree with annotators at a rate that is higher than the rate at which the annotators agree with each other. Despite advances in natural language processing (NLP), interpreting the semantics of the short informal texts automatically remains a hard problem. To classify disaster-related tweets, traditional classification approaches use batch learning with discrete representation of words. This approach has three major limitations. First, in the beginning of a disaster situation, there is no event labeled data available for training. Later, the labeled data arrives in small batches depending on the availability of geographically dispersed volunteers. These learning algorithms are dependent on the labeled data of the event for training. Due to the discrete word representations, they perform poor when trained on the data from previous events (out-of-event data). The second limitation is the offline learning style that inputs the complete labeled data and train a model. This is computational expensive in a disaster situation where labeled data is coming in batches. One would need to train a classifier from scratch every time a new batch of labeled data arrives. Thirdly, these approaches require to manually engineered features like cue words and TF-IDF vectors BIBREF3 for learning. Deep neural networks (DNNs) are based on online learning mechanism and have the flexibility to adaptively learn from new batches of labeled data without requiring to retrain from scratch. Due to their distributed word representation, they generalize well and make better use of the previously labeled data from other events to speed up the classification process in the beginning of a disaster. DNNs automatically learn latent features as distributed dense vectors, which generalize well and have shown to benefit various NLP tasks BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . In this paper, we propose a convolutional neural network (CNN) for the classification task. CNN captures the most salient $n$ -gram information by means of its convolution and max-pooling operations. On top of the typical CNN, we propose an extension that combines multilayer perceptron with a CNN. We present a series of experiments using different variations of the training data – event data only, out-of-event data only and a concatenation of both. Experiments are conducted for binary and multi-class classification tasks. For the event only binary classification task, the CNN model outperformed in four out of five tasks with an accuracy gain of up to 4.5 absolute points. In the scenario of no event data, the CNN model shows substantial improvement of up to 18 absolute points over the several non-neural models. This makes the neural network model an ideal choice in early hours of a disaster for tweet classification. When combined the event data with out-of-event data, we see similar results as in the case of event only training. For multi-class classification, the CNN model outperformed in similar fashion as in the case of binary classification. Our variation of the CNN model with multilayer perceptron (MLP-CNN) performed better than it's CNN counter part. In some cases, adding out-of-event data drops the performance. To reduce the effect of large out-of-event data and to make the most out of it, we apply a simple event selection technique based on TF-IDF and select only those events that are most similar to the event under consideration. We then train the classifiers on the concatenation of the event plus selected out-of-event data. The performance improves only for the event with small event data. To summarize, we show that neural network models can be used reliably with the already available out-of-event data for binary and multi-class classification. The automatic feature learning capabilities brings an additional value on top of non-neural classification methods. The MLP-CNN results show that there is still a roam for improvement on top of the best accuracy achieved. The rest of the paper is organized as follows. We summarize related work in Section "Related Work" . Section "Convolutional Neural Network" presents the convolutional neural model. In Section "Experimental Settings" , we describe the dataset and training settings of the models. In Section "Results" presents our results and analysis. We conclude and discuss future work in Section "Conclusion and Future Work" .

Question: Was any variation in results observed based on language typology?
Answer: It is observed some variability - but not significant. Bert does not seem to gain much more syntax information than with type level information.