Introduction
Time-critical analysis of social media data streams is important for many application areas. For instance, responders to humanitarian disasters (e.g., earthquake, flood) need information about the disasters to determine what help is needed and where. This information usually breaks out on social media before other sources. During the onset of a crisis situation, rapid analysis of messages posted on microblogging platforms such as Twitter can help humanitarian organizations like the United Nations gain situational awareness, learn about urgent needs of affected people at different locations, and decide on actions accordingly BIBREF0 , BIBREF1 .
Artificial Intelligence for Disaster Response (AIDR) is an online platform to support this cause BIBREF2 . During a disaster, any person or organization can use it to collect tweets related to the event. The total volume of all tweets can be huge, about 350 thousand tweets per minute. Filtering them using keywords helps cut down this volume to some extent. But, identifying different kinds of useful tweets that responders can act upon cannot be achieved using only keywords because a large number of tweets may contain the keywords but are of limited utility for the responders. The best-known solution to address this problem is to use supervised classifiers that would separate useful tweets from the rest.
Classifying tweets to identify their usefulness is difficult because: tweets are short – only 140 characters – and therefore, hard to understand without enough context; they often contain abbreviations, informal language and are ambiguous; and, finally, determining whether the tweet is useful in a disaster situation and identifying required actions for relief operations is a hard task because of its subjectivity. Individuals differ on their judgement about whether a tweet is useful or not and sometimes whether they belong to one topical class or another especially when there is information in a tweet that would be classified into multiple topical classes. Given this ambiguity, a computer cannot agree with annotators at a rate that is higher than the rate at which the annotators agree with each other. Despite advances in natural language processing (NLP), interpreting the semantics of the short informal texts automatically remains a hard problem.
To classify disaster-related tweets, traditional classification approaches use batch learning with discrete representation of words. This approach has three major limitations. First, in the beginning of a disaster situation, there is no event labeled data available for training. Later, the labeled data arrives in small batches depending on the availability of geographically dispersed volunteers. These learning algorithms are dependent on the labeled data of the event for training. Due to the discrete word representations, they perform poor when trained on the data from previous events (out-of-event data). The second limitation is the offline learning style that inputs the complete labeled data and train a model. This is computational expensive in a disaster situation where labeled data is coming in batches. One would need to train a classifier from scratch every time a new batch of labeled data arrives. Thirdly, these approaches require to manually engineered features like cue words and TF-IDF vectors BIBREF3 for learning.
Deep neural networks (DNNs) are based on online learning mechanism and have the flexibility to adaptively learn from new batches of labeled data without requiring to retrain from scratch. Due to their distributed word representation, they generalize well and make better use of the previously labeled data from other events to speed up the classification process in the beginning of a disaster. DNNs automatically learn latent features as distributed dense vectors, which generalize well and have shown to benefit various NLP tasks BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 .
In this paper, we propose a convolutional neural network (CNN) for the classification task. CNN captures the most salient $n$ -gram information by means of its convolution and max-pooling operations. On top of the typical CNN, we propose an extension that combines multilayer perceptron with a CNN. We present a series of experiments using different variations of the training data – event data only, out-of-event data only and a concatenation of both. Experiments are conducted for binary and multi-class classification tasks. For the event only binary classification task, the CNN model outperformed in four out of five tasks with an accuracy gain of up to 4.5 absolute points. In the scenario of no event data, the CNN model shows substantial improvement of up to 18 absolute points over the several non-neural models. This makes the neural network model an ideal choice in early hours of a disaster for tweet classification. When combined the event data with out-of-event data, we see similar results as in the case of event only training.
For multi-class classification, the CNN model outperformed in similar fashion as in the case of binary classification. Our variation of the CNN model with multilayer perceptron (MLP-CNN) performed better than it's CNN counter part. In some cases, adding out-of-event data drops the performance. To reduce the effect of large out-of-event data and to make the most out of it, we apply a simple event selection technique based on TF-IDF and select only those events that are most similar to the event under consideration. We then train the classifiers on the concatenation of the event plus selected out-of-event data. The performance improves only for the event with small event data.
To summarize, we show that neural network models can be used reliably with the already available out-of-event data for binary and multi-class classification. The automatic feature learning capabilities brings an additional value on top of non-neural classification methods. The MLP-CNN results show that there is still a roam for improvement on top of the best accuracy achieved.
The rest of the paper is organized as follows. We summarize related work in Section "Related Work" . Section "Convolutional Neural Network" presents the convolutional neural model. In Section "Experimental Settings" , we describe the dataset and training settings of the models. In Section "Results" presents our results and analysis. We conclude and discuss future work in Section "Conclusion and Future Work" .
Related Work
Studies have analyzed how Twitter can be useful during major disasters so as to gain insight into the situation as it unfolds BIBREF0 , BIBREF8 , BIBREF9 . A number of systems have been developed to classify, extract, and summarize crisis-relevant information from social media; for a detailed survey see BIBREF3 . Cameron, et al., describe a platform for emergency situation awareness BIBREF10 . They classify interesting tweets using an SVM classifier. Verma, et al., use Naive Bayes and MaxEnt classifiers to find situational awareness tweets from several crises BIBREF11 . Imran, et al., implemented AIDR to classify a Twitter data stream during crises BIBREF2 . They use a random forest classifier in an offline setting. After receiving every minibatch of 50 training examples, they replace the older model with a new one.
There is growing interest in recent years for DNNs and word embeddings with application to myriad of NLP problems. The emergence of tools such as word2vec BIBREF6 and GloVe BIBREF12 have enabled NLP researchers to learn word embeddings efficiently and use them to train better models.
Collobert et al. BIBREF4 presented a unified DNN architecture for solving various NLP tasks including part-of-speech tagging, chunking, named entity recognition and semantic role labeling. They showed that DNNs outperform traditional models in most of the tasks. They also proposed a multi-task learning framework for solving the tasks jointly.
Kim BIBREF13 and Kalchbrenner et al. BIBREF14 used convolutional neural networks (CNN) for sentence-level classification tasks (e.g., sentiment/polarity classification, question classification) and showed that CNNs outperform traditional methods (e.g., SVMs, MaxEnts). Despite these recent advancements, the application of CNNs to disaster response is novel to the best of our knowledge.
Convolutional Neural Network
In order to classify short and noisy Twitter messages effectively, possibly in the absence of any in-event training data, a classification model should use a distributed representation of words, which results in improved generalization, and should learn the key features at different levels of abstraction automatically. To this end, we use a Convolutional Neural Network (CNN) as proposed by Kim BIBREF13 .
Figure 1 demonstrates how a CNN works with an example tweet “guys if know any medical emergency around balaju area you can reach umesh HTTP doctor at HTTP HTTP”. Each word in the vocabulary $V$ is represented by a $D$ dimensional vector in a shared look-up table $L$ $\in $ $^{|V| \times D}$ . $L$ is considered a model parameter to be learned. We can initialize $L$ randomly or using pretrained word embedding vectors like word2vec BIBREF15 .
Given an input tweet $\mathbf {s} = (w_1, \cdots , w_T)$ , we first transform it into a feature sequence by mapping each word token $w_t \in \mathbf {s}$ to an index in $L$ . The look-up layer then creates an input vector $\mathbf {x_t}\in ^{D}$ for each token $w_t$ , which are passed through a sequence of convolution and pooling operations to learn high-level feature representations.
A convolution operation involves applying a filter $\mathbf {u} \in ^{L.D}$ to a window of $L$ words to produce a new feature
$$h_t = f(\mathbf {u} . \mathbf {x}_{t:t+L-1} + b_t)$$   (Eq. 4)
where $\mathbf {x}_{t:t+L-1}$ denotes the concatenation of $L$ input vectors, $b_t$ is a bias term, and $f$ is a nonlinear activation function (e.g., $, \tanh $ ). We apply this filter to each possible $L$ -word window in the tweet to generate a feature map $\mathbf {h}^i = [h_1, \cdots , h_{T+L-1}]$ . We repeat this process $N$ times with $N$ different filters to get $N$ different feature maps (i.e., $L$0 ). We use a wide convolution BIBREF14 (as opposed to narrow), which ensures that the filters reach the entire sentence, including the boundary words. This is done by performing zero-padding, where out-of-range ( $L$1 $L$2 1 or $L$3 $L$4 $L$5 ) vectors are assumed to be zero.
After the convolution, we apply a max-pooling operation to each feature map.
$$\mathbf {m} = [\mu _p(\mathbf {h}^1), \cdots , \mu _p(\mathbf {h}^N)] $$   (Eq. 5)
where $\mu _p(\mathbf {h}^i)$ refers to the $\max $ operation applied to each window of $p$ features in the feature map $\mathbf {h}^i$ . For instance, with $p=2$ , this pooling gives the same number of features as in the feature map (because of the zero-padding). Intuitively, the filters compose local $n$ -grams into higher-level representations in the feature maps, and max-pooling reduces the output dimensionality while keeping the most important aspects from each feature map.
Since each convolution-pooling operation is performed independently, the features extracted become invariant in locations (i.e., where they occur in the tweet), thus acts like bag-of- $n$ -grams. However, keeping the order information could be important for modeling sentences. In order to model interactions between the features picked up by the filters and the pooling, we include a dense layer of hidden nodes on top of the pooling layer
$$\mathbf {z} = f(V\mathbf {m} + \mathbf {b_h}) $$   (Eq. 6)
where $V$ is the weight matrix, $\mathbf {b_h}$ is a bias vector, and $f$ is a non-linear activation. The dense layer naturally deals with variable sentence lengths by producing fixed size output vectors $\mathbf {z}$ , which are fed to the final output layer for classification. Formally, the output layer defines a Bernoulli distribution:
$$p(y|\mathbf {s}, \theta )= (y| (\mathbf {w^T} \mathbf {z} + b )) $$   (Eq. 7)
where $$ refers to the sigmoid function, and $\mathbf {w}$ are the weights from the dense layer to the output layer and $b$ is a bias term. We fit the models by minimizing the cross-entropy between the predicted distributions $\hat{y}_{n\theta } = p(y_n|\mathbf {s}_n, \theta )$ and the target distributions $y_n$ (i.e., the gold labels).
$$J(\theta ) = \hspace{0.0pt} - \sum _{n} y_n \log \hat{y}_{n\theta } + (1-y_n) \log \left(1- \hat{y}_{n\theta } \right)$$   (Eq. 9)
Word Embedding and Fine-tuning
In our CNN model, we intend to avoid manual feature engineering efforts by using word embeddings as the only features. As mentioned before, we can initialize the embeddings $L$ randomly and learn them as part of model parameters by backpropagating the errors to the look-up layer. One issue with random initialization is that it may lead the training algorithm to get stuck in local minima. On the other hand, one can plug the readily available embeddings from external sources (e.g., Google embeddings BIBREF15 ) in the CNN model and use them as features without tuning them further for the task, as is done in any other machine learning model. However, this approach does not exploit the automatic feature learning capability of NN models, which is one of the main motivations of using them. In our work, we use the pre-trained word embeddings to better initialize our models, and we fine-tune them for our task in training, which turns out to be beneficial. More specifically, we initialize the word vectors in $L$ in two different ways.
1. Google Embedding: Mikolov et al. BIBREF15 propose two log-linear models for computing word embeddings from large (unlabeled) corpus efficiently: a bag-of-words model CBOW that predicts the current word based on the context words, and a skip-gram model that predicts surrounding words given the current word. They released their pre-trained 300-dimensional word embeddings (vocabulary size 3 million) trained by the skip-gram model on part of Google news dataset containing about 100 billion words.
2. Crisis Embedding: Since we work on disaster related tweets, which are quite different from news, we have also trained domain-specific embeddings (vocabulary size 20 million) using the Skip-gram model of word2vec tool BIBREF6 from a large corpus of disaster related tweets. The corpus contains $57,908$ tweets and $9.4$ million tokens. For comparison with Google, we learn word embeddings of 300-dimensions.
Incorporating Other Features
Although CNNs learn word features (i.e., embeddings) automatically, we may still be interested in incorporating other sources of information (e.g., TF-IDF vector representation of tweets) to build a more effective model. Additional features can also guide the training to learn a better model. However, unlike word embeddings, we want these features to be fixed during training. This can be done in our CNN model by creating another channel, which feeds these additional features directly to the dense layer. In that case, the dense layer in Equation 6 can be redefined as
$$\mathbf {z} = f(V^{\prime } \mathbf {m^{\prime }} + \mathbf {b_h})$$   (Eq. 13)
where $\mathbf {m^{\prime }}=\mathbf {[m;y]}$ is a concatenated (column) vector of feature maps $\mathbf {m}$ and additional features $\mathbf {y}$ , and $V^{\prime }$ is the associated weight matrix. Notice that by including this additional channel, this network combines a multi-layer perceptron (MLP) with a CNN.
Experimental Settings
In this section, we first describe the dataset that is used for the classification task. We then present the TF-IDF based features which are used to train the non-neural classification algorithms. In the end, we describe the model settings and training settings of non-neural and neural classification models.
Datasets
We use data from multiple sources: (1) CrisisLex BIBREF16 , (2) CrisisNLP BIBREF17 , and (3) AIDR BIBREF2 . The first two sources have tweets posted during several humanitarian crises and labeled by paid workers. The AIDR data consists of tweets from several crises events labeled by volunteers.
The dataset consists of various event types such as earthquakes, floods, typhoons, etc. In all the datasets, the tweets are labeled into various informative classes (e.g., urgent needs, donation offers, infrastructure damage, dead or injured people) and one not-related or irrelevant class. Table 1 provides a one line description of each class and also the total number of labels from all the sources. Other useful information and Not related or irrelevant are the most frequent classes in the dataset. Table 2 shows statistics about the events we use for our experiments. In order to access the difficulty of the classification task, we calculate the inter-annotator agreement (IAA) scores of the datasets obtained from CrisisNLP. The California Earthquake has the highest IAA of 0.85 and Typhoon Hagupit has the lowest IAA of 0.70 in the events under-consideration. The IAA of remaining three events are around 0.75. We aim to reach these levels of accuracy.
Data Preprocessing: We normalize all characters to their lower-cased forms, truncate elongations to two characters, spell out every digit to D, all twitter usernames to userID, and all URLs to HTTP. We remove all punctuation marks except periods, semicolons, question and exclamation marks. We further tokenize the tweets using the CMU TweetNLP tool BIBREF18 .
Data Settings: For a particular event such as Nepal earthquake, data from all other events under-consideration plus All others (see Table 2 ) are referred to as out-of-event data. We divide each event dataset into train (70%), validation (10%) and test sets (20%) using ski-learn toolkit's module BIBREF19 which ensured that the class distribution remains reasonably balanced in each subset.
Feature Extraction: We extracted unigram, bigram and trigram features from the tweets as features. The features are converted to TF-IDF vectors by considering each tweet as a document. Note that these features are used only in non-neural models. The neural models take tweets and their labels as input. For SVM classifier, we implemented feature selection using Chi Squared test to improve estimator's accuracy scores.
Non-neural Model Settings
To compare our neural models with the traditional approaches, we experimented with a number of existing models including: Support Vector Machine (SVM), a discriminative max-margin model; Logistic Regression (LR), a discriminative probabilistic model; and Random Forest (RF), an ensemble model of decision trees. We use the implementation from the scikit-learn toolkit BIBREF19 . All algorithms use the default value of their parameters.
Settings for Convolutional Neural Network
We train CNN models by optimizing the cross entropy in Equation 7 using the gradient-based online learning algorithm ADADELTA BIBREF20 . The learning rate andparameters were set to the values as suggested by the authors. Maximum number of epochs was set to 25. To avoid overfitting, we use dropout BIBREF21 of hidden units and early stopping based on the accuracy on the validation set. We experimented with $\lbrace 0.0, 0.2, 0.4, 0.5\rbrace $ dropout rates and $\lbrace 32, 64, 128\rbrace $ minibatch sizes. We limit the vocabulary ( $V$ ) to the most frequent $P\%$ ( $P\in \lbrace 80, 85, 90\rbrace $ ) words in the training corpus. The word vectors in $L$ were initialized with the pre-trained embeddings. See Section "Word Embedding and Fine-tuning" .
We use rectified linear units (ReLU) for the activation functions ( $f$ ), $\lbrace 100, 150, 200\rbrace $ filters each having window size ( $L$ ) of $\lbrace 2, 3, 4\rbrace $ , pooling length ( $p$ ) of $\lbrace 2,3, 4\rbrace $ , and $\lbrace 100, 150, 200\rbrace $ dense layer units. All the hyperparameters are tuned on the development set.
Results
For each event under consideration, we train classifiers on the event data only, on the out-of-event data only, and on a combination of both. We conduct experiments for the binary and multi-class classification task. For former, we merge all informative classes to create one general Informative one.
We initialized the CNN model using two types of pre-trained word embeddings. Crisis Embeddings CNN $_{I}$ : trained on all tweet crisis data Google Embeddings CNN $_{II}$ trained on the Google News dataset. The CNN model then fine-tuned the embeddings using the training data.
Binary Classification
Table 3 (left) presents the results of binary classification comparing several non-neural classifier with the CNN-based classifier. For the scenario of training on the event only data, CNNs performed better than all non-neural classifiers in every event.
The CNN performed substantially better than the non-neural model SVM by a margin of up to 4% when trained on the out-of-event data only. This shows robustness of the CNN model in learning from the out-of-event data when no event data is available. This can be very helpful during the early hours of a crisis when no event-specific labeled data is available.
When combined event data with out-of-event data, CNN also performed better than the non-neural models. However, comparing different data settings for CNN, we saw mixed results. In most of the cases, the performance dropped in comparison to the event only training. The large size of the out-of-event data down weights the benefits of the event data and skewed the probability distribution towards the out-of-event data. Table 3 (right) presents confusion matrix of SVM and CNN $_I$ classifiers trained and evaluated on the Nepal earthquake data. SVM prediction is inclined towards Informative class whereas CNN predicted more instances as non-informative than informative. In the case of out-of-event training, SVM predicted most of the instances as informative. Thus achieved high recall but very low precision. CNN, on the other hand, achieved quite balanced precision and recall.
To summarize, the neural network based classifier out-performed non-neural classifiers in all data settings. The performance of the models trained on out-of-event data are (as expected) lower than that in the other two training settings. However, in case of the CNN models, the results are reasonable to the extent that out-of-event data can be used to predict tweets informativeness when no event data is available. It is also worth mentioning here that aside from out-performing non-neural classifiers, neural network models do not require any feature engineering and learn features automatically from the data. Comparing CNN $_{I}$ with CNN $_{II}$ , we did not see any system consistently better than other. For further experimentation and comparison, we only consider the CNN $_{I}$ trained on crisis embedding.
Multi-class Classification
For the purpose of multi-class classification, we mainly compare the performance of two variations of the CNN-based classifier, CNN $_{I}$ and MLP-CNN $_{I}$ (combining multi-layer perception and CNN), against an SVM classifier.
All labeled data from other events may not be useful for the event under-consideration. Based on this intuition, we apply a data selection technique to select the data from out-of-events that is most liked by the event. To achieve this, we trained a classifier on the event data only and predicted the label of each tweet of the out-of-event data. We selected tweets that are correctly predicted by the event classifier and added them (M $_{event+adapt}$ ) to the event data for the training of the classifier for multi-class classification task.
Table 4 summarizes the accuracy and macro F1 scores of the multi-class classification task. Similar to the results of binary classification task, the CNN model outperformed SVM in almost data settings. The most promising results are combing MLP and CNN really improve performance of our system. The results on training with event plus out-of-event data did not have a clear improvement over the event only model. The results dropped slightly in some cases. In addition, when using simple domain adaptation technique, we get rid of the noisy tweets from out-of-event data and improve the performance of our models. The M $_{event+adapt}$ system in Table 4 shows the results. The domain adapted system consistently performed better than its respective baseline and with the system trained on event data only. The macro F1 scores show the per class performance of each classifier. The Nepal earthquake has the lowest per class classification performance. This could be due to the imbalance in the training data as can be seen in Table 2 .
In Table 5 , we show the confusion matrix of the CNN-MLP model tested on the Nepal Earthquake data. The class Other useful information is the second largest class in the data. The classifier got biased and learned to predict other tweets in this class. Not related or irrelevant is the most confused class among all classes. The precision-recall curve in Table 5 shows that Not related or irrelevant and Donation and Volunteering are easier to be predicted than other classes. Their AUC scores are $0.87$ and $0.70$ respectively. Meanwhile, AUC scores of Other useful information and Infrastructure and utilities class are lower than the random (0.5) level.
Conclusion and Future Work
We presented a deep neural network model for binary and multi-class classification tasks and showed that one can reliably use out-of-event data for the classification of new event when no in-event data is available. A convolution neural network model performed consistently for all five events under consideration, and worked better than all non-neural models in most of the cases. The performance of the classifiers degraded when out-of-event training samples was added to training samples from event data. Thus, we recommend using out-of-event training data during the first few hours of a disaster only after which the training data related to the event should be used.
In future, we would like to improve the performance of the system using domain adaptation by either model weighting, separately building models for event and out-of-event and combining them, or by data selection, or intelligently selecting the out-of-event data that is best suited the event data.

Question: what was their baseline comparison?
Answer: Support Vector Machine (SVM), Logistic Regression (LR), Random Forest (RF)