# Learning Word Embeddings with Chi-Square Weights for Healthcare Tweet Classification
## 4 Experimental Method
### Evaluation and Results  
We completed the \(\chi^{2}\) statistical test on the two Twitter datasets. Boxplots for the \(\chi^{2}\) values in both datasets are shown in Figure 1. As we can see, most of the words in the two corpora have a very low \(\chi^{2}\) value. We list the words in Table 3 that ranked highest by \(\chi^{2}\) value. Words with higher \(\chi^{2}\) value are  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Data** & **Healthcare-Related** & **Healthcare-Unrelated** & **Total** \\ \hline
**Train** & 868 & 1301 & 2169 \\
**Test** & 217 & 325 & 532 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Tweet counts for the healthcare dataset (from [20]).  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Data** & **Influenza-Related** & **Influenza-Unrelated** & **Total** \\ \hline
**Train** & 2148 & 1609 & 3757 \\
**Test** & 537 & 402 & 939 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Tweet counts for the influenza dataset (from [3]).  
recognizable as plausible keywords for the identification of the health-related tweets. Using the \(\chi^{2}\) statistics in Algorithms I and II, the result of the experiment is shown in Table 4.  
Since we assume equal importance for the identification of both of the two classes, related versus unrelated, we choose accuracy, the commonly used evaluation criteria, as the metric to measure the classification performance as shown in Formula (7).  
\[accuracy(y_{lab},y_{pred})=\frac{1}{n_{test}}\sum\limits_{i=1}^{n_{test}}1(y_{ pred}=y_{lab}) \tag{7}\]  
Since tweets in both classes (related versus unrelated) contain the keywords of the topic, it is not surprising that the keywords-based approach in the tf-idf + SVM baseline behaves poorly for both  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Method** & **Healthcare Dataset** & **Influenza Dataset** \\ \hline tf-idf + SVM baseline & 59.96 & 57.18 \\ Skip-gram + CNN baseline & 66.61 & 66.99 \\ CBOW + CNN baseline & 69.00 & 66.67 \\ Algorithm I + CNN & 69.19 & 72.31 \\ Algorithm II + CNN & 69.93 & 72.84 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of testset classification accuracy across the two datasets using word embeddings from various models. SVM: support vector machine; CNN: convolutional neural network; CBOW: continuous bag of words.  
Figure 1: Boxplot of the values of the Chi-square (\(\chi^{2}\)) statistical test for the healthcare dataset and the influenza dataset.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Healthcare Dataset** & **Influenza Dataset** \\ \hline
1 & headache & sick \\
2 & sick & vaccine \\
3 & allergies & throat \\
4 & feeling & fear \\
5 & flu & news \\
6 & surgery & swineflu \\
7 & cramps & bird \\
8 & throat & shot \\ \hline \hline \end{tabular}
\end{table}
Table 3: Words with highest \(\chi^{2}\) value for both datasets.  
datasets. For the two Word2 Vec baselines, the CBOW model performs better than the skip-gram model for the smaller (healthcare) dataset; they have very similar results in the influenza dataset (which is a larger dataset). Overall, Algorithms I and II improve over the CBOW baseline model by 1.35% and 9.23% respectively. We can see that Algorithm I which chooses the word with the maximum \(\chi^{2}\) value also performs well in terms of accuracy. As we noted earlier, we have a small context window size of 3(\(b=1\)). When the context window is larger, more context words are included. It might not be optimal to choose only the word with the maximum statistical measurement score to form the context representation. Our experimental results indicate that the \(\chi^{2}\) weighting scheme of Algorithm II generally outperforms the others.