# Large Scale Implementations for Twitter Sentiment Classification
## 7 Conclusions  
In the context of this work, we have presented a tool that analyzes microblogging messages regarding their sentiment using machine learning techniques. This novel distributed framework was implemented in Hadoop as well as in Spark. Our algorithm exploits the hashtags and emoticons inside a tweet, as sentiment labels, and proceeds to a classification procedure of diverse sentiment types in a parallel and distributed manner. In addition, we utilize Bloom filters to compact the storage size of intermediate data and boost the performance of our algorithm. In addition, some classification algorithms are implemented in the Apache Spark cloud framework using Apache Spark's Machine Learning library, entitled MLlib. Through an extensive experimental evaluation, we prove that our system is efficient, robust and scalable.  
In the near future, we plan to extend and improve our framework by exploring more features that may be added in the feature vector and will increase the classification performance. Furthermore, we wish to explore more strategies for \(F_{H}\) and \(F_{C}\) bounds in order to achieve better separation between the HFWs and CWs. One other future work will be the experimentation with different clusters so as to better evaluate Hadoop's and Spark's performance in regards to time and scalability. In addition, we plan to investigate the effect of different Bloom filter bit vector sizes, in classification performance and storage space compression. Moreover, we plan to compare the classification performance of our solution with other classification methods, such as Naive Bayes or Support Vector Machines.  
Another future consideration is the adoption of aforementioned heuristics (e.g., the occurrence of emoticons) for handling complex semantic issues, such as irony that is typical of messages in Twitter. Such similar works are the ones in [58; 59; 60; 61]. The corresponding studies investigate the automatic detection of irony based on lexical features as well as the adoption of lexical and pragmatic factors on machine learning effectiveness for identifying sarcastic utterances. Finally, we plan on creating an online service that takes advantage of Spark Streaming, which is an Apache Spark library for manipulating streams of data that provides users with real-time analytics about sentiments of requested topics.  
**Author Contributions:** [PERSON], [PERSON], [PERSON], [PERSON], [PERSON] and [PERSON] conceived of the idea, designed and performed the experiments, analyzed the results, drafted the initial manuscript and revised the final manuscript.  
\begin{table}
\begin{tabular}{l c c c c} \hline
**Features** & **Positive** & **Negative** & **Neutral** & **Total** \\ \hline
**Complete Feature Vector** & 0.717 & 0.75 & 0.617 & 0.696 \\
**w/o Unigrams** & 0.628 & 0.602 & 0.537 & 0.592 \\
**w/o Bigrams** & 0.714 & 0.769 & 0.629 & 0.705 \\
**w/o Trigrams** & 0.732 & 0.77 & 0.643 & 0.716 \\
**w/o User** & 0.718 & 0.751 & 0.618 & 0.698 \\
**w/o Hashtag** & 0.721 & 0.739 & 0.608 & 0.692 \\
**w/o URL** & 0.72 & 0.748 & 0.619 & 0.697 \\
**w/o POS Tags** & 0.716 & 0.748 & 0.617 & 0.695 \\ \hline \end{tabular}
\end{table}
Table 13: F-Measure for Ternary Classification for Naive Bayes for 12,500 tweets.  
\begin{table}
\begin{tabular}{l c c c c} \hline
**Features** & **Positive** & **Negative** & **Neutral** & **Total** \\ \hline
**Complete Feature Vector** & 0.628 & 0.592 & 0.542 & 0.591 \\
**w/o Unigrams** & 0.596 & 0.457 & 0.451 & 0.51 \\
**w/o Bigrams** & 0.616 & 0.6 & 0.546 & 0.59 \\
**w/o Trigrams** & 0.649 & 0.623 & 0.572 & 0.618 \\
**w/o User** & 0.625 & 0.6 & 0.54 & 0.592 \\
**w/o Hashtag** & 0.612 & 0.591 & 0.526 & 0.58 \\
**w/o URL** & 0.613 & 0.598 & 0.537 & 0.585 \\
**w/o POS Tags** & 0.646 & 0.585 & 0.512 & 0.587 \\ \hline \end{tabular}
\end{table}
Table 12: F-Measure for Ternary Classification for Logistic Regression for 12,500 tweets.