## 5 Evaluation
### Content-Level Classification  
For the evaluation of content classification, we wanted to test whether Hypothesis 2 (_micro_ features improve content classification over pure _contextless_ features), Hypothesis 3 (_meso features improve content classification in the absence of micro features_), and Hypothesis 4 (_meso\({}_{e}\), and community detection in particular, can improve classification compared to only using \(meso_{i}\) and \(meso_{r}\) features_) held true. To do so, we compared the performance of the following classifiers:  
* Simon [40] (_contextless_): A sentiment analysis model based on semantic similarity. The model can be trained with different datasets. In our evaluation, we compared with the Simon model trained on different datasets: STS, Vader, Sentiment140, and a combination of all three. * Sentiment140 ([[https://www.sentiment140.com](https://www.sentiment140.com)]([https://www.sentiment140.com](https://www.sentiment140.com))) service (_contextless_): This is a public sentiment analysis service, tailored to Twitter. It outputs three labels: positive, negative, and neutral. This results in lower accuracy for the negative and positive labels. In fact, of all the models tested, this was the one with the lowest accuracy. If all tweets labeled neutral by the service are ignored, its accuracy reaches standard levels (around 60%). Unfortunately, this means that around 80% of tweets have to be ignored. * Meaningcloud ([[https://www.meaningcloud.com/](https://www.meaningcloud.com/)]([https://www.meaningcloud.com/](https://www.meaningcloud.com/))) Sentiment Analysis (_contextless_): An enterprise service that provides several types of text analysis, including sentiment analysis. It poses the same restrictions for evaluation as Sentiment140, as it provides positive, negative, and neutral labels. Fortunately, the subjectivity detection of this service for our datasets was better than that of Sentiment140. * Average Content (AvgContent) (_micro_): Content is applied the same label as the majority of content by the same user, and users are labeled according to the majority label of their content. * Naive majority (AvgNeigh) (\(meso_{i}\) or \(meso_{r}\), depending on the context): Users are labeled with the majority label in their group of neighbors in the network. Unlabeled content is given the label of its creator. \begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline
**Dataset** & \(ratio_{e}\) & **Model \(ratio_{u}\)** & **AvgComm** & **AvgContent** & **AvgNeigh** & **CRANK** & **CrankNoComm** \\ \hline RT Mind & 0.25 & 0.25 & 0.536 & 0.692 & 0.540 & **0.883** & 0.815 \\  & & 0.50 & 0.661 & 0.670 & 0.651 & **0.950** & 0.939 \\  & & 0.75 & 0.954 & 0.642 & 0.791 & 0.962 & **0.985** \\  & 0.50 & 0.25 & 0.536 & 0.860 & 0.540 & **0 Majority in the community (AvgComm) (\(meso_{e}\)): Users are grouped into communities, and each user is given the majority label of the users in their community. Content is given the label of its creator. * CRANK without community detection (\(meso_{r}\) or \(meso_{i}\), depending on the context): The CRANK model described in Algorithm 1, but using original edges instead of applying community detection. * CRANK (\(meso_{e}\)): Before applying Algorithm 1, the communities between users are extracted and converted to user edges, i.e., users in the same community are connected by an edge. * Label propagation [38] (Speriosu): Based on the results reported in the original paper for these datasets. We compared the accuracy of each of these models for several combinations of known content and user labels (\(ratio_{c}\) and \(ratio_{u}\)). Table 4 shows a summary of the mean accuracy for each combination. We also provide a graph of the mean accuracy and standard deviation of each model (Figure 3-5). Similarly to the user-classification case, if we focus on the RT Mind dataset, the CRANK algorithm outperformed all other models by a wide margin. In general, the baseline models that used social context had higher accuracy in this dataset than any contextlessapproach.