## 5 Evaluation
### User-Level Classification  
For the evaluation of user classification, we wanted to test whether Hypothesis 1 (_meso features improve user classification in the absence of micro features_) and Hypothesis 4 (_meso_, _and community detection in particular, can improve classification compared to only using \(meso_{i}\) and \(meso_{r}\) features_) held true. In our case, Hypothesis 1 was tested by comparing the accuracy of the CRANK model to a simpler model that labeled each user using the majority label of his/her content. Hypothesis 4 was tested by comparing the CRANK model to CRANK without community detection.  
The following models were compared:  
* Average content (AvgContent) (_micro_): Content was applied the same label as the majority of content by the same user, and users were labeled according to the majority label of their content.
* Naive majority (AvgNeigh) (_meso_\(i\) or _meso_\({}_{r}\), depending on the context): Users were labeled with the majority label in their group of neighbors in the network. Unlabeled content was given the label of its creator.
* Majority in the community (AvgComm) (_meso_\(c\)): Users were grouped into communities, and each user was given the majority label of the users in their community. Content was given the label of its creator.
* CRANK without community detection (_meso_, or _meso_\(i\), depending on the context): The CRANK model described in Algorithm 1, but using original edges instead of applying community detection.
* CRANK (_meso_\(c\)): Before applying Algorithm 1, the communities between users were extracted and converted to user edges, i.e., users in the same community were connected by an edge.  
The results of the evaluation are shown in Table 3, where the highest value for each row is presented in bold. It also highlights in grey the highest value when the average content was ignored.  
If we focus on the results for the RT Mind dataset, we could conclude that CRANK significantly improved the classification in all scenarios, especially with lower \(ratio_{c}\) values. In other datasets, where the network of users was sparser and less cohesive, CRANK outperformed all the models, except for the average of content. This was expected, since _meso_ features in these datasets were rather weak, and the content mean and median values were close to one. In particular, the difference between the CRANK model and the baseline in the HCR dataset was relatively small (0.02). That indicated that there was little penalty to using CRANK even when there were few _meso_ edges between users. In the OMD dataset, which had low agreement between neighbors, the difference between CRANK and the baseline was higher, and it did not decrease with higher values of \(ratio_{u}\). This confirmed our suspicions that the _meso_ features in this dataset were not useful for our purposes.  
Regarding Hypothesis 4, we observed that CRANK outperformed its variant without community detection in most of the cases. The exceptions were cases where most of the user labels were known. In those cases, the accuracy of both methods was extremely high (above 0.95). This difference could be explained by interpreting community detection as an aggregate over several users. In general, all the users in a community shared the same sentiment. However, some members would have a different label from the majority in their community (i.e., outliers). Often, those outliers were users that were connected to users of other communities with a different sentiment. That information was lost when aggregating, so for those outliers, community detection was actually detrimental. The fewer users that were left unlabeled, the higher the effect of those outliers would be. Aggregating in those cases presented a higher variance, which combined with the high accuracy values also lowered the mean compared to not aggregating. Nevertheless, we could conclude that \(meso_{e}\) features improved user classification in most cases.