# PIMA: Parameter-Shared Intelligent Media Analytics Framework for Low Resource Languages
## 3 Materials and Methods
### Media Analytics Classification Dataset  
The media analytics classification dataset contains textual multi-task sentence and paragraph-level annotations for sentiment, irony, hate speech, and emotion detection. The mixture of sentence- and paragraph-level annotations in the dataset facilitates learning from minimum examples that contain any or all the task specific objectives, while allowing to generalize on a bigger body of text. The content was randomly selected from each source to gather individual articles, posts, and text clips as uniformly distributed as possible among each source, user, and topic. Non-Greek content was excluded and achieving a diversity of content for each class was prioritized in the process. Manual annotation was carried out by three expert annotators working professionally as journalists and was divided into batches to control the process. During the initial annotation period, a set of guidelines were devised and three rounds of 100 documents from each domain were assigned to each annotator. After each round, we evaluated the agreement of the annotators on these documents with [PERSON]'s kappa [44], as shown in Table 2, and discussed the disagreeing texts and adjusted the guidelines. The final dataset achieved a [PERSON]'s kappa of 0.82, which was measured among the final annotations on the complete dataset after the three pilot annotation rounds, indicating a high agreement rate. The final guidelines contained semantic descriptions of all targets in all four subjectivity aspects, limiting false annotations in specific types of texts for all categories, and included examples to resolve ambiguity where necessary. A few characteristic examples are described in the following: \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Domain** & **Category** & & \\ \hline \multirow{2}{*}{Internet} & Blogs & News sites, & \\  & 26,507 & aggregators & \\ \hline \multirow{2}{*}{Social media} & Facebook & Twitter & Instagram \\  & 144,813 & 144,530 & 18,798 \\ \hline \multirow{2}{*}{Press} & Free press & Sports & Economics & Local \\  & 5212 & 1930 & 38,656 & 31,149 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Alignment (domain adaption) dataset statistics. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Annotation Rounds** & **Cohen’s K** & **Interpretation** \\ \hline Round 1 & 0.72 & Substantial agreement \\ Round 2 & 0.77 & Substantial agreement \\ Round 3 & 0.86 & Near perfect agreement \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Cohen’s K during the initial annotation process for guideline adjustment. * Promotional texts in which the launch of a new product was announced were unilaterally defined as neutral sentiment, non ironic, non hate speech containing, and of no emotion. * Stock market reports from official financial outlets were described as neutral sentiment and no emotion. * Ironic texts usually contain hyperbole, contradictions, metaphors and/or proverbs, blunt language, language tone change (e.g., formal in an informal context), etc. Commonly, not all guidelines were all inclusive in terms of the subjectivity aspects, as the content can vary drastically. Evidently, there would be exceptions in these guidelines and given the expert status of the annotators they had the freedom to deviate from these guidelines in such scenarios. To resolve disagreements, we followed a majority voting strategy on texts in which at least two of the four subjectivity labels were agreed upon by all three annotators, while discarding the disagreeing instances. The final dataset contained 14579 annotated examples with all four subjectivity labels. Table 3 shows the labels and the distribution per label for each task. The evident imbalance in label distribution, especially for the irony and hate speech labels, can be attributed to the focus on domain-related topics of the collected content.