# Adapting Data-Driven Research to the Fields of Social Sciences and the Humanities
## 8 Data Acquisition and Preprocessing
### Knowledge-Aware Text Classification  
The next iteration of the domain-specificity component drew upon the formalized domain knowledge to calculate the probabilistic affiliation of a new and unknown text with a given set of categories. In the presented project, two categories--media criticism and not media criticism--were used, corresponding to relevant and irrelevant content.  
To gather the needed domain-specificity information, domain experts (i.e., communication scientists) collected a gold-standard corpus of both media-critical and non-media-critical documents. Special attention was paid to finding document pairs dealing with the same topic, where one document contained media criticism while the other one did not, as this helped to minimize the risk of topic-specific bias. In addition, the domain experts aimed to include a wide range of topics, such as sports, direct democracy, affirmative actions, disabilities, the Holocaust, offshore leaks, the [PERSON], Islam, climate change, the energy transition, and airplane accidents, in the gold standard. All selected documents were published in a Swiss medium after 2010. The media-critical documents in the sample only considered texts for which media criticism was clearly identified as the main topic and was not just peripherally mentioned. The final gold standard comprised 503 media-critical documents and 643 corresponding non-media-critical counterparts, which were used for training the classifier.  
This iteration used the Naive Bayes classification algorithm due to its simplicity and explainability, which allowed the determination of the reasons for a correct (or incorrect) classification result. Being able to observe and correct the process if necessary helped in building the domain expert's trust in the classifier, and this was therefore identified as a crucial step in creating the system.  
The overall probability of an unknown text was derived from the probabilities of the contained terms: First, the document was converted into a \"bag of words\" representation that also considered n-grams and skip-grams. A stopword filter and a frequency-based filter, which remove terms that have been used less than three times in all collected documents, were applied to speed up the computation time. The prior probability for each element of this bag of words was received from the knowledge base calculated previously. The overall probability of an article being media critical given its included terms \(P(M|t)\) was calculated with the cross-product of its terms' prior probability \(P(M)\) and the likelihood \(P(t|M)\) that they were contained divided by its evidence \(P(t)\).  
\[P(M|t)=\frac{\mathit{priorprobability}\times\mathit{likelihood}}{\mathit{evidence}}=\frac{P(M)P(t|M)}{P(t)} \tag{1}\]  
In this scenario, prior probability describes the general probability of an unknown text containing media criticism according to the training data, while likelihood refers to the probability of an unknown text's term being related to a media-critical text. Meanwhile, evidence means the probability of an unknown text's terms being contained in any of the trained categories (in this case media criticism \(M\) and not media criticism \(\
eg M\)).  
\[\mathit{evidence}=P(t)=P(M)P(t|M)+P(\
eg M)P(t|\
eg M) \tag{2}\]We further introduced ensemble heuristics, where the classification functions as a baseline, and further knowledge sources, such as (i) source whitelists, (ii) black- and whitelists, and (iii) named entity annotation and text patterns, function as regulators. Documents that contain no or only a few keywords received a penalty on the estimated probability of containing media criticism. The ensemble methods applied in this iteration achieved results with a recall of over 71%, which is already considerably better than the keyword-based approach.