# Adapting Data-Driven Research to the Fields of Social Sciences and the Humanities
## 8 Data Acquisition and Preprocessing
### Deep Learning for Text Classification  
The final iteration of the domain-specificity component drew upon a transformer architecture to perform the classification process. As outlined in Figure 2, the classifier used a Bidirectional Encoder Representations from Transformers (BERT) language model [36] with twelve transformer layers and a maximum sequence length of 512 tokens to create a contextualized representation of the input document, in which each token was represented by a 768-dimensional vector. Text classification tasks usually only consider the BERT symbol for classification output ([CLS]), which starts each document and provides a 768-dimensional vector representation of its content. Since combining multiple hidden layers has been shown to improve accuracy [36], we pooled the output of the last four hidden layers and fed it through two linear layers, which then provided the final classification result.  
We used the Hugging Face transformer library ([[https://huggingface.co](https://huggingface.co)]([https://huggingface.co](https://huggingface.co)), accessed on 21 February 2021) in conjunction with PyTorch ([[https://pytorch.org](https://pytorch.org)]([https://pytorch.org](https://pytorch.org)), accessed on 21 February 2021) to implement the classifier and evaluated it using the following two pre-trained transformer models:  
* German BERT base ([[https://huggingface.co/bert-base-german-cased](https://huggingface.co/bert-base-german-cased)]([https://huggingface.co/bert-base-german-cased](https://huggingface.co/bert-base-german-cased)), accessed on 21 February 2021): A case-sensitive BERT transformer that has been trained on over 10 GB of textual data comprising the German Wikipedia dump (6 GB), the OpenLegalData dump (2.4 GB), and 3.6 GB of news articles.  
Figure 2: Architecture of the deep learning classifier.  
* Multilingual BERT ([[https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment](https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment)]([https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment](https://huggingface.co/nlptown/bert-base-multilingual-uncased-sentiment)), accessed on 21 February 2021): A case-insensitive, multilingual BERT model that has been fine-tuned for sentiment analysis on product reviews in English, Dutch, German, French, Spanish, and Italian.  
Experiments that aimed at optimizing the model's hyperparameters yielded the following settings: a batch size of four, a dropout value of 0.3 to prevent overfitting, an Adam optimizer with a learning rate of \(\alpha=3\times 10^{-5}\), a binary cross-entropy (BCE) loss function, and a training limit of 35 epochs with early stopping if the validation loss does not improve.  
Both models were fine-tuned on the domain-specificity classification task using the gold standard created by our domain experts. In our experiments, the German BERT Base model achieved an F1 score of 92.5% with both a precision and a recall of 92.5%. The multilingual BERT model even topped this performance with an F1 score of 95.8% (recall: 95.6%, precision: 95.8%). This considerable boost in performance came at the cost of a lower explainability of the provided classification results.