Assessing In-context Learning and Fine-tuning for Topic Classification of German Web Data (2024)

Abstract: Researchers in the political and social sciences often rely on classification models to analyze trends in information consumption by examining browsing histories of millions of webpages. Automated scalable methods are necessary due to the impracticality of manual labeling. In this paper, we model the detection of topic-related content as a binary classification task and compare the accuracy of fine-tuned pre-trained encoder models against in-context learning strategies. Using only a few hundred annotated data points per topic, we detect content related to three German policies in a database of scraped webpages. We compare multilingual and monolingual models, as well as zero and few-shot approaches, and investigate the impact of negative sampling strategies and the combination of URL & content-based features. Our results show that a small sample of annotated data is sufficient to train an effective classifier. Finetuning encoder-based models yields better results than in-context learning. Classifiers using both URL & content-based features perform best, while using URLs alone provides adequate results when content is unavailable.

heading: 1 Introduction
text: Text classification of webpages is used to understand information consumption by categorizing large collections of individuals’ browsing histories (e.g., Stier et al. 2022a). By categorizing webpages, researchers can identify patterns of online news consumption (Flaxman et al., 2016) and quantify exposure to populist sentiments (Stier et al., 2022b). Analyzing browsing histories by topic often necessitates "finding the needle in the haystack", as typically just a small fraction of webpage visits correspond to a given domain, such as news sources (Wojcieszak et al., 2022). Therefore, identifying the few relevant pages among numerous unrelated visits makes manual labeling impractical. Machine learning classifiers are often used as
an automated and scalable alternative (Stier et al., 2022b).
Since the introduction of the transformer architecture, fine-tuning pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) has seen widespread adoption in text classification tasks. Applications include classifying public opinions about policies in digital media (Viehmann et al., 2023) and identifying protest-related content in newspaper articles (Re et al., 2021; Sebők and Kacsuk, 2021). Further applications encompass sentiment analysis on social media posts (Manias et al., 2023) and advertising (Jin et al., 2017). However, finetuning classifiers still requires hundreds to thousands of manually labeled documents. Given the multilingual nature of the web and the noisy data resulting from the scraping process, compiling a representative training set remains a complex and time-consuming task. Generative models such as Llama (Touvron et al., 2023) and Mistral (Jiang et al., 2023) are often inherently multilingual and can generalize to completely unseen tasks without the need for fine-tuning, potentially making them a promising alternative.
In this study, we investigate the use of large language models (LLMs) for the task of binary topic classification across a corpus of scraped webpages. We evaluate our approach by identifying webpages that provide information on three specific German policies discussed during data collection: (1) a policy introduced to combat child poverty, (2) the promotion of renewable energy, and (3) the amendment of cannabis legislation. We compare the classification accuracy between multilingual (Conneau et al., 2020) and monolingual (Chan et al., 2020) pre-trained language models by fine-tuning them on manually labeled data. Our analysis extends to generative models (Touvron et al., 2023; Chung et al., 2022), evaluating few-shot prompting for document classification and assessing the impact of demonstrator sampling strategies.
238

heading: 2 Related Work
text: Political and social sciences researchers increasingly use topic classification to filter large collections of webpages derived from browsing histories (Guess, 2021; Stier et al., 2022a). This task is commonly modeled as binary or multiclass classification, assigning text segments to one or more predefined categories. Until recently, researchers in these applied fields have relied on traditional NLP methods such as naive Bayes classifiers (Stier et al., 2022a) and logistic regression models (Guess, 2021).
The adaptation of BERT models created new opportunities by improving classification accuracy. For instance, Viehmann et al. (2023) fine-tuned BERT models to classify opinions on policies in digital media. Similarly, Re et al. (2021) explored the use of BERT variants for classifying sentences in newspaper articles to detect protest-related content. Osnabrügge et al. (2023) applied a logistic regression model for classifying the topics of parliamentary speeches. Research on webpage classification also includes the use of URL features (Kan and Thi, 2005), extracted content (Jin et al., 2017), graph representations (Wu et al., 2015), and visual features (Xu and Miller, 2015).

heading: 2.1 Feature-based Learning
text: Historically, text classification involved feature engineering by (1) extracting a vector representation of the text, followed by (2) feeding the extracted features into a classifier to determine the final label. Support vector machines (D’Orazio et al., 2014) and naive Bayes models (Scharkow, 2013), often combined with frequency-based tf-idf vectors, were the standard tools. More recently, approaches also rely on techniques such as Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), to obtain dense representations of vocabulary items.

heading: 2.2 Contextualized Embeddings
text: Recent advancements in text classification have been driven by models like BERT (Devlin et al., 2019) based on the transformer architecture, which utilize attention mechanisms (Vaswani et al., 2017) and are trained on extensive unlabeled text datasets through unsupervised pre-training prior to finetuning on downstream tasks such as document classification. For instance, mBERT was pre-trained on data from Wikipedias in 104 languages. XLMRoBERTa (Conneau et al., 2020), a multilingual
extension of RoBERTa (Zhuang et al., 2021), is pretrained on text from 100 languages. Subsequent fine-tuning of BERT models by replacing the last layer with a classification head for the final prediction has become a common approach (Re et al., 2021; Gnehm and Clematide, 2020; Viehmann et al., 2023; Manias et al., 2023).

heading: 2.3 Models Pre-trained on German Texts
text: A considerable amount of research has been dedicated to exploring text classification tasks specifically for the German language (Viehmann et al., 2023; Scharkow, 2013). Although not all recent studies utilize transformer models for German text classification (Graef, 2021), the majority of research underscores the superiority of BERT models in this domain (Gnehm and Clematide, 2020). DBMDZ BERT is comparable in size to BERT-base but is trained on the German segments of the OPUS corpus and Wikipedia. GBERT (Chan et al., 2020) is another German BERT variant that outperforms multilingual models and other German-trained BERT variants (Idrissi-Yaghir et al., 2023; Niklaus et al., 2023; Bornheim et al., 2021). GBERT includes additional data and implements training enhancements (Chan et al., 2020), as does the GELECTRA model (Clark et al., 2020), which is designed for more efficient learning by enabling the model to learn from entire sentences, rather than just the masked tokens.

heading: 2.4 In-context Learning
text: Large generative models like FLAN (Chung et al., 2022), Mistral (Jiang et al., 2023), and LLaMa (Touvron et al., 2023) are also transformerbased but use stacked decoder blocks instead of the encoder blocks used by BERT. Encoder blocks extract dense vector representations, used as features for classification tasks. Decoder blocks predict the next token to generate output sequences, allowing these models to perform different tasks due to their flexible output schema.
Generative models have demonstrated remarkable generalization across a broad spectrum of NLP tasks by incorporating the instruction directly into the input prompt, often alongside a few labeled examples, thereby eliminating the need for parameter updates. Due to their large training corpora, generative models typically possess some multilingual capabilities. For instance, FLAN is a model family based on the T5 model architecture (Chung et al., 2022), able to follow instructions in mul-
tiple languages, including English, German, and French. Larger models, like those based on the LLaMA (Touvron et al., 2023) architecture, are further optimized through reinforcement learning from human feedback (Ouyang et al., 2022; Bai et al., 2022), improving cross-domain generalization and reasoning skills. Aya (Üstün et al., 2024) and Vicuna are further examples. The former is trained on 101 languages including German, while the latter is fine-tuned on user-shared conversations, primarily in English.1
While neural networks have become the stateof-the-art text classification approach, current research lacks a thorough evaluation of LLMs for identifying topic-related content on German webpages. Here, we provide a comprehensive study to fill this gap, including a comparison to traditional feature-based approaches.

heading: 3 Dataset
text: For our experiments, we use a corpus of scraped webpages annotated by topic. We describe the data collection and annotation process in Section 3.1. The topic labels correspond to three German policies that were of interest during the period of data collection: (1) basic child support policy (Kindergrundsicherung), introduced to combat child poverty, (2) energy transition policy (Förderung erneuerbarer Energien), designed to promote renewable energy, and the (3) cannabis legalization amendment (Cannabislegalisierung). We refer to these policies as the children, energy, and cannabis policies throughout this paper. Our dataset contains substantially more topic-unrelated than relevant webpages. This exemplifies a common challenge in the social, political, and communication sciences: finding relevant content within a vast database of unrelated webpages.
1https://sharegpt.com

heading: 3.1 Data Collection and Annotation
text: The browsing traces are obtained as part of a broader project in which 1, 228 participants of a commercial web-tracked panel take part in an online experiment, during which they are instructed to inform themselves about the three policy topics (see Appendix A and C for details). In total, the participants visit 267k quasi-unique URLs. Given that only 1, 324 unique URLs (775 after filtering) are annotated as policy-related across the three topics, a research assistant augments our training data by manually searching the web for further policyrelated webpages. An additional 297 high-quality positive cases are added for each topic in this way (77, 83, and 137, respectively, for the topics children, energy, and cannabis).
Data from the collected URLs is scraped using the Python package requests2 and the plain text content is extracted from the HTML using the Python package selectolax.3
For each of the three topics, the browsing trace data are manually annotated with binary labels (topic-related or non-relevant) at the URL level. Given the amount of data, we employ a multilevel filtering and refinement approach, moving from hostname categories down to hostnames and finally individual URLs, at each step removing nonrelevant URLs. For details on the annotation procedure, see Appendix A.
After annotation of the successfully scraped webpages (156k out of 267k URLs), our highconfidence data set is comprised of 214 (children), 227 (energy), 228 (cannabis) webpages that are related to the respective topic, and 4,106 (children), 4,572 (energy), 3,857 (cannabis) non-relevant webpages. As a result of the multi-level annotation strategy, we also obtain 143k additional URLs with low-confidence labels that are predominantly neg-
2https://pypi.org/project/requests/ 3https://pypi.org/project/selectolax
ative cases (e.g., searches, YouTube videos, and social media posts), which we use in our evaluation of a real-world application scenario of classifying noisy web data. For further ablative testing on noisy data, we also construct an extended test set with low-confidence labels.

heading: 3.2 Data Preprocessing
text: We describe the processing steps for compiling the datasets for training and evaluation, including sampling train and test examples, as well as segmenting long webpages. We filter out cases where we were unable to retrieve the content, to allow for a 1-to-1 comparison of classification performance based on URLs alone versus using content as an additional feature.
Training and Test Sets. We partition the dataset for each topic into training and test sets, allocating 90% of the positive examples to training and 10% to testing, resulting in three datasets for three binary classification tasks (see Table 1). Only URLs with high-confidence labels are used for the training and test sets (see Section 3.1). The positive cases added during manual augmentation are used exclusively for training.
For our initial experiments, we aim for an even proportion of positive and negative cases in the training and test sets (we discuss suitable sampling strategies in Section 4.1). Further negative examples that are not included are assigned to a second, unbalanced test set (unbl) consisting of predominantly negative examples. This second data set mirrors the original proportion of topic-related and unrelated webpages in our data but still contains only high-confidence URLs. Finally, to assess the
performance of the classifiers under real-world conditions, we construct an extended test (extd) set comprised of low-confidence labels. This test set also includes difficult-to-scrape webpages, such as search engines, often resulting in non-useful HTML content due to disabled JavaScript. This dataset is even more unbalanced, containing an overwhelming number of negative cases.
Document Splitting. Due to the limited context window of the test LLMs (see Table 2), we divide webpage content into chunks using a recursive text splitter4. We utilize a maximum chunk size of 384 tokens for all models, including an overlap of 64 tokens. For each chunk, we assign the label of the parent URL.

heading: 4 Methods
text: We model the detection of topic-related content as a binary classification task for each of the three topics. We compare the F1-scores of fine-tuned encoder models (supervised) and in-context learning strategies (few/zero-shot) against suitable baselines. Figure 1 shows a schematic overview of the supervised training and classification pipeline. The evaluated LLMs are listed in Table 2. We make the code for our experiments publicly available.5
For supervised fine-tuning of monolingual and multilingual models, we experiment with using URL-based features on their own and in combination with content. Due to the small number of webpages related to the three topics, we also experiment with different strategies to sample from the
4https://python.langchain.com/docs/ 5https:/github.com/julianschelb/Topic-Classification
large number of negative examples. For in-context learning classification methods, we evaluate multiple models in zero- and few-shot scenarios, comparing different task demonstrator sampling strategies for the latter.
To aggregate the predicted labels for chunks into document level labels during inference, we assign a positive label to webpages if the label of at least one chunk is predicted to be topic-relevant.

heading: 4.1 Sampling Negative Examples
text: To address the imbalance of negative and positive examples in our dataset, we investigate three sampling strategies for negative training examples.
Random. We select a random subset of webpages classified as negative, aiming for an even number of topic-related and unrelated webpages in our training dataset.
Stratified. To prevent an overrepresentation of webpages from frequent domains, we group them into strata based on their domain, selecting the 128 most frequent URLs for individual groups and consolidating all remaining ones into a ’others’ group.
Cluster-based. Like Sun et al. 2023, we test KNN sampling. We create document vectors using TF-IDF with a dimensionality of 10,000, which we then reduce to 100 dimensions using PCA. Given the unknown total number of clusters, we utilize DBSCAN for clustering and sample webpages from each cluster, including the noise cluster.

heading: 4.2 Supervised Classification
text: We evaluate several monolingual encoder models that are pre-trained specifically on German texts, as well as multilingual encoder models that include at
least a portion of German text in their pre-training data. For fine-tuning, we use the same parameters across all models: a learning rate of 2× 10−5 over a maximum of 3 epochs. We use a warm-up of 500 steps at the beginning of training and a weight decay of 0.01.
We train one URL-based classifier and one combined URL & content classifier per topic. Since URLs often contain parts of the article title, categories, or search engine optimization (SEO) keywords, we expect them to be useful for classification (Aljofey et al., 2022; Kan and Thi, 2005). To avoid overfitting on specific domains only the path and parameter sections of the URL are utilized (see Figure 1).
Baselines. For URL-based classification, we use linear interpolation and backoff (LIB) as the baseline (Abramson and Aha, 2012). For URL & content classification, we use support vector machine (SVM) classifiers with TF-IDF vectors for feature extraction, similar to what is frequently employed in the literature (Idrissi-Yaghir et al., 2023; Kan and Thi, 2005; D’Orazio et al., 2014).

heading: 4.3 Zero- and Few-Shot Classification
text: We evaluate multiple generative models using incontext learning for classification tasks in both zero-shot and few-shot scenarios. We include Aya (Üstün et al., 2024) and two Vicuna variants (Chiang et al., 2023), as well as three FLANT5 variants (Chung et al., 2022) to assess the performance scaling with model size. Due to the limited context window of FLAN-T5, we evaluate them exclusively in a zero-shot setting. Due to the long inference times, we opted to only evaluate on the balanced test set. Our prompts combine a task de-
scription with "Yes" or "No" response instructions to simplify the parsing of the output. Figure 2 shows the used prompt template. We convert responses to lowercase to map the models’ output more easily to a binary label. For answer generation, we set the temperature to 0.3, top_k to 50, and top_p to 0.95. While the generative models tend to have longer context windows and would allow for larger webpage chunks, we use the same chunks as the supervised classification for comparison.
Demonstrator Sampling. Since the selection of task demonstrators included in the few-shot prompt affects prediction quality (Liu et al., 2022; Peng et al., 2024), we evaluate multiple sampling strategies: (1) random sampling over the training set, (2) random sampling with balanced classes to address class imbalance by ensuring equal representation of each class, and (3) KNN-based sampling, which selects training examples similar to the input (Sun et al., 2023). We calculate the cosine distance
based on embeddings extracted using a sentencetransformer (Reimers and Gurevych, 2019).

heading: 5 Results and Discussion

heading: 5.1 Supervised Classification Results
text: We evaluate all models using URL-only and URL & content as features and report the F1 scores for the three test datasets (test, unbalanced, and extended) and three topics in Table 3.
GELECTRA-Large, using URL & content features, achieves the best average F1 score of 0.430 across all topics on the complete test set (see Table 6), making it the overall best-performing model. Analyzing the results by topic, GELECTRA-Large achieves the best F1 scores of 0.583 for the children topic and 0.151 for the energy topic. Meanwhile, German-BERT-Base achieves the best score for the cannabis topic with an F1 score of 0.578.
We discuss the impact of feature selection and negative sampling methods and analyze performance differences between monolingual and multi-
lingual models, as well as base and large models.
URL & content. While the URL alone can be an adequate feature for many applications, our findings show that integrating webpage content improves classification performance. Across all topics and models, the average F1 score improved by 40.8% on the complete test set.
Classifiers on the children topic experienced the most notable improvement, with F1 scores increasing by 4.4% on the test set, 42.1% on the unbalanced set, an substantial 977.3% on the extended set, and 731.1% on the complete set, indicating that content helps the classifier to generalize. The
energy topic also showed enhanced performance with the inclusion of content features. Interestingly, the cannabis topic exhibited a decrease in average performance. This decrease may be attributed to ground truth labels being annotated at the URL level rather than the content level. Webpages on this topic might utilize URLs with highly expressive keywords, enabling the URL-only classifier to perform very effectively. Alternatively, as our manual error analysis suggests (see 5.3), webpages discussing this topic but lacking topic-relevant keywords in the URLs might have been missed during the annotation process.
In summary, classifiers trained on URL & content perform better, especially on the challenging extended test set.
Performance Comparison: Test Sets. All models perform well on the balanced test set with both URL & content-based features, but their performance significantly deteriorates on the unbalanced and extended test sets. The average performance across all topics decreases by 65.7% from the balanced to the unbalanced set and by 73.1% to the extended set. Although recall remains high, the drop in precision indicates an increase in false positives, confirming the greater difficulty of these datasets due to lower quality scraped content and less reliable labels. The results show that the classifiers struggle with noise in the extracted webpage content introduced by the scraping process.
Performance Comparison: Topics. Cannabisrelated webpages are generally the easiest to detect, while energy-related webpages are the most chal-
lenging. This observation aligns with our intuition, as cannabis represents a more specific topic. In contrast, the energy topic is considerably broader, overlapping with a range of areas that are unrelated to the topic of renewable energy, such as climate change. The precision-recall curves based on all available data, as depicted in Figure 3, further support this observation.
Monolingual vs. Multilingual Models. Monolingual models achieve a mean F1 score 25.9% higher than multilingual models on the complete test set across all topics when using URL & content features. Comparing the best monolingual model, GELECTRA-Large, with the best multilingual model, xlm-roberta-large, GELECTRA-Large achieves an F1 score that is 22.4% higher on the unbalanced dataset, 60.0% higher on the extended dataset, and 49.5% higher on the complete test set.
Negative Sampling. In Table 4, we report the results comparing three negative sampling strategies. We find that random sampling and stratified sampling perform comparably, with stratified sampling yielding slightly better performance overall.
Model Size and Runtime Analysis. Larger models generally outperform their base variants, with modest gains. On the unbalanced dataset, the average F1 score increases by 9.4% (from 0.32 to 0.35), while on the extended dataset, scores see a more substantial boost of 25% (from 0.24 to 0.30). These improvements highlight the benefits of larger models in handling more complex and varied data. However, this increased performance comes at a significant cost in processing time. As shown in Table 6, large variants achieve better F1 scores but
process only ~19 webpage chunks per second, compared to ~63 chunks for the base variants. This 28% gain in F1 score comes with a 200% increase in processing time. The SVM baseline is the fastest at ~1000 chunks per second but has the lowest F1 score. Measurements were conducted using an Nvidia Tesla P100 GPU and an Intel Xeon Gold 6132 CPU @ 3.700GHz.

heading: 5.2 Zero- and Few-shot Results
text: Our results demonstrate that zero-shot and few-shot methods deliver good performance (see Table 5). The best zero-shot model, determined by averaging the F1 scores across the three topics, is Vicuna 7b, which achieves an average F1 score of 0.930. The overall best model is Vicuna 13b with few-shot and random sampling of task demonstrators, which achieves an average F1 score of 0.970. For sampling task demonstrators, random and random balanced sampling strategies work better than KNNbased sampling. However, few-shot classification remains consistently inferior to fine-tuning, which is therefore the preferred approach for achieving optimal results if labeled data is available.

heading: 5.3 Manual Error Analysis
text: We perform a manual error analysis on the predictions of the best performing classifier, GELECTRA-Large with random negative sampling, by randomly sampling 50 misclassified webpage chunks from both the unbalanced and extended test sets per topic, yielding 300 chunks in total. The errors are categorized by type in Table 7 (for a more detailed breakdown, see Appendix E).
In 42 instances, the classifier’s prediction is correct and the ground truth is incorrect (GT error). This is not surprising since the extended test set consists primarily of webpages with low-confidence labels and the manual labeling is URL-based, while
the classifier analyzes individual chunks within the scraped content. In 85 instances, webpage chunks contained very general information pertaining to the topic but were not truly relevant (topic related). Examples include pharmacies selling cannabis, online solar panel shops, and energy price comparison portals. Conversely, in 50 instances, the classifier identified webpages from ministries or institutions discussing other laws as topic-relevant (law related). Both cases highlight the inherent difficulty in distinguishing topical information from specific legal content. Furthermore, we find that the classifier is sensitive to words like "legal," "Umwelt" (environment), and "Verkehr" (transportation), resulting in 56 misclassified cases (unrelated). Additionally, in 52 cases, the classifier misclassified boilerplate chunks, such as navigation elements or cookie banners, likely because all chunks inherit the webpage’s URL-based label (boilerplate). This caused some chunks to be labeled as topicrelevant without containing relevant information, introducing noise to the training dataset. Finally, in 15 cases, web scraping or preprocessing failed to produce meaningful content, which confused the classifier (content error). Errors include warnings about disabled JavaScript, login-protected content, or encoding issues.

heading: 6 Conclusion
text: We compare the performance of fine-tuned encoder models against in-context learning strategies for the classification of topic-related content. Using only a few hundred positively annotated data points per topic, we detect content related to three German policies in a database of scraped webpages. The best supervised classifier, GELECTRA-Large, using URL & content features, achieves an average F1 score of 0.430 over all topics, performance varies by topic. It performs well on the children and cannabis topics but performs suboptimal in terms of precision for the energy topic.
All fine-tuned models achieve strong performance on the high-quality balanced test set, re-
gardless of using URL or content-based features. However, performance declines substantially on lower-quality and unbalanced data, with high recall but lower precision due to more pages being falsely labeled as topic-related. While recall remains high across all topics and test sets, precision drops considerably, leading to a substantial number of false positives, which indicates that the model is overly sensitive to keywords that are topic-related but also occur in other contexts. Webpage content proved to be a strong signal for classification over URLbased baselines, and classifiers that combined URL & content-based features perform best. In cases where content-based analysis is infeasible, URLbased classifiers can provide an adequate baseline performance, although the precision-recall tradeoff in settings with real-world data requires a careful approach. However, a manual error analysis revealed that the classifiers struggle to distinguish between weak and strong relations to the topic, with URL-based labels leading to incorrect associations of boilerplate texts with the topic. An investigation of more elaborate chunk pooling and combination strategies in future work is needed. Additionally, incorporating loosely topic-related negative examples into the training data would likely improve classifier precision by enabling better differentiation between relevant and non-relevant instances. For instance, online shops that advertise cannabis or solar panels are relevant to the topic in general but not in the sense of political policy discussion.
Our evaluation shows high accuracy for zeroand few-shot prompting without fine-tuning, indicating their potential in data-constrained situations. Few-shot learning can be viable when runtime is less critical, but labeled data is expensive. However, fine-tuning encoder-based models generally yields better results and should be given preference over in-context learning for annotating large datasets.
Future Work. It is likely that classifier precision can be enhanced by filtering out topic-unrelated chunks and training a content-only classifier to remove unrelated content. To address the limited number of positive examples, data augmentation appears like a fruitful addition to the pipeline. For in-context learning, advanced prompting methods such as prompt chaining and chain-of-thought prompting are likely to enhance LLM reasoning.
Limitations
URL-based Labeling. Since we generated training data based on URL-level labeling of websites as a proxy for content-based labeling for reasons of feasibility, it is likely that our data (and therefore our findings) are biased. While the manual error analysis indicated that just 14% of errors are ground-truth errors, this amount is non-negligible. In settings where resources are available for proper content-based labeling, it is likely that this error can be reduced.
Website Chunking. Since we assign URL-level labels to webpage chunks, it is likely that chunks in the training data are labeled incorrectly. As described in Section 3.2, we split webpage content into chunks due to the 512-token input limit for our classifiers, with each chunk inheriting the URL’s label. Thus, if a webpage is labeled as topic-relevant, all chunks receive a positive label, even if some contain irrelevant text, such as navigation elements or cookie banners. As a result of this, the model sometimes associates boilerplate text with the positive class. The pragmatic solution here is to go with the times and use models with larger input sizes to avoid chunking altogether.
Scraping-induced Noise. Another source of noise stems from the web scraping process. For example, our web scraper did not support JavaScript, causing many webpages to display warnings or malfunction. In these cases, the URL label remains positive, indicating topic-related content, but the scraper failed to retrieve that content, further introducing noise in the training data. Similar issues occur with login protected webpages, dynamic content, cookie banners, YouTube videos, and PDFs.
Ethics Statement
The browsing traces from which we scraped the web data were provided by Bilendi GmbH, which hosts a web tracking panel. The company adheres to EU GDPR regulations, and participants were fully informed about the data collection process, including the option to temporarily disable tracking for privacy reasons. A letter of information was provided, and consent was requested from all participants upon first contact and then thereafter at each additional contact point. Ethics approval has been received by the University of Konstanz IRB under the number IRB23KN02-003/w.

heading: AI Policy Statement
text: In conducting our research and preparing this paper, we utilized AI assistants, specifically ChatGPT and GitHub Copilot. ChatGPT was employed for paraphrasing and refining the authors’ original content to enhance clarity and readability, without suggesting new content. GitHub Copilot assisted in coding tasks by providing code suggestions and completions.

heading: Acknowledgements
text: We owe many thanks to Katharina Jäger and Corinna Nitsch for their help with data annotation, and Anton Pogrebnjak for support in data scraping. We also extend our gratitude to Juhi Kulshrestha and Celina Kacperski for helpful discussions. This research was funded by the Deutsche Forschungsgemeinschaft (DFG – German Research Foundation) under Germany‘s Excellence Strategy – EXC2035/1 – 390681379. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

heading: A Data collection
text: The URLs forming the basis for the corpus of this study were obtained as part of a broader project in which individuals of a commercial web-tracked panel were invited to participate in an online experiment. Participants (N=1228) were randomly assigned to one of 3 groups: a control group, and two intervention groups (both instructed to search about the policy topics, but only one with a financial incentive), with weekly instructions to inform themselves about the three policy topics during a 20-30h window. The visited URLs were recorded (N= 761K), and the content was scraped.
Children. The "Kindergrundsicherung" (basic child support) policy aims to combat child poverty by providing a fixed amount, income-dependent supplement, and educational benefits.6
6https://www.bmfsfj.de/bmfsfj/service/gesetze/gesetzzur-einfuehrung-einer-kindergrundsicherungund-zur-aenderung-weiterer-bestimmungenbundeskindergrundsicherungsgesetz-bkg–230650
Energy. The EEG 2023 (Erneuerbare-EnergienGesetz, Renewable Energy Sources Act) aims to increase the share of renewable energies in gross electricity consumption to at least 80% by 2030.7
Cannabis. The CanG 2023 (Cannabisgesetz, Cannabis Control Act) will legalize the private cultivation of cannabis by adults for personal use and collective non-commercial cultivation.8

heading: B URL Annotation Process
text: During the 20h-30h windows of the experiment, participants visited ∼ 761K URLs comprising ∼ 267K quasi-unique URLs (i.e., the sum of the total unique URLs per topic). To obtain training examples, the URL annotation protocol followed a multi-level strategy:
1. Hostname category: Hostnames (N = 17, 207) were classified according to three categorizations: (1) base categories provided by the commercial panel (N = 48), and (2) the simplified categories (N = 46) and (3) IAB categories (N = 405) gathered via the Webshrinker service. Three researchers (two postdocs and one research assistant) indicated if the base and simplified categories were irrelevant to the topic, i.e., were unlikely to contain policy-related information; two annotators (one postdoc and one research assistant) did so for the IAB categories. Only URLs from unanimously irrelevant categories were discarded.
2. Hostname: We extracted the unique hostnames corresponding to the remaining URLs (homepages were excluded). One research assistant indicated that the hostname was irrelevant (i.e., unlikely to contain information relevant to the topic). If so, the hostname was discarded. As an exception, the next level directly included URLs corresponding to a curated list of news hostnames (N ≈ 700, Stier et al., 2020) because they are likely to include topic-related information (so checking those domains manually is unnecessary).
3. URL: URLs were sorted into categories (see Table 2). URLs that fall into the “Other” cat-
7https://www.bundesregierung.de/bregde/schwerpunkte/klimaschutz/novelle-eeg-gesetz-20232023972
8https://www.bundesgesundheitsministerium.de/themen/ cannabis/faq-cannabisgesetz
egory were not annotated (14.7%) because most would require visiting the URL. One of the authors checked the hostnames and judged them to be not very likely to contain relevant information. One annotator indicated if the remaining URLs were related to the policy topic.
For the experiments in the study, three annotated URL categories were excluded: (1) web searches because the post-hoc scraping would alter the results the participants encounter, (2) social media because the content is not accessible (via scraping), and (3) YouTube because the API was used instead of web-scraping (and the content does not strictly correspond to webpages).
In total, 4983 URLs for children, 5782 for energy, and 4834 for cannabis manually annotated URLs were used in this study; only 139, 180, and 76, respectively, were relevant to each topic.

heading: C Distribution of unique URLs
text: The distribution of annotated URLs according to their category and topic is presented in Table 1. During the multistep annotation process, some categories, such as social media and web searches, are discarded before manual analysis due to their unlikely relevance to the topic (see column "Used"). Categories with high-confidence labels (used = yes) include URLs with SEO-optimized titles, news without SEO-optimized titles, Wikipedia, and keyworded domains, while web searches, social media, YouTube shorts and videos, and other miscellaneous URLs have only low-confidence labels (used = no). The latter categories form the basis of our extended test set. The URL counts in Table 1 indicate the total number of URLs annotated. The number of webpages in our dataset used in our experiments is lower because cases where content cannot be retrieved using our web scraper are excluded.

heading: D Manually-augmented data
text: Given the scarcity of topic-relevant URLs among the annotated cases, a research assistant was instructed to complement our training dataset using the Google search engine. Three query terms were based on how the policy topics were referred to in the online survey experiment: "kindergrundsicherung", "gesetze zur förderung erneuerbarer energien", and “cannabis legalisierung". The process was twofold:
1. First, the assistant downloaded approximately 15 non-news results related to the topic among the top 30, limiting the search until July 31st, 2023.
2. Second, they performed nine monthlyrestricted news searches between November 1st, 2022, and July 31st, 2023, downloading those relevant to the topic among the top 10 results (top 20 for cannabis).
In total, 77, 83, and 137 webpages were added for each topic, respectively.

heading: E Manual Error Analysis
text: In our manual error analysis of GELECTRA-Large with random negative sampling, we examine 300 misclassified webpage chunks. Identifying these errors helps us refine labeling, enhance preprocessing, and adjust the model to better distinguish relevant from irrelevant content. See Table 2 for a detailed breakdown.
This analysis highlights areas for improvement in our model. For instance, in 52 cases, boilerplate text (e.g., navigation elements, cookie banners) is predicted as topic-relevant by the classifier, likely due to URL-based ground truth labels. The 512- token input limit necessitates chunking the webpage content. For URLs with positive labels, all chunks, sometimes including boilerplate, inherit the URL’s positive label. This causes the model to associate boilerplate text with the positive class during training. Using models with larger input sizes could mitigate this issue.
Noise from the web scraping process is another concern, as indicated by the 15 examples in our sample. Our web scraper does not support JavaScript, leading to errors when retrieving content from some webpages. This highlights the importance of URL-only classifiers as a fallback.

heading: Future Work
text: An investigation of more elaborate chunk pooling and combination strategies in future work is needed. Additionally, incorporating loosely topic-related negative examples into the training data would likely improve classifier precision by enabling better differentiation between relevant and non-relevant instances