Behind Closed Words: Creating and Investigating the forePLay Annotated Dataset for Polish Erotic Discourse
Authors: Anna Kołos, Katarzyna Lorenc, Emilia Wiśnios, Agnieszka Karlińska
Accepted at: acl25

Behind Closed Words: Creating and Investigating the forePLay Annotated Dataset for Polish Erotic Discourse

Warning: This paper contains examples of sexually explicit material

Anna Kołos*1, Katarzyna Lorenc*1, Emilia Wiśnios*2, Agnieszka Karlińska1
1 NASK National Research Institute
2 Independent Researcher
* Equal Contribution

Abstract

The surge in online content has created an urgent demand for robust detection systems, especially in non-English contexts where current tools demonstrate significant limitations. We introduce forePLay, a novel Polish-language dataset for erotic content detection, comprising over 24,000 annotated sentences. The dataset features a multidimensional taxonomy that captures ambiguity, violence, and socially unacceptable behaviors. Our comprehensive evaluation demonstrates that specialized Polish language models achieve superior performance compared to multilingual alternatives, with transformer-based architectures showing particular strength in handling imbalanced categories. The dataset and accompanying analysis establish essential frameworks for developing linguistically-aware content moderation systems, while highlighting critical considerations for extending such capabilities to morphologically complex languages.1

1 Introduction

The rapid growth of user-generated content online has created an urgent need for effective tools to detect and moderate harmful and inappropriate material. Traditional methods, such as manual review by editors or flagging by users, often fall short due to the sheer volume of content published daily. As a result, there has been an increasing reliance on automated solutions powered by advanced language models and natural language processing techniques.

Securing large language models (LLMs) against the generation of harmful content is another critical concern (Mahomed et al., 2024; Wu et al., 2024). While traditional content moderation tools are useful for user-generated content, they are less effective as input/output guardrails for LLMs due to their inability to adapt to new policies and distinguish between user-generated and AI-generated content (Inan et al., 2023). As a result, there is a growing need for datasets and specialized tools that can be integrated with LLMs to prevent the generation of explicit or harmful material, while maintaining their functionality across different applications.

While much of the previous work has focused on detecting toxicity, abusiveness, offensive language, or hate speech (Pavlopoulos et al., 2020; Gehman et al., 2020; MacAvaney et al., 2019), these categories do not encompass the full spectrum of undesired content (Markov et al., 2023). Erotic material, in particular, poses a significant risk, especially to minors. Detecting such content is essential not only for creating a safer online environment but also for upholding ethical standards. Although the Digital Services Act (DSA) primarily addresses non-consensual pornography and age-verification requirements, many online platforms, particularly those targeting minors, enforce stricter moderation policies that extend beyond legal obligations. Some platforms may prohibit explicit erotica or any 18+ material, even when it does not violate the law. One application of this research is to support such platforms in enforcing their internal guidelines, which vary depending on the service and its target audience.

Recent advances in deep learning have improved erotic content detection, but progress is hindered by the limited availability of training data, particularly open datasets. Existing tools are primarily designed for English-language content, limiting their effectiveness for other languages. This linguistic bias highlights the need for language-specific datasets and models that can capture the subtle semantic variations inherent in the expression of erotic content in different languages. Moreover, many existing datasets rely on overly simplified binary classification schemes, which fail to capture

1Dataset link: https://github.com/ZiLiAT-NASK/Fo rePLay
the complexities of erotic content, further limiting detection systems’ effectiveness in diverse cultural and linguistic contexts.

To address these gaps, we introduce fore-PLay, the first Polish language manually annotated dataset of erotic content. In contrast to prior efforts, we present a novel multidimensional taxonomy applied to a large Polish language corpus (\( n = 24,768 \)), incorporating fine-grained annotations across ambiguity, violence, and social unacceptability dimensions, while ensuring representation of LGBT-specific content from both literary and web-based sources. The corpus is constructed from user-generated content sourced from online fiction repositories and professionally published Polish literary works, offering diverse linguistic and stylistic coverage. The article provides a detailed description of the annotation process and discusses the challenges encountered during annotation, in particular human label variation. A major contribution is also the comprehensive evaluation of erotic content detection models, examining specialized Polish transformer-based models (HerBERT (Mroczkowski et al., 2021) and Polish RoBERTa (Dadas, 2023)) and Polish-specific LLMs such as the PLLuM family\footnote{For more information, see https://plum.org.pl/ and https://huggingface.co/CYFRAGOVPL} and Bielik (Ociepa et al., 2024). We also compare these models with state-of-the-art multilingual and general-purpose models, including both open-source (Mixtral (Jiang et al., 2024), Llama 3.1 (Dubey et al., 2024), and C4AI Command-R (CohererForAI, 2024)) and commercial solutions (GPT-4o (Hurst et al., 2024)).

2 Related Work

2.1 Existing Datasets

Prior work has established several benchmark datasets for analyzing textual content with erotic themes, though these exhibit notable limitations in scope and annotation granularity. The most comprehensive corpus, Triplex, comprises 27,000 literary works totaling 1.62 billion tokens extracted from Archive of Our Own (Achour, 2016), while subsequent contributions include the erotic-books\footnote{https://huggingface.co/datasets/AlekseyKorshuk/erotic-books} corpus (\( n = 646 \)) and the BeaverTails dataset containing a class of *sexually explicit, adult content* and developed for alignment research (Ji et al., 2023). Recent approaches to content analysis have leveraged large language models, as demonstrated in the erotica-analysis dataset\footnote{https://huggingface.co/datasets/openerotica/erotica-analysis} (\( n = 15,000 \)) utilizing GPT-3.5 for automated annotation. Various targeted datasets address binary classification of explicit content, including the Jigsaw corpus and its derivatives (\( n = 5,100 \)) (cjadams et al., 2019), though these typically employ simplified taxonomies. While specialized collections such as the sexting corpus\footnote{https://github.com/mathigatti/sexting-dataset} (\( n = 547 \)) examine specific discourse patterns, existing datasets predominantly focus on English-language content with binary classification schemes.

2.2 Erotic Content Detection

Existing methodological approaches to erotic content detection span a broad spectrum of techniques, though their effectiveness is often constrained by the limitations of available training data. Early work relied on classical machine learning approaches, employing Support Vector Machines (SVM) and Random Forest classifiers trained on social media content (Barrientos et al., 2020), while simpler approaches utilized Naive Bayes classification for web page filtering (Hu et al., 2007). More recent neural approaches have demonstrated superior performance, as evidenced by Hierarchical Attention Networks applied to historical Latin texts (\( n = 2,500 \) sentences), achieving significant improvements over token-based methods (Clerice, 2024). Contemporary research has increasingly leveraged large language models, as demonstrated by CENSORCHAT (Qiu et al., 2024), which employs knowledge distillation for monitoring dialogue systems. Advanced architectures combining semantic and statistical features through Convolutional Neural Networks (CNN) have shown promise (Wu and Hu, 2005). However, these approaches predominantly focus on English language content, limiting their applicability to other languages where linguistic nuances and cultural contexts play crucial roles in content interpretation.

2.3 Content Moderation Systems

One key application of erotic content detection datasets is content moderation, covering both user-generated and AI-generated material. Specialized systems like Llama Guard, a Llama2-7b safeguard model developed by Meta, use safety taxonomies
to categorize prompts, including sexual content (Inan et al., 2023). However, comparative studies indicate that general-purpose LLMs, such as GPT-4o (Hurst et al., 2024) and Gemini 1.5 Pro (Team et al., 2024), often outperform these systems by achieving a better balance between false positives and false negatives (AlDahoul et al., 2024).

Holistic approaches to content moderation, such as those proposed by Markov et al. (2023), combine careful taxonomy design, active learning, and lightweight transformer models. These methods excel in detecting rare harmful content, such as material involving minors. Similarly, the framework introduced by Wu et al. (2024) leverages conceptual features from LLM inference, achieving high accuracy for sexual content with minimal computational cost. However, their findings reveal that LLMs tend to self-censor compared to human-written text. Similar conclusions were drawn by Mahomed et al. (2024), whose evaluation of OpenAI’s moderation systems highlights GPT models’ fine-tuning to avoid generating sexual content.

These findings highlight the challenges of building scalable, linguistically inclusive moderation frameworks, especially for nuanced categories like erotic material. Most current tools rely on English-centric training and evaluation pipelines (Markov et al., 2023). To address cultural and linguistic nuances, particularly in morphologically complex languages like Polish, language-specific detection strategies are essential.

3 Data

3.1 Data Collection

We constructed a large-scale Polish language corpus (\( n = 24,768 \) sentences) through systematic sampling from two distinct sources: (i) user-generated content from online fiction repositories and (ii) Polish literary works, including translations of world literature. The data collection was carried out by an interdisciplinary team consisting of literary scholars with extensive expertise in contemporary Polish literature, sociologists, NLP specialists, and IT engineers. The corpus comprises demographically diverse material drawn from 905 unique text units and includes a substantial, manually curated representation of LGBTQ+ narratives. This representation was ensured by selecting queer-themed literary texts—primarily original Polish works, supplemented by a smaller number of translated pieces—based on scholarly research and expertise in Polish queer literature. Document-level context is preserved through unique file identifiers. While direct URLs are omitted due to the volatility of online sources, our sampling methodology ensures broad coverage across linguistic registers and social contexts.

Given the potential application of models in supporting moderation, we focused on internet language and amateur writing, which is predominantly anonymous. Consequently, a significant portion of the sentences (69%) originates from online stories. While the majority of these stories are erotic and non-professional, we included non-erotic stories in a 1:4 ratio to enhance diversity and reduce genre-specific biases. To filter erotic content, we relied on tags and category labels related to erotica provided by the online repositories. All non-professional stories were scraped from publicly available websites, see Appendix A for details.

To avoid overfitting to specific individual writing styles, we limited the dataset to a maximum of two stories per author. Additionally, incorporating professional literary works with varying degrees of erotic themes (31%) aimed to further diversify linguistic patterns. When selecting these texts, availability in digital form was also a key criterion. A total of 22 different literary texts by Polish and international authors were included. This number is significantly lower than the count of unique text units for non-professional content. However, the literary texts were considerably longer, and finding works that met the established criteria proved challenging. Considering the overall proportion of professional works in the corpus, this number should be sufficient.

3.2 Data Preprocessing

Our text segmentation pipeline employs the NLTK library (Bird and Loper, 2004) for sentence boundary detection, which exhibits robust performance despite the inherent variability in web-sourced content quality. To maximize ecological validity, we preserved the original linguistic characteristics of the source material, including non-standard language patterns and orthographic variations, thereby enabling downstream models to generalize effectively to real-world applications.

The corpus contained a total of 342,546 tokens, as counted using the NLTK tokenizer, with an average sentence length of 13.83 tokens (median 11.0, std 11.53). Detailed token statistics are presented in Table 1.
<table>
  <tr>
    <th>Subcorpus</th>
    <th>Mean</th>
    <th>Std</th>
    <th>25%</th>
    <th>50%</th>
    <th>75%</th>
  </tr>
  <tr>
    <td>Total</td>
    <td>13.83</td>
    <td>11.53</td>
    <td>7.0</td>
    <td>11.00</td>
    <td>17.00</td>
  </tr>
  <tr>
    <td>Non-professional</td>
    <td>12.43</td>
    <td>10.32</td>
    <td>6.0</td>
    <td>10.00</td>
    <td>16.00</td>
  </tr>
  <tr>
    <td>Professional</td>
    <td>16.90</td>
    <td>13.31</td>
    <td>9.0</td>
    <td>14.00</td>
    <td>21.00</td>
  </tr>
</table>

Table 1: Token Count Statistics

Inconsistencies in text segmentation and discrepancies in automatically detected sentences arose from the fact that non-professional writers often did not use proper punctuation, making it challenging for the NLTK library to accurately identify sentence boundaries.

4 Annotation Process

To assemble the annotation team, three female annotators were allocated from our internal organization. To ensure gender balance, three part-time male annotators were recruited as external contractors. All selected annotators underwent training and guidance to maintain annotation consistency and quality throughout the project.

This process resulted in a gender-balanced team of six annotators (3 male, 3 female), aged 20–40. We included annotators with backgrounds in linguistics, literature, or related fields to ensure sensitivity to language nuances. To minimize potential bias, the annotators were not provided with metadata that could link specific samples to genre types. Furthermore, samples from each genre type were distributed evenly among all annotators. Each sentence was independently assessed by three annotators. Final labels were determined by majority vote. In cases where all three labels differed (830 out of 24,768 samples, representing 3.35% of the total observations), a superannotator made the final decision. The superannotator, an NLP specialist and senior member of the annotation team, was responsible for overseeing consistency and quality throughout the annotation process. This role encompassed resolving disagreements among annotators, providing additional guidance, and conducting spot checks to ensure adherence to the defined guidelines.

4.1 Annotation Scheme

Based on the expert knowledge of the content, a set of five possible exclusive labels to annotate the samples was designed. This included the following categories: erotic (e), ambiguous (a), violence-related (v), socially unacceptable behaviors (u), and neutral (n). Some examples of the annotated text and its English translation are presented in Table 2.

Sentences labeled as erotic typically describe sexual activities and desires, advanced flirting with evident erotic undertones, references to past sexual activities, or explicit sexual fantasies. However, it is important to note that mere mention of genital terms or publicly accepted romantic behaviors, such as kissing or holding hands, does not warrant classification as sexual. Similarly, descriptions of physiological processes or neutral discussions of erotic or sexual topics (e.g., from the psychological perspective) are excluded from this category and considered neutral.

The violence-related category is reserved exclusively for sentences that include explicit sexual harassment, rape, lack of consent, or any other sort of non-consensual violence which concerns sexual intentions or activities. It is essential to distinguish consensual BDSM behaviors, as well as non-sexual acts of violence (based solely on the sentence-level analysis), as they do not fall under this category.

The category of socially unacceptable behaviors includes sentences that describe sexual behaviors considered illegal, generally taboo, or violative of social norms, such as zoophilia, necrophilia, pedophilia, incest, and other sexual deviations. This category takes precedence over violence-related label, meaning that if a sentence could be classified as both socially unacceptable and violence-related, it should be categorized under socially unacceptable behaviors.

Another label covers context-related sentences, which fall under the category of ambiguous samples. These sentences are identified based on the conviction that—given commonly recognized patterns of sexual descriptions—they evoke erotic or sexual connotations. However, in a neutral context, they could be interpreted as describing non-sexual behaviors or acts.

Lastly, sentences that did not fall into any of the above-mentioned categories were labeled as neutral. On one hand, these typically include non-sexual content on any topic. On the other, this category also comprises general statements about human sexuality or intimate parts that do not typically evoke erotic associations. While such sentences may be sex-related, they are not defined as erotic.

The annotation guidelines, developed by literary scholars and linguists based on domain expertise and material analysis, are described in detail in Appendix B.
<table>
  <tr>
    <th>Original sentence in Polish</th>
    <th>English translation</th>
    <th>Label</th>
  </tr>
  <tr>
    <td>“Mój mężczyzna...”— usłyszałem szept i zanim się zorientowałem zostałem bez koszulki, a Ty siedzi- alaś naprzeciw mnie, całując zachłannie moje usta i błądząc dłońmi po torsie, lekko go drapiąc.</td>
    <td>“My man...”— I heard a whisper and before I knew it I was left shirtless and you were sitting across from me, kissing my lips greedily and wandering your hands over my torso, lightly scratching it.</td>
    <td>erotic</td>
  </tr>
  <tr>
    <td>Był cały twardy.</td>
    <td>He was all hard.</td>
    <td>ambiguous</td>
  </tr>
  <tr>
    <td>Ten gad ja zdradzał!</td>
    <td>This rat was cheating on her!</td>
    <td>neutral</td>
  </tr>
</table>

Table 2: Examples of annotations in the dataset

<table>
  <tr>
    <th rowspan="2">Subcorpus</th>
    <th colspan="5">Label Categories</th>
  </tr>
  <tr>
    <th>Erotic</th>
    <th>Ambiguous</th>
    <th>Violence</th>
    <th>Unacceptable</th>
    <th>Neutral</th>
  </tr>
  <tr>
    <td>Total</td>
    <td>6,361</td>
    <td>1,344</td>
    <td>69</td>
    <td>116</td>
    <td>16,878</td>
  </tr>
  <tr>
    <td>Non-professional</td>
    <td>2,937</td>
    <td>1,277</td>
    <td>52</td>
    <td>95</td>
    <td>12,649</td>
  </tr>
  <tr>
    <td>Professional</td>
    <td>3,424</td>
    <td>67</td>
    <td>17</td>
    <td>21</td>
    <td>4,229</td>
  </tr>
</table>

Table 3: Distribution of Label Categories Across Dataset Subcorpora

Following the completion of the superannotation process, the final dataset consists of two primary classes: 68.14% of the samples are labeled as neutral, and 25.68% as erotic. The remaining categories are minor, with 5.43% labeled as ambiguous, 0.47% as socially unacceptable behaviors, and 0.28% as violence-related. Distribution of labels across the subcorpora are presented in Table 3.

The percentage of violence-related examples, as well as those representing socially unacceptable behaviors, was scarce, which was expected, as the data collection process did not specifically target texts exploring such themes. Additionally, while only 15% of the texts in the dataset were sourced from non-erotic stories, the randomization process employed to minimize bias resulted in the inclusion of neutral sentences even from texts classified as erotic, as these are not exclusive to non-erotic stories.

4.2 Annotation Quality

The process of quality evaluation revealed various challenges faced by the annotators, highlighting the demanding nature of the task. The designed labels were intended to be applied exclusively, meaning that each sample could only be categorized as either neutral, sex-related, or ambiguous. Within the sex-related category, overlaps were mitigated by a clear hierarchical structure. As previously mentioned, in cases where a sample exhibited characteristics of both violence and socially unacceptable behaviors, priority was given to the latter. All the other sex-related samples contributed to the erotic class.

Due to the distinctiveness of rare cases of sexual violence and non-acceptable sexual behaviors, this aspect did not pose significant concerns. Annotators applied these two labels only incidentally, with individual frequencies ranging from 0.2% to 0.6%.

Firstly, the annotation task itself—particularly given the wide range of non-normative language patterns found in non-professional short stories—was inherently subjective. Secondly, the prevalence of misspellings, controversial vocabulary, and grammatical errors, which diminished the clarity of the authors’ intentions and made rational decision-making exceedingly difficult. Thirdly, the category of erotic sentences turned out to be extremely broad, spanning from evident and explicit samples, often including vulgar language, to much subtler erotic tones. Consequently, the ambiguous label, designed to account for context-related uncertainty, offered a means of resolving indecision between sexual and neutral classifications. This created a potential risk of overusing the ambiguous label as a way to mitigate uncertainty in borderline cases.

Human label variation (HLV) (Plank, 2022), inherently present in our task of annotating erotic sentences, raised several questions regarding the dataset design (see the Section 7). To preserve the authenticity of the annotation process, we chose not to intervene beyond the initial training phase. For the dataset’s initial release, we opted to follow the conventional practice of data aggregation featuring a ground truth.

To guarantee annotation reliability, we calculated Cohen’s Kappa for each pair of annotators across all categories. Furthermore, we evaluated inter-annotator agreement for each category individually. Detailed results are presented in Table 4.

Agreement in most pairs was consistent, ranging from 0.66 to 0.71, regardless of annotators’ gender. However, pairs involving annotator Fem1 presented a notable deviation, as Fem1 emerged as an outlier,
<table>
  <tr>
    <th>Annotators</th>
    <th>all</th>
    <th>a</th>
    <th>e</th>
    <th>u</th>
    <th>v</th>
  </tr>
  <tr>
    <td>Fem1 / M1</td>
    <td>0.1750</td>
    <td>0.0532</td>
    <td>0.3602</td>
    <td>0.2545</td>
    <td>0.1984</td>
  </tr>
  <tr>
    <td>Fem1 / Fem2</td>
    <td>0.1808</td>
    <td>0.0370</td>
    <td>0.3193</td>
    <td>0.3952</td>
    <td>0.2911</td>
  </tr>
  <tr>
    <td>Fem1 / Fem3</td>
    <td>0.1415</td>
    <td>0.0151</td>
    <td>0.2890</td>
    <td>0.2740</td>
    <td>0.2271</td>
  </tr>
  <tr>
    <td>M1 / Fem2</td>
    <td>0.7044</td>
    <td>0.3505</td>
    <td>0.7513</td>
    <td>0.3728</td>
    <td>0.4602</td>
  </tr>
  <tr>
    <td>Fem2 / M2</td>
    <td>0.6910</td>
    <td>0.0833</td>
    <td>0.7149</td>
    <td>0.5074</td>
    <td>0.1696</td>
  </tr>
  <tr>
    <td>Fem2 / M3</td>
    <td>0.6610</td>
    <td>0.0037</td>
    <td>0.6923</td>
    <td>0.3973</td>
    <td>0.3237</td>
  </tr>
  <tr>
    <td>Fem3 / Fem2</td>
    <td>0.7187</td>
    <td>0.2683</td>
    <td>0.7728</td>
    <td>0.6718</td>
    <td>0.4078</td>
  </tr>
  <tr>
    <td>M2 / M3</td>
    <td>0.7030</td>
    <td>0.0368</td>
    <td>0.7261</td>
    <td>0.4191</td>
    <td>0.2481</td>
  </tr>
</table>

Table 4: Cohen’s Kappa values for each annotator pair across all categories and for each individual category, broken down by gender (Fem – female, M – male).

largely due to an excessive use of the ambiguous label.

In addition to pairwise agreement, the overall consistency of the entire sample was assessed using Krippendorff’s Alpha. The value was quite low (0.387). However, after excluding the Fem1 annotation, which significantly deviated from the other annotators, the value increased to 0.716, indicating a satisfactory level of agreement in the annotation process.

5 Experiments

We utilize a diverse set of language models for erotic content detection, ranging from specialized Polish encoder-based models and developed Polish-specific LLMs to state-of-the-art commercial LLMs. This variety enables a comprehensive comparison across different model architectures and training methodologies. Additionally, the translated English prompt for the LLMs is included in the appendix on Figure 2.

5.1 Datasets

The original dataset consists of five distinct label categories, but due to significant class imbalance, additional experiments were conducted with modified label groupings as outlined below:
• Basic: A binary classification setup where the dataset was reduced to two classes: neutral and erotic labels.
• Core: A three-class classification setup where the dataset included neutral, erotic, and ambiguous labels.
• Extended: A four-class classification setup that expanded the dataset to include neutral, erotic and ambiguous labels, with a merged category combining violence-related and socially unacceptable behaviors.
• Full: The full dataset containing all five original categories.

5.2 Specialized Polish Transformer-based Models

Encoder-based models for Polish were selected based on the KLEJ Benchmark leaderboard (Rybak et al., 2020). Two top-performing models across multiple downstream tasks were chosen: HerBERT and Polish RoBERTa. HerBERT is a Polish-language adaptation of BERT, trained on a Polish corpus, while Polish RoBERTa is an optimized version featuring a unigram tokenizer, whole-word masking, and an expanded vocabulary of 128k entries. In our experiments, both models were fine-tuned on each dataset configuration (Basic, Core, Extended, Full).

5.3 Polish Large Language Models

Our evaluation framework encompasses three variants from the PLLuM (Polish Large Language Model) family: Llama-3.1-8B-PLLuM, PLLuM-Mistral-12B, and PLLuM-Mistral-8x7B, adapted from their respective base architectures. These models underwent pre-training on 147B Polish language tokens and subsequent instruction tuning. We additionally employed Bielik 2.3, trained on 200B tokens and aligned using DPO-positive on 66,000 examples. To assess adaptation capabilities, we fine-tuned Llama-3.1-8B-PLLuM and PLLuM-Mistral-12B on our classification task. All base models were evaluated under 0-shot, 1-shot, and 5-shot configurations, while fine-tuned variants underwent standard supervised evaluation.

5.4 State of the Art Large Language Models

For comparison with LLMs fine-tuned strictly for Polish, we employed a selection of state-of-the-art multilingual and general-purpose models, including both open-sourced and commercial solutions. Among these, Mistral, characterized by its modular and scalable architecture, was utilized in both the 8x22B and 8x7B configurations, alongside Mistral 12B, designed as a lightweight alternative. We also tested Llama 3.1 70B-Instruct and its smaller counterpart Llama 3.1 8B-Instruct, both fine-tuned for instruction-following tasks, as well as C4AI Command-R, developed for command-following applications. Additionally, we incorporated GPT-40, frequently used for tasks requiring generalization across various domains.
6 Results

Table 5 highlights the macro-F1 performance metrics of the HerBERT and RoBERTa models across classification tasks involving different dataset configurations. In binary classification scenarios, both models exhibit robust performance. However, as the number of classes increases, a decline in macro-F1 scores is observed. Notably, RoBERTa, particularly in its Base configuration, demonstrates superior performance relative to HerBERT, with pronounced advantages evident in tasks on the Basic and Extended datasets.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">HerBERT</th>
    <th colspan="2">RoBERTa</th>
  </tr>
  <tr>
    <th>Base</th>
    <th>Large</th>
    <th>Base</th>
    <th>Large</th>
  </tr>
  <tr>
    <td>Basic</td>
    <td>0.929</td>
    <td>0.939</td>
    <td><b>0.944</b></td>
    <td>0.943</td>
  </tr>
  <tr>
    <td>Core</td>
    <td>0.702</td>
    <td>0.738</td>
    <td>0.738</td>
    <td><b>0.748</b></td>
  </tr>
  <tr>
    <td>Extended</td>
    <td>0.693</td>
    <td><b>0.746</b></td>
    <td>0.704</td>
    <td>0.734</td>
  </tr>
  <tr>
    <td>Full</td>
    <td>0.632</td>
    <td>0.648</td>
    <td><b>0.707</b></td>
    <td>0.664</td>
  </tr>
</table>

Table 5: Macro-F1 scores for HerBERT and RoBERTa models across different dataset versions.

In the case of LLMs, models from the PLLuM family consistently demonstrate strong performance across varying label counts and shot configurations, with particular emphasis on the PLLuM-Mixtral-8x7B, which achieves outstanding macro-F1 scores across a wide range of label counts, notably reaching 0.921 for Basic data and 0.670 for Core data in the 5-shot setting. Additionally Llama-3.1-8B-PLLuM (SFT) and PLLuM-Mistral-12B (SFT) show significant improvements when subjected to supervised fine-tuning, resulting in enhanced performance, with the highest macro-F1 scores observed across various label counts.

Fine-tuned LLMs exhibit performance levels comparable to traditional transformer-based architectures when evaluated on datasets labeled as Basic and Core, where all categories are sufficiently represented. This suggests that in scenarios characterized by balanced class distributions, where each class is supported by an adequate number of training examples, the performance differential between these methodological approaches is minimal. Conversely, for datasets labeled as Extended and Core, which are marked by pronounced class imbalance, models built upon the BERT architecture begin to demonstrate a distinct advantage. These models excel in effectively addressing the challenges posed by imbalanced data distributions, exhibiting superior capability in identifying underrepresented categories. This highlights their enhanced adaptability and robustness in handling more complex scenarios involving skewed training data distributions.

In the context of multilingual models trained with a primary emphasis on the English language, the results are generally inferior compared to models specifically adapted for the Polish language. The performance varies depending on the configuration, including the number of labels and the number of shots, with different models excelling under different conditions. However, overall, the Mixtrals and GPT-4o models consistently yielded the most favorable results.

The full results of all evaluated models—including the baseline models, the ones adapted to the Polish language from the PLLuM family, and the ones further fine-tuned on our dataset—are presented in Table 6. A detailed account of refusals and responses that could not be classified into any of the predefined categories is presented in Table 7. Among the evaluated models, Llama 3.1 8B-Instruct recorded the highest number of refusals, reaching a total of 848. In contrast, models that consistently provided clear and categorizable outputs included GPT-4o and Mixtral 8x22B, as well as those additionally fine-tuned on our dataset.

7 Discussion

The task of multi-dimensional eroticity annotation is highly complex and inherently subjective, necessitating further and more advanced reflection on the Human-Language Variation (HLV) framework, which has become an increasingly important open problem in recent years (Plank, 2022; Uma et al., 2021; Pavlick and Kwiatkowski, 2019). The researchers highlight the fact that disagreements in the annotation process should not be dismissed as mere noise in the search for a single ground truth. On the contrary, such disagreements can often reflect plausible alternative judgments, suggesting that these annotations may not always be categorical (De Marneffe et al., 2012). However, as demonstrated by our experience evaluating annotation quality, distinguishing between clear errors—where no plausible argument could justify the choice of a particular label—and alternative judgments is a critical issue that warrants deeper concern and further discussion. Within the context of erotic content detection, we believe it is valuable to advance current research by utilizing
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Basic</th>
    <th colspan="3">Core</th>
    <th colspan="3">Extended</th>
    <th colspan="3">Full</th>
  </tr>
  <tr>
    <th>0-shot</th><th>1-shot</th><th>5-shot</th>
    <th>0-shot</th><th>1-shot</th><th>5-shot</th>
    <th>0-shot</th><th>1-shot</th><th>5-shot</th>
    <th>0-shot</th><th>1-shot</th><th>5-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>0.888</td><td>0.873</td><td>0.891</td><td>0.640</td><td>0.605</td><td>0.618</td><td>0.425</td><td>0.410</td><td>0.415</td><td>0.340</td><td>0.334</td><td>0.349</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td><td>0.684</td><td>0.730</td><td>0.714</td><td>0.420</td><td>0.486</td><td>0.454</td><td>0.322</td><td>0.374</td><td>0.363</td><td>0.257</td><td>0.292</td><td>0.299</td>
  </tr>
  <tr>
    <td>Llama 3.1 8B-Instruct</td><td>0.789</td><td>0.736</td><td>0.709</td><td>0.562</td><td>0.517</td><td>0.514</td><td>0.388</td><td>0.364</td><td>0.378</td><td>0.325</td><td>0.283</td><td>0.316</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td><td>0.837</td><td>0.845</td><td>0.846</td><td>0.595</td><td>0.641</td><td>0.630</td><td>0.396</td><td>0.417</td><td>0.408</td><td>0.329</td><td>0.344</td><td>0.364</td>
  </tr>
  <tr>
    <td>Mistral 12B</td><td>0.872</td><td>0.847</td><td>0.862</td><td>0.58</td><td>0.579</td><td>0.610</td><td>0.423</td><td>0.411</td><td>0.42</td><td>0.354</td><td>0.349</td><td>0.338</td>
  </tr>
  <tr>
    <td>Mistral 8x7B</td><td>0.867</td><td>0.87</td><td>0.862</td><td>0.616</td><td>0.622</td><td>0.621</td><td>0.440</td><td>0.451</td><td>0.435</td><td>0.364</td><td>0.364</td><td>0.348</td>
  </tr>
  <tr>
    <td>Mistral 8x22B</td><td>0.872</td><td>0.854</td><td>0.833</td><td>0.633</td><td>0.642</td><td>0.610</td><td>0.424</td><td>0.440</td><td>0.406</td><td>0.335</td><td>0.361</td><td>0.344</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td><td>0.848</td><td>0.858</td><td>0.868</td><td>0.601</td><td>0.597</td><td>0.607</td><td>0.427</td><td>0.416</td><td>0.423</td><td>0.312</td><td>0.437</td><td>0.48</td>
  </tr>
  <tr>
    <td>PLLuM-Mistral-12B</td><td>0.894</td><td>0.842</td><td>0.844</td><td>0.656</td><td>0.610</td><td>0.613</td><td>0.456</td><td>0.441</td><td>0.427</td><td>0.401</td><td>0.359</td><td>0.401</td>
  </tr>
  <tr>
    <td>PLLuM-Mistral-8x7B</td><td>0.874</td><td>0.902</td><td>0.921</td><td>0.647</td><td>0.662</td><td>0.670</td><td>0.452</td><td>0.456</td><td>0.426</td><td>0.422</td><td>0.406</td><td>0.422</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLuM</td><td>0.641</td><td>0.737</td><td>0.709</td><td>0.425</td><td>0.446</td><td>0.425</td><td>0.301</td><td>0.324</td><td>0.326</td><td>0.314</td><td>0.274</td><td>0.273</td>
  </tr>
  <tr>
    <td>PLLuM-Mistral-12B (SFT)</td><td>0.946</td><td></td><td></td><td>0.792</td><td></td><td></td><td>0.458</td><td></td><td></td><td>0.488</td><td></td><td></td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLuM (SFT)</td><td>0.947</td><td></td><td></td><td>0.764</td><td></td><td></td><td>0.461</td><td></td><td></td><td>0.435</td><td></td><td></td>
  </tr>
</table>

Table 6: Macro-F1 scores for various models across different dataset configurations and shot numbers.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Basic</th>
    <th colspan="3">Core</th>
    <th colspan="3">Extended</th>
    <th colspan="3">Full</th>
    <th>Total</th>
  </tr>
  <tr>
    <th>0-shot</th><th>1-shot</th><th>5-shot</th>
    <th>0-shot</th><th>1-shot</th><th>5-shot</th>
    <th>0-shot</th><th>1-shot</th><th>5-shot</th>
    <th>0-shot</th><th>1-shot</th><th>5-shot</th>
    <th></th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td><td>9</td><td>2</td><td>14</td><td>9</td><td>5</td><td>10</td><td>7</td><td>1</td><td>9</td><td>4</td><td>1</td><td>14</td><td>95</td>
  </tr>
  <tr>
    <td>Llama 3.1 8B-Instruct</td><td>42</td><td>150</td><td>12</td><td>38</td><td>99</td><td>14</td><td>134</td><td>62</td><td>8</td><td>66</td><td>208</td><td>15</td><td>848</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td><td>8</td><td>0</td><td>0</td><td>33</td><td>0</td><td>0</td><td>46</td><td>0</td><td>0</td><td>49</td><td>0</td><td>0</td><td>136</td>
  </tr>
  <tr>
    <td>Mistral 12B</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>0</td><td>1</td>
  </tr>
  <tr>
    <td>Mistral 8x7B</td><td>0</td><td>0</td><td>2</td><td>1</td><td>0</td><td>2</td><td>1</td><td>5</td><td>7</td><td>3</td><td>0</td><td>4</td><td>25</td>
  </tr>
  <tr>
    <td>Mistral 8x22B</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td><td>116</td><td>1</td><td>0</td><td>115</td><td>2</td><td>0</td><td>148</td><td>1</td><td>3</td><td>2</td><td>5</td><td>1</td><td>394</td>
  </tr>
  <tr>
    <td>PLLuM-Mistral-12B</td><td>3</td><td>5</td><td>1</td><td>3</td><td>2</td><td>3</td><td>8</td><td>7</td><td>0</td><td>6</td><td>10</td><td>5</td><td>53</td>
  </tr>
  <tr>
    <td>PLLuM-Mistral-8x7B</td><td>0</td><td>0</td><td>1</td><td>0</td><td>0</td><td>2</td><td>0</td><td>0</td><td>2</td><td>0</td><td>1</td><td>2</td><td>8</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLuM</td><td>1</td><td>4</td><td>0</td><td>1</td><td>4</td><td>3</td><td>1</td><td>3</td><td>5</td><td>0</td><td>4</td><td>1</td><td>27</td>
  </tr>
  <tr>
    <td>PLLuM-Mistral-12B (SFT)</td><td>0</td><td></td><td></td><td>0</td><td></td><td></td><td>0</td><td></td><td></td><td>0</td><td></td><td></td><td>0</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLuM (SFT)</td><td>0</td><td></td><td></td><td>0</td><td></td><td></td><td>0</td><td></td><td></td><td>0</td><td></td><td></td><td>0</td>
  </tr>
</table>

Table 7: Number of unidentified responses or refusals.

datasets with multiple plausible labels and exploring the approach of Learning with Disagreements (LeWiDi) (Uma et al., 2021) or Multiple Ground Truth (Anand et al., 2024). This approach could offer more nuanced insights into the field by comparing the performance of models trained on aggregated annotations versus those trained on multiple possible labels.

Our experimental results demonstrate clear performance patterns across model architectures and linguistic contexts, with specialized Polish language models consistently outperforming multilingual alternatives in erotic content detection. The encoder-based architectures, particularly RoBERTa, achieve robust macro-F1 scores in binary classification (0.929 – 0.944), though performance degrades with increasing categorical complexity (dropping to ~ 0.66 for five-class classification). This degradation pattern reveals fundamental challenges in maintaining discriminative power across finer-grained categorical distinctions, particularly in cases requiring subtle cultural and contextual understanding.

The PLLuM family of models exhibits exceptional few-shot learning capabilities, with PLLuM-Mixtral-8x7B achieving macro-F1 scores of 0.921 in 5-shot settings for binary classification, while supervised fine-tuning of PLLuM-Mistral-12B yields the highest overall performance (macro-F1 = 0.946). However, the diminishing returns observed with increasing model size, coupled with the persistent challenge of handling ambiguous content in imbalanced datasets, suggests that architectural sophistication alone may not overcome the fundamental challenges of erotic content detection in morphologically complex languages. These
findings underscore the importance of language-specific model development for sensitive content detection tasks, rather than relying solely on increasingly large multilingual models.

Beyond overall performance metrics, understanding error patterns is crucial for content moderation deployment, where both false positives (over-censorship) and false negatives (missed inappropriate content) carry significant consequences. We calculated Type I error percentages as the number of false positives divided by the sum of false positives and false negatives to assess model precision across different classification scenarios. Full results regarding Type I and Type II errors for each model are presented in the Appendix D. The analysis reveals that Type I error rates decrease substantially with classification complexity—from approximately 60-80% in binary classification (Basic) to 40-50% in the full five-class scenario (Figure 1). This pattern indicates higher Type I error rates in simpler tasks and higher Type II rates in complex multi-class scenarios, highlighting the fundamental trade-off between precision and sensitivity in erotic content detection tasks.

8 Conclusions

This work presents forePLay, the first comprehensive Polish language dataset for erotic content detection, introducing a novel multidimensional taxonomy encompassing ambiguity, violence, and social acceptability dimensions. Through systematic empirical evaluation across multiple model architectures, our analysis demonstrates the superiority of specialized Polish language models over multilingual alternatives, particularly in handling nuanced categorical distinctions. The performance patterns observed in transformer-based architectures and few-shot learning scenarios provide valuable insights for developing language-specific content moderation systems. Our findings underscore the critical importance of culturally and linguistically adapted approaches in content detection tasks, while the presented dataset and evaluation framework establish essential groundwork for future research in multilingual content moderation systems, particularly for morphologically complex languages.

9 Data Availability

A subset of erotic and ambiguous sentences, totaling 3,704 samples, has been released on our public Github account. Due to ethical considerations, we opted to exclude the minor harmful classes of violence-related content and socially unacceptable behaviors. ForePLay Dataset Release 1.0 consists of 2,728 sentences labeled as erotic, and 976 sentences labeled as ambiguous, which corresponds to 43% and 73% of the original dataset, respectively. Neutral sentences were not included in this release, as such data is readily obtainable from publicly available corpora and may be easily used for comparative purposes in downstream tasks. This subset has undergone additional copyright verification and was made available following legal consultations. The released data and its license comply with new legal regulations that came into effect after the data collection and annotation process had been completed.

Limitations

Our study reveals several important limitations that warrant discussion. While our dataset represents a significant contribution to Polish language resources, it exhibits sampling biases through its predominance of content from online fiction repositories (69%). This may not fully capture the linguistic diversity of erotic discourse across different domains and registers. The relative underrepresentation of professional literary works (31%) potentially limits our understanding of more sophisticated expressions of erotic content.

A significant methodological limitation emerges from our binary approach to violence and socially unacceptable behaviors. The current taxonomy, while practical for annotation purposes, may oversimplify the complex spectrum of content severity
and social acceptability. This simplification could impact the generalizability of our models to real-world content moderation scenarios where more nuanced distinctions are crucial.

The annotation process presents limitations stemming from inherent subjectivity in erotic content interpretation. Despite achieving reasonable inter-annotator agreement (Krippendorff’s Alpha of 0.716 after outlier removal), the significant variation in individual annotator interpretations—particularly evident in the use of the ambiguous category—suggests potential instability in ground truth labels.

From a technical perspective, our evaluation framework primarily focuses on classification accuracy without deeply examining model behavior on edge cases or adversarial examples. While our study demonstrates the superiority of Polish-specific models over multilingual alternatives, we acknowledge limitations in comparing across model architectures with significantly different parameter counts and training regimes.

Furthermore, a specific limitation pertaining to the evaluation phase is that the definitions of the labels were not explicitly included in the prompts provided to the models. We confirm this omission may have contributed to the lower performance of the models in comparison to the baseline.

These limitations underscore the need for future work addressing dataset diversity, annotation methodology refinement, and more robust evaluation frameworks for erotic content detection systems.

Acknowledgements

The development of the PLLuM models used in the experiments was funded by the Polish Minister of Digital Affairs under a special purpose subsidy No. 1/WI/DBil/2023: Responsible Development of the Open Large Language Model, PLLuM (Polish Large Language Model), aimed at supporting breakthrough technologies in the public and economic sectors, including an open, Polish-language intelligent assistant for public administration clients. The work on the dataset, however, was conducted independently and was not financed by this project.

References

Sara Achour. 2016. A data driven analysis framework and erotica writing assistant. In SIGTBD.

Nouar AlDahoul, Myles Joshua Toledo Tan, Harishwar Reddy Kasireddy, and Yasir Zaki. 2024. Advancing content moderation: Evaluating large language models for detecting sensitive content across text, images, and videos. Preprint, arXiv:2411.17123.

Abhishek Anand, Negar Mokhberian, Prathyusha Naresh Kumar, Anweasha Saha, Zihao He, Ashwin Rao, Fred Morstatter, and Kristina Lerman. 2024. Don’t blame the data, blame the model: Understanding noise and bias when learning from subjective annotations. arXiv preprint arXiv:2403.04085.

Gonzalo Molpeceres Barrientos, Rocío Alaiz-Rodríguez, Víctor González-Castro, and Andrew C. Parnell. 2020. Machine learning techniques for the detection of inappropriate erotic content in text. International Journal of Computational Intelligence Systems, 13(1):591–603.

Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217. Barcelona, Spain. Association for Computational Linguistics.

cjadams, Daniel Borkan, inversion, Jeffrey Sorensen, Lucas Dixon, Lucy Vasserman, and nthium. 2019. Jigsaw unintended bias in toxicity classification. https://kaggle.com/competitions/jigsaw-unintended-bias-in-toxicity-classification. Kaggle.

Thibault Clerice. 2024. Detecting sexual content at the sentence level in first millennium Latin texts. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4772–4783. Torino, Italia. ELRA and ICCL.

CohereForAI. 2024. Command r +.

Slawomir Dadas. 2023. Polish roberta.

Marie-Catherine De Marneffe, Christopher D Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. Computational linguistics, 38(2):301–333.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, and et al. Abhishek Kadian. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462.

Weiming Hu, Ou Wu, Zhouyao Chen, Zhouyu Fu, and Steve Maybank. 2007. Recognition of pornographic web pages by classifying texts and images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6):1019–1034.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. Preprint, arXiv:2312.06674.

Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruixiang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beaver-tails: Towards improved safety alignment of llm via a human-preference dataset. arXiv preprint arXiv:2307.04657.

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L’elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024. Mix-trial of experts. ArXiv, abs/2401.04088.

Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. Hate speech detection: Challenges and solutions. PloS one, 14(8):e0221152.

Yaaseen Mahomed, Charlie M. Crawford, Sanjana Gautam, Sorelle A. Friedler, and Danaë Metaxa. 2024. Auditing gpt’s content moderation guardrails: Can chatgpt write your favorite tv show? In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’24, page 660–686, New York, NY, USA. Association for Computing Machinery.

Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. Preprint, arXiv:2208.03274.

Robert Mroczkowski, Piotr Rybak, Alina Wróblewska, and Ireneusz Gawlik. 2021. HERBERT: Efficiently pretrained transformer-based language model for Polish. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 1–10, Kyiv, Ukraine. Association for Computational Linguistics.

Krzysztof Ociepa, Łukasz Flis, Remigiusz Kinas, Adrian Gwoździej, and Krzysztof Wróbel. 2024. Bielik: A family of large language models for the polish language - development, insights, and evaluation.

Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694.

John Pavlopoulos, Jeffrey Sorensen, Lucas Dixon, Nithum Thain, and Ion Androutsopoulos. 2020. Toxicity detection: Does context really matter? arXiv preprint arXiv:2006.00998.

Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Huachuan Qiu, Shuai Zhang, Hongliang He, Anqi Li, and Zhenzhong Lan. 2024. Facilitating pornographic text detection for open-domain dialogue systems via knowledge distillation of large language models. CoRR, abs/2403.13250.

Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Ireneusz Gawlik. 2020. Klej: Comprehensive benchmark for polish language understanding. arXiv preprint arXiv:2005.00630.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338–347, Online. Association for Computational Linguistics.

Jialin Wu, Jiangyi Deng, Shengyuan Pang, Yanjiao Chen, Jiayang Xu, Xinfeng Li, and Wenyuan Xu. 2024. Legilimens: Practical and unified content moderation for large language model services. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, page 1151–1165, New York, NY, USA. Association for Computing Machinery.

Ou Wu and Weiming Hu. 2005. Web sensitive text filtering by combining semantics and statistics. In 2005 International Conference on Natural Language Processing and Knowledge Engineering, pages 663–667.
A Data Sources

All non-professional stories were scraped from publicly available websites including:

• opowiadaniaerotyczne-darmowo.com
• sexopowiadania.pl
• pornzone.com
• anonserek.pl
• opowi.pl (categories: o życiu, różne, milosne)
• opowiadania.pl
• polki.pl

B Annotation Guidelines Summary

This annotation task aims to create a manually annotated dataset for erotic content detection, enabling the development of automated detection methods and content moderation systems. These systems can assist models in identifying undesired content that may be potentially harmful. Each annotator is provided with a set of samples to label exclusively with one of the five predefined categories. The annotation process is conducted independently, meaning that annotators do not have access to the labels or work of other team members. For the purpose of this task, a sample is defined as a single sentence of varying length. The style and spelling of the sentences may vary significantly, and assessing the quality or correctness of the text is not part of the task.

Annotators are instructed to carefully read each sentence and assign one final label to it. Each sentence was presented in isolation, without additional context. They are also encouraged to pause or discontinue the task at any point if they feel uncomfortable with the content being evaluated.

Each sample should be exclusively assigned one of the following labels:
• Label e (for erotic): choose this label if the sentence describes sexual activities and desires, advanced flirting with evident erotic undertones, references to past sexual activities, or explicit sexual fantasies.
Please note that mere mentions of genital terms or publicly accepted romantic behaviors or neutral discussions of erotic or sexual topics (e.g., from the psychological perspective) are excluded from this category.
• Label v (for violence-related): choose this label if the sentence depicts explicit sexual harassment, rape, lack of consent, or any other sort of non-consensual violence which concerns sexual intentions or activities.
Please note that a sentence can include violence which is not sex-related. Such instances are excluded from this category. Consensual BDSM behaviors are likely to fall under the label erotic.
• Label u (for socially unacceptable behaviors): choose this label if the sentence describes sexual behaviors considered illegal, generally taboo, or violative of social norms, such as zoophilia, necrophilia, pedophilia, incest, and other sexual deviations.
Please note that if you classify a sample as both violence related and featuring socially unacceptable behaviors, the priority should be given to the latter. Such instances should be labeled as socially unacceptable behaviors.
• Label a (for ambiguous): choose this label for sentences that may evoke some degree of erotic connotation but whose interpretation depends heavily on the context. If the sentence can reasonably be interpreted as non-erotic in a neutral context, it should be labeled as ambiguous.
• Label n (for neutral): choose this label if the sentence does not meet the criteria of the above mentioned sex-related categories. This typically includes non-sexual content on various topics, as well as general statements about human sexuality or intimate anatomy that do not carry erotic connotations.

C Prompt

Prompt Structure:
You are a classifier, and your task is to analyze the text and assign one of the following labels:

1. Neutral text
2. Erotic text
3. Ambiguous text
4. Socially unacceptable behavior text
5. Violence-related text

Based on the text below, return the results in JSON format, which will contain the keys "Text" and "Prediction". The value in the "Prediction" field should contain only the name of the class.

Input text:

Figure 2: Classification prompt template used for evaluating language models (translated into English).
D  Analysis of Type I and Type II errors

D.1  Full

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Erotic</th>
    <th colspan="2">Ambiguous</th>
    <th colspan="2">Violence</th>
    <th colspan="2">Unacceptable</th>
    <th colspan="2">Neutral</th>
  </tr>
  <tr>
    <th>Type I</th><th>Type II</th>
    <th>Type I</th><th>Type II</th>
    <th>Type I</th><th>Type II</th>
    <th>Type I</th><th>Type II</th>
    <th>Type I</th><th>Type II</th>
  </tr>
  <tr>
    <td>HerBERT Base</td><td>44</td><td>42</td><td>29</td><td>39</td><td>2</td><td>4</td><td>0</td><td>2</td><td>58</td><td>46</td>
  </tr>
  <tr>
    <td>HerBERT Large</td><td>55</td><td>38</td><td>18</td><td>47</td><td>0</td><td>4</td><td>0</td><td>8</td><td>62</td><td>38</td>
  </tr>
  <tr>
    <td>RoBERTa Base</td><td>35</td><td>42</td><td>41</td><td>40</td><td>3</td><td>2</td><td>2</td><td>1</td><td>50</td><td>46</td>
  </tr>
  <tr>
    <td>RoBERTa Large</td><td>39</td><td>41</td><td>27</td><td>41</td><td>10</td><td>2</td><td>0</td><td>3</td><td>51</td><td>40</td>
  </tr>
  <tr>
    <td colspan="11"><b>0-shot</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>97</td><td>26</td><td>14</td><td>72</td><td>32</td><td>3</td><td>228</td><td>7</td><td>1</td><td>266</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td><td>165</td><td>89</td><td>99</td><td>61</td><td>18</td><td>4</td><td>329</td><td>8</td><td>9</td><td>519</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td><td>134</td><td>19</td><td>6</td><td>72</td><td>21</td><td>3</td><td>245</td><td>8</td><td>8</td><td>312</td>
  </tr>
  <tr>
    <td>Mixtral 8x22B</td><td>91</td><td>73</td><td>12</td><td>72</td><td>43</td><td>2</td><td>244</td><td>5</td><td>14</td><td>252</td>
  </tr>
  <tr>
    <td>Bielik-1IB-v2.3-Instruct</td><td>260</td><td>10</td><td>6</td><td>72</td><td>29</td><td>2</td><td>144</td><td>7</td><td>0</td><td>348</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td><td>55</td><td>108</td><td>34</td><td>70</td><td>11</td><td>1</td><td>50</td><td>8</td><td>104</td><td>64</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td><td>53</td><td>85</td><td>35</td><td>69</td><td>11</td><td>1</td><td>51</td><td>7</td><td>72</td><td>60</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td><td>5</td><td>238</td><td>1</td><td>72</td><td>2</td><td>3</td><td>7</td><td>8</td><td>312</td><td>6</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td><td>43</td><td>34</td><td>18</td><td>72</td><td>5</td><td>0</td><td>106</td><td>8</td><td>23</td><td>80</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td><td>47</td><td>26</td><td>39</td><td>72</td><td>1</td><td>3</td><td>76</td><td>8</td><td>27</td><td>49</td>
  </tr>
  <tr>
    <td colspan="11"><b>1-shot</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>99</td><td>26</td><td>14</td><td>72</td><td>33</td><td>3</td><td>255</td><td>7</td><td>1</td><td>296</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td><td>237</td><td>30</td><td>4</td><td>71</td><td>18</td><td>4</td><td>143</td><td>8</td><td>31</td><td>320</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td><td>83</td><td>50</td><td>15</td><td>72</td><td>21</td><td>3</td><td>250</td><td>6</td><td>8</td><td>246</td>
  </tr>
  <tr>
    <td>Mixtral 8x22B</td><td>87</td><td>70</td><td>19</td><td>71</td><td>26</td><td>2</td><td>178</td><td>6</td><td>30</td><td>185</td>
  </tr>
  <tr>
    <td>Bielik-1IB-v2.3-Instruct</td><td>189</td><td>14</td><td>14</td><td>71</td><td>19</td><td>3</td><td>104</td><td>6</td><td>4</td><td>236</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td><td>52</td><td>132</td><td>50</td><td>70</td><td>7</td><td>3</td><td>54</td><td>8</td><td>119</td><td>65</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td><td>66</td><td>100</td><td>65</td><td>67</td><td>8</td><td>2</td><td>90</td><td>6</td><td>64</td><td>109</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td><td>53</td><td>216</td><td>20</td><td>71</td><td>36</td><td>3</td><td>44</td><td>7</td><td>260</td><td>114</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td><td>35</td><td>52</td><td>34</td><td>70</td><td>7</td><td>1</td><td>102</td><td>8</td><td>23</td><td>65</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td><td>65</td><td>24</td><td>38</td><td>72</td><td>1</td><td>3</td><td>68</td><td>8</td><td>28</td><td>62</td>
  </tr>
  <tr>
    <td colspan="11"><b>5-shot</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td><td>90</td><td>34</td><td>26</td><td>72</td><td>43</td><td>1</td><td>231</td><td>8</td><td>1</td><td>276</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td><td>205</td><td>105</td><td>8</td><td>72</td><td>12</td><td>3</td><td>39</td><td>8</td><td>133</td><td>209</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td><td>85</td><td>53</td><td>15</td><td>71</td><td>24</td><td>1</td><td>272</td><td>8</td><td>6</td><td>269</td>
  </tr>
  <tr>
    <td>Mixtral 8x22B</td><td>142</td><td>48</td><td>16</td><td>72</td><td>33</td><td>2</td><td>154</td><td>5</td><td>17</td><td>233</td>
  </tr>
  <tr>
    <td>Bielik-1IB-v2.3-Instruct</td><td>172</td><td>19</td><td>7</td><td>72</td><td>24</td><td>2</td><td>138</td><td>6</td><td>1</td><td>239</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td><td>55</td><td>108</td><td>34</td><td>70</td><td>11</td><td>1</td><td>50</td><td>8</td><td>104</td><td>64</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td><td>53</td><td>85</td><td>35</td><td>69</td><td>11</td><td>1</td><td>51</td><td>7</td><td>72</td><td>60</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td><td>30</td><td>288</td><td>16</td><td>72</td><td>20</td><td>3</td><td>10</td><td>8</td><td>277</td><td>51</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td><td>43</td><td>34</td><td>18</td><td>72</td><td>5</td><td>0</td><td>106</td><td>8</td><td>23</td><td>80</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td><td>75</td><td>28</td><td>42</td><td>72</td><td>3</td><td>3</td><td>73</td><td>8</td><td>31</td><td>86</td>
  </tr>
</table>

Table 8: Detailed Type I and Type II error counts for various models for <i>Full</i> dataset type.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Erotic</th>
    <th colspan="2">Ambiguous</th>
    <th colspan="2">Violence + Unacceptable</th>
    <th colspan="2">Neutral</th>
  </tr>
  <tr>
    <th>Type I</th>
    <th>Type II</th>
    <th>Type I</th>
    <th>Type II</th>
    <th>Type I</th>
    <th>Type II</th>
    <th>Type I</th>
    <th>Type II</th>
  </tr>
  <tr>
    <td>HerBERT Base</td>
    <td>41</td>
    <td>45</td>
    <td>34</td>
    <td>40</td>
    <td>3</td>
    <td>7</td>
    <td>58</td>
    <td>44</td>
  </tr>
  <tr>
    <td>HerBERT Large</td>
    <td>33</td>
    <td>58</td>
    <td>41</td>
    <td>33</td>
    <td>6</td>
    <td>3</td>
    <td>54</td>
    <td>40</td>
  </tr>
  <tr>
    <td>RoBERTa Base</td>
    <td>45</td>
    <td>49</td>
    <td>38</td>
    <td>42</td>
    <td>8</td>
    <td>2</td>
    <td>48</td>
    <td>51</td>
  </tr>
  <tr>
    <td>RoBERTa Large</td>
    <td>42</td>
    <td>37</td>
    <td>30</td>
    <td>43</td>
    <td>4</td>
    <td>4</td>
    <td>47</td>
    <td>44</td>
  </tr>
  <tr>
    <th colspan="9">0-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>95</td>
    <td>33</td>
    <td>240</td>
    <td>34</td>
    <td>42</td>
    <td>10</td>
    <td>2</td>
    <td>262</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>75</td>
    <td>216</td>
    <td>26</td>
    <td>71</td>
    <td>9</td>
    <td>11</td>
    <td>265</td>
    <td>76</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td>
    <td>75</td>
    <td>101</td>
    <td>220</td>
    <td>34</td>
    <td>22</td>
    <td>9</td>
    <td>59</td>
    <td>232</td>
  </tr>
  <tr>
    <td>Mistral 8x2B</td>
    <td>52</td>
    <td>99</td>
    <td>225</td>
    <td>36</td>
    <td>39</td>
    <td>9</td>
    <td>39</td>
    <td>174</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>159</td>
    <td>42</td>
    <td>66</td>
    <td>63</td>
    <td>19</td>
    <td>10</td>
    <td>44</td>
    <td>160</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>78</td>
    <td>91</td>
    <td>40</td>
    <td>56</td>
    <td>100</td>
    <td>5</td>
    <td>55</td>
    <td>104</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>116</td>
    <td>44</td>
    <td>62</td>
    <td>39</td>
    <td>43</td>
    <td>9</td>
    <td>26</td>
    <td>112</td>
  </tr>
  <tr>
    <td>Llama-3.1-B-PLLaM</td>
    <td>7</td>
    <td>256</td>
    <td>3</td>
    <td>72</td>
    <td>2</td>
    <td>12</td>
    <td>334</td>
    <td>6</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>31</td>
    <td>47</td>
    <td>68</td>
    <td>12</td>
    <td>16</td>
    <td>12</td>
    <td>42</td>
    <td>37</td>
  </tr>
  <tr>
    <td>Llama-3.1-B-PLLaM (SFT)</td>
    <td>58</td>
    <td>24</td>
    <td>73</td>
    <td>36</td>
    <td>10</td>
    <td>12</td>
    <td>28</td>
    <td>49</td>
  </tr>
  <tr>
    <th colspan="9">1-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>117</td>
    <td>28</td>
    <td>241</td>
    <td>37</td>
    <td>44</td>
    <td>10</td>
    <td>1</td>
    <td>293</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>221</td>
    <td>32</td>
    <td>149</td>
    <td>70</td>
    <td>21</td>
    <td>10</td>
    <td>35</td>
    <td>310</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td>
    <td>82</td>
    <td>59</td>
    <td>201</td>
    <td>32</td>
    <td>42</td>
    <td>7</td>
    <td>12</td>
    <td>239</td>
  </tr>
  <tr>
    <td>Mistral 8x2B</td>
    <td>87</td>
    <td>66</td>
    <td>166</td>
    <td>43</td>
    <td>80</td>
    <td>7</td>
    <td>27</td>
    <td>185</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>194</td>
    <td>27</td>
    <td>95</td>
    <td>64</td>
    <td>35</td>
    <td>9</td>
    <td>6</td>
    <td>213</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>56</td>
    <td>105</td>
    <td>39</td>
    <td>65</td>
    <td>56</td>
    <td>5</td>
    <td>93</td>
    <td>55</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>67</td>
    <td>89</td>
    <td>96</td>
    <td>42</td>
    <td>56</td>
    <td>7</td>
    <td>56</td>
    <td>102</td>
  </tr>
  <tr>
    <td>Llama-3.1-B-PLLaM</td>
    <td>60</td>
    <td>208</td>
    <td>65</td>
    <td>71</td>
    <td>32</td>
    <td>11</td>
    <td>255</td>
    <td>120</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>43</td>
    <td>40</td>
    <td>122</td>
    <td>29</td>
    <td>21</td>
    <td>11</td>
    <td>29</td>
    <td>61</td>
  </tr>
  <tr>
    <td>Llama-3.1-B-PLLaM (SFT)</td>
    <td>56</td>
    <td>105</td>
    <td>39</td>
    <td>65</td>
    <td>56</td>
    <td>5</td>
    <td>93</td>
    <td>55</td>
  </tr>
  <tr>
    <th colspan="9">5-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>88</td>
    <td>44</td>
    <td>256</td>
    <td>35</td>
    <td>62</td>
    <td>9</td>
    <td>2</td>
    <td>281</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>193</td>
    <td>110</td>
    <td>70</td>
    <td>33</td>
    <td>20</td>
    <td>8</td>
    <td>136</td>
    <td>225</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td>
    <td>85</td>
    <td>52</td>
    <td>272</td>
    <td>31</td>
    <td>48</td>
    <td>7</td>
    <td>8</td>
    <td>258</td>
  </tr>
  <tr>
    <td>Mistral 8x2B</td>
    <td>130</td>
    <td>110</td>
    <td>70</td>
    <td>33</td>
    <td>20</td>
    <td>8</td>
    <td>136</td>
    <td>225</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>153</td>
    <td>23</td>
    <td>163</td>
    <td>53</td>
    <td>27</td>
    <td>9</td>
    <td>2</td>
    <td>239</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>64</td>
    <td>101</td>
    <td>50</td>
    <td>42</td>
    <td>38</td>
    <td>10</td>
    <td>100</td>
    <td>68</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>66</td>
    <td>89</td>
    <td>66</td>
    <td>41</td>
    <td>44</td>
    <td>10</td>
    <td>76</td>
    <td>76</td>
  </tr>
  <tr>
    <td>Llama-3.1-B-PLLaM</td>
    <td>19</td>
    <td>237</td>
    <td>24</td>
    <td>68</td>
    <td>25</td>
    <td>10</td>
    <td>296</td>
    <td>36</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>60</td>
    <td>28</td>
    <td>87</td>
    <td>34</td>
    <td>26</td>
    <td>11</td>
    <td>19</td>
    <td>80</td>
  </tr>
  <tr>
    <td>Llama-3.1-B-PLLaM (SFT)</td>
    <td>72</td>
    <td>26</td>
    <td>70</td>
    <td>44</td>
    <td>22</td>
    <td>10</td>
    <td>27</td>
    <td>80</td>
  </tr>
</table>

Table 9: Detailed Type I and Type II error counts for various models for *Extended* dataset type.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Erotic</th>
    <th colspan="2">Ambiguous</th>
    <th colspan="2">Neutral</th>
  </tr>
  <tr>
    <th>Type I</th>
    <th>Type II</th>
    <th>Type I</th>
    <th>Type II</th>
    <th>Type I</th>
    <th>Type II</th>
  </tr>
  <tr>
    <td>HerBERT Base</td>
    <td>53</td>
    <td>40</td>
    <td>47</td>
    <td>55</td>
    <td>56</td>
    <td>54</td>
  </tr>
  <tr>
    <td>HerBERT Large</td>
    <td>48</td>
    <td>36</td>
    <td>48</td>
    <td>45</td>
    <td>44</td>
    <td>59</td>
  </tr>
  <tr>
    <td>RoBERTa Base</td>
    <td>37</td>
    <td>46</td>
    <td>46</td>
    <td>46</td>
    <td>62</td>
    <td>51</td>
  </tr>
  <tr>
    <td>RoBERTa Large</td>
    <td>43</td>
    <td>40</td>
    <td>49</td>
    <td>43</td>
    <td>51</td>
    <td>55</td>
  </tr>
  <tr>
    <th colspan="7">0-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>74</td>
    <td>25</td>
    <td>301</td>
    <td>23</td>
    <td>1</td>
    <td>279</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>68</td>
    <td>238</td>
    <td>45</td>
    <td>68</td>
    <td>284</td>
    <td>88</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B-Instruct</td>
    <td>57</td>
    <td>96</td>
    <td>320</td>
    <td>30</td>
    <td>48</td>
    <td>237</td>
  </tr>
  <tr>
    <td>Mistral 8x2B</td>
    <td>58</td>
    <td>90</td>
    <td>241</td>
    <td>30</td>
    <td>35</td>
    <td>172</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>142</td>
    <td>21</td>
    <td>109</td>
    <td>59</td>
    <td>20</td>
    <td>178</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>87</td>
    <td>39</td>
    <td>98</td>
    <td>52</td>
    <td>36</td>
    <td>110</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>106</td>
    <td>32</td>
    <td>125</td>
    <td>49</td>
    <td>17</td>
    <td>144</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td>
    <td>10</td>
    <td>239</td>
    <td>8</td>
    <td>72</td>
    <td>305</td>
    <td>12</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>22</td>
    <td>56</td>
    <td>67</td>
    <td>34</td>
    <td>60</td>
    <td>21</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td>
    <td>32</td>
    <td>30</td>
    <td>79</td>
    <td>39</td>
    <td>38</td>
    <td>47</td>
  </tr>
  <tr>
    <th colspan="7">1-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>95</td>
    <td>21</td>
    <td>327</td>
    <td>28</td>
    <td>0</td>
    <td>329</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>206</td>
    <td>37</td>
    <td>232</td>
    <td>62</td>
    <td>36</td>
    <td>375</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B-Instruct</td>
    <td>62</td>
    <td>55</td>
    <td>243</td>
    <td>22</td>
    <td>11</td>
    <td>239</td>
  </tr>
  <tr>
    <td>Mistral 8x2B</td>
    <td>79</td>
    <td>61</td>
    <td>229</td>
    <td>31</td>
    <td>18</td>
    <td>193</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>140</td>
    <td>19</td>
    <td>191</td>
    <td>52</td>
    <td>2</td>
    <td>232</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>89</td>
    <td>57</td>
    <td>86</td>
    <td>60</td>
    <td>67</td>
    <td>113</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>70</td>
    <td>55</td>
    <td>169</td>
    <td>39</td>
    <td>24</td>
    <td>136</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td>
    <td>72</td>
    <td>197</td>
    <td>92</td>
    <td>69</td>
    <td>232</td>
    <td>127</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>34</td>
    <td>43</td>
    <td>70</td>
    <td>33</td>
    <td>50</td>
    <td>39</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td>
    <td>38</td>
    <td>24</td>
    <td>96</td>
    <td>31</td>
    <td>34</td>
    <td>58</td>
  </tr>
  <tr>
    <th colspan="7">5-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>88</td>
    <td>29</td>
    <td>303</td>
    <td>29</td>
    <td>0</td>
    <td>290</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>185</td>
    <td>117</td>
    <td>129</td>
    <td>71</td>
    <td>138</td>
    <td>236</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B-Instruct</td>
    <td>67</td>
    <td>57</td>
    <td>269</td>
    <td>19</td>
    <td>5</td>
    <td>265</td>
  </tr>
  <tr>
    <td>Mistral 8x2B</td>
    <td>105</td>
    <td>37</td>
    <td>253</td>
    <td>41</td>
    <td>16</td>
    <td>243</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>106</td>
    <td>31</td>
    <td>240</td>
    <td>44</td>
    <td>1</td>
    <td>244</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>102</td>
    <td>54</td>
    <td>62</td>
    <td>62</td>
    <td>67</td>
    <td>97</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>45</td>
    <td>75</td>
    <td>165</td>
    <td>38</td>
    <td>40</td>
    <td>103</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td>
    <td>24</td>
    <td>247</td>
    <td>43</td>
    <td>69</td>
    <td>290</td>
    <td>38</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>56</td>
    <td>28</td>
    <td>63</td>
    <td>38</td>
    <td>31</td>
    <td>50</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td>
    <td>45</td>
    <td>24</td>
    <td>64</td>
    <td>40</td>
    <td>46</td>
    <td>49</td>
  </tr>
</table>

Table 10: Detailed Type I and Type II error counts for various models for Core dataset type.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">Erotic</th>
    <th colspan="2">Neutral</th>
  </tr>
  <tr>
    <th>Type I</th>
    <th>Type II</th>
    <th>Type I</th>
    <th>Type II</th>
  </tr>
  <tr>
    <td>HerBERT Base</td>
    <td>34</td>
    <td>34</td>
    <td>34</td>
    <td>34</td>
  </tr>
  <tr>
    <td>HerBERT Large</td>
    <td>31</td>
    <td>28</td>
    <td>28</td>
    <td>31</td>
  </tr>
  <tr>
    <td>RoBERTa Base</td>
    <td>31</td>
    <td>23</td>
    <td>23</td>
    <td>31</td>
  </tr>
  <tr>
    <td>RoBERTa Large</td>
    <td>25</td>
    <td>29</td>
    <td>29</td>
    <td>25</td>
  </tr>
  <tr>
    <th colspan="5">0-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>115</td>
    <td>1</td>
    <td>1</td>
    <td>115</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>76</td>
    <td>191</td>
    <td>191</td>
    <td>76</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B-Instruct</td>
    <td>152</td>
    <td>19</td>
    <td>19</td>
    <td>152</td>
  </tr>
  <tr>
    <td>Mistral 8x22B</td>
    <td>95</td>
    <td>33</td>
    <td>33</td>
    <td>95</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>140</td>
    <td>18</td>
    <td>18</td>
    <td>140</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>87</td>
    <td>20</td>
    <td>20</td>
    <td>87</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>127</td>
    <td>4</td>
    <td>4</td>
    <td>127</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td>
    <td>6</td>
    <td>246</td>
    <td>246</td>
    <td>6</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>17</td>
    <td>34</td>
    <td>34</td>
    <td>17</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td>
    <td>31</td>
    <td>20</td>
    <td>20</td>
    <td>31</td>
  </tr>
  <tr>
    <th colspan="5">1-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>132</td>
    <td>1</td>
    <td>1</td>
    <td>132</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>263</td>
    <td>34</td>
    <td>34</td>
    <td>263</td>
  </tr>
  <tr>
    <td>Llama-3.1 70B-Instruct</td>
    <td>153</td>
    <td>10</td>
    <td>10</td>
    <td>153</td>
  </tr>
  <tr>
    <td>Mistral 8x22B</td>
    <td>134</td>
    <td>18</td>
    <td>18</td>
    <td>134</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>149</td>
    <td>1</td>
    <td>1</td>
    <td>149</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>127</td>
    <td>35</td>
    <td>35</td>
    <td>127</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>79</td>
    <td>19</td>
    <td>19</td>
    <td>79</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td>
    <td>76</td>
    <td>156</td>
    <td>156</td>
    <td>76</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>40</td>
    <td>47</td>
    <td>40</td>
    <td>40</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td>
    <td>127</td>
    <td>35</td>
    <td>35</td>
    <td>127</td>
  </tr>
  <tr>
    <th colspan="5">5-shot</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>113</td>
    <td>0</td>
    <td>0</td>
    <td>113</td>
  </tr>
  <tr>
    <td>C4AI Command-R</td>
    <td>199</td>
    <td>96</td>
    <td>96</td>
    <td>199</td>
  </tr>
  <tr>
    <td>Llama 3.1 70B-Instruct</td>
    <td>159</td>
    <td>4</td>
    <td>4</td>
    <td>159</td>
  </tr>
  <tr>
    <td>Mistral 8x22B</td>
    <td>167</td>
    <td>10</td>
    <td>10</td>
    <td>167</td>
  </tr>
  <tr>
    <td>Bielik-11B-v2.3-Instruct</td>
    <td>135</td>
    <td>3</td>
    <td>3</td>
    <td>135</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B</td>
    <td>111</td>
    <td>46</td>
    <td>46</td>
    <td>111</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-8x7B</td>
    <td>44</td>
    <td>32</td>
    <td>32</td>
    <td>44</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM</td>
    <td>28</td>
    <td>200</td>
    <td>200</td>
    <td>28</td>
  </tr>
  <tr>
    <td>PLuM-Mistral-12B (SFT)</td>
    <td>73</td>
    <td>15</td>
    <td>15</td>
    <td>73</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-PLLaM (SFT)</td>
    <td>43</td>
    <td>17</td>
    <td>17</td>
    <td>43</td>
  </tr>
</table>

Table 11: Detailed Type I and Type II error counts for various models for Basic dataset type.
E  Computational Details

The details provided below correspond to the Full version of the dataset.

The encoder-based models were trained for approximately 5 hours and 10 hours, respectively, for the Base and Large versions, using A100 (40GB) GPU with early stopping. The batch size was set to 8 for both model variants. Training of the PLLuM family models on our dataset took approximately 3 hours for the Llama-3.1-8B-PLLuM (SFT) model and 4 hours for the PLLuM-Mistral-12B (SFT) model. These models were trained on a distributed setup consisting of 2 nodes, each equipped with 4 H100 (96GB) GPUs, ensuring parallel processing across GPUs for enhanced training efficiency.

Reviewer paper summaries: This paper introduces forePLay, a manually annotated, Polish-language dataset for erotic content detection.

This paper introduces **forePLay**, a novel dataset for erotic content detection in Polish. It consists of 24,678 sentences, labeled with a taxonomy of 5 distinct categories: erotic, ambiguity, violence, social unacceptability, and neutral, expanding previous work on binary classification for the task. The authors define 4 different task definitions - binary, multi-class with 3, 4, and 5 classes. They explore various approaches, including Polish transformer-based models, Polish-specific LLMs, and state-of-the-art multilingual and general-purpose (open-source and proprietary) models to evaluate their erotic content detection capabilities. The experimental settings include 0-shot, few-shot, and fine-tuning.
Summary of strengths: 1. This paper addresses a gap in resources for erotic-content detection for non-English languages.
2. The authors take great care in ensuring the dataset is from diverse sources and that the annotations are high quality.
3. The baselines are thorough and provide a great starting point for future work on this task.
4. The dataset is fairly large (nearly 25K examples) for a manually annotated dataset.

**Novelty**
- The dataset provides a new level of granularity that goes beyond the largely explored binary classification tasks.
- Four task levels are defined by logically grouping categories based on the problem setting.

**Methodology**
- forePLay comprises 24k sentences in Polish, collected from 905 unique texts (both non-professional and professional), ensuring comprehensive coverage of erotic content.
- The dataset collection process is well-defined and meticulous. The authors share a detailed explanation of the applied criteria during collection.
- The annotation process contains important details regarding annotator selection, bias mitigation, task distribution, and label assignment decisions. Furthermore, the work evaluates annotation quality and discusses important problems identified during the annotation process.
- Experimental settings are well-defined and clear. 

**Findings**
- The study identifies specialized Polish language models as top-performing models compared to multilingual alternatives, highlighting the importance of language-specific models on subjective tasks.
Summary of weaknesses: 1. The paper does not mention how the annotators were recruited.
2. The authors report F1 scores for the tested models. However, given the task, it would be helpful to see label specific performance measures. In particular, it would be helpful to see if the models are more prone to Type I or Type II errors for erotic vs non erotic labels, and if the distribution of error types changes as the label space is expanded.

* The main weakness of the dataset is the class imbalance. While this probably reflects the actual distribution present in data sources, the imbalance topic is not thoroughly addressed in this work. For example, using a better sampling strategy or conducting balanced training (e.g., using focal loss).
* The metric used is F1 but I did not find a mention of the used average, e.g., macro, weighted. 
* Due to the imbalanced nature of the data, per-class evaluation is necessary. It is not possible to report this on every model but at the very least it can be reported for the best model of each group. 
* Insufficient error analysis. Most of the focus is on quantitative error analysis but the work will benefit from such a discussion, considering the close performance between the best models.
Comments, suggestions and typos: 1. Table 6 is mentioned before Table 5. These should be switched in ordering.
2. Capitalize “table” in lines 247 and 336.

- Capitalize all table references: table 1 -> Table 1 
- Binary task is defined as "Basic" but referenced as "Base" in several places in the paper (Table 5, Table 7, also in section 5.4 line 514)
- In experimental settings, the dataset split details are missing. This is needed for better reproducibility
- (Minor) The abbreviation of female annotators as F1 can be slightly misleading at first, considering F1-score is sometimes used as a metric for annotator quality & agreement.
Overall assessment scores: 4.5

4
Metareview score: 4
Metareview: This paper introduces forePLay, a new dataset for erotic content detection in Polish. 
The corpus contains 24K sentences, labeled for five distinct categories.
Existing Polish-specific and multilingual transformer models and LLMs are evaluated on erotic content detection.
*Note from AC: please fix the captions so they always appear below the tables and figures for camera-ready.*
Summary of reasons to publish: This new erotic content detection dataset not only provides more granular annotations but also mitigates the lack of data resources on non-English languages.
The dataset ensures diverse sources and is of high quality.
The paper also contains important details regarding annotator selection, bias mitigation, task distribution, and label assignment decisions.
A wide set of experiments are conducted, including different types of models, from 0-shot to fine-tuning, and on different classification scenarios.

In the rebuttal, the authors addressed the following concerns:
- Only a small amount of sentences are erotic in the dataset: this represents the distribution of non-erotic vs. erotic classes in real-world data -- reason for not upsampling erotic data; 
- Types of model misclassifications: (a) content being flagged as inappropriate, and (b) model assigning a class that fell outside the predefined set of responses;
- backgrounds of the annotators:  3 female annotators (members of the internal team) and 3 part-time male contractors
- train/dev/test partitions are also provided in the rebuttal