Content Moderation for Evolving Policies using Binary Question Answering
Year: 2023

Abstract:
Content moderation on social media is governed by policies that are intricate and frequently updated with evolving world events. However, automated content moderation systems often restrict easy adaptation to policy changes and are expected to learn policy intricacies from limited amounts of labeled data, which make effective policy compliance challenging. We propose to model content moderation as a binary question answering problem where the questions validate the loosely coupled themes constituting a policy. A decision logic is applied on top to aggregate the themespecific validations. This way the questions pass theme information to a transformer network as explicit policy prompts, that in turn enables explainability. This setting further allows for faster adaptation to policy updates by leveraging zero-shot capabilities of pre-trained transformers. We showcase improved recall for our proposed method at 95% precision on two proprietary datasets of social media posts and comments respectively annotated under curated Hate Speech and Commercial Spam policies.

heading: 1 Introduction
text: Social media platforms use content moderation to safeguard users from abuse, harassment, malicious attacks, spam, etc. This moderation process is governed by a set of community policies1. For example, to shield the users from undesired spammy advertising of illegal products/services, social media platforms generally maintain a Commercial Spam (CS) policy2. The large volume of content
1The professional community policy maintained by LinkedIn https://www.linkedin.com/legal/ professional-community-policies or Facebook Community standards https://transparency.fb.com/en-gb/ policies/community-standards/
2 LinkedIn’s illegal, dangerous, and inappropriate commercial policy: https://www.linkedin.com/help/ linkedin/answer/137373 and similarly Facebook’s commerce policy: https://www.facebook.com/policies_ center/commerce/
generated on social media platforms necessitates building automated systems for content moderation to scale policy-specific validations. (Fortuna and Nunes, 2018; MacAvaney et al., 2019).
Traditionally, automated content moderation systems are mostly binary classifiers (Hovold, 2006; Sakkis et al., 2001) often aided by pre-processing (Naseem et al., 2021) and additional tasks such as intent identification (Agarwal and Sureka, 2017). Recent approaches involve fine-tuned Large Language Models (LLMs) (Caselli et al., 2020; Tan et al., 2020), putting attention on suspicious pieces of text (Pavlopoulos et al., 2017), or reformulating the problem as multi-task learning (Kapil and Ekbal, 2020) or natural language inference (Yin et al., 2019; Goldzycher and Schneider, 2022).
However, policy compliance in automated content moderation still remains a challenge due to two primary reasons: (1) The governing policies are likely to contain intricacies arising from various aspects like content-specific edge cases, context driven interpretations, and exceptions. For example, Table 1 documents a typical Hate Speech policy (prohibiting hateful contents targeting inherent traits such as gender, race etc.) that can have complicated samples where decision making is difficult. (2) To keep up with world events and their direct impact on content distribution, policies may need to be updated somewhat frequently.
The common industry practice considers policy as a single atomic concept and formulates content moderation as binary classification problem. Here the policy appears to the classifier as a black-box abstract concept yet it is expected to learn even the minute intricacies of the policy only through the labeled data. This leads to three major production challenges: (1) Labeled data are limited in quantity. (2) The non-stationary distribution of content on social media continuously evolves in response to world events resulting in label and concept drift (Gama et al., 2014; Yamazaki et al., 2007). (3)
561
There is no direct way to reuse an existing model following policy update. Instead, one has to reannotate data for the updated policy and develop a fresh model. From industry perspective, this incurs additional labeling and development cost leading to compliance delays that leaves the user on the platform less protected for a prolonged period.
To get a better understanding of content policies we take an example CS policy that prohibits advertising/selling of illegal products from any of the three categories (hereafter called themes), namely Human Body Parts, Recreational Drugs, and Cryptocurrency (see Table 2). Thus, a policy can be seen as a collection of loosely coupled themes (the smallest, logically coherent, and well defined granularity of a policy) threaded together by a decision logic (here if the content violates any of the themes it will be marked as commercial spam). Breaking down a policy into themes has two benefits. (1) Themes are independent and focused thus they tend to be less ambiguous. (2) A policy update boils down to addition of new themes or removal of old themes with changes in the decision aggregation logic. For example, an updated CS policy may clear Cryptocurrency and introduce Pharmaceuticals as a new prohibited item (see Table 2).
When we consider the policy as a set of themes combined by a decision logic, it enables us to formulate the task of policy compliance as a binary Question Answering problem (Clark et al., 2019) that leverages a pre-trained Large Language Model (LLM) as described in Figure 1. This formulation has four advantages. (1) The theme information can be passed to the LLM in the form of explicit prompts, in this case, binary questions (answered Yes or No). This is similar to a prompt-based learning (Liu et al., 2023) approach that enables a better understanding of the policy. (2) Prompting enables
leveraging zero-shot capabilities of LLMs for understanding the question-content relation to validate less prevalent or newly added themes with no or few data samples. (3) The decision logic gets decoupled from the model. This simplifies learning and enables fast adaptation to policy changes. (4) The individual theme validations provide explainability useful for fine-grained monitoring and performance tuning (can be used for transparency and fairness requirements for social media).
The key highlights of this paper are as follows:. (1) In Section 3, we propose a binary Question Answering based Content Moderation (QnA-CM) system. Here, we leverage the policy structure that allows reformulating the problem of content moderation as a generic task of binary QnA. Going beyond Clark et al. (2019); Saeidi et al. (2021a) that deal with more syntactical and factual questions, with QnA-CM we aim to answer semantically involved theme-validations. (2) Contrary to BoolQ (Clark et al., 2019) in QnA-CM to maintain diversity and limit class imbalance in the training set, we undertake a sampling strategy detailed in Section 3. (3) We further propose a scalable multi-level inference strategy in Section 3 that enables QnACM to perform at near computational cost of binary classifiers while offering greater explainability. (4) Using questions QnA-CM leverages explicit policy knowledge and consequently gains agility to policy changes in a zero-shot manner, as demonstrated in a simulation study in Section 4.

heading: 2 Related Works
text: Content moderation systems usually employ binary (Sakkis et al., 2001) or multi-class classifiers (Founta et al., 2018) to label content. A binary classifier ignores themes altogether by treating policy as a black box seen through the lens of spam
and clear labels. Even though a multi-class classifier may consider themes it is not agile to policy changes. With the advent of LLMs (Devlin et al., 2018), models like TextCNN (Kim, 2014), XGBoost (Chen and Guestrin, 2016) using external word embeddings like Glove (Pennington et al., 2014) have been outperformed. Notable LLMbased content moderation systems, primarily designed for Hate Speech detection, usually fail in production due to language-specific interjections (Nozza, 2021), limited data availability (Uzan and HaCohen-Kerner, 2021), demand for fine grained subjective labels (Mollas et al., 2022), theme imbalance (Plaza-Del-Arco et al., 2021), and high response time (Goldzycher and Schneider, 2022).
A pre-trained LLM can be fine-tuned to perform text understanding tasks such as binary question answering as in BoolQ (Clark et al., 2019). However, BoolQ is trained to answer in Yes or No responses to content-specific factual questions thus cannot be directly applied to the task of policy compliance. Even though BoolQ inspired question answering along with rule based deductive reasoning have found some success in content validation (Saeidi et al., 2021b; Saeed et al., 2021), they use simple clearly defined policies and did not investigate the applicability in content moderation that requires answering semantically involved questions. To elaborate, policies governing content moderation often use legal language that are difficult to process by LLMs (Moro and Ragazzi, 2022; Khazaeli et al., 2021; Ravichander et al., 2019). For short, focused, and well defined insurance policies, expressing the
rules as decision trees may be useful (Kotonya et al., 2022) but that neither extends to capture the intricacies in social media content moderation policies or formally characterize their updates.

heading: 3 Methodology
text: Preliminaries: Let us take a set X = {x1,x2, · · · ,xn} of n text contents paired with a set of labels YP = {Spam,Clear}n where P is the underlying policy. Due to the likely imbalance between s Spam and c Clear contents n = c + s and c = rs where r > 1 is the imbalance ratio. A policy P as mentioned in Section 1 usually consists of a set of themes T and a decision logic (combination of logical operators like AND, OR, etc.) D to combine the theme-specific validations to reach a final Spam or Clear label or P = (T,D).
To elaborate, content policies in social media typically have a primary theme of “common intent" in T . For example, in CS this can be “sale of prohibited items" or in hate speech a “hateful sentiment". Evidently, if this primary theme is violated then only it makes sense to proceed with the checks on the other themes for finding a spam. Each of the other themes individually covers a certain “specific" rule under the policy such as a regulated product like recreational drugs in CS and inherent traits like gender in hate speech. The policy provides the decision tree D that in its commonly preferred form marks a content Spam if “common intent" is violated along with (logical AND) any (i.e. logical OR) other specific theme is contravened.
Now P when circulated to the user, should
phrase T as a guideline to assist the user in creating good quality content. However, when P is provided to human reviewers, T can be rephrased as questions Q as that is more intuitive for validating a content. For example, a publicly circulated Hate Speech policy may state “Please do not create hateful contents that target the inherent traits of an individual or group.” To a reviewer this may be rephrased as a set of binary answerable questions Q along with a decision logic D to aggregate the answers, as shown in Figure 2. Similarly, in QnACM mimicking a human reviewer we rephrase T as Q such that the answers of Q can be logically combined by D.
Formally, for a policy with k themes the set Q = {q1,q2, · · · ,qm} contains m validatory questions. Here, m ≥ k and equality is achieved when each theme has exactly one validating question. The decision tree D is a boolean function that maps from {0, 1}m to {0, 1} using logical operators. We are representing Yes as 1 and No as 0 to match the implementation while ⊗ and ⊕ respectively denote the logical operators AND and OR. Binary Question Answering Using LLMs: We follow from LLM classifiers (Devlin et al., 2018) and BoolQ (Clark et al., 2019), for validating a content x ∈ X against a question q ∈ Q. The input i concatenates [CLS], q, [SEP], and x in the order, where [CLS] and [SEP] are special tokens. The output f(i) of the LLM f summarizes the input i in the feature space at f ( i[CLS] ) i.e. the dimension corresponding to the [CLS] input token (Devlin et al., 2018). This f ( i[CLS] ) is sent through fully-connected layers to map to the two classes (Clark et al., 2019). After applying softmax to the logits, the model will output probability scores Pr(1|(q,x)) and Pr(0|(q,x)) respectively for 1 (Yes) and 0 (No) responses. A threshold θ converts the probability scores to binary labels. This can be trained end-to-end with a loss such as binary cross-entropy. Figure 4 illustrates the architecture. Training of QnA-CM: In the QnA-CM training set each sample is a question-content pair with 1 (Yes) or 0 (No) label. To form such a training data we ask the same set of questions Q to every content x in X . However, this may result in severe class imbalance depending on r and m. For simplicity without loss of generality, let us assume k = m, a spam content violates only one theme, and a positive theme violation corresponds to a Yes answer only. Thus, we have a total of sm(r + 1) questions in
the training data, where s of them are answered by Yes and the rest (s(m − 1) from spam and rsm from clear) are answered by No, resulting in an imbalance of (m+mr − 1). A naive solution of sampling random No answering question-content pairs may not provide a quality training set.
We sample diverse question-content pairs with label 0 (No) in three ways: (S1) Pair a Clear sample with a random question with probability νn and assign it a 0 (No) label. (S2) Take a Spam x that answers Yes to qj. Pair x with probability νs with any q ∈ Q \ {qj} and label it as 0 (No). (S3) Use theme-specific weak classifiers using models like TextCNN or pre-trained natural language inference models like BART (Lewis et al., 2019) to find the qj with highest confidence (above ω) that matches with a Clear sample x. With a probability νh, pair x with qj and label it as 0 (No).
The “common intent” of a policy is expressed through the top-level spam and clear labels. To utilize this additional information and learn the commonalities across themes to aid zero-shot generalization, we sample Spam and Clear contents respectively with probability ν+ and ν− and pair them with a “common intent” validating question in the training set. Note that, in the process of building the training set, we introduce five new data dependent hyperparameters in QnA-CM, namely νs, νh, νn, ν+, ν−, and ω.
Inference using QnA-CM: We propose a scalable multi-level inference strategy for QnA-CM as described in Figure 3. Here we exploit two peculiarities of the content moderation ecosystem. (1) Spammy content is commonly very less frequent than clear content. (2) In our two-level inference strategy. In the first level L1, we match the content against a single question representing the common intent of the policy. Only if the content is matched in L1, we proceed to the second level of L2 to validate it against all the m theme-specific questions, otherwise we directly mark it as clear. This way only the potential spammy content will be validated against all m theme-validating questions while the rest will be cleared in L1 with a similar computation cost of a binary classifier. In other words, the computational overhead will only be zm/(1+r) times in practice where z is the ratio of potential spam to actual spam content. Typically in production r >> z (tuning distinct θs for L1 and L2 offers finer control over z) and m is not large thus zm/(1 + r) remains close to 1, thus asserting

heading: Hate
text: scalability similar to traditional methods.

heading: 4 Experiments
text: Experimental Protocol: We use two proprietary long-tailed text datasets, both sampled from content publicly posted on social media during 2021-2023 (both are sampled from the same social media to maintain consistency). (D1) Comments in four languages namely English, Spanish, French, and Portuguese. (D2) Text part of Feed Posts made in English. For our experiments, we curate two typical policies by selectively amalgamating the ones used by various social media platforms. (P1) A CS policy that contains 17 themes (described in terms of Q and D in Appendix A.2). (P2) A Hate Speech policy that employs hateful Q and D
as shown in Figure 2. The D1 dataset is validated against P1 while D2 is labeled with P2 (datasets are detailed in Appendix A.3).
We take 4 English and 2 multilingual pre-trained LLMs for our experiments (listed in Appendix A.4 with additional network architecture and hyperparamater details). For all the experiments we compare a fine-tuned LLM-based baseline binary classifier that maps the [CLS] token embedding of the backbone to Spam or Clear labels by a multi-layer perceptron network againstst a QnA-CM model with the same backbone. Content moderation systems in production aim to achieve better recall with high precision such that the users (and reviewers) are minimally affected by false positives. Thus, we use recall value at 95% precision level for comparing
among contenders (note that accuracy is not suitable given the class imbalance while GMean is not informative as content moderators do not need to focus on true negatives (Mullick et al., 2020)).
We demonstrate performance of QnA-CM across two dimensions. (1) To train with the full dataset, we retain all the themes in training, validation, and test set mimicking a long standing static policy. Here we aim to evaluate how well QnA-CM employs prompting through questions to handle the long tallied theme distribution. (2) For the CS policy, we simulate a policy change (Hate Speech policy is commonly static as the inherent traits are well defined) where new themes are introduced in the policy (see Appendix A.5). Thus, we train the model only on previously existing themes while inferring on newly introduced ones to evaluate how QnA-CM employs prompting and zero-shot capability of LLM to adapt to policy update.
Performance of QnA-CM Compared to Binary Classifiers: We can observe from Table 3 (detailed in Table 12) that for both the policies, QnA-CM is achieving a better performance on average over the four backbone LLMs than the baseline binary classifiers in terms of R@95P. The performance improvement is more apparent in the case of simulated policy changes in CS, indicating better adaptability of QnA-CM to policy updates. Further, in the training with full data, the better performance of QnA-CM attests to the usefulness of theme-specific knowledge prompted through the questions.
Explainability of QnA-CM: In Table 4 we report the performance at 95% policy-level precision for three highly prevalent themes after training QnACM with full data. The Table 4 also lists down
the metrics at 95% policy-level precision for three themes newly introduced through policy change. This theme-level performance provided by QnACM allows finer monitoring and tuning. For example, we can prioritize data collection to improve recall for Human Body Parts while increasing the respective threshold can provide better precision.
Ablation Study: To understand how the proposed training strategy of QnA-CM aids in learning we perform an ablation study using the D1 English Comments datasets labeled with CS policy. In Table 5 we see that the performance of QnA-CM greatly improves as hard “No” answering questions are added on top of the randomly selected ones (i.e. S1+S2+S3 as in Section 3). QnA-CM further benefits, especially in a policy update situation, when the policy-level question for the “Common Intent” is additionally used during training.
Multilingual Inference on D1 Comments Dataset: We fine-tune two multilingual LLMs for QnA-CM using the D1 English comments training dataset. However, we infer on the samples from the three other languages along with the English test set. We keep the questions in English, to validate how well QnA-CM can adapt to multilingual content without explicit multilingual fine tuning. We see from Table 6 that QnA-CM elevates the performance of the models compared to the baselines indicating the usefulness of prompt-based learning through questions even for bilingual inputs. Importance of Prompted Learning: We showcase benefits of our prompt based learning framework QnA-CM on Hate Speech detection. In the real world industry setting, Hate Speech detection is often plagued with false positives that arise due to
binary classifiers getting confused between “hateful” content and “hate speech”. This distinction is important for fair assessment of the content severity and user regulation. Therefore, understanding of inherent traits along with hateful intent becomes crucial, which can be achieved via questions in QnA-CM. We demonstrate this setting by two experiments. (1) At 95% precision level we compare the average theme-specific recall over four backbone LLMs for each of the 6 inherent traits for the D2 Posts dataset labeled against the Hate Speech policy. From Table 7 (full results in Table 13) we can see that QnA-CM performs better in all cases irrespective of the imbalance thus validating that the questions are providing useful information to the model. (2) We take 116 English posts from social media that annotators marked as hateful but do not target any inherent traits thus are not Hate Speech. The Table 7 (and Table 14 in Appendix) shows QnA-CM achieving disentanglement among the “hateful” and inherent traits thus offering a lower false positive rate for Hate Speech.

heading: 5 Conclusion
text: In this work, we model content moderation as a binary question answering problem where the questions act like prompts, validating various themes belonging to a content policy. This further allows faster adaptation to policy updates by leveraging zero-shot capabilities of pre-trained transformers. Our experiments show 7% absolute improvement in recall over the binary classification setting. In case of policy updates we achieve 22% absolute recall improvement as well, without any additional training. Furthermore we show improved recall on multilingual data with QnA-CM fine-tuned only on English. We also show improved recall and reduced false positives for Hate Speech using QnA-
CM. All these facilitate an agile response to policy updates by prompt injection thus limiting member exposure to spam. In the future, we aim to investigate the applicability of open source datasets with recently developed large generative models with high natural language understanding and promptdriven zero-shot capabilities such as GPT series (Brown et al., 2020) or LLaMA (Touvron et al., 2023) in the QnA-CM framework.

heading: A Appendix

heading: A.1 Model Architecture
text: The model architecture used for QnA-CM is illustrated in the following Figure 4.

heading: A.2 Policies Curated for the Experiments
text: We have generated a CS policy for our experiments that marks a content as Spam if it comes with a primary common intent of “promoting, facilitating access to, distributing, or attempting to sell” any of the 17 types of illegal or regulated products or services (that correspond to 17 themes). Essentially Q contains one question for the common intent and 17 others covering the individual themes. The decision tree D is similar to Figure 2 where the themes are aggregated by logical OR and combined with the policy intent with a logical AND. Moreover, the inference directly follows from Figure 3. We formulate this CS policy by taking inspiration from the community policies publicly circulated by LinkedIn, Facebook (see footnote 2) and Twitter (https://business.twitter.com/en/help/ ads-policies.html). Note that, this CS policy is not an exact copy of any of the three social media but rather a selective amalgamation, with additions such as Cryptocurrency, Occult, etc and removal of themes like unauthorized sale of digital media, adult contents etc.
For the Hate Speech policy we have considered the traditional sense i.e. hateful content targeting inherent traits of person or group. Again this is inspired by the LinkedIn Hateful and Derogatory Content Policy (https://www.linkedin. com/help/linkedin/answer/a1339812),
Facebook Hate Speech Policy (https: //transparency.fb.com/en-gb/policies/ community-standards/hate-speech/), and Twitter policy on Hateful Conduct (https:// help.twitter.com/en/rules-and-policies/ hateful-conduct-policy). However, like the CS policy the Hate Speech policy curated by us does not directly follow any of the social media in particular rather it combines the essence of the three, focussing on some key inherent traits only. Moreover, we have observed that even highly capable LLMs like BART (Lewis et al., 2019) often get confused with particularly identifying fine grained inherent traits. For example, it is a complicated problem for the LLMs to disambiguate between Ethnicity, National Origin, Religious Affiliation, and Immigration Status as they often occur together and can even be interpreted synonymously in the same content. Similarly it has been found that LLMs are not powerful enough to properly distinguish between gender and sex as different inherent traits. Hence, while creating the question set as described in Figure 2 we joined these fine grained inherent traits together.
One key challenge for QnA-CM is to formulate questions from the policy definition as the same statement can be rephrased in multiple ways. To remove this possible ambiguity during training we left the question design to the unanimous decision by a team of 5 reviewers who are well acquainted with the two curated policies. We felt this is a reasonable approach as this directly reflects the human reviewers’ understanding of a content policy and best aids the mimicking of that in QnA-CM.

heading: A.3 D1 Comments and D2 Posts Datasets
text: For both the D1 Comments and D2 Posts dataset, each content is labeled against the respective policy by two annotators and conflicts are resolved through a third opinion. We have used a group of 5 annotators who are all trained on the two curated policies. The D1 Comments dataset has three primary features to replicate real world scenarios. (1) The distribution of examples over the languages is long tailed i.e. there is imbalance in the dataset across languages (2) The distribution of the samples over the themes is also long tailed for each of the four languages. (3) There is an imbalance between the number of Spam and Clear instances. We achieve this by using a couple of multinomial dis-
tributions respectively with distinct language and theme selection probabilities along with a biased Bernoulli distribution. The final data distribution over the language, content labels, and themes is documented in Figure 5.
For the D2 Posts dataset we have applied a strategy similar to the one used for Comments. Here the dataset contains only the text part of the English posts. Similar to comments here also we aim to maintain two key features to mimic real-life scenarios. (1) The distribution of the samples over the 6 inherent themes is long tailed. (2) The number of Spam examples is less than that of the Clear ones. Thus, we use a multinomial distribution with different probabilities for each theme and a Bernoulli distribution biased to the Clear contents to sample our Posts dataset.
We partition the D1 English Comments and D2 Posts datasets intro training, validation, and test sets by theme level stratified sampling (Singh and Mangat, 1996) The distributions of samples over the three sets for these two datasets are described in the following Table 10.

heading: A.4 Model Details for QnA-CM
text: In this study we employ the general purpose BERT-Large (Devlin et al., 2018), RoBERTa-Large (Liu et al., 2019), Albert-Large-V2 (Lan et al.,
2019), and DeBERTa-Large (He et al., 2020), along with multilingual models such as BERTBase-Multilingual (Devlin et al., 2018) and XLMRoBERTa-Base (Conneau et al., 2019), as the backbone LLM for both binary classifier and QnA-CM. All the LLM backbones used in QnA-CM are finetuned using the dataset under concern for a maximum number of 10 epochs with a learning rate of 1.00e-05 for the Adam (Kingma and Ba, 2014) optimizer. We measure the performance on the validation set after every 100 steps. An early termination criterion is used to check if the performance evaluation metrics such as Accuracy, precision, recall,
and F1 score have not improved on the validation set for the last e consecutive evaluation steps. For the CS policy e is set to 10 and the same for Hate Speech is kept to 5, as those choices found to be performing well on average. The hyperparameters introduced in QnA-CM are tuned using grid search. The search space and the final choices for these hyperparameters for the D1 Comments and the D2 Posts datasets are detailed in Table 11.

heading: A.5 Policy Update Simulation for CS Policy
text: To simulate an update for the CS policy, the low prevalent eight themes in the English dataset, namely Animals, Fabricated Items, Illegal Drug, Human Body Parts, Mail Order Bride, Weapons, Counterfeit Items, and Human Exploitation are removed from the training and validation set and added to the test set. Further, the corresponding validating questions for these eight themes are only used during inference. This is a viable strategy for simulating policy update as it is likely that the ex-
isting themes will have enough annotated samples while the same for the newly introduced themes will be less in number.

heading: A.6 Full Results
text: The complete results for Table 3 is available in Table 12 while the same for Table 7 is reported in Table 13. Moreover, the scores of QnA-CM and Baseline for two examples that are hateful but not Hate Speech are detailed in Table 14.

heading: Future Work
text: In the future, we aim to investigate the applicability of open source datasets with recently developed large generative models with high natural language understanding and promptdriven zero-shot capabilities such as GPT series (Brown et al. , 2020) or LLaMA (Touvron et al. , 2023) in the QnA-CM framework