BotPercent: Estimating Bot Populations in Twitter Communities
Authors: Zhaoxuan Tan
Shangbin Feng
Melanie Sclar
Herun Wan
Minnan Luo
Yejin Choi
Yulia Tsvetkov
Venue: EMNLP 2023 Findings (2023)
Keywords: Twitter bot detection
social network analysis

Abstract: Twitter bot detection is vital in combating misinformation and safeguarding the integrity of social media discourse. While malicious bots are becoming more and more sophisticated and personalized, standard bot detection approaches are still agnostic to social environments (henceforth, communities) the bots operate at. In this work, we introduce community-specific bot detection, estimating the percentage of bots given the context of a community. Our method---BotPercent---is an amalgamation of Twitter bot detection datasets and feature-, text-, and graph-based models, adjusted to a particular community on Twitter.  We introduce an approach that performs confidence calibration across bot detection models, which addresses generalization issues in existing community-agnostic models targeting individual bots and leads to more accurate community-level bot estimations. Experiments demonstrate that BotPercent achieves state-of-the-art performance in community-level Twitter bot detection across both balanced and imbalanced class distribution settings, presenting a less biased estimator of Twitter bot populations within the communities we analyze. We then analyze bot rates in several Twitter groups, including users who engage with partisan news media, political communities in different countries, and more. Our results reveal that the presence of Twitter bots is not homogeneous, but exhibiting a spatial-temporal distribution with considerable heterogeneity that should be taken into account for content moderation and social media policy making. The implementation of BotPercent is available at https://github.com/TamSiuhin/BotPercent.

BotPercent: Estimating Bot Populations in Twitter Communities
Zhaoxuan Tan
Shangbin Feng**
Melanie Sclar Herun Wan
Minnan Luo
Yejin Choi o Yulia Tsvetkov
University of Notre Dame University of Washington"
Xi'an an Jiaotong University" Allen Institute for AI ztan3ënd. edu; shangbin@cs washington. edu
Abstract
Communty
Twitter bot detection is vital in combating mis- information and safeguarding the integrity of social media discourse. While malicious bots are becoming more and more sophisticated personalized, standard bot detection ap- proaches are still agnostic to social environ- ments (henceforth, communities) the bots oper- ate at. this work, we introduce community- specific bot detection, estimating the percent- age of bots given the context of commu- nity. Our gamation of Twitter bot detection datasets and feature-. text-, and graph-basee models, ad- justed to a particular community Twitter. We introduce an approach that performs con- fidence calibration across bot detection mod- els, which addresses generalization issues in existing community-agnostic models targeting individual bots and leads more accurate community-level bot estimations. Experiments demonstrate that BotPercent achieves state-of- the-art performance community-level Twit- ter bot detection across both balanced and im- balanced class distribution settings, present- ing less biased estimator of Twitter bot pop- ulations within the communities we analyze. We then analyze bot rates several Twitter groups, including users who engage with parti- san news media, political communities in dif- ferent countries, and more. Our results reveal that the presence of Twitter bots is not homo- gencous, but exhibiting a pempaal-emmporl dis- tribution with considerable heterogeneity that should be taken into account for content mod- eration and social media policy making. The implementation of BotPercent is available https://ithub..om/Tnaa BattsihhhiinooPeem.....mm
intore
Individual Analysis
Community Analysic
Beyond individual detection, our work proposes socially contextualized community-lewe level bot detection and analysis of bot populations within differ- ent communities on Twitter.
Figure I:
an amal-
Twitter bots have been deployed with malicious intents, such as disinformation spread (Cui et al., 2020; Wang et al., 2020; Lu and Li, 2020; Huang et al., 2022), interference in elections (Howard et al., 2016; Bradshaw et al., 2017; Ferrara et al., 201 6a; Rossi et al., 2020), promotion of extremism (Ferrara et al., 2016b; Marcellino et al., 2020), and the spread of conspiracy theories (Ferrara, 2020; Ahmed et al., 2020). These triggered the devel- opment of automatic Twitter bot detection models aiming at mitigating harms from bots' malicious interference (Cresci, 2020).
Prior work has mainly focused on bot detection at the individual account level (Yang al., 2020; Echeverna al., 2018; 8: Feng et al., 2022a), whereas community-levee estimates of bot population and activity are under-explored, where "community" is defined as network proximity in line with prior works (Feng et al., 2021b, 2022b). Our work posits that the social context in which a bot operates is essential for accurate detection, with major im- plications for both the social media platform and users. From the platform side, it can allow decision- makers to efficiently distribute content moderation resources among communities, as well as inform community members about the risks of inauthen-
1 Introduction
Twitter accounts controlled by automated pro- grams, also known as Twitter bots, have become widely recognized, concerning, and studied phe- nomenon (Ferrara al., 2016a; Aiello et al., 2012). "These authors contributed equally to this work.

---

tic content. From user perspective, can en- hance awareness of potential opinion manipulation by clearly indicating the anticipated level of inau- thentic content within a social network. Finally, community-lveel bot detection can help alleviate privacy concerns by presenting collective statistics per-community, rather than probing tracking in- dividual users within the community, required by existing approaches to individual bot detection (Guerid al., 2013). These and other commercial and legal concerns have led to an increased inter- est in understanding the percentage of Twitter bots from groups to crowds (Varol, 2022).
tings. Extensive experiments demonstrate that BotPercent achieves state-of-the-art performance for community-level bot detection, presenting con- sistently more accurate and calibrated bot popula- tion estimations across both settings ten diverse Twitter communities. Armed with BotPercent, we investigate the bot populations in real-world Twitter communities of varying sizes and contexts, including political groups, news-sharing communi- ties, celebrity- cliques, and more. We find that (1) around 8- 8-14% of interactions with Elon Musk's Twitter poll' to reinstate Donald Trump were carried out by bots; (2) online communities that focus on politics and cryptocurrency are wit- nessing an elevated level of bot interference; (3) news media that are more partisan often face higher rates of bot- not-generated comments; (4) bot popula- tions in the democratic discourse around political issues (c.g.. abortion and immigration) often wax and wane due to major socio- picciopooitical events. To- gether our experiments and results demonstrate that the distribution of Twitter bots is not homogeneous, but rather has spatial-temporea temporal patterns with sig- nificant implications for bot behavior understand- ing, navigating socio-political events, social media moderation, and more.
Although state-of-the-art Twitter bot detection methods have achieved impressive results, their individual-bot focus renders them unsuitable for out-of-the tooo population-level predictions. Cur- rent approaches often employ models that analyze a single modality of user information in a sin- gle dataset, overfitting to certain types of Twitter bots (Echeverria et al., 2018; Yang et al., 2020). Such individual-bor bot detection approaches are of- ten poorly calibrated and thus cannot be casily adapted to estimating bot populations, since robust community-levee bot detection requires having an unbiased estimator of bot probability.
The contribution of BotPercent the largest and most comprehensive combination of exist- ing bot detection models and datasets with con- fidence calibration, addressing novel problem of community- llvell bot detection which provides bet- ter aggregated estimates of bot populations within social networks, leading new findings while avoiding the privacy risks of tracking individual accounts.
To this end, we propose BotPercent, a frame- bot detection that
work for community-oriemi aggregates and calibrates multiple existing models and datasets to devise accurate and generalizable es- timations of Twitter bot populations from groups to crowds. BotPercent combines feature, text, and graph-based approaches with complementary in- ductive biases, which facilitates robustness to shift- ing user domains. This ensemble aims to balance the over-specialization of single-modality models. Since our analysis shows that community-agnostic models are often miscalibrated, BotPercent also conducts model calibration for individual models (Guo et al., 2017) and combines their predictions while dynamically learning the weights of indi- vidual models in shifting contexts for more accu- rate bot percentage estimation. For training data, Bet merges multiple available Twitter bot detection datasets with 1,216,758 users to enhance generalization and overcome the data limitations of existing coamunntty-gnnoste approaches.
2 BotPercent Methodology
We propose BotPercent, novel system for community-level Twitter bot detection. As illus- trated in Figure 2, BotPercent first trains multiple models on mixture of datasets, then leverages con- fidence calibration perform unbiased bot percent- age estimation, and finally combines predictions with learnable weights.
2.1 Model Components
Motivated by the fact that different types of bot detection models have their strengths and weak- nesses in the face of customized bots in different
We first evaluate BotPercent with the 10 Twit- ter community datasets of the TwiBot-22 bench- mark (Feng et al., 2022b) through both balanced and imbalanced class distribution community set-
'https://twiie http://wwwee..eeeeeoo//ssttaeaae 15937679537669211

---

System Training
Inference
Twitter Data Col ection
Twitter APL
Selection
Calibration
Teature Data
Feature Based Models
VPV
Data Collection
beaed Jene aioestowia
II Jesee 201
1,26 Pellwien Poliroo
37.06 Polimar
Twitter Community
Text Data
Text Based Models ... ا ببب
Combination
werk Risso Use ...
Aceda audita Nenciin gengen Sukkyy Jaan you syow
Language Model
Σ
t Graph
e.x
Graph Bosed Models
Graph Naural Natsors
91.54%
8.66%
Figure 2: Overview of BotPercent. BotPercent first trains multiple models on merged datasets and then uses confidence calibration to conduct accurate bot percentage estimation. Finally, it combines the predictions with learnable weights to obtain final result.
communities (Sayyadiharikandeh et al., 2020), we propose unified framework that adopts represen- tative approaches. Existing Twitter bot detection approaches could be categorized based on their input data: feature-, text-, and graph-based mod- els use different kinds of user features, adopt dif- ferent classifier architectures, and have respective inductive biases (Feng et al., 2022b). We com- bine and calibrate these as modular components of BotPercent, enhancing individual models' perfor- mance and generalizability.
classification:
Pr =
where pt E R denotes the prediction logits, [-II-] denotes vector concatenation, t; and d; represent tweet and user description tokens respectively, and LM denotes one of the language models with mean pooling feature extractor over the final layer. These text-based models optimized with the cross entropy loss function.
Graph-based bot detection models leverage the Twitter network structure with graph neural net- works (GNNs) to analyze the contextual user in- teractions through local neighborhood information aggregation (Ali Alhosseini et al., 2019; Feng et al., 2022a). For graph- based models, employ four graph neural network-based approaches to bot de- tection: SimpleHGN (Lv et al., 2021), HGT (Hu et al., 2020), BoIRGCN (Feng et al., 2021c), and RGT (Feng et al., 2022a) in BotPercent since these models take into account the intrinsic hetero- geneity in social networks and have shown promis- ing bot detection performance (Feng et al., 2022b). The message-passingg paradigm of these models could be summarized as:
Feature-based models extract user features and adopt traditional classifiers (Varol al., 2017). To construct a comprehensive feature-based model as part BotPercent, we summarize features intro- duced in existing feature- based models and obtain a more comprehensive feature set consisting of 12 direct and 14 derived user features. Following previous works (Yang al., 2020; Knauth, 2019), BotPercent leverages random forest (Ho, 1995) and AdaBoost (Freund and Schapire, 1997) as two efficient feature-based classifiers and obtains bi- nary prediction logits py E 1 for each Twitter user representing its probabilities as bot or human.
Text-based bot detection models leverage users' tweets and descriptions identify Twitter bots and malicious content (Wei and Nguyen, 2019). Simi- larly, BotPercent leverages pretrained RoBERTa (Liu et al., 2019a) and T5 (Raffel et al., 2020) lan- guage models to encode user descriptions and 20 latest tweets while using a linear layer Linear for
Agg. (1) (Extr. (0 (h, h5 h VeE(..VE
(1-1) ,)),
where h." denotes the i-th user's representation in the I-th GNN layer, M2 represents the neighbor nodes of user i, and E(i.j) denotes all the edges

---

from user to i. Extr. represents the neigh- bor information extractor in i-th layer, which ex- tracts user information from source user's repre- sentation h, " with the target user representation hn-") and the edge e between two users as propa- gated message. Agg. (9) gathers the neighborhood information of source users via aggregation oper- ators, Different GNNs use different aggregation and extraction functions. SimpleHGN uses the at- tention mechanism with consideration of edge type Agg. (1) and an MLP as Extr. (1), HGT adopts the attention mechanism with regard edge type Agg. (4) and takes the edge type as different pro- jection matrixes in Extr. ()) BotRGCN takes the mean pooling as Agg. (1) and processes the edge type with different aggregation matrixes in Extr. (1) RGT propagates message under different relation types as Extr. and aggregate representation from different relation types with the attention mecha- nism in Agg. (1) With these different GNN archi- tectures and message-passingg mechanisms, we aim capture the diverse interaction patterns between multi-faceted bots and users.
BotPercent's efficiency and scalability for the large-scale real-time analysis of social network communities.
2.2 Confidence Calibration
Existing bot detection models generally leverage one dataset as training data, and existing datasets are limited in user domains, Twitter communi- ties, and data collection times (Feng al., 2021b). Thus, current models generalize poorly new user communities and emerging bots (Echeverria et al., 2018). In contrast, community-Level bot detection should generalize to diverse communities and time periods. To this end, we propose an approach to combining existing datasets for BotPercent train- ing. Specifically, we merge a wide variety of publicly available Twitter bot detection datasets?, and train models the resulting dataset. By jointly leveraging diverse and representative exist- ing Twitter bot detection datasets as training data, Percent presents a community- tion system designed for better generalization.
bot detec-
Although individual models provide scores in- dicating the likelihood of each account being a bot, they could not be trivially aggregated for community-level estimation since binary classi- fiers often produce scores that do not accurately reflect true probabilities (Platt al., 1999; Guo et al., 2017), i.c., models are often miscalibrated. To accurately estimate the probability of Twit- ter account to be a bot and obtain percentages of bots within communities, BotPercent performs confidence calibration for all sub-models to ensure alignment between estimated and true probabilities. Specifically, we leverage temperature scaling (Guo et al., 2017), a poot-pooessiing method that rescales confidence predictions by tuning a single scaling parameter over a held-out validation set. By re- peating this calibration process for each and every modular component covering the three modalities, BotPercent results in calibrated and less biased estimators for bot population results.
After modeling the Twitter network with L lay- ers of GNNs, we obtain the representation h hn(") for user i. BotPercent then employs a linear layer for two-way classification as Py = Linear(h (h.")) ())
Graph-based approaches are optimized with cross entropy loss function.
A unique challenge of employing graph-based approaches for community-level bot detection is that they face scalability issues inference time due to data dependency: when BotPercent ana- lyzes a specific user, it needs to encode its multi- hop following/follower neighborhood, which leads to exponential inference costs. Motivated by Zhang et al. (2021), we use knowledge distillation (Hinton al., 2015) to transfer the knowledge of graph- based detectors to efficient linear layers. The distil- lation training loss could be written as:
La = A Σ CE yo) + (1 Σ (Sollll))) BEV
VEV
2.3 Prediction Combination
where V denotes a batch of users, CE denotes the cross entropy loss, KI. denotes the KI-- divergence, À is hyperparameter, ye, PG. and yo denotes the prediction of the linear layer, GNNs, and the ground truth label. In this way, state- of-the-art graph- approaches are distilled into high-quality linear layers that serve as prox- ies for the computation-heary GNNs, improving
After obtaining the calibrated results of all sub- models, BotPercent combines their predictions
Dataset details are presented in Appendix I, datasets in- clude CRESCI-15 (Cresci et al., 2015), GILANI-17 (Gilani et al., 2017). CRESCI-17 (Cresci et al., 2017), MIDTERM- 18 (Yang et al., 2020), CRESCI-STOCK- 18 (Cresci et al.. 2018, 2019). CRESCI-RTBUST-ID (Mazza et al., 2019). COOMMTTER-PEERRACEE 19 (Feng et al., 2021b), and TwIBoT-22 (Feng ot al., 2022b).
1999(Ya) (Yang et al., 2019), TwiBon-20

---

Text
Graph
Muhi-MModel
BotPereent
Veegtal Rediooe
Buchace Grees
EdRO
Predicton
Gresd Tirk
30% Gold
10%
50%
Gold
Gold
Gold
Figure 3: Predicted bot percentages resampled communities from the TwiBot-22 benchmark that simulate imbalanced settings, with bot percentages ranging from 10% to 90%. BotPercent consistently yields accurate population estimations, while baselines severely under- over-estimate bot populations across Twitter communities.
bots. As a result, the correct estimation of Twit- ter bot populations in these communities would be 50%. We conduct community-level bot de- tection with BotPercent and compare with dif- ferent types of existing bot detectors: Yang et al. (2020), BotHunter (Gu et al., 2007), LOBO (Echev- erra et al., 2018), RoBERTa (Liu al., 2019b), BotRGCN (Feng al., 2021c), RGT (Feng et al., 2022a), Botometer (Yang et al., 2022a), and Bot- Buster (Ng and Carley, 2022) Figure shows that PotPerennt is significantly more accurate at predicting the true bot percentage for the 10 popula- tions analyzed, including state-of-the-art individual bot detection methods such RGT. In addition, feature- and text-based methods generally underes- timate the bot population, while graph-based meth- ods generally overestimate the percentage bots.
2 Phattorr
63%
43%
0%
18%
butteser
1,300
Romteree
buston
Hnlr
Nathure
Belf Beffersent
Figure 4: BotPercent and existing models' community- level bot population estimation for the balanced com- munity datasets in TwiBot-22 (ground truth is 50% for communities). BotPercent is significantly more ac- curate for all communities analyzed.
through weighted summation:
3.2 Imbalanced Twitter Communities
D p D (rgmax(o.1)
(fiia) K2 Σ
Σ
Ojk Pike
To better understand the performance BotPercent, and ensure does not simply learn the balanced bot distribution, we further resample imbalanced communities of 5,000 users from the TwiBot-22 (Feng et al., 2022b) benchmark based network proximity with bot percentages ranging from 10% 90%. We then conduct community-level bot detection on these resampled communities and compare BotPercent with representative bot detectors. Figure demon- strate that BotPercent consistently outperforms baseline models by offering consistent Twitter bot population estimations that are close to the ground truth. These results, along with the balanced settings performance, demonstrate the importance of BotPercent as multi-dataset multi-model pipeline to combine their inductive biases and improve generalization, as well as confidence calibration's effectiveness for accurate bot percentage estimation.
where D denotes the number of accounts in given Twitter community, f. f,t,g respectively denote feature-, text-, and graph-based approaches, Kj denotes the number of sub-modeis under the j-th modality, and jr denotes the learnable weight of the k-th sub-model under category j. which is opti- mized using the negative log likelihood (NLL) loss on the validation set with weights of sub-models kept frozen. The resulting is the BotPercent bot population estimation for community.
3 Experiments
3.1 Balanced Twitter Communities
We first evaluate BotPercent in balanced settings by leveraging the 10 Twitter communities pre- sented in recent TwiBot-22 benchmark (Feng et al., 2022b). Each community contains 10,000 users with an even split between genuine users and
Baseline details are presented Appendix O.

---

Favironment
Cappocumency
Entrrtiimment
Sports
Politics
@reeDDnndddTung 13,89%
@aantonop 45.09%%
e VitellaB oteriz 28.60%
40%
@ Doliitdon 10.64%
@ladygagg 7.48%
② AthereunToeepp 23.27%
SSBB PIX 20.77%
30%
@ Cobrmitne 14.90%
jeftivozoe 20.103 10%
SpenkorPelod 10.68%
@ DDD. strsi
@elnnmsk
@ Blamcaraib 6.17%
@ Kingjj amesd 8.600%
@ Tromm 7.80%
@ rogerfederer 0.35%
@byyoorsent11 9.40%
@ millimssthem
@SpookorrcCarrtyy 0.29%
6.91%
10%
Olaack 9.13%%
0%
L0
10'
@ benshappiro 8.57%
hnnaatrmmbrr Pollower Count @ Cummy 10.37%
(@neymad 1.93%
@Hkatypery 2.67%
@ Cristiano 4.75%
@rthanna 7.45%
9 9,26%
Figure 5: Bot percentage within the comment sections of celebrities from different interest domains. Celebrity- centric communitics in cryptoeurrency, tech, and polities are generally witnessing larger extent of bot manipulation.
4 Analysis of Twitter Bots
128
Fox Niass 10 64%
Section 3 demonstrates that BotPercent achieves state-of-the-art performance on community-level bot detection and produces well-calibrated bot per- centage estimation for diverse Twitter communi- ties. We now investigate diverse real-world Twitter communities and estimate their bot populations by leveraging BenPercent.
Duily Kas
10.13% CNN USA Todas DLSOR
5.5%%
Wichington Times
Bot Parcestage
9.33% AB
Revidbett
taciiagton
Assodated Press
10.0E%
tpp AP RIL
5..0%
Weehington
08
8.25%
The
7.50%5
6.9%%
4.1 Bot Population among User Interactions
lnf
Ien left
center
Joen right
7111
Political Learing
Comment sections of famous users' tweets are bat- dlegrounds public opinion (Weber, 2014). As result, we investigate the bot percentage in these comment sections and understand the extent to which celebrity-enntre and news-sharing groups are compromised by Twitter bots.
Figure 6: Bot percentage within the comment sections of news media with respect to their political lcaning. Bot interference higher in more partisan news media.
4.1.2 News Media
With the advent of online social networks, tradi- tional news media have also used social media to report on current events and provide political commentary. Although previous research has shed light on bot involvement in news media (Shao et al., 2017), it remains unclear to which extent the Twit- ter accounts of news outlets are targeted by bots for amplification or rebuttal, which could in turn cloud the judgment of social media users. To this end, select the official Twitter accounts of news me- dia with different political leanings as evaluated by AllSides.* 4 As shown Figure there is a strong correlation between the political stance news media and the percentage of bot accounts their Twitter account's 's comment section. Specifically, centrist news media generally have the lowest pro- portion of bot accounts in their comment section. As the political stance becomes more polarized,
4.1.1 Celebrities
We examine 28 celebrities from 6 interest domains: sports, entertainment, tech, politics, crypto, and environmentalism. We collect all accounts that commented on these users' tweets from Decem- ber 23rd to 31st, 2022. Results show that the bot percentage in the comment sections of cryptocur- rency celebrities is significantly higher than in other domains, and the bot percentage in tech also gen- erally above average (see Figure 5). This suggests a non-uniform spatial distribution of bots on so- cial networks. Although previous works mainly focused on Twitter bots in the political domain (Woolley, 2016; Forelle et al., 2015), findings reveal that Twitter bots heavily active in various domains, particularly cryptocurrency and technol- ogy. This highlights the importance of studying bot impacts beyond politics, with implications for financial fraud, market manipulation, and more.
"https://www www.:/wwwwwl..ds.....

---

30%
Farta Aqusement
Musim hon
Wade
Ecanomy
25%
Healtheare
26
2%
4.22%
Crime
20%
239
Supreme Court
353
Immigration
Gun Poliry
Bot
Formggn Policy
Aboction
5%
Rocism
Climate Chasge
0%
2022
ear
Figure 7: Temporal percentage trend 11 important political issues of the past decade. Bot activity is highly correlated with major socio-political events such as Trump's Muslim ban and Roe Wade' overturning leaks.
the proportion of bot accounts increases, and bot percentage within the comment sections of right- leaning media is slightly higher than that of left- leaning counterparts. Based on the above analysis, we infer that news media with centrist stance less susceptible to interference from bot accounts in their comment section, while media with more partisan political stances are more susceptible to manipulation from bot accounts. This suggests that social media users should practice more caution when reading and interacting with hyperpartisan online news media.
retweet
18%
Bot Percentage
11.7%%
10%
9:34%
9.76%
85%
0%/
Reinstate Trump
Musk Step Down
Figure 8: percentage among users that interacted with the two social media moderation votes held by Elon Musk. These votes witnessed substantial interference from Twitter bots, casting doubt Elon Musk's "Vox Populi, Vox Dei" principle.
4.2
Bot Population Changes through Time
Roe v. Wade. In addition to temporal trends, Figure 7 demonstrates that bot participation in immigra- tion and healthcare discussions is generally higher than other topics (6.89% and 6.66%, on average). Together, these results show that Twitter bot behav- ior exhibits variable temporal patterns and spikes in response major socio-polition events, suggest- ing that content moderators and day-to-day users should practice extra caution when moderating and engaging in discussions of emerging events.
Twitter and social media in general have become important medium for political discourse. Wor- ryingly, Twitter bots often operated by mali- cious actors to interfere with political discussions (Caldarelli et al., 2020). To better understand the patterns of political interference from Twitter bots, we investigate 11 political topics and use political keywords presented in Flores-Saviaga et al. (2022) to search for tweets posted during different time pe- riods and analyze the corresponding Twitter users. For each political topic, we collect tweets from 1000 users per quarter in the past decade from Jan- uary 2012 to December 2022. As shown in Fig- ure 7, the proportion of bot accounts changes in line with major socio-political events. For exam- ple, the spikes in bot participation regarding im- migration in spring 2017 coincides with Trump's 's Muslim ban (Pieree and Meissner, 2017). Bots dis- cussing climate change peaked in the summer of 2015, which could be attributed the negotiations and signing of the Paris Agreement, and the sud- den spike regarding abortion in early 2022 could be attributed to the Supreme Court leaks concerning
4.3 Bot Presence in Content Moderation Votes
Since Elon Musk's takeover of Twitter in 2022, he has held numerous votes on his personal account, with two of them having consequential content moderation outcomes: one vote to decide whether to reinstate Donald Trump' on Twitter and another vote to decide whether Musk should step down as Twitter CEO. While the policy of direct democ- racy for content moderation seems straightforward,
1593767953789921995
"https: /ttwtttr..wttttt/tt/eeeee http://wwwwweetteeeeeeeeeeee com//elommss//ttatus////////
664411743973124897

---

10:02%
Careda 12.159%
108
14.29%
PrEKe 5.38%
South Korm
0.38%
125
10.11%
9.1%%
Argendrr
151%
South Affica
Austriiia 13.4%%
7.82%
Figure 9: Bot populations within the political landscapes of different countries. The United States has the highest percentage bots the political community, while bot populations in English-speaking countries are generally above average.
it has numerous concerns, one of them being the interference from malicious actors through Twitter bots. We investigate the bot percentage in users
Individual
Community Bot %
Method
Accuracy
Fl-score
BotPercent
0.731
0.726
+3.9%
W/O FEAT BASED MODELS
0.649
0.698
+15.2%
that retweeted, commented, or liked the two votes
(the specific voting data is not publicly available).
GRAPH BASED MODELS
0.659
0.639
-33.8%
W/O CALIBRATION
0.653
0.693
+23.8%
Figure 8 shows that 8% to 14% of users that in- teracted with the two content moderation votes bots. Given the close results of both votes (51.8% v. 48.2%, 57.5 % 42.5%), this suggests that bots could have changed the outcome, putting into question the validity the "Vox Populi, Vox Dei" principle of social media moderation.
MEAN POULING AS COMB.
0,708
0.711
+7.05%
Table 1: Ablation study of BotPercent. Bold indicates the best performance, underline indicates the second best. In community-level bot detection, we compare the model estimation the ground truth percentage within 10 communities in the TwiBot-22 dataset. The ground truth is represented as 0%, with negative percent- ages indicating underestimation of the hot population, and positive percentages indicating an overestimation.
5
Bot Population in Different Countries' Politics
Existing research on the Twitter bot population mostly focuses on bots in U.S. politics (Bessi and Ferrara, 2016; Yang et al., 2020) while neglect- ing the political landscape of other countries that could have similar problems. We complement the scarce literature by investigating the bot popula- tion different countries' political communities. Specifically, we use the president or prime minis- ter's Twitter account as a starting point and sample their followers to serve as proxy for the politically engaged communities in different countries. Figure 97 illustrates that the percentage of bots in U.S. pol- itics is the highest, while other English-speaking countries also witness higher levels of bot interfer- ence. In addition, the percentages of bots in the political communities of Argentina, France, and Nigeria are the lowest, indicating more genuine and authentic political discourse. These results again reaffirm that Twitter bots have spatial pat-
terns across the whole Twitter network, while the impact of malicious Twitter bots in countries other than the U.S. warrants further research.
Ablation Study
As BotPercent outperforms various baselines in community-level bot detection tasks, we investigate the impact of each module in BotPercent verify their effectiveness. More specifically, we perform ablation studies on feature-, text-, and graph- based modules and test the effectiveness of calibration and weighted sum combination, as is shown in Ta- ble 1. First, it is illustrated that full BotPer cent outperforms ablated models, proving our design choice's 's effectiveness. Second, we observe signif- icant bot percentage bias without the graph-based module version model, indicating its indispensable role in BotPercent's strong community-level bot detection performance. Finally, the community- level bot detection performance dropped dramat- ically without confidence calibration, indicating
'Country border naturalearthdata.com
source:
https://wwi

---

that confidence calibration plays an essential role in accurate bot percentage estimation.
(Feng et al., 2021c; Lei al.. 2022; Peng et al., 2022). Researchers have also explored combining graph- and text-based methods (Guo al., 2021) or proposing new GNN architectures to take into ac- count heterogeneities in the Twitter network (Feng et al., 2022a). These graph-based approaches have been effective in addressing the challenges such bot clusters and bot disguises (Feng et al., 2021c).
7 Related Work
Existing Twitter bot detection models mainly fall into feature-based, text-based, and graph-based. Feature-based bot detection methods extract features from user timelines and metadata while using traditional classification algorithms to iden- tify bots. Features can be extracted from various sources such as metadata (Yang et al., 2020; Lee et al., 2011), I), user description (Hayawi et al., 2022), tweets (Miller et al., 2014), temporal tweeting pat- terns (Mazza et al., 2019), and the following rela- tionship (Feng et al., 2021c, 2022a). Later research focused on improving the scalability of feature- based approaches (Yang et al., 2020), automatically discovering new bots (Chavoshi et al., 2016), and finding the optimal balance between precision and recall (Morstatter et al., 2016). However, as bot operators become more aware of the features used by feature-based methods, they try to alter them to evade detection (Cui et al., 2020), which makes it difficult for feature-based based methods to effectively identify advanced bots.
There are also preliminary attempts to ensemble methods from different categories in bot detection. For example, Ng and Carley (2022) trains feature- and text-based experts for ensemble, and Liu et al. (2023) further ensembles three models each from feature- text-, and graph-based categories to boost community-agnostic bot detection. However, improve stability and generalizability, the ensemble size for both data and model needs to be increased.
Although these community-agnosee approaches have advanced the understanding of Twitter bots and their behavior, community-level bot detec- tion, aiming understand the extent to which bots compromised a given Twitter community, re- mains an underexplored yet important problem. With great implications for both platform mod- erators and everyday users, we propose a novel community-leved system BotPercent with multi- dataset training and the largest model ensemble to estimate the Twitter bot populations from groups to crowds.
Text-based methods propose to leverage tech- niques in natural language processing to analyze tweets and user descriptions. These methods em- ploy word embeddings (Wei and Nguyen, 2019), recurrent neural networks (Kudugunta and Ferrara, 2018), the attention mechanism (Feng al., 2022a), and pretrained language models (Dukić al., 2020) analyze Twitter text. Later research efforts com- bine tweet representations with user features (Cai et al., 2017), learn unsupervised user representa- tions (Feng et al., 2021a), and address the issue of multilingual tweets (Knauth, 2019). However, advanced bots counter text-based approaches by diluting malicious tweets with content stolen from genuine users (Cui et al., 2020), while Feng al. (2 found that relying solely on tweet content may not be robust or accurate enough to win the bot detection arms race.
8 Conclusion
We propose BotPercent, a multi-dataset multi- model Twitter bot detection system for estimat- ing the Twitter bot populations from groups to crowds. Experiments on the TwiBot-22 benchmark demonstrate that BotPercent achieves state-of-the- art performance on community-level bot detection while being significantly more robust to perturba- tions user features. Armed with BotPercent, we investigate the bot populations in Twitter com- munities, such as news commentators, politically engaged accounts, and more. Together these re- sults demonstrate that the existence of Twitter bots is not homogencous, rather having apatial-temporal patterns that have great implications for social me- dia moderation, warning of socio-political events, real-time monitoring, and BotPer 's potential to guide future research in specific communitics rather than general-purposs.
Graph-based methods for Twitter bot detection interpret Twitter as network and leverage con- cepts from network science and geometric deep learning. These approaches adopt node central- ity (Dehghan al., 2022), representation learning (Pham al., 2022), graph neural networks (GNNs) (Ali Alhosseini et al., 2019; Alothali et al., 2022; Yang et al., 2022b), and heterogeneous GNNs

---

Limitations
acknowledge support from NSF CAREER Grant No. ISS214279,, and the Alfred Sloan Founda- tion Fellowship. The views and conclusions con- tained herein are those of the authors and should not be interpreted as necessarily representing the offi- cial policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government, The U.S. Gov- ernment is authorized to reproduce and distribute reprints for governmental purposes notwithstand- ing any copyright annotation therein.
We identify three key limitations. Firstly, due to the limitations of the Twitter API, our estimation of Twitter bot populations sometimes limited in scale and information completeness. For instance, Twitter API limits the tweet lookup rate to 900 per 15 minutes, and we do not have access to user infor- mation that participated in Twitter votes. However, our work presents Bot and series of bot analysis proposals that are compatible with better API access and improved data sources. Secondly, BotPercent employed knowledge distillation to make graph-based models scalable to real-world analysis, while this approach may result in minor performance drops due to the absence of graph structure information in the inference stage. Fi- nally, BotPercent may introduce bias in the cold start circumstance, where users who have not en- gaged with the platform for long enough time are more likely be identified as bot accounts.
References
Wasim Ahmed, Francesc López Seguí, Josep Vidal- Alaball, Matthew S Katz, al. 2020. Covid-19 and the "film your hospital" conspiracy theory: social network analysis of twitter data. Journal of medical Internet research, 22(1044:23374
Luca Maria Aiello, Martina Deplano, Rossano Schi- fanella, and Giancarlo Ruffo. 2012. People are strange when you re stranger: Impact and influ- ence of bots on social networks. In Proceedings of the international AAAI conference on web and social media, volume 6, pages 10-17
Ethics Statement
We envision BotPercent as large-scale pre- screening tool and not as ultimate decision maker. Importantly, BotPercent and any other au- tomatic bot detection systems are imperfect proxies for bot detection and thus need to be used with care, in collaboration with human moderators to monitor or suspend suspicious accounts. Apart from that, as a combination of datasets and models, BotPercent may inherit the biases of its constituents. For ex- ample, pretrained language models could encode undesirable social biases and stereotypes (Nadeem et al., 2021; Liang et al., 2021; Bender et al., 2021), while graph neural networks could also discrim- inate against certain demographic groups (Dong et al., 2022) in decision makiing. We leave to future work on how to incorporate the bias detection and mitigation techniques developed in MI. research in bot detection systems.
Seyed Ali Alhosseini, Raad Bin Tareaf, Pejman Najafi, and Christoph Meinel. 2019, Detect me if you can: Spam bot detection using inductive representation learning. In Companion Proceedings of The 2019 World Wide Web Conference, pages 148-153.
Eiman Alothali, Motamen Salih, Kadhim Hayawi, and Hany Alashwal. 2022. Bot-mgat: A transfer learning model based on a multi-view graph attention network to detect social bots. Applied Sciences, 12(16):8 17..
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence Fairness, Accountability and Transparency, pages 610-623.
Alessandro Bessi and Emilio Ferrara. 2016. Social bots distort the 2016 presidential election online discussion. First monday, 21(11-7).
Samantha Bradshaw, Bence Kollanyi, Clementine De- sigaud, and Gillian Bolsover. 2017. Junk news and bots during the french presidential election: What are french voters sharing over twitter? Technical report, COMPROP Data Memo.
Acknowledgements
We thank the reviewers and area chair for their com- ments and feedback. This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Re- search Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200004, This ma- terial is funded by the DARPA Grant under Con- tract No. HROO1
Chiyu Cai, Linjing Li, and Daniel Zeng. 2017. Detect- ing social bots by jointly modeling deep behavior and content information. In Proceedings of the 2017 ACM on Conference Information and Knowledge Management, pages 1995-1998.
Guido Caldarelli, Rocco Nicola, Fabio Del Vigna, Marinella Petrocchi, and Fabio Saracco, 2020. The
We also gratefully

---

role of bot squads in the political propaganda on twitter. Communications Physics, 3(1):1-15.
Shangbin Feng, Zhaoxuan Tan, Rui Li, and Minnan Luo. 2022a. Meterogeneitt--waar aware twitter bot detection with relational graph transformers. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 36, pages 3977-3985.
Nikan Chavoshi, Hossein Hamooni, and Abdullah Mueen. 2016. Debot: Twitter bot detection via warped correlation. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 817-822. IEEE Computer Society.
Shangbin Feng, Zhaoxuan Tan, Herun Wan, Ningnan Wang. Zilong Chen, Binchi Zhang, Qinghua Zheng, Wengian Zhang, Zhenyu Lei, Shujie Yang, et al. 2022b. Twibot-22: Towards graph-based twitter bot detection. In Thirty-sixth Conference on Neural In- formation Processing Systems Datasets and Bench- marks Track.
Stefano Cresci. 2020. decade of social bot detection. Communications of the ACM, 63(10)72--83
Stefano Cresci, Roberto Di Pietro. Marinella Petroe- chi. Angelo Spognardi, and Maurizio Tesconi. 2015. Fame for sale: Efficient detection of fake twitter fol- lowers. Decision Support Systems, 80:56-71.
Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. 2021a. Satar: A self-supervised approach to twitter account representation learning and its application in bot detection. In Proceedings of the 30th ACM International Conference on Informa- tion & Knowledge Management, pages 3808-3817.
Stefano Cresci, Roberto Di Pietro. Marinella Petrocchi,
Angelo Spognardi, and Maurizio Tesconi. 2017. The
paradigm-shiff of social spambots: Evidence, theo- ries, and tools for the arms race. In Proceedings of the 26th international conference on world wide web companion. pages 963-972.
Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. 2021b. Twibot-20: A com- prehensive twitter bot detection benchmark. In Pro- ceedings the 30th ACM International Conference on Information & Knowledge Management, pages 4485-4494.
Stefano Cresci, Fabrizio Lillo, Daniele Regoli, Serena Tardelli, and Maurizio Tesconi. 2018. $ fake: Evi- dence of spam and bot activity in stock microblogs on twitter. In Twelffh international AAAI conference on web and social media.
Stefano Cresci, Fabrizio Lillo, Daniele Regoli, Serena Tardelli, and Maurizio Tesconi, 2019. Cashtag piggy- backing: Uncovering spam and bot activity in stock microblogs on twitter. ACM Transactions on the Web (TWEB), 13(2):1-27.
Shangbin Feng, Herun Wan, Ningnan Wang, and Min- nan Luo. 202 Botrgen: Twitter bot detection with relational graph convolutional networks. Proceed- ings of the 2021 IEEF/ACM International Conference on Advances in Social Networks Analysis and Mining, pages 236.
Limeng Cui, Haeseung Sco, Maryam Tabar, Fenglong Ma, Suhang Wang, and Dongwon Lee. 2020. Deter- rent: Knowledge guided graph attention network for detecting healthcare misinformation. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 492- 502.
Emilio Ferrara. 2020. What types of covid-19 conspira- cies are populated by twitter bots? First Monday.
Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menezer, and Alessandro Flammini. 2016a. The rise of social bots. Communications of the ACM, 59(7))96-106
Ashkan Dehghan, Kinga Siuta, Agata Skorupka, Ak- shat Dubey, Andrei Betlen, David Miller, Wei Xu, Bogumil Kaminski, and Pawel Pralat. 2022. Detect- ing bots in social-networks using node and structural embeddings.
Emilio Ferrara, Wen- Nen-Qiang Wang, Onur Varol, Alessan- dro Flammini, and Aram Galstyan. 2016b. Predicting online extremism, content adopters, and interaction reciprocity. In International conference on social informatics, pages 22-39. Springer.
Yushun Dong, Ninghao Liu, Brian Jalaian, and Jundong Li. 2022. Edits: Modeling and mitigating data bias for graph neural networks. In Proceedings of the ACM Web Conference 2022, pages 2259-1633
Matthias Fey and Jan Eric Lenssen. 2019. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: 1903.02428.
David Dukié, Dominik Keča, and Dominik Stipié. 2020. Are you human? detecting bots on twitter using bert. In 2020 IEE 7th International Conference on Data Science and Advonced Analytics (DSAA), pages 631- 636. IBEE
Claudia Flores- Savees-aviiga, Shangbin Feng, and Saiph Sav- age. 2022. Datavoidant: An ai system for address- ing political data voids social media. Proceed- ings of the ACM on Human- Human-Computer Interaction, 6(CSCW2):1-29.
Juan Echeverna, Emiliano De Cristofaro, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, and Shi Zhou. 2018. Lobo: Evaluation of generalization deficiencies in twitter bot classifiers. In Proceedings of the 34th annual computer security applications conference, pages 137-146.
Michelle Forelle, Phil Howard, Andrés Monroy- Hernández, and Saiph Savage. 2015. Political bots and the manipulation of public opinion in venezuela. arXiv preprint 1rXiv:1507.07100

---

Yoav Freund and Robert E Schapire. 1997. A decision- theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 159191119-139
Zhen Huang, Zhilong Lv. Xiaoyun Han, Binyang Li, Menglong Lu, and Dongsheng Li. 2022. Social bot- aware graph neural network for early rumor detec- tion. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 6680- 6690, Gyeongju, Republic of Korea. International Committce Computational Linguisties.
Zafar Gilani, Reza Farahbakhsh, Gareth Tyson, Liang Wang, and Jon Crowcroft. 2017. Of bots and humans (on twitter). In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Net- works Analysis and Mining 2017, pages 349-354,
Clayton Hutto and Eric Gilbert. 2014, Vader: A parsi- monious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216-225.
Guofei Gu, Phillip A Porras, Vinod Yegneswaran, Mar- tin W Fong, and Wenke Lec. 2007. Bothunter: De- tecting malware infection through ids-driven dialog correlation. In USENIX Security Symposium, vol- ume 7, pages 1-16.
Jürgen Knauth. 2019. Language-agnosti twitter-bo bot detection. Proceedings of the International Con- ference on Recent Advances in Natural Language Processing (RANLP 2019), pages 550-558.
Hachem Guerid, Karel Mittig, and Ahmed Serhrouchni. 2013. Privacy-preserving domain- fllux botnet detec- tion in a large scale network. In 2013 Fifth Interna- tional Conference on Communication Systems and Networks (COMSNETS), pages 1-9, IEEE.
Sneha Kudugunta and Emilio Ferrara. 2018. Deep neu- ral networks for bot detection. Information Sciences. 467:312-322.
Kyumin Lee, Brian Eoff, and James Caverlee. 2011 I. Seven months with the devils: long-term study of content polluters on twitter. In Proceedings of the international AAAI conference on web and social media, volume 5, pages 185-192.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International conference on machine learn- ing, pages 1321-1330. PMLR.
Zhenyu Lei, Herun Wan, Wenqian Zhang, Shanghin Feng. Zilong Chen, Qinghua Zheng, and Minnan Luo. 2022. Bic: Twitter bot detection with text-graph interaction and semantic consistency. arXiv preprint wrXiv:2208.08322
Qinglang Guo, Haiyong Xie, Yangyang Li, Wen Ma. and Chao Zhang. 2021. Social bots detection via fusing bert and graph convolutional networks. Sym- metry, 14(1):30.
Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cour- napeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, et al. 2020. Array programming with numpy. Nature, 585(7825)3357-66)
Vladimir I Levenshtein al. 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710. Soviet Union
Paul Pu Liang. Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In ICML., pages 6565-6576 PMLR.
Kadhim Hayawi, Sujith Mathew, Neethu Venugopal, Mohammad M Masud, and Pin-Han Ho. 2022. Deep- robot: a hybrid deep neural network model for social bot detection based user profile data. Social Net- work Analysis and Mining. 12(1):1-19.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1 1692.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv: 025311 2(7).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907, 11622.
Tin Kam Ho. 1995. Random decision forests. In Pro- ceedings of 3rd international conference docu- ment amalysis and recognition, volume 1, pages 278- 282. IEEE.
Yuhan Liu, Zhaoxuan Tan, Heng Wang, Shangbin Feng, Qinghua Zheng, and Minnan Luo. 2023. Botmoe: Twitter bot detection with community-awe mix- tures of modal-specific experts.
Philip N Howard, Bence Kollanyi, and Samuel Wool- ley. 2016. Bots and automation over twitter during the us election. Computational propaganda project: Working paper series, 21(8).
Yi-Ju Lu and Cheng Te Li. 2020. Gean: Graph- aware co-attention networks for explainable fake news de- tection social media. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 505-514.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. 2020. Heterogeneous graph transformer. In Pro- ceedings of The Web Conference 2020. pages 2704-.. 2710.

---

Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng. Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang. 2021. Are we really making much progress? revisiting, bench- marking and refining heterogeneous graph neural networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1150-1160.
Huailiang Peng. Yujun Zhang, Hao Sun, Xu Bai, Yangyang Li, and Shuhai Wang. 2022. Domain- aware federated social bot detection with multi- relational graph neural networks. In IJCNN, pages 1-8. IEFE
Phu Pham, Loan TT Nguyen, Bay Vo, and Unil Yun. 2022. Bot2vec: a general approach of intra- community oriented representation learning for bot detection in different types of social networks. Infor- mation Systems, 103: 101771
William Marcellino, Madeline Magnuson, Anne Stick- clls, Benjamin Boudreaux, Todd C. Helmus, Ed- ward Geist, and Zev Winkelman. 2020. Counter- radicalization bot research using social bots to fight violent extremism. Technical report, RAND CORP SANTA MONICA CA.
Sarah Pierce and Doris Meissner. 2017. Trump execu- tive order on refugees and travel ban: A brief review. Migration Policy Institute.
Michele Mazza, Stefano Cresci, Marco Avvenuti, Wal- ter Quattrociocchi. and Maurizio Tesconi. 2019, Ri- bust: Exploiting temporal patterns for botnet detec- tion on twitter. In Proceedings of the 10th ACM conference on web science, pages 183-192.
John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized like- lihood methods. Advances in large margin classifiers, 106)):1177.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 116
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, 2013. Efficient estimation of word representa- tions in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Joshua Roesslein. 2009, tweepy documentation. On- line] http://tweey.. readthedocs. io/en/v3, 5.
Zachary Miller, Brian Dickinson, William Deitrick, Wei Hu, and Alex Hai Wang. 2014. Twitter spammer detection using data stream clustering. Information Sciences, 260:64-73.
Sippo Rossi, Matti Rossi, Bikesh Upreti, and Yong Liu. 2020. Detecting political bots on twitter during the 2019 finnish parliamentary election. In Proceedings of the 53rd Hawaii international conference on sys- tem sciences.
Fred Morstatter, Liang Wu, Tahora Nazer, Kathleen M Carley, and Huan Liu. 2016. A new approach to bot. detection: striking the balance between precision and recall. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 533-540. IEBE.
Mohsen Sayyadiharikandeh, Onur Varol, Kai-Cheng Yang, Alessandro Flammini, and Filippo Menezer. 2020. Detection novel social bots by ensem- bles of specialized classifiers. In Proceedings of the 29th ACM international conference on information & knowledge management, pages 2725-2732.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5356-5371.
Chengcheng Shao, Giovanni Luca Ciampaglia, Onur Varol, Alessandro Flammini, and Filippo Menezer. 2017. The spread of fake news by social bots. arXiv preprint
96:104.
Lynnette Hui Xian Ng and Kathleen M Carley. 2022, Botbuster: Multi Multi-platform bot detection using a mix- ture of experts. arXiv preprint arXiv:2207.13653
Massimo Stella, Emilio Ferrara, and Manlio De Domenico. 2018 Bots increase exposure to negative and inflammatory content in online social systems. Proceedings of the National Academy of Sciences, 115(49):12435 12440.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, teg-peerformance deep learning library. Advances in neural information processing systems, 32.
Joshua Uyheng and Kathleen M Carley. 2020. Bot impacts on public sentiment and community struc- tures: Comparative analysis of three elections in the asia-pacific. In International conference on social computing, behavioral-culhe tion and behavior representation modeling and simulation, pages 12-22. Springer.
modeling and predic-
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gram- fort, Vincent Michel, Bertrand Thirion. Olivier Grisel. Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vin- cent Dubourg, al. 2011. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-8330
Onur Varol. 2022. Should we agrec to disagree about twitter's bot problem? arXiv preprint arXiv:2209.1000

---

Onur Varol, Emilio Ferrara. Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2017. Online human-bol interactions: Detection, estimation, and characterization. In ICWSM, volume 11, pages 280-
289.
Youze Wang, Shengsheng Qian, Jun Hu, Quan Fang, and Changsheng Xu. 2020. Fake news detection via knowledge-driven multimodal graph convolutional networks. In Proceedings of the 2020 International Conference Multimedia Retrieval, pages 540-547.
Patrick Weber. 2014.
Discussions in the comments section: Factors influencing participation and interac- tivity in online newspapers' reader comments. New media& & society, 16(6)991-997.
Feng Wei and Uyen Trang Nguyen. 2019. Twitter bot detection using bidirectional long short-term memory neural networks and word embeddings. In 2019 First IEEE International Conference Trust, Privacy and Security Intelligent Systems and Applications (TPS-ISA), pages 101-109. IEEE.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of oftte-of-the-rr natural language processing. In Proceedings of the 2020 con- ference on empirical methods natural language processing: system demonstrations, pages 38-45.
Samuel C Woolley. 2016. Automating power: Social bot interference global politics. First Monday.
Kai-Cheng Yang, Emilio Ferrra,, and Filippo Menezer. 2022a. Botometer 101: Social bot practicum for computational social scientists. arXiv preprint arXiv:2201.1608.
Kai-Cheng Yang, Onur Varol, Clayton A Davis, Emilio Ferrara, Alessandro Flammini, and Filippo Menezer. 2019. Arming the public with artificial intelligence to counter social bots. Human Behavior and Emerging Technologies, 1(1):48611.
Kai-Cheng Yang. Onur Varol, Pik-Mai Hui, and Filippo Menczer. 2020. Scalable and generalizable social bot detection through data selection. In Proceed- ings of the AAAI conference on artificial intelligence volume 34, pages 1096-1103.
Yingguang Yang, Renyu Yang, Yangyang Li, Kai Cui, Zhiqin Yang, Yue Wang, Jie Xu, and Haiyong Xie. 2022b. Rosgas: Adaptive social bot detection with reinforced self-supervised gnn architecture search. ACM Transactions on the Web.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. 2021. Graph-less neural networks: Teaching old mips new tricks via distillation. In ICLR.

---

A Discussion
Individual (F1-seor)
Community (bxt ()
Model
True
All Fulis
Random
All Trae
All False
Random
In this work, propose the novel setting of community-level bot detection, design the BotPercent pipeline, and investigate the bot popu- lation from groups to crowds. Our experiment and results demonstrate that the distribution of Twit- ter bots is not homogeneous, but rather possesses spatial-temporia patterns. The spatial and temporal nature of the bot populations has great implications for social media moderation, day-to-day users, and future research.
Yang al. LOBO
0.518
0.540
0.583
17.90%
23.90%
20.89%
0.272
0.476
0.380
8.01%
16,207
12.08%
BeIRGIN
1.640
0.456
8.01%5
42.00%
ROT
0.002
0.639
0.461
0.01%
83 234g
41 58%
BotBuster
0,008
23.21%
11.52%
BotPercent
0.656
0.672
0,668
47.67%
59,94%
55.21%
Table 2: Feature perturbation study where we alter user verification to all True, all False, or randomly assigned. We report F1 score for individual analysis and the bot percentage for community-level detection, where the ground truth is 50% and the closer the better.
Since the Twitter bot population has spatial pat- terns, it could be beneficial to invest more resources into high-stakes communities with higher bot pres- ence. For example, in this work, we demonstrated that the bot percentage in political and cryptocur- rency communities is much higher than in enter- tainment. This finding suggests that in addition building general-pus purpose bot detection systems, the fine-grained study of these "highly polluted" communities (Uyheng and Carley, 2020) also of integral importance.
Hy pergarameerr
Graph-Based
Text-Based
Distillation
LEARNING RATB
1 10-3
Ix 10
3 10-1
BATCH 5126
128
EPOURS
50
50
50
1.2 ERGULARITTTION
10-6
-1055
10-1
HIDDEN DIM
128
128
1024
DROPOUT
0.5
LAVER COUNT L
2
0.7
Table 3: Hyperparameter settings of BotPercent.
an important indicator (Yang et al., 2020), recently there were significant changes to Twitter's verifica- tion policy: existing verified users might lose their verified status, while previously unverified users could get a blue checkmark by subscribing to Twit- ter Blue. This has great implications for Twitter bot detection since verification was a widely adopted and essential feature across multiple types of bot detectors. As result, desirable bot detection system should be robust to such feature perturba- tions, especially for the verified binary feature. To this end, we evaluate BotPercent and baselines on three new settings: a) all users become verified users, b) all users become unverified users, and c) the user verification status is randomly assigned. This is to imitate a scenario where user verification is no longer reliable and how bot detectors would fare in this case. We present the results in Table 2. which demonstrates that disabling the verification feature would seriously cripple the performance of existing bot detection systems. On the contrary. BotPercent maintains steady performance both in individual (determing the bot-or-not of individual users) and community- approaches, thanks to its multi-modal and multi-model pipeline reducing the over-reliance on the verified feature.
Since the Twitter bot population has tempo- ral patterns, it could be beneficial to take socio- political events into account when investing con- tent moderation resources. For example, in this work, we showed that bot presence in the online discussion regarding immigration and healthcare often spikes around major U.S. elections. This find- ing suggests that informing social media users of potential interference and curbing the malicious im- pact of Twitter bots is especially important during elections (Bessi and Ferrara, 2016), referendums (Stella al., 2018), and other major socio-political events (Woolley, 2016).
B Problem Definition
Let u= (my, ue) be a social media account, where "f represents all its accessible features (metadata, textual, graph-based) and u. [human, bot) rep- resents whether the account is managed by hu- man or bot. Let be a group of users, and let Pu = Hu : u E U4 ue = bot}| [24] the true pro- portion of bots in the population U. Community- level bot detection targets to learn a precise estima- tor Pu of per, i.e. aiming to minimize pu pirl-
D Bot Percentage among Active Users
Feature Perturbation Study
We also provide an answer to an important and widely debated question: the overall percentage of bots among active Twitter users. We define ac-
While existing Twitter bot detection approaches heavily rely on the verified (i.e. blue check mark) as

---

Economy
Healtheare
Crime
Supeeme
Immigration
Bot Po.Percemlage
15%
Dot Percentage
20064
Bot Percentag
15%
Bot Percentage
10%
Percealage
2005
10%
1056
M
0%/
2014
2022
2022
0%
2022
Year
Year
Year
Year
Year
Gun Policy
Foreign Policy
Aburtion
Racism
Climate Change
Bot Porcelaag
15%4
Bot Percentage
10%
Bot Percetttpo
10%
Bot Percentage
10%
Percerinne
10%
10%
556
5%
5%
2014
2018
2022
2014
2018
2022
2014
2018
2022
2014
2013
2022
2014
2015
2022
Year
Year
Year
Year
Year
Figure 10: The temporal trends of the bot percentage on the 11 political issues over the past decade.
Model
Accuracy
Fl-score
F Details of Bot Percentage Trend in Different Topics
Yang et al. (Yang al., 2020)
0.623
0.395
BotHunter (Gu al., 2007)
0.614
0,370
LOBO (Echeverna et al., 2018)
0.552
0.198
To facilitate readability, we divide Figure 7 into 10 topics and individually present trends in Figure 10.
RoBIERTa (1.iu et al., 2019a)
0.633
0.432
BuRGCN (Feng et al., 2021e)
0.488
0.485
RGT (Peng et al., 2022n)
0.509
0,509
Butometer (Yang el al., 2022a)
0.755
0.585
G Communities Details
BotBuster (Ng and Carley, 2022)
0.627
0.439
BotPercent (Ours)
0.731
0.726
We adopt the 10 community datasets proposed in the TwiBot-22 dataset (Feng et al., 2022b) to eval- uate BotPercent in Section 3.1, which start from five closely connected sub-communities around @ BarackObama, @ @elonmusk, @ CNN, @NeurlP- SConf, and ladygaga. Then, Feng et al. (2022b) used K-means to cluster the Word2Vec (Mikolov et al., 2013) representations of hashtags and identi- fied users tweeting about similar bashtags into five additional sub- communities.
Table 4:
Individual bot detection performance. BotPercent achieves significantly higher Fl-score thanks to its balanced and well-calibrated predictions.
tive users those who generated content the platform (posts, comments, retweets) and exclude passive users (those that solely browsed or liked posts). This may be seen as a proxy understand how much of the genuine users' content consump- tion comes from bot accounts.
H Hyperparameter Details
The hyperparameters of BotPercent are presented in Table 3 to facilitate reproducibility.
Dataset Details
E Individual Bot Detection
The CRESCI-15 (Cresci et al., 2015) dataset mainly consists of accounts collected from volunteer base and active Italian Twitter users. Users in the GILANI-17 (Gilani et al., 2017) dataset are collected with the Twitter streaming API and are grouped into four categories based on the num- ber of followers. CRESCI-17 features three types of bots: traditional spambots, social spambots, and fake followers. The MIDTERM- 18 (Yang et al., 2020) dataset is filtered based on politi- cal tweets and active users collected during the 2018 U.S. midterm elections. For the CRESCI- STOCK-18 (Cresci al., 2018, 2019) dataset, bot users were identified by finding accounts with similar timelines among tweets containing the se- lected hashtags during five months in 2017. The
In addition achieving state-of-the-art perfor- mance on community-level bot detection, we also evaluate BotPercent at the individual level. We leverage the expert annotated dataset the TwiBot- 22 benchmark. BotPercent outperforms all base- lines by at least 14.1 F1-score (see Table 4). While Botometer achieves slightly higher accuracy, fur- ther examination shows that it greatly underesti- mates the bot population (also shown in Figure 4) by skewing its predictions heavily towards genuine users, which is also evident in its lower Fl-score re- sults. In summary, BotPercent achieves more bal- anced predictions and yields an overall improved performance.

---

User Metadata
Derived Feature
Calcufation
NAMI/USERY
digits
FOI LOWIR, ETINTT
TWERT PREQUENCY
status, count user, age
FRIEND CCUNTT
COUNT
No. URL descriptions
FAVORITE COUNT
BOT WORD DESCRIPTIONSO NAME/USERNAME
No. in descripiionseeee nomehoerrame string
LISTED COUNT
USERNAME ENTROPY
where normalized string length name string
DRFAULT
SURIIEN NAMIUU NIRNAMI/OESSSSEE LINGII
PROFILE_ USE, BAC.BBOCCCDDDDDOCC JMAGE
FOLLOWERS, GROWTHLRAD
followers, count uuser_age
FRIENDS RATE
usee_age
USER_ID
SCRBEN NAME/DESCRIPTION WASHTNG_ COUNT
No. bashing in string
FOLLOWER_ RATIO
follower, count
count
IOSAL.CATTION
USTRNAMMI LATTTAI DOUNT
No. cugetal letter usernume and
USERLAGE
SCREEN NAME/USERNAME UNICODE GROUP
group user's
Unicode groups
DESCRIPTION DEICKIPTIONSENTMMEYY JEENIMEET__ SCORE
sentiment sccces generated by VADER (Humo and Gilhert, 2014) Levensttiin distance (Levenenccn al., 1966) between two strings
USERNAME SCREEN
Table 5: Features adopted in the feature-based module in BotPercent. User metadata features are directly extracted from Twitter API, and derived features are calculated based on user metadata.
category
sub-model
weight
K Learned Weights for Each Model
Random Forest
0.544
To gain better understanding of the importance of each module and its implications in BotPercent, we present the learned weights for each sub-model in Table 6. We found that feature-based methods received the highest weight, while text-basee based meth- ods received the lowest weight. This indicates that fcature-based methods have the greatest impact on the prediction results, while graph-based methods have slightly greater impact than text-based meth- ods.
feature-based
1.127
Adaboost
0.583
RoBERTa
0.404
text-based
0.815
T5
0.411
HGT
0.247
SimpleHGN
0.205
graph-based
0.852
BoIRGCN 0.192
RGT 0.208
Table Learned weight ays for each sub-model (Sec- tion 2.3), where j E gf denotes feature-, text-, and graph-based methods respectively.
L Computation Details
CRESCI-RTBUST-I (Mazza et al., 2019) dataset was crawled from Italian retweets between 17- 30 June 2018. The BOTOMETER-FEDDACKK
We used a server with 8 NVIDIA 2080 GPUs, 178GB memory, and 24 CPU cores for BotPercent training. The inference stage is con- ducted on a PC with 4 CPU cores and 16GB mem- ory. Training BotPercent with the best hyperpa- rameters takes approximately hour on the merged dataset.
(Yang et al., 2019) dataset was constructed by man- ually labeling accounts annotated by feedback from Botometer users. TwiBor-20 (Feng al., 2021b) consists of users from four interest domains col- lected from July September 2022. TwIBoT- 22 (Feng et al., 2022b) uses diversity-aware BFS to collect users by expanding with the follow re- lationships. To prevent test data leakage, we re- moved user labels for the 10 communities, as well the expert-labeled users in the TwiBot-22 dataset. Moreover, we created dictionary structure project- ing from user ID their account information to prevent data leakage caused by user overlap. This ensures no overlap between the training data the merged version and the data used for evalua- tion.
M Scientific Artifacts
BotPercent is built with the help of many exist- ing scientific artifacts, including PyTorch (Paszke et al., 2019), torch geometric (Fey and Lenssen, 2019), Tweepy (Roesslein, 2009), Numpy (Harris et al., 2020), sklearn (Pedregosa al.. 2011), 1), and transformers (Wolf et al., 2020). We have made the BotPercent implementation publicly available.
N Active-User Collection Methodology
We use the StreamClient function in the Twitter API sample 1% of real-time tweets and corre- sponding users from December 12th 18th, 2022. The StreamClient Twitter API function only pro- vides posts or retweets but does not include users who only liked or browsed Twitter content.
J Feature Details
To facilitate further research, we present the list of features used in the feature-based module of BotPercent in Table 5.

---

Baseline Details
Yang et al. (2020) leverages types user metadata and 12 derived features to identify bot accounts with random forest.
BotHunter (Gu et al., 2007) leverages ran- dom forest to classify users based on account metadata, network attributes, content, and tim- ing information.
LOBO (Echeverna et al., 2018) extracts 26 user features and feeds them into random for- est for classification.
RoBERTa (Liu et al., 2019a; Feng et al., 2022b) leverages RoBERTa encode user tweets and descriptions and feed them into MLP distinguish bots from humans.
BotRGCN (Feng et al., 2021c) utilizes text information from descriptions and tweets, and numerical and categorical user property information as user features. Then BotRGCN constructs heterogeneous graph from the Twitter network based on user relationships and applics relational graph convolutional net- works (RGCN) for bot classification.
RGT (Feng et al.. 2022a) uses graph trans- formers and semantic attention network to model the intrinsic influence heterogeneity and relation heterogeneity in Twittersphere. RGT uses the same input BoIRGCN and feeds user features into RGT layers for bot identification.
Botometer (Yang al., 2022a) is a public website to check the activity of Twitter ac- count and returns score, where higher scores mean more bot-like activity. Botometer sys- tem leverages more than 1,000 features using available metadata and information extracted from interaction patterns and content.
BotBuster (Ng and Carley, 2022) enhances cross-platform bot detection by processing metadata and textual information with the mixture-offexpprts architecture.


Reviews:
[{"review_id": "6AUe5Ai99I", "reviewer": "Reviewer_TcPZ", "summary": "", "questions": "A. Can you elaborate on the utility of \"community\" or running bot-detection method on a certain \"community\" ? Is it rendering some information that is helpful to predict if a particular account is Bot or not. If yes, will this method fail for uncohesive or arbitrary community ? \n\nB. The jointly working group attacks are often well studied in data-mining literature, which means multiple accounts working together in a lock-step behavior. Does it make sense to compare and contrast with that literature as well ?\n\nC. [Individual vs Aggregate Metric] I would say though the aggregate metric i.e. percentage of bots in a community is interesting, but it is more investigative than intervention or action-oriented. Though the method performs well on both individual and aggregate metric, but I would want to know the motivation behind choosing a more investigative metric rather than more action-oriented metric to optimize on. I treat this not as a weakness especially the method works well on both of the said metrics. But if it wouldn't have then I would have preferred to opt in on more intervention based rather than investigative metric.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback!\n\n> Data leakage between evaluation and training set\n\nTo prevent test data leakage, we removed user labels for the 10 communities, as well as the expert labeled users in the TwiBot-22 dataset. Moreover, we created a dictionary structure projecting from user ID to their account information to prevent test data leakage caused by user overlap. This ensures no overlap between the training data in the merged version and the data used for evaluation.\n\n> The definition of community is not clear\n\nFollowing previous works [1-2], we define Twitter communities by network proximity. Specifically, we evaluate BotPercent on the 10 Twitter communities collected in [1], which were obtained using BFS search and cluster techniques. The TwiBot-22 work examined that these proximity-based communities do form loosely connected interest groups, thus we follow this definition. We will add this clarification to Section 3.\n\nRegarding the utility of \"community\", our design choice of distilling the graph neural network into an MLP allows BotPercent to effectively fit into arbitrary user groups or communities and estimate the bot population within them. This means that the method can be applied to different user groups, regardless of how they are defined.\n\nIn terms of data, the 10 user communities were obtained in the TwiBot-22 paper using BFS search and cluster techniques. For a more comprehensive understanding of the process, Please refer to Appendix H for detailed information.\n\n[1] Feng, S., Tan, Z., Wan, H., Wang, N., Chen, Z., Zhang, B., ... & Luo, M. (2022). TwiBot-22: Towards graph-based Twitter bot detection. Advances in Neural Information Processing Systems, 35, 35254-35269.\n[2] Feng, S., Wan, H., Wang, N., Li, J., & Luo, M. (2021, October). Twibot-20: A comprehensive twitter bot detection benchmark. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (pp. 4485-4494).\n\n> Performance gain from ensemble and calibration is not clear\n\nIn Appendix C, we present an ablation study to investigate the factors contributing to the performance gains of BotPercent. The study involves removing feature-, text-, graph-based methods, and confidence calibration individually to assess their impact. Our findings reveal that the uncalibrated ensemble performs 19.9% worse than the calibrated ensemble. Moreover, the calibrated ensemble outperforms the calibrated model without feature-, text-, and graph-based models by 11.3%, 5.4%, and 29.9%, respectively. Results indicate that both ensemble and confidence calibration contribute significantly to the performance gain on community-level Twitter bot detection.\n\n> Comparison with the joint working group attacks and lock-step behavior literature \n\nThanks for this great suggestion! We will discuss the joint working group attack and lock-step behavior literature, e.g.  [1, 2], in the final version.\n\n[1] Jiang, M., Cui, P., Beutel, A., Faloutsos, C., & Yang, S. (2016). Inferring lockstep behavior from connectivity pattern in large graphs. Knowledge and Information Systems, 48, 399-428.\n\n[2] Beutel, A., Xu, W., Guruswami, V., Palow, C., & Faloutsos, C. (2013, May). Copycatch: stopping group attacks by spotting lockstep behavior in social networks. In Proceedings of the 22nd international conference on World Wide Web (pp. 119-130).\n\n> Individual or aggregate metric\n\nThe choice of an aggregate metric, such as the percentage of bots in a community, stems from the need to gain a comprehensive understanding in situations where data access is limited and the dataset size is immense. In these cases, an aggregate metric allows us to probe and analyze the global perspective more effectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n2. The proposed method is an ensemble on top of existing approaches and then confidence is calibrated to identify if a particular account is a bot or not. The method outperforms multiple other baselines.\n3. The analysis section is very insightful and could have wide-spread impact. ", "reasons_to_reject": "The paper is centered around an important topic, but there are some drawbacks in the paper.\n\n[Training Dataset] The training dataset is an amalgamation of all merged datasets. One of the valid things to test out is that if there is any data leakage (temporal or other) between the evaluation set and the training set.\n\n[Definition of Community] The definition of community is somehow unclear in the experiments. The authors in most of the settings assume that the communities are given - or are estimated through the comment section of influential accounts. It is harder to understand if having an approach i.e. \"community-specific\" adding any value. Let's say if I arbitrarily group a set of users as \"community\" - will the estimates within that community be worse than any other actual \"community\" ?\n\n[Performance Gains] It is unclear if the performance gains are coming from ensemble of the models or some \"community-level\" calibration. Ideally the ensemble itself is a good contribution - but doesn't warrant enough to be a scientific article where the goal is to propose or unlock a new understanding. It would be ideal to see performance comparison among different ensembles such as BotRGCN or BotBuster or something else etc. It might also be helpful to clarify if how similar are individual methods in BotPercent to other baselines.", "questions_to_authors": "A. Can you elaborate on the utility of \"community\" or running bot-detection method on a certain \"community\" ? Is it rendering some information that is helpful to predict if a particular account is Bot or not. If yes, will this method fail for uncohesive or arbitrary community ? \n\nB. The jointly working group attacks are often well studied in data-mining literature, which means multiple accounts working together in a lock-step behavior. Does it make sense to compare and contrast with that literature as well ?\n\nC. [Individual vs Aggregate Metric] I would say though the aggregate metric i.e. percentage of bots in a community is interesting, but it is more investigative than intervention or action-oriented. Though the method performs well on both individual and aggregate metric, but I would want to know the motivation behind choosing a more investigative metric rather than more action-oriented metric to optimize on. I treat this not as a weakness especially the method works well on both of the said metrics. But if it wouldn't have then I would have preferred to opt in on more intervention based rather than investigative metric.", "paper_topic_and_main_contributions": "The paper is centered on estimating the number of bots (percentage) within a certain community. The authors claim that the current bot detection methods are unimodal and more importantly, do not take into account communities or social environment into consideration while trying to find bots. The paper then goes on to use multiple different unimodal models (feature , text, graph-based) and calibrate their output scores. This proposed framework is used to estimate percentage of bots in different communities.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. I think literature on lock-step behavior from data mining research could be interesting here.\n"}, "extra_text": {}}, {"review_id": "x3hdl7DJi0", "reviewer": "Reviewer_q2rU", "summary": "", "questions": "Please refer to W1 and W2 in my detailed comments above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback!\n\n> How to ensure the sampling does not affect the graph information?\n\nAs discussed in line 352 to 358, we conducted downsampling on the 50:50 dataset and got user communities with bot percentages ranging from 10% to 90%. \n\nFor our proposed BotPercent, thanks to our design choice that distills GNNs into MLP, we could conduct efficient and scalable inference without social links. That being said, BotPercent doesn’t need graph structure (social links) and is immune to the change of graph structure in the inference stage.\n\nRegarding the graph-based baseline models, the underlying graph structure remains unchanged throughout the experiments. The selection of users within a specific community for analysis does not alter the graph structure but rather serves as a means to evaluate the model's predictions within that particular context.\n\n> Ablations on user features and GNN/LM choice are needed\n\nThanks for pointing this out. We did an ablation study on feature-, text-, graph-based models, confidence calibration, and weighted sum aggregation in Table 1, Appendix. We agree that it would be beneficial to extend this ablation study to include the analysis of user features, different language models (e.g., DistilBERT), and specific GNN architectures or combinations for the text-based and graph-based components, respectively. We will add these ablation settings in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are a few strong points about this paper that I particularly like, which are:\n\nS1.\tThis paper studies the important problem of bot detection on Twitter, or more specifically the proportion of bots given a specific community. This is an interesting aspect of the bot detection problem but also one that needs more motivation. For instance, detecting individual users as bots allow for actions such as banning or suspending that account but know the proportion of bots allows for less direct actions, although is still useful in some ways.\n\nS2.\tThe proposed BotPercent model makes good use of multiple modality or input types of user features, text features and graph networks. Within each component, there is also a fair amalgamation of techniques for that specific domain, e.g., various types of GNN. \n\nS3.\tThere is a fairly comprehensive set of experiments on a relatively large bot dataset, with various bot configurations being studied such as a bot distribution from 10% to 90%. In these experiments BotPercent shows good performance over various baselines including the ones that uses a single input type or modality.\n\nS4.\tIn addition to the comparison experiments, there are also various interesting investigations into the distribution of Twitter bot population over different time periods and across different interest groups.\n\nS5.\tIn general, this is a nicely written and well structured paper that is overall a pleasure to read.", "reasons_to_reject": "There are a few weak aspects of the paper that the authors can clarify or improve on, which are:\n\nW1.\tThere is a good consideration of experiments based on two settings, a balanced 50:50 bot:real user setting and various bot percentages of 10% to 90%. Given that the proposed model includes a GNN component that uses the social links, how do you ensure that any sampling of bots for this percentage do not (either favourable or unfavourably) affect the proposed model or any baselines that uses graph information?\n\nW2.\tWhile the BotPercent model is well-described, certain architecture design considerations or intuitions are missing from the paper. For example, how are the 12 direct and 14 derived user features identified and selected, and what effect will another user feature set have? Similarly, how would another language model (e.g., DistilBERT instead of RoBERTa) or a specific GNN or GNN combination work for the text-based and graph-based ocmponents, respectively? An ablation study might be useful for investigating some of these factors.", "questions_to_authors": "Please refer to W1 and W2 in my detailed comments above.", "paper_topic_and_main_contributions": "This paper studies the problem of bot population detection on Twitter, which aims to determine the proportion of bots as compared to the tradition task of detecting whether a single user is a bot. To address this problem, the paper proposed BotPercent a multi-modality neural network that combines different architectures for different input types, e.g., random forests and AdaBoost for user features, RoBERTa and T5 for txt-based features, and various GNNs for the user network/graph. There is a comprehensive set of experiments showing the good performance of BotPercent against various baselines, as well as various interesting studies into the bot population on Twitter over time and across different interest groups.\n\nI wish the authors all the best as they proceed with this work.\n\nPost-rebuttal: I thank the authors for their responses and acknowledge that they have been read. My scores remain unchanged.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wwb2ySphKb", "reviewer": "Reviewer_Rurt", "summary": "", "questions": "Q-A: In section 4.3, did you check for overlaps among these accounts? Overlaps among the accounts that have had any of the three types of engagements and overlaps among the accounts that have interacted with either of these two polls.\nQ-B: In section 4.3, the authors have concluded that these percentages of bots could have changed the outcomes of the two polls. This means that most of the 8–14% of accounts identified as bots have voted for the reinstatement of Donald Trump on Twitter (since this is the only way the outcome could have changed). On the other hand, if 8–14% of accounts that have voted for Elon Musk Step Down are bots, most of them should have been against Elon Musk being the CEO (this is the only way they could have changed the outcome. Since currently, the poll asks him to step down and the supporters of stepping down are the winners of the poll). But, if Elon Musk as the CEO paves the way for Trump to reinstate, why should they change the outcome in the next poll (one month after the first) to step down? :)\nAssuming the analysis stated in the previous question (Q-A) is not done (at least, it is not reported in the present paper), I think such a claim cannot be realistic.\nQ-C: In section 4.3, what is the distribution (number, not percentages) of users in each poll?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback!\n\n> Presentation problem: too long, repetition, and verbose language\n\nThanks for your suggestion! We will streamline the introduction, methodology description, as well as experiments in the final version.\n\n> The individual-level bot detection performance is not clear. To what extent the performance is based on the correct prediction, like true positive and false positive?\n\nIn Appendix F, we present the individual-level bot detection performance (Acc and F1) on the expert annotated dataset in TwiBot-22. In the final version, we will include the individual-level bot detection performance metrics, such as recall and precision to assess the extent to which the achieved performance on the overall bot percentage is based on the correct detection of bots.\n\n> In section 4.3, did you check the overlaps between either of the two interaction groups?\n\nWe conducted the experiment, and the results are presented in the table below (in user count):\n\n|    | like & retweet  | like & comment | comment & retweet|\n|  ----  | :----:  | :----:  | :----:  |\n| **Reinstate Trump vote**  | 801 | 20 | 20 |\n| **Musk step-down vote**  | 892 | 48 | 32 |\n\n> In section 4.3, if the identified percentage of bot accounts could have changed the outcomes of the two polls, it would imply that most of the bots voted for the reinstatement of Donald Trump and against Elon Musk as CEO, which seems contradictory and unrealistic.\n\nWe examined the overlap between users who participated in the Reinstate Trump vote and the Musk step-down vote. Out of the 37,498 and 38,506 users involved, only 536 users overlapped, accounting for 0.91% and 0.88% respectively. This statistic suggests that the users or bots participating in both votes differ significantly, potentially holding contrasting opinions and voting for different stances. Consequently, we believe our claim that \"bots could have influenced the outcome\" to be reasonable.\n\nTo provide further context, as of 2023, Twitter has 353.9 million active users [1]. Based on our findings, approximately 28 million bot accounts exist, while the users engaged in these votes amount to 15 million and 17 million respectively. This further serves as evidence that the users involved in these two votes can be distinct, with minimal overlap.\n\n[1] https://www.statista.com/statistics/303681/twitter-users-worldwide/\n\n> In section 4.3, what’s the distribution of users in each pool?\n\nThe user count distribution is:\n\n|    | retweet  | comment | like |\n|  ----  | :----:  | :----:  | :----:  |\n| **Reinstate Trump vote**  | 11,079 | 5,634 | 21,626 |\n| **Musk step-down vote**  | 12,172 | 8,519 | 18,773 |\n\n> Typo in line 223, line 485, and figure order\n\n Thank you for pointing out! We have fixed these typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1: The presented work achieves state-of-the-art performance and performs fine on detecting bot percentages in the evaluated communities and datasets\nA2: The paper uses a variety of features, models, and tools from the literature to build a reliable bot detector and compares its performance with a benchmark\nA3: The paper is well-written, and the figures and tables have proper qualities\nA4: The authors have investigated several different communities in different domains to evaluate their approach and provide insights", "reasons_to_reject": "R1: The paper is too long, and important discussions such as the datasets, names and descriptions of features are left for the Appendix. \nR2: There are many repetitions and verbose language in the presented paper. For example, concepts like community-based bot detection and calibration are repeated several times before actually getting to the points the paper explains its contribution towards each.\nR3: While the aggregate/overall bot percentages for 10 populations/communities are presented in the paper, it is not clear how the BotPercent's individual bot detection performs in each of these communities and the whole datasets. For example, if the bot percentage is 40% instead of 50%, what portion of those 40% of accounts identified as bots are really bots? Such analysis would clarify to what extent the achieved performance on the overall bot percentage is based on correct detection of bots (true positives), and to what extent is because of false positives.", "questions_to_authors": "Q-A: In section 4.3, did you check for overlaps among these accounts? Overlaps among the accounts that have had any of the three types of engagements and overlaps among the accounts that have interacted with either of these two polls.\nQ-B: In section 4.3, the authors have concluded that these percentages of bots could have changed the outcomes of the two polls. This means that most of the 8–14% of accounts identified as bots have voted for the reinstatement of Donald Trump on Twitter (since this is the only way the outcome could have changed). On the other hand, if 8–14% of accounts that have voted for Elon Musk Step Down are bots, most of them should have been against Elon Musk being the CEO (this is the only way they could have changed the outcome. Since currently, the poll asks him to step down and the supporters of stepping down are the winners of the poll). But, if Elon Musk as the CEO paves the way for Trump to reinstate, why should they change the outcome in the next poll (one month after the first) to step down? :)\nAssuming the analysis stated in the previous question (Q-A) is not done (at least, it is not reported in the present paper), I think such a claim cannot be realistic.\nQ-C: In section 4.3, what is the distribution (number, not percentages) of users in each poll?", "paper_topic_and_main_contributions": "This paper proposes a multi-dataset multi-model approach based on user metadata and derived features, written text, and network structure for estimating the percentage of bots in Twitter communities. The authors combine and calibrate the modular components and compare the performance of their proposed bot detection with a recent benchmark (TwiBot-22) and achieve state-of-the-art performance. They also investigate the ratio of bots in several Twitter communities, such as those who have commented on celebrities' tweets, news media, or content moderation polls posted by Elon Musk on Twitter. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "T1: In line 223, I think \"i-th layer\" should be \"l-th layer\"\nT2: Figure 4 (section 3.1) comes before Figure 3 (section 3.2) in the text\nT3: Figure 1 is never referred to\nT4: Line 485, \"Figure 8 shows that ∼8% to 14% of users that interacted with the two content moderation votes are bots.\" Based on the figure, is it not 9% to 15%? If not (for example, if you take weighted averages on the three types of interactions), please clarify the percentages.\n"}}]

Author rebuttal:

Meta-review: The paper is centered on estimating the number of bots (percentage) within a certain community. The proposed method is an ensemble on top of existing approaches and then confidence is calibrated to identify if a particular account is a bot or not. 

The proposed approach is interesting although it shows some limitations and drawbacks:
1) It is not clear how the community has been defined in the evaluation set. 
2) Related work should be extended by considering more recent works on data mining field.
3) The authors need to explain how they ensure any sampling of bots for both experimental analysis.
Decision: Accept-Findings