Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a new graph-based Twitter bot detection dataset. The authors describe the dataset in detail together with the annotation method. TwiBot22 is significantly larger than other similar datasets, with almost 1 million users. The authors use an annotation method by employing several human annotators and using this manual annotation in a weak supervised manner to annotate the rest of the dataset. The paper evaluates TwiBot22 on several methods for bot detection, comparing the accuracy of the methods with other similar datasets. 


strengths: The paper is well-written, with an impressively large dataset. By comparison with previous data, TwiBot22 seems significantly larger. The annotation method seems comprehensive. The dataset seems to build on the methodology and data of TwiBot20 by extending the dataset and annotation. 
weaknesses: The authors perform an evaluation comparison between TwiBot and several other datasets on 35 proposed methods. While this analysis is particularly comprehensive, accuracy for TwiBot22 is not that high compared to the rest. Still it is on average over 70%, which seems fine, and it performs well for the graph-based methods, so this is not a big issue. 

I appreciate the detailed Appendix, the data description together with the additional results seem consistent with the results in the main text.  However, Table 8 seems to have some issues: it seems like it's showing the average model performance (F-1) in addition to the table in the main text (which shows accuracy), but the average F-1 for TwiBot22 seems significantly worse than for the other datasets. Unless it's a reporting issue (some typo), this seems significant. For example, by Rodriguez-Ruiz, C-15 is 82.6, C-17 is 86.6, TwiBot-20 is 75.9, while TwiBot22 is 0.8. I am assuming this is a reporting issue given that TwiBot20 is not that different. 
 
I appreciate the short discussion on societal impact and the potential of the dataset to be used to build bot evasion techniques. It would be great to have a slightly longer discussion on that and assess this potential by analyzing the related literature. 
correctness: The paper overall seems correct, with a clear explanation of the dataset construction. The evaluation methods are explicit. However, one concern I have is regarding Table 8 in the Appendix, which shows the F-1 score of several datasets (including TwiBot22) on 35 different evaluation methods. TwiBot22 has a very different score than the other datasets. I'm suspecting a typo, but this should be clarified.
clarity: The paper is well-written and clear. 
relation to prior work: The paper discusses related work well, by comparing previously collected datasets and describing methods for evaluation proposed in the field. The paper does a good job in explaining the different types of datasets used for bot detection, describing the differences between feature-based, text-based, and graph-based methods. 
additional feedback: Detail: missing period at the end of the line in line 599
review_title: Twitter dataset bot detection review 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper tackles the problem of bot detection on Twitter with graph-based methods. To that effect, the paper makes the following contributions:
1. Creates an annotated graph dataset, TwiBot-22, of human and bot Twitter users, with 1M users in total.
2. Presents an evaluation framework to standardize the measurement interface across various different datasets and methods.
strengths: The paper has the following strengths:
1. It tackles the very relevant problem of bot detection on Twitter, which is of consequence to various downstream objectives like spread of misinformation, propagation of rumors, etc.
2. The dataset released is heterogenous with 4 different node types and 14 different relation types, making the dataset close to real-world graphs.
3. Extensive experiments are conducted with a range of state of the art methods, including graph neural networks, to demonstrate the usefulness of the released dataset. The experiments also cover interesting ablation studies to demonstrate the usefulness of having graph signals.
weaknesses: 1. In the experiments section, I would have liked to see testing for statistical significance as well.
correctness: I found the data sampling methodology to be robust, specially accounting for distribution and value diversity. Here, the latter relates to the concept of homophilic behavior, which is important to control for when creating graph datasets.
clarity: The paper is very well-written and easy to follow.
relation to prior work: I found the coverage of prior work to be sufficient. In fact, there are very good comparisons made to the existing TwiBot-20 benchmark in terms of data and modeling trends.
additional feedback: None
review_title: Good benchmark for Twitter bot detection

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a new graph-based Twitter bot detection benchmark dataset. This dataset has been presented as an improvement over the already existing graph-based datasets in terms of dataset scale, extensiveness of the graph structure and annotation quality. The authors evaluated the proposed dataset, along with 8 more existing datasets, using 35 twitter bot detection baseline algorithms. Based on evaluation, the authors argue that the graph-based component of the sota algorithms is responsible for their better performance hence the need of an extensive graph-based dataset. Finally, the authors provide a publicly available evaluation framework which includes all the datasets and baselines used in the paper.
strengths: A. The dataset improves upon the existing graph-based datasets in terms of dataset scale, graph structure and annotation quality. 
B. The evaluation framework which includes implementation of many baseline algorithms and few existing datasets can facilitate bot detection related research.

weaknesses: A. Regarding the first stage of data collection process, this dataset appears to be a direct extension of a previous dataset “TwitBot20”[1] by the same authors with a minor change in the "diversity-aware" sampling.

B. Line 198 : "...We further evaluate annotation quality on the test set of expert annotations...". The test set implied here seems to correspond to the 200 users' set mentioned in line 181. However, in appendix A.5, the test set seems to consist of 1000 users (500 from TwitBot20 and 500 from TwitBot22). Need to address this ambiguity.

C. In section 4.4 (Generalization Study), more details about the 10 sub-communities could be included (similar to what is done in Table 1 for the whole dataset). This could be useful in inspecting the results presented in Figure 3.

D. The authors have stated a possible limitation to be the lack of multimedia content in the dataset. But they have indicated that it is possible to do so using media links in TwitBot-22. The solution being alluded to is not clear.

[1] Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. TwitBot-20: A comprehensive twitter bot detection benchmark. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 4485-4494, 2021.

correctness: Yes. Most of the claims are supported by evidence. However, it is not clear to me that how exactly future graph-based algorithms can leverage a more complex entity-relation structure of the dataset. 
clarity: Yes. The flow of the paper is good.
relation to prior work: This is not the first graph-based dataset. They have mentioned prior graph-based datasets and provided a comparison.
additional feedback: N/A
review_title: Recommending accept after the discussion phase

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper utilizes a graph based approach to create the largest Twitter bot detection dataset. The primary strength of the paper is the dataset which contains over 100,000 bot users which is very useful for research on social media data. Additionally, the authors perform extensive evaluation using latest methods and re-implemented previous baselines and compared the performance of their dataset. The following are the contributions.

1) A graph based approach, TwiBot-22, which contains bot detection dataset that establishes the largest benchmark
2) benchmark framework and re-implemented 35 baselines for evaluation
3) code available to reproduce and dataset available for research
strengths: The following are the strengths of the paper

1) Using labeling functions to create labels and utilizing weak supervision approach. This demonstrates scalability 
2) The dataset is useful as there is an immense need for bot detection techniques with increase in social media research
3) Extensive evaluation with several existing datasets to draw comparison with the graph based method 
weaknesses: The following are the weakness. These needs to be addressed to strengthen the paper

1) TwiBot-22 has less performance when compared to TwiBot-20. Authors made the statement on the decrease in the performance but have not discussed potential reasons for decrease in the performance. Is it because of adding more noise through labelling functions? 
2) The authors do not discuss any limitations of the approach or the dataset. A few limitations could add more perspective to the paper 
3) More details on the dataset is required. For example in Table 1, the authors list number of users and number of humans and number of bots in the dataset. But the authors fail to mention how many of tweets are actually bot tweets. This would be a great contribution factor and add value to the dataset if there are more bot tweets than user tweets. Tweet distribution must be added to the paper. 
correctness: The claims made in the paper are correct and the dataset is constructed in a sound way. The benchmark evaluation methods and experiment design are appropriate and performed correctly. 
clarity: Paper is well written. Please address the weakness points to add more value 
relation to prior work: Yes. Prior work is well discussed 
additional feedback: Authors need to add more information on the dataset and the distribution of tweets in the dataset.
review_title: A well evaluated graph based approach to identify twitter bots 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a large graph-based dataset for Twitter bot detection that helps prevent misinformation and maintain the integrity of the online discourse. Twibot-22, proposed by the authors, includes five times more users than the existing dataset and is a heterogeneous information network, including 4 entities and 14 relation types. It also uses a weak supervision learning strategy to generate high-quality annotations at a low cost. The authors conduct 35 benchmark tests on the proposed dataset. Experimental results imply that Twibot-22 can be used to develop improved automatic methods optimized for Twitter bot detection in more complex real-world situations.
strengths: (1) This paper significantly and robustly extends the data available in Twitter bot detection. It proposes a dataset 5 times larger than the previous dataset.

(2) The dataset contains rich information for Twitter bot detection, providing for the first time ever, a heterogeneous graph.

(3) Extensive experiments are conducted, and the results provide insight into future work. Also, a well-organized evaluation framework is proposed to allow easy access and reuse of datasets and all baselines.
weaknesses: (1) The user expansion process described in Section A.2 is not clear. The authors clarified that the number of users increased from the starting user by expanding to their following and follower users. However, how to achieve 1 million users consequently from starting users? For example, how many iterations of random selection are performed? Did the target number of users exist? More details are needed for clarity.

(2) In Section 3.3, annotation quality comparisons are conducted between Twibot-22 and Twibot-20 only. However, a comparison with other baseline datasets such as cresci-15 should be considered for fair comparison. A comparison of the quality with all other baseline datasets should also be presented that verify whether the proposed dataset has high-quality annotation labels compared to others.

(3) In Section 4.4, the generalization study, the criteria for selecting the five users (e.g., @ BarackObama, @elonmusk, …) for constructing sub-communities should be clarified. For example, what exactly is the “interest domain” used as the criteria? How are the users of the sub-community distributed? Do the characteristics of the sub-communities significantly statistically differ? If so, how the authors evaluate the differences? The authors should provide more details and discussion for verifying that 10 sub-communities could represent the real-world data.


correctness: The database seems quite soundly constructed, and evaluation methods and experiment design are appropriate and performed correctly.
clarity: The paper is clearly written.
relation to prior work: The paper details what previous datasets or approaches for detecting Twitter bots exist and how these datasets differ from the proposed dataset’s aspects of scale, graph type, and the quality of annotations.
additional feedback: Line 532: add a period.
review_title: A well-documented framework with benchmarks.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper provides a new, larger and more comprehensive dataset for Twitter bot detection. It benchmarks a large number of existing methods on the new dataset and 8 others from the literature. There are also some experiments on the importance of graphs in several models and on how well different models generalize.
strengths: 1. The new dataset contributed is significantly larger (more users) and more comprehensive (more data types/graph relations) than previous bot detection datasets. 

2. The benchmarking is extensive, with 35 models over 9 datasets.

3. The dataset and benchmarking codes are accessible.

This is an important real-world problem, with both social good and business applications. Given that and the points above, it seems highly likely that this work will be useful for future researchers.

weaknesses: 1. I have doubts that accuracy is an appropriate metric for the benchmark with the proposed dataset, due to class imbalance. This may affect reliability of some of the conclusions - at the least, I feel they should be justified with another metric. I elaborate in detail below (point #1 under Correctness).

2. Because the annotation process involves several specific model architectures, I am concerned the benchmarking might have some bias in favor of similar models (point #2 under Correctness).

3. There are a few parts which are not completely clear to me. Most importantly, some of the expert annotation process and corresponding evaluation of the dataset annotations (point #4 and part of point #5 under Clarity).


correctness: 1. In the main benchmarking results in Table 2, the paper reports accuracy. However, there is a sizeable class imbalance in their dataset TwiBot-22, with just under 14% bot and 86% human. So a model that always predicts "human" will achieve 86% accuracy, which is higher than any of the reported results. Therefore, it is not clear to me how much these results reflect models' performance vs. simply models' tendency to predict "human." Consequently I am also uncertain about conclusions drawn from these results, such as graph-based methods outperforming others.
The paper does report F1 results in an appendix (Table 8). While even F1 as a metric for binary classification is subject to debate, to my knowledge it is at least much better than accuracy when there is substantial class imbalance. I suggest at least replacing Table 2 with Table 8. Then in appendix present precision and recall individually to add more detail. It would be even better if the authors could add some discussion on the choice of metric. For example, what is standard in the literature? If the authors agree that accuracy is perhaps not appropriate here, is F1 justified or is there something better?

I think some of the conclusions would also need to be revised accordingly. Of the 4 conclusions lines 244-259, the second and third do not depend on the metric so would remain unchanged. Assuming we go by F1, the first would need revised numbers, but just eyeballing it perhaps the overall result that graph-based methods perform better on average still holds? The fourth, regarding performance drop between TwiBot-20 and TwiBot-22, seems much more extreme. Many of the models seem to decay by 30+ percentage points, with some even dropping to near 0. Do the authors have any hypotheses on the cause? Perhaps these models can't handle this extent of class imbalance? Have the authors done any experiments balancing the training data, for example sampling an equal number of human and bot users to train on?

Similar concerns apply to other accuracy results, especially Figure 3, where no F1 results are reported, and the class imbalance for the sub-clusters may vary from the overall dataset and could be even more extreme.

2. Certain models are part of the label annotation process, i.e. lines 183-194. Could that potentially bias the benchmarking in favor of similar models? For example, RGCN is one of the models in the annotation process, and then BotRGCN is noted to perform particularly well in the benchmarking. 
clarity: The paper is generally smooth to read. There are a few sections I have questions or concerns about:

1. For the expansion with k users discussed on lines 151-160, I couldn't find any explicit value of k. The paper does discuss adding 6 users at a time in appendix A.2, so my best guess is k=2. But from lines 154-155, it seems that there should be k*2 users added for a true-false metadata but k*3 for others, i.e. different values added depending on whether true-false or not. I suggest in the appendix explicitly saying the value of k, and if indeed different numbers of users are added depending on true-false or not, saying the two total numbers added per step (e.g. "we add 6 users per step, except for true-false metadata where we add 4"). If it's always 6 users regardless of true-false or not, then I suggest making k the total number of users added, and rewriting 151-160 in terms of fraction of k (e.g. k/3 for each of highest/lowest/random on line 154, k/2 for true-false on line 155, etc.).

2. When collecting additional tweets related to a hashtag, e.g. lines 597-598, does this include all tweets that include that hashtag during the collection period? Or only some subset, e.g. a fixed amount per hashtag?

3. In the generalization study, it would be helpful to elaborate further (e.g. in appendix) on exactly how the subcommunities are constructed. For the user-based ones, are they the one-hop neighborhoods (based on follow relation?) of the 5 selected users? For the hashtag clustering ones, how is word2vec applied, and is anything special done to make it work with hashtags (which are often not dictionary words)? When applying K-means to those representations, is K=5, or else what is the value and how was it chosen? Is a user included in the sub-community as long as they use a hashtag within it, or is there a stricter criterion?

4. On line 199 the labels achieve 90.5% accuracy compared to the expert hand-labeled test set. But then in figure 2c the paper reports accuracy against a different expert hand-labeled set and it's a bit under 80%. This difference seems somewhat large to be random? Do the authors have any comments, explanation, or hypothesis here?
Possibly related, it is not clear to me how the 5-way labels the experts provide (in the second case) are converted to the 2-way labels in the dataset to make this comparison. How are disagreements resolved between the 3 experts providing a label for each example? Similar question for the 1000 expert annotations used in the dataset construction itself. The paper explains each example is labeled by 5 experts as human/bot/not sure, and then majority voting is applied, but how are cases without a majority (e.g. 2x human 2x not sure 1x bot) resolved? What happens to cases with majority not sure, are they dropped, and if so is the 1000 the remainder after dropping such cases or is the actual number used somewhat less than 1000?

5. In Figure 2c, is the F1 reported for the positive class "bot" or something else e.g. macro-average? I would guess that the results in Table 8 are F1 for the positive class "bot" but would also be good to confirm.

There are a few typos such as:

Line 163: "stops until the user network contains..." -> should read "stops when the ..." or "continues until the ..."

Line 200: imporved -> improved

Line 221: bot reporistory -> bot repository

Line 599: period missing at end of sentence



relation to prior work: Generally yes. The paper discusses the data types different methods use, and summarizes how the benchmarked methods work. It discusses previous datasets for this task and how the new dataset presented compares. I think the literature review could be improved by adding discussion of performance metrics used in the bot detection literature (please see point #1 under Correctness for elaboration).
additional feedback: 1. Two expert hand-labeled subsets of users were collected during the annotation process. It might be interesting to see results of the different benchmarked models on these subsets, since in theory they are the most accurate labels available. Also, are they available or marked somewhere in the dataset (on github/drive/etc.)? I didn't find them, but only looked briefly.

2. The Snorkel model used for annotation seems like it might have better performance than any benchmarked model, at least according to the first set expert labels where it achieved 90.5% accuracy. Not crucial to the paper, but besides annotation, is it worth considering that as a future method? Would it perform well on the other datasets benchmarked here?

3. Overall, I feel this paper makes some clear and worthwhile contributions. For example, I fully agree with the authors that larger and more comprehensive data is important for this task, and the dataset contributed will surely be useful for future work on bot detection. However, as explained above, I have some concerns about the validity of the benchmarking results (chief among them the choice of accuracy as metric). There are also some sections where additional details would improve the clarity, not just in terms of ease of reading but for properly understanding and using the dataset and results. Therefore, I currently recommend reject, but would welcome rebuttal and/or revisions addressing these points.

Regardless of the final decision here,  the author's contributions to this important area and hope they will continue research along these lines. Thank you!


EDIT: the authors have done substantial revisions and new experiments that significantly improve the quality of the paper and address most of my concerns. I am raising my overall score accordingly (previously 5 now 7).
review_title: Some meaningful contributions but also some issues

Meta-review: metareview: In this work, the utilize a graph based approach to create the largest Twitter bot detection dataset, with over 100K bot users identified. The dataset is of great use for the social media mining community in many different aspects. The work was greatly improved after a very interactive and fruitful discussion period, making the contribution a lot more refined and useful.

Pros:
- Largest dataset available.
- Thorough evaluation
- Addition of most reviewer's suggestions have strengthen the details in the paper

Cons:
- Minor statistical rigor elements are missing, but not completely necessary.
 
confidence: 5: The area chair is absolutely certain
Decision: Accept