dataesr/scientific-paragraphs-categorization huggingface dataset:
g. ,  Chepfer et 2008 Chepfer et , 2013;; Feofilov et al. 2022 ). (ii) lidar measurements are averaged vertically every 480m, to improve the signal-to-noise ratio (SNR) while maintaining consistency with CloudSat data used to compare with COSP/radar outputs  (Marchand et al. , 2009; Haynes et al. , 2007). This value of 480m can be different in CLIMP as it can be changed in COSP/lidar, but averaging le lidar signal vertically before cloud detection should remain the way to increase ATLID SNR when needed for climate mode evaluation. (iii) cloud detection thresholds are chosen for consistency with COSP/lidar and to prevent false cloud detections in CALIOP L1 daytime data at full horizontal resolution and 480m-averaged vertical resolution. The cloud detection threshold can be modified in CLIMP but then should also be modified in COSP/lidar.
They also redirect to malicious sites for fraud. These bots are considered malicious  [16]. • Hybrid Bots or Cyborgs: This classification corresponds to user accounts that are managed by human users with the help or support of a bot or vice versa  [17]. • Crawlers and scrapers: They are bots that have been designed to index and archive webpages, for example, to make the websites available through search engines. • Chatbots: They are computer programs that converse with people in real-time using natural language. Frequently used by businesses for easier customer interaction, such as responding to frequently asked queries. • Spambots: They are software or computer systems that have been compromised by malware and are under the control of a third party. They are used to send out a large number of messages, advertising, comments, etc.
Moreover, in order to evaluate the accuracy of the extended credibility model, it is incorporated into T-CREo, a framework to perform credibility analysis in Twitter, in real time  [28]. In summary, the main contributions of this work are summarized as follows: • Two datasets for bot detection (English and Spanish), extending the ones proposed in  [29]  with content features, such as wavelet average and entropy; • An algorithm for Twitter bots detection in real time, based on predictive models; one for tweets in English and the other one for tweets in Spanish; • A comparative study for identifying the best features involved in the models that characterize the bots; • An extended credibility model that integrates a bot detection function to calculate the level of credibility of tweets, incorporated into T-CREo to demonstrate its functionality and suitability.
The remainder of this paper is organized as follows. Section II corresponds to the review of works related to bot detection in Twitter and some works which consider characteristics of the user of a Twitter account, in their credibility analysis. Section III describes the methods used for Twitter bot detection including the dataset configuration, the preprocessing, and the training and selection of predictive models. In Section IV, the original credibility model proposed in  [27]  is described. Section V presents the addition of bot detection in the extended credibility model. Finally, Section VI discuss the main results and the future work.
A. DETECTION OF BOTS ON TWITTER Bot detection in social networks is a topic that has been researched and still continues to be a challenge. Some reviews of the most recent techniques that have been published analyse the different classifiers used, the type of dataset used, and the features selected, as well as presenting comparisons among the evaluation techniques used for their validation  [22] ,  [30] -  [32]. Alothali et al. [22]  identify three common methods, based on graph, crowdsourcing, and machine learning, for social bot detection on social networks. For the machine learning algorithms, authors select 20 features of content, profile, and behavior from accounts on social networks. Orabi et al. [30]  elaborate a bibliographic review of published works between 2010 and 2019 resulting in a taxonomy of detection methods and techniques used to detect bots in social media and a comparison among them.
Aljabri et al. [31]  survey existing studies from 2015 to 2022, which apply supervised, semi-supervised, and unsupervised machine learning techniques for social media bot detection. Authors consider the detection of social bots, spambots, and sybil bots on Facebook, Instagram, LinkedIn, Twitter, and Weibo, and provide a novel taxonomy of social media bot detection using machine learning based techniques. Moreover, they present an analysis of the most commonly extracted features and datasets used on each social media platform. Beatson et al. [32]  compare a semi-automated method based on fixed criteria against a manual model based on open source intelligence (OSINT) to detect bots in Twitter. The evaluation is performed in terms of accuracy, user friendliness, efficiency or time costs, and simplicity of operation.
The accuracy of their classification model reaches 96% for binary prediction (bots and humans). Alarifi et al. [33]  propose a supervised classifier considering bots, hybrids, and human users as the three prediction categories. Authors performed data acquisition through Twitter API REST and Twitter Streaming API collecting information about 1. 8 million accounts. They then performed labeling of the data by a selected group of people trained in classification. After the tests performed, they obtained 96% of accuracy and a Kappa coefficient at 0. 61, resulting in a good percentage of agreement among the labeling group. They use four learning algorithms for training: Random Forest, BayesNet, Support Vector Machine (SVM), and multilayer artificial neural network. After the tests carried out, the predictive models obtained by Random Forest and BayesNet proved to have an accuracy percentage of 91%.
Kudugunta et al. [34]  describe a work using a neural network based on contextual short-term memory (LSTM), where both tweet content and metadata are considered. Social features of the user are also used, taking into account only the detection of bots at the tweet level, obtaining an accuracy of 96%. Barbon et al. [35]  propose an algorithm based on Discrete Wavelet Transform to obtain a pattern of writing style embedded in post contents in order to classify authors/users as being a human, a legitimate robot, or a malicious robot. Authors trained a classification model using the Random Forest algorithm over two real Twitter datasets (Super Bowl XLVIII event and Seatle Seahawks) and their model achieved an average accuracy of 94. 47%. Hayawi et al. [36]  propose a deep neural network designed with LSTM units and dense layers to classify human or bot accounts in Twitter.
The classification is based on user profile metadata, including username, screen name, tweet count, followers count, friends count, listed count, user created date, description, location, url, verified flag, obtained through the Twitter API. The proposed model is evaluated on several datasets using two manners: testing on a hold-out set of the same dataset and training with one dataset and testing with a different one. Results show that the proposed model achieves an area under curve (AUC) as high as 0. 97 with a selected set of features on the hold-out test set and 93% F1-score with a threshold=0,34. Shukla et al. [37]  propose a multi-layer condition-based model to identify fraudulent bots in Twitter. The proposed method analyzes Twitter-specific user profiles considering profile name, description, location, listed count, verification status, that are recovered with the Twitter API, and also compare the string-based features with a bag of words containing tokens that are likely to be prevalent in bots.
Authors performed a comparative evaluation of the proposed model with existing benchmark classifiers, such as SVN, Categorical Naïve Bayes, Bernoulli Naïve Bayes, Multilayer Perceptron, Decision Trees, Random Forest, and other automation identifiers, as well as with publicly available standard automation identifier API, such as Botometer, BotSentinel, and TweetBotOrNot. The Kaggle dataset is used for the experimentation, obtaining an accuracy of 99. 0049% and a testing AUC of 0. 996606. Verma et al. [38]  propose UCred, a model aimed at classifying user accounts as fake or real. UCred model separates text features of the user profile (e. g. , description) from numeric features (e. g. , followers, friends, verified flag) to be evaluated simultaneously by several machine learning and deep learning methods.
Text features are evaluated by post-trained model (i. e. , CNN, LSTM, BiLSTM) and pretrained methods (i. e. , BERT, RoBERT, DistilBERT), while numeric features are evaluated with Naïve Bayes, SVN, Random Forest, and K-Nearest Neighbor (KNN). Outputs from all these methods are submitted to a voting classifier to finally decide if the account is real or fake. To evaluate the performance, authors performed experiments on OSN dataset, which contains 1337 fake and 1481 real profiles. Results show that UCred reaches up to 98,96% of accuracy and 98,37% of F1-score. Currently, there exist some automatic tools for bot detection in tweets in English, such as Debot  [39] , Tweet-BotOrNot  [40] , Bot Sentinel  [41] , and Botometer 2   [42]  Debot  [39]  is a system developed at the University of New Mexico which performs a search for correlated accounts by making use of deformed correlation (correlation extended by dynamic time warping) when analyzing their activity, given the idea that humans cannot be highly synchronous over long periods of time.
The approach used in this system corresponds to an unsupervised method, not needing a labeled dataset to train a model beforehand. Debot works as a near real-time system, and has an accuracy of 94%, in addition to having an API that allows using it in external projects. TweetBotOrNot  [40]  is an R package developed by Michael W. Kearney in 2018. It is based on the gradient boosted supervised model to obtain the probability of a Twitter account being a bot. To do so, the model uses user-level metadata, tweet-level, and tweet text, including friends, followers, posts, account time creation, retweets. The classifier was trained on 10,000 Twitter accounts with over 7000 bots, from which TweetBotOrNot shows an accuracy of 93. 8%. Bot Sentinel  [41]  is another Twitter analytics free tool founded in 2018 by Christopher Bouzy.
This service requires Twitter authentication and authorization in order to use. Both Botometer and Twitter API have rate limits, meaning that users can only make a certain number of queries in a given time period. Botometer has a website and API endpoints with similar functionality. Using the Botometer API requires keys associated with a Twitter app, when querying the API, users are responsible to send the required data (i. e. , 200 most recent tweets by the account being checked and tweets mentioning this account) in a specified format through HTTPS requests. All these works demonstrate the interest and feasibility of bot detection on Twitter. While most of these works report good results after testing, they do not fully match our requirements, regarding the consideration of English and Spanish and a prediction performance in real time, for the development of the functionality in our previous credibility model.
Additionally, the datasets used are not available for general use; thus, a separate or further retraining is not possible. In the same way, although Alarifi et al. [33]  perform supervised training at the end of their data collection process and design and implement a browser plug-in, none of these resources are available or enabled for use or incorporation into another project. Debot, TweetBotOrNot, and Bot Sentinel are still working today, but the teams in charge of those projects do not provide more keys for the use of their API; while Botometer has the same limitations of use as the Twitter API. Table  1  summarizes the points discussed, considering also the number of attributes used by each work as a reference. The interest of this work is to be able to detect whether the user of the analyzed Twitter account is a bot or not, leaving the possibility of using additional analysis to subsequently infer whether the bot in question is a useful or dangerous bot.
B. CREDIBILITY ANALYSIS CONSIDERING THE ACCOUNTS/USERS Concerning on how to do the credibility analysis, the existing works consider the analysis of some types of information to calculate different credibility measures  [27] ,  [43] ,  [44]. This information mainly includes four dimensions or levels associated to text, user or account, topic, and social influence. Text dimension measures the level of relevance and accuracy of the text, independent of the referenced topic or with respect to a certain topic. It is mainly calculated through text analysis techniques. User dimension calculates the user account credibility based on attributes that describe it (e. g. , the account creation date, if the account is verified, user gender and age). Social dimension calculates the credibility of a publication, related or not to a topic, based on the available metadata that describe the social impact of the user account and the post itself, with respect to other users.
In this instantiated model authors calculate the credibility of a user considering whether the account is verified or not and the activity time of the account since its creation. Saputri et al. [51]  propose a method based on information gain and KNN to detect fake news related to COVID-19. The model is trained with a dataset built by authors and manually labeled as credible or not credible. Using the Twitter API, authors collected tweet with the hashtag #covid19, along with the users' profile, consisting on username, location, user description, account creation time, number of followers, as well as user verification, from which the author's credibility is determined and the manual labeling is done. From the text of tweets, the information gain is computed based on the TF-IDF method. Thus, from the user's credibility and the information gain, the KNN is able to identify tweets related to COVID-19 as fake news or valid information.
III. EVALUATION OF METHODS FOR BOT DETECTION IN TWITTER For the evaluation of algorithms for bot detection, we follow the methodology shown in Figure  1 , which includes steps from data collection and cleaning, to training of the predictive models and their evaluation. This methodology is inspired on the classical methodology for Knowledge Discovery in Databases (KDD), which proposes five phases for data processing (i. e. , data selection, preprocessing, transformation, data mining, and interpretation/evaluation). In this work, we correspond these phases such as database configuration (data selection), preproccesing (preprocessing and transformation), training phase (data mining), and testing and analysis (interpretation/evaluation), as Figure  1  illustrates. In the following, we detail each phase.
We further aggregate data on the text or content of the tweet, considering structural features as in  [34]. We have also considered a new feature based on the Discrete Wavelet Transform (DWT)  [35] , which have been used for text analysis. The features selected and used in this work for the construction of the training datasets and subsequent use in the training process include: The list of features corresponds to social features or quantitative information from the text itself. Most features can be retrieved from the Twitter account itself using the Twitter API. The features 3) Favourites_count, 6) Retweet_count, 8) Num_URLs, 9) Num_hashtags, and 10) Num_mentions were also used by Kudugunta et al. [34] , while features 1) Followers_count, 2) Favorite_count, 4) Listed_count, and 5) Friends_count were used by Alarifi et al.
2) Data extraction Although there are a variety of works that offer the possibility of requesting the datasets used for their studies based on Twitter, most of them only have the id of the labeled accounts and not their respective tweets. Moreover, we found that some of these datasets are obsolete with deleted accounts or they are now totally different from the period in which they were collected  [54]. Other authors have created their own datasets using the Twitter API, but they are not reported as public  [55]. The task of making a full dataset containing tweets and data related to them demands many resources, such as the labeling process and the data gathering from Twitter, where the API offers limitations for free accounts. For instance, authors in  [56]   Other studies, such as the one presented in  [57] , focus on the prediction models and use public datasets proposed by other works  [29] ,  [58].
In this work, we requested and used the datasets collected by authors in  [29]. These datasets contain different user accounts and their tweets in a variety of languages, such as English, Spanish, French, Japanese. No additional requests to the Twitter API were required. Table  3  shows the total number of user accounts contained in the different datasets (genuine, social spambots, traditional spambots, and fake followers), the number of tweets associated to them, and the year which represents the average of the creation years of the accounts that belong to the dataset. This table is a simplified version of the original table presented in  [29].
B. PREPROCESSING After obtaining the datasets (Table  3 ), an initial review showed that they contain several encoding errors (accents, punctual letters such as 'ñ' or capital letters), emojis, and characters that do not contribute to the content of the text of the tweet. Additionally, the text associated to tweets are in a variety of languages, such as English, Spanish, French, Japanese. For these reasons, the preprocessing to prepare the data for training involves data cleaning and data filter tasks, such as: • Fix encoding errors: The Python library ftfy is used to correct text encoding problems in tweets. This consideration is necessary mainly when preprocessing the information that would compose the training dataset in Spanish, since several characters had encoding problems, such as accents (á,é,í,ó, and ú), umlauts, or the letter 'ñ'.
• Feature scaling: Due to different feature scales, they are standardized by removing the mean and scaling to unit variance. For this, features are scaled using the Scikit-learn library StandardScaler. This function calculates the standard score of a sample x as: z = (x -u)/s, where u is the mean and s is the standard deviation of the training samples. • Filter by text language: A function is used to identify the language in which the tweet was written with the Python library langid; thus, it is possible to make a quick classification of texts written in English or Spanish tagging them as 'en' or 'es', respectively. Once the preprocessing is finished, the relevant records are collected between the group of genuine (human) accounts and the rest of the bot groups (social spambots #1, #2, and #3, traditional spambots #1 and #2), obtaining two datasets.
B. 1. Feature selection methods In the interest of reducing the high dimensionality of datasets and to provide more parsimonious models, different methods of feature selection have been studied and classified into three groups: filters, wrappers, and embedded  [59]. However, no single selection algorithm into these groups seems to be capable of ensuring optimal results in terms of predictive performance, robustness, stability, and interpretability. In this sense, we have explored two feature selection methods which have been used successfully in different domains. The first one corresponds to an embedded method, feature importance in Random Forest  [60] -  [63]. The second one considered is the SHapley Additive exPlanations (SHAP) method, which help understanding what are the main features that affect the output of the model  [64] , and is also being used as a feature selection mechanism  [65] -  [67].
1) Feature importance in Random Forest As an interesting result derived from the importance of the features in the training process of the data with the Random Forest, some features with higher variance were involved in the classification process (i. e. , favourites_count, statuses_count, followers_count, friends_count, and listed_count). Figure  2  and Figure  3  show these best features. The results correspond to the permutation feature importance of the Random Forest  3. This is a model inspection approach that can be used to any fitted estimator and is especially helpful for estimators that are non-linear or opaque. The permutation feature significance is defined as the reduction in model score caused by randomly rearranging a single feature value. The technique destroys the link between the feature and the goal, and the decrease in model score reflects how much the feature is relied upon.
This method has the advantage of being model independent and can be calculated numerous times with various permutations of the characteristic  [68]. The disadvantages of impurity-based feature importance are overcome by permutation feature importance since it does not favor high-cardinality features and may be computed on a test set with missing data. 2) SHAP method SHAP (SHapley Additive exPlanations), a unified framework for interpreting predictions, assigns each feature an importance value for a particular prediction. This method has also been adopted to provide a reasonable interpretable model  [64]. The SHAP value method has been used in a number of cases to calculate the importance of features in  machine learning models, so as to get an intuitive explanation  [69]. Figure  4  and Figure  5  show the SHAP values on our two datasets.
In furtherance of testing the impact of applying a feature selection method, we have reduced the number of features to 7 and 5 for Spanish and English datasets, respectively. Afterwards, we have retrained our classifiers, resulting Random Forest (as classifier) with the best metrics. Selected features correspond to an importance score greater than 1 in Table  5. Table  6  shows metrics resulting from the prediction on the testing sets using Random Forest. We can observe that the metrics are good with values over 90% and the selected features are associated to social features. We could infer from these results that these features are independent of the language and the tweet content (note that the associated features to language and content were not selected by the feature selection methods).
1) The experimental conditions In this work, we deal with a supervised learning problem with a binary classification task considering two classes: "bot" and "human". In this context, there exist several machine learning algorithms that have shown good results for bot detection  [70] ,  [71]. We have considered some of the most popular and relevant: • AdaBoost classifier: It is a meta-estimator that starts by fitting a classifier to the initial dataset and then fits additional copies of the classifier on the same dataset with the weights of instances that were mistakenly categorized. This is done such that future classifiers concentrate more on challenging situations  [72] ,  [73]. • Bagging classifier: It is an ensemble meta-estimator that applies basic classifiers to different random subsets of the initial dataset, and then combines each prediction (either by voting or by averaging) to get the final prediction.
The SVM is a popular algorithm due to its high generalization capability together with its ability to find global and non-linear classification solutions. In this work, we use Support Vector Classification (SVC), the scikit-learn implementation of this method for classification purposes. • Multi-layer Perceptron (MLP): MLP is a supervised learning algorithm that learns a function by training on a dataset. Given a set of features or input of m dimensions and a target of o dimension, it can learn a nonlinear function approximator for either classification or regression  [79]. To determine the best parameters, each algorithm was trained several times by exploring different parameter values using a GridSearchCV method  4. This method exhaustively considers all parameter combinations over specified parameter values for each classifier.