Approximating Naive Bayes on Unlabelled Categorical Data
Authors: Cormac Herley
Venue: Accepted by TMLR ()

Abstract: We address the question of binary classification when no labels are available and the input features are categorical. The lack of labels means supervised approaches can't be used, and the lack of a natural distance measure means that most unsupervised methods do poorly. For such problems, where the alternatives might be a) do nothing or b)  heuristic rules-based approaches, we offer a third alternative: a classifier that approximates Naive Bayes. Our primary scenarios are those that involve distinguishing scripted, or bot, web traffic from that of legitimate users.

Our main assumption is the existence of some attribute $x_*$ more prevalent in the benign than the scripted traffic; i.e., $P(x_*|\overline{\mbox{bot}}) = K \cdot P(x_*|\mbox{bot}),$ for $K>1.$ We show that any such disparity yields a lower bound on $P(\mbox{bot}|x_{j})$ even when we have no prior estimates of $P(x_*|\overline{\mbox{bot}}),$  $P(x_*|\mbox{bot})$ or $K$ (except that $K>1$). We show that when at least one bin of at least one feature receives no attack traffic then we under-estimate the actual conditional probability by a factor of $1-1/K.$ Thus, any attribute with a large disparity between prevalence in benign and abuse traffic (i.e., $K$ is large), allows good approximation of the Naive Bayes classifier without the benefit of labels.
 
The approach is particularly suited to problems where $K$ is high and thus the approximation is very accurate. Example problems (and relevant attributes) might be: password-guessing, if login attempts from legitimate users succeed at a much higher rate than those from password-guessing attackers; Credit Card Verification Value (CVV) guessing, if an attacker exhaustively tries all possible 3 or 4-digit values and fails at a higher rate than legitimate users; account registration, if legitimate users  use email addresses from services that do not allow fee anonymous accounts (e.g., {\tt .edu})  at a much higher rate than attackers; click-fraud if legitimate users  visit pages and services that contain no ads at a higher rate than click-fraud bots.

Published in Transactions on Machine Learning Research (09/2023)
Approximating Naive Bayes on Unlabelled Categorical Data
Cormac Herley Microsoft Research Redmond, WA
Reviewed on OpenReview: //pp::///peerevviee/ net/ forum? id= KpE22SSpp
Abstract
We address the question of binary classification when no labels are available and the input features are categorical. The lack labels means supervised approaches can't used, and the lack of natural distance mcasure means that most unsupervised methods do poorly. For such problems, where the alternatives might be a) do nothing or b) heuristic rules-based approaches, we offer third alternative: classifier that approximates Naive Bayes. Our primary scenarios are those that involve distinguishing scripted, bot, web traffic from that of legitimate users.
Our main assumption is the existence of some attribute more prevalent in the benign than the scripted traffic; i..., Plt,/bot) = K P(x,/bot), for such disparity yields a lower bound on P(bot(a,) even when we have no prior estimates of P(., (bo)), P(x=, (bot) or K (excopt that K > 1). We show that when least one bin of at least one feature receives no allack traflic then we under-estimate the actual conditional probability by factor of 1 - 1/K. Thus, any attribute with large disparity between prevalence in benign and abuse traffic (i.c., K is large), allows good approximation of the Naive Bayes classifier without the benefit of labels.
We show that any
The approach is particularly suited to problems where K is high and thus the approximation very accurate. Example problems (and relevant attributes) might be: haasword-guessingg login attempts from legitimate users succeed at a much higher rate than those from password-guessing attackers; Credit Card Verification Value (CVV) guessing, if an attacker exhaustively tries all possible 3 4-digit values and fails at higher rate than legitimate users; account registration, if legitimate users email addresses from services that do not allow fee anonymous accounts (c.g., edu) at much higher rate than attackers; click-fraud if Imgetimate users visit, pages and services that contain no ads at higher rate than click-fraud bots.
1 Introduction
Abuse of web services often involves automated traffic. This includes registration abuse (i.e., scripting the account creation process), unruth scraping, click-fraud, password guessing, fake engagement on social networking sites and so on. This traffic can be hard to detect. Supervised approaches can't be used since labels are almost never available (Chio & Freeman 2018). Anomaly detection and one-class learners that assume outlicrs are "few and rare" do not work well when abuse can range from 5% to 90% of traffic Chandola et al., 2200)). Most unsupervised methods assune that closeness in feature space implies greater likelihood of belonging to the same class. Since have categorical features there is natural idea of distance between samples; thus, unsupervised approaches like K means, One-Class Support Vector Machines (SVMs) [Schölkopf et al. 1999), ; and Isolation Forests (Liu et al. 2008) are not easily applied. Techniques that map categorical features to numeric (such as one-hot encoding) offer limited help when we have a small number of features (Zheng Casari 2018).
The scale of these abuse problems can be considerable. Twitter suspended 70 million fake accounts in June 2018. In April 2021, the information of 533 million Facebook and 500 million LinkedIn users were gathered by

---

Published in Transactions on Machine Learning Research (09/2023)
unauthorized scraping. Akamai reports that 43% of login attempts on their platform in 2017 were password- guessing attempts (Akamai). Varol et al. (Varol et al. 2017 estimate that at least 15% of Twitter's 's active accounts are bot-created fakes.
In this paper we show that if we can identify a binary feature 2s which is more prevalent in the benign traffic than in the abuse we can approximate Naive Bayes (NB) classifier. That is, if P(x. (ot_bott) = K P(x, for K > 1 we can approximate P(bot|zz) in any feature zj. The approximation is order-preserving: sorting list of samples by our estimates will have the same order the true NB estimates. Further, when K is large, the approximation is very accurate.
Examples of possible problems where the technique might be worth considering are: password-guessingg if login atlempls from legitimate users succeed al a much higher rale than those from password-guesst attackers; account registration, if legitimate users use email addresses from services that do not offer anony- mous free accounts a much higher rate than attackers; social networking, if the fraction of legitimate users for whom the follower/followed ratio is large to greatly exceed that for bot accounts.
For web-services the fields available will be restricted to those gathered from browser. A very common list of features will be those listed in the Apache web-server Combined Log Format (CLF) (Apache 2019). An example logged request might be:
127.0.0.1 [10/0ct/2000: 36 -0700] "GET /apache_pp _rtp__pbiit.. HTTP/1. O" 200 2326 "http://www exwmple. www.eeee....ettaa.eereeeee..aaae iectaattttmit html" "Mozilla/4. 08 [en] (Win98; I ;Nav)
These recorded features are: Client TP address, Timestamp, Request, HTTP response, Bytes Requested, Referer, Useragent. Note that, with the exception of Bytes Roquested, all of the features are categorical ordinal. Further note that we effectively have five or six features.
2 Related work
2.1 Unsupervised and weakly supervised learning
Most unsupervised learning methods assume that points that are close in feature space are more likely to belong to the same class (Hastie et al., (000).. This is true of clustering methods (e.g.. K means), Principal Components Analysis, Tsolation Forests (Liu et al. 2008) and One-Class SVMs (Schölkopf et al., 1999). Similarly, survey of anomaly detection approaches divides them into classification (i.c., supervised), neirest-, clustering and statistically based methods (Chandola et al.) 2009); all of these require a reliable measure of distance. When we deal with categorical inputs some modification of the input space must be porformed to generate numeric features.
A common way of accomplishing this is with one-hot encoding, where feat Fi, T2, ... Fd of cardinalities R1, Rg..... Ra will be mapped into numeric features each of which take the value zero or one. This mapping now allows us to use the unsupervised and anomaly detection techniques mentioned above. However, the distance in the new feature space is very constrained: the distance between any two points must be one of the discrete values 0, 1.... d. Unfortunately, this severely limits the efficacy of settance-based unsupervised algs This is particularly true when d is small (as is the case in our web-services scenario where d 5). It is easy to see that performance of an algorithm like K means will not perform well when there are only d 1 6 possible distances between points. further difficulty in applying One-Class SVMs is that the algorithm requires a parameter a which represents the fraction of samples that are expected to be anomalous. In anomaly detection applications we might expect this to be very small (e.g., v < 0.01) but in fraud and abuse problems we expect it to be much higher (e.g., 0.05 < 0.9). In addition, the amount of fraud might vary quite rapidly as attackers gain or lose access to resources or alter their strategies. The need to provide the One-Class SVM algorithm with prior is a significant shortcoming of the approach.
Our work belongs to the growing body of work on weak supervision. While most of the work on weak supervision involves noisy or unreliable labels, there are interesting variations. Parisi et al. show how rank several different classifiers without the benefit of labels (Paris et al. 2014). A similar weak supervision signal to ours is identified by Soman et al. (Soman al. 2022 They are also interested in the classification
2

---

Published in Transactions on Machine Learning Research (09/2023)
problem and identify markers that are more prevalent in one class than the other. They use this to evaluate the relative performance of competing models rather than to build a NB approximation as we do. Joyce et al. (Joyce et al. 2021 describe how some performance metrics for clustering and multi-class classification algorithms can be computed in the absence of ground-truth reference labels. The approach leverages prior domain knowledge and works best when the number of classes is large and grouping similar data points is casier than determining ground-truth labels. An ingenious approach to classification without labels is described in (Du Plessis et al. 2013) and developed further in (Kaji & Sugiyama 2019). They show how the classification boundary between Co and C1 may be obtained if we have two unlabelled collections of unknown mixtures of the two classes. While it builds classifier this approach does not estimate the class membership probabilities Pz) and P(C, Hence (unlike our method) a ranking of samples from most to least, bot-like isn't available; this can be problematic for abuse applications where sensitivity to false positives often means that we act only when P(bot(ar) is very high.
2.2 Detection of automated abuse
Despite the scale and importance of automated abuse it is unclear that Machine Learning (ML) has had significant impact. In fact, Sommer and Paxson describe the limitations of ML methods for socurity problems (Sommer& & Paxson 2010) observing that the absence of labels and available datasets are serious impediments. A more recent account of using MT to protect consumer properties is by Chio and Freeman (Chio & Freeman, 2018). Many of these problems are inherently unsupervised. This is not simply a question of cost or resources; even expert human labellers will have no basis to label individual web requests as malicious or benign. In fact, many abuse problems appear still addressed using hand-crafted rules and heuristics. They involve categorical variables, making direct use of one-class algorithms such as One-class SVM's (Schölkopf et al. 1999) and Isolation Forests (Liu et al. 2008) difficult.
TP reputation has often been suggested blocklist solutions might capture much of the abuse with minimal impact to the benign traffic (Stringhini et al. 2015 Unfortunately, in a recent large-scale honeypot study Li et al. find that 64.37% of bot IP addresses lie in the residential IP space (Li et al. 2021]. It's obviously not feasible for web-scale consumer platforms such as Facebook, Google, Amazon, Pinterest, Twitter etc to block huge portions of the consumer IP space. CAPTCHA's (Ahn et al. 2003) to tell humans and computers apart have been widely deployed but sulfer from a number of difficulties. They place a considerable burden on users and have been successfully broken by automated solvers (Bursztein et al. 2014).
solution. When abuse traffic comes from a small TP space
describe DNN approach to social bot detection. They use synthetic minority oversampling (Chawla et al. 2002) which generates large amount of hettc labelled data from a smaller number of actual labels. Jan et al (Jan et al. (020) also leverage small amount of labelled data to synthosize quantitics that can be used by neural networks. They echo some of findings of Sommer and Paxson (Sommer & Paxson 2010) on the limitations of ML methods. When pairs of features that are independent the benign traffic can be identified, deviations from rank-one in the associated pivot matrices can be used to classify abuse traffic (Herley 2022).
Kudugunta and Ferrara (Kudugunta & Ferrara 2018
3 Target Encoding: Replace tokens with conditional probabilities
We assumee that we receive unknown mixture of benign and malicious traffic:
P(x) = [1 P(bot)] P(x bot) + P(bot) P(z[bot),
(1)
where x is our vector of observed features.
We wish to build a classifier that distinguishes bot from bot traffic. An initial difficulty is that many MI. techniques take numeric features as input, and thus categorical features must first be transformed. We
discussed the significant problems of approaches such as one-hot encoding in Section Bilenko 2015 Zheng
& Casari 2018).
In supervised settings one way around this problem is to replace each category token with the conditional probability of the target class under the token value. That is (if Zykk is the k-th token of the j-th feature of
3

---

Published in Transactions on Machine Learning Research (09/2023)
x) replace 25 with P(bot|z, Tyk). When we have labels this can be done by simply counting the number of positive and negative instances (N+ and N respectively) in the 2ikk bin over a training window:
N
P(bot|æ, = zzx))=
N+ N++N-
When we apply to many features this is, of course, a component of the Naive Bayes classifier the condi- tional probability assuming per-class independence of features. The trick of using this as input to a more sophisticated classifier appears have been discovered and rediscovered many times (Chen al. 2009 Li
let al., 2010 Lee et al. 1998). Good treatments are given by Bilenko [Blenkoo] 2015) and by Zheng and Casari (Zheng & Casari 2018 Zheng and Casari point out that this transformation of the categorical one-hot encoding, such as being computationally more efficient, scaling to higher cardinality features, and working well with non-linear models.
variables has several advantages
Our first observation is that labels entered into the calculation of the conditional probability only via their aggregates N+ and N That is, if we are content with a Naive Bayes classifier we don't need the individual labels, just a way to estimate the ratio of the sum of their counts. If we had a mechanism estimate P(bot)a4z)) N+ N+N(N+ + N)) directly we would not need labels at all.
4 Lower bound conditional probabilities
Suppose have binary feature, Zx. such that P(xx > P(xx/bot); that is, x. has value True higher rate in the benign than in the abuse traffic (although we don't know P(a../bot) or P(x.//bot) | Consider the conditional probability of x. conditioned on Ijk. From I we get:
P(z,(2z)) [1 P(bot(223)) P(x./bot, 2zx) + P(bot(zza) (bt,,
Hence:
P(bot(azzk) = P(x, (bo,,bot, x,n) P ((r.(1)))
P(x. bot, xzn)) P(x. |bot zzx))
If we had labels could estimate this directly, but without them cannot. That is, while we can estimate P(r.(19)) by counting positive and negative instances of x,, we don't have a way of unravelling the values for P(rx/bot, zz4) and P(z,|bot,I,j
However, since we are interested in the Naive Bayes estimate we can simplify by imposing per-class in- dependence among features. This gives P(r./bot, P(x. (bot) and Plx.|bot, = P(x./bbot) and hence:
P(x.(az)) [1 P(bot)az))) P(x. bot) P(bot(2z3) (bot).
(2)
This simplifies our expression for the conditional probability to
P(bot|az5k) = P(x, ((t,)bot) - P(x., [z,n)
(3)
P(x, (bt) P(x, [bot)*
Also, observe that 2 says that ((x)) is a probability-weighted sum of P(x,, bot) and P(r./bot)) so it must between them. Since we know that P(x,, (bot) P(x,, (bot) we get the following ordering:
Plx,|bot) 2 (( 2 P (xx|bot).
(4)
We wish to be conservative in flagging traffic malicious; hence prefer to under- rather than over- estimate P(bot)/zzx) Thus, our main interest is in lower bound. Obviously, we can get a lower bound on 3 by decreasing the numeralor and/or increasing the denominator. Consider the denominator first. Since
implies P(x.|bot) > P(xx|bot) - (oot/bot)
bot) we get:
P(bot(a3h) 2 P(x.(bol)) - P(r. (13z)
(ot.
While v. input feature we denote separately from the other input features; Le.,, vn $ f01, "2.,,,. .aa).
4

---

Published in Transactions on Machine Learning Research (09/2023)
This still leaves us with P(xx (bot) to estimate bound. Again we invoke (4:: the observed distribution
P(x. lzgk) is lower than or equal to P(x.|bot), never higher. Thus maxj. P(r.//3j is lower bound for
P(z. (oo).. So we define:
P(x. bot)
max P(z,((z)) & P(r, bot).
(5)
J.
Here the maximization is over all bins of all features.
This gives our overall lower bound for (15o)
P(bot(azH) P(x. (z5k) S P(bot|/zja).
(6)
P(x. (bo)
That is, the left-hand side gives a lower bound for the Naive Bayes estimate in terms only of quantities observe.
A similar analysis gives lower bound on the fraction traffic that is from bots:
P(r.)
P(bot) &
S P(bot).
(7)
P(x, bot)
To build our overall classifier, observe that:
P(bot).
P(bot at)
P(æ:/bot)
P(x)
P(bot)
P(bot|(x:) P(ax).
P(x)
P(bot)
Replacing P(bot) and P(bot|xz) with their estimates, we get:
P(bot|x) A P(bot) P(x)
P(bot|a:)) P(x.) P(bot)
II
The NB estimator decides for bot if P(bot|æ) > ((bot(æ), and for bot otherwise. If we wish to trade true and false positives at some specific rate a threshold is easily incorporated the decision.
It worth noting, from (2), that least one bin of at least one feature unattacked (i.c., P(bot|azzk) for some x then P(z.|boi) = P(r.)bol). This is a reasonable expectation unless abuse is spread to all bins of all features. For example, if no abuse comes from a common browser like Opera, or a geographic region like Wisconsin, then our estimate of P(x. bot) (5) is exact.
4.1 Tightness of the bound
We assumed that P(x_|bot). > P(xx|bott). Let's call their ratio K:
(8) Here K simply captures the factor difference in P(x.) between benign and abuse traffic (e.g., K 1.1 then P(x.) is somewhat higher in the benign traffic, if K = 100 it is greatly so). We emphasize that we don't assume that we know (or have good prior estimates of) P(x. P(x./bot,ax Izk), P(c. Pbo,/bot,ar ajk) or K.
P(r. bot) = K P(r./bot).
Comparing 3 and
we note that our bound differs from the actual Naive Bayes estimate by a multiplica-
tive factor:
P(t./bot) P(x. (Tg)) P(x. bot) P(x,/bot) P(x,/bot)
P(x, [oo)) P(.. (ryzk)
If we assume that at least one bin of at least one feature receives no attack traffic (i.e., ((bot(z,x) = 0 for some 2z4) then P(zx|bot) = P(æ,|bot) and the amount by which we under-estimate becomes
P P(x, ((x.(bot) P(x, (bot) - 1/K.
P(z.
5

---

Published in Transactions on Machine Learning Research (09/2023)
That is, our estimated conditional probability is related to the actual value by:
p(bot)azx) = (1-11)10 1/K) P(bot[zjk)
(9)
Thus, when K is large get very accurate approximation of the Naive Bayes estimate. A similar analysis yields:
P(bot) = (1 - 1/K) P(bot).
(10)
If an unattacked bin does not exist the bound is still quite tight. To this, suppose that, in the least-
allacked bin, 5% of trallic is abuse. From ( and ( this gives P(.., (bol) = (0.95 + 0.05/K) P(x.|bot).
Hence we differ from the actual Naive Bayes estimate by multiplicative factor:
(0.95 + 0.05/K) K) ((r., (bol) ((rkalazk))
P(r. (bot) P( |bot)
P(z, bot) P(Zx/ITzH
(0.95 + 0.05 K) P(x bot)
The second term is (1 1/K) + (K)5/K), which ranges between (1 (1/K)/0.90 and (1 - 1/K) K ranges 1 + co. The first. term is close to 1 when P(r,, [bot) - P(z.11za is large; i.e, when the received distribution differs considerably from clean. Thus, when we do not have unattacked bins, but the volume of abuse is high, we still underestimate the Naive Bayes by so (1 (1-14 1-11/ /) so long as at least one bin of at least one feature sees little abuse traffic.
4.2 Preservation of order
Consider now the overall estimate P(bot|æ)). We next show that if P(bot(#) > P(bot|æ') then P(bot)æ:) > P(bot|z') (again assuming at least one bin of at least one feature receives no attack traffic).
Observe:
P(bot)
P(bot|æ)
((ootbot)
P(x)
= P(bot)
d P(bot)x:)) P(a,)
P(x)
P(bot)
i=1
(1 1/K)
P(bot) L/bot)((ot)
(1 - 1/K) (((xt)
P(w)
(1- 1/K) P(bot)
i=1
1/K) P(bot)
P(bot|x;). P((x))
P(x)
P(bot)
(1 1/K) P(bot|æ))
This is true independently of K (so long as K > 1). Thus (when we have unattacked bins) a ranking under our estimate ((bot(1y)) has the same order as a ranking under the true value P(bot|Jzy)) even if we are wrong about the exact values. Significantly this means that the Receiver Operator Characteristic (ROC) curve that method produces will be identical to that of the true Naive Bayes algorithm. We can use this to order browser versions, geographic regions, IP blocks, etc, by the fraction of abuse tralfic they contain.
The above analysis considers the behavior in the limit of arbitrarily large amounts of data. In practice of course, we must consider the effects of sampling. Our mothod behaves similarly to NB in this respect. That is. the true NB estimate is the ratio N W+/(N++* + counts of the labels. The two terms in our estimate (6p (i.c., P(x.) and P(x. bot)) are similarly found as ratios of counts, not of the labels but of x.. Naturally, confidence intervals should be considered in the maxinization calculation of (
4.3 Claims
For convenience we summarize which conclusions depend on which assumptions. Since we seek a Naive Bayes classifier we assume per-class independence of features. Tf, for some v., P(r. (bot) > P(r.(bot) then

---

Published in Transactions on Machine Learning Research (09/2023)
6 gives lower bound on the conditional probability P(bot(az)) If, in addition, at least bin of least one of the features in receives no attack traffic then P(bot|aya) P(bot|zyx). When this is true we underestinate ((bot(az)) by factor of (1-11/ 1/K), ranking of group of bins under our estimate will be the same as ranking under P(bot|zz3)) and the ROC curve has the same shape that of true NB.
4.4 Notes and Limitations
Whether least one bin of at least one feature is free of attack traffic will obviously be dataset-dependent. However, if any of the features has high cardinality the assumption will be satisfied unless every bin contains attack traffic. Web traffic, for example, may be grouped by city, region, IP subnet or Autonomous System Number (ASN). If any of these features has thousands of bins, the assumption will be satisfied unless allack traffic comes from every city, region, ASN, etc.
Our analysis neglects sampling and finite-data effects. In practice, confidence intervals for all quantitics should be calculated, and these may have significant effect on the quality of the approxination, even when the main assumptions are met. The approximation of NB will thus be far better for large datasets than for small ones.
5
Toy example: account registration
Consider the case of account registration at web service. Suppose (unrealistically) that benign traffic has equal volumes from all 50 states. Suppose most of the abuse traflic comes from 10 states where it makes up 30% of traffic, but there's also smaller amounts in the other 40 states where it makes up 1% of traffic.
On signup users are asked provide an email account. Popular free domains like gmail, hotmail and outlook obviously dominate. However, small fraction of legitimate users provide email addresses from paid service like AOL Comcast; c.g.; P(AOL/bot). 0.03. If attackers concentrate on free domains we might expect P(AOL|bot) > PotOL|bot). Thus, we use AOL our discriminative feature For definiteness suppose P(AOL|bot) = 0.002 (although, of course, this information isn't available to our estimator). Observe that:
P(AOL]state)
[1 P(bot|state)] P(AOL|bot) P(bot|staate) P(AOL
[1 (((boo/state) 0.03 P(bot|staae) 0.002.
In the most-attacked states abuse 30% volume, so P(AOL|statt) 0.7 0.03 + 0.3 x 0.002 0.0216 while the 40 less-attacked states P(AOL]state) = 0.99 x 0.03 + 0.01 x 0.002 0.0297.
Our estimate P(oo in will come from the least-attacked states: P(AOL|bot maxgtate ((AOLestate) 0.0297. Thus, from 99) in the states that receive most attack traffic estimate is - 0.021660.00220 = 0.273. In the states that receive little attack traffic our estimate is 1 - 000.0000977..0.0 /0.09/000097 = 0.
Thus, in the 10 states that account for 88% of atlack traffic we under-estimate P(bot|state) by / = 0.91. This is slightly worse than the 1 - 1/K = 1 - 0.002/0.03 = 0.933 predicted in Section 4.1 because in this example there were no unattacked bins. That we closcly approximate the true Naive Bayes estimate, even though the only assumptions we make are per-class independence of features, and that free accounts are more popular with attackers than paid-for ones (e.g., the estimator does not assume knowledge of ((x.(bot). P(x./bot) K).
6 Example: Password-spray guessing attacks
Consider the example of online password guessing. An authentication server receives login requests which are POST events containing a username and password (as well as other logged fealures such as useragent, timestamp, IP address, geo-location, etc). Traffic is an unknown mixture of benign (i.e., attempts by users) and malicious (i.c., guesses sent by attackers). Spray password-guessing attacks involve sending small number of guesses (e.g., less than 100) against each of millions of different accounts. For example, the 100 most common passwords cover approximately 3% of the distribution at large consumer platform [Bonneau.
7

---

Published in Transactions on Machine Learning Research (09/2023)
Joseph 2012 By trying 100 guesses against each of 1 million accounts attacker expect yield of 30,000 accounts. Since no individual account receives large number of guesses, heuristics such as locking an account after several failed attempts (i.e., "three strikes" type rules) have limited utility. Despite the prevalence of the problem there appears little beyond rulcs- approaches and heuristics to address it FForrecii al. 2.14).
Let Th be a feature that is True if the correct password is submitted, and False otherwise. We expect that legitimate users submit the correct password most of the time (failing perhaps =10% due to forgetting, typos, etc). However, attackers must fail more than 99% of the time (since even the most common password covers <1% of accounts). Thus, P(æ, > 0.9 and P(a,/bot) < 0.01. Thus, conservatively, we should have K > 90, so that, from 9 our estimate for any particular bucket of traffic being malicious will be - 1/90 x 0.989 of the true conditional probability. This allows us to identify which (if any) useragents, ISPs, IP address blocks send the highest, proportion of attack traffic. This turn allows us make finely grained decisions. Instead of blocking all requests from certain IP ranges based on rules with hard-coded thresholds (as appears common (Florêncio et al. 2014) we have a full Naive Bayes classifier: e.g.- refusing requests only from certain IP addresses at certain times that also use certain browsers, etc.
7 Evaluation
We evaluate the algorithm simulated data. We choose a random uniform Ry-dimensional vector for (where R, is the feature cardinality along dimension j). For the clean distribution we choose an Ry-dimensional Zipf vector with exponential factor of 2. For d features, with cardinalities R[j] this might be done in Python as follows:
import numpy
for j in range(d):
q[j] = www.or.nddmiiippo (2, R[j])
P[j] numpy.
rand (R (R[j])
P[j] /= P[j] sum
q[j] /= q[j] sum()
We zero one ranseml bin of one randomly chosen feature of Q and re-normalize. We generate a logstream by choosing a sample to be from either the clean or abuse distributions with probability 1- - P(bot) and P(bot) respectively. we choose from the clean distribution we generate the d columns of a data point:
for j in range(d): col[j] random. choices P(j])
where symb [j] is array containing the possible values of the j-th feature. Similarly, if we choose from the abuse distribution. This allows us to generate arbitrary amounts of data for testing. We showed in Section 4.2 that the ROC curve produced by our algorithm is unaffected by P(x, [bot), P(x, ((ot) and K so long as K > 1. Thus, we arbitrarily chose P(z,/bot) = 0.2, P(r.|bot) = 0.1 and K = 2; we verified that the shape of the empirical ROC curve does not depend on these values (i.e., by varying these choices). In our generated data we set v. to True with probability P(r,/bot) and P(x,(bot) for each sample of the clean and abuse data respectively. Again, we emphasize that none of these choices affect the shape of the ROC curve produced by our algorithm. Our algorithm had access to I., but obviously did not have access to the clean/ /anuse labels, the chosen values for P(z,|bot) P(z,/bot) or K. We compare our approach with true NB. The NB algorithm was trained on labelled training set, while our Approximate NB algorithm did not have access to the labels.
Figure shows our comparison ROC curves for increasing values of P(bot) when the clean and abuse data are chosen from Zipf and random distributions as above. Each set involved million samples, 4 categorical features, each of which had cardinality R₃ 20. As expected, the ROC curve of our approximation matches that of NB.
8

---

Published in Transactions on Machine Learning Research (09/2023)
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
5
0.4
0.4
0.2
02
0.2
Nalve Bayes
Nalve Bayes
Naive Bayes
0.0
Approx. NB w/o labels
0.0
Approx. NB w/o labels
0.0
Approx. NB wo labels
0.4
0.6
0.8
0.2
0.8
1.0
fp
f
Figure 1: ROC curves comparing Naive Bayes and our approximation for synthesized data and increasing levels of abuse. The data had d = 4 categorical features each of cardinality 20. The clean distribution was Zipf, while the abuse distribution was uniform. (a) P(bot) = 0.125, (b) P(bot) = 0.25, (c) P(bot) = 0.5. Note that the Naive Bayes classifier had access to labels while our approximate approach did not.
We also test with a second dataset. Instead of Zipf the clean data, P(aylbott, in this case is also formed from a randomly chosen Ry dimensional vector. This might be done in Python as:
for j in range(d)
[[j] numpy www.om.ndo.mmcomom rand (R [j]) Q[j] mampy.raddm..rr rand (R [j])
P[j] /= P[j].sum()
Q[j] /= Q[j] sum
candomly-chosen bin of one randomly chosen feature of Q re-normalize and generate 20 million sample logstream as before. This represents harder classification task since there is less distinction between clean and abuse distributions. Figure 2 the comparisons. While the classifier does worse on this task, again note that the approximate NB method closely mirrors the performance of true NB.
Again we
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
Naive Bayes
Naive Bayes
Naive Bayes
0.0
Approx. NB w/o labels
0.0
Approx. NB w/o labels
0.0
Approx. NB wio labels
1.0
fp
fp
fp
Figure 2: ROC curves comparing various methods for synthesized data and increasing levels of abuse. The data had d = 4 categorical features each of cardinality 20. Both clean and abuse distributions were uniform. (a) P(bot) = 0.125, (b) P(bot) = 0.25, (c) P(bot) 0.5. Note that the Naive Bayes classifier had access to labels while our approximate approach did not.

---

Published in Transactions on Machine Learning Research (09/2023)
8 Discussion
We've given method to approximate NB classifier. We rely on being able to identify binary feature x, that is more prevalent in the benign than in the abuse traflic and the Naive Bayes assumplion of per-class independence of the features.
Our approach is conservative in that we underestimate the odds that traffic is abuse. Under the assumption that at least one bin of at least one feature receives no allack traffic: when is large, our estimate will be closely approximate the true Naive Bayes conditional probability. The ROC curve of our method matches that of true NB (except for sampling effects when data is limited)).
We do not claim that this produces excellent classification decisions or that it will be state-of-the-rt for any particular application. Rather, it provides a new possibility when the alternatives are very limited. This is often the case in abuse problems where there can be high volumes of abuse and there is no possibility of labels. When this is so, a feature satisfying (8) can act as a very useful weak supervision signal. It also useful allowing us to rank subsets of the data by the estimated amount of abuse traffic.
9 Conclusion
We have shown an approach that allows us to approximate the Naive Bayes estimator when we have cate- gorical ures and no labels are available. We require a feature x, that is more prevalent in the legitimate than the scripted traffic.
References
L.V. Ahn, M. Blum, N.J. Hopper, and J. Langford. CAPTCHA: Using hard AI problems for security. In Proceedings of the 22nd International Conference on Theory and Applicalions of Cryplographic Techniques, pp. 294-30 311. Springer-Verlag, 2003.
Akamai. State of the internet / security Q4 2017 http://wwwrpp///.aaaaaaaaaaaaaaa
Apache. Combined Log Format. 2019. httpg://attcc.pcccc..o/////////
Misha Bilenko. Many Shades of Scale: Big Learning Beyond Big Data. 2015. https://www. http:://wwwalidesaaree
Bonneau, Joseph. The Science of Guessing: Analyzing Anonymized Corpus of 70 Million Passwords. In Proc. 2012 IEEE Symp. on Security and Privacy, pp. 538-552, 2012.
Elie Bursztein, Jonathan Aigrain, Angelika Moscicki, and John C Mitchell. The end is nigh: Generic solving bext- captchas. In 8th USENIX Workshop on Offensive Technologies (WOOT 14), 2014.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly Detection: A Survey. ACM Compuling Surveys (CSUR), 41(3):15, 2009.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. SMOTE: Synthetic Minority Over-Sampling Technique. Journal of Artificial Intelligence Research, 16:321-353 2002.
Ye Chen, Dmitry Pavlov, and John F Canny. Large-scale behavioral targeting. In Proceedings of the 15th
ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 209 218, 2009.
Clarence Chio and David Freeman. Machine Learning and Security, O'Reilly Media, 2018
Marthinus Christoffel Du Plessis, Gang Niu, and Masashi Sugiyama. Clustcring unclustered data: Unsuper- vised binary labeling of two datasets having different class balances. In 2013 Conference on Technologies and Applications of Artificial Intelligence, pp. 1-6. TEFE, 2013.
10

---

Published in Transactions on Machine Learning Research (09/2023)
Dinei Florêncio, Cormac Herley, and Paul C van Oorschot. An administrator's guide to internet password research. In USENIX LTSA14: 28th Large Installation System Administration Conference, pp. 14-61,
2014.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistionl Learning. Springer Series in Statistics. New York, NY, USA: Springer New York Inc, 2001.
Cormac Herley, Automated detection of automated traffic. In 31st USENIX Security Symposium (USENIX Security 22), pp. 1615-1632, 2022.
Steve TK Jan; Qingying Hao, Tianrui Hu, Jiameng Pu, Sonal Oswal, Gang Wang, and Bimal Viswanath. Throwing Darts in the Dark? Detecting Bots with Limited Data using Neural Data. Augmentation. In The 41st IEEE Symposium on Security and Privacy (IEEE SP), 2020.
Robert J Joyce, Edward Raff, and Charles Nicholas. A framework for cluster and classifier evaluation in the absence of reference labels. In Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security, pp. 73-84, 2021.
Hirotaka Kaji and Masashi Sugiyama. Binary classification only from unlabeled data by iterative unlabeled- unlabeled classification. In TCASSP 2019-22 9019-2019 TEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3527 3531. LEEE, 2019.
Sneha Kudugunta and Emilio Ferrara. Deep Neural Networks for Bot Detection. Information Sciences, 467: 312-322, 2018.
Wenke Lee, Salvatore Stolfo, and Kui W Mok. Mining audit data to build intrusion detection models. In Proceedings of the 4th ACM SIGKDD international conference on Knowledge discovery and data måning, 1998.
Wei Li, Xuerui Wang, Ruofei Zhang, Ying Cui, Jianchang Mao, and Rong Jin. Exploitation and exploration in a performance based contextual advertising system. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 27 27-36, 2010.
Xigao Li, Babak Amin Azad, Amir Rahmati, and Nick Nikiforakis. Good bot, bad bot: Characterizing automated browsing activity. IEEE Symp. on Security & Privacy, 2021.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation Forest. In 2008 Eighth IEEE International Conference on Data Mining, pp. 413-422 IEFE, 2008.
Fabio Parisi, Francesco Strino, Boaz Nadler, and Yuval Kluger. Ranking and combining multiple predictors without labeled data. Proceedings of the National Academy of Sciences, (4)::2233-1258 2014.
Bernhard Schölkopf, Robert C Williamson, Alexander J Smola, John Shawe- Taylor, John C Platt, et al. Support Vector Method for Novelty Detection. In NIPS, volume 12, pp. 582 588. Citeseer, 1999.
Bhavna Soman, Ali Torkamani, Michael Morais, Jeffrey Bickford, and Baris Coskun. evaluation using weak signals. arXio preprint ar arXiw:2207.0082) 2022.
Firenze: Model
Robin Sommer and Vern Paxson. Outside the Closed World: On using Machine Learning for Network Intrusion Detection. In Security and Privacy (SP), 2010 IEEE Symposium on, pp. 305-316. IEEE, 2010.
Gianluca Stringhini, Pierre Mourlanne, Gregoire Jacob, Manuel Fgele, Christopher Kruegel, and Giovanni Vigna. (EVILCOHORT): Detecting communities of malicious accounts on online services. In 24th {USENIX} Security Symposium (USENIX)) Security 15), pp. 563-578, 2015.
Onur Varol, Emilio Ferrara, Clayton Davis, Filippo Menezer, and Alessandro Flammini. Online human- bot interactions: Detection, estimation, and characterization. In Proceedings of the International AAAI Conference Web and Social Media, volume 11, 2017.
Alice Zheng and Amanda Casari. Feature engineering for machine learning: principles and techniques for dala sciendists. O'Reilly Media, Inc., 2018
11


Reviews:
[{"review_id": "DAnvgLzL66", "reviewer": "Reviewer_2CVy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea behind the paper is clever, and gets at a niche - but very real - problem in the information security space. The theory is simple and makes sense in how they approached the problem and potentially gives life and utility to the often inferior naive Bayes. \n\nWhile very desirable, I think the most significant item missing from this work is some empirical evaluation. Even if industry data can not be obtained (e.g., the password attack example), many datasets exist and could be used to explore the efficacy of this approach. Given the very empirical motivation of the paper, it is thus hard to tell if the approach is actually of utility. This is notable because the idea behind the paper has been explored in other works in the information security realm, and the strategy can be leveraged at large without relying on the specific approach given (i.e., use noisy label sources and train models assuming label noise occurs). \n\n-----\n\nTo avoid confusion, the author's current stance is that the paper is only claiming math and application. In this light, I think the paper as a whole needs major re-writing as the examples, notation, and motivation all lean heavily to imply a practical use in information security. However, this renders the paper - in my opinion and reading of the guidelines - uninteresting to the TMLR audience. Given the guidelines state that if there is any doubt, one should assume it would be applicable: I am confident that it would not be of interest. I hope this clears up confusion on this reviewer's stance on what actions can be taken to improve the manuscript. I do earnestly hope it is headed as there is a niche idea that could prove insightful, but I don't believe any insight can be drawn without some empirical results on real data. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The idea behind this work is that, if we are willing to accept a model as naive as naive bayes, we can exploit prior information about the conditional probability of the class label given specific features to approximate the naive bayes solution. This is valuable in cases where the labeling of a non-trivial amount of data is very hard, as is often the case in information security problems. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": " In particular, the obvious ablation seems to be:\n\n1. Train Naive Bayes on the ground truth labels $y$\n2. Train this approach on some 1-5 features that are most biased toward $y$, but hold back the true $y$ labels. \n3. Train naive bayes using $\\hat{y} = x_*$ to quantify the empirical benefit of using this approach over assuming that the feature should instead be used as a proxy label. \n\nThere are also some related works from the information security space that seem critically relevant given the approach in the paper. Most notable is the Firenze approach presented at CAMLIS 2022 ( https://arxiv.org/abs/2207.00827 )  and the Approximate Ground Truth Refinement (AGTR) from AISec 2021 ( https://arxiv.org/abs/2109.11126 ) . Both of these works deal with the evaluation of malware classified on unlabeled data by using some form of side or prior information. While different from the proposed work, they are more directly relevant than the cited literature as those in the IS community have worked on tackling these issues already. \n\nWhile the AGTR should be noted in related work, I do think it is appropriate to perform a comparison against a (mildly adapted) Firenze, as it is employing the same key insight: some features are known apriori to occur more frequently in $y$ and in $\\neg y$, and that can be achieved with a small number of such features. Firenze also provides the foundation for this exploration via the Ember dataset. \n\nI think this would make the work complete and demonstrate that the proposal is worth consideration and use, and fully support the claim that this is a viable approach to leverage. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FGm30sdGR3", "reviewer": "Reviewer_cWPS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n1. The proposed idea is quite interesting and seems promising under specific real scenarios (unlabeled data, categorical data). The paper highlight several practical settings that well fit the limitations of the assumptions (existing feature with a higher benign chance + independence).\n2. Two toy examples are given to show fairly good approximations by the proposed NB estimator and both example settings seem quite practical.\n\n### Weakness\n1. The proposed approach targets for specific scenarios due to assumption constraints.\n2. Additional empirical evaluation would help a lot to the paper. As mentioned in the related work that existing unsupervised classifier with one-hot encoding and one-class SVM could likely perform poorly under unlabeled categorical settings. However, it would be very helpful to have some existing common methods evaluated in the provided toy examples as well to confirm this point. In specific the paper claims the distance between points and the fraction of samples expected to be anomalous are the main limitation factors. It would be interesting to see, comparing the proposed method to K-means and One-class SVM, how the estimation tightness changes in terms of changing these factors. This would be very helpful to guide real users when to use the proposed NB classifier over the others.\n3. It would be great to show empirical ablation study of estimation tightness in terms of states and K for one toy example with specific feature like AOL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an Approximated Naive Bayes (NB) classifier targeting for unlabeled and categorical data points. It relies on the assumption that we can have a feature x* has higher chance in the benign setting over abused ones. Also, another assumption is per-class independence of the features. The paper find good motivating settings for the real use cases of this proposed NB classifier.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness above for evaluation comments.\n\nIn terms of writing, I would suggest having a subsection in related work, mentioning (1) in general how people dealing with related abnormal detection in existing works, and (2) how the investigated problem scenarios in this paper differ from existing ones.", "broader_impact_concerns": "No ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7VPm06JHn3", "reviewer": "Reviewer_YFuT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe method is simple and reasonable, and the theoretical guarantees are quite good given what is assumed.  As the authors argue, the method might be practically useful in settings where no other method easily applies.\n\n\nWeaknesses\n\nI think an empirical evaluation of the method would make the paper much stronger, and better support the claims.  Also see detailed comments below for some minor issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a theoretically-principled method for approximating Naive Bayes on unlabeled datasets with categorical features.  The key assumption is there exists a binary feature that is more prevalent in one class than in the other.  The relevant prevalence controls the acuracy of approximation.  Under further assumptions, the method preserves the ordering of data points, and thus produces the same ROC curve as Naive Bayes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(Also including detailed comments here.  The authors can use their judgment regarding the more subjective suggestions.)\n\nLast sentence on page 1, \"Varol et al\"  (and later in the paper): missing \".\"; consider using \\citeauthor?\n\n\"Legitimate users use paid-for email services\": I'm not sure to what extent this is still true --- any evidence?\n\nRelated work, one-hot encoding: in principle one can define the distance in richer ways.  For example, Win98 is probably closer to WinXP than to MacOS 10.0.\n\nSecond paragraph of sec 3: \"discus\".\n\nNext paragraph, \"replace each token with the conditional probability\": in general information is lost in this process, right?  Any theoretical justification of this approach?  (Well I see that very soon you mention the independence assumption...)\n\nParagraph after eq (4), \"(4) implies ...\": I guess what you really mean is P(x_* | bot) \\ge 0 (and the inequality doesn't need to be strict)...\n\nSec 4.3: \"at least one bin is free of attack\": this is a bit tricky when considered together with confidence intervals.  As the authors argue, it is quite plausible that no attack comes from some city / region / ASN, but then taking into consideration confidence intervals one can't just say the true probability of an attack is 0.  In fact there's a tradeoff between \"seeing no attack in the dataset\" and \"being sure that the true probability isn't too much higher\", and I think it's somewhat unfair to emphasize only the former.", "broader_impact_concerns": "I don't have concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)