Internet
Infrastructure
Review
May 2019

Vol.

42

Periodic Observation Report

SOC Report
Focused Research (1)Deep-Learning for Log Analysis
to Detect Malicious Communications
Focused Research (2)Designing a Large-scale
Email System

Internet Inf rast ruc ture Review
May 2019 Vol.42

Executive Summary ........................ ................................................ 3
1. Periodic Observation Report .......... ................................................ 4
1.1
1.2

Introduction ............................................................................... 4

Observational Data ........................................................................ 4

1.2.1 Attacks Involving Cryptocurrencies . ....... ................................................ 4
1.2.2 SYN/ACK Reflection Attack ................ ................................................ 6
1.2.3 Resurgence of Attacks Targeting Known Vulnerabilities . ................................... 7
1.3

Detecting Malicious Transmissions Using Machine Learning . ............................... 8

1.3.1 Application to DNS Query Data ............ ................................................ 8
1.3.2 Application to Web Proxy Data . ........................................................... 8
1.4

Conclusion ................................ ................................................ 9

2. Focused Research (1) .................. ............................................... 10
2.1
2.2
2.3
2.4

Introduction ............................... ............................................... 10

Background . ............................................................................. 10
Detecting Communications with Malware C&C (C2) Servers ............................... 11

Exploit Kit Detection ....................... ............................................... 15

3. Focused Research (2) .................. ............................................... 18
3.1
3.2

Introduction ............................... ............................................... 18
Challenges and Goals of the System Overhaul ............................................. 18

3.2.1 Review of the Architecture . ............... ............................................... 18

3.2.2 Better Filtering ............................. ............................................... 19
3.2.3 Avoiding Over-reliance on Any One Vendor . .............................................. 19
3.3
3.4
3.5
3.6

Single-stage MTA for Effective Use of Hardware Resources . .............................. 20
Making Antivirus/Antispam Engines Interchangeable ....................................... 20
Decision to Develop MTA In-house ......... ............................................... 22

Overhaul Outcomes . ...................... ............................................... 23

3.6.1 Achieving our Development Goals . ....................................................... 23

3.6.2 Secondary Benefits ....................................................................... 23

2© Internet Initiative Japan Inc.

May 2019

Vol.

42

Executive Summary

Executive Summary

Conventional DDoS attacks primarily used botnets consisting of a large number of malware-infected PCs toflood the attack target with connections. Then from around 2013, we began to observe large-scale DDoS attacks where the attacker had taken advantage of poorly configured IoT devices, such as home routers and internet cameras, exploiting their vulnerabilities to infect them with malware.
We expect the number of internet-connected IoT devices to continue rising strongly ahead, and preventing
them from being used in DDoS attacks will be crucial to keeping the internet safe. In response to such threats,
on February 1, 2019, Japan’s Ministry of Internal Affairs and Communications, the National Institute of Information Communications and Technology (NICT), and telecommunications carriers unveiled a project to survey
IoT devices and alert users called “NOTICE (National Operation Towards IoT Clean Environment)”.
Through the NOTICE project, the NICT will survey IoT devices on the internet to identify devices vulnerable tocyberattacks and pass information about relevant devices onto the telecommunications carriers. The carriers
will then identify the users of those devices and alert them to the issue. This is a collaboration between government and the private sector aimed at enhancing the safety of internet-connected IoT devices, and IIJ is anactive participant.
The IIR introduces the wide range of technology that IIJ researches and develops, comprising periodic observation reports that provide an outline of various data IIJ obtains through the daily operation of services, as well as
focused research examining specific areas of technology.
As our periodic observation report for this edition, Chapter 1 discusses our SOC Report. IIJ’s SOC collates andanalyzes a range of logs, including those from security devices provided as IIJ services, on its Data Analytics
Platform, and we release up-to-date information on threats observed in blog format via wizSafe Security Signal.
In this edition, we draw from past wizSafe Security Signal posts and look at three types of notable activity revealed using our Data Analytics Platform and also describe the use of our Data Analytics Platform for machinelearning.
Our first focused research report in Chapter 2, titled “Deep-Learning Analysis of Logs to Detect Malicious Communications”, is a restructured version of the content of a presentation given by IIJ staff at Black Hat Europe
2018. The report looks at general-purpose methods for detecting threats using logs from commonly available
servers and network devices, rather than using specialized equipment or security devices. These huge logs require complex processing, but our research has confirmed there is potential to make effective use of them ifthey are properly optimized for deep learning.
Our second focused research report in Chapter 3 discusses the overhaul of IIJ Secure MX Service, the email
gateway service that IIJ provides. More than 10 years have passed since the service was first launched. It
remains a key service for IIJ and continues to see strong growth in subscriptions to this day. That said, withthe changes in the landscape over those 10 years, the system had grown obsolete and was saddled by various
issues. The chapter presents a report from one of the engineers involved in development process, which included revising system architecture to solve those issues and the decision to develop the system in-house, and wehope this serves as a good reference.
Through activities such as these, IIJ strives to improve and develop its services on a daily basis while maintaining the stability of the Internet. We will continue to provide a variety of services and solutions that our customers can take full advantage of as infrastructure for their corporate activities.

Junichi Shimagami
Mr. Shimagami is a Senior Executive Officer and the CTO of IIJ. His interest in the Internet led to him joining IIJ in
September 1996. After engaging in the design and construction of the A-Bone Asia region network spearheaded by IIJ,
as well as IIJ’s backbone network, he was put in charge of IIJ network services. Since 2015, he has been responsible
for network, cloud, and security technology across the board as CTO. In April 2017, he became chairman of theTelecom Services Association of Japan MVNO Council.

© Internet Initiative Japan Inc.

3

1. Periodic Observation Report

SOC Report

1.1 Introduction

Section 1.2 describes information on threats revealed via

IIJ announced its new security business brand wizSafe*1 on

the Data Analytics Platform in 2018, and Section 1.3 de-October 31, 2016 and is constantly working to bring about

scribes new initiatives using the Data Analytics Platform.

a world in which customers can use the Internet safely. As

The observations for 2018 are summarized in wizSafe

part of such efforts, we release up-to-date information on

Security Signal*4.

security threats observed at our SOC in blog format via wizSafe Security Signal*2. This includes some information on

1.2 Observational Data

threats identified through IIJ’s Data Analytics Platform. For

First, we look at activity identified using the Data Analytics

an overview of the Data Analytics Platform, see Internet

Platform that is particularly noteworthy. This information is

Infrastructure Review (IIR) Vol. 38*3.

taken from wizSafe Security Signal posts from last year.

Here, we give an overview of analysis using the Data

1.2.1 Attacks Involving Cryptocurrencies

Analytics Platform. The logs collected on the platform nat-

Attempts to monetize attacks using cryptocurrencies at-urally include those from security devices such as firewalls,

tracted attention in 2018. Analysis on IIJ’s Data Analytics

IPS/IDS, and antivirus solutions provided as IIJ services, as

Platform also revealed several cases of attackers attempting

well as logs of DNS queries, Web access, incoming/outgo-

to exploit cryptocurrencies.

ing email, and so forth. Characteristically, these logs contain
only a tiny amount of abnormal traffic (threats) among a

The first example involves manipulating a website to embedlarge amount of normal traffic. We therefore need to thinka mining script. Our SOC’s observations uncovered multiple

about how to go about aggregating and visualizing the datacases of mining scripts embedded in websites that do not

so that we can identify threats clearly.

appear to have been put there intentionally by the website

WebLogic server
1.Attack

DL.ps1

DL & run

3.DL & run

Attacker

Neutrino.ps1

C2 traffic part
Mining part2.Download & run

DL.php3.DL & run

WMI.ps1

3.DL

Runs

lsass.eXe

One of:*x86.exe*x86_VMP.exe*64.exe*x64_VMP.exe

Figure 1: Overview of GhostMiner Attack

4

Response*1

IIJ announces new security business brand wizSafe (https://www.iij.ad.jp/en/news/pressrelease/2016/1031.html).

*2

wizSafe (in Japanese at https://wizsafe.iij.ad.jp).

*3

IIJ, Internet Infrastructure Review (IIR) Vol. 38 (https://www.iij.ad.jp/en/dev/iir/038.html).

*4

wizSafe, “wizSafe Security Signal 2018 Annual Summary” (in Japanese at https://wizsafe.iij.ad.jp/2019/03/601/).

© Internet Initiative Japan Inc.

C2 server

May 2019

Vol.

42

1. Periodic Observation Reportadministrator. By exploiting website vulnerabilities and so

Yet another example involves not mining but attempts toforth, attackers can embed mining scripts into Web pages.

illegally transfer funds. In December 2018, we observed

When a user views a tainted site, the user’s computer runs

scanning activity (Figure 2) targeting the JSON-RPC pro-the cryptocurrency mining script, and any mined proceeds

tocol used in an Ethereum client*8. The scanning activity

go to the attacker.

was looking for Ethereum clients that are accessible via theInternet due to a misconfiguration. We note that a number

The above is an example of an attack aimed at clients,of conditions must be met for the funds transfer to actually

but we have also observed cryptocurrency mining attacks

complete successfully.

on servers. One specific example is an attack campaign*5
called GhostMiner (Figure 1). The GhostMiner campaign

Cryptocurrencies are appealing to attackers because attacks

was observed in March 2018 and exploits a vulnerabilityon them can be monetized directly and because, depending(CVE-2017-10271) in Oracle WebLogic Server. The vulner-

on the type, they offer a high degree of anonymity. Also,

ability allows the execution of remote code, so the attacker

any environment that has computational resources can be

ultimately attempts to get the Web server to mine crypto-

attacked, as evidenced by the variety of attacks geared

currency. We have also observed several other attempts toto cryptocurrency mining, which target both clients anduse remote code execution vulnerabilities to get servers toservers. We expect attackers to continue to target crypto-mine cryptocurrency

currencies as one means of monetizing attacks.

.

*6*7

(Number of detections)
1400
1200
1000
800
600
400
■BG
■US
■RU
■Other200

2000h, Dec. 31, ’180000h, Dec. 31, ’18

0400h, Dec. 30, ’18

0800h, Dec. 29, ’18

1200h, Dec. 28, ’18

1600h, Dec. 27, ’18

2000h, Dec. 26, ’180000h, Dec. 26, ’18

0400h, Dec. 25, ’18

0800h, Dec. 24, ’18

1200h, Dec. 23, ’18

1600h, Dec. 22, ’18

2000h, Dec. 21, ’180000h, Dec. 21, ’18

0400h, Dec. 20, ’18

0800h, Dec. 19, ’18

1200h, Dec. 18, ’18

1600h, Dec. 17, ’18

2000h, Dec. 16, ’180000h, Dec. 16, ’18

0400h, Dec. 15, ’18

0800h, Dec. 14, ’18

1200h, Dec. 13, ’18

1600h, Dec. 12, ’18

2000h, Dec. 11, ’180000h, Dec. 11, ’18

0400h, Dec. 10, ’18

0800h, Dec. 9, ’18

1200h, Dec. 8, ’18

1600h, Dec. 7, ’18

2000h, Dec. 6, ’180000h, Dec. 6, ’18

0400h, Dec. 5, ’18

0800h, Dec. 4, ’18

1200h, Dec. 3, ’18

1600h, Dec. 2, ’18

2000h, Dec. 1, ’180000h, Dec. 1, ’180(Date & time)

Figure 2: Scanning of 8545/TCP (Dec. 2018)*5

wizSafe, “GhostMiner infections spreading” (in Japanese at https://wizsafe.iij.ad.jp/2018/04/323/).

*6

wizSafe, “wizSafe Security Signal January 2018 Observation Report” (in Japanese at https://wizsafe.iij.ad.jp/2018/02/247/).

*7

wizSafe, “wizSafe Security Signal February 2018 Observation Report” (in Japanese at https://wizsafe.iij.ad.jp/2018/03/286/).

*8

wizSafe, “Ethereum JSON-RPC scans observed” (in Japanese at https://wizsafe.iij.ad.jp/2019/01/541/).

© Internet Initiative Japan Inc.

5

1.2.2 SYN/ACK Reflection Attack

not be able to realize an attack of the scale envisaged. Also,

One peculiar example of a DDoS attack that the SOC ob-

because only a small volume of SYN packets is received perserved in 2018 is a SYN/ACK reflection attack using 80/

server, one can infer that the attacker is probably sending

TCP. This was included in wizSafe Security Signal forout SYN packets far and wide.

September 2018*9 (Figure 3). The attack sends TCP SYN
packets with a spoofed source address to many addresses

The attack described above uses port 80/TCP, and servers

simultaneously, thereby effectively recruiting the resulting

on which 80/TCP is open can generally be considered to

SYN/ACK packet responses to perform a DDoS attack on

be Web servers. Hence, normal Web access traffic on suchthe source address.

servers do contain a small amount of SYN packets, the typeof packet used in a SYN/ACK reflection attack, and it isThis SYN/ACK reflection attack was observed by the SOC

thus difficult to determine whether any of those packets areon September 26, 2018, but it has also been observed on abeing used in an attack. In this example, we detected thesmall scale since October, and attacks of the same type areattack by cross-analyzing the multiple customer firewall logs

detected daily via the Data Analytics Platform. One featurepresent on our Data Analytics Platform.

of the DDoS attack observed on September 26 is that thesource invokes the attack by sending a small amount of SYN

Because firewall logs reveal information about internal andpackets to servers on which the 80/TCP port is open to theexternal access, such attacks can be detected when multi-Internet. If an attacker sends a high volume of SYN packets

ple firewall logs indicate that 80/TCP responses are being

to a single server, the administrator of the receiving server

generated for a single specific IP address (the spoofed IP ad-

is liable to think that a TCP SYN flood attack*10 is underway

dress being attacked). However, this feature could indicate

and block further traffic. If this happens, the attacker may

scanning activity rather than a DDoS attack. We therefore■ Traffic count for the single IP address■ Traffic count for other IP addresses

(%)
14
12
10
8
6
4
2

00
00
h,
Se
p.
01
26
00
,’
h,
18
Se
p
02
.2
00
6,
h,
’
18
Se
p.
03
26
00
,
h,
’
18
Se
p.
04
26
00
,’
h,
18
Se
p.
05
2
00
6,
h,
’
18
Se
p.
06
2
00
6,
h,
’
18
Se
p.
07
26
00
,’
h,
18
Se
p.
08
26
00
,’
h,
18
Se
p.
09
2
00
6,
h,
’
18
Se
p.
10
2
00
6,
h,
’
18
Se
p.
11
26
00
,’
h,
18
Se
p.
12
26
00
,’
h,
18
Se
p.
13
2
00
6,
h,
’
18
Se
p.
14
26
00
,
h,
’
18
Se
p.
15
26
00
,’
h,
18
Se
p.
16
2
00
6,
h,
’
18
Se
p.
17
2
00
6,
h,
’
18
Se
p.
18
26
00
,’
h,
18
Se
p.
19
26
00
,’
h,
18
Se
p.
20
2
00
6,
h,
’
18
Se
p.
21
2
00
6,
h,
’
18
Se
p.
22
26
00
,’
h,
18
Se
p.
23
26
00
,’
h,
18
Se
p.
26
,’
18

0

(Date & time)

Figure 3: Increase in 80/TCP Traffic from a Single IP Address*9*10

wizSafe, “wizSafe Security Signal September 2018 Observation Report” (in Japanese at https://wizsafe.iij.ad.jp/2018/10/470/).
In a TCP SYN flood attack, the attacker sends a large amount of SYN packets—requests used to establish a TCP connection—to the target system, causing it toprepare for a large number of connections and thereby wasting processing power, memory, etc.

6

© Internet Initiative Japan Inc.

May 2019

Vol.

42

1. Periodic Observation Report

differentiate between DDoS attacks and scanning activity

The vulnerability that the malware exploits is a buffer over-based on total bytes sent/received, duration, and so on as

flow issue with the Microsoft Office Equation Editor that

calculated from the firewall logs.

allows remote code execution. Microsoft issued a patch thatfixes this vulnerability in November 2017. As a workaround,

The SYN/ACK reflection attack that we wrote about inusers can also disable the Equation Editor as a means of

September 2018 was also observed in IIJ’s honeypots. We

avoiding this attack without applying the patch.

reported about this in detail in an IIJ-SECT blog post (inJapanese): “SYN/ACK reflection attack using IoT devices as

We observed an attack targeting this vulnerability via thea springboard”*11. The post describes changes in the ports

Data Analytics Platform in September 2018, almost a year

used in the attack and reveals that it is a complex DDoS

after the fix had been issued (Figure 4)*12. The attackers

attack that uses the UDP protocol, so we encourage you tosent malware that exploits the vulnerability as an email

read through it.

attachment. We think this was a deliberate attempt to target systems in which a fix was never applied or in which

1.2.3 Resurgence of Attacks Targeting Known Vulnerabilities

the Equation Editor had only temporarily been disabled as

One notable of the 2018 analysis performed on the Dataa means of avoiding the attack, and that it was timed for

Analytics Platform is that some attacks targeting vulnerabili-

when awareness of this vulnerability had faded somewhat.

ties that have already been disclosed and for which patches
have been made available have re-emerged after being dor-In addition to the Microsoft Office vulnerability discussed

mant for some time. One example is malware that exploits

here, we have observed multiple other similar cases via thea vulnerability (CVE-2017-11822) in the Microsoft Office

Data Analytics Platform*13. A lesson to be learned from theseEquation Editor.

observations is that whether one implements a fundamental

Other

5.03%

Trojan.Win32.Neurevt
1.26%
Trojan.HTML.Phish
Trojan.Java.Agent

Email-Worm.Win32.Mydoom
42.88%

1.28%
1.91%

Trojan-Spy.Win32.Zbot
2.10%
Trojan-Downloader.
MSWord.Agent
2.26%
Backdoor.Win32.Androm
7.93%
Trojan-PSW.Win32.Fareit
8.51%
Trojan-Spy.Win32.Noon
11.49%

Exploit.MSOffice.
CVE-2017-11882
15.35%

Figure 4: Breakdown of Malware Types Detected
in Received Emails (Sep. 2018)*11

IIJ-SECT Security Diary, “SYN/ACK reflection attack using IoT devices as a springboard” (in Japanese at https://sect.iij.ad.jp/d/2019/02/128021.html).

*12

wizSafe, “wizSafe Security Signal September 2018 Observation Report” (in Japanese at https://wizsafe.iij.ad.jp/2018/10/470/).

*13

wizSafe, “wizSafe Security Signal November 2017 Observation Report” (in Japanese at https://wizsafe.iij.ad.jp/2017/12/184/).

© Internet Initiative Japan Inc.

7

fix to a vulnerability or, depending on the circumstances,announced at USENIX Security ’18. As the conference

any of the various workarounds available, it is crucial that

paper on FANCI explains, the system combines domain fea-such measures are kept in place indefinitely.

tures inspired by those used in past research with a machinelearning algorithm known as random forests, and it general-

1.3 Detecting Malicious Transmissions		
Using Machine Learningizes very well.

Characteristically, the data analyzed on our Data Analytics

As the first step in our follow-up research, we aim to stick

Platform include only a tiny amount of abnormal traffic

to the methodology described in the paper as much as pos-(threats) among a large amount of normal traffic. Efforts are

sible and use the DNS query data available from our Databeing made to use machine learning to discover such threats.

Analytics Platform. This first step is intended to assess

The main task handled is that of detecting anomalies fromwhether the methodology can be applied unmodified to theimbalanced data. Here, we describe two such projects that

Data Analytics Platform’s data. We do this because any

are underway, along with the challenges they face.

given methodology will not necessarily produce the same
results when different data are used. Next, if we determine

1.3.1 Application to DNS Query Data

that the methodology cannot be applied as is, and that it is

The domain names of the C2 (command & control) servers

possible to investigate why and implement a solution, weused by malware may be generated algorithmically using

intend to work toward a practical implementation that may

a DGA (domain generation algorithm). The domain names

include additional performance enhancements. Potential per-generated by DGAs differ widely depending on the type offormance enhancements could, for example, come from thealgorithm and the parameters used when running it. This canuse of gradient boosting decision trees, a popular method inmake it difficult to blacklist the malware’s servers ahead ofrecent years, or ensemble learning that combines undersam-

time or to create an expression for the detection signature.

pling and bagging.

So in this project, we aim to solve the problem by combining

The volume of data passing through and processed by thethe Data Analytics Platform’s DNS query data with machine

Data Analytics Platform is large, however, so as a matterlearning. We take this approach because tasks that humansof practicality, the model needs to have high throughput.

find difficult to construct rules for can be amenable to ma-

We aim to strike a balance between the increasing compu-

chine-learning solutions. A desirable property of machine

tational load that results from the use of more complicatedlearning algorithms is that they can autonomously acquire

models and workflows and the performance enhancements

the ability to classify anomalies when provided with datathat can be obtained, and with some fine tuning, we ul-containing features that are effective in identifying those

timately aim to build the system into our Data Analytics

anomalies. For example, in IIR Vol. 41, we looked at URL

Platform to provide one means of detecting these anomalies.

strings and described an approach to identifying rogue sites
1.3.2 Application to Web Proxy Data

using neural networks*14.

One other project we are pursuing aims to detect comWe know of several attempts to use machine learning tomunications sent to C2 servers by malware in Web proxydetect DGAs, including some that have already been put

data. We are currently running validation tests with theinto real-world use. Currently, we are engaged in research

objective of applying the methodology presented by IIJ en-

that follows on from the FANCI (Feature-based Automatedgineers at Black Hat Europe 2018*16 to our Data Analytics

NXDomain

Platform’s logs. The methodology presented at BlackClassification

andIntelligence)*15

system*14

IIJ, Internet Infrastructure Review (IIR) Vol. 41 (https://www.iij.ad.jp/en/dev/iir/041.html).

*15

USENIX, “FANCI: Feature-based Automated NXDomain Classification and Intelligence” (https://www.usenix.org/conference/usenixsecurity18/ presentation/schuppen).

*16

Black Hat, “Deep Impact: Recognizing Unknown Malicious Activities from Zero Knowledge” (https://www.blackhat.com/eu-18/briefings/schedule/#deep-impact-recognizing-unknown-malicious-activities-from-zero-knowledge-12276).

8© Internet Initiative Japan Inc.

May 2019

Vol.

42

1. Periodic Observation Report

Hat Europe 2018 is described later in this edition underto improve the accuracy of machine learnings models, and“Focused Research (1): Deep-Learning Analysis of Logs to

is focused on building systems that improve quality without

Detect Malicious Communications”.

putting any additional load on current security operations.

The project uses convolutional neural networks, which are

One attempt to improve accuracy entails feature engineer-commonly applied to image recognition tasks, to discern

ing. There is a strong perception that feature engineering

trends in normal traffic and anomalous traffic (with a C2

involves adding features expected to be effective on theserver). The key here is learning model performance and its

basis of data analysis, but other approaches also exist. For

evaluation.

example, various statistics can be calculated from existing
features, combined with the data from which they were de-If there were, say, a model capable of producing 95% ac-

rived, and then used for learning and evaluation. We will

curacy or better, this would generally be regarded as good

also use various other methods to repeatedly add and eval-

performance. But because the volume of logs collected onuate features as we work to enhance model.

the Data Analytics Platform is enormous, 1% of this data
is not the sort of volume that a human could process by

1.4 Conclusioneye. Even if false positives do arise, the model needs toIn this edition, we provided an overview of analysis using

provide a level of accuracy that is tolerable when put into

the Data Analytics Platform, went over some actual obser-

operation. This, of course, is the case when only machine

vations from 2018, and described our efforts with respectlearning is used, and approaches that reduce false positives

to machine learning. The final machine learning approachthrough non-machine-learning systematic processing arethat we described has the potential to further expand thealso conceivable.

detectable range for threats where detection with conventional methods is difficult or subject to limitations. We are

Aside from accuracy, we also need to be aware of differ-able to pursue these efforts entirely because we are able

ences in the distributions of the datasets we are dealing

to use traffic logs received from customers on the Data

with. It is quite possible that the distributions of datasets

Analytics Platform, subject to customer consent. Machineused by reportedly well-performing machine learning mod-learning approaches require large volumes of data in par-els presented at conferences, academic events, and the liketicular, so it is fair to say that we are only able to pursue

are characteristically different from the dataset distributions

these efforts because we have access to these traffic logs

encountered by our SOC, so follow-up research is needed.

via the Data Analytics Platform. We will continue to provide
up-to-date information on threats through wizSafe Security

In view of the above, the SOC takes the overall design and

Signal and the IIJ-SECT blog, and we will continue striving

operation of systems that use machine learning models into

to bring about a world in which the Internet is even safer forconsideration, conducting follow-up research and working

customers to use.

Satoshi Kobayashi
Data Analyst, Security Operations Center, Security Business Department, Advanced Security Division, IIJ

Shun Morita
Data Analyst, Security Operations Center, Security Business Department, Advanced Security Division, IIJ© Internet Initiative Japan Inc.

9

2. Focused Research (1)Deep-Learning for Log Analysis
to Detect Malicious Communications
2.1 Introduction

Some of the solutions described earlier are also very

Deep learning can be used to discover malicious commu-

expensive and thus not necessarily something that all orga-

nications. Here, we describe two methods of detecting

nizations can deploy. The aim of our work, therefore, was tomalware communications caused by malware and by exploit

develop a general-purpose solution that would enable many

kits in the huge volumes of logs generated by commonlyorganizations to detect threats based on the logs createdused devices such as firewalls and Web proxy servers.

by common types of servers and network devices, such as
Web proxy servers, routers, and firewalls, rather than spe-

This chapter is a retelling of a presentation titled “Deepcialized equipment and security devices. These logs have

Impact: Recognizing Unknown Malicious Activities from Zerorarely been used effectively in the past, with their use being

Knowledge”*1 given as part of the Briefings sessions at thelimited to cases such as the following.

Black Hat Europe 2018 international security conference.
• Anomaly detection based on communication vol-2.2 Backgroundume, frequency, etc.

In most cases at present, the following methods are used

• SIEM event correlation

to detect malicious activities, including malware infections.

• Detection using IoCs (indicators of compromise)when they are available*2• Pattern matching (including blacklists and whitelists)• Behavioral analysis

If we can make use of these sorts of logs, which con-• Event correlation

sume valuable disk space, many organizations may be able
to achieve greater levels of security without making large

However, sophisticated and unknown attacks can circum-changes to network structure or additional investments.

vent these solutions. And even with attacks that are notparticularly sophisticated or unknown, detection rules for

One possible reason why these sorts of logs have not been

pattern matching, for instance, need to be changed in re-

put to effective use is that, although somewhat dependent

sponse to even small changes in an attack’s pattern. Thison system and organizational scale, the logs themselves

is because the existing detection methods are

based onare very large, preventing effective analysis that involves

information that an attacker can easily alter, such as the C2

high computational complexity. Deep learning, however, isserver domain name, IP address, and the executable’s bi-

known to be suitable for big data analysis; for example, it isnary pattern. Hence, if it is possible to use detection criteria

capable of processing hundreds of millions of images eachthat do not rely on these existing methods and that apply toon the order of tens to hundreds of kilobytes in size. So if

essential aspects of an attack that are difficult for attackers

such logs can be optimized for deep learning, it may be pos-to alter, we can combine such criteria with existing methods

sible to solve this problem.

to achieve an ever greater level of security.

Continually checks forand executes commands

Sets command(s)AttackerC2 server

Infected device

Figure 1: Bot or RAT Continually Communicatingwith a Command & Control (C2) Server*1

Deep Impact: Recognizing Unknown Malicious Activities from Zero Knowledge (https://www.blackhat.com/eu-18/briefings/schedule/index.html#deep-impact-recognizing-unknown-malicious-activities-from-zero-knowledge-12276).

*2

For example, IoCs may be obtained from host names discovered via anomaly detection or analysis of suspicious devices reported by users, as well as from externalreports, etc.

10

© Internet Initiative Japan Inc.

May 2019

Vol.

42

2. Focused Research (1)2.3 Detecting Communications 			with Malware C&C (C2) Servers

it would be able to detect malware communications. This

Some types of malware such as bots and RATs continu-

the like, so it should detect malware that existing detection

ally connect to their C2 servers to receive commands frommethods miss.

method does not rely on DNS name, IP address, URL, andattackers, which they then execute (Figure 1). Typically,
they use polling intervals that range from several dozen sec-

Here, we divide the logs up by client and server and countonds to several minutes or so. The longer the interval, thecommunications in 1-minute buckets for each 1-hour pe-

longer the program waits for any single command, whichriod. We thus convert the logs into pseudo 60-dot images

makes it difficult for the attacker to take action. Conversely,on which we perform image recognition using CNNs (con-the shorter the interval, the easier it is for the attacker tovolution neural networks), a class of neural network usedact, but the easier defenders can detect the activity, sincein deep learning. It is known that, depending on the model

the activity will appear toward the top in a simple analysis

used, CNNs can outperform human recognition accuracy,of communication frequency broken down by destination

and we therefore transform the logs into images and use

hosts . What this boils down to is that adjusting the fre-this class of network in the hopes that this will prove more

quency of communication presents both an important task

effective than other deep learning models.

*3

and a tough decision for the attacker. Meanwhile, whenordinary users within an organization communicate with

Our training dataset is constructed as follows. We use 1.5

external networks, such as when accessing the Web, it ismillion “images” created from Web proxy logs as our benign

rare for those communications to occur frequently or persist

sample set. For our malicious samples, we use no actual

over a long time. Figure 2 illustrates what communication(in-the-wild) malware communication patterns and instead

frequency looks like over the course of an hour when aemulate patterns of periodic communications. With thisuser accesses harmless Web servers (left) and when mal-approach, simply by generating a range of conceivable ma-ware is continually communicating with a C2 server (right).

licious patterns and training our model on them, we should

Different communication patterns almost always arise, so

be able to detect malware even for which no real-world sam-

we thought that if our system could learn the differences,

ples are available. This is what we mean by the term “zero

0

1

2

3

4

5

6

7

8

9

0

9

0

0

0

0

0

0

0

0

0

10

0

0

0

0

8

1

0

0

0

20

0

0

0

0

0

0

0

0

30

0

0

0

0

0

0

0

40

0

0

0

0

0

0

50

0

0

0

0

0

0

(min.)

0

1

2

3

4

5

6

7

8

9

0

1

0

1

0

1

0

1

0

1

0

0

10

1

0

1

0

1

0

1

0

1

0

0

0

20

1

0

1

0

1

0

1

0

1

0

0

0

0

30

1

0

1

0

1

0

1

0

1

0

0

0

0

0

40

1

0

1

0

1

0

1

0

1

0

0

0

0

0

50

1

0

1

0

1

0

1

0

1

0

(1) To a legitimate Web server(min.)(2) To a C2 server

Figure 2: Illustration of Communication with a Legitimate Web Server (left) and Continual Communication with a C2 Server (right)*3

Some recent malware samplesreceive sleep times from a C2 server. Only when the attacker is active, sleep times are short and communication is frequent. At other times,
the programs sleep for long periods. These sorts of techniques can make it difficult to detect the anomalous communication based on, for example, daily average times.

© Internet Initiative Japan Inc.

11

knowledge” in the subtitle of our Black Hat presentation.

see if our system can detect these. Our investigation covers

We emulate patterns for a wide range of intervals, from 3

the following malware families*5.

seconds up to 12 minutes (Figure 3). As a special case, we
also generate patterns that include sleeps of several minutes• PlugX

following several minutes of continuous activity (Figure 4).

• Asruex

Additionally, to account for patterns that differ only slightly• xxmmfrom the ones we have come up with and to better resist• himawari/ReadLeaves

CNN attacks , we also (a) apply rotations that shift each dot• ChChesin the generated patterns along a number of intervals and (b)• Elirks

randomly set existing values to zero. We thus generate a total• Logedrutof around one million patterns.

• ursnif/gozi*4• Shiz/Shifu
Moving on to our test dataset, we use around 4.5 million im-• Vawtrak

ages constructed from Web proxy logs (in the same manner• KINS

as for our training dataset) as our benign samples. For our
malicious samples, we use images constructed from logs ofUsing the model we built, we were able to detect all of thesemalware communications taken from in-the-wild incidents tomalware families. And as shown below, the rates of false

0

1

2

3

4

5

6

7

8

9

0

20

20

20

20

20

20

20

20

20

20

10

20

20

20

20

20

20

20

20

20

20

(min.)

...

0

1

2

3

4

5

6

7

8

9

0

1

0

0

0

0

0

0

0

0

0

10

0

0

1

0

0

0

0

0

0

0

20

0

0

0

0

1

0

0

0

0

0

20

20

20

20

20

20

20

20

20

20

20

30

20

20

20

20

20

20

20

20

20

20

30

0

0

0

0

0

0

1

0

0

0

40

20

20

20

20

20

20

20

20

20

20

40

0

0

0

0

0

0

0

0

1

0

50

20

20

20

20

20

20

20

20

20

20

50

0

0

0

0

0

0

0

0

0

0

Emulation of communication once every 3 seconds(min.)

Emulation of communication every 12 minutes

Figure 3: Emulation of Malicious Communications(min.)

0

1

2

3

4

5

6

7

8

9

0

1

1

1

0

0

0

1

1

1

0

0

10

0

0

1

1

1

0

0

0

1

1

1

1

20

1

0

0

0

1

1

1

0

0

0

1

0

0

30

1

1

1

0

0

0

1

1

1

0

0

0

1

1

40

0

0

1

1

1

0

0

0

1

1

1

1

0

0

50

0

0

0

0

1

1

1

0

0

0

0

1

2

3

4

5

6

7

8

9

0

1

1

0

0

1

1

0

0

1

1

10

0

0

1

1

0

0

1

1

0

20

1

1

0

0

1

1

0

0

30

0

0

1

1

0

0

1

40

1

1

0

0

1

1

50

0

0

1

1

0

0

2 min. of continual communication followed by 2-min. sleep(min.)...

3 min. of continual communication followed by 3-min. sleep

Figure 4: Emulation of Malicious Communications (2)*4*5Simple Black-Box Adversarial Perturbations for Deep Networks (https://arxiv.org/abs/1612.06299).
We do not obtain patterns simply by acquiring and running malware samples in a closed environment; instead, we use patterns obtained from actual incidents inwhich malware was connected to a C2 server. This is because the sleep times observed when malware is connected to an in-the-wild C2 server may differ fromthose observed when it is simply run in a closed environment (see footnote *3).

12

© Internet Initiative Japan Inc.

May 2019

Vol.

42

2. Focused Research (1)positives for our benign samples were also low. So we be-That said, it is conceivable that Web pages that frequently

lieve our system will be effective if we filter out these FQDNs

reload, such as Web mail interfaces and sports sites, could

using a whitelist.

register as false positives. So to reduce false positives when
the system is in operation, we can take steps like excluding• Benign sample set 1

such sites by whitelisting them or regarding communication

Accuracy: 1,565,139/1,566,109 (99.94%)

with a Web server as legitimate when alerts from many

False positive FQDNs: 64/246,190

users are raised for that same destination.

• Benign sample set 2
Accuracy: 1,540,419/1,541,050 (99.96%)

Figures 5–9 show examples of communication patterns suc-

False positive FQDNs: 72/243,106

cessfully detected from actual malware communications. It• Benign sample set 3

is evident that communications from the actual samples are

Accuracy: 1,528,936/1,529,617 (99.96%)

not perfectly periodic, but the system takes the differences inFalse positive FQDNs: 65/243,185

stride and detects the patterns using deep learning. Logedrut

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0

6

6

0

3

6

3

0

6

6

0

0

96

95

92

96

95

97

96

97

101

95

10

3

6

3

0

6

6

0

3

7

2

10

97

93

95

96

95

98

92

93

95

101

20

0

6

6

0

3

7

2

0

6

6

20

96

95

94

93

88

98

95

97

97

96

30

0

3

8

1

0

6

6

0

3

8

30

97

88

94

96

94

101

98

97

97

96

40

1

0

6

6

0

3

8

1

0

6

40

95

95

91

93

91

101

96

100

97

89

50

6

0

4

8

0

0

7

5

0

5

50

92

94

96

98

94

98

98

92

94

95

(min.)(min.)

Figure 5: PlugX Communication Pattern

0

1

2

3

4

5

6

7

8

9

0

0

0

0

0

0

0

0

0

0

4

10

0

2

0

2

0

2

0

0

2

20

0

2

2

0

2

0

0

2

30

0

2

0

0

2

0

2

40

0

2

0

0

2

0

50

2

0

2

0

2

0

0

1

2

3

4

5

6

7

8

9

0

1

0

1

1

0

1

1

0

1

1

0

10

0

1

1

0

1

0

1

0

1

0

0

2

20

1

0

1

0

1

0

0

1

1

0

0

2

0

30

0

1

1

0

0

1

1

0

0

1

0

2

0

2

40

1

0

0

1

0

1

0

1

0

1

2

2

0

2

50

0

1

0

1

0

0

1

1

0

0

(min.)

Figure 6: Asruex Communication Pattern(min.)

Figure 7: Elirks Communication Pattern

0

1

2

3

4

5

6

7

8

9

0

1

0

0

0

1

1

0

0

0

1

0

10

1

0

0

1

1

0

0

0

1

1

0

1

20

0

0

0

1

1

1

0

0

1

0

0

0

0

30

1

0

0

1

0

1

0

0

1

0

0

0

0

0

40

1

0

0

1

0

0

0

0

0

0

0

0

0

0

50

0

0

0

0

0

0

0

0

0

0

0

1

2

3

4

5

6

7

8

9

0

0

0

0

0

0

0

1

0

0

0

10

0

0

0

0

0

0

0

1

0

20

0

0

0

0

0

0

0

0

30

0

0

0

0

0

0

0

40

1

0

0

0

0

0

50

0

1

0

0

0

0

(min.)

Figure 8: Logedrug Communication Pattern(min.)

Figure 9: Vawtrak Communication Pattern© Internet Initiative Japan Inc.

13

(Figure 8) communicates infrequently, only once every 12

For space reasons, we do not show all malware families andminutes, but it was still detected as being distinct from themodel details here. Further details can be found in our slides*1

benign sample. In the case of Vawtrak (Figure 9), no com-on the Black Hat Europe 2018 website. Since we are notmunications occur in the final 16 minutes, but the systembuilding complicated models here, we believe the models candetects the malware in cases like these as well.

be trained on CPU-based systems.

1. Landing

2. Exploit

3. Malware

.HTML

.SWF

.EXE

ek.example.com

Figure 10: Sequence of Content-Types Sent by an Exploit Kit Server

Substance

Content-Type

URL path & parameters

Landing pagetext/html/?
NTI0OTU5&RCDUIv&oJhtJNm=dGFraW5n&wouMDc=Y2FwaXRhbA==&JgtXjOEttIAHrI=Y2FwaXRhbA==&TKCcodYFxdiy=dGhpbmdz&tNDodvGjF=Y2FwaXRhbA==&pHtonQrvp=bG9jYXRlZA==&kl345dfdfg234fsd=UDQTpjkGELQNmyN9ZAF1G9P2s
3EeBzhWZiMHT-RTZZA4QrZSQR7Rt3VzyxrckQPskg1TH6mI&pWjLlCBUIUSRIw=Y2FwaXRhbA==&nR45dsgd54lsCs=xXrQ
MvWfbRXQDJ3EKvjcT6NAMVHRGUCL2YqdmrHXefjaf1WkzrfFTF_3ozKATASG6_ZtdfJ

Substance

Content-Type

URL path & parameters

Flash Exploit

application/x-shockwave-flash/?
NTQ0NjEw&zWuWFX&lskPeVWn=dW5rbm93bg==&NCDmQdmxCxapA=dW5rbm93bg==&eLCxfNVxDhHqBH=Y29uc2lkZXI=&nzZHzkCNdL=cmVwb3J0&HZELKhjPUenym=cG9wdWxhcg==&nR45dsgd54lsCs=wnrQMvXcKxXQFYbDKuXDSKZDKU7
WG0aVw4-dhMG3YpjNfynz1ezURnL1tASVVFiRrbMdKL&kl345dfdfg234fsd=VYOQfk20LUKgEzm9sJVFhBo66tjUmDmBCd1
JLX-UeLMg9DqZOSHbIL0Vz0zLMRQIgigECy&rZpDUeqxIDnMQL=bG9jYXRlZA==&LENxPZQZ=cmVwb3J0

Substance

Content-Type

URL path & parameters

Malwareapplication/x-msdownload/?
MjEwNzA1&mTONXmiGJttk&nR45dsgd54lsCs=wXrQMvXcJwDQDobGMvrESLtGNknQA0KK2Iv2_dqyEoH9fWnihNzUSkr16
B2aCm3W&UEiQzsUEYQeeS=Y2FwaXRhbA==&jeeGWAgbhZSFoHh=bG9jYXRlZA==&KRssZN=bG9jYXRlZA==&BWeciQa
XKEgAey=bG9jYXRlZA==&SOymAmL=cG9wdWxhcg==&uLNyyCiGt=cG9wdWxhcg==&wlNBeZFOQXgP=dW5rbm93bg==&
kl345dfdfg234fsd=_fcpKeRXaVKziULVLwczyIlbUVJFpqj6i0SAmxDPhcGD_hKEUQ1M-5KREYFmmF7F

Figure 11: Example of Rig Exploit Kit’s Content-Type Sequence

Substance Content-Type

URL path &parameters

Landing
page/sm/

text/html

Substance Content-type

URL path &parameters

Substance Content-Type

URL path &parameters

Substance Content-Type

URL path &parameters

Flash loader
script/sm/swfobject.js

IE Exploit/sm/main.html

Java Exploit/sm/NeIsFp.jar

application/x-javascript

text/html

Substance Content-Type
MalwareURL path &parameters

application/octet-stream /dwm.exe

Figure 12: Example of KaiXin Exploit Kit’s Content-Type Sequence

14© Internet Initiative Japan Inc.

application/java-archive42

May 2019

Vol.

2. Focused Research (1)2.4 Exploit Kit Detection

Figures 11 and 12 show examples of the Content-Type

When a PC accessing the Web is redirected to an exploit

sequences when, respectively, Rig Exploit Kit and KaiXinkit, the exploit kit server sends content in the order shown

Exploit Kit are observed. The figures show that theirin Figure 10.

Content-Type transitions are indeed as described above.

1.Landing page: Identifies the PC’s Web browser environ-On the other hand, we can think of almost no cases inment and loads the next stage of exploit content. May

which normal Web browsing would produce these sorts ofalso include an exploit(s) for the browser itself. The

sequences in Content-Type sent by a server. In large-scale

Content-Type is text/html.

Web services, for example, dedicated servers are typically

2.Exploit content: Content file containing exploit(s) for

set up to handle each Content-Type, so content that is sub-the browser and its plug-ins. Content-Types include ap-

ject to exploits, such as Flash and Java, and HTML content

plication/x-shockwave-flash, application/x-java-archive,

like landing pages tend to come from different servers, asapplication/x-silverlight-app, application/pdf.

shown in Figure 13. And in cases where a single server

3.Malware: If the previous stage’s exploit(s) succeeds,

hosts all of a service’s content, data like images and CSS,

malware that infects the PC itself is loaded. In mostwhich recent exploit kits do not use all that much, comecases, the Content-Type is application/octet-stream orfrom the same server, as shown in Figure 14.

application/x-msdownload.

.HTML

.CSS

.CSS

.JS

.HTML

.CSS

.JS

static.example.com

.PNG

.JPG

.SWF

.PNG

.SWF

.JPG

.PNG

img.example.com

Figure 13: Example Content-Type Sequences for a Web Service with Separate Servers for Content-Types

.HTML

.JS

.CSS

.HTML

.CSS

.HTML

.SWF

.JPG

.PNG

www.example.org

Figure 14: Example Content-Type Sequence for a Service Hosted on a Single Web Server© Internet Initiative Japan Inc.

15

Given the above, we thought that if analysis of Web proxyclient PC – destination server pairs and convert series oflogs could differentiate between the sequences for exploit

requests and responses into sequences*6. To improve noise

kit servers and the sequences for normal Web server con-tolerance, we eliminate duplicate Content-Types from within

nections, it might be possible to detect exploit kits without

each sequence. We also limit sequences to a length of five*7,

relying on techniques like pattern matching. The Content-

deleting any lines beyond that. Finally, we convert each line

Type sequences peculiar to the exploit kits mentioned abovein the sequence to an 84-dimension vector. The first 83

are strongly related to the fundamental functions by whichslots in the vector represent the Content-Type in one-hot

exploit kits force a Web browser to run exploits and infectencoding, and the final slot is a flag indicating whether thea PC with malware. We should, therefore, be able to detectreferer and request URL contain the same domain name.

unknown exploit kits that operate in a similar manner. For thesame reason, it should not be easy for an exploit kit author to

Our training dataset comprises a benign sample of aroundevade detection by altering the sequence.

580,000 sequences constructed from some 3.9 million lines
from Web proxy logs, and a malicious sample of aroundTo differentiate between sequences, we use a class of neu-

300,000 sequences designed to emulate conceivable ex-

ral networks called RNNs (recurrent neural network), which

ploit kit patterns. Instead of using patterns observed in theare used in natural language processing and the processing

wild for our malicious sample, we generate a comprehensiveof time-series data such as video and audio streams. So first,

range of patterns representing content sequences that could

we need to convert the Web proxy logs into a form that anconceivably be produced by exploit kits. Figure 15 shows

RNN model can process. We split the logs into individual

examples of such pseudo-sequences. Our sample includes

A

.HTML

B

.HTML

C

.HTML

D

.HTML

.SWF

.EXE

.JS

.SWF

.SWF

.JAR

DATA

Typical sequence for recent exploit kits

.EXE

Exploits succeed multiple times(exploit in landing page and separately loaded Flash exploit)

DATA

Multiple exploit content downloaded.SWF

Exploit unsuccessful, no malware loaded

Figure 15: Examples of Generated Pseudo-sequences for Exploit Kits

*6

To be precise, we split log lines up according to client PC – destination server pairs, and then further split them into Web browser sessions (being the series ofrequests and responses caused by a Web browser in order to display the Web page at a given URL). The Web proxy logs we used have each Web browser session
recorded separately. In more common environments, it is possible to split up Web proxy logs according to fields such as the timestamp.

*7

With the exploit kits we have observed recently, in almost no cases is content sent by the server more than five times. However, if such cases were to rise in
future, we think the upper limit would need to be raised.

16© Internet Initiative Japan Inc.

May 2019

Vol.

42

2. Focused Research (1)sequences that represent cases where several types of ex-• Benign sample set 1

ploit content are loaded, cases where an exploit succeeds

Sequences: 562,390

multiple times in a row, and cases where the exploit is un-

False positives: 642

successful and no malware download takes place.

Accuracy: 0.9988• Benign sample set 2

To test our model, we used a malicious sample constructed

Sequences: 574,452

from actual communication data for the following 14 exploit

False positives: 681

kits, and a benign sample of around 1.7 million sequences

Accuracy: 0.9988

constructed from Web proxy logs for a time period that dif-

• Benign sample set 3

fers from that of the training set.

Sequences: 576,294
False positives: 639• Rig

Accuracy: 0.9988• Nebula• Terror

We have confirmed that using a whitelist of around 15 lines• Sundown

can halve the number of false positives listed above. When

• KaiXin

applying a system like ours to production environments, we• Neutrino

would recommend combining it with other methods to nar-

• Anglerrow down the alerts, such as whitelists, host reputation,• Nuclear

anomaly detection, and automated sandbox analysis.

• Magnitude• Fiesta

Our slides*1 available on the Black Hat Europe 2018 website• Sweet Orange

also present a method of identifying Rig Exploit Kit from

• GoonWeb proxy logs using an MLP (multilayer perceptron) model.

• Infinity

This method focuses on features of individual exploit kits’• Astrum

URLs, so while it is not suited to detecting unknown exploit
kits, it is useful for identifying known exploit kits and track-

The model we built was able to detect all of the exploit kits

ing their variants. Using it with the RNN-based exploit kit

listed above. And as shown below, false positive rates for

detection method described here can improve the accuracy

our benign samples were also relatively low.

of detection of known exploit kits.

Hiroshi Suzuki
Malware & Forensic Analyst, Office of Emergency Response and Clearinghouse for Security Information, Advanced Security Division, IIJ
As a member of IIJ-SECT, Mr. Suzuki is a malware analyst and a forensic investigator. He has dedicated over 13 years to the areas. As afrequent speaker and trainer for international conferences, he has given presentations and trainings at Black Hat (USA, Europe and Asia) and
FIRST TC multiple times.

Hisao Nashiwa
Threat Analyst, Office of Emergency Response and Clearinghouse for Security Information, Advanced Security Division, IIJ
Mr. Nashiwa is a member of IIJ-SECT, which is IIJ’s private CSIRT.
His work includes incident response, malware analysis and network traffic analysis, and he has thus been investigating malicious activities
over nine years. He has been researching cyber crimes such as those involving exploit kits and malware for many years and has expertisein malware analysis. He is a frequent conference speaker and has given talks and hands-on training sessions multiple times at international
conferences such as Black Hat and FIRST TC.

© Internet Initiative Japan Inc.

17

3. Focused Research (2)Designing a Large-scale Email System

3.1 Introduction

3.2 Challenges and Goals of the System Overhaul

In April 2017, we conducted a full system overhaul for IIJ

To begin with, we set a number of goals in view of issues

Secure MX Service (SMX), which was launched in Octoberwith the old delivery system.

2006. The SMX service’s primary focus is on providing
email gateway functionality. Before customers’ systems

3.2.1 Review of the Architecture

receive emails, the emails first pass through IIJ’s email

The first goal was to revise the old architecture, which had

servers, where email-borne threats are filtered out using abeen expanded excessively. Classic large-scale email sys-

variety of technologies, including virus filtering, spam fil-tems commonly use architectures that line up simple mailtering, sender authentication filtering, backscatter filtering,

servers (message transfer agent, MTA) in series as shownand sandbox filtering. The safe emails are then delivered toin Figure 1. Pre-overhaul, SMX’s delivery system also usedcustomers’ email systems.

a similar architecture.

Over 10 years have passed since launch, and the landscapeThe biggest benefit of multistage MTA architectures like thisfor emails services has changed substantially over that time,is the high extensibility. Delivery system functionality can bein terms of the volume and size of emails that pass throughexpanded easily by linking in MTAs with the desired addi-

as well as other factors, from server specs through to email

tional functionality. Also, because the MTAs are connected

system requirements. SMX has seen system expansions

via SMTP, the standard protocol for email transmissions,on top of expansions to cope with those changes and thus

linking in products from different vendors does not raise any

now has a vastly different makeup from the system that itconcerns in terms of interface compatibility, and it is gen-started out as. In many cases, however, fundamental as-

erally rare for product mixing and matching to cause anypects of system architecture can be traced all the way backproblems.

to launch, and we have thus felt for quite a while now thatthe system has its limitations.

This method of expanding a system, however, comes withside effects and should thus not be overdone. The biggest

In our recent overhaul, therefore, we revised the entire

side effect of concern is the increases in storage I/O andsystem from the architecture up. This report describes theoperating costs associated with adding more MTAs. In

design of the SMX email system, particularly its delivery

multistage MTA delivery systems, received data is written

system, along with some background to the revisions.

to storage every time an email passes through an MTA,

Customer email system
Internet

Senderauthentication

MTA

Relay

Relay

Relay

Antivirus

Antispam

Sandbox

MTA

MTA

MTA

MailboxFigure 1: Architecture of a Classical Email System

18© Internet Initiative Japan Inc.

May 2019

Vol.

42

3. Focused Research (2)resulting in storage I/O volumes several times that of the

3.2.2 Better Filtering

email size. Emails with virtually the same content are writ-

The second goal was to improve the accuracy of virus filter-ten to storage over and over, so taken as a whole, thising (antivirus) and spam filtering (antispam). Antivirus andsort of delivery system architecture results in a plethora ofantispam functions are crucial and constitute the dual cen-

unnecessary storage I/O operations. When network storage

terpiece of email security services.

solutions such as NAS and SAN are included, these I/O operations alone put a strain on the storage network, creating

While many security vendors provide both antivirus and anti-

a bottleneck for the delivery system as a whole. Not only

spam services and products, the security industry is subject

was storage a bottleneck for the old SMX delivery system,

to rapid change; new attack methods constantly arise, andthe system was also unable to take full advantage of ad-vendors are constantly developing new technologies tovances in hardware speed and capacity because CPU andcounter them. What this means is that Vendor A’s productmemory resources were being left idle.

may offer high detection accuracy on one day, while Vendor
B’s service may offer high detection accuracy on another,

Also, lining up different MTAs in series means filling theand then the situation may suddenly change when Vendor

delivery system with a mix of various MTAs that operate in

C releases a new product offering superb accuracy. Put dif-different ways. Not only does this increase the knowledge

ferently, this also means that sticking with any one vendor’s

required—in terms of, e.g., how to view the logs, perform

engine comes with the risk of declining detection accuracy

tasks, and deal with problems, and the build procedures toas the technology used by the engine becomes obsolete. In

follow when expanding the delivery system—each MTA canview of this, we felt we needed a mechanism for keeping

also create forks in the delivery route, send out notification

virus and spam email detection accuracy high.

emails, and so on, which complicates the flow of emails
within the delivery system and makes it tough to understand3.2.3 Avoiding Over-reliance on Any One Vendor

the delivery system as a whole.

The third goal was to avoid over-reliance on any specificvendor. To provide a wide range of functionality, the SMXWith SMX, the delivery system was repeatedly expanded

system incorporates numerous vendor products and ser-as scale increased, making for a complicated and expensive

vices. Although many of the products and services offereddelivery system, further expansion of which had become

are appealing, to avoid vendor lock-in, we felt we needed todifficult.

retain control over how reliant we are on any single offering.

© Internet Initiative Japan Inc.

19

It is not uncommon, particularly among foreign vendors,

processing tasks, making it possible to more efficiently use

for a company to be suddenly bought out by a competi-

CPU resources across the entire delivery system (Figure 3).

tor, resulting in its products and services being terminated.
Suddenly eliminating the ability to use a product has no

Because SMX employs multiple antivirus and antispam

small impact on customers who incorporate it into their sys-engines, the single-stage MTA architecture requires sev-

tems. Although we cannot reduce the impact of this to zero,eral engines to be loaded into any one server’s memory.

we do need to take steps to minimize it.

In general, antivirus and antispam engines hold a lot ofdata in memory and thus tend to consume a lot of mem-

3.3 Single-stage MTA for Effective Use 		of Hardware Resources

ory resources. Loading several such engines into memory

In overhauling the delivery system in line with the above

servers could not accommodate, but the increasingly highgoals, we first sought to harness the plentiful CPU andperformance of commodity server hardware , as mentioned

memory resources afforded by the increasingly high perfor-

above, has made this sort of setup possible.

necessitates the sort of total memory capacity that older

mance of commodity server hardware and to reduce storage
I/O, which was a bottleneck for the system overall as well
as a factor in costs.

3.4 Making Antivirus/Antispam			Engines Interchangeable
Next, we designed the overall system to allow the antivirus

The architecture we arrived at is the polar opposite of theand antispam engines to be replaceable at any time.

previous one. In short, all processing will be completed
within a single, multifunctional MTA, with this single-stage

IIJ does not develop antivirus or antispam engines in-house

setup eliminating the unnecessary relaying of emails be-

but instead provides this functionality by incorporating such

tween MTAs (Figure 2). With this architecture, email isengines from security vendors into the delivery system.

only written to storage once, so storage I/O operations areThis means that IIJ cannot directly address any problems

greatly reduced compared with the old delivery system,that arise with detection accuracy. Taking the opposite

which wrote the same content to storage multiple times.

perspective, however, by taking advantage of the ability to
cut obsolete technology loose and swiftly incorporate fresh

Also in the old delivery system, the storage I/O bottleneck

technology, we aimed to design a system that is not tightly

meant that the CPU resources of most servers were left idle

tied to any specific antivirus or antispam engine.

while CPU utilization rates were high on only some servers, where high-CPU-load tasks such as virus scans wereFirst, we evaluated the antivirus and antispam engines ofbeing run. By using identically configured MTAs in parallel,each security vendor. We ran scans for viruses and spam

previously idle CPU resources can be diverted for high-loadon emails received by honeypots run by IIJ, accumulating

Sender authentication

Antivirus

Customer email system

Internet
Antispam

Sandbox

MTA
Figure 2: Architecture of the Overhauled Email System

20© Internet Initiative Japan Inc.

Mailbox

May 2019

Vol.

42

3. Focused Research (2)statistics on detection performance over several months toafter they begin to circulate. Engine A may be quicker tofacilitate comparisons. Virus and spam emails can exhibit ep-

detect such emails during one campaign, while Engine B

idemic-like qualities, so any evaluation of countermeasures

may be quicker during another, and so on. Combining mul-over short periods can be heavily influenced by detection

tiple engines enables us to reduce detection misses duringperformance with respect to whatever spam campaign*1 hap-

the early stages of a campaign. In the case of antispam

pened to be active at the time, and thus may not provide aengines, in particular, we combine engines that use different

proper idea of long-term detection performance. This is why

approaches so that the weaknesses of any one engine are

we used a fairly long validation period. During the validationcompensated for by other engines.

period, we received product guidance from a range of security vendors. The approaches taken for antispam engines,

Our main aim in combining multiple engines was to improvein particular, are quite distinct across the different products

detection accuracy, but it also had secondary effects. One

available, and the results of our validation exercise are veryis the reduction of scan errors. It is not uncommon for virusinteresting in that they reflect the differences in approach. It

and spam scans to fail to complete successfully because of

was an arduous process, but it gave us an understanding ofcorrupted email headers/attachments or deliberate content

the detection accuracy and tendencies of each engine.

manipulation, for instance. Running scans with multiple engines makes it possible to significantly reduce the number

To achieve enhanced detection accuracy, we combine, re-of emails that cannot be scanned at all. Also, in rare cases,

spectively, multiple antivirus engines and multiple antispam

antivirus engines and antispam engines can crash whenengines. We then combine the results given by each to ar-scanning specific emails or attachments. In such cases, you

rive at the final detection results.

have the option of disconnecting the engine experiencing
the problem as an emergency measure to allow emails to

It is evident from the results of this validation process, andactually make it through the system. It also makes it possi-

also simply from intuition gained through day-to-day oper-

ble to minimize the impact of a security vendor being bought

ations, that no single engine is an out-and-out winner inout and its products becoming unavailable.

terms of how long it takes to detect virus and spam emails

BeforeAfter
Sender authentication
Antivirus
Antispam

Senderauthenticationzzz

Antivirus

Antispamzzz

Internet

Internet

Heavy load on one server’s CPU, other servers’ CPUs idle

Heavy processing loads can be allocated to all servers’ CPUs

Multistage MTA architecture

Single-stage MTA architecture

Figure 3: Effective Use of Hardware Resources*1

Mass mailout of identical or similar spam emails.

© Internet Initiative Japan Inc.

21

3.5 Decision to Develop MTA In-house

IIJ continues to actively expand SMX’s functionality with a

The biggest problem we had when overhauling the delivery

close eye not only on customer feedback but also on the lat-system was how to implement this design. This is what

est trends in the email and security industries. So the SMX

prompted our decision to develop the MTA in-house, butdelivery system may at times require functionality that other

other methods were available to us, such as combining open

operators do not require. In such situations, getting a ven-source MTAs like Postfix and Sendmail, or the use of MTAs

dor to add features that an MTA lacks is generally difficult.

made by MTA vendors.

MTA vendors serve many customers, so they will inevitably
put priority on developing functionality that many custom-

Postfix and Sendmail provide an interface called Milter,ers require, and on functionality that key customers want.

which provides advanced functionality and makes it possible

Developing highly idiosyncratic and niche functionality will

to easily and safely implement email control and rewrite ca-

naturally be of low priority. Vendor MTAs would appear topabilities. On the other hand, given the architecture, adding

be suitable when the required specifications are clear and

Milter onto the system would result in a large I/O overhead

when few specification changes are likely occur in future,in particular. And we also had to admit that it lacks the func-

but whether such offerings could support our proactive ap-

tionality for realizing a complex system like SMX. Althoughproach to expanding SMX is an unclear.

one idea could be to modify Postfix or Sendmail directly, thecost of keeping up with updates to the official package is far

Vendor MTAs also come with the risk of the vendor being

greater than you might imagine.

bought out. Because the system as a whole is heavily reliant
on the MTA, were such a corporate acquisition to occur,

The option of adopting a vendor-produced MTA seemed

the impact would be immense relative to any impact thatlike a very realistic one. Several MTA vendors develop MTA

might result in relation to the antivirus and antispam en-

products for ISPs and large-scale senders. In these products

gines. Indeed, several acquisitions involving MTA vendors

as well, the idea of performing all processing in a single

and products have taken place in the past few years. In

stage, as described above, is the mainstream approach.

absolute terms, not many vendors develop MTAs, so in per-They also allow various security vendors’ antivirus and an-

centage terms, this is a risk that cannot be ignored.

tispam engines to be swapped in and out, and they allow
flexible, extremely fine-grained customizations that meet

The final option is in-house development, but this is also

the detailed requirements of large-scale delivery systems,

not an easy option. MTAs that support the huge levels ofand as such, MTAs from vendors are far more versatilecommunication flows seen at the ISP level require extremely

and powerful than the open-source offerings. For us, thishigh levels of stability, robustness, and performance. Such awould have been the quickest and easiest way of creating

system would also need to provide the diverse functionality

the delivery system we had in mind. Indeed, many ISPs andand flexibility of a system like SMX. And the technical capa-

large-scale senders use vendor MTAs, and parts of SMX’s

bilities to support future expansions in functionality wouldold delivery system also incorporated vendor MTAs.

also be needed.

The only, and biggest, concern we had with taking on a

The decision to develop such an MTA from scratch in-house

vendor MTA was that SMX would become one with thatis perhaps not a very easy one. IIJ, however, does haveparticular MTA. In a single-stage MTA architecture, theexperience and knowhow in developing many email system

MTA itself is the delivery system. And the entire systemcomponents itself. It also has a battery of reliable develop-

beyond the delivery system, from the service specifications

ment teams, and this is why we were able to pull the trigger

through to the operation procedures, would depend heavily

on developing our system in-house—the risk was high, buton the MTA. In other words, what the MTA can do, SMX

we also saw that we had much to gain.

can do; and vice versa.

22© Internet Initiative Japan Inc.

May 2019

Vol.

42

3. Focused Research (2)3.6 Overhaul Outcomes

however, the operations and development teams can work

3.6.1 Achieving our Development Goals

closely with one another. This means that the root cause

The overhaul project encompassed the entire system, in-of any problems, in particular, can be identified extremely

cluding the MTA component. More than a full year passed

quickly, and proper provisional and permanent responses

before the first release was out. It was the biggest develop-

implemented.

ment project I have ever been part of.
Also, and while this is not something that affords direct comA lengthy development period and repeated testing wereparisons, I feel that developing the system in-house resulted

needed to complete the system, but ultimately it met all of

in strong motivation levels for the team. When developing a

the goals we set. Our flexible, versatile MTA made it pos-system that incorporates MTAs or other products from ven-sible to create a delivery system with a single-stage MTA

dors, the development team develops ways of interfacingarchitecture. In accord with our design, we reduced stor-

with those vendor products, but this frequently involves a

age I/O operations, which had been an issue with the oldseemingly futile struggle against unclear product specifica-

system, thereby enhancing the performance of the deliv-

tions and is often not much fun on a personal level. In-house

ery system overall. Major reviews of the virus filters anddevelopment, on the other hand, entails a far, far greaterspam filters also resulted in improved detection accuracy.

amount of work, and we were concerned about the bur-

And constantly keeping an eye on the detection rates ofden this would place on the development team. Yet, while

each engine we use has also enabled us to take swift action

the team was certainly busy, I never felt an atmosphere ofwhenever detection rates change.

fatigue or exhaustion setting in. In the end, I think there issomething exciting about building a large-scale system on

Developing an MTA in-house has freed us from the risk ofyour own and seeing it gradually come together.

MTA vendors being acquired, and it has also enabled us tominimize the effect of security vendors being acquired be-

We released the new system in April 2017, and we spentcause we are now able to swap engines in and out. Althougha full year migrating over to it from the old one. We have

perhaps quite obvious, I think the most important outcome

received various opinions and requests since release,of this system overhaul is that we have laid foundations

prompting us to add functions and fix issues, and I thinkthat will enable IIJ to run its own services as it sees fit, es-

we have been able to so swiftly make these improvements

sentially immune to the vicissitudes of vendor acquisitions.

precisely because we developed the system in-house.

3.6.2 Secondary Benefits

I would note that my intention is not to criticize systems

We also realized some secondary benefits beyond the goals

that use vendor products across the board. Both approaches

that we originally set.

have their advantages and disadvantages, so you need to
select the right balance depending on the situation. Finally,

First, troubleshooting speed improved. When problems occur

I would also note that SMX incorporates many vendor prod-

with a vendor MTA, details of the condition are reported toucts, including some from foreign vendors, into its system,

the vendor and a fix requested. But if it is unclear how to

and we work closely with vendors on a daily basis as we

reproduce the problem, or if the details cannot be passed toprovide our services to customers.

the vendor because they contain customer data, then it may
take considerable time for the vendor to confirm the problem

Here, I have described the design of the SMX delivery sys-or, as is often the case, the vendor may be unable to ascer-tem. With its newly acquired architecture, we will continue

tain what the problem is at all. With in-house development,

to evolve the SMX service going forward.

Takahiko Suzuki
Senior Engineer, Service Development Section, Application Service Department, Network Division, IIJ
Since joining IIJ in 2004, Mr. Suzuki has continued to work on email service development.
He is the developer of yenma, an open-source sender authentication filtering program.

© Internet Initiative Japan Inc.

23

42
May 2019 Vol.
About Internet Initiative Japan Inc. (IIJ)
IIJ was established in 1992, mainly by a group of engineers who
had been involved in research and development activities related
to the Internet, under the concept of promoting the widespreaduse of the Internet in Japan.
IIJ currently operates one of the largest Internet backbones
in Japan, manages Internet infrastructures, and provides
comprehensive high-quality system environments (includingInternet access, systems integration, and outsourcing services,etc.) to high-end business users including the government andother public offices and financial institutions.
In addition, IIJ actively shares knowledge accumulated throughservice development and Internet backbone operation, andis making efforts to expand the Internet used as a social
infrastructure.­­The copyright of this document remains in Internet Initiative Japan Inc.
(“IIJ”) and the document is protected under the Copyright Law of Japanand treaty provisions. You are prohibited to reproduce, modify, or makethe public transmission of or otherwise whole or a part of this document
without IIJ’s prior written permission. Although the content of thisdocument is paid careful attention to, IIJ does not warrant the accuracy andusefulness of the information in this document.
Initiative Japan Inc. All rights reserved.
© Internet
IIJ-MKTG020-0040

Internet Initiative Japan Inc.
Address: Iidabashi Grand Bloom, 2-10-2 Fujimi, Chiyoda-ku,
Tokyo 102-0071, Japan
Email: info@iij.ad.jp URL: https://www.iij.ad.jp/en/