3. Focused Research (2)Designing a Large-scale Email System

3.1 Introduction

3.2 Challenges and Goals of the System Overhaul

In April 2017, we conducted a full system overhaul for IIJ

To begin with, we set a number of goals in view of issues

Secure MX Service (SMX), which was launched in Octoberwith the old delivery system.

2006. The SMX service’s primary focus is on providing
email gateway functionality. Before customers’ systems

3.2.1 Review of the Architecture

receive emails, the emails first pass through IIJ’s email

The first goal was to revise the old architecture, which had

servers, where email-borne threats are filtered out using abeen expanded excessively. Classic large-scale email sys-

variety of technologies, including virus filtering, spam fil-tems commonly use architectures that line up simple mailtering, sender authentication filtering, backscatter filtering,

servers (message transfer agent, MTA) in series as shownand sandbox filtering. The safe emails are then delivered toin Figure 1. Pre-overhaul, SMX’s delivery system also usedcustomers’ email systems.

a similar architecture.

Over 10 years have passed since launch, and the landscapeThe biggest benefit of multistage MTA architectures like thisfor emails services has changed substantially over that time,is the high extensibility. Delivery system functionality can bein terms of the volume and size of emails that pass throughexpanded easily by linking in MTAs with the desired addi-

as well as other factors, from server specs through to email

tional functionality. Also, because the MTAs are connected

system requirements. SMX has seen system expansions

via SMTP, the standard protocol for email transmissions,on top of expansions to cope with those changes and thus

linking in products from different vendors does not raise any

now has a vastly different makeup from the system that itconcerns in terms of interface compatibility, and it is gen-started out as. In many cases, however, fundamental as-

erally rare for product mixing and matching to cause anypects of system architecture can be traced all the way backproblems.

to launch, and we have thus felt for quite a while now thatthe system has its limitations.

This method of expanding a system, however, comes withside effects and should thus not be overdone. The biggest

In our recent overhaul, therefore, we revised the entire

side effect of concern is the increases in storage I/O andsystem from the architecture up. This report describes theoperating costs associated with adding more MTAs. In

design of the SMX email system, particularly its delivery

multistage MTA delivery systems, received data is written

system, along with some background to the revisions.

to storage every time an email passes through an MTA,

Customer email system
Internet

Senderauthentication

MTA

Relay

Relay

Relay

Antivirus

Antispam

Sandbox

MTA

MTA

MTA

MailboxFigure 1: Architecture of a Classical Email System

18© Internet Initiative Japan Inc.

May 2019

Vol.

42

3. Focused Research (2)resulting in storage I/O volumes several times that of the

3.2.2 Better Filtering

email size. Emails with virtually the same content are writ-

The second goal was to improve the accuracy of virus filter-ten to storage over and over, so taken as a whole, thising (antivirus) and spam filtering (antispam). Antivirus andsort of delivery system architecture results in a plethora ofantispam functions are crucial and constitute the dual cen-

unnecessary storage I/O operations. When network storage

terpiece of email security services.

solutions such as NAS and SAN are included, these I/O operations alone put a strain on the storage network, creating

While many security vendors provide both antivirus and anti-

a bottleneck for the delivery system as a whole. Not only

spam services and products, the security industry is subject

was storage a bottleneck for the old SMX delivery system,

to rapid change; new attack methods constantly arise, andthe system was also unable to take full advantage of ad-vendors are constantly developing new technologies tovances in hardware speed and capacity because CPU andcounter them. What this means is that Vendor A’s productmemory resources were being left idle.

may offer high detection accuracy on one day, while Vendor
B’s service may offer high detection accuracy on another,

Also, lining up different MTAs in series means filling theand then the situation may suddenly change when Vendor

delivery system with a mix of various MTAs that operate in

C releases a new product offering superb accuracy. Put dif-different ways. Not only does this increase the knowledge

ferently, this also means that sticking with any one vendor’s

required—in terms of, e.g., how to view the logs, perform

engine comes with the risk of declining detection accuracy

tasks, and deal with problems, and the build procedures toas the technology used by the engine becomes obsolete. In

follow when expanding the delivery system—each MTA canview of this, we felt we needed a mechanism for keeping

also create forks in the delivery route, send out notification

virus and spam email detection accuracy high.

emails, and so on, which complicates the flow of emails
within the delivery system and makes it tough to understand3.2.3 Avoiding Over-reliance on Any One Vendor

the delivery system as a whole.

The third goal was to avoid over-reliance on any specificvendor. To provide a wide range of functionality, the SMXWith SMX, the delivery system was repeatedly expanded

system incorporates numerous vendor products and ser-as scale increased, making for a complicated and expensive

vices. Although many of the products and services offereddelivery system, further expansion of which had become

are appealing, to avoid vendor lock-in, we felt we needed todifficult.

retain control over how reliant we are on any single offering.

© Internet Initiative Japan Inc.

19

It is not uncommon, particularly among foreign vendors,

processing tasks, making it possible to more efficiently use

for a company to be suddenly bought out by a competi-

CPU resources across the entire delivery system (Figure 3).

tor, resulting in its products and services being terminated.
Suddenly eliminating the ability to use a product has no

Because SMX employs multiple antivirus and antispam

small impact on customers who incorporate it into their sys-engines, the single-stage MTA architecture requires sev-

tems. Although we cannot reduce the impact of this to zero,eral engines to be loaded into any one server’s memory.

we do need to take steps to minimize it.

In general, antivirus and antispam engines hold a lot ofdata in memory and thus tend to consume a lot of mem-

3.3 Single-stage MTA for Effective Use 		of Hardware Resources

ory resources. Loading several such engines into memory

In overhauling the delivery system in line with the above

servers could not accommodate, but the increasingly highgoals, we first sought to harness the plentiful CPU andperformance of commodity server hardware , as mentioned

memory resources afforded by the increasingly high perfor-

above, has made this sort of setup possible.

necessitates the sort of total memory capacity that older

mance of commodity server hardware and to reduce storage
I/O, which was a bottleneck for the system overall as well
as a factor in costs.

3.4 Making Antivirus/Antispam			Engines Interchangeable
Next, we designed the overall system to allow the antivirus

The architecture we arrived at is the polar opposite of theand antispam engines to be replaceable at any time.

previous one. In short, all processing will be completed
within a single, multifunctional MTA, with this single-stage

IIJ does not develop antivirus or antispam engines in-house

setup eliminating the unnecessary relaying of emails be-

but instead provides this functionality by incorporating such

tween MTAs (Figure 2). With this architecture, email isengines from security vendors into the delivery system.

only written to storage once, so storage I/O operations areThis means that IIJ cannot directly address any problems

greatly reduced compared with the old delivery system,that arise with detection accuracy. Taking the opposite

which wrote the same content to storage multiple times.

perspective, however, by taking advantage of the ability to
cut obsolete technology loose and swiftly incorporate fresh

Also in the old delivery system, the storage I/O bottleneck

technology, we aimed to design a system that is not tightly

meant that the CPU resources of most servers were left idle

tied to any specific antivirus or antispam engine.

while CPU utilization rates were high on only some servers, where high-CPU-load tasks such as virus scans wereFirst, we evaluated the antivirus and antispam engines ofbeing run. By using identically configured MTAs in parallel,each security vendor. We ran scans for viruses and spam

previously idle CPU resources can be diverted for high-loadon emails received by honeypots run by IIJ, accumulating

Sender authentication

Antivirus

Customer email system

Internet
Antispam

Sandbox

MTA
Figure 2: Architecture of the Overhauled Email System

20© Internet Initiative Japan Inc.

Mailbox

May 2019

Vol.

42

3. Focused Research (2)statistics on detection performance over several months toafter they begin to circulate. Engine A may be quicker tofacilitate comparisons. Virus and spam emails can exhibit ep-

detect such emails during one campaign, while Engine B

idemic-like qualities, so any evaluation of countermeasures

may be quicker during another, and so on. Combining mul-over short periods can be heavily influenced by detection

tiple engines enables us to reduce detection misses duringperformance with respect to whatever spam campaign*1 hap-

the early stages of a campaign. In the case of antispam

pened to be active at the time, and thus may not provide aengines, in particular, we combine engines that use different

proper idea of long-term detection performance. This is why

approaches so that the weaknesses of any one engine are

we used a fairly long validation period. During the validationcompensated for by other engines.

period, we received product guidance from a range of security vendors. The approaches taken for antispam engines,

Our main aim in combining multiple engines was to improvein particular, are quite distinct across the different products

detection accuracy, but it also had secondary effects. One

available, and the results of our validation exercise are veryis the reduction of scan errors. It is not uncommon for virusinteresting in that they reflect the differences in approach. It

and spam scans to fail to complete successfully because of

was an arduous process, but it gave us an understanding ofcorrupted email headers/attachments or deliberate content

the detection accuracy and tendencies of each engine.

manipulation, for instance. Running scans with multiple engines makes it possible to significantly reduce the number

To achieve enhanced detection accuracy, we combine, re-of emails that cannot be scanned at all. Also, in rare cases,

spectively, multiple antivirus engines and multiple antispam

antivirus engines and antispam engines can crash whenengines. We then combine the results given by each to ar-scanning specific emails or attachments. In such cases, you

rive at the final detection results.

have the option of disconnecting the engine experiencing
the problem as an emergency measure to allow emails to

It is evident from the results of this validation process, andactually make it through the system. It also makes it possi-

also simply from intuition gained through day-to-day oper-

ble to minimize the impact of a security vendor being bought

ations, that no single engine is an out-and-out winner inout and its products becoming unavailable.

terms of how long it takes to detect virus and spam emails

BeforeAfter
Sender authentication
Antivirus
Antispam

Senderauthenticationzzz

Antivirus

Antispamzzz

Internet

Internet

Heavy load on one server’s CPU, other servers’ CPUs idle

Heavy processing loads can be allocated to all servers’ CPUs

Multistage MTA architecture

Single-stage MTA architecture

Figure 3: Effective Use of Hardware Resources*1

Mass mailout of identical or similar spam emails.

© Internet Initiative Japan Inc.

21

3.5 Decision to Develop MTA In-house

IIJ continues to actively expand SMX’s functionality with a

The biggest problem we had when overhauling the delivery

close eye not only on customer feedback but also on the lat-system was how to implement this design. This is what

est trends in the email and security industries. So the SMX

prompted our decision to develop the MTA in-house, butdelivery system may at times require functionality that other

other methods were available to us, such as combining open

operators do not require. In such situations, getting a ven-source MTAs like Postfix and Sendmail, or the use of MTAs

dor to add features that an MTA lacks is generally difficult.

made by MTA vendors.

MTA vendors serve many customers, so they will inevitably
put priority on developing functionality that many custom-

Postfix and Sendmail provide an interface called Milter,ers require, and on functionality that key customers want.

which provides advanced functionality and makes it possible

Developing highly idiosyncratic and niche functionality will

to easily and safely implement email control and rewrite ca-

naturally be of low priority. Vendor MTAs would appear topabilities. On the other hand, given the architecture, adding

be suitable when the required specifications are clear and

Milter onto the system would result in a large I/O overhead

when few specification changes are likely occur in future,in particular. And we also had to admit that it lacks the func-

but whether such offerings could support our proactive ap-

tionality for realizing a complex system like SMX. Althoughproach to expanding SMX is an unclear.

one idea could be to modify Postfix or Sendmail directly, thecost of keeping up with updates to the official package is far

Vendor MTAs also come with the risk of the vendor being

greater than you might imagine.

bought out. Because the system as a whole is heavily reliant
on the MTA, were such a corporate acquisition to occur,

The option of adopting a vendor-produced MTA seemed

the impact would be immense relative to any impact thatlike a very realistic one. Several MTA vendors develop MTA

might result in relation to the antivirus and antispam en-

products for ISPs and large-scale senders. In these products

gines. Indeed, several acquisitions involving MTA vendors

as well, the idea of performing all processing in a single

and products have taken place in the past few years. In

stage, as described above, is the mainstream approach.

absolute terms, not many vendors develop MTAs, so in per-They also allow various security vendors’ antivirus and an-

centage terms, this is a risk that cannot be ignored.

tispam engines to be swapped in and out, and they allow
flexible, extremely fine-grained customizations that meet

The final option is in-house development, but this is also

the detailed requirements of large-scale delivery systems,

not an easy option. MTAs that support the huge levels ofand as such, MTAs from vendors are far more versatilecommunication flows seen at the ISP level require extremely

and powerful than the open-source offerings. For us, thishigh levels of stability, robustness, and performance. Such awould have been the quickest and easiest way of creating

system would also need to provide the diverse functionality

the delivery system we had in mind. Indeed, many ISPs andand flexibility of a system like SMX. And the technical capa-

large-scale senders use vendor MTAs, and parts of SMX’s

bilities to support future expansions in functionality wouldold delivery system also incorporated vendor MTAs.

also be needed.

The only, and biggest, concern we had with taking on a

The decision to develop such an MTA from scratch in-house

vendor MTA was that SMX would become one with thatis perhaps not a very easy one. IIJ, however, does haveparticular MTA. In a single-stage MTA architecture, theexperience and knowhow in developing many email system

MTA itself is the delivery system. And the entire systemcomponents itself. It also has a battery of reliable develop-

beyond the delivery system, from the service specifications

ment teams, and this is why we were able to pull the trigger

through to the operation procedures, would depend heavily

on developing our system in-house—the risk was high, buton the MTA. In other words, what the MTA can do, SMX

we also saw that we had much to gain.

can do; and vice versa.

22© Internet Initiative Japan Inc.

May 2019

Vol.

42

3. Focused Research (2)3.6 Overhaul Outcomes

however, the operations and development teams can work

3.6.1 Achieving our Development Goals

closely with one another. This means that the root cause

The overhaul project encompassed the entire system, in-of any problems, in particular, can be identified extremely

cluding the MTA component. More than a full year passed

quickly, and proper provisional and permanent responses

before the first release was out. It was the biggest develop-

implemented.

ment project I have ever been part of.
Also, and while this is not something that affords direct comA lengthy development period and repeated testing wereparisons, I feel that developing the system in-house resulted

needed to complete the system, but ultimately it met all of

in strong motivation levels for the team. When developing a

the goals we set. Our flexible, versatile MTA made it pos-system that incorporates MTAs or other products from ven-sible to create a delivery system with a single-stage MTA

dors, the development team develops ways of interfacingarchitecture. In accord with our design, we reduced stor-

with those vendor products, but this frequently involves a

age I/O operations, which had been an issue with the oldseemingly futile struggle against unclear product specifica-

system, thereby enhancing the performance of the deliv-

tions and is often not much fun on a personal level. In-house

ery system overall. Major reviews of the virus filters anddevelopment, on the other hand, entails a far, far greaterspam filters also resulted in improved detection accuracy.

amount of work, and we were concerned about the bur-

And constantly keeping an eye on the detection rates ofden this would place on the development team. Yet, while

each engine we use has also enabled us to take swift action

the team was certainly busy, I never felt an atmosphere ofwhenever detection rates change.

fatigue or exhaustion setting in. In the end, I think there issomething exciting about building a large-scale system on

Developing an MTA in-house has freed us from the risk ofyour own and seeing it gradually come together.

MTA vendors being acquired, and it has also enabled us tominimize the effect of security vendors being acquired be-

We released the new system in April 2017, and we spentcause we are now able to swap engines in and out. Althougha full year migrating over to it from the old one. We have

perhaps quite obvious, I think the most important outcome

received various opinions and requests since release,of this system overhaul is that we have laid foundations

prompting us to add functions and fix issues, and I thinkthat will enable IIJ to run its own services as it sees fit, es-

we have been able to so swiftly make these improvements

sentially immune to the vicissitudes of vendor acquisitions.

precisely because we developed the system in-house.

3.6.2 Secondary Benefits

I would note that my intention is not to criticize systems

We also realized some secondary benefits beyond the goals

that use vendor products across the board. Both approaches

that we originally set.

have their advantages and disadvantages, so you need to
select the right balance depending on the situation. Finally,

First, troubleshooting speed improved. When problems occur

I would also note that SMX incorporates many vendor prod-

with a vendor MTA, details of the condition are reported toucts, including some from foreign vendors, into its system,

the vendor and a fix requested. But if it is unclear how to

and we work closely with vendors on a daily basis as we

reproduce the problem, or if the details cannot be passed toprovide our services to customers.

the vendor because they contain customer data, then it may
take considerable time for the vendor to confirm the problem

Here, I have described the design of the SMX delivery sys-or, as is often the case, the vendor may be unable to ascer-tem. With its newly acquired architecture, we will continue

tain what the problem is at all. With in-house development,

to evolve the SMX service going forward.

Takahiko Suzuki
Senior Engineer, Service Development Section, Application Service Department, Network Division, IIJ
Since joining IIJ in 2004, Mr. Suzuki has continued to work on email service development.
He is the developer of yenma, an open-source sender authentication filtering program.

© Internet Initiative Japan Inc.

23