Main sponsor

Micro Services – Java the Unix way
James Lewis

James Lewis
Senior Engineer / Principal Consultant / TAB memberYour Storyteller todayradarlatest Edition released last week

Micro-services
Embedded Servlet ContainersOr how we designed and nearly built a Resource Oriented, Event
Driven System out of applications about 1000 lines long…

WHAT I DID LAST SUMMER

In the beginning…
• There was a new product being developedby an organisation in London• The organisation had gathered their list ofhigh level requirements• And they asked ThoughtWorks if we could
help them design and build it…

So we took a look at theirrequirements• Me and my mates at ThoughtWorks• Worked out to be about 5000 points worthof User Stories– At 7 points per pair of developers per weekComplete

Half waythrough0

openedthe box

End day
1

Cows
come
homehellfreezes

pigs flyHeat
death oftheUniverse

Tip 1Divide and conquer
Start	  on	  the	  outside	  and	  model	  business	  capabili3es

ProductAnalytics

Real Time andBatch Interfaces

ProductMobile

ProductCMS

Product

Ecommerce

DemoSite

ExternalReporting

Product

3rd party Gateway

SMS Gateway

ProductProductProductProductConfig
Application

Call Centre
Application

Marketing
Application

Reporting
Application

A / E Services

Config Services

Account
Services

User Services

User / Role
Repository

Raw Txn Store

Account Store

Member Store

Product Data
Product
Product/ static Catalog
Repository
Product/ static Catalog
Product / static Catalog

Access and Entitlement

Batch Lifecyle
services

Config andMetadata
servcesRules Engine

Config /metadata store

Rules store

Batch Interface

Metadata

Each small box represents a capability,
composed of one or more services

Product
Rules Config
Application

Rules Engine

ReportingServices
Reportingdatastore
Reporting Services• The only way we could hit anything like thetimescales required was to scale theprogramme quickly• And that meant multiple teams in multiple
workstreams

And there were some, umm,
interesting non-functional
requirements tooProductAnalytics

Real Time andBatch Interfaces

Product

MobileProductCMS

Product

Ecommerce

DemoSite

External
Reporting

Product

3rd party Gateway

SMS Gateway

Product

Product

Product

Product

ConfigApplication

Call Centre
Application

Marketing
Application

Reporting
Application

A / E Services

Config Services

Account
Services

User Services

User / Role
Repository

Raw Txn Store

Account Store

Member Store

Product Data
Product
Product/ static Catalog
Repository
Product/ static Catalog
Product / static Catalog

Access and Entitlement

Batch Lifecyle
services

Config andMetadata
servcesRules Engine

Config /metadata store

Rules store

Batch Interface

Metadata

Thisbit

Product
Rules Config
Application

Rules Engine

ReportingServices
Reportingdatastore
Reporting Services

Had to handle 1000TPS with a 99th percentile latency of < 2 seconds

ProductAnalytics

Real Time andBatch Interfaces

Product

MobileProduct

CMS

Product
Demo
Site

Ecommerce

External
Reporting

Product

3rd party Gateway

SMS Gateway

Product

Product

Product

Product

ConfigApplication

Call Centre
Application

Marketing
Application

Reporting
Application

A / E Services

Config Services

AccountServices

User Services

User / Role
Repository

Raw Txn Store

Account Store

Member Store

Product Data
Product
Product/ static Catalog
Repository
Product/ static Catalog
Product / static Catalog

Access and Entitlement

Batch Lifecyle
services

Config andMetadata
servcesRules EngineConfig /metadata store

Rules store

Batch Interface

Metadata

Product
Rules Config
Application

Rules Engine

Support a user base of 100 million active customers

ReportingServices
Reportingdatastore
Reporting Services

ThisbitProductAnalytics

Real Time andBatch Interfaces

ProductMobile

ProductCMS

Product

Ecommerce

DemoSite

External
Reporting

Product

3rd party Gateway

SMS Gateway

ProductProductProductProductConfig
Application

Call Centre
Application

Marketing
Application

Reporting
Application

A / E Services

Config Services

Account
Services

User Services

User / Role
Repository

Raw Txn Store

Account Store

Member Store

Product Data
Product
Product/ static Catalog
Repository
Product/ static Catalog
Product / static Catalog

Access and Entitlement

Batch Lifecyle
services

Config andMetadata
servcesRules Engine

Config /metadata store

Rules store

Batch Interface

Metadata

Thisbit

Product
Rules Config
Application

Rules Engine

ReportingServices
Reportingdatastore
Reporting Services

Needed to support bulk loads of 30 – 90 million records nightly(and keep them for six months)

Did I mention PCI Level 1?

Finally, this is a product build.
So it needed to be modular /<cough> “infinitely configurable”
And deployable on Cloud and Tin

The product need to to be…
•Performance– fairly high throughput both transactional and batch•

Fault tolerant– One thing about the cloud, you are designing for failure right?

•Configurable– On a per install or PaaS basis•

Portable– Fortunately not to Windows…

•Maintainable– over multiple versions and years•

Supporting big data sets
– Billions of transactions available– Millions of customers available

and capable of being built quickly without sacrificing the other principles

Plus ça change, plus c'est la même chose.

(The	  more	  things	  change	  the	  more	  they	  stay	  the	  same)	  So, after five weeks we had broken the problem down into capabilties

Now we had to start scaling the teams to deliver these capabilities

Tip 2
Use Conway’s Law to structure teams“…organiza3ons	  which	  design	  systems	  …	  are	  constrained	  to	  produce	  designs	  which	  are	  copies	  of	  the	  communica3on	  structure	  of	  those	  organiza3ons”	  Melvin	  Conway,	  1968	  The first business capability - Users• Responsible for creation and maintenance ofusers in the system– Up to 100 million of them per instance of the product• Used by many clients with many usage patterns– Call centre and website – CRUD
– Inbound batch files – CRUD x hundreds of thousands
per night• Many downstream consumers of the data
– Fulfilment systems for example

Tip 3
The Last Responsible Moment

Don’t decide everything at the point you know leastaWe started with a business process…
NotifyExternalSystemInvalidFileNotify Sys
Admin

Batch
Enrolment
Results File

Batch Failed

Apply rules
Batch File

file
received

Validate filestructure

Validate Line
Item

Create User

User
CreatedSend Letter

Link AccountInvalid
User

Send results
file
Notify Sys
Admin

and noticed something funny…

Batchmonitoring

User
Monitoring

Fulfilment
Monitoring

Account
Monitoring

EventsMonitoring
Services

Batch
Event

Fulfilment
EventUser
Collection

Triggering
Triggering

Triggering

Batch
Processing
ServiceTriggering

Triggering

Triggering

BankAccountServiceFulfilment
ServiceRules EngineBank
Account
Creation

fulfilment

Rules store

BankAccount Storefulfilment

MemberService

Rules Engine

Address
Validation

Results File
Creation

UserValidation

Batch StoreUser
Creation

File Structure
Validation
Ad-hocUsers

User StoreTriggering

Triggering

EmailI know what you are thinking…

*ESB* Dan North coined the term Enterprise Night Bus…

Or you could use the web

Tip 4

Be of the web, not behind the web

RFC 5023 to be precise

/user-request
application/atom+json/user-request/1223

Eventqueue

Eventstore

UserService

User
store

Queue
processing
engineUsers Capability/users/142

application/vnd.user+JSON/users

and this is what we built

/user-request
application/atom+json/user-request/1223

Standard resource representations using well known web
standards – atom+json
Event
Event
queue

store

User
Service

User
store

Queue
processing
engineUsers Capability/users/142/usersapplication/vnd.user+JSON

/user-request
application/atom+json/user-request/1223

Reified the request to create a user. Clients POST a request
Event
to create a user as an entryEventto an atomcollection.
queue

store

User
Service

User
store

Queue
processing
engineUsers Capability/users/142/usersapplication/vnd.user+JSON

Tip 5
If something is important, make it anexplicit part of your design
Reify
to convert into or regard as a concrete thing: to reify aconcept.

/user-request
application/atom+json/user-request/1223

Event
queue

Event
store

Queue
Event queue has the
single responsibility of managing stateprocessing
enginetransitions for the request
to create a user

User
Service

User
store

Users Capability/users/142/usersapplication/vnd.user+JSON

/user-request
application/atom+json/user-request/1223

Event
queue

Eventstore

UserService

User
store

Queue
processing
engineUsers Capability

Queue Processing Engine implemented the Competing
Consumer pattern using
Conditional GET, PUT and Etags/users/142
application/vnd.user+JSON
against the atom collection exposed by the event queue/users/user-request
application/atom+json/user-request/1223

Event
queue

Eventstore

UserService

User
store

Queue
processing
engineUsers Capability

User Service and store is the system of record for users/users/142/usersapplication/vnd.user+JSON

/user-request
application/atom+json/user-request/1223

Event
queue

Eventstore

Queue
processing
engineAfter creation, representations of Users are available atcanonical locations in well defined formats and creation
UserUserevents added to anotheratom collection
Servicestore
Users Capability/users/142/users

application/vnd.user+JSON

Where they are available for consumption
by other downstream systems

Batchmonitoring

User
Monitoring

Fulfilment
Monitoring

Monitoring

Account
Monitoring

Monitoring
Services

Batch
Event

Fulfilment
Event

User
Collection

Triggering
Triggering

Triggering

Batch
Processing
ServiceTriggering

Triggering

Triggering

Bank
Account
Service

Fulfilment
Service

Rules EngineBank
Account
Creation

fulfilment

Rules store

Bank
Account Storefulfilment

MemberService

Rules Engine

Address
Validation

Results File
Creation

User
Validation

Batch StoreUser
CreationFile StructureValidation
Ad-hocUsers

User StoreTriggering

Triggering

EmailFulfilment

Our micro-services• User Request Queue– Forms the transactional boundary of the system• Request Queue Processor– Competing Consumer processes events on thequeue and POSTs them to• User Service– System of record for Users in the system– Responsible for all state changes of those users– Exposes events on those users to other systems

CHARACTERISTICS OF
MICRO-SERVICESSmall with a single responsibility• Each application only does one thing• Small enough to fit in your head
– James’ heuristic– “If a class is bigger than my head then it is toobig”• Small enough that you can throw them away
– Rewrite over Maintain

Containerless and installed as well
behaved Unix services• Embedded web container– Jetty / SimpleMind– This has a lot of benefits for testing (inproctester forexample) and eases deployment• Packaged as a single executable jar– Along with their configuration– And unix standard rc.d scripts• Installed in the same way you would install httpd or
any other application– Why recreate the wheel? Daemons seem to work ok foreverything else. Unless you are *special*?

Located in different VCS roots•

Each application is completely separate•

Software developers see similarities and abstractions– And before you know it you have One Domain To Rule Them All•

Domain Driven Design / Conways Law– Domains in different bounded contexts should be distinct – and its ok tohave duplication– Use physical separation to enforce this•

There will be common code, but it should be library andinfrastructure code
– Treat it as you would any other open source library– Stick it in a nexus repo somewhere and treat it as a binary dependency

Provisioned automatically• The way to manage the complexity ofmany small applications is declarative
provisioning– UAT:• 2 * service A, Load Balanced, Auto-Scaled• 2 * service B, Load Balanced, Auto-Scaled• 1 * database cluster

Status aware and auto-scaling• What good is competing consumer if you only
have one consumer?
– We don’t want to wake Peter up at three in themorning any more to start a new process• Use watchdog processes to monitor in-app
status pages– Each app exposes metrics about itself– In our case, queue-depth for example– This allows others services to auto-scale to meetthroughput requirements

/user-request
application/atom+json/user-request/1223

Event
queue

Eventstore

A single capability composed ofmany small applications andexposing a uniform interface of Atom
Collections
Queueprocessing
engineUser
Service

User
store

Users Capability/users/142/usersapplication/vnd.user+JSON

How the capabilities form a productThey interact via the uniforminterface• HTTP– Don’t fight the battles already won– Use no-brainer force multipliers like reverse proxies• HATEOS– Link relations drive state changes– Its an anti-corruption layer that allows the capability toevolve independently of its clients• Standard media types– Can be used by many different clients– You can monitor it using a feed reader if you want…

atom+json / HTTP (AJOH)

Monitoring
Capability

atom+XML / HTTP

Reporting
Capability(AJOH)(AJOH)

Fulfilment
Capability

User
Capability(AJOH)

Capabilities poll waiting(AJOH)
for events that they know(AJOH)
how to deal
with. Forming an eventually consistent systemInbound Batch

Call Centre
ExternalSuppliers

Tip 6Favour	  service	  choreography	  over	  orchestra3onatom+json / HTTP (AJOH)

Monitoring
Capability

atom+XML / HTTP

Reporting
Capability(AJOH)(AJOH)

Each is entirely decoupled from it’s clients, scalable, testableand deployable individuallyFulfilment
Capability

User
Capability(AJOH)(AJOH)

Inbound Batch(AJOH)

Call Centre
External
Suppliers

Tip 7

Use hypermedia controls to decouple servicesatom+json / HTTP (AJOH)

Monitoring
Capability

atom+XML / HTTP

Reporting
Capability(AJOH)(AJOH)

Each developed by a separate team,using whatever tech they choose
Fulfilment
CapabilityUser
Capability(AJOH)(AJOH)

Inbound Batch(AJOH)

Call Centre
ExternalSuppliersOur stack• Embedded Jetty (current project uses SimpleWeb)• PicoContainer for DI• Hibernate (but wrote our own SQL)
• Abdera for Atom
• Smoothie charts• Metrics @codehass• Graphite

Infrastructure automation stack• Fabric with boto
• AWS, but deployable to anything with SSH• Maven (boo)• Git• Puppet for provisioning

NO SILVER BULLETS

This stuff is hard
• We haven’t even talked about– Versioning– Integration– Testing– Deployment• Eventual Consistency can be tricky for people to getthere head around• Developers like using enterprisy software– No one got fired for choosing an ESB– Convincing people to use the web is hard

SUMMARYThe Unix Philosophy
Lions commentary on Unix 2nd edition:s/pipes/hJp/	  	  Consistent and reinforcing practices
Hexagonal Business capabilities composed of:Micro Services that you canRewrite rather than maintain and which form
A Distributed Bounded Context.
Deployed as containerless OS services
With standardised application protocols and message semantics
Which are auto-scaling and designed for failureIs hiring!

Thanks!
jalewis@thoughtworks.com@boicyhttp://bovon.org