Patterns to Introduce Continuous Integration to
Organizations
Kenichiro Ota

Hiroko Tamagawa

Lei Wang

Shift inc.
Tokyo
Japan

Shift inc.
Tokyo
Japan

Shift inc.
Tokyo
Japan

oota_ken@hotmail.com
kenichiro.ota@shiftinc.jp

hiroko.tamagawa@shiftinc.jp

lei.wang@shiftinc.jp

ABSTRACT
Continuous Integration (CI) is to incrementally integrate softwaredevelopment work within a team by executing a build process
whenever changes of work occur. This build process includes
compilation, automated testing, static analysis and finallydeployment.
From our experiences of introducing the concept of CI to several
organizations applying to different projects, we noticed common
successful patterns and anti-patterns, which are further organized
into a pattern catalog. This paper uses pattern languages to
describe the relationship of those patterns.
We believe that people in such roles as team leaders, software
configuration management engineers, test automation engineersand process improvement engineers can benefit from our pattern
catalog when employing CI in their organizations.

Categories and Subject Descriptors
Continuous Integration

identification and resolution of anti-patterns, we explain patterns
that should be adopted.
We list categories of patterns by grouping them, and draw apattern map that describes the relationship of the patterns.

2. Pattern Category List
We classify patterns into management, introduction and operation
categories.
Table 1. Pattern Category List
Management

Anti-patterns

Patterns

Over expectation of
stakeholders

Temporary expedientand effective expedientInvolve architect
Local CI

Introduction

Agile Development

Automated test first

Procedure first

Automate all

Write tests for newcode

DevOpsStatic analysis
testability

Test Automation

General Terms

Operation

Development ProcessOver
dependency
on CI Server

for

Use existing codes asbaseline
Operation without CI
serverAutomation
Verification

Table 2. Category Description

1. INTRODUCTIONManagement

As agile development methodology prevails in contemporary
software development practices [1], CI, one of the key concepts inagile development practice has become prominent because it cangreatly reinforce software quality. But if organizations or teams
have no experiences of automations in CI, they may soon faceanti-patterns and fail the same way as dealing with automations inother domains.

Patterns of managing expectations andrequirements of stakeholder when introducing
CI

Introduction

Patterns of priorities
introducing CI

Operation

Patterns of scopes and metrics of operation ofCI

Coming along with CI, we will introduce several automations thatare static analysis, code coverage analysis, test automation,
packaging deployment and provisioning for test environment all
together which will be more difficult than just one singular
automation such as test automation though the subjects in bothcases are similar.
This pattern catalog explains typical anti-patterns that could
happen in the introduction of CI to organizations. Followed byandscopeswhen

3. Pattern Map
Pattern map describes relationship and orders of patterns. The
simplex arrow shows the order to adopt patterns: from an antipattern to a pattern, we can adopt a pattern after we identify andresolve its corresponding anti-pattern. Lines without arrows are
not directional, meaning there are feedbacks between two patterns.

It is recommended to start with the identification of Automatedtest first anti-pattern though the automation of testing process is abenefit from CI. In this case one should start to resolve
Automated tests first anti-pattern.
A better approach to demonstrating the advantages of CI is toadopt Temporary expedient and effective expedient pattern andthen to share the results with stakeholders.

Refactored solution

Specific solution for the anti-pattern

Relatedpatterns

Any other anti-patterns that are related tothis oneanti-

5. Anti-Patterns
In the introduction of CI, we sometimes have to confront antipatterns before we can adopt proper patterns. Therefore describing
those potential anti-patterns are necessary.

5.1 Management anti-patterns
5.1.1 Over expectation of stakeholders
NameOver expectation of stakeholders

Background

Investigation when introducing CI

Symptomandconsequences

If adopting Automated test first antipattern, one starts test automation firstand only focuses on it, then the
introduction of CI fails.
If adopting Automate all anti-pattern,
one automates all build process. Cost of
maintenance and operations will greatly
increase.

Figure 1. Pattern map of patterns to introduce CI

4. Pattern template

Anecdotal
evidence

If we install CI servers, then all
development process will be automated.

4.1 Template for patterns
We use the following templates for patterns.

If we introduce CI, we will be able toautomate tests.

Typical causes

Lack of understanding of CI among
stakeholders

Name

Coined word that explain pattern name properly

Context

Context that we adopt this pattern

Problem

Problem that we must resolve

Solution

Solution for the problem

ResultSituation that we can resolve the problem

Temporary expedient and effective
expedient

Cause

Cause that we select the solution

Involve architect

Lack of breakdown and priority
management for tasks that we can do inorder to introduce CI
Refactored patterns

Refactored solution

4.2 Template for anti-patterns
We use the following templates for anti-patterns. These templates
are from “AntiPatterns: Refactoring Software, Architectures, andProjects in Crisis “[2].
NameCoined word that explain the anti-pattern

Background

Process that the anti-pattern occurs

Symptomconsequences

andNegative results if the ant-pattern isadoptedAnecdotal evidence

Conversations being heard in context of
the anti-pattern

Typical causes

Typical causes of the anti-pattern

Refactored patterns

Refactored patterns that can resolve theanti-pattern

If adopting Temporary expedient andeffective expedient pattern, one canclassify build process to temporary
expedient and effective expedient. Tasksof temporary expedient including
compilation and deployment can takeeffects immediately. Tasks of effective
expedient including test automation anduse of metrics require process
improvement.
One can breakdown tasks of temporary
expedient and effective expedient andcalculate cost and profit. Thus theresults of breakdown and the orders ofpriority of the tasks can be shared withstakeholders.
If adopting effective expedient, process
and product improvements are needed.
Therefore the Involve architect patternis also required, which basically asks towork with the architect and improve
process and product together.
Relatedpatterns

anti-Automated all

Name

Automate all

Background

Introduction of CI

Symptomandconsequences

If trying to automate all tasks including
testing and maintenance, cost will begreatly increased and the execution of
automated tests will be time consuming.

Anecdotal
evidence

We should automate all tasks including
GUI testing

5.2 Introduction anti-patterns
5.2.1 Automated test first
Name

Automated test first

Background

Investigation when introducing CI

Symptomandconsequences

When trying to automate test first in theenvironment where there are no
automated tests, the introduction of CI
fails.

Anecdotal
evidence

If we introduce CI, we will be able toautomate tests.

Typical causes

Lack of understanding of the position oftest automation in CI

Refactored
patterns

Temporary expedient and effective
expedient

We can see results of static analysis andmetrics from a CI server too.
Typical causes

Lack of analysis of cost and effect for
tasks that can be accomplished by CI

Refactored
patterns

Procedure first

Refactored
solution

Although having enough skills, server
capacity and capability to automate all,rapidly increased cost of maintenanceand execution occurs.

Involve architect

One can adopt Procedure first pattern
and automate only the tasks of temporary
expedient.

Static analysis for testability

If improving testability by change ofarchitecture and design is needed, one
can adopt Involve architect pattern tocooperate with the architect to automate
tests.

Write tests for new codes
Involve architect
Refactored
solution

It is commonly perceived that CI and test
automation are the necessary pair and themost effective automation within CI istest automation.
However test automation is not
temporary expedient but effective
expedientthat
requiresprocess
improvement.
Therefore test automation can only beachieved after Temporary expedientand effective expedient pattern isadopted and the order of priority isclarified.
If trying to automate tests for legacy
code that has no previously automatedtests, 	
  we adopt Static analysis for
testability pattern and Write tests for
new code pattern for new code to
improve members’ skills and producttestability.
If product testability is low, adopting
Involve architect pattern is necessary tochange the architecture and design of the
product.

Relatedpatterns

anti-Automated all

5.2.2 Automate all

Relatedpatterns

anti-Automated test first

5.3 Operation anti-patterns
5.3.1 Over dependency on CI Server
NameOver dependency on CI Server

Background

Operation of CI

Symptomandconsequences

Build process can be automated, but if
the CI server stops, nothing can be built
anymore. Even worse, build with manualor semi-automated process cannot bedone as well.

Anecdotal
evidence

As our CI server stops, we can build
nothing.

Typical causes

Over dependency on CI servers
Lackof
understandingoftheresponsibilities of the necessary tools

Refactored
patterns

Operation without CI Server

Refactored
solution

Over dependency on the CI server
should be avoided. Otherwise the CI
server will be the single point of failure
during the build process. Its risk is sameas manual build or specific machinebuild.

Relatedpatterns

anti-

and use of metrics.

It is suggested to adopt Operation
without CI Server pattern, by
separating responsibilities between CI
servers and other tools like building tools
or testing tools. A CI server should be
the hub that links all the other tools
together.

6.1.2 Involve architect

Automated all

6. Patterns

Name

Involve architect

Context

Trying test automation and use of metrics, which
are the tasks of effective expedient, have been
decided.

Problem

Because of low testability of the product, it isdifficult to automate tests.

Solution

It is required to involve the architect who is theoriginal or current designer by explaining the needof product improvement and profit after the
improvement.

After the above anti-patterns are identified, we can start to draw
and adopt patterns to solve the corresponding problems. Somepatterns are particularly aiming to resolve a particular anti-pattern,
and if that is the case, we also mention that anti-pattern in theform.

One can change architecture and design with thearchitect.
Working with the architect one can develop
template code of test scripts which can be usedwithin the team to facilitate test automation easily.

6.1 Management Patterns
6.1.1 Temporary expedient and effective expedientName

Temporary expedient and effective expedientContext

Automated build tasks in CI include automatedtesting, deployment and provisioning, static
analysis, and metrics measurement.

Problem

The first automated task needs to be decided

Solution

Tasks in CI are classified into temporary
expedient and effective expedient. Temporary
expedient doesn’t have to improve development
process or product, while effective expedientneeds. Which to choose depends on therequirements from stakeholders.

If the architect is skeptical to test automation, one
can adopt Write tests for new code pattern andautomate tests for utility modules that are
independent to the existing architecture.
ResultOne can try the tasks of effective expedient thatneeds the change of architecture and design.
Therefore CI adoption can be promoted.

Cause

Testability is low because of the architecture anddesign of product. Thus to improve testability one
needs to change architecture and design.
If it is the case, one needs an agreement from theoriginal designers and team.
As the change of architecture and design affects
productivity of the team, the engineer that theentire team relies on should lead it.

If stakeholders want immediate outputs, one canadopt Procedure first pattern, and the tasks of
temporary expedient can be firstly attempted.
If stakeholders agree on the introduction of CI fora long-term improvement, One could firstly
estimate ROI and then adopt Write tests for newcode pattern, thus try the tasks of effective
expedient.
ResultOne can clarify the first automated task.
One can avoid Automated test first anti-pattern.
One can avoid Over expectation of stakeholders
anti-pattern.

Cause

Automated tasks in CI have two types: tasks of
temporary expedient and tasks of effective
expedient. Temporary expedient automates tasks
that we already execute many times manually
including compilation and deployment. Its costeffectiveness is low, but it is easy to try and its
benefit of saving time is clear.
Tasks of effective expedient require process andproduct improvement including automated testing

6.1.3 Local CI
Name

Local CI

Context

We are investigating CI. But we can’t see what we
can do.

Problem

How to find out automated tasks that the team canintroduce and work on.

Solution

One can create CI servers and examine feasibility
in the local environment.
If trying to change the design and source code of
the product, a local branch can be created in theversion control system.
With provisioning tools one can automate theconfiguration of the CI servers that is for
deployment for the team.
If under examination the architecture and design

are required to change, adopting
architect pattern is suggested.
ResultInvolve

Trying test automation in the introduction of CI.

Problem

Need to identify the areas of the product in whichtest automation can be introduced.

Solution

One can create automated tests for newly added
product code.

One can find out tasks to do for the team.
Because the configuration of CI servers can be
automated, one can complete deployment tasks
fairly quickly.

Cause

Context

Because of cost restriction it is difficult to set up
physical servers on which investigation can be
performed.

One can create automated tests for utility classes
in newly added code. Utility classes are statelessand independent to architecture and design of the
product.

Creating virtual servers in local environment iseasy due to rapid evolution of virtualization
technology.

If can adopt Involve architect pattern, one canchange architecture and design of the product for
testability.

Using provisioning tools one can copy theconfigurations of CI servers that are created on a
virtual server.

ResultOne can automate tests without changing theexisting architecture and design.

Cause

If for current product code there are no automatedtests, the testability of the code must be low. In
this case it is difficult to automate tests for the
existing code.

6.2 Introduction Patterns
6.2.1 Procedure first

Therefore the architecture and design of the
product need to change in order to improve thetestability. But it is very hard.

Name

Procedure firstContext

Although stakeholders agree on the introductionof CI, they expect immediate outputs.

6.2.3 Static analysis for testability

The most effective task that has high profit in ashort period of time is expected.

Name

Static analysis for testability

Context

When trying test automation in the introduction ofCI.

Problem

Member’s skills
automation

Solution

One can use static analysis tools and metrics
measurement tools to improve testability.

Problem
Solution

One can start procedure tasks. Procedure tasks
include compilation, deployment and packaging.
Procedure tasks are tasks that we run manually or
semi-automatically with many times.
Automation for procedure tasks doesn’t have tochange architecture and design of the product. So
particular members can do the work and the teamworkload will be low.

ResultTasks of effective expedient need to change
development process and need all members to get
involved.

Name

Write tests for new code

for

test

One should run those essential rules on CI servers
and create reports from static analysis.

Tasks of effective expedient include proceduretasks such as compilation, deployment andpackaging.

6.2.2 Write tests for new codeenoughAs rules of static analysis tools are in large
quantity, we must select essential rules that caneffectively help improve the testability and detect
critical bugs.

Tasks of effective expedient are of high cost andtake a long time to take effects.

As procedure tasks are tasks that we run manuallyor semi-automatically with many times, we can
automate them in a short period of time thus save
execution time.

not

As static analysis tools and metrics measurement
tools can detect code that is nested deeply or highcoupling, from which team members can learn
how to write testable code.

One can have outputs in a short period of time.
One can prepare for the tasks of effective
expedient.

Cause

are

After team members acquire enough skills towrite testable codes, Write tests for new code
pattern can be adopted.
ResultTeam members can acquire enough skills to write
testable codes.
Team members can adopt Write tests for newcodes pattern.

Cause

If trying to adopt Write tests for new codes
pattern, developers must know how to write
testable codes.
It is very difficult to write automated tests for thecode with low testability.
Static analysis tools can check testability of code.

6.3 Operation Patterns
6.3.1 Use existing codes as baseline
Name

Use existing codes as baseline

Context

Stakeholders request to use existing code on CI.

Problem

When trying to use existing codes on CI, one does
not possess finance and skill to do tasks ofeffective expedient like test automation.

Solution

ResultVersion management of CI servers is not enoughcomparing to version management system.
One should manage all configurations of build
tasks in a version management system.
One should use tools for the objectives.

7. Relationship with other CI patterns andBest Practice

One can use static analysis report, code metrics
and code coverage of existing code as baseline.

There exist many literatures discussing patterns and anti-patterns
about CI in which ”Continuous Integration Patterns and Antipatterns”[3] describes a lot of patterns and anti-patterns especiallyfor CI operations.

Existing code does not need to be corrected but
used as baseline by which the quality of new code
can be measured.

Moreover, ”Automation for the people: Continuous Integration
anti-patterns”[4] explains operational anti-patterns in greater
depth.

One can fulfil stakeholders’ request of using
existing code as baseline.

“Practices of Continuous Integration”[5] describes best practices
of CI including its introduction. Best practices are not patterns,
but if the best practices have valid reasons, they can be treated as
patterns.

Existing code can be used not to increase
workload of team members.
Cause

One should be able to build manually or semiautomatically even if CI servers have stopped.

It takes time and cost to adopt effective expedientfor existing code.
It is easy to run static analysis tools or metrics
measurement tools for existing code.
Static analysis tools or metrics measurement tools
can identify many potential issues.

We describe relationship between our patterns and those patterns
as following.

7.1.1 Local CI and Private Workspace
We can start our investigation for CI by adopting Private
Workspace pattern from ”Continuous Integration Patterns andAnti-patterns”[2]. Private Workspace pattern prevents ourinvestigation effecting mainstream tasks.

6.3.2 Operation without CI Server

7.1.2 Operation without CI Server, Minimal
Dependencies and Single Command

Name

Operation without CI Server

We should keep minimal dependencies not only on CI servers, butalso on pre-installed tools.

Context

Running so many tasks like compilation,
deployment, static analysis, metrics measurement
and provisioning on the CI server.

Minimal Dependencies pattern from ”Continuous Integration
Patterns and Anti-patterns”[3] recommends minimal dependencies
on pre-installed tools.

Problem

Even if CI servers have stopped, build process
still needs to continue.

Solution

One should automate or semi-automate build
tasks without CI servers.

Single Command pattern from ”Continuous Integration Patterns
and Anti-patterns”[3] recommends using single command on
build tasks. We can use Single Command pattern to accomplish
Operation without CI Server pattern.

A CI server is the hub to run some automatedtasks. But one still can run other tasks from build
tools.
One can manage configurations of build tasks onversion management systems. Thus they can bereverted whenever necessary.
ResultEven if CI servers have stopped, one can stillbuild in our local environment.
Even if the configurations of build tasks become
invalid, one can still revert them easily.

Cause

If depending on CI servers heavily and when CI
servers have stopped, build processes become
discontinued.

7.1.3 Use existing codes as baseline and Continuous
Inspection
We can use Continuous Inspection pattern from ”Continuous
Integration Patterns and Anti-patterns”[3] for Use existing codes
as baseline pattern.

7.1.4 Automate all, Bloated builds delay rapid
feedback and Keep the Build Fast
Automate all anti-pattern leads to long build time. If we canresolve Bloated builds delay rapid feed anti-pattern from
“Automation for the people: Continuous Integration antipatterns”[4] and Keep the Build Fast practice from “Practices ofContinuous Integration”[5] by using build pipeline, problem oflong build time can be resolved. This is another solution for slowbuild.

7.1.5 Procedure first and Automate Deployment
Procedure first pattern recommends automating deployment astask of temporary expedient. Automate deployment practicefrom “Practices of Continuous Integration”[5] also asks toautomate deployment by other reasons. The reasons are safe
deployment and easy rollback. Automated deployment can reduce
cost by reducing mistakes of the manual operations.

8. Use cases of Patterns
We below describe some use cases of patterns for some particularsituations.

8.1.1 Stakeholders’ expectation is highSome stakeholders think that if CI is introduced, one can automate
everything. By identifying Automate all and Over expectationof stakeholders anti-patterns, one can resolve the issues andadopt the appropriate patterns.

8.1.2 Stakeholders can’t decide what should start
One can classify tasks in CI into tasks of temporary expedient andeffective expedient. Temporary expedient does not have toimprove development process or product, but effective expedientdoes need. Thus one can adopt Temporary expedient andeffective expedient pattern and reach at the same page withstakeholders on ROI of the tasks.
If one selects temporary expedient as first task, Procedure firstpattern can be used. One should evaluate automated tasks ofprocedure by adopting Local CI pattern, then spread to the team.

8.1.3 Want to start tasks of effective expedientAs tasks of effective expedient need cooperation of the architect
who understands the existing architecture and design of the
product, one can adopt Involve architect pattern.
Even if one can refactor architecture and design, existing code
may not have automated tests and may not be testable. In that case,
one can adopt Use existing code as baseline pattern and Write
test code for new code pattern.

8.1.4 Can’t build without CI servers
Often Over dependency for CI Server anti-pattern brings
challenge, but one can resolve this anti-pattern by adopting
Operation without CI Server pattern, Minimal Dependencies
pattern and Single Command pattern.

9. ACKNOWLEDGMENTS
We created this pattern catalog from our experience ofintroducing CI to several organizations for various projects. My
appreciation goes to the architects and team members who haveinspired us.
And we also would like to thank our colleagues, CI experts and CI
community members who reviewed our paper.
Lastly we want to thank the authors who wrote CI patterns such as
from “Continuous Integration Patterns and Anti-patterns [2]” forsharing their profound knowledge.
We hope that this pattern catalog can bring beneficial outcomes toteam leaders, software configuration management engineers, test
automation engineers and process improvement engineers whentrying to employ CI within organizations.

10. REFERENCES[1] State of Agile Report
http://stateofagile.versionone.com/[2] William J. Brown, Raphael C. Malveau, Hays W. "Skip"
McCormick, Thomas J. Mowbray. AntiPatterns: Refactoring
Software, Architectures, and Projects in Crisis. ISBN-13:
978-0471197133[3] Continuous Integration Patterns and Anti-pattern.
http://refcardz.dzone.com/refcardz/continuousintegration#refcard-download-social-buttons-display[4] Automation for the people: Continuous Integration antipatterns
http://www.ibm.com/developerworks/java/library/j-ap11297/[5] Practices of Continuous Integration
http://martinfowler.com/articles/continuousIntegration.html[6] John Ferguson Smart. Jenkins: The Definitive Guide.
O'Reilly Media. ISBN-13: 978-1449305352[7] Paul M. Duvall, Steve Matyas, Andrew Glover. Continuous
Integration: Improving Software Quality and Reducing Risk.
ISBN-13: 978-0321336385