UNCLASSIFIED

CLEAREDFor Open Publication
Sep 12, 2019
Department of Defense

OFFICE OF PREPUBLICATION AND SECURITY REVIEWDoD Enterprise DevSecOps
Reference Design
Version 1.0
12 August 2019
Department of Defense (DoD)
Chief Information Officer

DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.

i
UNCLASSIFIEDUNCLASSIFIED

Document Approvals
Prepared By:

signed by
LAM.NGOAN.THOM Digitally
LAM.NGOAN.THOMAS.1229438960
Date: 2019.09.05 11:52:32 -04'00'
AS.1229438960________________________________________________________
Thomas Lam
Acting Director of Architecture and Engineering
Department of Defense, Office of the Chief Information Officer (DoD CIO)
signed by
CHAILLAN.NICOLAS. Digitally
CHAILLAN.NICOLAS.MAXIME.153505652
4
MAXIME.1535056524
Date: 2019.09.05 12:01:37 -04'00'________________________________________________________

Nicolas Chaillan
Special Advisor for Cloud Security and DevSecOps
Department of Defense, Office the Undersecretary of Acquisition and Sustainment (A&S)(currently: Chief Software Officer, Department of Defense, United States Air Force, SAF/AQ)

Approved By:
signed by
RANKS.PETER.TH Digitally
RANKS.PETER.THOMAS.1284
OMAS.12846166 616665
Date: 2019.09.05 21:41:37
65-04'00'________________________________________________________

Peter Ranks
Deputy Chief Information Officer for Information Enterprise (DCIO IE)
Department of Defense, Office of the Chief Information Officer (DoD CIO)

ii
UNCLASSIFIEDUNCLASSIFIED

Trademark Information
Names, products, and services referenced within this document may be the trade names,
trademarks, or service marks of their respective owners. References to commercial vendors andtheir products or services are provided strictly as a convenience to our readers, and do notconstitute or imply endorsement by the Department of any non-Federal entity, event, product,
service, or enterprise.

iii
UNCLASSIFIEDUNCLASSIFIED

Executive Summary
Legacy software acquisition and development practices in the DoD do not provide the agility to
deploy new software “at the speed of operations”. In addition, security is often an afterthought,
not built in from the beginning of the lifecycle of the application and underlying infrastructure.
DevSecOps is the industry best practice for rapid, secure software development.
DevSecOps is an organizational software engineering culture and practice that aims at unifying
software development (Dev), security (Sec) and operations (Ops). The main characteristic of
DevSecOps is to automate, monitor, and apply security at all phases of the software lifecycle:plan, develop, build, test, release, deliver, deploy, operate, and monitor. In DevSecOps, testingand security are shifted to the left through automated unit, functional, integration, and security
testing - this is a key DevSecOps differentiator since security and functional capabilities aretested and built simultaneously.
The benefits of adopting DevSecOps include:••••

Reduced mean-time to production: the average time it takes from when new softwarefeatures are required until they are running in production;
Increased deployment frequency: how often a new release can be deployed into the
production environment;
Fully automated risk characterization, monitoring, and mitigation across the application
lifecycle;
Software updates and patching at "the speed of operations".

This DoD Enterprise DevSecOps Reference Design describes the DevSecOps lifecycle,
supporting pillars, and DevSecOps ecosystem; lists the tools and activities for DevSecOps
software factory and ecosystem; introduces the DoD enterprise DevSecOps container service thatprovides hardened DevSecOps tools and deployment templates to the program application
DevSecOps teams to select; and showcases a sampling of software factory reference designs andapplication security operations. This DoD Enterprise DevSecOps Reference Design provides
implementation and operational guidance to Information Technology (IT) capability providers,
IT capability consumers, application teams, and Authorizing Officials.

iv
UNCLASSIFIEDUNCLASSIFIED

Table of Contents
1

2

3

Introduction ......................................................................................................................... 10
1.1

Background ................................................................................................................... 10

1.2

Purpose .......................................................................................................................... 11

1.3

Scope .............................................................................................................................. 11

1.4

Document Overview ..................................................................................................... 12

Assumptions and Principles................................................................................................ 13
2.1

Assumptions .................................................................................................................. 13

2.2

Principles ....................................................................................................................... 13

DevSecOps Concepts ........................................................................................................... 15
3.1

Key Terms ..................................................................................................................... 15

3.1.1
3.2

DevSecOps Lifecycle .................................................................................................... 18

3.3

DevSecOps Pillars ........................................................................................................ 19

3.3.1

Organization............................................................................................................ 20

3.3.2

Process .................................................................................................................... 21

3.3.3

Technology ............................................................................................................. 23

3.3.4

Governance ............................................................................................................. 23

3.4

4

Conceptual Model ................................................................................................... 18

3.3.4.1

Management Structure ..................................................................................... 23

3.3.4.2

Authorizing Official ........................................................................................ 25

DevSecOps Ecosystem.................................................................................................. 26

3.4.1

Planning .................................................................................................................. 27

3.4.2

Software Factory ..................................................................................................... 28

3.4.3

Operations ............................................................................................................... 29

3.4.4

External Systems ..................................................................................................... 29

DevSecOps Tools and Activities ......................................................................................... 31
4.1

Planning Tools and Activities ...................................................................................... 31

4.2

Software Factory Tools and Activities ....................................................................... 34v
UNCLASSIFIEDUNCLASSIFIED
4.2.1

CI/CD Orchestrator ................................................................................................. 34

4.2.2

Develop ................................................................................................................... 35

4.2.3

Build........................................................................................................................ 38

4.2.4

Test .......................................................................................................................... 40

4.2.5

Release and Deliver ................................................................................................ 45

4.3

Production Operation Tools and Activities ............................................................... 46

4.3.1

5

4.3.1.1

Virtual Machine deployment ........................................................................... 46

4.3.1.2

Container deployment...................................................................................... 47

4.3.2

Operate .................................................................................................................... 49

4.3.3

Monitor ................................................................................................................... 50

4.4

Security Tools and Activities Summary ..................................................................... 53

4.5

Configuration Management Tools and Activities Summary ................................... 54

4.6

Database Management Tools and Activities Summary ............................................ 55

DoD Enterprise DevSecOps Container Service ................................................................ 57
5.1

DoD Enterprise DevSecOps Container Factory ........................................................ 57

5.1.1

DoD Hardened Containers ...................................................................................... 57

5.1.2

Container Hardening Process .................................................................................. 58

5.2
6

Deploy ..................................................................................................................... 46

5.1.2.1

Select the Container Base Image ..................................................................... 58

5.1.2.2

Harden the Container ....................................................................................... 59

5.1.2.3

Store the Hardened Container.......................................................................... 59

5.1.2.4

Documentation................................................................................................. 59

5.1.2.5

Continuous Engineering .................................................................................. 60

5.1.2.6

Cybersecurity ................................................................................................... 60

DoD Centralized Artifact Repository ......................................................................... 60

DevSecOps Ecosystem Reference Designs ......................................................................... 61
6.1

Containerized Software Factory ................................................................................. 61

6.1.1

Hosting Environment .............................................................................................. 62

6.1.2

Container Orchestration .......................................................................................... 63
vi
UNCLASSIFIEDUNCLASSIFIED

7

6.1.3

Software Factory Using Hardened Containers ....................................................... 63

6.1.4

DoD Applications ................................................................................................... 64

6.2

Software Factory using Cloud DevSecOps Services ................................................. 65

6.3

Serverless Support........................................................................................................ 66

6.4

Application Security Operations................................................................................. 68

6.4.1

Continuous Deployment ......................................................................................... 68

6.4.2

Continuous Operation ............................................................................................. 68

6.4.3

Continuous Monitoring ........................................................................................... 69

6.4.4

Sidecar Container Security Stack............................................................................ 70

Conclusion ............................................................................................................................ 75

Appendix A

Acronym Table .................................................................................................. 76

Appendix B

Glossary of Key Terms ..................................................................................... 79

Appendix C

References .......................................................................................................... 88

vii
UNCLASSIFIEDUNCLASSIFIED

List of Figures
Figure 1: Containers ................................................................................................................... 17
Figure 2: Conceptual Model ...................................................................................................... 18
Figure 3: DevSecOps Software Lifecycle .................................................................................. 19
Figure 4: DevSecOps Pillars ...................................................................................................... 20
Figure 5: Application DevSecOps Processes ............................................................................ 22
Figure 6: Five Principles of Next Generation Governance ..................................................... 25
Figure 7: Assessment and Authorization Inheritance ............................................................. 26
Figure 8: DevSecOps Ecosystem................................................................................................ 27
Figure 9: DevSecOps Software Factory .................................................................................... 28
Figure 10: DoD Enterprise DevSecOps Container Service Architecture .............................. 57
Figure 11: Major Steps in the Container Hardening Process................................................. 58
Figure 12: Containerized Software Factory Reference Design .............................................. 62
Figure 13: DevSecOps Platform Options .................................................................................. 63
Figure 14: Software Factory Phases in the Application Lifecycle .......................................... 64
Figure 15: Software Factory using Cloud DevSecOps Services ............................................. 66
Figure 16: Operational Efficiency ............................................................................................. 67
Figure 17: Logging and Log Analysis Process ......................................................................... 70
Figure 18: Sidecar Pattern ......................................................................................................... 71
Figure 19: Sidecar Components ................................................................................................ 72
Figure 20: Sidecar Container Security Stack Interactions ..................................................... 74
Figure 21: Hypervisor with Virtual Machines ......................................................................... 84

viii
UNCLASSIFIEDUNCLASSIFIED

List of Tables
Table 1: Key Terms .................................................................................................................... 15
Table 2: Roles of Authorizing Officials in DevSecOps ............................................................ 26
Table 3: Plan Phase Tools .......................................................................................................... 31
Table 4: Plan Phase Activities .................................................................................................... 33
Table 5: CI/CD Orchestrator ..................................................................................................... 35
Table 6: Develop Phase Tools .................................................................................................... 36
Table 7: Develop Phase Activities .............................................................................................. 37
Table 8: Build Phase Tools ......................................................................................................... 38
Table 9: Build Phase Activities .................................................................................................. 39
Table 10: Test Phase Tools ......................................................................................................... 40
Table 11: Test Phase Activities .................................................................................................. 43
Table 12: Release and Deliver Phase Tools .............................................................................. 45
Table 13: Release and Deliver Phase Activities ........................................................................ 46
Table 14: Deploy Phase Tools .................................................................................................... 47
Table 15: Deploy Phase Activities ............................................................................................. 48
Table 16: Operate Phase Tools .................................................................................................. 50
Table 17: Operate Phase Activities ........................................................................................... 50
Table 18: Monitor Phase Tools .................................................................................................. 51
Table 19: Monitor Phase Activities ........................................................................................... 52
Table 20: Security Activities Summary .................................................................................... 53
Table 21: Configuration Management Activities Summary ................................................... 54
Table 22: Database Management Activities Summary ........................................................... 56
Table 23: Sidecar Container Security Stack Components ...................................................... 72

ix
UNCLASSIFIEDUNCLASSIFIED

1Introduction

1.1 Background
DevSecOps is an organizational software engineering culture and practice that aims at unifying
software development (Dev), security (Sec) and operations (Ops). The main characteristic of
DevSecOps is to improve customer outcomes and mission value by automating, monitoring, andapplying security at all phases of the software lifecycle: plan, develop, build, test, release,deliver, deploy, operate, and monitor. Practicing DevSecOps provides demonstrable quality andsecurity improvements over the traditional software lifecycle, which can be measured with these
metrics:•

Mean-time to production: the average time it takes from when new software features are
required until they are running in production.

•

Average lead-time: how long it takes for a new requirement to be delivered and deployed.

•

Deployment speed: how fast a new version of the application can be deployed into the
production environment.

•

Deployment frequency: how often a new release can be deployed into the production
environment.

•

Production failure rate: how often software fails during production.

•

Mean-time to recovery: how long it takes applications in the production stage to recoverfrom failure.

In addition, DevSecOps practice enables:•

Fully automated risk characterization, monitoring, and mitigation across the application
lifecycle.

•

Software updates and patching at a pace that allows the addressing of security
vulnerabilities and code weaknesses.

DevSecOps practice enables application security, secure deployment, and secure operations inclose alignment with mission objectives. In DevSecOps, testing and security are shifted to theleft through automated unit, functional, integration, and security testing - this is a key
DevSecOps differentiator since security and functional capabilities are tested and built
simultaneously. In addition, some security features are automatically injected into the application
without developer intervention via a sidecar container.

10
UNCLASSIFIEDUNCLASSIFIED

1.2 Purpose
The main purpose of this document is to provide a logical description of the key designcomponents and processes to provide a repeatable reference design that can be used to instantiatea DoD DevSecOps software factory.
The target audiences for this document include:•

DoD Enterprise DevSecOps capability providers who build DoD Enterprise DevSecOps
hardened containers and provide a DevSecOps hardened container access service•

DoD organization DevSecOps teams who manage (instantiate and maintain) DevSecOps
software factories and associated pipelines for its programs•

DoD program application teams who use DevSecOps software factories to develop,
secure, and operate mission applications•

Authorizing Officials (AOs)

The DoD Enterprise DevSecOps reference design leverages a set of hardened DevSecOps tools
and deployment templates that enable DevSecOps teams to select the appropriate template forthe program application capability to be developed. For example, these templates will be
specialized around a specific programming language or around different types of capabilities
such as web application, transactional, big data, or artificial intelligence (AI) capabilities. A
program selects a DevSecOps template and toolset; the program then uses these to instantiate a
DevSecOps software factory and the associated pipelines that enable Continuous Integration andContinuous Delivery (CI/CD) of the mission application.
This reference design aligns with these reference documents:•

DoD Cloud Computing Strategy [1]•

DoD Cloud Computing Security Requirements Guide [2]•DoD Secure Cloud Computing Architecture (SCCA) [3]•

Presidential Executive Order on Strengthening the Cybersecurity of Federal Networks
and Critical Infrastructure (Executive Order (EO) 1380) [4]•

National Institute of Standards and Technology (NIST) Cybersecurity Framework [5]•DoD Container Hardening Security Requirements Guide [6].

1.3 ScopeThis document describes the reference design to enable DevSecOps to scale across thedepartment. DevSecOps is an established mature capability in industry, and it is already usedwithin some pockets of the Government; this reference design formalizes its usage across the11
UNCLASSIFIEDUNCLASSIFIED
DoD. This design is product agnostic and provides execution guidance for use by software
teams. It is applicable to developing new capabilities and to sustaining existing capabilities inboth business and weapons systems software, including business transactions, C2, embedded
systems, big data, and Artificial Intelligence (AI).
This document does not address policy or acquisition.

1.4 Document Overview
The documentation is organized as follows.
•

Section 1 describes the background, purpose and scope of this document.

•

Section 2 describes the assumptions made in developing this reference design, as well as
stating foundational principles.

•

Section 3 describes the DevSecOps lifecycle, the four pillars to assist DevSecOps
adoption, and a technical architecture of the DevSecOps software factory and its
ecosystem.

•

Section 4 describes the DevSecOps ecosystem tools and the activities along software
lifecycle phases.

•

Section 5 describes the DoD Enterprise DevSecOps Service. The target audience of thissection is DoD Enterprise DevSecOps capability providers.

•

Section 6 describes the reference design for DoD programs to build their DevSecOps
software factory and ecosystem.

12
UNCLASSIFIEDUNCLASSIFIED

2

Assumptions and Principles

2.1 Assumptions
This document makes the following assumptions:•

For most organizations, deploying to a certified and monitored cloud environment will
become their preferred solution technically and culturally.

•

Rapidly changing technology dictates designing the DevSecOps pipelines and patterns
for flexibility as new development capabilities enter/exit the commercial product market.

•

The DoD Enterprise DevSecOps software factory is designed to avoid vendor lock-in andleverage Open Container Initiative (OCI) compliant containers and Cloud Native
Computing Foundation (CNCF) certified Kubernetes to orchestrate and manage thecontainers.

•

The government must balance open source integration risks vs. using pre-integrated
Commercial Off-The-Shelf (COTS) products that have vendor “cost of exit” and vendor
insider risks.

•

It must be possible to host a DevSecOps software factory in any DoD general-purpose
cloud environment, as well as in disconnected and classified environments.

•

The DevSecOps architecture must have the capability to scale to any type of operational
requirement needing a software solution, including:o Business systems
o Command and Control systems
o Embedded and Weapon systems
o Intelligence analysis systems
o Autonomous systems
o Assisted human operations

2.2

Principles

There are several key principles to implementing a successful DevSecOps approach:•Remove bottlenecks (including human ones) and manual actions.

•

Automate as much of the development and deployment activities as possible.

•

Adopt common tools from planning and requirements through deployment andoperations.

13
UNCLASSIFIEDUNCLASSIFIED•

Leverage agile software principles and favor small, incremental, frequent updates overlarger, more sporadic releases.

•

Apply the cross-functional skill sets of Development, Cybersecurity, and Operations
throughout the software lifecycle, embracing a continuous monitoring approach inparallel instead of waiting to apply each skill set sequentially.

•

Security risks of the underlying infrastructure must be measured and quantified, so thatthe total risks and impacts to software applications are understood.

•

Deploy immutable infrastructure, such as containers. The concept of immutableinfrastructure is an IT strategy in which deployed components are replaced in their
entirety, rather than being updated in place. Deploying immutable infrastructure requires
standardization and emulation of common infrastructure components to achieve
consistent and predictable results.

14
UNCLASSIFIEDUNCLASSIFIED

3

DevSecOps Concepts

DevSecOps describes an organization’s culture and practices enabling organizations to bridge
the gap between developers, security team, and operations team; improve processes throughcollaborative and agile workflows; drive for faster and more secure software delivery via
technology; and achieve consistent governance and control. There is no uniform DevSecOps
practice. Each DoD organization needs to tailor its culture and its DevSecOps practices to its
own unique processes, products, security requirements, and operational procedures. Embracing
DevSecOps requires organizations to shift their culture, evolve existing processes, adopt newtechnologies, and strengthen governance.
This section will briefly discuss the DevSecOps lifecycle, supporting pillars, and the DevSecOps
ecosystem.

3.1 Key Terms
Here are some key terms used throughout the document. Refer to the glossary in Appendix B for
the full list.
Table 1: Key Terms
Term
DevSecOps Ecosystem

Definition
A collection of tools and process workflows created andexecuted on the tools to support all the activities throughout
the full DevSecOps lifecycle.
The process workflows may be fully automated, semiautomated, or manual.

Software Factory

A software assembly plant that contains multiple pipelines,
which are equipped with a set of tools, process workflows,
scripts, and environments, to produce a set of software
deployable artifacts with minimal human intervention. It
automates the activities in the develop, build, test, release,and deliver phases. The software factory supports multitenancy.

CI/CD Pipeline

The set of tools and the associated process workflows toachieve continuous integration and continuous delivery withbuild, test, security, and release delivery activities, which
are steered by a CI/CD orchestrator and automated as muchas practice allows.
15
UNCLASSIFIEDUNCLASSIFIED
TermDefinition

CI/CD Pipeline Instance

A single process workflow and the tools to execute theworkflow for a specific software language and application
type for a project. As much of the pipeline process is
automated as is practicable.

Environment

Sets a runtime boundary for the software component to be
deployed and executed. Typical environments include
development, integration, test, pre-production, andproduction.

Software Factory Artifact
Repository

Code

A local repository tied to the software factory. It stores
artifacts pulled from DoD Centralized Artifact Repository(DCAR) as well as locally developed artifacts to be used in
DevSecOps processes. The artifacts include, but are notlimited to, virtual machine (VM) images, container images,
binary executables, archives, and documentation. It supports
multi-tenancy.
Note that programs may have a single artifact repository anduse tags to distinguish the content types. It is also possible
to have separate artifact repositories to store local artifacts
and released artifacts.
Software instructions for a computer, written in a
programming language. These instructions may be in theform of either human-readable source code, or machinecode, which is source code that has been compiled intomachine executable instructions.

16
UNCLASSIFIEDUNCLASSIFIED
TermContainers

Definition
A standard unit of software that packages up code and all its
dependencies, down to, but not including the Operating
System (OS). It is a lightweight, standalone, executable
package of software that includes everything needed to run
an application except the OS: code, runtime, system tools,
system libraries and settings.
Several containers can run in the same OS without
conflicting with one another.

Figure 1: Containers
Containers run on the OS, so no hypervisor (virtualization)is necessary (though the OS itself may be running on a
hypervisor).
Containers are much smaller than a VM, typically by afactor of 1,000 (MB vs GB), partly because they don’t needto include the OS. Using containers allows denser packing
of applications than VMs.
Unlike VMs, containers are portable between clouds or
between clouds and on-premise servers. This helps alleviate
Cloud Service Provider (CSP) lock-in, though anapplication may still be locked-in to a CSP, if it uses CSPspecific services.
Containers also start much faster than a VM (seconds vs.
minutes), partly because the OS doesn’t need to boot.

17
UNCLASSIFIEDUNCLASSIFIED
3.1.1

Conceptual Model

The following conceptual model shows some of the most important concepts described in thispaper along with their relationships. It should help to clarify these relationships. When reading
text along an arrow, follow the direction of the arrow. So, a DevSecOps Ecosystem contains one
or more software factories, and each software factory contains one or more pipelines. The
diagram also shows that each software factory contains only one CI/CD Orchestrator, and thatmany software factories use DCAR.

Figure 2: Conceptual Model

3.2 DevSecOps Lifecycle
The DevSecOps software lifecycle phases are illustrated in Figure 3. There are nine phases: plan,develop, build, test, release, deliver, deploy, operate, and monitor. Security is embedded within

18
UNCLASSIFIEDUNCLASSIFIED
each phase.

Figure 3: DevSecOps Software Lifecycle
With DevSecOps, the software development lifecycle is not a monolithic linear process. The“big bang” style delivery of the Waterfall process is replaced with small but more frequentdeliveries, so that it is easier to change course as necessary. Each small delivery is accomplishedthrough a fully automated process or semi-automated process with minimal human intervention
to accelerate continuous integration and delivery. The DevSecOps lifecycle is adaptable and has
many feedback loops for continuous improvement.

3.3 DevSecOps Pillars
DevSecOps are supported by four pillars: organization, process, technology, and governance, as
illustrated in Figure 4.

19
UNCLASSIFIEDUNCLASSIFIED

Figure 4: DevSecOps Pillars
For each DoD organization, the practice of DevSecOps starts with buy-in of the DevSecOps
philosophy by senior leaders within the organization. This leads to a change to the organizational
culture, along with the development of new collaborative processes, technologies and tools toautomate the process and to apply consistent governance. A project must advance in all fourareas to be successful.
3.3.1

Organization

The organization should embrace the following philosophies and ideas and incorporate them into
their daily activities and software lifecycle management processes.
•

Change the organizational culture to take a holistic view and share the responsibility ofsoftware development, security and operations. Train staff with DevSecOps concepts andnew technologies. Gradually gain buy-in from all stakeholders.

•

Break down organizational silos. Increase the team communication and collaboration in
all phases of the software lifecycle.

•

Actionable security and quality assurance (QA) information, such as security alerts or
QA reports, must be automatically available to the teams at each software lifecycle phaseto make collaborative actions possible.
20
UNCLASSIFIEDUNCLASSIFIED•

Build a culture of safety by sharing after-action reports on both positive and negative
events across the entire organization. Teams should use both success and failure as
learning opportunities to improve the system design, harden the implementation, andenhance the incident response capability as part of the DevSecOps practice.

•

Make many small, incremental changes instead of fewer large changes. The scope ofsmaller changes is more limited and thus easier to manage.

•

Embrace feedback and user driven change to respond to new, emerging, and unforeseen
requirements.

•

Plan and budget for continuous code refactoring to ensure constant buy down ofaccumulated technical debt.

3.3.2

Process

Depending on the mission environment, system complexity, system architecture, software design
choices, risk tolerance level, and system maturity level, each program’s software lifecycle has its
own unique management processes.
For example, suppose a mature web application software system has adopted a microservices
design. Its development, pre-production, and production environment are on the same cloud. The
test procedures are fully automated. This system could have a process flow to automate thedevelop, build, test, secure, and delivery tasks to push updates into production quickly without
human intervention. On the other hand, a complex mission critical embedded system, such as aweapons system, may have a different process that requires some tests that cannot be fully
automated. The software lifecycle process for that system will be significantly different from theprocess for the web application system.
To adopt a DevSecOps process successfully, implement it in multiple, iterative phases. Start
small with some tasks that are easy to automate, then gradually build up the DevSecOps
capability and adjust the processes to match. Figure 5 illustrates this concept; it shows that asoftware system can start with a Continuous Build pipeline, which only automates the build
process after the developer commits code. Over time, it can then progress to Continuous
Integration, Continuous Delivery, Continuous Deployment, Continuous Operation, and finally
Continuous Monitoring, to achieve the full closed loop of DevSecOps. A program could startwith a suitable process and then grow progressively from there. The process improvement isfrequent, and it responds to feedback to improve both the application and the process itself.

21
UNCLASSIFIEDUNCLASSIFIED

Figure 5: Application DevSecOps Processes
There is no “one size fits all” solution for process design. Each software team has its own uniquerequirements and constraints. Below is a list of some best practices to guide the process design:
1. The process design is a collective effort from multidisciplinary teams.
2. Most of the processes should be automatable via tools and technologies.
3. The DevSecOps lifecycle is an iterative closed loop. Start small and build it up
progressively to strive for continuous improvement. Set up human intervention at thecontrol gates when necessary, depending on the maturity level of the process and theteam’s confidence level in the automation. Start with more human intervention andgradually decrease it as possible.
4. AO should consider automating the Authority to Operate (ATO) process as much as
possible.

22
UNCLASSIFIEDUNCLASSIFIED
To help organizations evolve their DevSecOps capabilities and processes, the DoD has
developed a DevSecOps Maturity Model. This model details many steps that organizations cantake to move incrementally towards a higher DevSecOps maturity level. That maturity model ispresented in the DoD DevSecOps Playbook [7].
3.3.3Technology

Many DevSecOps tools automate many tasks in the software lifecycle without human
involvement. Other DevSecOps tools, such as collaboration and communication tools, facilitateand stimulate human interaction to improve productivity. Some DevSecOps tools aim to help anactivity at a specific lifecycle phase. For example, an Integrated Development Environment(IDE) DevSecOps plug-in for develop phase, or a static application security test tool for the build
phase. Most tools assist a particular set of activities. The tags added to artifacts in the artifact
repository help guarantee that the same set of artifacts move together along a pipeline. Section 4
will introduce a variety of DevSecOps tools.
The instantiation of the DevSecOps environments can be orchestrated from configuration files
instead of setting up one component at a time manually. The infrastructure configuration files,
the DevSecOps tool configuration scripts, and the application run-time configuration scripts arereferred to as Infrastructure as Code (IaC). Taking the same approach as IaC, security teams
program security policies directly into configuration code, as well as implement security
compliance checking and auditing as code, which are referred as Security as Code (SaC). BothIaC and SaC are treated as software and go through the rigorous software development processes
including design, development, version control, peer review, static analysis, and test.
Technologies and tools play a key role in DevSecOps practice to shorten the software lifecycle
and increase efficiency. They not only enable software production automation as part of asoftware factory, but also allow operations and security process orchestration.
3.3.4

Governance

Governance actively assesses and manages the risks associated with the mission program
throughout the lifecycle. Governance activities do not stop after ATO but continue throughout
the software lifecycle, including operations and monitoring. DevSecOps can facilitate andautomate many governance activities.
3.3.4.1

Management Structure

The management objective of DevSecOps must be both “top-down” and “bottom-up” to balancelarger strategic goals. Studies (e.g., [8]) have shown that senior leader buy-in is crucial for
success. But buy-in at the staff level is also important to engender a sense of ownership, toencourage the appropriate implementation of processes related to governance, and to enable teammembers to support continuous process improvement. Continuous process improvement –seeking opportunities to simplify and automate whenever and wherever possible – is essential forgovernance to keep pace with a rapidly changing world.
23
UNCLASSIFIEDUNCLASSIFIED
Early DevSecOps efforts in the DoD, such as [9] have leveraged and adopted commercial bestpractices. That document identifies Five Fundamental Principles of Next Generation Governance(NGG):
1. Run IT with Mission Discipline: Tie requirements back to your organization’s mission. Everyaction should be aligned to the mission. If they are not, then an evaluation should be
performed with Continuous Process Improvement to address how to tie actions to missions.
2. Invest in Automation: Automate everything possible, including actions, business processes,
decisions, approvals, documentation, and more. Automation, including designs, interfaces,
functional and security tests, and their related documentation, create the Artifacts of Recordthat provide the body of evidence required by the Risk Management Framework (RMF) andfor historical audits when needed.
3. Embrace Adaptability: Accept that change can be required at any time, and all options areavailable to achieve it. Fail fast, fail small, and fail forward. An example of failing forward iswhen a developer finds that a release does not work. Then instead of restoring the server to
its pre-deployment state with the previous software, the developer’s change should bediscrete enough that they can fix it and address the issue through a newer release.
4. Promote Transparency: Offer open access across the organization to view the activities
occurring within the automated process and to view the auto-generated Artifacts of Record.
Transparency generates an environment for sharing ideas and developing solutions
comprised of Subject Matter Experts (SMEs) or leads from across the enterprise in the formof cross-functional teams to avoid the “silo effect.” When composed of all representative
stakeholders, the team possesses the skills needed to build a mission system and thecollective ingenuity necessary to overcome all encountered challenges.
5. Inherent Accountability: Push down or delegate responsibility to the lowest level:•

Strategic: This is related to the Change Control Board (CCB) or Technical Review Board(TRB); it involves “Big Change” unstructured decisions. These infrequent and high-risk
decisions have the potential to shape the strategy and mission of an organization.

•

Operational: (Various Scrum) Cross-cutting, semi-structured decisions. In these frequentand high-risk decisions, a series of small, interconnected decisions are made by different
groups as part of a collaborative, end-to-end decision process.

•Tactical: (Global Enterprise Partners (GEP)/Product Owner/Developers Activities)
Delegated, structured decisions. These frequent and low-risk decisions are effectively
handled by an individual or working team, with limited input from others.

These 5 principles are summarized in Figure 6, which is from [9].

24
UNCLASSIFIEDUNCLASSIFIED

Figure 6: Five Principles of Next Generation Governance3.3.4.2

Authorizing Official

DoD Instruction (DoDI) 8510.01 [10] is the ultimate governance policy and states processes thatall DoD information system and platform information technology system must follow. It is underrevision and the following DevSecOps related governance information will be incorporated in
the future release.
For initial standup of a new DevSecOps software factory instance and a production operations
environment, the RMF process follows an enterprise level process. The assessment andauthorization (A&A) should inherit the certifications and authorizations of the underlying
infrastructure (e.g., a DoD cloud provisional authorization) and of the DoD Enterprise Hardened
Containers, without having to re-certify them. The program should have a formal Service Level
Agreement (SLA) with the underlying infrastructure provider about what services are included
and what authorizations can be inherited. This affects the status of applicable assessment
procedures and prepares the stage for inheritance into the operations environment andapplication. Once the instance is authorized and operational, the specialty AO for the functional

25
UNCLASSIFIEDUNCLASSIFIED
area or the local AO for the program has cognizance for Continuous Authorization of theenvironment. Figure 7 illustrates the A&A inheritance.
The specialty AO for the functional area or the local AO for the program has cognizance forContinuous Authorization of the mission applications.

Figure 7: Assessment and Authorization Inheritance
Table 2: Roles of Authorizing Officials in DevSecOps
Capability

Authorizing Official

DoD Enterprise Hardened Containers
DevSecOps software factory instances
Continuous Process Improvement / Continuous Authorizationof DevSecOps software factory instances
AO for mission applications

Enterprise AO (e.g., Defense Information
Systems Agency (DISA))Enterprise AO (e.g., DISA, Military
Department (MilDep) CIO)
Specialty or Local AO (e.g., Program executive
Officer (PEO))
Specialty or Local AO (e.g., PEO)

3.4 DevSecOps Ecosystem
The DevSecOps ecosystem is a collection of tools and the process workflows created andexecuted on the tools to support all the activities throughout the full DevSecOps lifecycle. As
illustrated in Figure 8, the DevSecOps ecosystem is composed of three subsystems: planning, asoftware factory, and production operations. The DevSecOps ecosystem interacts with external
enterprise services to get all dependency support, and with enterprise and local AO to gain
operation authorization.
The DevSecOps administration team is responsible for administrating the ecosystem tools andautomating the process workflows. The mission application team focuses on the development,
testing, security and operations tasks using the ecosystem.

26
UNCLASSIFIEDUNCLASSIFIED

Figure 8: DevSecOps Ecosystem
3.4.1

Planning

The plan phase involves activities that help the project manage time, cost, quality, risk andissues. These activities include business-need assessment, project plan creation, feasibility
analysis, risk analysis, business requirements gathering, business process creation, systemdesign, DevSecOps design and ecosystem instantiation, etc. The plan phase repeats when
DevSecOps the lifecycle recycles. It is a best practice to develop a minimum viable product(MVP) for critical business needs as the first thing to develop. Then get into the feedback loop
process as quickly as possible; this is recommended in the Lean Startup methodology [11]. The
DevSecOps design creates the DevSecOps processes and control gates, which will guide theautomation throughout the lifecycle. DevSecOps ecosystem tools will facilitate process
automation and consistent process execution.
The DevSecOps planning subsystem supports the activities in the plan phase using a set ofcommunication, collaboration, project management, and change management tools. In thisphase, the process workflows are not fully automated. The planning tools assist humaninteraction and increase team productivity.

27
UNCLASSIFIEDUNCLASSIFIED
3.4.2

Software Factory

A software factory, illustrated in Figure 9, contains multiple pipelines, which are equipped with aset of tools, process workflows, scripts, and environments, to produce a set of software
deployable artifacts with minimal human intervention. It automates the activities in the develop,
build, test, release, and deliver phases. The environments that are set up in the software factoryshould be orchestrated with scripts that include IaC and SaC and which run on various tools. A
software factory must be designed for multi-tenancy and automate software production for
multiple projects. A DoD organization may need multiple pipelines for different types ofsoftware systems, such as web applications or embedded systems.

Figure 9: DevSecOps Software Factory
The factory starts with the development team developing application code and IaC, QA
developing test scripts, and the security team developing SaC in their suitable IDEs. The entire
Software Factory should leverage OCI compliant containers and DoD Hardened Containers
whenever available. Once COTS, Government off the Shelf (GOTS), or newly developed codeand scripts are committed into the Software Factory’s code repository, the assembly line
automation kicks in. There could be multiple CI pipeline instances as assembly lines. Each is fora specific application subsystem, such as a JavaScript assembly line for a web front-end, aPython or R assembly line for data analytics, or a GoLang assembly line for a backend
application. The CI assembly line guides the subsystem through continuous integration bybuilding the code and incorporating dependencies (such as libraries) from the local artifact
28
UNCLASSIFIEDUNCLASSIFIED
repository. In addition, it performs tests in the development and test environments, such as unit
tests, static code analysis, functional tests, interface tests, dynamic code analysis, etc. The
subsystems that pass CI assembly line control gate policies will move into the pre-production
environment for systems integration. The CD assembly line takes over control from this point.
More tests and security scans are performed in this environment, such as performance tests,
acceptance test, security compliance scan, etc. The CD assembly line releases and delivers the
final product package to the released artifact repository if the control gate policies are met.
Developing applications using a DevSecOps software factory provides many benefits:•

Improved software product consistency and quality•

Shortened time to market and increased productivity•

Simplified governance

3.4.3

Operations

In the production environment, the released software is pulled from the released artifact
repository and deployed. Operations, operation monitoring, and security monitoring are
performed. Production operation tools aim to streamline and automate the deployment,
operations, and monitoring activities. Tools should be selected based on system functional
requirements and their suitability for the production environment infrastructure.
3.4.4

External Systems

The DevSecOps ecosystem itself and program applications depend on some DoD enterpriseservices to acquire the necessary baseline tools, application dependencies, and security services
to operate.
••

DoD Centralized Artifact Repository (DCAR) holds the hardened VM images andhardened OCI compliant container images of: DevSecOps tools, container security
tools, and common program platform components (e.g. COTS or open source products)that DoD program software teams can utilize as a baseline to facilitate the authorization
process.
DoD Common Security Services are DoD enterprise-level common services thatfacilitate cybersecurity enforcement and IT management. One security service will
perform traffic inspection and filtering to protect the mission enclave and mission
applications. Some security service examples include firewalls; Intrusion Detection
System (IDS)/Intrusion Prevention System (IPS); malware detection; data loss
prevention; host-based security; log/telemetry aggregation and analysis; and Identity,
Credential, and Access Management (ICAM). A Cybersecurity Service Provider(CSSP) will provide additional services, including Attack Sensing and Warning(ASW), Forensic Media Analysis (FMA), Assurance Vulnerability Management(AVM), Incident Reporting (IR), Incident Handling Response (IHR), Information
Operations Condition (INFOCON), Cyber Protection Condition (CPCON), Malware
Notification Protection (MNP), and Network Security Monitoring (NSM).
29
UNCLASSIFIEDUNCLASSIFIEDThe DevSecOps ecosystem interacts with the enterprise AO for the initial software factory ATO
and initial application ATO, as well as the local AO for continuous ATO for the application.

30
UNCLASSIFIEDUNCLASSIFIED

4

DevSecOps Tools and Activities

This section describes both tools for the DevSecOps ecosystem and DevSecOps activities foreach phase. Activities and tools are listed in table format. The Baseline column in the tool tables
has two values: Minimal Viable Product (MVP) and objective. They indicate whether the tool
must be available in the DevSecOps ecosystem MVP as threshold or if the tool is an objective tobe reached as the ecosystem matures. Activity tables list a wide range of activities for
DevSecOps practice. DoD organizations should define their own processes, choose proper
activities, and then select tools suitable for their systems to build software factories and
DevSecOps ecosystems. With the DevSecOps maturity progression, the level of activity
automation will increase.

4.1 Planning Tools and Activities
The Planning tools support software development planning, which includes configuration
management planning, change management planning, project management planning, systemdesign, software design, test planning, and security planning. Table 3 lists some tools that can
assist the planning process. Some tools will be used throughout the software lifecycle, such as ateam collaboration tool, an issue tracking system, and a project management system. Some tools
are shared at the enterprise level across programs. Policy and enforcement strategy should beestablished for access controls on various tools.
Table 3: Plan Phase Tools
Tool
Team
collaboration
systemIssue tracking
systemAssetinventory
management

Features
Audio/videoconferencing;chat/messaging;brainstorming
discussion board;group calendars;
file sharing;
Wiki website
Bugs and defectmanagement;
Feature and change
management;
Prioritization
management;
Assignment
management;
Escalation
management;
Knowledge basemanagement
Collect information
about all IT assets;

Benefits

InputsSimplify
communication
and boost teamefficiency

Team meetings;
Design notes;
Documentation

Organized
teamwork;
Versioncontrolled
documents

MVP

Easy to detect
defect trends
Improve
software productquality
Reduce cost andimprove Returnon Investment(ROI)

Bug reportFeature/change
requestRoot cause
analysis
SolutionsIssuesfeature/change
tickets.
Issue
resolution
tracking
historyMVP

Increasesituationawareness

IT assets(applications,softwareAsset
inventory

Objective

31
UNCLASSIFIED

Outputs

Baseline

UNCLASSIFIED
Tool

Configuration
management
database(CMDB)

Projectmanagement
systemSoftwaresystem design
tool

Threatmodeling tool

Data modelingtool

Features
Maintain a “real-time”
inventory of all
applications, software
licenses, libraries,
operating systems, andversioning information
Auto-discovery;
Dependency mapping;Integration with othertools;
Configuration auditing

Task management
Scheduling and time
management
Resource management
Budget management
Risk management
Assist requirement
gathering,system architecture
design, components
design, and interface
designDocument systemsecurity design;
Analyze the design forpotential security
issues;
Review and analysis
against common attack
patterns;
Suggest and manage
mitigation
Model theinterrelationship andflows between different
data elements

Benefits

Centralized
database used bymany systems(such as asset
management,
configuration
management,
incident
management,etc.) during
development
and operations
phases.
Assist projectprogresstracking
Optimize
resource
allocation
Independent ofprogramming
languages
Helps visualize
the softwaresystem design

Inputslicenses,libraries,
operatingsystems, andversioning
information)
IT hardware
and software
components
information

Outputs

Baseline

Configuration
items

Objective

Tasks,
scheduling,
resource
allocation, etc.

Project plan

MVP

User
requirements
Design ideas

System design
documents,
Functiondesigndocument,
Test plan,
Systemdeployment
environment
configuration
plan
Potential
threats andmitigation plan

Objective

Data model

Objective ifusing adatabase

Allows softwarearchitects to
identify andmitigate
potential
security issues
early.

System design

Ensure therequired data
objects by thesystem areaccurately
represented

Systemrequirement;
Business logic

32
UNCLASSIFIED

ObjectiveUNCLASSIFIED
The activities supported by the plan phase are listed in Table 4. Some activities are suitable atenterprise or program level, such as DevSecOps ecosystem design, project team onboarding
planning, and change management planning. Others fit at the project level and are considered
continuous in the DevSecOps lifecycle.
Table 4: Plan Phase Activities
Activities

Description

Inputs

Outputs

DevSecOps
ecosystem design

Design the
DevSecOps process
workflows that are
specific to this project

Project teamonboarding
planning

Plan the project teamonboarding process,interface, access
control policyPlan the change
control process- Changemanagement
process;- System design;- Release plan &schedule.
Organization policy

DevSecOps process
flow chart;
DevSecOps ecosystem
tool selection;
Deployment platform
selection
Onboarding plan

- Organizational
policy;- Softwaredevelopment bestpractice.

Change control
procedures;
Review procedures;
Control review board;change managementplan
CM processes and plan;
CM tool selection;
Responsible
configuration items;

Changemanagement
planning

Configuration
management(CM) planning

Plan the configuration
control process;
Identify configuration
items

Softwarerequirement
analysis

Gather therequirements from all
stakeholders

System design

Design the system
based therequirements- Softwaredevelopment,
security andoperations bestpractice;- IT infrastructure
asset;- Software systemcomponents.
- Stakeholder inputs
or feedback;- Operation
monitoring
feedback;- Test feedback.
Requirements
documents

33
UNCLASSIFIED

Tool
Dependencies
Team
collaboration
systemTeam
collaboration
systemTeam
collaboration
system;Issue tracking
systemTeam
collaboration
system;Issue tracking
systemTagging strategy-Feature requirements-Performancerequirements-Privacy requirements-Security requirements

Team
collaboration
system;Issue tracking
systemDocuments:-System architecture-Functional design-Data flow diagrams-Test plan
-Infrastructure
configuration plan
-Tool selections-Development tool
-Test tool
-Deployment platform

Team
collaboration
system;Issue tracking
systemSoftware systemdesign tools

UNCLASSIFIED
Activities

Description

Project planning

Project task
management
Release planning

Risk management

Risk assessment

Configuration
identification

Discover or manual
input configuration
items into CMDB;
Establish systembaselines

Threat modeling

Identify potential
threats, weaknesses
and vulnerabilities.
Define the mitigation
plan
Data modeling;database selection;
Database deployment
topology

Database design

Inputs

Outputs
Task plan & schedule;
Release plan &schedule.

Tool
Dependencies
Team
collaboration
system;Projectmanagement
system- Systemarchitecture;- Supply chain
information;- Security risks.
-IT infrastructure
asset;- Software systemcomponents(include DevSecOps
tools);-code baselines-document
baselines.
System design

Risk management planTeam
collaboration
system;Configuration items

CMDB;
Source code
repository;
Artifact
repository;Team
collaboration
systemPotential threats andmitigation plan

Threat modeling
tool

System requirement;
System design

Database design
document

Data modeling
tool;
Team
collaboration
systemTeam
collaboration
systemTeam
collaboration
systemDesign review

Review and approveplans and documents

Plans and design
documents;

Review comments;
Action items

Documentation
version control

Track design changes

Plans and design
documents;

Version controlled
documents

4.2 Software Factory Tools and Activities
Software factory tools include a CI/CD orchestrator, a set of development tools, and a group oftools in the build, test, release, and deliver phases that are pluggable to the CI/CD orchestrator.
4.2.1

CI/CD Orchestrator

The CI/CD Orchestrator is the central automation engine of the CI/CD pipeline. It manages
pipeline creation, modification, execution, and termination.
The DevSecOps team creates a pipeline workflow in the Orchestrator by specifying a set ofstages, stage conditions, stage entrance and exit control rules, and stage activities. The
34
UNCLASSIFIEDUNCLASSIFIED
Orchestrator automates the pipeline workflow by validating the stage control rules. If all theentrance rules of a stage are met, the Orchestrator will transition the pipeline into that stage andperform the defined activities by coordinating the tools via plugins. If all the exit rules of thecurrent stage are met, the pipeline exits out the current stage and starts to validate the entrance
rules of the next stage.
Table 5 shows the features, benefits, and inputs and outputs of the CI/CD Orchestrator.
Table 5: CI/CD Orchestrator
Tool

FeaturesBenefits

Inputs

Outputs

CI/CD
Orchestrator

Create pipeline
workflow

Customizable
pipeline
solution

Pipeline workflow
configuration

Orchestratepipeline
workflow
execution bycoordinatingother plugin
tools or scripts.

Automate the
CI/CD tasks;
Auditable trailof activities

Human input about:• A set of stages• A set of event
triggers• Each stage
entrance and exit
control gate
• Activities in eachstage
Event triggers (such ascode commit, test
results, human input,etc.);
Artifacts from theartifact repository

4.2.2

BaselineMVP

Pipeline workflow
execution results(such as control
gate validation,
stage transition,
activity execution,etc.);
Event and activity
audit logs

Develop

The Develop phase uses tools to support the development activities that convert requirements
into source code. the source code includes application code, test scripts, Infrastructure as Code,
Security as Code, DevSecOps workflow scripts, etc. The development team may rely on a single
modern integrated development environment (IDE) for multiple programming language support.
The IDE code assistance feature aids developers with code completion, semantic coloring, andlibrary management to improve coding speed and quality. The integrated compiler, interpreter,
lint tools, and static code analysis plugins can catch code mistakes and suggest fixes before
developers check code into the source code repository. Source code peer review or pair
programming are other ways to ensure code quality control. All the code generated during
development must be committed to the source code repository and thus version controlled.
Committed code that breaks the build should be checked in on a branch and not merged into thetrunk until it is fixed.
The following tables list the components that facilitate code development, along with their inputs
and outputs.
35
UNCLASSIFIEDUNCLASSIFIED
Table 6: Develop Phase Tools
Tool

FeaturesBenefits

Inputs

Outputs

Integrated
development
environment(IDE)

Source code editor
Intelligent code completion
Compiler or interpreter
Debugger
Build automation(integration with a build
tool)

Developer
coding input

Source codeMVP

Integrated
development
environment(IDE)
securityplugins
Source code
repository

Scan and analyze the code
as the developer writes it,
notify developer ofpotential code weakness
and may suggest
remediation
Source code version
control
Branching and merging
Collaborative code review

Visualrepresentation
Increase
efficiency
Faster coding withless effort
Improved bug
fixing speed
Reproduciblebuilds via scripts
Address sourcecode weaknesses
and aid developers
to improve secure
coding skills

Source code;
known
weaknesses

source code
weakness
findings

Objective

Source code
Infrastructure
as codeVersioncontrolled
source code

MVP

Source code
repository
securitypluginCheck the changes for
suspicious content such as
Secure Shell (SSH) keys,
authorization tokens,passwords and othersensitive information
before pushing the changes
to the main repository.
If it finds suspicious
content, it notifies thedeveloper and blocks thecommit.
View code changes,
identify defects, reject orapprove the changes, andmake comments on specificlines. Sets review rules andautomatic notifications toensure that reviews arecompleted on time.

Compare files,
identify
differences, andmerge the changes
if needed before
committing.
Keep track ofapplication builds
Helps prevent
passwords andother sensitive
data from beingcommitted into aversion control
repository

Locally
committed
source code

Security
findings andwarnings

Objective

Automates thereview process
which in turnminimizes the taskof reviewing thecode.

Source code

Review
results (rejector accept),code
comments

Objective

Code quality
review tool

The activities supported by the develop phase are listed in Table 7.

36
UNCLASSIFIED

Baseline

UNCLASSIFIED
Table 7: Develop Phase Activities
Activities
Application codedevelopment
Infrastructure codedevelopment

Security codedevelopment
Test development

Databasedevelopment

Code commit
Code commit scan

Code review

Documentation

Static code scan
before commit

Container or VM
hardening

Description

InputsApplication coding-System components andinfrastructure orchestration
coding-Individual component
configuration script coding
Security policy enforcement
script coding
Develop detailed test
procedures, test data, test
scripts, test scenario
configuration on the specifictest tool
Implement the data modelusing data definition
language or data structure
supported by the database;
Implement triggers, views orapplicable scripts;
Implement test scripts, test
data generation scripts.
Commit source code into
version control system
Check the changes forsensitive information before
pushing the changes to themain repository.
If it finds suspicious content,
it notifies the developer andblocks the commit.
Perform code review to all
source code. Note that pair
programming counts.
Detailed implementation
documentation

Scan and analyze the code
as the developer writes it.
Notify developers ofpotential code weakness andsuggest remediation.
Harden the deliverable for
production deployment

Developer coding
input
Developer coding
input

Source code

Tool
Dependencies
IDE

Source code

IDE

Developer coding
input
Test plan

Source code

IDE

Test procedure
document;
Test data file;
Test scripts

IDE;
Specific test tool

Data model

IDE or tools
come with the
database
softwareLocally committed
source code

Database artifacts(including data
definition,
triggers, view
definitions, test
data, test data
generation scripts,
test scripts, etc.)
Version controlled
source code
Security findings
and warnings

Source code

Review comments

Code quality
review tool

User input;
Source code

Documentation;
Auto generated
Application
Programming
Interface (API)documentation
source code
weakness findings

IDE ordocument editoror build tool

Vulnerability
report andrecommended
mitigation

Container
security tool
Security
compliance tool

Source code

Source code;
known
weaknesses

Running VM orcontainer

37
UNCLASSIFIED

Outputs

Source code
repository
Source code
repository
security plugin

IDE security
plugins

UNCLASSIFIED
4.2.3

Build

The build tools perform the tasks of building and packaging applications, services, andmicroservices into artifacts. For languages like C++, building starts with compiling and linking.
The former is the act of turning source code into object code and the latter is the act of
combining object code with libraries to create an executable file. For Java Virtual Machine(JVM) based languages, building starts with compiling to class files, then building a compressed
file such as a jar, war or ear file, which includes some metadata, and may include other files suchas icon images. For interpreted languages, such as Python or JavaScript, there is no need tocompile, but lint tools help to check for some potential errors such as syntax errors. Building
should also include generating documentation, such as Javadoc, copying files like libraries or
icons to appropriate locations, and creating a distributable file such as a tar or zip file. The build
script should also include targets for running automated unit tests.
Modern build tools can also be integrated into both an IDE and a source code repository toenable building both during development and after committing. For those applications that use
containers, the build stage also includes a containerization tool.
The following tables list build-related tools along with their inputs and outputs.
Table 8: Build Phase Tools
Tool
Build tool

Lint tool

Containerbuilder

Features
Dependency
Management
CompileLink (if appropriate)
Built-in lint stylistic
checking
Integration with IDE
Analyzes source code toflag programming
errors, bugs, stylistic
errors, and suspicious
constructs.
Applicable to bothcompiled or interpreted
languages
Build a container image
based on a build
instruction file

Benefits

Inputs

Outputs

Reduces human
mistakes
Saves time

Source code
under version
control
Artifacts

Binaryartifacts
stored in
the Artifact
repository

MVP

Improve codereadability;
Pre-code review;
Finding (syntax)
errors before
execution forinterpreted
languages
Container image
build automation

Source code or
scripts

Analyze
results

Objective

Container baseimage;
Container build
file

OCI
compliantcontainer
image

MVP

38
UNCLASSIFIED

Baseline

UNCLASSIFIED
Tool
Artifact
Repository

FeaturesBenefits

Binary artifact version
control;
Container registry

Separate binary
control from source
control to avoid
external access tosource control
system.

Inputs

Outputs

Baseline

Artifacts

Versioncontrolled
artifacts

MVP

Source code;
known
vulnerabilities
and weaknesses

Static code
scan report
andrecommended
mitigation.

MVP

Dependency list
or BOM list

Vulnerabili
ty report

Objective

Improved build
stability by
reducing relianceon external
repositories.

StaticApplication
Security Test
(SAST) tool

Dependency
checking /Billof Materials(BOM)
checking tool

SAST analyzes
application static codes,
such as source code,
byte code, binary code,
while they are in a nonrunning state to detect
the conditions thatindicate code
weaknesses.
Identify vulnerabilities
in the dependent
components based on
publicly disclosed open
source vulnerabilities

Better quality
software by
avoiding outdated
artifacts withknown issues.
Catch code
weaknesses at an
early stage.
Continuous
assessment during
development.

Secure the overall
application;
Manage the supply
chain risk

The activities supported by the build phase are listed in Table 9.
Table 9: Build Phase Activities
Activities

Description

Inputs

Build

Compile and link

Source code;
dependencies

Binary artifacts

Staticapplication
security test andscan
Dependency
vulnerability
checking
Containerize

Perform SAST to thesoftware system

Source code;
known
vulnerabilities andweaknesses
Dependency list or
BOM list

Static code
scan report andrecommended
mitigation.
Vulnerability
reportContainer baseimage;
Container build file

Container
image

Identify vulnerabilities
in the open source
dependent components
Packages all requiredcomponents OS,
developed code,

39
UNCLASSIFIED

Outputs

Tool
Dependencies
Build tool;
Lint tool;
Artifact repository
SAST tool

Dependency checking /BOM checking tool
Container builder

UNCLASSIFIED
libraries, etc.) into ahardened container
Package binary artifacts,
container or VM images,
infrastructure
configuration scripts,
proper test scripts,
documentation,
checksum, digital
signatures, and release
notes as a package.
Store artifacts to theartifact repository

Releasepackaging

Store artifacts

Buildconfiguration
control and audit

4.2.4

Track build results,
SAST and dependency
checking report;
Generate action items;
Make go/no-go decision
to the next phase

Binary artifacts;
Scripts;
Documentation;
Release notes

Releasedpackage withchecksum anddigitalsignature

Release packaging tool

Binary artifacts;
Database artifacts;
Scripts;
Documentation;
Container images
Build results;
SAST report;
Dependency
checking report

Versioned
controlled
artifacts

Artifact Repository

Versioncontrolledbuild report;
Action items;
Go/no-godecision

Team collaboration
system;Issue tracking system;
CI/CD orchestrator

Test

Test tools support continuous testing across the software development lifecycle. Test activities
may include, but are not limited to, unit test, functional test, integration test, system test,
regression test, acceptance test, performance test, and variety of security tests. . Mission
programs can select their own test activities and merge several tests together based on the natureof their software and environment. All tests start with test development, which develops detailed
test procedures, test scenarios, test scripts, and test data. Automated test can be executed byrunning a set of test scripts or running a set of test scenarios on the specific test tool without
human intervention. If full automation is not possible, the highest percentage of automation isdesired. It is highly recommended to leverage emulation and simulation to test proper integration
between components such as microservices and various sensors/systems, so integration testing
can be automated as much as possible. Automation will help achieve high test coverage andmake continuous ATO practicable, as well as significantly increase the quality of delivered
software.
The components involved with the test phase are listed in the following table.
Table 10: Test Phase Tools
Tool

FeaturesBenefits

InputsTestdevelopment
tool

Assists test scenario, test
script, and test data
development.
The specific tool varies,
depending on the test
activity (such as unit test,

Increase theautomation andrate of testing

Test plan

40
UNCLASSIFIED

Outputs
test scenarios,
test scripts,
test data

BaselineMVP

UNCLASSIFIED
Tool

Test data
generator

Test tool
suite

Test
coverage tool

Test
Management
Tool

Non-security
compliancescan
Softwarelicensecompliancechecker
Dynamic
Application
Security Test(DAST) toolInteractive
Application
Security Test
(IAST) tool

Features

Inputs

Outputs

Baseline

Increase test
fidelity

Test scenario,
test data

Input data forthe system
under test

Objective

Increase test
automation, speed

Test scenario,
test scripts,
test data

Test results,
test report

MVP

Shows the fidelityof the test results

Applicationcode,
automated
tests

MVP

Manages requirements,
streamlines test case designfrom requirements, plans
test activities, manages test
environment, tracks test
status and results.
Such as Section 508
accessibility compliance

Increases QA
team collaborationand streamlines
test processes.

Requirements, test cases,
test results

The
percentage ofcode that is
exercised by
the tests.
Test progress,
test results
statistics

Ensures
compliance

Artifacts

CompliancereportObjective

Inventory software license;
Audit the compliance.

Software licensecompliance andsoftware asset
management
Catch the dynamic
code weakness inruntime and undercertainenvironment
setting.
Identify and fix
issues duringcontinuous
integration.
Provide accurate
results for fast
triage; pinpoint
the source ofvulnerabilities

Purchased
license info;
Softwareinstances
Running
softwareapplication;
fuzz inputs

CompliancereportObjective

dynamic code
scan report andrecommended
mitigation.

Objective

Running
application,and operating
systems;
Fuzz inputs

Analysisreport andrecommended
mitigation.

Objective

penetration test) and theapplication type (e.g., web
application, or Hadoop data
analytics)
Generates test data for thesystem (such as network
traffic generator, web
request generator)
A set of test tools toperform unit test, interface
test, system test, integration
test, performance test andacceptance test of thesoftware system.
Generate test report
Specific tool varies
depending on the type oftests, software application,and programming language
Measures how much codeis exercised while theautomated tests are running

DAST tools analyze arunning application
dynamically and canidentify runtime
vulnerabilities andenvironment related issues.

Analyze code for security
vulnerabilities while theapplication is run by anauto-test, human tester, or
any activity “interacting”

Benefits

41
UNCLASSIFIED

ObjectiveUNCLASSIFIED
ToolContainer
security tool

Container
policyenforcement

Security
compliancetool

Network
security test
tool

Database test
tool suite

Databasesecurity scanand test tool

Featureswith the application
functionality
Container image scan
OS check

Support for Security
Content Automation
Protocol (SCAP) andContainer configuration
policies. These policies canbe defined as needed.
Scan and report forcompliance regulations,such as DISA Security
Technical Implementation
Guides (STIGs), NIST
800-53.
Simulate real-world
legitimate traffic,
distributed denial of service(DDOS), exploits,
malware, and fuzzing.
Tools that facilitate
database test;
It includes test data
generator, databasefunctional test tool,
database load test tool;
Find the database common
security vulnerabilities,
such as weak password,
known configuration risks,missing patches;
Structured Query Language(SQL) injection test tool;
Data access control test;
User access control test;
Denial of service test

Benefits

Inputs

Outputs

Baseline

Ease the container
hardening process

Vulnerability
report andrecommended
mitigation.
CompliancereportMVP

Automated policy
enforcement

Container
images or
running
containers
Policies in
SCAP form.

Speed up ATO
process.

Container
images.

Vulnerability
report andrecommended
mitigation.

Objective

Validate systemsecurity; increaseattack readiness;
reduce the risk ofsystemdegradation.
Automate or semiautomate thedatabase tests

Test
configuration

Test traffic

Objective

Test data;
Test scenario

Test results

Objective
if using a
database

Reduce thesecurity risks

Test data;
Test scenarios

Vulnerability
findings;
Recommended
mitigation
actions

Objective
if using a
databaseMVP

The activities supported by the test phase are listed in Table 11. These activities happen at
different test stages.
•

Development stage: unit test, SAST discussed in the build phase•

System test stage: DAST or IAST, integration test, system test•

Pre-production stage: manual security test, performance test, regression test, acceptancetest, container policy enforcement, and compliance scan

Test audit, test deployment, and configuration audit happen at all stages.
42
UNCLASSIFIEDUNCLASSIFIED
Table 11: Test Phase Activities
Activities

Description

Unit test

Assist unit test script
development and unit test
execution. It is typicallylanguage specific.

Dynamic
application
security test andscan

Perform DAST or IAST
testing to the software system

Integration test

Develops the integration test
scripts and execute the scripts
to test several software units
as a group with the interaction
between the units as the focus.
System test uses a set of tools
to test the complete softwaresystem and its interaction withusers or other external
systems.

System test

Manual security
test

Performance test

Regression test

Such as penetration test,
which uses a set of tools andprocedures to evaluate thesecurity of the system byinjecting authorized simulated
cyber-attacks to the system.
CI/CD orchestrator does not
automate the test, but the test
results can be a control pointin the pipeline.
Ensure applications will
perform well under the
expected workload. The test
focus is on application
response time, reliability,
resource usage and scalability.
A type of software testing toconfirm that a recent program
or code change has notadversely affected existing
features.

Inputs

Outputs

Unit test script,
individual software
unit under test (afunction, method oran interface), test
input data, andexpected output data
Running application
and underlying OS;
fuzz inputs

Test report todeterminewhether theindividual
software unit
performs as
designed.
Vulnerability,
static code
weakness and/or
dynamic code
weakness report
andrecommended
mitigation
Test report about
whether the
integrated units
performed as
designed.
Test result about
if the system
performs as
designed.

Integration test
scripts, the software
units under test, test
input data, andexpected output data
System test scripts,
the software systemand external
dependencies, test
input data andexpected output data
Running
application,
underlying OS, andhosting environment

Tool
Dependencies
Test tool suite,
Test coverage
tool

DAST tool or
IAST tool

Test tool suite

Test tool suite

Vulnerability
report andrecommended
mitigation

Varies tools andscripts (mayinclude network
security test
tool)

Test case, test data,
and the softwaresystemPerformance
metrics

Test tool suite,
Test data
generator

Functional and nonfunctional
regression test
cases; the softwaresystemTest report

Test tool suite

43
UNCLASSIFIEDUNCLASSIFIED
Activities

Description

Inputs

Acceptance test

Conduct operational readinesstest of the system. It generally
includes:Accessibility and usability test
failover and recovery test
performance, stress andvolume test
security and penetration test
interoperability testcompatibility test
supportability andmaintainability
Check developed containers tobe sure they meet container
policies
Compliance audit

The tested system
Supporting system
Test data

Test report

Container, Policies
in SCAP form

Container
compliancereportCompliancereports

Test audit keeps who performs
what test at what time and test
results in records
Deploy application and set up
testing environment using
Infrastructure as Code

Test activity and test
results

Test audit log

Artifacts(application
artifacts, test code)
Infrastructure as
Code
Test data

The
environment
ready to run
tests
Test results

Database test
tools

Test data;
Test scenarios

Test results

Database test
tools

Test data;
Test scenarios

Test results

Test results;
Security scan andcompliance scanreportVersioncontrolled test
results;
Action items;
Go/no-godecision

Vulnerability
findings;
Recommended
mitigation
actionsTeam
collaboration
system;Issue tracking
system;CI/CDorchestrator

Container policy
enforcement
Compliancescan

Test audit

Test deployment

Databasefunctional test

Database nonfunctional test
Databasesecurity test

Test
configuration
control and audit

Perform unit test andfunctional test to database toverify the data definition,
triggers, constrains are
implemented as expected
Conduct performance test,
load test, and stress test;
Conduct failover test
Perform security scan;
Security test

Track test and security scan
results;
Generate action items;
Make go/no-go decision to thenext phase.

Artifacts;
Software instances;
System components

44
UNCLASSIFIED

Outputs

Tool
Dependencies
Test tool suite,
Non-security
compliance scan

Container policy
enforcement
Non-security
compliancescan;
Software licensecompliancechecker;
Security
compliance tool
Test
management
tool
Configuration
automation tool;
IaC

UNCLASSIFIED
Activities

Description

Inputs

Outputs

Tool
Dependencies(There may be several
iterations for several tests
across stages)

4.2.5

Release and Deliver

In the release and deliver phase, the software artifacts are digitally signed to verify that they havepassed build, all tests, and security scans. They are then delivered to the artifact repository. The
content of the artifacts depends on the application. It may include, but is not limited to, container
images, VM images, binary executables (such as jar, war, ear files), test results, security scan
results, and Infrastructure as Code deployment scripts. Artifacts will be tagged with the release
tag if GO release decision is made based on the configuration audit results. The artifacts with therelease tag are delivered to production.
Table 12: Release and Deliver Phase Tools
Tool

FeaturesBenefits

Inputs

Outputs

Baseline

Releasepackagingtool

Package binary
artifacts, container or
VM images,
infrastructure
configuration scripts,
proper test scripts,
documentation, release
notes as a package;generate checksum anddigital signature for the
package.

Release package(such as a bundleof artifacts, selfextracting
software installer,
software tar file,etc.)

Binary artifacts,base containers
or VM images,
infrastructure
configuration
scripts, proper
test scripts,
documentation,
release notes

Releasepackage withchecksumand digital
signature (abundle ofartifacts, suchas a selfextracting
softwareinstaller, or atar file, etc.)

MVP ifusing
VMs

The package may be
prepared for a specificinstaller or it is a selfextracting installer
itself.

The mission program could have more than one artifact repository, though more likely there is acentralized one and tags separate artifact types. One artifact repository (or set of tags) is used in
the build stage to store build results. The test deployment activity can fetch the artifacts from thebuild stage artifact repository to deploy the application into various environments (development,
test, or pre-production). Another artifact repository (or set of tags) may be used by the
production environment, which is the one that the store artifacts stage uses to push the final
deliverables to production. The production deployment will get all the artifacts from theproduction artifact repository to deploy the application.
Some mission program application systems have geographically distributed operational regions
across the country or even overseas. In order to increase deployment velocity, a remote
operational region may have its own local artifact repository that replicates the artifact repository
45
UNCLASSIFIEDUNCLASSIFIED
completely or partially. During release, a new artifact is pushed into the artifact repository andthen replicated to other regional artifact repositories.
The activities supported by the release and deliver phase are listed below.
Table 13: Release and Deliver Phase Activities
Activities

Description

Inputs

Outputs
go / no-go
decision;
Artifacts
are taggedwithrelease tag
if go
decision ismadeNew
release in
the artifact
repository
Artifacts inall regional
artifact
repositories

Release go / nogo decisionThis is part of configuration audit;
Decision on whether to release artifacts tothe artifact repository for the production
environment.

Designdocumentation;
Test reports;
Security test
and scanreports;
Artifacts

Deliver released
artifacts

Push released artifacts to the artifact
repository

The releasepackage

Artifacts
replication

Replicate newly release artifacts to all
regional artifact repositories

Artifacts

Tool
Dependency
CI/CD
Orchestrator

Artifacts
repository

Artifacts
repositories(release,regional)

4.3 Production Operation Tools and Activities
The production operations tools provide the capability to deploy artifacts, including containers
and VM images, to the production environment, and to monitor their operations. For Cloudnative applications, it is recommended to use containers whenever possible over virtual machines
due to the baked-in security provided by the Sidecar Container Security Stack.
4.3.1

DeployThe tools used in deploy phase are environment and deployment mode dependent.
4.3.1.1

Virtual Machine deployment

While it is highly recommended to leverage containers for new system design anddevelopment, if the application is deployed as a VM, the virtualization manager in the
hosting environment is the key component with which IaC will interface to deploy andconfigure the application system. The virtualization manager manages the virtual compute,storage and network resources. In some hosting environments, such as a general-purpose
cloud, the virtualization manager also provides some security capabilities, such as microsegmentation, which creates security zones to isolate VMs from one another and secure them
individually. Several capabilities of the virtualization manager are keys to the success of
mission application runtime operation and security, such as health checking, virtual resource
46
UNCLASSIFIEDUNCLASSIFIED
monitoring, and scaling. The application production environment infrastructure has toleverage these capabilities in its architecture and configuration.
The use of “clones” from a master image library enables VMs to be created quickly. A clone
is made from a snapshot of the master image. The use of clones also enables the concept of
immutable infrastructure by pushing updated, clean images to the VM each time it is started.
Only the master image needs to be patched or updated with the latest developed code; eachrunning image is restarted to pick up these changes.
4.3.1.2

Container deployment

A container manager provides capabilities that check for new versions of containers, deploys
the containers to the production environment, and performs post-deployment checkout.
The container manager consists of an OCI-compliant container runtime and a CNCF-certified
Kubernetes, which is an orchestration tool for managing microservices or containerized
applications across a cluster of nodes. The nodes could be bare metal servers or VMs. The
container manager may be owned by a mission program or provided by the cloud hosting
environment. It simplifies container management tasks, such as instantiation, configuration,scaling, monitoring, and rolling updates. The CNCF-certified Kubernetes interacts with theunderlying virtualization manager in the cloud environment to ensure each node’s health andperformance, and scale it as needed. This scaling includes container scaling within the
CNCF-certified Kubernetes cluster, but when running in a cloud, it also includes the ability
to auto-scale number of nodes in a cluster by adding or deleting VMs.
The following tables list deployment-related tools and their inputs and outputs.
Table 14: Deploy Phase Tools
Tool

FeaturesBenefits

Inputs

Outputs

Baseline

Virtualization
Manager

VM instancemanagement
VM resource monitoring(provided on hosting
environment)
Container grouping using
pods
Health checks and selfhealing
Horizontal infrastructure
scaling
Container auto-scalability
Domain Name Service(DNS) management
Load balancing
Rolling update or
rollback
Resource monitoring andlogging

Centralized VM
instantiation,scaling, andmonitoring

VM instancespecificationand monitoring
policyRunning VM

MVP if using
VMs

Simplify
operations bydeployment andupdateautomation

Container
instance
specificationand monitoring
policyRunning
containerMVP

CNCFcertified
Kubernetes

Scale resources
and applications
in real time
Cost savings by
optimizing
infrastructure
resources

47
UNCLASSIFIEDUNCLASSIFIED
Tool

FeaturesBenefits

Inputs

Outputs

Baseline

Data masking
tool

Shield personallyidentifiable informationor other confidential data

Original data

Masked data

Objective if
database
contains
sensitive data

Databaseencryption
tool

Encrypt data at rest andin transit

Provide data
privacy;
Reduce the riskof data loss
during data
breachProvide data
privacy andsecurity;
Prevent data loss

Original data

Encrypted data

Database
automation
tool

Automate database tasks,
such as deployments,
upgrades, discoveringand troubleshooting
anomalies, recoveringfrom failures, topology
changes, running
backups, verifying data
integrity, and scaling.
Execute the configuration
scripts to provision theinfrastructure, security
policy, environment, andthe application systemcomponents.
Ability to create a
network of deployed
services with loadbalancing, service-toservice authentication,and monitoring.

Simplify
database
operations andreduce humanerrorsDatabase
artifacts;
Data;
Running status
and events

Status report;
Warnings;
alertsMVP ifdatabase
contains
highlysensitive data
Objective ifusing adatabase

Configuration
automation
Consistent
provisioning

Infrastructure
configuration
scripts
Infrastructure
configuration
data
Control plane:
servicecommunication
routing
policies,
authentication
certificates.
Data plane:
servicecommunication
data

Provisioned
deployment
infrastructureMVP

Control plane:
service status
reports
Data plane:routed service
communication
data

MVP

Configuration
automation
tools

Service mesh

Support for
microservice
interactions.

The activities supported by the deploy phase are listed in Table 15.
Table 15: Deploy Phase Activities
Activities

Description

Inputs

Outputs

Artifact download

Download newly release artifacts from theartifact repository

Requested
artifacts

Infrastructure
provisioning
automation

Infrastructure systems auto provisioning(such as software defined networking,firewalls, DNS, auditing and logging
system, user/group permissions, etc.)

Artifact
downloadrequestInfrastructure
configuration
scripts /recipes /manifests /playbooks

48
UNCLASSIFIED

Provisioned
andconfigured
infrastructure

Tool
Dependency
Artifact
repository
Configuration
automation
tools; IaC

UNCLASSIFIED
Activities

Description

InputsCreate linked
clone of VM
master image

Instantiate VM by creating a link clone ofparent VM with master image

New VM
instance

Deliver container
to container
registry

Upload the hardened container andassociated artifacts to the container registry

VM parent
New VM
instance
parameters
Hardened
container

Post-deployment
security scan

System and infrastructure security scan

Security
vulnerability
findings

Post-deployment
checkout

Run automated test to make sure theimportant functions of system are working

Test results

Test scripts

Database
installation anddatabase artifact
deployment

Database software installation; Cluster orhigh availability setup;
Database artifacts deployment and dataloading

Access to
systemcomponents
andinfrastructurecomponents
Smoke test
scenarios andtest scripts
Artifacts intherepository;data

CNCFcertified
Kubernetes;
Artifact
repository
container
registry
Security
compliancetool

Running
databasesystemArtifact
repository;
Database
automation
tool;
Data masking
or encryption
tool if needed

4.3.2

Outputs

New
container
instance

Tool
Dependency
Virtualization
Manager

OperateThe Operate phase uses tools for system scaling, load balancing, and backup.
Load balancing monitors resource consumption and demand, and then distributes the workloads
across the system resources. Scaling helps dynamic resource allocation based on demand. Bothvirtualization manager and CNCF-certified Kubernetes support load balancing and scaling
capabilities. CNCF-certified Kubernetes handles the load balancing and scaling at the container
level. The virtualization manager works at the VM level.
Application deployment must have proper load balancing and scaling policies configured withthe virtualization manager or the CNCF-certified Kubernetes based on VM deployment or
container deployment respectively. During runtime, the management layer will continuously
monitor the resources. If the configured threshold is met (for example if memory or Central
Processing Unit (CPU) usage meets a pre-set threshold), then the system triggers the loadbalancing or scaling action automatically. Auto-scaling must be able to scale both up and down.

49
UNCLASSIFIEDUNCLASSIFIED
Table 16: Operate Phase Tools
Tool

FeaturesBenefits

Inputs

Outputs

Backupmanagement

Data backup
System components (VMor container) snapshot
Provide operators a visualview of operations status,
alerts, and actions.

Improvefailurerecovery
Improve
operations
management

Access to the
backup source

Backup data
System VM orcontainer snapshot
Dashboard display

Operations
dashboard

All operational
monitoring
status, alerts,
andrecommended
actions

BaselineMVP

Objective

The activities supported by the operate phase are listed in the table below.
Table 17: Operate Phase Activities
Activities

Description

Inputs

Outputs

Backup

Data backup;
System backup

Access to backup
systemBackup data or
image

Scale

Scale manages
VMs/containers as agroup. The number of
VMs/containers in thegroup can be
dynamically changed
based on the demandand policy.

Real-time demand andVM/container
performance measures
Scale policy (demandor Key Performance
Indicator(KPI)threshold;minimum, desired, andmaximum number of
VMs/containers)
Load balance policy
Real time traffic load
and VM/container
performance measures

Optimized
resource
allocation

Load balancing

Load balancing
equalizes the resource
utilization

Tool
Dependency
Backupmanagement;
Database
automation tool
VM management
capability on the
hostingenvironment;
Container
management on the
hosting environment

Balancedresource
utilization

VM management
capability on the
hostingenvironment;
Container
management on the
hosting environment

4.3.3

MonitorIn the monitor phase, tools are utilized to collect and assess key information about the use of theapplication to discover trends and identify problem areas. Monitoring spans the underlying
hardware resources, network transport, applications / microservices, containers, interfaces,
normal and anomalous endpoint behavior, and security event log analysis.
It also includes behavior and signature-based detection in the runtime environment. All thesesecurity capabilities are mapped against the NIST controls and follow NIST Special Publication
800-190 Application Container Security Guide [12] for continuous compliance.
50
UNCLASSIFIEDUNCLASSIFIED
Table 18: Monitor Phase Tools
Tool

FeaturesBenefits

Inputs

Logging

Logging events for alluser, network, application,and data activities

Assisttroubleshooting
the issues.
Assistdetection ofadvancedpersistent
threats andforensics.

All user,network,
application,and data
activities

Event logs

MVP

Log
aggregator

Filter log files for eventsof interest (e.g., security),
and transform intocanonical format
Analyze and audit todetect malicious threats /activity;
Automated alerting andworkflows for response
Forensics for damage
assessment
Report various
performance metrics suchas resource utilization
rates, number ofconcurrent user sessions,and Input/Output (IO)
rates;
Provide dashboards todisplay performance;
Alert performance issues
Establish a baseline forcomparison
Monitor network security
Monitor personnel activity
Monitor configuration
changes
Perform periodical
security scan to all systemcomponents
Monitor the IT assets anddetect deviations fromsecurity, fault tolerance,
performance bestpractices.
Monitor and analyze log
files
Audit IT asset’s
configuration compliance
Detect and block
malicious code

Logs

Aggregated,filtered,
formatted event
log
Alert messages,
emails, etc.
Remediation
report and log

MVP

Log
analysis &auditing

Operations
monitoring

Information
Security
Continuous
Monitoring(ISCM)

Logs

Outputs

BaselineMVP

Improve
operations
continuity
Identify thearea to improve
Better end-user
experience

Performance
KPI andService Level
Agreement(SLA)

Performancestatistics
PerformancealertsMVP

Detectunauthorized
personnel,
connections,devices, andsoftwareIdentify
cybersecurity
vulnerability
Detect security
andcomplianceviolation
Verify theeffectivenessof protective
measures

IT asset
Network
Personnel
activities
Known
vulnerabilities

Vulnerabilities
Incompliance
Findings,
assessments andrecommendations

MVP

51
UNCLASSIFIEDUNCLASSIFIED
Tool

Alerting
andnotification

Databasemonitoring
tool
Databasesecurity
audit tool

Features
Continuous security
vulnerability assessments
and scans
Provide browse, filter,search, visualize, analysis
capabilities
Generate findings,
assessments andrecommendations.
Provide recommendations
and/or tools for
remediating any noncompliant IT asset and/or
IT workload.
Notify security teams
and/or administrators
about detected events.
Support automatic
remediation of highpriority time-criticalevents.

Baseline database
performance and databasetraffic;
Detect anomalies
Perform user access anddata access audit;
Detect anomalies fromevents correlation;
Detect SQL injection;
Generate alert

Benefits

Inputs

Outputs

Improve
visibility ofsystem events
Reduce system
downtime
Improvecustomer
serviceLogs,
monitoringdata.
Supportautomatic
remediationof highpriority timecriticalevents.
Running
database

Alert messages,
emails, etc.
Remediation
reportIssue ticket

MVP

Logs;
Warnings andalertsObjective ifusing adatabase

Running
database

Audit logs;
Warnings andalertsMVP if using
a database

Improve
database
operations
continuity
Enhance
databasesecurity

Baseline

The activities supported by the monitor phase are listed in Table 19.
Table 19: Monitor Phase Activities
Activities

Description

Logging

Log system events

Log analysis &auditing

Filter or aggregate logs;
Analyze and correlate logs

Systemperformancemonitoring

Monitor system hardware,
software, database, andnetwork performance;Baselining system
performance;

Inputs

Outputs

All user,network,
application, anddata activities
Logs

Logs

Running system

Performance KPI
measures;
Recommended
actions;
Warnings or alerts

52
UNCLASSIFIED

Alerts andremediation report

Tool
Dependencies
Logging

Log aggregator
Log analysis &auditing
Operation
monitoring
Issue tracking
system; Alertingand notification;

UNCLASSIFIED
Detect anomalies
System Security
monitoring

Monitor security of all systemcomponents
Security vulnerability
assessment
System security compliancescan

Running system

Vulnerabilities;
Incompliance
Findings; assessments
andrecommendations;
Warnings and alerts.

Asset Inventory

Inventory system IT assets

IT assets

Asset inventory

Systemconfiguration
monitoring

System configuration(infrastructure components
and software) compliancechecking, analysis, andreporting

Running system
configuration;
Configuration
baseline

Compliance report;
Recommended
actions;
Warnings and alerts

Databasemonitoring andsecurity auditing

Database performance andactivities monitoring andauditing

Database traffic,
event, andactivities

Logs;
Warnings and alerts

Operations
dashboard
ISCM;
Issue tracking
system;Alerting andnotification;
Operations
dashboard
Inventory
Management;
ISCM;
Issue tracking
system; Alertingand notification;
Operations
dashboard
Databasemonitoring tool;
Databasesecurity audit
tool;Issue tracking
system;Alerting andnotification;
Operations
dashboard

4.4 Security Tools and Activities Summary
Security is not a separate phase of the DevSecOps lifecycle; rather security activities occur in allphases. This DevSecOps security practice facilitates automated risk characterization, monitoring,and mitigation across the application lifecycle. Table 20 summarizes the security activities of allphases.
Table 20: Security Activities Summary
Activities

Phase

Threat modeling

Plan

Activities
Table
Reference
Table 4Tool Dependencies

Tool Table
Reference

Security codedevelopment
Static code scan before
commitCode commit scan

Develop

Table 7

IDE

Table 6Develop

Table 7

IDE security plugins

Table 6Develop

Table 7

Table 6

Container or virtual
machine hardening
Static application security
test and scan

Develop

Table 7

Build

Table 9Source code repository security
pluginContainer security tool
Security compliance tool
SAST toolThreat modeling tool

Table 353
UNCLASSIFIED

Table 10
Table 8

UNCLASSIFIED
Dependency
vulnerability checking
Dynamic application
security test and scan
Manual security testing(such as penetration test)
Container policy
enforcement
Post-deployment security
scan
System Security
monitoring

Build

Table 9

Table 8

Table 11

Dependency checking / BOM
checking tool
DAST tool or
IAST tool
Varies tools and scripts (mayinclude network security test tool)
Container policy enforcement

Test

Table 11

Test

Table 11

Test
DeployTable 15

Security compliance tool

Table 10

MonitorTable 19

Information Security Continuous
Monitoring (ISCM)

Table 18

Table 10
Table 10
Table 10

4.5 Configuration Management Tools and Activities Summary
Configuration management plays a key role in DevSecOps practice. It ensures the configurationof a software system’s infrastructure, software components, and functionalities are not only
known initially but also knowable and well controlled throughout the DevSecOps lifecycle.
Configuration management consists of three sets of activities:•

Configuration identification: identify the configuration items. This can be done manuallyor with assistance from a discovery tool. The configuration items include infrastructurecomponents, COTS or open source software components used in the system, documented
software design, features, software code or scripts, artifacts, etc.

•

Configuration control: control the changes of the configuration items. Each configuration
item has its own attributes, such as model number, version, configuration setup, license,
etc. The CMDB, source code repository, and artifact repository are tools to track andcontrol the changes. The source code repository is used primarily during development.
The other two are used in both development and operations.

•

Configuration verification and audit: verify that the configuration items meet the
documented requirements and design. Configuration verification and audit are control
gates along a pipeline to control the go/no-go decision to the next phase.
Table 21: Configuration Management Activities Summary
Activities

Phase

Configuration
management
planning
Configuration
identification
Design review

Plan

Activities Table
Reference
Table 4Tool Dependencies

Plan

Table 4

CMDB

Table 3

Plan

Table 4

Team collaboration system

Table 3Documentation
version control
Code review

Plan

Table 4

Team collaboration system

Table 3Develop

Table 7

Code quality review tool

Table 6

Team collaboration system;
Issue tracking system

54
UNCLASSIFIED

Tool Table
Reference
Table 3

UNCLASSIFIED
Activities

Phase

Code Commit

Develop

Activities Table
Reference
Table 7Tool Dependencies
Source code repository

Tool Table
Reference
Table 6Store artifacts

Build

Table 9Artifact repository

Table 8

Build phase
configuration control
and audit
Test phase
configuration control
and audit
Release go / no-go
decision
Infrastructure
provisioning
automation
Post-deployment
security scan
Post-deployment
checkout
Asset inventory

Build

Table 9

Table 3
Table 5

Test

Table 11

ReleaseTable 13

Team collaboration system;
Issue tracking system
CI/CD orchestrator
Team collaboration system;
Issue tracking system
CI/CD orchestrator
CI/CD orchestrator

DeployTable 15

Configuration automation
tool

Table 14

DeployTable 15

Security compliance tool

Table 10

DeployTable 15

Test scripts

MonitorTable 19

Asset inventory tool

Table 18

System performancemonitoring

MonitorTable 19

Table 3
Table 16
Table 18

System configuration
monitoring

MonitorTable 19

Operation monitoring
Issue tracking system;
Alerting and notification;
Operations dashboard
ISCM;
Issue tracking system;
Alerting and notification;
Operations dashboard

Table 3
Table 5
Table 5

Table 3
Table 16
Table 18

4.6 Database Management Tools and Activities Summary
Databases are commonly used in the DoD software systems. They hold some of the most criticalinformation of an enterprise or a mission and are typically the center piece of the softwaresystem. Data security and privacy protection are paramount to enterprises and missions.
Relational databases continue to be a prime target for data thieves, and security vulnerabilities
are compound by adoption of big data platforms, such as Hadoop, NoSQL databases, andDatabase as a Service (DBaaS) in the cloud. Here we discuss some database activities throughout
the DevSecOps lifecycle to improve database security and operations.
In development phases, database design, development, and testing activities generate database
artifacts, which are data models, database schema files, trigger definitions, view definition, test
data, test data generation scripts, test scripts, etc. These database artifacts must be underconfiguration management control. During test phase, database functional test is like application
code unit test and functional test to validate the schema, triggers, and data compliance. The nonfunctional test includes load testing, stress test, and performance test. The security test focuseson vulnerability scan, user authentication and authorization, unauthorized access to data, dataencryption, privilege elevation, SQL injection, and denial of service.
55
UNCLASSIFIEDUNCLASSIFIED
During operations, the deployment and operational activities can be automated via databaseautomation tools. The continuous monitoring is achieved using database monitoring tool andsecurity audit tool.
Table 22 summarizes the database activities of all phases.
Table 22: Database Management Activities Summary
Activities

Phase

Database design

Plan

Activities Table
Reference
Table 4

Database development

Develop

Table 7

Code review (database
schemas, codes)
Code Commit (database
schemas, codes
Store artifacts (database
artifacts)
Database functional test

Develop

Tool Dependencies
Data modeling tool

Tool Table
Reference
Table 3
Table 6

Table 7

IDE or tools come with the
database software
Code quality review tool

Develop

Table 7

Source code repository

Table 6

Build

Table 9Artifact repository

Table 8

Test

Table 11

Database test tool

Table 10

Database non-functional
test
Database security test

Test

Table 11

Database test tool

Table 10

Test

Table 11

Database security test tool

Table 10

Database installation anddatabase artifact
deployment

Test and
DeployTable 11
Table 15

Table 8
Table 14

BackupOperateTable 17

Artifact repository;
Database automation tool;
Data masking or encryption
tool if needed
Database automation tool

Database monitoring andsecurity auditing

MonitorTable 19

Database monitoring tool;
Database security audit tool

Table 1856
UNCLASSIFIED

Table 6

Table 14

UNCLASSIFIED

5

DoD Enterprise DevSecOps Container Service

The DoD Enterprise DevSecOps Container Service creates DevSecOps hardened containers andprovides hardened container access service to DoD programs to instantiate their own DevSecOps
ecosystem.

Figure 10: DoD Enterprise DevSecOps Container Service Architecture
Figure 10 illustrates the DoD Enterprise DevSecOps Container Service architecture. It contains a
DoD Enterprise DevSecOps Container Factory and a DoD Centralized Artifact Repository(DCAR). The Container Factory takes public container images as input and automates thecontainer hardening process to produce the hardened container images. The DCAR stores thehardened container images and allows DoD programs access these images.

5.1 DoD Enterprise DevSecOps Container Factory
The Container Factory produces the hardened containers of DevSecOps tools. It is imperative for
the Container Factory to automate its hardening process as much as possible. It does this byleveraging CI/CD pipelines in an instance of the software factory specifically configured for
hardening of DevSecOps tool containers.
5.1.1

DoD Hardened Containers

A DoD hardened container is an Open Container Image (OCI) compliant image that is secured
and made compliant with the DoD Container Hardening Security Requirements Guide [6].
57
UNCLASSIFIEDUNCLASSIFIED
Container images should adhere to the OCI Image Format Specification to ensure portability.
Each hardened container includes the “global configuration”, which includes all security
hardening configuration. The packaged container is tagged with integrity metadata, such as adigital signature or a digital hash. Tags may be implemented as metadata bound to the object, oras attributes in a file associated with the object. Some configuration values can be changed for
local use, such as the DNS location. These “local configuration” values are outside the scope of
the integrity tag, and do not “break” the chain of trust for the hardened container.
Artifacts related to the hardened container should include the Information Assurance (IA)controls that the hardened container has successfully addressed, so that users of the container
know which controls they can inherit, versus which controls they must address. This capability
may not exist in the MVP, but it is an objective that enables reciprocity.
The Hardened Container Factory produces following types of containers:•

Hardened containers of DevSecOps CI/CD pipeline tools•

Sidecar Container Security Stack (see details in Section 6.4.4) containers to be used inruntime environments for container security•

Common containers (such as OS, database, web servers, etc.) to be used as a program
development baseline

5.1.2

Container Hardening Process

The Container hardening process, including required documentation, sustainment of thehardened containers, and cybersecurity requirements, is fully described in the DoD Enterprise
DevSecOps Initiative Hardening Containers [13]. The basic process is depicted in Figure 11.
Select the containerbase image

Harden the container

Store the hardened
container

Cybersecurity

ContinuousEngineering

Documentation

Figure 11: Major Steps in the Container Hardening Process
5.1.2.1

Select the Container Base Image

A base image is a container image that comes from a vendor or an open source community; it isused as the starting point to create a hardened container image. Use the base image without
creating forks to enable direct coupling with its updates. The container should be built startingwith the respective DoD hardened base OS STIG image.

58
UNCLASSIFIEDUNCLASSIFIED
5.1.2.2

Harden the Container

The Container Factory uses a set of instructions given in [13] to harden the container to mitigate
findings and ensure proper DoD compliance. Reuse the instructions as much as possible between
versions so that the hardening will be consistent across versions. It is possible that new versions
bring new features, which may require additional hardening.
The DoD Centralized Container Source Code Repository (DCCSCR) is used to store instruction
files to build container images, associated checksums, and various documentation. The sourcecode repository is centrally hosted so hardeners can store their code and leverage a CI/CD
pipeline (Container Factory). This is what feeds the container hardening process and DCAR
repository.
Use the CI/CD orchestration tool; download the DCCSCR folder content into the pipeline anduse the Instructions file to build the container.
The CI/CD pipeline will then run the required Container Hardening Scanners, scanning thecontainer image. Based on the findings of the Container Hardening Scanners, add instructions to
mitigate the findings as needed. Rebuild and rescan until findings are mitigated or accepted.
5.1.2.3

Store the Hardened Container

The DCAR is used to store the hardened containers, associated checksums, and various
documentation. This repository will be centrally hosted. Each container will have its own folder
in the DCAR. Subfolders should be used for versioning.
Store the hardened container and checksum inside the DCAR. It will be tagged as “preproduction” as well until the artifact receives an ATO, in which case it will then be tagged for“production”.
5.1.2.4

Documentation

Content and documentation provided for the hardened container will include:••

A description of the container, how to deploy it, the functional capabilities it provides,
and its interfaces.
Scripts, including the instructions file for building the container, and related
configuration files for deploying and scaling the hardened image or container•

Security test results including findings, false positives, and a recommended mitigation
plan.

•

Accepted risks, including a Plan of Action and Milestones (POA&M) for critical andhigh findings that are not yet resolved.

•

Change log of significant changes since the last version.

•

Human-readable licenses for all products are included within the container. COTS license
keys will not be included, and they will need to be acquired separately.
59
UNCLASSIFIEDUNCLASSIFIED
5.1.2.5

Continuous Engineering

The hardened container factory CI/CD pipeline searches for and downloads new base images
that are posted by the vendor or in an open source community repository and runs the steps in the
Section 5.1.2.2. These steps are triggered automatically, as soon as a new image is released into
the open source repository. If the build passes all scans, it should automatically store the newcontainer into DCAR, where it will be tagged as “pre-production”. It also automatically notifies
the team if a build fails to pass any of the scans.
5.1.2.6

Cybersecurity

The hardened containers produced by the factory should meet the related cybersecurity
requirements, which include NIST Special Publication (SP) 800-53 [14], NIST SP 800-37 [15],the DoD DISA Security Technical Implementation Guides (STIGs) and Security Requirements
Guides (SRGs), and industry best practices.

5.2 DoD Centralized Artifact Repository
The DCAR holds the DoD hardened container images that the DoD Enterprise DevSecOps
Container Factory produces. DoD program DevSecOps teams can utilize these to instantiate theirown DevSecOps ecosystem and software factory. The DCAR also holds the DoD hardened
containers for base operating systems, web servers, application servers, databases, API gateways,
message buses, and additional enterprise capabilities for use by DoD program software teams as
a program system deployment baseline. Separate hardened container images are created for
different versions of base images. DCAR hardened container images are version controlled.
These hardened containers, along with security accreditation reciprocity, greatly simplify andspeed the process of obtaining an Approval to Connect (ATC) or Authority to Operate (ATO).
The DCAR provides the capability to allow DoD programs (including approved DoD
contractors) to search, list information about, and download artifacts from the repository for DoD
software development on the approved environment.

60
UNCLASSIFIEDUNCLASSIFIED

6

DevSecOps Ecosystem Reference Designs

This section will discuss two software factory reference designs. One is based on the DoDEnterprise DevSecOps Container Service offering to create a software factory using DevSecOps
tool hardened containers from DCAR. The other is based on DoD authorized cloud DevSecOps
service offerings as provided by a CSP.
This section also discusses secure operations for containerized applications in a production
environment by leveraging container security tools in the DCAR.

6.1 Containerized Software Factory
A containerized software factory can be instantiated using a set of DevSecOps hardened
containers that are offered in the DCAR. These enterprise containers are preconfigured andsecured to reduce the certification and accreditation burden and are often available as apredetermined pattern or pipeline that will need limited or no configuration. Figure 12 illustrates
a containerized software factory reference design. The software factory is built on an underlying
container orchestration layer and a host environment. It produces DoD applications as the
product. These applications use different sets of hardened containers from the DCAR than theones used to create the software factory.
DoD programs may have already implemented a DevSecOps platform. One of the pain points issustaining that platform. It is highly recommended that, as incremental updates are made to theexisting platform, the program migrates capabilities to the DoD Enterprise DevSecOps hardened
containers. For those cases where a DoD Enterprise hardened container is not available, or
requires a custom policy, the program in conjunction with the DoD Enterprise DevSecOps
program office is encouraged to create, sustain, and deliver the hardened container to the DCAR.

61
UNCLASSIFIEDUNCLASSIFIED

Figure 12: Containerized Software Factory Reference Design
6.1.1

Hosting Environment

The reference design does not restrict the software factory hosting environment, which could be
DoD-approved Cloud Service Providers, DoD data centers or even on-premises servers. The
62
UNCLASSIFIEDUNCLASSIFIED
hosting environment provides compute, storage, and network resources in either physical or
virtual form.
6.1.2

Container Orchestration

In order to support containerized software factory tools, the underlying container orchestrationmust use CNCF certified Kubernetes and support OCI compliant containers. CNCF-certified
Kubernetes orchestrates containers, interacts with underlying hosting environment resources, andcoordinates clusters of nodes at scale in development, testing and pre-production in an efficientmanner. There are two options for the container orchestration layer as illustrated in Figure 13.

Figure 13: DevSecOps Platform Options
In Option A, it is the mission program’s responsibility to build and maintain the container
orchestration layer (CNCF-certified Kubernetes) using COTS solutions. The container
orchestration layer can be deployed on top of a DoD authorized cloud environment, a DoD data
center, or on bare metal servers. The container orchestration system components are subject tomonitoring and security control under the DoD policy in that hosting environment, such as the
DoD Cloud Computing Security Requirements Guide (SRG) [2] and DISA’s Secure Cloud
Computing Architecture (SCCA) [3] for the cloud environment.
In Option B, the mission program uses a CSP container service, which must have a DoD
provisional authorization and must be based on CNCF-certified Kubernetes.
6.1.3

Software Factory Using Hardened Containers

The software factory leverages technologies and tools to automate the CI/CD pipeline processes
defined in the DevSecOps lifecycle plan phase. There are no “one size fits all” or hard rules
63
UNCLASSIFIEDUNCLASSIFIED
about what CI/CD processes should look like and what tools must be used. Each software teamneeds to embrace the DevSecOps culture and define its processes that suit its software systemarchitectural choices. The tool chain selection is specific to the software programming language
choices, application type, tasks in each software lifecycle phase, and the system deployment
platform.
Software factory building itself follows the DevSecOps philosophy and goes through its own
design, instantiate, verify, operate and monitor phases. It evolves through the application
lifecycle iteration. Figure 14 illustrates the software factory phases, activities, and therelationship with the application lifecycle. Security must be applied across the software factory
phases. Sidecar Container Security Stack (SCSS) discussed in 6.4.4 can be used for software
factory runtime cybersecurity monitoring.

Figure 14: Software Factory Phases in the Application Lifecycle
Figure 12 is a software factory reference design. It includes the tools and process workflows todevelop, build, test, secure, release, and deliver software application for production deployment.
All the tools are based on the DoD enterprise DevSecOps hardened containers. Committing code
into the code repository kicks off the automated factory CI/CD pipeline workflow. The CI/CD
orchestrator executes the workflow by coordinating different tools to perform various tasks.
Some tasks are completed by a set of DevSecOps tools, such as build, static code analysis, unit
test, publish artifacts, build container image, etc. Other tasks may need assistance fromunderlying container orchestration layer, such as deploy application to test, pre-production, andfinal production environments. Some test and security tasks may need human involvement.
6.1.4

DoD Applications

The term “DoD Application” refers to a DoD software program hosted by an information system[16], which spreads widely from legacy monolithic infrastructure-dependent applications to64
UNCLASSIFIEDUNCLASSIFIED
modern modular infrastructure-agnostic applications. Most systems are in brownfield withlegacy applications or mixed legacy and modern applications. Programs should consider thenature of their application and the deployment environment when designing their software
factory. It is recommended to leverage the Strangler Pattern [17] [18] to refactor legacy
applications to modern microservices/containerized applications.
•

DoD application environments at all phases (development, test, pre-production,
production) are subject to security control from DoD common security services.

•

While refactoring legacy code or writing new code, it is highly recommended to leverage
the DoD hardened containers or hardening scripts to facilitate the application’s ATO.

6.2 Software Factory using Cloud DevSecOps Services
The DoD authorized cloud may already or will soon offer DevSecOps services, such as code
repository, artifact repository, build service, code deploy service, etc. Programs should consider
using these native managed services as alternatives to self-built and self-maintained DevSecOps
tool sets, but they should understand that using them may lead to vendor lock-in with the CSP.
The CSP is responsible for maintaining the service offering and the DoD Provisional
Authorization (PA) for each service. The program still needs to follow the software factory
lifecycle and performs the design, DevSecOps service selection instead of tool selection, CI/CD
pipeline process workflow automation, verify the workflows, operates and monitors theworkflows.
Another consideration is that the CSP may not offer a full solution set. For the capability that a
DevSecOps service with a DoD PA is not available, the corresponding DoD Enterprise hardened
container that provides the proper capability should be used. A CNCF-certified Kubernetes
container orchestration service is required for container runtime. Figure 15: Software Factory
illustrates a software factory using both cloud DevSecOps services and self-maintained security
tools.

65
UNCLASSIFIEDUNCLASSIFIED

Figure 15: Software Factory using Cloud DevSecOps Services

6.3 Serverless Support
So-called “serverless computing” is becoming more popular in the DoD, and it is being used
extensively in industry. This is a kind of Platform as a Service (PaaS) that is sometimes called
Function as a Service (FaaS). Despite the “serverless” moniker, a FaaS still needs servers, butdevelopers don’t have to worry about the servers, but rather how to deploy the code to them, how
to set up autoscaling, and other deployment tasks. This frees the developers to focus on the code.
Figure 16 illustrates that a FaaS can reduce development complexity and increase efficiency overan Infrastructure as a Service (IaaS), a Containers as a Service (CaaS), or some Platform as aService (PaaS) offerings. Although a Software as a Service (SaaS) offering is even more
efficient, and good to use when it meets requirements, a SaaS is typically focused on only a few
capabilities, and cannot provide the flexibility the DoD needs to develop custom applications. A
good FaaS, on the other hand, does provide that flexibility, although some applications will need
the even greater flexibility of an IaaS.

66
UNCLASSIFIEDUNCLASSIFIED

Figure 16: Operational Efficiency
The FaaS concept has made its way into the Kubernetes environment. The basic concept is tohand the FaaS some code, then the FaaS will build an image from the code and start it running on
Kubernetes.
The FaaS must have these features:
1) Build – builds containers from source codea) Uses container images as the deployment unitb) Given source code, build the code and create a container to house it2) Serving – runs the containers created by Build and automatically scales them up or down asnecessarya) Uses CNCF Kubernetes as the underlying container orchestration layerb) Auto-scale up (given that the code is written to allow this)c) Auto-scale down all the way to zerod) Gradual rollouts of new versionse) Network routing within the cluster, and ingress connections into the cluster
3) Eventing – allows functions/applications to publish and subscribe to event streams, to enable
loosely-coupled, event-driven systemsa) Universal subscription, delivery, and management of eventsb) Bind events to functions or containersc) Trigger functions when called via and Hypertext Transfer Protocol (HTTP) requestsd) Automatically scale from a few events per day to live streams
67
UNCLASSIFIEDUNCLASSIFIED
One popular open source product that implements FaaS for Kubernetes is Knative. Another open
source product is Kubeless. The DevSecOps Software Factories must offer Knative support.
They may also support Kubeless or another FaaS for Kubernetes.

6.4 Application Security Operations
This section focuses on the software application lifecycle in the production environment.
Continuous Deployment, Continuous Operation, and Continuous Monitoring are keys tostreamlined and secure operations.
6.4.1

Continuous Deployment

Continuous deployment is triggered by the successful delivery of released artifacts to the artifact
repository and may be subject to control with human intervention according to the nature of the
program application. The typical activities for continuous deployment include, but are notlimited to, deploying a new software release to the production environment, applying necessary
infrastructure and security configuration changes, running a smoke test to make sure essentialfunctionality is working, and performing security scans. Each activity is completed by specifictools or configuration/orchestration scripts. The selection of tools and theconfiguration/orchestration system depends on the application and the production platform. For
example, if the application is containerized and the production platform uses Kubernetes, theorchestration is done by Kubernetes. In this case, the configuration scripts would be Kubernetes
Operators or Helm charts. On the other hand, if the application is VM based, theconfiguration/orchestration tool could be Chef, Puppet or Ansible.
Continuous deployment interacts with other DevSecOps components, such as the artifact
repository for retrieving new releases, the log storage and retrieval service for logging
deployment events, and the issue tracking system for recording any deployment issues. The firsttime deployment may involve heavy infrastructure provisioning, dependency system
configuration (such as monitoring tools, logging tools, scanning tools, backup tools, etc.), andexternal system connectivity (such as DoD common security services, etc.).
6.4.2

Continuous Operation

Continuous operation is an extension of continuous deployment. It is triggered by a successful
deployment to the production environment, so that it operates the latest stable software release.
The activities of continuous operation include, but are not limited to, system patching,compliance scanning, data backup, system recovery if failure happens, and resource optimization
with load balancing and scaling. The selection of the tools that facilitate the activities areapplication and environment dependent. The resource optimization heavily depends on theunderlying platform. A containerized application can rely on Kubernetes to automatically scale
containers across cluster nodes. On the other hand, a VM-based application with no containers
can rely on the underlying CSP’s scaling service.
68
UNCLASSIFIEDUNCLASSIFIED
Continuous operation interacts with the logging system, issue tracking system, and theunderlying infrastructure platform.
6.4.3

Continuous Monitoring

Continuous monitoring is an extension to continuous operation. It continuously inventories all
system components, monitors the performance and security of all components, and logs
application and system events. Figure 20 in Section 6.4.4 illustrates the components formonitoring a containerized application deployed on Kubernetes. Figure 17 illustrates a simplified
sample process of monitoring, logging, and log analysis and alerting, which also applies todeployments to non-containerized environments.
The process starts with application logging, compute resource monitoring, storage monitoring,
network monitoring, security monitoring, and data monitoring at the Kubernetes pod level in thecase of containerized deployment or individual subsystem level in the case of VM deployment.
Each application will need to determine how it is divided into subsystems, the number ofsubsystems, and the specific monitoring mechanisms within the subsystems. The security tools
within each subsystem (e.g., the Sidecar Container Security Stack) will aggregate and forward
the event logs gathered from monitoring to a locally centralized aggregated logs database on the
mission program platform. This should be automated within the Kubernetes cluster. The
aggregated logs will be further forwarded to the Logs/Telemetry Analysis in the DoD Common
Security Services after passing the program application configured log filter. The program’s
local log analysis capability will analyze the aggregated logs and generate incident alerts andreports. Incidents will be forwarded to the mission program incident management system tofacilitate change request generation for incident resolution. The mission program incident
management should alert or notify the responsible personnel about the incidents. The change
request may be created to address the incident. These actions make the DevSecOps pipeline afull closed loop from secure operations back to planning.

69
UNCLASSIFIEDUNCLASSIFIED

Figure 17: Logging and Log Analysis Process
6.4.4

Sidecar Container Security Stack

A new service that is enabled by DevSecOps and the container-based Kubernetes runtime
environment is the Sidecar Container Security Stack (SCSS). This security stack enables:
correlated and centralized logs, container security, east/west traffic management, a zero-trustmodel, a whitelist, Role-Based Access Control (RBAC), continuous monitoring, signature-based
continuous scanning using Common Vulnerabilities and Exposures (CVEs), runtime behavior
analysis, and container policy enforcement.
One advantage of using the SCSS is that Kubernetes can inject the sidecar automatically, without
the team having to do anything, once it’s configured.
The sidecar pattern is depicted in Figure 18. A container group or pod is a set of containers thatare deployed together. A sidecar is a container running inside a pod alongside an application
container. If there is one application container and one sidecar container in the container group,
then we have the sidecar pattern as depicted in Figure 18.
70
UNCLASSIFIEDUNCLASSIFIED

Figure 18: Sidecar Pattern
The sidecar can share state with the application container. In particular, the two containers canshare disk and network resources while their running components are isolated from one another.
A sidecar is a general container pattern. The Sidecar Container Security Stack has a sidecar
container that contains a security stack, along with some supporting services that run in the
hosting environment, such as a logging service. The security stack in the security sidecar
container will include:
1. A logging agent to push logs to a platform centralized logging service.
2. Container policy enforcement. This includes ensuring container hardening from DCAR
containers are preserved and complies with the NIST 800-190 requirements [12].
3. Runtime Defense, this can perform both signature-based and behavior-based detection.
This can also be used to send notifications when there is anomalous behavior.
4. Vulnerability Management
5. A service mesh proxy to connect to the service mesh
6. Zero Trust down to the container level. Zero trust requires strict controls, never trust
anything by default and always verify. Key aspects of zero trust at the container level
include mutual Transport Layer Security authentication (mTLS), an encrypted
communication tunnel between containers, strong identities per Pod using certificates,
and whitelisting rather than blacklisting.
In addition to the components in the sidecar, there are a few services that support the security
sidecar. These include:
71
UNCLASSIFIEDUNCLASSIFIED
1. Program-specific Log Storage and Retrieval Service
2. Service Mesh
3. Program-specific artifact repository
4. Runtime Behavior Analysis Artificial Intelligence (AI) service
5. DCAR for the hardened containers
6. Common Vulnerabilities and Exposures (CVE)Service / host-based security to provide
CVEs for the security sidecar container
The interaction of these services with the sidecar components is depicted in Figure 19. The
arrows show the direction of the data flow. The items in purple are services provided by the
DoD, while blue indicates components that are provided by the DoD but instantiated andoperated by the program.

Figure 19: Sidecar Components
Table 23: Sidecar Container Security Stack Components
Tool

FeaturesBenefits

Logging agent

Send logs to a logging service

Logging Storageand Retrieval
Service

Stores logs and allows searching logs

Standardize log collection to acentral location. This can also be
used to send notifications when
there is anomalous behavior.
Place to store logs

72
UNCLASSIFIED

BaselineMVP

MVP

UNCLASSIFIED
Tool

FeaturesLog visualization
and analysis
Container policy
enforcement

Ability to visualize log data in various
ways and perform basic log analysis.
Support for Security Content
Automation Protocol (SCAP) andcontainer configuration policies. These
policies can be defined as needed.
Creates runtime behavior models,
including whitelist and least privilege
Ties to the Service Mesh. Used with amicroservices architecture.
Used for a microservices architecture

Helps to find anomalous patterns

Objective

Automated policy enforcement

MVP

Dynamic, adaptive cybersecurity

MVP

Enables use of the service mesh.

Objective

Better microservice management.

Objective

Vulnerability
Management

Provides vulnerability managementMVP

CVE Service / Host
Based Security

Provides CVEs. Used by thevulnerability management agent in the
security sidecar container.
Storage and retrieval for artifacts suchas containers.
Provides strong identities per Pod withcertificates, mTLS tunneling andwhitelisting of East-West traffic down
to the Pod level.

Makes sure everything is properly
patched to avoid known
vulnerabilities
Makes sure the system is aware of
known vulnerabilities incomponents.
One location to obtain hardened
artifacts such as containers
Reduces attack surface and improves
baked-in security

Runtime Defense
Service Meshproxy
Service Mesh

Artifact Repository
Zero Trust model
down to thecontainer level

Benefits

Baseline

MVP

MVP
MVP

Figure 20 depicts another view of the sidecar, along with some of the other DevSecOps
components. Again, the arrows show the direction of the data flow, and not all interactions
between the depicted components are indicated. The program dashboard displays information
about the application. The dashboard is built partly using visualizations from the log
visualization service. The items in purple and blue are either services or components that areprovide by the DoD. But those in blue will be stood up by the program. For example, the Service
Mesh is provided as a hardened container. Similarly, the Sidecar Container Security Stack isprovided as a hardened container that the program installs in each pod (container group); this is
injected by Kubernetes automatically without application developer involvement.

73
UNCLASSIFIEDUNCLASSIFIED

Figure 20: Sidecar Container Security Stack Interactions

74
UNCLASSIFIEDUNCLASSIFIED

7

Conclusion

We have introduced key DevSecOps concepts, described the DevSecOps Ecosystem, including
the Software Factory and the Sidecar Container Security Stack, and indicated how the ecosystemshould be set up and used. More detail on the components described here, such as DCAR, can befound in other DoD CIO documents.
Moving to DevSecOps improves agility and speeds new capabilities into the field. But it also
requires new policies, processes and culture change. More information on DevSecOps culture,
metrics, and the maturity model can be found in the DoD DevSecOps Playbook [7].

75
UNCLASSIFIEDUNCLASSIFIED

Appendix A Acronym Table
AcronymDefinition

A&A

Assessment and Authorization

A&S

Acquisition and Sustainment

AI

Artificial Intelligence

AO

Authorizing Official

API

Application Programming Interface

AQ

Acquisition

ASW

Attack Sensing and Warning

ATC

Approval to Connect (ATC)ATO

Authority to Operate (ATO)

AVM

Assurance Vulnerability Management

BOM

Bill of Materials

CaaS

Containers as a Service

CCB

Change Control Board

CD

Continuous Delivery

CI

Continuous Integration

CIO

Chief Information Officer

CM

Configuration Management

CMDB

Configuration Management Data Base

CNCF

Cloud Native Computing Foundation

CNSS

Committee on National Security Systems

CNSSI

Committee on National Security Systems Instruction

COTS

Commercial Off The Shelf

CPCON

Cyber Protection Condition

CPU

Central Processing Unit

CSP

Cloud Service Provider

CSSP

Cybersecurity Service Provider

CVE

Common Vulnerabilities and Exposures

DAST

Dynamic Application Security TestDCAR

DoD Centralized Artifact Repository

DCCSCRDoD Centralized Container Source Code Repository

DCIO

Deputy Chief Information Officer

DBaaS

Database as a Service

DDOS

Distributed Denial of Service

DevSecOps

Development, Security, and Operations

DISA

Defense Information Systems Agency

76
UNCLASSIFIEDUNCLASSIFIED
Acronym
DNS

Definition
Domain Name Service

DoD

Department of Defense

DoDI

DoD Instruction

DTRA

Defense Threat Reduction Agency

EO

Executive Order

FaaS

Function as a Service

FMA

Forensic Media Analysis

GB

Gigabyte

GEP

Global Enterprise Partners

GOTS

Government Off The Shelf

HTTP

Hypertext Transfer Protocol

IA

Information Assurance

IaaS

Infrastructure as a Service

IaC

Infrastructure as Code

IASTInteractive Application Security Test

ICAM

Identity, Credential, and Access Management

IDE

Integrated Development Environment

IDS

Intrusion Detection System

IE

Information Enterprise

IHR

Incident Handling Response

INFOCON

Information Operations Condition

IO

Input/Output

IPS

Intrusion Prevention System

IR

Incident Reporting

ISCM

Information Security Continuous Monitoring

IT

Information Technology

JVM

Java Virtual Machine

KPI

Key Performance IndicatorMB

Megabyte

MilDepMilitary Department

MNP

Malware Notification Protection

mTLS

mutual Transport Layer Security authentication

MVP

Minimum Viable Product

NGG

Next Generation Governance

NIST

National Institute of Standards and Technology

NoSQL

Non SQL

NSM

Network Security Monitoring

77
UNCLASSIFIEDUNCLASSIFIED
AcronymDefinition

OCI

Open Container InitiativeOS

Operating System

PA

Provisional Authorization

PaaS

Platform as a Service

PEO

Program executive Officer

POA&M

Plan of Action and Milestones

QA

Quality Assurance

RBAC

Role-Based Access Control

RMF

Risk Management Framework

ROI

Return on Investment

SaaS

Software as a Service

SaC

Security as Code

SAF

Secretary of the Air Force

SAST

Static Application Security Test

SCAP

Security Content Automation Protocol

SCCA

Secure Cloud Computing Architecture

SCSS

Sidecar Container Security Stack

SLA

Service Level Agreement

SRG

Security Requirements Guide

SQL

Structured Query Language

SSH

Secure Shell

STIG

Security Technical Implementation Guide

TRB

Technical Review Board

VM

Virtual Machine

78
UNCLASSIFIEDUNCLASSIFIED

Appendix B Glossary of Key Terms
Following are the key terms used in describing the reference design in this document.
Term

Definition

Artifact

An artifact is a consumable piece of software produced during thesoftware development process. Except for interpreted languages,
the artifact is or contains compiled software. Important examples
of artifacts include container images, virtual machine images,
binary executables, jar files, test scripts, test results, security scan
results, configuration scripts, Infrastructure as a Code,documentation, etc. Artifacts are usually accompanied bymetadata, such as an id, version, name, license, dependencies,
build date and time, etc.

Software Artifact

Note that items such as source code, test scripts, configuration
scripts, build scripts, and Infrastructure as Code are checked into
the source code repository, not the artifact repository, and are notconsidered artifacts.
Artifact Repository

An artifact repository is a system for storage, retrieval, andmanagement of artifacts and their associated metadata.
Note that programs may have separate artifact repositories to store
local artifacts and released artifacts. It is also possible to have asingle artifact repository and use tags to distinguish the content
types.

Bare Metal
Bare Metal Server

BinaryBinary File

Build
Software Build

A bare metal or bare metal server refers to a traditional physicalcomputer server that is dedicated to a single tenant and which does
not run a hypervisor. This term is used to distinguish physicalcompute resources from modern forms of virtualization and cloud
hosting.
Binary refers to a data file or computer executable file that isstored in binary format (as opposed to text), which is computerreadable, but not human-readable. Examples include images,
audio/video files, exe files, and jar/war/ear files.
The process of creating a set of executable code that is producedby compiling source code and linking binary code.
79
UNCLASSIFIEDUNCLASSIFIED
Build tools
Software Build Tools
CI/CD Orchestrator

Used to retrieve software source code, build software, and generate
artifacts.
CI/CD orchestrator is a tool that enables fully or semi-automatedshort duration software development cycles through integration ofbuild, test, secure, store artifacts tools.
CI/CD orchestrator is the central automation engine of the CI/CD
pipeline

CI/CD Pipeline

CI/CD pipeline is the set of tools and the associated process
workflows to achieve continuous integration and continuous
delivery with build, test, security, and release delivery activities,
which are steered by a CI/CD orchestrator and automated as muchas practice allows.

CI/CD Pipeline
Instance

CI/CD pipeline instance is a single process workflow and the tools
to execute the workflow for a specific software language andapplication type for a project. As much of the pipeline process is
automated as is practicable.

Cloud Native
Computing Foundation(CNCF)“CNCF is an open source software foundation dedicated to making
cloud native computing universal and sustainable. Cloud native
computing uses an open source software stack to deploy
applications as microservices, packaging each part into its own
container, and dynamically orchestrating those containers to
optimize resource utilization. Cloud native technologies enablesoftware developers to build great products faster.” https://www.cncf.io/

CNCF-certified
Kubernetes

CNCF has created a Certified Kubernetes Conformance Program.
Software conformance ensures that every vendor’s version of
Kubernetes supports the required APIs. Conformance guarantees
interoperability between Kubernetes from different vendors. Mostof the world’s leading vendors and cloud computing providers
have CNCF certified Kubernetes offerings.

Code

Software instructions for a computer, written in a programming
language. These instructions may be in the form of either humanreadable source code, or machine code, which is source code thathas been compiled into machine executable instructions.
80
UNCLASSIFIEDUNCLASSIFIED
Configuration
Management

Capability to establish and maintain a specific configuration within
operating systems and applications.

Container

A standard unit of software that packages up code and all its
dependencies, down to, but not including the OS. It is alightweight, standalone, executable package of software that
includes everything needed to run an application except the OS:
code, runtime, system tools, system libraries and settings.

Continuous Build

Continuous build is an automated process to compile and build
software source code into artifacts. The common activities in thecontinuous build process include compiling code, running static
code analysis such as code style checking, binary linking (in thecase of languages such as C++), and executing unit tests. The
outputs from continuous build process are build results, build
reports (e.g., the unit test report, and a static code analysis report),
and artifacts stored into Artifact Repository. The trigger to thisprocess could be a developer code commit or a code merge of abranch into the main trunk.

Continuous Delivery

Continuous delivery is an extension of continuous integration toensure that a team can release the software changes to production
quickly and in a sustainable way.
The additional activities involved in continuous integration include
release control gate validation and storing the artifacts in theartifact repository, which may be different than the build artifact
repository.
The trigger to these additional activities is successful integration,
which means all automation tests and security scans have been
passed.
The human input from the manual test and security activities
should be included in the release control gate.
The outputs of continuous delivery are a release go/no-go decisionand released artifacts, if the decision is to release.

Continuous
Deployment

Continuous deployment is an extension of continuous delivery. It
is triggered by a successful delivery of released artifacts to theartifact repository.
81
UNCLASSIFIEDUNCLASSIFIED
The additional activities for continuous deployment include, but
are not limited to, deploying a new release to the production
environment, running a smoke test to make sure essentialfunctionality is working, and a security scan.
The output of continuous deployment includes the deployment
status. In the case of a successful deployment, it also provides a
new software release running in production. On the other hand, afailed deployment causes a rollback to the previous release.
Continuous Integration

Continuous integration goes one step further than continuous
build. It extends continuous build with more automated tests andsecurity scans. Any test or security activities that require humanintervention can be managed by separate process flows.
The automated tests include, but are not limited to, integration
tests, a system test, and regression tests. The security scansinclude, but are not limited to, dynamic code analysis, test
coverage, dependency/BOM checking, and compliance checking.
The outputs from continuous integration include the continuous
build outputs, plus automation test results and security scan results.
The trigger to the automated tests and security scan is a successfulbuild.

Continuous monitoring

Continuous monitoring is an extension to continuous operation. It
continuously monitors and inventories all system components,
monitors the performance and security of all the components, andaudits & logs the system events.

Continuous Operation

Continuous operation is an extension to continuous deployment. It
is triggered by a successful deployment. The production
environment operates continuously with the latest stable software
release.
The activities of continuous operation include, but are not limited
to: system patching, compliance scanning, data backup, andresource optimization with load balancing and scaling (bothhorizontal and vertical).

Cybersecurity,
Software Cybersecurity

The preventative methods used to protect software from threats,
weaknesses and vulnerabilities.
82
UNCLASSIFIEDUNCLASSIFIEDDoD Centralized
Artifact Repository(DCAR)

Holds the hardened container images of DevSecOps componentsthat DoD mission software teams can utilize to instantiate theirown DevSecOps pipeline. It also holds the hardened containers forbase operating systems, web servers, application servers,
databases, API gateways, message busses for use by DoD mission
software teams as a mission system deployment baseline. These
hardened containers, along with security accreditation reciprocity,
greatly simplifies and speeds the process of obtaining an Approval
to Connect (ATC) or Authority to Operate (ATO).

Delivery

The process by which a released software is placed into a artifact
repository that operational environment can download.

Deployment

The process by which the released software is downloaded anddeployed to the production environment.

DevSecOps

DevSecOps is a software engineering culture and practice thataims at unifying software development (Dev), security (Sec) andoperations (Ops). The main characteristic of DevSecOps is toautomate, monitor, and apply security at all phases of softwaredevelopment: plan, develop, build, test, release, deliver, deploy,
operate, and monitor.

DevSecOps Ecosystem

A collection of tools and process workflows created and executedon the tools to support all the activities throughout the full
DevSecOps lifecycle.
The process workflows may be fully automated, semi-automated,or manual.

DevSecOps Pipeline

DevSecOps pipeline is a collection of DevSecOps tools, uponwhich the DevSecOps process workflows can be created andexecuted.

DevSecOps phase

The software development, security, and operation activities in thesoftware lifecycle are divided into phases. Each phase completes apart of related activities using tools.

Environment

Sets a runtime boundary for the software component to be
deployed and executed. Typical environments include
development, integration, test, pre-production, and production.
83
UNCLASSIFIEDUNCLASSIFIED
Factory,
Software Factory

Software Factory
Artifact Repository

Hypervisor

A software assembly plant that contains multiple pipelines, which
are equipped with a set of tools, process workflows, scripts, andenvironments, to produce a set of software deployable artifacts
with minimal human intervention. It automates the activities in thedevelop, build, test, release, and deliver phases. The software
factory supports multi-tenancy.
Stores artifacts pulled from DCAR as well as locally developed
artifacts to be used in DevSecOps processes. The artifacts include,
but are not limited to, VM images, container images, binary
executables, archives, and documentation. It supports multitenancy.
Note that program could have separate artifact repositories to store
local artifacts and released artifacts. It is also possible to have asingle artifact repository and use tags to distinguish the contents.
A hypervisor is a kind of low-level software that creates and runs
virtual machines (VMs). Each VM has its own Operating System(OS). Several VMs can run on one physical machine.

Figure 21: Hypervisor with Virtual Machines
Image Management,
Software ImageManagement,

The process of centralizing, organizing, distributing, and trackingof software artifacts.

Binary ImageManagement,
Container ImageManagement,
VM Image
Management
Immutable
infrastructure

An infrastructure paradigm in which servers are never modified
after they're deployed. If something needs to be updated, fixed, or
84
UNCLASSIFIEDUNCLASSIFIED
modified in any way, new servers built from a common image
with the appropriate changes are provisioned to replace the oldones. After they're validated, they're put into use and the old ones
are decommissioned.
The benefits of an immutable infrastructure include moreconsistency and reliability in your infrastructure and a simpler,
more predictable deployment process. (from:https://www.digitalocean.com/community/tutorials/what-isimmutable-infrastructure)
Infrastructure as CodeThe management of infrastructure (networks, virtual machines,
load balancers, and connection topology) in a descriptive model,using the same versioning that the DevSecOps team uses forsource code. Infrastructure as Code evolved to solve the problem
of environment drift in the release pipeline.

Kubernetes

An open-source system for automating deployment, scaling, andmanagement of containerized applications. It was originallydesigned by Google and is now maintained by the CNCF. Many
vendors also provide their own branded Kubernetes. It works witha range of container runtimes. Many cloud services offer a
Kubernetes-based platform as a service.

Lockdown

The closing or removal of weaknesses and vulnerabilities fromsoftware.

Microservices

Microservices are both an architecture and an approach to softwaredevelopment in which a monolith application is broken down intoa suite of loosely coupled independent services that can be altered,
updated, or taken down without affecting the rest of theapplication.

Mission Application
Platform

The mission application platform is the underlying hosting
environment resources and capabilities, plus any mission program
enhanced capabilities that form the base upon which the mission
software application operates.

Monitoring

The regular observation, recording, and presentation of activities.

Security Monitoring

85
UNCLASSIFIEDUNCLASSIFIED
NodeCluster node

A node is a worker machine in CNCF Kubernetes. A node may be
a VM or physical machine, depending on the cluster. Each nodecontains the services necessary to run pods and is managed by themaster components, including the node controller.

OCI“An open governance structure for the express purpose of creating
open industry standards around container formats and runtime” https://www.opencontainers.org/

OCI Compliant
Container

The image of the OCI compliant container must conform with the
OCI Image Specification.

OCI Compliant
Container Runtime

A container runtime is software that executes containers andmanages container images on a node. OCI compliant container
runtime must conform with the OCI Runtime Specification.

Orchestration

Automated configuration, coordination, and management.

Platform

A platform is a group of resources and capabilities that form a baseupon which other capabilities or services are built and operated.

Pod

A group of containers that run on the same CNCF Kubernetes
worker node and share libraries and OS resources.

Provisioning

Instantiation, configuration, and management of software or theenvironments that host or contain software.

ReportingAn account or statement describing an event.

Repository

A central place in which data is aggregated and maintained in an
organized way.

Resource

CPU, Memory, Disk, Networking

Scanning,

The evaluation of software for cybersecurity weaknesses andvulnerabilities.

Security Scanning
Sidecar Container
Security Stack

A sidecar container security stack is a stack of sidecar containers
aimed to enhance the security capabilities of the main containers inthe same Pod.

86
UNCLASSIFIEDUNCLASSIFIED
SidecarA sidecar is a container used to extend or enhance the functionalityof an application container without strong coupling between two.
When using CNCF Kubernetes, a pod is composed of one or morecontainers. A sidecar is a utility container in the pod and its
purpose is to support the main application container or containers
inside the same pod. For more information, see Section 6.4.4 andKubernetes documentation.

Telemetry

Capability to take measurements and collect and distribute thedata.

Test Coverage,

Test coverage is a measure used to describe what percentage ofapplication code is exercised when a test suite runs. A higher
percentage indicates more source code executed during testing,which suggests a lower chance of containing undetected bugs.

Code Coverage

Virtual Machine (VM)

Emulates a physical computer in software. Several VMs can run
on the same physical device.

Virtual Network

Networks constructed of software-defined devices.

Virtual Storage

Storage constructed of software-defined devices.

87
UNCLASSIFIEDUNCLASSIFIED

Appendix C References
The following references are utilized in the development of the DevSecOps reference design
document:[1] Department of Defense, "DoD Cloud Computing Strategy," December 2018.
[2] DISA, "Department Of Defense Cloud Computing Security Requirements Guide, V1R2,"
18 March, 2016.
[3] DISA, "DoD Secure Cloud Computing Architecture (SCCA) Functional Requirements,"
January 31, 2017.
[4] White House, "Presidential Executive Order on Strengthening the Cybersecurity of Federal
Networks and Critical Infrastructure (EO 1380)," May 11, 2017.
[5] National Institute of Standards and Technology, Framework for Improving Critical
Infrastructure Cybersecurity, 2018.
[6] DISA, "Department Of Defense Container Hardening Security Requirements Guide(Draft)," 2019.
[7] Office of the DoD CIO, "DoD DevSecOps Playbook," 2019. [Online]. Available:
https://www.milsuite.mil/book/groups/dod-enterprise-devsecops.
[8] Puppet Labs, "Puppet State of DevOps Report 2018," Puppet Labs, 2018.
[9] Defense Threat Reduction Agency (DTRA), "Next-Generation Technology Governance,"
2018.
[10] DoD, "DoDI 8510.01: Risk Management Framework for DoD Information Technology,"
24 May 2016.
[11] E. Ries, "The Lean Startup," [Online]. Available: http://theleanstartup.com/principles.
[Accessed 15 July 2019].
[12] NIST, "NIST Special Publication 800-190, Application Container Security Guide,"
September 2017.

88
UNCLASSIFIEDUNCLASSIFIED[13] N. M. Chaillan, "DoD Enterprise DevSecOps Initiative Hardening Containers," DRAFT,
2019.
[14] NIST, "Security and Privacy Controls for Federal Information Systems and Organizations,"
NIST SP 800-53 Revision 4, 2013.
[15] NIST, "Risk Management Framework for Information Systems and Organizations," NIST
SP 800-37 Revision 2, 2018.
[16] Committee on National Security Systems (CNSS) , "Committee on National Security
Systems Instruction (CNSSI) 4009: Committee on National Security Systems (CNSS)
Glossary," 2015.
[17] M. Fowler, "Strangler Fig Application," 29 June 2004. [Online]. Available:
https://martinfowler.com/bliki/StranglerFigApplication.html. [Accessed 12 July 2019].
[18] P. Hammant, "Legacy applicaion Strangulation: Case Studies," 14 July 2013. [Online].
Available: https://paulhammant.com/2013/07/14/legacy-application-strangulation-casestudies/. [Accessed 12 July 2019].

89
UNCLASSIFIED