DevSecOps Series: Shifting Security Left
By: Lucas Kauffman
Tags: Ci Cd Pipeline, Devsecops, Security, DevOps, Appsec

Mandatory Disclaimer: this post is by no means endorsed by Amazon or AWS. It is a pure expression of my personal experiences and opinion.

I’ve been looking for a while to create a series on DevSecOps. In my professional life I have helped organisations such as banks and insurance companies plan and implement their security processes within DevOps. In this series I want to share my experiences and provide insights on what the typical approaches are for DevSecOps.

I will describe the different components of running a DevSecOps program as well as what some of the (hard) lessons I learned through the years.

This introduction aims to provide the overall view of the why, what and how of implementing DevSecOps with subsequent stories diving deeper into areas such as people management, process design, pipeline management, branching strategies, automated security testing, container security and others.

Similar to DevOps or any other IT strategy, there is no such thing as a single size fits all or silver bullet to tackle it, and therefore, it’s important to understand your own organization, its business goals, its challenges and the current point of their DevOps/DevSecOps journey for you to move forward.

Shifting Security Left aims to do more security upfront in the Software Development Life-cycle and takes a pragmatic approach to getting more assurance in a more friction-less and automated way.

A word on risk management

In this article you will notice I highlight various risk levels along the way, these are what I found to be working for organisations who start their DevSecOps journey. The more mature you get, the lesser the risk tolerance should be for each of the highlighted gates as this ensures issues and risks are addressed earlier in the process.

Every organisation should define what their risk appetite and tolerance is to be implemented in the gating process. Most organisations do have present and unknown risks. Choose a pragmatic approach and a set of achievable goals to remediate it.

What is Agile, DevOps, CI/CD?

Before we can explain DevSecOps we first will discuss what lead us to Agile and DevOps in the first place.

A Very Brief History of Development Methodologies

The original way of doing development in modern times has generally been Waterfall where you have a non-iterative and structured approach that goes through a set of defined phases as shown below:

Image from medium

The main issue with Waterfall is that the time between defining a feature that a system should have to the time of delivery is long (months to years). A lot of modern enterprises require to respond to market changes at a very rapid pace. The waterfall model simply became too slow and lacked agility to keep up with the changing market demands.

The lack of agility resulted in … well you guessed it: Agile. Agile isn’t a single methodology but rather a set of principles which are implemented through other methodologies such as SCRUM or Kanban.

In the old world of Waterfall there was a clear split between development (Dev) and operations (Ops) and each had their own priorities. One tries to push out as much code as possible where the other tries to maintain stability. Dev and Ops started to clash, as with the Agile methodologies features and new code would be created quite rapidly, but the push to production would be lacking as Ops would prefer to move slower in order to maintain stability.

Traditionally both are part of different organisations with opposing KPIs and goals. This clashing culture is called the “Wall of Confusion”. So in order to fix this the concept of DevOps emerged which is a culture where Dev and Ops come together and work as a single team with the same goals, responsibilities and KPIs. The idea behind it is that this single team will work better together, produce better code and create more stable platforms as there is a shared pain across all stakeholders if things go wrong.

The DevOps culture helps operationalising the Agile methodology from inception to deployment and running of code.

A note on People and Processes within DevOps

DevOps is first and foremost a shift in culture. Secondary to the shift in culture there is a need to change processes to enable Agile delivery. Traditional testing (functional, regression, load or security testing) where you take weeks, now may be restricted to days or even a few hours. Where you needed to sign-off for code to be sent to production in a steering committee may now be an automated form instead. This is logical as with Waterfall you were deploying once every few months, where a day or two delay is not a big deal — but with Agile you may be deploying several 10’s of times a day.

Most organizations I’ve worked with would already consider it a success story if they managed to deploy once a week or once a month. From the business perspective this is crucial in order to push new functions or react to customer feedback or competitors with new products or offerings in a very short time frame.

Security needs to take note of this because we tend to have bolted on traditional controls that may align well with the Waterfall model but not with the Agile development processes and therefore will clash with the business objectives. This doesn’t mean we should throw out our controls, rather be smarter on how we apply them to get greater assurance.

DevOps Tools: CI/CD

In order to support the processes one of the key DevOps set of tools is the CI/CD pipeline. CI/CD stands for continuous integration and continuous deployment. CI/CD bridges the gaps between development and operation activities and teams by enforcing automation in building, testing and deployment of applications.

Continuous Integration (CI) aims to integrate and merge changes as often as possible. The developer’s changes are validated by creating a build and running automated tests against the build. By doing so, you avoid integration challenges that can happen when waiting for release day to merge changes into the release branch.

Continious Delivery/Deployment extends CI since it automatically deploys all changes to a testing and/or production environment after the build stage.

This means that on top of automated testing, you have an automated release process and you can deploy your application any time by clicking a button or after your automated tests have succeeded.

What is DevSecOps?

DevSecOps is doing security as part of the DevOps process. While the original definition of DevOps included security, DevSecOps aims to make it more explicit and provide more guidance on how to make security easy for development teams.

DevSecOps to me focusses on a few things:

People — Build a security culture

— Build a security culture Process — Shift Security Left

— Shift Security Left Tools — Automate Security Testing and Assurance Processes

— Automate Security Testing and Assurance Processes Governance — Enable, Standardise and promote re-use

People — Building a security culture

Building a security culture is one of the make-it-or-break-it components of DevSecOps.

The two issues I generally faced were:

Lack of skills or knowledge on security within the development teams

KPIs dis-incentivizing security

Security Champions program to build skills

When working with development teams I realized that a lot of security issues are introduced due to a lack of understanding of security. Security is a niche area and the reality is that it’s hard for development teams to understand some of the concepts. After building numerous secure coding guidelines and doing OWASP top 10 presentations I realized that neither was very effective.

In order to up-skill your development you need to create a hands-on training program as well as a champions program with tangible incentives to do more security. The best training, in my own personal view, are hands-on training programs that gamify the learning experience. One of the examples (and I am by no means affiliated) is Secure Code Warrior. It’s one of the few tools that seemed to resonate really well with the developers because it was developer centric. From here you can move to other trainings depending on the role of the developer or architect (e.g. CSSLP, AWS Security Specialty, etc).

Scaling through champions

The biggest issue we have is scalability. We simply can’t find enough security people to put a security specialist in every team. To fix this you need a tiered support structure so you can spread the knowledge of a few people across a lot of teams.

This is what a champions program tries to address. It provides training, badges (levels based on skills acquired and demonstrated), incentives and a network to more security knowledgeable people. The idea is that easy questions can be resolved at the lower tiers, which are covered by developers that took security training (and became a Security Champion) and if more difficult questions or problems come up that can’t be addressed by the lower tier, it can be asked to the next tier which can consist of more elite champions or security specialists. It’s similar to how Spotify has Guilds.

I will cover doing up a security champions program in more detail in a separate story in the future.

Getting KPIs right

KPI issues stem from developers being measured on how fast they can push out features and code by a certain time. Therefore they are incentivized to de-prioritise security features or issues. This is because generally security covers a non-functional requirements which are not visible. In order to fix this there are a few KPIs you can consider implementing:

Metrics on product owners and team collaboration

KPIs on teams rather than individuals

Metrics to drive quality and security (security being a metric of quality)-

Coverage of automated (security) testing

Number of penetration test results showing preventable issues with automated testing

# of trained security champions per development team

Hours of dedicated time for security activities within a scrum team (ideally this is 8 hours per 120 hours of development in a team)

KPIs should be holistic across departments so that you can have both the bottom-up as well as top-down incentive to get this right and promote more collaboration.

Change Management

A culture shift also means change management. When doing any large program it’s important to listen to your champions and stakeholders throughout the roll out and management of the program. Your champions can be very good indicators on adoption and issues faced on the ground. Use this feedback and address it where required. Without proper change management it’s unlikely the program will be very successful. Note that culture change is an on-going activity and not a one-off. Therefore you need to have someone to continuously manage the human component of the program.

Process and Tools — Shift Security Left and Automate Security Testing

Left shifting testing means you do more testing earlier in development lifecycle as to avoid finding bugs during later stages. The main reason for this is because the later you find bugs, the more expensive they become.

Similarly for doing security we need to shift our security activities left and do more upfront. This doesn’t just cover security testing but also security requirements gathering and procurement processes. One of the documents that describes this more in-depth is for instance the CSA Secure by Design document.

From a testing perspective the DevSecOps process should take create a gating process that consists of :

Soft Gates : gates where you highlight and flag out the defects or short comings but do not prevent the development team moving forward, it just will add to your delivery risk and potential delays if not addressed.

: gates where you highlight and flag out the defects or short comings but do not prevent the development team moving forward, it just will add to your delivery risk and potential delays if not addressed. Hard Gates: gates where the defects and short comings prevent you from moving forward and will block you for instance from going to production.

The strategy I generally adopt is to define the soft gates at the left side and subsequently hard gates at the right side (e.g. production or go-live). Each gate has it’s own exit criteria and acceptable risks levels. The goal is to balance risk versus development flexibility.

When designing the process you want to make sure the testing can be automated as much as possible because it will also make it easier for you to push down governance and make data driven decisions based on the output of the tools.

Security Processes within DevOps: CI/CD

The typical DevOps Process and CI/CD pipeline looks like this:

Within each of the blue boxes there are different activities and sub processes — for each step I will highlight what the typical security activities are.

Requirements Gathering (Req)

During the requirements gathering phase for the normal DevOps process the objective is to get the requirements from the business on what problem they want to solve or what product they want to build. From a security perspective it is important to be involved as it will allow you to:

Understand the initial application purpose

Understand the type of information or data the application will be using

Do an initial high level risk assessment and elicit what the original set of security requirements might be

Review what security skills are needed for the project (e.g. cloud security skills, secure coding skills, data security specialists like GDPR) and ensure your dev team covers those skills or is trained before work starts

Req Gating

Soft Gate: Ensure the above activities are done and the initial risk profile is created as well as ensuring we have the right skills present in the team

There are no special tools or technologies I’ve used in this phase, any GRC tool would work or even something as basic as Excel.

Plan

This is the part of the project where you organize the tasks, schedules and set up your tools. The idea is to plan tasks using the user story process from the agile methodology. Writing tickets in the form of a user story will allow developers and Ops to understand what development needs to be done and why.

From a security perspective we normally cover:

Perform Threat Modelling

Translate Threat models into Abuser/Misuser stories which are part of each relevant user story. Also ensure you capture non-functional requirements and require passing security testing to be part of the definition of done

Ensure you define the boundaries, security checks or requirements explicitly and ideally map them back to for instance OWASP ASVS guidelines

Ensure you use security patterns (or create re-usable codified security patterns) where possible

Once you have this defined it also becomes easier to understand what types of tests can be relevant or what functional security tests to write manually if they can’t be covered by an automated testing tool.

From a tooling perspective you generally use the typical tools developers are already using and integrate with their instances. Some examples might be JIRA, Trello, Asana, Confluence or even SharePoint where you can input your abuser story information, threat models or other information. Do not try to manage this seperately in an external tool as it will increase friction for developers to adopt.

Threat modelling can be done in different ways, either through a white boarding sessions, Microsoft Visio, Threat Modeler or other diagramming tools. Some commercial tools like Threat Modeler work semi automated but also come with a cost. Threat modelling is not a very scalable activity, so when you decide to do threat modeling, you should focus on your high value assets.

Plan Gating

Soft Gate: Ensure that all user stories that go into the backlog have abuser stories tagged to them. Ensure there is a threat model present and that there is continuous iteration throughout the DevOps lifecycle to update the threat model.

Code

Within the code phase your developers would start coding and code should also be reviewed (automatically and manually) both from a code quality and security perspective. You want to ensure developers can commit code to repositories that are subsequently tested and reviewed before moving into a building stage.

Code (Repo management: GitFlow)

The most common approach to secure a code repository and instill discipline is to use a GitFlow pattern where you have various feature branches where your developers commit their code and a master branch which represents the code that is deployed in production. The access to the master branch is secured through the concept of pull requests.

It’s important to set the right access controls onto the code repository and who can create pull and approve pull requests. In general you want senior developers and security engineers or champions to review and approve the code before it is pushed out, both for quality and security purposes.

Alongside you would also want tooling to be able to reject the code if it does not meet the security baselines (e.g. presence of vulnerabilities of a certain risk).

Code (Automated Testing Tools)

From a tooling perspective you want to do a few things:

IDE Plugins: these are plugins your developers can use to provide hints while they are coding. Commercial tools like Fortify and Snyk offer tools for developers to immediately run security from their IDE. This is useful as developers can check their own code from their own machine immediately.

these are plugins your developers can use to provide hints while they are coding. Commercial tools like Fortify and Snyk offer tools for developers to immediately run security from their IDE. This is useful as developers can check their own code from their own machine immediately. Pre-commit hooks: used before code is committed and can be used to scan for secrets, passwords or other strings that should not be present in your code. If you are using infrastructure as code you can also use Linters to validate your configurations (e.g. Terraform)

used before code is committed and can be used to scan for secrets, passwords or other strings that should not be present in your code. If you are using infrastructure as code you can also use Linters to validate your configurations (e.g. Terraform) Pull Request to dev: once developers create a pull request code into a development branch you can automatically trigger both SAST and SCA scans to ensure code does not contain vulnerabilities in your code (SAST) or code in third party dependencies (SCA). Note that different SAST/SCA tools work differently and sometimes they require a built package rather than source code. In this case you would only be able to run these scans after the build step. My personal preference is to run it on the source code instead of built packages as this happens earlier and you don’t waste time with building if vulnerabilities are present.

Based on the risk level of security issues you can either use an automated or manual review step before allowing the pull request to merge the code.

Code Gating

Hard Gate for Dev branch:

Positive presence of secrets in the pre-commit security hook

Vulnerable Third Party Dependencies with a risk level of High or Critical within your dependencies

SAST issues with a risk level of High or Critical

Soft Gate for Dev branch:

Vulnerable Third Party Dependencies with a risk level of Low or higher

SAST issues with a risk level of Low or higher

Hard Gate for Master branch:

Positive presence of secrets in the pre-commit security hook

Vulnerable Third Party Dependencies with a risk level of Low or higher

SAST issues with a risk level of Medium or higher

We want to give some flexibility to the development teams to continue working on the project even if certain vulnerabilities are present, but also report on it for visibility. This is to avoid getting into the situation where security issues go unnoticed until closer to the release checkpoint.

Of course mitigating controls (e.g. the development environment is not hosted on the Internet) should be present to prevent compromise of your development environment.

Build

During the build phase you will create artifacts. An artifact is a by-product produced during the software development process. It may consist of the project source code, dependencies, binaries or resources as well as output of scans and test results.

Wherever you store your artifacts, they require to be stored with high integrity (meaning access should be limited to read-only). My preference is to have the artifacts to be immutable and signed. Meaning if a build runs twice, you should generally see a new set of artifacts. This is to increase the reliability and integrity of the audit trail as you can always map back the deployed binary to a set of artifacts that were required (and inspect whether the process was followed).

For the build phase there are also security scanning activities that need to run on the binaries and container images you just created:

The SAST/SCA scans if the package requires to be built in order for your tool to be able to scan it

The container images you built which will need to be scanned with a container scanner and review if hardened properly

AMI images you built which will need to be vetted for vulnerabilities and hardening

Ensuring you have a binary and container signing process to certify your artifacts and to have assurance on file integrity

If there are issues, you can consider breaking the build and invalidating the artifact (e.g. not signing it).

Note that when generating artifacts, you normally deploy the same set in each environment and do not build “special” versions for your production environment.

That is why you abstract configurations outside of the applications and use tools to provision configuration parameters and secrets at runtime. For instance tools like Hashicorp Vault help you abstract these parameters so that developers do not need access to secrets or other sensitive information.

Build Gating

Hard Gate:

Vulnerable Third Party Dependencies with a risk level of High or Critical

SAST issues with a risk level of High or Critical

Container or Golden Image issues with a risk level of High or Critical

Soft Gate:

Vulnerable Third Party Dependencies with a risk level of Low or higher

SAST issues with a risk level of Low or higher

Container issues with a risk level of Low or higher

Test (Dynamic and Interactive Application Security Testing)

During the test phase your binaries will be deployed into a running machine (or container) and your code will be deployed inside. Your application will effectively be running.

This allows you to run DAST tools like OWASP Zap, Burpsuite Pro or Netsparker for instance. DAST tools can complement the SAST process to scan for security headers and run-time vulnerabilities.

Another tool that is becoming more common is IAST (Interactive Application Security Testing) which is agent based on runs “within” the application. The benefit of IAST is that you would encounter a lot less false positives than for instance SAST or DAST. It is a relative new technology but seems very promising in the sense that it may provide the coverage of an SAST with the vulnerability confidence of DAST.

If you are running more classic infrastructure you can also consider doing Vulnerability Assessments with tools such as Nessus or Nexpose.

Test (Unit Testing)

Ideally your developers should also codify security test cases to ensure security boundaries are respected and input/output validation is performed. From the OWASP testing guidelines:

A good practice for developers is to build security test cases as a generic security test suite that is part of the existing unit testing framework. A generic security test suite could be derived from previously defined use and misuse cases to security test functions, methods and classes. A generic security test suite might include security test cases to validate both positive and negative requirements for security controls such as: - Identity, Authentication & Access Control - Input Validation & Encoding - Encryption - User and Session Management - Error and Exception Handling - Auditing and Logging

Test (Penetration Testing)

Penetration testing may still be required, but the scope of the penetration test is vastly reduced. Instead of trying to find injection vulnerabilities such as XSS or SQL injection (which DAST and SAST tools are good at), the penetration tester should focus mostly on business logic. By doing so we can reduce the overall scope of the tester. The penetration tester can also advice on the coverage of the security unit test cases and where they can be improved.

Based on the features released you may also opt not to do a penetration test at all. For instance when your changes are largely cosmetic.

Test Gating

Hard Gate:

DAST issues with a risk of Medium or higher

Soft Gate:

DAST issues with a risk of Low or higher

Security Processes within DevOps: CD

Once you have created your binary you can now deploy it. Deployment processes should verify that the artifacts created alongside the binary have not been tampered with (e.g. signing the artefacts) and also vet that none of the artifacts contain reporting of risk levels outside of the tolerance level. If there are there should be an additional manual approval.

Release and Management: Platform security

Aside from securing the pipeline you also need to secure your platform, regardless whether that’s running on your private kubernetes, AWS Fargate, Azure Kubernetes or some other platform.

Platform security should minimally include:

2FA always

Splitting up accounts/groups and limit access for dev, SIT, UAT and prod

Auditing and accounting of all account and system events

No access for developers to production

Set up guardrails to prevent risky behavior

Asset tagging (e.g. to declare prod/non-prod, sensitivity of information, etc)

Usage of secrets management tools such as Azure KeyVault, AWS Key Management Service or Hashicorp Vault

Enabling of security services that help ensure the platform integrity and alert on certain changes such as automated scanning for platform compliance, this can be done with third party tools such as Redlock, Cloud Conformity or native tools similar to AWS Config or Azure Policy

Make use of infrastructure as code, you can use repositories to manage IaC similar to code. There are different tools available such as CloudFormation on AWS or platform agnostic technology like Terraform. Both have security scanning tools available like CloudFormation Guard or tfsec

These require on-going management and processing. It’s important to implement this earlier on as your environment will keep growing, and refactoring will keep becoming more complex and expensive.

Monitoring

Once your application is running in production you will still need to do active monitoring, especially with regard to third party dependencies, container images or other images. This is because new vulnerabilities are published on a daily basis. This means your dependencies inside your application or containers may become subject to new vulnerabilities. It’s important to monitor and remediate this as to avoid a situation faced by for instance Equifax that wasn’t aware of a security vulnerability in one of their libraries and got breached.

Tools that help scan for vulnerabilities often have a monitoring function you can enable. Make sure you set SLAs for remediation and have escalation paths where needed to ensure a swift remediation.

Incident Response

Things will go wrong at some point, having a good framework and response plans/playbooks are therefore important. Aside from having these artifacts, practicing and simulating security incidents should be performed as well.

A typical plan looks like this:

Prepare : ensure you have playbooks, communication plans, contact details etc available. You can also consider putting in place incident response automation. Cloud platforms lend themselves really well for this. Look for instance at this example on AWS or this AWS CleanRoom Forensics example for EC2

: ensure you have playbooks, communication plans, contact details etc available. You can also consider putting in place incident response automation. Cloud platforms lend themselves really well for this. Look for instance at this example on AWS or this AWS CleanRoom Forensics example for EC2 Detect: in order to detect issues you first need to put in place systems that can well … detect issues. This can be done through tools such as AWS GuardDuty, Azure Sentinel or through custom detection rules that are application specific rules

in order to detect issues you first need to put in place systems that can well … detect issues. This can be done through tools such as AWS GuardDuty, Azure Sentinel or through custom detection rules that are application specific rules Contain and Eradicate: once you have detected an incident it’s important to respond to it and track it to closure. An operational process needs to be put in place to track and report on these incidents, to avoid situations where no response is given. Digital forensics may be required to further analyse the issue, so ensure you have some capability in this area or have a retainer with a vendor that can support such activities

once you have detected an incident it’s important to respond to it and track it to closure. An operational process needs to be put in place to track and report on these incidents, to avoid situations where no response is given. Digital forensics may be required to further analyse the issue, so ensure you have some capability in this area or have a retainer with a vendor that can support such activities Recovery: past the incident lies recovery, where systems are brought back to their normal state of operations. Always ensure you put in place mitigating controls as well as additional monitoring to verify that the issue has been resolved and there are no active attempts to re-exploit it again

past the incident lies recovery, where systems are brought back to their normal state of operations. Always ensure you put in place mitigating controls as well as additional monitoring to verify that the issue has been resolved and there are no active attempts to re-exploit it again Lessons Learned: after each incident it’s good to do a lessons learned. this is to review where things can be improved, be it in terms of automation, tools used or communication protocols exercised.

Governance — Standardise and promote re-use

Good governance is important in order to sustain and improve the DevSecOps program. The governance should look at:

Listening to what is happening on the ground and intervene if required (either from the top down or bottom up)

Improve the processes and use value stream mapping to iterate and elevate the DevSecOps process so that friction is reduced and automation can be leveraged more effectively

Create re-usable artifacts such as libraries and standardise pipelines and processes across teams. For instance authentication libraries, sanitization/input validation libraries, crypto libraries, …

Policies should keep automation in mind and be pragmatic. If people don’t understand the policy they will just ignore it.

Have senior management buy-in and education: this is of the upmost importance as your senior leadership will be the ones that can help prioritize security and enforce remediation actions. Explain that if they do not act now, they are just creating security debt and it will become more costly to fix it in the future. Security mishaps may break customer confidence and repairing that confidence takes a long time.

CI/CD — Bringing it all together

Here’s a conceptional overview of what it may look like end-to-end:

In conclusion: DevSecOps is a journey not a destination

The above activities may seem a lot and daunting at first. The reality of it all is that you should start small, introduce certain activities (my recommendation are automated testing tools) and subsequently tackle the more complex items.

DevSecOps is an upfront investment. Activities like Threat Modelling and SAST will become more and more cumbersome the longer you postpone executing these activities. Similar to technical debt you may end up creating security debt if you don’t perform these often and early.

The key takeaways are:

Ensure you enable developers and do not become a burden, find a flexible approach with pragmatic risk management

Start small first with a handful of teams, create some success stories and subsequently scale out

Make use of soft gates first to find the balance of what controls can be enforced and which ones may be too early and cumbersome. It’s important to strike a balance between risk and ensuring development isn’t impeded

Invest in your security champions, ensure they get the time to do security

Waterfall is not dead

One thing I want to highlight is that Waterfall is not dead. If you have a structured project which is unlikely to have changes throughout the development lifecycle, Waterfall may still be a good development strategy. Generally Waterfall is still useful when:

All the requirements are well-known, definite, and fixed.

You have all the resources necessary for the delivery.

There is no need for additional researches.

You don’t have obscure requirements.

Your project is short.

Some organisations often combine both Waterfall and Agile and do a form of “iterative waterfall”. It’s not because you are doing waterfall you can’t benefit from some of the testing strategies of DevSecOps like automated code scans and use of code repos with gitflow branching strategies.

References

Additional Reading Materials

Some of the reading materials that helped me to better appreciate