Package: clair-cicd
Clair,released by CoreOS in Nov
’16,
is a very effective tool for statically analyzing docker images to
determine which known vulnerabilities exist in the images. Integrating
Clair into a CI/CD pipeline:is complex (believe this is partly a documentation challenge)creates performance problems (building the Clair required Postgres
database of vulnerabilities is slow)in and of itself is insufficient from a risk assessment point of view
because once vulnerabilities are identified there’s a lack of
prescriptive guidance on how to act on the identified vulnerabilitiesThis repo was created to address the above challenges.BackgroundThe roots of this repo center around the following beliefs:when inserted into a CI/CD pipelineClaircan be a very effective
foundation for the automated assessment of Docker image vulnerability
riskservices should be run in Docker containers and thus CI/CD pipelines
should be focused on the automated generation, assessment and
ultimately deployment of Docker imagesunderstanding and assessing the risk profile of services is important
ie. security is importantDocker images shouldnotbe pushed to a Docker registry until
their risk profile is understood (this is an important one)the CI/CD pipeline has to be fast. how fast? ideally < 5 minutes
between code commit and automated (CD) deployment begins rolling out
a changethere should be a clear division of responsibilities between those
who create a docker image (service engineer) and those who determine
the risk of vulnerabilities in a docker image (security analyst)the risk assessment process should generate evidence which can be
used to understand the risk assessment decisionKey Participantsservice engineer - responsible for implementing a service that is
packaged in a docker imagesecurity analyst - responsible for defining whitelists which are
consumed byclair_cicdto influence Docker image risk assessment
decisionsHow to UseGetting StartedTo start usingclair-cicd, a service engineer inserts a single line
of code into a service’s CI pipeline. The single line of code runs the
shell scriptassess-image-risk.sh.
Part of the CI pipeline’s responsibility is to build the docker image
and then push that docker image to a docker registry. The single line ofclair-cicdcode should appear after the docker image is built and
tested but before the docker image is pushed to a docker registry.In this simple case,assess-image-risk.shreturns a zero exit status if the docker image contains no known
vulnerabilities above a medium severity. If the docker image contains
any known vulnerabilities with a severity higher than medium,assess-image-risk.shreturns a non-zero exit status and the build fails ie. the build should
fail before the docker image is pushed to a docker registry.The example illustrates what’s described about for thealpine:3.4
docker image.~>curl-s-L\https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh|\bash-s--alpine:3.4~>echo$?0~>Understanding the Risk Assessment DecisionTo understand howassess-image-risk.shis making its risk assessment
decision try using the-vflag.~>curl-s-L\https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh|\bash-s---valpine:3.42020-01-1216:43:35pullingclairdatabaseimage'simonsdave/clair-cicd-database:latest'2020-01-1216:44:17successfullypulledclairdatabaseimage2020-01-1216:44:17startingclairdatabasecontainer'clair-db-c1dbb5f93ae98755'2020-01-1216:44:23waitingfordatabaseserverincontainer'clair-db-c1dbb5f93ae98755'tostart...........................2020-01-1216:44:54successfullystartedclairdatabasecontainer2020-01-1216:44:54clairconfigurationin'/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.ElAlhGNl'2020-01-1216:44:59pullingclairimage'simonsdave/clair-cicd-clair:latest'2020-01-1216:45:13successfullypulledclairimage'simonsdave/clair-cicd-clair:latest'2020-01-1216:45:13startingclaircontainer'clair-e9573ae537134fa0'2020-01-1216:45:15successfullystartedclaircontainer'clair-e9573ae537134fa0'2020-01-1216:45:15savingdockerimage'alpine:3.4'to'/tmp/tmp.IaNHCH'2020-01-1216:45:16successfullysaveddockerimage'alpine:3.4'2020-01-1216:45:16startingtocreateclairlayers2020-01-1216:45:16creatingclairlayer'378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'2020-01-1216:45:16successfullycreatedclairlayer'378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'2020-01-1216:45:16donecreatingclairlayers2020-01-1216:45:16startingtogetvulnerabilitiesforclairlayers2020-01-1216:45:16savingvulnerabilitiestodirectory'/tmp/tmp.MDncHN'2020-01-1216:45:16gettingvulnerabilitiesforlayer'378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'2020-01-1216:45:16successfullygotvulnerabilitiesforlayer'378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'2020-01-1216:45:16donegettingvulnerabilitiesforclairlayers2020-01-1221:45:17INFOio:89Lookingforvulnerabilitiesindirectory'/tmp/tmp.MDncHN'2020-01-1221:45:17INFOio:95Found1fileswithvulnerabilitiesindirectory'/tmp/tmp.MDncHN'2020-01-1221:45:17INFOio:104Lookingforvulnerabilitiesin'/tmp/tmp.MDncHN/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'2020-01-1221:45:17INFOio:122Found0vulnerabilitiesin'/tmp/tmp.MDncHN/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'2020-01-1221:45:17INFOio:133Found0vulnerabilitiesin1filesindirectory'/tmp/tmp.MDncHN'2020-01-1221:45:17INFOassessor:19Assessmentstarts2020-01-1221:45:17INFOassessor:26Assessmentends-pass~>echo$?0~>Adding a Vulnerability WhitelistIn the above examples a default vulnerability whitelist was used. When
specified as a JSON doc, this whitelist would be:{"ignoreSevertiesAtOrBelow":"medium"}By default,assess-image-risk.shreturns a non-zero exit status if any vulnerabilities are identified in
the image with a severity higher than medium. The medium is derived from
the default vulnerability whitelist.The example below illustrate how to specify a vulnerability whitelist
and with a severity other than medium. Note the use of thejson://prefix to indicate this is an inline whitelist.~>curl-s-L\https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh|\bash-s---v--whitelist'json://{"ignoreSevertiesAtOrBelow": "negligible"}'ubuntu:18.042020-01-1216:46:56pullingclairdatabaseimage'simonsdave/clair-cicd-database:latest'2020-01-1216:46:58successfullypulledclairdatabaseimage2020-01-1216:46:58startingclairdatabasecontainer'clair-db-3b0811925f7e8bc2'2020-01-1216:46:59waitingfordatabaseserverincontainer'clair-db-3b0811925f7e8bc2'tostart.............................2020-01-1216:47:32successfullystartedclairdatabasecontainer2020-01-1216:47:32clairconfigurationin'/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.BXCs3Giy'2020-01-1216:47:34pullingclairimage'simonsdave/clair-cicd-clair:latest'2020-01-1216:47:36successfullypulledclairimage'simonsdave/clair-cicd-clair:latest'2020-01-1216:47:36startingclaircontainer'clair-fc579c71e7daba57'2020-01-1216:47:38successfullystartedclaircontainer'clair-fc579c71e7daba57'2020-01-1216:47:38savingdockerimage'ubuntu:18.04'to'/tmp/tmp.lPDhNd'2020-01-1216:47:43successfullysaveddockerimage'ubuntu:18.04'2020-01-1216:47:43startingtocreateclairlayers2020-01-1216:47:43creatingclairlayer'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'2020-01-1216:47:43successfullycreatedclairlayer'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'2020-01-1216:47:43creatingclairlayer'27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'2020-01-1216:47:44successfullycreatedclairlayer'27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'2020-01-1216:47:44creatingclairlayer'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'2020-01-1216:47:44successfullycreatedclairlayer'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'2020-01-1216:47:44creatingclairlayer'1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'2020-01-1216:47:44successfullycreatedclairlayer'1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'2020-01-1216:47:44donecreatingclairlayers2020-01-1216:47:44startingtogetvulnerabilitiesforclairlayers2020-01-1216:47:44savingvulnerabilitiestodirectory'/tmp/tmp.dkfgmI'2020-01-1216:47:44gettingvulnerabilitiesforlayer'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'2020-01-1216:47:44successfullygotvulnerabilitiesforlayer'cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd'2020-01-1216:47:44gettingvulnerabilitiesforlayer'27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'2020-01-1216:47:44successfullygotvulnerabilitiesforlayer'27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54'2020-01-1216:47:44gettingvulnerabilitiesforlayer'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'2020-01-1216:47:44successfullygotvulnerabilitiesforlayer'd80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e'2020-01-1216:47:44gettingvulnerabilitiesforlayer'1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'2020-01-1216:47:44successfullygotvulnerabilitiesforlayer'1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d'2020-01-1216:47:44donegettingvulnerabilitiesforclairlayers2020-01-1221:47:45INFOio:89Lookingforvulnerabilitiesindirectory'/tmp/tmp.dkfgmI'2020-01-1221:47:45INFOio:95Found4fileswithvulnerabilitiesindirectory'/tmp/tmp.dkfgmI'2020-01-1221:47:45INFOio:104Lookingforvulnerabilitiesin'/tmp/tmp.dkfgmI/27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54.json'2020-01-1221:47:45INFOio:122Found33vulnerabilitiesin'/tmp/tmp.dkfgmI/27a911bb510bf1e9458437f0f44216fd38fd08c462ed7aa026d91aab8c054e54.json'2020-01-1221:47:45INFOio:104Lookingforvulnerabilitiesin'/tmp/tmp.dkfgmI/cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd.json'2020-01-1221:47:45INFOio:122Found33vulnerabilitiesin'/tmp/tmp.dkfgmI/cc59b0ca1cf21d77c81a98138703008daa167b1ab1a115849d498dba64e738dd.json'2020-01-1221:47:45INFOio:104Lookingforvulnerabilitiesin'/tmp/tmp.dkfgmI/1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d.json'2020-01-1221:47:45INFOio:122Found33vulnerabilitiesin'/tmp/tmp.dkfgmI/1ee34a985f7aef86436a5519f5ad83f866a74c7d9a0c22e47c4213ee9cb64e6d.json'2020-01-1221:47:45INFOio:104Lookingforvulnerabilitiesin'/tmp/tmp.dkfgmI/d80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e.json'2020-01-1221:47:45INFOio:122Found33vulnerabilitiesin'/tmp/tmp.dkfgmI/d80735acaa72040a0a98ca3ae6891f9abb4e2f5d627b4099c4fefdc3ce1e696e.json'2020-01-1221:47:45INFOio:133Found33vulnerabilitiesin4filesindirectory'/tmp/tmp.dkfgmI'2020-01-1221:47:45INFOassessor:19Assessmentstarts2020-01-1221:47:45INFOassessor:34AssessingvulnerabilityCVE-2018-11236-start2020-01-1221:47:45INFOassessor:52VulnerabilityCVE-2018-11236@severitymediumgreaterthanwhitelistseverity@negligible-fail2020-01-1221:47:45INFOassessor:36AssessingvulnerabilityCVE-2018-11236-finish2020-01-1221:47:45INFOassessor:23Assessmentends-fail~>echo$?1~>The above is an example of an inline whitelist. It’s also possible to
specify a whitelist in a file. The example below illustrates the usage.
Note use of thefile://prefix to indicate the whitelist is
contained in a file.~>catwhitelist.json{"ignoreSevertiesAtOrBelow":"medium"}~>curl-s-L\https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh|\bash-s---v--whitelistfile://whitelist.jsonalpine:3.42020-01-1216:48:41pullingclairdatabaseimage'simonsdave/clair-cicd-database:latest'2020-01-1216:48:42successfullypulledclairdatabaseimage2020-01-1216:48:42startingclairdatabasecontainer'clair-db-191152e37b864e4b'2020-01-1216:48:43waitingfordatabaseserverincontainer'clair-db-191152e37b864e4b'tostart.............................2020-01-1216:49:16successfullystartedclairdatabasecontainer2020-01-1216:49:16clairconfigurationin'/var/folders/7x/rr443kj575s8zz54jrbrp4jc0000gn/T/tmp.GdlBNmiG'2020-01-1216:49:19pullingclairimage'simonsdave/clair-cicd-clair:latest'2020-01-1216:49:20successfullypulledclairimage'simonsdave/clair-cicd-clair:latest'2020-01-1216:49:20startingclaircontainer'clair-747d1c50606fba7e'2020-01-1216:49:21successfullystartedclaircontainer'clair-747d1c50606fba7e'2020-01-1216:49:22savingdockerimage'alpine:3.4'to'/tmp/tmp.Eldkbe'2020-01-1216:49:23successfullysaveddockerimage'alpine:3.4'2020-01-1216:49:23startingtocreateclairlayers2020-01-1216:49:23creatingclairlayer'378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'2020-01-1216:49:23successfullycreatedclairlayer'378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'2020-01-1216:49:23donecreatingclairlayers2020-01-1216:49:23startingtogetvulnerabilitiesforclairlayers2020-01-1216:49:23savingvulnerabilitiestodirectory'/tmp/tmp.pCOhlL'2020-01-1216:49:23gettingvulnerabilitiesforlayer'378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'2020-01-1216:49:23successfullygotvulnerabilitiesforlayer'378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d'2020-01-1216:49:23donegettingvulnerabilitiesforclairlayers2020-01-1221:49:23INFOio:89Lookingforvulnerabilitiesindirectory'/tmp/tmp.pCOhlL'2020-01-1221:49:23INFOio:95Found1fileswithvulnerabilitiesindirectory'/tmp/tmp.pCOhlL'2020-01-1221:49:23INFOio:104Lookingforvulnerabilitiesin'/tmp/tmp.pCOhlL/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'2020-01-1221:49:23INFOio:122Found0vulnerabilitiesin'/tmp/tmp.pCOhlL/378cb6b4a17e08c366cebd813d218f60889848387fa61a56ac054ca027a4890d.json'2020-01-1221:49:23INFOio:133Found0vulnerabilitiesin1filesindirectory'/tmp/tmp.pCOhlL'2020-01-1221:49:23INFOassessor:19Assessmentstarts2020-01-1221:49:23INFOassessor:26Assessmentends-pass~>echo$?0~>Whitelists can also contain vulnerabilities. The example below
illustrates this capability. If you add the-v(verbose) flag toassess-image-risk.shyou see exactly how the vulnerabilities impact
the risk assessment with statements likeVulnerabilityCVE-2019-13627in whitelist - pass~>curl-s-L\https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh|\bash-s----whitelist'json://{"ignoreSevertiesAtOrBelow":"low"}'ubuntu:18.04~>echo$?1~>catwhitelist.json{"ignoreSevertiesAtOrBelow":"low","vulnerabilities":[{"cveId":"CVE-2018-20839","rationale":"reason #1"},{"cveId":"CVE-2019-5188","rationale":"reason #2"},{"cveId":"CVE-2018-11236","rationale":"reason #3"},{"cveId":"CVE-2019-13627","rationale":"reason #4"},{"cveId":"CVE-2019-13050","rationale":"reason #5"},{"cveId":"CVE-2018-11237","rationale":"reason #6"},{"cveId":"CVE-2018-19591","rationale":"reason #7"}]}~>curl-s-L\https://raw.githubusercontent.com/simonsdave/clair-cicd/master/bin/assess-image-risk.sh|\bash-s----whitelist'file://whitelist.json'ubuntu:18.04~>echo$?0~>Responsibilitiesmaintained by security analystnotservice engineerchecked into source code control and appropriate change management
processes are used to make changes (code reviews, feature branches,
etc)How it Works + Requirements/AssumptionsThere are 3 moving pieces:assess-image-risk.shis bash script which does the heavy lifting to co-ordinate the
interaction of the 2 other moving piecesClair’svulnerability database
which is packaged inside the docker imagesimonsdave/clair-database- aCircleCIcron job is used to rebuildsimonsdave/clair-database3 days per week to ensure the vulnerability database is currenta set of Python and Bash risk assessment scripts packaged in thesimonsdave/clair-cicd-clairdocker image which is based on the docker imagequay.io/coreos/clairwhich packages upClairFrom the samples at the start of this doc you’ll see the approach of
curl’ing the latest release ofassess-image-risk.shinto a localy run bash shell.assess-image-risk.shthen spins up a container using thesimonsdave/clair-database.
Another container is then run usingsimonsdave/clair-cicd-clairwith thesimonsdave/clair-cicd-claircontainer being able to talk with thesimonsdave/clair-databasecontainer. Once thesimonsdave/clair-cicd-claircontainer is running,assess-image-risk.shdocker exec’sthis bash
scriptwhich does the actual
risk assessment.Armed with the understanding of howclair-cicdworks you’ll
appreciate that the ability to executeassess-image-risk.shis what defines the requirements for the execution environment.assess-image-risk.shis a bash script used to launch the risk assessment process and as such
it’s this script which defines the bulk of the assumptions/requirements
forclair-cicd- the script uses docker, sed and openssl so all
these need to be available in the environment runningclair-cicdReferences29 Jan ’20 - What Is DevSecOps, and How Is It Different from
DevOps?1 Oct ’18 - Baking Compliance in your CI/CD
Pipeline11 Sep ’18 - What is
DevSecOps?20 Jun ’18 - Where in the DevOps cycle do you do
security?8 May ’18 - DevSecOps: 7 habits of strong security
organizations18 Apr ’18 - The Cloudcast #343 - Container Vulnerability
Scanning21 Feb ’18 - Automated Compliance Testing Tool Accelerates
DevSecOps20 Feb ’18 - 6 Requirements for Achieving
DevSecOps22 Jan ’18 - DevOps and Security: How to Overcome Cultural
Challenges and Transform to True
DevSecOps15 Jan ’18 - Why DevSecOps matters to IT
leaders27 Nov ’17 - What is vulnerability management? Processes and
software for prioritizing
threats23 Oct ’17 - The Ten Cybersecurity
Commandments9 Oct ’17 - 10 layers of Linux container
security5 Oct ’17 - How to Maintain Security when Rolling out
DevOps26 Jan ’17 - DevOps and Separation of
Duties26 Jul ’16 - Injecting security into Continuous
Delivery5 Jun ’16 - <— Shifting Security to the
LeftFive Secrets and Two Common “Gotchas” of Vulnerability
Scanning