Open-Source
Supply Chain Security
at Google
Russ Cox (he/him)
ACM SCORED
November 2023
go.dev/s/acmscoredOpen-Source
Supply Chain Security
at Google
Russ Cox (he/him)
ACM SCORED
November 2023
go.dev/s/acmscoredgo.dev/s/acmscored“Supply chain security”

ftWhat is a so ware supply chain?
What does it mean to be secure?

go.dev/s/acmscored“Supply chain security”
What is a so ware supply chain?
What does it mean to be secure?

ft

ft

Dra : Supply chain security is the engineering of defenses
against supply chain attacks.

go.dev/s/acmscored“Supply chain attack”
What is a supply chain attack?

go.dev/s/acmscored“Supply chain attack”
What is a supply chain attack?
A (so ware) supply chain attack is
the nefarious alteration of trusted so ware
before delivery.

ft

fift

(tweaking a de nition by Kim Zetter)go.dev/s/acmscoredgo.dev/s/acmscoredgo.dev/s/acmscoredgo.dev/s/acmscoredgo.dev/s/acmscoredgo.dev/s/acmscored“Open-source software supply chain attack”

ft

An open-source so ware supply chain attack is
the nefarious alteration of a trusted open-source component
before delivery used later in a trusted program.

go.dev/s/acmscoredgo.dev/s/acmscoredgo.dev/s/acmscoredgo.dev/s/acmscored“Open-source supply chain vulnerability”

ft

An open source supply chain vulnerability is
an exploitable weakness in trusted so warecaused by an open source component.

go.dev/s/acmscored“Open-source software supply chain vulnerability”
An open source supply chain vulnerability is
an exploitable weakness in a trusted so ware package
caused by one of that package’s open source components.

ft

ftft

‣ Trusted so ware need not be open-source (Minecra is not).

go.dev/s/acmscored“Open-source supply chain security”
Earlier dra :Supply chain security is the engineering of defenses against
supply chain attacks.

ftOpen source supply chain security is
the engineering of defenses against
open source supply chain attacks andopen source supply chain vulnerabilities.

go.dev/s/acmscored

Open-Source
Supply Chain Security
at Google

ffDisclaimers‣ Not exhaustive about e orts at Google.
‣ No intent to discount work being done elsewhere.
Just reporting about Google.

Thgo.dev/s/acmscored

Open-source software supply chain security at Google
ree main approaches:‣ Understanding the supply chain‣ Strengthening the supply chain‣ Monitoring the supply chain

go.dev/s/acmscored

Understanding
the software supply chainftTh( e so ware supply chain is all the places
where a supply chain attack might happen.)go.dev/s/acmscored

Understanding the software supply chain

Build Graphgo.dev/s/acmscored

Understanding the software supply chain

Server Graph (~2020, outdated)go.dev/s/acmscored

Understanding the software supply chain

Dependency Graphgo.dev/s/acmscored

Understanding the software supply chain

Dependency Graphgo.dev/s/acmscored

Understanding
the software supply chain

go.dev/s/acmscored

Strengthening
the software supply chain> Defend against attacks
Find vulnerabilities

go.dev/s/acmscored

Strengthening the software supply chain > Attacks

Cryptographic Signatures
Cryptographic signatures make it impossible
to nefariously alter code between signing and verifying.
Removes download infrastructure, hosting,network middleboxes as potential attack sites.
Introduces key distribution problems.

go.dev/s/acmscored

Strengthening the software supply chain > Attacks

Go Checksum Database
Map from (module, version) -> SHA256 of le tree
Signed by private key; public key hard-coded in Go distribution
Every download of public module
checks (possibly cached) checksum database entry.

fifi

Checksum database assumes rst observed copy of code is “correct”.
Makes (module, version) -> code mapping immutable.

go.dev/s/acmscored

Strengthening the software supply chain > Attacks

Sigstorego.dev/s/acmscored

Strengthening the software supply chain > Attacks

Sigstorego.dev/s/acmscored

Strengthening the software supply chain > Attacks

Computer System Security
All these boxes need to be secured too.
Dedicated build systems can provide better security, reproducibility:
Google Cloud Build, GitHub Actions.
(But engineering workstations, laptops need security too.)go.dev/s/acmscored

Strengthening the software supply chain > Attacks

Reproducible builds

go.dev/s/acmscored

Strengthening the software supply chain > Attacks

Two-Person Approvals

go.dev/s/acmscored

Strengthening the software supply chain > Attacks

Supply-chain Levels for Software Artifacts (SLSA)

go.dev/s/acmscored

Strengthening the software supply chain > Attacks

OpenSSF Security Scorecards

go.dev/s/acmscored

Strengthening the software supply chain > Attacks

Capslock% capslock -output=verbose
Capslock is an experimental tool for static analysis of Go packages.
Analyzed packages:
golang.org/x/text v0.9.0
CAPABILITY_READ_SYSTEM_STATE: 2 references (2 direct, 0 transitive)
Example callpath:
rsc.io/sampler.Hellosampler.go:32:27:rsc.io/sampler.DefaultUserPrefs
sampler.go:21:22:os.Getenv
CAPABILITY_UNANALYZED: 5 references (0 direct, 5 transitive)
Example callpath:
rsc.io/sampler.DefaultUserPrefs
sampler.go:22:39:golang.org/x/text/language.Make
language.go:44:21:(golang.org/x/text/language.CanonType).Make
language.go:50:17:(golang.org/x/text/language.CanonType).Parse
parse.go:58:29:golang.org/x/text/language.canonicalize
language.go:128:44:(golang.org/x/text/internal/language.Language).Canonicalize
lookup.go:50:17:golang.org/x/text/internal/language.normLang
lookup.go:55:18:sort.Searchgo.dev/s/acmscored

Strengthening the software supply chain > AttacksCapslock Limitations
Code operating within its capabilities can still be nefarious.
‣ String comparison in a password checker.
‣ YAML parser in a production system.
‣ XcodeGhostgo.dev/s/acmscored

Strengthening
the software supply chain
Defend against attacks> Find vulnerabilities

go.dev/s/acmscored

Strengthening the software supply chain > Vulnerabilities

OSS Fuzz

go.dev/s/acmscored

Strengthening the software supply chain > Vulnerabilities

Syzkaller

go.dev/s/acmscored

Strengthening the software supply chain > Safe Languages

Internet Worm, November 1988

“It has raised the public awareness to aconsiderable degree. It is likely to makepeople more careful and more attentive tovulnerabilities in the future.”– Robert H. Morris,
quoted in the next day’s paper

go.dev/s/acmscored

Strengthening the software supply chain > Safe Languages

Memory-Safe Languages

go.dev/s/acmscored

Strengthening the software supply chain > Safe Languages

Memory-Safe Languages

go.dev/s/acmscored

Strengthening the software supply chain > Safe Languages

Memory-Safe Languages

go.dev/s/acmscored

Strengthening the software supply chain > Safe Languages

Memory-Safe Languages

go.dev/s/acmscored

Strengthening the software supply chain > Safe Languages

Serious Vulnerabilities by Languages

flff

C/C++: Bu er over ow => Remote code execution

go.dev/s/acmscored

Strengthening the software supply chain > Safe Languages

Serious Vulnerabilities by Languages

fl

fl

ff

C/C++: Bu er over ow => Remote code execution
Java: Misuse of re ection, code loading => Remote code execution

go.dev/s/acmscored

Strengthening the software supply chain > Safe Languages

Serious Vulnerabilities by Languages
C/C++: Bu er over ow => Remote code execution
Java: Misuse of re ection, code loading => Remote code execution

flflff

Go: Large or malformed inputs => Denial of service
Rust: Large or malformed inputs => Denial of service

go.dev/s/acmscored

Monitoring
the software supply chain

go.dev/s/acmscored

Monitoring the software supply chain

What is a Software Bill of Materials (SBOM)?
I don’t know. Do you?

go.dev/s/acmscored

Monitoring the software supply chain

What is a Software Bill of Materials (SBOM)?
I don’t know. Do you?

fiDe nitely includes list of packages and version.

go.dev/s/acmscored

Monitoring the software supply chain

What is a Software Bill of Materials (SBOM)?
“An SBOM is e ectively a nested inventory, a list of ingredients that make up so warecomponents. An SBOM identi es and lists so ware components, information about thosecomponents, and supply chain relationships between them. e amount and type of
information included in a particular SBOM may vary depending on factors such as theindustry or sector and the needs of SBOM consumers.”

ft

ft

Thft

ftfi

ff

ft– “Framing So ware Component Transparency: Establishing a Common So ware Bill ofMaterials (SBOM)”, NTIA Multistakeholder Process on So ware Component
Transparency Framing Working Group, 2021

go.dev/s/acmscored

Monitoring the software supply chain

What is a Software Bill of Materials (SBOM)?

fi

De nitely includes list of packages and version.

go.dev/s/acmscored

Monitoring the software supply chain

Go “SBOM”% go version -m $HOME/bin/gomote/Users/rsc/bin/gomote: go1.21.0
path golang.org/x/build/cmd/gomotemodgolang.org/x/build
v0.0.0-20230809040836-4f3589752dd4
dep
cloud.google.com/go/compute/metadata v0.2.3
dep
github.com/aws/aws-sdk-go
v1.30.15
dep
github.com/golang/groupcache
v0.0.0-20210331224755-41bb18bfe9dadep
github.com/golang/protobuf
v1.5.3dep
github.com/google/s2a-go
v0.1.4dep
github.com/google/uuid
v1.3.0dep
github.com/googleapis/enterprise-certificate-proxy v0.2.3
dep
github.com/googleapis/gax-go/v2 v2.10.0
dep
github.com/jmespath/go-jmespath v0.4.0
dep
go.opencensus.iov0.24.0
dep
golang.org/x/cryptov0.9.0dep
golang.org/x/net
v0.10.0
dep
golang.org/x/oauth2
v0.8.0...

h1:gK+EqJ6LNNP/...
h1:mg4jlk7mCAj6...
h1:Sd8QDVzzE8Sl...
h1:oI5xCqsCo564...
h1:KhyjKVUg7Usr...
h1:1kZ/sQM3sreP...
h1:t6JiXgmwXMjE...
h1:yk9/cqRKtT9w...
h1:ebSgKfMxynOd...
h1:BEgLn5cpjn8U...
h1:y73uSU6J157Q...
h1:LF6fAI+IutBo...
h1:X2//UzNDwYmt...
h1:6dkIjl3j3LtZ...

go.dev/s/acmscored

Monitoring the software supply chain

Go Vulnerability Databasego.dev/s/acmscored

Monitoring the software supply chain

Open-Source Vulnerability Database (OSV)

go.dev/s/acmscored

Monitoring the software supply chain

Vulnerability Scanning% govulncheck -mode=binary gomote
Scanning your binary for known vulnerabilities...
Vulnerability #1: GO-2023-2153
Denial of service from HTTP/2 Rapid Reset in google.golang.org/grpc
More info: https://pkg.go.dev/vuln/GO-2023-2153
Module: google.golang.org/grpc
Found in: google.golang.org/grpc@v1.55.0
Fixed in: google.golang.org/grpc@v1.58.3
Example traces found:#1: grpc.Server.Serve#2: transport.NewServerTransport
Vulnerability #2: GO-2023-2043
Improper handling of special tags within script contexts in html/template
More info: https://pkg.go.dev/vuln/GO-2023-2043
Standard library
Found in: html/template@go1.21
Fixed in: html/template@go1.21.1
Example traces found:#1: template.Template.Execute#2: template.Template.ExecuteTemplatego.dev/s/acmscored

Monitoring the software supply chain

Vulnerability Scanning% govulncheck ./gomote
Scanning your code and 399 packages across 23 dependent modules for known vulnerabilities...
=== Informational ===Found 2 vulnerabilities in packages that you import, but there are no callstacks leading to the use of these vulnerabilities. You may not need totake any action. See https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheckfor details.
Vulnerability #1: GO-2023-2153
Denial of service from HTTP/2 Rapid Reset in google.golang.org/grpc
More info: https://pkg.go.dev/vuln/GO-2023-2153
Module: google.golang.org/grpc
Found in: google.golang.org/grpc@v1.58.2
Fixed in: google.golang.org/grpc@v1.58.3
Vulnerability #2: GO-2023-2102...
No vulnerabilities found.

go.dev/s/acmscored

Monitoring the software supply chain

Vulnerability Scanning in IDE

go.dev/s/acmscored

Monitoring the software supply chain

Vulnerability Scanning in IDE

go.dev/s/acmscored

Monitoring the software supply chain

Vulnerability Scanning in IDE

go.dev/s/acmscored

Monitoring the software supply chain

Vulnerability Scanning in Production

go.dev/s/acmscored

Open-Source
Supply Chain Security
at Google

go.dev/s/acmscored

Open-Source
Supply Chain Security
Historical Perspective

go.dev/s/acmscored

Air Force review of Multics, 1972–1974

go.dev/s/acmscored

Air Force review of Multics, 1972–1974

go.dev/s/acmscored

Air Force review of Multics, 1972–1974

go.dev/s/acmscored

Air Force review of Multics, 1972–1974

go.dev/s/acmscored

Ken Thompson’s Turing Award Lecturego.dev/s/acmscored

Ken Thompson’s Actual Code

go.dev/s/acmscored

Do We Learn From History?

Th

1974 Multics report
1983 ompson lecture
1988 Internet worm…

go.dev/s/acmscored

Do We Learn From History?
1974 Multics report
1983 ompson lecture
1988 Internet worm
…

ThNo.

go.dev/s/acmscored

Do We Learn From History?
1974 Multics report
1983 ompson lecture
1988 Internet worm…

ThNo.
But why are things not worse?

go.dev/s/acmscored

Hope For the Future
Industry can x problems when it wants to.
‣ HTTP to HTTPS‣ 2-Factor Auth and Security Keys
‣ Passkeys?

fifiMaybe we want to x supply chain security next.
Hopefully we will.

Open-Source
Supply Chain Security
at Google
Russ Cox (he/him)
ACM SCORED
November 2023
go.dev/s/acmscored