Repository: google/exposure-notifications-server
Pull request #948 (2020-09-02 16:21:43)

Smoke test terraform deployment
- Add a `terraform-e2e` module as a thin wrapper of `terraform` module, just for e2e tests
- Add a script for smoke testing terraform deployment

google-oss-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/google/exposure-notifications-server/pull/948#" title="Author self-approved">chaodaiG</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **mikehelmick**
You can assign the PR to them by writing `/assign @mikehelmick` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=google%2Fexposure-notifications-server).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/google/exposure-notifications-server/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["mikehelmick"]} -->
chaodaiG: /test pull-en-server-release-e2e-canary
google-oss-robot: @chaodaiG: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-en-server-release-e2e-canary | 9b40b85e7c37acf68a652e40b53be32970f197d9 | [link](https://oss-prow.knative.dev/view/gs/oss-prow/pr-logs/pull/google_exposure-notifications-server/948/pull-en-server-release-e2e-canary/1301193847279194113) | `/test pull-en-server-release-e2e-canary`

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
chaodaiG: /test pull-en-server-release-e2e-canary
google-oss-robot: @chaodaiG: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-en-server-release-e2e-canary | 67b5ae2dd367731ee531825fd8ec32489b9358e3 | [link](https://oss-prow.knative.dev/view/gs/oss-prow/pr-logs/pull/google_exposure-notifications-server/948/pull-en-server-release-e2e-canary/1301194224795914241) | `/test pull-en-server-release-e2e-canary`

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
chaodaiG: Since #950 merged and image pushed up
/test pull-en-server-release-e2e-canary
chaodaiG: /test pull-en-server-release-e2e-canary
google-oss-robot: @chaodaiG: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-en-server-release-e2e-canary | 67b5ae2dd367731ee531825fd8ec32489b9358e3 | [link](https://oss-prow.knative.dev/view/gs/oss-prow/pr-logs/pull/google_exposure-notifications-server/948/pull-en-server-release-e2e-canary/1301219369883275264) | `/test pull-en-server-release-e2e-canary`

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
chaodaiG: /test pull-en-server-release-e2e-canary

Patch:
diff --git a/docs/design/verification_protocol.md b/docs/design/verification_protocol.md
index 07b8ab9..dc14f57 100644
--- a/docs/design/verification_protocol.md
+++ b/docs/design/verification_protocol.md
@@ -73,7 +73,7 @@ authorities.
 We also prescribe a set of private claims to transmit data from the PHA
 verification server to the exposure notification key server.
 
-* `tekhmac` : _REQUIRED_ The HMAC of the TEKs that was presented to the PHA verification
+* `tekmac` : _REQUIRED_ The HMAC of the TEKs that was presented to the PHA verification
 server. This must be calculated in a specific way (see below). This is REQUIRED.
 Base64 encoded string property.
 * `reportType` : _REQUIRED_ One of 'confirmed', 'likely', or 'negative' to indicate the
diff --git a/internal/integration/helpers.go b/internal/integration/helpers.go
index 6b607bb..a16cd49 100644
--- a/internal/integration/helpers.go
+++ b/internal/integration/helpers.go
@@ -95,18 +95,11 @@ func NewTestServer(tb testing.TB, exportPeriod time.Duration) (*serverenv.Server
 		}
 	}
 
-	km, err := keys.NewInMemory(ctx)
-	if err != nil {
-		tb.Fatal(err)
-	}
-	if _, err := km.CreateEncryptionKey("tokenkey"); err != nil {
-		tb.Fatal(err)
-	}
-	if _, err := km.CreateSigningKey(ctx, "signing", "signingkey"); err != nil {
-		tb.Fatal(err)
-	}
+	kms := keys.TestKeyManager(tb)
+	tokenKey := keys.TestEncryptionKey(tb, kms)
+
 	// create an initial revision key.
-	revisionDB, err := revdb.New(db, &revdb.KMSConfig{WrapperKeyID: "tokenkey", KeyManager: km})
+	revisionDB, err := revdb.New(db, &revdb.KMSConfig{WrapperKeyID: tokenKey, KeyManager: kms})
 	if err != nil {
 		tb.Fatal(err)
 	}
@@ -144,7 +137,7 @@ func NewTestServer(tb testing.TB, exportPeriod time.Duration) (*serverenv.Server
 		serverenv.WithAuthorizedAppProvider(aap),
 		serverenv.WithBlobStorage(bs),
 		serverenv.WithDatabase(db),
-		serverenv.WithKeyManager(km),
+		serverenv.WithKeyManager(kms),
 		serverenv.WithSecretManager(sm),
 	)
 	// Note: don't call env.Cleanup() because the database helper closes the
@@ -208,7 +201,7 @@ func NewTestServer(tb testing.TB, exportPeriod time.Duration) (*serverenv.Server
 	// TODO: this is a grpc listener and requires a lot of setup.
 
 	revConfig := revision.Config{
-		KeyID:     "tokenkey",
+		KeyID:     tokenKey,
 		AAD:       []byte{1, 2, 3},
 		MinLength: 28,
 	}
diff --git a/internal/publish/config.go b/internal/publish/config.go
index 99202d5..5c8d274 100644
--- a/internal/publish/config.go
+++ b/internal/publish/config.go
@@ -56,6 +56,9 @@ type Config struct {
 	MaxMagnitudeSymptomOnsetDays uint          `env:"MAX_SYMPTOM_ONSET_DAYS, default=14"`
 	CreatedAtTruncateWindow      time.Duration `env:"TRUNCATE_WINDOW, default=1h"`
 
+	ResponsePaddingMinBytes int64 `env:"RESPONSE_PADDING_MIN_BYTES, default=1024"`
+	ResponsePaddingRange    int64 `env:"RESPONSE_PADDING_RANGE, default=1024"`
+
 	RevisionKeyCacheDuration time.Duration `env:"REVISION_KEY_CACHE_DURATION, default=1m"`
 
 	// API Versions.
diff --git a/internal/publish/publish.go b/internal/publish/publish.go
index bd92a2a..a8fe607 100644
--- a/internal/publish/publish.go
+++ b/internal/publish/publish.go
@@ -17,9 +17,11 @@ package publish
 
 import (
 	"context"
+	"crypto/rand"
 	"encoding/base64"
 	"errors"
 	"fmt"
+	"math/big"
 	"net/http"
 	"time"
 
@@ -119,6 +121,39 @@ type response struct {
 	count       int // metricCount
 }
 
+func (r *response) padResponse(c *Config) error {
+	if r.pubResponse == nil {
+		return fmt.Errorf("no publish response exists to pad")
+	}
+
+	minBytes := c.ResponsePaddingMinBytes
+	if minBytes <= 0 {
+		minBytes = 1024
+	}
+	padRange := c.ResponsePaddingRange
+	if padRange <= 0 {
+		padRange = 1024
+	}
+
+	bi, err := rand.Int(rand.Reader, big.NewInt(padRange))
+	if err != nil {
+		return fmt.Errorf("padding: failed to generate random number: %w", err)
+	}
+	i := int(bi.Int64() + minBytes)
+
+	b := make([]byte, i)
+	n, err := rand.Read(b)
+	if err != nil {
+		return fmt.Errorf("padding: failed to read bytes: %w", err)
+	}
+	if n < i {
+		return fmt.Errorf("padding: wrote less bytes than expected")
+	}
+
+	r.pubResponse.Padding = fmt.Sprintf("%q", base64.StdEncoding.EncodeToString(b))
+	return nil
+}
+
 // versionBridge closes the gap in up-leveling v1alpha1 to v1 API.
 type versionBridge struct {
 	AdditionalRegions []string
diff --git a/internal/publish/publish_v1.go b/internal/publish/publish_v1.go
index 89f3533..007008a 100644
--- a/internal/publish/publish_v1.go
+++ b/internal/publish/publish_v1.go
@@ -24,6 +24,7 @@ import (
 
 	"github.com/google/exposure-notifications-server/internal/jsonutil"
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/mikehelmick/go-chaff"
 )
 
@@ -60,9 +61,15 @@ func (h *PublishHandler) Handle() http.Handler {
 		http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 			response := h.handleRequest(w, r)
 
+			ctx := r.Context()
+			metrics := h.serverenv.MetricsExporter(ctx)
+
+			if err := response.padResponse(h.config); err != nil {
+				metrics.WriteInt("padding-failed", true, 1)
+				logging.FromContext(ctx).Errorw("failed to padd response", "error", err)
+			}
+
 			if response.metric != "" {
-				ctx := r.Context()
-				metrics := h.serverenv.MetricsExporter(ctx)
 				metrics.WriteInt(response.metric, true, response.count)
 			}
 
diff --git a/internal/publish/publish_v1alpha1.go b/internal/publish/publish_v1alpha1.go
index acff957..139988e 100644
--- a/internal/publish/publish_v1alpha1.go
+++ b/internal/publish/publish_v1alpha1.go
@@ -25,6 +25,7 @@ import (
 	"github.com/google/exposure-notifications-server/internal/jsonutil"
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1"
 	"github.com/google/exposure-notifications-server/pkg/api/v1alpha1"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/mikehelmick/go-chaff"
 )
 
@@ -41,6 +42,7 @@ func (h *PublishHandler) handleV1Apha1Request(w http.ResponseWriter, r *http.Req
 			status:      code,
 			pubResponse: &verifyapi.PublishResponse{ErrorMessage: message}, // will be down-converted in ServeHTTP
 			metric:      "publish-bad-json", count: 1}
+
 	}
 
 	// Upconvert the exposure key records.
@@ -75,6 +77,14 @@ func (h *PublishHandler) HandleV1Alpha1() http.Handler {
 		http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 			response := h.handleV1Apha1Request(w, r)
 
+			ctx := r.Context()
+			metrics := h.serverenv.MetricsExporter(ctx)
+
+			if err := response.padResponse(h.config); err != nil {
+				metrics.WriteInt("padding-failed", true, 1)
+				logging.FromContext(ctx).Errorw("failed to padd response", "error", err)
+			}
+
 			// Downgrade the v1 response to a v1alpha1 response.
 			alpha1Response := v1alpha1.PublishResponse{
 				RevisionToken:     response.pubResponse.RevisionToken,
@@ -84,8 +94,6 @@ func (h *PublishHandler) HandleV1Alpha1() http.Handler {
 			}
 
 			if response.metric != "" {
-				ctx := r.Context()
-				metrics := h.serverenv.MetricsExporter(ctx)
 				metrics.WriteInt(response.metric, true, response.count)
 			}
 
diff --git a/pkg/keys/config.go b/pkg/keys/config.go
index 57d379b..1d4f46a 100644
--- a/pkg/keys/config.go
+++ b/pkg/keys/config.go
@@ -22,7 +22,7 @@ const (
 	KeyManagerTypeAzureKeyVault  KeyManagerType = "AZURE_KEY_VAULT"
 	KeyManagerTypeGoogleCloudKMS KeyManagerType = "GOOGLE_CLOUD_KMS"
 	KeyManagerTypeHashiCorpVault KeyManagerType = "HASHICORP_VAULT"
-	KeyManagerTypeInMemory       KeyManagerType = "IN_MEMORY"
+	KeyManagerTypeFilesystem     KeyManagerType = "FILESYSTEM"
 )
 
 // Config defines configuration.
@@ -34,4 +34,7 @@ type Config struct {
 	// Adherence to this config setting is optional and based
 	// upon the key manager implementation and underlying capabilities.
 	CreateHSMKeys bool `env:"CREATE_HSM_KEYS, default=true"`
+
+	// FilesystemRoot is the root path where keys are managed on the filesystem.
+	FilesystemRoot string `env:"KEY_FILESYSTEM_ROOT"`
 }
diff --git a/pkg/keys/filesystem.go b/pkg/keys/filesystem.go
new file mode 100644
index 0000000..adc894c
--- /dev/null
+++ b/pkg/keys/filesystem.go
@@ -0,0 +1,418 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package keys
+
+import (
+	"bytes"
+	"context"
+	"crypto"
+	"crypto/aes"
+	"crypto/cipher"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+var _ EncryptionKeyManager = (*Filesystem)(nil)
+var _ KeyManager = (*Filesystem)(nil)
+var _ SigningKeyManager = (*Filesystem)(nil)
+
+// Filesystem is a key manager that uses the filesystem to store and retrieve
+// keys. It should only be used for local development and testing.
+type Filesystem struct {
+	root string
+	mu   sync.RWMutex
+}
+
+// NewFilesystem creates a new KeyManager backed by the local filesystem. It
+// should only be used for development and testing.
+//
+// If root is provided and does not exist, it will be created. If root is a
+// relative path, it's relative to where the process is currently executing. If
+// root is not supplied, all data is dumped in the current working directory.
+//
+// In general, root should either be a hardcoded path like $(pwd)/local or a
+// temporary directory like os.TempDir().
+func NewFilesystem(ctx context.Context, root string) (*Filesystem, error) {
+	if root != "" {
+		if err := os.MkdirAll(root, 0700); err != nil {
+			return nil, err
+		}
+	}
+
+	return &Filesystem{
+		root: root,
+	}, nil
+}
+
+// NewSigner creates a new signer from the given key. If the key does not exist,
+// it returns an error. If the key is not a signing key, it returns an error.
+func (k *Filesystem) NewSigner(ctx context.Context, keyID string) (crypto.Signer, error) {
+	k.mu.RLock()
+	defer k.mu.RUnlock()
+
+	pth := filepath.Join(k.root, keyID)
+	b, err := ioutil.ReadFile(pth)
+	if err != nil {
+		return nil, fmt.Errorf("failed to read signing key: %w", err)
+	}
+
+	pk, err := x509.ParseECPrivateKey(b)
+	if err != nil {
+		return nil, fmt.Errorf("failed to parse signing key: %w", err)
+	}
+
+	return pk, nil
+}
+
+// Encrypt encrypts the given plaintext and aad with the key. If the key does
+// not exist, it returns an error.
+func (k *Filesystem) Encrypt(ctx context.Context, keyID string, plaintext []byte, aad []byte) ([]byte, error) {
+	k.mu.RLock()
+	defer k.mu.RUnlock()
+
+	// Find the most recent DEK - that's what we'll use for encryption
+	pth := filepath.Join(k.root, keyID)
+	infos, err := ioutil.ReadDir(pth)
+	if err != nil {
+		return nil, fmt.Errorf("failed to list keys: %w", err)
+	}
+	if len(infos) < 1 {
+		return nil, fmt.Errorf("there are no key versions")
+	}
+	var latest os.FileInfo
+	for _, info := range infos {
+		if info.Name() == "metadata" {
+			continue
+		}
+		if latest == nil {
+			latest = info
+			continue
+		}
+		if info.Name() > latest.Name() {
+			latest = info
+		}
+	}
+	if latest == nil {
+		return nil, fmt.Errorf("key %q does not exist", keyID)
+	}
+
+	latestPath := filepath.Join(pth, latest.Name())
+	dek, err := ioutil.ReadFile(latestPath)
+	if err != nil {
+		return nil, fmt.Errorf("failed to read encryption key: %w", err)
+	}
+
+	block, err := aes.NewCipher(dek)
+	if err != nil {
+		return nil, fmt.Errorf("bad cipher block: %w", err)
+	}
+	aesgcm, err := cipher.NewGCM(block)
+	if err != nil {
+		return nil, fmt.Errorf("failed to wrap cipher block: %w", err)
+	}
+	nonce := make([]byte, aesgcm.NonceSize())
+	if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
+		return nil, fmt.Errorf("failed to generate nonce: %w", err)
+	}
+	ciphertext := aesgcm.Seal(nonce, nonce, plaintext, aad)
+
+	// Append the keyID to the ciphertext so we know which key to use to decrypt.
+	id := []byte(latest.Name() + ":")
+	ciphertext = append(id, ciphertext...)
+
+	return ciphertext, nil
+}
+
+// Decrypt decrypts the ciphertext. It returns an error if decryption fails or
+// if the key does not exist.
+func (k *Filesystem) Decrypt(ctx context.Context, keyID string, ciphertext []byte, aad []byte) ([]byte, error) {
+	k.mu.RLock()
+	defer k.mu.RUnlock()
+
+	// Figure out which DEK to use
+	parts := bytes.SplitN(ciphertext, []byte(":"), 2)
+	if len(parts) < 2 {
+		return nil, fmt.Errorf("invalid ciphertext: missing version")
+	}
+	version, ciphertext := parts[0], parts[1]
+
+	versionPath := filepath.Join(k.root, keyID, string(version))
+	dek, err := ioutil.ReadFile(versionPath)
+	if err != nil {
+		return nil, fmt.Errorf("failed to read encryption key: %w", err)
+	}
+
+	block, err := aes.NewCipher(dek)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create cipher from dek: %w", err)
+	}
+
+	aesgcm, err := cipher.NewGCM(block)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create gcm from dek: %w", err)
+	}
+
+	size := aesgcm.NonceSize()
+	if len(ciphertext) < size {
+		return nil, fmt.Errorf("malformed ciphertext")
+	}
+	nonce, ciphertextPortion := ciphertext[:size], ciphertext[size:]
+
+	plaintext, err := aesgcm.Open(nil, nonce, ciphertextPortion, aad)
+	if err != nil {
+		return nil, fmt.Errorf("failed to decrypt ciphertext with dek: %w", err)
+	}
+
+	return plaintext, nil
+}
+
+// SigningKeyVersions lists all the versions for the given parent. If the
+// provided parent is not a signing key, it returns an error.
+func (k *Filesystem) SigningKeyVersions(ctx context.Context, parent string) ([]SigningKeyVersion, error) {
+	k.mu.RLock()
+	defer k.mu.RUnlock()
+
+	metadata, err := k.metadataForKey(parent)
+	if err != nil {
+		return nil, fmt.Errorf("failed to list signing keys: %w", err)
+	}
+	if metadata.KeyType != "signing" {
+		return nil, fmt.Errorf("failed to list signing keys: key is not a signing key type")
+	}
+
+	pth := filepath.Join(k.root, parent)
+	var versions []SigningKeyVersion
+	if err := filepath.Walk(pth, func(curr string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+
+		if info.IsDir() {
+			return nil
+		}
+
+		if info.Name() == "metadata" {
+			return nil
+		}
+
+		b, err := ioutil.ReadFile(curr)
+		if err != nil {
+			return err
+		}
+
+		pk, err := x509.ParseECPrivateKey(b)
+		if err != nil {
+			return fmt.Errorf("failed to parse signing key: %w", err)
+		}
+
+		versions = append(versions, &filesystemSigningKey{
+			name:    strings.TrimPrefix(curr, k.root),
+			created: info.ModTime(),
+			pk:      pk,
+		})
+
+		return nil
+	}); err != nil {
+		return nil, fmt.Errorf("failed to list keys: failed to walk: %w", err)
+	}
+
+	// Sort keys descending so the newest is first
+	sort.Slice(versions, func(i, j int) bool {
+		a := versions[i].(*filesystemSigningKey).name
+		b := versions[j].(*filesystemSigningKey).name
+		return a > b
+	})
+
+	return versions, nil
+}
+
+// CreateSigningKey creates a signing key. For this implementation, that means
+// it creates a folder on disk (but no keys inside). If the folder already
+// exists, it returns its name.
+func (k *Filesystem) CreateSigningKey(_ context.Context, parent, name string) (string, error) {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+
+	pth := filepath.Join(k.root, parent, name)
+	if err := os.MkdirAll(pth, 0700); err != nil {
+		return "", fmt.Errorf("failed to create directory for key: %w", err)
+	}
+
+	metadataPath := filepath.Join(pth, "metadata")
+	b, err := ioutil.ReadFile(metadataPath)
+	if err != nil && !os.IsNotExist(err) {
+		return "", fmt.Errorf("failed to read metadata file: %w", err)
+	}
+	if len(b) > 0 {
+		var metadata filesystemKeyInfo
+		if err := json.Unmarshal(b, &metadata); err != nil {
+			return "", fmt.Errorf("failed to parse metadata: %w", err)
+		}
+		if metadata.KeyType != "signing" {
+			return "", fmt.Errorf("found key, but is not signing type")
+		}
+		return strings.TrimPrefix(pth, k.root), nil
+	}
+
+	// If we got this far, the metadata file does not exist, so create it.
+	metadata := &filesystemKeyInfo{KeyType: "signing"}
+	b, err = json.Marshal(metadata)
+	if err != nil {
+		return "", fmt.Errorf("failed to generate metadata file: %w", err)
+	}
+	if err := ioutil.WriteFile(metadataPath, b, 0600); err != nil {
+		return "", fmt.Errorf("failed to create metadata file: %w", err)
+	}
+	return strings.TrimPrefix(pth, k.root), nil
+}
+
+// CreateEncryptionKey creates an encryption key. For this implementation, that
+// means it creates a folder on disk (but no keys inside). If the folder already
+// exists, it returns its name.
+func (k *Filesystem) CreateEncryptionKey(_ context.Context, parent, name string) (string, error) {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+
+	pth := filepath.Join(k.root, parent, name)
+	if err := os.MkdirAll(pth, 0700); err != nil {
+		return "", fmt.Errorf("failed to create directory for key: %w", err)
+	}
+
+	metadataPath := filepath.Join(pth, "metadata")
+	b, err := ioutil.ReadFile(metadataPath)
+	if err != nil && !os.IsNotExist(err) {
+		return "", fmt.Errorf("failed to read metadata file: %w", err)
+	}
+	if len(b) > 0 {
+		var metadata filesystemKeyInfo
+		if err := json.Unmarshal(b, &metadata); err != nil {
+			return "", fmt.Errorf("failed to parse metadata: %w", err)
+		}
+		if metadata.KeyType != "encryption" {
+			return "", fmt.Errorf("found key, but is not encryption type")
+		}
+		return strings.TrimPrefix(pth, k.root), nil
+	}
+
+	// If we got this far, the metadata file does not exist, so create it.
+	metadata := &filesystemKeyInfo{KeyType: "encryption"}
+	b, err = json.Marshal(metadata)
+	if err != nil {
+		return "", fmt.Errorf("failed to generate metadata file: %w", err)
+	}
+	if err := ioutil.WriteFile(metadataPath, b, 0600); err != nil {
+		return "", fmt.Errorf("failed to create metadata file: %w", err)
+	}
+	return strings.TrimPrefix(pth, k.root), nil
+}
+
+// CreateKeyVersion creates a new key version for the parent. If the parent is a
+// signing key, it creates a signing key. If the parent is an encryption key, it
+// creates an encryption key. If the parent does not exist, it returns an error.
+func (k *Filesystem) CreateKeyVersion(_ context.Context, parent string) (string, error) {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+
+	metadata, err := k.metadataForKey(parent)
+	if err != nil {
+		return "", fmt.Errorf("failed to create key version: %w", err)
+	}
+
+	switch t := metadata.KeyType; t {
+	case "signing":
+		pk, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+		if err != nil {
+			return "", fmt.Errorf("failed to generate signing key: %w", err)
+		}
+		b, err := x509.MarshalECPrivateKey(pk)
+		if err != nil {
+			return "", fmt.Errorf("failed to marshal signing key: %w", err)
+		}
+		pth := filepath.Join(k.root, parent, strconv.FormatInt(time.Now().UnixNano(), 10))
+		if err := ioutil.WriteFile(pth, b, 0600); err != nil {
+			return "", fmt.Errorf("failed to write signing key to disk: %w", err)
+		}
+		return strings.TrimPrefix(pth, k.root), nil
+	case "encryption":
+		ek := make([]byte, 32)
+		if _, err := io.ReadFull(rand.Reader, ek); err != nil {
+			return "", fmt.Errorf("failed to generate encryption key: %w", err)
+		}
+		pth := filepath.Join(k.root, parent, strconv.FormatInt(time.Now().UnixNano(), 10))
+		if err := ioutil.WriteFile(pth, ek, 0600); err != nil {
+			return "", fmt.Errorf("failed to write encryption key to disk: %w", err)
+		}
+		return strings.TrimPrefix(pth, k.root), nil
+	default:
+		return "", fmt.Errorf("unknown key type %q", t)
+	}
+}
+
+// DestroyKeyVersion destroys the given key version. It does nothing if the key
+// does not exist.
+func (k *Filesystem) DestroyKeyVersion(_ context.Context, id string) error {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+
+	pth := filepath.Join(k.root, id)
+	if err := os.Remove(pth); err != nil && !os.IsNotExist(err) {
+		return fmt.Errorf("failed to destroy key version: %w", err)
+	}
+	return nil
+}
+
+type filesystemSigningKey struct {
+	name    string
+	created time.Time
+	pk      *ecdsa.PrivateKey
+}
+
+func (k *filesystemSigningKey) KeyID() string          { return k.name }
+func (k *filesystemSigningKey) CreatedAt() time.Time   { return k.created }
+func (k *filesystemSigningKey) DestroyedAt() time.Time { return time.Time{} }
+func (k *filesystemSigningKey) Signer(_ context.Context) (crypto.Signer, error) {
+	return k.pk, nil
+}
+
+type filesystemKeyInfo struct {
+	KeyType string `json:"t"`
+}
+
+func (k *Filesystem) metadataForKey(parent string) (*filesystemKeyInfo, error) {
+	metadataPath := filepath.Join(k.root, parent, "metadata")
+	b, err := ioutil.ReadFile(metadataPath)
+	if err != nil {
+		return nil, fmt.Errorf("failed to open metadata (does the key exist?): %w", err)
+	}
+
+	var metadata filesystemKeyInfo
+	if err := json.Unmarshal(b, &metadata); err != nil {
+		return nil, fmt.Errorf("failed to parse metadata: %w", err)
+	}
+	return &metadata, nil
+}
diff --git a/pkg/keys/google_cloud_kms.go b/pkg/keys/google_cloud_kms.go
index cfb1590..e0e33b6 100644
--- a/pkg/keys/google_cloud_kms.go
+++ b/pkg/keys/google_cloud_kms.go
@@ -46,18 +46,9 @@ type CloudKMSSigningKeyVersion struct {
 	keyManager  *GoogleCloudKMS
 }
 
-func (k *CloudKMSSigningKeyVersion) KeyID() string {
-	return k.keyID
-}
-
-func (k *CloudKMSSigningKeyVersion) CreatedAt() time.Time {
-	return k.createdAt
-}
-
-func (k *CloudKMSSigningKeyVersion) DestroyedAt() time.Time {
-	return k.destroyedAt
-}
-
+func (k *CloudKMSSigningKeyVersion) KeyID() string          { return k.keyID }
+func (k *CloudKMSSigningKeyVersion) CreatedAt() time.Time   { return k.createdAt }
+func (k *CloudKMSSigningKeyVersion) DestroyedAt() time.Time { return k.destroyedAt }
 func (k *CloudKMSSigningKeyVersion) Signer(ctx context.Context) (crypto.Signer, error) {
 	return k.keyManager.NewSigner(ctx, k.keyID)
 }
diff --git a/pkg/keys/hashicorp_vault.go b/pkg/keys/hashicorp_vault.go
index 117518c..9bfa74f 100644
--- a/pkg/keys/hashicorp_vault.go
+++ b/pkg/keys/hashicorp_vault.go
@@ -92,18 +92,9 @@ type vaultKeyVersion struct {
 	publicKey crypto.PublicKey
 }
 
-func (v *vaultKeyVersion) KeyID() string {
-	return fmt.Sprintf("%s/%d", v.name, v.version)
-}
-
-func (v *vaultKeyVersion) CreatedAt() time.Time {
-	return v.createdAt.UTC()
-}
-
-func (v *vaultKeyVersion) DestroyedAt() time.Time {
-	return time.Time{}
-}
-
+func (v *vaultKeyVersion) KeyID() string          { return fmt.Sprintf("%s/%d", v.name, v.version) }
+func (v *vaultKeyVersion) CreatedAt() time.Time   { return v.createdAt.UTC() }
+func (v *vaultKeyVersion) DestroyedAt() time.Time { return time.Time{} }
 func (v *vaultKeyVersion) Signer(ctx context.Context) (crypto.Signer, error) {
 	return &HashiCorpVaultSigner{
 		client:    v.client,
diff --git a/pkg/keys/in_memory.go b/pkg/keys/in_memory.go
deleted file mode 100644
index 73632c7..0000000
--- a/pkg/keys/in_memory.go
+++ /dev/null
@@ -1,193 +0,0 @@
-// Copyright 2020 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package keys
-
-import (
-	"context"
-	"crypto"
-	"crypto/aes"
-	"crypto/cipher"
-	"crypto/ecdsa"
-	"crypto/elliptic"
-	"crypto/rand"
-	"fmt"
-	"io"
-	"sync"
-)
-
-var _ KeyManager = (*InMemory)(nil)
-var _ SigningKeyAdder = (*InMemory)(nil)
-var _ EncryptionKeyAdder = (*InMemory)(nil)
-
-// InMemory is useful for testing. Do NOT use in a running system as all
-// keys are only kept in memory and will be lost across server reboots.
-type InMemory struct {
-	mu          sync.RWMutex
-	signingKeys map[string]*ecdsa.PrivateKey
-	cryptoKeys  map[string][]byte
-}
-
-// NewInMemory creates a new, local, in memory KeyManager.
-func NewInMemory(ctx context.Context) (*InMemory, error) {
-	return &InMemory{
-		signingKeys: make(map[string]*ecdsa.PrivateKey),
-		cryptoKeys:  make(map[string][]byte),
-	}, nil
-}
-
-// CreateSigningKey generates a new ECDSA P256 Signing Key identified by
-// the provided keyID
-func (k *InMemory) CreateSigningKey(ctx context.Context, parent, name string) (string, error) {
-	k.mu.Lock()
-	defer k.mu.Unlock()
-
-	keyID := fmt.Sprintf("%s/%s", parent, name)
-	if _, ok := k.signingKeys[keyID]; ok {
-		return "", fmt.Errorf("key already exists: %v", keyID)
-	}
-
-	pk, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
-	if err != nil {
-		return "", fmt.Errorf("unable to generate private key: %w", err)
-	}
-
-	k.signingKeys[keyID] = pk
-	return keyID, nil
-}
-
-// AddSigningKey adds a new ECDSA P256 Signing Key identified by the provided
-// keyID.
-func (k *InMemory) AddSigningKey(keyID string, pk *ecdsa.PrivateKey) error {
-	k.mu.Lock()
-	defer k.mu.Unlock()
-
-	if _, ok := k.signingKeys[keyID]; ok {
-		return fmt.Errorf("key already exists: %v", keyID)
-	}
-
-	k.signingKeys[keyID] = pk
-	return nil
-}
-
-// CreateEncryptionKey generates and stores new encryption key identified by the
-// provided keyID.
-func (k *InMemory) CreateEncryptionKey(keyID string) ([]byte, error) {
-	k.mu.Lock()
-	defer k.mu.Unlock()
-
-	if _, ok := k.cryptoKeys[keyID]; ok {
-		return nil, fmt.Errorf("key already exists: %v", keyID)
-	}
-
-	key := make([]byte, 32)
-	if _, err := io.ReadFull(rand.Reader, key); err != nil {
-		return nil, fmt.Errorf("failed to read random bytes: %w", err)
-	}
-
-	k.cryptoKeys[keyID] = key
-	return key, nil
-}
-
-// AddEncryptionKey stores the key on the system.
-func (k *InMemory) AddEncryptionKey(keyID string, key []byte) error {
-	k.mu.Lock()
-	defer k.mu.Unlock()
-
-	if _, ok := k.cryptoKeys[keyID]; ok {
-		return fmt.Errorf("key already exists: %v", keyID)
-	}
-
-	k.cryptoKeys[keyID] = key
-	return nil
-}
-
-func (k *InMemory) NewSigner(ctx context.Context, keyID string) (crypto.Signer, error) {
-	var pk *ecdsa.PrivateKey
-	{
-		k.mu.RLock()
-		defer k.mu.RUnlock()
-
-		if k, ok := k.signingKeys[keyID]; !ok {
-			return nil, fmt.Errorf("key not found")
-		} else {
-			pk = k
-		}
-	}
-
-	return pk, nil
-}
-
-func (k *InMemory) getDEK(keyID string) ([]byte, error) {
-	k.mu.RLock()
-	defer k.mu.RUnlock()
-
-	if dek, ok := k.cryptoKeys[keyID]; ok {
-		return dek, nil
-	}
-	return nil, fmt.Errorf("key not found")
-}
-
-func (k *InMemory) Encrypt(ctx context.Context, keyID string, plaintext []byte, aad []byte) ([]byte, error) {
-	dek, err := k.getDEK(keyID)
-	if err != nil {
-		return nil, err
-	}
-
-	block, err := aes.NewCipher(dek)
-	if err != nil {
-		return nil, fmt.Errorf("bad cipher block: %w", err)
-	}
-	aesgcm, err := cipher.NewGCM(block)
-	if err != nil {
-		return nil, fmt.Errorf("failed to wrap cipher block: %w", err)
-	}
-	nonce := make([]byte, aesgcm.NonceSize())
-	if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
-		return nil, fmt.Errorf("failed to generate nonce: %w", err)
-	}
-	ciphertext := aesgcm.Seal(nonce, nonce, plaintext, aad)
-
-	return ciphertext, nil
-}
-
-func (k *InMemory) Decrypt(ctx context.Context, keyID string, ciphertext []byte, aad []byte) ([]byte, error) {
-	dek, err := k.getDEK(keyID)
-	if err != nil {
-		return nil, err
-	}
-
-	block, err := aes.NewCipher(dek)
-	if err != nil {
-		return nil, fmt.Errorf("failed to create cipher from dek: %w", err)
-	}
-
-	aesgcm, err := cipher.NewGCM(block)
-	if err != nil {
-		return nil, fmt.Errorf("failed to create gcm from dek: %w", err)
-	}
-
-	size := aesgcm.NonceSize()
-	if len(ciphertext) < size {
-		return nil, fmt.Errorf("malformed ciphertext")
-	}
-	nonce, ciphertextPortion := ciphertext[:size], ciphertext[size:]
-
-	plaintext, err := aesgcm.Open(nil, nonce, ciphertextPortion, aad)
-	if err != nil {
-		return nil, fmt.Errorf("failed to decrypt ciphertext with dek: %w", err)
-	}
-
-	return plaintext, nil
-}
diff --git a/pkg/keys/keys.go b/pkg/keys/keys.go
index db44c46..f9b4243 100644
--- a/pkg/keys/keys.go
+++ b/pkg/keys/keys.go
@@ -23,7 +23,6 @@ package keys
 import (
 	"context"
 	"crypto"
-	"crypto/ecdsa"
 	"crypto/x509"
 	"encoding/pem"
 	"fmt"
@@ -59,11 +58,6 @@ type KeyVersionCreator interface {
 	CreateKeyVersion(ctx context.Context, parent string) (string, error)
 }
 
-// EncryptionKeyAdder supports creating encryption keys.
-type EncryptionKeyAdder interface {
-	AddEncryptionKey(string, []byte) error
-}
-
 // KeyVersionDestroyer supports destroying a key version.
 type KeyVersionDestroyer interface {
 	// DestroyKeyVersion destroys the given key version, if it exists. If the
@@ -71,11 +65,6 @@ type KeyVersionDestroyer interface {
 	DestroyKeyVersion(ctx context.Context, id string) error
 }
 
-// SigningKeyAdder supports creating signing keys.
-type SigningKeyAdder interface {
-	AddSigningKey(string, *ecdsa.PrivateKey) error
-}
-
 // SigningKeyVersion represents the necessary details that this application needs
 // to manage signing keys in an external KMS.
 type SigningKeyVersion interface {
@@ -100,6 +89,17 @@ type SigningKeyManager interface {
 	KeyVersionDestroyer
 }
 
+// EncryptionKeyManager supports extended management of encryption keys,
+// versions, and rotation.
+type EncryptionKeyManager interface {
+	// CreateEncryptionKey creates a new encryption key in the given parent,
+	// returning the id. If the key already exists, it returns the key's id.
+	CreateEncryptionKey(ctx context.Context, parent, name string) (string, error)
+
+	KeyVersionCreator
+	KeyVersionDestroyer
+}
+
 // KeyManagerFor returns the appropriate key manager for the given type.
 func KeyManagerFor(ctx context.Context, config *Config) (KeyManager, error) {
 	typ := config.KeyManagerType
@@ -112,8 +112,8 @@ func KeyManagerFor(ctx context.Context, config *Config) (KeyManager, error) {
 		return NewGoogleCloudKMS(ctx, config)
 	case KeyManagerTypeHashiCorpVault:
 		return NewHashiCorpVault(ctx)
-	case KeyManagerTypeInMemory:
-		return NewInMemory(ctx)
+	case KeyManagerTypeFilesystem:
+		return NewFilesystem(ctx, config.FilesystemRoot)
 	}
 
 	return nil, fmt.Errorf("unknown key manager type: %v", typ)
diff --git a/pkg/observability/resource.go b/pkg/observability/resource.go
index 4da3f1d..d13931d 100644
--- a/pkg/observability/resource.go
+++ b/pkg/observability/resource.go
@@ -16,6 +16,7 @@ package observability
 
 import (
 	"context"
+	"strings"
 
 	"github.com/google/exposure-notifications-server/pkg/logging"
 
@@ -39,7 +40,7 @@ type stackdriverMonitoredResource struct {
 // generating.
 //
 // NOTE: This code is focused on support GCP Cloud Run Managed. If you are
-// runniing in a different environment, you may see weird results.
+// running in a different environment, you may see weird results.
 func NewStackdriverMonitoredResource(ctx context.Context, c *StackdriverConfig) monitoredresource.Interface {
 	logger := logging.FromContext(ctx).Named("stackdriver")
 
@@ -72,6 +73,14 @@ func NewStackdriverMonitoredResource(ctx context.Context, c *StackdriverConfig)
 	}
 	labels["location"] = region
 
+	// Metadata often returns the following: "projects/111111111111/regions/us-central1"
+	pieces := strings.Split(region, "/")
+	if len(pieces) == 4 {
+		labels["location"] = pieces[3]
+	} else {
+		logger.Errorw("region did not match expected format", "region", region)
+	}
+
 	labels["namespace"] = c.Namespace
 
 	filteredLabels := removeUnusedLabels(resource, labels)
diff --git a/pkg/observability/stackdriver.go b/pkg/observability/stackdriver.go
index de33cf2..4974330 100644
--- a/pkg/observability/stackdriver.go
+++ b/pkg/observability/stackdriver.go
@@ -27,7 +27,11 @@ import (
 	"go.opencensus.io/trace"
 )
 
-var _ Exporter = (*stackdriverExporter)(nil)
+var (
+	_ Exporter = (*stackdriverExporter)(nil)
+
+	SDExportFrequency = time.Minute * 2
+)
 
 type stackdriverExporter struct {
 	exporter *stackdriver.Exporter
@@ -49,7 +53,7 @@ func NewStackdriver(ctx context.Context, config *StackdriverConfig) (Exporter, e
 	exporter, err := stackdriver.NewExporter(stackdriver.Options{
 		Context:                 ctx,
 		ProjectID:               projectID,
-		ReportingInterval:       time.Minute, // stackdriver export interval minimum
+		ReportingInterval:       SDExportFrequency,
 		MonitoredResource:       monitoredResource,
 		DefaultMonitoringLabels: &stackdriver.Labels{},
 		OnError: func(err error) {
@@ -65,6 +69,10 @@ func NewStackdriver(ctx context.Context, config *StackdriverConfig) (Exporter, e
 
 // StartExporter starts the exporter.
 func (e *stackdriverExporter) StartExporter() error {
+	if err := registerViews(); err != nil {
+		return fmt.Errorf("failed to register views: %w", err)
+	}
+
 	if err := e.exporter.StartMetricsExporter(); err != nil {
 		return fmt.Errorf("failed to start stackdriver exporter: %w", err)
 	}
@@ -73,7 +81,9 @@ func (e *stackdriverExporter) StartExporter() error {
 		DefaultSampler: trace.ProbabilitySampler(e.config.SampleRate),
 	})
 	trace.RegisterExporter(e.exporter)
+
 	view.RegisterExporter(e.exporter)
+	view.SetReportingPeriod(SDExportFrequency)
 
 	return nil
 }
diff --git a/pkg/server/server.go b/pkg/server/server.go
index 22d57ec..72ea365 100644
--- a/pkg/server/server.go
+++ b/pkg/server/server.go
@@ -115,8 +115,9 @@ func (s *Server) ServeHTTP(ctx context.Context, srv *http.Server) error {
 func (s *Server) ServeHTTPHandler(ctx context.Context, handler http.Handler) error {
 	return s.ServeHTTP(ctx, &http.Server{
 		Handler: &ochttp.Handler{
-			Handler:     handler,
-			Propagation: &tracecontext.HTTPFormat{},
+			Handler:          handler,
+			IsPublicEndpoint: true,
+			Propagation:      &tracecontext.HTTPFormat{},
 		},
 	})
 }
diff --git a/scripts/terraform.sh b/scripts/terraform.sh
new file mode 100755
index 0000000..81cbff5
--- /dev/null
+++ b/scripts/terraform.sh
@@ -0,0 +1,121 @@
+#!/usr/bin/env bash
+
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# terrafrom.sh script is used for running e2e test in prow, especially smoke test terrafrom conigurations
+
+set -eEuo pipefail
+
+PROGNAME="$(basename $0)"
+ROOT="$(cd "$(dirname "$0")/.." &>/dev/null; pwd -P)"
+
+if [[ -z "${PROJECT_ID:-}" ]]; then
+  echo "✋ PROJECT_ID must be set"
+fi
+
+if [[ -z "${GOOGLE_APPLICATION_CREDENTIALS:-}" ]]; then
+  echo "This is local development, authenticate using gcloud"
+  echo "gcloud auth login"
+  echo "gcloud auth application-default login"
+  echo "gcloud auth application-default set-quota-project "${PROJECT_ID}""
+else
+  echo "GOOGLE_APPLCIATION_CREDENTIALS defined, use it for authentication."
+  echo "For local development, run 'unset GOOGLE_APPLCIATION_CREDENTIALS', "
+  echo "then rerun this script"
+fi
+
+function deploy() {
+  pushd "${ROOT}/terraform-e2e" > /dev/null
+
+  # Preparing for deployment
+  echo "project = \"${PROJECT_ID}\"" > ./terraform.tfvars
+  # Don't fail if it already exists
+  gsutil mb -p ${PROJECT_ID} gs://${PROJECT_ID}-tf-state 2>/dev/null || true
+  cat <<EOF > "${ROOT}/terraform-e2e/state.tf"
+terraform {
+  backend "gcs" {
+    bucket = "${PROJECT_ID}-tf-state"
+  }
+}
+EOF
+
+  # google_app_engine_application.app is global, cannot be deleted once created,
+  # if this project already has it created then terraform apply will fail,
+  # importing it can solve this problem
+  terraform init --upgrade
+  terraform get --update
+  terraform import module.en.google_app_engine_application.app ${PROJECT_ID} || true
+
+  # Terraform deployment might fail intermittently with certain cloud run 
+  # services not up, retry to make it more resilient
+  local failed=1
+  for i in 1 2 3; do
+    if [[ "${failed}" == "0" ]]; then
+      break
+    fi
+    if terraform apply -auto-approve; then
+      failed=0
+    fi
+  done
+  popd > /dev/null
+  return $failed
+}
+
+function destroy() {
+  pushd "${ROOT}/terraform-e2e" > /dev/null
+  terraform get --update
+  local db_inst_name
+  db_inst_name="$(terraform output -json 'en' | jq '. | .db_inst_name' | tr -d \")"
+  # DB often failed to be destroyed by terraform due to "used by other process",
+  # so delete it manually
+  gcloud sql instances delete ${db_inst_name} -q --project=${PROJECT_ID} || true
+  # Clean up states after manual DB delete
+  terraform state rm module.en.google_sql_user.user || true
+  terraform state rm module.en.google_sql_ssl_cert.db-cert || true
+  terraform destroy -auto-approve
+  popd > /dev/null
+}
+
+function smoke() {
+  # Best effort destroy before applying
+  destroy || true
+  deploy
+  trap "destroy || true" EXIT
+}
+
+# help prints help.
+function help() {
+  echo 1>&2 "Usage: ${PROGNAME} <command>"
+  echo 1>&2 ""
+  echo 1>&2 "Commands:"
+  echo 1>&2 "  deploy       deploy server"
+  echo 1>&2 "  destroy      destroy server"
+  echo 1>&2 "  smoke        deploy then destroy server"
+}
+
+case "${1:-}" in
+  "" | "help" | "-h" | "--help" )
+    help
+    ;;
+
+  "deploy" | "destroy" | "smoke" )
+    $1
+    ;;
+
+  *)
+    help
+    exit 1
+    ;;
+esac
diff --git a/terraform/database.tf b/terraform/database.tf
index 8c42c4f..8c38d36 100644
--- a/terraform/database.tf
+++ b/terraform/database.tf
@@ -180,6 +180,10 @@ output "db_user" {
   value = google_sql_user.user.name
 }
 
+output "db_inst_name" {
+  value = google_sql_database_instance.db-inst.name
+}
+
 output "db_password" {
   value = google_secret_manager_secret_version.db-secret-version["password"].name
 }
diff --git a/tools/admin-console/templates/healthauthority.html b/tools/admin-console/templates/healthauthority.html
index 53f62a2..94e73f5 100644
--- a/tools/admin-console/templates/healthauthority.html
+++ b/tools/admin-console/templates/healthauthority.html
@@ -33,7 +33,7 @@ health authority.</p>
     <div class="col-sm-6">
       <input type="text" id="Audience" name="Audience" size="50" value="{{.ha.Audience}}">
       <small id="AudienceHelpBlock" class="form-text text-muted">
-        The valud for audience is defined by the operator of this server. It should be
+        The value for audience is defined by the operator of this server. It should be
         communicated to the health authority for inclusion in the verification
         certificates. This is the 'aud' field of the certificate JWT.
         <strong>Once set, this field should not be edited.</strong></small>
diff --git a/tools/b64decode/main.go b/tools/b64decode/main.go
new file mode 100644
index 0000000..0f01add
--- /dev/null
+++ b/tools/b64decode/main.go
@@ -0,0 +1,40 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package main
+
+import (
+	"encoding/hex"
+	"log"
+	"os"
+
+	"github.com/google/exposure-notifications-server/pkg/base64util"
+)
+
+// Small to base64 decode data and print some info
+func main() {
+	if len(os.Args) != 2 {
+		log.Fatal("requires 1 argument which will be interpreted as a base64 encoded byte array")
+	}
+
+	encoded := os.Args[1]
+	bytes, err := base64util.DecodeString(encoded)
+	if err != nil {
+		log.Fatalf("base64util.DecodeString: %v : %v", encoded, err)
+	}
+	log.Printf("Number of bytes: %v", len(bytes))
+
+	hex := hex.EncodeToString(bytes)
+	log.Printf("As HEX: %v", hex)
+}


Test patch:
diff --git a/internal/keyrotation/rotate_test.go b/internal/keyrotation/rotate_test.go
index 41b4147..d9e0ecd 100644
--- a/internal/keyrotation/rotate_test.go
+++ b/internal/keyrotation/rotate_test.go
@@ -35,11 +35,9 @@ func TestRotateKeys(t *testing.T) {
 	t.Parallel()
 	ctx := context.Background()
 
-	kms, _ := keys.NewInMemory(context.Background())
-	keyID := "testKeyID"
-	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
-		t.Fatal(err)
-	}
+	kms := keys.TestKeyManager(t)
+	keyID := keys.TestEncryptionKey(t, kms)
+
 	config := &Config{
 		RevisionToken:      revision.Config{KeyID: keyID},
 		DeleteOldKeyPeriod: 14 * 24 * time.Hour, // two weeks
diff --git a/internal/keyrotation/server_test.go b/internal/keyrotation/server_test.go
index 3a36d5f..f243771 100644
--- a/internal/keyrotation/server_test.go
+++ b/internal/keyrotation/server_test.go
@@ -31,7 +31,7 @@ func TestNewRotationHandler(t *testing.T) {
 	ctx := context.Background()
 	testDB := database.NewTestDatabase(t)
 
-	kms, _ := keys.NewInMemory(context.Background())
+	kms := keys.TestKeyManager(t)
 
 	testCases := []struct {
 		name string
@@ -61,10 +61,8 @@ func TestNewRotationHandler(t *testing.T) {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
 
-			keyID := "test" + t.Name()
-			if _, err := kms.CreateEncryptionKey(keyID); err != nil {
-				t.Fatal(err)
-			}
+			keyID := keys.TestEncryptionKey(t, kms)
+
 			config := &Config{
 				RevisionToken: revision.Config{KeyID: keyID},
 			}
diff --git a/internal/publish/publish_test.go b/internal/publish/publish_test.go
index 5af0969..42d71ab 100644
--- a/internal/publish/publish_test.go
+++ b/internal/publish/publish_test.go
@@ -387,17 +387,13 @@ func TestPublishWithBypass(t *testing.T) {
 		}
 
 		ctx := context.Background()
+
 		// Database init for all modules that will be used.
 		testDB := coredb.NewTestDatabase(t)
-		// Make key manager
-		kms, err := keys.NewInMemory(ctx)
-		if err != nil {
-			t.Fatalf("can't make kms: %v", err)
-		}
-		keyID := "rev"
-		if _, err := kms.CreateEncryptionKey(keyID); err != nil {
-			t.Fatal(err)
-		}
+
+		kms := keys.TestKeyManager(t)
+		keyID := keys.TestEncryptionKey(t, kms)
+
 		tokenAAD := make([]byte, 16)
 		if _, err := rand.Read(tokenAAD); err != nil {
 			t.Fatalf("not enough entropy: %v", err)
@@ -434,6 +430,8 @@ func TestPublishWithBypass(t *testing.T) {
 				}
 				config.RevisionToken.AAD = tokenAAD
 				config.RevisionToken.KeyID = keyID
+				config.ResponsePaddingMinBytes = 100
+				config.ResponsePaddingRange = 100
 				env := serverenv.New(ctx,
 					serverenv.WithDatabase(testDB),
 					serverenv.WithAuthorizedAppProvider(aaProvider),
@@ -536,6 +534,9 @@ func TestPublishWithBypass(t *testing.T) {
 				if resp.StatusCode != tc.Code {
 					t.Fatalf("http response code want: %v, got %v.", tc.Code, resp.StatusCode)
 				}
+				if len(response.Padding) == 0 {
+					t.Errorf("padding is empty on response: %+v", response)
+				}
 
 				if ver == useV1 {
 					// The extended data validation only happens on verifyapi.
@@ -707,17 +708,13 @@ func TestKeyRevision(t *testing.T) {
 	}
 
 	ctx := context.Background()
+
 	// Database init for all modules that will be used.
 	testDB := coredb.NewTestDatabase(t)
-	// Make key manager
-	kms, err := keys.NewInMemory(ctx)
-	if err != nil {
-		t.Fatalf("can't make kms: %v", err)
-	}
-	keyID := "rev"
-	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
-		t.Fatal(err)
-	}
+
+	kms := keys.TestKeyManager(t)
+	keyID := keys.TestEncryptionKey(t, kms)
+
 	tokenAAD := make([]byte, 16)
 	if _, err := rand.Read(tokenAAD); err != nil {
 		t.Fatalf("not enough entropy: %v", err)
@@ -741,6 +738,8 @@ func TestKeyRevision(t *testing.T) {
 	config.MaxKeysOnPublish = 20
 	config.MaxSameStartIntervalKeys = 2
 	config.MaxIntervalAge = 14 * 24 * time.Hour
+	config.ResponsePaddingMinBytes = 100
+	config.ResponsePaddingRange = 100
 	aaProvider, err := authorizedapp.NewDatabaseProvider(ctx, testDB, config.AuthorizedAppConfig())
 	if err != nil {
 		t.Fatal(err)
diff --git a/internal/revision/database/revision_test.go b/internal/revision/database/revision_test.go
index 2e1d8a2..2c4bd46 100644
--- a/internal/revision/database/revision_test.go
+++ b/internal/revision/database/revision_test.go
@@ -35,14 +35,8 @@ func TestRevisionKey(t *testing.T) {
 	testDB := database.NewTestDatabase(t)
 	ctx := context.Background()
 
-	kms, err := keys.NewInMemory(ctx)
-	if err != nil {
-		t.Fatalf("unable to cerate in memory KMS")
-	}
-	keyID := "funkey"
-	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
-		t.Fatalf("unable to generate key: %v", err)
-	}
+	kms := keys.TestKeyManager(t)
+	keyID := keys.TestEncryptionKey(t, kms)
 
 	cfg := KMSConfig{keyID, kms}
 	revDB, err := New(testDB, &cfg)
@@ -71,14 +65,8 @@ func TestMultipleRevisionKeys(t *testing.T) {
 	testDB := database.NewTestDatabase(t)
 	ctx := context.Background()
 
-	kms, err := keys.NewInMemory(ctx)
-	if err != nil {
-		t.Fatalf("unable to cerate in memory KMS")
-	}
-	keyID := "funkey"
-	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
-		t.Fatalf("unable to generate key: %v", err)
-	}
+	kms := keys.TestKeyManager(t)
+	keyID := keys.TestEncryptionKey(t, kms)
 
 	cfg := KMSConfig{keyID, kms}
 	revDB, err := New(testDB, &cfg)
diff --git a/internal/revision/revision_test.go b/internal/revision/revision_test.go
index 3f2f238..a1d7790 100644
--- a/internal/revision/revision_test.go
+++ b/internal/revision/revision_test.go
@@ -16,9 +16,7 @@ package revision
 
 import (
 	"context"
-	"crypto/rand"
 	"fmt"
-	"io"
 	"testing"
 	"time"
 
@@ -151,18 +149,8 @@ func TestEncryptDecrypt(t *testing.T) {
 	testDB := database.NewTestDatabase(t)
 	ctx := context.Background()
 
-	kms, err := keys.NewInMemory(ctx)
-	if err != nil {
-		t.Fatalf("unable to cerate in memory KMS")
-	}
-	keyID := "skeleton"
-	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
-		t.Fatalf("unable to generate key: %v", err)
-	}
-	key := make([]byte, 32)
-	if _, err := io.ReadFull(rand.Reader, key); err != nil {
-		t.Fatalf("unable to generate AES key: %v", err)
-	}
+	kms := keys.TestKeyManager(t)
+	keyID := keys.TestEncryptionKey(t, kms)
 
 	cfg := revisiondb.KMSConfig{
 		WrapperKeyID: keyID,
diff --git a/internal/setup/setup_test.go b/internal/setup/setup_test.go
index 809bfcf..c1d41f7 100644
--- a/internal/setup/setup_test.go
+++ b/internal/setup/setup_test.go
@@ -61,7 +61,7 @@ func (t *testConfig) DatabaseConfig() *database.Config {
 
 func (t *testConfig) KeyManagerConfig() *keys.Config {
 	return &keys.Config{
-		KeyManagerType: keys.KeyManagerType("IN_MEMORY"),
+		KeyManagerType: keys.KeyManagerType("FILESYSTEM"),
 	}
 }
 
@@ -186,8 +186,8 @@ func TestSetupWith(t *testing.T) {
 			t.Errorf("expected key manager to exist")
 		}
 
-		if _, ok := km.(*keys.InMemory); !ok {
-			t.Errorf("expected %T to be InMemory", km)
+		if _, ok := km.(*keys.Filesystem); !ok {
+			t.Errorf("expected %T to be Filesystem", km)
 		}
 	})
 
diff --git a/pkg/keys/filesystem_test.go b/pkg/keys/filesystem_test.go
new file mode 100644
index 0000000..d25096a
--- /dev/null
+++ b/pkg/keys/filesystem_test.go
@@ -0,0 +1,389 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package keys
+
+import (
+	"bytes"
+	"context"
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestNewFilesystem(t *testing.T) {
+	t.Parallel()
+
+	ctx := context.Background()
+
+	t.Run("root_empty", func(t *testing.T) {
+		t.Parallel()
+
+		if _, err := NewFilesystem(ctx, ""); err != nil {
+			t.Fatal(err)
+		}
+	})
+
+	t.Run("root_relative", func(t *testing.T) {
+		t.Parallel()
+
+		t.Cleanup(func() {
+			if err := os.RemoveAll("tmp1"); err != nil {
+				t.Fatal(err)
+			}
+		})
+
+		fs, err := NewFilesystem(ctx, "tmp1")
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if _, err := fs.CreateSigningKey(ctx, "foo", "bar"); err != nil {
+			t.Fatal(err)
+		}
+
+		if _, err := os.Stat("tmp1/foo/bar"); err != nil {
+			t.Fatal(err)
+		}
+	})
+
+	t.Run("root_absolute", func(t *testing.T) {
+		t.Parallel()
+
+		tmp, err := ioutil.TempDir("", "")
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		t.Cleanup(func() {
+			if err := os.RemoveAll(tmp); err != nil {
+				t.Fatal(err)
+			}
+		})
+
+		fs, err := NewFilesystem(ctx, tmp)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if _, err := fs.CreateSigningKey(ctx, "foo", "bar"); err != nil {
+			t.Fatal(err)
+		}
+
+		if _, err := os.Stat(tmp + "/foo/bar"); err != nil {
+			t.Fatal(err)
+		}
+	})
+}
+
+func TestFilesystem_NewSigner(t *testing.T) {
+	t.Parallel()
+
+	cases := []struct {
+		name  string
+		keyID string
+		setup func(string) error
+		err   string
+	}{
+		{
+			name:  "error_key_not_exist",
+			keyID: "totally_not_valid",
+			err:   "failed to read signing key",
+		},
+		{
+			name:  "error_key_not_ecdsa",
+			keyID: "banana",
+			setup: func(dir string) error {
+				pth := filepath.Join(dir, "banana")
+				return ioutil.WriteFile(pth, []byte("dafd"), 0600)
+			},
+			err: "failed to parse signing key",
+		},
+		{
+			name:  "happy",
+			keyID: "apple",
+			setup: func(dir string) error {
+				pk, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+				if err != nil {
+					return err
+				}
+				b, err := x509.MarshalECPrivateKey(pk)
+				if err != nil {
+					return err
+				}
+
+				pth := filepath.Join(dir, "apple")
+				return ioutil.WriteFile(pth, b, 0600)
+			},
+		},
+	}
+
+	for _, tc := range cases {
+		tc := tc
+
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			ctx := context.Background()
+
+			dir, err := ioutil.TempDir("", "")
+			if err != nil {
+				t.Fatal(err)
+			}
+			t.Cleanup(func() { os.RemoveAll(dir) })
+
+			if tc.setup != nil {
+				if err := tc.setup(dir); err != nil {
+					t.Fatal(err)
+				}
+			}
+
+			fs, err := NewFilesystem(ctx, dir)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			if _, err := fs.NewSigner(ctx, tc.keyID); err != nil {
+				if tc.err == "" {
+					t.Fatal(err)
+				}
+
+				if !strings.Contains(err.Error(), tc.err) {
+					t.Fatalf("expected %#v to contain %#v", err.Error(), tc.err)
+				}
+			}
+		})
+	}
+}
+
+func TestFilesystem_EncryptDecrypt(t *testing.T) {
+	t.Parallel()
+
+	cases := []struct {
+		name      string
+		keyID     string
+		plaintext []byte
+		aad       []byte
+		setup     func(*Filesystem) error
+		err       string
+	}{
+		{
+			name:  "error_key_not_exist",
+			keyID: "totally_not_valid",
+			err:   "failed to list keys",
+		},
+		{
+			name:  "no_key_versions",
+			keyID: "banana",
+			setup: func(fs *Filesystem) error {
+				dir := filepath.Join(fs.root, "banana")
+				return os.MkdirAll(dir, 0700)
+			},
+			err: "no key versions",
+		},
+		{
+			name:      "happy",
+			keyID:     "apple",
+			plaintext: []byte("bacon"),
+			setup: func(fs *Filesystem) error {
+				ctx := context.Background()
+				id, err := fs.CreateEncryptionKey(ctx, "", "apple")
+				if err != nil {
+					return err
+				}
+				if _, err := fs.CreateKeyVersion(ctx, id); err != nil {
+					return err
+				}
+				return nil
+			},
+		},
+		{
+			name:      "multi",
+			keyID:     "apple",
+			plaintext: []byte("bacon"),
+			setup: func(fs *Filesystem) error {
+				ctx := context.Background()
+				id, err := fs.CreateEncryptionKey(ctx, "", "apple")
+				if err != nil {
+					return err
+				}
+
+				for i := 0; i < 3; i++ {
+					if _, err := fs.CreateKeyVersion(ctx, id); err != nil {
+						return err
+					}
+				}
+				return nil
+			},
+		},
+	}
+
+	for _, tc := range cases {
+		tc := tc
+
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			ctx := context.Background()
+
+			dir, err := ioutil.TempDir("", "")
+			if err != nil {
+				t.Fatal(err)
+			}
+			t.Cleanup(func() { os.RemoveAll(dir) })
+
+			fs, err := NewFilesystem(ctx, dir)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			if tc.setup != nil {
+				if err := tc.setup(fs); err != nil {
+					t.Fatal(err)
+				}
+			}
+
+			ciphertext, err := fs.Encrypt(ctx, tc.keyID, tc.plaintext, tc.aad)
+			if err != nil {
+				if tc.err == "" {
+					t.Fatal(err)
+				}
+
+				if !strings.Contains(err.Error(), tc.err) {
+					t.Fatalf("expected %#v to contain %#v", err.Error(), tc.err)
+				}
+			}
+
+			if len(ciphertext) > 0 {
+				// Create another key version - this will ensure our ciphertext -> key
+				// version mapping works.
+				for i := 0; i < 3; i++ {
+					if _, err := fs.CreateKeyVersion(ctx, tc.keyID); err != nil {
+						t.Fatal(err)
+					}
+				}
+
+				plaintext, err := fs.Decrypt(ctx, tc.keyID, ciphertext, tc.aad)
+				if err != nil {
+					t.Fatal(err)
+				}
+
+				if got, want := plaintext, tc.plaintext; !bytes.Equal(got, want) {
+					t.Errorf("expected %#v to be %#v", got, want)
+				}
+			}
+		})
+	}
+}
+
+func TestFilesystem_SigningKeyVersions(t *testing.T) {
+	t.Parallel()
+
+	cases := []struct {
+		name  string
+		keyID string
+		setup func(*Filesystem) error
+		exp   int
+		err   string
+	}{
+		{
+			name:  "error_key_not_exist",
+			keyID: "totally_not_valid",
+			err:   "failed to open metadata",
+		},
+		{
+			name:  "happy",
+			keyID: "apple",
+			setup: func(fs *Filesystem) error {
+				ctx := context.Background()
+				id, err := fs.CreateSigningKey(ctx, "", "apple")
+				if err != nil {
+					return err
+				}
+				if _, err := fs.CreateKeyVersion(ctx, id); err != nil {
+					return err
+				}
+				return nil
+			},
+			exp: 1,
+		},
+		{
+			name:  "multi",
+			keyID: "apple",
+			setup: func(fs *Filesystem) error {
+				ctx := context.Background()
+				id, err := fs.CreateSigningKey(ctx, "", "apple")
+				if err != nil {
+					return err
+				}
+
+				for i := 0; i < 3; i++ {
+					if _, err := fs.CreateKeyVersion(ctx, id); err != nil {
+						return err
+					}
+				}
+				return nil
+			},
+			exp: 3,
+		},
+	}
+
+	for _, tc := range cases {
+		tc := tc
+
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+
+			ctx := context.Background()
+
+			dir, err := ioutil.TempDir("", "")
+			if err != nil {
+				t.Fatal(err)
+			}
+			t.Cleanup(func() { os.RemoveAll(dir) })
+
+			fs, err := NewFilesystem(ctx, dir)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			if tc.setup != nil {
+				if err := tc.setup(fs); err != nil {
+					t.Fatal(err)
+				}
+			}
+
+			versions, err := fs.SigningKeyVersions(ctx, tc.keyID)
+			if err != nil {
+				if tc.err == "" {
+					t.Fatal(err)
+				}
+
+				if !strings.Contains(err.Error(), tc.err) {
+					t.Fatalf("expected %#v to contain %#v", err.Error(), tc.err)
+				}
+			}
+
+			if got, want := len(versions), tc.exp; got != want {
+				t.Errorf("expected %d version to be %d", got, want)
+			}
+		})
+	}
+}
diff --git a/pkg/keys/testing.go b/pkg/keys/testing.go
new file mode 100644
index 0000000..b5b027e
--- /dev/null
+++ b/pkg/keys/testing.go
@@ -0,0 +1,92 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package keys
+
+import (
+	"context"
+	"io/ioutil"
+	"os"
+	"testing"
+)
+
+// TestKeyManager creates a new key manager suitable for use in tests.
+func TestKeyManager(tb testing.TB) KeyManager {
+	tb.Helper()
+
+	ctx := context.Background()
+
+	tmpdir, err := ioutil.TempDir("", "")
+	if err != nil {
+		tb.Fatal(err)
+	}
+	tb.Cleanup(func() {
+		if err := os.RemoveAll(tmpdir); err != nil {
+			tb.Fatal(err)
+		}
+	})
+
+	kms, err := NewFilesystem(ctx, tmpdir)
+	if err != nil {
+		tb.Fatal(err)
+	}
+
+	return kms
+}
+
+// TestEncryptionKey creates a new encryption key and key version in the given
+// key manager. If the provided key manager does not support managing encryption
+// keys, it calls t.Fatal.
+func TestEncryptionKey(tb testing.TB, kms KeyManager) string {
+	tb.Helper()
+
+	typ, ok := kms.(EncryptionKeyManager)
+	if !ok {
+		tb.Fatal("kms cannot manage encryption keys")
+	}
+
+	ctx := context.Background()
+	parent, err := typ.CreateEncryptionKey(ctx, "parent", "key")
+	if err != nil {
+		tb.Fatal(err)
+	}
+	if _, err := typ.CreateKeyVersion(ctx, parent); err != nil {
+		tb.Fatal(err)
+	}
+
+	return parent
+}
+
+// TestSigningKey creates a new signing key and key version in the given key
+// manager. If the provided key manager does not support managing signing keys,
+// it calls t.Fatal.
+func TestSigningKey(tb testing.TB, kms KeyManager) string {
+	tb.Helper()
+
+	typ, ok := kms.(SigningKeyManager)
+	if !ok {
+		tb.Fatal("kms cannot manage signing keys")
+	}
+
+	ctx := context.Background()
+	parent, err := typ.CreateSigningKey(ctx, "parent", "key")
+	if err != nil {
+		tb.Fatal(err)
+	}
+	if _, err := typ.CreateKeyVersion(ctx, parent); err != nil {
+		tb.Fatal(err)
+	}
+
+	return parent
+}
diff --git a/scripts/e2e-test.sh b/scripts/e2e-test.sh
new file mode 100755
index 0000000..606abb9
--- /dev/null
+++ b/scripts/e2e-test.sh
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# e2e-test.sh script is the entrypoint for running e2e tests in prow
+
+set -eEuo pipefail
+
+ROOT="$(cd "$(dirname "$0")/.." &>/dev/null; pwd -P)"
+
+# TODO: use project pool for this
+# PROW_JOB_ID is an env var set by prow, use project for prow when it's in prow
+if [[ -z "${PROJECT_ID:-}" && -n "${PROW_JOB_ID:-}" ]]; then
+    echo "Using project for prow since"
+    export PROJECT_ID=serious-physics-287516
+fi
+
+${ROOT}/scripts/terraform.sh smoke
diff --git a/scripts/e2e_test.sh b/scripts/e2e_test.sh
deleted file mode 100755
index aa3b9e7..0000000
--- a/scripts/e2e_test.sh
+++ /dev/null
@@ -1,57 +0,0 @@
-#!/usr/bin/env bash
-
-# Copyright 2020 Google LLC
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-set -eEuo pipefail
-
-ROOT="$(cd "$(dirname "$0")/.." &>/dev/null; pwd -P)"
-
-
-echo "🌳 Set up environment variables"
-if [[ -z "${PROJECT_ID:-}" ]]; then
-  echo "✋ PROJECT_ID must be set"
-  exit 1
-fi
-
-
-if [[ -z "${DB_CONN:-}" ]]; then # Allow custom database
-  echo "🔨 Provision servers"
-  pushd terraform
-  # TODO(chaodaiG): terraform init; terraform apply; trap "terraform destroy"
-  export DB_CONN="$(terraform output db_conn)"
-  export DB_NAME="$(terraform output db_name)"
-  export DB_USER="$(terraform output db_user)"
-  export DB_PASSWORD="secret://$(terraform output db_password)"
-  export DB_SSLMODE=disable
-  popd
-fi
-
-
-echo "🔨 Run cloud sql proxy"
-which cloud_sql_proxy || {
-  echo "✋ Download cloud_sql_proxy from https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#install"
-  exit 1
-}
-cloud_sql_proxy -instances=${DB_CONN}=tcp:5432 &
-CLOUD_SQL_PROXY_PID=$!
-trap "kill $CLOUD_SQL_PROXY_PID || true" EXIT
-
-
-echo "🧪 Test"
-go test \
-  -count=1 \
-  -race \
-  -timeout=10m \
-  ./internal/integration
diff --git a/terraform-e2e/README.md b/terraform-e2e/README.md
new file mode 100644
index 0000000..e37a7ce
--- /dev/null
+++ b/terraform-e2e/README.md
@@ -0,0 +1,54 @@
+# terraform-e2e
+
+This directory contains terraform configuration to be used for deploying key servers with terraform, while being able to reuse a project to deploy repeatedly. Should only be used for running e2e tests.
+
+## Prerequisite
+
+Follow steps from [Terrafrom instructions](https://github.com/google/exposure-notifications-server/tree/main/terraform), going through from top until finishing `Instructions` section, then change into `terraform-e2e` directory:
+
+```shell
+cd terraform-e2e/
+echo "project = \"${PROJECT_ID}\"" >> ./terraform.tfvars
+```
+
+## Terraform Apply
+
+### Fresh GCP project
+
+If it's a fresh GCP project, just run `terraform init; terrform apply` under current directory.
+
+### Existing GCP project
+
+If it's a project that had verification server deployed before, run these before `terraform apply`:
+
+```shell
+terraform import module.en.google_app_engine_application.app ${PROJECT_ID}
+```
+
+#### Troubleshooting: `terraform import` failure
+
+If you ran `terraform import` and see the error of invalid provider configuration like:
+
+```text
+Error: Invalid provider configuration
+
+  on ../terraform/main.tf line 15:
+  15: provider "google" {
+
+The configuration for
+module.en.provider["registry.terraform.io/hashicorp/google"] depends on values
+that cannot be determined until apply.
+```
+
+This can be mitigated by updating terraform to version equal or greater than 1.13.1
+
+## Terraform Destroy
+
+Terraform destroy always fails while trying to delete db instance. To be reliably successfully destroy, run these ahead:
+
+```shell
+db_inst_name="$(terraform output -json 'en' | jq '. | .db_inst_name' | tr -d \")"
+gcloud sql instances delete ${db_inst_name} -q --project=${PROJECT_ID}
+terraform state rm module.en.google_sql_user.user
+terraform state rm module.en.google_sql_ssl_cert.db-cert
+```
diff --git a/terraform-e2e/main.tf b/terraform-e2e/main.tf
new file mode 100644
index 0000000..8cefae8
--- /dev/null
+++ b/terraform-e2e/main.tf
@@ -0,0 +1,54 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+variable "project" {
+  type = string
+}
+
+resource "random_string" "suffix" {
+  length  = 5
+  special = false
+  number  = false
+  upper   = false
+}
+
+module "en" {
+  source = "../terraform"
+
+  project                           = var.project
+  cloudsql_disk_size_gb             = 500
+  db_name                           = "en-server-${random_string.suffix.result}"
+  kms_export_signing_key_ring_name  = "export-signing-${random_string.suffix.result}"
+  kms_revision_tokens_key_ring_name = "revision-tokens-${random_string.suffix.result}"
+
+  create_env_file = true
+
+  service_environment = {
+    export = {
+      LOG_DEBUG = "true"
+    }
+
+    exposure = {
+      LOG_DEBUG = "true"
+    }
+
+    generate = {
+      LOG_DEBUG = "true"
+    }
+  }
+}
+
+output "en" {
+  value = module.en
+}
diff --git a/terraform-e2e/state.tf b/terraform-e2e/state.tf
new file mode 100644
index 0000000..ffd9877
--- /dev/null
+++ b/terraform-e2e/state.tf
@@ -0,0 +1,5 @@
+terraform {
+  backend "gcs" {
+    bucket = "serious-physics-287516-tf-state"
+  }
+}