Repository: google/exposure-notifications-server
Pull request #840 (2020-08-07 18:13:49)

Integration test works with existing GCP PostgresSQL
- Allow working with existing DB
- Add script bridging terraform with integration test

This PR has been tested with the database provisioned by terraform. Follow up would be figuring out how to test servers provisioned by terraform

Part of: #723 

google-oss-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/google/exposure-notifications-server/pull/840#" title="Author self-approved">chaodaiG</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **mikehelmick**
You can assign the PR to them by writing `/assign @mikehelmick` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=google%2Fexposure-notifications-server).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/google/exposure-notifications-server/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["mikehelmick"]} -->

Patch:
diff --git a/RELEASING.md b/RELEASING.md
index 679d6e1..f2219df 100644
--- a/RELEASING.md
+++ b/RELEASING.md
@@ -12,14 +12,35 @@ You need the kubernetes release-notes tool to generate release notes.
 GO111MODULE=on go get k8s.io/release/cmd/release-notes
 ```
 
+You'll also need a GitHub Personal Access Token with permission to read repositories. It should be exported as `GITHUB_TOKEN`.
+
 # Cutting a release
 
+Set a target version:
+
+```shell
+# Note: no "v" prefix here!
+export RELEASE_VERSION="0.3.0"
+```
+
 ## Generate release notes
 
-This is a diff from the last time a release was generated. Adjust the `start-rev` and filename output as appropriate. The start rev should be the last release tag.
+This is a diff from the last time a release was generated.
 
-```
-release-notes --github-org google --github-repo exposure-notifications-server --required-author="" --start-rev=v0.2.0 --end-rev=main --output=/tmp/relnotes-0.3.0.md --repo-path=/tmp/relnotes-repo --dependencies=true  --branch=main
+```shell
+export LAST_RELEASE_TAG="TODO" # e.g. v0.2.2
+
+release-notes \
+  --github-org "google" \
+  --github-repo "exposure-notifications-server" \
+  --branch "main" \
+  --required-author ""  \
+  --start-rev "${LAST_RELEASE_TAG}" \
+  --end-rev "main" \
+  --release-version "${RELEASE_VERSION}" \
+  --output "/tmp/relnotes-${RELEASE_VERSION}.md" \
+  --repo-path "/tmp/relnotes-repo" \
+  --dependencies true
 ```
 
 ## Update the release notes
@@ -29,20 +50,40 @@ database migrations, possible breaking changes, and new environment variables th
 
 ## Create a release branch
 
-Ensure that you are on main, that you are up to date (the commit you want to cut the release at). This requires repo admin privelages.
+Ensure that you are on `main`, that you are up to date (the commit you want to cut the release at). This requires repo admin privelages.
 
+```shell
+git checkout -b release-${RELEASE_VERSION%??}
+git push --set-upstream origin release-${RELEASE_VERSION%??}
 ```
-git branch release-0.3
-git checkout release-0.3
-git push --set-upstream upstream release-0.3
+
+If you are cutting a patch release, the `release-x.y` branch likely already exists:
+
+```shell
+git fetch --all
+git checkout -t origin/release-${RELEASE_VERSION%??}
+# cherry-pick or decide the best way to bring over changes
+git push origin release-${RELEASE_VERSION%??}
+```
+
+## Create and push tag
+
+Ensure you are on the **release branch** and create a new tag:
+
+```shell
+git tag -a -s -m "Release v${RELEASE_VERSION}" v${RELEASE_VERSION}
+git push origin --tags
 ```
 
-## Draft a new release on github
+## Draft a new release on GitHub
+
+1. Find the tag you just pushed on the repo:
 
-Visit: https://github.com/google/exposure-notifications-server/releases/new
+    - [main](https://github.com/google/exposure-notifications-server/tags)
+    - [verification](https://github.com/google/exposure-notifications-verification-server/tags)
 
-Set the tag, including the patch level, i.e. `v0.3.0`
+1. Click the "..." and choose "create release".
 
-Copy the release notes into the web form.
+1. Copy the release notes into the web form.
 
-# Go do the same on exposure-notifications-verification-server
\ No newline at end of file
+# Go do the same on exposure-notifications-verification-server
diff --git a/cmd/cleanup-export/main.go b/cmd/cleanup-export/main.go
index b8d1225..bdcf9e3 100644
--- a/cmd/cleanup-export/main.go
+++ b/cmd/cleanup-export/main.go
@@ -21,16 +21,18 @@ import (
 	"net/http"
 
 	"github.com/google/exposure-notifications-server/internal/cleanup"
-	"github.com/google/exposure-notifications-server/internal/logging"
-	_ "github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	_ "github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/cmd/cleanup-exposure/main.go b/cmd/cleanup-exposure/main.go
index b44b256..d39b19d 100644
--- a/cmd/cleanup-exposure/main.go
+++ b/cmd/cleanup-exposure/main.go
@@ -21,16 +21,18 @@ import (
 	"net/http"
 
 	"github.com/google/exposure-notifications-server/internal/cleanup"
-	"github.com/google/exposure-notifications-server/internal/logging"
-	_ "github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	_ "github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/cmd/debugger/main.go b/cmd/debugger/main.go
index ba46869..5c0091f 100644
--- a/cmd/debugger/main.go
+++ b/cmd/debugger/main.go
@@ -20,15 +20,17 @@ import (
 	"fmt"
 
 	"github.com/google/exposure-notifications-server/internal/debugger"
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/cmd/export/main.go b/cmd/export/main.go
index db3d7db..f0f0518 100644
--- a/cmd/export/main.go
+++ b/cmd/export/main.go
@@ -20,16 +20,18 @@ import (
 	"fmt"
 
 	"github.com/google/exposure-notifications-server/internal/export"
-	"github.com/google/exposure-notifications-server/internal/logging"
-	_ "github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	_ "github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/cmd/exposure/main.go b/cmd/exposure/main.go
index eb53513..5843bcb 100644
--- a/cmd/exposure/main.go
+++ b/cmd/exposure/main.go
@@ -20,17 +20,19 @@ import (
 	"fmt"
 	"net/http"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
-	_ "github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/publish"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	_ "github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/cmd/federationin/main.go b/cmd/federationin/main.go
index c77e381..44a3f91 100644
--- a/cmd/federationin/main.go
+++ b/cmd/federationin/main.go
@@ -22,16 +22,18 @@ import (
 	"net/http"
 
 	"github.com/google/exposure-notifications-server/internal/federationin"
-	"github.com/google/exposure-notifications-server/internal/logging"
-	_ "github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	_ "github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/cmd/federationout/main.go b/cmd/federationout/main.go
index 9815d06..9c7d82d 100644
--- a/cmd/federationout/main.go
+++ b/cmd/federationout/main.go
@@ -25,17 +25,19 @@ import (
 	"go.opencensus.io/plugin/ocgrpc"
 
 	"github.com/google/exposure-notifications-server/internal/federationout"
-	"github.com/google/exposure-notifications-server/internal/logging"
-	_ "github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/pb"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	_ "github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/cmd/generate/main.go b/cmd/generate/main.go
index c5e42ad..71e9c07 100644
--- a/cmd/generate/main.go
+++ b/cmd/generate/main.go
@@ -22,16 +22,18 @@ import (
 	"net/http"
 
 	"github.com/google/exposure-notifications-server/internal/generate"
-	"github.com/google/exposure-notifications-server/internal/logging"
-	_ "github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	_ "github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/cmd/key-rotation/main.go b/cmd/key-rotation/main.go
index 20058eb..46ee998 100644
--- a/cmd/key-rotation/main.go
+++ b/cmd/key-rotation/main.go
@@ -20,16 +20,18 @@ import (
 	"fmt"
 
 	"github.com/google/exposure-notifications-server/internal/keyrotation"
-	"github.com/google/exposure-notifications-server/internal/logging"
-	_ "github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	_ "github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/server"
 	"github.com/sethvargo/go-signalcontext"
 )
 
 func main() {
 	ctx, done := signalcontext.OnInterrupt()
-	logger := logging.FromContext(ctx)
+
+	logger := logging.NewLogger(true)
+	ctx = logging.WithLogger(ctx, logger)
 
 	err := realMain(ctx)
 	done()
diff --git a/docs/getting-started/deploying.md b/docs/getting-started/deploying.md
index b8cb000..e97c254 100644
--- a/docs/getting-started/deploying.md
+++ b/docs/getting-started/deploying.md
@@ -161,7 +161,7 @@ To enable distributed tracing, please ensure your environment has these variable
 Variable|Values|Comment
 ---|---|---
 OBSERVABILITY_EXPORTER|If unset, no exporting shall be done. Use any of "stackdriver", "prometheus", or "ocagent" otherwise|Note: when using "prometheus" a METRICS_PORT environment variable should also be set
-PROJECT_ID|The ProjectID of your associated Google Cloud Platform project on which this application shall be deployed|Required if you use "stackdrver"
+PROJECT_ID|The ProjectID of your associated Google Cloud Platform project on which this application shall be deployed|Required if you use "stackdriver"
 
 ## Running migrations
 
diff --git a/docs/server_functional_requirements.md b/docs/server_functional_requirements.md
index e32d785..06f7efe 100644
--- a/docs/server_functional_requirements.md
+++ b/docs/server_functional_requirements.md
@@ -74,7 +74,8 @@ Minimum required fields, followed by a JSON example:
   * `ExposureKey` object properties
     * `key` (**REQUIRED**)
       * Type: String
-      * Description: Base64 encoded temporary exposure key from the device
+      * Description: Base64 encoded temporary exposure key from the device.
+      The server MUST validate that the length of the decoded key is exactly 16 bytes.
     * `rollingStartNumber` (**REQUIRED**)
       * Type: integer (uint32)
       * Description: Intervals are 10 minute increments since the UTC epoch
@@ -86,23 +87,20 @@ Minimum required fields, followed by a JSON example:
       * Description: Number of intervals that the key is valid for
     * `transmissionRisk` (**REQUIRED**)
       * Type: Integer
-      * **The values and meanings of this enum are not finalized at this time.** //TODO(llatif): check status
       * Constraints:
         * Valid values range from 0-8
-      * Description: //TODO(llatif): Add description
-* `regions` (**REQUIRED**)
-  [ISO 3166 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) format.
-  * Type: Array of string
-  * Description: 2 letter country to identify the region(s) a key is valid for.
-* `appPackageName` (**REQUIRED**)
+      * Description: recommend practice is to use the diagnosis verification
+        protocol with a 'reportType' provided in the verification certificate.
+        The server can then backfill transmission risk correctly.
+* `healthAuthorityID` (**REQUIRED**)
   * Type: string (lowercase)
-  * Description: Name of the application being used to send the request. This
-    is used to determine what app is uploading keys and if it is an allowed
-    region for that app. (BundleID on iOS)
+  * Description: Name of the health authority for whom these keys are being
+    shared.
 * `verificationPayload` (**REQUIRED FOR VERIFICATION PROTOCOL**)
   * Type: String
   * Description: verificationPayload is a signed certificate from a public
-    health authority, indicating a confirmed diagnosis
+    health authority, indicating a confirmed diagnosis. Use of the verification
+    protocol is HIGHLY recommended.
 * `hmackey` (**REQUIRED FOR VERIFICATION PROTOCOL**)
   * Type: String
   * Description: The device generated, random key that was used to create the
@@ -110,6 +108,18 @@ Minimum required fields, followed by a JSON example:
   data to calculate the HMAC over is defined as part of the
   [verification protocol](design/verification_protocol.md). This field is base64
   encoded.
+* `symptomOnsetInterval`
+  * Type: int (int32)
+  * Description:  An interval number that aligns with the symptom onset date.
+    * Uses the same interval system as TEK timing.
+    * Will be rounded down to the start of the UTC day provided.
+    * Will be used to calculate the days +/- symptom onset for provided keys.
+    * MUST be no more than 14 days ago.
+    * Does not have to be within range of any of the provided keys (i.e. future key uploads)
+* `revisionToken`
+  * Type: string
+  * Description: An opaque string that must be passed intact on additional publish
+     requests from the same device, where the same TEKs may be published again.
 * `padding`
   * Type: String
   * Constraints:
@@ -122,31 +132,44 @@ The following snippet is an example POST request payload in JSON format.
 ```json
 {
   "temporaryExposureKeys": [
-    {"key": "base64 KEY1", "rollingStartNumber": 12345, "rollingPeriod": 144, "transmissionRisk": 5},
-    {"key": "base64 KEY2", "rollingStartNumber": 12489, "rollingPeriod": 10, "transmissionRisk": 6},
-    {"key": "base64 KEYN", "rollingStartNumber": 12499, "rollingPeriod": 100, "transmissionRisk": 7}],
-  "regions": ["US", "CA", "MX"],
-  "appPackageName": "com.foo.app",
+    {"key": "base64 KEY1 (16 bytes)", "rollingStartNumber": 12345, "rollingPeriod": 144, "transmissionRisk": 5},
+    {"key": "base64 KEY2 (16 bytes)", "rollingStartNumber": 12489, "rollingPeriod": 10, "transmissionRisk": 6},
+    {"key": "base64 KEYN (16 bytes)", "rollingStartNumber": 12499, "rollingPeriod": 100, "transmissionRisk": 7}],
+  "healthAuthorityID": "gov.state.doh",
   "verificationPayload": "signed JWT issued by public health authority",
   "hmackey": "base64 encoded HMAC key used in preparing the data for the verification server",
+  "symptomOnsetInterval": 12345,
+  "revisionToken": "empty or result of previous publish"
   "padding": "random string data..."
 }
 ```
 
-### Requirements and recommendations
+The publish request should respond with a structure described in [exposure_types.go](https://github.com/google/exposure-notifications-server/blob/main/pkg/api/v1alpha1/exposure_types.go).
 
-* Required: An allow-list check for `appPackageName` and the regions in
-which the app is allowed to report on. Individual applications should also be
-configured for one or more public health authorities they at they accept
-diagnosis verifications from.
+The revision token is a critical piece of ensuring that revised keys that ensures that a different device cannot revise a previously uploaded key. We have a reference implementation in the [revision package](https://github.com/google/exposure-notifications-server/blob/main/internal/revision/revision.go).
 
-* Recommended: To discourage abuse, only failures in processing should
-return retry-able error codes to clients. For example, invalid device
-attestations should return success, with the data only saved for abuse
-analysis.
+```json
+{
+  "revisionToken": "oaque token that needs to be sent on future publish.",
+  "insertedExposures": 14,
+  "error": "omitted, or error message",
+  "code": "omitted or standard error code",
+  "padding": "padding to normalize response size"
+}
+```
 
-* Appropriate denial of service protection should be put in place.
+### Requirements and recommendations
 
+* Required 
+   * An allow-list check for `healthAuthorityID` that specifies the 
+  verification certificate signing keys allowed and the region information
+  for those keys.
+  * Validation of all uploaded data, in particular that TEK data is 16 bytes in length
+    and that the interval number is valid (>=14days ago not in the future), the 
+    interval count is valid (1-144) and that the rest of the metadata that will
+    appear in an export file is valid.
+* Recommended
+  * Appropriate denial of service protection should be put in place.
 * Recommended: Our overall security and privacy recommendation is to, from
   the mobile application, periodically send chaff requests to the sever so that
   ALL users appear to be reporting themselves as infected multiple times per
diff --git a/internal/authorizedapp/database_provider.go b/internal/authorizedapp/database_provider.go
index 4cf1266..cec74b4 100644
--- a/internal/authorizedapp/database_provider.go
+++ b/internal/authorizedapp/database_provider.go
@@ -25,8 +25,7 @@ import (
 	"github.com/google/exposure-notifications-server/internal/authorizedapp/model"
 	"github.com/google/exposure-notifications-server/internal/database"
 	"github.com/google/exposure-notifications-server/pkg/cache"
-
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 )
 
 // Compile-time check to assert implementation.
diff --git a/internal/cleanup/cleanup.go b/internal/cleanup/cleanup.go
index 8cb32f3..6e7dc6e 100644
--- a/internal/cleanup/cleanup.go
+++ b/internal/cleanup/cleanup.go
@@ -26,9 +26,9 @@ import (
 	"github.com/google/exposure-notifications-server/internal/export/database"
 	publishdb "github.com/google/exposure-notifications-server/internal/publish/database"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/serverenv"
 	"github.com/google/exposure-notifications-server/internal/storage"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 )
 
 const (
diff --git a/internal/cleanup/config.go b/internal/cleanup/config.go
index 99766bb..65879fd 100644
--- a/internal/cleanup/config.go
+++ b/internal/cleanup/config.go
@@ -18,9 +18,9 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
 	"github.com/google/exposure-notifications-server/internal/storage"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 )
 
diff --git a/internal/database/connection.go b/internal/database/connection.go
index 2da6ff8..9d6e06e 100644
--- a/internal/database/connection.go
+++ b/internal/database/connection.go
@@ -21,7 +21,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 
 	"github.com/jackc/pgx/v4/pgxpool"
 )
diff --git a/internal/database/lock.go b/internal/database/lock.go
index b37dd49..856d189 100644
--- a/internal/database/lock.go
+++ b/internal/database/lock.go
@@ -20,7 +20,7 @@ import (
 	"fmt"
 	"time"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	pgx "github.com/jackc/pgx/v4"
 )
 
diff --git a/internal/debugger/handler_debug.go b/internal/debugger/handler_debug.go
index 010903b..80c0ca5 100644
--- a/internal/debugger/handler_debug.go
+++ b/internal/debugger/handler_debug.go
@@ -27,7 +27,7 @@ import (
 	authorizedappmodel "github.com/google/exposure-notifications-server/internal/authorizedapp/model"
 	exportdatabase "github.com/google/exposure-notifications-server/internal/export/database"
 	exportmodel "github.com/google/exposure-notifications-server/internal/export/model"
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/kelseyhightower/run"
 	"golang.org/x/oauth2/google"
 	"google.golang.org/api/iam/v1"
diff --git a/internal/export/batcher.go b/internal/export/batcher.go
index 6af64e7..e4f2681 100644
--- a/internal/export/batcher.go
+++ b/internal/export/batcher.go
@@ -26,7 +26,7 @@ import (
 	"github.com/google/exposure-notifications-server/internal/export/model"
 	publishmodel "github.com/google/exposure-notifications-server/internal/publish/model"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 )
 
 // handleCreateBatches is a handler to iterate the rows of ExportConfig and
diff --git a/internal/export/config.go b/internal/export/config.go
index aefffac..5b39669 100644
--- a/internal/export/config.go
+++ b/internal/export/config.go
@@ -19,10 +19,10 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
 	"github.com/google/exposure-notifications-server/internal/storage"
 	"github.com/google/exposure-notifications-server/pkg/keys"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 )
 
diff --git a/internal/export/database/export.go b/internal/export/database/export.go
index 9336d16..865629b 100644
--- a/internal/export/database/export.go
+++ b/internal/export/database/export.go
@@ -25,8 +25,8 @@ import (
 
 	"github.com/google/exposure-notifications-server/internal/database"
 	"github.com/google/exposure-notifications-server/internal/export/model"
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/storage"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 
 	pgx "github.com/jackc/pgx/v4"
 )
diff --git a/internal/export/worker.go b/internal/export/worker.go
index 0693c68..e5983aa 100644
--- a/internal/export/worker.go
+++ b/internal/export/worker.go
@@ -32,7 +32,7 @@ import (
 	"github.com/google/exposure-notifications-server/internal/export/model"
 	publishmodel "github.com/google/exposure-notifications-server/internal/publish/model"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1alpha1"
 )
@@ -129,8 +129,13 @@ func (s *Server) exportBatch(ctx context.Context, eb *model.ExportBatch, emitInd
 	groups := []*group{}
 	nextGroup := group{}
 	totalNewKeys, totalRevisedKeys := 0, 0
+	droppedKeys := 0
 
 	_, err := publishdatabase.New(db).IterateExposures(ctx, criteria, func(exp *publishmodel.Exposure) error {
+		if len(exp.ExposureKey) != verifyapi.KeyLength {
+			droppedKeys++
+			return nil
+		}
 		nextGroup.exposures = append(nextGroup.exposures, exp)
 		totalNewKeys++
 		if nextGroup.Length() == s.config.MaxRecords {
@@ -146,6 +151,10 @@ func (s *Server) exportBatch(ctx context.Context, eb *model.ExportBatch, emitInd
 	// go get the revised keys.
 	criteria.OnlyRevisedKeys = true
 	_, err = publishdatabase.New(db).IterateExposures(ctx, criteria, func(exp *publishmodel.Exposure) error {
+		if len(exp.ExposureKey) != verifyapi.KeyLength {
+			droppedKeys++
+			return nil
+		}
 		nextGroup.revised = append(nextGroup.revised, exp)
 		totalRevisedKeys++
 		if nextGroup.Length() == s.config.MaxRecords {
@@ -158,6 +167,11 @@ func (s *Server) exportBatch(ctx context.Context, eb *model.ExportBatch, emitInd
 		return fmt.Errorf("iterating revised exposures: %w", err)
 	}
 
+	if droppedKeys > 0 {
+		logger.Errorf("Export found keys of invalid length, %v keys were dropped", droppedKeys)
+		s.env.MetricsExporter(ctx).WriteInt("export-bad-key-length", false, droppedKeys)
+	}
+
 	// If the last group has anything, add it to the list.
 	if nextGroup.Length() > 0 {
 		groups = append(groups, &nextGroup)
diff --git a/internal/federationin/config.go b/internal/federationin/config.go
index e7eb2e8..a853ffc 100644
--- a/internal/federationin/config.go
+++ b/internal/federationin/config.go
@@ -19,8 +19,8 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 )
 
diff --git a/internal/federationin/federationin.go b/internal/federationin/federationin.go
index 8fb835a..712eba4 100644
--- a/internal/federationin/federationin.go
+++ b/internal/federationin/federationin.go
@@ -30,13 +30,13 @@ import (
 	coredb "github.com/google/exposure-notifications-server/internal/database"
 	"github.com/google/exposure-notifications-server/internal/federationin/database"
 	"github.com/google/exposure-notifications-server/internal/federationin/model"
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/metrics"
 	"github.com/google/exposure-notifications-server/internal/pb"
 	publishdb "github.com/google/exposure-notifications-server/internal/publish/database"
 	publishmodel "github.com/google/exposure-notifications-server/internal/publish/model"
 	"github.com/google/exposure-notifications-server/internal/serverenv"
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1alpha1"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 
 	"google.golang.org/api/idtoken"
 	"google.golang.org/grpc"
diff --git a/internal/federationout/config.go b/internal/federationout/config.go
index 8329d97..e60f2ac 100644
--- a/internal/federationout/config.go
+++ b/internal/federationout/config.go
@@ -18,8 +18,8 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 )
 
diff --git a/internal/federationout/federationout.go b/internal/federationout/federationout.go
index d22749d..c8e6e14 100644
--- a/internal/federationout/federationout.go
+++ b/internal/federationout/federationout.go
@@ -24,8 +24,8 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/federationin/model"
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/pb"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 
 	coredb "github.com/google/exposure-notifications-server/internal/database"
 	"github.com/google/exposure-notifications-server/internal/federationout/database"
diff --git a/internal/generate/config.go b/internal/generate/config.go
index 313b768..5ede2d7 100644
--- a/internal/generate/config.go
+++ b/internal/generate/config.go
@@ -19,9 +19,9 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/publish/model"
 	"github.com/google/exposure-notifications-server/internal/setup"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 )
 
@@ -47,7 +47,7 @@ type Config struct {
 	MaxIntervalAge               time.Duration `env:"MAX_INTERVAL_AGE_ON_PUBLISH, default=360h"`
 	MaxMagnitudeSymptomOnsetDays uint          `env:"MAX_SYMPTOM_ONSET_DAYS, default=21"`
 	CreatedAtTruncateWindow      time.Duration `env:"TRUNCATE_WINDOW, default=1h"`
-	DefaultRegion                string        `env:"DEFAULT_REGOIN, default=US"`
+	DefaultRegion                string        `env:"DEFAULT_REGION, default=US"`
 	ChanceOfKeyRevision          int           `env:"CHANCE_OF_KEY_REVISION, default=30"` // 0-100 are valid values.
 	KeyRevisionDelay             time.Duration `env:"KEY_REVISION_DELAY, default=2h"`     // key revision will be forward dates this amount.
 }
diff --git a/internal/generate/handler.go b/internal/generate/handler.go
index e392082..f55bb48 100644
--- a/internal/generate/handler.go
+++ b/internal/generate/handler.go
@@ -24,12 +24,12 @@ import (
 
 	"go.opencensus.io/trace"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/publish/database"
 	"github.com/google/exposure-notifications-server/internal/publish/model"
 	"github.com/google/exposure-notifications-server/internal/serverenv"
 	"github.com/google/exposure-notifications-server/internal/verification"
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/google/exposure-notifications-server/pkg/util"
 )
 
diff --git a/internal/integration/client.go b/internal/integration/client.go
index 2832d1c..6c46c46 100644
--- a/internal/integration/client.go
+++ b/internal/integration/client.go
@@ -94,6 +94,19 @@ func (c *Client) ExportBatches() error {
 	return nil
 }
 
+func (c *Client) RotateKeys() error {
+	resp, err := c.client.Get("/key-rotation/rotate-keys")
+	if err != nil {
+		return fmt.Errorf("failed to GET /key-rotation/rotate-keys: %w", err)
+	}
+
+	body, err := checkResp(resp)
+	if err != nil {
+		return fmt.Errorf("failed to GET /key-rotation/rotate-keys: %w: %s", err, body)
+	}
+	return nil
+}
+
 func (c *Client) StartExportWorkers() error {
 	resp, err := c.client.Get("/export/do-work")
 	if err != nil {
diff --git a/internal/integration/helpers.go b/internal/integration/helpers.go
index 298794a..d524e31 100644
--- a/internal/integration/helpers.go
+++ b/internal/integration/helpers.go
@@ -30,6 +30,7 @@ import (
 	"github.com/google/exposure-notifications-server/internal/database"
 	"github.com/google/exposure-notifications-server/internal/export"
 	"github.com/google/exposure-notifications-server/internal/federationin"
+	"github.com/google/exposure-notifications-server/internal/keyrotation"
 	"github.com/google/exposure-notifications-server/internal/publish"
 	"github.com/google/exposure-notifications-server/internal/revision"
 	revdb "github.com/google/exposure-notifications-server/internal/revision/database"
@@ -40,6 +41,7 @@ import (
 	"github.com/google/exposure-notifications-server/pkg/keys"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 	"github.com/google/exposure-notifications-server/pkg/server"
+	"github.com/sethvargo/go-envconfig"
 	"github.com/sethvargo/go-retry"
 
 	authorizedappmodel "github.com/google/exposure-notifications-server/internal/authorizedapp/model"
@@ -50,15 +52,16 @@ import (
 )
 
 var (
-	ExportDir    = "my-bucket"
+	// ExportDir is the default export bucket of blob storage
+	ExportDir = "my-bucket"
+	// FileNameRoot is the default file path root under blob storage
 	FileNameRoot = "/"
-	jwtCfg       = testutil.JWTConfig{}
 )
 
 // NewTestServer sets up clients used for integration tests
-func NewTestServer(tb testing.TB, exportPeriod time.Duration) (*serverenv.ServerEnv, *Client, *database.DB) {
+func NewTestServer(tb testing.TB, exportPeriod time.Duration) (*serverenv.ServerEnv, *Client, *database.DB, testutil.JWTConfig) {
 	ctx := context.Background()
-	env, client := testServer(tb)
+	env, client, jwtCfg := testServer(tb)
 	db := env.Database()
 	enClient := &Client{client: client}
 
@@ -103,14 +106,17 @@ func NewTestServer(tb testing.TB, exportPeriod time.Duration) (*serverenv.Server
 		tb.Fatal(err)
 	}
 
-	return env, enClient, db
+	return env, enClient, db, jwtCfg
 }
 
 // testServer sets up mocked local servers for running tests
-func testServer(tb testing.TB) (*serverenv.ServerEnv, *http.Client) {
+func testServer(tb testing.TB) (*serverenv.ServerEnv, *http.Client, testutil.JWTConfig) {
 	tb.Helper()
 
-	ctx := context.Background()
+	var (
+		ctx    = context.Background()
+		jwtCfg = testutil.JWTConfig{}
+	)
 
 	aa, err := authorizedapp.NewMemoryProvider(ctx, nil)
 	if err != nil {
@@ -130,15 +136,31 @@ func testServer(tb testing.TB) (*serverenv.ServerEnv, *http.Client) {
 	}
 
 	db := database.NewTestDatabase(tb)
+	if v := os.Getenv("DB_NAME"); v != "" && !testing.Short() {
+		dbConfig := &database.Config{}
+		sm, err := secrets.SecretManagerFor(ctx, secrets.SecretManagerTypeGoogleSecretManager)
+		if err != nil {
+			tb.Fatalf("unable to connect to secret manager: %v", err)
+		}
+		if err := envconfig.ProcessWith(ctx, dbConfig, envconfig.OsLookuper(),
+			secrets.Resolver(sm, &secrets.Config{})); err != nil {
+			tb.Fatalf("error loading environment variables: %v", err)
+		}
+
+		db, err = database.NewFromEnv(ctx, dbConfig)
+		if err != nil {
+			tb.Fatalf("unable to connect to database: %v", err)
+		}
+	}
 
 	km, err := keys.NewInMemory(ctx)
 	if err != nil {
 		tb.Fatal(err)
 	}
-	if err := km.AddEncryptionKey("tokenkey"); err != nil {
+	if _, err := km.CreateEncryptionKey("tokenkey"); err != nil {
 		tb.Fatal(err)
 	}
-	if err := km.AddSigningKey("signingkey"); err != nil {
+	if _, err := km.CreateSigningKey("signingkey"); err != nil {
 		tb.Fatal(err)
 	}
 	// create an initial revision key.
@@ -235,7 +257,8 @@ func testServer(tb testing.TB) (*serverenv.ServerEnv, *http.Client) {
 	if err != nil {
 		tb.Fatal(err)
 	}
-	mux.Handle("/export/", http.StripPrefix("/export", exportServer.Routes(ctx)))
+	exportHandler := http.StripPrefix("/export", exportServer.Routes(ctx))
+	mux.Handle("/export/", exportHandler)
 
 	// Federation
 	federationInConfig := &federationin.Config{
@@ -243,18 +266,36 @@ func testServer(tb testing.TB) (*serverenv.ServerEnv, *http.Client) {
 		TruncateWindow: 1 * time.Hour,
 	}
 
-	mux.Handle("/federation-in", federationin.NewHandler(env, federationInConfig))
+	federationinHandler := federationin.NewHandler(env, federationInConfig)
+	mux.Handle("/federation-in", federationinHandler)
 
 	// Federation out
 	// TODO: this is a grpc listener and requires a lot of setup.
 
+	revConfig := revision.Config{
+		KeyID:     "tokenkey",
+		AAD:       []byte{1, 2, 3},
+		MinLength: 28,
+	}
+
+	// Key Rotation
+	keyRotationConfig := &keyrotation.Config{
+		RevisionToken: revConfig,
+
+		// Very accellerated schedule for testing.
+		NewKeyPeriod:       100 * time.Millisecond,
+		DeleteOldKeyPeriod: 100 * time.Millisecond,
+	}
+
+	rotationServer, err := keyrotation.NewServer(keyRotationConfig, env)
+	if err != nil {
+		tb.Fatal(err)
+	}
+	mux.Handle("/key-rotation/", http.StripPrefix("/key-rotation", rotationServer.Routes(ctx)))
+
 	// Publish
 	publishConfig := &publish.Config{
-		RevisionToken: revision.Config{
-			KeyID:     "tokenkey",
-			AAD:       []byte{1, 2, 3},
-			MinLength: 28,
-		},
+		RevisionToken:            revConfig,
 		MaxKeysOnPublish:         15,
 		MaxSameStartIntervalKeys: 2,
 		MaxIntervalAge:           360 * time.Hour,
@@ -288,7 +329,7 @@ func testServer(tb testing.TB) (*serverenv.ServerEnv, *http.Client) {
 	// Create a client
 	client := testClient(tb, srv)
 
-	return env, client
+	return env, client, jwtCfg
 }
 
 type prefixRoundTripper struct {
@@ -349,6 +390,7 @@ func Eventually(tb testing.TB, times uint64, f func() error) {
 	}
 }
 
+// IndexFilePath returns the filepath of index file under blob storage
 func IndexFilePath() string {
 	return path.Join(FileNameRoot, "index.txt")
 }
diff --git a/internal/keyrotation/config.go b/internal/keyrotation/config.go
index db39d01..c21dbd2 100644
--- a/internal/keyrotation/config.go
+++ b/internal/keyrotation/config.go
@@ -18,10 +18,10 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/revision"
 	"github.com/google/exposure-notifications-server/internal/setup"
 	"github.com/google/exposure-notifications-server/pkg/keys"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 )
 
diff --git a/internal/keyrotation/rotate.go b/internal/keyrotation/rotate.go
index e61ad5b..098ad3a 100644
--- a/internal/keyrotation/rotate.go
+++ b/internal/keyrotation/rotate.go
@@ -21,8 +21,8 @@ import (
 	"net/http"
 	"time"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/revision/database"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/hashicorp/go-multierror"
 	"go.opencensus.io/trace"
 )
@@ -62,6 +62,7 @@ func (s *Server) handleRotateKeys(ctx context.Context) http.HandlerFunc {
 
 func (s *Server) doRotate(ctx context.Context) error {
 	metrics := s.env.MetricsExporter(ctx)
+	logger := logging.FromContext(ctx).Named("keyrotation.doRotate")
 
 	effectiveID, allowed, err := s.revisionDB.GetAllowedRevisionKeys(ctx)
 	if err != nil {
@@ -77,6 +78,7 @@ func (s *Server) doRotate(ctx context.Context) error {
 		}
 		effectiveID = key.KeyID
 		previousCreated = key.CreatedAt
+		logger.Info("Created new revision key.")
 		metrics.WriteInt("revision-keys-created", true, 1)
 	} else {
 		previousCreated = allowed[0].CreatedAt
@@ -92,8 +94,10 @@ func (s *Server) doRotate(ctx context.Context) error {
 		}
 		previousCreated = key.CreatedAt
 	}
-
-	metrics.WriteInt("revision-keys-deleted", true, deleted)
+	if deleted > 0 {
+		logger.Infof("Deleted %d old revision keys.", deleted)
+		metrics.WriteInt("revision-keys-deleted", true, deleted)
+	}
 	return result
 }
 
diff --git a/internal/logging/logger.go b/internal/logging/logger.go
deleted file mode 100644
index becf2a3..0000000
--- a/internal/logging/logger.go
+++ /dev/null
@@ -1,51 +0,0 @@
-// Copyright 2020 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-// Package logging sets up and configures logging.
-package logging
-
-import (
-	"context"
-
-	"go.uber.org/zap"
-)
-
-type loggerKey struct{}
-
-var fallbackLogger *zap.SugaredLogger
-
-func init() {
-	// Set up Stackdriver keys (see: https://cloud.google.com/run/docs/logging#special-fields)
-	config := zap.NewProductionConfig()
-	config.EncoderConfig.MessageKey = "message"
-	config.EncoderConfig.LevelKey = "severity"
-	config.Level = zap.NewAtomicLevelAt(zap.DebugLevel)
-
-	if logger, err := config.Build(); err != nil {
-		fallbackLogger = zap.NewNop().Sugar()
-	} else {
-		fallbackLogger = logger.Named("default").Sugar()
-	}
-}
-
-func WithLogger(ctx context.Context, logger *zap.SugaredLogger) context.Context {
-	return context.WithValue(ctx, loggerKey{}, logger)
-}
-
-func FromContext(ctx context.Context) *zap.SugaredLogger {
-	if logger, ok := ctx.Value(loggerKey{}).(*zap.SugaredLogger); ok {
-		return logger
-	}
-	return fallbackLogger
-}
diff --git a/internal/metrics/exporter.go b/internal/metrics/exporter.go
index e99604d..cc7becf 100644
--- a/internal/metrics/exporter.go
+++ b/internal/metrics/exporter.go
@@ -18,7 +18,7 @@ package metrics
 import (
 	"context"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"go.uber.org/zap"
 )
 
diff --git a/internal/observability/observability.go b/internal/observability/observability.go
deleted file mode 100644
index ae1d746..0000000
--- a/internal/observability/observability.go
+++ /dev/null
@@ -1,81 +0,0 @@
-// Copyright 2020 Google LLC
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//      http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-// Package observability sets up and configures observability tools.
-package observability
-
-import (
-	"context"
-	"fmt"
-	"time"
-
-	"contrib.go.opencensus.io/exporter/ocagent"
-	"contrib.go.opencensus.io/exporter/stackdriver"
-	"github.com/google/exposure-notifications-server/internal/logging"
-)
-
-// Exporter defines the minimum shared functionality for an observability exporter
-// used by this application.
-type Exporter interface {
-	InitExportOnce() error
-	Flush()
-}
-
-// NewFromEnv returns the observability exporter given the provided configuration, or an error
-// if it failed to be created.
-func NewFromEnv(ctx context.Context, config *Config) (Exporter, error) {
-	switch config.ExporterType {
-	default:
-		return &NoopExporter{}, nil
-
-	case ExporterNoop:
-		return &NoopExporter{}, nil
-
-	case ExporterStackdriver:
-		if config.StackdriverConfig.ProjectID == "" {
-			return nil, fmt.Errorf("configuration PROJECT_ID is required to use the Stackdriver observability exporter")
-		}
-		logger := logging.FromContext(ctx).Named("stackdriver")
-
-		monitoredResource := NewStackdriverMonitoredResoruce(&config.StackdriverConfig)
-
-		sde, err := stackdriver.NewExporter(stackdriver.Options{
-			ProjectID:         config.StackdriverConfig.ProjectID,
-			ReportingInterval: time.Minute, // stackdriver export interval minimum
-			MonitoredResource: monitoredResource,
-			OnError: func(err error) {
-				logger.Errorf("stackdriver export error: %v", err)
-			},
-		})
-		if err != nil {
-			return nil, fmt.Errorf("failed to create Stackdriver observability exporter: %v", err)
-		}
-		return &GenericExporter{sde, config.TraceProbabilitySampleRate, func() { sde.Flush() }}, nil
-
-	case ExporterOCAgent, ExporterPrometheus:
-		var opts []ocagent.ExporterOption
-		if config.OCAgentConfig.Insecure {
-			opts = append(opts, ocagent.WithInsecure())
-		}
-		if config.OCAgentConfig.Endpoint != "" {
-			opts = append(opts, ocagent.WithAddress(config.OCAgentConfig.Endpoint))
-		}
-
-		oce, err := ocagent.NewExporter(opts...)
-		if err != nil {
-			return nil, fmt.Errorf("failed to create OpenCensus observability exporter: %v", err)
-		}
-		return &GenericExporter{oce, config.TraceProbabilitySampleRate, func() { oce.Flush() }}, nil
-	}
-}
diff --git a/internal/performance/README.md b/internal/performance/README.md
index 698a293..8ebb963 100644
--- a/internal/performance/README.md
+++ b/internal/performance/README.md
@@ -3,3 +3,13 @@
 This directory contains configs and tests for performance metrics collection,
 the data can be visualized at
 [Mako.dev](https://mako.dev/benchmark?benchmark_key=4790091207671808)
+
+To run these tests, execute [`./scripts/performance.sh <--dev> <#publishes>` ](../../scripts/performance.sh) from the root fo the project. This will also require the setting of env `GOOGLE_APPLICATION_CREDENTIALS`, if this is needed contact @chaodaiG or @MushuEE.
+
+The `--dev` option will run the [development](#Dev) version of the test.
+
+## Production
+This uses the production [banchmark_config](./export_test_benchmark.config) and writes results to a [Mako](https://mako.dev/benchmark?benchmark_key=4790091207671808) instance with the provided key.
+
+## Dev
+This uses the development [banchmark_config](./export_test_benchmark.dev.config) and writes results to a [Mako](https://mako.dev/benchmark?benchmark_key=5698752440434688) instance with the provided key. You may want to set the number of Publishes within this test for more keys, set that from the the second argument to [performance.sh](../../scripts/performance.sh). Ex. `./scripts/performance.sh --dev 2000`. \**defaults to 1000 for dev and 100000 for prod*
\ No newline at end of file
diff --git a/internal/performance/helper.go b/internal/performance/helper.go
index 2b4504d..ca6a48c 100644
--- a/internal/performance/helper.go
+++ b/internal/performance/helper.go
@@ -36,23 +36,39 @@ import (
 )
 
 const (
-	benchmarkConfigFile = "export_test_benchmark.config"
+	benchmarkProdConfigFile = "export_test_benchmark.config"
+	benchmarkDevConfigFile  = "export_test_benchmark.dev.config"
 )
 
-type config struct {
+type makoConfig struct {
 	MakoPort uint `env:"MAKO_PORT,required"`
+	Dev      bool `env:"PERFORMANCE_DEV"`
 }
 
 // setup sets up client used for performance test
 func setup(tb testing.TB) (*quickstore.Quickstore, func(context.Context)) {
-	ctx := context.Background()
+	var (
+		ctx  = context.Background()
+		data []byte
+		err  error
+	)
 	ctxWithTimeout, cancel := context.WithTimeout(ctx, 10*time.Second)
 	defer cancel()
 
+	c := makoConfig{}
+	if err := envconfig.ProcessWith(context.Background(), &c, envconfig.OsLookuper()); err != nil {
+		tb.Fatalf("unable to process env: %v", err)
+	}
+	microservice := fmt.Sprintf("localhost:%d", c.MakoPort)
+
 	benchmarkConfig := &mpb.BenchmarkInfo{}
-	data, err := ioutil.ReadFile(benchmarkConfigFile)
+	if c.Dev {
+		data, err = ioutil.ReadFile(benchmarkDevConfigFile)
+	} else {
+		data, err = ioutil.ReadFile(benchmarkProdConfigFile)
+	}
 	if err != nil {
-		tb.Fatal(err)
+		tb.Fatalf("unable to read benchmark config: %v", err)
 	}
 	if err = proto.UnmarshalText(string(data), benchmarkConfig); err != nil {
 		tb.Fatal(err)
@@ -61,12 +77,6 @@ func setup(tb testing.TB) (*quickstore.Quickstore, func(context.Context)) {
 		BenchmarkKey: proto.String(*benchmarkConfig.BenchmarkKey),
 	}
 
-	c := config{}
-	if err := envconfig.ProcessWith(context.Background(), &c, envconfig.OsLookuper()); err != nil {
-		tb.Fatalf("unable to process env: %v", err)
-	}
-	microservice := fmt.Sprintf("localhost:%d", c.MakoPort)
-
 	q, close, err := quickstore.NewAtAddress(ctxWithTimeout, input, microservice)
 	if err != nil {
 		tb.Fatalf("quickstore.NewAtAddress() = %v", err)
diff --git a/internal/publish/config.go b/internal/publish/config.go
index a858cd5..40b0898 100644
--- a/internal/publish/config.go
+++ b/internal/publish/config.go
@@ -20,12 +20,12 @@ import (
 
 	"github.com/google/exposure-notifications-server/internal/authorizedapp"
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/publish/model"
 	"github.com/google/exposure-notifications-server/internal/revision"
 	"github.com/google/exposure-notifications-server/internal/setup"
 	"github.com/google/exposure-notifications-server/internal/verification"
 	"github.com/google/exposure-notifications-server/pkg/keys"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 )
 
diff --git a/internal/publish/database/exposure.go b/internal/publish/database/exposure.go
index 30257bd..1737b3d 100644
--- a/internal/publish/database/exposure.go
+++ b/internal/publish/database/exposure.go
@@ -26,11 +26,11 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/pb"
 	"github.com/google/exposure-notifications-server/internal/publish/model"
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1alpha1"
 	"github.com/google/exposure-notifications-server/pkg/base64util"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 
 	pgx "github.com/jackc/pgx/v4"
 )
diff --git a/internal/publish/model/exposure_model.go b/internal/publish/model/exposure_model.go
index d8a325f..0aab4d6 100644
--- a/internal/publish/model/exposure_model.go
+++ b/internal/publish/model/exposure_model.go
@@ -24,9 +24,9 @@ import (
 	"strings"
 	"time"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/verification"
 	"github.com/google/exposure-notifications-server/pkg/base64util"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1"
 )
@@ -53,7 +53,7 @@ type Exposure struct {
 	LocalProvenance  bool
 	FederationSyncID int64
 
-	// These fileds are nullable to maintain backwards compatibility with
+	// These fields are nullable to maintain backwards compatibility with
 	// older versions that predate their existence.
 	HealthAuthorityID     *int64
 	ReportType            string
diff --git a/internal/publish/publish.go b/internal/publish/publish.go
index 94fbb8c..fbea2ef 100644
--- a/internal/publish/publish.go
+++ b/internal/publish/publish.go
@@ -26,7 +26,6 @@ import (
 	"go.opencensus.io/trace"
 
 	"github.com/google/exposure-notifications-server/internal/authorizedapp"
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/pb"
 	"github.com/google/exposure-notifications-server/internal/publish/database"
 	"github.com/google/exposure-notifications-server/internal/publish/model"
@@ -37,6 +36,7 @@ import (
 	verifydb "github.com/google/exposure-notifications-server/internal/verification/database"
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1"
 	"github.com/google/exposure-notifications-server/pkg/base64util"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/mikehelmick/go-chaff"
 )
 
@@ -72,7 +72,7 @@ func NewHandler(ctx context.Context, config *Config, env *serverenv.ServerEnv) (
 
 	aadBytes := config.RevisionToken.AAD
 	if len(aadBytes) == 0 {
-		return nil, fmt.Errorf("must provide ADD for revision token encryption in REVISION_TOKEN_AAD env variable: %w", err)
+		return nil, fmt.Errorf("must provide ADD for revision token encryption in REVISION_TOKEN_AAD env variable")
 	}
 	revisionKeyConfig := revisiondb.KMSConfig{
 		WrapperKeyID: config.RevisionToken.KeyID,
@@ -249,6 +249,7 @@ func (h *PublishHandler) process(ctx context.Context, data *verifyapi.Publish, b
 
 	// Examine the revision token. It is expected that it is missing in most cases.
 	var token *pb.RevisionTokenData
+	decryptFail := false
 	if len(data.RevisionToken) != 0 {
 		encryptedToken, err := base64util.DecodeString(data.RevisionToken)
 		if err != nil {
@@ -256,8 +257,9 @@ func (h *PublishHandler) process(ctx context.Context, data *verifyapi.Publish, b
 		} else {
 			token, err = h.tokenManager.UnmarshalRevisionToken(ctx, encryptedToken, h.tokenAAD)
 			if err != nil {
-				logger.Errorf("unable to unmarshall / descrypt revision token: %v", err)
+				logger.Errorf("unable to unmarshall / decrypt revision token: %v", err)
 				token = nil // just in case.
+				decryptFail = true
 			}
 		}
 	}
@@ -283,14 +285,14 @@ func (h *PublishHandler) process(ctx context.Context, data *verifyapi.Publish, b
 		var logMessage, errorMessage, errorCode string
 		metric := "publish-revision-token-issue"
 		switch {
+		case decryptFail || errors.Is(err, database.ErrExistingKeyNotInToken) || errors.Is(err, database.ErrRevisionTokenMetadataMismatch):
+			logMessage = fmt.Sprintf("revision token present, but invalid: %v", err)
+			errorMessage = "revision token is invalid"
+			errorCode = verifyapi.ErrorInvalidRevisionToken
 		case errors.Is(err, database.ErrNoRevisionToken):
 			logMessage = "no revision token"
 			errorMessage = "no revision token, but sent existing keys"
 			errorCode = verifyapi.ErrorMissingRevisionToken
-		case errors.Is(err, database.ErrExistingKeyNotInToken) || errors.Is(err, database.ErrRevisionTokenMetadataMismatch):
-			logMessage = fmt.Sprintf("revision token present, but invalid: %v", err)
-			errorMessage = "revision token is invalid"
-			errorCode = verifyapi.ErrorInvalidRevisionToken
 		default:
 			logMessage = fmt.Sprintf("error writing exposure record: %v", err)
 			errorMessage = http.StatusText(http.StatusInternalServerError)
diff --git a/internal/revision/database/revision.go b/internal/revision/database/revision.go
index a499d17..deac671 100644
--- a/internal/revision/database/revision.go
+++ b/internal/revision/database/revision.go
@@ -34,8 +34,8 @@ import (
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/pkg/keys"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/jackc/pgx/v4"
 )
 
diff --git a/internal/revision/revision.go b/internal/revision/revision.go
index 0399352..f7fcb44 100644
--- a/internal/revision/revision.go
+++ b/internal/revision/revision.go
@@ -28,10 +28,10 @@ import (
 	"sync"
 	"time"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/pb"
 	"github.com/google/exposure-notifications-server/internal/publish/model"
 	"github.com/google/exposure-notifications-server/internal/revision/database"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"google.golang.org/protobuf/proto"
 )
 
diff --git a/internal/serverenv/env.go b/internal/serverenv/env.go
index 36cabbe..caf86f8 100644
--- a/internal/serverenv/env.go
+++ b/internal/serverenv/env.go
@@ -23,9 +23,9 @@ import (
 	"github.com/google/exposure-notifications-server/internal/authorizedapp"
 	"github.com/google/exposure-notifications-server/internal/database"
 	"github.com/google/exposure-notifications-server/internal/metrics"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/storage"
 	"github.com/google/exposure-notifications-server/pkg/keys"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 )
 
@@ -178,7 +178,9 @@ func (s *ServerEnv) Close(ctx context.Context) error {
 	}
 
 	if s.observabilityExporter != nil {
-		s.observabilityExporter.Flush()
+		if err := s.observabilityExporter.Close(); err != nil {
+			return nil
+		}
 	}
 
 	return nil
diff --git a/internal/setup/setup.go b/internal/setup/setup.go
index 81f9ab4..219199a 100644
--- a/internal/setup/setup.go
+++ b/internal/setup/setup.go
@@ -21,12 +21,12 @@ import (
 
 	"github.com/google/exposure-notifications-server/internal/authorizedapp"
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/logging"
 	"github.com/google/exposure-notifications-server/internal/metrics"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/serverenv"
 	"github.com/google/exposure-notifications-server/internal/storage"
 	"github.com/google/exposure-notifications-server/pkg/keys"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 	"github.com/sethvargo/go-envconfig"
 )
@@ -170,18 +170,21 @@ func SetupWith(ctx context.Context, config interface{}, l envconfig.Lookuper) (*
 	// Configure and initialize the observability exporter.
 	if provider, ok := config.(ObservabilityExporterConfigProvider); ok {
 		logger.Info("configuring observability exporter")
+
 		oeConfig := provider.ObservabilityExporterConfig()
 		oe, err := observability.NewFromEnv(ctx, oeConfig)
 		if err != nil {
-			return nil, fmt.Errorf("unable to create ObservabilityExporter provider: %v", err)
+			return nil, fmt.Errorf("unable to create observability provider: %v", err)
 		}
-		if err := oe.InitExportOnce(); err != nil {
-			return nil, fmt.Errorf("error initializing observability exporter: %v", err)
+		if err := oe.StartExporter(); err != nil {
+			return nil, fmt.Errorf("failed to start observability: %w", err)
 		}
+		exporter := serverenv.WithObservabilityExporter(oe)
 
-		serverEnvOpts = append(serverEnvOpts, serverenv.WithObservabilityExporter(oe))
+		// Update serverEnv setup.
+		serverEnvOpts = append(serverEnvOpts, exporter)
 
-		logger.Infow("observability exporter", "config", oeConfig)
+		logger.Infow("observability", "config", oeConfig)
 	}
 
 	// Configure blob storage.
diff --git a/pkg/api/v1/exposure_types.go b/pkg/api/v1/exposure_types.go
index 4650ee4..766263c 100644
--- a/pkg/api/v1/exposure_types.go
+++ b/pkg/api/v1/exposure_types.go
@@ -62,7 +62,7 @@ const (
 //  - Android: The App Package AppPackageName
 //  - iOS: The BundleID
 // verificationPayload: The Verification Certificate from a verification server.
-// hmacKey: the device generated secret that is used to recalcualte the HMAC value
+// hmacKey: the device generated secret that is used to recalculate the HMAC value
 //  that is present in the verification payload.
 //
 // symptomOnsetInterval: An interval number that aligns with the symptom onset date.
@@ -77,8 +77,8 @@ const (
 //  keys of a "traveler" who has left the home region represented by this server
 //  (or by the home health authority in case of a multi-tenant installation).
 //
-// revisionToken: An opaque string that must be passed in-tact from on additional
-//   publish requests from the same device, there the same TEKs may be published
+// revisionToken: An opaque string that must be passed intact on additional
+//   publish requests from the same device, where the same TEKs may be published
 //   again.
 //
 // Padding: random base64 encoded data to obscure the request size. The server will
diff --git a/pkg/api/v1alpha1/exposure_types.go b/pkg/api/v1alpha1/exposure_types.go
index 8b492f7..b797c50 100644
--- a/pkg/api/v1alpha1/exposure_types.go
+++ b/pkg/api/v1alpha1/exposure_types.go
@@ -45,7 +45,7 @@ const (
 //  - Android: The App Package AppPackageName
 //  - iOS: The BundleID
 // VerifcationPayload: The Verification Certificate from a verification server.
-// HMACKey: the device generated secret that is used to recalcualte the HMAC value
+// HMACKey: the device generated secret that is used to recalculate the HMAC value
 //  that is present in the verification payload.
 //
 // SymptomOnsetInterval: An interval number that aligns with the symptom onset date.
@@ -56,8 +56,8 @@ const (
 //  - Does not have to be within range of any of the provided keys (i.e. future
 //    key uploads)
 //
-// revisionToken: An opaque string that must be passed in-tact from on additional
-//   publish requests from the same device, there the same TEKs may be published
+// revisionToken: An opaque string that must be passed intact on additional
+//   publish requests from the same device, where the same TEKs may be published
 //   again.
 //
 // Padding: random base64 encoded data to obscure the request size.
diff --git a/pkg/keys/in_memory.go b/pkg/keys/in_memory.go
index 47498ea..5b43ae3 100644
--- a/pkg/keys/in_memory.go
+++ b/pkg/keys/in_memory.go
@@ -27,6 +27,12 @@ import (
 	"sync"
 )
 
+var _ KeyManager = (*InMemory)(nil)
+var _ SigningKeyCreator = (*InMemory)(nil)
+var _ SigningKeyAdder = (*InMemory)(nil)
+var _ EncryptionKeyCreator = (*InMemory)(nil)
+var _ EncryptionKeyAdder = (*InMemory)(nil)
+
 // InMemory is useful for testing. Do NOT use in a running system as all
 // keys are only kept in memory and will be lost across server reboots.
 type InMemory struct {
@@ -43,42 +49,68 @@ func NewInMemory(ctx context.Context) (*InMemory, error) {
 	}, nil
 }
 
-// AddSigningKey generates a new ECDSA P256 Signing Key identified by
+// CreateSigningKey generates a new ECDSA P256 Signing Key identified by
 // the provided keyID
-func (k *InMemory) AddSigningKey(keyID string) error {
+func (k *InMemory) CreateSigningKey(keyID string) (*ecdsa.PrivateKey, error) {
 	k.mu.Lock()
 	defer k.mu.Unlock()
 
 	if _, ok := k.signingKeys[keyID]; ok {
-		return fmt.Errorf("key already exists: %v", keyID)
+		return nil, fmt.Errorf("key already exists: %v", keyID)
 	}
 
 	pk, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 	if err != nil {
-		return fmt.Errorf("unable to generate private key: %w", err)
+		return nil, fmt.Errorf("unable to generate private key: %w", err)
+	}
+
+	k.signingKeys[keyID] = pk
+	return pk, nil
+}
+
+// AddSigningKey adds a new ECDSA P256 Signing Key identified by the provided
+// keyID.
+func (k *InMemory) AddSigningKey(keyID string, pk *ecdsa.PrivateKey) error {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+
+	if _, ok := k.signingKeys[keyID]; ok {
+		return fmt.Errorf("key already exists: %v", keyID)
 	}
 
 	k.signingKeys[keyID] = pk
 	return nil
 }
 
-// AddEncryptionKey generates a new encryption key identified by
-// the provided keyID.
-func (k *InMemory) AddEncryptionKey(keyID string) error {
+// CreateEncryptionKey generates and stores new encryption key identified by the
+// provided keyID.
+func (k *InMemory) CreateEncryptionKey(keyID string) ([]byte, error) {
 	k.mu.Lock()
 	defer k.mu.Unlock()
 
 	if _, ok := k.cryptoKeys[keyID]; ok {
-		return fmt.Errorf("key already exists: %v", keyID)
+		return nil, fmt.Errorf("key already exists: %v", keyID)
 	}
 
 	key := make([]byte, 32)
 	if _, err := io.ReadFull(rand.Reader, key); err != nil {
-		return fmt.Errorf("failed to read random bytes: %w", err)
+		return nil, fmt.Errorf("failed to read random bytes: %w", err)
 	}
 
 	k.cryptoKeys[keyID] = key
+	return key, nil
+}
+
+// AddEncryptionKey stores the key on the system.
+func (k *InMemory) AddEncryptionKey(keyID string, key []byte) error {
+	k.mu.Lock()
+	defer k.mu.Unlock()
 
+	if _, ok := k.cryptoKeys[keyID]; ok {
+		return fmt.Errorf("key already exists: %v", keyID)
+	}
+
+	k.cryptoKeys[keyID] = key
 	return nil
 }
 
diff --git a/pkg/keys/keys.go b/pkg/keys/keys.go
index 1ae1e93..d8bdf15 100644
--- a/pkg/keys/keys.go
+++ b/pkg/keys/keys.go
@@ -23,6 +23,7 @@ package keys
 import (
 	"context"
 	"crypto"
+	"crypto/ecdsa"
 	"fmt"
 )
 
@@ -48,6 +49,26 @@ type KeyManager interface {
 	Decrypt(ctx context.Context, keyID string, ciphertext []byte, aad []byte) ([]byte, error)
 }
 
+// EncryptionKeyCreator supports creating encryption keys.
+type EncryptionKeyCreator interface {
+	CreateEncryptionKey(string) ([]byte, error)
+}
+
+// EncryptionKeyAdder supports creating encryption keys.
+type EncryptionKeyAdder interface {
+	AddEncryptionKey(string, []byte) error
+}
+
+// SigningKeyCreator supports creating signing keys.
+type SigningKeyCreator interface {
+	CreateSigningKey(string) (*ecdsa.PrivateKey, error)
+}
+
+// SigningKeyAdder supports creating signing keys.
+type SigningKeyAdder interface {
+	AddSigningKey(string, *ecdsa.PrivateKey) error
+}
+
 // KeyManagerFor returns the appropriate key manager for the given type.
 func KeyManagerFor(ctx context.Context, typ KeyManagerType) (KeyManager, error) {
 	switch typ {
diff --git a/pkg/logging/logger.go b/pkg/logging/logger.go
new file mode 100644
index 0000000..fbc9877
--- /dev/null
+++ b/pkg/logging/logger.go
@@ -0,0 +1,156 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package logging sets up and configures logging.
+package logging
+
+import (
+	"context"
+	"sync"
+	"time"
+
+	"go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
+)
+
+// contextKey is a private string type to prevent collisions in the context map.
+type contextKey string
+
+// loggerKey points to the value in the context where the logger is stored.
+const loggerKey = contextKey("logger")
+
+var (
+	// defaultLogger is the default logger. It is initialized once per package
+	// include upon calling DefaultLogger.
+	defaultLogger     *zap.SugaredLogger
+	defaultLoggerOnce sync.Once
+)
+
+// NewLogger creates a new logger with the given configuration.
+func NewLogger(debug bool) *zap.SugaredLogger {
+	config := &zap.Config{
+		Level:            zap.NewAtomicLevelAt(zap.InfoLevel),
+		Development:      false,
+		Sampling:         samplingConfig,
+		Encoding:         encodingJSON,
+		EncoderConfig:    encoderConfig,
+		OutputPaths:      outputStderr,
+		ErrorOutputPaths: outputStderr,
+	}
+
+	// Add more details if logging is in debug mode.
+	if debug {
+		config.Level = zap.NewAtomicLevelAt(zap.DebugLevel)
+		config.Development = true
+		config.Sampling = nil
+	}
+
+	logger, err := config.Build()
+	if err != nil {
+		logger = zap.NewNop()
+	}
+
+	return logger.Sugar()
+}
+
+// DefaultLogger returns the default logger for the package.
+func DefaultLogger() *zap.SugaredLogger {
+	defaultLoggerOnce.Do(func() {
+		defaultLogger = NewLogger(false)
+	})
+	return defaultLogger
+}
+
+// WithLogger creates a new context with the provided logger attached.
+func WithLogger(ctx context.Context, logger *zap.SugaredLogger) context.Context {
+	return context.WithValue(ctx, loggerKey, logger)
+}
+
+// FromContext returns the logger stored in the context. If no such logger
+// exists, a default logger is returned.
+func FromContext(ctx context.Context) *zap.SugaredLogger {
+	if logger, ok := ctx.Value(loggerKey).(*zap.SugaredLogger); ok {
+		return logger
+	}
+	return DefaultLogger()
+}
+
+const (
+	timestamp  = "timestamp"
+	severity   = "severity"
+	logger     = "logger"
+	caller     = "caller"
+	message    = "message"
+	stacktrace = "stacktrace"
+
+	levelDebug     = "DEBUG"
+	levelInfo      = "INFO"
+	levelWarning   = "WARNING"
+	levelError     = "ERROR"
+	levelCritical  = "CRITICAL"
+	levelAlert     = "ALERT"
+	levelEmergency = "EMERGENCY"
+
+	encodingJSON = "json"
+)
+
+var outputStderr = []string{"stderr"}
+
+var encoderConfig = zapcore.EncoderConfig{
+	TimeKey:        timestamp,
+	LevelKey:       severity,
+	NameKey:        logger,
+	CallerKey:      caller,
+	MessageKey:     message,
+	StacktraceKey:  stacktrace,
+	LineEnding:     zapcore.DefaultLineEnding,
+	EncodeLevel:    levelEncoder(),
+	EncodeTime:     timeEncoder(),
+	EncodeDuration: zapcore.SecondsDurationEncoder,
+	EncodeCaller:   zapcore.ShortCallerEncoder,
+}
+
+var samplingConfig = &zap.SamplingConfig{
+	Initial:    250,
+	Thereafter: 250,
+}
+
+// levelEncoder transforms a zap level to the associated stackdriver level.
+func levelEncoder() zapcore.LevelEncoder {
+	return func(l zapcore.Level, enc zapcore.PrimitiveArrayEncoder) {
+		switch l {
+		case zapcore.DebugLevel:
+			enc.AppendString(levelDebug)
+		case zapcore.InfoLevel:
+			enc.AppendString(levelInfo)
+		case zapcore.WarnLevel:
+			enc.AppendString(levelWarning)
+		case zapcore.ErrorLevel:
+			enc.AppendString(levelError)
+		case zapcore.DPanicLevel:
+			enc.AppendString(levelCritical)
+		case zapcore.PanicLevel:
+			enc.AppendString(levelAlert)
+		case zapcore.FatalLevel:
+			enc.AppendString(levelEmergency)
+		}
+	}
+}
+
+// timeEncoder encodes the time as RFC3339 nano
+func timeEncoder() zapcore.TimeEncoder {
+	return func(t time.Time, enc zapcore.PrimitiveArrayEncoder) {
+		enc.AppendString(t.Format(time.RFC3339Nano))
+	}
+}
diff --git a/internal/observability/config.go b/pkg/observability/config.go
similarity index 76%
rename from internal/observability/config.go
rename to pkg/observability/config.go
index 316d477..cd39a3e 100644
--- a/internal/observability/config.go
+++ b/pkg/observability/config.go
@@ -29,30 +29,31 @@ const (
 type Config struct {
 	ExporterType ExporterType `env:"OBSERVABILITY_EXPORTER, default=STACKDRIVER"`
 
-	OpenCensusConfig
-	StackdriverConfig
-	OCAgentConfig
+	OpenCensus  *OpenCensusConfig
+	Stackdriver *StackdriverConfig
 }
 
 // OpenCensusConfig holds the configuration options for the open census exporter
 type OpenCensusConfig struct {
-	TraceProbabilitySampleRate float64 `env:"TRACE_PROBABILITY, default=0.40"`
+	SampleRate float64 `env:"TRACE_PROBABILITY, default=0.40"`
+
+	Insecure bool   `env:"OCAGENT_INSECURE"`
+	Endpoint string `env:"OCAGENT_TRACE_EXPORTER_ENDPOINT"`
 }
 
 // StackdriverConfig holds the configuration options for the stackdriver exporter
 type StackdriverConfig struct {
-	ProjectID string `env:"PROJECT_ID, default=$GOOGLE_CLOUD_PROJECT"`
-	// Knative+Cloud Run container contract env vars
-	// https://cloud.google.com/run/docs/reference/container-contract#env-vars
-	// If present, can be used to configured the Stackdriver MonitoredResource correctly.
-	Service  string `env:"K_SERVICE"`
-	Revision string `env:"K_REVISION"`
+	SampleRate float64 `env:"TRACE_PROBABILITY, default=0.40"`
 
-	Namespace string `env:"K_CONFIGURATION, default=ens"`
-}
+	ProjectID string `env:"PROJECT_ID, default=$GOOGLE_CLOUD_PROJECT"`
 
-// OCAgentConfig holds the configuration options for the default opencensus exporter
-type OCAgentConfig struct {
-	Insecure bool   `env:"OCAGENT_INSECURE, default=true"`
-	Endpoint string `env:"OCAGENT_TRACE_EXPORTER_ENDPOINT"`
+	// Knative+Cloud Run container contract envvars:
+	//
+	// https://cloud.google.com/run/docs/reference/container-contract#env-vars
+	//
+	// If present, can be used to configured the Stackdriver MonitoredResource
+	// correctly.
+	Service   string `env:"K_SERVICE"`
+	Revision  string `env:"K_REVISION"`
+	Namespace string `env:"K_CONFIGURATION, default=en"`
 }
diff --git a/internal/observability/noop.go b/pkg/observability/noop.go
similarity index 69%
rename from internal/observability/noop.go
rename to pkg/observability/noop.go
index c66633d..93a4e63 100644
--- a/internal/observability/noop.go
+++ b/pkg/observability/noop.go
@@ -15,14 +15,22 @@
 // Package observability sets up and configures observability tools.
 package observability
 
+import "context"
+
 // Compile-time check to verify implements interface.
-var _ Exporter = (*NoopExporter)(nil)
+var _ Exporter = (*noopExporter)(nil)
+
+// noopExporter is an observability exporter that does nothing.
+type noopExporter struct{}
 
-// NoopExporter is an observability exporter that does nothing.
-type NoopExporter struct{}
+func NewNoop(_ context.Context) (Exporter, error) {
+	return &noopExporter{}, nil
+}
 
-func (g *NoopExporter) InitExportOnce() error {
+func (g *noopExporter) StartExporter() error {
 	return nil
 }
 
-func (g *NoopExporter) Flush() {}
+func (g *noopExporter) Close() error {
+	return nil
+}
diff --git a/internal/observability/generic.go b/pkg/observability/observability.go
similarity index 52%
rename from internal/observability/generic.go
rename to pkg/observability/observability.go
index 95102bd..425d907 100644
--- a/internal/observability/generic.go
+++ b/pkg/observability/observability.go
@@ -16,66 +16,50 @@
 package observability
 
 import (
+	"context"
 	"fmt"
-	"sync"
+	"io"
 
 	"go.opencensus.io/plugin/ocgrpc"
 	"go.opencensus.io/plugin/ochttp"
 	"go.opencensus.io/stats/view"
-	"go.opencensus.io/trace"
 )
 
-// Compile-time check to verify implements interface.
-var _ Exporter = (*GenericExporter)(nil)
-
-var initExporterOnce sync.Once
-
-type FlushFn func()
-
-type traceAndViewExporter interface {
-	trace.Exporter
-	view.Exporter
-}
-
-// GenericExporter is a standard implementation of an exporter that wraps the opencensus interfaces
-// with custom configuration
-type GenericExporter struct {
-	exporter   traceAndViewExporter
-	sampleRate float64
-	flushFn    FlushFn
-}
-
-func (g *GenericExporter) InitExportOnce() error {
-	var err error
-	initExporterOnce.Do(func() {
-		err = g.initExporter()
-	})
-	return err
-}
-
-func (g *GenericExporter) Flush() {
-	g.flushFn()
+// Exporter defines the minimum shared functionality for an observability exporter
+// used by this application.
+type Exporter interface {
+	io.Closer
+	StartExporter() error
 }
 
-func (g *GenericExporter) initExporter() error {
-	if g.exporter == nil {
-		return nil
+// NewFromEnv returns the observability exporter given the provided configuration, or an error
+// if it failed to be created.
+func NewFromEnv(ctx context.Context, config *Config) (Exporter, error) {
+	switch config.ExporterType {
+	case ExporterNoop:
+		return NewNoop(ctx)
+	case ExporterStackdriver:
+		return NewStackdriver(ctx, config.Stackdriver)
+	case ExporterOCAgent, ExporterPrometheus:
+		return NewOpenCensus(ctx, config.OpenCensus)
+	default:
+		return nil, fmt.Errorf("unknown observability exporter type %v", config.ExporterType)
 	}
-	trace.ApplyConfig(trace.Config{
-		DefaultSampler: trace.ProbabilitySampler(g.sampleRate),
-	})
-	trace.RegisterExporter(g.exporter)
-	view.RegisterExporter(g.exporter)
+}
 
+// registerViews registers the necessary tracing views.
+func registerViews() error {
 	// Record the various HTTP view to collect metrics.
 	httpViews := append(ochttp.DefaultServerViews, ochttp.DefaultClientViews...)
 	if err := view.Register(httpViews...); err != nil {
-		return fmt.Errorf("failed to register http views for observability exporter: %v", err)
+		return fmt.Errorf("failed to register http views: %w", err)
 	}
+
 	// Register the various gRPC views to collect metrics.
 	gRPCViews := append(ocgrpc.DefaultServerViews, ocgrpc.DefaultClientViews...)
 	if err := view.Register(gRPCViews...); err != nil {
-		return fmt.Errorf("failed to register grpc views for observability exporter: %v", err)
+		return fmt.Errorf("failed to register grpc views: %w", err)
 	}
+
 	return nil
 }
diff --git a/pkg/observability/opencensus.go b/pkg/observability/opencensus.go
new file mode 100644
index 0000000..24ce19c
--- /dev/null
+++ b/pkg/observability/opencensus.go
@@ -0,0 +1,76 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package observability
+
+import (
+	"context"
+	"fmt"
+
+	"contrib.go.opencensus.io/exporter/ocagent"
+	"go.opencensus.io/stats/view"
+	"go.opencensus.io/trace"
+)
+
+var _ Exporter = (*opencensusExporter)(nil)
+
+type opencensusExporter struct {
+	exporter *ocagent.Exporter
+	config   *OpenCensusConfig
+}
+
+// NewOpenCensus creates a new metrics and trace exporter for OpenCensus.
+func NewOpenCensus(ctx context.Context, config *OpenCensusConfig) (Exporter, error) {
+	var opts []ocagent.ExporterOption
+	if config.Insecure {
+		opts = append(opts, ocagent.WithInsecure())
+	}
+	if config.Endpoint != "" {
+		opts = append(opts, ocagent.WithAddress(config.Endpoint))
+	}
+
+	oc, err := ocagent.NewExporter(opts...)
+	if err != nil {
+		return nil, fmt.Errorf("failed to create opencensus exporter: %w", err)
+	}
+	return &opencensusExporter{oc, config}, nil
+}
+
+// StartExporter starts the exporter.
+func (e *opencensusExporter) StartExporter() error {
+	trace.ApplyConfig(trace.Config{
+		DefaultSampler: trace.ProbabilitySampler(e.config.SampleRate),
+	})
+	trace.RegisterExporter(e.exporter)
+	view.RegisterExporter(e.exporter)
+
+	if err := registerViews(); err != nil {
+		return fmt.Errorf("failed to start opencensus exporter: %w", err)
+	}
+
+	return nil
+}
+
+// Close halts the exporter.
+func (e *opencensusExporter) Close() error {
+	if err := e.exporter.Stop(); err != nil {
+		return fmt.Errorf("failed to stop exporter: %w", err)
+	}
+
+	// Unregister the exporter
+	trace.UnregisterExporter(e.exporter)
+	view.UnregisterExporter(e.exporter)
+
+	return nil
+}
diff --git a/internal/observability/resource.go b/pkg/observability/resource.go
similarity index 90%
rename from internal/observability/resource.go
rename to pkg/observability/resource.go
index 46e297a..c195c78 100644
--- a/internal/observability/resource.go
+++ b/pkg/observability/resource.go
@@ -22,9 +22,9 @@ import (
 	"github.com/google/uuid"
 )
 
-var _ monitoredresource.Interface = (*StackdriverMonitoredResoruce)(nil)
+var _ monitoredresource.Interface = (*stackdriverMonitoredResource)(nil)
 
-type StackdriverMonitoredResoruce struct {
+type stackdriverMonitoredResource struct {
 	resource string
 	labels   map[string]string
 }
@@ -65,12 +65,12 @@ func NewStackdriverMonitoredResoruce(c *StackdriverConfig) monitoredresource.Int
 	}
 	labels["namespace"] = c.Namespace
 
-	return &StackdriverMonitoredResoruce{
+	return &stackdriverMonitoredResource{
 		resource: resource,
 		labels:   labels,
 	}
 }
 
-func (s *StackdriverMonitoredResoruce) MonitoredResource() (string, map[string]string) {
+func (s *stackdriverMonitoredResource) MonitoredResource() (string, map[string]string) {
 	return s.resource, s.labels
 }
diff --git a/pkg/observability/stackdriver.go b/pkg/observability/stackdriver.go
new file mode 100644
index 0000000..b091a0b
--- /dev/null
+++ b/pkg/observability/stackdriver.go
@@ -0,0 +1,90 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package observability sets up and configures observability tools.
+package observability
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"contrib.go.opencensus.io/exporter/stackdriver"
+	"github.com/google/exposure-notifications-server/pkg/logging"
+	"go.opencensus.io/stats/view"
+	"go.opencensus.io/trace"
+)
+
+var _ Exporter = (*stackdriverExporter)(nil)
+
+type stackdriverExporter struct {
+	exporter *stackdriver.Exporter
+	config   *StackdriverConfig
+}
+
+// NewStackdriver creates a new metrics and trace exporter for Stackdriver.
+func NewStackdriver(ctx context.Context, config *StackdriverConfig) (Exporter, error) {
+	logger := logging.FromContext(ctx).Named("stackdriver")
+
+	projectID := config.ProjectID
+	if projectID == "" {
+		return nil, fmt.Errorf("missing PROJECT_ID in Stackdriver exporter")
+	}
+
+	monitoredResource := NewStackdriverMonitoredResoruce(config)
+
+	exporter, err := stackdriver.NewExporter(stackdriver.Options{
+		ProjectID:         projectID,
+		ReportingInterval: time.Minute, // stackdriver export interval minimum
+		MonitoredResource: monitoredResource,
+		OnError: func(err error) {
+			logger.Errorw("failed to export metric", "error", err)
+		},
+	})
+
+	if err != nil {
+		return nil, fmt.Errorf("failed to create Stackdriver exporter: %w", err)
+	}
+	return &stackdriverExporter{exporter, config}, nil
+}
+
+// StartExporter starts the exporter.
+func (e *stackdriverExporter) StartExporter() error {
+	if err := e.exporter.StartMetricsExporter(); err != nil {
+		return fmt.Errorf("failed to start stackdriver exporter: %w", err)
+	}
+
+	trace.ApplyConfig(trace.Config{
+		DefaultSampler: trace.ProbabilitySampler(e.config.SampleRate),
+	})
+	trace.RegisterExporter(e.exporter)
+	view.RegisterExporter(e.exporter)
+
+	return nil
+}
+
+// Close halts the exporter.
+func (e *stackdriverExporter) Close() error {
+	// Flush any existing metrics
+	e.exporter.Flush()
+
+	// Stop the exporter
+	e.exporter.StopMetricsExporter()
+
+	// Unregister the exporter
+	trace.UnregisterExporter(e.exporter)
+	view.UnregisterExporter(e.exporter)
+
+	return nil
+}
diff --git a/pkg/secrets/resolver.go b/pkg/secrets/resolver.go
index 24577a0..4bd5496 100644
--- a/pkg/secrets/resolver.go
+++ b/pkg/secrets/resolver.go
@@ -23,7 +23,7 @@ import (
 	"path/filepath"
 	"strings"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"github.com/sethvargo/go-envconfig"
 )
 
diff --git a/pkg/server/metrics.go b/pkg/server/metrics.go
index 073bd52..27ec976 100644
--- a/pkg/server/metrics.go
+++ b/pkg/server/metrics.go
@@ -23,7 +23,7 @@ import (
 	"strings"
 
 	"contrib.go.opencensus.io/exporter/prometheus"
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 )
 
 // ServeMetricsIfPrometheus serves the opencensus metrics at /metrics when OBSERVABILITY_EXPORTER set to "prometheus"
diff --git a/pkg/server/server.go b/pkg/server/server.go
index 9ae7d2f..0505a77 100644
--- a/pkg/server/server.go
+++ b/pkg/server/server.go
@@ -28,7 +28,7 @@ import (
 	"strconv"
 	"time"
 
-	"github.com/google/exposure-notifications-server/internal/logging"
+	"github.com/google/exposure-notifications-server/pkg/logging"
 	"go.opencensus.io/plugin/ochttp"
 	"google.golang.org/grpc"
 )
diff --git a/scripts/performance.sh b/scripts/performance.sh
index 9e6f15e..1271706 100755
--- a/scripts/performance.sh
+++ b/scripts/performance.sh
@@ -14,6 +14,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+if [[ "$1" == "--dev" || "$1" == "-d" ]]; then
+ export PERFORMANCE_DEV=1
+ export PUBLISHES=${2:-1000}
+fi
+
 set -eEuo pipefail
 
 echo "🌳 Set up environment variables"
diff --git a/tools/export-analyzer/main.go b/tools/export-analyzer/main.go
index 2c49341..d7c7ce5 100644
--- a/tools/export-analyzer/main.go
+++ b/tools/export-analyzer/main.go
@@ -20,12 +20,16 @@ import (
 	"flag"
 	"io/ioutil"
 	"log"
+	"time"
 
 	"github.com/google/exposure-notifications-server/internal/export"
+	exportpb "github.com/google/exposure-notifications-server/internal/pb/export"
+	"github.com/google/exposure-notifications-server/internal/publish/model"
 )
 
 var (
 	filePath = flag.String("file", "", "Path to the export zip file.")
+	quiet    = flag.Bool("q", false, "run in quiet mode")
 )
 
 func main() {
@@ -45,9 +49,44 @@ func main() {
 		log.Fatalf("error unmarshaling export file: %v", err)
 	}
 
+	// Do some basic data validation.
+	if !*quiet {
+		checkExportFile(keyExport)
+	}
+
 	prettyJSON, err := json.MarshalIndent(keyExport, "", "  ")
 	if err != nil {
 		log.Fatalf("error pretty printing export: %v", err)
 	}
 	log.Printf("%v", string(prettyJSON))
 }
+
+func checkExportFile(export *exportpb.TemporaryExposureKeyExport) {
+	now := time.Now().UTC()
+	floor := model.IntervalNumber(now.Add(-14 * 24 * time.Hour))
+	ceiling := model.IntervalNumber(now)
+
+	checkKeys("keys", export.Keys, floor, ceiling)
+	checkKeys("revisedKeys", export.RevisedKeys, floor, ceiling)
+}
+
+func checkKeys(kType string, keys []*exportpb.TemporaryExposureKey, floor, ceiling int32) {
+	for i, k := range keys {
+		if l := len(k.KeyData); l != 16 {
+			log.Printf("%s #%d: invald key length: want 16, got: %v", kType, i, l)
+		}
+		if s := k.GetRollingStartIntervalNumber(); s < floor {
+			log.Printf("%s #%d: rolling interval start number is > 14 days ago, want >= %d, got %d", kType, i, floor, s)
+		} else if s > ceiling {
+			log.Printf("%s #%d: rolling interval start number in the future, want < %d, got %d", kType, i, ceiling, s)
+		}
+		if r := k.GetRollingPeriod(); r < 1 || r > 144 {
+			log.Printf("%s #%d: rolling period invalid, want >= 1 && <= 144, got %d", kType, i, r)
+		}
+		if k.DaysSinceOnsetOfSymptoms != nil {
+			if d := k.GetDaysSinceOnsetOfSymptoms(); d < -14 || d > 14 {
+				log.Printf("%s #%d: days_since_onset_of_symptoms is outside of expectd range, -14..14, got: %d", kType, i, d)
+			}
+		}
+	}
+}


Test patch:
diff --git a/internal/export/worker_test.go b/internal/export/worker_test.go
index 9d0d59c..9dcddeb 100644
--- a/internal/export/worker_test.go
+++ b/internal/export/worker_test.go
@@ -19,6 +19,7 @@ import (
 
 	"github.com/google/exposure-notifications-server/internal/publish/model"
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1alpha1"
+	"github.com/google/exposure-notifications-server/pkg/util"
 	"google.golang.org/protobuf/proto"
 )
 
@@ -60,7 +61,6 @@ func TestEnsureMinExposures(t *testing.T) {
 	// Insert a few exposures - that will be used to base the interval information off of.
 	exposures := []*model.Exposure{
 		{
-			ExposureKey:           []byte{1},
 			TransmissionRisk:      verifyapi.TransmissionRiskConfirmedStandard,
 			IntervalNumber:        123456,
 			IntervalCount:         144,
@@ -68,7 +68,6 @@ func TestEnsureMinExposures(t *testing.T) {
 			ReportType:            verifyapi.ReportTypeConfirmed,
 		},
 		{
-			ExposureKey:           []byte{2},
 			TransmissionRisk:      verifyapi.TransmissionRiskConfirmedStandard,
 			IntervalNumber:        123456 + 144,
 			IntervalCount:         144,
@@ -76,7 +75,6 @@ func TestEnsureMinExposures(t *testing.T) {
 			ReportType:            verifyapi.ReportTypeConfirmed,
 		},
 		{
-			ExposureKey:           []byte{3},
 			TransmissionRisk:      verifyapi.TransmissionRiskConfirmedStandard,
 			IntervalNumber:        123456 - 144,
 			IntervalCount:         144,
@@ -85,6 +83,14 @@ func TestEnsureMinExposures(t *testing.T) {
 		},
 	}
 
+	for _, k := range exposures {
+		b64, err := util.GenerateKey()
+		if err != nil {
+			t.Fatalf("unable to generate exposure keys: %v", err)
+		}
+		k.ExposureKey = util.DecodeKey(b64)
+	}
+
 	numKeys := 100
 	variance := 10
 	m := make(map[int32]int)
diff --git a/internal/integration/integration_test.go b/internal/integration/integration_test.go
index 88faa93..59848f8 100644
--- a/internal/integration/integration_test.go
+++ b/internal/integration/integration_test.go
@@ -46,7 +46,7 @@ func TestIntegration(t *testing.T) {
 	t.Parallel()
 
 	ctx := context.Background()
-	env, client, db := NewTestServer(t, 2*time.Second)
+	env, client, db, jwtCfg := NewTestServer(t, 2*time.Second)
 
 	// Set query criteria (used throughout)
 	criteria := publishdb.IterateExposuresCriteria{
@@ -64,9 +64,14 @@ func TestIntegration(t *testing.T) {
 	verification, salt := testutil.IssueJWT(t, jwtCfg)
 	payload.VerificationPayload = verification
 	payload.HMACKey = salt
+	var revisionToken string
 	if resp, err := client.PublishKeys(payload); err != nil {
 		t.Fatal(err)
 	} else {
+		if resp.RevisionToken == "" {
+			t.Fatal("empty revision token")
+		}
+		revisionToken = resp.RevisionToken
 		t.Logf("response: %+v", resp)
 	}
 
@@ -208,6 +213,26 @@ func TestIntegration(t *testing.T) {
 		}
 	}
 
+	// Rotate Keys. Should Genereate a new key.
+	time.Sleep(200 * time.Millisecond) // Ensure DeleteOldKeyPeriod is elapsed
+	if err := client.RotateKeys(); err != nil {
+		t.Fatalf("Error rotating keys: %v", err)
+	}
+
+	// Rotate Keys. Should Delete the original key.
+	time.Sleep(200 * time.Millisecond) // Ensure DeleteOldKeyPeriod is elapsed
+	if err := client.RotateKeys(); err != nil {
+		t.Fatalf("Error rotating keys: %v", err)
+	}
+
+	// Re-publish with the original token. This key is now not-allowed.
+	payload.RevisionToken = revisionToken
+	if _, err := client.PublishKeys(payload); err == nil {
+		t.Fatal(err)
+	} else if !strings.Contains(err.Error(), verifyapi.ErrorInvalidRevisionToken) {
+		t.Fatal(err)
+	}
+
 	// Publish some new keys so we can generate a new batch
 	keys = util.GenerateExposureKeys(3, -1, false)
 	payload.Keys = keys
diff --git a/internal/keyrotation/rotate_test.go b/internal/keyrotation/rotate_test.go
index f1587cc..41b4147 100644
--- a/internal/keyrotation/rotate_test.go
+++ b/internal/keyrotation/rotate_test.go
@@ -37,7 +37,9 @@ func TestRotateKeys(t *testing.T) {
 
 	kms, _ := keys.NewInMemory(context.Background())
 	keyID := "testKeyID"
-	kms.AddEncryptionKey(keyID)
+	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
+		t.Fatal(err)
+	}
 	config := &Config{
 		RevisionToken:      revision.Config{KeyID: keyID},
 		DeleteOldKeyPeriod: 14 * 24 * time.Hour, // two weeks
diff --git a/internal/keyrotation/server_test.go b/internal/keyrotation/server_test.go
index 7511d18..3a36d5f 100644
--- a/internal/keyrotation/server_test.go
+++ b/internal/keyrotation/server_test.go
@@ -62,7 +62,9 @@ func TestNewRotationHandler(t *testing.T) {
 			t.Parallel()
 
 			keyID := "test" + t.Name()
-			kms.AddEncryptionKey(keyID)
+			if _, err := kms.CreateEncryptionKey(keyID); err != nil {
+				t.Fatal(err)
+			}
 			config := &Config{
 				RevisionToken: revision.Config{KeyID: keyID},
 			}
diff --git a/internal/performance/export_test.go b/internal/performance/export_test.go
index c8878b5..31523f4 100644
--- a/internal/performance/export_test.go
+++ b/internal/performance/export_test.go
@@ -30,44 +30,60 @@ import (
 	publishdb "github.com/google/exposure-notifications-server/internal/publish/database"
 	publishmodel "github.com/google/exposure-notifications-server/internal/publish/model"
 	"github.com/google/exposure-notifications-server/internal/storage"
+	testutil "github.com/google/exposure-notifications-server/internal/utils"
 	verifyapi "github.com/google/exposure-notifications-server/pkg/api/v1"
 	"github.com/google/exposure-notifications-server/pkg/base64util"
 	"github.com/google/exposure-notifications-server/pkg/util"
 	pgx "github.com/jackc/pgx/v4"
+	"github.com/sethvargo/go-envconfig"
 	"github.com/sethvargo/go-retry"
 )
 
+type testConfig struct {
+	Publishes int  `env:"PUBLISHES,default=1000"`
+	Dev       bool `env:"PERFORMANCE_DEV"`
+}
+
 func TestExport(t *testing.T) {
 	const (
 		keysPerPublish = 14
-		// Publishes in 24 hours
-		numPublishes = 100000
-		// Consider the above publishes are evenly distributed in 24 hours, and
-		// period is 10 minutes
-		exportPeriod  = 10 * time.Minute
-		totalBatches  = 24 * 6
-		roughPerBatch = numPublishes/totalBatches + 1
+		exportPeriod   = 10 * time.Minute
+		totalBatches   = 24 * 6
 	)
 	var (
 		ctx      = context.Background()
 		criteria = publishdb.IterateExposuresCriteria{
 			OnlyLocalProvenance: false,
 		}
-		want           = keysPerPublish * numPublishes
+		numPublishes = 100000
+		// Consider the above publishes are evenly distributed in 24 hours, and
+		// period is 10 minutes
 		batchStartTime = time.Now().Add(time.Duration(-totalBatches-10) * exportPeriod)
 	)
+	c := testConfig{}
+	if err := envconfig.ProcessWith(context.Background(), &c, envconfig.OsLookuper()); err != nil {
+		t.Fatalf("unable to process env: %v", err)
+	}
+
+	if c.Dev && c.Publishes > 0 {
+		numPublishes = c.Publishes
+	}
+	want := keysPerPublish * numPublishes
+	roughPerBatch := numPublishes/totalBatches + 1
 
 	makoQuickstore, cancel := setup(t)
 	defer cancel(context.Background())
 
-	env, client, db := integration.NewTestServer(t, exportPeriod)
+	env, client, db, jwtCfg := integration.NewTestServer(t, exportPeriod)
+	keys := util.GenerateExposureKeys(keysPerPublish, -1, false)
 	payload := &verifyapi.Publish{
-		Keys:              util.GenerateExposureKeys(keysPerPublish, -1, false),
+		Keys:              keys,
 		HealthAuthorityID: "com.example.app",
-
-		// TODO: hook up verification
-		VerificationPayload: "TODO: ",
 	}
+	jwtCfg.ExposureKeys = keys
+	verification, salt := testutil.IssueJWT(t, jwtCfg)
+	payload.VerificationPayload = verification
+	payload.HMACKey = salt
 	if _, err := client.PublishKeys(payload); err != nil {
 		t.Fatal(err)
 	}
diff --git a/internal/performance/export_test_benchmark.config b/internal/performance/export_test_benchmark.config
index 7dd4d23..4c09646 100644
--- a/internal/performance/export_test_benchmark.config
+++ b/internal/performance/export_test_benchmark.config
@@ -15,11 +15,6 @@
 # For assistance using the command line tool see go/mako-help.
 #
 
-# Development benchmark configs. Uncomment lines below and comment lines below
-# `Prod benchmark configs` for debugging.
-# benchmark_name: "Development Export"
-# benchmark_key: "5698752440434688"
-
 # Prod benchmark configs
 benchmark_name: "Export"
 benchmark_key: "4790091207671808"
@@ -27,6 +22,7 @@ benchmark_key: "4790091207671808"
 project_name: "Exposure Notifications Server"
 owner_list: "kgood@google.com"
 owner_list: "chaodai@google.com"
+owner_list: "gmccloskey@google.com"
 owner_list: "prow-mako-service-account@oss-prow-build-apollo-server.iam.gserviceaccount.com"
 input_value_info: {
   value_key: "t"
diff --git a/internal/performance/export_test_benchmark.dev.config b/internal/performance/export_test_benchmark.dev.config
new file mode 100644
index 0000000..5dcf778
--- /dev/null
+++ b/internal/performance/export_test_benchmark.dev.config
@@ -0,0 +1,40 @@
+# This file is a protobuff text format representation of a Mako
+# BenchmarkInfo message.
+#
+# For more information about text format protobuffs see: go/proto-text-format
+#
+# For more information about BenchmarkInfo and the fields it contains see
+# https://github.com/google/mako/blob/master/spec/proto/mako.proto
+#
+# This particular message is configured to be passed into a create_benchmark
+# command via the mako command line tool.
+#
+# The difference between this file and update_benchmark.config is that the
+# benchmark_key is not defined.
+#
+# For assistance using the command line tool see go/mako-help.
+#
+
+# Development benchmark configs.
+benchmark_name: "Development Export"
+benchmark_key: "5698752440434688"
+
+project_name: "Exposure Notifications Server"
+owner_list: "kgood@google.com"
+owner_list: "chaodai@google.com"
+owner_list: "gmccloskey@google.com"
+owner_list: "prow-mako-service-account@oss-prow-build-apollo-server.iam.gserviceaccount.com"
+input_value_info: {
+  value_key: "t"
+  label: "time"
+  type: TIMESTAMP
+}
+metric_info_list: {
+  value_key: "m1"
+  label: "Time_to_generate_file_ms"
+}
+metric_info_list: {
+  value_key: "m2"
+  label: "Time_to_cleanup_export_ms"
+}
+description: "Exposure Notifications Server's time to measure export/cleanup"
diff --git a/internal/publish/publish_test.go b/internal/publish/publish_test.go
index 6445efa..644b764 100644
--- a/internal/publish/publish_test.go
+++ b/internal/publish/publish_test.go
@@ -371,7 +371,9 @@ func TestPublishWithBypass(t *testing.T) {
 			t.Fatalf("can't make kms: %v", err)
 		}
 		keyID := "rev"
-		kms.AddEncryptionKey(keyID)
+		if _, err := kms.CreateEncryptionKey(keyID); err != nil {
+			t.Fatal(err)
+		}
 		tokenAAD := make([]byte, 16)
 		if _, err := rand.Read(tokenAAD); err != nil {
 			t.Fatalf("not enough entropy: %v", err)
@@ -682,7 +684,9 @@ func TestKeyRevision(t *testing.T) {
 		t.Fatalf("can't make kms: %v", err)
 	}
 	keyID := "rev"
-	kms.AddEncryptionKey(keyID)
+	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
+		t.Fatal(err)
+	}
 	tokenAAD := make([]byte, 16)
 	if _, err := rand.Read(tokenAAD); err != nil {
 		t.Fatalf("not enough entropy: %v", err)
diff --git a/internal/revision/database/revision_test.go b/internal/revision/database/revision_test.go
index a020e94..2e1d8a2 100644
--- a/internal/revision/database/revision_test.go
+++ b/internal/revision/database/revision_test.go
@@ -40,7 +40,7 @@ func TestRevisionKey(t *testing.T) {
 		t.Fatalf("unable to cerate in memory KMS")
 	}
 	keyID := "funkey"
-	if err := kms.AddEncryptionKey(keyID); err != nil {
+	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
 		t.Fatalf("unable to generate key: %v", err)
 	}
 
@@ -76,7 +76,7 @@ func TestMultipleRevisionKeys(t *testing.T) {
 		t.Fatalf("unable to cerate in memory KMS")
 	}
 	keyID := "funkey"
-	if err := kms.AddEncryptionKey(keyID); err != nil {
+	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
 		t.Fatalf("unable to generate key: %v", err)
 	}
 
diff --git a/internal/revision/revision_test.go b/internal/revision/revision_test.go
index 379dc66..91817eb 100644
--- a/internal/revision/revision_test.go
+++ b/internal/revision/revision_test.go
@@ -128,7 +128,7 @@ func TestEncryptDecrypt(t *testing.T) {
 		t.Fatalf("unable to cerate in memory KMS")
 	}
 	keyID := "skeleton"
-	if err := kms.AddEncryptionKey(keyID); err != nil {
+	if _, err := kms.CreateEncryptionKey(keyID); err != nil {
 		t.Fatalf("unable to generate key: %v", err)
 	}
 	key := make([]byte, 32)
diff --git a/internal/setup/setup_test.go b/internal/setup/setup_test.go
index f20afa5..809bfcf 100644
--- a/internal/setup/setup_test.go
+++ b/internal/setup/setup_test.go
@@ -23,10 +23,10 @@ import (
 
 	"github.com/google/exposure-notifications-server/internal/authorizedapp"
 	"github.com/google/exposure-notifications-server/internal/database"
-	"github.com/google/exposure-notifications-server/internal/observability"
 	"github.com/google/exposure-notifications-server/internal/setup"
 	"github.com/google/exposure-notifications-server/internal/storage"
 	"github.com/google/exposure-notifications-server/pkg/keys"
+	"github.com/google/exposure-notifications-server/pkg/observability"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 	"github.com/sethvargo/go-envconfig"
 )
@@ -231,9 +231,10 @@ func TestSetupWith(t *testing.T) {
 		if oe == nil {
 			t.Errorf("expected observability exporter to exist")
 		}
-
-		if _, ok := oe.(*observability.NoopExporter); !ok {
-			t.Errorf("expected %T to be GenericExporter", oe)
-		}
+		defer func() {
+			if err := oe.Close(); err != nil {
+				t.Fatal(err)
+			}
+		}()
 	})
 }
diff --git a/pkg/logging/logger_test.go b/pkg/logging/logger_test.go
new file mode 100644
index 0000000..7a13fd6
--- /dev/null
+++ b/pkg/logging/logger_test.go
@@ -0,0 +1,67 @@
+// Copyright 2020 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package logging_test
+
+import (
+	"context"
+	"testing"
+
+	"github.com/google/exposure-notifications-server/pkg/logging"
+)
+
+func TestNewLogger(t *testing.T) {
+	t.Parallel()
+
+	logger := logging.NewLogger(true)
+	if logger == nil {
+		t.Fatal("expected logger to never be nil")
+	}
+}
+
+func TestDefaultLogger(t *testing.T) {
+	t.Parallel()
+
+	logger1 := logging.DefaultLogger()
+	if logger1 == nil {
+		t.Fatal("expected logger to never be nil")
+	}
+
+	logger2 := logging.DefaultLogger()
+	if logger2 == nil {
+		t.Fatal("expected logger to never be nil")
+	}
+
+	// Intentionally comparing identities here
+	if logger1 != logger2 {
+		t.Errorf("expected %#v to be %#v", logger1, logger2)
+	}
+}
+
+func TestContext(t *testing.T) {
+	t.Parallel()
+
+	ctx := context.Background()
+	logger1 := logging.FromContext(ctx)
+	if logger1 == nil {
+		t.Fatal("expected logger to never be nil")
+	}
+
+	ctx = logging.WithLogger(ctx, logger1)
+
+	logger2 := logging.FromContext(ctx)
+	if logger1 != logger2 {
+		t.Errorf("expected %#v to be %#v", logger1, logger2)
+	}
+}
diff --git a/scripts/e2e_test.sh b/scripts/e2e_test.sh
new file mode 100755
index 0000000..8b94f12
--- /dev/null
+++ b/scripts/e2e_test.sh
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eEuo pipefail
+
+ROOT="$(cd "$(dirname "$0")/.." &>/dev/null; pwd -P)"
+
+
+echo "🌳 Set up environment variables"
+if [[ -z "${PROJECT_ID:-}" ]]; then
+  echo "✋ PROJECT_ID must be set"
+fi
+
+
+if [[ -z "${DB_CONN:-}" ]]; then # Allow custom database
+  echo "🔨 Provision servers"
+  pushd terraform
+  # TODO(chaodaiG): terraform init; terraform apply; trap "terraform destroy"
+  export DB_CONN="$(terraform output 'db_conn')"
+  export DB_NAME="$(terraform output 'db_name')"
+  export DB_USER="$(terraform output 'db_user')"
+  export DB_PASSWORD="secret://$(terraform output db_pass_secret)"
+  export DB_SSLMODE=disable
+  popd
+fi
+
+
+echo "🔨 Run cloud sql proxy"
+which cloud_sql_proxy || {
+  echo "✋ Download cloud_sql_proxy from https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#install"
+  exit 1
+}
+cloud_sql_proxy -instances=${DB_CONN}=tcp:5432 &
+CLOUD_SQL_PROXY_PID=$!
+trap "kill $CLOUD_SQL_PROXY_PID || true" EXIT
+
+
+echo "🧪 Test"
+go test \
+  -count=1 \
+  -race \
+  -timeout=10m \
+  ./internal/integration