Repository: google/exposure-notifications-server
Pull request #1049 (2020-10-02 15:20:23)

E2e test: export test
As export endpoints are not publicly available, testing exporting by relying on cloud scheduler:

- Make cloud scheduler cron string configurable, and each runs every minute in e2e test
- Worst case scenario is 60 seconds for export batch, and another 60 seconds for export file, let e2e test wait 150 seconds to allow this happen


google-oss-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/google/exposure-notifications-server/pull/1049#" title="Author self-approved">chaodaiG</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **mikehelmick**
You can assign the PR to them by writing `/assign @mikehelmick` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=google%2Fexposure-notifications-server).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/google/exposure-notifications-server/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["mikehelmick"]} -->

Patch:
diff --git a/docs/server_migration.md b/docs/server_migration.md
new file mode 100644
index 0000000..6ce36c4
--- /dev/null
+++ b/docs/server_migration.md
@@ -0,0 +1,94 @@
+---
+layout: default
+---
+# Migration to Google Exposure Notifications Server
+
+This document outlines the process for migrating an existing
+Exposure Notifications server to the Google Exposure Notifications
+server. This process applies regardless of whether your Exposure
+Notifications server uses this codebase.
+
+## Definitions
+
+* For purposes of this document, we'll discuss migration from a __state__ to a __national__
+  key server.
+
+* __export-importer__: This server contains a job (`./cmd/export-importer`) that is capable  
+  reading signed export files from other servers and importing those TEKs to the
+  local storage for re-exporting. We use this in the migration process.
+
+* __Full migration__: The goal of such a migration is to have a state's client
+  application write to and read from the national key server and to _decomission_
+  the state's key server infrastructure.
+
+* __Read-only migration__: The goal of such a migration is to have a state's client
+  application reading from the national exports, but continuing to upload
+  to the state server. This could be because of a custom verification protocol.
+  In this case, decomissioning the state's key sever infrastructure is a non goal.
+
+## Assumptions
+
+* The client / mobile application is live, and requires an update in order
+  to switch servers. This update could be done via dynamic configuration
+  or require a new mobile app version.
+
+* There can only be one active export signing key for the application.
+
+* The __state__ key server has a text file index of available zip files that is
+  sorted in ascending time order (newest files last). If a state server does
+  not have this, it needs to be added before migration can occur.
+
+# Migration Process
+
+## Configure state exports for a second location
+
+The state server must generate export files in a second location
+(for consuption by the national sever). This is necessary because the
+state sever will need to continue to produce state-level exports
+for migration reasons, but will need to keep client-facing exports
+in their current location.
+
+It is recommended to use a separate signing key for these exports than the one already used
+for exports destined for clients. This is only a recommendation and is not necessary since
+the current signing key will be rotated away from client usage.
+
+## Start importing keys
+
+Configure the __export-importer__ job to import keys from the __state__ server
+to the __national__ server.
+
+This requres that the public key portion of the export signing key for the state
+is shared with the national server.
+
+These imported keys will start being shared as part of the national export.
+
+## Export mirroring
+
+These next actions need to happen as quickly as possible, and as close together as possible.
+
+The __state__ will rotate it's signing key to be that of the __national__ server
+by informing Apple and Google of the change. It's best to request an expedited
+rotation if available.
+
+Configure the __state__ server to mirror the __national__ exports on the currently
+known client CDN / download location.
+
+### End state
+
+All client apps for the state will now be uploading keys to the __state__ server
+and downloading keys from the __national__ server. This gives all users full coverage.
+
+This is a terminal state for the __read-only migration__ option.
+
+## Full migration / application update
+
+The __state__ will now push a client application update that directs clients
+to upload to and download from the national server directly.
+
+When this is done, the state server can be decommissioned and the export-importer
+job turned down.
+
+If this is being down through an App Store / Play Store application update,
+this may take a while and it is up to the state to determine
+what an appropriate amount of time to run in this exprort-import-mirror state is.
+
diff --git a/internal/integration/helpers.go b/internal/integration/helpers.go
index 3b6a76a..6160f7e 100644
--- a/internal/integration/helpers.go
+++ b/internal/integration/helpers.go
@@ -243,6 +243,7 @@ func Seed(tb testing.TB, ctx context.Context, db *database.DB, exportPeriod time
 
 	if v := os.Getenv("GOOGLE_CLOUD_BUCKET"); v != "" && !testing.Short() {
 		bucketName = v
+		tb.Logf("Bucket name is %q", bucketName)
 	}
 
 	// create a signing key
@@ -371,9 +372,9 @@ func testRandomID(tb testing.TB, size int) string {
 // Eventually retries the given function n times, sleeping 1s between each
 // invocation. To mark an error as retryable, wrap it in retry.RetryableError.
 // Non-retryable errors return immediately.
-func Eventually(tb testing.TB, times uint64, f func() error) {
+func Eventually(tb testing.TB, times uint64, interval time.Duration, f func() error) {
 	ctx := context.Background()
-	b, err := retry.NewConstant(1 * time.Second)
+	b, err := retry.NewConstant(interval)
 	if err != nil {
 		tb.Fatalf("failed to create retry: %v", err)
 	}
diff --git a/terraform/service_cleanup_export.tf b/terraform/service_cleanup_export.tf
index bf22d8a..a83db5f 100644
--- a/terraform/service_cleanup_export.tf
+++ b/terraform/service_cleanup_export.tf
@@ -145,7 +145,7 @@ resource "google_cloud_run_service_iam_member" "cleanup-export-invoker" {
 resource "google_cloud_scheduler_job" "cleanup-export-worker" {
   name             = "cleanup-export-worker"
   region           = var.cloudscheduler_location
-  schedule         = "0 */6 * * *"
+  schedule         = var.cleanup_export_worker_cron_schedule
   time_zone        = "America/Los_Angeles"
   attempt_deadline = "600s"
 
diff --git a/terraform/service_cleanup_exposure.tf b/terraform/service_cleanup_exposure.tf
index 79e7900..a073f18 100644
--- a/terraform/service_cleanup_exposure.tf
+++ b/terraform/service_cleanup_exposure.tf
@@ -139,7 +139,7 @@ resource "google_cloud_run_service_iam_member" "cleanup-exposure-invoker" {
 resource "google_cloud_scheduler_job" "cleanup-exposure-worker" {
   name             = "cleanup-exposure-worker"
   region           = var.cloudscheduler_location
-  schedule         = "0 */4 * * *"
+  schedule         = var.cleanup_exposure_worker_cron_schedule
   time_zone        = "America/Los_Angeles"
   attempt_deadline = "600s"
 
diff --git a/terraform/service_export.tf b/terraform/service_export.tf
index 09a68cf..936cc63 100644
--- a/terraform/service_export.tf
+++ b/terraform/service_export.tf
@@ -153,7 +153,7 @@ resource "google_cloud_run_service_iam_member" "export-invoker" {
 resource "google_cloud_scheduler_job" "export-worker" {
   name             = "export-worker"
   region           = var.cloudscheduler_location
-  schedule         = "* * * * *"
+  schedule         = var.export_worker_cron_schedule
   time_zone        = "America/Los_Angeles"
   attempt_deadline = "600s"
 
@@ -180,7 +180,7 @@ resource "google_cloud_scheduler_job" "export-worker" {
 resource "google_cloud_scheduler_job" "export-create-batches" {
   name             = "export-create-batches"
   region           = var.cloudscheduler_location
-  schedule         = "*/5 * * * *"
+  schedule         = var.export_create_batches_cron_schedule
   time_zone        = "America/Los_Angeles"
   attempt_deadline = "600s"
 
diff --git a/terraform/storage.tf b/terraform/storage.tf
index 5a2d3ef..264b69f 100644
--- a/terraform/storage.tf
+++ b/terraform/storage.tf
@@ -46,3 +46,7 @@ resource "google_storage_bucket_iam_member" "public" {
   role   = "roles/storage.objectViewer"
   member = "allUsers"
 }
+
+output "export_bucket" {
+  value = google_storage_bucket.export.name
+}
diff --git a/terraform/vars.tf b/terraform/vars.tf
index 221a864..7734f49 100644
--- a/terraform/vars.tf
+++ b/terraform/vars.tf
@@ -132,6 +132,34 @@ variable "cloudsql_backup_location" {
   description = "Location in which to backup the database."
 }
 
+variable "export_worker_cron_schedule" {
+  type    = string
+  default = "* * * * *"
+
+  description = "Schedule to execute the export worker service."
+}
+
+variable "export_create_batches_cron_schedule" {
+  type    = string
+  default = "*/5 * * * *"
+
+  description = "Schedule to execute the export create batches service."
+}
+
+variable "cleanup_exposure_worker_cron_schedule" {
+  type    = string
+  default = "0 */4 * * *"
+
+  description = "Schedule to execute the cleanup exposure worker service."
+}
+
+variable "cleanup_export_worker_cron_schedule" {
+  type    = string
+  default = "0 */6 * * *"
+
+  description = "Schedule to execute the cleanup export worker service."
+}
+
 variable "generate_cron_schedule" {
   type    = string
   default = "0 0 1 1 0"


Test patch:
diff --git a/internal/e2e/e2e_test.go b/internal/e2e/e2e_test.go
index c8f383a..323b5ca 100644
--- a/internal/e2e/e2e_test.go
+++ b/internal/e2e/e2e_test.go
@@ -20,18 +20,23 @@ import (
 	"context"
 	"encoding/base64"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"io/ioutil"
 	"net/http"
+	"strings"
 	"testing"
 	"time"
 
 	"github.com/google/exposure-notifications-server/internal/database"
+	exportapi "github.com/google/exposure-notifications-server/internal/export"
 	"github.com/google/exposure-notifications-server/internal/integration"
 	publishdb "github.com/google/exposure-notifications-server/internal/publish/database"
+	"github.com/google/exposure-notifications-server/internal/storage"
 	"github.com/google/exposure-notifications-server/pkg/secrets"
 	"github.com/google/exposure-notifications-server/pkg/util"
 	"github.com/sethvargo/go-envconfig"
+	"github.com/sethvargo/go-retry"
 
 	publishmodel "github.com/google/exposure-notifications-server/internal/publish/model"
 	testutil "github.com/google/exposure-notifications-server/internal/utils"
@@ -39,11 +44,10 @@ import (
 )
 
 type testConfig struct {
-	DbName       string `env:"DB_NAME"`
-	ExposureURL  string `env:"EXPOSURE_URL"`
-	ExportBucket string `env:"EXPORT_BUCKET"`
-	ProjectID    string `env:"PROJECT_ID"`
-	DBConfig     *database.Config
+	DbName      string `env:"DB_NAME"`
+	ExposureURL string `env:"EXPOSURE_URL"`
+	ProjectID   string `env:"PROJECT_ID"`
+	DBConfig    *database.Config
 }
 
 func initConfig(tb testing.TB, ctx context.Context) *testConfig {
@@ -109,7 +113,8 @@ func TestPublishEndpoint(t *testing.T) {
 	if err != nil {
 		t.Fatalf("unable to connect to database: %v", err)
 	}
-	jwtCfg, _, _, appName := integration.Seed(t, ctx, db, 2*time.Second)
+	jwtCfg, bucketName, filenameRoot, appName := integration.Seed(t, ctx, db, 2*time.Minute)
+
 	keys := util.GenerateExposureKeys(3, -1, false)
 
 	// Publish 3 keys
@@ -117,6 +122,12 @@ func TestPublishEndpoint(t *testing.T) {
 		Keys:              keys,
 		HealthAuthorityID: appName,
 	}
+
+	var wantExport []string
+	for _, key := range keys {
+		wantExport = append(wantExport, key.Key)
+	}
+
 	jwtCfg.ExposureKeys = keys
 	jwtCfg.JWTWarp = time.Duration(0)
 	verification, salt := testutil.IssueJWT(t, jwtCfg)
@@ -146,6 +157,63 @@ func TestPublishEndpoint(t *testing.T) {
 			t.Fatalf("Want published key %q not exist in exposures", want.Key)
 		}
 	}
+
+	blobStore, err := storage.NewGoogleCloudStorage(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	gotExported := make(map[string]bool)
+	integration.Eventually(t, 30, 5*time.Second, func() error {
+		// Attempt to get the index
+		index, err := blobStore.GetObject(ctx, bucketName, integration.IndexFilePath(filenameRoot))
+		if err != nil {
+			if errors.Is(err, storage.ErrNotFound) {
+				return retry.RetryableError(fmt.Errorf("Can not find index file: %w", err))
+			}
+			return err
+		}
+
+		// Find the latest file in the index
+		lines := strings.Split(string(index), "\n")
+		latest := ""
+		for _, entry := range lines {
+			if strings.HasSuffix(entry, "zip") {
+				if entry > latest {
+					latest = entry
+				}
+			}
+		}
+		if latest == "" {
+			return retry.RetryableError(errors.New("failed to find latest export"))
+		}
+
+		// Download the latest export file contents
+		data, err := blobStore.GetObject(ctx, bucketName, latest)
+		if err != nil {
+			return fmt.Errorf("failed to open %s/%s: %w", bucketName, latest, err)
+		}
+
+		// Process contents as an export
+		key, _, err := exportapi.UnmarshalExportFile(data)
+		if err != nil {
+			return fmt.Errorf("failed to extract export data: %w", err)
+		}
+
+		for _, k := range key.Keys {
+			gotExported[base64.StdEncoding.EncodeToString(k.KeyData)] = true
+		}
+		allExported := true
+		for _, want := range wantExport {
+			if _, ok := gotExported[want]; !ok {
+				allExported = false
+			}
+		}
+		if !allExported {
+			return retry.RetryableError(errors.New("Not all keys are exported yet, keep waiting"))
+		}
+
+		return nil
+	})
 }
 
 // getExposures finds the exposures that match the given criteria.
diff --git a/internal/integration/integration_test.go b/internal/integration/integration_test.go
index c5946ef..eb5b21a 100644
--- a/internal/integration/integration_test.go
+++ b/internal/integration/integration_test.go
@@ -122,7 +122,7 @@ func TestIntegration(t *testing.T) {
 
 			// Get the exported exposures
 			var exported *export.TemporaryExposureKeyExport
-			Eventually(t, 30, func() error {
+			Eventually(t, 30, time.Second, func() error {
 				// Trigger an export
 				if err := client.ExportBatches(); err != nil {
 					return err
@@ -290,7 +290,7 @@ func TestIntegration(t *testing.T) {
 
 			// Wait for the export to be created and get the list of files
 			var batchFiles []string
-			Eventually(t, 30, func() error {
+			Eventually(t, 30, time.Second, func() error {
 				// Trigger an export
 				if err := client.ExportBatches(); err != nil {
 					return err
@@ -358,7 +358,7 @@ func TestIntegration(t *testing.T) {
 			}
 
 			// Ensure the export was deleted
-			Eventually(t, 30, func() error {
+			Eventually(t, 30, time.Second, func() error {
 				// Trigger cleanup
 				if err := client.CleanupExports(); err != nil {
 					return err
diff --git a/internal/performance/export_test.go b/internal/performance/export_test.go
index 174ac1d..a8afa7a 100644
--- a/internal/performance/export_test.go
+++ b/internal/performance/export_test.go
@@ -165,7 +165,7 @@ func TestExport(t *testing.T) {
 
 	// Start measurement
 	startTime := time.Now()
-	integration.Eventually(t, 30, func() error {
+	integration.Eventually(t, 30, time.Second, func() error {
 		// Export batch again to make the rest of batches
 		if err := client.ExportBatches(); err != nil {
 			t.Fatal(err)
@@ -245,7 +245,7 @@ func TestExport(t *testing.T) {
 	}
 
 	var remainings []string
-	integration.Eventually(t, 30, func() error {
+	integration.Eventually(t, 30, time.Second, func() error {
 		index, err := env.Blobstore().GetObject(ctx, exportDir,
 			path.Join(exportRoot, "index.txt"))
 		if err != nil {
diff --git a/scripts/e2e-test.sh b/scripts/e2e-test.sh
index 97668af..7f7de37 100755
--- a/scripts/e2e-test.sh
+++ b/scripts/e2e-test.sh
@@ -48,8 +48,8 @@ function incremental() {
   export_terraform_output db_name DB_NAME
   export_terraform_output db_user DB_USER
   export_terraform_output db_password DB_PASSWORD
+  export_terraform_output export_bucket GOOGLE_CLOUD_BUCKET
   export_terraform_output exposure_url EXPOSURE_URL
-  
   export DB_PASSWORD="secret://${DB_PASSWORD}"
   export DB_SSLMODE=disable
 
diff --git a/terraform-e2e/main.tf b/terraform-e2e/main.tf
index 5c39239..9c2056a 100644
--- a/terraform-e2e/main.tf
+++ b/terraform-e2e/main.tf
@@ -32,6 +32,11 @@ module "en" {
   kms_export_signing_key_ring_name  = "export-signing-${random_string.suffix.result}"
   kms_revision_tokens_key_ring_name = "revision-tokens-${random_string.suffix.result}"
 
+  cleanup_export_worker_cron_schedule = "* * * * *"
+  cleanup_exposure_worker_cron_schedule = "* * * * *"
+  export_worker_cron_schedule = "* * * * *"
+  export_create_batches_cron_schedule = "* * * * *"
+
   create_env_file = true
   deploy_debugger = true