Repository: apigee/registry
Pull request #631 (2022-06-25 05:03:11)

Minimal initial benchmarks
This pares down @giteshk's benchmarks to a single set of CRUD tests on API resources that is hopefully easily extensible to other resources and scenarios. I'm not sure that we need to do this for every resource since the underlying operations don't vary much per resource. This alone seems enough to do some interesting comparisons of various hosting alternatives.

codecov[bot]: # [Codecov](https://codecov.io/gh/apigee/registry/pull/631?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee) Report
> Merging [#631](https://codecov.io/gh/apigee/registry/pull/631?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee) (a06c1f1) into [main](https://codecov.io/gh/apigee/registry/commit/c180a8e3a5056ab60b4e7e5177fbca9c5c8a2420?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee) (c180a8e) will **decrease** coverage by `0.14%`.
> The diff coverage is `8.69%`.

```diff
@@            Coverage Diff             @@
##             main     #631      +/-   ##
==========================================
- Coverage   52.53%   52.39%   -0.15%     
==========================================
  Files          71       72       +1     
  Lines        6974     6997      +23     
==========================================
+ Hits         3664     3666       +2     
- Misses       2915     2936      +21     
  Partials      395      395              
```


| [Impacted Files](https://codecov.io/gh/apigee/registry/pull/631?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee) | Coverage Δ | |
|---|---|---|
| [tests/benchmark/benchmark\_helpers.go](https://codecov.io/gh/apigee/registry/pull/631/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee#diff-dGVzdHMvYmVuY2htYXJrL2JlbmNobWFya19oZWxwZXJzLmdv) | `8.69% <8.69%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/apigee/registry/pull/631?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apigee/registry/pull/631?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee). Last update [c180a8e...a06c1f1](https://codecov.io/gh/apigee/registry/pull/631?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apigee).

timburks: @theganyo it works for me... I was originally running on my Chromebook but just verified that it works on a MacBook (with a small change to line 21 of the script). How did you run this?
theganyo: `./tests/benchmark/BENCH.sh` needs +x perm set
theganyo: I've found the issue. It's no coincidence that the errors are for ID 10: It's the [default page size](https://github.com/apigee/registry/blob/main/cmd/registry/cmd/rpc/generated/list-apis.go#L28) for the ListXXXCmd actions which are being used in wipeout.
timburks: Well, I thought the problem was with wipeout but it seems instead that the function passed to Run() gets called multiple times without cleaning up after itself. Fixing...
theganyo: On another note re: schema. The wipeout code seems to make it clear we're not using foreign key constraints in the database? Manually maintaining DB integrity can be quite error prone... Shall I create an issue to address this?
timburks: @theganyo Agreed - after starting a `wipeout_test.go`, I'd prefer to add that in a separate PR because I expect a lot of discussion about the test structure.
timburks: Done (added pkg/README.md)

Patch:
diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml
index ba19eaa..25413e4 100644
--- a/.github/workflows/go.yml
+++ b/.github/workflows/go.yml
@@ -82,6 +82,17 @@ jobs:
     - name: Test registry server with PostgreSQL
       run: go test ./server/registry -postgresql
 
+    - name: Verify that benchmarks run
+      env:
+        APG_REGISTRY_ADDRESS: localhost:8080 
+        APG_REGISTRY_AUDIENCES: http://localhost:8080
+        APG_REGISTRY_INSECURE: 1
+        PROJECTID: bench
+        ITERATIONS: 1
+      run: |
+        registry rpc admin create-project --project_id=$PROJECTID --json
+        go test ./tests/benchmark --bench=. --project_id=$PROJECTID --benchtime=${ITERATIONS}x --timeout=0
+
     - name: Compute code coverage
       env:
         CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
@@ -95,4 +106,4 @@ jobs:
         gpgv codecov.SHA256SUM.sig codecov.SHA256SUM
         shasum -a 256 -c codecov.SHA256SUM
         chmod +x codecov
-        ./codecov -t ${CODECOV_TOKEN}
\ No newline at end of file
+        ./codecov -t ${CODECOV_TOKEN}
diff --git a/go.mod b/go.mod
index 43b95d3..4b02fdd 100644
--- a/go.mod
+++ b/go.mod
@@ -52,10 +52,10 @@ require (
 	github.com/jinzhu/inflection v1.0.0 // indirect
 	github.com/jinzhu/now v1.1.2 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
-	github.com/lib/pq v1.10.3 // indirect
+	github.com/lib/pq v1.10.3
 	github.com/magiconair/properties v1.8.5 // indirect
 	github.com/mailru/easyjson v0.7.7 // indirect
-	github.com/mattn/go-sqlite3 v1.14.8 // indirect
+	github.com/mattn/go-sqlite3 v1.14.8
 	github.com/mitchellh/mapstructure v1.4.1 // indirect
 	github.com/pelletier/go-toml v1.9.3 // indirect
 	github.com/pkg/errors v0.9.1 // indirect
diff --git a/pkg/README.md b/pkg/README.md
new file mode 100644
index 0000000..b20f51c
--- /dev/null
+++ b/pkg/README.md
@@ -0,0 +1,3 @@
+# pkg
+
+This directory contains libraries that can be used by external applications.
\ No newline at end of file
diff --git a/pkg/wipeout/wipeout.go b/pkg/wipeout/wipeout.go
new file mode 100644
index 0000000..54fd898
--- /dev/null
+++ b/pkg/wipeout/wipeout.go
@@ -0,0 +1,204 @@
+// Copyright 2022 Google LLC. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package wipeout
+
+import (
+	"context"
+
+	"github.com/apigee/registry/cmd/registry/core"
+	"github.com/apigee/registry/connection"
+	"github.com/apigee/registry/log"
+	"github.com/apigee/registry/rpc"
+)
+
+// Set this global variable in calling code to run verbosely.
+var Verbose bool
+
+// This collects one (arbitrary) error that occurs during wipeout.
+// It's a global and gets assigned from concurrent goroutines,
+// but if it fails, it fails.
+var _err error
+
+// Wipeout deletes all resources in a project using the specified number of parallel worker jobs.
+func Wipeout(ctx context.Context, client connection.Client, projectid string, jobs int) error {
+	if Verbose {
+		log.Infof(ctx, "Deleting everything in project %s", projectid)
+	}
+	project := "projects/" + projectid + "/locations/global"
+	// Wipeout resources in groups to ensure that children are deleted before parents.
+	{
+		if Verbose {
+			log.Infof(ctx, "Deleting artifacts")
+		}
+		taskQueue, wait := core.WorkerPool(ctx, jobs)
+		wipeoutArtifacts(ctx, client, taskQueue, project+"/apis/-/versions/-/specs/-")
+		wipeoutArtifacts(ctx, client, taskQueue, project+"/apis/-/versions/-")
+		wipeoutArtifacts(ctx, client, taskQueue, project+"/apis/-/deployments/-")
+		wipeoutArtifacts(ctx, client, taskQueue, project+"/apis/-")
+		wipeoutArtifacts(ctx, client, taskQueue, project)
+		wait()
+	}
+	{
+		if Verbose {
+			log.Infof(ctx, "Deleting specs")
+		}
+		taskQueue, wait := core.WorkerPool(ctx, jobs)
+		wipeoutApiSpecs(ctx, client, taskQueue, project+"/apis/-/versions/-")
+		wait()
+	}
+	{
+		if Verbose {
+			log.Infof(ctx, "Deleting versions")
+		}
+		taskQueue, wait := core.WorkerPool(ctx, jobs)
+		wipeoutApiVersions(ctx, client, taskQueue, project+"/apis/-")
+		wait()
+	}
+	{
+		if Verbose {
+			log.Infof(ctx, "Deleting deployments")
+		}
+		taskQueue, wait := core.WorkerPool(ctx, jobs)
+		wipeoutApiDeployments(ctx, client, taskQueue, project+"/apis/-")
+		wait()
+	}
+	{
+		if Verbose {
+			log.Infof(ctx, "Deleting apis")
+		}
+		taskQueue, wait := core.WorkerPool(ctx, jobs)
+		wipeoutApis(ctx, client, taskQueue, project)
+		wait()
+	}
+	if Verbose {
+		log.Infof(ctx, "Wipeout complete with err=%+v", _err)
+	}
+	return _err
+}
+
+type ResourceKind int
+
+const (
+	API ResourceKind = iota
+	Version
+	Spec
+	Deployment
+	Artifact
+)
+
+type deleteResourceTask struct {
+	client connection.Client
+	name   string
+	kind   ResourceKind
+}
+
+func (task *deleteResourceTask) String() string {
+	return "delete " + task.name
+}
+
+func (task *deleteResourceTask) Run(ctx context.Context) error {
+	var err error
+	switch task.kind {
+	case API:
+		if Verbose {
+			log.Infof(ctx, "Deleting %s", task.name)
+		}
+		err = task.client.DeleteApi(ctx, &rpc.DeleteApiRequest{Name: task.name})
+	case Version:
+		if Verbose {
+			log.Infof(ctx, "Deleting %s", task.name)
+		}
+		err = task.client.DeleteApiVersion(ctx, &rpc.DeleteApiVersionRequest{Name: task.name})
+	case Spec:
+		if Verbose {
+			log.Infof(ctx, "Deleting %s", task.name)
+		}
+		err = task.client.DeleteApiSpec(ctx, &rpc.DeleteApiSpecRequest{Name: task.name})
+	case Deployment:
+		if Verbose {
+			log.Infof(ctx, "Deleting %s", task.name)
+		}
+		err = task.client.DeleteApiDeployment(ctx, &rpc.DeleteApiDeploymentRequest{Name: task.name})
+	case Artifact:
+		if Verbose {
+			log.Infof(ctx, "Deleting %s", task.name)
+		}
+		err = task.client.DeleteArtifact(ctx, &rpc.DeleteArtifactRequest{Name: task.name})
+	default:
+		if Verbose {
+			log.Infof(ctx, "Unknown resource type %s", task.name)
+		}
+	}
+	if err != nil {
+		log.FromContext(ctx).WithError(err).Errorf("Deletion %s failed", task.name)
+		_err = err
+	}
+	return nil
+}
+
+func wipeoutArtifacts(ctx context.Context, registryClient connection.Client, taskQueue chan<- core.Task, parent string) {
+	it := registryClient.ListArtifacts(ctx, &rpc.ListArtifactsRequest{Parent: parent})
+	for artifact, err := it.Next(); err == nil; artifact, err = it.Next() {
+		taskQueue <- &deleteResourceTask{
+			client: registryClient,
+			name:   artifact.Name,
+			kind:   Artifact,
+		}
+	}
+}
+
+func wipeoutApiDeployments(ctx context.Context, registryClient connection.Client, taskQueue chan<- core.Task, parent string) {
+	it := registryClient.ListApiDeployments(ctx, &rpc.ListApiDeploymentsRequest{Parent: parent})
+	for deployment, err := it.Next(); err == nil; deployment, err = it.Next() {
+		taskQueue <- &deleteResourceTask{
+			client: registryClient,
+			name:   deployment.Name,
+			kind:   Deployment,
+		}
+	}
+}
+
+func wipeoutApiSpecs(ctx context.Context, registryClient connection.Client, taskQueue chan<- core.Task, parent string) {
+	it := registryClient.ListApiSpecs(ctx, &rpc.ListApiSpecsRequest{Parent: parent})
+	for spec, err := it.Next(); err == nil; spec, err = it.Next() {
+		taskQueue <- &deleteResourceTask{
+			client: registryClient,
+			name:   spec.Name,
+			kind:   Spec,
+		}
+	}
+}
+
+func wipeoutApiVersions(ctx context.Context, registryClient connection.Client, taskQueue chan<- core.Task, parent string) {
+	it := registryClient.ListApiVersions(ctx, &rpc.ListApiVersionsRequest{Parent: parent})
+	for version, err := it.Next(); err == nil; version, err = it.Next() {
+		taskQueue <- &deleteResourceTask{
+			client: registryClient,
+			name:   version.Name,
+			kind:   Version,
+		}
+	}
+}
+
+func wipeoutApis(ctx context.Context, registryClient connection.Client, taskQueue chan<- core.Task, parent string) {
+	it := registryClient.ListApis(ctx, &rpc.ListApisRequest{Parent: parent})
+	for api, err := it.Next(); err == nil; api, err = it.Next() {
+		taskQueue <- &deleteResourceTask{
+			client: registryClient,
+			name:   api.Name,
+			kind:   API,
+		}
+	}
+}


Test patch:
diff --git a/tests/benchmark/BENCH.sh b/tests/benchmark/BENCH.sh
new file mode 100755
index 0000000..9edcbe9
--- /dev/null
+++ b/tests/benchmark/BENCH.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+#
+# Copyright 2022 Google LLC. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+export PROJECTID=bench
+
+# Assume that "bench" is a local project and all other names are hosted.
+if [[ "$PROJECTID" == "bench" ]]; then
+  . auth/LOCAL.sh
+  # Redirect the following command to /dev/null to ignore errors due to nonexistent projects.
+  registry rpc admin delete-project --name=projects/$PROJECTID --force --json &> /dev/null
+  registry rpc admin create-project --project_id=$PROJECTID --json
+else
+  . auth/HOSTED.sh
+fi
+
+# Increase this to get better sampling.
+export ITERATIONS=1
+
+go test ./tests/benchmark -parallel=1 --bench=. --project_id=$PROJECTID --benchtime=${ITERATIONS}x --timeout=0
diff --git a/tests/benchmark/README.md b/tests/benchmark/README.md
new file mode 100644
index 0000000..c9b78a2
--- /dev/null
+++ b/tests/benchmark/README.md
@@ -0,0 +1,3 @@
+# Benchmark tests
+
+See `BENCH.sh` for an example showing how to run these.
\ No newline at end of file
diff --git a/tests/benchmark/apis_test.go b/tests/benchmark/apis_test.go
deleted file mode 100644
index 56823a1..0000000
--- a/tests/benchmark/apis_test.go
+++ /dev/null
@@ -1,177 +0,0 @@
-// Copyright 2021 Google LLC. All Rights Reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//    http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-package benchmark
-
-import (
-	"context"
-	"fmt"
-	"testing"
-
-	"github.com/apigee/registry/connection"
-	"github.com/apigee/registry/rpc"
-	"github.com/google/uuid"
-)
-
-const (
-	rootResource = "projects/bench/locations/global"
-)
-
-func uniqueID() string {
-	return fmt.Sprintf("%.8s", uuid.New())
-}
-
-func BenchmarkCreateApi(b *testing.B) {
-	ctx := context.Background()
-	client, err := connection.NewClient(ctx)
-	if err != nil {
-		b.Fatalf("Setup: Failed to create client: %s", err)
-	}
-
-	b.StopTimer()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		req := &rpc.CreateApiRequest{
-			Parent: rootResource,
-			ApiId:  uniqueID(),
-			Api:    &rpc.Api{},
-		}
-
-		b.StartTimer()
-		api, err := client.CreateApi(ctx, req)
-		b.StopTimer()
-
-		if err != nil {
-			b.Errorf("CreateApi(%+v) returned unexpected error: %s", req, err)
-		}
-
-		if err := client.DeleteApi(ctx, &rpc.DeleteApiRequest{Name: api.GetName()}); err != nil {
-			b.Errorf("Cleanup: DeleteApi(%q) returned unexpected error: %s", api.GetName(), err)
-		}
-	}
-}
-
-func BenchmarkGetApi(b *testing.B) {
-	ctx := context.Background()
-	client, err := connection.NewClient(ctx)
-	if err != nil {
-		b.Fatalf("Setup: Failed to create client: %s", err)
-	}
-
-	b.StopTimer()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		api, err := client.CreateApi(ctx, &rpc.CreateApiRequest{
-			Parent: rootResource,
-			ApiId:  uniqueID(),
-			Api:    &rpc.Api{},
-		})
-		if err != nil {
-			b.Fatalf("Setup: Failed to create API: %s", err)
-		}
-
-		req := &rpc.GetApiRequest{
-			Name: api.GetName(),
-		}
-
-		b.StartTimer()
-		_, err = client.GetApi(ctx, req)
-		b.StopTimer()
-
-		if err != nil {
-			b.Errorf("GetApi(%+v) returned unexpected error: %s", req, err)
-		}
-
-		if err := client.DeleteApi(ctx, &rpc.DeleteApiRequest{Name: api.GetName()}); err != nil {
-			b.Errorf("Cleanup: DeleteApi(%q) returned unexpected error: %s", api.GetName(), err)
-		}
-	}
-}
-
-func BenchmarkListApis(b *testing.B) {
-	b.Skip("TODO: Create ListApi benchmark test")
-}
-
-func BenchmarkUpdateApi(b *testing.B) {
-	ctx := context.Background()
-	client, err := connection.NewClient(ctx)
-	if err != nil {
-		b.Fatalf("Setup: Failed to create client: %s", err)
-	}
-
-	b.StopTimer()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		api, err := client.CreateApi(ctx, &rpc.CreateApiRequest{
-			Parent: rootResource,
-			ApiId:  uniqueID(),
-			Api:    &rpc.Api{},
-		})
-		if err != nil {
-			b.Fatalf("Setup: Failed to create API: %s", err)
-		}
-
-		req := &rpc.UpdateApiRequest{
-			Api: &rpc.Api{
-				Name:        api.GetName(),
-				Description: uniqueID(),
-			},
-		}
-
-		b.StartTimer()
-		_, err = client.UpdateApi(ctx, req)
-		b.StopTimer()
-
-		if err != nil {
-			b.Errorf("UpdateApi(%+v) returned unexpected error: %s", req, err)
-		}
-
-		if err := client.DeleteApi(ctx, &rpc.DeleteApiRequest{Name: api.GetName()}); err != nil {
-			b.Errorf("Cleanup: DeleteApi(%q) returned unexpected error: %s", api.GetName(), err)
-		}
-	}
-}
-
-func BenchmarkDeleteApi(b *testing.B) {
-	ctx := context.Background()
-	client, err := connection.NewClient(ctx)
-	if err != nil {
-		b.Fatalf("Setup: Failed to create client: %s", err)
-	}
-
-	b.StopTimer()
-	b.ResetTimer()
-	for i := 0; i < b.N; i++ {
-		api, err := client.CreateApi(ctx, &rpc.CreateApiRequest{
-			Parent: rootResource,
-			ApiId:  uniqueID(),
-			Api:    &rpc.Api{},
-		})
-		if err != nil {
-			b.Fatalf("Setup: Failed to create API: %s", err)
-		}
-
-		req := &rpc.DeleteApiRequest{
-			Name: api.GetName(),
-		}
-
-		b.StartTimer()
-		err = client.DeleteApi(ctx, req)
-		b.StopTimer()
-
-		if err != nil {
-			b.Errorf("DeleteApi(%+v) returned unexpected error: %s", req, err)
-		}
-	}
-}
diff --git a/tests/benchmark/benchmark_apis_test.go b/tests/benchmark/benchmark_apis_test.go
new file mode 100644
index 0000000..da4f503
--- /dev/null
+++ b/tests/benchmark/benchmark_apis_test.go
@@ -0,0 +1,170 @@
+// Copyright 2022 Google LLC. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package benchmark
+
+import (
+	"context"
+	"fmt"
+	"testing"
+
+	"github.com/apigee/registry/connection"
+	"github.com/apigee/registry/rpc"
+	"google.golang.org/api/iterator"
+	"google.golang.org/genproto/protobuf/field_mask"
+)
+
+func apiId(i int) string {
+	return fmt.Sprintf("test-%d", i)
+}
+
+func apiName(apiId string) string {
+	return fmt.Sprintf("%s/apis/%s", root(), apiId)
+}
+
+func getApi(b *testing.B, ctx context.Context, client connection.Client, apiId string) error {
+	b.Helper()
+	_, err := client.GetApi(ctx, &rpc.GetApiRequest{Name: apiName(apiId)})
+	return err
+}
+
+func listApis(b *testing.B, ctx context.Context, client connection.Client) error {
+	b.Helper()
+	it := client.ListApis(ctx, &rpc.ListApisRequest{Parent: root()})
+	for _, err := it.Next(); err != iterator.Done; _, err = it.Next() {
+		if err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func createApi(b *testing.B, ctx context.Context, client connection.Client, apiId string) error {
+	b.Helper()
+	_, err := client.CreateApi(ctx, &rpc.CreateApiRequest{
+		Parent: root(),
+		ApiId:  apiId,
+		Api: &rpc.Api{
+			DisplayName: apiId,
+			Description: fmt.Sprintf("Description for %s", apiId),
+		},
+	})
+	return err
+}
+
+func updateApi(b *testing.B, ctx context.Context, client connection.Client, apiId string) error {
+	b.Helper()
+	_, err := client.UpdateApi(ctx, &rpc.UpdateApiRequest{
+		Api: &rpc.Api{
+			Name:        apiName(apiId),
+			DisplayName: fmt.Sprintf("Updated %s", apiId),
+		}, UpdateMask: &field_mask.FieldMask{
+			Paths: []string{"display_name"},
+		},
+	})
+	return err
+}
+
+func deleteApi(b *testing.B, ctx context.Context, client connection.Client, apiId string) error {
+	b.Helper()
+	return client.DeleteApi(ctx, &rpc.DeleteApiRequest{Name: apiName(apiId)})
+}
+
+func BenchmarkGetApi(b *testing.B) {
+	ctx, client := setup(b)
+	if err := createApi(b, ctx, client, apiId(1)); err != nil {
+		b.Fatalf("%s", err)
+	}
+	b.Run("GetApi", func(b *testing.B) {
+		for i := 1; i <= b.N; i++ {
+			if err := getApi(b, ctx, client, apiId(1)); err != nil {
+				b.Fatalf("%s", err)
+			}
+		}
+	})
+	teardown(b, ctx, client)
+}
+
+func BenchmarkListApis(b *testing.B) {
+	ctx, client := setup(b)
+	for i := 1; i <= 10; i++ {
+		if err := createApi(b, ctx, client, apiId(i)); err != nil {
+			b.Fatalf("%s", err)
+		}
+	}
+	b.Run("ListApis", func(b *testing.B) {
+		for i := 1; i <= b.N; i++ {
+			if err := listApis(b, ctx, client); err != nil {
+				b.Fatalf("%s", err)
+			}
+		}
+	})
+	teardown(b, ctx, client)
+}
+
+func BenchmarkCreateApi(b *testing.B) {
+	ctx, client := setup(b)
+	b.Run("CreateApi", func(b *testing.B) {
+		for i := 1; i <= b.N; i++ {
+			if err := createApi(b, ctx, client, apiId(i)); err != nil {
+				b.Fatalf("%s", err)
+			}
+		}
+		b.StopTimer()
+		for i := 1; i <= b.N; i++ {
+			if err := deleteApi(b, ctx, client, apiId(i)); err != nil {
+				b.Fatalf("%s", err)
+			}
+		}
+		b.StartTimer()
+	})
+
+	teardown(b, ctx, client)
+}
+
+func BenchmarkUpdateApi(b *testing.B) {
+	ctx, client := setup(b)
+	for i := 1; i <= 1; i++ {
+		if err := createApi(b, ctx, client, apiId(i)); err != nil {
+			b.Fatalf("%s", err)
+		}
+	}
+	b.Run("UpdateApi", func(b *testing.B) {
+		for i := 1; i <= b.N; i++ {
+			if err := updateApi(b, ctx, client, apiId(1)); err != nil {
+				b.Fatalf("%s", err)
+			}
+		}
+	})
+	teardown(b, ctx, client)
+}
+
+func BenchmarkDeleteApi(b *testing.B) {
+	ctx, client := setup(b)
+	b.Run("DeleteApi", func(b *testing.B) {
+		b.StopTimer()
+		for i := 1; i <= b.N; i++ {
+			if err := createApi(b, ctx, client, apiId(i)); err != nil {
+				b.Fatalf("%s", err)
+			}
+		}
+		b.StartTimer()
+		for i := 1; i <= b.N; i++ {
+			if err := deleteApi(b, ctx, client, apiId(i)); err != nil {
+				b.Fatalf("%s", err)
+			}
+		}
+	})
+	teardown(b, ctx, client)
+}
diff --git a/tests/benchmark/benchmark_helpers.go b/tests/benchmark/benchmark_helpers.go
new file mode 100644
index 0000000..ec4ce0a
--- /dev/null
+++ b/tests/benchmark/benchmark_helpers.go
@@ -0,0 +1,64 @@
+// Copyright 2022 Google LLC. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package benchmark
+
+import (
+	"context"
+	"flag"
+	"fmt"
+	"testing"
+
+	"github.com/apigee/registry/connection"
+	"github.com/apigee/registry/pkg/wipeout"
+)
+
+// The level of parallelism to use for wipeout.
+// If this is too high, the test could be rate-limited.
+const jobs = 10
+
+var (
+	projectID string
+)
+
+func init() {
+	flag.StringVar(&projectID, "project_id", "bench", "registry project ID")
+}
+
+func root() string {
+	return fmt.Sprintf("projects/%s/locations/global", projectID)
+}
+
+func setup(b *testing.B) (context.Context, connection.Client) {
+	b.Helper()
+	ctx := context.Background()
+	client, err := connection.NewClient(ctx)
+	if err != nil {
+		b.Logf("Unable to connect to registry server. Is it running?")
+		b.FailNow()
+	}
+	if err := wipeout.Wipeout(ctx, client, projectID, jobs); err != nil {
+		b.Logf("Pre-test wipeout failed")
+		b.FailNow()
+	}
+	return ctx, client
+}
+
+func teardown(b *testing.B, ctx context.Context, client connection.Client) {
+	b.Helper()
+	if err := wipeout.Wipeout(ctx, client, projectID, jobs); err != nil {
+		b.Logf("Post-test wipeout failed")
+		b.FailNow()
+	}
+}