Repository: oasisprotocol/oasis-sdk
Pull request #299 (2021-07-20 14:04:57)

test/benchmark: fixes and improvements
- use release instead of the debug benchmarks build
- use compute+storage worker nodes instead of separate workers
- bump some benchmark parameters

codecov[bot]: # [Codecov](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol) Report
> Merging [#299](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol) (d19858e) into [main](https://codecov.io/gh/oasisprotocol/oasis-sdk/commit/a8b8ce46ee3fea973b4ee447e4a349ab41e1852d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol) (a8b8ce4) will **decrease** coverage by `2.85%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/graphs/tree.svg?width=650&height=150&src=pr&token=uscfoYjSZn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol)](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol)

```diff
@@            Coverage Diff             @@
##             main     #299      +/-   ##
==========================================
- Coverage   70.70%   67.85%   -2.86%     
==========================================
  Files          57        8      -49     
  Lines        3455      252    -3203     
==========================================
- Hits         2443      171    -2272     
+ Misses        992       61     -931     
  Partials       20       20              
```


| [Impacted Files](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol) | Coverage Δ | |
|---|---|---|
| [runtime-sdk/src/crypto/multisig/test.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-cnVudGltZS1zZGsvc3JjL2NyeXB0by9tdWx0aXNpZy90ZXN0LnJz) | | |
| [runtime-sdk-macros/src/test\_utils.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-cnVudGltZS1zZGstbWFjcm9zL3NyYy90ZXN0X3V0aWxzLnJz) | | |
| [runtime-sdk/src/testing/keys.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-cnVudGltZS1zZGsvc3JjL3Rlc3Rpbmcva2V5cy5ycw==) | | |
| [tests/runtimes/benchmarking/src/runtime/mod.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-dGVzdHMvcnVudGltZXMvYmVuY2htYXJraW5nL3NyYy9ydW50aW1lL21vZC5ycw==) | | |
| [runtime-sdk/src/error.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-cnVudGltZS1zZGsvc3JjL2Vycm9yLnJz) | | |
| [runtime-sdk/src/crypto/multisig/mod.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-cnVudGltZS1zZGsvc3JjL2NyeXB0by9tdWx0aXNpZy9tb2QucnM=) | | |
| [runtime-sdk/src/types/message.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-cnVudGltZS1zZGsvc3JjL3R5cGVzL21lc3NhZ2UucnM=) | | |
| [tests/runtimes/simple-consensus/src/main.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-dGVzdHMvcnVudGltZXMvc2ltcGxlLWNvbnNlbnN1cy9zcmMvbWFpbi5ycw==) | | |
| [tests/runtimes/simple-keyvalue/src/main.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-dGVzdHMvcnVudGltZXMvc2ltcGxlLWtleXZhbHVlL3NyYy9tYWluLnJz) | | |
| [tests/runtimes/simple-consensus/src/lib.rs](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol#diff-dGVzdHMvcnVudGltZXMvc2ltcGxlLWNvbnNlbnN1cy9zcmMvbGliLnJz) | | |
| ... and [39 more](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol). Last update [a8b8ce4...d19858e](https://codecov.io/gh/oasisprotocol/oasis-sdk/pull/299?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=oasisprotocol).


Patch:
diff --git a/.dockerignore b/.dockerignore
index e1a91bbe12..771e35d32a 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -2,7 +2,7 @@
 **
 
 # Allow runtime benchmarking binary.
-!target/debug/test-runtime-benchmarking
+!target/release/test-runtime-benchmarking
 
 # Allow downloaded oasis-core binaries.
 !tests/untracked/


Test patch:
diff --git a/tests/benchmark/Dockerfile b/tests/benchmark/Dockerfile
index c977617906..23dc92e4c1 100644
--- a/tests/benchmark/Dockerfile
+++ b/tests/benchmark/Dockerfile
@@ -7,6 +7,6 @@ RUN apt-get -y update && apt-get install -y bubblewrap
 COPY tests/untracked/oasis_core_${OASIS_CORE_VERSION}_linux_amd64/oasis-node /oasis/bin/oasis-node
 COPY tests/untracked/oasis_core_${OASIS_CORE_VERSION}_linux_amd64/oasis-core-runtime-loader /oasis/bin/oasis-core-runtime-loader
 COPY tests/benchmark/benchmark /oasis/bin/benchmark
-COPY target/debug/test-runtime-benchmarking /oasis/lib/oasis-runtime
+COPY target/release/test-runtime-benchmarking /oasis/lib/oasis-runtime
 
 ENV PATH "/oasis/bin:${PATH}"
diff --git a/tests/benchmark/build-benchmarks.sh b/tests/benchmark/build-benchmarks.sh
index 7c08eaa043..6eebb424c5 100755
--- a/tests/benchmark/build-benchmarks.sh
+++ b/tests/benchmark/build-benchmarks.sh
@@ -12,5 +12,5 @@ go build
 
 echo "Building test benchmarking runtime."
 pushd "${TESTS_DIR}"/runtimes/benchmarking
-    cargo build
+    cargo build --release
 popd
diff --git a/tests/benchmark/cmd/fixture.go b/tests/benchmark/cmd/fixture.go
index f57edbc6bb..7ebe916a93 100644
--- a/tests/benchmark/cmd/fixture.go
+++ b/tests/benchmark/cmd/fixture.go
@@ -93,12 +93,14 @@ func fixture() *oasis.NetworkFixture {
 			{NodeFixture: oasis.NodeFixture{ExtraArgs: clientExtraArgs}},
 		},
 		StorageWorkers: []oasis.StorageWorkerFixture{
-			{Entity: 1, Runtimes: []int{0}, Backend: badger.BackendName},
+			{NodeFixture: oasis.NodeFixture{Name: "compute-storage-1"}, Entity: 1, Runtimes: []int{0}, Backend: badger.BackendName},
+			{NodeFixture: oasis.NodeFixture{Name: "compute-storage-2"}, Entity: 1, Runtimes: []int{0}, Backend: badger.BackendName},
+			{NodeFixture: oasis.NodeFixture{Name: "compute-storage-3"}, Entity: 1, Runtimes: []int{0}, Backend: badger.BackendName},
 		},
 		ComputeWorkers: []oasis.ComputeWorkerFixture{
-			{Entity: 1, Runtimes: []int{0}, NodeFixture: oasis.NodeFixture{ExtraArgs: computeExtraArgs}},
-			{Entity: 1, Runtimes: []int{0}, NodeFixture: oasis.NodeFixture{ExtraArgs: computeExtraArgs}},
-			{Entity: 1, Runtimes: []int{0}, NodeFixture: oasis.NodeFixture{ExtraArgs: computeExtraArgs}},
+			{NodeFixture: oasis.NodeFixture{Name: "compute-storage-1", ExtraArgs: computeExtraArgs}, Entity: 1, Runtimes: []int{0}},
+			{NodeFixture: oasis.NodeFixture{Name: "compute-storage-2", ExtraArgs: computeExtraArgs}, Entity: 1, Runtimes: []int{0}},
+			{NodeFixture: oasis.NodeFixture{Name: "compute-storage-3", ExtraArgs: computeExtraArgs}, Entity: 1, Runtimes: []int{0}},
 		},
 		Seeds: []oasis.SeedFixture{{}},
 		Runtimes: []oasis.RuntimeFixture{
@@ -124,8 +126,8 @@ func fixture() *oasis.NetworkFixture {
 					ProposerTimeout:   5,
 				},
 				Storage: registry.StorageParameters{
-					GroupSize:               1,
-					MinWriteReplication:     1,
+					GroupSize:               3,
+					MinWriteReplication:     3,
 					MaxApplyWriteLogEntries: 100_000,
 					MaxApplyOps:             2,
 				},
diff --git a/tests/benchmark/run-benchmarks.sh b/tests/benchmark/run-benchmarks.sh
index 7a5157cbb9..9bc92a8390 100755
--- a/tests/benchmark/run-benchmarks.sh
+++ b/tests/benchmark/run-benchmarks.sh
@@ -1,11 +1,10 @@
 #!/usr/bin/env bash
 set -o nounset -o pipefail -o errexit -x
 
-
 # Kill all dangling processes on exit.
 cleanup() {
-	pkill -P $$ || true
-	wait || true
+    pkill -P $$ || true
+    wait || true
 }
 trap "cleanup" EXIT
 
@@ -23,8 +22,8 @@ mkdir -p /tmp/oasis-net-runner-benchmarks
     fixture \
     --node.binary "${TEST_NODE_BINARY}" \
     --runtime.id "8000000000000000000000000000000000000000000000000000000000000000" \
-    --runtime.binary ../../target/debug/test-runtime-benchmarking \
-    --runtime.loader "${TEST_RUNTIME_LOADER}" > /tmp/oasis-net-runner-benchmarks/fixture.json
+    --runtime.binary ../../target/release/test-runtime-benchmarking \
+    --runtime.loader "${TEST_RUNTIME_LOADER}" >/tmp/oasis-net-runner-benchmarks/fixture.json
 
 "${TEST_NET_RUNNER}" \
     --fixture.file /tmp/oasis-net-runner-benchmarks/fixture.json \
@@ -47,7 +46,7 @@ echo "Advancing epoch."
 echo "Waiting for all nodes to be registered."
 "${TEST_NODE_BINARY}" debug control wait-nodes \
     --address "${CLIENT_SOCK}" \
-    --nodes 5 \
+    --nodes 4 \
     --wait
 
 echo "Advancing epoch."
@@ -58,7 +57,7 @@ echo "Advancing epoch."
 sleep 2
 
 ./benchmark \
-	--address "${CLIENT_SOCK}" \
-	--runtime.id "8000000000000000000000000000000000000000000000000000000000000000" \
-	--benchmarks accounts_transfers \
-	--benchmarks.concurrency 100
+    --address "${CLIENT_SOCK}" \
+    --runtime.id "8000000000000000000000000000000000000000000000000000000000000000" \
+    --benchmarks accounts_transfers \
+    --benchmarks.concurrency 6000