Repository: conbench/conbench (language: python)

Issue:
ci instability: cb-on-minikube may fail during postgres-operator deployment
Seen here: https://github.com/conbench/conbench/pull/692#issuecomment-1420616849
```
Wait for port forwarding to take effect
The command curl --location --silent --output /dev/null http://127.0.0.1:8080/clusters timed out
==== FAILURE: OPERATOR DID NOT START OR PORT FORWARDING DID NOT WORK
If you observe issues with minikube VM not starting/not proceeding, consider deleting the .minikube dir and/or rebooting before re-running the script
Error: Process completed with exit code 1.
```

Went away upon re-run, proving that this is a non-deterministic effect.

Maybe trying longer than 1 minute is a solution: https://github.com/zalando/postgres-operator/blob/933b45aba0d64243368e1ed8619ce1cbfb3e7ac6/run_operator_locally.sh#L34

Maybe _not_ swallowing stderr is of help: https://github.com/zalando/postgres-operator/blob/933b45aba0d64243368e1ed8619ce1cbfb3e7ac6/run_operator_locally.sh#L170






Discussion / hints:
@austin3dickey had this twice yesterday. Right?
Depending on the platform this is seemingly way more likely to hit in. Needs tackling.
Seen by @alistaire47 here: https://github.com/conbench/conbench/actions/runs/4138141592/jobs/7154181896 but that was before adding the latest improvements on this front.

@austin3dickey I'd really love to see the current state on your machine. Thanks!
Just observed this again locally:
```
Wait for port forwarding to take effect
Wait for port forwarding to take effect
Wait for port forwarding to take effect
Wait for port forwarding to take effect
Wait for port forwarding to take effect
Wait for port forwarding to take effect
The command curl --location --silent --output /dev/null http://127.0.0.1:8080/clusters timed out
==== FAILURE: OPERATOR DID NOT START OR PORT FORWARDING DID NOT WORK
```

Means that https://github.com/conbench/conbench/pull/725 did not help, which was one of two expected outcomes. We've learned that this needs further debugging.

It's entirely possible that the port-forward was never set up properly before going into the curl loop. We wouldn't see a corresponding error as I've pointed out in the initial comment above:

> Maybe not swallowing stderr is of help
Submitted this patch: https://github.com/zalando/postgres-operator/pull/2218. Will start jgehrcke/postgres-operator (with the patch branch) for our CI for now.

Fix patch:
diff --git a/Makefile b/Makefile
index ed3b317b0..227870e74 100644
--- a/Makefile
+++ b/Makefile
@@ -162,7 +162,7 @@ conbench-on-minikube: build-conbench-container-image start-minikube
 	@echo "     log in with admin/admin"
 	@echo
 	@echo "Conbench UI port-forward:"
-	@echo "     kubectl port-forward svc/conbench-service 8000:conbench-service-port"
+	@echo "     run: kubectl port-forward svc/conbench-service 8000:conbench-service-port"
 	@echo "     then open the Conbench UI at: http://localhost:8000 "
 	@make -s minikube-conbench-url
 
@@ -187,6 +187,7 @@ minikube-conbench-url:
 start-minikube:
 	minikube delete --profile mk-conbench
 	minikube start --profile mk-conbench --cpus=2 --memory=4g \
+		--driver=docker \
 		--disable-metrics \
 		--kubernetes-version=v1.24.10 \
 		--bootstrapper=kubeadm \


Test patch:
diff --git a/ci/minikube/test-conbench-on-mk.sh b/ci/minikube/test-conbench-on-mk.sh
index 5e34df9f5..c7c68b910 100644
--- a/ci/minikube/test-conbench-on-mk.sh
+++ b/ci/minikube/test-conbench-on-mk.sh
@@ -28,25 +28,23 @@ minikube addons disable metrics-server --profile mk-conbench
 # Great docs: https://postgres-operator.readthedocs.io/en/latest/user/
 # Running ./run_operator_locally.sh means installing this manifest:
 # https://github.com/zalando/postgres-operator/blob/v1.9.0/manifests/minimal-postgres-manifest.yaml
-git clone https://github.com/zalando/postgres-operator
+#git clone https://github.com/zalando/postgres-operator
+git clone https://github.com/jgehrcke/postgres-operator
 pushd postgres-operator
-    git checkout v1.9.0  # release from 2023-01-30
+    # git checkout v1.9.0  # release from 2023-01-30
+    # Use this patch for better robustness for now, also see
+    # https://github.com/conbench/conbench/issues/693
+    git checkout jp/run-local-robustness
 
     # Set number of Postgres instances to 1. Need to be conservative with k8s
     # cluster resources, because GHA offers limited resources.
     sed -i.bak 's|numberOfInstances: 2|numberOfInstances: 1|g' manifests/minimal-postgres-manifest.yaml
     cat manifests/minimal-postgres-manifest.yaml | grep numberOfInstances
 
-    # Timeout after four minutes instead of one minute. On some platforms this
-    # takes longish. See https://github.com/conbench/conbench/issues/693.
-    sed -i.bak 's|{1..20}|{1..80}|g' ./run_operator_locally.sh
+    # Remove 'start_minikube' from `run_operator_locally.sh` (the minikube
+    # cluster is already up and running at this point).
+    sed -i.bak 's|^    start_minikube$|#start_minikube|g' ./run_operator_locally.sh
 
-    # alchemy: Remove 'clean_up' and 'start_minikube' from
-    # `run_operator_locally.sh` (the minikube cluster is already up and running
-    # at this point). Do this via line number deletion. In the original file,
-    # delete line 256 and 257. That is safe, because a specific commit of this
-    # file was checked out.
-    sed -i.bak '256d;257d' run_operator_locally.sh
     bash ./run_operator_locally.sh
 popd