Repository: apache/solr-operator

Issue:
Getting unauthorized requests for cluster/replicas/balance in v0.8.0
Hi Team,

We've recently changed the operator version from v0.7.0 to v0.8.0 for our SolrCloud cluster (9.4). 
It seems that the new version requests the status of an async task named `balance-replicas-ScaleUp` every 60 seconds. Once it doesn't find one running, it sends a "replica/balance" request to the target Solr, therefore we're getting the following error for every attempt:

```
Recieved bad response code of 403 from solr with response: {
\"servlet\":\"default\",
\"message\":\"Unauthorized request, Response code: 403\",
\"url\":\"/solr/____v2/cluster/replicas/balance\",
\"status\":\"403\"
}
```

I found that the relevant metadata for this use case is written into the annotation field of the statefulsets object: `"solr.apache.org/clusterOpsLock": "{\"operation\":\"BalanceReplicas\",\"lastStartTime\":\"2023-11-09T14:28:18Z\",\"metadata\":\"ScaleUp\"}"`  

We're using the default security.json credentials and I'm not certain if there's anything to be changed in our settings for this matter.

Thanks!


Hints from issue comments:
@mmoscher: Not sure, but could be a similar/related issue.
After upgrade from v0.7.0 to v0.8.0 for our SolrCloud cluster (9.0), we're getting a 404 from Solr API

`Error returned from Solr API: 404. no core retrieved for core name:  null. Path : /cluster/replicas/balance`

```
2023-12-22T06:37:54Z    INFO    Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler  {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "01215b5c-6555-404c-85f5-2a5246ef41cb"}
2023-12-22T06:37:54Z    ERROR   Reconciler error        {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"solr","namespace":"solr"}, "namespace": "solr", "name": "solr", "reconcileID": "01215b5c-6555-404c-85f5-2a5246ef41cb", "error": "Error returned from Solr API: 404. no core retrieved for core name:  null. Path : /cluster/replicas/balance"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:227
``` 

Same error on Solr side:

`2023-12-22 06:40:54.937 ERROR (qtp1942828992-2283) [] o.a.s.a.V2HttpCall >> path: '/cluster/replicas/balance'
2023-12-22 06:40:54.937 ERROR (qtp1942828992-2283) [] o.a.s.a.V2HttpCall Error in init() => org.apache.solr.common.SolrException: no core retrieved for core name:  null. Path : /cluster/replicas/balance
        at org.apache.solr.api.V2HttpCall.init(V2HttpCall.java:155)
org.apache.solr.common.SolrException: no core retrieved for core name:  null. Path : /cluster/replicas/balance`

why is the operator hitting this path?

Thanks for any help.@HoustonPutman: @mmoscher , that is expected. After the completion of an ephemeral rolling restart, the Solr Operator now tries to balance the cluster. If Solr doesn't support that command (and 9.0 does not), it just completes the operation. But the only way that the Solr Operator can know if it's supported is to try running it.

@ozlerhakan I think that is something that we missed. I'll try to add a PR for that soon.@bcbrockway: Is there a workaround for this? We're getting the same error after starting up a brand new cluster.@jstaf: As a workaround, you can manually add `/____v2/cluster/replicas/balance` in the security screen to the `k8s-oper` and `admin` users on a new cluster. 

However just a warning that rebalancing doesn't seem to work very well so you may just want to leave it off... it seems to just create a whole bunch of replicas then time out, then leaves a bunch of replicas on the new nodes without deleting the ones on the old node. After failing, it runs rebalance again, then instantly declares it was a "success" even though the replicas never got moved properly and now you have more replicas than you started with.

```
2025-03-15 00:59:28.579 INFO  (OverseerThreadFactory-120-thread-2) [c: s: r: x: t:] o.a.s.c.a.c.ReplicaMigrationUtils Timed out waiting for 1 leader replicas to recover
2025-03-15 00:59:28.579 INFO  (OverseerThreadFactory-120-thread-2) [c: s: r: x: t:] o.a.s.c.a.c.ReplicaMigrationUtils Failed to create some replicas. Cleaning up all newly created replicas.
org.apache.solr.common.SolrException: collection,shard,replica are required params
"	at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.checkRequired(CollectionHandlingUtils.java:196) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.api.collections.DeleteReplicaCmd.deleteReplica(DeleteReplicaCmd.java:86) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:130) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]"
2025-03-15 00:59:28.579 ERROR (OverseerThreadFactory-120-thread-2) [c: s: r: x: t:] o.a.s.c.a.c.CollectionHandlingUtils Operation balance_replicas failed => org.apache.solr.common.SolrException: collection,shard,replica are required params
"	at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.checkRequired(CollectionHandlingUtils.java:196) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.api.collections.DeleteReplicaCmd.deleteReplica(DeleteReplicaCmd.java:86) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.api.collections.ReplicaMigrationUtils.migrateReplicas(ReplicaMigrationUtils.java:217) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.api.collections.BalanceReplicasCmd.call(BalanceReplicasCmd.java:81) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:130) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:363) ~[solr-solrj-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]"
2025-03-15 00:59:28.579 WARN  (OverseerThreadFactory-120-thread-2) [c: s: r: x: t:] o.a.s.c.a.c.ReplicaMigrationUtils Error deleting replica  => org.apache.solr.common.SolrException: collection,shard,replica are required params
"	at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.checkRequired(CollectionHandlingUtils.java:196)"
"	at org.apache.solr.cloud.api.collections.ReplicaMigrationUtils.migrateReplicas(ReplicaMigrationUtils.java:217) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.api.collections.BalanceReplicasCmd.call(BalanceReplicasCmd.java:81) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.api.collections.CollApiCmds$TraceAwareCommand.call(CollApiCmds.java:225) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:564) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:363) ~[solr-solrj-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]"
"	at java.base/java.lang.Thread.run(Unknown Source) [?:?]"
"	at org.apache.solr.cloud.api.collections.CollectionHandlingUtils.checkRequired(CollectionHandlingUtils.java:196)"
org.apache.solr.common.SolrException: collection,shard,replica are required params
"	at org.apache.solr.cloud.api.collections.CollApiCmds$TraceAwareCommand.call(CollApiCmds.java:225) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:564) ~[solr-core-9.6.1.jar:9.6.1 d7f7166567f52f1b31e3315b0188e11f2c4c9b60 - houston - 2024-05-23 13:50:22]"
"	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]"
"	at java.base/java.lang.Thread.run(Unknown Source) [?:?]"
```

Solution patch:
diff --git a/controllers/util/solr_security_util.go b/controllers/util/solr_security_util.go
index eab99d2..417d6b9 100644
--- a/controllers/util/solr_security_util.go
+++ b/controllers/util/solr_security_util.go
@@ -117,6 +117,9 @@ func reconcileForBasicAuthWithBootstrappedSecurityJson(ctx context.Context, clie
 
 		// supply the bootstrap security.json to the initContainer via a simple BASE64 encoding env var
 		security.SecurityJson = string(bootstrapSecret.Data[SecurityJsonFile])
+		security.SecurityJsonSrc = &corev1.EnvVarSource{
+			SecretKeyRef: &corev1.SecretKeySelector{
+				LocalObjectReference: corev1.LocalObjectReference{Name: bootstrapSecret.Name}, Key: SecurityJsonFile}}
 		basicAuthSecret = authSecret
 	}
 
@@ -393,7 +396,9 @@ func generateSecurityJson(solrCloud *solr.SolrCloud) map[string][]byte {
           { "name": "k8s-metrics", "role":"k8s", "collection": null, "path":"/admin/metrics" },
           { "name": "k8s-zk", "role":"k8s", "collection": null, "path":"/admin/zookeeper/status" },
           { "name": "k8s-ping", "role":"k8s", "collection": "*", "path":"/admin/ping" },
-          { "name": "read", "role":["admin","users"] },
+          { "name": "k8s-replica-balancing", "role":"k8s", "collection": null, "path":"/____v2/cluster/replicas/balance" },
+          { "name": "collection-admin-edit", "role":"k8s" },
+          { "name": "read", "role":["admin","users","k8s"] },
           { "name": "update", "role":["admin"] },
           { "name": "security-read", "role": ["admin"] },
           { "name": "security-edit", "role": ["admin"] },
diff --git a/docs/solr-cloud/solr-cloud-crd.md b/docs/solr-cloud/solr-cloud-crd.md
index 5eaf6e9..52027f0 100644
--- a/docs/solr-cloud/solr-cloud-crd.md
+++ b/docs/solr-cloud/solr-cloud-crd.md
@@ -1032,6 +1032,16 @@ Take a moment to review these authorization rules so that you're aware of the ro
         "collection": "*",
         "path": "/admin/ping"
       },
+      {
+        "name": "k8s-replica-balancing",
+        "role": "k8s",
+        "collection": null,
+        "path": "/____v2/cluster/replicas/balance"
+      },
+      {
+        "name": "collection-admin-edit",
+        "role": "k8s"
+      },
       {
         "name": "read",
         "role": [ "admin", "users" ]
@@ -1165,6 +1175,12 @@ Users need to ensure their `security.json` contains the user supplied in the `ba
 /admin/metrics
 /admin/ping (for collection="*")
 /admin/zookeeper/status
+/____v2/cluster/replicas/balance
+```
+
+And the following named permissions:
+```aiignore
+collection-admin-edit
 ```
 _Tip: see the authorization rules defined by the default `security.json` as a guide for configuring access for the operator user_
 



Test patch:
diff --git a/tests/e2e/solrcloud_security_json_test.go b/tests/e2e/solrcloud_security_json_test.go
index 1ceef5e..8bc8695 100644
--- a/tests/e2e/solrcloud_security_json_test.go
+++ b/tests/e2e/solrcloud_security_json_test.go
@@ -20,9 +20,14 @@ package e2e
 import (
 	"context"
 	solrv1beta1 "github.com/apache/solr-operator/api/v1beta1"
+	"github.com/apache/solr-operator/controllers"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
 	"k8s.io/utils/pointer"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"time"
 )
 
 var _ = FDescribe("E2E - SolrCloud - Security JSON", func() {
@@ -35,10 +40,6 @@ var _ = FDescribe("E2E - SolrCloud - Security JSON", func() {
 	})
 
 	JustBeforeEach(func(ctx context.Context) {
-		By("generating the security.json secret and basic auth secret")
-		generateSolrSecuritySecret(ctx, solrCloud)
-		generateSolrBasicAuthSecret(ctx, solrCloud)
-
 		By("creating the SolrCloud")
 		Expect(k8sClient.Create(ctx, solrCloud)).To(Succeed())
 
@@ -50,20 +51,64 @@ var _ = FDescribe("E2E - SolrCloud - Security JSON", func() {
 		solrCloud = expectSolrCloudToBeReady(ctx, solrCloud)
 
 		By("creating a first Solr Collection")
-		createAndQueryCollection(ctx, solrCloud, "basic", 1, 1)
+		createAndQueryCollection(ctx, solrCloud, "basic", 2, 1)
 	})
 
-	FContext("Provided Zookeeper", func() {
-		BeforeEach(func() {
+	FContext("Provided Security JSON", func() {
+		BeforeEach(func(ctx context.Context) {
 			solrCloud.Spec.ZookeeperRef = &solrv1beta1.ZookeeperRef{
 				ProvidedZookeeper: &solrv1beta1.ZookeeperSpec{
 					Replicas:  pointer.Int32(1),
 					Ephemeral: &solrv1beta1.ZKEphemeral{},
 				},
 			}
+
+			solrCloud.Spec.SolrSecurity = &solrv1beta1.SolrSecurityOptions{
+				AuthenticationType: "Basic",
+				BasicAuthSecret:    solrCloud.Name + "-basic-auth-secret",
+				BootstrapSecurityJson: &corev1.SecretKeySelector{
+					LocalObjectReference: corev1.LocalObjectReference{
+						Name: solrCloud.Name + "-security-secret",
+					},
+					Key: "security.json",
+				},
+			}
+
+			By("generating the security.json secret and basic auth secret")
+			generateSolrSecuritySecret(ctx, solrCloud)
+			generateSolrBasicAuthSecret(ctx, solrCloud)
 		})
 
 		// All testing will be done in the "JustBeforeEach" logic, no additional tests required here
 		FIt("Starts correctly", func(ctx context.Context) {})
 	})
+
+	FContext("Bootstrapped Security", func() {
+
+		BeforeEach(func() {
+			solrCloud.Spec.SolrSecurity = &solrv1beta1.SolrSecurityOptions{
+				AuthenticationType: "Basic",
+			}
+		})
+
+		FIt("Scales up with replica migration", func(ctx context.Context) {
+			originalSolrCloud := solrCloud.DeepCopy()
+			solrCloud.Spec.Replicas = pointer.Int32(int32(2))
+			By("triggering a scale up via solrCloud replicas")
+			Expect(k8sClient.Patch(ctx, solrCloud, client.MergeFrom(originalSolrCloud))).To(Succeed(), "Could not patch SolrCloud replicas to initiate scale down")
+
+			By("make sure scaleDown happens without a clusterLock and eventually the replicas are removed")
+			// Once the scale down actually occurs, the statefulSet annotations should be removed very soon
+			expectStatefulSetWithChecksAndTimeout(ctx, solrCloud, solrCloud.StatefulSetName(), time.Minute*2, time.Millisecond*500, func(g Gomega, found *appsv1.StatefulSet) {
+				g.Expect(found.Spec.Replicas).To(HaveValue(BeEquivalentTo(2)), "StatefulSet should eventually have 2 pods.")
+				clusterOp, err := controllers.GetCurrentClusterOp(found)
+				g.Expect(err).ToNot(HaveOccurred(), "Error occurred while finding clusterLock for SolrCloud")
+				g.Expect(clusterOp).To(BeNil(), "StatefulSet should have a ScaleDown lock after scaling is complete.")
+			})
+
+			queryCollection(ctx, solrCloud, "basic", 0)
+
+			// TODO: When balancing is in all Operator supported Solr versions, add a test to make sure balancing occurred
+		})
+	})
 })
diff --git a/tests/e2e/suite_test.go b/tests/e2e/suite_test.go
index 1c2aec5..c746443 100644
--- a/tests/e2e/suite_test.go
+++ b/tests/e2e/suite_test.go
@@ -106,6 +106,7 @@ var _ = SynchronizedBeforeSuite(func(ctx context.Context) {
 	var err error
 	k8sConfig, err = config.GetConfig()
 	Expect(err).NotTo(HaveOccurred(), "Could not load in default kubernetes config")
+	k8sConfig.Timeout = time.Minute
 	Expect(zkApi.AddToScheme(scheme.Scheme)).To(Succeed())
 	Expect(certManagerApi.AddToScheme(scheme.Scheme)).To(Succeed())
 	k8sClient, err = client.New(k8sConfig, client.Options{Scheme: scheme.Scheme})
@@ -338,7 +339,7 @@ func writeAllSolrInfoToFiles(ctx context.Context, directory string, namespace st
 	}
 
 	foundServices := &corev1.ServiceList{}
-	Expect(k8sClient.List(ctx, foundServices, listOps)).To(Succeed(), "Could not fetch Solr pods")
+	Expect(k8sClient.List(ctx, foundServices, listOps)).To(Succeed(), "Could not fetch Solr services")
 	Expect(foundServices).ToNot(BeNil(), "No Solr services could be found")
 	for _, service := range foundServices.Items {
 		writeAllServiceInfoToFiles(
@@ -346,6 +347,25 @@ func writeAllSolrInfoToFiles(ctx context.Context, directory string, namespace st
 			&service,
 		)
 	}
+
+	// Unfortunately the secrets don't have a technology label
+	req, err = labels.NewRequirement("solr-cloud", selection.Exists, make([]string, 0))
+	Expect(err).ToNot(HaveOccurred())
+
+	labelSelector = labels.Everything().Add(*req)
+	listOps = &client.ListOptions{
+		Namespace:     namespace,
+		LabelSelector: labelSelector,
+	}
+
+	foundSecrets := &corev1.SecretList{}
+	Expect(k8sClient.List(ctx, foundSecrets, listOps)).To(Succeed(), "Could not fetch Solr secrets")
+	for _, secret := range foundSecrets.Items {
+		writeAllSecretInfoToFiles(
+			directory+secret.Name+".secret",
+			&secret,
+		)
+	}
 }
 
 // writeSolrClusterStatusInfoToFile writes the following each to a separate file with the given base name & directory.
@@ -401,6 +421,19 @@ func writeAllServiceInfoToFiles(baseFilename string, service *corev1.Service) {
 	Expect(writeErr).ToNot(HaveOccurred(), "Could not write service json to file")
 }
 
+// writeAllSecretInfoToFiles writes the following each to a separate file with the given base name & directory.
+//   - Service
+func writeAllSecretInfoToFiles(baseFilename string, secret *corev1.Secret) {
+	// Write service to a file
+	statusFile, err := os.Create(baseFilename + ".json")
+	defer statusFile.Close()
+	Expect(err).ToNot(HaveOccurred(), "Could not open file to save secret status: %s", baseFilename+".json")
+	jsonBytes, marshErr := json.MarshalIndent(secret, "", "\t")
+	Expect(marshErr).ToNot(HaveOccurred(), "Could not serialize secret json")
+	_, writeErr := statusFile.Write(jsonBytes)
+	Expect(writeErr).ToNot(HaveOccurred(), "Could not write secret json to file")
+}
+
 // writeAllPodInfoToFile writes the following each to a separate file with the given base name & directory.
 //   - Pod Spec/Status
 //   - Pod Events
diff --git a/tests/e2e/test_utils_test.go b/tests/e2e/test_utils_test.go
index 203b93e..240ee4c 100644
--- a/tests/e2e/test_utils_test.go
+++ b/tests/e2e/test_utils_test.go
@@ -25,6 +25,7 @@ import (
 	"fmt"
 	"io"
 	"os"
+	"regexp"
 	"strconv"
 	"strings"
 	"time"
@@ -233,7 +234,7 @@ func createAndQueryCollectionWithGomega(ctx context.Context, solrCloud *solrv1be
 	}
 
 	additionalOffset += 1
-	g.EventuallyWithOffset(additionalOffset, func(innerG Gomega) {
+	g.EventuallyWithOffset(additionalOffset, func(innerG Gomega, ctx context.Context) {
 		response, err := callSolrApiInPod(
 			ctx,
 			solrCloud,
@@ -246,7 +247,7 @@ func createAndQueryCollectionWithGomega(ctx context.Context, solrCloud *solrv1be
 	}).Within(time.Second*10).WithContext(ctx).Should(Succeed(), "Collection creation command start was not successful")
 	// Only wait 5 seconds when trying to create the asyncCommand
 
-	g.EventuallyWithOffset(additionalOffset, func(innerG Gomega) {
+	g.EventuallyWithOffset(additionalOffset, func(innerG Gomega, ctx context.Context) {
 		response, err := callSolrApiInPod(
 			ctx,
 			solrCloud,
@@ -271,7 +272,7 @@ func createAndQueryCollectionWithGomega(ctx context.Context, solrCloud *solrv1be
 		innerG.Expect(response).To(ContainSubstring("\"state\":\"completed\""), "Did not finish creating Solr Collection in time")
 	}).Within(time.Second*40).WithContext(ctx).Should(Succeed(), "Collection creation was not successful")
 
-	g.EventuallyWithOffset(additionalOffset, func(innerG Gomega) {
+	g.EventuallyWithOffset(additionalOffset, func(innerG Gomega, ctx context.Context) {
 		response, err := callSolrApiInPod(
 			ctx,
 			solrCloud,
@@ -296,7 +297,7 @@ func queryCollection(ctx context.Context, solrCloud *solrv1beta1.SolrCloud, coll
 }
 
 func queryCollectionWithGomega(ctx context.Context, solrCloud *solrv1beta1.SolrCloud, collection string, docCount int, g Gomega, additionalOffset ...int) {
-	g.EventuallyWithOffset(resolveOffset(additionalOffset), func(innerG Gomega) {
+	g.EventuallyWithOffset(resolveOffset(additionalOffset), func(innerG Gomega, ctx context.Context) {
 		response, err := callSolrApiInPod(
 			ctx,
 			solrCloud,
@@ -476,6 +477,20 @@ func callSolrApiInPod(ctx context.Context, solrCloud *solrv1beta1.SolrCloud, htt
 		queryParamsString = "?" + queryParamsString
 	}
 
+	toolOpts := ""
+	if solrCloud.Spec.SolrSecurity != nil && solrCloud.Spec.SolrSecurity.AuthenticationType == solrv1beta1.Basic {
+		basicAuthSecretName := solrCloud.BasicAuthSecretName()
+		basicAuthSecret := &corev1.Secret{}
+		if err = k8sClient.Get(ctx, resourceKey(solrCloud, basicAuthSecretName), basicAuthSecret); err != nil {
+			return "", err
+		}
+		toolOpts =
+			"JAVA_TOOL_OPTIONS=\"-Dbasicauth=" +
+				string(basicAuthSecret.Data[corev1.BasicAuthUsernameKey]) + ":" + string(basicAuthSecret.Data[corev1.BasicAuthPasswordKey]) +
+				" -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory\""
+	}
+	GinkgoLogr.Info(toolOpts)
+
 	command := []string{
 		"solr",
 		"api",
@@ -489,6 +504,12 @@ func callSolrApiInPod(ctx context.Context, solrCloud *solrv1beta1.SolrCloud, htt
 			apiPath,
 			queryParamsString),
 	}
+	if toolOpts != "" {
+		commandString := fmt.Sprintf("%s %s", toolOpts, strings.Join(command, " "))
+		commandString = regexp.MustCompile(`\s+`).ReplaceAllString(strings.TrimSpace(commandString), " ")
+
+		command = []string{"sh", "-c", fmt.Sprintf("%q", commandString)}
+	}
 	return runExecForContainer(ctx, util.SolrNodeContainer, solrCloud.GetRandomSolrPodName(), solrCloud.Namespace, command)
 }
 
@@ -655,17 +676,6 @@ func generateBaseSolrCloudWithSecurityJSON(replicas int) *solrv1beta1.SolrCloud
 		solrCloud.Spec.SolrSecurity = &solrv1beta1.SolrSecurityOptions{}
 	}
 
-	solrCloud.Spec.SolrSecurity.BootstrapSecurityJson = &corev1.SecretKeySelector{
-		LocalObjectReference: corev1.LocalObjectReference{
-			Name: solrCloud.Name + "-security-secret",
-		},
-		Key: "security.json",
-	}
-
-	solrCloud.Spec.SolrSecurity.AuthenticationType = "Basic"
-
-	solrCloud.Spec.SolrSecurity.BasicAuthSecret = solrCloud.Name + "-basic-auth-secret"
-
 	return solrCloud
 }
 
diff --git a/tests/scripts/manage_e2e_tests.sh b/tests/scripts/manage_e2e_tests.sh
index 9af0c5b..2da32c4 100755
--- a/tests/scripts/manage_e2e_tests.sh
+++ b/tests/scripts/manage_e2e_tests.sh
@@ -76,7 +76,7 @@ if [[ -z "${KUBERNETES_VERSION:-}" ]]; then
   KUBERNETES_VERSION="v1.26.6"
 fi
 if [[ -z "${SOLR_IMAGE:-}" ]]; then
-  SOLR_IMAGE="${SOLR_VERSION:-9.4.0}"
+  SOLR_IMAGE="${SOLR_VERSION:-9.8.1}"
 fi
 if [[ "${SOLR_IMAGE}" != *":"* ]]; then
   SOLR_IMAGE="solr:${SOLR_IMAGE}"