Repository: apache/solr-operator

Issue:
Security.json is never copied to Zookeeper while creating Solr 9.8.0 Clusters
When creating a SolrCloud 9.8.0 cluster using version 0.9.0 of Solr Operator, security.json is never coped to Zookeeper. The `setup-zk` container appears to have all the correct information, including the $SECURITY_JSON variable. The only logs are a single line: `No security.json found. Put new security.json in ZK`

I have other SolrCloud clusters that have been upgraded from 9.4 -> 9.8 and they work fine, but they have a completely different command in `setup-zk`

One thing I noticed is that all of the `setup-zk` init containers finish running before a zookeeper is online, possibly creating a race condition

#720 Seems related

I believe 0e76152755bc1b66fb2f703b58df2271158444ca introduced the new behavior

Discussion:
@janhoy: Can you whip together a **minimum reproduction** of this? E.g.

1. Install solr operator 0.9.0
2. Create a secret with a security.json as follows: bla bla kubectl apply bla bla
3. Install a solr cluster with this yaml specifying security etc - kubectl apply etc
4. Observe how cluster is created without applying security

Your linked issue was a different error preventing 9.7 to work properly. There were then fixes in operator (as linked to) and solr, but there could be another regression here.@jdgiddings: @janhoy Here is a minimal example where the bug occurs. Please excuse the Azure stuff, we're hosting it on AKS

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: security-json
  namespace: dev
data:
  security.json: REDACTED
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: solr-basic-auth
  namespace: dev
data:
  password: REDACTED
  username: REDACTED
type: kubernetes.io/basic-auth
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: solr-config
  namespace: dev  
data:
  solr.xml: |
    <?xml version="1.0" encoding="UTF-8" ?>
    <solr>
        <int name="maxBooleanClauses">10240</int>
        <solrcloud>
        <str name="host">${host:}</str>
        <int name="hostPort">${hostPort:80}</int>
        <str name="hostContext">${hostContext:solr}</str>
        <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
        <int name="zkClientTimeout">${zkClientTimeout:30000}</int>
        <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
        <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
        <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str>
        <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str>
        </solrcloud>
        <shardHandlerFactory name="shardHandlerFactory"
        class="HttpShardHandlerFactory">
        <int name="socketTimeout">${socketTimeout:600000}</int>
        <int name="connTimeout">${connTimeout:60000}</int>
        </shardHandlerFactory>      
    </solr>
---
apiVersion: solr.apache.org/v1beta1
kind: SolrCloud
metadata:
  name: test
  namespace: dev
spec:
  availability:
    podDisruptionBudget:
      enabled: false
  busyBoxImage:
    repository: busybox
    tag: 1.28.0-glibc
  customSolrKubeOptions:
    configMapOptions:
      providedConfigMap: solr-config
    podOptions:
      nodeSelector:
        kubernetes.azure.com/scalesetpriority: spot
      tolerations:
        - effect: NoSchedule
          key: kubernetes.azure.com/scalesetpriority
          operator: Equal
          value: spot
  dataStorage:
    persistent:
      pvcTemplate:
        metadata: {}
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 8Gi
          storageClassName: premium-ssd-v2
      reclaimPolicy: Retain
  replicas: 3
  scaling:
    populatePodsOnScaleUp: true
    vacatePodsOnScaleDown: true
  solrAddressability:
    commonServicePort: 80
    podPort: 8983
  solrImage:
    repository: solr
    tag: 9.8.0
  solrJavaMem: '-Xms8G -Xmx8G'
  solrLogLevel: INFO
  solrSecurity:
    authenticationType: Basic
    basicAuthSecret: solr-basic-auth
    bootstrapSecurityJson:
      key: security.json
      name: security-json
  zookeeperRef:
    provided:
      image:
        pullPolicy: IfNotPresent
        repository: pravega/zookeeper
      maxUnavailableReplicas: 1
      persistence:
        reclaimPolicy: Retain
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
          storageClassName: standard-ssd-zrs
      replicas: 3
      zookeeperPodPolicy:
        nodeSelector:
          kubernetes.azure.com/scalesetpriority: spot
        securityContext:
          fsGroup: 1000
        tolerations:
          - effect: NoSchedule
            key: kubernetes.azure.com/scalesetpriority
            operator: Equal
            value: spot
```@jdgiddings: I have confirmed it is not a race condition. I spun up the zookeepers first and then spun up the first solr node. The issue appears to lie in the command on `setup-zk`
@jdgiddings: I ran some further tests. I added the SECURITY_JSON environment variable to the `solrcloud-node` container and (roughly) ran the commands from `setup-zk` and they worked! Here's the commands in the order I ran them

1. `solr zk cp zk:/security.json /tmp/current_security.json` Worked
2. `cat /tmp/current_security.json` was {}
3. `echo $SECURITY_JSON > /tmp/security.json` Worked
4. `cat /tmp/security.json` was the full contents of security.json
5. `solr zk cp /tmp/security.json zk:/security.json` Worked

This *should be* pretty much identical to the commands that run in `setup-zk`, but I see the logs from `setup-zk` are `No security.json found. Put new security.json in ZK` which indicates that `solr zk cp zk:/security.json /tmp/current_security.json` is exiting with code 1. I'm not sure why this would happen, considering that it exits with code 0 when running it in `solrcloud-node`

the `ZK_HOST`, `ZK_CHROOT`, and `ZK_SERVER` environment variables are identical between the two containers
@jdgiddings: @janhoy I found a temporary solution. the `solr zk` command is not respecting the ZK_HOST environment variable. I created an init container identical to `setup-zk` but, I added --zk-host ${ZK_HOST} to all the `solr zk` commands. It works perfect! Here's the script:

```
solr zk cp --zk-host ${ZK_HOST} zk:/security.json /tmp/current_security.json >/dev/null 2>&1; GET_CURRENT_SECURITY_JSON_EXIT_CODE=$?; if [ ${GET_CURRENT_SECURITY_JSON_EXIT_CODE} -eq 0 ]; then if [ ! -s /tmp/current_security.json ] || grep -q '^{}$' /tmp/current_security.json ]; then echo $SECURITY_JSON > /tmp/security.json; solr zk cp --zk-host ${ZK_HOST} /tmp/security.json zk:/security.json >/dev/null 2>&1; echo 'Blank security.json found. Put new security.json in ZK'; fi; elif [ ${GET_CURRENT_SECURITY_JSON_EXIT_CODE} -eq 1 ]; then echo $SECURITY_JSON > /tmp/security.json; solr zk cp --zk-host ${ZK_HOST} /tmp/security.json zk:/security.json >/dev/null 2>&1; echo 'No security.json found. Put new security.json in ZK'; fi
```@janhoy: Thanks for testing further. Turns out many CLI commands document that `ZK_HOST` can be used in place of `--zk-host`, but they never attempt to read the environment. What `CLIUtils#getZkHost` does instead is to fall back to connecting to Solr itself to query the system-info endpoint to get zkHost. While this works in many cases locally, when run in our init container, Solr is not yet running, so it won't work.

I believe this shuold be solved in Solr CLI so it respects `ZK_HOST` variable.@gerlowskija: > I believe this shuold be solved in Solr CLI so it respects ZK_HOST variable.

We should definitely fix this in Solr, but it might also be worth an operator workaround, since the operator currently aims to support a range of versions that will include 9.8 for quite some time.

There's a PR [here](https://github.com/apache/solr-operator/pull/756) to update the `bin/solr zk` invocation that's seemingly already most of the way there...@janhoy: > There's a PR [here](https://github.com/apache/solr-operator/pull/756) to update the `bin/solr zk` invocation that's seemingly already most of the way there...

Fixing it in Solr for 9.8.1 would hopefully make it work for all 9.x versions except 9.8.0, but I have not double checked that "solr zk" in 9.0->9.7 actually pick up ZK_HOST correctly. So a operator fix may not be necessary but would probably not hurt either.@gerlowskija: Agreed - most folks will gravitate to the latest bugfix release (i.e. 9.8.1) by default once it's available, but I imagine some folks will still run 9.8.0.  So I'd love to fix this for those folks.

Solution patch:
diff --git a/controllers/util/solr_security_util.go b/controllers/util/solr_security_util.go
index 2eb8679..c11e98c 100644
--- a/controllers/util/solr_security_util.go
+++ b/controllers/util/solr_security_util.go
@@ -237,17 +237,17 @@ func addHostHeaderToProbe(httpGet *corev1.HTTPGetAction, host string) {
 }
 
 func cmdToPutSecurityJsonInZk() string {
-	cmd := " solr zk cp zk:/security.json /tmp/current_security.json >/dev/null 2>&1; " +
+	cmd := " solr zk cp zk:/security.json /tmp/current_security.json -z $ZK_HOST >/dev/null 2>&1; " +
 		" GET_CURRENT_SECURITY_JSON_EXIT_CODE=$?; " +
 		"if [ ${GET_CURRENT_SECURITY_JSON_EXIT_CODE} -eq 0 ]; then " + // JSON already exists
 		"if [ ! -s /tmp/current_security.json ] || grep -q '^{}$' /tmp/current_security.json ]; then " + // File doesn't exist, is empty, or is just '{}'
 		" echo $SECURITY_JSON > /tmp/security.json;" +
-		" solr zk cp /tmp/security.json zk:/security.json >/dev/null 2>&1; " +
+		" solr zk cp /tmp/security.json zk:/security.json -z $ZK_HOST >/dev/null 2>&1; " +
 		" echo 'Blank security.json found. Put new security.json in ZK'; " +
 		"fi; " + // TODO: Consider checking a diff and still applying over the top
 		"elif [ ${GET_CURRENT_SECURITY_JSON_EXIT_CODE} -eq 1 ]; then " + // JSON doesn't exist, but not other error types
 		" echo $SECURITY_JSON > /tmp/security.json;" +
-		" solr zk cp /tmp/security.json zk:/security.json >/dev/null 2>&1; " +
+		" solr zk cp /tmp/security.json zk:/security.json -z $ZK_HOST >/dev/null 2>&1; " +
 		" echo 'No security.json found. Put new security.json in ZK'; " +
 		"fi"
 	return cmd



Test patch:
diff --git a/controllers/solrcloud_controller_basic_auth_test.go b/controllers/solrcloud_controller_basic_auth_test.go
index c271393..90eeef2 100644
--- a/controllers/solrcloud_controller_basic_auth_test.go
+++ b/controllers/solrcloud_controller_basic_auth_test.go
@@ -351,12 +351,12 @@ func expectBasicAuthConfigOnPodTemplateWithGomega(g Gomega, solrCloud *solrv1bet
 
 func expectPutSecurityJsonInZkCmd(g Gomega, expInitContainer *corev1.Container) {
 	g.Expect(expInitContainer).To(Not(BeNil()), "Didn't find the setup-zk InitContainer in the sts!")
-	expCmd := "solr zk cp zk:/security.json /tmp/current_security.json >/dev/null 2>&1;  " +
+	expCmd := "solr zk cp zk:/security.json /tmp/current_security.json -z $ZK_HOST >/dev/null 2>&1;  " +
 		"GET_CURRENT_SECURITY_JSON_EXIT_CODE=$?; if [ ${GET_CURRENT_SECURITY_JSON_EXIT_CODE} -eq 0 ]; then " +
 		"if [ ! -s /tmp/current_security.json ] || grep -q '^{}$' /tmp/current_security.json ]; then  " +
-		"echo $SECURITY_JSON > /tmp/security.json; solr zk cp /tmp/security.json zk:/security.json >/dev/null 2>&1; " +
+		"echo $SECURITY_JSON > /tmp/security.json; solr zk cp /tmp/security.json zk:/security.json -z $ZK_HOST >/dev/null 2>&1; " +
 		" echo 'Blank security.json found. Put new security.json in ZK'; fi; elif [ ${GET_CURRENT_SECURITY_JSON_EXIT_CODE} -eq 1 ]; then " +
-		" echo $SECURITY_JSON > /tmp/security.json; solr zk cp /tmp/security.json zk:/security.json >/dev/null 2>&1; " +
+		" echo $SECURITY_JSON > /tmp/security.json; solr zk cp /tmp/security.json zk:/security.json -z $ZK_HOST >/dev/null 2>&1; " +
 		" echo 'No security.json found. Put new security.json in ZK'; fi"
 	g.Expect(expInitContainer.Command[2]).To(ContainSubstring(expCmd), "setup-zk initContainer not configured to bootstrap security.json!")
 }