Package: deploy-to-kubernetes
Deploying a Distributed AI Stack to Kubernetes on CentOSInstall and manage a Kubernetes cluster (version 1.13.3) with helm on a single CentOS 7 vm or in multi-host mode that runs the cluster on 3 CentOS 7 vms. Once running, you can deploy a distributed, scalable python stack capable of delivering a resilient REST service with JWT for authentication and Swagger for development. This service uses a decoupled REST API with two distinct worker backends for routing simple database read and write tasks vs long-running tasks that can use a Redis cache and do not need a persistent database connection. This is handy for not only simple CRUD applications and use cases, but also serving a secure multi-tenant environment where multiple users manage long-running tasks like training deep neural networks that are capable of making near-realtime predictions.This guide was built for deploying theAntiNex stack of docker containersand theStock Analysis Engineon a Kubernetes single host or multi-host cluster.Managing a Multi-Host Kubernetes Cluster with an External DNS ServerCert Manager with Let’s Encrypt SSL supportA Native Ceph Cluster for Persistent Volume Management with KVMA Third-party Rook Ceph Cluster for Persistent VolumesMinio S3 Object StoreRedisPostgresDjango REST API with JWT and SwaggerDjango REST API Celery WorkersJupyterCore Celery WorkerspgAdmin4(Optional) Splunk with TCP and HEC Service EndpointsGetting StartedNotePlease ensure for single-vm hosting that the CentOS machine has at least 4 CPU cores and more than 8 GB ram. Here is a screenshot of the CPU utilization during AI training with only 3 cores:OverviewThis guide installs the following systems and a storage solutionRook with Ceph cluster (default)or NFS volumes to prepare the host for running containers and automatically running them on host startup:KubernetesHelm and TillerMinio S3 StoragePersistent Storage Volumes using Rook with Ceph clusteror optional NFS Volumes mounted at:/data/k8/redis,/data/k8/postgres,/data/k8/pgadminFlannel CNIInstallHere is a video showing how to prepare the host to run a local Kubernetes cluster:Preparing the host to run Kubernetes requires run this as rootsudo su
./prepare.shNoteThis has only been tested on CentOS 7 and Ubuntu 18.04 and requires commenting out all swap entries in/etc/fstabto workWarningThis guide used to install the cluster on Ubuntu 18.04, but after seeing high CPU utilization after a few days of operation this guide was moved to CentOS 7. The specific issues on Ubuntu were logged injournalctl-xeand appeared to be related to “volumes not being found” and “networking disconnects”.ValidateInstall Kubernetes ConfigRun as your usermkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configOr use the script:./user-install-kubeconfig.shCheck the Kubernetes Versionkubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"b1b29978270dc22fecc592ac55d903350454310a", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?Confirm the Kubernetes Pods Are Runningkubectl get pods -n kube-systemNAME                            READY     STATUS    RESTARTS   AGE
coredns-78fcdf6894-k8srv        1/1       Running   0          4m
coredns-78fcdf6894-xx8bt        1/1       Running   0          4m
etcd-dev                        1/1       Running   0          3m
kube-apiserver-dev              1/1       Running   0          3m
kube-controller-manager-dev     1/1       Running   0          3m
kube-flannel-ds-m8k9w           1/1       Running   0          4m
kube-proxy-p4blg                1/1       Running   0          4m
kube-scheduler-dev              1/1       Running   0          3m
tiller-deploy-759cb9df9-wxvp8   1/1       Running   0          4mDeploy Redis and Postgres and the Nginx IngressHere is a video showing how to deploy Postgres, Redis, Nginx Ingress, and the pgAdmin4 as pods in the cluster:NotePostgres, pgAdmin4 and Redis use Rook Ceph to persist dataHere are the commands to deploy Postgres, Redis, Nginx Ingress, and pgAdmin4 in the cluster:NotePlease ensure helm is installed and the tiller pod in thekube-systemnamespace is theRunningstate or Redis will encounter deployment issuesInstall Go using the./tools/install-go.sh scriptor with the commands:# note go install has only been tested on CentOS 7 and Ubuntu 18.04:
sudo su
GO_VERSION="1.11"
GO_OS="linux"
GO_ARCH="amd64"
go_file="go${GO_VERSION}.${GO_OS}-${GO_ARCH}.tar.gz"
curl https://dl.google.com/go/${go_file} --output /tmp/${go_file}
export GOPATH=$HOME/go/bin
export PATH=$PATH:$GOPATH:$GOPATH/bin
tar -C $HOME -xzf /tmp/${go_file}
$GOPATH/go get github.com/blang/expenv
# make sure to add GOPATH and PATH to ~/.bashrc./user-install-kubeconfig.sh
./deploy-resources.shIf you want to deploy splunk you can add it as an argument:./deploy-resources.sh splunkIf you want to deploy splunk with Let’s Encrypt make sure to addprodas an argument:./deploy-resources.sh splunk prodStart ApplicationsHere is a video showing how to start the Django REST Framework, Celery Workers, Jupyter, and the AntiNex Core as pods in the cluster:Start all applications as your user with the command:./start.shIf you want to deploy the splunk-ready application builds, you can add it as an argument:./start.sh splunkIf you want to deploy the splunk-ready application builds integrated with Let’s Encrypt TLS encryption, just addprodas an argument:./start.sh splunk prodNoteTheCert Manageris set to staging mode by default and requires theprodargument to prevent accidentally getting blocked due to Lets Encrypt rate limitsConfirm Pods are RunningDepending on how fast your network connection is the initial container downloads can take a few minutes. Please wait until all pods areRunningbefore continuing.kubectl get podsRun a Database MigrationHere is a video showing how to apply database schema migrations in the cluster:To apply new Django database migrations, run the following command:./api/migrate-db.shAdd Ingress Locations to /etc/hostsWhen running locally (also known in these docs asdevmode), all ingress urls need to resolve on the network. Please append the following entries to your local/etc/hostsfile on the127.0.0.1line:sudo vi /etc/hostsAppend the entries to the existing127.0.0.1line:127.0.0.1   <leave-original-values-here> api.example.com jupyter.example.com pgadmin.example.com splunk.example.com s3.example.com ceph.example.com minio.example.comUsing the Minio S3 Object StoreBy default, the Kubernetes cluster has aMinio S3 object store running on a Ceph Persistent Volume. S3 is a great solution for distributing files, datasets, configurations, static assets, build artifacts and many more across components, regions, and datacenters using an S3 distributed backend. Minio can also replicate some of theAWS Lambda event-based workflowswithMinio bucket event listeners.For reference, Minio was deployed using this script:./minio/run.shView the Verification Tests on the Minio DashboardLogin with:access key:trexaccesskeysecret key:trex123321https://minio.example.com/minio/s3-verification-tests/Test Minio S3 with Bucket Creation and File Upload and DownloadRun from inside the API container./api/ssh.sh
source /opt/venv/bin/activate && run_s3_test.pyExample logs:creating test file: run-s3-test.txt
connecting: http://minio-service:9000
checking bucket=s3-verification-tests exists
upload_file(run-s3-test.txt, s3-verification-tests, s3-worked-on-2018-08-12-15-21-02)
upload_file(s3-verification-tests, s3-worked-on-2018-08-12-15-21-02, download-run-s3-test.txt)
download_filename=download-run-s3-test.txt contents: tested on: 2018-08-12 15:21:02
exitRun from outside the Kubernetes clusterNoteThis tool requires the pythonboto3pip is installedsource ./minio/envs/ext.env
./minio/run_s3_test.pyVerify the files were uploaded to Miniohttps://minio.example.com/minio/s3-verification-tests/Using the Rook Ceph ClusterBy default, the Kubernetes cluster is running aRook Ceph cluster for storagewhich provides HA persistent volumes and claims.You can review the persistent volumes and claims using the Ceph Dashboard:https://ceph.example.comCreate a UserCreate the usertrexwith password123321on the REST API../api/create-user.shDeployed Web ApplicationsHere are the hosted web application urls. These urls are made accessible by the included nginx-ingress.View Django REST FrameworkLogin with:user:trexpassword:123321https://api.example.comView SwaggerLogin with:user:trexpassword:123321https://api.example.com/swaggerView JupyterLogin with:password:adminhttps://jupyter.example.comView pgAdminLogin with:user:admin@admin.compassword:123321https://pgadmin.example.comView Minio S3 Object StorageLogin with:access key:trexaccesskeysecret key:trex123321https://minio.example.comView Cephhttps://ceph.example.comView SplunkLogin with:user:trexpassword:123321https://splunk.example.comTraining AI with the Django REST APIThese steps install theAntiNex python clientfor training a deep neural network to predict attack packets from recorded network data (all of which is already included in the docker containers).Create a virtual environment and install the clientvirtualenv -p python3 /opt/venv && source /opt/venv/bin/activate
pip install antinex-clientWatch the application logsFrom a separate terminal, you can tail the Django REST API logs with the command:./api/logs.shFrom a separate terminal, you can tail the Django Celery Worker logs with the command:./worker/logs.shFrom a separate terminal, you can tail the AntiNex Core Worker logs with the command:./core/logs.shNoteUsectrl + cto stop these log tailing commandsTrain a Deep Neural Network on KubernetesWith virtual environment set up, we can use the client to train a deep neural network with the included datasets:Notethis can take a few minutes to finish depending on your hosting resourcesai -a https://api.example.com -u trex -p 123321 -s -f ./tests/scaler-full-django-antinex-simple.jsonWhile you wait, here is a video showing the training and get results:Get the AI Job Recordai_get_job.py -a https://api.example.com -u trex -p 123321 -i 1Get the AI Training Job Resultsai_get_results.py -a https://api.example.com -u trex -p 123321 -i 1 -sStandalone DeploymentsBelow are steps to manually deploy each component in the stack with Kubernetes.Deploy Redis./redis/run.shOr manually with the commands:echo "deploying persistent volume for redis"
kubectl apply -f ./redis/pv.yml
echo "deploying Bitnami redis stable with helm"
helm install \
    --name redis stable/redis \
    --set rbac.create=true \
    --values ./redis/redis.ymlConfirm ConnectivityThe following commands assume you haveredis-toolsinstalled (sudoapt-getinstallredis-tools).redis-cli -h $(kubectl describe pod redis-master-0 | grep IP | awk '{print $NF}') -p 6379
10.244.0.81:6379> info
10.244.0.81:6379> exitDebug Redis ClusterExamine Redis Masterkubectl describe pod redis-master-0Examine Persistent Volume Claimkubectl get pvc
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
redis-ceph-data           Bound     pvc-1a88e3a6-9df8-11e8-8047-0800270864a8   8Gi        RWO            rook-ceph-block   46mExamine Persistent Volumekubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                             STORAGECLASS      REASON    AGE
pvc-1a88e3a6-9df8-11e8-8047-0800270864a8   8Gi        RWO            Delete           Bound     default/redis-ceph-data           rook-ceph-block             46mPossible ErrorsCreate the Persistent VolumesWarning  FailedMount       2m               kubelet, dev       MountVolume.SetUp failed for volume "redis-pv" : mount failed: exit status 32./pvs/create-pvs.shDelete Redishelm del --purge redis
release "redis" deletedDelete Persistent Volume and ClaimDelete Claimkubectl delete pvc redis-data-redis-master-0Delete Volumekubectl delete pv redis-pv
persistentvolume "redis-pv" deletedDeploy PostgresInstall GoUsing Crunchy Data’s postgres containers requires having go installed. Go can be installed using the./tools/install-go.sh scriptor with the commands:# note go install has only been tested on CentOS 7 and Ubuntu 18.04:
sudo su
GO_VERSION="1.11"
GO_OS="linux"
GO_ARCH="amd64"
go_file="go${GO_VERSION}.${GO_OS}-${GO_ARCH}.tar.gz"
curl https://dl.google.com/go/${go_file} --output /tmp/${go_file}
export GOPATH=$HOME/go/bin
export PATH=$PATH:$GOPATH:$GOPATH/bin
tar -C $HOME -xzf /tmp/${go_file}
$GOPATH/go get github.com/blang/expenv
# make sure to add GOPATH and PATH to ~/.bashrcStartStart thePostgres containerwithin Kubernetes:./postgres/run.shDebug PostgresExamine Postgreskubectl describe pod primary

Type    Reason     Age   From               Message
----    ------     ----  ----               -------
Normal  Scheduled  2m    default-scheduler  Successfully assigned default/primary to dev
Normal  Pulling    2m    kubelet, dev       pulling image "crunchydata/crunchy-postgres:centos7-10.4-1.8.3"
Normal  Pulled     2m    kubelet, dev       Successfully pulled image "crunchydata/crunchy-postgres:centos7-10.4-1.8.3"
Normal  Created    2m    kubelet, dev       Created container
Normal  Started    2m    kubelet, dev       Started containerExamine Persistent Volume Claimkubectl get pvc
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
pgadmin4-http-data        Bound     pvc-19031825-9df8-11e8-8047-0800270864a8   400M       RWX            rook-ceph-block   46m
primary-pgdata            Bound     pvc-17652595-9df8-11e8-8047-0800270864a8   400M       RWX            rook-ceph-block   46mExamine Persistent Volumekubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                             STORAGECLASS      REASON    AGE
pvc-17652595-9df8-11e8-8047-0800270864a8   400M       RWX            Delete           Bound     default/primary-pgdata            rook-ceph-block             47m
pvc-19031825-9df8-11e8-8047-0800270864a8   400M       RWX            Delete           Bound     default/pgadmin4-http-data        rook-ceph-block             47mDeploy pgAdminPlease confirm go is installed with theInstall Go section.StartStart thepgAdmin4 containerwithin Kubernetes:./pgadmin/run.shGet Logs./pgadmin/logs.shSSH into pgAdmin./pgadmin/ssh.shDeploy Django REST APIUse these commands to manage theDjango REST Framework podswithin Kubernetes.Start./api/run.shRun a Database MigrationTo apply a django database migration run the following command:./api/migrate-db.shGet Logs./api/logs.shSSH into the API./api/ssh.shDeploy Django Celery WorkersUse these commands to manage theDjango Celery Worker podswithin Kubernetes.Start./worker/run.shGet Logs./worker/logs.shSSH into the Worker./worker/ssh.shDeploy AntiNex CoreUse these commands to manage theBackend AntiNex Core podswithin Kubernetes.Start./core/run.shGet Logs./core/logs.shSSH into the API./core/ssh.shDeploy JupyterUse these commands to manage theJupyter podswithin Kubernetes.Start./jupyter/run.shLogin to JupyterLogin with:password:adminhttps://jupyter.example.comGet Logs./jupyter/logs.shSSH into Jupyter./jupyter/ssh.shDeploy SplunkUse these commands to manage theSplunk containerwithin Kubernetes.Start./splunk/run.shLogin to SplunkLogin with:user:trexpassword:123321https://splunk.example.comSearching in SplunkHere is the splunk searching command line tool I use with these included applications:https://github.com/jay-johnson/spylunkingWith search example documentation:https://spylunking.readthedocs.io/en/latest/scripts.html#examplesSearch using SpylunkingFind logs in splunk using thespcommand line tool:sp -q 'index="antinex" | reverse' -u trex -p 123321 -a $(./splunk/get-api-fqdn.sh) -i antinexFind Django REST API Logs in Splunksp -q 'index="antinex" AND name=api | head 20 | reverse' -u trex -p 123321 -a $(./splunk/get-api-fqdn.sh) -i antinexFind Django Celery Worker Logs in Splunksp -q 'index="antinex" AND name=worker | head 20 | reverse' -u trex -p 123321 -a $(./splunk/get-api-fqdn.sh) -i antinexFind Core Logs in Splunksp -q 'index="antinex" AND name=core | head 20 | reverse' -u trex -p 123321 -a $(./splunk/get-api-fqdn.sh) -i antinexFind Jupyter Logs in Splunksp -q 'index="antinex" AND name=jupyter | head 20 | reverse' -u trex -p 123321 -a $(./splunk/get-api-fqdn.sh) -i antinexExample for debuggingspsplunk connectivity from inside an API Pod:kubectl exec -it api-59496ccb5f-2wp5t -n default echo 'starting search' && /bin/bash -c "source /opt/venv/bin/activate && sp -q 'index="antinex" AND hostname=local' -u trex -p 123321 -a 10.101.107.205:8089 -i antinex"Get Logs./splunk/logs.shSSH into Splunk./splunk/ssh.shDeploy Nginx IngressThis project is currently using thenginx-ingressinstead of theKubernetes Ingress using nginx. Use these commands to manage and debug the nginx ingress within Kubernetes.NoteThe default Yaml file annotations only work with thenginx-ingress customizationsStart./ingress/run.shGet Logs./ingress/logs.shSSH into the Ingress./ingress/ssh.shView Ingress Nginx ConfigWhen troubleshooting the nginx ingress, it is helpful to view the nginx configs inside the container. Here is how to view the configs:./ingress/view-configs.shView a Specific Ingress ConfigurationIf you know the pod name and the namespace for the nginx-ingress, then you can view the configs from the command line with:app_name="jupyter"
app_name="pgadmin"
app_name="api"
use_namespace="default"
pod_name=$(kubectl get pods -n ${use_namespace} | awk '{print $1}' | grep nginx | head -1)
kubectl exec -it ${pod_name} -n ${use_namespace} cat /etc/nginx/conf.d/${use_namespace}-${app_name}-ingress.confDeploy SplunkStartTo deploy splunk you can add the argumentsplunkto the./deploy-resources.sh splunkscript. Or you can manually run it with the command:./splunk/run.shOr if you want to use Let’s Encrypt for SSL:./splunk/run.sh prodDeploy Splunk-Ready ApplicationsAfter deploying the splunk pod, you can deploy the splunk-ready applications with the command:./start.sh splunkGet Logs./splunk/logs.shSSH into Splunk./splunk/ssh.shView Ingress Config./splunk/view-ingress-config.shCreate your own self-signed x509 TLS Keys, Certs and Certificate Authority with AnsibleIf you have openssl installed you can use this ansible playbook to create your own certificate authority (CA), keys and certs.Create the CA, Keys and Certificatescd ansible
ansible-playbook -i inventory_dev create-x509s.ymlCheck the CA, x509, keys and certificates for the client and server were createdls -l ./sslDeploying Your Own x509 TLS Encryption files as Kubernetes SecretsThis is a work in progress, but indevmode the cert-manager is not in use. Instead the cluster utilizes pre-generated x509s TLS SSL files created with theincluded ansible playbook create-x509s.yml. Once created, you can deploy them as Kubernetes secrets using thedeploy-secrets.shscript and reload them at any time in the future.Deploy SecretsRun this to create the TLS secrets:./ansible/deploy-secrets.shList Secretskubectl get secrets | grep tls
tls-ceph                kubernetes.io/tls                     2         36m
tls-client              kubernetes.io/tls                     2         36m
tls-database            kubernetes.io/tls                     2         36m
tls-docker              kubernetes.io/tls                     2         36m
tls-jenkins             kubernetes.io/tls                     2         36m
tls-jupyter             kubernetes.io/tls                     2         36m
tls-k8                  kubernetes.io/tls                     2         36m
tls-kafka               kubernetes.io/tls                     2         36m
tls-kibana              kubernetes.io/tls                     2         36m
tls-minio               kubernetes.io/tls                     2         36m
tls-nginx               kubernetes.io/tls                     2         36m
tls-pgadmin             kubernetes.io/tls                     2         36m
tls-phpmyadmin          kubernetes.io/tls                     2         36m
tls-rabbitmq            kubernetes.io/tls                     2         36m
tls-redis               kubernetes.io/tls                     2         36m
tls-restapi             kubernetes.io/tls                     2         36m
tls-s3                  kubernetes.io/tls                     2         36m
tls-splunk              kubernetes.io/tls                     2         36m
tls-webserver           kubernetes.io/tls                     2         36mReload SecretsIf you want to deploy new TLS secrets at any time, use thereloadargument with thedeploy-secrets.shscript. Doing so will delete the original secrets and recreate all of them using the new TLS values:./ansible/deploy-secrets.sh -rDeploy Cert Manager with Let’s EncryptUse these commands to manage theCert Manager with Let’s Encrypt SSL supportwithin Kubernetes. By default, the cert manager is deployed only inprodmode. If you run it in production mode, then it will install real, valid x509 certificates fromLet’s Encryptinto the nginx-ingress automatically.Start with Let’s Encrypt x509 SSL CertificatesStart the cert manager inprodmode to enable Let’s Encrypt TLS Encryption with the command:./start.sh prodOr manually with the command:./cert-manager/run.sh prodIf you have splunk you can just add it to the arguments:./start.sh splunk prodView LogsWhen using the production mode, make sure to view the logs to ensure you are not being blocked due to rate limiting:./cert-manager/logs.shStop the Cert ManagerIf you notice things are not working correctly, you can quickly prevent yourself from getting blocked by stopping the cert manager with the command:./cert-manager/_uninstall.shNoteIf you get blocked due to rate-limits it will show up in the cert-manager logs like:I0731 07:53:43.313709       1 sync.go:273] Error issuing certificate for default/api.antinex.com-tls: error getting certificate from acme server: acme: urn:ietf:params:acme:error:rateLimited: Error finalizing order :: too many certificates already issued for exact set of domains: api.antinex.com: see https://letsencrypt.org/docs/rate-limits/
E0731 07:53:43.313738       1 sync.go:182] [default/api.antinex.com-tls] Error getting certificate 'api.antinex.com-tls': secret "api.antinex.com-tls" not foundDebuggingTo reduce debugging issues, the cert manager ClusterIssuer objects use the same name for staging and production mode. This is nice because you do not have to update all the annotations to deploy on production vs staging:The cert manager starts and defines the issuer name for both production and staging as:--set ingressShim.defaultIssuerName=letsencrypt-issuerMake sure to set any nginx ingress annotations that need Let’s Encrypt SSL encryption to these values:annotations:
  kubernetes.io/tls-acme: "true"
  kubernetes.io/ingress.class: "nginx"
  certmanager.k8s.io/cluster-issuer: "letsencrypt-issuer"TroubleshootingCustomize Minio and How to TroubleshootChange the Minio Access and Secret KeysChange the secrets file:minio/secrets/default_access_keys.ymlChange theaccess_keyandsecret_keyvalues after generating the new base64 string values for the secrets file:echo -n "NewAccessKey" | base64
TmV3QWNjZXNzS2V5
# now you can replace the access_key's value in the secrets file with the string: TmV3QWNjZXNzS2V5echo -n "NewSecretKey" | base64
TmV3U2VjcmV0S2V5
# now you can replace the secret_key's value in the secrets file with the string: TmV3QWNjZXNzS2V5Deploy the secrets filekubectl apply -f ./minio/secrets/default_access_keys.ymlRestart the Minio Podkubectl delete pod -l app=minioIf you have changed the default access and secret keys, then you will need to export the following environment variables as needed to make sure the./minio/run_s3_test.pytest script works:export S3_ACCESS_KEY=<minio access key: trexaccesskey - default>
export S3_SECRET_KEY=<minio secret key: trex123321 - default>
export S3_REGION_NAME=<minio region name: us-east-1 - default>
export S3_ADDRESS=<minio service endpoint: external address found with the script ./minio/get-s3-endpoint.sh and the internal cluster uses the service: minio-service:9000>
# examples of setting up a minio env files are in: ./minio/envsView the Minio DashboardLogin with:access key:trexaccesskeysecret key:trex123321https://minio.example.comGet S3 Internal EndpointIf you want to use the Minio S3 service within the cluster please use the endpoint:minio-service:9000or source the internal environment file:source ./minio/envs/int.envGet S3 External EndpointIf you want to use the Minio S3 service from outside the cluser please use the endpoint provided by the script:./minio/get-s3-endpoint.sh
# which for this documentation was the minio service's Endpoints:
# 10.244.0.103:9000or source the external environment file:source ./minio/envs/ext.envDebugging StepsLoad the Minio S3 external environment variables:source ./minio/envs/ext.envRun the S3 Verification test script./minio/run_s3_test.pyConfirm Verification Keys are showing up in this Minio S3 buckethttps://minio.example.com/minio/s3-verification-tests/If not please use the describe tools in./minio/describe-*.shto grab the logs andplease file a GitHub issueDescribe Pod./minio/describe-service.shDescribe Service./minio/describe-service.shDescribe Ingress./minio/describe-ingress.shUninstall Minio./minio/_uninstall.shCeph TroubeshootingPlease refer to theRook Common Issuesfor the latest updates on how to use your Rook Ceph cluster.NoteBy default Ceph is not hosting the S3 solution unlesscephs3is passed in as an argument todeploy-resource.sh.There are included troubleshooting tools in the./rookdirectory with an overview of each below:Validate Ceph System Pods are Running./rook/view-system-pods.sh

-----------------------------------------
Getting the Rook Ceph System Pods:
kubectl -n rook-ceph-system get pod
NAME                                  READY     STATUS    RESTARTS   AGE
rook-ceph-agent-g9vzm                 1/1       Running   0          7m
rook-ceph-operator-78d498c68c-tbsdf   1/1       Running   0          7m
rook-discover-h9wj9                   1/1       Running   0          7mValidate Ceph Pods are Running./rook/view-ceph-pods.sh

-----------------------------------------
Getting the Rook Ceph Pods:
kubectl -n rook-ceph get pod
NAME                                  READY     STATUS      RESTARTS   AGE
rook-ceph-mgr-a-9c44495df-7jksz       1/1       Running     0          6m
rook-ceph-mon0-rxxsl                  1/1       Running     0          6m
rook-ceph-mon1-gqblg                  1/1       Running     0          6m
rook-ceph-mon2-7xfsq                  1/1       Running     0          6m
rook-ceph-osd-id-0-7d4d4c8794-kgr2d   1/1       Running     0          6m
rook-ceph-osd-prepare-dev-kmsn9       0/1       Completed   0          6m
rook-ceph-tools                       1/1       Running     0          6mValidate Persistent Volumes are Boundkubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                             STORAGECLASS      REASON    AGE
pvc-03e6e4ef-9df8-11e8-8047-0800270864a8   1Gi        RWO            Delete           Bound     default/certs-pv-claim            rook-ceph-block             46m
pvc-0415de24-9df8-11e8-8047-0800270864a8   1Gi        RWO            Delete           Bound     default/configs-pv-claim          rook-ceph-block             46m
pvc-0441307f-9df8-11e8-8047-0800270864a8   1Gi        RWO            Delete           Bound     default/datascience-pv-claim      rook-ceph-block             46m
pvc-0468ef73-9df8-11e8-8047-0800270864a8   1Gi        RWO            Delete           Bound     default/frontendshared-pv-claim   rook-ceph-block             46m
pvc-04888222-9df8-11e8-8047-0800270864a8   1Gi        RWO            Delete           Bound     default/staticfiles-pv-claim      rook-ceph-block             46m
pvc-1c3e359d-9df8-11e8-8047-0800270864a8   10Gi       RWO            Delete           Bound     default/minio-pv-claim            rook-ceph-block             46mValidate Persistent Volume Claims are Boundkubectl get pvc
NAME                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
certs-pv-claim            Bound     pvc-03e6e4ef-9df8-11e8-8047-0800270864a8   1Gi        RWO            rook-ceph-block   47m
configs-pv-claim          Bound     pvc-0415de24-9df8-11e8-8047-0800270864a8   1Gi        RWO            rook-ceph-block   47m
datascience-pv-claim      Bound     pvc-0441307f-9df8-11e8-8047-0800270864a8   1Gi        RWO            rook-ceph-block   47m
frontendshared-pv-claim   Bound     pvc-0468ef73-9df8-11e8-8047-0800270864a8   1Gi        RWO            rook-ceph-block   47m
minio-pv-claim            Bound     pvc-1c3e359d-9df8-11e8-8047-0800270864a8   10Gi       RWO            rook-ceph-block   46mCreate a Persistent Volume ClaimGoing forward, Ceph will automatically create a persistent volume if one is not available for binding to an available Persistent Volume Claim. To create a new persistent volume, just create a claim and verify the Rook Ceph cluster created the persistent volume and both are bound to each other.kubectl apply -f pvs/pv-staticfiles-ceph.ymlVerify the Persistent Volume is Boundkubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                          STORAGECLASS      REASON    AGE
pvc-77afbc7a-9ade-11e8-b293-0800270864a8   20Gi       RWO            Delete           Bound     default/staticfiles-pv-claim   rook-ceph-block             2sVerify the Persistent Volume Claim is Boundkubectl get pvc
NAME                   STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
staticfiles-pv-claim   Bound     pvc-77afbc7a-9ade-11e8-b293-0800270864a8   20Gi       RWO            rook-ceph-block   11sDescribe Persistent Volumeskubectl describe pv pvc-c88fc37b-9adf-11e8-9fae-0800270864a8
Name:            pvc-c88fc37b-9adf-11e8-9fae-0800270864a8
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by=ceph.rook.io/block
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    rook-ceph-block
Status:          Bound
Claim:           default/certs-pv-claim
Reclaim Policy:  Delete
Access Modes:    RWO
Capacity:        20Gi
Node Affinity:   <none>
Message:
Source:
    Type:       FlexVolume (a generic volume resource that is provisioned/attached using an exec based plugin)
    Driver:     ceph.rook.io/rook-ceph-system
    FSType:     xfs
    SecretRef:  <nil>
    ReadOnly:   false
    Options:    map[clusterNamespace:rook-ceph image:pvc-c88fc37b-9adf-11e8-9fae-0800270864a8 pool:replicapool storageClass:rook-ceph-block]
Events:         <none>Show Ceph Cluster Status./rook/show-ceph-status.sh

----------------------------------------------
Getting the Rook Ceph Status with Toolbox:
kubectl -n rook-ceph exec -it rook-ceph-tools ceph status
cluster:
    id:     7de1988c-03ea-41f3-9930-0bde39540552
    health: HEALTH_OK

services:
    mon: 3 daemons, quorum rook-ceph-mon2,rook-ceph-mon0,rook-ceph-mon1
    mgr: a(active)
    osd: 1 osds: 1 up, 1 in

data:
    pools:   1 pools, 100 pgs
    objects: 12 objects, 99 bytes
    usage:   35443 MB used, 54756 MB / 90199 MB avail
    pgs:     100 active+cleanShow Ceph OSD Status./rook/show-ceph-osd-status.sh

----------------------------------------------
Getting the Rook Ceph OSD Status with Toolbox:
kubectl -n rook-ceph exec -it rook-ceph-tools ceph osd status
+----+-------------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| id |                 host                |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+-------------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | rook-ceph-osd-id-0-7d4d4c8794-kgr2d | 34.6G | 53.4G |    0   |     0   |    0   |     0   | exists,up |
+----+-------------------------------------+-------+-------+--------+---------+--------+---------+-----------+Show Ceph Free Space./rook/show-ceph-df.sh

----------------------------------------------
Getting the Rook Ceph df with Toolbox:
kubectl -n rook-ceph exec -it rook-ceph-tools ceph df
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED
    90199M     54756M       35443M         39.29
POOLS:
    NAME            ID     USED     %USED     MAX AVAIL     OBJECTS
    replicapool     1        99         0        50246M          12Show Ceph RDOS Free Space./rook/show-ceph-rados-df.sh

----------------------------------------------
Getting the Rook Ceph rados df with Toolbox:
kubectl -n rook-ceph exec -it rook-ceph-tools rados df
POOL_NAME   USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD   WR_OPS WR
replicapool   99      12      0     12                  0       0        0    484 381k     17 7168

total_objects    12
total_used       35443M
total_avail      54756M
total_space      90199MOut of IP AddressesFlannel can exhaust all available ip addresses in the CIDR network range. When this happens please run the following command to clean up the local cni network files:./tools/reset-flannel-cni-networks.shAntiNex Stack StatusHere are the AntiNex repositories, documentation and build reports:ComponentBuildDocs LinkDocs BuildREST APIDocsCore WorkerDocsNetwork PipelineDocsAI UtilsDocsClientDocsReset ClusterHere is a video showing how to reset the local Kubernetes cluster.Please be careful as these commands will shutdown all containers and reset the Kubernetes cluster.Run as root:sudo su
kubeadm reset -f
./prepare.shOr use the file:sudo su
./tools/cluster-reset.shOr the full reset and deploy once ready:sudo su
cert_env=dev; ./tools/reset-flannel-cni-networks.sh; ./tools/cluster-reset.sh ; ./user-install-kubeconfig.sh ; sleep 30; ./deploy-resources.sh splunk ${cert_env}
exit
# as your user
./user-install-kubeconfig.sh
# depending on testing vs prod:
# ./start.sh splunk
# ./start.sh splunk prodDevelopmentRight now, the python virtual environment is only used to bring in ansible for running playbooks, but it will be used in the future with the kubernetes python client as I start using it more and more.virtualenv -p python3 /opt/venv && source /opt/venv/bin/activate && pip install -e .Testingpy.testorpython setup.py testLicenseApache 2.0 - Please refer to theLICENSEfor more details