Change:
What happened: Creation of dynamic volume failed using Kubernetes plugin. Volume successfully created and mounted when created by heketi-cli. The problem is most probably on Kubernetes side. What you expected to happen: Dynamic volume created and bound successfully on Kubernetes. How to reproduce it (as minimally and precisely as possible): 1) Prepare configuration to Heketi API and storage class 2) Install Jupyterhub using helm. Secret: apiVersion: v1 kind: Secret metadata: name: heketi-secret namespace: default type: "" data: key: Zfsfvddf1pbjEyMw== Storage class: apiVersion: kind: StorageClass metadata: name: gfs-new provisioner: parameters: resturl: "http://qpm-gluster01:8080" clusterid: "" restauthenabled: "true" restuser: "admin" secretNamespace: "default" secretName: "heketi-secret" volumetype: "replicate:2" Jupyther hub PVC: [root jupyterhub]# kubectl describe persistentvolumeclaims hub-db-dir Name: hub-db-dir Namespace: default StorageClass: gfs-new Status: Pending Volume: Labels: app=jupyterhub chart=jupyterhub-0.8.2 component=hub heritage=Tiller release=jhub Annotations: Finalizers: [] Capacity: Access Modes: VolumeMode: Filesystem Mounted By: hub--qwnhf Anything else we need to know?: K8S error: K8S Error: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 71s (x415 over 17h) persistentvolume-controller Failed to provision volume with StorageClass "gfs-new": failed to create volume: failed to update endpoint &Endpoints{ObjectMeta:k8sioapimachinerypkgapismetav1.ObjectMeta{Name:glusterfs-dynamic--deb1-4a31-b3fd-,GenerateName:,Namespace:default,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil,DeletionGracePeriodSeconds:nil,Labels:map[string]string{ hub-db-dir,},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Subsets:[{[{qpm-gluster02 <nilnil} {qpm-gluster01 <nilnil}] [] [{ 1 TCP}]}],}: <nilHeketi log: Sep 24 22:25:37 qpm-gluster01 systemd[1]: Started Heketi Server. Sep 24 22:25:37 qpm-gluster01 heketi[]: Heketi 8.0.0 Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Loaded ssh executor Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 GlusterFS Application Loaded Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Started Node Health Cache Monitor Sep 24 22:25:37 qpm-gluster01 heketi[]: Listening on port 8080 Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Starting Node Health Status refresh Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster02 Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] WARNING 2019/09/24 22:25:47 Failed to create SSH connection to qpm-gluster02:22: dial tcp 10.20.105.61:22: getsockopt: Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] ERROR 2019/09/24 22:25:47 dial tcp 10.20.105.61:22: getsoc Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Periodic health check status: node up=false Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster01 Sep 24 22:25:48 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:25:48 Host: qpm-gluster01:22 Command: /bin Sep 24 22:25:48 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:25:48 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:25:48 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:25:46 CST; 2s ago Sep 24 22:25:48 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:25:48 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:25:48 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:25:48 qpm-gluster01 heketi[]: CGroup: Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfsd -s qpm-gluster01 --volfile-id -p /var/run/gluster/vols Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p -l /var/l Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Starting GlusterFS, a clustered file-system server... Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Started GlusterFS, a clustered file-system server. Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Periodic health check status: node up=true Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Cleaned 0 nodes from health cache Sep 24 22:27:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:27:37 Starting Node Health Status refresh Sep 24 22:27:37 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:27:37 Check Glusterd service status in node qpm-gluster02 Sep 24 22:27:38 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:27:38 Host: qpm-gluster02:22 Command: /bin Sep 24 22:27:38 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:27:38 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:27:38 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:21:20 CST; 1min 47s ago Sep 24 22:27:38 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:27:38 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:27:38 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:27:38 qpm-gluster01 heketi[]: CGroup: Sep 24 22:27:38 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO ...skipping... Sep 25 18:02:12 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 404.314µs Sep 25 18:02:13 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:13 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 47.059µs Sep 25 18:02:14 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:14 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 48.526µs Sep 25 18:02:15 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:15 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 947.302µs Sep 25 18:02:16 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:16 Host: qpm-gluster02:22 Command: /bin Sep 25 18:02:16 qpm-gluster01 heketi[]: Result: volume start: vol55484bcd5ff2f73c76a1b732899bb08e: success Sep 25 18:02:16 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:02:16 Create Volume succeeded Sep 25 18:02:16 qpm-gluster01 heketi[]: [asynchttp] INFO 2019/09/25 18:02:16 Completed job in 5.788973955s Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Completed 303 See Other in 63.842µs Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Started GET /volumes/ Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 1.387158ms Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Started GET /clusters/ Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 279.552µs Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Started GET /nodes/ Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 933.614µs Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Started GET /nodes/ Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 760.05µs Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Started DELETE /volumes/ Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Completed 202 Accepted in 1.56933ms Sep 25 18:02:16 qpm-gluster01 heketi[]: [asynchttp] INFO 2019/09/25 18:02:16 Started job Sep 25 18:02:16 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:02:16 Started async operation: Delete Volume Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:16 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 37.443µs Sep 25 18:02:16 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:16 Host: qpm-gluster01:22 Command: /bin Sep 25 18:02:16 qpm-gluster01 heketi[]: Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?Sep 25 18:02:16 qpm-gluster01 heketi[]: <cliOutputSep 25 18:02:16 qpm-gluster01 heketi[]: <opRet0</opRetSep 25 18:02:16 qpm-gluster01 heketi[]: <opErrno0</opErrnoSep 25 18:02:16 qpm-gluster01 heketi[]: <opErrstr/Sep 25 18:02:16 qpm-gluster01 heketi[]: <snapListSep 25 18:02:16 qpm-gluster01 heketi[]: <count0</countSep 25 18:02:16 qpm-gluster01 heketi[]: </snapListSep 25 18:02:16 qpm-gluster01 heketi[]: </cliOutputSep 25 18:02:17 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:17 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 984.585µs [root ~]# journalctl -u heketi -- Logs begin at Tue 2019-09-24 22:25:31 CST, end at Wed 2019-09-25 18:02:27 CST. -- Sep 24 22:25:37 qpm-gluster01 systemd[1]: Started Heketi Server. Sep 24 22:25:37 qpm-gluster01 heketi[]: Heketi 8.0.0 Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Loaded ssh executor Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 GlusterFS Application Loaded Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Started Node Health Cache Monitor Sep 24 22:25:37 qpm-gluster01 heketi[]: Listening on port 8080 Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Starting Node Health Status refresh Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster02 Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] WARNING 2019/09/24 22:25:47 Failed to create SSH connection to qpm-gluster02:22: dial tcp 10.20.105.61:22: getsockopt: Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] ERROR 2019/09/24 22:25:47 dial tcp 10.20.105.61:22: getsoc Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Periodic health check status: node up=false Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster01 Sep 24 22:25:48 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:25:48 Host: qpm-gluster01:22 Command: /bin Sep 24 22:25:48 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:25:48 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:25:48 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:25:46 CST; 2s ago Sep 24 22:25:48 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:25:48 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:25:48 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:25:48 qpm-gluster01 heketi[]: CGroup: Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfsd -s qpm-gluster01 --volfile-id -p /var/run/gluster/vols Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p -l /var/l Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Starting GlusterFS, a clustered file-system server... Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Started GlusterFS, a clustered file-system server. Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Periodic health check status: node up=true Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Cleaned 0 nodes from health cache Sep 24 22:27:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:27:37 Starting Node Health Status refresh Sep 24 22:27:37 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:27:37 Check Glusterd service status in node qpm-gluster02 Sep 24 22:27:38 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:27:38 Host: qpm-gluster02:22 Command: /bin Sep 24 22:27:38 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:27:38 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:27:38 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:21:20 CST; 1min 47s ago Sep 24 22:27:38 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:27:38 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:27:38 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:27:38 qpm-gluster01 heketi[]: CGroup: Sep 24 22:27:38 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO ...skipping... Sep 25 18:02:22 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 42.104µs Sep 25 18:02:23 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:23 Host: qpm-gluster01:22 Command: /bin Sep 25 18:02:23 qpm-gluster01 heketi[]: Result: volume stop: vol55484bcd5ff2f73c76a1b732899bb08e: success Sep 25 18:02:23 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:23 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 338.922µs Sep 25 18:02:24 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:24 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 1.116144ms Sep 25 18:02:25 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:25 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 724.84µs Sep 25 18:02:26 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:26 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 1.210774ms Sep 25 18:02:27 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:27 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 37.461µs Sep 25 18:02:28 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:28 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 27.736µs Sep 25 18:02:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:29 Host: qpm-gluster01:22 Command: /bin Sep 25 18:02:29 qpm-gluster01 heketi[]: Result: volume delete: vol55484bcd5ff2f73c76a1b732899bb08e: success Sep 25 18:02:29 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:02:29 Deleting brick Sep 25 18:02:29 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:02:29 Deleting brick Sep 25 18:02:29 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:29 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 326.536µs Sep 25 18:02:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:29 Host: qpm-gluster01:22 Command: /bin Sep 25 18:02:29 qpm-gluster01 heketi[]: Result: Sep 25 18:02:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:29 Host: qpm-gluster02:22 Command: /bin Sep 25 18:02:29 qpm-gluster01 heketi[]: Result: Sep 25 18:02:30 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:30 Host: qpm-gluster01:22 Command: /bin Sep 25 18:02:30 qpm-gluster01 heketi[]: Result: Sep 25 18:02:30 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:30 Host: qpm-gluster02:22 Command: /bin Sep 25 18:02:30 qpm-gluster01 heketi[]: Result: Sep 25 18:02:30 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:30 Host: qpm-gluster01:22 Command: /bin Sep 25 18:02:30 qpm-gluster01 heketi[]: Result: Logical volume "brick6af28af0e6eabac6e672d9efbc2d3316" successfully removed Sep 25 18:02:30 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:02:30 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 562.671µs Sep 25 18:02:30 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:30 Host: qpm-gluster02:22 Command: /bin Sep 25 18:02:30 qpm-gluster01 heketi[]: Result: Logical volume "bricke199d4c88ca6fa987604176a12fa9db3" successfully removed Sep 25 18:02:30 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:30 Host: qpm-gluster01:22 Command: /bin Sep 25 18:02:30 qpm-gluster01 heketi[]: Result: 0 Sep 25 18:02:30 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:02:30 Host: qpm-gluster02:22 Command: /bin Sep 25 18:02:30 qpm-gluster01 heketi[]: Result: 0 [root ~]# journalctl -u heketi -- Logs begin at Tue 2019-09-24 22:25:31 CST, end at Wed 2019-09-25 18:03:09 CST. -- Sep 24 22:25:37 qpm-gluster01 systemd[1]: Started Heketi Server. Sep 24 22:25:37 qpm-gluster01 heketi[]: Heketi 8.0.0 Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Loaded ssh executor Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 GlusterFS Application Loaded Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Started Node Health Cache Monitor Sep 24 22:25:37 qpm-gluster01 heketi[]: Listening on port 8080 Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Starting Node Health Status refresh Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster02 Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] WARNING 2019/09/24 22:25:47 Failed to create SSH connection to qpm-gluster02:22: dial tcp 10.20.105.61:22: getsockopt: Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] ERROR 2019/09/24 22:25:47 dial tcp 10.20.105.61:22: getsoc Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Periodic health check status: node up=false Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster01 Sep 24 22:25:48 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:25:48 Host: qpm-gluster01:22 Command: /bin Sep 24 22:25:48 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:25:48 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:25:48 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:25:46 CST; 2s ago Sep 24 22:25:48 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:25:48 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:25:48 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:25:48 qpm-gluster01 heketi[]: CGroup: Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfsd -s qpm-gluster01 --volfile-id -p /var/run/gluster/vols Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p -l /var/l Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Starting GlusterFS, a clustered file-system server... Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Started GlusterFS, a clustered file-system server. Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Periodic health check status: node up=true Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Cleaned 0 nodes from health cache Sep 24 22:27:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:27:37 Starting Node Health Status refresh Sep 24 22:27:37 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:27:37 Check Glusterd service status in node qpm-gluster02 Sep 24 22:27:38 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:27:38 Host: qpm-gluster02:22 Command: /bin Sep 24 22:27:38 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:27:38 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:27:38 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:21:20 CST; 1min 47s ago Sep 24 22:27:38 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:27:38 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:27:38 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:27:38 qpm-gluster01 heketi[]: CGroup: Sep 24 22:27:38 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO ...skipping... Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: meta-data=/dev/mapper/vg0698a82595cf974afcef3cccfa143966-bricke8e2ba7d0e74e6224f94f7a7f58fa978 isize=512 agcount=8, Sep 25 18:03:09 qpm-gluster01 heketi[]: = sectsz=4096 attr=2, projid32bit=1 Sep 25 18:03:09 qpm-gluster01 heketi[]: = crc=1 finobt=0, sparse=0 Sep 25 18:03:09 qpm-gluster01 heketi[]: data = bsize=4096 blocks=, imaxpct=25 Sep 25 18:03:09 qpm-gluster01 heketi[]: = sunit=64 swidth=64 blks Sep 25 18:03:09 qpm-gluster01 heketi[]: naming =version 2 bsize=8192 ascii-ci=0 ftype=1 Sep 25 18:03:09 qpm-gluster01 heketi[]: log =internal log bsize=4096 blocks=2560, version=2 Sep 25 18:03:09 qpm-gluster01 heketi[]: = sectsz=4096 sunit=1 blks, lazy-count=1 Sep 25 18:03:09 qpm-gluster01 heketi[]: realtime =none extsz=4096 blocks=0, rtextents=0 Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:03:09 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 53.775µs Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/25 18:03:09 Creating volume vol6d948239054f70f44dbf39c0310671f4 replica 2 Sep 25 18:03:09 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:09 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:09 qpm-gluster01 heketi[]: Result: volume create: vol6d948239054f70f44dbf39c0310671f4: success: please start the volume to access data Sep 25 18:03:10 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:03:10 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 51.199µs Sep 25 18:03:11 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:03:11 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 46.972µs Sep 25 18:03:12 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:03:12 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 47.692µs [root ~]# journalctl -u heketi -- Logs begin at Tue 2019-09-24 22:25:31 CST, end at Wed 2019-09-25 18:03:30 CST. -- Sep 24 22:25:37 qpm-gluster01 systemd[1]: Started Heketi Server. Sep 24 22:25:37 qpm-gluster01 heketi[]: Heketi 8.0.0 Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Loaded ssh executor Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 GlusterFS Application Loaded Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Started Node Health Cache Monitor Sep 24 22:25:37 qpm-gluster01 heketi[]: Listening on port 8080 Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Starting Node Health Status refresh Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster02 Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] WARNING 2019/09/24 22:25:47 Failed to create SSH connection to qpm-gluster02:22: dial tcp 10.20.105.61:22: getsockopt: Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] ERROR 2019/09/24 22:25:47 dial tcp 10.20.105.61:22: getsoc Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Periodic health check status: node up=false Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster01 Sep 24 22:25:48 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:25:48 Host: qpm-gluster01:22 Command: /bin Sep 24 22:25:48 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:25:48 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:25:48 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:25:46 CST; 2s ago Sep 24 22:25:48 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:25:48 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:25:48 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:25:48 qpm-gluster01 heketi[]: CGroup: Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfsd -s qpm-gluster01 --volfile-id -p /var/run/gluster/vols Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p -l /var/l Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Starting GlusterFS, a clustered file-system server... Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Started GlusterFS, a clustered file-system server. Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Periodic health check status: node up=true Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Cleaned 0 nodes from health cache Sep 24 22:27:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:27:37 Starting Node Health Status refresh Sep 24 22:27:37 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:27:37 Check Glusterd service status in node qpm-gluster02 Sep 24 22:27:38 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:27:38 Host: qpm-gluster02:22 Command: /bin Sep 24 22:27:38 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:27:38 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:27:38 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:21:20 CST; 1min 47s ago Sep 24 22:27:38 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:27:38 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:27:38 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:27:38 qpm-gluster01 heketi[]: CGroup: Sep 24 22:27:38 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO ...skipping... Sep 25 18:03:26 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 37.874µs Sep 25 18:03:27 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:27 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:27 qpm-gluster01 heketi[]: Result: volume delete: vol6d948239054f70f44dbf39c0310671f4: success Sep 25 18:03:27 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:03:27 Deleting brick Sep 25 18:03:27 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:03:27 Deleting brick Sep 25 18:03:27 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:03:27 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 332.418µs Sep 25 18:03:27 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:27 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:27 qpm-gluster01 heketi[]: Result: Sep 25 18:03:27 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:27 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:27 qpm-gluster01 heketi[]: Result: Sep 25 18:03:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:28 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:28 qpm-gluster01 heketi[]: Result: Sep 25 18:03:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:28 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:28 qpm-gluster01 heketi[]: Result: Sep 25 18:03:28 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:03:28 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 1.237352ms Sep 25 18:03:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:28 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:28 qpm-gluster01 heketi[]: Result: Logical volume "bricke8e2ba7d0e74e6224f94f7a7f58fa978" successfully removed Sep 25 18:03:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:28 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:28 qpm-gluster01 heketi[]: Result: Logical volume "brick4a1f48053a83f22e54494cb7dc2a9dd2" successfully removed Sep 25 18:03:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:28 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:28 qpm-gluster01 heketi[]: Result: 0 Sep 25 18:03:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:28 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:28 qpm-gluster01 heketi[]: Result: 0 Sep 25 18:03:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:29 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:29 qpm-gluster01 heketi[]: Result: Logical volume "tpe6c2bab18eea73d91dc8b82a41f8cf20" successfully removed Sep 25 18:03:29 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:03:29 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 51.186µs Sep 25 18:03:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:29 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:29 qpm-gluster01 heketi[]: Result: Logical volume "tp4a1f48053a83f22e54494cb7dc2a9dd2" successfully removed Sep 25 18:03:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:29 Host: qpm-gluster01:22 Command: /bin Sep 25 18:03:29 qpm-gluster01 heketi[]: Result: Sep 25 18:03:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:03:29 Host: qpm-gluster02:22 Command: /bin Sep 25 18:03:29 qpm-gluster01 heketi[]: Result: Sep 25 18:03:29 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:03:29 Delete Volume succeeded Sep 25 18:03:29 qpm-gluster01 heketi[]: [asynchttp] INFO 2019/09/25 18:03:29 Completed job in 15.398688681s Sep 25 18:03:30 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:03:30 qpm-gluster01 heketi[]: [negroni] Completed 204 No Content in 44.008µs [root ~]# journalctl -u heketi -- Logs begin at Tue 2019-09-24 22:25:31 CST, end at Wed 2019-09-25 18:04:30 CST. -- Sep 24 22:25:37 qpm-gluster01 systemd[1]: Started Heketi Server. Sep 24 22:25:37 qpm-gluster01 heketi[]: Heketi 8.0.0 Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Loaded ssh executor Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 GlusterFS Application Loaded Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Started Node Health Cache Monitor Sep 24 22:25:37 qpm-gluster01 heketi[]: Listening on port 8080 Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Starting Node Health Status refresh Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster02 Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] WARNING 2019/09/24 22:25:47 Failed to create SSH connection to qpm-gluster02:22: dial tcp 10.20.105.61:22: getsockopt: Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] ERROR 2019/09/24 22:25:47 dial tcp 10.20.105.61:22: getsoc Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Periodic health check status: node up=false Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster01 Sep 24 22:25:48 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:25:48 Host: qpm-gluster01:22 Command: /bin Sep 24 22:25:48 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:25:48 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:25:48 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:25:46 CST; 2s ago Sep 24 22:25:48 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:25:48 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:25:48 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:25:48 qpm-gluster01 heketi[]: CGroup: Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfsd -s qpm-gluster01 --volfile-id -p /var/run/gluster/vols Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p -l /var/l Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Starting GlusterFS, a clustered file-system server... Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Started GlusterFS, a clustered file-system server. Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Periodic health check status: node up=true Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Cleaned 0 nodes from health cache Sep 24 22:27:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:27:37 Starting Node Health Status refresh Sep 24 22:27:37 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:27:37 Check Glusterd service status in node qpm-gluster02 Sep 24 22:27:38 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:27:38 Host: qpm-gluster02:22 Command: /bin Sep 24 22:27:38 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:27:38 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:27:38 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:21:20 CST; 1min 47s ago Sep 24 22:27:38 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:27:38 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:27:38 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:27:38 qpm-gluster01 heketi[]: CGroup: Sep 24 22:27:38 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO ...skipping... Sep 25 18:04:26 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 37.624µs Sep 25 18:04:27 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:26 Host: qpm-gluster02:22 Command: /bin Sep 25 18:04:27 qpm-gluster01 heketi[]: Result: volume delete: volb5e4543ce096afe05a4a840eff1f75dc: success Sep 25 18:04:27 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:04:26 Deleting brick Sep 25 18:04:27 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:04:26 Deleting brick Sep 25 18:04:27 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:27 Host: qpm-gluster02:22 Command: /bin Sep 25 18:04:27 qpm-gluster01 heketi[]: Result: Sep 25 18:04:27 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:04:27 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 40.113µs Sep 25 18:04:27 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:27 Host: qpm-gluster01:22 Command: /bin Sep 25 18:04:27 qpm-gluster01 heketi[]: Result: Sep 25 18:04:27 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:27 Host: qpm-gluster02:22 Command: /bin Sep 25 18:04:27 qpm-gluster01 heketi[]: Result: Sep 25 18:04:27 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:27 Host: qpm-gluster01:22 Command: /bin Sep 25 18:04:27 qpm-gluster01 heketi[]: Result: Sep 25 18:04:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:28 Host: qpm-gluster02:22 Command: /bin Sep 25 18:04:28 qpm-gluster01 heketi[]: Result: Logical volume "brick0fb883071f2725dea564de4b2a997eca" successfully removed Sep 25 18:04:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:28 Host: qpm-gluster01:22 Command: /bin Sep 25 18:04:28 qpm-gluster01 heketi[]: Result: Logical volume "brick1b55942eec775bed13dd998d8603970d" successfully removed Sep 25 18:04:28 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:04:28 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 359.086µs Sep 25 18:04:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:28 Host: qpm-gluster02:22 Command: /bin Sep 25 18:04:28 qpm-gluster01 heketi[]: Result: 0 Sep 25 18:04:28 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:28 Host: qpm-gluster01:22 Command: /bin Sep 25 18:04:28 qpm-gluster01 heketi[]: Result: 0 Sep 25 18:04:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:29 Host: qpm-gluster01:22 Command: /bin Sep 25 18:04:29 qpm-gluster01 heketi[]: Result: Logical volume "tp1b55942eec775bed13dd998d8603970d" successfully removed Sep 25 18:04:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:29 Host: qpm-gluster02:22 Command: /bin Sep 25 18:04:29 qpm-gluster01 heketi[]: Result: Logical volume "tpc014cca72deab875e543b2c702e60de1" successfully removed Sep 25 18:04:29 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:04:29 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 850.294µs Sep 25 18:04:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:29 Host: qpm-gluster01:22 Command: /bin Sep 25 18:04:29 qpm-gluster01 heketi[]: Result: Sep 25 18:04:29 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:29 Host: qpm-gluster02:22 Command: /bin Sep 25 18:04:29 qpm-gluster01 heketi[]: Result: Sep 25 18:04:29 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:04:29 Delete Volume succeeded Sep 25 18:04:29 qpm-gluster01 heketi[]: [asynchttp] INFO 2019/09/25 18:04:29 Completed job in 15.143158493s Sep 25 18:04:30 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:04:30 qpm-gluster01 heketi[]: [negroni] Completed 204 No Content in 37.995µs [root ~]# journalctl -u heketi -- Logs begin at Tue 2019-09-24 22:25:31 CST, end at Wed 2019-09-25 18:04:57 CST. -- Sep 24 22:25:37 qpm-gluster01 systemd[1]: Started Heketi Server. Sep 24 22:25:37 qpm-gluster01 heketi[]: Heketi 8.0.0 Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Loaded ssh executor Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 GlusterFS Application Loaded Sep 24 22:25:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:37 Started Node Health Cache Monitor Sep 24 22:25:37 qpm-gluster01 heketi[]: Listening on port 8080 Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Starting Node Health Status refresh Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster02 Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] WARNING 2019/09/24 22:25:47 Failed to create SSH connection to qpm-gluster02:22: dial tcp 10.20.105.61:22: getsockopt: Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] ERROR 2019/09/24 22:25:47 dial tcp 10.20.105.61:22: getsoc Sep 24 22:25:47 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:47 Periodic health check status: node up=false Sep 24 22:25:47 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:25:47 Check Glusterd service status in node qpm-gluster01 Sep 24 22:25:48 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:25:48 Host: qpm-gluster01:22 Command: /bin Sep 24 22:25:48 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:25:48 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:25:48 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:25:46 CST; 2s ago Sep 24 22:25:48 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:25:48 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:25:48 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:25:48 qpm-gluster01 heketi[]: CGroup: Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfsd -s qpm-gluster01 --volfile-id -p /var/run/gluster/vols Sep 24 22:25:48 qpm-gluster01 heketi[]: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p -l /var/l Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Starting GlusterFS, a clustered file-system server... Sep 24 22:25:48 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Started GlusterFS, a clustered file-system server. Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Periodic health check status: node up=true Sep 24 22:25:48 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:25:48 Cleaned 0 nodes from health cache Sep 24 22:27:37 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/24 22:27:37 Starting Node Health Status refresh Sep 24 22:27:37 qpm-gluster01 heketi[]: [cmdexec] INFO 2019/09/24 22:27:37 Check Glusterd service status in node qpm-gluster02 Sep 24 22:27:38 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/24 22:27:38 Host: qpm-gluster02:22 Command: /bin Sep 24 22:27:38 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 24 22:27:38 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 24 22:27:38 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:21:20 CST; 1min 47s ago Sep 24 22:27:38 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 24 22:27:38 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 24 22:27:38 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 24 22:27:38 qpm-gluster01 heketi[]: CGroup: Sep 24 22:27:38 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO ...skipping... Sep 25 18:04:59 qpm-gluster01 heketi[]: [negroni] Started GET /nodes/ Sep 25 18:04:59 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 764.283µs Sep 25 18:04:59 qpm-gluster01 heketi[]: [negroni] Started DELETE /volumes/ Sep 25 18:04:59 qpm-gluster01 heketi[]: [negroni] Completed 202 Accepted in 1.494867ms Sep 25 18:04:59 qpm-gluster01 heketi[]: [asynchttp] INFO 2019/09/25 18:04:59 Started job Sep 25 18:04:59 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:04:59 Started async operation: Delete Volume Sep 25 18:04:59 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:04:59 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 35.251µs Sep 25 18:04:59 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:59 Host: qpm-gluster01:22 Command: /bin Sep 25 18:04:59 qpm-gluster01 heketi[]: Result: glusterd.service - GlusterFS, a clustered file-system server Sep 25 18:04:59 qpm-gluster01 heketi[]: Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Sep 25 18:04:59 qpm-gluster01 heketi[]: Active: active (running) since Tue 2019-09-24 22:25:46 CST; 19h ago Sep 25 18:04:59 qpm-gluster01 heketi[]: Docs: man:glusterd(8) Sep 25 18:04:59 qpm-gluster01 heketi[]: Process: ExecStart=/usr/sbin/glusterd -p --log-level GLUSTERDOPTIONS (code=exited, stat Sep 25 18:04:59 qpm-gluster01 heketi[]: Main PID: (glusterd) Sep 25 18:04:59 qpm-gluster01 heketi[]: CGroup: Sep 25 18:04:59 qpm-gluster01 heketi[]: /usr/sbin/glusterd -p --log-level INFO Sep 25 18:04:59 qpm-gluster01 heketi[]: /usr/sbin/glusterfsd -s qpm-gluster01 --volfile-id -p /var/run/gluster/vols Sep 25 18:04:59 qpm-gluster01 heketi[]: /bin/bash /var/lib/glusterd/hooks/1/create/post/S10selinux-label- --volname=vol1146042f3ba2f7586c9630a6cd286b2e Sep 25 18:04:59 qpm-gluster01 heketi[]: /usr/sbin/glusterfsd -s qpm-gluster01 --volfile-id Sep 25 18:04:59 qpm-gluster01 heketi[]: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p -l /var/l Sep 25 18:04:59 qpm-gluster01 heketi[]: /usr/bin/python -Es /usr/sbin/semanage fcontext --add -t glusterdbrickt -r s0 /var/lib/heketi/mounts/vgdd6fade2c4f5e1 Sep 25 18:04:59 qpm-gluster01 heketi[]: /sbin/loadpolicy Sep 25 18:04:59 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Starting GlusterFS, a clustered file-system server... Sep 25 18:04:59 qpm-gluster01 heketi[]: Sep 24 22:25:46 qpm-gluster01 systemd[1]: Started GlusterFS, a clustered file-system server. Sep 25 18:04:59 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:04:59 Periodic health check status: node up=true Sep 25 18:04:59 qpm-gluster01 heketi[]: [heketi] INFO 2019/09/25 18:04:59 Cleaned 0 nodes from health cache Sep 25 18:04:59 qpm-gluster01 heketi[]: [cmdexec] DEBUG 2019/09/25 18:04:59 Host: qpm-gluster01:22 Command: /bin Sep 25 18:04:59 qpm-gluster01 heketi[]: Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?Sep 25 18:04:59 qpm-gluster01 heketi[]: <cliOutputSep 25 18:04:59 qpm-gluster01 heketi[]: <opRet0</opRetSep 25 18:04:59 qpm-gluster01 heketi[]: <opErrno0</opErrnoSep 25 18:04:59 qpm-gluster01 heketi[]: <opErrstr/Sep 25 18:04:59 qpm-gluster01 heketi[]: <snapListSep 25 18:04:59 qpm-gluster01 heketi[]: <count0</countSep 25 18:04:59 qpm-gluster01 heketi[]: </snapListSep 25 18:04:59 qpm-gluster01 heketi[]: </cliOutputSep 25 18:05:00 qpm-gluster01 heketi[]: [negroni] Started GET /queue/ Sep 25 18:05:00 qpm-gluster01 heketi[]: [negroni] Completed 200 OK in 1.139606ms Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-07-18T09:09:21Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux release 7.6.1810 (Core) Kernel (e.g. ): Linux qpm-k8s-master01 4.4.186- SMP Sun Jul 21 04:06:52 EDT 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: heketi.x8664 8.0.0-1.el7 heketi-client.x8664 8.0.0-1.el7
/sig storage
/assign
I only see the following errors: I don't see any IP validation error, am I overlooking something?
The endpoint contains and which are hostname would not be updated, could you paste the topology info of heketi?
[root ~]# heketi-cli topology info Cluster Id: File: true Block: true Volumes: Nodes: Node Id: State: online Cluster Id: Zone: 2 Management Hostnames: qpm-gluster02 Storage Hostnames: qpm-gluster02 Devices: Id: Name:/dev/sdh State:online Size (GiB):447 Used (GiB):0 Free (GiB):447 Bricks: Id: Name:/dev/sdg State:online Size (GiB):447 Used (GiB):0 Free (GiB):447 Bricks: Node Id: State: online Cluster Id: Zone: 1 Management Hostnames: qpm-gluster01 Storage Hostnames: qpm-gluster01 Devices: Id: Name:/dev/sdg State:online Size (GiB):447 Used (GiB):0 Free (GiB):447 Bricks: Id: Name:/dev/sdh State:online Size (GiB):447 Used (GiB):0 Free (GiB):447 Bricks:
Heketi is not running on K8S and we don't use external DNS, name resolution is from synchronized hosts files.
needed to replace with ip.
Many thanks. It worked for me. :) Would this behavior be changed by ?
Yes, I will update later.

Discussion:
"fmt" "math" "math/rand" <ins> "net" </ins> "os" "path/filepath" "runtime"

gcli "github.com/heketi/heketi/client/api/go-client" gapi "github.com/heketi/heketi/pkg/glusterfs/api" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

return nil, fmt.Errorf("failed to get host ipaddress: %v", err) } ipaddr := dstrings.Join(nodeInfo.NodeAddRequest.Hostnames.Storage, "") <ins> // IP validates if a string is a valid IP address. ip := net.ParseIP(ipaddr) if ip == nil { return nil, fmt.Errorf("glusterfs server node ip address %s must be a valid IP address, (e.g. 10.9.8.7)", ipaddr) } </ins> dynamicHostIps = append(dynamicHostIps, ipaddr) } klog.V(3).Infof("host list :%v", dynamicHostIps)