Repository: gluster/glusterfs

Issue #2443: Core dumps on Gluster 9 - 3 replicas 
Created: 2021-05-17T18:34:44Z
Description of problem:
A number of services use a 3 replicated gluster mount, rebooting one of the replicas always results in a core dump on the machine taking over the services. Once coredumped, the directory shows Socket not connected.
How to trigger: rebooting one of the replicas. Recreated the volume from scratch, still same problem. The services in HA might hold the mounted gluster for a reasonably long time and a smooth umount might not occur. However, in 3 replica this shouldn't matter.
gdb /usr/sbin/glusterfs -c core.3549
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/sbin/glusterfsd...Reading symbols from /usr/sbin/glusterfsd...(no debugging symbols found)...done.
(no debugging symbols found)...done.
warning: core file may not match specified executable file.
[New LWP 3605]
[New LWP 3551]
[New LWP 3553]
[New LWP 3549]
[New LWP 3557]
[New LWP 3556]
[New LWP 3558]
[New LWP 3604]
[New LWP 3648]
[New LWP 3559]
[New LWP 3647]
[New LWP 3646]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/glusterfs --process-name fuse --volfile-server=10.141.255.254 --volfi'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f27bb5a985b in __insert_and_merge () from /usr/lib64/glusterfs/9.2/xlator/protocol/client.so
Missing separate debuginfos, use: debuginfo-install glusterfs-fuse-9.2-1.el7.x86_64
Mandatory info:
- The output of the gluster volume info command:
gluster volume info local
Volume Name: local
Type: Replicate
Volume ID: 04e9d8b5-2225-46c2-bcd2-78356e0581f1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.141.255.254:/gluster/local
Brick2: 10.141.255.253:/gluster/local
Brick3: 10.141.11.1:/glusterssd/local
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
performance.write-behind: off
performance.flush-behind: off
cluster.granular-entry-heal: enable
(core dump error is also there without the granular)
- The output of the gluster volume status command:
gluster volume status local
Status of volume: local
Gluster process                             TCP Port  RDMA Port  Online  Pid
Brick 10.141.255.254:/gluster/local         49154     0          Y       2680
Brick 10.141.255.253:/gluster/local         49154     0          Y       2757
Brick 10.141.11.1:/glusterssd/local         49154     0          Y       6150
Self-heal Daemon on localhost               N/A       N/A        Y       2693
Self-heal Daemon on 10.141.11.1             N/A       N/A        Y       6196
Self-heal Daemon on 10.141.255.253          N/A       N/A        Y       2770
Task Status of Volume local
There are no active volume tasks
- The output of the gluster volume heal command:
gluster volume heal local
Launching heal operation to perform index self heal on volume local has been successful
Use heal info commands to check status.
Will not solve it
gluster volume heal local info
Brick 10.141.255.254:/gluster/local
Status: Connected
Number of entries: 0
Brick 10.141.255.253:/gluster/local
Status: Connected
Number of entries: 0
Brick 10.141.11.1:/glusterssd/local
Status: Connected
Number of entries: 0
Socket still disconnected
**- Provide logs present on following locations of client and server nodes -
/var/log/glusterfs/
The message "W [MSGID: 114061] [client-common.c:2895:client_pre_lk_v2] 0-local-client-1: remote_fd is -1. EBADFD [{gfid=c8963045-a4b5-4dd6-b794-7ea4acb6614d}, {errno=77}, {error=File descriptor in
bad state}]" repeated 37 times between [2021-05-17 16:57:50.016590 +0000] and [2021-05-17 16:57:50.106029 +0000]
pending frames:
frame : type(1) op(LOOKUP)
frame : type(1) op(LK)
frame : type(1) op(FSYNC)
frame : type(1) op(FSYNC)
frame : type(1) op(FSYNC)
frame : type(1) op(FSYNC)
frame : type(1) op(FSYNC)
frame : type(1) op(FSYNC)
frame : type(1) op(FSYNC)
frame : type(1) op(FSYNC)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2021-05-17 16:57:50 +0000
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 9.2
/lib64/libglusterfs.so.0(+0x28d2f)[0x7f9b7b6dcd2f]
/lib64/libglusterfs.so.0(gf_print_trace+0x36a)[0x7f9b7b6e7dba]
/lib64/libc.so.6(+0x36400)[0x7f9b79912400]
/usr/lib64/glusterfs/9.2/xlator/protocol/client.so(+0x3b85b)[0x7f9b6c52d85b]
/usr/lib64/glusterfs/9.2/xlator/protocol/client.so(+0x3cc00)[0x7f9b6c52ec00]
/usr/lib64/glusterfs/9.2/xlator/protocol/client.so(+0x5903d)[0x7f9b6c54b03d]
/lib64/libgfrpc.so.0(+0xf7f1)[0x7f9b7b4867f1]
/lib64/libgfrpc.so.0(+0xfb65)[0x7f9b7b486b65]
/lib64/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f9b7b483133]
/usr/lib64/glusterfs/9.2/rpc-transport/socket.so(+0x4418)[0x7f9b6f23b418]
/usr/lib64/glusterfs/9.2/rpc-transport/socket.so(+0x9d21)[0x7f9b6f240d21]
/lib64/libglusterfs.so.0(+0x8e13c)[0x7f9b7b74213c]
/lib64/libpthread.so.0(+0x7ea5)[0x7f9b7a114ea5]
/lib64/libc.so.6(clone+0x6d)[0x7f9b799da9fd]
**- Is there any crash ? Provide the backtrace and coredump
I don't see the debug rpm in the repo?
Additional info:

- The operating system / glusterfs version:
9.2 rpm repo,

Duplicate issue #2563: Crash
Created: 2021-06-22T19:51:06Z
Description of problem:
3 node replicated cluster running ubuntu server 20 with glusterfs 9.2.  Got this immediately after accessing login page for radarr. ls of mount location reports "ls: cannot open directory '.': Transport endpoint is not connected"
Expected results:
No crashing.
- The output of the gluster volume info command:
Volume Name: gv0
Type: Replicate
Volume ID: e2d38fea-3c32-4c9f-a805-dad9c61a45e1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: h0.local.example.com:/home/user/gfs/brick0
Brick2: h1.local.example.com:/home/user/gfs/brick0
Brick3: h2.local.example.com:/home/user/gfs/brick0
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
cluster.granular-entry-heal: on
storage.owner-uid: 1000
storage.owner-gid: 1000

- The output of the gluster volume status command:
Status of volume: gv0
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick h0.local.example.com:/home/user/g
fs/brick0                                   49152     0          Y       1098
Brick h1.local.example.com:/home/user/g
fs/brick0                                   49152     0          Y       18989
Brick h2.local.example.com:/home/user/g
fs/brick0                                   49152     0          Y       14591
Self-heal Daemon on localhost               N/A       N/A        Y       1127
Self-heal Daemon on h2.local.example.com N/A       N/A        Y       14608
Self-heal Daemon on h1.local.example.com N/A       N/A        Y       19006
Task Status of Volume gv0
------------------------------------------------------------------------------
There are no active volume tasks

- The output of the gluster volume heal command:
user@h0:/mnt/docker0$ sudo gluster volume heal gv0
Launching heal operation to perform index self heal on volume gv0 has been successful
Use heal info commands to check status.
user@h0:/mnt/docker0$ sudo gluster volume heal gv0 info
Brick h0.local.example.com:/home/user/gfs/brick0
Status: Connected
Number of entries: 0

Brick h1.local.example.com:/home/user/gfs/brick0
Status: Connected
Number of entries: 0

Brick h2.local.example.com:/home/user/gfs/brick0
Status: Connected
Number of entries: 0

**- Provide logs present on following locations of client and server nodes -
/var/log/glusterfs/
**- Is there any crash ? Provide the backtrace and coredump
[2021-06-22 18:56:14.780185 +0000] I [MSGID: 100030] [glusterfsd.c:2685:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=9.2}, {cmdlinestr=/usr/sbin/glusterfs --process-name fuse --volfile-server=localhost --volfile-id=/gv0 /mnt/docker0}]
[2021-06-22 18:56:14.786395 +0000] I [glusterfsd.c:2421:daemonize] 0-glusterfs: Pid of current running process is 1128
[2021-06-22 18:56:14.791675 +0000] I [MSGID: 101190] [event-epoll.c:669:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}]
[2021-06-22 18:56:14.792855 +0000] I [MSGID: 101190] [event-epoll.c:669:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}]
[2021-06-22 18:56:15.839347 +0000] I [glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of available volfile servers: h1.local.example.com:24007 h2.local.example.com:24007
[2021-06-22 18:56:15.852398 +0000] I [io-stats.c:3706:ios_sample_buf_size_configure] 0-gv0: Configure ios_sample_buf  size is 1024 because ios_sample_interval is 0
[2021-06-22 18:56:15.855219 +0000] I [MSGID: 114020] [client.c:2319:notify] 0-gv0-client-0: parent translators are ready, attempting connect on transport []
[2021-06-22 18:56:15.855413 +0000] I [MSGID: 114020] [client.c:2319:notify] 0-gv0-client-1: parent translators are ready, attempting connect on transport []
[2021-06-22 18:56:15.855556 +0000] I [MSGID: 114020] [client.c:2319:notify] 0-gv0-client-2: parent translators are ready, attempting connect on transport []
[2021-06-22 18:56:15.855570 +0000] E [MSGID: 114058] [client-handshake.c:1201:client_query_portmap_cbk] 0-gv0-client-0: failed to get the port number for remote subvolume. Please run gluster volume status on server to see if brick process is running []
[2021-06-22 18:56:15.855603 +0000] I [socket.c:848:__socket_shutdown] 0-gv0-client-0: intentional socket shutdown(12)
[2021-06-22 18:56:15.855624 +0000] I [MSGID: 114018] [client.c:2227:client_rpc_notify] 0-gv0-client-0: disconnected from client, process will keep trying to connect glusterd until brick's port is available [{conn-name=gv0-client-0}]
[2021-06-22 18:56:15.855665 +0000] E [MSGID: 108006] [afr-common.c:6140:__afr_handle_child_down_event] 0-gv0-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
Final graph:
+------------------------------------------------------------------------------+
  1: volume gv0-client-0
  2:     type protocol/client
  3:     option ping-timeout 42
  4:     option remote-host h0.local.example.com
  5:     option remote-subvolume /home/user/gfs/brick0
  6:     option transport-type socket
  7:     option transport.address-family inet
  8:     option username aacd9a85-26a8-486a-9467-6a05fa42949a
  9:     option password 2f76f7fd-bb1d-42e5-bf37-6ff32e546e54
 10:     option transport.socket.ssl-enabled off
 11:     option transport.tcp-user-timeout 0
 12:     option transport.socket.keepalive-time 20
 13:     option transport.socket.keepalive-interval 2
 14:     option transport.socket.keepalive-count 9
 15:     option strict-locks off
 16:     option send-gids true
 17: end-volume
 18:
 19: volume gv0-client-1
 20:     type protocol/client
 21:     option ping-timeout 42
 22:     option remote-host h1.local.example.com
 23:     option remote-subvolume /home/user/gfs/brick0
 24:     option transport-type socket
 25:     option transport.address-family inet
 26:     option username aacd9a85-26a8-486a-9467-6a05fa42949a
 27:     option password 2f76f7fd-bb1d-42e5-bf37-6ff32e546e54
 28:     option transport.socket.ssl-enabled off
 29:     option transport.tcp-user-timeout 0
 30:     option transport.socket.keepalive-time 20
 31:     option transport.socket.keepalive-interval 2
 32:     option transport.socket.keepalive-count 9
 33:     option strict-locks off
 34:     option send-gids true
 35: end-volume
 36:
 37: volume gv0-client-2
 38:     type protocol/client
 39:     option ping-timeout 42
 40:     option remote-host h2.local.example.com
 41:     option remote-subvolume /home/user/gfs/brick0
 42:     option transport-type socket
 43:     option transport.address-family inet
 44:     option username aacd9a85-26a8-486a-9467-6a05fa42949a
 45:     option password 2f76f7fd-bb1d-42e5-bf37-6ff32e546e54
 46:     option transport.socket.ssl-enabled off
 47:     option transport.tcp-user-timeout 0
 48:     option transport.socket.keepalive-time 20
 49:     option transport.socket.keepalive-interval 2
 50:     option transport.socket.keepalive-count 9
 51:     option strict-locks off
 52:     option send-gids true
 53: end-volume
 54:
 55: volume gv0-replicate-0
 56:     type cluster/replicate
 57:     option afr-pending-xattr gv0-client-0,gv0-client-1,gv0-client-2
 58:     option volume-id 8c66b64e-fbd8-4374-b914-013b6e761c6f
 59:     option granular-entry-heal on
 60:     option use-compound-fops off
 61:     option use-anonymous-inode yes
 62:     subvolumes gv0-client-0 gv0-client-1 gv0-client-2
 63: end-volume
 64:
 65: volume gv0-dht
 66:     type cluster/distribute
 67:     option lock-migration off
 68:     option force-migration off
 69:     subvolumes gv0-replicate-0
 70: end-volume
 71:
 72: volume gv0-utime
 73:     type features/utime
 74:     option noatime on
 75:     subvolumes gv0-dht
 76: end-volume
 77:
 78: volume gv0-write-behind
 79:     type performance/write-behind
 80:     subvolumes gv0-utime
 81: end-volume
 82:
 83: volume gv0-quick-read
 84:     type performance/quick-read
 85:     subvolumes gv0-write-behind
 86: end-volume
 87:
 88: volume gv0-md-cache
 89:     type performance/md-cache
 90:     subvolumes gv0-quick-read
 91: end-volume
 92:
 93: volume gv0
 94:     type debug/io-stats
 95:     option log-level INFO
 96:     option threads 16
 97:     option latency-measurement off
 98:     option count-fop-hits off
 99:     option global-threading off
100:     subvolumes gv0-md-cache
101: end-volume
102:
103: volume meta-autoload
104:     type meta
105:     subvolumes gv0
106: end-volume
107:
+------------------------------------------------------------------------------+
[2021-06-22 18:56:15.856235 +0000] I [rpc-clnt.c:1967:rpc_clnt_reconfig] 0-gv0-client-2: changing port to 49153 (from 0)
[2021-06-22 18:56:15.856257 +0000] I [socket.c:848:__socket_shutdown] 0-gv0-client-2: intentional socket shutdown(14)
[2021-06-22 18:56:15.856274 +0000] I [rpc-clnt.c:1967:rpc_clnt_reconfig] 0-gv0-client-1: changing port to 49153 (from 0)
[2021-06-22 18:56:15.856290 +0000] I [socket.c:848:__socket_shutdown] 0-gv0-client-1: intentional socket shutdown(13)
[2021-06-22 18:56:15.856778 +0000] I [MSGID: 114057] [client-handshake.c:1126:select_server_supported_programs] 0-gv0-client-2: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
[2021-06-22 18:56:15.856922 +0000] I [MSGID: 114057] [client-handshake.c:1126:select_server_supported_programs] 0-gv0-client-1: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
[2021-06-22 18:56:15.857278 +0000] I [MSGID: 114046] [client-handshake.c:855:client_setvolume_cbk] 0-gv0-client-2: Connected, attached to remote volume [{conn-name=gv0-client-2}, {remote_subvol=/home/user/gfs/brick0}]
[2021-06-22 18:56:15.857290 +0000] I [MSGID: 108005] [afr-common.c:6062:__afr_handle_child_up_event] 0-gv0-replicate-0: Subvolume 'gv0-client-2' came back up; going online.
[2021-06-22 18:56:15.857419 +0000] I [MSGID: 114046] [client-handshake.c:855:client_setvolume_cbk] 0-gv0-client-1: Connected, attached to remote volume [{conn-name=gv0-client-1}, {remote_subvol=/home/user/gfs/brick0}]
[2021-06-22 18:56:15.857446 +0000] I [MSGID: 108002] [afr-common.c:6434:afr_notify] 0-gv0-replicate-0: Client-quorum is met
[2021-06-22 18:56:15.858626 +0000] I [fuse-bridge.c:5292:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.31
[2021-06-22 18:56:15.858640 +0000] I [fuse-bridge.c:5924:fuse_graph_sync] 0-fuse: switched to graph 0
[2021-06-22 18:56:18.856198 +0000] I [rpc-clnt.c:1967:rpc_clnt_reconfig] 0-gv0-client-0: changing port to 49152 (from 0)
[2021-06-22 18:56:18.856599 +0000] I [MSGID: 114057] [client-handshake.c:1126:select_server_supported_programs] 0-gv0-client-0: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
[2021-06-22 18:56:18.857001 +0000] I [MSGID: 114046] [client-handshake.c:855:client_setvolume_cbk] 0-gv0-client-0: Connected, attached to remote volume [{conn-name=gv0-client-0}, {remote_subvol=/home/user/gfs/brick0}]
[2021-06-22 18:56:21.916593 +0000] I [MSGID: 108031] [afr-common.c:3201:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-0
[2021-06-22 18:56:32.822476 +0000] E [fuse-helpers.c:201:frame_fill_groups] 0-fuse: failed to open /proc/0/status: No such file or directory
The message "I [MSGID: 108031] [afr-common.c:3201:afr_local_discovery_cbk] 0-gv0-replicate-0: selecting local read_child gv0-client-0" repeated 2 times between [2021-06-22 18:56:21.916593 +0000] and [2021-06-22 18:57:57.604168 +0000]
pending frames:
frame : type(1) op(READ)
frame : type(1) op(LK)
frame : type(1) op(FLUSH)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2021-06-22 18:58:58 +0000
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 9.2
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x25ca4)[0x7f4d49b20ca4]
/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x486)[0x7f4d49b2c0f6]
/lib/x86_64-linux-gnu/libc.so.6(+0x46210)[0x7f4d498cf210]
/usr/lib/x86_64-linux-gnu/glusterfs/9.2/xlator/protocol/client.so(+0x3a281)[0x7f4d44341281]
/usr/lib/x86_64-linux-gnu/glusterfs/9.2/xlator/protocol/client.so(+0x3b06b)[0x7f4d4434206b]
/usr/lib/x86_64-linux-gnu/glusterfs/9.2/xlator/protocol/client.so(+0x582ed)[0x7f4d4435f2ed]
/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfde6)[0x7f4d49acbde6]
/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x1013d)[0x7f4d49acc13d]
/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x2e)[0x7f4d49ac8ade]
/usr/lib/x86_64-linux-gnu/glusterfs/9.2/rpc-transport/socket.so(+0x566c)[0x7f4d4544466c]
/usr/lib/x86_64-linux-gnu/glusterfs/9.2/rpc-transport/socket.so(+0xb80c)[0x7f4d4544a80c]
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x8a9c3)[0x7f4d49b859c3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f4d49a86609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f4d499ab293]

- The operating system / glusterfs version:
ubuntu server 20 with glusterfs 9.2

Closing comment (2021-06-23T17:10:58Z): This is a duplicate of #2443, already fixed, yet to be released. Thanks for taking the time to post all the details! Really appreciate your help!