Repository: gluster/glusterfs

Issue #2462: Repeated mkdirs with Invalid arguments are coming in glustershd.log after upgrade to v9.x
Created: 2021-05-21T14:55:51Z
Following logs show up in glustershd.log continuously:
[MSGID: 108027] [afr-common.c:2925:afr_attempt_readsubvol_set] 0-trinity-replicate-0: no read subvols for /local/docker-registry" repeated 31 times between [2021-05-20 16:03:03.866727 +0000] and [2021-05-20 16:04:52.390320 +0000]
[2021-05-20 16:05:07.873483 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-trinity-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-05-20 16:05:07.873510 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-trinity-client-4: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]
[2021-05-20 16:05:07.873883 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-trinity-client-3: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]

Op version of glusterd:
[root@master02 glusterfs]#  gluster volume get all cluster.op-version
Option                                   Value
------                                   -----
cluster.op-version                       70200

cluster.use-anonymous-inode is supposed to be enabled only when the op-version is 9.x but it is enabled even without it. This is leading to issues where the heals won't complete etc.
[root@balin ~]# gluster volume get trinity cluster.use-anonymous-inode
Option                                   Value
------                                   -----
cluster.use-anonymous-inode              yes

Duplicate issue #2728: No healing, errno 22 on third replica when upgrading a 10x3 gluster from 7.9 to 9.3
Created: 2021-08-24T13:45:07Z
Description of problem:
I've tried to do a rolling upgrade on two moderately busy 10x3 distributed/replicated gluster clusters. The plan was to upgrade the bricks with the first replicas first, then the bricks with the second replicas, finally the bricks with the third replicas. The upgrade was done "by the book", i.e. brick by brick with appropriate wait time to empty the heal log in between. In both cases the upgrade went fine until the first brick of the third replica. After the new glusterd came back up, the gluster did not proceed to heal as expected.  I'm seeing lots messages like the following in glustershd.log of the first and second replicas:
[2021-08-24 12:59:50.250749 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1053:afr_selfheal_entry_do] 0-stor1-replicate-0: performing entry selfheal on b729d06b-1c8e-4bb8-aeae-2568ba44d311 
[2021-08-24 12:59:50.252616 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 
[2021-08-24 12:59:50.252635 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 
[2021-08-24 12:59:50.252643 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 
[2021-08-24 12:59:50.253948 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1053:afr_selfheal_entry_do] 0-stor1-replicate-0: performing entry selfheal on 3d704524-3fc7-4571-ad3e-b5d3ea6b54b6 
[2021-08-24 12:59:50.255679 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 
[2021-08-24 12:59:50.255727 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 
[2021-08-24 12:59:50.255738 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 
[2021-08-24 12:59:50.256956 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1053:afr_selfheal_entry_do] 0-stor1-replicate-0: performing entry selfheal on 27f1b3d1-35eb-44d6-9bed-7430efa686e8 
[2021-08-24 12:59:50.259229 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 
[2021-08-24 12:59:50.259229 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 
[2021-08-24 12:59:50.259257 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 0-stor1-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}] 

Picking out the first GFID here I see the following:
First replica:
root@dostor-chassis1-node1:~# ls -lain /glusterpool/brick/.glusterfs/b7/29/b729d06b-1c8e-4bb8-aeae-2568ba44d311/
total 570
 41517984 drwxr-xr-x   3 23000 23000   3 Aug 24 14:59 .
 41515934 drwxr-xr-x 102 23000 23000 102 Aug 24 13:36 ..
104761613 drwxr-xr-x  16 23000 23000  16 Aug 24 14:47 629803853205615187
root@dostor-chassis1-node1:~# ls -lain /glusterpool/brick/SPACE/ww3/temp/session/87
total 570
 41517984 drwxr-xr-x   3 23000 23000   3 Aug 24 14:59 .
 41515934 drwxr-xr-x 102 23000 23000 102 Aug 24 13:36 ..
104761613 drwxr-xr-x  16 23000 23000  16 Aug 24 14:47 629803853205615187

Second replica:
root@dostor-chassis2-node1:~# ls -lain /glusterpool/brick/.glusterfs/b7/29/b729d06b-1c8e-4bb8-aeae-2568ba44d311/
total 570
 41547345 drwxr-xr-x   3 23000 23000   3 Aug 24 14:59 .
 41546272 drwxr-xr-x 102 23000 23000 102 Aug 24 13:36 ..
104838579 drwxr-xr-x  16 23000 23000  16 Aug 24 14:47 629803853205615187
root@dostor-chassis2-node1:~# ls -lain /glusterpool/brick/SPACE/ww3/temp/session/87
total 570
 41547345 drwxr-xr-x   3 23000 23000   3 Aug 24 14:59 .
 41546272 drwxr-xr-x 102 23000 23000 102 Aug 24 13:36 ..
104838579 drwxr-xr-x  16 23000 23000  16 Aug 24 14:47 629803853205615187

Third replica (upgraded last):
root@dostor-chassis3-node1:~# ls -lain /glusterpool/brick/.glusterfs/b7/29/b729d06b-1c8e-4bb8-aeae-2568ba44d311/
total 611
 41582506 drwxr-xr-x   5 23000 23000   5 Aug 24 14:59 .
 41581076 drwxr-xr-x 102 23000 23000 102 Aug 24 13:36 ..
103437617 drwxr-xr-x 173 23000 23000 173 Aug 23 14:18 629695002169562587
104426605 drwxr-xr-x  13 23000 23000  13 Aug 23 13:53 629712799250765987
 39864845 drwxr-xr-x  16 23000 23000  16 Aug 24 14:47 629803853205615187
root@dostor-chassis3-node1:~# ls -lain /glusterpool/brick/SPACE/ww3/temp/session/87
total 611
 41582506 drwxr-xr-x   5 23000 23000   5 Aug 24 14:59 .
 41581076 drwxr-xr-x 102 23000 23000 102 Aug 24 13:36 ..
103437617 drwxr-xr-x 173 23000 23000 173 Aug 23 14:18 629695002169562587
104426605 drwxr-xr-x  13 23000 23000  13 Aug 23 13:53 629712799250765987
 39864845 drwxr-xr-x  16 23000 23000  16 Aug 24 14:47 629803853205615187

Apparently during the time that the third brick was being upgraded, these two directories were deleted on bricks chassis1-node1 and chassis2-node1.
The exact command to reproduce the issue:
Standard upgrade procedure:
 # systemctl stop glusterd
 # systemctl stop glustereventsd
 # killall -9 glusterd glusterfs glusterfsd
 # apt-get dist-upgrade
 # shutdown -fr now

The full output of the command that failed:
No single command failure observed.
Expected results:
"gluster volume heal ... info" would show growing heal-backlog while glusterfsd was stopped; then after the reboot, heal backlog would shrink quickly to zero entries.
Mandatory info:
- The output of the gluster volume info command:
# gluster v info
 
Volume Name: stor1
Type: Distributed-Replicate
Volume ID: efd1bbd5-68ab-4d1b-8169-5b08316904fe
Status: Started
Snapshot Count: 0
Number of Bricks: 10 x 3 = 30
Transport-type: tcp
Bricks:
Brick1: 10.111.53.10:/glusterpool/brick
Brick2: 10.111.53.34:/glusterpool/brick
Brick3: 10.111.53.58:/glusterpool/brick
Brick4: 10.111.53.12:/glusterpool/brick
Brick5: 10.111.53.36:/glusterpool/brick
Brick6: 10.111.53.60:/glusterpool/brick
Brick7: 10.111.53.14:/glusterpool/brick
Brick8: 10.111.53.38:/glusterpool/brick
Brick9: 10.111.53.62:/glusterpool/brick
Brick10: 10.111.53.16:/glusterpool/brick
Brick11: 10.111.53.40:/glusterpool/brick
Brick12: 10.111.53.64:/glusterpool/brick
Brick13: 10.111.53.18:/glusterpool/brick
Brick14: 10.111.53.42:/glusterpool/brick
Brick15: 10.111.53.66:/glusterpool/brick
Brick16: 10.111.53.20:/glusterpool/brick
Brick17: 10.111.53.44:/glusterpool/brick
Brick18: 10.111.53.68:/glusterpool/brick
Brick19: 10.111.53.22:/glusterpool/brick
Brick20: 10.111.53.46:/glusterpool/brick
Brick21: 10.111.53.70:/glusterpool/brick
Brick22: 10.111.53.24:/glusterpool/brick
Brick23: 10.111.53.48:/glusterpool/brick
Brick24: 10.111.53.72:/glusterpool/brick
Brick25: 10.111.53.26:/glusterpool/brick
Brick26: 10.111.53.50:/glusterpool/brick
Brick27: 10.111.53.74:/glusterpool/brick
Brick28: 10.111.53.28:/glusterpool/brick
Brick29: 10.111.53.52:/glusterpool/brick
Brick30: 10.111.53.76:/glusterpool/brick
Options Reconfigured:
performance.client-io-threads: on
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
network.inode-lru-limit: 200000
performance.stat-prefetch: on
server.event-threads: 4
client.event-threads: 4
network.ping-timeout: 17
server.tcp-user-timeout: 17
cluster.locking-scheme: granular

- The output of the gluster volume status command:
Status of volume: stor1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.111.53.10:/glusterpool/brick       49152     0          Y       4270 
Brick 10.111.53.34:/glusterpool/brick       49152     0          Y       1944 
Brick 10.111.53.58:/glusterpool/brick       49153     0          Y       568008
Brick 10.111.53.12:/glusterpool/brick       49152     0          Y       3447 
Brick 10.111.53.36:/glusterpool/brick       49152     0          Y       1719 
Brick 10.111.53.60:/glusterpool/brick       49152     0          Y       3594 
Brick 10.111.53.14:/glusterpool/brick       49152     0          Y       3137 
Brick 10.111.53.38:/glusterpool/brick       49152     0          Y       3730 
Brick 10.111.53.62:/glusterpool/brick       49152     0          Y       3698 
Brick 10.111.53.16:/glusterpool/brick       49152     0          Y       3416 
Brick 10.111.53.40:/glusterpool/brick       49152     0          Y       4240 
Brick 10.111.53.64:/glusterpool/brick       49152     0          Y       3865 
Brick 10.111.53.18:/glusterpool/brick       49152     0          Y       2656 
Brick 10.111.53.42:/glusterpool/brick       49152     0          Y       3300 
Brick 10.111.53.66:/glusterpool/brick       49152     0          Y       1368 
Brick 10.111.53.20:/glusterpool/brick       49152     0          Y       2235 
Brick 10.111.53.44:/glusterpool/brick       49152     0          Y       4487 
Brick 10.111.53.68:/glusterpool/brick       49152     0          Y       4529 
Brick 10.111.53.22:/glusterpool/brick       49152     0          Y       2676 
Brick 10.111.53.46:/glusterpool/brick       49152     0          Y       3469 
Brick 10.111.53.70:/glusterpool/brick       49152     0          Y       5170 
Brick 10.111.53.24:/glusterpool/brick       49152     0          Y       2964 
Brick 10.111.53.48:/glusterpool/brick       49152     0          Y       2194 
Brick 10.111.53.72:/glusterpool/brick       49152     0          Y       4790 
Brick 10.111.53.26:/glusterpool/brick       49152     0          Y       3116 
Brick 10.111.53.50:/glusterpool/brick       49152     0          Y       2496 
Brick 10.111.53.74:/glusterpool/brick       49152     0          Y       5801 
Brick 10.111.53.28:/glusterpool/brick       49152     0          Y       4767 
Brick 10.111.53.52:/glusterpool/brick       49152     0          Y       10800
Brick 10.111.53.76:/glusterpool/brick       49152     0          Y       30383
Self-heal Daemon on localhost               N/A       N/A        Y       568019
Self-heal Daemon on 10.111.53.24            N/A       N/A        Y       2975 
Self-heal Daemon on 10.111.53.26            N/A       N/A        Y       3137 
Self-heal Daemon on 10.111.53.42            N/A       N/A        Y       3311 
Self-heal Daemon on 10.111.53.10            N/A       N/A        Y       4281 
Self-heal Daemon on 10.111.53.36            N/A       N/A        Y       1730 
Self-heal Daemon on 10.111.53.74            N/A       N/A        Y       5812 
Self-heal Daemon on 10.111.53.28            N/A       N/A        Y       4778 
Self-heal Daemon on 10.111.53.46            N/A       N/A        Y       3484 
Self-heal Daemon on 10.111.53.34            N/A       N/A        Y       1973 
Self-heal Daemon on 10.111.53.18            N/A       N/A        Y       2667 
Self-heal Daemon on 10.111.53.52            N/A       N/A        Y       10811
Self-heal Daemon on 10.111.53.50            N/A       N/A        Y       2507 
Self-heal Daemon on 10.111.53.72            N/A       N/A        Y       4801 
Self-heal Daemon on 10.111.53.14            N/A       N/A        Y       3154 
Self-heal Daemon on 10.111.53.62            N/A       N/A        Y       3709 
Self-heal Daemon on 10.111.53.12            N/A       N/A        Y       3458 
Self-heal Daemon on 10.111.53.70            N/A       N/A        Y       5181 
Self-heal Daemon on 10.111.53.20            N/A       N/A        Y       2246 
Self-heal Daemon on 10.111.53.22            N/A       N/A        Y       2687 
Self-heal Daemon on 10.111.53.16            N/A       N/A        Y       3427 
Self-heal Daemon on 10.111.53.38            N/A       N/A        Y       3741 
Self-heal Daemon on 10.111.53.48            N/A       N/A        Y       2205 
Self-heal Daemon on 10.111.53.40            N/A       N/A        Y       4251 
Self-heal Daemon on 10.111.53.68            N/A       N/A        Y       4540 
Self-heal Daemon on 10.111.53.44            N/A       N/A        Y       4498 
Self-heal Daemon on 10.111.53.60            N/A       N/A        Y       3609 
Self-heal Daemon on 10.111.53.64            N/A       N/A        Y       3876 
Self-heal Daemon on 10.111.53.66            N/A       N/A        Y       1379 
Self-heal Daemon on 10.111.53.76            N/A       N/A        Y       30487
 
Task Status of Volume stor1
------------------------------------------------------------------------------
There are no active volume tasks

- The output of the gluster volume heal command:
*** = redacted due to GDPR
root@dostor-chassis3-node1:~# gluster v heal stor1
Launching heal operation to perform index self heal on volume stor1 has been successful 
Use heal info commands to check status.
root@dostor-chassis3-node1:~# gluster v heal stor1 info
Brick 10.111.53.10:/glusterpool/brick
/SPACE/ww3/miscdirs/de/lernsax/hpos/users/biologie5a/logo 
<gfid:df8f63f9-9678-4125-aa39-e27f4b69eb99> 
/SPACE/ww3/temp/session/11 
<gfid:bd4c733e-a7b0-45af-9fb1-f0baded7e667> 
/SPACE/ww3/temp/session/38 
/SPACE/ww3/temp/session/13 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/miscdirs/de/l***/users/*** 
/SPACE/ww3/temp/session/26 
/SPACE/ww3/temp/session/29 
/SPACE/ww3/temp/session/44 
/SPACE/ww3/temp/session/35 
<gfid:a9fdac85-a0f7-4c5b-8af8-5ae92ec9d5d7> 
<gfid:3741c9ca-266f-4404-821c-f6a738db98ce> 
/SPACE/ww3/temp/session/64 
/SPACE/ww3/temp/session/43 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
<gfid:62bec4cb-6197-45f0-8d34-6b55255bec8e> 
/SPACE/ww3/temp/session/63 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
<gfid:6d9fa27a-5e98-407f-913e-1eb2bbd7eaf0> 
/SPACE/ww3/temp/session/25 
/SPACE/ww3/temp/session/36 
/SPACE/ww3/temp/session/52 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/temp/session/01 
/SPACE/ww3/temp/session/66 
/SPACE/ww3/webdirs/de/l***/users 
/SPACE/ww3/temp/session/06 
<gfid:ac2cca47-6c52-45d4-beb2-5e28d7c83681> 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/temp/session/05 
/SPACE/ww3/maildirs/de/l***/cur 
<gfid:8ca6ac36-512e-4453-a470-7139dd2ee032> 
/SPACE/ww3/temp/session/47 
/SPACE/ww3/temp/session/33 
/SPACE/ww3/temp/session/54 
<gfid:c59bad84-e323-4567-9be4-0a580392928c> 
/SPACE/ww3/temp/session/87 
/SPACE/ww3/temp/session/08 
<gfid:34a26afa-6729-4121-8ede-f86787e71d37> 
/SPACE/ww3/temp/session/14 
/SPACE/ww3/temp/session/59 
/SPACE/ww3/temp/session/20 
/SPACE/ww3/temp/session/98 
/SPACE/ww3/temp/session/45 
/SPACE/ww3/temp/session/62 
/SPACE/ww3/temp/session/82 
/SPACE/ww3/maildirs/de/l***/users 
/SPACE/ww3/temp/session/67 
/SPACE/ww3/temp/session/91 
/SPACE/ww3/temp/session/21 
/SPACE/ww3/maildirs/de/l***/Maildir/new 
<gfid:9471c9ef-9af1-4bca-ab64-fa40fa7b89f9> 
<gfid:b72cd903-79e1-4ebd-9b4b-249a483c2f11> 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
<gfid:24bbd036-a060-4f29-a789-245c93d1bdef> 
/SPACE/ww3/temp/session/37 
/SPACE/ww3/temp/session/48 
<gfid:d7fc6697-5207-44f9-93e3-d9b17e536e2c> 
/SPACE/ww3/temp/session/50 
/SPACE/ww3/maildirs/de/l***/Maildir/new 
/SPACE/ww3/temp/session/34 
<gfid:85dd8adb-7212-4bfa-a448-4dd1ec190d95> 
/SPACE/ww3/temp/session/70 
Status: Connected
Number of entries: 66

Brick 10.111.53.34:/glusterpool/brick
/SPACE/ww3/temp/session/64 
<gfid:3741c9ca-266f-4404-821c-f6a738db98ce> 
/SPACE/ww3/temp/session/43 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/temp/session/29 
/SPACE/ww3/temp/session/44 
/SPACE/ww3/temp/session/26 
<gfid:a9fdac85-a0f7-4c5b-8af8-5ae92ec9d5d7> 
/SPACE/ww3/temp/session/35 
<gfid:1a76b061-70bb-4bbc-994f-7b80cd3ec6ff> 
/SPACE/ww3/temp/session/52 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/temp/session/01 
/SPACE/ww3/temp/session/63 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
<gfid:62bec4cb-6197-45f0-8d34-6b55255bec8e> 
<gfid:6d9fa27a-5e98-407f-913e-1eb2bbd7eaf0> 
/SPACE/ww3/temp/session/36 
/SPACE/ww3/temp/session/25 
<gfid:df8f63f9-9678-4125-aa39-e27f4b69eb99> 
/SPACE/ww3/miscdirs/de/l***/logo 
/SPACE/ww3/temp/session/11 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/temp/session/13 
/SPACE/ww3/miscdirs/de/l***/users/*** 
<gfid:bd4c733e-a7b0-45af-9fb1-f0baded7e667> 
/SPACE/ww3/temp/session/38 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/maildirs/de/l***/Maildir/new 
<gfid:9471c9ef-9af1-4bca-ab64-fa40fa7b89f9> 
<gfid:b72cd903-79e1-4ebd-9b4b-249a483c2f11> 
<gfid:24bbd036-a060-4f29-a789-245c93d1bdef> 
/SPACE/ww3/temp/session/37 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/temp/session/45 
/SPACE/ww3/temp/session/62 
/SPACE/ww3/temp/session/98 
/SPACE/ww3/temp/session/21 
/SPACE/ww3/temp/session/91 
/SPACE/ww3/temp/session/82 
/SPACE/ww3/maildirs/de/l***/users 
/SPACE/ww3/temp/session/67 
<gfid:85dd8adb-7212-4bfa-a448-4dd1ec190d95> 
/SPACE/ww3/temp/session/70 
/SPACE/ww3/temp/session/34 
/SPACE/ww3/temp/session/48 
/SPACE/ww3/maildirs/de/l***/Maildir/new 
/SPACE/ww3/temp/session/46 
/SPACE/ww3/temp/session/50 
/SPACE/ww3/temp/session/05 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
/SPACE/ww3/temp/session/33 
/SPACE/ww3/temp/session/47 
/SPACE/ww3/maildirs/de/l***/Maildir/cur 
<gfid:8ca6ac36-512e-4453-a470-7139dd2ee032> 
/SPACE/ww3/temp/session/66 
/SPACE/ww3/temp/session/06 
<gfid:ac2cca47-6c52-45d4-beb2-5e28d7c83681> 
/SPACE/ww3/webdirs/de/l***/users 
/SPACE/ww3/temp/session/14 
/SPACE/ww3/temp/session/20 
/SPACE/ww3/temp/session/59 
<gfid:c59bad84-e323-4567-9be4-0a580392928c> 
/SPACE/ww3/temp/session/87 
/SPACE/ww3/temp/session/54 
/SPACE/ww3/temp/session/08 
<gfid:34a26afa-6729-4121-8ede-f86787e71d37> 
Status: Connected
Number of entries: 67

Brick 10.111.53.58:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.12:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.36:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.60:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.14:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.38:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.62:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.16:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.40:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.64:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.18:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.42:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.66:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.20:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.44:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.68:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.22:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.46:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.70:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.24:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.48:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.72:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.26:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.50:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.74:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.28:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.52:/glusterpool/brick
Status: Connected
Number of entries: 0

Brick 10.111.53.76:/glusterpool/brick
Status: Connected
Number of entries: 0

**- Provide logs present on following locations of client and server nodes -
/var/log/glusterfs/
Will provide log download location in follow-up.
**- Is there any crash ? Provide the backtrace and coredump
None observed.
Additional info:
The glusterfs upgrade was done as a part of an overall upgrade of Debian 10 to Debian 11, glusterfs 7.9 to 9.3, ZFS from 0.8.5 to 2.0.5.
- The operating system / glusterfs version:
Pre-upgrade:
root@dostor-chassis3-node2:~# cat /etc/debian_version 
10.7

root@dostor-chassis3-node2:~# uname -a
Linux dostor-chassis3-node2 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) x86_64 GNU/Linux

root@dostor-chassis3-node2:~# dpkg -l glusterfs-server zfsutils-linux
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name             Version      Architecture Description
+++-================-============-============-================================================
ii  glusterfs-server 7.9-1        amd64        clustered file-system (server package)
ii  zfsutils-linux   0.8.5-pve1   amd64        command-line tools to manage OpenZFS filesystems

Post-upgrade:
root@dostor-chassis3-node1:~# cat /etc/debian_version 
11.0

root@dostor-chassis3-node1:~# uname -a
Linux dostor-chassis3-node1 5.11.22-3-pve #1 SMP PVE 5.11.22-6 (Wed, 28 Jul 2021 10:51:12 +0200) x86_64 GNU/Linux

root@dostor-chassis3-node1:~# dpkg -l glusterfs-server zfsutils-linux
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name             Version      Architecture Description
+++-================-============-============-================================================
ii  glusterfs-server 9.3-1        amd64        clustered file-system (server package)
ii  zfsutils-linux   2.0.5-pve1   amd64        command-line tools to manage OpenZFS filesystems

Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration

Closing comment (2021-08-24T13:49:59Z): Related to #2462