client: close dirfrag whem rmdir

Fixes: #10387
Signed-off-by: Yan, Zheng <zyan@redhat.com>