Destroy FUSE object as late as possible.