check if a directory exists before we try to remove it

For #6155 

cc @PVince81 @karlitschek @DeepDiver1975