Only delete directory if exists