Do not try to remove directories we do not own