UrlFile::sweep, avoid freeing smart allocated objects

Thanks @jdelong, for pointing me in the right direction.