LdapLink::sweep, avoid freeing smart allocated objects

This is totally untested, but if #1643 is correct, then this likely is as well.