device: Don't enslave unmanaged devices

They don't notify of their removal as they don't transition to UNMANAGED
state in remove_device() and thus leave hanging in bridges despite the
actual device is gone. It's probably a good idea to ignore them altogether.