context: minor optmization of the rescan

Add the devices after removing the ones that do not exists
anymore, this way we avoid checking the new ones.