Only call ipset if it is not in the cache