Determine access source addresses to add and to remove