Only build token replica map as needed