Two things added:
1. Added possibility to add attributes to the links.  This allows me page easier with javascript:
(tagBuilder, page) => tagBuilder.Attributes["data-page"] = page.ToString(CultureInfo.InvariantCulture))

2. Added id for pager control.  This allows me to replace the whole containing div with a template.  In retrospect if I put the containing div in the template the problem would maybe be solved as well.