andrewrjones/ant-task-sitemap — pull request #3 by eric-brechemier (2012-08-03 22:08:00+00:00)
Title: New Feature: automatic generation of sitemap index when maximum number of URLs is reached  

Andrew, let me know what you think, when you get the time to review this new feature.

A sitemap file may only contain up to 50,000 URLs. Once this limit is reached, the sitemap must be cut into several files, and a separate index should be created to list the sitemaps.

This can be handled smoothly, by using the same name, "sitemap.xml" for a single sitemap and for the sitemap index when multiple sitemaps are created. This way, the declaration in robots.txt remains unchanged:

Sitemap: http://www.example.com/sitemap.xml

I forked the original SitemapGen4j project to implement this feature, and updated the library included in lib folder. I tested the generation of the sitemap index in new unit tests part of this project. I did not try it as part of the Ant target. This can be done by generating more than 50,000 URLs in a loop, as shown in the sample code on the home page of the original project:

WebSitemapGenerator wsg = new WebSitemapGenerator("http://www.example.com", myDir);
for (int i = 0; i < 60000; i++) wsg.addUrl("http://www.example.com/doc"+i+".html");
wsg.write(); // generate sitemap1.xml and sitemap2.xml + sitemap index as sitemap.xml
// wsg.writeSitemapsWithIndex(); // generate the sitemap_index.xml

The difference is that there is no need to call wsg.writeSitemapsWithIndex() anymore; the sitemap index is generated automatically when needed.