Documentation, cleanup, add URL pattern check

This adds some comments, does some cleanup, and checks the URLs are
properly formed, based on my experimentation with a search size of 5000.