Miscellaneous improvements.

 - Reorganize background task code.
 - Add Go subrepo page.
 - Add more project reference regexps.
Improve package footer.

- Add whitespace and horiontal line between main document and footer.
- Move errors to footer.
Add meta robots NOINDEX to more pages.

- Don't index search results and directory listings.
- Move directory listing pages from pkg.* templates to new dir.*
  templates.
Use hash for path -> id map.
Add importers API endpoint.