Added FTP web support.
DefaultURLExtractor now recognizes URLs starting with "URL=" (e.g. meta
http-equiv="refresh").
Added BASIC and DIGEST support
Updated changes.xml
Added doc about authentication
Now supports in-page robot instructions.  Via "ROBOTS" meta tag first,
or "X-Robots-Tag" tag if present in HTTP header.
Now supports in-page robot instructions.  Via "ROBOTS" meta tag first,
or "X-Robots-Tag" tag if present in HTTP header.
Merge branch 'develop' of https://github.com/Norconex/collectors into develop

Conflicts:
	norconex-collector-http/src/changes/changes.xml
Removed done task.
Improved documentation.
Added html files as test resources
Significant changes to URL processing, with support for sitemap added.
Default sitemap implementation coming.
Fixed refactor bugs.
Sitemap and sitemap index (plain or gzip) implemented.
Updated notes.
More todos.
removed done tasks.
Merge branch 'feature-sitemap' into develop
Done for sitemap: Only load/parse sitemap locations matching root url
    - Implement lenient attribute
    - Implement xml load/save.
Sitemap now resumes well when resuming after job failure.  Sitemap
documented.