Now supports in-page robot instructions.  Via "ROBOTS" meta tag first,
or "X-Robots-Tag" tag if present in HTTP header.
Now supports in-page robot instructions.  Via "ROBOTS" meta tag first,
or "X-Robots-Tag" tag if present in HTTP header.
Merge branch 'develop' of https://github.com/Norconex/collectors into develop

Conflicts:
	norconex-collector-http/src/changes/changes.xml