Moves the URLItem status setting concern out of Engine.
Bundles robotexclusionrulesparser because it is not hosted in PyPI.

Resolves pip insecurity warning.
Updates readme with links and third-party code notice.
Fixes robots.txt handling logic flow.