processor.py: Fixes url_item not processed when robots.txt denies a URL.