openhatch/oh-bugimporters — pull request #23 by paulproteus (2012-10-31 22:34:40+00:00)
Title: Feature/trac handles existing bug urls

This adds a test (and fixes the implementation) for the Trac bug importer handling the existing_bug_urls field in configuration files. This handles the Trac bugimporter case for http://openhatch.org/bugs/issue772 . I have local patches that seem to handle it for other importers, too, but wanted to land this separately and first.

It also adds a sample input file that demonstrates this feature.

To test:

export BUG_TRACKER_LIST=examples/sample_configuration_larger.yaml

export SCRAPY_RESULT_FILE=/tmp/results.jsonlines

export SCRAPY_LOG=/tmp/log

env/bin/scrapy runspider bugimporters/main.py -a input_filename="$BUG_TRACKER_LIST" -s TELNETCONSOLE_ENABLED=0 -s WEBSERVICE_ENABLED=0 -s FEED_FORMAT=jsonlines -s FEED_URI="$SCRAPY_RESULT_FILE" -s LOG_FILE="$SCRAPY_LOG" -s CONCURRENT_REQUESTS_PER_DOMAIN=1 -s CONCURRENT_REQUESTS=200 -s DEPTH_PRIORITY=1 -s SCHEDULER_DISK_QUEUE=scrapy.squeue.PickleFifoDiskQueue -s SCHEDULER_MEMORY_QUEUE=scrapy.squeue.FifoMemoryQueue

Sorry that's absurdly long. I should write a helper script.