Split up unit tests into multiple files according to module.
Handle 404 and missing DB rows.
Introduce config file in YAML format (initially for repo settings).
Return unicode string from WARP.get() and Repository.get_response().
Program spider by chaining builder methods.

Spider.at(url_p) returns a RegistrationBuilder object which has a method
for each of the possible instructions: do(action), process(proc) and
extract_links_with(ex=None, xpath=None, re=None).

Example:
(spider.at(r'/index/.*')
       .do(Sth())
       .do(More())
       .extract_links_with(xpath="//a[@rel='next']")
       .process(Parser())
Program spider through spider.instruct(at,**kwargs).

All kwargs are translated to RegistrationBuilder method invocations.
Since kwargs can only appear once, values can also be lists.

Example:
spider.instruct(
  at=r'/index/.*',
  do=[Sth(),More()],
  extract_links_with_xpath="//a[@rel='next']",
  process=Parser())
Encapsulate traversal logic, make it transparent to Spider.go().

- FifoTraversal.class
- DepthFirstTraversal.class
- PriorityQueueTraversal.class
New config options: INFO_DB, INFO_SIDECAR.

- usage of a DB now optional
- data can also be dumped into and retrieved from a sidecar file
Binary resource support for WARP and repository.

- new config options: CUSTOM_404, ACCESS_WEB
- simplify WARP.process_query() with dict as intermediate data structure
Simple GUI for spiders (based on QT).

- control: init, next, go, pause
- state: queue (total, finished, remaining)
- current URL
Log panel for the GUI.

Also some refactoring:
- move traversal classes to traversal.py
- move event-related classes events.py
GUI's spider actions report back by emitting QT signals.
Event bus: support for subscription by type.

On registration those types are wrapped in ClassSubscriptionWrapper
objects which derive from Subscription.class.
GUI: display extracted URLs.
GUI: QListView for log panel.
GUI: QListView for queue panel.
WARP: indicate download progress.

- new config options: DL_SIZE_INFO_THRESHOLD, DL_PROGRESS,
  DL_BLOCK_SIZE
- constructor accepts event_bus object that is used to fire
  events of type DLStatusEvent
Spider: GeneratorTraversal and download failure handling.

- accept traverser object in constructor
- handle download failures with retries
- Spider.url_range_generator()
- GeneratorTraversal.class
GUI: display processing results with view based on QtWebKit.
WARP HTTP proxy server & spider browser.

Proxy Server
- translates HTTP requests into WARP requests
- minimalist socket-based implementation
- parallel request handler threads

Spider Browser
- display live or downloaded page
- load page resources from internet or WARP proxy