added dynamic processor function
Merge pull request #2 from nextlanding/dynamic-processor

added dynamic processor function
added ProcessBasedUtils to run spiders in-process
Merge pull request #4 from nextlanding/processed-based-utils

added ProcessBasedUtils to run spiders in-process
added follow pagination type