refactored auto path and pipelines, ready for the real scrape process