syntax fix
magic encoding
lots of small fixes to get the scrapers going
sidekiq can get very hungry
make sha1 unique