Write sample meta in a separate method. Order sample fieldnames without relying on an explicit list in settings file.
Update README
Refactor echonest pipeline to populate echonest fields in process_item instead of a separate function.
Refactor S3 operations into a util class so they can be shared between pipelines and extensions. Move generation of license, profile, manifest, cue, etc. into item classes. Update settings file.
Add TODO file
Remove obsolete hierarchy doc
Specify field export order and modify DDL to match export order.
Write to S3 in lowercase.
Check for CC license and download data in pipeline. Take advantage of Twisted deferreds for pipeline requests.
Update TODO file
Refactor metadata exports. Remove generation of files such as license, cue, manifest, etc from items.py -- in the future we may want to move away from using items to export samples.
Update TODO
Remove LBYL SPECFilter in favor of EAFP Request errback
Update TODO
Decode crawl_start field
Refactor cli option parsing into it's own module that can be shared across spiders. Fix bugs in how options were used in soundclick spider. Minor formatting.
Specify processors are for input
Use dont_filter request param for artist page requests instead of DUPEFILTER_CLASS setting
Pull sensitive settings data from envars
Ignore .env file