change user agent default to mediawords bot
Add Data::Structure::Utile
Merge branch 'master' of github.com:berkmancenter/mediacloud
Merge branch 'master' of github.com:berkmancenter/mediacloud
add more prefixes to remove from normalized urls
use MediaWords::Util::URL::normalize_url instead of local version
use MediaWords::Util::URL to normalize url in generate_medium_url_and_name_from_url; only reload new media in lookup_medium_by_url
Merge branch 'master' of github.com:berkmancenter/mediacloud
accept arguments
increase number of crawlers and extractors; change gearman jobs to use numprocs to make them stoppable
add default feed name
add query_encoded_json
validate <link> urls
optimize _get_dashboard_dates
update to use supervisor / gearman setup
enqueue gearman add_default_feed job for each added medium
remove debug print
fix bug in <link> scraping
change to pass json directly through from solr query rather than decoding / re-encoding
added activities table to migration (got lost in merging)