change user agent default to mediawords bot
Add Data::Structure::Utile
added missing Data::Structure::Util module to cpanfile (required by the lib/MediaWords/Controller/Admin/Query.pm)
renamed doc/carton.txt to doc/carton.markdown so GitHub renders Markdown nicely
Merge branch 'master' of github.com:berkmancenter/mediacloud
Merge branch 'master' of github.com:berkmancenter/mediacloud
add more prefixes to remove from normalized urls
use MediaWords::Util::URL::normalize_url instead of local version
use MediaWords::Util::URL to normalize url in generate_medium_url_and_name_from_url; only reload new media in lookup_medium_by_url
Merge branch 'master' of github.com:berkmancenter/mediacloud
accept arguments
increase number of crawlers and extractors; change gearman jobs to use numprocs to make them stoppable
add default feed name
add query_encoded_json
validate <link> urls
optimize _get_dashboard_dates
update to use supervisor / gearman setup
enqueue gearman add_default_feed job for each added medium
remove debug print
fix bug in <link> scraping