Package: newspeak
What it doesNewspeak is a feed aggregator with advanced features for keyword filtering
and link content extraction, implemented as a standaloone Django application.ArchitectureNewspeak performs the following tasks (in order):Fetch specified RSS/Atom feeds as per theFeedmodel (in parallel).Parses the feeds usingfeedparser.(Optionally) applies per-feed inclusive/exclusive keyword filters on the title and/or summary, based on theKeywordFiltermodel.(Optionally) extract summary data using an XPath expression from feed entry’s link URL, usinglxml.(Optionally) extract enclosure information using XPath expressions from the feed entry’s link URL, usinglxml.Store the resulting feed information locally in a database.Serve the aggregate of all the feed entries in a single RSS/Atom feed.The flow of feed data through the application is roughly as follows (given some example feeds and keyword filters):[Feed 1]-[Keyword filter 1]-[Keyword filter 2]-[XPath content extraction]-----------------------------`\
[Feed 2]--------------------[Keyword filter 3]-[XPath summary extraction]-[XPath content extraction ] -+--[Aggregate output feed]
[Feed 3]-[Keyword filter 3]-[Keyword filter 4]---------------------------------------------------------/InstallingGetting started with newspeak is really easy thanks to David Cramer’s awesomeloganfor making standalone Django apps.
Simply perform the following steps:Install such that you can easily code along:pip install -e \
  git+https://github.com/bitsoffreedom/newspeak.git#egg=newspeakIf you’re smart and like to keep your Python environment clean, do this
in aVirtualEnv.Initialize configuration in~/.newspeak/newspeak.conf.py:newspeak initPerform (optional) configuration by editing the settings file. Because
Newspeak is based on Django, allavailable Django settingscan be used. Furthermore, there are some Newspeak-specific settings:NEWSPEAK_THREADS: The number of (lightweight) threads used for crawling
feed data.NEWSPEAK_METADATA: Metadata used in the generated output feed.For a more thorough description and an example of these settings, please
have a look at the initial settings file generated in the previous step.(Optionally) Run the tests:newspeak test newspeakThis might take a while, so go fetch a cup of coffee. If something fails,
please supply the output of the commandnewspeak test newspeak –tracebackin anissueon GitHub.Create admin user and SQLite database (proper database is optional):newspeak syncdb --migrateStart the local webserver:newspeak run_gunicornOpenhttp://127.0.0.1:8000/admin/in your browser, add some feed. Only
the URL is required, the description and title will be fetched
automatically, as well as the first set of entries.(Optionally) Configure one or more keyword-based filters for your feed(s).Make sure the following command gets executed to update the feeds:newspeak update_feeds(Optionally, add-v <1|2|3>to get more feedback on the process.)Look at the pretty feeds: openhttp://127.0.0.1:8000/all/rss/orhttp://127.0.0.1:8000/all/atom/in your favorite feed reader. All input
feeds will be aggregated there.Alternatively, the original feeds, keywords and XPath expressions as used
by Bits of Freedom are contained in a fixture calledfeeds_bof.json. This
fixture can be loaded using:newspeak loaddata feeds_bofSetup aCronjobto automatically
update the feed data using thenewspeak update_feedscommand. For
example, a cron job updating the feeds every hour could look as follows:0 * * * *  <full_path_to_>/newspeak update_feedsUpgradingRun the PIP installation command again:pip install -e \
  git+https://github.com/bitsoffreedom/newspeak.git#egg=newspeak(Optionally) Run the tests:newspeak test newspeakApply any database migrations:newspeak migrate