Package: processor
BadgesSimple rulesPython processor is a tool for creating chained pipelines for dataprocessing.
It have very few key concepts:Data objectAny python dict with two required fields:sourceandtype.SourceAn iterable sequence ofdata objectsor a function which returnsdata objects.
Seefull list of sourcesin the docs.OutputA function which accepts adata objectas input and could output another. Seefull list of outputsin the docs.
(or same)data objectas result.PredicatePipeline consists from sources outputs, butpredicatedecides whichdata objectshould be processed by whichoutput.Quick exampleHere is example of pipeline which reads IMAP folder and sends all emails to Slack chat:run_pipeline(sources.imap('imap.gmail.com''username','password''INBOX'),[prepare_email_for_slack,outputs.slack(SLACK_URL)])Here you construct a pipeline, which usessources.imapfor reading imap folder
“INBOX” ofusername@gmail.com. In more complex caseoutputs.fanoutcan be used for routing dataobjects to different processors andsources.mixcan
be used to merge items two or more sources into a one stream.Functionsprepare_email_to_slackandoutputs.slack(SLACK_URL)are processors. First one
is a simple function which accepts data object, returned by imap source and transforming
it to the data object which could be used by slack.output. We need that because slack
requires a different set of fields. Call tooutputs.slack(SLACK_URL)returns a
function which gets an object and send it to the specified Slack’s endpoint.It is just example, for working snippets, continue reading this documention ;-)NoteBy the way, did you know there is a Lisp dialect which runs on Python
virtual machine? It’s name is HyLang, and python processor is written in this
language.InstallationCreate a virtual environment with python3::virtualenv --python=python3 env
source env/bin/activateInstall required version of hylang (this step is necessary because Hy syntax is not
final yet and frequently changed by language maintainers)::pip install -U 'git+git://github.com/hylang/hy.git@a3bd90390cb37b46ae33ce3a73ee84a0feacce7d#egg=hy'If you are on OSX, then install lxml on OSX separately::STATIC_DEPS=true pip install lxmlIf you want to access IMAP over SSL on OSX, then you need to installopensslvia homebrew, and then installpyopenssllike this::brew install openssl
env LDFLAGS="-L$(brew --prefix openssl)/lib" \
    CFLAGS="-I$(brew --prefix openssl)/include" \
    pip install -U --force-reinstall pyopensslThen install theprocessor::pip install processorUsageNow create an executable python script, where you’ll place your pipline’s configuration.
For example, this simple code creates a process line which searches new results in Twitter
and outputs them to console. Of cause, you can output them not only to console, but also
post by email, to Slack chat or everywhere else if there is an output for it:#!env/bin/python3importosfromprocessorimportrun_pipeline,sources,outputsfromtwiggy_goodies.setupimportsetup_loggingfor_any_message=lambdamsg:Truedefprepare(tweet):return{'text':tweet['text'],'from':tweet['user']['screen_name']}setup_logging('twitter.log')run_pipeline(sources=[sources.twitter.search('My Company',consumer_key='***',consumer_secret='***',access_token='***',access_secret='***',)],rules=[(for_any_message,[prepare,outputs.debug()])])Running this code, will fetch new results for search by queryMy Companyand output them on the screen. Of course, you could use any otheroutput,
supported by theprocessor. Browse online documentation to find out
which sources and outputs are supported and for to configure them.Ideas for Sources and Outputsweb-hookendpoint(in progress).tailsource which reads file and outputs lines appeared in a file between invocations
or is able to emulatetail-fbehaviour. Python moduletailercould be used here.grepoutput – a filter to grep some fields using patterns. Withtailandgrepyou could build a pipeline which watch on a log and send errors by email or to the chat.xmppoutput.ircoutput.rss/atom feed reader.weathersource which tracks tomorrow’s weather forecast and outputs a message if it was
changed significantly, for example from “sunny” to “rainy”.githubsome integrations with github API?jiraor other task tracker of your choice?suggest your ideas!Documentationhttps://python-processor.readthedocs.org/DevelopmentTo run the all tests run:toxAuthorsAlexander Artemenko -http://dev.svetlyak.ruChangelog0.10.0 (2016-01-04)IMAP source was fixed to work with new IMAPClient’s API and
supportIMAPClient > 1.0.0.Datastorage was fixed to getfilenamefromPROCESSOR_DBenvironment variable in case if it was setup usingos.environ['PROCESSOR_DB']= 'some.db'after the imports.0.9.0 (2015-12-06)Code was fixed to work with HyLang froma3bd90390cb37b46ae33ce3a73ee84a0feacce7dcommit. Please, use this pinned version of HyLang andsubscribeon future
release notes to know when this requirement will change.0.8.0 (2015-11-16)Code was fixed to work with latest Hy, from GitHub.Addedtwitter.mentionssource, to read stream of mentions from the Twitter.Fixed a way how number of messages from IMAP folder is limited. Previously
limit was applied even when we already know an ID of the last seen message,
but now limit is ignored in this case and only applied when visiting the
folder first time.0.7.0 (2015-05-05)New output – XMPP was added and now processor is able
to notify Jabber users.0.6.0 (2015-05-01)The biggest change in this release is a new source –github.releases.
It is able to read all new releases in given repository and send them into
processing pipeline. This works as for public repositories, and for private
too.Read the docsfor futher details.Other changes are:Storage backend now saves JSON database nicely pretty printed for you could read and edit it in your favorite editor. This is Emacs, right?Twitter.search source now saves state after the tweet was processed. This way processor shouldn’t loose tweets if there was exception somewhere in processing pipeline.IMAP source was fixed and now is able to fetch emails from really big folders.0.5.0 (2015-04-15)Good news, everyone! New output was added -email.
Now Processor is able to notify you via email about any event.0.4.0 (2015-04-06)Functionrun_piplinewas simplified and now accepts only one source and one ouput.
To implement more complex pipelines, usesources.mixandoutputs.fanouthelpers.0.3.0 (2015-04-01)Added aweb.hooksource.Nowsourcecould be not only a iterable object, but any function which returns values.0.2.1 (2015-03-30)Fixed error inimport-or-errormacro, which prevented from using 3-party libraries.0.2.0 (2015-03-30)Most 3-party libraries are optional now. If you want to use
some extension which requires external library, it will issue
an error and callsys.exit(1)until you satisfy this
requirement.This should make life easier for thouse, who does not want
to userssoutput which requiresfeedgenwhich requireslxmlwhich is hard to build because it is C extension.0.1.0 (2015-03-18)First release on PyPI.