Package: lineup
|Build Status| # LineUp - Distributed Pipeline Framework for PythonLineup is a redis-based`pipeline <http://en.wikipedia.org/wiki/Pipeline_(software)>`__framework that turns horizontal scalling seamless.It's currently providing parallelism through python threads and ispretty useful for writing systems where the workers make lots of networkI/O.It scales horizontally, so you can simply run more workers in as manymachines you want.Installing----------.. code:: basheasy_install curdlingcurd install lineupPhilosophy----------Lineup focus in:1. Simplicity: easy to create otherwise complex pipelines2. Easy-scale: just run more workers and you're good.Defining steps--------------Steps must always implement the method ``consume(self, instructions)``and always call ``self.produce()`` with it's portion of work.Example: a pipeline that downloads files~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~This is a copy of the `examples <examples>`__ folder... code:: pythonimport reimport codecsimport requestsfrom lineup import Stepfrom lineup import Pipelineclass Download(Step):def after_consume(self, instructions):self.log("Done downloading %s",instructions['url'],)def before_consume(self):self.log("The downloader is ready")def consume(self, instructions):url = instructions['url']method = instructions.get('method', 'get').lower()http_request = getattr(requests, method)response = http_request(url)instructions['download'] = {'content': response.content,'headers': dict(response.headers),'status_code': response.status_code,}self.produce(instructions)class Cache(Step):def after_consume(self, instructions):self.log("Done caching %s", instructions.keys())def before_consume(self):self.log("The cacher is also ready")def get_slug(self, url):url = re.sub(r'^https?://', '', url)url = re.sub(r'\W+', '-', url)return urldef consume(self, instructions):url = instructions['url']name = self.get_slug(url)with codecs.open(name, 'wb', 'utf-8') as fd:fd.write(instructions['download']['content'])fd.close()instructions['cached_at'] = {'filename': fd.name,}self.produce(instructions)# Defining pipelinesclass SimpleUrlDownloader(Pipeline):name = 'downloader'steps = [Download, Cache]Command line------------When running from the command line, lineup will recursively try toimport all python files given as ``--working-dir`` argument, whichdefaults to the relative equivalent to ``os.getcwd()``.In other words it will find your stuff automatically in dir you run thepipeline from, or from the ``--working-dir`` arg.Type ``lineup --help`` for more info.Listing available pipelines~~~~~~~~~~~~~~~~~~~~~~~~~~~LineUp scans the given working directory recursively for pipeline classdeclarations, here is how you can list all the pipelines that lineup canfind:.. code:: bashlineup list pipelinesRunning a pipeline in foreground~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.. code:: bash# Lineup will connect to this redisexport LINEUP_REDIS_URI='redis://0@localhost:6379'lineup downloader run --output=rpush@example-output.. figure:: example/run.png:alt: example/run.pngexample/run.pngFeeding a pipeline through command line~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.. code:: bashlineup downloader push {"url": "http://github.com/gabrielfalcao.keys"}.. figure:: example/push.png:alt: example/run.pngexample/run.pngFeeding a pipeline programatically~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.. code:: pythonfrom lineup import JSONRedisBackendfrom example.pipelines import SimpleUrlDownloaderpipeline = SimpleUrlDownloader(JSONRedisBackend)pipeline.input.put({"url": "http://github.com/gabrielfalcao.keys"})Stopping all running pipelines~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~This broadcasts a "stop" signal in the backend... code:: bashlineup downloader stopContributing------------Install Dependencies~~~~~~~~~~~~~~~~~~~~.. code:: bashcurd install -r development.txtRun tests~~~~~~~~~.. code:: bashmake test.. |Build Status| image:: https://travis-ci.org/weedlabs/lineup.png?branch=master:target: https://travis-ci.org/weedlabs/lineup